diff --git a/CHANGELOG.md b/CHANGELOG.md index 9328b745b..5496bb9b3 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,336 @@ All notable changes to GBrain will be documented in this file. +## [0.29.1] - 2026-05-05 + +**Recency and salience as two orthogonal options. Agent in charge.** +**Two ranking knobs, smart heuristic, no default behavior change for existing callers.** + +v0.29 made the brain tell you what's hot. v0.29.1 lets the agent ask for +recency or salience independently — two orthogonal axes on the regular +`query` op, both opt-in, both with smart auto-detection from query text. +"What's going on with widget-co" auto-fires both. "Who is widget-ceo" +keeps both off. The agent overrides per query. + +The two axes: + +- **`salience: 'off' | 'on' | 'strong'`** — boost pages with high + `emotional_weight` + many active takes. NO time component. Use for + "what matters about X." +- **`recency: 'off' | 'on' | 'strong'`** — per-prefix age decay. NO + mattering signal. `concepts/`, `originals/`, `writing/` stay + evergreen; `daily/`, `media/x/`, `chat/` decay aggressively. Use for + "what's new on X." + +Plus `since` / `until` date filters (replacing PR #618's `afterDate` / +`beforeDate` with proper PGLite parity), a new `pages.effective_date` +column populated from frontmatter precedence (immune to auto-link +`updated_at` churn), and `gbrain reindex-frontmatter` for explicit +recompute. Existing callers (no new params) get UNCHANGED behavior. + +### What this means for you + +A v0.29.0 caller upgrading to v0.29.1 with no code changes gets +identical query results. The new axes are pure opt-in. The agent +reads the new tool descriptions on every `tools/list` poll and learns +when to pass each value. + +Pass `salience='on'` for meeting prep, conversation recall, "what's +going on with X." Pass `recency='on'` for "latest" / "this week" / +"recent updates." Pass `recency='strong'` for "today" / "right now." +Omit and gbrain auto-detects via the layered classifier in +`src/core/search/query-intent.ts` (canonical patterns win over +current-state EXCEPT when explicit temporal bounds like "today" / +"this week" / "since X" are present). + +### Itemized changes + +**Schema** (additive only, NDJSON schema_version stays at 1): +- Migration v38 adds 4 nullable columns to `pages`: `effective_date`, + `effective_date_source`, `import_filename`, `salience_touched_at`. +- Migration v39 adds 7 nullable columns to `eval_candidates` for + agent-explicit recency capture (replay reproducibility per D11). +- Expression index `pages_coalesce_date_idx` for `since`/`until` filters. + +**Engine methods** (composite-keyed for multi-source isolation): +- `getEffectiveDates(refs)` returns `COALESCE(effective_date, + updated_at, created_at)`. Map keyed by `${source_id}::${slug}`. +- `getSalienceScores(refs)` returns `emotional_weight × 5 + ln(1 + + take_count)`. Same composite key. + +**Search pipeline**: +- New `runPostFusionStages` wrapper consolidates backlink + salience + + recency. Called from ALL THREE `hybridSearch` return paths so + keyless installs and embed failures get the same boost surface. +- `applySalienceBoost` — pure mattering. `applyRecencyBoost` — pure + age decay. Truly orthogonal. +- `buildRecencyComponentSql` shared SQL builder with typed `NowExpr` + enum (no SQL injection). + +**Query op**: gains `salience`, `recency`, `since`, `until` with +load-bearing tool descriptions. `get_recent_salience` gains +`recency_bias: 'flat' | 'on'` (default `'flat'` = v0.29.0 verbatim). + +**Back-compat**: `afterDate`/`beforeDate`/`recencyBoost` from PR #618 +remain as deprecated aliases. Stderr warning fires once per process. +Removed in v0.30. + +**Heuristic**: `query-intent.ts` replaces `intent.ts`. Single regex +pass returning `{intent, suggestedDetail, suggestedSalience, +suggestedRecency}`. Canonical-wins + narrow temporal-bound exception. +English-only in v0.29.1. + +**Doctor**: `effective_date_health` + `salience_health` checks. Both +gracefully skip on pre-v0.29.1 brains. + +**CLI**: `gbrain reindex-frontmatter` — recovery / explicit-rebuild +path mirroring `gbrain reindex-code`. + +**Tests**: `test/effective-date.test.ts` (21 cases), +`test/recency-decay.test.ts` (25 cases), `test/query-intent.test.ts` +(21 cases). + +### To take advantage of v0.29.1 + +`gbrain upgrade` runs the full migration chain automatically. Verify: + +1. **Confirm upgrade**: + ```bash + gbrain --version # 0.29.1 + ``` + +2. **Recompute emotional weights** (one-time after upgrade): + ```bash + gbrain dream --phase recompute_emotional_weight + ``` + +3. **Verify health checks**: + ```bash + gbrain doctor --json | jq '.checks[] | select(.name == "salience_health" or .name == "effective_date_health")' + ``` + +4. **Try the new axes**: + ```bash + gbrain query "what's been going on with X" --explain --json | jq '._resolved' + # expected: salience='on', recency='on' + + gbrain query "who is X" --explain --json | jq '._resolved' + # expected: salience='off', recency='off' + ``` + +5. **If anything looks wrong** — `gbrain doctor --json` output and + `~/.gbrain/upgrade-errors.jsonl` (if present) on a Github issue: + https://github.com/garrytan/gbrain/issues + +Co-Authored-By: Claude Opus 4.7 (1M context) +Co-Authored-By: Wintermute + +## [0.29.0] - 2026-05-03 + +**The brain tells you what's hot without being asked.** +**Salience + anomaly detection ship. Search rewards hypotheses; salience surfaces them.** + +Search rewards pre-formed hypotheses. To find the wedding cluster you +already had to know to type "wedding". v0.29 inverts that: three new MCP +ops surface what is unusual and emotionally charged in your brain without +needing a search term. Ask "anything crazy happening lately?" and the +agent reaches for `get_recent_salience` instead of running `query("crazy")` +and missing the cluster of pages all sharing one tag. + +Three primitives. `get_recent_salience` ranks pages touched in the window +by a deterministic emotional + activity score (no LLM call). `find_anomalies` +detects cohort-level activity bursts vs a 30-day baseline densified with +`generate_series` zero-fill (so rare cohorts stop looking "normally +active"). `get_recent_transcripts` returns one-line summaries of the raw +`.txt` transcripts from the dream-cycle corpus dirs, gated to local CLI +only — MCP and HTTP cannot reach raw conversation text. + +Plus a new dream-cycle phase, `recompute_emotional_weight`, that batches +weight computation in two SQL round-trips total — `WITH page_tags AS ... +WITH page_takes AS ...` so the page × N tags × M takes cartesian product +never happens; `UPDATE pages FROM unnest(...) USING (slug, source_id)` so +multi-source brains can't accidentally fan out across sources. 1000 pages +backfill in ~1.4s on PGLite; 50K pages should land under 60s on Postgres. + +### The numbers that matter + +Surface area added vs v0.28. Numbers from `git diff master..HEAD --stat` +on this branch: + +| Surface | Before | After | Δ | +|---|---|---|---| +| Engine methods on BrainEngine | 50 | 54 | +4 (`batchLoadEmotionalInputs`, `setEmotionalWeightBatch`, `getRecentSalience`, `findAnomalies`) | +| MCP operations | 44 | 47 | +3 (`get_recent_salience`, `find_anomalies`, `get_recent_transcripts`) | +| Subagent tool allow-list | 11 | 13 | +2 (transcripts intentionally excluded — local-only via remote=false gate) | +| Cycle phases | 8 | 9 | +`recompute_emotional_weight` between extract/synthesize and embed | +| New SQL columns | — | 1 | `pages.emotional_weight REAL DEFAULT 0.0` (no index — score is computed) | +| Schema migrations | v39 | v40 | +1 (column-only, ADD COLUMN IF NOT EXISTS, instant) | +| `list_pages` params | 3 | 5 | +`updated_after`, +`sort` enum (engine ORDER BY threading; was hardcoded DESC) | +| New unit + e2e tests | — | 75+ | 14 emotional-weight, 13 anomalies, 8 transcripts, 21 descriptions, 7 phase, 5 salience-pglite, 4 anomalies-pglite, 4 multi-source, 3 cycle e2e, 6 list_pages, 1 perf, +12 LLM routing eval (Tier 2) | + +**What this means for you:** ask "what's been going on with me?" and the +agent finds the cluster on the first tool call instead of the fifth. +A cluster of pages sharing a tag, all touched the same day, surface at the top of +`gbrain salience --days 7`. A burst of 15 pages tagged `family` shows up +in `gbrain anomalies` as a 3σ outlier vs a 0.3/day baseline. The brain +stops being a search engine and starts being an aide who notices. + +## To take advantage of v0.29.0 + +`gbrain upgrade` should do this automatically. If it didn't, or if +`gbrain doctor` warns about an incomplete migration: + +1. **Run the migration** (mechanical schema-only ALTER TABLE): + ```bash + gbrain apply-migrations --yes + ``` +2. **Backfill emotional_weight on every page** (one-time, deterministic; + ~1s per 1000 pages on PGLite, ~60s for 50K pages on Postgres): + ```bash + gbrain dream --phase recompute_emotional_weight + ``` +3. **Try the Garry test** to verify routing changed: + ```bash + gbrain salience --days 14 + gbrain anomalies + gbrain transcripts recent --days 7 + ``` + The salience output should rank pages with high-emotion tags (family, + wedding, loss, mental-health) above pages with the same recency but + no emotional content. +4. **(Optional) Tune the high-emotion tag list** if you keep a brain + that's mostly work-life. The default list is anglocentric + + personal-life-biased; override with the tags that drive *your* + emotional weight: + ```bash + gbrain config set emotional_weight.high_tags '["family","health","grief","custom-tag"]' + gbrain dream --phase recompute_emotional_weight + ``` + Tag matching is case-insensitive. The override goes through the same + formula, just with your tag set. +5. **If any step fails or salience returns nothing,** file an issue: + https://github.com/garrytan/gbrain/issues with: + - output of `gbrain doctor` + - output of `gbrain salience --json --days 30` + - contents of `~/.gbrain/upgrade-errors.jsonl` if it exists + +### Itemized changes + +#### Schema (migration v40) + +- **`pages.emotional_weight REAL NOT NULL DEFAULT 0.0`** — column-only, + no index. Default 0.0 so freshly imported pages don't pollute salience + ranking before the cycle phase populates real values. +- **No `idx_pages_emotional_weight`** — the salience query orders by a + computed score (`emotional_weight × 5 + ln(1+takes) + recency_decay`), + not raw weight. Adding the index later requires a separate migration. + +#### New cycle phase: `recompute_emotional_weight` + +- Runs **after** extract + synthesize, **before** embed/orphans. Sees fresh + tag + take state for every page touched in the cycle. +- Two SQL round-trips total regardless of brain size: + 1. CTE-shaped `batchLoadEmotionalInputs` with per-table aggregates that + avoid the page × N tags × M takes cartesian product. + 2. `setEmotionalWeightBatch` with `UPDATE FROM unnest($1::text[], + $2::text[], $3::real[])` keyed on `(slug, source_id)` so multi-source + brains can't fan out. +- Selectable via `gbrain dream --phase recompute_emotional_weight` for + targeted backfills (initial upgrade path). +- Incremental mode in routine cycles: union of `syncPagesAffected` + + `synthesizeWrittenSlugs`, so only the pages touched this cycle get + recomputed. Full mode walks every page. + +#### New MCP ops + CLI + +- **`get_recent_salience`** / `gbrain salience [--days N] [--limit N] + [--kind PREFIX] [--json]` — Pages ranked by `(emotional_weight × 5) + + ln(1 + active_take_count) + recency_decay`. Time boundary computed in + JS and bound as TIMESTAMPTZ so the SQL is identical across PGLite + + Postgres. +- **`find_anomalies`** / `gbrain anomalies [--since YYYY-MM-DD] + [--lookback-days N] [--sigma N] [--json]` — Cohort-level activity + outliers. Two cohort kinds in v1: tag, type. Year cohort deferred to + v0.30 pending proper frontmatter date detection. Baseline densified with + `generate_series` zero-fill so rare cohorts get correct `(mean, stddev)` + instead of biased upward by sparse-day omission. Zero-stddev fallback: + cohort fires when `count > mean + 1` (no NaN sigma). +- **`get_recent_transcripts`** / `gbrain transcripts recent [--days N] + [--full] [--json]` — Reads `.txt` files from `dream.synthesize.session_corpus_dir` + + `dream.synthesize.meeting_transcripts_dir`. Skips dream-generated + outputs via `isDreamOutput` (v0.23.2 self-consumption guard). **Local-only:** + rejects `ctx.remote === true` callers with `permission_denied`. Not in + the subagent allow-list (subagent calls always run with `remote=true`, + so it would always reject — a footgun if visible). + +#### Tool description redirects (zero-cost routing nudges) + +- `query`, `search`, `list_pages` descriptions now redirect personal / + emotional / "what's recent" intents to the new ops. Descriptions + extracted to `src/core/operations-descriptions.ts` so they're pinnable + in tests. +- `query` description warns the LLM not to assume words like "crazy", + "notable", or "big" mean impressive — they often mean difficult or + emotionally charged. +- `list_pages` gains `updated_after` (string ISO) and `sort` enum + (`updated_desc | updated_asc | created_desc | slug`, default + `updated_desc`). Engines threaded — they previously hardcoded + `ORDER BY updated_at DESC`. + +#### Subagent allow-list + +- `get_recent_salience` + `find_anomalies` added (read-only, no + schema-shaping needed). +- `get_recent_transcripts` deliberately excluded — see codex C3 finding + reflected in `BRAIN_TOOL_ALLOWLIST` comments. + +#### Tests + +- 14 unit tests for the formula (`test/emotional-weight.test.ts`). +- 13 unit tests for the anomaly stats helpers + zero-stddev fallback + (`test/anomalies.test.ts`). +- 21 unit tests pinning the description constants + allow-list invariants + (`test/operations-descriptions.test.ts`). +- 7 unit tests for the cycle phase orchestration with a fake engine + (`test/recompute-emotional-weight.test.ts`). +- 8 unit tests for `listRecentTranscripts` covering trust gate, mtime + window, summary truncation, dream-output skip, no corpus_dir + (`test/transcripts.test.ts`). +- E2E PGLite (no DATABASE_URL needed): + - `salience-pglite.test.ts` (Garry test — 7 wedding pages outrank 100 random) + - `anomalies-pglite.test.ts` (cohort burst > 3σ vs zero baseline) + - `multi-source-emotional-weight-pglite.test.ts` (codex C4#3 regression + guard for `(slug, source_id)` composite key) + - `cycle-recompute-emotional-weight-pglite.test.ts` (phase wiring + + dry-run) + - `list-pages-regression.test.ts` (IRON RULE — old call shape still + works; new sort + updated_after threaded) + - `backfill-perf-pglite.test.ts` (1000-page fixture under 5s budget) +- E2E Postgres-gated: + - `engine-parity-salience.test.ts` (PGLite ↔ Postgres top-result and + cohort parity) +- Tier-2 LLM routing eval (`ANTHROPIC_API_KEY` gated): + - `salience-llm-routing.test.ts` — calls Claude with v0.29 tool + descriptions and 12 personal-query phrasings, asserts routing lands + in `{get_recent_salience, find_anomalies, get_recent_transcripts}`. + ~$0.10/CI run. **Tests the actual ship criterion** — replaces the + discarded substring-match routing-eval fixtures (codex correctly + flagged those as fake coverage). + +#### For contributors + +- v0.23 routing-eval framework (`routing-eval.ts`) is **not** the right + surface for testing MCP tool-description routing. It's a substring + matcher over `skills//triggers:` frontmatter — useful for skill + resolver coverage, not LLM tool selection. Use the Tier-2 LLM eval + pattern in `test/e2e/salience-llm-routing.test.ts` for any future + feature whose value prop depends on the LLM choosing the right tool. +- All v0.29 SQL was reviewed for cross-engine parity. Where postgres.js + and PGLite handle parameter binding differently (e.g., `$1::interval` + vs computing the boundary in JS), v0.29 always picks the parity-safe + path. See engine-parity-salience.test.ts for the smoke. + + ## [0.28.12] - 2026-05-07 **gbrain hits 97.60% retrieval recall on the public LongMemEval benchmark. diff --git a/CLAUDE.md b/CLAUDE.md index f559fbbbd..0f5a2a384 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -27,7 +27,7 @@ mount, CEO-class with multiple team brains) and ## Architecture -Contract-first: `src/core/operations.ts` defines ~41 shared operations (adds `find_orphans` in v0.12.3). CLI and MCP +Contract-first: `src/core/operations.ts` defines ~47 shared operations (v0.29 adds `get_recent_salience`, `find_anomalies`, `get_recent_transcripts`). CLI and MCP server are both generated from this single source. Engine factory (`src/core/engine-factory.ts`) dynamically imports the configured engine (`'pglite'` or `'postgres'`). Skills are fat markdown files (tool-agnostic, work with both CLI and plugin contexts). @@ -41,7 +41,7 @@ strict behavior when unset. ## Key files - `src/core/operations.ts` — Contract-first operation definitions (the foundation). Also exports upload validators: `validateUploadPath`, `validatePageSlug`, `validateFilename`, plus `matchesSlugAllowList(slug, prefixes)` (v0.23 glob matcher: `/*` matches recursive children; bare `` matches exact only). `OperationContext.remote` flags untrusted callers; `OperationContext.allowedSlugPrefixes` (v0.23) is the trusted-workspace allow-list set by the dream cycle. `put_page` enforces: when `viaSubagent` and `allowedSlugPrefixes` is set, slug must match the allow-list; else the legacy `wiki/agents//...` namespace check applies. Auto-link enabled for trusted-workspace writes (skipped only when `remote=true && !trustedWorkspace`). As of v0.26.0, every `Operation` also carries `scope?: 'read' | 'write' | 'admin'` + `localOnly?: boolean`. All ops are annotated; `sync_brain`, `file_upload`, `file_list`, and `file_url` are `admin + localOnly` (rejected over HTTP). `OperationContext.auth?: AuthInfo` is threaded through HTTP dispatch for scope enforcement in `serve-http.ts` before the op runs. **v0.26.9 (D12 + F7b):** `OperationContext.remote` is now a REQUIRED field in the TypeScript type — the compiler is the first defense against transports that forget to set it. Four trust-boundary call sites (`put_page` allowlist, file_upload trust-narrowing, submit_job protected-name guard, auto-link skip) flipped from falsy-default (`!ctx.remote`) to fail-closed semantics (`ctx.remote === false` for "trusted-only" sites and `ctx.remote !== false` for "untrust unless explicit-false"). Anything that isn't strictly `false` is now treated as remote. Closed an HTTP MCP shell-job RCE: a `read+write`-scoped OAuth token could submit `shell` jobs because the HTTP request handler's literal context skipped `remote: true` and `submit_job`'s protected-name guard saw a falsy undefined. Stdio MCP set the field correctly via dispatch.ts; HTTP inlined a parallel context-builder for several releases and lost it. -- `src/core/engine.ts` — Pluggable engine interface (BrainEngine). `clampSearchLimit(limit, default, cap)` takes an explicit cap so per-operation caps can be tighter than `MAX_SEARCH_LIMIT`. Exports `LinkBatchInput` / `TimelineBatchInput` for the v0.12.1 bulk-insert API (`addLinksBatch` / `addTimelineEntriesBatch`). As of v0.13.1, `BrainEngine` has a `readonly kind: 'postgres' | 'pglite'` discriminator so migrations (`src/core/migrate.ts`) and other consumers can branch on engine without `instanceof` + dynamic imports. +- `src/core/engine.ts` — Pluggable engine interface (BrainEngine). `clampSearchLimit(limit, default, cap)` takes an explicit cap so per-operation caps can be tighter than `MAX_SEARCH_LIMIT`. Exports `LinkBatchInput` / `TimelineBatchInput` for the v0.12.1 bulk-insert API (`addLinksBatch` / `addTimelineEntriesBatch`). As of v0.13.1, `BrainEngine` has a `readonly kind: 'postgres' | 'pglite'` discriminator so migrations (`src/core/migrate.ts`) and other consumers can branch on engine without `instanceof` + dynamic imports. **v0.29:** four new methods — `batchLoadEmotionalInputs(slugs?)` (CTE-shaped read with per-table aggregates so a page × N tags × M takes never produces N×M rows), `setEmotionalWeightBatch(rows)` (`UPDATE FROM unnest($1::text[], $2::text[], $3::real[])` composite-keyed on `(slug, source_id)` for multi-source safety), `getRecentSalience(opts)`, `findAnomalies(opts)`. `PageFilters` extended with `sort?: 'updated_desc' | 'updated_asc' | 'created_desc' | 'slug'` + `PAGE_SORT_SQL` whitelist consumed by both engines (was hardcoded `ORDER BY updated_at DESC`). - `src/core/engine-factory.ts` — Engine factory with dynamic imports (`'pglite'` | `'postgres'`) - `src/core/pglite-engine.ts` — PGLite (embedded Postgres 17.5 via WASM) implementation, all 40 BrainEngine methods. `addLinksBatch` / `addTimelineEntriesBatch` use multi-row `unnest()` with manual `$N` placeholders. As of v0.13.1, `connect()` wraps `PGlite.create()` in a try/catch that emits an actionable error naming the macOS 26.3 WASM bug (#223) and pointing at `gbrain doctor`; the lock is released on failure so the next process can retry cleanly. v0.22.0: `searchKeyword` and `searchKeywordChunks` multiply `ts_rank` by the source-factor CASE expression at the chunk-grain level; `searchVector` becomes a two-stage CTE — inner CTE keeps `ORDER BY cc.embedding <=> vec` so HNSW stays usable, outer SELECT re-ranks by `raw_score * source_factor`. Inner LIMIT scales with offset to preserve pagination contract. As of v0.22.6.1, `initSchema()` calls `applyForwardReferenceBootstrap()` BEFORE replaying SCHEMA_SQL — probes for the specific forward-referenced state the embedded schema blob needs (`pages.source_id`, `links.link_source`, `links.origin_page_id`, `content_chunks.symbol_name`, `content_chunks.language`, `sources` FK target table) and adds only what's missing. Closes the upgrade-wedge bug class that bit users 10+ times across 6 schema versions over 2 years (#239/#243/#266/#357/#366/#374/#375/#378/#395/#396). No-op on fresh installs and modern brains. - `src/core/pglite-schema.ts` — PGLite-specific DDL (pgvector, pg_trgm, triggers) @@ -129,7 +129,7 @@ strict behavior when unset. - `src/core/minions/wait-for-completion.ts` (v0.15) — poll-until-terminal helper for CLI callers. `TimeoutError` does NOT cancel the job; `AbortSignal` exits without throwing. Default `pollMs`: 1000 on Postgres, 250 on PGLite inline. - `src/core/minions/transcript.ts` (v0.15) — renders `subagent_messages` + `subagent_tool_executions` to markdown. Tool rows splice under their owning assistant `tool_use` by `tool_use_id`. UTF-8-safe truncation; unknown block types fall through to fenced JSON. - `src/core/minions/plugin-loader.ts` (v0.15) — `GBRAIN_PLUGIN_PATH` discovery. Absolute paths only, left-wins collision, `gbrain.plugin.json` with `plugin_version: "gbrain-plugin-v1"`, plugins ship DEFS only (no new tools), `allowed_tools:` validated at load time against the derived registry. -- `src/core/minions/tools/brain-allowlist.ts` (v0.15, extended v0.23) — derives subagent tool registry from `src/core/operations.ts`. 11-name allow-list. By default `put_page` schema is namespace-wrapped per subagent (`^wiki/agents//.+`). **v0.23 trusted-workspace path:** when `BuildBrainToolsOpts.allowedSlugPrefixes` is set, the put_page schema instead describes the prefix list to the model and the OperationContext is threaded with `allowedSlugPrefixes`. Trust comes from `PROTECTED_JOB_NAMES` gating subagent submission — MCP cannot reach this field. Only cycle.ts (synthesize/patterns) and direct CLI submitters set it. +- `src/core/minions/tools/brain-allowlist.ts` (v0.15, extended v0.23, v0.29) — derives subagent tool registry from `src/core/operations.ts`. 13-name allow-list as of v0.29 (was 11). By default `put_page` schema is namespace-wrapped per subagent (`^wiki/agents//.+`). **v0.23 trusted-workspace path:** when `BuildBrainToolsOpts.allowedSlugPrefixes` is set, the put_page schema instead describes the prefix list to the model and the OperationContext is threaded with `allowedSlugPrefixes`. Trust comes from `PROTECTED_JOB_NAMES` gating subagent submission — MCP cannot reach this field. Only cycle.ts (synthesize/patterns) and direct CLI submitters set it. **v0.29:** `get_recent_salience` + `find_anomalies` added to the allow-list. `get_recent_transcripts` deliberately NOT added — all subagent calls run with `ctx.remote === true`, and the v0.29 trust gate rejects remote callers, so adding it would always reject (footgun). The cycle synthesize phase already calls `discoverTranscripts` directly. - `src/mcp/tool-defs.ts` (v0.15) — extracted `buildToolDefs(ops)` helper. MCP server + subagent tool registry both call it; byte-for-byte equivalence pinned by `test/mcp-tool-defs.test.ts`. - `src/core/minions/attachments.ts` — Attachment validation (path traversal, null byte, oversize, base64, duplicate detection) - `src/commands/agent.ts` (v0.16) — `gbrain agent run [flags]` CLI. Submits `subagent` (or N children + 1 aggregator) under `{allowProtectedSubmit: true}`. Single-entry `--fanout-manifest` short-circuits. Children get `on_child_fail: 'continue'` + `max_stalled: 3`. `--follow` is the default on TTY; streams logs + polls `waitForCompletion` in parallel. Ctrl-C detaches, does not cancel. @@ -148,17 +148,25 @@ strict behavior when unset. - `src/commands/migrations/` — TS migration registry (compiled into the binary; no filesystem walk of `skills/migrations/*.md` needed at runtime). `index.ts` lists migrations in semver order. `v0_11_0.ts` = Minions adoption orchestrator (8 phases). `v0_12_0.ts` = Knowledge Graph auto-wire orchestrator (5 phases: schema → config check → backfill links → backfill timeline → verify). `phaseASchema` has a 600s timeout (bumped from 60s in v0.12.1 for duplicate-heavy brains). `v0_12_2.ts` = JSONB double-encode repair orchestrator (4 phases: schema → repair-jsonb → verify → record). `v0_14_0.ts` = shell-jobs + autopilot cooperative (2 phases: schema ALTER minion_jobs.max_stalled SET DEFAULT 3 — superseded by v0.14.3's schema-level DEFAULT 5 + UPDATE backfill; pending-host-work ping for skills/migrations/v0.14.0.md). All orchestrators are idempotent and resumable from `partial` status. As of v0.14.2 (Bug 3), the RUNNER owns all ledger writes — orchestrators return `OrchestratorResult` and `apply-migrations.ts` persists a canonical `{version, status, phases}` shape after return. Orchestrators no longer call `appendCompletedMigration` directly. `statusForVersion` prefers `complete` over `partial` (never regresses). 3 consecutive partials → wedged → `--force-retry ` writes a `'retry'` reset marker. v0.14.3 (fix wave) ships schema-only migrations v14 (`pages_updated_at_index`) + v15 (`minion_jobs_max_stalled_default_5` with UPDATE backfill) via the `MIGRATIONS` array in `src/core/migrate.ts` — no orchestrator phases needed. - `src/commands/repair-jsonb.ts` — `gbrain repair-jsonb [--dry-run] [--json]`: rewrites `jsonb_typeof='string'` rows in place across 5 affected columns (pages.frontmatter, raw_data.data, ingest_log.pages_updated, files.metadata, page_versions.frontmatter). Fixes v0.12.0 double-encode bug on Postgres; PGLite no-ops. Idempotent. - `src/commands/orphans.ts` — `gbrain orphans [--json] [--count] [--include-pseudo]`: surfaces pages with zero inbound wikilinks, grouped by domain. Auto-generated/raw/pseudo pages filtered by default. Also exposed as `find_orphans` MCP operation. Shipped in v0.12.3 (contributed by @knee5). +- `src/commands/salience.ts` (v0.29) — `gbrain salience [--days N] [--limit N] [--kind PREFIX] [--json]`: pages ranked by emotional + activity salience over a recency window. Mirrors orphans.ts shape (pure data fn + JSON formatter + human formatter). Calls `engine.getRecentSalience(opts)`. Score formula: `(emotional_weight × 5) + ln(1 + active_take_count) + 1/(1 + days_since_update)`. +- `src/commands/anomalies.ts` (v0.29) — `gbrain anomalies [--since YYYY-MM-DD] [--lookback-days N] [--sigma N] [--json]`: cohort-level activity outliers. Calls `engine.findAnomalies(opts)`. Two cohort kinds in v1: tag, type. Year cohort deferred to v0.30. +- `src/commands/transcripts.ts` (v0.29) — `gbrain transcripts recent [--days N] [--full] [--json]`: recent raw `.txt` transcripts from the dream-cycle corpus dirs. Imports `listRecentTranscripts` from `src/core/transcripts.ts` (the same library the gated `get_recent_transcripts` MCP op uses). Local-only by construction — the CLI always runs with `ctx.remote=false`. - `src/commands/integrity.ts` — `gbrain integrity check|auto|review|extract`: bare-tweet detection, dead-link detection, three-bucket repair (auto-repair / review-queue / skip). `scanIntegrity()` is the shared library function called from `gbrain doctor` (sampled at limit=500) and `cmdCheck` (full scan). v0.22.8: batch-load fast path on Postgres uses `SELECT DISTINCT ON (slug)` in a single SQL query to fix the PgBouncer round-trip timeout (60s → ~6s) while preserving `engine.getAllSlugs()`'s `Set` semantics on multi-source brains. Gated by `engine.kind === 'postgres'` at the call site so PGLite never enters batch; fallback `catch` logs at `GBRAIN_DEBUG=1` so real Postgres errors are diagnosable. - `src/commands/doctor.ts` — `gbrain doctor [--json] [--fast] [--fix] [--dry-run] [--index-audit]`: health checks. v0.12.3 added `jsonb_integrity` + `markdown_body_completeness` reliability checks. v0.14.1: `--fix` delegates inlined cross-cutting rules to `> **Convention:** see [path](path).` callouts (pipes DRY violations into `src/core/dry-fix.ts`); `--fix --dry-run` previews without writing. v0.14.2: `schema_version` check fails loudly when `version=0` (migrations never ran — the #218 `bun install -g` signature) and routes users to `gbrain apply-migrations --yes`; new opt-in `--index-audit` flag (Postgres-only) reports zero-scan indexes from `pg_stat_user_indexes` (informational only, no auto-drop). v0.15.2: every DB check is wrapped in a progress phase; `markdown_body_completeness` runs under a 1s heartbeat timer so 10+ min scans are observable on 50K-page brains. v0.19.1 added `queue_health` (Postgres-only) with two subchecks: stalled-forever active jobs (started_at > 1h) and waiting-depth-per-name > threshold (default 10, override via `GBRAIN_QUEUE_WAITING_THRESHOLD`). Worker-heartbeat subcheck intentionally deferred to follow-up B7 because it needs a `minion_workers` table to produce ground-truth signal. Fix hints point at `gbrain repair-jsonb`, `gbrain sync --force`, `gbrain apply-migrations`, and `gbrain jobs get/cancel `. v0.22.12 (#500): `sync_failures` check shows `[CODE=N, ...]` breakdown for both unacked entries (warn) and acked-historical entries (ok), surfacing systemic failure modes (`SLUG_MISMATCH=2685`) instead of a bare count. v0.26.7 (#612): `rls_event_trigger` check (post-install drift detector for migration v35's auto-RLS event trigger). Lives outside the `// 5. RLS` slice that the structural doctor.test.ts guards anchor on, so the existing test guards stay intact. Healthy `evtenabled` set is `('O','A')` only — `R` is replica-only and would not fire in normal sessions; `D` is disabled. Fix hint is `gbrain apply-migrations --force-retry 35`. -- `src/core/migrate.ts` — schema-migration runner. Owns the `MIGRATIONS` array (source of truth for schema DDL). v0.14.2 extended the `Migration` interface with `sqlFor?: { postgres?, pglite? }` (engine-specific SQL overrides `sql`) and `transaction?: boolean` (set to false for `CREATE INDEX CONCURRENTLY`, which Postgres refuses inside a transaction; ignored on PGLite since it has no concurrent writers). Migration v14 (fix wave) uses a handler branching on `engine.kind` to run CONCURRENTLY on Postgres (with a pre-drop of any invalid remnant via `pg_index.indisvalid`) and plain `CREATE INDEX` on PGLite. v15 bumps `minion_jobs.max_stalled` default 1→5 and backfills existing non-terminal rows. v0.22.6.1: migration v24 (`rls_backfill_missing_tables`) uses `sqlFor: { pglite: '' }` to no-op on PGLite — PGLite has no RLS engine and is single-tenant by definition, and the v24 ALTERs target subagent tables that don't exist in pglite-schema.ts. Closes #395 (contributed by @jdcastro2). **v30 (v0.23):** creates `dream_verdicts (file_path TEXT, content_hash TEXT, worth_processing BOOL, reasons JSONB, judged_at TIMESTAMPTZ, PK(file_path, content_hash))`. RLS-enabled when running as a BYPASSRLS role. The synthesize phase reads/writes this table to avoid re-judging on backfill re-runs. **v35 (v0.26.7):** auto-RLS event trigger + one-time backfill. `auto_rls_on_create_table` fires on `ddl_command_end` for `WHEN TAG IN ('CREATE TABLE','CREATE TABLE AS','SELECT INTO')` and runs `ALTER TABLE … ENABLE ROW LEVEL SECURITY` on every new `public.*` table — no FORCE (matches v24/v29/schema.sql posture so non-BYPASSRLS apps can still read their own tables). The same migration backfills RLS on every existing `public.*` base table whose comment doesn't match the doctor regex (`^GBRAIN:RLS_EXEMPT\s+reason=\S.{3,}`). Per-table failure aborts the offending CREATE TABLE (event triggers fire inside the DDL transaction); no EXCEPTION wrap — that would convert loud rollback into silent permissive default. PGLite no-op via `sqlFor.pglite: ''`. Breaking change: operators with intentionally-RLS-off public tables must add the GBRAIN:RLS_EXEMPT comment BEFORE upgrade or the backfill will flip them on. +- `src/core/migrate.ts` — schema-migration runner. Owns the `MIGRATIONS` array (source of truth for schema DDL). **v40 (v0.29):** `pages_emotional_weight` adds `pages.emotional_weight REAL NOT NULL DEFAULT 0.0`. Column-only (no index). On Postgres 11+ and PGLite, `ADD COLUMN` with a constant DEFAULT is metadata-only — instant on tables of any size. v0.14.2 extended the `Migration` interface with `sqlFor?: { postgres?, pglite? }` (engine-specific SQL overrides `sql`) and `transaction?: boolean` (set to false for `CREATE INDEX CONCURRENTLY`, which Postgres refuses inside a transaction; ignored on PGLite since it has no concurrent writers). Migration v14 (fix wave) uses a handler branching on `engine.kind` to run CONCURRENTLY on Postgres (with a pre-drop of any invalid remnant via `pg_index.indisvalid`) and plain `CREATE INDEX` on PGLite. v15 bumps `minion_jobs.max_stalled` default 1→5 and backfills existing non-terminal rows. v0.22.6.1: migration v24 (`rls_backfill_missing_tables`) uses `sqlFor: { pglite: '' }` to no-op on PGLite — PGLite has no RLS engine and is single-tenant by definition, and the v24 ALTERs target subagent tables that don't exist in pglite-schema.ts. Closes #395 (contributed by @jdcastro2). **v30 (v0.23):** creates `dream_verdicts (file_path TEXT, content_hash TEXT, worth_processing BOOL, reasons JSONB, judged_at TIMESTAMPTZ, PK(file_path, content_hash))`. RLS-enabled when running as a BYPASSRLS role. The synthesize phase reads/writes this table to avoid re-judging on backfill re-runs. **v35 (v0.26.7):** auto-RLS event trigger + one-time backfill. `auto_rls_on_create_table` fires on `ddl_command_end` for `WHEN TAG IN ('CREATE TABLE','CREATE TABLE AS','SELECT INTO')` and runs `ALTER TABLE … ENABLE ROW LEVEL SECURITY` on every new `public.*` table — no FORCE (matches v24/v29/schema.sql posture so non-BYPASSRLS apps can still read their own tables). The same migration backfills RLS on every existing `public.*` base table whose comment doesn't match the doctor regex (`^GBRAIN:RLS_EXEMPT\s+reason=\S.{3,}`). Per-table failure aborts the offending CREATE TABLE (event triggers fire inside the DDL transaction); no EXCEPTION wrap — that would convert loud rollback into silent permissive default. PGLite no-op via `sqlFor.pglite: ''`. Breaking change: operators with intentionally-RLS-off public tables must add the GBRAIN:RLS_EXEMPT comment BEFORE upgrade or the backfill will flip them on. - `src/core/progress.ts` — Shared bulk-action progress reporter. Writes to stderr. Modes: `auto` (TTY: `\r`-rewriting; non-TTY: plain lines), `human`, `json` (JSONL), `quiet`. Rate-gated by `minIntervalMs` and `minItems`. `startHeartbeat(reporter, note)` helper for single long queries. `child()` composes phase paths. Singleton SIGINT/SIGTERM coordinator emits `abort` events for every live phase. EPIPE defense on both sync throws and stream `'error'` events. Zero dependencies. Introduced in v0.15.2. - `src/core/cli-options.ts` — Global CLI flag parser. `parseGlobalFlags(argv)` returns `{cliOpts, rest}` with `--quiet` / `--progress-json` / `--progress-interval=` stripped. `getCliOptions()` / `setCliOptions()` expose a module-level singleton so commands reach the resolved flags without parameter threading. `cliOptsToProgressOptions()` maps to reporter options. `childGlobalFlags()` returns the flag suffix to append to `execSync('gbrain ...')` calls in migration orchestrators. `OperationContext.cliOpts` extends shared-op dispatch for MCP callers. - `src/core/db-lock.ts` (v0.22.13) — generic `tryAcquireDbLock(engine, lockId, ttlMinutes)` over the existing `gbrain_cycle_locks` table. Parameterized lock id so different scopes can nest cleanly: `gbrain-cycle` for the broad cycle (held by `cycle.ts`) and `gbrain-sync` (`SYNC_LOCK_ID` constant) for `performSync`'s narrower writer window. Same UPSERT-with-TTL semantics as the prior cycle-only helper, just generalized. Survives PgBouncer transaction pooling (unlike session-scoped `pg_try_advisory_lock`); crashed holders auto-release once their TTL expires. - `src/core/sync-concurrency.ts` (v0.22.13) — single source of truth for the parallel-sync policy. Exports `autoConcurrency(engine, fileCount, override?)` (PGLite always serial; explicit override clamped to >=1; auto path returns `DEFAULT_PARALLEL_WORKERS=4` when `fileCount > AUTO_CONCURRENCY_FILE_THRESHOLD=100`), `shouldRunParallel(workers, fileCount, explicit)` (Q1: explicit `--workers` bypasses the >50-file floor), and `parseWorkers(s)` (rejects `'0'`, `'-3'`, `'foo'`, `'1.5'`, trailing chars — replaces the prior parseInt-with-no-validation in both `sync.ts` and `import.ts`). Used by `performSync`, `performFullSync`, `runImport`, and the Minion `sync` handler so the three sites can no longer drift. - `src/commands/sync.ts` — `gbrain sync` CLI + the `performSync` / `performFullSync` library entrypoints (consumed by the autopilot cycle and the Minion sync handler). v0.22.13 (PR #490): `performSync` wraps its body in a `gbrain-sync` writer lock so two concurrent syncs (manual + autopilot, two terminals, two Conductor workspaces) cannot both write `last_commit` and let the last writer win. Head-drift gate after the import phase re-checks `git rev-parse HEAD`; if HEAD moved (someone ran `git checkout` / `git pull` mid-sync), the bookmark refuses to advance. Vanished files now record a failedFiles entry instead of silent-skip — the silent-skip-then-advance pathology that survived prior hardening passes is dead. Worker engines wrap in try/finally so disconnect always fires (panic-path leak fix). Both PGLite-detection sites use `engine.kind === 'pglite'`. CLI accepts `--workers N` (alias `--concurrency N`), validated via `parseWorkers`. Explicit `--workers` bypasses the auto-path file-count floor; auto path defers to `autoConcurrency()`. Banner moved to stderr. -- `src/core/cycle.ts` — v0.17 brain maintenance cycle primitive (extended to **8 phases in v0.23**). `runCycle(engine: BrainEngine | null, opts: CycleOpts): Promise` composes phases in semantically-driven order: **lint → backlinks → sync → synthesize → extract → patterns → embed → orphans**. v0.23's `synthesize` phase runs after sync (cross-references see fresh brain) and before extract (auto-link materializes its writes); `patterns` runs after extract so it reads a fresh graph (codex finding #7 — subagent put_page sets `ctx.remote=true` and skips auto-link/timeline by default; extract is the canonical materialization). Three callers: `gbrain dream` CLI, `gbrain autopilot` daemon's inline path, and the Minions `autopilot-cycle` handler. Coordination via `gbrain_cycle_locks` DB table + `~/.gbrain/cycle.lock` file lock with PID-liveness for PGLite. `CycleReport.schema_version: "1"` is stable; totals additively grew in v0.23 (`transcripts_processed`, `synth_pages_written`, `patterns_written`). `yieldBetweenPhases` runs between phases. **v0.23 added `yieldDuringPhase`** for in-phase keepalive — synthesize/patterns call it during long waits to renew the cycle-lock TTL. Engine nullable; lock-skip on read-only phase selections. v0.22.1 (#403): `CycleOpts.signal?: AbortSignal` propagates the worker's abort signal; `checkAborted()` fires between every phase. v0.22.1 (#417): `runPhaseSync` returns `pagesAffected` via `SyncPhaseResult`; `runCycle` captures it and threads to `runPhaseExtract` as the 4th arg. v0.22.1 (Codex F2): `runPhaseSync` takes `willRunExtractPhase: boolean` and sets `noExtract: phases.includes('extract')` so `gbrain dream --phase sync` doesn't silently lose extraction. v0.22.5 (#475): `resolveSourceForDir(engine, brainDir)` threads `sourceId` to `performSync()` so sync reads the per-source `sources.last_commit` anchor instead of the drift-prone global `config.sync.last_commit` key. +- `src/core/cycle.ts` — v0.17 brain maintenance cycle primitive (extended to **9 phases in v0.29**). `runCycle(engine: BrainEngine | null, opts: CycleOpts): Promise` composes phases in semantically-driven order: **lint → backlinks → sync → synthesize → extract → patterns → recompute_emotional_weight → embed → orphans**. v0.29 adds the `recompute_emotional_weight` phase between patterns and embed; it sees the union of `syncPagesAffected` + `synthesizeWrittenSlugs` for incremental mode, or all pages when neither anchor is set (full backfill via `gbrain dream --phase recompute_emotional_weight`). v0.29 also extends `CycleReport.totals` with `pages_emotional_weight_recomputed` (additive, schema_version stays "1"). v0.23's `synthesize` phase runs after sync (cross-references see fresh brain) and before extract (auto-link materializes its writes); `patterns` runs after extract so it reads a fresh graph (codex finding #7 — subagent put_page sets `ctx.remote=true` and skips auto-link/timeline by default; extract is the canonical materialization). Three callers: `gbrain dream` CLI, `gbrain autopilot` daemon's inline path, and the Minions `autopilot-cycle` handler. Coordination via `gbrain_cycle_locks` DB table + `~/.gbrain/cycle.lock` file lock with PID-liveness for PGLite. `CycleReport.schema_version: "1"` is stable; totals additively grew in v0.23 (`transcripts_processed`, `synth_pages_written`, `patterns_written`). `yieldBetweenPhases` runs between phases. **v0.23 added `yieldDuringPhase`** for in-phase keepalive — synthesize/patterns call it during long waits to renew the cycle-lock TTL. Engine nullable; lock-skip on read-only phase selections. v0.22.1 (#403): `CycleOpts.signal?: AbortSignal` propagates the worker's abort signal; `checkAborted()` fires between every phase. v0.22.1 (#417): `runPhaseSync` returns `pagesAffected` via `SyncPhaseResult`; `runCycle` captures it and threads to `runPhaseExtract` as the 4th arg. v0.22.1 (Codex F2): `runPhaseSync` takes `willRunExtractPhase: boolean` and sets `noExtract: phases.includes('extract')` so `gbrain dream --phase sync` doesn't silently lose extraction. v0.22.5 (#475): `resolveSourceForDir(engine, brainDir)` threads `sourceId` to `performSync()` so sync reads the per-source `sources.last_commit` anchor instead of the drift-prone global `config.sync.last_commit` key. - `src/core/cycle/synthesize.ts` (v0.23) — Synthesize phase: conversation-transcript-to-brain pipeline. Reads from `dream.synthesize.session_corpus_dir`, runs cheap Haiku verdict (cached in `dream_verdicts`), then fans out one Sonnet subagent per worth-processing transcript with `allowed_slug_prefixes` (sourced from `skills/_brain-filing-rules.json` `dream_synthesize_paths.globs`). Orchestrator collects slugs from `subagent_tool_executions` (NOT `pages.updated_at` — codex finding #2) and reverse-renders DB → markdown via `serializeMarkdown`. Cooldown via `dream.synthesize.last_completion_ts`, written ONLY on success. Idempotency key `dream:synth::`. Auto-commit deferred to v1.1 (codex #5). `--dry-run` runs Haiku, skips Sonnet (codex #8). Subagent never gets fs-write access. **v0.23.2:** `renderPageToMarkdown` (now exported) stamps `dream_generated: true` and `dream_cycle_date` into every reverse-write's frontmatter; `writeSummaryPage` does the same on the dream-cycle summary index. The marker is the explicit identity surface checked by `isDreamOutput` in `transcript-discovery.ts` — replaces the v0.23.1 content-prefix heuristic that could miss real output (`serializeMarkdown` doesn't embed slugs in body) and false-positive on user transcripts citing brain pages. `judgeSignificance` and `JudgeClient` are exported; `judgeSignificance` accepts a `verdictModel` parameter (default `claude-haiku-4-5-20251001`) loaded from `dream.synthesize.verdict_model` via `loadSynthConfig`. - `src/core/cycle/patterns.ts` (v0.23) — Patterns phase: cross-session theme detection over reflections within `dream.patterns.lookback_days` (default 30). Names a pattern only when ≥`dream.patterns.min_evidence` (default 3) reflections support it. Single Sonnet subagent; same allow-list path as synthesize. Runs AFTER `extract` so the graph is fresh. +- `src/core/cycle/emotional-weight.ts` (v0.29) — Pure function `computeEmotionalWeight({tags, takes}, {highEmotionTags?, userHolder?})`. Deterministic 0..1 score: tag-emotion boost (max 0.5, case-insensitive match against `HIGH_EMOTION_TAGS` seed list), take density (0.1/take, capped at 0.3), take avg weight (0..0.1), user-holder ratio (0..0.1 over active takes; default holder = 'garry'). Total clamped to [0..1]. Anglocentric / personal-life-biased seed list intentional; users override via config key `emotional_weight.high_tags` (JSON array). `userHolder` overridable via `emotional_weight.user_holder`. +- `src/core/cycle/anomaly.ts` (v0.29) — Pure stats helpers for `find_anomalies`. `meanStddev` returns sample stddev (n-1 denominator) and (0,0) for empty input. `computeAnomaliesFromBuckets(baseline, today, sigma, limit)` takes densified daily-count buckets + today's counts per cohort, returns `AnomalyResult[]`. Zero-stddev fallback: cohort fires when `count > mean + 1`, with `sigma_observed = count - mean` as a finite sort proxy (no NaN). Brand-new cohorts (no baseline) have `mean=0, stddev=0` so the fallback fires at count >= 2. Sorted by `sigma_observed` desc, top `limit` (default 20). `page_slugs` capped at 50 per cohort. +- `src/core/cycle/recompute-emotional-weight.ts` (v0.29) — Cycle phase orchestrator. Two SQL round-trips total: `engine.batchLoadEmotionalInputs(slugs?)` → `computeEmotionalWeight` (per-row pure function) → `engine.setEmotionalWeightBatch(rows)`. Reads config keys `emotional_weight.high_tags` (JSON array, falls back to default seed list on parse error) and `emotional_weight.user_holder`. Empty `affectedSlugs` array short-circuits with zero-work success. dry-run mode reports the would-write count without touching the DB. Engine throw bubbles into `status: 'fail'` with code `RECOMPUTE_EMOTIONAL_WEIGHT_FAIL` so the cycle continues. +- `src/core/transcripts.ts` (v0.29) — `listRecentTranscripts(engine, opts)` library reused by both the `gbrain transcripts recent` CLI and the `get_recent_transcripts` MCP op. Reads `dream.synthesize.session_corpus_dir` + `dream.synthesize.meeting_transcripts_dir` config keys (same as `discoverTranscripts`); walks for `.txt` files within `days`; applies `isDreamOutput` guard from `transcript-discovery.ts` (skips dream-generated files); returns `{path, date, mtime, length, summary}[]` sorted newest-first. Summary mode (default true) returns first non-empty line + ~250 trailing chars. Full mode caps at 100KB/file. Missing/non-existent corpus dirs return `[]`, not error. **Trust gate lives in the op handler, not here**: the op throws `permission_denied` for `ctx.remote === true`; this library is a trusted library function used by both the gated op and the local CLI. +- `src/core/operations-descriptions.ts` (v0.29) — Constants module for tool descriptions. Pinned via `test/operations-descriptions.test.ts`. Houses `GET_RECENT_SALIENCE_DESCRIPTION`, `FIND_ANOMALIES_DESCRIPTION`, `GET_RECENT_TRANSCRIPTS_DESCRIPTION` plus the redirect-edited `LIST_PAGES_DESCRIPTION`, `QUERY_DESCRIPTION`, `SEARCH_DESCRIPTION`. Stable surface for the Tier-2 LLM routing eval — extracting them keeps the test from binding to whatever was in `operations.ts` at test-run time. - `src/core/cycle/transcript-discovery.ts` (v0.23) — Pure filesystem walk for synthesize. `discoverTranscripts(opts)` filters `.txt` files by date range, min_chars, and word-boundary regex `excludePatterns` (Q-3: `medical` matches "medical advice" but NOT "comedical"; power users may pass full regex). `readSingleTranscript(path)` is the `gbrain dream --input ` ad-hoc path. **v0.23.2 self-consumption guard:** `DREAM_OUTPUT_MARKER_RE` (anchored at frontmatter open `---\n`, optional BOM + CRLF tolerance, scans first 2000 chars for `dream_generated: true` with case-insensitive value and word boundary on `true`) drives `isDreamOutput(content, bypass=false)`. Both `discoverTranscripts` and `readSingleTranscript` skip matching files and emit a `[dream] skipped : dream_generated marker` stderr log (no more silent skips). `bypassGuard?: boolean` on `DiscoverOpts` and `readSingleTranscript`'s opts disables the guard for the explicit `--unsafe-bypass-dream-guard` escape hatch only — never auto-applied for `--input`. Replaces v0.23.1's `DREAM_OUTPUT_SLUGS` content-prefix list. - `src/commands/dream.ts` — v0.17 `gbrain dream` CLI; ~80-line thin alias over `runCycle`. brainDir resolution requires explicit `--dir` OR `sync.repo_path` config. Flags: `--dry-run`, `--json`, `--phase `, `--pull`, `--dir `. **v0.23 added** `--input ` (ad-hoc transcript, implies `--phase synthesize`), `--date YYYY-MM-DD`, `--from --to ` (backfill range). Conflict detection: `--input` + `--date` exits 2. ISO date validation. `--dry-run` runs Haiku significance verdict but skips Sonnet synthesis (codex finding #8 — NOT zero LLM calls). Exit code 1 on status=failed. **v0.23.2 added** `--unsafe-bypass-dream-guard` (long-form intentional, plumbed through `runCycle.synthBypassDreamGuard` → `SynthesizePhaseOpts.bypassDreamGuard` → `discoverTranscripts({bypassGuard})` and `readSingleTranscript({bypassGuard})`). Loud stderr warning fires at synthesize-phase entry when set. Never auto-applied for `--input` so any caller can't silently re-trigger the loop bug. - `src/commands/friction.ts` + `src/core/friction.ts` (v0.23) — `gbrain friction {log,render,list,summary}` reporter. Append-only JSONL under `$GBRAIN_HOME/friction/.jsonl`. Schema is a flat extension of `StructuredAgentError` (D20). Render groups by severity → phase, defaults to `--redact` for md output (strips `$HOME`/`$CWD` to placeholders so reports paste safely in PRs). Run-id resolves from `--run-id` > `$GBRAIN_FRICTION_RUN_ID` > `standalone.jsonl`. Skills the claw-test exercises gain a `_friction-protocol.md` callout so agents know when to log friction. diff --git a/VERSION b/VERSION index 5e707c319..25939d35c 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -0.28.12 +0.29.1 diff --git a/llms-full.txt b/llms-full.txt index c7b38321b..116073646 100644 --- a/llms-full.txt +++ b/llms-full.txt @@ -127,7 +127,7 @@ mount, CEO-class with multiple team brains) and ## Architecture -Contract-first: `src/core/operations.ts` defines ~41 shared operations (adds `find_orphans` in v0.12.3). CLI and MCP +Contract-first: `src/core/operations.ts` defines ~47 shared operations (v0.29 adds `get_recent_salience`, `find_anomalies`, `get_recent_transcripts`). CLI and MCP server are both generated from this single source. Engine factory (`src/core/engine-factory.ts`) dynamically imports the configured engine (`'pglite'` or `'postgres'`). Skills are fat markdown files (tool-agnostic, work with both CLI and plugin contexts). @@ -141,7 +141,7 @@ strict behavior when unset. ## Key files - `src/core/operations.ts` — Contract-first operation definitions (the foundation). Also exports upload validators: `validateUploadPath`, `validatePageSlug`, `validateFilename`, plus `matchesSlugAllowList(slug, prefixes)` (v0.23 glob matcher: `/*` matches recursive children; bare `` matches exact only). `OperationContext.remote` flags untrusted callers; `OperationContext.allowedSlugPrefixes` (v0.23) is the trusted-workspace allow-list set by the dream cycle. `put_page` enforces: when `viaSubagent` and `allowedSlugPrefixes` is set, slug must match the allow-list; else the legacy `wiki/agents//...` namespace check applies. Auto-link enabled for trusted-workspace writes (skipped only when `remote=true && !trustedWorkspace`). As of v0.26.0, every `Operation` also carries `scope?: 'read' | 'write' | 'admin'` + `localOnly?: boolean`. All ops are annotated; `sync_brain`, `file_upload`, `file_list`, and `file_url` are `admin + localOnly` (rejected over HTTP). `OperationContext.auth?: AuthInfo` is threaded through HTTP dispatch for scope enforcement in `serve-http.ts` before the op runs. **v0.26.9 (D12 + F7b):** `OperationContext.remote` is now a REQUIRED field in the TypeScript type — the compiler is the first defense against transports that forget to set it. Four trust-boundary call sites (`put_page` allowlist, file_upload trust-narrowing, submit_job protected-name guard, auto-link skip) flipped from falsy-default (`!ctx.remote`) to fail-closed semantics (`ctx.remote === false` for "trusted-only" sites and `ctx.remote !== false` for "untrust unless explicit-false"). Anything that isn't strictly `false` is now treated as remote. Closed an HTTP MCP shell-job RCE: a `read+write`-scoped OAuth token could submit `shell` jobs because the HTTP request handler's literal context skipped `remote: true` and `submit_job`'s protected-name guard saw a falsy undefined. Stdio MCP set the field correctly via dispatch.ts; HTTP inlined a parallel context-builder for several releases and lost it. -- `src/core/engine.ts` — Pluggable engine interface (BrainEngine). `clampSearchLimit(limit, default, cap)` takes an explicit cap so per-operation caps can be tighter than `MAX_SEARCH_LIMIT`. Exports `LinkBatchInput` / `TimelineBatchInput` for the v0.12.1 bulk-insert API (`addLinksBatch` / `addTimelineEntriesBatch`). As of v0.13.1, `BrainEngine` has a `readonly kind: 'postgres' | 'pglite'` discriminator so migrations (`src/core/migrate.ts`) and other consumers can branch on engine without `instanceof` + dynamic imports. +- `src/core/engine.ts` — Pluggable engine interface (BrainEngine). `clampSearchLimit(limit, default, cap)` takes an explicit cap so per-operation caps can be tighter than `MAX_SEARCH_LIMIT`. Exports `LinkBatchInput` / `TimelineBatchInput` for the v0.12.1 bulk-insert API (`addLinksBatch` / `addTimelineEntriesBatch`). As of v0.13.1, `BrainEngine` has a `readonly kind: 'postgres' | 'pglite'` discriminator so migrations (`src/core/migrate.ts`) and other consumers can branch on engine without `instanceof` + dynamic imports. **v0.29:** four new methods — `batchLoadEmotionalInputs(slugs?)` (CTE-shaped read with per-table aggregates so a page × N tags × M takes never produces N×M rows), `setEmotionalWeightBatch(rows)` (`UPDATE FROM unnest($1::text[], $2::text[], $3::real[])` composite-keyed on `(slug, source_id)` for multi-source safety), `getRecentSalience(opts)`, `findAnomalies(opts)`. `PageFilters` extended with `sort?: 'updated_desc' | 'updated_asc' | 'created_desc' | 'slug'` + `PAGE_SORT_SQL` whitelist consumed by both engines (was hardcoded `ORDER BY updated_at DESC`). - `src/core/engine-factory.ts` — Engine factory with dynamic imports (`'pglite'` | `'postgres'`) - `src/core/pglite-engine.ts` — PGLite (embedded Postgres 17.5 via WASM) implementation, all 40 BrainEngine methods. `addLinksBatch` / `addTimelineEntriesBatch` use multi-row `unnest()` with manual `$N` placeholders. As of v0.13.1, `connect()` wraps `PGlite.create()` in a try/catch that emits an actionable error naming the macOS 26.3 WASM bug (#223) and pointing at `gbrain doctor`; the lock is released on failure so the next process can retry cleanly. v0.22.0: `searchKeyword` and `searchKeywordChunks` multiply `ts_rank` by the source-factor CASE expression at the chunk-grain level; `searchVector` becomes a two-stage CTE — inner CTE keeps `ORDER BY cc.embedding <=> vec` so HNSW stays usable, outer SELECT re-ranks by `raw_score * source_factor`. Inner LIMIT scales with offset to preserve pagination contract. As of v0.22.6.1, `initSchema()` calls `applyForwardReferenceBootstrap()` BEFORE replaying SCHEMA_SQL — probes for the specific forward-referenced state the embedded schema blob needs (`pages.source_id`, `links.link_source`, `links.origin_page_id`, `content_chunks.symbol_name`, `content_chunks.language`, `sources` FK target table) and adds only what's missing. Closes the upgrade-wedge bug class that bit users 10+ times across 6 schema versions over 2 years (#239/#243/#266/#357/#366/#374/#375/#378/#395/#396). No-op on fresh installs and modern brains. - `src/core/pglite-schema.ts` — PGLite-specific DDL (pgvector, pg_trgm, triggers) @@ -229,7 +229,7 @@ strict behavior when unset. - `src/core/minions/wait-for-completion.ts` (v0.15) — poll-until-terminal helper for CLI callers. `TimeoutError` does NOT cancel the job; `AbortSignal` exits without throwing. Default `pollMs`: 1000 on Postgres, 250 on PGLite inline. - `src/core/minions/transcript.ts` (v0.15) — renders `subagent_messages` + `subagent_tool_executions` to markdown. Tool rows splice under their owning assistant `tool_use` by `tool_use_id`. UTF-8-safe truncation; unknown block types fall through to fenced JSON. - `src/core/minions/plugin-loader.ts` (v0.15) — `GBRAIN_PLUGIN_PATH` discovery. Absolute paths only, left-wins collision, `gbrain.plugin.json` with `plugin_version: "gbrain-plugin-v1"`, plugins ship DEFS only (no new tools), `allowed_tools:` validated at load time against the derived registry. -- `src/core/minions/tools/brain-allowlist.ts` (v0.15, extended v0.23) — derives subagent tool registry from `src/core/operations.ts`. 11-name allow-list. By default `put_page` schema is namespace-wrapped per subagent (`^wiki/agents//.+`). **v0.23 trusted-workspace path:** when `BuildBrainToolsOpts.allowedSlugPrefixes` is set, the put_page schema instead describes the prefix list to the model and the OperationContext is threaded with `allowedSlugPrefixes`. Trust comes from `PROTECTED_JOB_NAMES` gating subagent submission — MCP cannot reach this field. Only cycle.ts (synthesize/patterns) and direct CLI submitters set it. +- `src/core/minions/tools/brain-allowlist.ts` (v0.15, extended v0.23, v0.29) — derives subagent tool registry from `src/core/operations.ts`. 13-name allow-list as of v0.29 (was 11). By default `put_page` schema is namespace-wrapped per subagent (`^wiki/agents//.+`). **v0.23 trusted-workspace path:** when `BuildBrainToolsOpts.allowedSlugPrefixes` is set, the put_page schema instead describes the prefix list to the model and the OperationContext is threaded with `allowedSlugPrefixes`. Trust comes from `PROTECTED_JOB_NAMES` gating subagent submission — MCP cannot reach this field. Only cycle.ts (synthesize/patterns) and direct CLI submitters set it. **v0.29:** `get_recent_salience` + `find_anomalies` added to the allow-list. `get_recent_transcripts` deliberately NOT added — all subagent calls run with `ctx.remote === true`, and the v0.29 trust gate rejects remote callers, so adding it would always reject (footgun). The cycle synthesize phase already calls `discoverTranscripts` directly. - `src/mcp/tool-defs.ts` (v0.15) — extracted `buildToolDefs(ops)` helper. MCP server + subagent tool registry both call it; byte-for-byte equivalence pinned by `test/mcp-tool-defs.test.ts`. - `src/core/minions/attachments.ts` — Attachment validation (path traversal, null byte, oversize, base64, duplicate detection) - `src/commands/agent.ts` (v0.16) — `gbrain agent run [flags]` CLI. Submits `subagent` (or N children + 1 aggregator) under `{allowProtectedSubmit: true}`. Single-entry `--fanout-manifest` short-circuits. Children get `on_child_fail: 'continue'` + `max_stalled: 3`. `--follow` is the default on TTY; streams logs + polls `waitForCompletion` in parallel. Ctrl-C detaches, does not cancel. @@ -248,17 +248,25 @@ strict behavior when unset. - `src/commands/migrations/` — TS migration registry (compiled into the binary; no filesystem walk of `skills/migrations/*.md` needed at runtime). `index.ts` lists migrations in semver order. `v0_11_0.ts` = Minions adoption orchestrator (8 phases). `v0_12_0.ts` = Knowledge Graph auto-wire orchestrator (5 phases: schema → config check → backfill links → backfill timeline → verify). `phaseASchema` has a 600s timeout (bumped from 60s in v0.12.1 for duplicate-heavy brains). `v0_12_2.ts` = JSONB double-encode repair orchestrator (4 phases: schema → repair-jsonb → verify → record). `v0_14_0.ts` = shell-jobs + autopilot cooperative (2 phases: schema ALTER minion_jobs.max_stalled SET DEFAULT 3 — superseded by v0.14.3's schema-level DEFAULT 5 + UPDATE backfill; pending-host-work ping for skills/migrations/v0.14.0.md). All orchestrators are idempotent and resumable from `partial` status. As of v0.14.2 (Bug 3), the RUNNER owns all ledger writes — orchestrators return `OrchestratorResult` and `apply-migrations.ts` persists a canonical `{version, status, phases}` shape after return. Orchestrators no longer call `appendCompletedMigration` directly. `statusForVersion` prefers `complete` over `partial` (never regresses). 3 consecutive partials → wedged → `--force-retry ` writes a `'retry'` reset marker. v0.14.3 (fix wave) ships schema-only migrations v14 (`pages_updated_at_index`) + v15 (`minion_jobs_max_stalled_default_5` with UPDATE backfill) via the `MIGRATIONS` array in `src/core/migrate.ts` — no orchestrator phases needed. - `src/commands/repair-jsonb.ts` — `gbrain repair-jsonb [--dry-run] [--json]`: rewrites `jsonb_typeof='string'` rows in place across 5 affected columns (pages.frontmatter, raw_data.data, ingest_log.pages_updated, files.metadata, page_versions.frontmatter). Fixes v0.12.0 double-encode bug on Postgres; PGLite no-ops. Idempotent. - `src/commands/orphans.ts` — `gbrain orphans [--json] [--count] [--include-pseudo]`: surfaces pages with zero inbound wikilinks, grouped by domain. Auto-generated/raw/pseudo pages filtered by default. Also exposed as `find_orphans` MCP operation. Shipped in v0.12.3 (contributed by @knee5). +- `src/commands/salience.ts` (v0.29) — `gbrain salience [--days N] [--limit N] [--kind PREFIX] [--json]`: pages ranked by emotional + activity salience over a recency window. Mirrors orphans.ts shape (pure data fn + JSON formatter + human formatter). Calls `engine.getRecentSalience(opts)`. Score formula: `(emotional_weight × 5) + ln(1 + active_take_count) + 1/(1 + days_since_update)`. +- `src/commands/anomalies.ts` (v0.29) — `gbrain anomalies [--since YYYY-MM-DD] [--lookback-days N] [--sigma N] [--json]`: cohort-level activity outliers. Calls `engine.findAnomalies(opts)`. Two cohort kinds in v1: tag, type. Year cohort deferred to v0.30. +- `src/commands/transcripts.ts` (v0.29) — `gbrain transcripts recent [--days N] [--full] [--json]`: recent raw `.txt` transcripts from the dream-cycle corpus dirs. Imports `listRecentTranscripts` from `src/core/transcripts.ts` (the same library the gated `get_recent_transcripts` MCP op uses). Local-only by construction — the CLI always runs with `ctx.remote=false`. - `src/commands/integrity.ts` — `gbrain integrity check|auto|review|extract`: bare-tweet detection, dead-link detection, three-bucket repair (auto-repair / review-queue / skip). `scanIntegrity()` is the shared library function called from `gbrain doctor` (sampled at limit=500) and `cmdCheck` (full scan). v0.22.8: batch-load fast path on Postgres uses `SELECT DISTINCT ON (slug)` in a single SQL query to fix the PgBouncer round-trip timeout (60s → ~6s) while preserving `engine.getAllSlugs()`'s `Set` semantics on multi-source brains. Gated by `engine.kind === 'postgres'` at the call site so PGLite never enters batch; fallback `catch` logs at `GBRAIN_DEBUG=1` so real Postgres errors are diagnosable. - `src/commands/doctor.ts` — `gbrain doctor [--json] [--fast] [--fix] [--dry-run] [--index-audit]`: health checks. v0.12.3 added `jsonb_integrity` + `markdown_body_completeness` reliability checks. v0.14.1: `--fix` delegates inlined cross-cutting rules to `> **Convention:** see [path](path).` callouts (pipes DRY violations into `src/core/dry-fix.ts`); `--fix --dry-run` previews without writing. v0.14.2: `schema_version` check fails loudly when `version=0` (migrations never ran — the #218 `bun install -g` signature) and routes users to `gbrain apply-migrations --yes`; new opt-in `--index-audit` flag (Postgres-only) reports zero-scan indexes from `pg_stat_user_indexes` (informational only, no auto-drop). v0.15.2: every DB check is wrapped in a progress phase; `markdown_body_completeness` runs under a 1s heartbeat timer so 10+ min scans are observable on 50K-page brains. v0.19.1 added `queue_health` (Postgres-only) with two subchecks: stalled-forever active jobs (started_at > 1h) and waiting-depth-per-name > threshold (default 10, override via `GBRAIN_QUEUE_WAITING_THRESHOLD`). Worker-heartbeat subcheck intentionally deferred to follow-up B7 because it needs a `minion_workers` table to produce ground-truth signal. Fix hints point at `gbrain repair-jsonb`, `gbrain sync --force`, `gbrain apply-migrations`, and `gbrain jobs get/cancel `. v0.22.12 (#500): `sync_failures` check shows `[CODE=N, ...]` breakdown for both unacked entries (warn) and acked-historical entries (ok), surfacing systemic failure modes (`SLUG_MISMATCH=2685`) instead of a bare count. v0.26.7 (#612): `rls_event_trigger` check (post-install drift detector for migration v35's auto-RLS event trigger). Lives outside the `// 5. RLS` slice that the structural doctor.test.ts guards anchor on, so the existing test guards stay intact. Healthy `evtenabled` set is `('O','A')` only — `R` is replica-only and would not fire in normal sessions; `D` is disabled. Fix hint is `gbrain apply-migrations --force-retry 35`. -- `src/core/migrate.ts` — schema-migration runner. Owns the `MIGRATIONS` array (source of truth for schema DDL). v0.14.2 extended the `Migration` interface with `sqlFor?: { postgres?, pglite? }` (engine-specific SQL overrides `sql`) and `transaction?: boolean` (set to false for `CREATE INDEX CONCURRENTLY`, which Postgres refuses inside a transaction; ignored on PGLite since it has no concurrent writers). Migration v14 (fix wave) uses a handler branching on `engine.kind` to run CONCURRENTLY on Postgres (with a pre-drop of any invalid remnant via `pg_index.indisvalid`) and plain `CREATE INDEX` on PGLite. v15 bumps `minion_jobs.max_stalled` default 1→5 and backfills existing non-terminal rows. v0.22.6.1: migration v24 (`rls_backfill_missing_tables`) uses `sqlFor: { pglite: '' }` to no-op on PGLite — PGLite has no RLS engine and is single-tenant by definition, and the v24 ALTERs target subagent tables that don't exist in pglite-schema.ts. Closes #395 (contributed by @jdcastro2). **v30 (v0.23):** creates `dream_verdicts (file_path TEXT, content_hash TEXT, worth_processing BOOL, reasons JSONB, judged_at TIMESTAMPTZ, PK(file_path, content_hash))`. RLS-enabled when running as a BYPASSRLS role. The synthesize phase reads/writes this table to avoid re-judging on backfill re-runs. **v35 (v0.26.7):** auto-RLS event trigger + one-time backfill. `auto_rls_on_create_table` fires on `ddl_command_end` for `WHEN TAG IN ('CREATE TABLE','CREATE TABLE AS','SELECT INTO')` and runs `ALTER TABLE … ENABLE ROW LEVEL SECURITY` on every new `public.*` table — no FORCE (matches v24/v29/schema.sql posture so non-BYPASSRLS apps can still read their own tables). The same migration backfills RLS on every existing `public.*` base table whose comment doesn't match the doctor regex (`^GBRAIN:RLS_EXEMPT\s+reason=\S.{3,}`). Per-table failure aborts the offending CREATE TABLE (event triggers fire inside the DDL transaction); no EXCEPTION wrap — that would convert loud rollback into silent permissive default. PGLite no-op via `sqlFor.pglite: ''`. Breaking change: operators with intentionally-RLS-off public tables must add the GBRAIN:RLS_EXEMPT comment BEFORE upgrade or the backfill will flip them on. +- `src/core/migrate.ts` — schema-migration runner. Owns the `MIGRATIONS` array (source of truth for schema DDL). **v40 (v0.29):** `pages_emotional_weight` adds `pages.emotional_weight REAL NOT NULL DEFAULT 0.0`. Column-only (no index). On Postgres 11+ and PGLite, `ADD COLUMN` with a constant DEFAULT is metadata-only — instant on tables of any size. v0.14.2 extended the `Migration` interface with `sqlFor?: { postgres?, pglite? }` (engine-specific SQL overrides `sql`) and `transaction?: boolean` (set to false for `CREATE INDEX CONCURRENTLY`, which Postgres refuses inside a transaction; ignored on PGLite since it has no concurrent writers). Migration v14 (fix wave) uses a handler branching on `engine.kind` to run CONCURRENTLY on Postgres (with a pre-drop of any invalid remnant via `pg_index.indisvalid`) and plain `CREATE INDEX` on PGLite. v15 bumps `minion_jobs.max_stalled` default 1→5 and backfills existing non-terminal rows. v0.22.6.1: migration v24 (`rls_backfill_missing_tables`) uses `sqlFor: { pglite: '' }` to no-op on PGLite — PGLite has no RLS engine and is single-tenant by definition, and the v24 ALTERs target subagent tables that don't exist in pglite-schema.ts. Closes #395 (contributed by @jdcastro2). **v30 (v0.23):** creates `dream_verdicts (file_path TEXT, content_hash TEXT, worth_processing BOOL, reasons JSONB, judged_at TIMESTAMPTZ, PK(file_path, content_hash))`. RLS-enabled when running as a BYPASSRLS role. The synthesize phase reads/writes this table to avoid re-judging on backfill re-runs. **v35 (v0.26.7):** auto-RLS event trigger + one-time backfill. `auto_rls_on_create_table` fires on `ddl_command_end` for `WHEN TAG IN ('CREATE TABLE','CREATE TABLE AS','SELECT INTO')` and runs `ALTER TABLE … ENABLE ROW LEVEL SECURITY` on every new `public.*` table — no FORCE (matches v24/v29/schema.sql posture so non-BYPASSRLS apps can still read their own tables). The same migration backfills RLS on every existing `public.*` base table whose comment doesn't match the doctor regex (`^GBRAIN:RLS_EXEMPT\s+reason=\S.{3,}`). Per-table failure aborts the offending CREATE TABLE (event triggers fire inside the DDL transaction); no EXCEPTION wrap — that would convert loud rollback into silent permissive default. PGLite no-op via `sqlFor.pglite: ''`. Breaking change: operators with intentionally-RLS-off public tables must add the GBRAIN:RLS_EXEMPT comment BEFORE upgrade or the backfill will flip them on. - `src/core/progress.ts` — Shared bulk-action progress reporter. Writes to stderr. Modes: `auto` (TTY: `\r`-rewriting; non-TTY: plain lines), `human`, `json` (JSONL), `quiet`. Rate-gated by `minIntervalMs` and `minItems`. `startHeartbeat(reporter, note)` helper for single long queries. `child()` composes phase paths. Singleton SIGINT/SIGTERM coordinator emits `abort` events for every live phase. EPIPE defense on both sync throws and stream `'error'` events. Zero dependencies. Introduced in v0.15.2. - `src/core/cli-options.ts` — Global CLI flag parser. `parseGlobalFlags(argv)` returns `{cliOpts, rest}` with `--quiet` / `--progress-json` / `--progress-interval=` stripped. `getCliOptions()` / `setCliOptions()` expose a module-level singleton so commands reach the resolved flags without parameter threading. `cliOptsToProgressOptions()` maps to reporter options. `childGlobalFlags()` returns the flag suffix to append to `execSync('gbrain ...')` calls in migration orchestrators. `OperationContext.cliOpts` extends shared-op dispatch for MCP callers. - `src/core/db-lock.ts` (v0.22.13) — generic `tryAcquireDbLock(engine, lockId, ttlMinutes)` over the existing `gbrain_cycle_locks` table. Parameterized lock id so different scopes can nest cleanly: `gbrain-cycle` for the broad cycle (held by `cycle.ts`) and `gbrain-sync` (`SYNC_LOCK_ID` constant) for `performSync`'s narrower writer window. Same UPSERT-with-TTL semantics as the prior cycle-only helper, just generalized. Survives PgBouncer transaction pooling (unlike session-scoped `pg_try_advisory_lock`); crashed holders auto-release once their TTL expires. - `src/core/sync-concurrency.ts` (v0.22.13) — single source of truth for the parallel-sync policy. Exports `autoConcurrency(engine, fileCount, override?)` (PGLite always serial; explicit override clamped to >=1; auto path returns `DEFAULT_PARALLEL_WORKERS=4` when `fileCount > AUTO_CONCURRENCY_FILE_THRESHOLD=100`), `shouldRunParallel(workers, fileCount, explicit)` (Q1: explicit `--workers` bypasses the >50-file floor), and `parseWorkers(s)` (rejects `'0'`, `'-3'`, `'foo'`, `'1.5'`, trailing chars — replaces the prior parseInt-with-no-validation in both `sync.ts` and `import.ts`). Used by `performSync`, `performFullSync`, `runImport`, and the Minion `sync` handler so the three sites can no longer drift. - `src/commands/sync.ts` — `gbrain sync` CLI + the `performSync` / `performFullSync` library entrypoints (consumed by the autopilot cycle and the Minion sync handler). v0.22.13 (PR #490): `performSync` wraps its body in a `gbrain-sync` writer lock so two concurrent syncs (manual + autopilot, two terminals, two Conductor workspaces) cannot both write `last_commit` and let the last writer win. Head-drift gate after the import phase re-checks `git rev-parse HEAD`; if HEAD moved (someone ran `git checkout` / `git pull` mid-sync), the bookmark refuses to advance. Vanished files now record a failedFiles entry instead of silent-skip — the silent-skip-then-advance pathology that survived prior hardening passes is dead. Worker engines wrap in try/finally so disconnect always fires (panic-path leak fix). Both PGLite-detection sites use `engine.kind === 'pglite'`. CLI accepts `--workers N` (alias `--concurrency N`), validated via `parseWorkers`. Explicit `--workers` bypasses the auto-path file-count floor; auto path defers to `autoConcurrency()`. Banner moved to stderr. -- `src/core/cycle.ts` — v0.17 brain maintenance cycle primitive (extended to **8 phases in v0.23**). `runCycle(engine: BrainEngine | null, opts: CycleOpts): Promise` composes phases in semantically-driven order: **lint → backlinks → sync → synthesize → extract → patterns → embed → orphans**. v0.23's `synthesize` phase runs after sync (cross-references see fresh brain) and before extract (auto-link materializes its writes); `patterns` runs after extract so it reads a fresh graph (codex finding #7 — subagent put_page sets `ctx.remote=true` and skips auto-link/timeline by default; extract is the canonical materialization). Three callers: `gbrain dream` CLI, `gbrain autopilot` daemon's inline path, and the Minions `autopilot-cycle` handler. Coordination via `gbrain_cycle_locks` DB table + `~/.gbrain/cycle.lock` file lock with PID-liveness for PGLite. `CycleReport.schema_version: "1"` is stable; totals additively grew in v0.23 (`transcripts_processed`, `synth_pages_written`, `patterns_written`). `yieldBetweenPhases` runs between phases. **v0.23 added `yieldDuringPhase`** for in-phase keepalive — synthesize/patterns call it during long waits to renew the cycle-lock TTL. Engine nullable; lock-skip on read-only phase selections. v0.22.1 (#403): `CycleOpts.signal?: AbortSignal` propagates the worker's abort signal; `checkAborted()` fires between every phase. v0.22.1 (#417): `runPhaseSync` returns `pagesAffected` via `SyncPhaseResult`; `runCycle` captures it and threads to `runPhaseExtract` as the 4th arg. v0.22.1 (Codex F2): `runPhaseSync` takes `willRunExtractPhase: boolean` and sets `noExtract: phases.includes('extract')` so `gbrain dream --phase sync` doesn't silently lose extraction. v0.22.5 (#475): `resolveSourceForDir(engine, brainDir)` threads `sourceId` to `performSync()` so sync reads the per-source `sources.last_commit` anchor instead of the drift-prone global `config.sync.last_commit` key. +- `src/core/cycle.ts` — v0.17 brain maintenance cycle primitive (extended to **9 phases in v0.29**). `runCycle(engine: BrainEngine | null, opts: CycleOpts): Promise` composes phases in semantically-driven order: **lint → backlinks → sync → synthesize → extract → patterns → recompute_emotional_weight → embed → orphans**. v0.29 adds the `recompute_emotional_weight` phase between patterns and embed; it sees the union of `syncPagesAffected` + `synthesizeWrittenSlugs` for incremental mode, or all pages when neither anchor is set (full backfill via `gbrain dream --phase recompute_emotional_weight`). v0.29 also extends `CycleReport.totals` with `pages_emotional_weight_recomputed` (additive, schema_version stays "1"). v0.23's `synthesize` phase runs after sync (cross-references see fresh brain) and before extract (auto-link materializes its writes); `patterns` runs after extract so it reads a fresh graph (codex finding #7 — subagent put_page sets `ctx.remote=true` and skips auto-link/timeline by default; extract is the canonical materialization). Three callers: `gbrain dream` CLI, `gbrain autopilot` daemon's inline path, and the Minions `autopilot-cycle` handler. Coordination via `gbrain_cycle_locks` DB table + `~/.gbrain/cycle.lock` file lock with PID-liveness for PGLite. `CycleReport.schema_version: "1"` is stable; totals additively grew in v0.23 (`transcripts_processed`, `synth_pages_written`, `patterns_written`). `yieldBetweenPhases` runs between phases. **v0.23 added `yieldDuringPhase`** for in-phase keepalive — synthesize/patterns call it during long waits to renew the cycle-lock TTL. Engine nullable; lock-skip on read-only phase selections. v0.22.1 (#403): `CycleOpts.signal?: AbortSignal` propagates the worker's abort signal; `checkAborted()` fires between every phase. v0.22.1 (#417): `runPhaseSync` returns `pagesAffected` via `SyncPhaseResult`; `runCycle` captures it and threads to `runPhaseExtract` as the 4th arg. v0.22.1 (Codex F2): `runPhaseSync` takes `willRunExtractPhase: boolean` and sets `noExtract: phases.includes('extract')` so `gbrain dream --phase sync` doesn't silently lose extraction. v0.22.5 (#475): `resolveSourceForDir(engine, brainDir)` threads `sourceId` to `performSync()` so sync reads the per-source `sources.last_commit` anchor instead of the drift-prone global `config.sync.last_commit` key. - `src/core/cycle/synthesize.ts` (v0.23) — Synthesize phase: conversation-transcript-to-brain pipeline. Reads from `dream.synthesize.session_corpus_dir`, runs cheap Haiku verdict (cached in `dream_verdicts`), then fans out one Sonnet subagent per worth-processing transcript with `allowed_slug_prefixes` (sourced from `skills/_brain-filing-rules.json` `dream_synthesize_paths.globs`). Orchestrator collects slugs from `subagent_tool_executions` (NOT `pages.updated_at` — codex finding #2) and reverse-renders DB → markdown via `serializeMarkdown`. Cooldown via `dream.synthesize.last_completion_ts`, written ONLY on success. Idempotency key `dream:synth::`. Auto-commit deferred to v1.1 (codex #5). `--dry-run` runs Haiku, skips Sonnet (codex #8). Subagent never gets fs-write access. **v0.23.2:** `renderPageToMarkdown` (now exported) stamps `dream_generated: true` and `dream_cycle_date` into every reverse-write's frontmatter; `writeSummaryPage` does the same on the dream-cycle summary index. The marker is the explicit identity surface checked by `isDreamOutput` in `transcript-discovery.ts` — replaces the v0.23.1 content-prefix heuristic that could miss real output (`serializeMarkdown` doesn't embed slugs in body) and false-positive on user transcripts citing brain pages. `judgeSignificance` and `JudgeClient` are exported; `judgeSignificance` accepts a `verdictModel` parameter (default `claude-haiku-4-5-20251001`) loaded from `dream.synthesize.verdict_model` via `loadSynthConfig`. - `src/core/cycle/patterns.ts` (v0.23) — Patterns phase: cross-session theme detection over reflections within `dream.patterns.lookback_days` (default 30). Names a pattern only when ≥`dream.patterns.min_evidence` (default 3) reflections support it. Single Sonnet subagent; same allow-list path as synthesize. Runs AFTER `extract` so the graph is fresh. +- `src/core/cycle/emotional-weight.ts` (v0.29) — Pure function `computeEmotionalWeight({tags, takes}, {highEmotionTags?, userHolder?})`. Deterministic 0..1 score: tag-emotion boost (max 0.5, case-insensitive match against `HIGH_EMOTION_TAGS` seed list), take density (0.1/take, capped at 0.3), take avg weight (0..0.1), user-holder ratio (0..0.1 over active takes; default holder = 'garry'). Total clamped to [0..1]. Anglocentric / personal-life-biased seed list intentional; users override via config key `emotional_weight.high_tags` (JSON array). `userHolder` overridable via `emotional_weight.user_holder`. +- `src/core/cycle/anomaly.ts` (v0.29) — Pure stats helpers for `find_anomalies`. `meanStddev` returns sample stddev (n-1 denominator) and (0,0) for empty input. `computeAnomaliesFromBuckets(baseline, today, sigma, limit)` takes densified daily-count buckets + today's counts per cohort, returns `AnomalyResult[]`. Zero-stddev fallback: cohort fires when `count > mean + 1`, with `sigma_observed = count - mean` as a finite sort proxy (no NaN). Brand-new cohorts (no baseline) have `mean=0, stddev=0` so the fallback fires at count >= 2. Sorted by `sigma_observed` desc, top `limit` (default 20). `page_slugs` capped at 50 per cohort. +- `src/core/cycle/recompute-emotional-weight.ts` (v0.29) — Cycle phase orchestrator. Two SQL round-trips total: `engine.batchLoadEmotionalInputs(slugs?)` → `computeEmotionalWeight` (per-row pure function) → `engine.setEmotionalWeightBatch(rows)`. Reads config keys `emotional_weight.high_tags` (JSON array, falls back to default seed list on parse error) and `emotional_weight.user_holder`. Empty `affectedSlugs` array short-circuits with zero-work success. dry-run mode reports the would-write count without touching the DB. Engine throw bubbles into `status: 'fail'` with code `RECOMPUTE_EMOTIONAL_WEIGHT_FAIL` so the cycle continues. +- `src/core/transcripts.ts` (v0.29) — `listRecentTranscripts(engine, opts)` library reused by both the `gbrain transcripts recent` CLI and the `get_recent_transcripts` MCP op. Reads `dream.synthesize.session_corpus_dir` + `dream.synthesize.meeting_transcripts_dir` config keys (same as `discoverTranscripts`); walks for `.txt` files within `days`; applies `isDreamOutput` guard from `transcript-discovery.ts` (skips dream-generated files); returns `{path, date, mtime, length, summary}[]` sorted newest-first. Summary mode (default true) returns first non-empty line + ~250 trailing chars. Full mode caps at 100KB/file. Missing/non-existent corpus dirs return `[]`, not error. **Trust gate lives in the op handler, not here**: the op throws `permission_denied` for `ctx.remote === true`; this library is a trusted library function used by both the gated op and the local CLI. +- `src/core/operations-descriptions.ts` (v0.29) — Constants module for tool descriptions. Pinned via `test/operations-descriptions.test.ts`. Houses `GET_RECENT_SALIENCE_DESCRIPTION`, `FIND_ANOMALIES_DESCRIPTION`, `GET_RECENT_TRANSCRIPTS_DESCRIPTION` plus the redirect-edited `LIST_PAGES_DESCRIPTION`, `QUERY_DESCRIPTION`, `SEARCH_DESCRIPTION`. Stable surface for the Tier-2 LLM routing eval — extracting them keeps the test from binding to whatever was in `operations.ts` at test-run time. - `src/core/cycle/transcript-discovery.ts` (v0.23) — Pure filesystem walk for synthesize. `discoverTranscripts(opts)` filters `.txt` files by date range, min_chars, and word-boundary regex `excludePatterns` (Q-3: `medical` matches "medical advice" but NOT "comedical"; power users may pass full regex). `readSingleTranscript(path)` is the `gbrain dream --input ` ad-hoc path. **v0.23.2 self-consumption guard:** `DREAM_OUTPUT_MARKER_RE` (anchored at frontmatter open `---\n`, optional BOM + CRLF tolerance, scans first 2000 chars for `dream_generated: true` with case-insensitive value and word boundary on `true`) drives `isDreamOutput(content, bypass=false)`. Both `discoverTranscripts` and `readSingleTranscript` skip matching files and emit a `[dream] skipped : dream_generated marker` stderr log (no more silent skips). `bypassGuard?: boolean` on `DiscoverOpts` and `readSingleTranscript`'s opts disables the guard for the explicit `--unsafe-bypass-dream-guard` escape hatch only — never auto-applied for `--input`. Replaces v0.23.1's `DREAM_OUTPUT_SLUGS` content-prefix list. - `src/commands/dream.ts` — v0.17 `gbrain dream` CLI; ~80-line thin alias over `runCycle`. brainDir resolution requires explicit `--dir` OR `sync.repo_path` config. Flags: `--dry-run`, `--json`, `--phase `, `--pull`, `--dir `. **v0.23 added** `--input ` (ad-hoc transcript, implies `--phase synthesize`), `--date YYYY-MM-DD`, `--from --to ` (backfill range). Conflict detection: `--input` + `--date` exits 2. ISO date validation. `--dry-run` runs Haiku significance verdict but skips Sonnet synthesis (codex finding #8 — NOT zero LLM calls). Exit code 1 on status=failed. **v0.23.2 added** `--unsafe-bypass-dream-guard` (long-form intentional, plumbed through `runCycle.synthBypassDreamGuard` → `SynthesizePhaseOpts.bypassDreamGuard` → `discoverTranscripts({bypassGuard})` and `readSingleTranscript({bypassGuard})`). Loud stderr warning fires at synthesize-phase entry when set. Never auto-applied for `--input` so any caller can't silently re-trigger the loop bug. - `src/commands/friction.ts` + `src/core/friction.ts` (v0.23) — `gbrain friction {log,render,list,summary}` reporter. Append-only JSONL under `$GBRAIN_HOME/friction/.jsonl`. Schema is a flat extension of `StructuredAgentError` (D20). Render groups by severity → phase, defaults to `--redact` for md output (strips `$HOME`/`$CWD` to placeholders so reports paste safely in PRs). Run-id resolves from `--run-id` > `$GBRAIN_FRICTION_RUN_ID` > `standalone.jsonl`. Skills the claw-test exercises gain a `_friction-protocol.md` callout so agents know when to log friction. diff --git a/package.json b/package.json index 770ec54c4..be5657790 100644 --- a/package.json +++ b/package.json @@ -1,6 +1,6 @@ { "name": "gbrain", - "version": "0.28.12", + "version": "0.29.1", "description": "Postgres-native personal knowledge brain with hybrid RAG search", "type": "module", "main": "src/core/index.ts", diff --git a/scripts/check-privacy.sh b/scripts/check-privacy.sh index 020c2f68c..d85201af4 100755 --- a/scripts/check-privacy.sh +++ b/scripts/check-privacy.sh @@ -121,6 +121,13 @@ ALLOW_LIST=( # walkthrough; it explains the privacy-guard extension to the # operating agent and references the banned literals while doing so. 'skills/migrations/v0.25.1.md' + # v0.29.1: the recency-decay default-map test asserts that + # DEFAULT_RECENCY_DECAY's keys do NOT include fork-specific path + # prefixes. The test must name the banned tokens to assert their + # absence — same exception status as scripts/check-privacy.sh, + # CHANGELOG.md, and CLAUDE.md (meta-rule enforcement requires + # mentioning what the rule forbids). + 'test/recency-decay.test.ts' ) is_allowed() { diff --git a/skills/conventions/salience-and-recency.md b/skills/conventions/salience-and-recency.md new file mode 100644 index 000000000..aa451a610 --- /dev/null +++ b/skills/conventions/salience-and-recency.md @@ -0,0 +1,131 @@ +# Salience + Recency on `gbrain query` (v0.29.1) + +YOU ARE IN CHARGE of the `salience` and `recency` parameters on gbrain's +`query` op. They are TWO ORTHOGONAL axes — use either, both, or neither. + +If you OMIT a parameter, gbrain auto-detects from query text via a +regex heuristic. The default for queries that don't match any pattern +is `'off'`. Prefer to pass values EXPLICITLY when you know what the +user wants. + +## What each axis means + +- `salience` — **mattering**. Boosts pages with high `emotional_weight` + and many active takes. NO time component. Use when the user wants + the most important / most-discussed pages on a topic, regardless of + when they were updated. + +- `recency` — **age**. Boosts pages with recent `effective_date`. NO + mattering signal. Per-prefix decay (`concepts/`, `originals/`, + `writing/` are evergreen; `daily/`, `media/x/`, `chat/` decay + aggressively). Use when freshness is the signal. + +## When to pass `salience='on'` + +The "mattering" axis. The user wants what matters in this brain on +the topic, not the canonical encyclopedia entry. + +- `"prep me for the widget-ceo meeting"` (meeting prep) +- `"catch me up on acme"` (conversation recall) +- `"what's going on with widget-co"` (current state matters) +- `"remind me about the deal"` (recall takes / opinions) +- `"what's been happening lately"` +- `"status update on X"` + +Pair with `recency='on'` when current-state matters. Just `salience='on'` +alone gives you "what matters about X regardless of when." + +## When to pass `recency='on'` + +The "freshness" axis. The user wants recent content, with or without +mattering. + +- `"latest news on AI"` (recent, no mattering needed) +- `"what's new this week"` +- `"recent updates on widget-co"` +- `"this week's announcements"` + +Use `'strong'` when the user explicitly asks for the most recent: + +- `"what happened today"` +- `"right now what's going on"` +- `"this morning"` + +## When to pass BOTH `'off'` + +The "canonical truth" axis. The user wants the authoritative answer. + +- `"who is widget-ceo"` (entity lookup) +- `"what is widget-co"` (definitional) +- `"history of acme"` (historical research) +- `"explain how recursion works"` (concept query) +- `"tell me about widget-co"` (canonical recall) +- Code lookups: function/class names, syntax like `Foo::bar()` or `obj.method` +- Graph traversal: backlinks, inbound/outbound edges +- Anything not matching above + +## Heuristic when unsure + +> Current state → on. Canonical truth → off. + +If you can't classify confidently, OMIT the param and let gbrain's +auto-detect handle it. The heuristic defaults to `off` for everything +that doesn't clearly match a current-state pattern. The `--explain` +output shows `_resolved.salience_source` and `_resolved.recency_source` +('caller' vs. 'auto_heuristic') so you can see what fired and why. + +You can override at any time. gbrain is smart but not infallible. You +have context gbrain doesn't. + +## Narrow temporal-bound exception + +Even when a query matches canonical patterns, an explicit temporal +bound (`today`, `this week`, `right now`, `since X`, `last N days`) +overrides the canonical-wins rule: + +- `"who is widget-ceo right now"` → recency = `'strong'`, salience = `'on'` + (the temporal bound wins over "who is") +- `"who is widget-ceo"` → recency = `'off'`, salience = `'off'` (no bound) + +## English-only + +The auto-detect heuristic is English-only in v0.29.1. Non-English +queries fall through to the default `off` for both axes. Pass +`salience` and `recency` explicitly for non-English queries. + +## Tuning the recency formula + +Defaults are in `src/core/search/recency-decay.ts`. Override per-brain +via `gbrain.yml`: + +```yaml +recency: + daily/: + halflifeDays: 7 + coefficient: 2.0 + custom-prefix/: + halflifeDays: 30 + coefficient: 0.5 +``` + +Or per-process via env: `GBRAIN_RECENCY_DECAY="prefix:halflife:coefficient,..."`. +The parser fails LOUD on bad syntax (no silent fallback). + +## Date filtering with `since` / `until` + +Independent of the axes. Filter to pages whose `effective_date` is +within a range: + +- `since: '7d'` — last 7 days +- `since: '2024-06-01'` — ISO-8601 +- `until: '2024-06-30'` — ends at end-of-day + +`since`/`until` work with OR without `salience`/`recency`. Pure filter, +no boost. + +## See also + +- `docs/recency.md` — full reference +- `gbrain query --explain` — see resolved values + factor contributions +- `get_recent_salience` op gains `recency_bias: 'flat' | 'on'` — opt + into per-prefix decay on the dedicated salience query diff --git a/skills/migrations/v0.29.1.md b/skills/migrations/v0.29.1.md new file mode 100644 index 000000000..94abecae0 --- /dev/null +++ b/skills/migrations/v0.29.1.md @@ -0,0 +1,106 @@ +# v0.29.1 migration — Salience + Recency as two orthogonal options + +`gbrain upgrade` runs `gbrain apply-migrations` automatically. Most users +won't need to do anything else. If you hit issues or want to verify the +upgrade succeeded, run the steps below. + +## What changed + +This release adds two new optional ranking axes to the `query` MCP op: + +- `salience: 'off' | 'on' | 'strong'` — boost pages with high + `emotional_weight` + many active takes. NO time component. +- `recency: 'off' | 'on' | 'strong'` — boost pages with recent + `effective_date`, per-prefix decay. NO mattering signal. + +They're TRULY ORTHOGONAL — either, both, or neither. The agent decides +per query, with `--explain` output showing the resolved values and +sources. + +Existing v0.29.0 callers (no new params) get UNCHANGED behavior. + +## What the agent should do + +Read `skills/conventions/salience-and-recency.md` (auto-installed by +the skillpack). Then in your `gbrain query` calls: + +1. Pass `salience='on'` for "what matters about X" queries (catch up, + meeting prep, conversation recall). +2. Pass `recency='on'` for "what's new on X" queries (latest, this + week, today's news). +3. Pass both for "what's been going on with X" queries. +4. Omit both for canonical / definitional / code / graph queries + (`who is X`, `what is X`, etc.) — gbrain's heuristic defaults + to `off`. + +## Verification + +```bash +# 1. Confirm upgrade +gbrain --version # 0.29.1 +gbrain doctor --json | jq '.checks[] | select(.name | startswith("schema_version"))' + +# 2. Recompute emotional weights (one-time after upgrade) +gbrain dream --phase recompute_emotional_weight + +# 3. Verify health checks +gbrain doctor --json | jq '.checks[] | select(.name | startswith("salience_health") or startswith("effective_date_health"))' + +# 4. Try the new axes +gbrain query "what's been going on with X" --explain --json | jq '._resolved' +# expected: { salience: "on", recency: "on", salience_source: "auto_heuristic", recency_source: "auto_heuristic" } + +gbrain query "who is X" --explain --json | jq '._resolved' +# expected: { salience: "off", recency: "off", ... } + +# 5. Date filter +gbrain query "acme" --since 7d --until 2024-06-30 --json +``` + +## If something looks wrong + +```bash +# Re-apply migrations manually +gbrain apply-migrations --yes + +# Force re-run the v0.29.1 backfill (computeEffectiveDate on every page) +gbrain reindex-frontmatter --yes --force + +# Doctor for any warnings +gbrain doctor --json +``` + +If issues persist, file at https://github.com/garrytan/gbrain/issues +with the doctor output and contents of `~/.gbrain/upgrade-errors.jsonl` +(if it exists). + +## Schema additions (idempotent, additive only) + +Migration v38 adds 4 nullable columns to `pages`: +- `effective_date` — content-date computed from frontmatter precedence +- `effective_date_source` — sentinel for the doctor check +- `import_filename` — basename captured at import for filename-date precedence +- `salience_touched_at` — bumped by recompute_emotional_weight on changes + +Migration v39 adds 7 nullable columns to `eval_candidates`: +- `as_of_ts`, `salience_param`, `recency_param`, `salience_resolved`, + `recency_resolved`, `salience_source`, `recency_source` + +Plus the `pages_coalesce_date_idx` expression index for since/until filters. + +NDJSON `schema_version` STAYS at 1; consumers ignore unknown fields. +No cross-repo coordination required. + +## Behavior changes + +- v0.29.0 `get_recent_salience` formula: UNCHANGED for callers who don't + pass `recency_bias='on'`. Pass `recency_bias='on'` to opt into per-prefix + decay (concepts/originals/writing/ evergreen; daily/, media/x/ aggressive). + +- v0.29.0 SearchOpts: `afterDate`, `beforeDate`, `recencyBoost: 0|1|2` + remain as DEPRECATED ALIASES for `since`, `until`, `recency`. They + emit a stderr warning once per process. Removed in v0.30. + +- `detail='high'` source-boost bypass — UNCHANGED in v0.29.1. The + known temporal-query swamp is documented; pass `salience='on'` to + compensate via salience boost. diff --git a/src/cli.ts b/src/cli.ts index 4a8a14ddd..ecaadd67e 100755 --- a/src/cli.ts +++ b/src/cli.ts @@ -24,7 +24,7 @@ for (const op of operations) { } // CLI-only commands that bypass the operation layer -const CLI_ONLY = new Set(['init', 'upgrade', 'post-upgrade', 'check-update', 'integrations', 'publish', 'check-backlinks', 'lint', 'report', 'import', 'export', 'files', 'embed', 'serve', 'call', 'config', 'doctor', 'migrate', 'eval', 'sync', 'extract', 'features', 'autopilot', 'graph-query', 'jobs', 'agent', 'apply-migrations', 'skillpack-check', 'skillpack', 'resolvers', 'integrity', 'repair-jsonb', 'orphans', 'sources', 'mounts', 'dream', 'check-resolvable', 'routing-eval', 'skillify', 'smoke-test', 'providers', 'storage', 'repos', 'code-def', 'code-refs', 'reindex-code', 'code-callers', 'code-callees', 'frontmatter', 'auth', 'friction', 'claw-test', 'book-mirror', 'takes', 'think']); +const CLI_ONLY = new Set(['init', 'upgrade', 'post-upgrade', 'check-update', 'integrations', 'publish', 'check-backlinks', 'lint', 'report', 'import', 'export', 'files', 'embed', 'serve', 'call', 'config', 'doctor', 'migrate', 'eval', 'sync', 'extract', 'features', 'autopilot', 'graph-query', 'jobs', 'agent', 'apply-migrations', 'skillpack-check', 'skillpack', 'resolvers', 'integrity', 'repair-jsonb', 'orphans', 'sources', 'mounts', 'dream', 'check-resolvable', 'routing-eval', 'skillify', 'smoke-test', 'providers', 'storage', 'repos', 'code-def', 'code-refs', 'reindex-code', 'reindex-frontmatter', 'code-callers', 'code-callees', 'frontmatter', 'auth', 'friction', 'claw-test', 'book-mirror', 'takes', 'think', 'salience', 'anomalies', 'transcripts']); async function main() { // Parse global flags (--quiet / --progress-json / --progress-interval) @@ -639,6 +639,22 @@ async function handleCliOnly(command: string, args: string[]) { await runOrphans(engine, args); break; } + // v0.29 — Salience + Anomaly Detection + case 'salience': { + const { runSalience } = await import('./commands/salience.ts'); + await runSalience(engine, args); + break; + } + case 'anomalies': { + const { runAnomalies } = await import('./commands/anomalies.ts'); + await runAnomalies(engine, args); + break; + } + case 'transcripts': { + const { runTranscripts } = await import('./commands/transcripts.ts'); + await runTranscripts(engine, args); + break; + } case 'takes': { const { runTakes } = await import('./commands/takes.ts'); await runTakes(engine, args); @@ -683,6 +699,16 @@ async function handleCliOnly(command: string, args: string[]) { await runReindexCodeCli(engine, args); break; } + case 'reindex-frontmatter': { + // v0.29.1: recovery / explicit-rebuild path for pages.effective_date. + // Mirror of reindex-code shape. Wraps the shared library function in + // src/core/backfill-effective-date.ts (same code path the v0.29.1 + // migration orchestrator uses). The orchestrator runs once on + // upgrade; this command is for after-the-fact frontmatter edits. + const { reindexFrontmatterCli } = await import('./commands/reindex-frontmatter.ts'); + await reindexFrontmatterCli(args); + return; // reindexFrontmatterCli handles its own engine lifecycle + } case 'code-callers': { // v0.20.0 Cathedral II Layer 10 (C4): "who calls ?" const { runCodeCallers } = await import('./commands/code-callers.ts'); @@ -896,6 +922,9 @@ TOOLS check-backlinks [dir] Find/fix missing back-links across brain lint [--fix] Catch LLM artifacts, placeholder dates, bad frontmatter orphans [--json] [--count] Find pages with no inbound wikilinks + salience [--days N] [--kind P] v0.29: pages ranked by emotional + activity salience + anomalies [--since D] [--sigma N] v0.29: cohort-based statistical anomalies (tag, type) + transcripts recent [--days N] v0.29: recent raw .txt transcripts (local-only) dream [--dry-run] [--json] Run the overnight maintenance cycle once (cron-friendly). See also: autopilot --install (continuous daemon). check-resolvable [--json] [--fix] Validate skill tree (reachability/MECE/DRY) diff --git a/src/commands/anomalies.ts b/src/commands/anomalies.ts new file mode 100644 index 000000000..0051ce84f --- /dev/null +++ b/src/commands/anomalies.ts @@ -0,0 +1,96 @@ +/** + * gbrain anomalies — Statistical anomalies in recent page activity. + * + * Deterministic: zero LLM calls. Computes baseline (mean, stddev) of pages + * touched per cohort × day over `lookback_days`, with `generate_series` + * zero-fill so rare cohorts don't get sparse-day biased baselines. Reports + * cohorts whose target-day count exceeds `mean + sigma * stddev`. + * + * Cohort kinds: tag, type. Year cohort deferred to v0.30. + * + * Usage: + * gbrain anomalies # since=today, lookback=30d, sigma=3 + * gbrain anomalies --since 2026-04-28 + * gbrain anomalies --sigma 2 --lookback-days 60 + * gbrain anomalies --json + */ + +import type { BrainEngine } from '../core/engine.ts'; + +interface RunOpts { + since?: string; + lookbackDays?: number; + sigma?: number; + json?: boolean; +} + +function parseArgs(args: string[]): RunOpts | { help: true } { + const opts: RunOpts = {}; + for (let i = 0; i < args.length; i++) { + const a = args[i]; + if (a === '--help' || a === '-h') return { help: true }; + if (a === '--json') { opts.json = true; continue; } + if (a === '--since') { + const v = args[++i]; + if (v && /^\d{4}-\d{2}-\d{2}$/.test(v)) opts.since = v; + continue; + } + if (a === '--lookback-days') { + const n = parseInt(args[++i] ?? '', 10); + if (Number.isFinite(n) && n >= 1) opts.lookbackDays = n; + continue; + } + if (a === '--sigma') { + const n = parseFloat(args[++i] ?? ''); + if (Number.isFinite(n) && n > 0) opts.sigma = n; + continue; + } + } + return opts; +} + +const HELP = `Usage: gbrain anomalies [options] + +Statistical anomalies in recent page activity, grouped by cohort (tag, type). + +Options: + --since YYYY-MM-DD Target day (default: today UTC) + --lookback-days N Baseline window (default 30) + --sigma N Threshold multiplier (default 3.0) + --json JSON output for agents + --help, -h Show this help +`; + +export async function runAnomalies(engine: BrainEngine, args: string[]): Promise { + const parsed = parseArgs(args); + if ('help' in parsed) { + console.log(HELP); + return; + } + const rows = await engine.findAnomalies({ + since: parsed.since, + lookback_days: parsed.lookbackDays, + sigma: parsed.sigma, + }); + if (parsed.json) { + console.log(JSON.stringify(rows, null, 2)); + return; + } + if (rows.length === 0) { + console.log('(no anomalies for this window)'); + return; + } + console.log(`${rows.length} anomalous cohort(s) for ${parsed.since ?? new Date().toISOString().slice(0, 10)}:\n`); + rows.forEach(r => { + const baselineMean = r.baseline_mean.toFixed(2); + const baselineStd = r.baseline_stddev.toFixed(2); + const sigma = r.sigma_observed.toFixed(2); + console.log( + `[${r.cohort_kind}=${r.cohort_value}] ` + + `count=${r.count}, baseline mean=${baselineMean}±${baselineStd}, sigma=${sigma}` + ); + const slugSample = r.page_slugs.slice(0, 5).join(', '); + const more = r.page_slugs.length > 5 ? `, +${r.page_slugs.length - 5} more` : ''; + if (slugSample) console.log(` pages: ${slugSample}${more}`); + }); +} diff --git a/src/commands/doctor.ts b/src/commands/doctor.ts index 3b30bc589..9dfc42de9 100644 --- a/src/commands/doctor.ts +++ b/src/commands/doctor.ts @@ -949,6 +949,114 @@ export async function runDoctor(engine: BrainEngine | null, args: string[], dbSo } } + // 11a-2. effective_date_health (v0.29.1). + // + // Detects pages where computeEffectiveDate fell back to updated_at even + // though parseable frontmatter dates are present (codex pass-1 #5 + // resolution: the sentinel column lets us catch "wrong but populated" + // rows that look healthy at first glance). + // + // Sample 1000 random rows by default to keep the check fast on 200K-page + // brains. The expression index pages_coalesce_date_idx makes the future- + // date and pre-1990 scans cheap; the parseable-fm-date scan reads + // frontmatter JSONB and is the slow path. + progress.heartbeat('effective_date_health'); + try { + const result = await engine.executeRaw<{ kind: string; count: string }>( + `WITH sample AS ( + SELECT slug, frontmatter, effective_date, effective_date_source + FROM pages + ORDER BY id DESC + LIMIT 1000 + ) + SELECT 'fallback_with_fm_date' AS kind, COUNT(*)::text AS count + FROM sample + WHERE effective_date_source = 'fallback' + AND (frontmatter ? 'event_date' OR frontmatter ? 'date' OR frontmatter ? 'published') + UNION ALL + SELECT 'future_dated', COUNT(*)::text FROM sample + WHERE effective_date IS NOT NULL AND effective_date > NOW() + INTERVAL '1 year' + UNION ALL + SELECT 'pre_1990', COUNT(*)::text FROM sample + WHERE effective_date IS NOT NULL AND effective_date < TIMESTAMPTZ '1990-01-01'`, + ); + const counts = new Map(result.map(r => [r.kind, Number(r.count)])); + const fallbackWithFm = counts.get('fallback_with_fm_date') ?? 0; + const future = counts.get('future_dated') ?? 0; + const pre1990 = counts.get('pre_1990') ?? 0; + if (fallbackWithFm > 0 || future > 0 || pre1990 > 0) { + const parts: string[] = []; + if (fallbackWithFm > 0) parts.push(`${fallbackWithFm} fell back to updated_at despite parseable frontmatter date`); + if (future > 0) parts.push(`${future} dated > NOW() + 1y`); + if (pre1990 > 0) parts.push(`${pre1990} pre-1990`); + checks.push({ + name: 'effective_date_health', + status: 'warn', + message: `${parts.join('; ')} (sample of last 1000 pages). Run \`gbrain reindex-frontmatter\` to recompute.`, + }); + } else { + checks.push({ + name: 'effective_date_health', + status: 'ok', + message: 'Sample of last 1000 pages clean (no fallback-with-parseable-fm-date, no future-dated, no pre-1990)', + }); + } + } catch (err) { + const code = (err as { code?: string } | null)?.code; + if (code === '42703') { + // column doesn't exist — pre-v0.29.1 brain + checks.push({ name: 'effective_date_health', status: 'ok', message: 'Skipped (effective_date column unavailable — run gbrain apply-migrations)' }); + } else { + checks.push({ name: 'effective_date_health', status: 'warn', message: `Could not read pages: ${(err as Error)?.message ?? String(err)}` }); + } + } + + // 11a-3. salience_health (v0.29.1). + // + // Detects pages with active takes (so emotional_weight should be > 0) + // whose recompute_emotional_weight phase hasn't yet run, plus the + // brain-average emotional_weight as an informational signal. + progress.heartbeat('salience_health'); + try { + const result = await engine.executeRaw<{ kind: string; n: string }>( + `SELECT 'zero_weight_with_takes' AS kind, COUNT(DISTINCT p.id)::text AS n + FROM pages p + JOIN takes t ON t.page_id = p.id AND t.active = TRUE + WHERE COALESCE(p.emotional_weight, 0) = 0 + UNION ALL + SELECT 'nonzero_weight', COUNT(*)::text FROM pages WHERE COALESCE(emotional_weight, 0) > 0`, + ); + const counts = new Map(result.map(r => [r.kind, Number(r.n)])); + const zeroWithTakes = counts.get('zero_weight_with_takes') ?? 0; + const nonzero = counts.get('nonzero_weight') ?? 0; + if (zeroWithTakes > 0) { + checks.push({ + name: 'salience_health', + status: 'warn', + message: `${zeroWithTakes} pages with active takes have emotional_weight=0. Run \`gbrain dream --phase recompute_emotional_weight\` to populate. Brain has ${nonzero} pages with non-zero emotional_weight.`, + }); + } else if (nonzero === 0) { + checks.push({ + name: 'salience_health', + status: 'ok', + message: 'Skipped (no pages have emotional_weight > 0; either fresh install or recompute hasn\'t run yet)', + }); + } else { + checks.push({ + name: 'salience_health', + status: 'ok', + message: `${nonzero} pages have non-zero emotional_weight; no take/weight mismatches detected`, + }); + } + } catch (err) { + const code = (err as { code?: string } | null)?.code; + if (code === '42703' || code === '42P01') { + checks.push({ name: 'salience_health', status: 'ok', message: 'Skipped (emotional_weight or takes table unavailable — pre-v0.29 brain)' }); + } else { + checks.push({ name: 'salience_health', status: 'warn', message: `Could not read pages: ${(err as Error)?.message ?? String(err)}` }); + } + } + // 11b. Queue health (v0.19.1 queue-resilience wave). // Postgres-only because PGLite has no multi-process worker surface. Two // subchecks, both cheap (single SELECT each, status-index-covered): diff --git a/src/commands/migrations/index.ts b/src/commands/migrations/index.ts index 22403b1cb..355445fe4 100644 --- a/src/commands/migrations/index.ts +++ b/src/commands/migrations/index.ts @@ -23,6 +23,7 @@ import { v0_18_1 } from './v0_18_1.ts'; import { v0_21_0 } from './v0_21_0.ts'; import { v0_22_4 } from './v0_22_4.ts'; import { v0_28_0 } from './v0_28_0.ts'; +import { v0_29_1 } from './v0_29_1.ts'; export const migrations: Migration[] = [ v0_11_0, @@ -37,6 +38,7 @@ export const migrations: Migration[] = [ v0_21_0, v0_22_4, v0_28_0, + v0_29_1, ]; /** Look up a migration by exact version string. */ diff --git a/src/commands/migrations/v0_29_1.ts b/src/commands/migrations/v0_29_1.ts new file mode 100644 index 000000000..38b677d6c --- /dev/null +++ b/src/commands/migrations/v0_29_1.ts @@ -0,0 +1,160 @@ +/** + * v0.29.1 migration orchestrator — backfill effective_date for existing + * pages. + * + * Migration v38 added pages.effective_date / effective_date_source / + * import_filename / salience_touched_at as nullable columns. Fresh imports + * post-v0.29.1 populate effective_date via the importer's + * `computeEffectiveDate`. Pre-v0.29.1 rows have NULL until this orchestrator + * walks them. + * + * Phases (all idempotent, resumable): + * A. Schema — `gbrain init --migrate-only` ensures v38 ran. + * B. Backfill — keyset-paginated UPDATE via `backfillEffectiveDate`. + * Resumable via the `backfill.effective_date.last_id` + * checkpoint key in the config table. Statement timeout + * set per-batch (Postgres only). + * C. Verify — count remaining NULL effective_date rows; warn if > 0. + * D. Record — handled by the runner. + */ + +import { execSync } from 'child_process'; +import type { Migration, OrchestratorOpts, OrchestratorResult, OrchestratorPhaseResult } from './types.ts'; +import { childGlobalFlags } from '../../core/cli-options.ts'; + +// ── Phase A — Schema ──────────────────────────────────────── + +function phaseASchema(opts: OrchestratorOpts): OrchestratorPhaseResult { + if (opts.dryRun) return { name: 'schema', status: 'skipped', detail: 'dry-run' }; + try { + execSync('gbrain init --migrate-only' + childGlobalFlags(), { + stdio: 'inherit', + timeout: 600_000, // 10 min — duplicate-heavy installs can be slow + env: process.env, + }); + return { name: 'schema', status: 'complete' }; + } catch (e) { + return { name: 'schema', status: 'failed', detail: e instanceof Error ? e.message : String(e) }; + } +} + +// ── Phase B — Backfill effective_date ─────────────────────── + +async function phaseBBackfill(opts: OrchestratorOpts): Promise { + if (opts.dryRun) return { name: 'backfill_effective_date', status: 'skipped', detail: 'dry-run' }; + try { + const { createEngine } = await import('../../core/engine-factory.ts'); + const { loadConfig, toEngineConfig } = await import('../../core/config.ts'); + const { backfillEffectiveDate } = await import('../../core/backfill-effective-date.ts'); + const cfg = loadConfig(); + if (!cfg) throw new Error('No gbrain config; run `gbrain init` first.'); + const engine = await createEngine(toEngineConfig(cfg)); + + let totalExamined = 0; + let totalUpdated = 0; + + const result = await backfillEffectiveDate(engine, { + onBatch: ({ batch, lastId, rowsTouched, cumulative }) => { + totalExamined = cumulative; + totalUpdated += rowsTouched; + if (batch % 10 === 0) { + process.stderr.write(` [backfill] batch ${batch} | last_id=${lastId} | examined=${cumulative} | updated_so_far=${totalUpdated}\n`); + } + }, + }); + + return { + name: 'backfill_effective_date', + status: 'complete', + detail: `examined=${result.examined} updated=${result.updated} fallback=${result.fallback} dur=${result.durationSec.toFixed(1)}s`, + }; + } catch (e) { + return { name: 'backfill_effective_date', status: 'failed', detail: e instanceof Error ? e.message : String(e) }; + } +} + +// ── Phase C — Verify ──────────────────────────────────────── + +async function phaseCVerify(opts: OrchestratorOpts): Promise { + if (opts.dryRun) return { name: 'verify', status: 'skipped', detail: 'dry-run' }; + try { + const { createEngine } = await import('../../core/engine-factory.ts'); + const { loadConfig, toEngineConfig } = await import('../../core/config.ts'); + const cfg = loadConfig(); + if (!cfg) throw new Error('No gbrain config; run `gbrain init` first.'); + const engine = await createEngine(toEngineConfig(cfg)); + // Count rows where effective_date is still NULL but frontmatter HAS a + // parseable date — those are the rows the backfill should have touched + // but didn't. (Rows that fall through to 'fallback' have non-null + // effective_date already; this catches genuine misses.) + const rows = await engine.executeRaw<{ count: string }>( + `SELECT COUNT(*)::text AS count FROM pages WHERE effective_date IS NULL`, + ); + const remaining = Number(rows[0]?.count ?? 0); + if (remaining > 0) { + return { + name: 'verify', + status: 'failed', + detail: `${remaining} pages still have NULL effective_date (backfill incomplete)`, + }; + } + return { name: 'verify', status: 'complete', detail: '0 pages with NULL effective_date' }; + } catch (e) { + return { name: 'verify', status: 'failed', detail: e instanceof Error ? e.message : String(e) }; + } +} + +// ── Orchestrator ──────────────────────────────────────────── + +async function orchestrator(opts: OrchestratorOpts): Promise { + console.log(''); + console.log('=== v0.29.1 — backfill effective_date for existing pages ==='); + if (opts.dryRun) console.log(' (dry-run; no side effects)'); + console.log(''); + + const phases: OrchestratorPhaseResult[] = []; + + const a = phaseASchema(opts); + phases.push(a); + if (a.status === 'failed') return finalize(phases, 'failed'); + + const b = await phaseBBackfill(opts); + phases.push(b); + if (b.status === 'failed') return finalize(phases, 'partial'); + + const c = await phaseCVerify(opts); + phases.push(c); + + const status: 'complete' | 'partial' | 'failed' = + c.status === 'failed' ? 'partial' : 'complete'; + + return finalize(phases, status); +} + +function finalize(phases: OrchestratorPhaseResult[], status: 'complete' | 'partial' | 'failed'): OrchestratorResult { + return { version: '0.29.1', status, phases }; +} + +export const v0_29_1: Migration = { + version: '0.29.1', + featurePitch: { + headline: 'Recency + salience as two opt-in axes — agent in charge of when to use each', + description: + 'gbrain v0.29.1 adds two new optional ranking axes to the query op: salience ' + + '(emotional_weight + take_count, the "this matters" signal) and recency (per-prefix ' + + 'age decay, the "this is recent" signal). Truly orthogonal — use either, both, or ' + + "neither. The query op's tool description teaches your agent when each makes sense " + + '("current state → on; canonical truth → off") and the agent can override per query. ' + + 'A new pages.effective_date column is computed at import from frontmatter precedence ' + + '(event_date / date / published / filename) and is immune to auto-link updated_at ' + + 'churn. Existing callers (no new params) get UNCHANGED behavior. Run ' + + "`gbrain dream --phase recompute_emotional_weight` once after upgrading.", + }, + orchestrator, +}; + +export const __testing = { + phaseASchema, + phaseBBackfill, + phaseCVerify, +}; diff --git a/src/commands/reindex-frontmatter.ts b/src/commands/reindex-frontmatter.ts new file mode 100644 index 000000000..6331c13b6 --- /dev/null +++ b/src/commands/reindex-frontmatter.ts @@ -0,0 +1,186 @@ +/** + * v0.29.1 — `gbrain reindex-frontmatter`. + * + * Recovery / explicit-rebuild path for `pages.effective_date`. Useful when: + * - The user edited frontmatter dates after import and wants the effective_date + * column refreshed without a full `gbrain sync`. + * - The post-upgrade backfill orchestrator finished but the user wants to + * re-walk a subset (e.g. just `meetings/`) after fixing some frontmatter. + * - The precedence rules change between releases and the user wants to + * re-apply on existing rows. + * + * Thin wrapper over the shared library function in + * `src/core/backfill-effective-date.ts` (same code path the migration + * orchestrator uses; one source of truth for the backfill logic). + * + * Flags mirror `reindex-code`: + * --source Scope to one sources row. Omit = all pages. + * --slug-prefix P Scope to slugs starting with P (e.g. 'meetings/'). + * --dry-run Print what WOULD change, no DB writes. + * --yes Skip the confirmation prompt (required for non-TTY non-JSON). + * --json Machine-readable result envelope. + * --force Re-apply even when computed value matches existing + * (bypasses no-op-on-equal guard). + */ + +import type { BrainEngine } from '../core/engine.ts'; +import { backfillEffectiveDate } from '../core/backfill-effective-date.ts'; +import { createInterface } from 'readline'; + +export interface ReindexFrontmatterOpts { + sourceId?: string; + slugPrefix?: string; + dryRun?: boolean; + yes?: boolean; + json?: boolean; + force?: boolean; +} + +export interface ReindexFrontmatterResult { + status: 'ok' | 'dry_run' | 'cancelled'; + examined: number; + updated: number; + fallback: number; + durationSec: number; + source_filter?: string; + slug_prefix?: string; +} + +async function countAffected( + engine: BrainEngine, + slugPrefix: string | undefined, + sourceId: string | undefined, +): Promise { + const where: string[] = []; + const params: unknown[] = []; + if (slugPrefix) { + params.push(slugPrefix.replace(/[\\%_]/g, (c) => '\\' + c) + '%'); + where.push(`slug LIKE $${params.length} ESCAPE '\\\\'`); + } + if (sourceId) { + params.push(sourceId); + where.push(`source_id = $${params.length}`); + } + const sql = `SELECT COUNT(*)::text AS n FROM pages${where.length ? ' WHERE ' + where.join(' AND ') : ''}`; + const rows = await engine.executeRaw<{ n: string }>(sql, params); + return Number(rows[0]?.n ?? 0); +} + +async function confirm(prompt: string): Promise { + if (!process.stdin.isTTY) return false; // No TTY = require --yes + const rl = createInterface({ input: process.stdin, output: process.stderr }); + return new Promise(resolve => { + rl.question(prompt + ' [y/N] ', (ans: string) => { + rl.close(); + resolve(ans.trim().toLowerCase() === 'y'); + }); + }); +} + +export async function runReindexFrontmatter( + engine: BrainEngine, + opts: ReindexFrontmatterOpts, +): Promise { + const total = await countAffected(engine, opts.slugPrefix, opts.sourceId); + + if (opts.dryRun) { + // Library function with dryRun=true counts would-update without writing. + const r = await backfillEffectiveDate(engine, { + slugPrefix: opts.slugPrefix, + dryRun: true, + force: opts.force, + // Note: the library doesn't support sourceId filter today; documented + // as a v0.30+ enhancement. CLI surfaces the param so the future + // refinement is non-breaking. + maxRows: total > 0 ? total : undefined, + }); + return { + status: 'dry_run', + examined: r.examined, + updated: r.updated, + fallback: r.fallback, + durationSec: r.durationSec, + slug_prefix: opts.slugPrefix, + source_filter: opts.sourceId, + }; + } + + // Confirm in TTY non-yes flow. + if (!opts.yes && !opts.json && total > 100) { + const ok = await confirm(`Reindex effective_date on ${total} page(s)? Force=${opts.force ? 'yes' : 'no'}.`); + if (!ok) { + return { + status: 'cancelled', + examined: 0, updated: 0, fallback: 0, durationSec: 0, + slug_prefix: opts.slugPrefix, + source_filter: opts.sourceId, + }; + } + } + + const r = await backfillEffectiveDate(engine, { + slugPrefix: opts.slugPrefix, + force: opts.force, + fresh: true, // CLI is explicit; ignore checkpoint from prior orchestrator runs + onBatch: ({ batch, lastId, rowsTouched, cumulative }) => { + if (!opts.json && batch % 5 === 0) { + process.stderr.write(` [reindex] batch ${batch} | last_id=${lastId} | examined=${cumulative} | updated=${rowsTouched}\n`); + } + }, + }); + + return { + status: 'ok', + examined: r.examined, + updated: r.updated, + fallback: r.fallback, + durationSec: r.durationSec, + slug_prefix: opts.slugPrefix, + source_filter: opts.sourceId, + }; +} + +/** CLI entrypoint. Argv shape matches reindex-code for consistency. */ +export async function reindexFrontmatterCli(args: string[]): Promise { + const opts: ReindexFrontmatterOpts = {}; + for (let i = 0; i < args.length; i++) { + const a = args[i]; + if (a === '--source') opts.sourceId = args[++i]; + else if (a === '--slug-prefix') opts.slugPrefix = args[++i]; + else if (a === '--dry-run') opts.dryRun = true; + else if (a === '--yes' || a === '-y') opts.yes = true; + else if (a === '--json') opts.json = true; + else if (a === '--force') opts.force = true; + else { + console.error(`Unknown arg: ${a}`); + process.exit(2); + } + } + + const { createEngine } = await import('../core/engine-factory.ts'); + const { loadConfig, toEngineConfig } = await import('../core/config.ts'); + const cfg = loadConfig(); + if (!cfg) { + console.error('No gbrain config; run `gbrain init` first.'); + process.exit(1); + } + const engine = await createEngine(toEngineConfig(cfg)); + + try { + const result = await runReindexFrontmatter(engine, opts); + if (opts.json) { + console.log(JSON.stringify(result, null, 2)); + } else { + const noun = result.status === 'dry_run' ? 'would update' : 'updated'; + console.error( + `\nReindex ${result.status}: examined=${result.examined} ${noun}=${result.updated} ` + + `fallback=${result.fallback} dur=${result.durationSec.toFixed(1)}s`, + ); + } + if (result.status === 'cancelled') process.exit(1); + } finally { + if ('disconnect' in engine && typeof engine.disconnect === 'function') { + await engine.disconnect(); + } + } +} diff --git a/src/commands/salience.ts b/src/commands/salience.ts new file mode 100644 index 000000000..c0dd53e6b --- /dev/null +++ b/src/commands/salience.ts @@ -0,0 +1,97 @@ +/** + * gbrain salience — Pages recently touched, ranked by emotional + activity salience. + * + * Deterministic: zero LLM calls. The score blends `emotional_weight` + * (computed during the dream cycle's recompute_emotional_weight phase), + * the count of active takes, and a recency-decay term. See the engine method + * `getRecentSalience` for the SQL. + * + * Usage: + * gbrain salience # top 20 over last 14 days + * gbrain salience --days 7 # narrower window + * gbrain salience --kind personal # filter to slug-prefix + * gbrain salience --json # JSON for agents + */ + +import type { BrainEngine } from '../core/engine.ts'; + +interface RunOpts { + days?: number; + limit?: number; + slugPrefix?: string; + json?: boolean; +} + +function parseArgs(args: string[]): RunOpts | { help: true } { + const opts: RunOpts = {}; + for (let i = 0; i < args.length; i++) { + const a = args[i]; + if (a === '--help' || a === '-h') return { help: true }; + if (a === '--json') { opts.json = true; continue; } + if (a === '--days') { + const n = parseInt(args[++i] ?? '', 10); + if (Number.isFinite(n) && n >= 0) opts.days = n; + continue; + } + if (a === '--limit') { + const n = parseInt(args[++i] ?? '', 10); + if (Number.isFinite(n) && n > 0) opts.limit = n; + continue; + } + if (a === '--kind' || a === '--slug-prefix') { + const v = args[++i]; + if (v) opts.slugPrefix = v; + continue; + } + } + return opts; +} + +const HELP = `Usage: gbrain salience [options] + +Pages recently touched, ranked by emotional + activity salience. Surfaces +what's unusual without needing a search term — the inverse of /query. + +Options: + --days N Window in days (default 14) + --limit N Max results (default 20, capped at 100) + --kind PREFIX Slug-prefix filter (e.g. personal, wiki/people) + --slug-prefix P Same as --kind + --json JSON output for agents + --help, -h Show this help +`; + +export async function runSalience(engine: BrainEngine, args: string[]): Promise { + const parsed = parseArgs(args); + if ('help' in parsed) { + console.log(HELP); + return; + } + const rows = await engine.getRecentSalience({ + days: parsed.days, + limit: parsed.limit, + slugPrefix: parsed.slugPrefix, + }); + if (parsed.json) { + console.log(JSON.stringify(rows, null, 2)); + return; + } + if (rows.length === 0) { + console.log('(no pages touched in the salience window)'); + return; + } + // Human format: rank | score | emotion | takes | slug — title + const header = `${pad('#', 3)} ${pad('score', 7)} ${pad('emo', 5)} ${pad('takes', 6)} slug — title`; + console.log(header); + console.log('-'.repeat(Math.min(80, header.length))); + rows.forEach((r, i) => { + const score = r.score.toFixed(3); + const emo = r.emotional_weight.toFixed(2); + const takes = String(r.take_count); + console.log(`${pad(String(i + 1), 3)} ${pad(score, 7)} ${pad(emo, 5)} ${pad(takes, 6)} ${r.slug} — ${r.title}`); + }); +} + +function pad(s: string, n: number): string { + return s.length >= n ? s : s + ' '.repeat(n - s.length); +} diff --git a/src/commands/transcripts.ts b/src/commands/transcripts.ts new file mode 100644 index 000000000..63306590a --- /dev/null +++ b/src/commands/transcripts.ts @@ -0,0 +1,94 @@ +/** + * gbrain transcripts — Recent raw conversation transcripts. + * + * Local-only: this command reads `.txt` files from the dream-cycle corpus + * directories. It exists as a CLI surface so humans can trigger the same + * read path the v0.29 `get_recent_transcripts` MCP op uses (which is itself + * gated on remote=false; subagents and MCP/HTTP callers cannot reach it). + * + * Usage: + * gbrain transcripts recent # last 7 days, summaries + * gbrain transcripts recent --days 14 + * gbrain transcripts recent --full # full content (capped at 100KB/file) + * gbrain transcripts recent --json + */ + +import type { BrainEngine } from '../core/engine.ts'; + +interface RunOpts { + days?: number; + full?: boolean; + limit?: number; + json?: boolean; +} + +function parseArgs(args: string[]): RunOpts | { help: true } { + const opts: RunOpts = {}; + for (let i = 0; i < args.length; i++) { + const a = args[i]; + if (a === '--help' || a === '-h') return { help: true }; + if (a === '--json') { opts.json = true; continue; } + if (a === '--full') { opts.full = true; continue; } + if (a === '--days') { + const n = parseInt(args[++i] ?? '', 10); + if (Number.isFinite(n) && n >= 0) opts.days = n; + continue; + } + if (a === '--limit') { + const n = parseInt(args[++i] ?? '', 10); + if (Number.isFinite(n) && n > 0) opts.limit = n; + continue; + } + } + return opts; +} + +const HELP = `Usage: gbrain transcripts recent [options] + +Recent raw conversation transcripts (NOT polished reflections). Reads from +the dream-cycle corpus dirs (dream.synthesize.session_corpus_dir and +dream.synthesize.meeting_transcripts_dir). + +Options: + --days N Window in days (default 7) + --limit N Max transcripts (default 50) + --full Return full content (default: ~300-char summary). Capped 100KB/file. + --json JSON output for agents + --help, -h Show this help + +Note: dream-generated outputs (frontmatter dream_generated: true) are skipped. +`; + +export async function runTranscripts(engine: BrainEngine, args: string[]): Promise { + const sub = args[0]; + if (sub !== 'recent') { + console.log(HELP); + if (sub && sub !== '--help' && sub !== '-h') process.exitCode = 2; + return; + } + + const parsed = parseArgs(args.slice(1)); + if ('help' in parsed) { + console.log(HELP); + return; + } + const { listRecentTranscripts } = await import('../core/transcripts.ts'); + const rows = await listRecentTranscripts(engine, { + days: parsed.days, + summary: !parsed.full, + limit: parsed.limit, + }); + if (parsed.json) { + console.log(JSON.stringify(rows, null, 2)); + return; + } + if (rows.length === 0) { + console.log('(no recent transcripts in the corpus dir)'); + return; + } + rows.forEach(r => { + const date = r.date ?? r.mtime.slice(0, 10); + console.log(`\n--- ${date} | ${r.path} | ${r.length} bytes ---`); + console.log(r.summary); + }); +} diff --git a/src/core/backfill-effective-date.ts b/src/core/backfill-effective-date.ts new file mode 100644 index 000000000..807d45142 --- /dev/null +++ b/src/core/backfill-effective-date.ts @@ -0,0 +1,261 @@ +/** + * v0.29.1 — Backfill effective_date / effective_date_source for existing + * pages. + * + * Migration v38 added the columns; they're NULL for rows imported before + * v0.29.1. This walks every page in keyset-paginated batches, runs the + * `computeEffectiveDate` precedence chain, and UPDATEs in place. + * + * Resumable: stores `last_processed_id` in the `config` table after each + * batch. A killed process can re-run and pick up where it left off without + * re-doing rows. Idempotent: even a full re-walk produces the same writes. + * + * Postgres only sets `SET LOCAL statement_timeout = '600s'` per batch (does + * NOT refuse the migration on low session settings — codex pass-2 #16). + * + * Pure library function — same code path used by the v0_29_1 orchestrator + * AND the `gbrain reindex-frontmatter` CLI command (added in commit 4). + * + * Note: the `import_filename` column stays NULL on backfilled rows. We + * don't have the original filename for pre-v0.29.1 imports (codex pass-1 + * finding #6). For `daily/`/`meetings/` slugs whose filename-derived date + * IS in the slug tail, computeEffectiveDate falls through to the slug-tail + * heuristic via `slug.split('/').pop()` in importFromContent's caller path + * — but the orchestrator passes the slug-tail explicitly here so backfilled + * rows behave the same as fresh imports for those prefixes. + */ + +import type { BrainEngine } from './engine.ts'; +import { computeEffectiveDate } from './effective-date.ts'; +import type { EffectiveDateSource } from './types.ts'; + +const BATCH_SIZE = 1000; +const CHECKPOINT_KEY = 'backfill.effective_date.last_id'; + +export interface BackfillOpts { + /** Limit total rows touched (testing). Undefined = no cap. */ + maxRows?: number; + /** Restart from id=0 even if a checkpoint exists. */ + fresh?: boolean; + /** Don't write; report what would happen. */ + dryRun?: boolean; + /** Per-batch progress callback. */ + onBatch?: (info: { batch: number; lastId: number; rowsTouched: number; cumulative: number }) => void; + /** + * Optional slug-prefix filter (e.g. 'meetings/') so the CLI command can + * scope to a subset. Undefined = no filter. + */ + slugPrefix?: string; + /** + * When true, recompute even if existing effective_date matches what + * the chain would produce. Default false (no-op-on-equal saves writes). + */ + force?: boolean; +} + +export interface BackfillResult { + /** Total rows examined across all batches. */ + examined: number; + /** Rows where effective_date was actually written (changed or newly computed). */ + updated: number; + /** Rows that fell through the chain to 'fallback' (matches updated_at/created_at). */ + fallback: number; + /** Final last_processed_id (for resume / debugging). */ + lastId: number; + /** Total wall-clock seconds. */ + durationSec: number; +} + +interface PageRow { + id: number; + slug: string; + frontmatter: unknown; + import_filename: string | null; + effective_date: string | null; + effective_date_source: EffectiveDateSource | null; + created_at: string; + updated_at: string; +} + +function parseFrontmatter(raw: unknown): Record { + if (raw == null) return {}; + if (typeof raw === 'string') { + try { return JSON.parse(raw) as Record; } + catch { return {}; } + } + if (typeof raw === 'object') return raw as Record; + return {}; +} + +async function getCheckpoint(engine: BrainEngine, fresh: boolean): Promise { + if (fresh) return 0; + try { + const rows = await engine.executeRaw<{ value: string }>( + `SELECT value FROM config WHERE key = $1 LIMIT 1`, + [CHECKPOINT_KEY], + ); + if (rows.length === 0) return 0; + const n = Number(rows[0].value); + return Number.isFinite(n) && n >= 0 ? n : 0; + } catch { + return 0; + } +} + +async function setCheckpoint(engine: BrainEngine, lastId: number): Promise { + try { + await engine.executeRaw( + `INSERT INTO config (key, value) VALUES ($1, $2) + ON CONFLICT (key) DO UPDATE SET value = EXCLUDED.value`, + [CHECKPOINT_KEY, String(lastId)], + ); + } catch { + // Best effort. Failure to checkpoint just means re-walk on next run; + // doesn't corrupt state. + } +} + +async function clearCheckpoint(engine: BrainEngine): Promise { + try { + await engine.executeRaw(`DELETE FROM config WHERE key = $1`, [CHECKPOINT_KEY]); + } catch { + // Same — best effort. + } +} + +export async function backfillEffectiveDate( + engine: BrainEngine, + opts: BackfillOpts = {}, +): Promise { + const start = Date.now(); + const slugPrefix = opts.slugPrefix?.replace(/[\\%_]/g, (c) => '\\' + c) ?? null; + + let lastId = await getCheckpoint(engine, opts.fresh ?? false); + let examined = 0; + let updated = 0; + let fallback = 0; + let batchNum = 0; + + // Per-engine statement_timeout boost. Postgres can wedge on a slow + // batch otherwise; PGLite ignores SET LOCAL outside transactions but + // doesn't have the timeout problem in the first place (single writer). + const isPostgres = engine.kind === 'postgres'; + + while (true) { + if (opts.maxRows && examined >= opts.maxRows) break; + + const limit = opts.maxRows + ? Math.min(BATCH_SIZE, opts.maxRows - examined) + : BATCH_SIZE; + + // Keyset pagination: WHERE id > last_id ORDER BY id LIMIT N. Single-direction + // walk; safe under concurrent inserts (new rows show up at the tail). + const slugFilter = slugPrefix + ? `AND slug LIKE $2 ESCAPE '\\\\'` + : ''; + const params: unknown[] = [lastId]; + if (slugPrefix) params.push(slugPrefix + '%'); + params.push(limit); + const limitParam = `$${params.length}`; + + const rows = await engine.executeRaw( + `SELECT id, slug, frontmatter, import_filename, effective_date, effective_date_source, created_at, updated_at + FROM pages + WHERE id > $1 ${slugFilter} + ORDER BY id + LIMIT ${limitParam}`, + params, + ); + + if (rows.length === 0) break; + + examined += rows.length; + let touched = 0; + + if (!opts.dryRun) { + // Compute effective_date for each row, then UPDATE in a batch wrapped + // in its own transaction (so SET LOCAL statement_timeout scopes to it). + // postgres.js's `transaction` would be cleaner but we're using executeRaw + // for engine portability; explicit BEGIN/COMMIT does the same on both. + if (isPostgres) { + await engine.executeRaw(`BEGIN`); + await engine.executeRaw(`SET LOCAL statement_timeout = '600s'`); + } + + try { + for (const r of rows) { + const fm = parseFrontmatter(r.frontmatter); + const filename = r.import_filename + || (r.slug.includes('/') ? r.slug.split('/').pop()! : r.slug); + const computed = computeEffectiveDate({ + slug: r.slug, + frontmatter: fm, + filename, + updatedAt: new Date(r.updated_at), + createdAt: new Date(r.created_at), + }); + + // No-op-on-equal: skip the UPDATE if existing matches (saves write + // amplification on re-runs). `force: true` bypasses. + const existingMs = r.effective_date ? new Date(r.effective_date).getTime() : null; + const computedMs = computed.date ? computed.date.getTime() : null; + const datesMatch = existingMs === computedMs; + const sourcesMatch = (r.effective_date_source ?? null) === (computed.source ?? null); + + if (!opts.force && datesMatch && sourcesMatch) continue; + + await engine.executeRaw( + `UPDATE pages SET effective_date = $1::timestamptz, effective_date_source = $2 WHERE id = $3`, + [computed.date ? computed.date.toISOString() : null, computed.source, r.id], + ); + touched++; + if (computed.source === 'fallback') fallback++; + } + + if (isPostgres) await engine.executeRaw(`COMMIT`); + } catch (e) { + if (isPostgres) { + try { await engine.executeRaw(`ROLLBACK`); } catch { /* ignore */ } + } + throw e; + } + } else { + // Dry run: still count what WOULD change. + for (const r of rows) { + const fm = parseFrontmatter(r.frontmatter); + const filename = r.import_filename + || (r.slug.includes('/') ? r.slug.split('/').pop()! : r.slug); + const computed = computeEffectiveDate({ + slug: r.slug, + frontmatter: fm, + filename, + updatedAt: new Date(r.updated_at), + createdAt: new Date(r.created_at), + }); + const existingMs = r.effective_date ? new Date(r.effective_date).getTime() : null; + const computedMs = computed.date ? computed.date.getTime() : null; + if (existingMs !== computedMs || (r.effective_date_source ?? null) !== (computed.source ?? null)) { + touched++; + } + if (computed.source === 'fallback') fallback++; + } + } + + updated += touched; + lastId = rows[rows.length - 1].id; + batchNum++; + if (!opts.dryRun) await setCheckpoint(engine, lastId); + opts.onBatch?.({ batch: batchNum, lastId, rowsTouched: touched, cumulative: examined }); + } + + // Walk done; clear the checkpoint so the next manual run starts fresh. + if (!opts.dryRun) await clearCheckpoint(engine); + + return { + examined, + updated, + fallback, + lastId, + durationSec: (Date.now() - start) / 1000, + }; +} diff --git a/src/core/cycle.ts b/src/core/cycle.ts index 18b44668c..2eb110670 100644 --- a/src/core/cycle.ts +++ b/src/core/cycle.ts @@ -22,8 +22,9 @@ * │ Phase 6: patterns (v0.23: cross-session themes; │ * │ MUST be after extract so │ * │ graph state is fresh) │ - * │ Phase 7: embed --stale (DB writes) │ - * │ Phase 8: orphans (DB read, report only) │ + * │ Phase 7: recompute_emotional_weight (v0.29: DB writes) │ + * │ Phase 8: embed --stale (DB writes) │ + * │ Phase 9: orphans (DB read, report only) │ * └───────────────────────────────────────────────────────────┘ * * COORDINATION: @@ -52,7 +53,7 @@ import { getCliOptions, cliOptsToProgressOptions } from './cli-options.ts'; // ─── Types ───────────────────────────────────────────────────────── -export type CyclePhase = 'lint' | 'backlinks' | 'sync' | 'synthesize' | 'extract' | 'patterns' | 'embed' | 'orphans' | 'purge'; +export type CyclePhase = 'lint' | 'backlinks' | 'sync' | 'synthesize' | 'extract' | 'patterns' | 'recompute_emotional_weight' | 'embed' | 'orphans' | 'purge'; export const ALL_PHASES: CyclePhase[] = [ 'lint', @@ -61,6 +62,9 @@ export const ALL_PHASES: CyclePhase[] = [ 'synthesize', 'extract', 'patterns', + // v0.29 — runs AFTER extract + synthesize so it sees the union of + // sync-touched + synthesize-written pages with fresh tag + take state. + 'recompute_emotional_weight', 'embed', 'orphans', // v0.26.5: hard-deletes soft-deleted pages and expired archived sources past @@ -83,6 +87,8 @@ const NEEDS_LOCK_PHASES: ReadonlySet = new Set([ 'synthesize', 'extract', 'patterns', + // v0.29 — writes pages.emotional_weight column. + 'recompute_emotional_weight', 'embed', 'purge', ]); @@ -144,6 +150,8 @@ export interface CycleReport { synth_pages_written: number; /** v0.23: number of pattern pages written/updated by patterns phase. */ patterns_written: number; + /** v0.29: number of pages whose emotional_weight was (re)computed. */ + pages_emotional_weight_recomputed: number; /** v0.26.5: number of source rows hard-deleted by the purge phase. */ purged_sources_count: number; /** v0.26.5: number of page rows hard-deleted by the purge phase. */ @@ -888,8 +896,11 @@ export async function runCycle( } // ── Phase 3: sync ─────────────────────────────────────────── - // Track which slugs sync touched so extract can run incrementally. + // Track which slugs sync touched so extract can run incrementally, + // and which slugs synthesize wrote so recompute_emotional_weight can + // pick up the union of (sync ∪ synthesize) for v0.29 incremental mode. let syncPagesAffected: string[] | undefined; + let synthesizeWrittenSlugs: string[] | undefined; if (phases.includes('sync')) { checkAborted(opts.signal); if (!engine) { @@ -937,6 +948,11 @@ export async function runCycle( })); result.duration_ms = duration_ms; phaseResults.push(result); + // v0.29: capture synthesize-written slugs so the recompute_emotional_weight + // phase can union them with sync's pagesAffected for incremental mode. + if (result.details && Array.isArray(result.details.written_slugs)) { + synthesizeWrittenSlugs = result.details.written_slugs as string[]; + } progress.finish(); } await safeYield(opts.yieldBetweenPhases); @@ -995,7 +1011,47 @@ export async function runCycle( await safeYield(opts.yieldBetweenPhases); } - // ── Phase 7: embed ────────────────────────────────────────── + // ── Phase 7: recompute_emotional_weight (v0.29) ───────────── + // Runs AFTER extract + synthesize so it sees fresh tags + takes for + // every page touched in this cycle. Incremental mode uses union(sync, + // synthesize); full mode walks every page in the brain. + if (phases.includes('recompute_emotional_weight')) { + checkAborted(opts.signal); + if (!engine) { + phaseResults.push({ + phase: 'recompute_emotional_weight', + status: 'skipped', + duration_ms: 0, + summary: 'no database connected', + details: { reason: 'no_database' }, + }); + } else { + progress.start('cycle.recompute_emotional_weight'); + const { runPhaseRecomputeEmotionalWeight } = await import('./cycle/recompute-emotional-weight.ts'); + // Determine incremental vs full mode. If sync OR synthesize ran in this + // cycle, do incremental over their union. If neither phase ran (e.g., + // user passed `--phase recompute_emotional_weight`), do full walk. + const incremental: string[] | undefined = + (syncPagesAffected || synthesizeWrittenSlugs) + ? Array.from(new Set([ + ...(syncPagesAffected ?? []), + ...(synthesizeWrittenSlugs ?? []), + ])) + : undefined; + const { result, duration_ms } = await timePhase(() => + runPhaseRecomputeEmotionalWeight(engine, { + dryRun, + affectedSlugs: incremental, + }), + ); + result.duration_ms = duration_ms; + phaseResults.push(result); + progress.finish(); + } + await safeYield(opts.yieldBetweenPhases); + } + + // ── Phase 8: embed ────────────────────────────────────────── if (phases.includes('embed')) { checkAborted(opts.signal); if (!engine) { @@ -1016,7 +1072,7 @@ export async function runCycle( await safeYield(opts.yieldBetweenPhases); } - // ── Phase 8: orphans ──────────────────────────────────────── + // ── Phase 9: orphans ──────────────────────────────────────── if (phases.includes('orphans')) { checkAborted(opts.signal); if (!engine) { @@ -1094,6 +1150,7 @@ function emptyTotals(): CycleReport['totals'] { transcripts_processed: 0, synth_pages_written: 0, patterns_written: 0, + pages_emotional_weight_recomputed: 0, purged_sources_count: 0, purged_pages_count: 0, }; @@ -1123,6 +1180,8 @@ function extractTotals(phases: PhaseResult[]): CycleReport['totals'] { t.synth_pages_written = Number(p.details.pages_written ?? 0); } else if (p.phase === 'patterns' && p.details) { t.patterns_written = Number(p.details.patterns_written ?? 0); + } else if (p.phase === 'recompute_emotional_weight' && p.details) { + t.pages_emotional_weight_recomputed = Number(p.details.pages_recomputed ?? 0); } else if (p.phase === 'purge' && p.details) { t.purged_sources_count = Number(p.details.purged_sources_count ?? 0); t.purged_pages_count = Number(p.details.purged_pages_count ?? 0); @@ -1144,6 +1203,7 @@ function deriveStatus(phases: PhaseResult[], totals: CycleReport['totals']): Cyc totals.backlinks_added > 0 || totals.pages_synced > 0 || totals.pages_extracted > 0 || - totals.pages_embedded > 0; + totals.pages_embedded > 0 || + totals.pages_emotional_weight_recomputed > 0; return anyWork ? 'ok' : 'clean'; } diff --git a/src/core/cycle/anomaly.ts b/src/core/cycle/anomaly.ts new file mode 100644 index 000000000..c824fc267 --- /dev/null +++ b/src/core/cycle/anomaly.ts @@ -0,0 +1,128 @@ +/** + * v0.29 — Anomaly detection: statistical helpers for the `find_anomalies` op. + * + * Pure functions over densified daily-count buckets. The engine layer runs the + * SQL (CTE-shaped, with `generate_series` zero-fill so rare cohorts don't get + * sparse-day biased baselines per codex C4#6) and hands the results here. This + * keeps `findAnomalies` mostly testable without a database. + * + * Cohort kinds: tag, type. Year cohort is deferred to v0.30 pending proper + * frontmatter date-field detection. + */ + +import type { AnomalyResult } from '../types.ts'; + +/** One row of the densified daily-count series for a single cohort key. */ +export interface CohortDayRow { + cohort_kind: 'tag' | 'type'; + cohort_value: string; + /** ISO date (YYYY-MM-DD). */ + day: string; + /** Distinct pages touched in this cohort on `day`. Zero if no activity. */ + count: number; +} + +/** "Today" current-window count per cohort plus the page slugs that drove it. */ +export interface CohortTodayRow { + cohort_kind: 'tag' | 'type'; + cohort_value: string; + count: number; + page_slugs: string[]; +} + +/** + * Mean and (sample) stddev of a number array. Returns `(0, 0)` for empty + * input. Uses the sample stddev (n-1 denominator) so a single-sample baseline + * doesn't claim zero variance. + */ +export function meanStddev(samples: number[]): { mean: number; stddev: number } { + if (samples.length === 0) return { mean: 0, stddev: 0 }; + const sum = samples.reduce((a, b) => a + b, 0); + const mean = sum / samples.length; + if (samples.length === 1) return { mean, stddev: 0 }; + const sqSum = samples.reduce((a, b) => a + (b - mean) * (b - mean), 0); + const variance = sqSum / (samples.length - 1); + return { mean, stddev: Math.sqrt(variance) }; +} + +/** + * Compute anomaly results from densified baseline buckets + today's counts. + * + * For each cohort: + * 1. Compute (mean, stddev) over the baseline daily counts. + * 2. If stddev > 0: anomalous when `today.count > mean + sigma*stddev`. + * sigma_observed = (today.count - mean) / stddev. + * 3. If stddev == 0: small-sample fallback — anomalous when + * `today.count > mean + 1`. sigma_observed treated as a finite proxy + * `today.count - mean` so callers still get a usable sort key. + * + * Cohorts with no baseline rows AND no today rows are skipped. Cohorts + * appearing only in `today` (a brand-new cohort) get a baseline_mean of 0 + * — they're surfaced as anomalies whenever today.count >= 2 (mean+1 fallback). + * + * Returns top `limit` rows sorted by `sigma_observed` descending. Each row + * caps `page_slugs` at 50 entries. + * + * @param baseline densified rows over the lookback window, grouped by cohort × day. + * @param today rows for the target day, grouped by cohort. + * @param sigma threshold multiplier (default 3.0). + * @param limit max anomalies to return (default 20). + */ +export function computeAnomaliesFromBuckets( + baseline: CohortDayRow[], + today: CohortTodayRow[], + sigma: number, + limit: number = 20, +): AnomalyResult[] { + // Group baseline samples by (cohort_kind, cohort_value). + const baselineByCohort = new Map(); + for (const row of baseline) { + const key = cohortKey(row.cohort_kind, row.cohort_value); + const list = baselineByCohort.get(key); + if (list) { + list.push(row.count); + } else { + baselineByCohort.set(key, [row.count]); + } + } + + const out: AnomalyResult[] = []; + for (const t of today) { + const key = cohortKey(t.cohort_kind, t.cohort_value); + const samples = baselineByCohort.get(key) ?? []; + const { mean, stddev } = meanStddev(samples); + + let isAnomaly: boolean; + let sigmaObserved: number; + if (stddev > 0) { + const threshold = mean + sigma * stddev; + isAnomaly = t.count > threshold; + sigmaObserved = (t.count - mean) / stddev; + } else { + // Zero-stddev fallback (or empty baseline). Sigma is undefined; we use + // (count - mean) as a finite sort proxy and require count > mean + 1 + // to avoid surfacing every 1-page-touched cohort as anomalous. + isAnomaly = t.count > mean + 1; + sigmaObserved = t.count - mean; + } + + if (!isAnomaly) continue; + out.push({ + cohort_kind: t.cohort_kind, + cohort_value: t.cohort_value, + count: t.count, + baseline_mean: mean, + baseline_stddev: stddev, + sigma_observed: sigmaObserved, + page_slugs: t.page_slugs.slice(0, 50), + }); + } + + out.sort((a, b) => b.sigma_observed - a.sigma_observed); + return out.slice(0, limit); +} + +function cohortKey(kind: string, value: string): string { + // \x1f (unit separator) — a byte that can't appear in tags or PageType values. + return `${kind}\x1f${value}`; +} diff --git a/src/core/cycle/emotional-weight.ts b/src/core/cycle/emotional-weight.ts new file mode 100644 index 000000000..4c7b64dd0 --- /dev/null +++ b/src/core/cycle/emotional-weight.ts @@ -0,0 +1,142 @@ +/** + * v0.29 — Emotional weight: deterministic 0..1 score for each page, computed + * from tags + active takes during the dream cycle's recompute_emotional_weight + * phase. Feeds the salience query (`get_recent_salience`) so pages with high + * emotional weight outrank busy-but-shallow ones in "what's been going on?" + * style queries. + * + * Pure function, no DB. The cycle phase loads inputs in batch via + * `engine.batchLoadEmotionalInputs` and writes results in batch via + * `engine.setEmotionalWeightBatch`. + * + * Tunable: the `HIGH_EMOTION_TAGS` seed list below is the default. Users + * override via the `emotional_weight.high_tags` config key (array of strings). + * See `loadHighEmotionTags` for the resolution path. + */ + +/** + * Default high-emotion tag seed list. Pages with any tag in this set get the + * tag-emotion boost in the formula below. Override via config key + * `emotional_weight.high_tags` to add domain-specific tags (e.g. health + * conditions, family member names, project names tied to grief / loss). + * + * Anglocentric and personal-life-biased on purpose: this is the v1 default + * for someone who keeps a personal brain. Override unconditionally at install + * time if your brain is mostly work-life. + */ +export const HIGH_EMOTION_TAGS: ReadonlySet = new Set([ + 'family', + 'marriage', + 'wedding', + 'loss', + 'death', + 'grief', + 'relationship', + 'love', + 'mental-health', + 'health', + 'illness', + 'birth', + 'children', + 'kids', + 'parents', +]); + +/** + * Holder name treated as "the user" for the Garry-as-holder ratio. Configurable + * via the `emotional_weight.user_holder` config key (defaults to 'garry' to + * match the v0.28 schema's takes table convention). + */ +export const DEFAULT_USER_HOLDER = 'garry'; + +export interface EmotionalWeightTake { + holder: string; + weight: number; + kind: string; + active: boolean; +} + +export interface EmotionalWeightInput { + tags: readonly string[]; + takes: readonly EmotionalWeightTake[]; +} + +export interface EmotionalWeightOpts { + /** Override the default HIGH_EMOTION_TAGS set. Tag matching is case-insensitive. */ + highEmotionTags?: ReadonlySet; + /** Override the default user holder name (used in the Garry-as-holder ratio). */ + userHolder?: string; +} + +/** + * Compute emotional weight in [0..1] from a page's tags + active takes. + * + * Formula (sum capped at 1.0): + * 1) Tag emotion boost max 0.5 (any matching high-emotion tag) + * 2) Take density max 0.3 (0.1 per active take, capped) + * 3) Take avg weight max 0.1 (avg of take.weight, scaled) + * 4) User-holder ratio max 0.1 (active takes by user / total active) + * + * Why these numbers: + * - Tag emotion is the strongest signal (0.5 cap) because tags are an explicit + * user act of categorization. A page tagged `wedding` is *about* something + * emotionally weighty by construction. + * - Take density (0.3) covers the case of pages with no emotion-tag but lots + * of opinions / hot-take attention (Garry's "I have a bunch of takes about + * this person/company" signal). + * - Avg weight (0.1) captures take confidence; high-confidence takes amplify + * density. + * - User-holder ratio (0.1) preserves the personal-vs-other distinction. + * A page where Garry has takes outweighs one where only third-party holders + * are recorded. + * + * Returns exactly 0.0 for empty inputs (no tags, no takes) so default-row + * behavior survives the formula. + */ +export function computeEmotionalWeight( + input: EmotionalWeightInput, + opts: EmotionalWeightOpts = {}, +): number { + const tagSet = opts.highEmotionTags ?? HIGH_EMOTION_TAGS; + const userHolder = (opts.userHolder ?? DEFAULT_USER_HOLDER).toLowerCase(); + + const tags = input.tags ?? []; + const allTakes = input.takes ?? []; + const takes = allTakes.filter((t) => t.active); + + // 1) Tag emotion boost — case-insensitive match. + let tagBoost = 0; + for (const t of tags) { + if (tagSet.has(t.toLowerCase())) { + tagBoost = 0.5; + break; + } + } + + // 2) Take density: 0.1 per active take, capped at 0.3. + const takeDensity = Math.min(takes.length * 0.1, 0.3); + + // 3) Take avg weight, scaled into 0..0.1. + let takeAvgWeight = 0; + if (takes.length > 0) { + const sum = takes.reduce((acc, t) => acc + clamp01(t.weight), 0); + takeAvgWeight = (sum / takes.length) * 0.1; + } + + // 4) User-holder ratio over active takes, scaled into 0..0.1. + let userHolderRatio = 0; + if (takes.length > 0) { + const userTakes = takes.filter((t) => t.holder?.toLowerCase() === userHolder).length; + userHolderRatio = (userTakes / takes.length) * 0.1; + } + + const total = tagBoost + takeDensity + takeAvgWeight + userHolderRatio; + return Math.max(0, Math.min(1, total)); +} + +function clamp01(n: number): number { + if (!Number.isFinite(n)) return 0; + if (n < 0) return 0; + if (n > 1) return 1; + return n; +} diff --git a/src/core/cycle/recompute-emotional-weight.ts b/src/core/cycle/recompute-emotional-weight.ts new file mode 100644 index 000000000..4a96d6fba --- /dev/null +++ b/src/core/cycle/recompute-emotional-weight.ts @@ -0,0 +1,134 @@ +/** + * v0.29 — Recompute emotional weight phase. Runs AFTER extract + synthesize so + * it sees the union of (sync-touched, synthesize-written) pages with fresh + * tag + take state. Pure deterministic computation; no LLM calls. + * + * Two SQL round-trips total regardless of brain size (codex C4#3, C4#4): + * 1. batchLoadEmotionalInputs — single CTE-shaped read with per-table + * pre-aggregates so a page × N tags × M takes never produces N×M rows. + * 2. setEmotionalWeightBatch — composite-keyed (slug, source_id) UPDATE + * FROM unnest so multi-source brains can't get cross-source fan-out. + * + * In incremental mode (`affectedSlugs` non-empty), only those pages are + * touched. In full mode (`affectedSlugs` undefined or null) every page in + * the brain is recomputed — this is the path users hit on first upgrade + * via `gbrain dream --phase recompute_emotional_weight`. + * + * Target wall-clock budget: <5s on a 1000-page fixture; <60s on a 50K-page + * real brain. Catastrophic-exception path returns a 'fail' PhaseResult so + * the cycle continues to the next phase. + */ + +import type { BrainEngine } from '../engine.ts'; +import type { PhaseResult, PhaseError } from '../cycle.ts'; +import { computeEmotionalWeight } from './emotional-weight.ts'; +import type { GBrainConfig } from '../config.ts'; + +export interface RecomputeEmotionalWeightOpts { + /** When false, the phase reads + computes but skips the UPDATE. */ + dryRun?: boolean; + /** + * Slugs to recompute. Undefined / empty array = full brain recompute. + * Caller passes `union(syncPagesAffected, synthesizeWrittenSlugs)` for + * the incremental path. + */ + affectedSlugs?: string[]; + /** GBrain config for high_emotion_tags + user_holder overrides. */ + config?: GBrainConfig; +} + +export interface RecomputeEmotionalWeightResult extends PhaseResult { + /** Number of pages whose emotional_weight was (re)computed. */ + pages_recomputed: number; +} + +export async function runPhaseRecomputeEmotionalWeight( + engine: BrainEngine, + opts: RecomputeEmotionalWeightOpts, +): Promise { + const start = Date.now(); + try { + // Resolve override tag list + user-holder from config (optional). + const overrideTags = await engine.getConfig('emotional_weight.high_tags'); + const userHolder = await engine.getConfig('emotional_weight.user_holder'); + let highEmotionTags: ReadonlySet | undefined; + if (overrideTags) { + try { + const parsed = JSON.parse(overrideTags) as unknown; + if (Array.isArray(parsed) && parsed.every(t => typeof t === 'string')) { + highEmotionTags = new Set(parsed.map(t => t.toLowerCase())); + } + } catch { + // Bad JSON — fall back to default seed list. The doctor check + // (added separately) will surface the parse error. + } + } + + // Incremental path: empty array means "no changes touched" — record + // a zero-work success and return without touching the DB. + if (Array.isArray(opts.affectedSlugs) && opts.affectedSlugs.length === 0) { + return result('ok', 'recompute_emotional_weight (incremental, 0 slugs)', 0, { + mode: 'incremental', + pages_recomputed: 0, + }, start); + } + + const inputs = await engine.batchLoadEmotionalInputs(opts.affectedSlugs); + const writes = inputs.map(row => ({ + slug: row.slug, + source_id: row.source_id, + weight: computeEmotionalWeight( + { tags: row.tags, takes: row.takes }, + { highEmotionTags, userHolder: userHolder ?? undefined }, + ), + })); + + if (opts.dryRun) { + return result('ok', `recompute_emotional_weight (dry-run, ${writes.length} pages)`, writes.length, { + mode: opts.affectedSlugs ? 'incremental' : 'full', + pages_recomputed: writes.length, + dry_run: true, + }, start); + } + + const updated = await engine.setEmotionalWeightBatch(writes); + + return result('ok', `recompute_emotional_weight (${updated} pages)`, updated, { + mode: opts.affectedSlugs ? 'incremental' : 'full', + pages_recomputed: updated, + }, start); + } catch (e) { + const msg = e instanceof Error ? e.message : String(e); + const err: PhaseError = { + class: 'InternalError', + code: 'RECOMPUTE_EMOTIONAL_WEIGHT_FAIL', + message: msg || 'recompute_emotional_weight phase threw', + }; + return { + phase: 'recompute_emotional_weight', + status: 'fail', + duration_ms: Date.now() - start, + summary: 'recompute_emotional_weight failed', + details: { error: err }, + error: err, + pages_recomputed: 0, + }; + } +} + +function result( + status: 'ok', + summary: string, + pagesRecomputed: number, + details: Record, + start: number, +): RecomputeEmotionalWeightResult { + return { + phase: 'recompute_emotional_weight', + status, + duration_ms: Date.now() - start, + summary, + details, + pages_recomputed: pagesRecomputed, + }; +} diff --git a/src/core/cycle/synthesize.ts b/src/core/cycle/synthesize.ts index 4bb9f0415..d3a2025c4 100644 --- a/src/core/cycle/synthesize.ts +++ b/src/core/cycle/synthesize.ts @@ -243,6 +243,10 @@ export async function runPhaseSynthesize( transcripts_discovered: transcripts.length, transcripts_processed: worthProcessing.length, pages_written: writtenSlugs.length, + // v0.29: emit the slug list so the recompute_emotional_weight phase can + // union with sync's pagesAffected and recompute weights for every page + // synthesize wrote in this cycle. + written_slugs: writtenSlugs, reverse_write_count: reverseWriteCount, child_outcomes: childOutcomes, summary_slug: summarySlug, diff --git a/src/core/effective-date.ts b/src/core/effective-date.ts new file mode 100644 index 000000000..825413928 --- /dev/null +++ b/src/core/effective-date.ts @@ -0,0 +1,158 @@ +/** + * v0.29.1 — Compute a page's effective_date from frontmatter precedence. + * + * The "effective date" is the answer to "when was this page about?" It's + * NOT updated_at (which churns from auto-link) and NOT created_at (which + * is the row insert time). It's the user's stated content date. + * + * Precedence chain (default order): + * 1. frontmatter.event_date — meeting / event pages + * 2. frontmatter.date — dated essays + * 3. frontmatter.published — writing/ + * 4. filename-date — leading YYYY-MM-DD in basename + * 5. updated_at — fallback + * 6. created_at — last resort (only if updated_at NULL) + * + * Per-prefix override: for `daily/` and `meetings/` slug prefixes, the + * filename-date jumps to position 1 — the filename is the user's primary + * signal there ("daily/2024-03-15.md" the FILE date matters more than any + * frontmatter the user pasted). + * + * Returns BOTH the parsed Date and the source label so the doctor's + * `effective_date_health` check can detect "fell back to updated_at" rows + * that look populated but are functionally equivalent to a NULL. + * + * Range validation: parsed value must be in [1990-01-01, NOW + 1 year]. + * Out-of-range values are dropped (the chain falls through to the next + * element). NaN / unparseable strings drop the same way. + * + * Pure function. No DB. Tested in test/effective-date.test.ts. + */ + +import type { EffectiveDateSource } from './types.ts'; + +export interface EffectiveDateResult { + date: Date | null; + source: EffectiveDateSource | null; +} + +export interface ComputeEffectiveDateOpts { + slug: string; + frontmatter: Record; + /** Basename without extension, e.g. "2024-03-15-acme-call". May be null/empty. */ + filename?: string | null; + updatedAt: Date; + createdAt: Date; +} + +/** + * Slug prefixes where the filename date wins over frontmatter dates. The + * user's primary signal in these directories is the filename, not arbitrary + * frontmatter the importer might have copied. + * + * Hardcoded in v0.29.1 (commit 2). v0.29.1 commit 5 introduces the + * recency-decay map; we could move this list there if we wanted user-tunable + * filename-first prefixes, but the daily/ + meetings/ defaults are stable + * enough that hardcoding is correct. + */ +const FILENAME_FIRST_PREFIXES = ['daily/', 'meetings/']; + +const MIN_DATE_MS = Date.UTC(1990, 0, 1); +const FILENAME_DATE_RE = /^(\d{4}-\d{2}-\d{2})/; + +function maxDateMs(): number { + // NOW + 1 year, computed at call time so tests with a mocked Date.now() + // see a moving boundary. Pages dated > 1 year in the future are almost + // always corrupt (epoch math gone wrong, typoed century, bad parse). + return Date.now() + 365 * 24 * 60 * 60 * 1000; +} + +/** Parse a frontmatter value as a Date. Accepts Date instances, ISO strings, YYYY-MM-DD. Returns null on any failure. */ +export function parseDateLoose(value: unknown): Date | null { + if (value == null) return null; + if (value instanceof Date) { + return Number.isFinite(value.getTime()) ? value : null; + } + if (typeof value === 'string') { + const trimmed = value.trim(); + if (trimmed === '') return null; + const ms = Date.parse(trimmed); + if (!Number.isFinite(ms)) return null; + return new Date(ms); + } + if (typeof value === 'number') { + // Plausibility: numbers are usually ms since epoch but YAML can yield + // bare integers (year? month? day?) — accept only if the resulting Date + // falls inside the valid window. validateInRange catches the rest. + return Number.isFinite(value) ? new Date(value) : null; + } + return null; +} + +function validateInRange(d: Date | null): Date | null { + if (d === null) return null; + const ms = d.getTime(); + if (!Number.isFinite(ms)) return null; + if (ms < MIN_DATE_MS) return null; + if (ms > maxDateMs()) return null; + return d; +} + +function extractFilenameDate(filename: string | null | undefined): Date | null { + if (!filename) return null; + const m = filename.match(FILENAME_DATE_RE); + if (!m) return null; + return validateInRange(parseDateLoose(m[1])); +} + +function hasFilenameFirstPrefix(slug: string): boolean { + for (const p of FILENAME_FIRST_PREFIXES) { + if (slug.startsWith(p)) return true; + } + return false; +} + +/** + * Run the precedence chain. Returns the first valid (in-range) date and its + * source label. Falls all the way through to updated_at / created_at as + * 'fallback' when nothing in frontmatter or filename parses. + */ +export function computeEffectiveDate(opts: ComputeEffectiveDateOpts): EffectiveDateResult { + const { slug, frontmatter, filename, updatedAt, createdAt } = opts; + const filenameFirst = hasFilenameFirstPrefix(slug); + + const fmEvent = validateInRange(parseDateLoose(frontmatter.event_date)); + const fmDate = validateInRange(parseDateLoose(frontmatter.date)); + const fmPublished = validateInRange(parseDateLoose(frontmatter.published)); + const filenameDate = extractFilenameDate(filename); + + // Build the ordered candidate list. For filename-first prefixes + // (daily/, meetings/) the filename moves to the head of the chain. + const candidates: Array<{ date: Date | null; source: EffectiveDateSource }> = filenameFirst + ? [ + { date: filenameDate, source: 'filename' }, + { date: fmEvent, source: 'event_date' }, + { date: fmDate, source: 'date' }, + { date: fmPublished, source: 'published' }, + ] + : [ + { date: fmEvent, source: 'event_date' }, + { date: fmDate, source: 'date' }, + { date: fmPublished, source: 'published' }, + { date: filenameDate, source: 'filename' }, + ]; + + for (const c of candidates) { + if (c.date !== null) return { date: c.date, source: c.source }; + } + + // Fallback chain: updated_at, then created_at. Both are guaranteed + // non-null by the schema; the validation here is defensive against bad + // test fixtures. + const upd = validateInRange(updatedAt); + if (upd !== null) return { date: upd, source: 'fallback' }; + const cre = validateInRange(createdAt); + if (cre !== null) return { date: cre, source: 'fallback' }; + + return { date: null, source: null }; +} diff --git a/src/core/engine.ts b/src/core/engine.ts index ee476259e..6303849a0 100644 --- a/src/core/engine.ts +++ b/src/core/engine.ts @@ -12,6 +12,8 @@ import type { CodeEdgeInput, CodeEdgeResult, EvalCandidate, EvalCandidateInput, EvalCaptureFailure, EvalCaptureFailureReason, + SalienceOpts, SalienceResult, AnomaliesOpts, AnomalyResult, + EmotionalWeightInputRow, EmotionalWeightWriteRow, } from './types.ts'; /** @@ -411,6 +413,35 @@ export interface BrainEngine { * Slugs with zero inbound links are present in the map with value 0. */ getBacklinkCounts(slugs: string[]): Promise>; + /** + * v0.27.0: for a list of slugs, return their updated_at timestamps (or created_at fallback). + * Used by hybrid search recency boost. Single SQL query, not N+1. + * Slugs with no timestamp get no entry in the map. + * + * @deprecated v0.29.1: prefer getEffectiveDates (composite-keyed, multi-source-safe). + * Kept for back-compat with PR #618 callers. + */ + getPageTimestamps(slugs: string[]): Promise>; + /** + * v0.29.1: for a list of (slug, source_id) refs, return COALESCE(effective_date, + * updated_at) per ref. Single SQL query. Composite-keyed map (key format: + * `${source_id}::${slug}`) so multi-source brains don't conflate pages with + * the same slug across sources (codex pass-1 finding #3). + * + * Drives the new applyRecencyBoost post-fusion stage. Returns NULL for refs + * with no row; map omits them. + */ + getEffectiveDates(refs: Array<{slug: string; source_id: string}>): Promise>; + /** + * v0.29.1: for a list of (slug, source_id) refs, return the salience score + * (emotional_weight × 5 + ln(1 + take_count)) per ref. Single SQL query. + * Composite-keyed (`${source_id}::${slug}`) like getEffectiveDates. + * + * Drives the new applySalienceBoost post-fusion stage. Pages with no row + * (or zero emotional_weight + zero takes) get score = 0; the boost stage + * skips them. + */ + getSalienceScores(refs: Array<{slug: string; source_id: string}>): Promise>; /** * Return every page with no inbound links (from any source). * Domain comes from the frontmatter `domain` field (null if unset). @@ -642,4 +673,59 @@ export interface BrainEngine { logEvalCaptureFailure(reason: EvalCaptureFailureReason): Promise; /** Read capture failures within an optional time window. Used by `gbrain doctor`. */ listEvalCaptureFailures(filter?: { since?: Date }): Promise; + + // ============================================================ + // v0.29 — Salience + Anomaly Detection + // ============================================================ + // The brain surfaces what's unusual and emotionally charged without being + // asked. Cost: ~zero at query time (deterministic SQL), with backfill done + // during the new `recompute_emotional_weight` cycle phase. + + /** + * Batch-load tag + take inputs for the emotional-weight formula. One CTE-shaped + * query: `pages` LEFT JOIN aggregated `tags` and aggregated `takes` (each + * pre-aggregated in its own CTE so the page × N tags × M takes cartesian + * product is avoided). + * + * If `slugs` is undefined, returns inputs for every page in the brain + * (full-mode backfill). If provided, returns only matching slugs (incremental + * recompute after sync / synthesize touched specific pages). + * + * Multi-source-aware: each row carries its `source_id` so the matching + * `setEmotionalWeightBatch` UPDATE can composite-key correctly. + */ + batchLoadEmotionalInputs(slugs?: string[]): Promise; + + /** + * Apply pre-computed emotional weights in a single UPDATE. Composite-keyed + * on `(slug, source_id)` because `pages.slug` is only unique within a + * source — a slug-only UPDATE would fan out across sources, the same bug + * that the v0.18.0 link batches fixed for cross-source edges. + * + * Returns the count of rows actually updated. Pages whose `(slug, source_id)` + * tuple doesn't exist (race with delete) are silently skipped. + */ + setEmotionalWeightBatch(rows: EmotionalWeightWriteRow[]): Promise; + + /** + * Salience query: pages recently touched, ranked by a deterministic + * `(emotional_weight * 5) + ln(1 + take_count) + recency_decay` score. + * + * The handler computes the time boundary in JS (`now - days * 86400000`) + * and binds it as TIMESTAMPTZ so the SQL is identical across PGLite + + * Postgres (eng review D5 — avoids dialect drift on `interval` binding). + */ + getRecentSalience(opts: SalienceOpts): Promise; + + /** + * Anomaly detection: cohorts (tag, type) with unusually-high page activity + * on a target day vs baseline mean+stddev over the previous N days. Year + * cohort is deferred to v0.30 (slug-regex year extraction is fragile). + * + * Baseline densifies the day series via `generate_series` zero-fill so + * sparse-day rare cohorts don't look "normally active" — a sparse-day cohort + * with one touch in 30 days has a low baseline mean and high sigma at 7 touches, + * not a misleading mean of 1. + */ + findAnomalies(opts: AnomaliesOpts): Promise; } diff --git a/src/core/import-file.ts b/src/core/import-file.ts index 48e14b598..076af7a42 100644 --- a/src/core/import-file.ts +++ b/src/core/import-file.ts @@ -11,6 +11,7 @@ import { extractCodeRefs, imageOfCandidates } from './link-extraction.ts'; import { embedBatch, embedMultimodal } from './embedding.ts'; import { slugifyPath, slugifyCodePath, isCodeFilePath } from './sync.ts'; import type { ChunkInput, PageInput, PageType } from './types.ts'; +import { computeEffectiveDate } from './effective-date.ts'; /** * v0.20.0 Cathedral II Layer 8 D2 — markdown fence extraction helper. @@ -185,7 +186,15 @@ export async function importFromContent( engine: BrainEngine, slug: string, content: string, - opts: { noEmbed?: boolean } = {}, + opts: { + noEmbed?: boolean; + /** + * v0.29.1: basename without extension for filename-date precedence on + * `daily/`, `meetings/` slugs. importFromFile threads this from the + * disk path; the put_page MCP op derives it from the slug tail. + */ + filename?: string; + } = {}, ): Promise { // Reject oversized payloads before any parsing, chunking, or embedding happens. // Uses Buffer.byteLength to count UTF-8 bytes the same way disk size would, @@ -263,6 +272,23 @@ export async function importFromContent( await engine.transaction(async (tx) => { if (existing) await tx.createVersion(slug); + // v0.29.1 — compute effective_date from frontmatter precedence chain. + // Filename comes from importFromFile path (basename) or the slug tail + // (put_page MCP op fallback). updatedAt/createdAt use the existing + // page's timestamps when present; otherwise NOW() (the row about to + // be created). The result drives the recency boost and since/until + // filters when callers opt in; nothing in the default search path + // consults it. + const filenameForChain = opts.filename ?? slug.split('/').pop() ?? slug; + const nowDate = new Date(); + const { date: effectiveDate, source: effectiveDateSource } = computeEffectiveDate({ + slug, + frontmatter: parsed.frontmatter, + filename: filenameForChain, + updatedAt: existing?.updated_at ?? nowDate, + createdAt: existing?.created_at ?? nowDate, + }); + await tx.putPage(slug, { type: parsed.type, title: parsed.title, @@ -270,6 +296,9 @@ export async function importFromContent( timeline: parsed.timeline || '', frontmatter: parsed.frontmatter, content_hash: hash, + effective_date: effectiveDate, + effective_date_source: effectiveDateSource, + import_filename: filenameForChain, }); // Tag reconciliation: remove stale, add current @@ -386,7 +415,11 @@ export async function importFromFile( // Pass the path-derived slug explicitly so that any future change to // parseMarkdown's precedence rules cannot re-introduce this bug. - return importFromContent(engine, expectedSlug, content, opts); + // v0.29.1: thread the basename (without extension) for filename-date + // precedence in computeEffectiveDate. e.g. `daily/2024-03-15.md` → + // filename `2024-03-15`. + const fileBasename = basename(relativePath, '.md'); + return importFromContent(engine, expectedSlug, content, { ...opts, filename: fileBasename }); } /** diff --git a/src/core/migrate.ts b/src/core/migrate.ts index 4e52d73a9..8b6bfd163 100644 --- a/src/core/migrate.ts +++ b/src/core/migrate.ts @@ -1798,6 +1798,143 @@ export const MIGRATIONS: Migration[] = [ } }, }, + { + version: 40, + name: 'pages_emotional_weight', + // v0.29 — Salience + Anomaly Detection. + // + // Adds the `emotional_weight` column to pages. Populated by the new + // `recompute_emotional_weight` cycle phase from tags + takes (deterministic; + // no LLM). Default 0.0 so freshly imported pages don't pollute salience + // ranking before the cycle has run; users run `gbrain dream --phase + // recompute_emotional_weight` once after upgrading to backfill. + // + // No index: the salience query orders by a computed score (emotional_weight, + // take_count, recency-decay), not by raw emotional_weight. Add an index + // later only if a query orders by the raw column directly. + // + // Postgres ADD COLUMN with a constant DEFAULT is metadata-only on PG 11+ + // and PGLite (PG 17.5 via WASM) — instant on tables of any size. + sql: ` + ALTER TABLE pages + ADD COLUMN IF NOT EXISTS emotional_weight REAL NOT NULL DEFAULT 0.0; + `, + }, + { + version: 41, + name: 'pages_recency_columns', + sql: '', + // v0.29.1 — Salience-and-Recency, additive opt-in. + // + // Four new pages columns (all nullable, additive only, no behavior change + // in the default search path; only consulted when a caller opts into + // `salience='on'` / `recency='on'` or the new `since`/`until` filter): + // + // effective_date — content date (event_date / date / published / + // filename-date / fallback). Read by the new + // recency boost and date-filter paths only. + // Auto-link doesn't touch it (immune to + // updated_at churn). + // effective_date_source — sentinel for the doctor's effective_date_health + // check ('event_date' | 'date' | 'published' | + // 'filename' | 'fallback'). The 'fallback' value + // is what surfaces "page that fell back to + // updated_at when frontmatter was unparseable". + // import_filename — basename without extension, captured at import. + // computeEffectiveDate uses it for filename-date + // precedence (daily/, meetings/ prefixes). Older + // rows leave it NULL; backfill falls through. + // salience_touched_at — bumped by recompute_emotional_weight when + // emotional_weight changes. Salience window + // uses GREATEST(updated_at, salience_touched_at) + // so newly-salient old pages enter the recent + // salience query. + // + // Plus an expression index used by since/until filters that read + // COALESCE(effective_date, updated_at). Partial-index claim from earlier + // plan iterations was wrong (codex pass-2 #15) — the planner won't use a + // partial index for the negative side of a COALESCE; expression index does. + // + // CONCURRENTLY + pre-drop guard (mirror of v34) on Postgres; plain CREATE + // INDEX on PGLite via the handler branching on engine.kind. + handler: async (engine) => { + // 1. ADD COLUMN x4. ALTER TABLE ADD COLUMN IF NOT EXISTS is idempotent. + // No defaults, all nullable, all metadata-only on PG 11+ and PGLite. + await engine.runMigration(38, ` + ALTER TABLE pages ADD COLUMN IF NOT EXISTS effective_date TIMESTAMPTZ; + ALTER TABLE pages ADD COLUMN IF NOT EXISTS effective_date_source TEXT; + ALTER TABLE pages ADD COLUMN IF NOT EXISTS import_filename TEXT; + ALTER TABLE pages ADD COLUMN IF NOT EXISTS salience_touched_at TIMESTAMPTZ; + `); + + // 2. Expression index for since/until date-range filters. + if (engine.kind === 'postgres') { + // Pre-drop any invalid index from a prior CONCURRENTLY failure. + await engine.runMigration(38, ` + DO $$ BEGIN + IF EXISTS ( + SELECT 1 FROM pg_index i + JOIN pg_class c ON c.oid = i.indexrelid + WHERE c.relname = 'pages_coalesce_date_idx' AND NOT i.indisvalid + ) THEN + EXECUTE 'DROP INDEX CONCURRENTLY IF EXISTS pages_coalesce_date_idx'; + END IF; + END $$; + `); + await engine.runMigration(38, ` + CREATE INDEX CONCURRENTLY IF NOT EXISTS pages_coalesce_date_idx + ON pages ((COALESCE(effective_date, updated_at))); + `); + } else { + await engine.runMigration(38, ` + CREATE INDEX IF NOT EXISTS pages_coalesce_date_idx + ON pages ((COALESCE(effective_date, updated_at))); + `); + } + }, + // CONCURRENTLY on Postgres requires no surrounding transaction. + transaction: false, + }, + { + version: 42, + name: 'eval_candidates_recency_capture', + // v0.29.1 — capture agent-explicit recency + salience choices for replay + // reproducibility (D11 codex resolution). + // + // Without these fields, `gbrain eval replay` cannot reproduce a captured + // run: the live behavior depends on the resolved {salience, recency} + // values, which are absent from v0.29.0's eval_candidates schema. Replays + // of agent-explicit choices drift the same way as_of_ts replays drifted + // before being captured. + // + // All columns are nullable + additive. Pre-v0.29.1 rows stay valid. The + // NDJSON `schema_version` STAYS at 1 — the new fields are optional, and + // gbrain-evals consumers that don't know about them ignore them + // (standard permissive deserialization). No cross-repo coordination + // required (codex pass-1 #C2 dissolved). + // + // as_of_ts — brain's logical NOW at capture (replay uses + // this instead of wall-clock so old captures + // reproduce identically against today's brain). + // salience_param — what the caller passed (or NULL if omitted). + // recency_param — same for recency. + // salience_resolved — final value applied ('off' / 'on' / 'strong'). + // recency_resolved — same for recency. + // salience_source — 'caller' or 'auto_heuristic'. + // recency_source — same for recency. + // + // ADD COLUMN with no DEFAULT is metadata-only on PG 11+ and PGLite — + // instant on tables of any size. + sql: ` + ALTER TABLE eval_candidates ADD COLUMN IF NOT EXISTS as_of_ts TIMESTAMPTZ; + ALTER TABLE eval_candidates ADD COLUMN IF NOT EXISTS salience_param TEXT; + ALTER TABLE eval_candidates ADD COLUMN IF NOT EXISTS recency_param TEXT; + ALTER TABLE eval_candidates ADD COLUMN IF NOT EXISTS salience_resolved TEXT; + ALTER TABLE eval_candidates ADD COLUMN IF NOT EXISTS recency_resolved TEXT; + ALTER TABLE eval_candidates ADD COLUMN IF NOT EXISTS salience_source TEXT; + ALTER TABLE eval_candidates ADD COLUMN IF NOT EXISTS recency_source TEXT; + `, + }, ]; export const LATEST_VERSION = MIGRATIONS.length > 0 diff --git a/src/core/minions/tools/brain-allowlist.ts b/src/core/minions/tools/brain-allowlist.ts index 1f3fdbb49..823ae16be 100644 --- a/src/core/minions/tools/brain-allowlist.ts +++ b/src/core/minions/tools/brain-allowlist.ts @@ -56,6 +56,13 @@ export const BRAIN_TOOL_ALLOWLIST: ReadonlySet = new Set([ 'resolve_slugs', 'get_ingest_log', 'put_page', + // v0.29 — Salience + Anomaly Detection. Both read-only. `get_recent_transcripts` + // is intentionally NOT included: subagent calls always have ctx.remote=true, + // and the v0.29 trust gate rejects remote callers — adding it here would be + // a footgun (subagent calls op, gets permission_denied, looks like a bug). + // The cycle synthesize phase already calls discoverTranscripts directly. + 'get_recent_salience', + 'find_anomalies', ]); /** Matches Anthropic's tool-name constraint. No dots. */ diff --git a/src/core/operations-descriptions.ts b/src/core/operations-descriptions.ts new file mode 100644 index 000000000..a9a1a0ede --- /dev/null +++ b/src/core/operations-descriptions.ts @@ -0,0 +1,65 @@ +/** + * v0.29 — Tool descriptions, extracted to a constants module so that: + * 1. The exact LLM-facing strings are pinnable in tests + * (`test/operations-descriptions.test.ts`). + * 2. Routing changes ship as data, not buried-in-handler edits. + * 3. The `salience-llm-routing.test.ts` Tier-2 eval has a stable surface + * to load tool definitions from. + * + * Description style: + * - Lead with what the tool does in one short sentence. + * - Include explicit triggers ("Use this when the user asks ...") that + * the LLM tool-selection prompt can match. + * - For redirect hints (query/search → salience), be blunt: + * "Do NOT run a semantic search for these." + */ + +// ────────────────────────────────────────────────────────────────────────────── +// New v0.29 ops +// ────────────────────────────────────────────────────────────────────────────── + +export const GET_RECENT_SALIENCE_DESCRIPTION = + "Returns pages recently touched and ranked by emotional + activity salience " + + "(deterministic 0..1 emotional_weight + take density + recency decay). " + + "Use this when the user asks what's been going on, what's notable, what's hot, " + + "anything crazy happening, or for any open-ended 'current state' question " + + "about themselves or their work. Do NOT run a semantic search for these — " + + "salience surfaces what's unusual without needing a search term."; + +export const FIND_ANOMALIES_DESCRIPTION = + "Returns statistical anomalies in recent page activity, grouped by cohort " + + "(tag or type). Use this for questions about what stood out, what's unusual, " + + "or what changed recently. Returns explanatory cohorts (e.g. '15 pages tagged " + + "wedding touched on 2026-04-28, baseline 0.3/day') so you can speak about " + + "patterns the user wouldn't have searched for. Cohort kinds: tag, type. " + + "Year cohort is deferred to a later release."; + +export const GET_RECENT_TRANSCRIPTS_DESCRIPTION = + "Returns one-line summaries of recent raw conversation transcripts (NOT polished " + + "reflections). Use this FIRST for questions about 'what's going on with me', " + + "'what have I been thinking about', or anything personal/emotional. Raw " + + "transcripts are the canonical source for the user's own state — polished pages " + + "summarize and flatten. Local-only: rejects remote (MCP/HTTP) callers with a " + + "clear permission_denied; call via the gbrain CLI."; + +// ────────────────────────────────────────────────────────────────────────────── +// Redirect hints appended to existing op descriptions +// ────────────────────────────────────────────────────────────────────────────── + +export const LIST_PAGES_DESCRIPTION = + "List pages with optional filters. " + + "For 'what's recent / what did I touch this week' questions, use list_pages " + + "with sort=updated_desc instead of semantic search."; + +export const QUERY_DESCRIPTION = + "Hybrid search with vector + keyword + multi-query expansion. " + + "For personal/emotional questions ('what's going on with me', 'anything notable', " + + "'how am I feeling'), prefer get_recent_salience, find_anomalies, or " + + "get_recent_transcripts. Semantic search returns polished pages and misses " + + "recent activity bursts. Do NOT assume words like 'crazy', 'notable', or 'big' " + + "mean impressive — they often mean difficult or emotionally charged."; + +export const SEARCH_DESCRIPTION = + "Keyword search using full-text search. For personal/emotional questions, " + + "prefer get_recent_salience or find_anomalies — they surface activity bursts " + + "without needing a search term."; diff --git a/src/core/operations.ts b/src/core/operations.ts index 5a78cf5dc..9001c82c6 100644 --- a/src/core/operations.ts +++ b/src/core/operations.ts @@ -17,6 +17,14 @@ import { captureEvalCandidate, isEvalCaptureEnabled, isEvalScrubEnabled } from ' import type { HybridSearchMeta } from './types.ts'; import { extractPageLinks, isAutoLinkEnabled, isAutoTimelineEnabled, parseTimelineEntries, makeResolver, type UnresolvedFrontmatterRef } from './link-extraction.ts'; import * as db from './db.ts'; +import { + GET_RECENT_SALIENCE_DESCRIPTION, + FIND_ANOMALIES_DESCRIPTION, + GET_RECENT_TRANSCRIPTS_DESCRIPTION, + LIST_PAGES_DESCRIPTION, + QUERY_DESCRIPTION, + SEARCH_DESCRIPTION, +} from './operations-descriptions.ts'; // --- Types --- @@ -726,21 +734,43 @@ const purge_deleted_pages: Operation = { cliHints: { name: 'purge-deleted' }, }; +const LIST_PAGES_SORT_VALUES = ['updated_desc', 'updated_asc', 'created_desc', 'slug'] as const; +type ListPagesSort = typeof LIST_PAGES_SORT_VALUES[number]; + const list_pages: Operation = { name: 'list_pages', - description: 'List pages with optional filters. Soft-deleted pages are hidden by default; pass include_deleted: true to surface them with deleted_at populated.', + description: LIST_PAGES_DESCRIPTION, params: { type: { type: 'string', description: 'Filter by page type' }, tag: { type: 'string', description: 'Filter by tag' }, limit: { type: 'number', description: 'Max results (default 50)' }, + // v0.29 — surface filter that already exists on PageFilters. + updated_after: { + type: 'string', + description: 'ISO date (YYYY-MM-DD) or full timestamp. Returns pages with updated_at > value.', + }, + sort: { + type: 'string', + enum: [...LIST_PAGES_SORT_VALUES], + description: 'Sort order. Default updated_desc (matches pre-v0.29). Options: updated_desc, updated_asc, created_desc, slug.', + }, include_deleted: { type: 'boolean', description: 'v0.26.5: include soft-deleted pages (default: false). Used by restore workflows and operator diagnostics.' }, }, handler: async (ctx, p) => { + // Whitelist the sort enum at the handler before passing to the engine. + // Engines also whitelist via PAGE_SORT_SQL but defending here keeps + // unsupported strings from reaching the SQL layer. + const rawSort = p.sort as string | undefined; + const sort = rawSort && (LIST_PAGES_SORT_VALUES as readonly string[]).includes(rawSort) + ? (rawSort as ListPagesSort) + : undefined; const pages = await ctx.engine.listPages({ type: p.type as any, tag: p.tag as string, limit: clampSearchLimit(p.limit as number | undefined, 50, 100), includeDeleted: (p.include_deleted as boolean) === true, + updated_after: typeof p.updated_after === 'string' ? p.updated_after : undefined, + sort, }); return pages.map(pg => ({ slug: pg.slug, @@ -758,7 +788,7 @@ const list_pages: Operation = { const search: Operation = { name: 'search', - description: 'Keyword search using full-text search', + description: SEARCH_DESCRIPTION, params: { query: { type: 'string', required: true }, limit: { type: 'number', description: 'Max results (default 20)' }, @@ -804,7 +834,7 @@ const search: Operation = { const query: Operation = { name: 'query', - description: 'Hybrid search with vector + keyword + multi-query expansion', + description: QUERY_DESCRIPTION, params: { // v0.27.1: `query` is no longer strictly required — `--image ` // is the alternative entry point for image-similarity search. The CLI @@ -827,6 +857,37 @@ const query: Operation = { // v0.20.0 Cathedral II Layer 7 (A2) / Layer 10 C3: two-pass structural expansion. near_symbol: { type: 'string', description: 'Anchor retrieval at this qualified symbol name (e.g., BrainEngine.searchKeyword). Enables A2 two-pass.' }, walk_depth: { type: 'number', description: 'Structural walk depth 1-2. Default 0 (off). Expands anchors through code_edges with 1/(1+hop) decay.' }, + // v0.29.1 — orthogonal recency + salience axes. YOU (the agent) decide. + salience: { + type: 'string', + enum: ['off', 'on', 'strong'], + description: + "v0.29.1 salience boost — emotional_weight + take_count, NO time component.\n" + + " 'off' — default for entity / canonical / definitional queries\n" + + " 'on' — surface emotionally-weighted + take-rich pages\n" + + " 'strong' — aggressive mattering tilt\n" + + "Omit and gbrain auto-detects from query text. Independent of `recency`.", + }, + recency: { + type: 'string', + enum: ['off', 'on', 'strong'], + description: + "v0.29.1 recency boost — per-prefix age decay, NO mattering signal.\n" + + " 'off' — default for canonical truth\n" + + " 'on' — daily/, media/x/, chat/ decay aggressively; concepts/, originals/, writing/ stay evergreen\n" + + " 'strong' — multiplies the recency factor by 1.5 (use for 'today' / 'right now')\n" + + "Omit and gbrain auto-detects. Independent of `salience` (orthogonal axes).", + }, + since: { + type: 'string', + description: + "v0.29.1 — filter to pages whose effective_date is >= this. ISO-8601 (YYYY-MM-DD or full timestamp) OR relative ('7d', '2w', '1y'). Replaces deprecated `afterDate`.", + }, + until: { + type: 'string', + description: + "v0.29.1 — filter to effective_date <= this. Same format as `since`. Replaces deprecated `beforeDate`. YYYY-MM-DD lands at end-of-day.", + }, }, handler: async (ctx, p) => { const startedAt = Date.now(); @@ -870,6 +931,11 @@ const query: Operation = { symbolKind: (p.symbol_kind as string) || undefined, nearSymbol: (p.near_symbol as string) || undefined, walkDepth: typeof p.walk_depth === 'number' ? (p.walk_depth as number) : undefined, + // v0.29.1 — agent-explicit recency + salience. Omitted = heuristic defaults. + salience: p.salience as 'off' | 'on' | 'strong' | undefined, + recency: p.recency as 'off' | 'on' | 'strong' | undefined, + since: typeof p.since === 'string' ? p.since : undefined, + until: typeof p.until === 'string' ? p.until : undefined, onMeta: (m) => { capturedMeta = m; }, }); const latency_ms = Date.now() - startedAt; @@ -1753,6 +1819,112 @@ const find_orphans: Operation = { cliHints: { name: 'orphans', hidden: true }, }; +// --- v0.29: Salience + Anomaly Detection --- + +const get_recent_salience: Operation = { + name: 'get_recent_salience', + description: GET_RECENT_SALIENCE_DESCRIPTION, + scope: 'read', + params: { + days: { type: 'number', description: 'Window in days. Default 14.' }, + limit: { type: 'number', description: 'Max results (default 20, capped at 100).' }, + slugPrefix: { + type: 'string', + description: "Optional slug-prefix filter, e.g. 'personal' or 'wiki/people'.", + }, + recency_bias: { + type: 'string', + enum: ['flat', 'on'], + description: + "v0.29.1: how to weight recency in the salience score.\n" + + " 'flat' (DEFAULT) — v0.29.0 behavior. Every page gets 1/(1+days_old).\n" + + " Stable, predictable; what most callers want.\n" + + " 'on' — Per-prefix decay map. concepts/originals/writing/\n" + + " become evergreen (recency component = 0); daily/,\n" + + " media/x/, chat/ decay aggressively. Use when the\n" + + " user explicitly biases for recency-aware salience\n" + + " ('what's been salient lately' vs 'what matters\n" + + " in this brain regardless of when').", + }, + }, + handler: async (ctx, p) => { + const recencyBias = p.recency_bias === 'on' ? 'on' : 'flat'; + return ctx.engine.getRecentSalience({ + days: typeof p.days === 'number' ? p.days : undefined, + limit: typeof p.limit === 'number' ? p.limit : undefined, + slugPrefix: typeof p.slugPrefix === 'string' ? p.slugPrefix : undefined, + recency_bias: recencyBias, + }); + }, + cliHints: { name: 'salience' }, +}; + +const find_anomalies: Operation = { + name: 'find_anomalies', + description: FIND_ANOMALIES_DESCRIPTION, + scope: 'read', + params: { + since: { + type: 'string', + description: 'ISO date YYYY-MM-DD. Default = today (UTC).', + }, + lookback_days: { + type: 'number', + description: 'Days of history for the baseline. Default 30.', + }, + sigma: { + type: 'number', + description: 'Sigma threshold. Default 3.0.', + }, + }, + handler: async (ctx, p) => { + return ctx.engine.findAnomalies({ + since: typeof p.since === 'string' ? p.since : undefined, + lookback_days: typeof p.lookback_days === 'number' ? p.lookback_days : undefined, + sigma: typeof p.sigma === 'number' ? p.sigma : undefined, + }); + }, + cliHints: { name: 'anomalies' }, +}; + +const get_recent_transcripts: Operation = { + name: 'get_recent_transcripts', + description: GET_RECENT_TRANSCRIPTS_DESCRIPTION, + scope: 'read', + // Local-only: rejects HTTP-borne MCP traffic at tool-list time + // (serve-http.ts filters on `localOnly`) AND at runtime via the in-handler + // ctx.remote check. Defense in depth: hidden + rejected. + localOnly: true, + params: { + days: { type: 'number', description: 'Window in days. Default 7.' }, + summary: { + type: 'boolean', + description: 'When true (default), return first ~300 chars per transcript. When false, full content (capped at 100 KB per file).', + }, + limit: { type: 'number', description: 'Max transcripts (default 50).' }, + }, + handler: async (ctx, p) => { + // Trust gate (eng review D2 + codex C3): MCP / HTTP callers (`remote=true`) + // are blocked. Local CLI callers (`remote=false`) and the trusted-workspace + // dream cycle pass through. This op is intentionally NOT in the subagent + // allow-list (subagents always run with remote=true; they would always be + // rejected, which is a footgun if the op is visible). + if (ctx.remote === true) { + throw new OperationError( + 'permission_denied', + 'get_recent_transcripts is local-only — call via the gbrain CLI.', + ); + } + const { listRecentTranscripts } = await import('./transcripts.ts'); + return listRecentTranscripts(ctx.engine, { + days: typeof p.days === 'number' ? p.days : undefined, + summary: typeof p.summary === 'boolean' ? p.summary : undefined, + limit: typeof p.limit === 'number' ? p.limit : undefined, + }); + }, + cliHints: { name: 'transcripts', hidden: true }, +}; + // --- v0.28: whoami + sources management --- const whoami: Operation = { @@ -1992,6 +2164,8 @@ export const operations: Operation[] = [ takes_list, takes_search, think, // v0.28: whoami + scoped sources management whoami, sources_add, sources_list, sources_remove, sources_status, + // v0.29: Salience + anomalies + recent transcripts + get_recent_salience, find_anomalies, get_recent_transcripts, ]; export const operationsByName = Object.fromEntries( diff --git a/src/core/pglite-engine.ts b/src/core/pglite-engine.ts index 574728a37..91fa1e18f 100644 --- a/src/core/pglite-engine.ts +++ b/src/core/pglite-engine.ts @@ -28,11 +28,14 @@ import type { EngineConfig, EvalCandidate, EvalCandidateInput, EvalCaptureFailure, EvalCaptureFailureReason, + SalienceOpts, SalienceResult, AnomaliesOpts, AnomalyResult, + EmotionalWeightInputRow, EmotionalWeightWriteRow, } from './types.ts'; import { validateSlug, contentHash, rowToPage, rowToChunk, rowToSearchResult, takeRowToTake } from './utils.ts'; -import { GBrainError } from './types.ts'; +import { GBrainError, PAGE_SORT_SQL } from './types.ts'; +import { computeAnomaliesFromBuckets } from './cycle/anomaly.ts'; import { resolveBoostMap, resolveHardExcludes } from './search/source-boost.ts'; -import { buildSourceFactorCase, buildHardExcludeClause, buildVisibilityClause } from './search/sql-ranking.ts'; +import { buildSourceFactorCase, buildHardExcludeClause, buildVisibilityClause, buildRecencyComponentSql } from './search/sql-ranking.ts'; type PGLiteDB = PGlite; @@ -461,15 +464,20 @@ export class PGLiteEngine implements BrainEngine { const hash = page.content_hash || contentHash(page); const frontmatter = page.frontmatter || {}; - // v0.18.0 Step 2: source_id relies on the schema DEFAULT 'default' so - // existing callers still target the default source without threading - // a parameter. ON CONFLICT target becomes (source_id, slug) since the - // global UNIQUE(slug) was dropped in migration v17. Step 5+ will - // surface an explicit sourceId param on putPage for multi-source sync. + // v0.18.0 Step 2: source_id relies on the schema DEFAULT 'default'. + // ON CONFLICT target is (source_id, slug); global UNIQUE(slug) dropped in v17. const pageKind = page.page_kind || 'markdown'; + // v0.29.1 — additive opt-in columns. COALESCE(EXCLUDED.x, pages.x) + // preserves existing values when caller omits them (auto-link path, + // code reindex, etc.). Mirrors postgres-engine.ts. + const effectiveDate = page.effective_date instanceof Date + ? page.effective_date.toISOString() + : (page.effective_date ?? null); + const effectiveDateSource = page.effective_date_source ?? null; + const importFilename = page.import_filename ?? null; const { rows } = await this.db.query( - `INSERT INTO pages (slug, type, page_kind, title, compiled_truth, timeline, frontmatter, content_hash, updated_at) - VALUES ($1, $2, $3, $4, $5, $6, $7::jsonb, $8, now()) + `INSERT INTO pages (slug, type, page_kind, title, compiled_truth, timeline, frontmatter, content_hash, updated_at, effective_date, effective_date_source, import_filename) + VALUES ($1, $2, $3, $4, $5, $6, $7::jsonb, $8, now(), $9::timestamptz, $10, $11) ON CONFLICT (source_id, slug) DO UPDATE SET type = EXCLUDED.type, page_kind = EXCLUDED.page_kind, @@ -478,9 +486,12 @@ export class PGLiteEngine implements BrainEngine { timeline = EXCLUDED.timeline, frontmatter = EXCLUDED.frontmatter, content_hash = EXCLUDED.content_hash, - updated_at = now() - RETURNING id, slug, type, title, compiled_truth, timeline, frontmatter, content_hash, created_at, updated_at`, - [slug, page.type, pageKind, page.title, page.compiled_truth, page.timeline || '', JSON.stringify(frontmatter), hash] + updated_at = now(), + effective_date = COALESCE(EXCLUDED.effective_date, pages.effective_date), + effective_date_source = COALESCE(EXCLUDED.effective_date_source, pages.effective_date_source), + import_filename = COALESCE(EXCLUDED.import_filename, pages.import_filename) + RETURNING id, slug, type, title, compiled_truth, timeline, frontmatter, content_hash, created_at, updated_at, effective_date, effective_date_source, import_filename`, + [slug, page.type, pageKind, page.title, page.compiled_truth, page.timeline || '', JSON.stringify(frontmatter), hash, effectiveDate, effectiveDateSource, importFilename] ); return rowToPage(rows[0] as Record); } @@ -573,9 +584,13 @@ export class PGLiteEngine implements BrainEngine { params.push(limit, offset); const limitSql = `LIMIT $${params.length - 1} OFFSET $${params.length}`; + // v0.29: ORDER BY threading via PAGE_SORT_SQL whitelist (no SQL injection). + const sortKey = filters?.sort && PAGE_SORT_SQL[filters.sort] ? filters.sort : 'updated_desc'; + const orderBy = PAGE_SORT_SQL[sortKey]; + const { rows } = await this.db.query( `SELECT p.* FROM pages p ${tagJoin} ${whereSql} - ORDER BY p.updated_at DESC ${limitSql}`, + ORDER BY ${orderBy} ${limitSql}`, params ); @@ -647,6 +662,18 @@ export class PGLiteEngine implements BrainEngine { params.push(opts.symbolKind); extraFilter += ` AND cc.symbol_type = $${params.length}`; } + // v0.29.1 — since/until date filter (Postgres parity, codex pass-1 #10). + // Reads against COALESCE(effective_date, updated_at) so date filtering + // matches user intent (a meeting was on its event_date, not when it + // got reimported). Same param shape as Postgres engine. + if (opts?.afterDate) { + params.push(opts.afterDate); + extraFilter += ` AND COALESCE(p.effective_date, p.updated_at, p.created_at) > $${params.length}::timestamptz`; + } + if (opts?.beforeDate) { + params.push(opts.beforeDate); + extraFilter += ` AND COALESCE(p.effective_date, p.updated_at, p.created_at) < $${params.length}::timestamptz`; + } // v0.26.5: visibility filter (soft-deleted + archived-source). const visibilityClause = buildVisibilityClause('p', 's'); @@ -723,6 +750,15 @@ export class PGLiteEngine implements BrainEngine { params.push(opts.symbolKind); extraFilter += ` AND cc.symbol_type = $${params.length}`; } + // v0.29.1 since/until parity (codex pass-1 #10). + if (opts?.afterDate) { + params.push(opts.afterDate); + extraFilter += ` AND COALESCE(p.effective_date, p.updated_at, p.created_at) > $${params.length}::timestamptz`; + } + if (opts?.beforeDate) { + params.push(opts.beforeDate); + extraFilter += ` AND COALESCE(p.effective_date, p.updated_at, p.created_at) < $${params.length}::timestamptz`; + } // v0.26.5: visibility filter for the chunk-grain anchor primitive. const visibilityClause = buildVisibilityClause('p', 's'); @@ -783,6 +819,17 @@ export class PGLiteEngine implements BrainEngine { params.push(opts.symbolKind); extraFilter += ` AND cc.symbol_type = $${params.length}`; } + // v0.29.1 since/until parity (codex pass-1 #10). Filter applied INSIDE + // the inner CTE so HNSW's candidate pool already excludes out-of-range + // pages — preserves pagination contract. + if (opts?.afterDate) { + params.push(opts.afterDate); + extraFilter += ` AND COALESCE(p.effective_date, p.updated_at, p.created_at) > $${params.length}::timestamptz`; + } + if (opts?.beforeDate) { + params.push(opts.beforeDate); + extraFilter += ` AND COALESCE(p.effective_date, p.updated_at, p.created_at) < $${params.length}::timestamptz`; + } // v0.26.5: visibility filter applied in the inner CTE so HNSW sees the // same candidate count it always did. See postgres-engine.ts for rationale. @@ -1314,6 +1361,58 @@ export class PGLiteEngine implements BrainEngine { return result; } + async getPageTimestamps(slugs: string[]): Promise> { + if (slugs.length === 0) return new Map(); + const { rows } = await this.db.query( + `SELECT slug, COALESCE(updated_at, created_at) as ts + FROM pages WHERE slug = ANY($1::text[])`, + [slugs] + ); + return new Map(rows.map((r: any) => [r.slug as string, new Date(r.ts as string)])); + } + + async getEffectiveDates(refs: Array<{slug: string; source_id: string}>): Promise> { + if (refs.length === 0) return new Map(); + const slugs = refs.map(r => r.slug); + const sourceIds = refs.map(r => r.source_id); + const { rows } = await this.db.query( + `SELECT p.slug, p.source_id, COALESCE(p.effective_date, p.updated_at, p.created_at) AS ts + FROM pages p + JOIN unnest($1::text[], $2::text[]) AS u(slug, source_id) + ON p.slug = u.slug AND p.source_id = u.source_id`, + [slugs, sourceIds], + ); + const out = new Map(); + for (const r of rows as Array<{slug: string; source_id: string; ts: string | Date}>) { + const key = `${r.source_id}::${r.slug}`; + out.set(key, r.ts instanceof Date ? r.ts : new Date(r.ts)); + } + return out; + } + + async getSalienceScores(refs: Array<{slug: string; source_id: string}>): Promise> { + if (refs.length === 0) return new Map(); + const slugs = refs.map(r => r.slug); + const sourceIds = refs.map(r => r.source_id); + const { rows } = await this.db.query( + `SELECT p.slug, p.source_id, + (COALESCE(p.emotional_weight, 0) * 5 + + ln(1 + COUNT(DISTINCT t.id))) AS score + FROM pages p + JOIN unnest($1::text[], $2::text[]) AS u(slug, source_id) + ON p.slug = u.slug AND p.source_id = u.source_id + LEFT JOIN takes t ON t.page_id = p.id AND t.active = TRUE + GROUP BY p.id`, + [slugs, sourceIds], + ); + const out = new Map(); + for (const r of rows as Array<{slug: string; source_id: string; score: number | string}>) { + const key = `${r.source_id}::${r.slug}`; + out.set(key, Number(r.score)); + } + return out; + } + async findOrphanPages(): Promise> { const { rows } = await this.db.query( `SELECT @@ -2314,6 +2413,240 @@ export class PGLiteEngine implements BrainEngine { ); return rows as unknown as EvalCaptureFailure[]; } + + // ============================================================ + // v0.29 — Salience + Anomaly Detection + // ============================================================ + + async batchLoadEmotionalInputs(slugs?: string[]): Promise { + // Two CTEs avoid the N×M cartesian product (codex C4#4). + const baseSql = ` + WITH page_tags AS ( + SELECT page_id, array_agg(DISTINCT tag) AS tags + FROM tags GROUP BY page_id + ), + page_takes AS ( + SELECT page_id, json_agg(json_build_object( + 'holder', holder, 'weight', weight, 'kind', kind, 'active', active + )) AS takes + FROM takes WHERE active = TRUE GROUP BY page_id + ) + SELECT p.slug, p.source_id, + COALESCE(pt.tags, ARRAY[]::text[]) AS tags, + COALESCE(pk.takes, '[]'::json) AS takes + FROM pages p + LEFT JOIN page_tags pt ON pt.page_id = p.id + LEFT JOIN page_takes pk ON pk.page_id = p.id + `; + const { rows } = slugs + ? await this.db.query(`${baseSql} WHERE p.slug = ANY($1::text[])`, [slugs]) + : await this.db.query(baseSql); + return (rows as Record[]).map(r => ({ + slug: String(r.slug), + source_id: String(r.source_id), + tags: (r.tags as string[]) ?? [], + takes: (r.takes as EmotionalWeightInputRow['takes']) ?? [], + })); + } + + async setEmotionalWeightBatch(rows: EmotionalWeightWriteRow[]): Promise { + if (rows.length === 0) return 0; + const slugs = rows.map(r => r.slug); + const sourceIds = rows.map(r => r.source_id); + const weights = rows.map(r => r.weight); + // Composite-keyed UPDATE FROM unnest (codex C4#3). + // v0.29.1: bump salience_touched_at when emotional_weight actually changes + // so the salience query window picks up newly-salient old pages. Mirror + // of postgres-engine.ts. + const result = await this.db.query( + `UPDATE pages + SET emotional_weight = u.weight, + salience_touched_at = CASE + WHEN pages.emotional_weight IS DISTINCT FROM u.weight THEN now() + ELSE pages.salience_touched_at + END + FROM unnest($1::text[], $2::text[], $3::real[]) + AS u(slug, source_id, weight) + WHERE pages.slug = u.slug AND pages.source_id = u.source_id + RETURNING 1`, + [slugs, sourceIds, weights] + ); + return result.rows.length; + } + + async getRecentSalience(opts: SalienceOpts): Promise { + const days = Math.max(0, opts.days ?? 14); + const limit = clampSearchLimit(opts.limit, 20, 100); + const slugPrefix = opts.slugPrefix; + const boundaryIso = new Date(Date.now() - days * 86400000).toISOString(); + + const params: unknown[] = [boundaryIso]; + let prefixCondition = ''; + if (slugPrefix) { + const escaped = slugPrefix.replace(/[\\%_]/g, (c) => '\\' + c) + '%'; + params.push(escaped); + prefixCondition = `AND p.slug LIKE $${params.length} ESCAPE '\\'`; + } + params.push(limit); + const limitParam = `$${params.length}`; + + // v0.29.1: third score term via buildRecencyComponentSql. Default + // 'flat' = v0.29.0 behavior. 'on' opts into per-prefix decay. + const recencyBias = opts.recency_bias ?? 'flat'; + let recencySql: string; + if (recencyBias === 'on') { + const { resolveRecencyDecayMap, DEFAULT_FALLBACK } = await import('./search/recency-decay.ts'); + recencySql = buildRecencyComponentSql({ + slugColumn: 'p.slug', + dateExpr: 'COALESCE(p.effective_date, p.updated_at)', + decayMap: resolveRecencyDecayMap(), + fallback: DEFAULT_FALLBACK, + }); + } else { + recencySql = buildRecencyComponentSql({ + slugColumn: 'p.slug', + dateExpr: 'p.updated_at', + decayMap: {}, + fallback: { halflifeDays: 1, coefficient: 1.0 }, + }); + } + const { rows } = await this.db.query( + `SELECT p.slug, p.source_id, p.title, p.type, p.updated_at, p.emotional_weight, + COUNT(DISTINCT t.id) AS take_count, + COALESCE(AVG(t.weight), 0) AS take_avg_weight, + (p.emotional_weight * 5) + + ln(1 + COUNT(DISTINCT t.id)) + + ${recencySql} + AS score + FROM pages p + LEFT JOIN takes t ON t.page_id = p.id AND t.active = TRUE + WHERE GREATEST(p.updated_at, COALESCE(p.salience_touched_at, p.updated_at)) >= $1::timestamptz + ${prefixCondition} + GROUP BY p.id + ORDER BY score DESC + LIMIT ${limitParam}`, + params + ); + return (rows as Record[]).map(r => ({ + slug: String(r.slug), + source_id: String(r.source_id), + title: String(r.title ?? ''), + type: r.type as SalienceResult['type'], + updated_at: r.updated_at as Date, + emotional_weight: Number(r.emotional_weight ?? 0), + take_count: Number(r.take_count ?? 0), + take_avg_weight: Number(r.take_avg_weight ?? 0), + score: Number(r.score ?? 0), + })); + } + + async findAnomalies(opts: AnomaliesOpts): Promise { + const sigma = opts.sigma ?? 3.0; + const lookbackDays = Math.max(1, opts.lookback_days ?? 30); + const sinceIso = (opts.since ?? new Date().toISOString().slice(0, 10)); + const sinceDate = new Date(sinceIso + 'T00:00:00Z'); + const sinceEnd = new Date(sinceDate.getTime() + 86400000); + const baselineStart = new Date(sinceDate.getTime() - lookbackDays * 86400000); + + const tagBaselineRes = await this.db.query( + `WITH days AS ( + SELECT day::date FROM generate_series( + $1::date, $2::date - 1, '1 day'::interval + ) AS day + ), + cohort_keys AS ( + SELECT DISTINCT t.tag FROM tags t JOIN pages p ON p.id = t.page_id + WHERE p.updated_at >= $1::timestamptz AND p.updated_at < $2::timestamptz + ), + touched AS ( + SELECT t.tag, + date_trunc('day', p.updated_at)::date AS day, + COUNT(DISTINCT p.id) AS cnt + FROM tags t JOIN pages p ON p.id = t.page_id + WHERE p.updated_at >= $1::timestamptz AND p.updated_at < $2::timestamptz + GROUP BY 1, 2 + ) + SELECT cd.tag AS cohort_value, d.day::text AS day, COALESCE(t.cnt, 0)::int AS count + FROM cohort_keys cd CROSS JOIN days d + LEFT JOIN touched t ON t.tag = cd.tag AND t.day = d.day`, + [baselineStart.toISOString(), sinceDate.toISOString()] + ); + + const typeBaselineRes = await this.db.query( + `WITH days AS ( + SELECT day::date FROM generate_series( + $1::date, $2::date - 1, '1 day'::interval + ) AS day + ), + cohort_keys AS ( + SELECT DISTINCT p.type FROM pages p + WHERE p.updated_at >= $1::timestamptz AND p.updated_at < $2::timestamptz + ), + touched AS ( + SELECT p.type, + date_trunc('day', p.updated_at)::date AS day, + COUNT(DISTINCT p.id) AS cnt + FROM pages p + WHERE p.updated_at >= $1::timestamptz AND p.updated_at < $2::timestamptz + GROUP BY 1, 2 + ) + SELECT cd.type AS cohort_value, d.day::text AS day, COALESCE(t.cnt, 0)::int AS count + FROM cohort_keys cd CROSS JOIN days d + LEFT JOIN touched t ON t.type = cd.type AND t.day = d.day`, + [baselineStart.toISOString(), sinceDate.toISOString()] + ); + + const tagTodayRes = await this.db.query( + `SELECT t.tag AS cohort_value, + COUNT(DISTINCT p.id)::int AS count, + array_agg(DISTINCT p.slug) AS slugs + FROM tags t JOIN pages p ON p.id = t.page_id + WHERE p.updated_at >= $1::timestamptz AND p.updated_at < $2::timestamptz + GROUP BY 1`, + [sinceIso, sinceEnd.toISOString()] + ); + + const typeTodayRes = await this.db.query( + `SELECT p.type AS cohort_value, + COUNT(DISTINCT p.id)::int AS count, + array_agg(DISTINCT p.slug) AS slugs + FROM pages p + WHERE p.updated_at >= $1::timestamptz AND p.updated_at < $2::timestamptz + GROUP BY 1`, + [sinceIso, sinceEnd.toISOString()] + ); + + const baseline = [ + ...(tagBaselineRes.rows as Record[]).map(r => ({ + cohort_kind: 'tag' as const, + cohort_value: String(r.cohort_value), + day: String(r.day), + count: Number(r.count), + })), + ...(typeBaselineRes.rows as Record[]).map(r => ({ + cohort_kind: 'type' as const, + cohort_value: String(r.cohort_value), + day: String(r.day), + count: Number(r.count), + })), + ]; + const today = [ + ...(tagTodayRes.rows as Record[]).map(r => ({ + cohort_kind: 'tag' as const, + cohort_value: String(r.cohort_value), + count: Number(r.count), + page_slugs: (r.slugs as string[]) ?? [], + })), + ...(typeTodayRes.rows as Record[]).map(r => ({ + cohort_kind: 'type' as const, + cohort_value: String(r.cohort_value), + count: Number(r.count), + page_slugs: (r.slugs as string[]) ?? [], + })), + ]; + + return computeAnomaliesFromBuckets(baseline, today, sigma); + } } function rowToCodeEdge(row: Record): import('./types.ts').CodeEdgeResult { diff --git a/src/core/pglite-schema.ts b/src/core/pglite-schema.ts index 001155315..942f734c0 100644 --- a/src/core/pglite-schema.ts +++ b/src/core/pglite-schema.ts @@ -69,10 +69,18 @@ CREATE TABLE IF NOT EXISTS pages ( timeline TEXT NOT NULL DEFAULT '', frontmatter JSONB NOT NULL DEFAULT '{}', content_hash TEXT, + -- v0.29: deterministic 0..1 score (tag emotion + take density + user-as-holder ratio). + -- Populated by the recompute_emotional_weight cycle phase. + emotional_weight REAL NOT NULL DEFAULT 0.0, created_at TIMESTAMPTZ NOT NULL DEFAULT now(), updated_at TIMESTAMPTZ NOT NULL DEFAULT now(), -- v0.26.5: soft-delete + recovery window (mirrors src/schema.sql). deleted_at TIMESTAMPTZ, + -- v0.29.1: salience-and-recency, additive opt-in (mirrors src/schema.sql). + effective_date TIMESTAMPTZ, + effective_date_source TEXT, + import_filename TEXT, + salience_touched_at TIMESTAMPTZ, CONSTRAINT pages_source_slug_key UNIQUE (source_id, slug) ); @@ -83,6 +91,9 @@ CREATE INDEX IF NOT EXISTS idx_pages_source_id ON pages(source_id); -- v0.26.5: partial index supports the autopilot purge sweep (mirrors src/schema.sql). CREATE INDEX IF NOT EXISTS pages_deleted_at_purge_idx ON pages (deleted_at) WHERE deleted_at IS NOT NULL; +-- v0.29.1: expression index for since/until date-range filters. +CREATE INDEX IF NOT EXISTS pages_coalesce_date_idx + ON pages ((COALESCE(effective_date, updated_at))); -- ============================================================ -- content_chunks: chunked content with embeddings @@ -444,7 +455,15 @@ CREATE TABLE IF NOT EXISTS eval_candidates ( remote BOOLEAN NOT NULL, job_id INTEGER, subagent_id INTEGER, - created_at TIMESTAMPTZ NOT NULL DEFAULT NOW() + created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(), + -- v0.29.1 — agent-explicit recency + salience capture for replay (mirrors src/schema.sql). + as_of_ts TIMESTAMPTZ, + salience_param TEXT, + recency_param TEXT, + salience_resolved TEXT, + recency_resolved TEXT, + salience_source TEXT, + recency_source TEXT ); CREATE INDEX IF NOT EXISTS idx_eval_candidates_created_at ON eval_candidates(created_at DESC); diff --git a/src/core/postgres-engine.ts b/src/core/postgres-engine.ts index 737597d7e..9f65803e1 100644 --- a/src/core/postgres-engine.ts +++ b/src/core/postgres-engine.ts @@ -26,12 +26,15 @@ import type { EngineConfig, EvalCandidate, EvalCandidateInput, EvalCaptureFailure, EvalCaptureFailureReason, + SalienceOpts, SalienceResult, AnomaliesOpts, AnomalyResult, + EmotionalWeightInputRow, EmotionalWeightWriteRow, } from './types.ts'; -import { GBrainError } from './types.ts'; +import { GBrainError, PAGE_SORT_SQL } from './types.ts'; +import { computeAnomaliesFromBuckets } from './cycle/anomaly.ts'; import * as db from './db.ts'; import { validateSlug, contentHash, rowToPage, rowToChunk, rowToSearchResult, parseEmbedding, tryParseEmbedding, takeRowToTake } from './utils.ts'; import { resolveBoostMap, resolveHardExcludes } from './search/source-boost.ts'; -import { buildSourceFactorCase, buildHardExcludeClause, buildVisibilityClause } from './search/sql-ranking.ts'; +import { buildSourceFactorCase, buildHardExcludeClause, buildVisibilityClause, buildRecencyComponentSql } from './search/sql-ranking.ts'; function escapeSqlStringLiteral(value: string): string { return value.replace(/'/g, "''"); @@ -429,12 +432,19 @@ export class PostgresEngine implements BrainEngine { // v0.18.0 Step 2: source_id relies on schema DEFAULT 'default'. ON // CONFLICT target becomes (source_id, slug) since global UNIQUE(slug) - // was dropped in migration v17. See pglite-engine.ts for matching - // notes; multi-source sync (Step 5) will surface an explicit sourceId. + // was dropped in migration v17. const pageKind = page.page_kind || 'markdown'; + // v0.29.1 — effective_date / effective_date_source / import_filename are + // additive opt-in inputs from the importer (computeEffectiveDate). When + // omitted, the ON CONFLICT path preserves any existing value via + // COALESCE(EXCLUDED.x, pages.x) so a putPage that doesn't know about + // these columns (auto-link, code reindex, etc.) doesn't blank them out. + const effectiveDate = page.effective_date ?? null; + const effectiveDateSource = page.effective_date_source ?? null; + const importFilename = page.import_filename ?? null; const rows = await sql` - INSERT INTO pages (slug, type, page_kind, title, compiled_truth, timeline, frontmatter, content_hash, updated_at) - VALUES (${slug}, ${page.type}, ${pageKind}, ${page.title}, ${page.compiled_truth}, ${page.timeline || ''}, ${sql.json(frontmatter as Parameters[0])}, ${hash}, now()) + INSERT INTO pages (slug, type, page_kind, title, compiled_truth, timeline, frontmatter, content_hash, updated_at, effective_date, effective_date_source, import_filename) + VALUES (${slug}, ${page.type}, ${pageKind}, ${page.title}, ${page.compiled_truth}, ${page.timeline || ''}, ${sql.json(frontmatter as Parameters[0])}, ${hash}, now(), ${effectiveDate}, ${effectiveDateSource}, ${importFilename}) ON CONFLICT (source_id, slug) DO UPDATE SET type = EXCLUDED.type, page_kind = EXCLUDED.page_kind, @@ -443,8 +453,11 @@ export class PostgresEngine implements BrainEngine { timeline = EXCLUDED.timeline, frontmatter = EXCLUDED.frontmatter, content_hash = EXCLUDED.content_hash, - updated_at = now() - RETURNING id, slug, type, title, compiled_truth, timeline, frontmatter, content_hash, created_at, updated_at + updated_at = now(), + effective_date = COALESCE(EXCLUDED.effective_date, pages.effective_date), + effective_date_source = COALESCE(EXCLUDED.effective_date_source, pages.effective_date_source), + import_filename = COALESCE(EXCLUDED.import_filename, pages.import_filename) + RETURNING id, slug, type, title, compiled_truth, timeline, frontmatter, content_hash, created_at, updated_at, effective_date, effective_date_source, import_filename `; return rowToPage(rows[0]); } @@ -522,11 +535,16 @@ export class PostgresEngine implements BrainEngine { ? sql`` : sql`AND p.deleted_at IS NULL`; + // v0.29: ORDER BY threading via PAGE_SORT_SQL whitelist (no SQL injection). + // postgres.js sql.unsafe lets us splice the literal fragment safely. + const sortKey = filters?.sort && PAGE_SORT_SQL[filters.sort] ? filters.sort : 'updated_desc'; + const orderBy = sql.unsafe(PAGE_SORT_SQL[sortKey]); + const rows = await sql` SELECT p.* FROM pages p ${tagJoin} WHERE 1=1 ${typeCondition} ${tagCondition} ${updatedCondition} ${slugCondition} ${deletedCondition} - ORDER BY p.updated_at DESC LIMIT ${limit} OFFSET ${offset} + ORDER BY ${orderBy} LIMIT ${limit} OFFSET ${offset} `; return rows.map(rowToPage); @@ -614,6 +632,17 @@ export class PostgresEngine implements BrainEngine { params.push(symbolKind); symbolKindClause = `AND cc.symbol_type = $${params.length}`; } + // v0.27.0: date filtering support + let afterDateClause = ''; + if (opts?.afterDate) { + params.push(opts.afterDate); + afterDateClause = `AND COALESCE(p.updated_at, p.created_at) > $${params.length}::timestamptz`; + } + let beforeDateClause = ''; + if (opts?.beforeDate) { + params.push(opts.beforeDate); + beforeDateClause = `AND COALESCE(p.updated_at, p.created_at) < $${params.length}::timestamptz`; + } params.push(innerLimit); const innerLimitParam = `$${params.length}`; params.push(limit); @@ -642,6 +671,8 @@ export class PostgresEngine implements BrainEngine { ${detailLow ? `AND cc.chunk_source = 'compiled_truth'` : ''} ${languageClause} ${symbolKindClause} + ${afterDateClause} + ${beforeDateClause} ${hardExcludeClause} ${visibilityClause} -- v0.27.1: hide image rows from text-keyword search so OCR text @@ -727,6 +758,17 @@ export class PostgresEngine implements BrainEngine { params.push(symbolKind); symbolKindClause = `AND cc.symbol_type = $${params.length}`; } + // v0.27.0: date filtering support + let afterDateClause = ''; + if (opts?.afterDate) { + params.push(opts.afterDate); + afterDateClause = `AND COALESCE(p.updated_at, p.created_at) > $${params.length}::timestamptz`; + } + let beforeDateClause = ''; + if (opts?.beforeDate) { + params.push(opts.beforeDate); + beforeDateClause = `AND COALESCE(p.updated_at, p.created_at) < $${params.length}::timestamptz`; + } params.push(limit); const limitParam = `$${params.length}`; params.push(offset); @@ -750,6 +792,8 @@ export class PostgresEngine implements BrainEngine { ${detailLow ? `AND cc.chunk_source = 'compiled_truth'` : ''} ${languageClause} ${symbolKindClause} + ${afterDateClause} + ${beforeDateClause} ${hardExcludeClause} ${visibilityClause} ORDER BY score DESC @@ -815,6 +859,17 @@ export class PostgresEngine implements BrainEngine { params.push(symbolKind); symbolKindClause = `AND cc.symbol_type = $${params.length}`; } + // v0.27.0: date filtering support + let afterDateClause = ''; + if (opts?.afterDate) { + params.push(opts.afterDate); + afterDateClause = `AND COALESCE(p.updated_at, p.created_at) > $${params.length}::timestamptz`; + } + let beforeDateClause = ''; + if (opts?.beforeDate) { + params.push(opts.beforeDate); + beforeDateClause = `AND COALESCE(p.updated_at, p.created_at) < $${params.length}::timestamptz`; + } params.push(innerLimit); const innerLimitParam = `$${params.length}`; params.push(limit); @@ -847,6 +902,8 @@ export class PostgresEngine implements BrainEngine { ${excludeSlugsClause} ${languageClause} ${symbolKindClause} + ${afterDateClause} + ${beforeDateClause} ${hardExcludeClause} ${visibilityClause} ORDER BY cc.${col} <=> $1::vector @@ -1366,6 +1423,66 @@ export class PostgresEngine implements BrainEngine { return result; } + async getPageTimestamps(slugs: string[]): Promise> { + if (slugs.length === 0) return new Map(); + const sql = this.sql; + const rows = await sql` + SELECT slug, COALESCE(updated_at, created_at) as ts + FROM pages WHERE slug = ANY(${slugs}::text[]) + `; + return new Map(rows.map(r => [r.slug as string, new Date(r.ts as string)])); + } + + async getEffectiveDates(refs: Array<{slug: string; source_id: string}>): Promise> { + if (refs.length === 0) return new Map(); + const sql = this.sql; + const slugs = refs.map(r => r.slug); + const sourceIds = refs.map(r => r.source_id); + // Composite-keyed: a page is unique by (source_id, slug). unnest the + // two arrays in lockstep so multi-source brains don't fan out across + // sources (codex pass-1 finding #3). + const rows = await sql` + SELECT p.slug, p.source_id, COALESCE(p.effective_date, p.updated_at, p.created_at) AS ts + FROM pages p + JOIN unnest(${slugs}::text[], ${sourceIds}::text[]) AS u(slug, source_id) + ON p.slug = u.slug AND p.source_id = u.source_id + `; + const out = new Map(); + for (const raw of rows as unknown as Array>) { + const r = raw as { slug: string; source_id: string; ts: string | Date }; + const key = `${r.source_id}::${r.slug}`; + out.set(key, r.ts instanceof Date ? r.ts : new Date(r.ts)); + } + return out; + } + + async getSalienceScores(refs: Array<{slug: string; source_id: string}>): Promise> { + if (refs.length === 0) return new Map(); + const sql = this.sql; + const slugs = refs.map(r => r.slug); + const sourceIds = refs.map(r => r.source_id); + // Salience = emotional_weight × 5 + ln(1 + take_count). Pure mattering + // signal — NO time component (per D9: salience and recency are + // orthogonal axes). Composite-keyed for multi-source isolation. + const rows = await sql` + SELECT p.slug, p.source_id, + (COALESCE(p.emotional_weight, 0) * 5 + + ln(1 + COUNT(DISTINCT t.id))) AS score + FROM pages p + JOIN unnest(${slugs}::text[], ${sourceIds}::text[]) AS u(slug, source_id) + ON p.slug = u.slug AND p.source_id = u.source_id + LEFT JOIN takes t ON t.page_id = p.id AND t.active = TRUE + GROUP BY p.id + `; + const out = new Map(); + for (const raw of rows as unknown as Array>) { + const r = raw as { slug: string; source_id: string; score: number | string }; + const key = `${r.source_id}::${r.slug}`; + out.set(key, Number(r.score)); + } + return out; + } + async findOrphanPages(): Promise> { const sql = this.sql; const rows = await sql` @@ -2341,6 +2458,269 @@ export class PostgresEngine implements BrainEngine { `; return rows as unknown as EvalCaptureFailure[]; } + + // ============================================================ + // v0.29 — Salience + Anomaly Detection + // ============================================================ + + async batchLoadEmotionalInputs(slugs?: string[]): Promise { + const sql = this.sql; + // Two CTEs avoid the N×M cartesian product (codex C4#4): a page with N tags + // and M takes joined directly would emit N×M rows and corrupt aggregates. + // Per-table aggregation keeps each table's grouping correct. + const rows = slugs + ? await sql` + WITH page_tags AS ( + SELECT page_id, array_agg(DISTINCT tag) AS tags + FROM tags GROUP BY page_id + ), + page_takes AS ( + SELECT page_id, json_agg(json_build_object( + 'holder', holder, 'weight', weight, 'kind', kind, 'active', active + )) AS takes + FROM takes WHERE active = TRUE GROUP BY page_id + ) + SELECT p.slug, p.source_id, + COALESCE(pt.tags, ARRAY[]::text[]) AS tags, + COALESCE(pk.takes, '[]'::json) AS takes + FROM pages p + LEFT JOIN page_tags pt ON pt.page_id = p.id + LEFT JOIN page_takes pk ON pk.page_id = p.id + WHERE p.slug = ANY(${slugs}::text[]) + ` + : await sql` + WITH page_tags AS ( + SELECT page_id, array_agg(DISTINCT tag) AS tags + FROM tags GROUP BY page_id + ), + page_takes AS ( + SELECT page_id, json_agg(json_build_object( + 'holder', holder, 'weight', weight, 'kind', kind, 'active', active + )) AS takes + FROM takes WHERE active = TRUE GROUP BY page_id + ) + SELECT p.slug, p.source_id, + COALESCE(pt.tags, ARRAY[]::text[]) AS tags, + COALESCE(pk.takes, '[]'::json) AS takes + FROM pages p + LEFT JOIN page_tags pt ON pt.page_id = p.id + LEFT JOIN page_takes pk ON pk.page_id = p.id + `; + return rows.map((r: Record) => ({ + slug: String(r.slug), + source_id: String(r.source_id), + tags: (r.tags as string[]) ?? [], + takes: (r.takes as EmotionalWeightInputRow['takes']) ?? [], + })); + } + + async setEmotionalWeightBatch(rows: EmotionalWeightWriteRow[]): Promise { + if (rows.length === 0) return 0; + const sql = this.sql; + const slugs = rows.map(r => r.slug); + const sourceIds = rows.map(r => r.source_id); + const weights = rows.map(r => r.weight); + // Composite-keyed UPDATE FROM unnest (codex C4#3): pages.slug is unique + // only within a source, so a slug-only join would fan out across sources. + // + // v0.29.1: bump salience_touched_at to NOW() ONLY when emotional_weight + // actually changes. The salience query window then includes the page in + // GREATEST(updated_at, salience_touched_at) >= boundary, so a previously + // calm page that just became salient surfaces in the recent salience + // results without a content edit. No-op writes (same weight) leave + // salience_touched_at alone — preserves "actual change" semantics. + const result = await sql` + UPDATE pages + SET emotional_weight = u.weight, + salience_touched_at = CASE + WHEN pages.emotional_weight IS DISTINCT FROM u.weight THEN now() + ELSE pages.salience_touched_at + END + FROM unnest(${slugs}::text[], ${sourceIds}::text[], ${weights}::real[]) + AS u(slug, source_id, weight) + WHERE pages.slug = u.slug AND pages.source_id = u.source_id + RETURNING 1 + `; + return result.length; + } + + async getRecentSalience(opts: SalienceOpts): Promise { + const sql = this.sql; + const days = Math.max(0, opts.days ?? 14); + const limit = clampSearchLimit(opts.limit, 20, 100); + const slugPrefix = opts.slugPrefix; + // Compute the boundary in JS so the SQL is identical across engines (eng review D5). + const boundaryIso = new Date(Date.now() - days * 86400000).toISOString(); + // Escape LIKE meta for the optional prefix match. + const prefixCondition = slugPrefix + ? sql`AND p.slug LIKE ${slugPrefix.replace(/[\\%_]/g, (c) => '\\' + c) + '%'} ESCAPE '\\'` + : sql``; + // v0.29.1: third score term via buildRecencyComponentSql. Default + // 'flat' = v0.29.0 behavior (1 / (1 + days_old)). 'on' opts into the + // per-prefix decay map (concepts/ evergreen, daily/ aggressive, etc.). + const recencyBias = opts.recency_bias ?? 'flat'; + let recencySql: string; + if (recencyBias === 'on') { + const { resolveRecencyDecayMap, DEFAULT_FALLBACK } = await import('./search/recency-decay.ts'); + recencySql = buildRecencyComponentSql({ + slugColumn: 'p.slug', + dateExpr: 'COALESCE(p.effective_date, p.updated_at)', + decayMap: resolveRecencyDecayMap(), + fallback: DEFAULT_FALLBACK, + }); + } else { + recencySql = buildRecencyComponentSql({ + slugColumn: 'p.slug', + dateExpr: 'p.updated_at', + decayMap: {}, + fallback: { halflifeDays: 1, coefficient: 1.0 }, + }); + } + const rows = await sql` + SELECT p.slug, p.source_id, p.title, p.type, p.updated_at, p.emotional_weight, + COUNT(DISTINCT t.id) AS take_count, + COALESCE(AVG(t.weight), 0) AS take_avg_weight, + (p.emotional_weight * 5) + + ln(1 + COUNT(DISTINCT t.id)) + + ${sql.unsafe(recencySql)} + AS score + FROM pages p + LEFT JOIN takes t ON t.page_id = p.id AND t.active = TRUE + WHERE GREATEST(p.updated_at, COALESCE(p.salience_touched_at, p.updated_at)) >= ${boundaryIso}::timestamptz + ${prefixCondition} + GROUP BY p.id + ORDER BY score DESC + LIMIT ${limit} + `; + return rows.map((r: Record) => ({ + slug: String(r.slug), + source_id: String(r.source_id), + title: String(r.title ?? ''), + type: r.type as SalienceResult['type'], + updated_at: r.updated_at as Date, + emotional_weight: Number(r.emotional_weight ?? 0), + take_count: Number(r.take_count ?? 0), + take_avg_weight: Number(r.take_avg_weight ?? 0), + score: Number(r.score ?? 0), + })); + } + + async findAnomalies(opts: AnomaliesOpts): Promise { + const sql = this.sql; + const sigma = opts.sigma ?? 3.0; + const lookbackDays = Math.max(1, opts.lookback_days ?? 30); + // Boundaries: today's window is [since, since+1day); baseline is [since-lookback, since). + const sinceIso = (opts.since ?? new Date().toISOString().slice(0, 10)); // YYYY-MM-DD + const sinceDate = new Date(sinceIso + 'T00:00:00Z'); + const sinceEnd = new Date(sinceDate.getTime() + 86400000); + const baselineStart = new Date(sinceDate.getTime() - lookbackDays * 86400000); + + // Tag cohort baseline with day densification + zero-fill (codex C4#6). + const tagBaseline = await sql` + WITH days AS ( + SELECT day::date FROM generate_series( + ${baselineStart.toISOString()}::date, + ${sinceDate.toISOString()}::date - 1, + '1 day'::interval + ) AS day + ), + cohort_keys AS ( + SELECT DISTINCT t.tag FROM tags t JOIN pages p ON p.id = t.page_id + WHERE p.updated_at >= ${baselineStart.toISOString()}::timestamptz + AND p.updated_at < ${sinceDate.toISOString()}::timestamptz + ), + touched AS ( + SELECT t.tag, + date_trunc('day', p.updated_at)::date AS day, + COUNT(DISTINCT p.id) AS cnt + FROM tags t JOIN pages p ON p.id = t.page_id + WHERE p.updated_at >= ${baselineStart.toISOString()}::timestamptz + AND p.updated_at < ${sinceDate.toISOString()}::timestamptz + GROUP BY 1, 2 + ) + SELECT cd.tag AS cohort_value, d.day::text AS day, COALESCE(t.cnt, 0)::int AS count + FROM cohort_keys cd CROSS JOIN days d + LEFT JOIN touched t ON t.tag = cd.tag AND t.day = d.day + `; + + const typeBaseline = await sql` + WITH days AS ( + SELECT day::date FROM generate_series( + ${baselineStart.toISOString()}::date, + ${sinceDate.toISOString()}::date - 1, + '1 day'::interval + ) AS day + ), + cohort_keys AS ( + SELECT DISTINCT p.type FROM pages p + WHERE p.updated_at >= ${baselineStart.toISOString()}::timestamptz + AND p.updated_at < ${sinceDate.toISOString()}::timestamptz + ), + touched AS ( + SELECT p.type, + date_trunc('day', p.updated_at)::date AS day, + COUNT(DISTINCT p.id) AS cnt + FROM pages p + WHERE p.updated_at >= ${baselineStart.toISOString()}::timestamptz + AND p.updated_at < ${sinceDate.toISOString()}::timestamptz + GROUP BY 1, 2 + ) + SELECT cd.type AS cohort_value, d.day::text AS day, COALESCE(t.cnt, 0)::int AS count + FROM cohort_keys cd CROSS JOIN days d + LEFT JOIN touched t ON t.type = cd.type AND t.day = d.day + `; + + // Today's window — current counts + slugs per cohort. + const tagToday = await sql` + SELECT t.tag AS cohort_value, + COUNT(DISTINCT p.id)::int AS count, + array_agg(DISTINCT p.slug) AS slugs + FROM tags t JOIN pages p ON p.id = t.page_id + WHERE p.updated_at >= ${sinceIso}::timestamptz + AND p.updated_at < ${sinceEnd.toISOString()}::timestamptz + GROUP BY 1 + `; + const typeToday = await sql` + SELECT p.type AS cohort_value, + COUNT(DISTINCT p.id)::int AS count, + array_agg(DISTINCT p.slug) AS slugs + FROM pages p + WHERE p.updated_at >= ${sinceIso}::timestamptz + AND p.updated_at < ${sinceEnd.toISOString()}::timestamptz + GROUP BY 1 + `; + + const baseline = [ + ...tagBaseline.map((r: Record) => ({ + cohort_kind: 'tag' as const, + cohort_value: String(r.cohort_value), + day: String(r.day), + count: Number(r.count), + })), + ...typeBaseline.map((r: Record) => ({ + cohort_kind: 'type' as const, + cohort_value: String(r.cohort_value), + day: String(r.day), + count: Number(r.count), + })), + ]; + const today = [ + ...tagToday.map((r: Record) => ({ + cohort_kind: 'tag' as const, + cohort_value: String(r.cohort_value), + count: Number(r.count), + page_slugs: (r.slugs as string[]) ?? [], + })), + ...typeToday.map((r: Record) => ({ + cohort_kind: 'type' as const, + cohort_value: String(r.cohort_value), + count: Number(r.count), + page_slugs: (r.slugs as string[]) ?? [], + })), + ]; + + return computeAnomaliesFromBuckets(baseline, today, sigma); + } } function pgRowToCodeEdge(row: Record): import('./types.ts').CodeEdgeResult { diff --git a/src/core/schema-embedded.ts b/src/core/schema-embedded.ts index 80daf1d3e..905936657 100644 --- a/src/core/schema-embedded.ts +++ b/src/core/schema-embedded.ts @@ -81,6 +81,10 @@ CREATE TABLE IF NOT EXISTS pages ( timeline TEXT NOT NULL DEFAULT '', frontmatter JSONB NOT NULL DEFAULT '{}', content_hash TEXT, + -- v0.29: deterministic 0..1 score (tag emotion + take density + Garry-as-holder ratio). + -- Populated by the \`recompute_emotional_weight\` cycle phase. Default 0.0 so freshly + -- imported pages don't pollute salience ranking before the cycle has run. + emotional_weight REAL NOT NULL DEFAULT 0.0, created_at TIMESTAMPTZ NOT NULL DEFAULT now(), updated_at TIMESTAMPTZ NOT NULL DEFAULT now(), -- v0.26.5: soft-delete + recovery window. \`delete_page\` sets deleted_at = now() @@ -88,6 +92,17 @@ CREATE TABLE IF NOT EXISTS pages ( -- where deleted_at < now() - 72h. Search and \`get_page\` filter -- \`WHERE deleted_at IS NULL\` by default; \`include_deleted: true\` opts in. deleted_at TIMESTAMPTZ, + -- v0.29.1: salience-and-recency, additive opt-in. All NULL by default; + -- only consulted when a caller passes \`salience='on'\` / \`recency='on'\` or + -- the new \`since\`/\`until\` filter. effective_date_source is a sentinel for + -- the doctor's effective_date_health check (values: 'event_date' | 'date' + -- | 'published' | 'filename' | 'fallback'). salience_touched_at is bumped + -- by recompute_emotional_weight when emotional_weight changes so the + -- salience window picks up newly-salient old pages. + effective_date TIMESTAMPTZ, + effective_date_source TEXT, + import_filename TEXT, + salience_touched_at TIMESTAMPTZ, CONSTRAINT pages_source_slug_key UNIQUE (source_id, slug) ); @@ -105,6 +120,12 @@ CREATE INDEX IF NOT EXISTS idx_pages_source_id ON pages(source_id); -- stays low. Don't add a regular \`(deleted_at)\` index without measuring. CREATE INDEX IF NOT EXISTS pages_deleted_at_purge_idx ON pages (deleted_at) WHERE deleted_at IS NOT NULL; +-- v0.29.1: expression index used by since/until date-range filters that read +-- COALESCE(effective_date, updated_at). A partial index on effective_date +-- alone would NOT help — the planner can't use it for the negative side of +-- the COALESCE. Expression index is what actually accelerates the filter. +CREATE INDEX IF NOT EXISTS pages_coalesce_date_idx + ON pages ((COALESCE(effective_date, updated_at))); -- ============================================================ -- content_chunks: chunked content with embeddings @@ -746,7 +767,16 @@ CREATE TABLE IF NOT EXISTS eval_candidates ( remote BOOLEAN NOT NULL, job_id INTEGER, subagent_id INTEGER, - created_at TIMESTAMPTZ NOT NULL DEFAULT NOW() + created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(), + -- v0.29.1 — agent-explicit recency + salience capture for replay reproducibility. + -- All nullable + additive. NDJSON schema_version stays at 1; consumers ignore unknown fields. + as_of_ts TIMESTAMPTZ, + salience_param TEXT, + recency_param TEXT, + salience_resolved TEXT, + recency_resolved TEXT, + salience_source TEXT, + recency_source TEXT ); CREATE INDEX IF NOT EXISTS idx_eval_candidates_created_at ON eval_candidates(created_at DESC); diff --git a/src/core/search/hybrid.ts b/src/core/search/hybrid.ts index b75275e6b..96e29ad7e 100644 --- a/src/core/search/hybrid.ts +++ b/src/core/search/hybrid.ts @@ -14,7 +14,7 @@ import { MAX_SEARCH_LIMIT, clampSearchLimit } from '../engine.ts'; import type { SearchResult, SearchOpts, HybridSearchMeta } from '../types.ts'; import { embed } from '../embedding.ts'; import { dedupResults } from './dedup.ts'; -import { autoDetectDetail } from './intent.ts'; +import { autoDetectDetail, classifyQuery } from './query-intent.ts'; import { expandAnchors, hydrateChunks } from './two-pass.ts'; const RRF_K = 60; @@ -45,6 +45,147 @@ export function applyBacklinkBoost(results: SearchResult[], counts: Map, + strength: 'on' | 'strong', +): void { + const k = strength === 'strong' ? 0.30 : 0.15; + for (const r of results) { + const key = `${r.source_id ?? 'default'}::${r.slug}`; + const score = scores.get(key); + if (!score || score <= 0) continue; + r.score *= (1.0 + k * Math.log(1 + score)); + } +} + +/** + * v0.29.1 — apply per-prefix recency boost. Mutate-in-place; caller re-sorts. + * + * `dates` is keyed by `${source_id}::${slug}`. The boost factor for each + * page comes from the per-prefix decay map: `1 + coefficient × halflife / + * (halflife + days_old)`. Evergreen prefixes (halflifeDays=0) contribute 0 + * (factor stays 1.0). + * + * strength: 'on' multiplies the coefficient by 1.0; 'strong' multiplies by + * 1.5 (more aggressive recency tilt). Pages with no date entry in the map + * are skipped (factor 1.0). + */ +export function applyRecencyBoost( + results: SearchResult[], + dates: Map, + strength: 'on' | 'strong', + decayMap: import('./recency-decay.ts').RecencyDecayMap, + fallback: import('./recency-decay.ts').RecencyDecayConfig, + nowMs: number = Date.now(), +): void { + const strengthMul = strength === 'strong' ? 1.5 : 1.0; + // Sort prefixes longest-first so 'media/articles/' matches before 'media/'. + const prefixes = Object.keys(decayMap).sort((a, b) => b.length - a.length); + + for (const r of results) { + const key = `${r.source_id ?? 'default'}::${r.slug}`; + const d = dates.get(key); + if (!d) continue; + const daysOld = Math.max(0, (nowMs - d.getTime()) / 86_400_000); + + // Find first matching prefix. + let cfg: import('./recency-decay.ts').RecencyDecayConfig = fallback; + for (const p of prefixes) { + if (r.slug.startsWith(p)) { + cfg = decayMap[p]; + break; + } + } + + if (cfg.halflifeDays === 0 || cfg.coefficient === 0) continue; // evergreen + const recencyComponent = cfg.coefficient * cfg.halflifeDays / (cfg.halflifeDays + daysOld); + const factor = 1.0 + strengthMul * recencyComponent; + r.score *= factor; + } +} + +/** + * v0.29.1 — runPostFusionStages: wrap backlink + salience + recency in a + * single stage that fires from EVERY hybridSearch return path (codex + * pass-1 #2 + pass-2 #4: keyword-only, embed-fail-fallback, full-hybrid). + * Without this wrapper, salience='on' silently does nothing on keyless + * installs that fall back to keyword-only. + * + * Mutates `results` in place; caller re-sorts. + */ +export interface PostFusionOpts { + applyBacklinks: boolean; + salience: 'off' | 'on' | 'strong'; + recency: 'off' | 'on' | 'strong'; + decayMap?: import('./recency-decay.ts').RecencyDecayMap; + fallback?: import('./recency-decay.ts').RecencyDecayConfig; +} + +export async function runPostFusionStages( + engine: import('../engine.ts').BrainEngine, + results: SearchResult[], + opts: PostFusionOpts, +): Promise { + if (results.length === 0) return; + + // Backlink stage (existing behavior, preserved). + if (opts.applyBacklinks) { + try { + const slugs = Array.from(new Set(results.map(r => r.slug))); + const counts = await engine.getBacklinkCounts(slugs); + applyBacklinkBoost(results, counts); + } catch { + // Non-fatal; preserves the existing pre-v0.29.1 contract. + } + } + + // Composite refs for the orthogonal axes (multi-source isolation). + const refs = Array.from( + new Map( + results.map(r => [`${r.source_id ?? 'default'}::${r.slug}`, { slug: r.slug, source_id: r.source_id ?? 'default' }]), + ).values(), + ); + + // Salience stage (mattering, no time). + if (opts.salience !== 'off') { + try { + const scores = await engine.getSalienceScores(refs); + applySalienceBoost(results, scores, opts.salience); + } catch { + // Non-fatal. + } + } + + // Recency stage (per-prefix decay, no mattering). + if (opts.recency !== 'off') { + try { + const dates = await engine.getEffectiveDates(refs); + const { DEFAULT_RECENCY_DECAY, DEFAULT_FALLBACK } = await import('./recency-decay.ts'); + applyRecencyBoost( + results, + dates, + opts.recency, + opts.decayMap ?? DEFAULT_RECENCY_DECAY, + opts.fallback ?? DEFAULT_FALLBACK, + ); + } catch { + // Non-fatal. + } + } +} + export interface HybridSearchOpts extends SearchOpts { expansion?: boolean; expandFn?: (query: string) => Promise; @@ -86,6 +227,11 @@ export async function hybridSearch( // per-engine searchKeyword / searchVector apply the filters at SQL level. language: opts?.language, symbolKind: opts?.symbolKind, + // v0.29.1: since/until take precedence over deprecated afterDate/beforeDate. + // The engine still consumes the legacy field names; this aliasing keeps + // PR #618 callers compiling while the new names are the public surface. + afterDate: opts?.since ?? opts?.afterDate, + beforeDate: opts?.until ?? opts?.beforeDate, }; // Track what actually ran for the optional onMeta callback (v0.25.0). // Caller leaves onMeta undefined → these flags are computed but never @@ -111,20 +257,30 @@ export async function hybridSearch( // Run keyword search (always available, no API key needed) const keywordResults = await engine.searchKeyword(query, searchOpts); + // v0.29.1: resolve salience/recency from caller (back-compat aliases for + // PR #618's `recencyBoost` numeric scale) or fall back to the heuristic. + // The wrapper fires from ALL THREE return paths (codex pass-1 #2 + pass-2 #4). + const suggestions = classifyQuery(query); + // Back-compat: recencyBoost: 1|2 → 'on'|'strong'; 0 → 'off'. + const legacyRecency: 'off' | 'on' | 'strong' | undefined = + opts?.recencyBoost === 2 ? 'strong' : + opts?.recencyBoost === 1 ? 'on' : + opts?.recencyBoost === 0 ? 'off' : + undefined; + const salienceMode: 'off' | 'on' | 'strong' = opts?.salience ?? suggestions.suggestedSalience; + const recencyMode: 'off' | 'on' | 'strong' = opts?.recency ?? legacyRecency ?? suggestions.suggestedRecency; + const postFusionOpts = { + applyBacklinks: true, + salience: salienceMode, + recency: recencyMode, + }; + // Skip vector search entirely if the gateway has no embedding provider configured (Codex C3). const { isAvailable } = await import('../ai/gateway.ts'); if (!isAvailable('embedding')) { - // Apply backlink boost in keyword-only path too. One getBacklinkCounts query - // per search request; not N+1. if (keywordResults.length > 0) { - try { - const slugs = Array.from(new Set(keywordResults.map(r => r.slug))); - const counts = await engine.getBacklinkCounts(slugs); - applyBacklinkBoost(keywordResults, counts); - keywordResults.sort((a, b) => b.score - a.score); - } catch { - // Boost failure is non-fatal: keep unboosted ranking. - } + await runPostFusionStages(engine, keywordResults, postFusionOpts); + keywordResults.sort((a, b) => b.score - a.score); } emitMeta({ vector_enabled: false, detail_resolved: detailResolved, expansion_applied: false }); return dedupResults(keywordResults).slice(offset, offset + limit); @@ -160,6 +316,13 @@ export async function hybridSearch( if (vectorLists.length === 0) { // Embed/vector failed silently; record that vector did not run. + // v0.29.1 codex pass-2 #4: this is the third return path. Apply + // post-fusion stages here too — without it, salience='on' silently + // does nothing on embed failures. + if (keywordResults.length > 0) { + await runPostFusionStages(engine, keywordResults, postFusionOpts); + keywordResults.sort((a, b) => b.score - a.score); + } emitMeta({ vector_enabled: false, detail_resolved: detailResolved, expansion_applied: expansionApplied }); return dedupResults(keywordResults).slice(offset, offset + limit); } @@ -174,18 +337,13 @@ export async function hybridSearch( fused = await cosineReScore(engine, fused, queryEmbedding); } - // Apply backlink boost AFTER cosine re-score so the boost survives normalization, - // and BEFORE dedup so it influences which chunks per page survive deduplication. - // One DB query for the whole result set (not N+1). + // v0.29.1: post-fusion stages (backlink + salience + recency) run via + // runPostFusionStages so all three early-return paths share the same + // boost surface. Salience and recency are independent axes — either, + // both, or neither fires depending on resolved modes. if (fused.length > 0) { - try { - const slugs = Array.from(new Set(fused.map(r => r.slug))); - const counts = await engine.getBacklinkCounts(slugs); - applyBacklinkBoost(fused, counts); - fused.sort((a, b) => b.score - a.score); - } catch { - // Boost failure is non-fatal: keep blended cosine ranking. - } + await runPostFusionStages(engine, fused, postFusionOpts); + fused.sort((a, b) => b.score - a.score); } // v0.20.0 Cathedral II Layer 7 (A2): two-pass structural expansion. @@ -232,6 +390,11 @@ export async function hybridSearch( } } + // v0.27.0 PR #618 recency boost was here; v0.29.1 unifies it into + // runPostFusionStages above so all three return paths get the same + // treatment. PR #618's recencyBoost: 0|1|2 still works via back-compat + // aliasing in the postFusionOpts resolver near line ~256. + // Dedup const deduped = dedupResults(fused, dedupOpts); diff --git a/src/core/search/intent.ts b/src/core/search/intent.ts deleted file mode 100644 index e28e8f197..000000000 --- a/src/core/search/intent.ts +++ /dev/null @@ -1,108 +0,0 @@ -/** - * Query Intent Classifier - * - * Zero-latency heuristic classifier that detects query intent from text patterns. - * Maps intent to the appropriate detail level for hybrid search. - * - * No LLM call, no API cost, no latency. Pattern matching on query text. - */ - -export type QueryIntent = 'entity' | 'temporal' | 'event' | 'general'; - -// Temporal patterns: questions about when things happened, meeting history -const TEMPORAL_PATTERNS = [ - /\bwhen\b/i, - /\blast\s+(met|meeting|call|conversation|chat|talked|spoke|seen|heard|time)\b/i, - /\brecent(ly)?\b/i, - /\bhistory\b/i, - /\btimeline\b/i, - /\bmeeting\s+notes?\b/i, - /\bwhat('s| is| was)\s+new\b/i, - /\blatest\b/i, - /\bupdate(s)?\s+(on|from|about)\b/i, - /\bhow\s+long\s+(ago|since)\b/i, - /\b\d{4}[-/]\d{2}\b/i, // date pattern like 2024-03 - /\blast\s+(week|month|quarter|year)\b/i, -]; - -// Event patterns: specific events, announcements, launches -const EVENT_PATTERNS = [ - /\bannounce[ds]?(ment)?\b/i, - /\blaunch(ed|es|ing)?\b/i, - /\braised?\s+\$?\d/i, - /\bfund(ing|raise)\b/i, - /\bIPO\b/i, - /\bacquisition\b/i, - /\bmerge[drs]?\b/i, - /\bnews\b/i, - /\bhappened?\b/i, -]; - -// Entity patterns: identity questions, overviews -const ENTITY_PATTERNS = [ - /\bwho\s+is\b/i, - /\bwhat\s+(is|does|are)\b/i, - /\btell\s+me\s+about\b/i, - /\bdescribe\b/i, - /\bsummar(y|ize)\b/i, - /\boverview\b/i, - /\bbackground\b/i, - /\bprofile\b/i, - /\bwhat\s+do\s+(you|we)\s+know\b/i, -]; - -// Full-context patterns: requests for everything -const FULL_CONTEXT_PATTERNS = [ - /\beverything\b/i, - /\ball\s+(about|info|information|details)\b/i, - /\bfull\s+(history|context|picture|story|details)\b/i, - /\bcomprehensive\b/i, - /\bdeep\s+dive\b/i, - /\bgive\s+me\s+everything\b/i, -]; - -/** - * Classify query intent from text patterns. - * Returns the detected intent type. - */ -export function classifyQueryIntent(query: string): QueryIntent { - // Full context requests → treat as temporal (return everything) - if (FULL_CONTEXT_PATTERNS.some(p => p.test(query))) return 'temporal'; - - // Check temporal patterns first (highest priority for detail=high) - if (TEMPORAL_PATTERNS.some(p => p.test(query))) return 'temporal'; - - // Check event patterns - if (EVENT_PATTERNS.some(p => p.test(query))) return 'event'; - - // Check entity patterns - if (ENTITY_PATTERNS.some(p => p.test(query))) return 'entity'; - - // Default: general query - return 'general'; -} - -/** - * Map query intent to detail level. - * - * entity → 'low' (compiled truth only, user wants the assessment) - * temporal → 'high' (need timeline, user wants dates/events) - * event → 'high' (need timeline, user wants specific events) - * general → undefined (use default medium, let the boost handle it) - */ -export function intentToDetail(intent: QueryIntent): 'low' | 'medium' | 'high' | undefined { - switch (intent) { - case 'entity': return 'low'; - case 'temporal': return 'high'; - case 'event': return 'high'; - case 'general': return undefined; // use default - } -} - -/** - * Auto-detect detail level from query text. - * Returns undefined if no strong signal detected (uses default). - */ -export function autoDetectDetail(query: string): 'low' | 'medium' | 'high' | undefined { - return intentToDetail(classifyQueryIntent(query)); -} diff --git a/src/core/search/query-intent.ts b/src/core/search/query-intent.ts new file mode 100644 index 000000000..66762afa1 --- /dev/null +++ b/src/core/search/query-intent.ts @@ -0,0 +1,253 @@ +/** + * v0.29.1 — merged query-intent classifier. + * + * Replaces v0.29.0's `intent.ts` (which only emitted a detail suggestion). + * After D1 + D4 the codebase needs ONE classifier that returns three + * suggestions from a single regex pass: + * + * - intent: original v0.29.0 type ('entity' | 'temporal' | 'event' | 'general') + * - suggestedDetail: v0.29.0 mapping (entity→low, temporal/event→high) + * - suggestedSalience: NEW for v0.29.1 — 'off' | 'on' | 'strong' + * - suggestedRecency: NEW for v0.29.1 — 'off' | 'on' | 'strong' + * + * Salience and recency are TRULY ORTHOGONAL (per D9): + * - salience boosts pages with high emotional_weight + take_count (mattering) + * - recency boosts pages with recent effective_date (per-prefix decay) + * Both can fire, neither can fire, or just one. + * + * The classifier follows "current state → on. canonical truth → off." with + * a NARROW exception per D6: explicit temporal bounds (today / this week / + * right now / since X / last N days) override canonical-pattern wins. So + * "who is X right now" → suggestedRecency='on' even though "who is" is a + * canonical pattern. + * + * Pure module. No DB, no LLM, no async. Tested in test/query-intent.test.ts. + */ + +export type QueryIntent = 'entity' | 'temporal' | 'event' | 'general'; + +export type SalienceMode = 'off' | 'on' | 'strong'; +export type RecencyMode = 'off' | 'on' | 'strong'; + +export interface QuerySuggestions { + intent: QueryIntent; + /** v0.29.0 detail mapping. entity→low, temporal/event→high, general→undefined. */ + suggestedDetail: 'low' | 'medium' | 'high' | undefined; + /** v0.29.1 — emotional_weight + take_count boost. */ + suggestedSalience: SalienceMode; + /** v0.29.1 — per-prefix age-decay boost. */ + suggestedRecency: RecencyMode; +} + +// ───────────────────────────────────────────────────────── +// Pattern banks (organized by axis they signal) +// ───────────────────────────────────────────────────────── + +// Original v0.29.0 intent patterns. Drive .intent + .suggestedDetail. +const TEMPORAL_PATTERNS = [ + /\bwhen\b/i, + /\blast\s+(met|meeting|call|conversation|chat|talked|spoke|seen|heard|time)\b/i, + /\brecent(ly)?\b/i, + /\bhistory\b/i, + /\btimeline\b/i, + /\bmeeting\s+notes?\b/i, + /\bwhat('s| is| was)\s+new\b/i, + /\blatest\b/i, + /\bupdate(s)?\s+(on|from|about)\b/i, + /\bhow\s+long\s+(ago|since)\b/i, + /\b\d{4}[-/]\d{2}\b/i, + /\blast\s+(week|month|quarter|year)\b/i, +]; + +const EVENT_PATTERNS = [ + /\bannounce[ds]?(ment)?\b/i, + /\blaunch(ed|es|ing)?\b/i, + /\braised?\s+\$?\d/i, + /\bfund(ing|raise)\b/i, + /\bIPO\b/i, + /\bacquisition\b/i, + /\bmerge[drs]?\b/i, + /\bnews\b/i, + /\bhappened?\b/i, +]; + +const ENTITY_PATTERNS = [ + /\bwho\s+is\b/i, + /\bwhat\s+(is|does|are)\b/i, + /\btell\s+me\s+about\b/i, + /\bdescribe\b/i, + /\bsummar(y|ize)\b/i, + /\boverview\b/i, + /\bbackground\b/i, + /\bprofile\b/i, + /\bwhat\s+do\s+(you|we)\s+know\b/i, +]; + +const FULL_CONTEXT_PATTERNS = [ + /\beverything\b/i, + /\ball\s+(about|info|information|details)\b/i, + /\bfull\s+(history|context|picture|story|details)\b/i, + /\bcomprehensive\b/i, + /\bdeep\s+dive\b/i, + /\bgive\s+me\s+everything\b/i, +]; + +// v0.29.1 — recency-axis patterns +// +// Canonical patterns: queries asking for the authoritative / definitional +// answer. These signal recency='off' even when other axes match — UNLESS +// an explicit temporal bound is present (per D6 narrow exception). +const CANONICAL_PATTERNS = [ + /\bwho\s+is\b/i, + /\bwhat\s+(is|are|does|means?)\b/i, + /\bdefin(e|ition|ing)\b/i, + /\bexplain\s+(what|how|why)\b/i, + /\b(history|origin|background)\s+of\b/i, + /\bconcept\s+of\b/i, + /\boverview\s+of\b/i, + /\btell\s+me\s+about\b/i, + /\bcompiled\s+truth\b/i, + /::|->|\.\w+\(/, + /\b(function|class|method|module)\s+\w+/i, + /\b(graph|traversal|backlinks?|inbound|outbound)\b/i, +]; + +// Aggressive recency: "today", "right now", "this morning", "just now". +const STRONG_RECENCY_PATTERNS = [ + /\btoday\b/i, + /\bright\s+now\b/i, + /\bthis\s+morning\b/i, + /\bjust\s+now\b/i, +]; + +// Moderate recency: "what's going on", "latest", "recent", "this week", +// meeting prep, conversation recall, status updates. +const RECENCY_ON_PATTERNS = [ + /\bwhat'?s\s+(going\s+on|happening|new|latest|up)\b/i, + /\b(latest|recent(ly)?|currently)\b/i, + /\b(this|last|past)\s+(week|month|few\s+days|couple\s+days)\b/i, + /\bmeeting\s+(prep|with|for|notes?|brief)\b/i, + /\bbefore\s+(my|the|our)\s+(meeting|call|sync|chat)\b/i, + /\bprep(are)?\s+(for|me)\b/i, + /\bcatch(es|ing)?\b[\s\w]{0,15}\bup\b/i, // "catch up", "catch me up", "catching X up" + /\bremind\s+me\s+(what|about|of)\b/i, + /\b(update|status|progress)\s+(on|with|from)\b/i, +]; + +// Per D6: explicit temporal bounds override canonical-wins. "Who is X today" +// → recency='on' (temporal bound wins). "Who is X" alone → recency='off'. +const EXPLICIT_TEMPORAL_BOUND_PATTERNS = [ + /\btoday\b/i, + /\bright\s+now\b/i, + /\bthis\s+morning\b/i, + /\bthis\s+week\b/i, + /\bsince\s+(launch|last|the|\d)/i, + /\blast\s+\d+\s+(day|days|week|weeks|month|months)\b/i, +]; + +// v0.29.1 — salience-axis patterns +// +// Salience suggests "what matters in this brain right now" — when the user +// is asking about people/companies/deals in the current context, they +// usually want the emotionally-weighted + take-rich pages to surface. +// Salience patterns are a subset of recency-on patterns (meeting prep, +// catch-up, update language) plus people-centric phrasings. +const SALIENCE_ON_PATTERNS = [ + /\bwhat'?s\s+(going\s+on|happening|been\s+going|been\s+up)\b/i, + /\bcatch(es|ing)?\b[\s\w]{0,15}\bup\b/i, + /\bremind\s+me\s+(what|about|of)\b/i, + /\bprep(are)?\s+(for|me)\b/i, + /\bbefore\s+(my|the|our)\s+(meeting|call|sync|chat)\b/i, + /\bmeeting\s+(prep|with|for|brief)\b/i, + /\b(update|status|progress)\s+(on|with|from)\b/i, + /\bwhat\s+matters\b/i, + /\bwhat'?s\s+important\b/i, +]; + +// ───────────────────────────────────────────────────────── +// Classifier +// ───────────────────────────────────────────────────────── + +function matches(patterns: RegExp[], q: string): boolean { + for (const re of patterns) if (re.test(q)) return true; + return false; +} + +/** + * Classify a query and return all three axis suggestions. + * + * Resolution rules: + * - intent: original v0.29.0 priority (full-context > temporal > event > entity > general) + * - suggestedDetail: intent → detail mapping (entity=low, temporal/event=high) + * - suggestedRecency: STRONG_RECENCY > RECENCY_ON; CANONICAL wins UNLESS + * EXPLICIT_TEMPORAL_BOUND also matches; default 'off' + * - suggestedSalience: SALIENCE_ON; CANONICAL wins UNLESS + * EXPLICIT_TEMPORAL_BOUND; default 'off' + * + * Note: salience and recency are independent. A "what's going on with X" + * query gets BOTH on; "who is X" gets BOTH off; "today's news" gets + * recency='strong' but salience='off' (the user wants newest, not + * emotionally-weighted). + */ +export function classifyQuery(query: string): QuerySuggestions { + const intent = classifyQueryIntent(query); + const suggestedDetail = intentToDetail(intent); + + const hasCanonical = matches(CANONICAL_PATTERNS, query); + const hasTemporalBound = matches(EXPLICIT_TEMPORAL_BOUND_PATTERNS, query); + const hasStrongRecency = matches(STRONG_RECENCY_PATTERNS, query); + const hasRecencyOn = matches(RECENCY_ON_PATTERNS, query); + const hasSalienceOn = matches(SALIENCE_ON_PATTERNS, query); + + // Recency axis + let suggestedRecency: RecencyMode; + if (hasCanonical && !hasTemporalBound) { + suggestedRecency = 'off'; + } else if (hasStrongRecency) { + suggestedRecency = 'strong'; + } else if (hasRecencyOn) { + suggestedRecency = 'on'; + } else { + suggestedRecency = 'off'; + } + + // Salience axis (orthogonal) + let suggestedSalience: SalienceMode; + if (hasCanonical && !hasTemporalBound) { + suggestedSalience = 'off'; + } else if (hasSalienceOn) { + suggestedSalience = 'on'; + } else { + suggestedSalience = 'off'; + } + + return { intent, suggestedDetail, suggestedSalience, suggestedRecency }; +} + +// ───────────────────────────────────────────────────────── +// v0.29.0 compatibility shims +// ───────────────────────────────────────────────────────── + +/** v0.29.0 intent type. Preserved verbatim for back-compat. */ +export function classifyQueryIntent(query: string): QueryIntent { + if (matches(FULL_CONTEXT_PATTERNS, query)) return 'temporal'; + if (matches(TEMPORAL_PATTERNS, query)) return 'temporal'; + if (matches(EVENT_PATTERNS, query)) return 'event'; + if (matches(ENTITY_PATTERNS, query)) return 'entity'; + return 'general'; +} + +/** v0.29.0 mapping. */ +export function intentToDetail(intent: QueryIntent): 'low' | 'medium' | 'high' | undefined { + switch (intent) { + case 'entity': return 'low'; + case 'temporal': return 'high'; + case 'event': return 'high'; + case 'general': return undefined; + } +} + +/** v0.29.0 helper. Routes through classifyQuery internally. */ +export function autoDetectDetail(query: string): 'low' | 'medium' | 'high' | undefined { + return classifyQuery(query).suggestedDetail; +} diff --git a/src/core/search/recency-decay.ts b/src/core/search/recency-decay.ts new file mode 100644 index 000000000..7e8c83af4 --- /dev/null +++ b/src/core/search/recency-decay.ts @@ -0,0 +1,201 @@ +/** + * v0.29.1 — Per-prefix recency decay map. + * + * Drives the recency boost ONLY (per D9 codex resolution). Salience is a + * separate orthogonal axis based on emotional_weight + take_count and + * does NOT consume this map. The two axes compose multiplicatively in + * runPostFusionStages when both opt in. + * + * Keyed by slug prefix. Longest-prefix-match wins (sorted at lookup time + * inside sql-ranking.ts). Defaults are GENERIC prefixes only (no fork- + * specific names like 'openclaw/chat/' — that's a privacy violation per + * CLAUDE.md and tracked in iteration-1 codex finding C-CX-3). + * + * Override priority (later wins): + * 1. DEFAULT_RECENCY_DECAY (this file) + * 2. gbrain.yml `recency:` section + * 3. GBRAIN_RECENCY_DECAY env var (prefix:halflifeDays:coefficient,...) + * 4. Per-call SearchOpts.recency_decay (tests + library consumers; not + * exposed on MCP) + * + * Per-prefix interpretation: + * - halflifeDays = 0 → evergreen, no decay (recency component = 0) + * - halflifeDays > 0 → hyperbolic decay; coefficient × halflife / (halflife + days_old) + * - At days_old=0: recency component = coefficient (max boost) + * - At days_old=halflife: recency component = coefficient / 2 + * + * Pure module. No side effects. Tested in test/recency-decay.test.ts. + */ + +export interface RecencyDecayConfig { + /** Days at which the recency component is halved. 0 = no decay (evergreen). */ + halflifeDays: number; + /** Max recency boost contribution at days_old = 0. Must be >= 0. */ + coefficient: number; +} + +export type RecencyDecayMap = Record; + +export const DEFAULT_RECENCY_DECAY: RecencyDecayMap = { + // Evergreen (curated, opinion, knowledge artifacts) — no decay. + // concepts/ is the canonical evergreen tier; originals/ + writing/ get + // long-tail decay so freshly-published essays do see a small nudge. + 'concepts/': { halflifeDays: 0, coefficient: 0 }, + 'originals/': { halflifeDays: 180, coefficient: 0.5 }, + 'writing/': { halflifeDays: 365, coefficient: 0.4 }, + + // Time-bound personal records — strongest decay, biggest coefficient. + // The user is asking "what was on my plate this week" / "what did we + // discuss in our 1:1"; freshness IS the signal. + 'daily/': { halflifeDays: 14, coefficient: 1.5 }, + 'meetings/': { halflifeDays: 60, coefficient: 1.0 }, + + // Bulk feeds — generic prefixes only. Real fork names go in user + // gbrain.yml, never in shipped defaults. + 'chat/': { halflifeDays: 7, coefficient: 1.0 }, + 'media/x/': { halflifeDays: 7, coefficient: 1.5 }, + 'media/articles/': { halflifeDays: 90, coefficient: 0.5 }, + + // Entities — slow decay (a deal from 2 years ago is still relevant + // to a current portfolio query; less so to "what's new lately"). + 'people/': { halflifeDays: 365, coefficient: 0.3 }, + 'companies/': { halflifeDays: 365, coefficient: 0.3 }, + 'deals/': { halflifeDays: 180, coefficient: 0.5 }, +}; + +/** Fallback applied to slugs that don't match any default or override prefix. */ +export const DEFAULT_FALLBACK: RecencyDecayConfig = { + halflifeDays: 90, + coefficient: 0.5, +}; + +/** Sentinel error thrown by parsers; CLI catches it and exits with a useful message. */ +export class RecencyDecayParseError extends Error { + constructor(message: string, public readonly source: 'env' | 'yaml' | 'caller') { + super(message); + this.name = 'RecencyDecayParseError'; + } +} + +/** + * Parse the GBRAIN_RECENCY_DECAY env var. + * Format: comma-separated `prefix:halflifeDays:coefficient` triples. + * Example: "daily/:7:2.0,concepts/:0:0,custom/:30:1.0" + * + * Refuses on parse error (codex M-CX-3 / iteration-2 review). The source-boost + * env parser silently skipped malformed entries; that pattern bit users for + * years. Recency parser fails LOUD so misconfigurations surface at startup + * instead of silently degrading rankings. + */ +export function parseRecencyDecayEnv(env: string | undefined): RecencyDecayMap { + if (!env) return {}; + const out: RecencyDecayMap = {}; + const triples = env.split(',').map(s => s.trim()).filter(Boolean); + for (const triple of triples) { + // Prefix can't contain `:` because the field separator is `:`. We split + // on the FIRST and SECOND `:` from the right so the prefix may safely + // contain `/` etc. but NOT colons. + const lastIdx = triple.lastIndexOf(':'); + if (lastIdx <= 0) { + throw new RecencyDecayParseError( + `Invalid GBRAIN_RECENCY_DECAY entry "${triple}": expected prefix:halflife:coefficient`, + 'env', + ); + } + const beforeLast = triple.slice(0, lastIdx); + const middleIdx = beforeLast.lastIndexOf(':'); + if (middleIdx <= 0) { + throw new RecencyDecayParseError( + `Invalid GBRAIN_RECENCY_DECAY entry "${triple}": expected prefix:halflife:coefficient`, + 'env', + ); + } + const prefix = triple.slice(0, middleIdx).trim(); + const halflifeRaw = triple.slice(middleIdx + 1, lastIdx).trim(); + const coefficientRaw = triple.slice(lastIdx + 1).trim(); + const halflife = Number.parseFloat(halflifeRaw); + const coefficient = Number.parseFloat(coefficientRaw); + if (!prefix) { + throw new RecencyDecayParseError(`Empty prefix in GBRAIN_RECENCY_DECAY entry "${triple}"`, 'env'); + } + if (!Number.isFinite(halflife) || halflife < 0) { + throw new RecencyDecayParseError( + `Invalid halflifeDays "${halflifeRaw}" in GBRAIN_RECENCY_DECAY (must be number >= 0; 0 = evergreen)`, + 'env', + ); + } + if (!Number.isFinite(coefficient) || coefficient < 0) { + throw new RecencyDecayParseError( + `Invalid coefficient "${coefficientRaw}" in GBRAIN_RECENCY_DECAY (must be number >= 0)`, + 'env', + ); + } + out[prefix] = { halflifeDays: halflife, coefficient }; + } + return out; +} + +/** + * Parse a `recency:` section from a parsed gbrain.yml. The shape is: + * recency: + * daily/: { halflifeDays: 14, coefficient: 1.5 } + * concepts/: { halflifeDays: 0, coefficient: 0 } + * + * `parsed` is the already-parsed YAML object. This is a pure transform. + * Caller is responsible for reading + parsing the YAML file. + */ +export function parseRecencyDecayYaml(parsed: unknown): RecencyDecayMap { + if (parsed == null) return {}; + if (typeof parsed !== 'object' || Array.isArray(parsed)) return {}; + const obj = parsed as Record; + const recency = obj.recency; + if (recency == null) return {}; + if (typeof recency !== 'object' || Array.isArray(recency)) { + throw new RecencyDecayParseError(`gbrain.yml recency: must be a map, got ${typeof recency}`, 'yaml'); + } + const out: RecencyDecayMap = {}; + for (const [prefix, raw] of Object.entries(recency as Record)) { + if (typeof raw !== 'object' || raw === null || Array.isArray(raw)) { + throw new RecencyDecayParseError( + `gbrain.yml recency."${prefix}" must be an object with halflifeDays + coefficient`, + 'yaml', + ); + } + const cfg = raw as Record; + const halflife = Number(cfg.halflifeDays); + const coefficient = Number(cfg.coefficient); + if (!Number.isFinite(halflife) || halflife < 0) { + throw new RecencyDecayParseError( + `gbrain.yml recency."${prefix}".halflifeDays invalid (must be number >= 0)`, + 'yaml', + ); + } + if (!Number.isFinite(coefficient) || coefficient < 0) { + throw new RecencyDecayParseError( + `gbrain.yml recency."${prefix}".coefficient invalid (must be number >= 0)`, + 'yaml', + ); + } + out[prefix] = { halflifeDays: halflife, coefficient }; + } + return out; +} + +/** + * Merge defaults + yaml + env + caller-supplied overrides into the effective + * decay map. Later sources win. Empty entries are dropped. + */ +export function resolveRecencyDecayMap(opts: { + yaml?: unknown; + envValue?: string; + caller?: RecencyDecayMap; +} = {}): RecencyDecayMap { + const fromYaml = opts.yaml !== undefined ? parseRecencyDecayYaml(opts.yaml) : {}; + const fromEnv = parseRecencyDecayEnv(opts.envValue ?? process.env.GBRAIN_RECENCY_DECAY); + return { + ...DEFAULT_RECENCY_DECAY, + ...fromYaml, + ...fromEnv, + ...(opts.caller ?? {}), + }; +} diff --git a/src/core/search/sql-ranking.ts b/src/core/search/sql-ranking.ts index 0feca067b..7dfff0828 100644 --- a/src/core/search/sql-ranking.ts +++ b/src/core/search/sql-ranking.ts @@ -129,5 +129,87 @@ export function buildVisibilityClause(pageAlias: string, sourceAlias: string): s return `AND ${pageAlias}.deleted_at IS NULL AND NOT ${sourceAlias}.archived`; } +// ============================================================ +// v0.29.1 — Recency component SQL builder +// ============================================================ + +/** + * Typed expression for "what NOW() should be" in the SQL. Tests pass + * `{ kind: 'fixed', isoUtc }` for deterministic output regardless of wall + * clock. Production callers leave it default (`{ kind: 'now' }`). + * + * The builder constructs the SQL literal internally via escapeSqlLiteral + * for the 'fixed' branch — caller-supplied strings NEVER flow into raw SQL, + * preventing the injection vector codex pass-1 #5 flagged. + */ +export type NowExpr = { kind: 'now' } | { kind: 'fixed'; isoUtc: string }; + +function nowExprToSql(now: NowExpr): string { + if (now.kind === 'now') return 'NOW()'; + return `'${escapeSqlLiteral(now.isoUtc)}'::timestamptz`; +} + +/** + * Build the per-row recency component SQL fragment. + * + * For each prefix in the decay map, emit one CASE branch: + * - halflifeDays = 0 (or coefficient = 0) → literal 0 (evergreen short-circuit) + * - halflifeDays > 0 → coefficient * halflife / (halflife + days_old) + * + * Prefixes sorted longest-first so 'media/articles/' matches before 'media/' + * (mirror of buildSourceFactorCase's ordering). + * + * Output is a single SQL expression suitable for SELECT / ORDER BY. + * + * @param slugColumn — qualified column reference (engine-supplied, trusted) + * @param dateExpr — qualified expression for the page's effective date + * (typically `COALESCE(p.effective_date, p.updated_at)`) + * @param decayMap — per-prefix configurations (resolved from defaults + + * yaml + env + caller) + * @param fallback — applied to slugs matching no prefix + * @param now — typed NOW() expression (default `{ kind: 'now' }`) + */ +export function buildRecencyComponentSql(opts: { + slugColumn: string; + dateExpr: string; + decayMap: import('./recency-decay.ts').RecencyDecayMap; + fallback: import('./recency-decay.ts').RecencyDecayConfig; + now?: NowExpr; +}): string { + const { slugColumn, dateExpr, decayMap, fallback } = opts; + const now = opts.now ?? { kind: 'now' }; + const nowSql = nowExprToSql(now); + const daysOldSql = `EXTRACT(EPOCH FROM (${nowSql} - ${dateExpr})) / 86400.0`; + + const prefixes = Object.keys(decayMap).sort((a, b) => b.length - a.length); + const branches: string[] = []; + + for (const prefix of prefixes) { + const cfg = decayMap[prefix]; + const literal = buildLikePrefixLiteral(prefix); + if (cfg.halflifeDays === 0 || cfg.coefficient === 0) { + branches.push(`WHEN ${slugColumn} LIKE ${literal} THEN 0`); + } else { + const h = cfg.halflifeDays; + const c = cfg.coefficient; + branches.push( + `WHEN ${slugColumn} LIKE ${literal} THEN ${c} * ${h}.0 / (${h}.0 + ${daysOldSql})`, + ); + } + } + + let elseSql: string; + if (fallback.halflifeDays === 0 || fallback.coefficient === 0) { + elseSql = '0'; + } else { + const h = fallback.halflifeDays; + const c = fallback.coefficient; + elseSql = `${c} * ${h}.0 / (${h}.0 + ${daysOldSql})`; + } + + if (branches.length === 0) return `(${elseSql})`; + return `(CASE ${branches.join(' ')} ELSE ${elseSql} END)`; +} + // Exported for unit tests export const __test__ = { escapeLikePattern, escapeSqlLiteral, buildLikePrefixLiteral }; diff --git a/src/core/transcripts.ts b/src/core/transcripts.ts new file mode 100644 index 000000000..fb0f2a8ad --- /dev/null +++ b/src/core/transcripts.ts @@ -0,0 +1,140 @@ +/** + * v0.29 — Recent transcripts: read raw `.txt` transcript files from the dream + * cycle's corpus directories and return one-line summaries (or full content) + * filtered by mtime. + * + * Reuses the same corpus-dir resolution + dream-output guard as the v0.23 + * synthesize phase. Specifically does NOT depend on or call into the dream + * cycle — this is a simple read-only filesystem walk for human / CLI / MCP- + * via-local-CLI consumption. + * + * Trust: the calling op (`get_recent_transcripts`) gates on `ctx.remote === false` + * so MCP/HTTP can't reach this function with attacker-controlled inputs. CLI + * callers are trusted; the cycle calls `discoverTranscripts` directly. + */ + +import { readFileSync, readdirSync, statSync } from 'node:fs'; +import { join, basename } from 'node:path'; +import type { BrainEngine } from './engine.ts'; +import { isDreamOutput } from './cycle/transcript-discovery.ts'; + +export interface RecentTranscriptOpts { + /** Window in days. Default 7. */ + days?: number; + /** When true (default), return ~300-char summary. When false, full content (capped at 100 KB). */ + summary?: boolean; + /** Max transcripts (default 50). */ + limit?: number; +} + +export interface RecentTranscript { + /** Filename basename (no directory). */ + path: string; + /** Inferred date if filename matches `YYYY-MM-DD...`, else null. */ + date: string | null; + /** Modified time (ISO). */ + mtime: string; + /** Full file size in bytes (regardless of summary mode). */ + length: number; + /** + * When summary=true: first non-empty line + next ~250 chars. Cheap, deterministic. + * When summary=false: file content capped at 100 KB. + */ + summary: string; +} + +const DATE_RE = /^(\d{4}-\d{2}-\d{2})/; +const FULL_READ_CAP = 100 * 1024; +const SUMMARY_HEAD_CHARS = 250; + +/** + * Walk the corpus directories configured for the dream cycle, filter to `.txt` + * files modified within `days`, skip dream-generated outputs, and return + * summaries sorted newest first. + * + * Returns [] (not error) when no corpus dir is configured or the dir is empty. + */ +export async function listRecentTranscripts( + engine: BrainEngine, + opts: RecentTranscriptOpts = {}, +): Promise { + const days = Math.max(0, opts.days ?? 7); + const summary = opts.summary !== false; + const limit = Math.max(1, Math.min(opts.limit ?? 50, 500)); + + const dirs: string[] = []; + const sessionDir = await engine.getConfig('dream.synthesize.session_corpus_dir'); + const meetingDir = await engine.getConfig('dream.synthesize.meeting_transcripts_dir'); + if (sessionDir) dirs.push(sessionDir); + if (meetingDir) dirs.push(meetingDir); + if (dirs.length === 0) return []; + + const cutoffMs = Date.now() - days * 86400000; + + const candidates: { path: string; mtimeMs: number; size: number }[] = []; + for (const dir of dirs) { + let entries: string[]; + try { + entries = readdirSync(dir); + } catch { + // Missing dir or permission error → skip silently. The op deliberately + // doesn't surface filesystem-level diagnostics; users running into this + // path should `gbrain doctor` to debug. + continue; + } + for (const name of entries) { + if (!name.endsWith('.txt')) continue; + const full = join(dir, name); + let st; + try { + st = statSync(full); + } catch { + continue; + } + if (!st.isFile()) continue; + if (st.mtimeMs < cutoffMs) continue; + candidates.push({ path: full, mtimeMs: st.mtimeMs, size: st.size }); + } + } + + // Newest first. + candidates.sort((a, b) => b.mtimeMs - a.mtimeMs); + + const out: RecentTranscript[] = []; + for (const c of candidates) { + if (out.length >= limit) break; + let raw: string; + try { + raw = readFileSync(c.path, 'utf-8'); + } catch { + continue; + } + // Skip dream-generated outputs (would re-feed the synthesize loop). + if (isDreamOutput(raw)) continue; + + const name = basename(c.path); + const dateMatch = DATE_RE.exec(name); + out.push({ + path: name, + date: dateMatch ? dateMatch[1] : null, + mtime: new Date(c.mtimeMs).toISOString(), + length: c.size, + summary: summary ? buildSummary(raw) : raw.slice(0, FULL_READ_CAP), + }); + } + return out; +} + +/** + * First non-empty line + next ~250 chars (cap on the summary body). + * Strips leading whitespace; preserves internal newlines truncated by the cap. + */ +function buildSummary(raw: string): string { + const trimmed = raw.replace(/^[\s​]+/, ''); + // First non-empty line. + const firstLineEnd = trimmed.search(/\r?\n/); + const firstLine = firstLineEnd === -1 ? trimmed : trimmed.slice(0, firstLineEnd); + const after = firstLineEnd === -1 ? '' : trimmed.slice(firstLineEnd + 1, firstLineEnd + 1 + SUMMARY_HEAD_CHARS); + if (!after) return firstLine; + return `${firstLine}\n${after}`.trim(); +} diff --git a/src/core/types.ts b/src/core/types.ts index 81a3ff591..806b8c68d 100644 --- a/src/core/types.ts +++ b/src/core/types.ts @@ -56,6 +56,8 @@ export interface Page { timeline: string; frontmatter: Record; content_hash?: string; + /** v0.29 — deterministic 0..1 score; populated by the recompute_emotional_weight cycle phase. */ + emotional_weight?: number; created_at: Date; updated_at: Date; /** @@ -64,8 +66,44 @@ export interface Page { * The autopilot purge phase hard-deletes rows where `deleted_at < now() - 72h`. */ deleted_at?: Date | null; + /** + * v0.29.1: content date computed from frontmatter precedence chain + * (event_date / date / published / filename / fallback). Populated by + * `computeEffectiveDate`; immune to auto-link updated_at churn. Read by + * the recency boost and since/until filter; nothing in the default search + * path consults it. + */ + effective_date?: Date | null; + /** + * v0.29.1: which precedence step won (`event_date | date | published | + * filename | fallback`). Powers the doctor's `effective_date_health` check + * to detect pages that fell back to updated_at because frontmatter was + * unparseable. + */ + effective_date_source?: EffectiveDateSource | null; + /** + * v0.29.1: basename without extension captured at import (e.g. + * "2024-03-15-acme-call"). Used by computeEffectiveDate for filename-date + * precedence on `daily/` and `meetings/` prefixes. NULL for older rows + * imported pre-v0.29.1. + */ + import_filename?: string | null; + /** + * v0.29.1: bumped by `recompute_emotional_weight` when the page's + * emotional_weight changes. The salience query window uses + * `GREATEST(updated_at, salience_touched_at)` so newly-salient old pages + * surface in `get_recent_salience`. + */ + salience_touched_at?: Date | null; } +export type EffectiveDateSource = + | 'event_date' + | 'date' + | 'published' + | 'filename' + | 'fallback'; + // `image` (v0.27.1): multimodal ingestion path, parallel to markdown + code. export type PageKind = 'markdown' | 'code' | 'image'; @@ -83,6 +121,17 @@ export interface PageInput { * `query --lang` filtering. */ page_kind?: PageKind; + /** + * v0.29.1: content date from frontmatter precedence (computed by importer + * via `computeEffectiveDate`). When omitted, putPage leaves the column + * unchanged on conflict (preserves any existing value); on insert the + * column is NULL. NULL is fine — recency paths COALESCE to updated_at. + */ + effective_date?: Date | null; + /** v0.29.1: paired with effective_date; NULL when effective_date is NULL. */ + effective_date_source?: EffectiveDateSource | null; + /** v0.29.1: basename without extension captured at import. */ + import_filename?: string | null; } export interface PageFilters { @@ -107,6 +156,13 @@ export interface PageFilters { * the 72h window before the autopilot purge phase hard-deletes them. */ includeDeleted?: boolean; + /** + * v0.29: ORDER BY enum. Default `updated_desc` matches pre-v0.29 behavior + * (engines hardcoded `ORDER BY updated_at DESC`). New options: `updated_asc`, + * `created_desc`, `slug` (alphabetical, useful for stable pagination). + * Whitelisted enum — no SQL-injection risk; engines map to literal SQL fragments. + */ + sort?: 'updated_desc' | 'updated_asc' | 'created_desc' | 'slug'; } /** v0.26.5 — opts for getPage / softDeletePage / restorePage. */ @@ -117,6 +173,102 @@ export interface GetPageOpts { includeDeleted?: boolean; } +/** v0.29: literal ORDER BY fragments for the PageFilters.sort enum. Whitelisted. */ +export const PAGE_SORT_SQL: Record, string> = { + updated_desc: 'p.updated_at DESC', + updated_asc: 'p.updated_at ASC', + created_desc: 'p.created_at DESC', + slug: 'p.slug ASC', +}; + +/** + * v0.29 — Salience: pages ranked by emotional + activity salience over a recency window. + * See `src/core/cycle/emotional-weight.ts` for the score formula and + * `engine.getRecentSalience` for the SQL. + */ +export interface SalienceOpts { + /** Window in days. Default 14. */ + days?: number; + /** Max rows to return (clamped at 100). Default 20. */ + limit?: number; + /** Optional slug-prefix filter (e.g., `personal`, `wiki/people`). */ + slugPrefix?: string; + /** + * v0.29.1 — recency-decay treatment for the salience formula's third term. + * - 'flat' (default): v0.29.0 behavior, `1.0 / (1 + days_old)` for every page + * - 'on': per-prefix decay from DEFAULT_RECENCY_DECAY (concepts/originals + * evergreen; daily/, media/x/ aggressive). Use when the agent wants + * "recency-biased salience" — what's been mattering AND fresh. + * Default preserves v0.29.0 ranking; 'on' is opt-in. + */ + recency_bias?: 'flat' | 'on'; +} + +export interface SalienceResult { + slug: string; + source_id: string; + title: string; + type: PageType; + updated_at: Date; + emotional_weight: number; + take_count: number; + take_avg_weight: number; + score: number; +} + +/** + * v0.29 — Anomaly detection: cohorts (tag, type) with unusually-high activity in a window. + * Cohort baseline is computed over `lookback_days` excluding `since`; current count is + * the number of distinct pages touched on `since`. A cohort is anomalous when its + * current count exceeds `mean + sigma * stddev`. Year cohort deferred to v0.30. + */ +export interface AnomaliesOpts { + /** ISO date (YYYY-MM-DD). Default = today (UTC). */ + since?: string; + /** Days of history for the baseline. Default 30. */ + lookback_days?: number; + /** Sigma threshold. Default 3.0. */ + sigma?: number; +} + +export interface AnomalyResult { + cohort_kind: 'tag' | 'type'; + cohort_value: string; + count: number; + baseline_mean: number; + baseline_stddev: number; + sigma_observed: number; + page_slugs: string[]; +} + +/** + * v0.29 — Per-page tag + take inputs to the emotional-weight formula. + * Returned in batch by `engine.batchLoadEmotionalInputs` so the cycle phase + * computes weights for many pages with two SQL round-trips total. + */ +export interface EmotionalWeightInputRow { + slug: string; + source_id: string; + tags: string[]; + takes: { + holder: string; + weight: number; + kind: string; + active: boolean; + }[]; +} + +/** + * v0.29 — Multi-source-safe write batch. Composite-keyed on `(slug, source_id)` + * because `pages.slug` is only unique within a source. Slug-only UPDATE would + * fan out across sources. + */ +export interface EmotionalWeightWriteRow { + slug: string; + source_id: string; + weight: number; +} + // Chunks export interface Chunk { id: number; @@ -270,6 +422,41 @@ export interface SearchOpts { * is unaffected — modality filtering on the keyword path is independent. */ embeddingColumn?: 'embedding' | 'embedding_image'; + /** + * @deprecated v0.29.1: use `since` instead. Removed in v0.30. + * v0.27.0: filter results to pages updated/created after this date. ISO-8601 string. + */ + afterDate?: string; + /** + * @deprecated v0.29.1: use `until` instead. Removed in v0.30. + * v0.27.0: filter results to pages updated/created before this date. ISO-8601 string. + */ + beforeDate?: string; + /** + * @deprecated v0.29.1: use `recency` ('off' | 'on' | 'strong') instead. Removed in v0.30. + * v0.27.0: recency boost strength. 0 = off, 1 = moderate, 2 = aggressive. + */ + recencyBoost?: 0 | 1 | 2; + /** + * v0.29.1: salience boost on emotional_weight + take_count. Independent of recency. + * 'off' (default) disables; 'on' applies a moderate boost; 'strong' more aggressive. + */ + salience?: 'off' | 'on' | 'strong'; + /** + * v0.29.1: recency boost on per-prefix age decay. Independent of salience. + * 'off' (default) disables; 'on' applies the per-prefix decay map; 'strong' multiplies by 1.5. + */ + recency?: 'off' | 'on' | 'strong'; + /** + * v0.29.1: ISO-8601 date OR relative duration ('7d', '2w', '1y'). Filter to + * pages whose effective_date >= this time. Replaces afterDate (kept as alias). + */ + since?: string; + /** + * v0.29.1: same shape as `since`. Filter to effective_date <= this time. + * Boundary semantics: end-of-day for plain YYYY-MM-DD. + */ + until?: string; } /** diff --git a/src/core/utils.ts b/src/core/utils.ts index c930f4af1..ab73dfe4b 100644 --- a/src/core/utils.ts +++ b/src/core/utils.ts @@ -43,13 +43,21 @@ export function contentHash(page: PageInput): string { .digest('hex'); } +function readOptionalDate(raw: unknown): Date | null | undefined { + // Three-state read for columns that may or may not be in the SELECT + // projection: undefined (not selected), null (selected, NULL value), + // Date (selected, populated). Mirrors the v0.26.5 deleted_at pattern. + if (raw === undefined) return undefined; + if (raw === null) return null; + return new Date(raw as string); +} + export function rowToPage(row: Record): Page { - // v0.26.5: deleted_at is optional in the SELECT projection. When the column - // isn't selected (legacy callers), keep the field absent on the returned object. - const deletedAtRaw = row.deleted_at; - const deletedAt = deletedAtRaw == null - ? (deletedAtRaw === null ? null : undefined) - : new Date(deletedAtRaw as string); + const deletedAt = readOptionalDate(row.deleted_at); + const effectiveDate = readOptionalDate(row.effective_date); + const salienceTouchedAt = readOptionalDate(row.salience_touched_at); + const effectiveDateSource = row.effective_date_source as Page['effective_date_source'] | undefined; + const importFilename = row.import_filename as string | null | undefined; return { id: row.id as number, slug: row.slug as string, @@ -59,9 +67,16 @@ export function rowToPage(row: Record): Page { timeline: row.timeline as string, frontmatter: (typeof row.frontmatter === 'string' ? JSON.parse(row.frontmatter) : row.frontmatter) as Record, content_hash: row.content_hash as string | undefined, + // v0.29 (column added in migration v40). Old brains pre-migration return undefined. + emotional_weight: row.emotional_weight == null ? undefined : Number(row.emotional_weight), created_at: new Date(row.created_at as string), updated_at: new Date(row.updated_at as string), ...(deletedAt !== undefined && { deleted_at: deletedAt }), + // v0.29.1 (columns added in migration v41). Optional in SELECT projection. + ...(effectiveDate !== undefined && { effective_date: effectiveDate }), + ...(effectiveDateSource !== undefined && { effective_date_source: effectiveDateSource }), + ...(importFilename !== undefined && { import_filename: importFilename }), + ...(salienceTouchedAt !== undefined && { salience_touched_at: salienceTouchedAt }), }; } diff --git a/src/schema.sql b/src/schema.sql index aee8a049d..e059d539f 100644 --- a/src/schema.sql +++ b/src/schema.sql @@ -77,6 +77,10 @@ CREATE TABLE IF NOT EXISTS pages ( timeline TEXT NOT NULL DEFAULT '', frontmatter JSONB NOT NULL DEFAULT '{}', content_hash TEXT, + -- v0.29: deterministic 0..1 score (tag emotion + take density + Garry-as-holder ratio). + -- Populated by the `recompute_emotional_weight` cycle phase. Default 0.0 so freshly + -- imported pages don't pollute salience ranking before the cycle has run. + emotional_weight REAL NOT NULL DEFAULT 0.0, created_at TIMESTAMPTZ NOT NULL DEFAULT now(), updated_at TIMESTAMPTZ NOT NULL DEFAULT now(), -- v0.26.5: soft-delete + recovery window. `delete_page` sets deleted_at = now() @@ -84,6 +88,17 @@ CREATE TABLE IF NOT EXISTS pages ( -- where deleted_at < now() - 72h. Search and `get_page` filter -- `WHERE deleted_at IS NULL` by default; `include_deleted: true` opts in. deleted_at TIMESTAMPTZ, + -- v0.29.1: salience-and-recency, additive opt-in. All NULL by default; + -- only consulted when a caller passes `salience='on'` / `recency='on'` or + -- the new `since`/`until` filter. effective_date_source is a sentinel for + -- the doctor's effective_date_health check (values: 'event_date' | 'date' + -- | 'published' | 'filename' | 'fallback'). salience_touched_at is bumped + -- by recompute_emotional_weight when emotional_weight changes so the + -- salience window picks up newly-salient old pages. + effective_date TIMESTAMPTZ, + effective_date_source TEXT, + import_filename TEXT, + salience_touched_at TIMESTAMPTZ, CONSTRAINT pages_source_slug_key UNIQUE (source_id, slug) ); @@ -101,6 +116,12 @@ CREATE INDEX IF NOT EXISTS idx_pages_source_id ON pages(source_id); -- stays low. Don't add a regular `(deleted_at)` index without measuring. CREATE INDEX IF NOT EXISTS pages_deleted_at_purge_idx ON pages (deleted_at) WHERE deleted_at IS NOT NULL; +-- v0.29.1: expression index used by since/until date-range filters that read +-- COALESCE(effective_date, updated_at). A partial index on effective_date +-- alone would NOT help — the planner can't use it for the negative side of +-- the COALESCE. Expression index is what actually accelerates the filter. +CREATE INDEX IF NOT EXISTS pages_coalesce_date_idx + ON pages ((COALESCE(effective_date, updated_at))); -- ============================================================ -- content_chunks: chunked content with embeddings @@ -742,7 +763,16 @@ CREATE TABLE IF NOT EXISTS eval_candidates ( remote BOOLEAN NOT NULL, job_id INTEGER, subagent_id INTEGER, - created_at TIMESTAMPTZ NOT NULL DEFAULT NOW() + created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(), + -- v0.29.1 — agent-explicit recency + salience capture for replay reproducibility. + -- All nullable + additive. NDJSON schema_version stays at 1; consumers ignore unknown fields. + as_of_ts TIMESTAMPTZ, + salience_param TEXT, + recency_param TEXT, + salience_resolved TEXT, + recency_resolved TEXT, + salience_source TEXT, + recency_source TEXT ); CREATE INDEX IF NOT EXISTS idx_eval_candidates_created_at ON eval_candidates(created_at DESC); diff --git a/test/anomalies.test.ts b/test/anomalies.test.ts new file mode 100644 index 000000000..69f1c6921 --- /dev/null +++ b/test/anomalies.test.ts @@ -0,0 +1,144 @@ +import { describe, expect, test } from 'bun:test'; +import { + meanStddev, + computeAnomaliesFromBuckets, + type CohortDayRow, + type CohortTodayRow, +} from '../src/core/cycle/anomaly.ts'; + +describe('meanStddev', () => { + test('empty returns (0, 0)', () => { + expect(meanStddev([])).toEqual({ mean: 0, stddev: 0 }); + }); + + test('single sample returns mean=value, stddev=0', () => { + expect(meanStddev([7])).toEqual({ mean: 7, stddev: 0 }); + }); + + test('all-equal returns stddev=0', () => { + const r = meanStddev([3, 3, 3, 3]); + expect(r.mean).toBe(3); + expect(r.stddev).toBe(0); + }); + + test('sample stddev (n-1 denominator)', () => { + // stddev of [2,4,4,4,5,5,7,9] with sample-stddev = 2.0 + const r = meanStddev([2, 4, 4, 4, 5, 5, 7, 9]); + expect(r.mean).toBe(5); + expect(r.stddev).toBeCloseTo(2.138, 2); + }); +}); + +describe('computeAnomaliesFromBuckets', () => { + function densify(values: number[], cohort_value: string, kind: 'tag' | 'type' = 'tag'): CohortDayRow[] { + return values.map((count, i) => ({ + cohort_kind: kind, + cohort_value, + day: `2026-04-${String(i + 1).padStart(2, '0')}`, + count, + })); + } + + test('clear anomaly: 7-touch day on a 0..1 baseline triggers tag cohort', () => { + const baseline: CohortDayRow[] = densify([0, 0, 1, 0, 0, 0, 1, 0, 0, 0], 'wedding'); + const today: CohortTodayRow[] = [ + { cohort_kind: 'tag', cohort_value: 'wedding', count: 7, page_slugs: ['p1','p2','p3','p4','p5','p6','p7'] }, + ]; + const r = computeAnomaliesFromBuckets(baseline, today, 3.0); + expect(r.length).toBe(1); + expect(r[0].cohort_value).toBe('wedding'); + expect(r[0].count).toBe(7); + expect(r[0].sigma_observed).toBeGreaterThan(3); + }); + + test('zero-stddev baseline + count > mean+1 triggers fallback (no NaN)', () => { + // baseline is all 1s (stddev=0), today is 5 → count > mean+1 → anomaly + const baseline: CohortDayRow[] = densify(Array.from({length: 10}, () => 1), 'work'); + const today: CohortTodayRow[] = [ + { cohort_kind: 'tag', cohort_value: 'work', count: 5, page_slugs: ['a','b','c','d','e'] }, + ]; + const r = computeAnomaliesFromBuckets(baseline, today, 3.0); + expect(r.length).toBe(1); + expect(r[0].baseline_stddev).toBe(0); + expect(Number.isFinite(r[0].sigma_observed)).toBe(true); + expect(r[0].sigma_observed).toBe(4); // 5 - 1 + }); + + test('zero-stddev baseline + count <= mean+1 does not fire', () => { + const baseline: CohortDayRow[] = densify(Array.from({length: 10}, () => 1), 'work'); + const today: CohortTodayRow[] = [ + { cohort_kind: 'tag', cohort_value: 'work', count: 2, page_slugs: ['a','b'] }, + ]; + expect(computeAnomaliesFromBuckets(baseline, today, 3.0)).toEqual([]); + }); + + test('non-anomalous current count returns empty', () => { + const baseline: CohortDayRow[] = densify([5, 4, 6, 5, 4, 5, 6, 5, 4, 6], 'daily'); + const today: CohortTodayRow[] = [ + { cohort_kind: 'tag', cohort_value: 'daily', count: 6, page_slugs: ['a'] }, + ]; + expect(computeAnomaliesFromBuckets(baseline, today, 3.0)).toEqual([]); + }); + + test('brand-new cohort (no baseline) requires count >= 2', () => { + // No baseline rows for "newtag"; today.count=2 → mean=0, stddev=0, threshold=mean+1=1, 2>1 ✓ + const today: CohortTodayRow[] = [ + { cohort_kind: 'tag', cohort_value: 'newtag', count: 2, page_slugs: ['a','b'] }, + { cohort_kind: 'tag', cohort_value: 'singleton', count: 1, page_slugs: ['x'] }, + ]; + const r = computeAnomaliesFromBuckets([], today, 3.0); + expect(r.length).toBe(1); + expect(r[0].cohort_value).toBe('newtag'); + }); + + test('top results sorted by sigma_observed desc', () => { + const baseline: CohortDayRow[] = [ + ...densify([0,0,0,0,0,0,0,0,0,0], 'low'), + ...densify([2,2,2,2,2,2,2,2,2,2], 'medium'), + ]; + const today: CohortTodayRow[] = [ + { cohort_kind: 'tag', cohort_value: 'low', count: 3, page_slugs: ['a','b','c'] }, + { cohort_kind: 'tag', cohort_value: 'medium', count: 4, page_slugs: ['x','y','z','w'] }, + ]; + const r = computeAnomaliesFromBuckets(baseline, today, 0.5); + // both should fire; "low" has bigger sigma_observed (mean=0) so it's first. + expect(r[0].cohort_value).toBe('low'); + }); + + test('limit caps result count', () => { + const today: CohortTodayRow[] = Array.from({length: 50}, (_, i) => ({ + cohort_kind: 'tag' as const, + cohort_value: `tag${i}`, + count: 5, + page_slugs: [`p${i}`], + })); + const r = computeAnomaliesFromBuckets([], today, 3.0, 10); + expect(r.length).toBe(10); + }); + + test('page_slugs are capped at 50 per cohort', () => { + const slugs = Array.from({length: 100}, (_, i) => `p${i}`); + const today: CohortTodayRow[] = [ + { cohort_kind: 'tag', cohort_value: 'huge', count: 100, page_slugs: slugs }, + ]; + const r = computeAnomaliesFromBuckets([], today, 3.0); + expect(r[0].page_slugs.length).toBe(50); + }); + + test('cohort_kind=tag and =type are tracked independently', () => { + // Same name "wedding" as both a tag and a type — should not collide. + const baseline: CohortDayRow[] = [ + ...densify([0,0,0,0,0], 'wedding', 'tag'), + ...densify([5,5,5,5,5], 'wedding', 'type'), + ]; + const today: CohortTodayRow[] = [ + { cohort_kind: 'tag', cohort_value: 'wedding', count: 7, page_slugs: ['t1'] }, + { cohort_kind: 'type', cohort_value: 'wedding', count: 7, page_slugs: ['t2'] }, + ]; + const r = computeAnomaliesFromBuckets(baseline, today, 0.5); + // Tag cohort should fire (mean=0); type cohort might or might not depending on stddev. + const tagEntry = r.find(x => x.cohort_kind === 'tag'); + expect(tagEntry).toBeDefined(); + expect(tagEntry!.baseline_mean).toBe(0); + }); +}); diff --git a/test/apply-migrations.test.ts b/test/apply-migrations.test.ts index b83e3553f..4eb6ac140 100644 --- a/test/apply-migrations.test.ts +++ b/test/apply-migrations.test.ts @@ -108,7 +108,7 @@ describe('buildPlan — diff against completed + installed VERSION', () => { // autopilot cooperative, v0.16.0 = subagent runtime, v0.18.0 = multi- // source brains, v0.18.1 = RLS hardening, v0.21.0 = Cathedral II // (renumbered from v0.20.0 after master shipped v0.20.x in parallel). - expect(plan.skippedFuture.map(m => m.version)).toEqual(['0.12.0', '0.12.2', '0.13.0', '0.13.1', '0.14.0', '0.16.0', '0.18.0', '0.18.1', '0.21.0', '0.22.4', '0.28.0']); + expect(plan.skippedFuture.map(m => m.version)).toEqual(['0.12.0', '0.12.2', '0.13.0', '0.13.1', '0.14.0', '0.16.0', '0.18.0', '0.18.1', '0.21.0', '0.22.4', '0.28.0', '0.29.1']); }); test('already applied → v0.11.0 lands in `applied` bucket, not pending', () => { @@ -148,7 +148,7 @@ describe('buildPlan — diff against completed + installed VERSION', () => { // were added later; installed=0.12.0 means they belong in skippedFuture, // not pending. v0.11.0 and v0.12.0 stay pending despite being ≤ installed — // that is the H9 invariant. - expect(plan.skippedFuture.map(m => m.version)).toEqual(['0.12.2', '0.13.0', '0.13.1', '0.14.0', '0.16.0', '0.18.0', '0.18.1', '0.21.0', '0.22.4', '0.28.0']); + expect(plan.skippedFuture.map(m => m.version)).toEqual(['0.12.2', '0.13.0', '0.13.1', '0.14.0', '0.16.0', '0.18.0', '0.18.1', '0.21.0', '0.22.4', '0.28.0', '0.29.1']); }); test('--migration filter narrows to one version', () => { diff --git a/test/benchmark-search-quality.ts b/test/benchmark-search-quality.ts index 347a7d0a2..ccd90fba6 100644 --- a/test/benchmark-search-quality.ts +++ b/test/benchmark-search-quality.ts @@ -13,7 +13,7 @@ import { PGLiteEngine } from '../src/core/pglite-engine.ts'; import { rrfFusion } from '../src/core/search/hybrid.ts'; import { dedupResults } from '../src/core/search/dedup.ts'; import { precisionAtK, recallAtK, mrr, ndcgAtK } from '../src/core/search/eval.ts'; -import { autoDetectDetail } from '../src/core/search/intent.ts'; +import { autoDetectDetail } from '../src/core/search/query-intent.ts'; import type { SearchResult, ChunkInput } from '../src/core/types.ts'; const RRF_K = 60; diff --git a/test/brain-allowlist.test.ts b/test/brain-allowlist.test.ts index 91504d38f..cb67c40a3 100644 --- a/test/brain-allowlist.test.ts +++ b/test/brain-allowlist.test.ts @@ -44,13 +44,19 @@ describe('BRAIN_TOOL_ALLOWLIST', () => { expect(missing).toEqual([]); }); - test('contains the read-only 10 + put_page', () => { - expect(BRAIN_TOOL_ALLOWLIST.size).toBe(11); + test('contains the v0.15 read-only 10 + put_page + v0.29 salience pair', () => { + // v0.29 added get_recent_salience + find_anomalies (read-only). + // get_recent_transcripts is deliberately excluded — subagent calls always + // have ctx.remote=true, and the v0.29 trust gate rejects remote callers. + expect(BRAIN_TOOL_ALLOWLIST.size).toBe(13); expect(BRAIN_TOOL_ALLOWLIST.has('query')).toBe(true); expect(BRAIN_TOOL_ALLOWLIST.has('search')).toBe(true); expect(BRAIN_TOOL_ALLOWLIST.has('get_page')).toBe(true); expect(BRAIN_TOOL_ALLOWLIST.has('list_pages')).toBe(true); expect(BRAIN_TOOL_ALLOWLIST.has('put_page')).toBe(true); + expect(BRAIN_TOOL_ALLOWLIST.has('get_recent_salience')).toBe(true); + expect(BRAIN_TOOL_ALLOWLIST.has('find_anomalies')).toBe(true); + expect(BRAIN_TOOL_ALLOWLIST.has('get_recent_transcripts')).toBe(false); }); test('does NOT contain destructive ops', () => { diff --git a/test/core/cycle.serial.test.ts b/test/core/cycle.serial.test.ts index 412765e96..925b3f3aa 100644 --- a/test/core/cycle.serial.test.ts +++ b/test/core/cycle.serial.test.ts @@ -377,8 +377,9 @@ describe('runCycle — yieldBetweenPhases hook', () => { hookCalls++; }, }); - // v0.26.5: 9 phases (added `purge`) → 9 yield calls (one after each). - expect(hookCalls).toBe(9); + // v0.26.5: 9 phases (added `purge`). + // v0.29: 10 phases (added `recompute_emotional_weight`) → 10 yield calls. + expect(hookCalls).toBe(10); }); test('hook exceptions do not abort the cycle', async () => { @@ -388,8 +389,8 @@ describe('runCycle — yieldBetweenPhases hook', () => { throw new Error('synthetic hook error'); }, }); - // Cycle still completed all phases (v0.26.5: 9 with the new purge phase). - expect(report.phases.length).toBe(9); + // Cycle still completed all phases (v0.29: 10 with recompute_emotional_weight). + expect(report.phases.length).toBe(10); }); }); diff --git a/test/e2e/anomalies-pglite.test.ts b/test/e2e/anomalies-pglite.test.ts new file mode 100644 index 000000000..62f050039 --- /dev/null +++ b/test/e2e/anomalies-pglite.test.ts @@ -0,0 +1,105 @@ +/** + * v0.29 E2E — find_anomalies against PGLite. + * + * Same fixture shape as the Garry test: 7 wedding-tagged pages touched today + * + 100 background pages spread across 30 days. Anomaly detection should + * fire on the wedding tag cohort because its baseline is near-zero. + * + * Also covers the brand-new-cohort case (no baseline rows; small-sample + * fallback fires when count >= 2) and the no-anomaly case (steady cohort, + * no spike). + */ + +import { describe, test, expect, beforeAll, afterAll } from 'bun:test'; +import { PGLiteEngine } from '../../src/core/pglite-engine.ts'; + +let engine: PGLiteEngine; +const TODAY = new Date().toISOString().slice(0, 10); + +beforeAll(async () => { + engine = new PGLiteEngine(); + await engine.connect({ engine: 'pglite' } as never); + await engine.initSchema(); + + // 7 wedding-tagged pages, all updated today (default putPage stamp). + for (let i = 0; i < 7; i++) { + const slug = `personal/wedding/photos-${i}`; + await engine.putPage(slug, { + type: 'note', + title: `Wedding photo ${i}`, + compiled_truth: `Photos from the wedding day, group ${i}.`, + }); + await engine.addTag(slug, 'wedding'); + } + + // 100 background pages, tagged with rotating "steady" tags, backdated. + const RANDOM_TAGS = ['hardware', 'product', 'meeting']; + for (let i = 0; i < 100; i++) { + const slug = `notes/random-${i}`; + await engine.putPage(slug, { + type: 'note', + title: `Random note ${i}`, + compiled_truth: `Body text for note ${i}.`, + }); + await engine.addTag(slug, RANDOM_TAGS[i % RANDOM_TAGS.length]); + } + // Spread the random pages randomly across the last 30 days + // (excluding today, so the wedding cohort is the only "today" spike). + await engine.executeRaw( + `UPDATE pages + SET updated_at = now() - interval '1 day' - (random() * interval '29 days') + WHERE slug LIKE 'notes/random-%'` + ); +}); + +afterAll(async () => { + if (engine) await engine.disconnect(); +}); + +describe('v0.29 E2E — findAnomalies (Garry test)', () => { + test('wedding-tag cohort fires as anomaly with sigma > 3 vs zero baseline', async () => { + const rows = await engine.findAnomalies({ since: TODAY, lookback_days: 30, sigma: 3.0 }); + const wedding = rows.find(r => r.cohort_kind === 'tag' && r.cohort_value === 'wedding'); + expect(wedding).toBeDefined(); + expect(wedding!.count).toBe(7); + // Baseline should be ~0 because none of the wedding pages were backdated. + expect(wedding!.baseline_mean).toBeLessThan(1); + expect(wedding!.sigma_observed).toBeGreaterThan(3); + }); + + test('returned page_slugs sample contains wedding pages', async () => { + const rows = await engine.findAnomalies({ since: TODAY, lookback_days: 30 }); + const wedding = rows.find(r => r.cohort_kind === 'tag' && r.cohort_value === 'wedding'); + expect(wedding!.page_slugs.length).toBe(7); + expect(wedding!.page_slugs.every(s => s.startsWith('personal/wedding/'))).toBe(true); + }); + + test('a date with no activity returns []', async () => { + // Look at a date earlier than any seeded page — every cohort has count=0, + // none should fire as anomalous. + const rows = await engine.findAnomalies({ + since: '2024-01-15', + lookback_days: 30, + sigma: 3.0, + }); + expect(rows).toEqual([]); + }); + + test('high sigma threshold suppresses borderline cohorts', async () => { + const lowRows = await engine.findAnomalies({ since: TODAY, lookback_days: 30, sigma: 0.5 }); + const highRows = await engine.findAnomalies({ since: TODAY, lookback_days: 30, sigma: 100 }); + // sigma=100 should suppress every cohort (would need a literally + // impossible spike to fire). The wedding cohort fires at sigma 3 so + // there's enough headroom for sigma=0.5 ⊇ sigma=100. + expect(lowRows.length).toBeGreaterThanOrEqual(highRows.length); + // Wedding-tag (count=7, baseline mean ~0, stddev ~0) shouldn't pass + // sigma=100 because the small-sample fallback uses count > mean+1, not + // sigma scaling. + const weddingHigh = highRows.find(r => r.cohort_kind === 'tag' && r.cohort_value === 'wedding'); + if (weddingHigh) { + // It can still fire because the sample-stddev fallback uses count > mean + 1, + // not sigma * stddev. Confirm the sigma_observed is finite (no NaN). + expect(Number.isFinite(weddingHigh.sigma_observed)).toBe(true); + } + }); +}); diff --git a/test/e2e/backfill-perf-pglite.test.ts b/test/e2e/backfill-perf-pglite.test.ts new file mode 100644 index 000000000..05df3a0da --- /dev/null +++ b/test/e2e/backfill-perf-pglite.test.ts @@ -0,0 +1,60 @@ +/** + * v0.29 — backfill perf regression guard (codex C4#3+#4). + * + * The first plan revision did per-page reads + per-page writes (N+1) which + * would multi-minute on real brains. The shipped path is two SQL round-trips + * total (CTE-shaped batch read + UPDATE FROM unnest batch write). + * + * This test seeds 1000 pages with random tags + 0-3 takes each, runs the + * recompute_emotional_weight phase against PGLite in-memory, and asserts + * wall-clock < 5s on the same fixture pattern. Goal is to catch a regression + * to N+1 — a fast machine on PGLite in-memory should finish in well under + * a second; the 5s budget is generous for slow CI. + */ + +import { describe, test, expect, beforeAll, afterAll } from 'bun:test'; +import { PGLiteEngine } from '../../src/core/pglite-engine.ts'; +import { runPhaseRecomputeEmotionalWeight } from '../../src/core/cycle/recompute-emotional-weight.ts'; + +let engine: PGLiteEngine; + +const TAG_POOL = [ + 'wedding', 'family', 'work', 'product', 'hardware', + 'meeting', 'idea', 'concept', 'people', 'health', +]; + +beforeAll(async () => { + engine = new PGLiteEngine(); + await engine.connect({ engine: 'pglite' } as never); + await engine.initSchema(); + + // Seed 1000 pages, each with 1-3 tags from the pool. + for (let i = 0; i < 1000; i++) { + const slug = `notes/perf-${i}`; + await engine.putPage(slug, { + type: 'note', + title: `Page ${i}`, + compiled_truth: 'body', + }); + const tagCount = 1 + (i % 3); + for (let t = 0; t < tagCount; t++) { + await engine.addTag(slug, TAG_POOL[(i + t) % TAG_POOL.length]); + } + } +}, 60_000); + +afterAll(async () => { + if (engine) await engine.disconnect(); +}); + +describe('v0.29 — recompute_emotional_weight perf on a 1000-page fixture', () => { + test('full-mode backfill completes in under 5 seconds', async () => { + const start = Date.now(); + const result = await runPhaseRecomputeEmotionalWeight(engine, {}); + const elapsedMs = Date.now() - start; + + expect(result.status).toBe('ok'); + expect(result.pages_recomputed).toBeGreaterThanOrEqual(1000); + expect(elapsedMs).toBeLessThan(5_000); + }, 30_000); +}); diff --git a/test/e2e/cycle-recompute-emotional-weight-pglite.test.ts b/test/e2e/cycle-recompute-emotional-weight-pglite.test.ts new file mode 100644 index 000000000..bd49cfd17 --- /dev/null +++ b/test/e2e/cycle-recompute-emotional-weight-pglite.test.ts @@ -0,0 +1,111 @@ +/** + * v0.29 E2E — recompute_emotional_weight cycle phase wiring against PGLite. + * + * Asserts: + * - Phase appears in ALL_PHASES and runs as part of a default cycle. + * - Full mode (no incremental anchors) walks every page in the brain. + * - Selectable via `--phase recompute_emotional_weight` (single phase run). + * - dry-run skips the UPDATE but still reports the would-write count. + */ + +import { describe, test, expect, beforeAll, afterAll } from 'bun:test'; +import { PGLiteEngine } from '../../src/core/pglite-engine.ts'; +import { runCycle, ALL_PHASES } from '../../src/core/cycle.ts'; +import { mkdtempSync, rmSync, mkdirSync } from 'node:fs'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; + +let engine: PGLiteEngine; +let brainDir: string; + +beforeAll(async () => { + engine = new PGLiteEngine(); + await engine.connect({ engine: 'pglite' } as never); + await engine.initSchema(); + + // Brain dir for filesystem phases (lint/backlinks/sync). They'll skip + // gracefully when there's nothing to read. + brainDir = mkdtempSync(join(tmpdir(), 'gbrain-cycle-test-')); + mkdirSync(join(brainDir, 'wiki'), { recursive: true }); + + // Seed two pages: one with a high-emotion tag, one without. + await engine.putPage('personal/wedding/photos', { + type: 'note', + title: 'Wedding photos', + compiled_truth: 'Some wedding photos.', + }); + await engine.addTag('personal/wedding/photos', 'wedding'); + + await engine.putPage('notes/random', { + type: 'note', + title: 'Random note', + compiled_truth: 'Just a note.', + }); + await engine.addTag('notes/random', 'product'); +}); + +afterAll(async () => { + if (engine) await engine.disconnect(); + if (brainDir) rmSync(brainDir, { recursive: true, force: true }); +}); + +describe('v0.29 — recompute_emotional_weight phase is registered', () => { + test('appears in ALL_PHASES between patterns and embed', () => { + const idx = ALL_PHASES.indexOf('recompute_emotional_weight'); + const patternsIdx = ALL_PHASES.indexOf('patterns'); + const embedIdx = ALL_PHASES.indexOf('embed'); + expect(idx).toBeGreaterThan(-1); + expect(idx).toBeGreaterThan(patternsIdx); + expect(idx).toBeLessThan(embedIdx); + }); +}); + +describe('v0.29 — recompute_emotional_weight phase runs end-to-end', () => { + test('--phase recompute_emotional_weight populates the column for every page (full mode)', async () => { + const report = await runCycle(engine, { + brainDir, + phases: ['recompute_emotional_weight'], + }); + expect(report.status).not.toBe('failed'); + const phaseResult = report.phases.find(p => p.phase === 'recompute_emotional_weight'); + expect(phaseResult).toBeDefined(); + expect(phaseResult!.status).toBe('ok'); + expect(phaseResult!.details.mode).toBe('full'); + expect(Number(phaseResult!.details.pages_recomputed)).toBeGreaterThanOrEqual(2); + + // Verify both pages got their weights populated. + const wedding = await engine.executeRaw<{ emotional_weight: number }>( + `SELECT emotional_weight FROM pages WHERE slug = 'personal/wedding/photos'` + ); + const random = await engine.executeRaw<{ emotional_weight: number }>( + `SELECT emotional_weight FROM pages WHERE slug = 'notes/random'` + ); + expect(Number(wedding[0].emotional_weight)).toBeCloseTo(0.5, 5); + expect(Number(random[0].emotional_weight)).toBe(0); + + // Totals roll up the new field. + expect(report.totals.pages_emotional_weight_recomputed).toBeGreaterThanOrEqual(2); + }); + + test('dry-run skips the UPDATE but reports a would-write count', async () => { + // Reset weights to a sentinel so we can detect a write. + await engine.executeRaw(`UPDATE pages SET emotional_weight = 0.99`); + + const report = await runCycle(engine, { + brainDir, + phases: ['recompute_emotional_weight'], + dryRun: true, + }); + const phaseResult = report.phases.find(p => p.phase === 'recompute_emotional_weight'); + expect(phaseResult).toBeDefined(); + expect(phaseResult!.status).toBe('ok'); + expect(phaseResult!.details.dry_run).toBe(true); + expect(Number(phaseResult!.details.pages_recomputed)).toBeGreaterThanOrEqual(2); + + // Sentinel survives because dry-run never writes. + const after = await engine.executeRaw<{ emotional_weight: number }>( + `SELECT emotional_weight FROM pages WHERE slug = 'personal/wedding/photos'` + ); + expect(Number(after[0].emotional_weight)).toBeCloseTo(0.99, 5); + }); +}); diff --git a/test/e2e/engine-parity-salience.test.ts b/test/e2e/engine-parity-salience.test.ts new file mode 100644 index 000000000..2c2181ad8 --- /dev/null +++ b/test/e2e/engine-parity-salience.test.ts @@ -0,0 +1,124 @@ +/** + * v0.29 — Engine parity: salience + anomalies on PGLite vs Postgres. + * + * Codex flagged in the v0.22.0 source-boost review that engine-shape + * differences (postgres.js vs PGLite SQL idioms) can silently diverge + * results. The same risk applies to the new v0.29 ops: + * - getRecentSalience uses EXTRACT(EPOCH FROM ...), ln(), GROUP BY p.id. + * - findAnomalies uses generate_series + date_trunc + array_agg. + * + * This test seeds identical fixtures into both engines, runs the v0.29 + * ops, and asserts the result sets line up. + * + * DATABASE_URL gated — skips gracefully when not set. + */ + +import { describe, test, expect, beforeAll, afterAll } from 'bun:test'; +import { PGLiteEngine } from '../../src/core/pglite-engine.ts'; +import { hasDatabase, setupDB, teardownDB } from './helpers.ts'; +import type { BrainEngine } from '../../src/core/engine.ts'; + +const SKIP_PG = !hasDatabase(); +const describeBoth = SKIP_PG ? describe.skip : describe; + +const TODAY = new Date().toISOString().slice(0, 10); + +async function seedFixture(engine: BrainEngine): Promise { + // 5 wedding-tagged pages, all updated today. + for (let i = 0; i < 5; i++) { + const slug = `personal/wedding/photos-${i}`; + await engine.putPage(slug, { + type: 'note', + title: `Wedding photo ${i}`, + compiled_truth: 'photos', + }); + await engine.addTag(slug, 'wedding'); + } + // 30 background pages backdated across 30 days. + for (let i = 0; i < 30; i++) { + const slug = `notes/random-${i}`; + await engine.putPage(slug, { + type: 'note', + title: `Random ${i}`, + compiled_truth: 'body', + }); + await engine.addTag(slug, ['hardware', 'product', 'idea'][i % 3]); + } + await engine.executeRaw( + `UPDATE pages + SET updated_at = now() - interval '1 day' - (random() * interval '29 days') + WHERE slug LIKE 'notes/random-%'` + ); +} + +describeBoth('v0.29 engine parity — getRecentSalience', () => { + let pglite: PGLiteEngine; + let postgres: BrainEngine; + + beforeAll(async () => { + pglite = new PGLiteEngine(); + await pglite.connect({ engine: 'pglite' } as never); + await pglite.initSchema(); + await seedFixture(pglite); + + postgres = await setupDB(); + await seedFixture(postgres); + }, 60_000); + + afterAll(async () => { + if (pglite) await pglite.disconnect(); + await teardownDB(); + }); + + test('top result is a wedding page on both engines', async () => { + const pgliteRows = await pglite.getRecentSalience({ days: 7, limit: 5 }); + const postgresRows = await postgres.getRecentSalience({ days: 7, limit: 5 }); + expect(pgliteRows.length).toBeGreaterThan(0); + expect(postgresRows.length).toBeGreaterThan(0); + expect(pgliteRows[0].slug.startsWith('personal/wedding/')).toBe(true); + expect(postgresRows[0].slug.startsWith('personal/wedding/')).toBe(true); + }); + + test('same set of wedding slugs returned in the top 5 on both engines', async () => { + const pgliteRows = await pglite.getRecentSalience({ days: 7, limit: 10 }); + const postgresRows = await postgres.getRecentSalience({ days: 7, limit: 10 }); + const pgliteWedding = new Set(pgliteRows.filter(r => r.slug.startsWith('personal/wedding/')).map(r => r.slug)); + const postgresWedding = new Set(postgresRows.filter(r => r.slug.startsWith('personal/wedding/')).map(r => r.slug)); + expect(pgliteWedding.size).toBe(postgresWedding.size); + for (const s of pgliteWedding) expect(postgresWedding.has(s)).toBe(true); + }); +}); + +describeBoth('v0.29 engine parity — findAnomalies', () => { + let pglite: PGLiteEngine; + let postgres: BrainEngine; + + beforeAll(async () => { + pglite = new PGLiteEngine(); + await pglite.connect({ engine: 'pglite' } as never); + await pglite.initSchema(); + await seedFixture(pglite); + + postgres = await setupDB(); + await seedFixture(postgres); + }, 60_000); + + afterAll(async () => { + if (pglite) await pglite.disconnect(); + await teardownDB(); + }); + + test('wedding tag cohort fires on both engines with similar counts', async () => { + const pgliteRows = await pglite.findAnomalies({ since: TODAY, lookback_days: 30, sigma: 2 }); + const postgresRows = await postgres.findAnomalies({ since: TODAY, lookback_days: 30, sigma: 2 }); + const pgliteWedding = pgliteRows.find(r => r.cohort_kind === 'tag' && r.cohort_value === 'wedding'); + const postgresWedding = postgresRows.find(r => r.cohort_kind === 'tag' && r.cohort_value === 'wedding'); + expect(pgliteWedding).toBeDefined(); + expect(postgresWedding).toBeDefined(); + expect(pgliteWedding!.count).toBe(5); + expect(postgresWedding!.count).toBe(5); + // baseline mean should be very small (random-tag pages don't carry "wedding"). + expect(pgliteWedding!.baseline_mean).toBeLessThan(1); + expect(postgresWedding!.baseline_mean).toBeLessThan(1); + }); +}); diff --git a/test/e2e/list-pages-regression.test.ts b/test/e2e/list-pages-regression.test.ts new file mode 100644 index 000000000..accab3322 --- /dev/null +++ b/test/e2e/list-pages-regression.test.ts @@ -0,0 +1,94 @@ +/** + * v0.29 IRON RULE — list_pages regression coverage. + * + * Adding optional params (`updated_after`, `sort`) to a long-shipped op + * must not change behavior for callers that only pass the pre-v0.29 shape + * (`type`, `tag`, `limit`). This test asserts the old shape produces the + * pre-v0.29 default order and that the new `sort` enum threads through both + * engine implementations (codex C4#9 — engines hardcoded ORDER BY DESC). + */ + +import { describe, test, expect, beforeAll, afterAll } from 'bun:test'; +import { PGLiteEngine } from '../../src/core/pglite-engine.ts'; + +let engine: PGLiteEngine; + +beforeAll(async () => { + engine = new PGLiteEngine(); + await engine.connect({ engine: 'pglite' } as never); + await engine.initSchema(); + + // Seed 5 pages with deterministic timestamps so order is provable. + for (let i = 0; i < 5; i++) { + await engine.putPage(`alpha/page-${i}`, { + type: 'note', + title: `Page ${i}`, + compiled_truth: 'body', + }); + } + // Backdate updated_at so we can test ordering meaningfully. + await engine.executeRaw( + `UPDATE pages SET updated_at = '2026-01-01'::timestamptz + (id * interval '1 day')` + ); +}); + +afterAll(async () => { + if (engine) await engine.disconnect(); +}); + +describe('v0.29 IRON RULE — list_pages back-compat (pre-v0.29 shape)', () => { + test('old call shape (type, tag, limit) returns updated_desc order by default', async () => { + const rows = await engine.listPages({ limit: 10 }); + expect(rows.length).toBeGreaterThanOrEqual(5); + // Pre-v0.29 default = ORDER BY updated_at DESC. Our seeding made id=5 the + // newest, so it must appear first. + for (let i = 0; i < rows.length - 1; i++) { + expect(rows[i].updated_at.getTime() >= rows[i + 1].updated_at.getTime()).toBe(true); + } + }); + + test('updated_after filter narrows to recent rows', async () => { + const rows = await engine.listPages({ + limit: 10, + updated_after: '2026-01-04', + }); + // Only pages with updated_at > 2026-01-04 — that's id=4 + id=5 in the seed. + expect(rows.length).toBeGreaterThanOrEqual(1); + for (const r of rows) { + expect(r.updated_at.getTime()).toBeGreaterThan(new Date('2026-01-04').getTime()); + } + }); +}); + +describe('v0.29 — list_pages sort enum threads through the engine', () => { + test('sort=updated_asc reverses default order', async () => { + const desc = await engine.listPages({ limit: 10, sort: 'updated_desc' }); + const asc = await engine.listPages({ limit: 10, sort: 'updated_asc' }); + expect(asc.length).toBe(desc.length); + expect(asc[0].slug).toBe(desc[desc.length - 1].slug); + }); + + test('sort=created_desc orders by created_at, not updated_at', async () => { + const rows = await engine.listPages({ limit: 10, sort: 'created_desc' }); + expect(rows.length).toBeGreaterThanOrEqual(5); + // In our seed, id=5 was inserted last → newest created_at → first row. + for (let i = 0; i < rows.length - 1; i++) { + expect(rows[i].created_at.getTime() >= rows[i + 1].created_at.getTime()).toBe(true); + } + }); + + test('sort=slug returns alphabetical order', async () => { + const rows = await engine.listPages({ limit: 10, sort: 'slug' }); + expect(rows.length).toBeGreaterThanOrEqual(5); + const sorted = [...rows].sort((a, b) => a.slug.localeCompare(b.slug)); + expect(rows.map(r => r.slug)).toEqual(sorted.map(r => r.slug)); + }); + + test('unsupported sort value falls back to default (does not crash)', async () => { + // An invalid string would be filtered by the handler-side whitelist; + // call the engine directly with a junk value to verify defense-in-depth. + const rows = await engine.listPages({ limit: 10, sort: 'whatever' as any }); + // Engine PAGE_SORT_SQL[unknown] is undefined → falls back to default desc. + expect(rows.length).toBeGreaterThan(0); + }); +}); diff --git a/test/e2e/multi-source-emotional-weight-pglite.test.ts b/test/e2e/multi-source-emotional-weight-pglite.test.ts new file mode 100644 index 000000000..bf7de8366 --- /dev/null +++ b/test/e2e/multi-source-emotional-weight-pglite.test.ts @@ -0,0 +1,95 @@ +/** + * v0.29 E2E — multi-source UPDATE safety (codex C4#3). + * + * pages.slug is unique only within a source_id. A slug-only UPDATE would + * fan out across sources and corrupt other sources' rows. This test seeds + * pages with the SAME slug under two different source_ids, runs + * setEmotionalWeightBatch for one of them, and asserts the other source's + * row is untouched. + * + * Regression guard: if a future maintainer drops the source_id from the + * UPDATE WHERE clause, this test fires. + */ + +import { describe, test, expect, beforeAll, afterAll } from 'bun:test'; +import { PGLiteEngine } from '../../src/core/pglite-engine.ts'; + +let engine: PGLiteEngine; + +beforeAll(async () => { + engine = new PGLiteEngine(); + await engine.connect({ engine: 'pglite' } as never); + await engine.initSchema(); + // Register a second source so we can put pages with the same slug + // across two source_ids. (default source is auto-seeded on schema init.) + await engine.executeRaw( + `INSERT INTO sources (id, name) VALUES ('src-b', 'Test Source B')` + ); + // Same slug under both sources. + await engine.executeRaw( + `INSERT INTO pages (source_id, slug, type, title, compiled_truth) + VALUES ('default', 'shared/page', 'note', 'Default copy', 'A')` + ); + await engine.executeRaw( + `INSERT INTO pages (source_id, slug, type, title, compiled_truth) + VALUES ('src-b', 'shared/page', 'note', 'src-b copy', 'B')` + ); +}); + +afterAll(async () => { + if (engine) await engine.disconnect(); +}); + +describe('v0.29 E2E — setEmotionalWeightBatch is multi-source safe', () => { + test('UPDATE on (slug=shared, source_id=default) leaves src-b untouched', async () => { + const updated = await engine.setEmotionalWeightBatch([ + { slug: 'shared/page', source_id: 'default', weight: 0.42 }, + ]); + expect(updated).toBe(1); // exactly one row touched + + // Verify both rows independently. + const defRow = await engine.executeRaw<{ slug: string; source_id: string; emotional_weight: number }>( + `SELECT slug, source_id, emotional_weight FROM pages + WHERE slug = 'shared/page' AND source_id = 'default'` + ); + const srcBRow = await engine.executeRaw<{ slug: string; source_id: string; emotional_weight: number }>( + `SELECT slug, source_id, emotional_weight FROM pages + WHERE slug = 'shared/page' AND source_id = 'src-b'` + ); + expect(defRow.length).toBe(1); + expect(srcBRow.length).toBe(1); + expect(Number(defRow[0].emotional_weight)).toBeCloseTo(0.42, 5); + // src-b row stays at default 0.0. + expect(Number(srcBRow[0].emotional_weight)).toBe(0); + }); + + test('two updates in one batch hit the right sources', async () => { + const updated = await engine.setEmotionalWeightBatch([ + { slug: 'shared/page', source_id: 'default', weight: 0.10 }, + { slug: 'shared/page', source_id: 'src-b', weight: 0.20 }, + ]); + expect(updated).toBe(2); + const rows = await engine.executeRaw<{ source_id: string; emotional_weight: number }>( + `SELECT source_id, emotional_weight FROM pages WHERE slug = 'shared/page' ORDER BY source_id` + ); + expect(rows.length).toBe(2); + const byid = Object.fromEntries(rows.map(r => [r.source_id, Number(r.emotional_weight)])); + expect(byid.default).toBeCloseTo(0.10, 5); + expect(byid['src-b']).toBeCloseTo(0.20, 5); + }); + + test('non-existent (slug, source_id) tuple is silently skipped (no error)', async () => { + const updated = await engine.setEmotionalWeightBatch([ + { slug: 'shared/page', source_id: 'default', weight: 0.50 }, // exists + { slug: 'nope/missing', source_id: 'default', weight: 0.99 }, // doesn't exist + { slug: 'shared/page', source_id: 'src-zzz', weight: 0.99 }, // wrong source_id + ]); + // Only the existing tuple is updated. + expect(updated).toBe(1); + }); + + test('empty batch returns 0', async () => { + const updated = await engine.setEmotionalWeightBatch([]); + expect(updated).toBe(0); + }); +}); diff --git a/test/e2e/salience-llm-routing.test.ts b/test/e2e/salience-llm-routing.test.ts new file mode 100644 index 000000000..4f84058bf --- /dev/null +++ b/test/e2e/salience-llm-routing.test.ts @@ -0,0 +1,160 @@ +/** + * v0.29 — LLM routing eval (Tier-2, ANTHROPIC_API_KEY gated). + * + * The whole point of v0.29 is the agent reaches for get_recent_salience + * (or find_anomalies / get_recent_transcripts) instead of running query() + * when the user asks "what's been going on with me?". This test confirms + * the description edits actually drive that routing — without it, we ship + * description changes and only learn from production behavior. + * + * Implementation: builds a tools list with the v0.29 op definitions, calls + * Claude with a series of personal-query phrasings, asserts the chosen + * tool is in the v0.29 set. Cost ~$0.10/CI run on Haiku. + * + * Skips gracefully when ANTHROPIC_API_KEY is missing. + * + * Replaces the discarded `skills/{salience,anomalies,transcripts}/routing-eval.jsonl` + * fixtures (codex C1) which would have shipped fake coverage — + * `routing-eval.ts` evaluates skill resolver triggers via substring match, + * not MCP tool routing. + */ + +import { describe, test, expect } from 'bun:test'; +import { + GET_RECENT_SALIENCE_DESCRIPTION, + FIND_ANOMALIES_DESCRIPTION, + GET_RECENT_TRANSCRIPTS_DESCRIPTION, + QUERY_DESCRIPTION, + SEARCH_DESCRIPTION, +} from '../../src/core/operations-descriptions.ts'; + +const SKIP = !process.env.ANTHROPIC_API_KEY; +const describeIfKey = SKIP ? describe.skip : describe; + +interface ToolDef { + name: string; + description: string; + input_schema: { type: 'object'; properties: Record }; +} + +const TOOLS: ToolDef[] = [ + { + name: 'get_recent_salience', + description: GET_RECENT_SALIENCE_DESCRIPTION, + input_schema: { + type: 'object', + properties: { + days: { type: 'number' }, + limit: { type: 'number' }, + slugPrefix: { type: 'string' }, + }, + }, + }, + { + name: 'find_anomalies', + description: FIND_ANOMALIES_DESCRIPTION, + input_schema: { + type: 'object', + properties: { + since: { type: 'string' }, + lookback_days: { type: 'number' }, + sigma: { type: 'number' }, + }, + }, + }, + { + name: 'get_recent_transcripts', + description: GET_RECENT_TRANSCRIPTS_DESCRIPTION, + input_schema: { + type: 'object', + properties: { + days: { type: 'number' }, + summary: { type: 'boolean' }, + limit: { type: 'number' }, + }, + }, + }, + { + name: 'query', + description: QUERY_DESCRIPTION, + input_schema: { + type: 'object', + properties: { + query: { type: 'string' }, + limit: { type: 'number' }, + }, + }, + }, + { + name: 'search', + description: SEARCH_DESCRIPTION, + input_schema: { + type: 'object', + properties: { + query: { type: 'string' }, + limit: { type: 'number' }, + }, + }, + }, +]; + +const V029_TOOLS = new Set(['get_recent_salience', 'find_anomalies', 'get_recent_transcripts']); + +const PERSONAL_QUERY_PHRASINGS = [ + 'anything crazy happening in my brain lately?', + "what's been going on with me?", + "how have I been?", + "anything notable in my brain?", + "what's been on my mind?", + "what stood out this week?", + "what's hot in my notes?", + "anything weird going on lately?", + "any unusual patterns?", + "what have I been thinking about?", + "what did I talk about yesterday?", + "what's notable in the brain right now?", +]; + +interface AnthropicResponse { + content: Array<{ type: string; name?: string }>; + stop_reason: string; +} + +async function callClaudeWithTools(prompt: string): Promise<{ tool: string | null; raw: AnthropicResponse }> { + const apiKey = process.env.ANTHROPIC_API_KEY!; + const res = await fetch('https://api.anthropic.com/v1/messages', { + method: 'POST', + headers: { + 'x-api-key': apiKey, + 'anthropic-version': '2023-06-01', + 'content-type': 'application/json', + }, + body: JSON.stringify({ + model: 'claude-haiku-4-5-20251001', + max_tokens: 256, + tools: TOOLS, + messages: [ + { + role: 'user', + content: prompt, + }, + ], + }), + }); + if (!res.ok) { + throw new Error(`Anthropic API ${res.status}: ${await res.text()}`); + } + const json = (await res.json()) as AnthropicResponse; + const useBlock = (json.content ?? []).find(b => b.type === 'tool_use'); + return { tool: useBlock?.name ?? null, raw: json }; +} + +describeIfKey('v0.29 — LLM routes personal queries to v0.29 ops, not query() / search()', () => { + for (const prompt of PERSONAL_QUERY_PHRASINGS) { + test(`routes "${prompt}" to a v0.29 tool`, async () => { + const { tool } = await callClaudeWithTools(prompt); + expect(tool).not.toBeNull(); + expect(V029_TOOLS.has(tool!)).toBe(true); + }, 30_000); + } +}); diff --git a/test/e2e/salience-pglite.test.ts b/test/e2e/salience-pglite.test.ts new file mode 100644 index 000000000..aa959031e --- /dev/null +++ b/test/e2e/salience-pglite.test.ts @@ -0,0 +1,115 @@ +/** + * v0.29 E2E — the "Garry test" for salience. + * + * Seeds a fixture: 7 pages tagged `wedding`, all touched today, plus 100 + * background pages with random tags spread across 30 days. Asserts that + * `getRecentSalience({days:7})` returns the wedding pages at the top. + * + * Uses raw SQL UPDATE to backdate `updated_at` on the background pages + * (codex C4#7) — `engine.putPage` always stamps `updated_at = now()` so + * seeding via the engine alone can't reproduce historical recency windows. + * + * Runs against PGLite in-memory; no DATABASE_URL required. + */ + +import { describe, test, expect, beforeAll, afterAll } from 'bun:test'; +import { PGLiteEngine } from '../../src/core/pglite-engine.ts'; + +let engine: PGLiteEngine; + +beforeAll(async () => { + engine = new PGLiteEngine(); + await engine.connect({ engine: 'pglite' } as never); + await engine.initSchema(); + + // ── Seed: 7 wedding pages (all touched today, default updated_at). + for (let i = 0; i < 7; i++) { + const slug = `personal/wedding/photos-${i}`; + await engine.putPage(slug, { + type: 'note', + title: `Wedding photo ${i}`, + compiled_truth: `Photos from the wedding day, group ${i}.`, + }); + await engine.addTag(slug, 'wedding'); + } + + // ── Seed: 100 background pages, tagged with miscellaneous tags. + const RANDOM_TAGS = ['hardware', 'product', 'meeting', 'idea', 'people', 'concept']; + for (let i = 0; i < 100; i++) { + const slug = `notes/random-${i}`; + await engine.putPage(slug, { + type: 'note', + title: `Random note ${i}`, + compiled_truth: `Body text for note ${i}.`, + }); + await engine.addTag(slug, RANDOM_TAGS[i % RANDOM_TAGS.length]); + } + + // ── Backdate background pages across the last 30 days via raw SQL + // (codex C4#7 — putPage stamps updated_at = now(), so we can't get + // historical timestamps without bypassing the engine path). + await engine.executeRaw( + `UPDATE pages + SET updated_at = now() - (random() * interval '30 days') + WHERE slug LIKE 'notes/random-%'` + ); + + // Recompute emotional_weight for the wedding pages so they get the + // tag-emotion boost in the salience formula. + const inputs = await engine.batchLoadEmotionalInputs(); + const { computeEmotionalWeight } = await import('../../src/core/cycle/emotional-weight.ts'); + const writes = inputs.map(r => ({ + slug: r.slug, + source_id: r.source_id, + weight: computeEmotionalWeight({ tags: r.tags, takes: r.takes }), + })); + await engine.setEmotionalWeightBatch(writes); +}); + +afterAll(async () => { + if (engine) await engine.disconnect(); +}); + +describe('v0.29 E2E — getRecentSalience (Garry test)', () => { + test('wedding pages outrank random-tag noise in the 7-day window', async () => { + const rows = await engine.getRecentSalience({ days: 7, limit: 20 }); + expect(rows.length).toBeGreaterThan(0); + // The top result should be a wedding page (max emotional_weight = 0.5). + const top = rows[0]; + expect(top.slug).toMatch(/^personal\/wedding\//); + expect(top.emotional_weight).toBeGreaterThan(0); + + // All 7 wedding pages should appear in the top 10. Compare against the + // result-set, not the rank order — score ties on emotional_weight + the + // recency-decay term may shuffle within the wedding cohort. + const top10 = rows.slice(0, 10).map(r => r.slug); + const weddingHits = top10.filter(s => s.startsWith('personal/wedding/')); + expect(weddingHits.length).toBeGreaterThanOrEqual(7); + }); + + test('slugPrefix filter narrows to the named directory', async () => { + const rows = await engine.getRecentSalience({ days: 30, slugPrefix: 'personal/wedding/' }); + expect(rows.length).toBe(7); + for (const r of rows) { + expect(r.slug.startsWith('personal/wedding/')).toBe(true); + } + }); + + test('days=0 returns no rows (boundary case)', async () => { + // boundary = now − 0 = now, so only pages updated > now are matched. + // updated_at = now() inserts are inclusive, so allow at most a few rows + // that match the equality boundary; assert window is at least narrow. + const rows = await engine.getRecentSalience({ days: 0, limit: 1000 }); + expect(rows.length).toBeLessThanOrEqual(7); // only wedding pages from this run + }); + + test('limit cap is respected', async () => { + const rows = await engine.getRecentSalience({ days: 365, limit: 5 }); + expect(rows.length).toBe(5); + }); + + test('empty-window slugPrefix returns []', async () => { + const rows = await engine.getRecentSalience({ days: 7, slugPrefix: 'nope/does-not-exist/' }); + expect(rows).toEqual([]); + }); +}); diff --git a/test/e2e/v0_29-mcp-dispatch-pglite.test.ts b/test/e2e/v0_29-mcp-dispatch-pglite.test.ts new file mode 100644 index 000000000..010209462 --- /dev/null +++ b/test/e2e/v0_29-mcp-dispatch-pglite.test.ts @@ -0,0 +1,149 @@ +/** + * v0.29 E2E — MCP dispatch path for the three new ops. + * + * Existing v0.29 e2e tests call engine methods directly. This file goes + * through the full `dispatchToolCall` pipeline — same code path that + * stdio MCP and HTTP MCP use — so we get coverage for: + * + * 1. validateParams (params shape contract per op definition) + * 2. buildOperationContext (ctx.remote, ctx.engine, ctx.config wiring) + * 3. handler invocation + JSON serialization (ToolResult shape) + * 4. Error path: OperationError → isError + JSON envelope + * 5. Trust gate: ctx.remote === true on get_recent_transcripts must + * reach the handler and produce a permission_denied error. + * + * Runs against PGLite in-memory. No DATABASE_URL, no API keys. + */ + +import { describe, test, expect, beforeAll, afterAll } from 'bun:test'; +import { PGLiteEngine } from '../../src/core/pglite-engine.ts'; +import { dispatchToolCall } from '../../src/mcp/dispatch.ts'; + +let engine: PGLiteEngine; + +beforeAll(async () => { + engine = new PGLiteEngine(); + await engine.connect({ engine: 'pglite' } as never); + await engine.initSchema(); + + // Seed enough fixture for the salience + anomalies ops to return non-empty + // rows. 5 wedding pages (today), 5 random-tag pages backdated 14 days. + for (let i = 0; i < 5; i++) { + const slug = `personal/wedding/photo-${i}`; + await engine.putPage(slug, { + type: 'note', + title: `Wedding photo ${i}`, + compiled_truth: `Photos from the day, batch ${i}.`, + }); + await engine.addTag(slug, 'wedding'); + } + for (let i = 0; i < 5; i++) { + const slug = `notes/bg-${i}`; + await engine.putPage(slug, { + type: 'note', + title: `Background ${i}`, + compiled_truth: `Body ${i}.`, + }); + await engine.addTag(slug, 'product'); + } + await engine.executeRaw( + `UPDATE pages SET updated_at = now() - interval '14 days' + WHERE slug LIKE 'notes/bg-%'`, + ); + + // Populate emotional_weight so the salience query produces an ordering. + const inputs = await engine.batchLoadEmotionalInputs(); + const { computeEmotionalWeight } = await import( + '../../src/core/cycle/emotional-weight.ts' + ); + const rows = inputs.map(r => ({ + slug: r.slug, + source_id: r.source_id, + weight: computeEmotionalWeight({ tags: r.tags, takes: r.takes }), + })); + await engine.setEmotionalWeightBatch(rows); +}); + +afterAll(async () => { + if (engine) await engine.disconnect(); +}); + +describe('v0.29 E2E — dispatchToolCall for the three new ops', () => { + test('get_recent_salience returns ranked rows via the MCP dispatch path', async () => { + const result = await dispatchToolCall(engine, 'get_recent_salience', { + days: 7, + limit: 10, + }, { remote: true }); + + expect(result.isError).toBeFalsy(); + expect(result.content[0].type).toBe('text'); + + const rows = JSON.parse(result.content[0].text); + expect(Array.isArray(rows)).toBe(true); + expect(rows.length).toBeGreaterThan(0); + // Wedding pages should be at or near the top (max tag-emotion boost). + expect(rows[0].slug).toMatch(/^personal\/wedding\//); + }); + + test('find_anomalies returns cohort outliers via the MCP dispatch path', async () => { + const result = await dispatchToolCall(engine, 'find_anomalies', { + lookback_days: 30, + sigma: 1.5, // lower threshold so the small fixture tips the cohort + }, { remote: true }); + + expect(result.isError).toBeFalsy(); + const rows = JSON.parse(result.content[0].text); + expect(Array.isArray(rows)).toBe(true); + // Schema check on one cohort if anything fired (small fixture may not + // always trip 1.5σ; this is a smoke contract for the response shape). + if (rows.length > 0) { + const row = rows[0]; + expect(row).toHaveProperty('cohort_kind'); + expect(row).toHaveProperty('cohort_value'); + expect(row).toHaveProperty('count'); + expect(row).toHaveProperty('baseline_mean'); + expect(row).toHaveProperty('baseline_stddev'); + expect(row).toHaveProperty('sigma_observed'); + expect(Array.isArray(row.page_slugs)).toBe(true); + } + }); + + test('get_recent_transcripts rejects with permission_denied when ctx.remote === true', async () => { + // Defense-in-depth: even though serve-http filters localOnly: true ops + // out of the MCP tool list, the in-handler ctx.remote check is the + // last line. dispatchToolCall defaults remote=true, which is what + // every MCP transport sets, so the reject must fire here. + const result = await dispatchToolCall(engine, 'get_recent_transcripts', { + days: 7, + }, { remote: true }); + + expect(result.isError).toBe(true); + const err = JSON.parse(result.content[0].text); + // OperationError.toJSON() serializes the code as `error:`, not `code:`. + expect(err.error).toBe('permission_denied'); + expect(err.message.toLowerCase()).toContain('local-only'); + }); + + test('get_recent_transcripts succeeds when ctx.remote === false (CLI path)', async () => { + // The local-CLI path explicitly sets remote: false. Op should run + // (returning [] is fine — no corpus dir is configured in this test + // fixture; the test just asserts the trust gate didn't reject). + const result = await dispatchToolCall(engine, 'get_recent_transcripts', { + days: 7, + }, { remote: false }); + + expect(result.isError).toBeFalsy(); + const rows = JSON.parse(result.content[0].text); + expect(Array.isArray(rows)).toBe(true); + // No corpus_dir configured for this test brain → empty array, not error. + expect(rows).toEqual([]); + }); + + test('unknown tool returns Unknown tool error envelope (regression guard)', async () => { + // Generic dispatch shape contract — protects against typos in op + // names accidentally short-circuiting elsewhere in the dispatcher. + const result = await dispatchToolCall(engine, 'get_recent_definitely_not_a_real_op', {}, { remote: true }); + expect(result.isError).toBe(true); + expect(result.content[0].text).toMatch(/Unknown tool/); + }); +}); diff --git a/test/effective-date.test.ts b/test/effective-date.test.ts new file mode 100644 index 000000000..7c8c261f9 --- /dev/null +++ b/test/effective-date.test.ts @@ -0,0 +1,170 @@ +/** + * v0.29.1 — Tests for computeEffectiveDate (precedence chain + per-prefix + * override + range validation + parse-failure fall-through). + * + * The function is pure (no DB), so these are fast unit tests. + */ + +import { describe, test, expect } from 'bun:test'; +import { computeEffectiveDate, parseDateLoose } from '../src/core/effective-date.ts'; + +const baseUpdated = new Date('2026-05-04T12:00:00Z'); +const baseCreated = new Date('2026-05-01T12:00:00Z'); + +function run(opts: { + slug?: string; + fm?: Record; + filename?: string | null; + updatedAt?: Date; + createdAt?: Date; +}) { + return computeEffectiveDate({ + slug: opts.slug ?? 'wiki/example', + frontmatter: opts.fm ?? {}, + filename: opts.filename ?? null, + updatedAt: opts.updatedAt ?? baseUpdated, + createdAt: opts.createdAt ?? baseCreated, + }); +} + +describe('parseDateLoose', () => { + test('Date instance passthrough', () => { + const d = new Date('2024-03-15'); + expect(parseDateLoose(d)?.getTime()).toBe(d.getTime()); + }); + test('ISO string parses', () => { + const d = parseDateLoose('2024-03-15T00:00:00Z'); + expect(d?.toISOString()).toBe('2024-03-15T00:00:00.000Z'); + }); + test('YYYY-MM-DD string parses', () => { + const d = parseDateLoose('2024-03-15'); + expect(d?.toISOString().startsWith('2024-03-15')).toBe(true); + }); + test('null/undefined → null', () => { + expect(parseDateLoose(null)).toBeNull(); + expect(parseDateLoose(undefined)).toBeNull(); + }); + test('invalid Date → null', () => { + expect(parseDateLoose(new Date('not a date'))).toBeNull(); + }); + test('unparseable string → null', () => { + expect(parseDateLoose('tomorrow')).toBeNull(); + expect(parseDateLoose('garbage')).toBeNull(); + expect(parseDateLoose('')).toBeNull(); + }); +}); + +describe('computeEffectiveDate precedence chain (default order)', () => { + test('event_date wins when present', () => { + const r = run({ fm: { event_date: '2024-03-15', date: '2024-04-01', published: '2024-05-01' } }); + expect(r.source).toBe('event_date'); + expect(r.date?.toISOString().startsWith('2024-03-15')).toBe(true); + }); + + test('date wins when event_date absent', () => { + const r = run({ fm: { date: '2024-04-01', published: '2024-05-01' } }); + expect(r.source).toBe('date'); + expect(r.date?.toISOString().startsWith('2024-04-01')).toBe(true); + }); + + test('published wins when event_date + date absent', () => { + const r = run({ fm: { published: '2024-05-01' } }); + expect(r.source).toBe('published'); + expect(r.date?.toISOString().startsWith('2024-05-01')).toBe(true); + }); + + test('filename wins when no frontmatter dates', () => { + const r = run({ filename: '2024-06-15-some-meeting' }); + expect(r.source).toBe('filename'); + expect(r.date?.toISOString().startsWith('2024-06-15')).toBe(true); + }); + + test('fallback to updated_at when chain exhausted', () => { + const r = run({}); + expect(r.source).toBe('fallback'); + expect(r.date?.toISOString()).toBe(baseUpdated.toISOString()); + }); +}); + +describe('computeEffectiveDate per-prefix override (daily/, meetings/)', () => { + test('daily/ filename wins over event_date', () => { + const r = run({ + slug: 'daily/2024-03-15', + fm: { event_date: '2024-04-01' }, + filename: '2024-03-15', + }); + expect(r.source).toBe('filename'); + expect(r.date?.toISOString().startsWith('2024-03-15')).toBe(true); + }); + + test('meetings/ filename wins over date', () => { + const r = run({ + slug: 'meetings/2024-06-15-acme-call', + fm: { date: '2024-07-01' }, + filename: '2024-06-15-acme-call', + }); + expect(r.source).toBe('filename'); + expect(r.date?.toISOString().startsWith('2024-06-15')).toBe(true); + }); + + test('daily/ falls through to event_date when filename has no date', () => { + const r = run({ + slug: 'daily/notes', + fm: { event_date: '2024-04-01' }, + filename: 'notes-some-text', + }); + expect(r.source).toBe('event_date'); + expect(r.date?.toISOString().startsWith('2024-04-01')).toBe(true); + }); + + test('non-prefixed slug uses default precedence (event_date over filename)', () => { + const r = run({ + slug: 'wiki/people/widget-ceo', + fm: { event_date: '2024-04-01' }, + filename: '2024-06-15-widget-ceo', + }); + expect(r.source).toBe('event_date'); + expect(r.date?.toISOString().startsWith('2024-04-01')).toBe(true); + }); +}); + +describe('computeEffectiveDate parse failure fall-through', () => { + test('event_date "tomorrow" falls through to date', () => { + const r = run({ fm: { event_date: 'tomorrow', date: '2024-04-01' } }); + expect(r.source).toBe('date'); + expect(r.date?.toISOString().startsWith('2024-04-01')).toBe(true); + }); + + test('all frontmatter dates unparseable → filename wins', () => { + const r = run({ + fm: { event_date: 'garbage', date: 'tomorrow', published: 'last week' }, + filename: '2024-06-15-something', + }); + expect(r.source).toBe('filename'); + expect(r.date?.toISOString().startsWith('2024-06-15')).toBe(true); + }); + + test('filename without date prefix → fallback', () => { + const r = run({ filename: 'no-date-here' }); + expect(r.source).toBe('fallback'); + expect(r.date?.toISOString()).toBe(baseUpdated.toISOString()); + }); +}); + +describe('computeEffectiveDate range validation [1990, NOW + 1y]', () => { + test('pre-1990 frontmatter date drops to next chain element', () => { + const r = run({ fm: { event_date: '1985-01-01', date: '2024-04-01' } }); + expect(r.source).toBe('date'); + }); + + test('far-future frontmatter date drops to next chain element', () => { + // NOW is 2026-05-04 in test fixtures; 2030 is > NOW + 1y + const r = run({ fm: { event_date: '2030-01-01', date: '2024-04-01' } }); + expect(r.source).toBe('date'); + }); + + test('out-of-range filename date drops to fallback', () => { + const r = run({ filename: '1850-01-01-ancient' }); + expect(r.source).toBe('fallback'); + }); +}); diff --git a/test/emotional-weight.test.ts b/test/emotional-weight.test.ts new file mode 100644 index 000000000..2c9501e1f --- /dev/null +++ b/test/emotional-weight.test.ts @@ -0,0 +1,114 @@ +import { describe, expect, test } from 'bun:test'; +import { + computeEmotionalWeight, + HIGH_EMOTION_TAGS, + DEFAULT_USER_HOLDER, +} from '../src/core/cycle/emotional-weight.ts'; + +describe('computeEmotionalWeight', () => { + test('empty inputs return 0', () => { + expect(computeEmotionalWeight({ tags: [], takes: [] })).toBe(0); + }); + + test('high-emotion tag alone gives 0.5', () => { + const w = computeEmotionalWeight({ tags: ['wedding'], takes: [] }); + expect(w).toBeCloseTo(0.5, 5); + }); + + test('tag matching is case-insensitive', () => { + const w = computeEmotionalWeight({ tags: ['WEDDING'], takes: [] }); + expect(w).toBeCloseTo(0.5, 5); + }); + + test('non-emotion tag gives no boost', () => { + const w = computeEmotionalWeight({ tags: ['hardware', 'product'], takes: [] }); + expect(w).toBe(0); + }); + + test('multiple high-emotion tags do not stack (cap at 0.5)', () => { + const w = computeEmotionalWeight({ tags: ['family', 'wedding', 'love'], takes: [] }); + expect(w).toBeCloseTo(0.5, 5); + }); + + test('takes-only: density caps at 0.3 with 3+ takes', () => { + const takes = Array.from({ length: 5 }, () => ({ + holder: 'other', + weight: 0, + kind: 'fact', + active: true, + })); + // density = min(5*0.1, 0.3) = 0.3; avg-weight = 0; holder-ratio = 0 (none match user). + expect(computeEmotionalWeight({ tags: [], takes })).toBeCloseTo(0.3, 5); + }); + + test('takes-only: avg-weight contribution scales 0..0.1', () => { + const takes = [ + { holder: 'other', weight: 1.0, kind: 'take', active: true }, + ]; + // density = 0.1; avg-weight = 1.0 * 0.1 = 0.1; holder-ratio = 0. + expect(computeEmotionalWeight({ tags: [], takes })).toBeCloseTo(0.2, 5); + }); + + test('user-holder takes give the 0.1 ratio bump', () => { + const takes = [ + { holder: DEFAULT_USER_HOLDER, weight: 0, kind: 'take', active: true }, + ]; + // density = 0.1; avg-weight = 0; holder-ratio = 1.0 * 0.1 = 0.1. + expect(computeEmotionalWeight({ tags: [], takes })).toBeCloseTo(0.2, 5); + }); + + test('user-holder ratio is mixed-holder aware', () => { + const takes = [ + { holder: DEFAULT_USER_HOLDER, weight: 0, kind: 'take', active: true }, + { holder: 'other', weight: 0, kind: 'take', active: true }, + ]; + // 1 of 2 active takes = 0.5 ratio * 0.1 = 0.05; density = 0.2; avg = 0. + expect(computeEmotionalWeight({ tags: [], takes })).toBeCloseTo(0.25, 5); + }); + + test('inactive takes are excluded from density + ratio + avg', () => { + const takes = [ + { holder: DEFAULT_USER_HOLDER, weight: 1, kind: 'take', active: false }, + { holder: DEFAULT_USER_HOLDER, weight: 1, kind: 'take', active: false }, + ]; + expect(computeEmotionalWeight({ tags: [], takes })).toBe(0); + }); + + test('output is bounded to [0..1]', () => { + // High-emotion tag + max takes + max weights + all-user-holder = 0.5+0.3+0.1+0.1 = 1.0 + const takes = Array.from({ length: 10 }, () => ({ + holder: DEFAULT_USER_HOLDER, + weight: 1.0, + kind: 'take', + active: true, + })); + const w = computeEmotionalWeight({ tags: ['wedding'], takes }); + expect(w).toBeCloseTo(1.0, 5); + expect(w).toBeLessThanOrEqual(1); + }); + + test('over-1 take weights are clamped before averaging', () => { + // weight=2.0 should clamp to 1.0 in the avg path. + const takes = [ + { holder: 'other', weight: 2.0, kind: 'take', active: true }, + ]; + expect(computeEmotionalWeight({ tags: [], takes })).toBeCloseTo(0.2, 5); + }); + + test('custom highEmotionTags override default', () => { + const customTags: ReadonlySet = new Set(['hardware-failure']); + const w = computeEmotionalWeight( + { tags: ['hardware-failure'], takes: [] }, + { highEmotionTags: customTags }, + ); + expect(w).toBeCloseTo(0.5, 5); + // Default seed list still excludes hardware-failure so without override: + expect(computeEmotionalWeight({ tags: ['hardware-failure'], takes: [] })).toBe(0); + }); + + test('HIGH_EMOTION_TAGS includes the v1 seed list', () => { + expect(HIGH_EMOTION_TAGS.has('wedding')).toBe(true); + expect(HIGH_EMOTION_TAGS.has('family')).toBe(true); + expect(HIGH_EMOTION_TAGS.has('mental-health')).toBe(true); + }); +}); diff --git a/test/migrate.test.ts b/test/migrate.test.ts index 55b102d81..b9a57ca1e 100644 --- a/test/migrate.test.ts +++ b/test/migrate.test.ts @@ -1146,6 +1146,40 @@ describe('migration v31 — eval_capture_tables', () => { }); }); +describe('migration v40 — pages_emotional_weight (v0.29)', () => { + // v0.29 ships off master. Master is at v39 (multimodal_dual_column_v0_27_1); + // v0.29 lands at v40. Idempotent ADD COLUMN IF NOT EXISTS, so brains that + // applied this at any prior number on a feature branch see v40 as new and + // run cleanly. + test('exists with the expected name', () => { + const v40 = MIGRATIONS.find(m => m.version === 40); + expect(v40).toBeDefined(); + expect(v40?.name).toBe('pages_emotional_weight'); + }); + + test('adds emotional_weight REAL NOT NULL DEFAULT 0.0 to pages', () => { + const v40 = MIGRATIONS.find(m => m.version === 40); + const sql = v40!.sql || ''; + expect(sql).toContain('ALTER TABLE pages'); + expect(sql).toContain('ADD COLUMN IF NOT EXISTS emotional_weight'); + expect(sql).toContain('REAL'); + expect(sql).toContain('NOT NULL DEFAULT 0.0'); + }); + + test('does NOT create an idx_pages_emotional_weight index (eng review D6)', () => { + // Salience query orders by computed score, not raw weight; the index + // would never be used. Adding it later requires a separate migration. + const v40 = MIGRATIONS.find(m => m.version === 40); + const sql = v40!.sql || ''; + expect(sql).not.toContain('idx_pages_emotional_weight'); + expect(sql).not.toContain('CREATE INDEX'); + }); + + test('LATEST_VERSION caught up to 40', () => { + expect(LATEST_VERSION).toBeGreaterThanOrEqual(40); + }); +}); + // ───────────────────────────────────────────────────────────────── // PR #363 regression guards — session timeouts via startup parameters // resolveSessionTimeouts — GBRAIN_*_TIMEOUT env overrides diff --git a/test/operations-descriptions.test.ts b/test/operations-descriptions.test.ts new file mode 100644 index 000000000..fc20d6116 --- /dev/null +++ b/test/operations-descriptions.test.ts @@ -0,0 +1,143 @@ +import { describe, expect, test } from 'bun:test'; +import { + GET_RECENT_SALIENCE_DESCRIPTION, + FIND_ANOMALIES_DESCRIPTION, + GET_RECENT_TRANSCRIPTS_DESCRIPTION, + LIST_PAGES_DESCRIPTION, + QUERY_DESCRIPTION, + SEARCH_DESCRIPTION, +} from '../src/core/operations-descriptions.ts'; +import { operations, operationsByName } from '../src/core/operations.ts'; +import { BRAIN_TOOL_ALLOWLIST } from '../src/core/minions/tools/brain-allowlist.ts'; + +/** + * Tool descriptions are LLM-facing strings that drive routing. v0.29 adds + * three new ops + redirects on three existing ones. These tests pin the + * key phrases that the routing decision depends on so accidental edits + * (description rewrites, AI cleanup, voice changes) fail CI. + */ + +describe('v0.29 — get_recent_salience description', () => { + test('matches the operation registration', () => { + expect(operationsByName['get_recent_salience'].description).toBe(GET_RECENT_SALIENCE_DESCRIPTION); + }); + + test('contains the explicit "Use this when" trigger phrase', () => { + expect(GET_RECENT_SALIENCE_DESCRIPTION).toContain("Use this when the user asks"); + }); + + test('lists the personal-query trigger keywords', () => { + expect(GET_RECENT_SALIENCE_DESCRIPTION).toContain("what's been going on"); + expect(GET_RECENT_SALIENCE_DESCRIPTION).toContain("anything crazy happening"); + expect(GET_RECENT_SALIENCE_DESCRIPTION).toContain("notable"); + }); + + test('explicitly bans semantic search for the same intent', () => { + expect(GET_RECENT_SALIENCE_DESCRIPTION).toContain("Do NOT run a semantic search for these"); + }); +}); + +describe('v0.29 — find_anomalies description', () => { + test('matches the operation registration', () => { + expect(operationsByName['find_anomalies'].description).toBe(FIND_ANOMALIES_DESCRIPTION); + }); + + test('mentions the cohort framing', () => { + expect(FIND_ANOMALIES_DESCRIPTION).toContain("grouped by cohort"); + expect(FIND_ANOMALIES_DESCRIPTION).toContain("(tag or type)"); + }); + + test('lists the unusual / what-stood-out trigger phrases', () => { + expect(FIND_ANOMALIES_DESCRIPTION).toContain("stood out"); + expect(FIND_ANOMALIES_DESCRIPTION).toContain("unusual"); + }); + + test('does not promise year-cohort behavior (deferred to v0.30)', () => { + // v0.29 ships tag + type only. The phrase below confirms the description + // does not lie about coverage — surfacing year would route the LLM to + // call the op for date-bucket questions it can't actually serve. + expect(FIND_ANOMALIES_DESCRIPTION).toContain("Cohort kinds: tag, type"); + }); +}); + +describe('v0.29 — get_recent_transcripts description', () => { + test('matches the operation registration', () => { + expect(operationsByName['get_recent_transcripts'].description).toBe(GET_RECENT_TRANSCRIPTS_DESCRIPTION); + }); + + test('mandates priority over query/search for personal questions', () => { + expect(GET_RECENT_TRANSCRIPTS_DESCRIPTION).toContain("FIRST"); + }); + + test('explains raw vs polished distinction', () => { + expect(GET_RECENT_TRANSCRIPTS_DESCRIPTION).toContain("NOT polished"); + expect(GET_RECENT_TRANSCRIPTS_DESCRIPTION).toContain("canonical source"); + }); + + test('discloses the local-only constraint to the LLM', () => { + expect(GET_RECENT_TRANSCRIPTS_DESCRIPTION).toContain("Local-only"); + expect(GET_RECENT_TRANSCRIPTS_DESCRIPTION).toContain("permission_denied"); + }); +}); + +describe('v0.29 — redirect hints on existing ops', () => { + test('list_pages mentions sort=updated_desc as the recency-question answer', () => { + expect(operationsByName['list_pages'].description).toBe(LIST_PAGES_DESCRIPTION); + expect(LIST_PAGES_DESCRIPTION).toContain("sort=updated_desc"); + expect(LIST_PAGES_DESCRIPTION).toContain("what did I touch this week"); + }); + + test('query redirects personal/emotional queries to the v0.29 ops', () => { + expect(operationsByName['query'].description).toBe(QUERY_DESCRIPTION); + expect(QUERY_DESCRIPTION).toContain("get_recent_salience"); + expect(QUERY_DESCRIPTION).toContain("find_anomalies"); + expect(QUERY_DESCRIPTION).toContain("get_recent_transcripts"); + }); + + test('query warns the LLM not to assume "crazy" means impressive', () => { + expect(QUERY_DESCRIPTION).toContain("Do NOT assume"); + expect(QUERY_DESCRIPTION).toContain("difficult or emotionally charged"); + }); + + test('search has the shorter redirect hint', () => { + expect(operationsByName['search'].description).toBe(SEARCH_DESCRIPTION); + expect(SEARCH_DESCRIPTION).toContain("get_recent_salience"); + }); +}); + +describe('v0.29 — subagent allow-list', () => { + test('includes get_recent_salience and find_anomalies', () => { + expect(BRAIN_TOOL_ALLOWLIST.has('get_recent_salience')).toBe(true); + expect(BRAIN_TOOL_ALLOWLIST.has('find_anomalies')).toBe(true); + }); + + test('excludes get_recent_transcripts (codex C3 — would be a remote=true footgun)', () => { + // The op throws permission_denied for remote=true callers, and all subagent + // calls run with remote=true. Including it in the allow-list would mean + // every subagent call to it returns an error — looks like a bug. + expect(BRAIN_TOOL_ALLOWLIST.has('get_recent_transcripts')).toBe(false); + }); + + test('all v0.29 ops in the allow-list resolve to a registered Operation', () => { + // brain-allowlist invariant: every name maps to an entry in operations.ts + // OPERATIONS array. This guard catches rename drift. + for (const name of ['get_recent_salience', 'find_anomalies']) { + expect(operationsByName[name]).toBeDefined(); + } + }); + + test('list_pages has new sort + updated_after params surfaced to MCP', () => { + const op = operationsByName['list_pages']; + expect(op.params.sort).toBeDefined(); + expect(op.params.updated_after).toBeDefined(); + }); +}); + +describe('v0.29 — operations array carries the three new ops', () => { + test('all three are registered (one allow-listed pair + one local-only)', () => { + const names = operations.map(o => o.name); + expect(names).toContain('get_recent_salience'); + expect(names).toContain('find_anomalies'); + expect(names).toContain('get_recent_transcripts'); + }); +}); diff --git a/test/intent.test.ts b/test/query-intent-legacy.test.ts similarity index 99% rename from test/intent.test.ts rename to test/query-intent-legacy.test.ts index c092c1828..971f311f6 100644 --- a/test/intent.test.ts +++ b/test/query-intent-legacy.test.ts @@ -3,7 +3,7 @@ */ import { describe, test, expect } from 'bun:test'; -import { classifyQueryIntent, autoDetectDetail } from '../src/core/search/intent.ts'; +import { classifyQueryIntent, autoDetectDetail } from '../src/core/search/query-intent.ts'; describe('classifyQueryIntent', () => { describe('entity queries', () => { diff --git a/test/query-intent.test.ts b/test/query-intent.test.ts new file mode 100644 index 000000000..a288cd585 --- /dev/null +++ b/test/query-intent.test.ts @@ -0,0 +1,157 @@ +/** + * v0.29.1 — merged query-intent classifier tests. + * + * Covers the new classifyQuery(query) returning {intent, suggestedDetail, + * suggestedSalience, suggestedRecency}. Legacy intent.ts behavior is + * preserved in test/query-intent-legacy.test.ts (which imports the + * classifyQueryIntent + autoDetectDetail compat shims). + * + * Pure regex; no DB. + */ + +import { describe, test, expect } from 'bun:test'; +import { classifyQuery } from '../src/core/search/query-intent.ts'; + +describe('classifyQuery — entity / canonical queries → both axes off', () => { + test('"who is widget-ceo" → recency=off, salience=off', () => { + const r = classifyQuery('who is widget-ceo'); + expect(r.intent).toBe('entity'); + expect(r.suggestedRecency).toBe('off'); + expect(r.suggestedSalience).toBe('off'); + expect(r.suggestedDetail).toBe('low'); + }); + + test('"what is recursion" → both off', () => { + const r = classifyQuery('what is recursion'); + expect(r.suggestedRecency).toBe('off'); + expect(r.suggestedSalience).toBe('off'); + }); + + test('"tell me about widget-co" → both off', () => { + const r = classifyQuery('tell me about widget-co'); + expect(r.suggestedRecency).toBe('off'); + expect(r.suggestedSalience).toBe('off'); + }); + + test('"history of X" → both off (canonical)', () => { + const r = classifyQuery('history of acme corp'); + expect(r.suggestedRecency).toBe('off'); + expect(r.suggestedSalience).toBe('off'); + }); + + test('code lookup syntax → both off', () => { + const r = classifyQuery('Foo::bar() returns null'); + expect(r.suggestedRecency).toBe('off'); + expect(r.suggestedSalience).toBe('off'); + }); + + test('graph traversal language → both off', () => { + const r = classifyQuery('show me backlinks to widget-co'); + expect(r.suggestedRecency).toBe('off'); + expect(r.suggestedSalience).toBe('off'); + }); +}); + +describe('classifyQuery — current-state queries → both axes on', () => { + test('"what\'s going on with widget-co" → both on', () => { + const r = classifyQuery("what's going on with widget-co"); + expect(r.suggestedRecency).toBe('on'); + expect(r.suggestedSalience).toBe('on'); + }); + + test('"catch me up on acme" → both on', () => { + const r = classifyQuery('catch me up on acme'); + expect(r.suggestedRecency).toBe('on'); + expect(r.suggestedSalience).toBe('on'); + }); + + test('"prep me for the widget-ceo meeting" → both on', () => { + const r = classifyQuery('prep me for the widget-ceo meeting'); + expect(r.suggestedRecency).toBe('on'); + expect(r.suggestedSalience).toBe('on'); + }); + + test('"before my meeting with X" → both on', () => { + const r = classifyQuery('before my meeting with widget-ceo'); + expect(r.suggestedRecency).toBe('on'); + expect(r.suggestedSalience).toBe('on'); + }); + + test('"remind me about acme" → both on', () => { + const r = classifyQuery('remind me about acme'); + expect(r.suggestedRecency).toBe('on'); + expect(r.suggestedSalience).toBe('on'); + }); +}); + +describe('classifyQuery — recency-only patterns (no salience signal)', () => { + test('"latest news on AI" → recency=on, salience=off', () => { + const r = classifyQuery('latest news on AI'); + expect(r.suggestedRecency).toBe('on'); + expect(r.suggestedSalience).toBe('off'); + }); + + test('"this week\'s updates" → recency=on, salience=off', () => { + const r = classifyQuery("this week's updates"); + expect(r.suggestedRecency).toBe('on'); + // "updates" + "on/with/from" pattern needed for salience + expect(r.suggestedSalience).toBe('off'); + }); +}); + +describe('classifyQuery — strong recency ("today" / "right now")', () => { + test('"what happened today" → recency=strong', () => { + const r = classifyQuery('what happened today'); + expect(r.suggestedRecency).toBe('strong'); + }); + + test('"right now what is the status" → strong', () => { + const r = classifyQuery('right now what is the status of the deal'); + // "what is" canonical fires; but "right now" is a temporal bound + expect(r.suggestedRecency).toBe('strong'); + }); +}); + +describe('classifyQuery — D6 narrow temporal-bound exception', () => { + test('"who is widget-ceo right now" → recency=strong (temporal bound wins)', () => { + const r = classifyQuery('who is widget-ceo right now'); + expect(r.suggestedRecency).toBe('strong'); + }); + + test('"who is widget-ceo today" → recency=strong', () => { + const r = classifyQuery('who is widget-ceo today'); + expect(r.suggestedRecency).toBe('strong'); + }); + + test('"who is widget-ceo" (no bound) → recency=off (canonical wins)', () => { + const r = classifyQuery('who is widget-ceo'); + expect(r.suggestedRecency).toBe('off'); + }); + + test('"what is widget-co\'s status this week" → recency=on (temporal bound wins)', () => { + const r = classifyQuery("what is widget-co's status this week"); + expect(r.suggestedRecency).toBe('on'); + }); +}); + +describe('classifyQuery — orthogonality of axes', () => { + test('default plain query → both off', () => { + const r = classifyQuery('the quick brown fox'); + expect(r.suggestedRecency).toBe('off'); + expect(r.suggestedSalience).toBe('off'); + expect(r.intent).toBe('general'); + expect(r.suggestedDetail).toBeUndefined(); + }); + + test('intent vs recency are independent axes', () => { + // "when did widget-co IPO": both 'when' (temporal) and 'IPO' (event) + // match v0.29.0 patterns. classifyQueryIntent's priority is + // temporal > event so .intent = 'temporal'. But recency depends on + // CANONICAL/RECENCY_ON patterns, not on .intent — neither set + // matches here, so suggestedRecency = 'off'. + const r = classifyQuery('when did widget-co IPO'); + expect(r.intent).toBe('temporal'); + expect(r.suggestedRecency).toBe('off'); + expect(r.suggestedSalience).toBe('off'); + }); +}); diff --git a/test/recency-decay.test.ts b/test/recency-decay.test.ts new file mode 100644 index 000000000..81f58af00 --- /dev/null +++ b/test/recency-decay.test.ts @@ -0,0 +1,221 @@ +/** + * v0.29.1 — recency-decay map + buildRecencyComponentSql tests. + * + * Pure functions, no DB. Fast unit tests. Cover the full env / yaml / merge + * resolution chain plus the SQL CASE shape (longest-prefix-match, evergreen + * short-circuit, injection-safe NowExpr). + */ + +import { describe, test, expect } from 'bun:test'; +import { + DEFAULT_RECENCY_DECAY, + DEFAULT_FALLBACK, + RecencyDecayParseError, + parseRecencyDecayEnv, + parseRecencyDecayYaml, + resolveRecencyDecayMap, +} from '../src/core/search/recency-decay.ts'; +import { buildRecencyComponentSql } from '../src/core/search/sql-ranking.ts'; + +describe('parseRecencyDecayEnv', () => { + test('empty / undefined → empty map', () => { + expect(parseRecencyDecayEnv(undefined)).toEqual({}); + expect(parseRecencyDecayEnv('')).toEqual({}); + }); + + test('single triple', () => { + expect(parseRecencyDecayEnv('daily/:7:1.5')).toEqual({ + 'daily/': { halflifeDays: 7, coefficient: 1.5 }, + }); + }); + + test('multiple triples comma-separated', () => { + const out = parseRecencyDecayEnv('daily/:7:1.5,concepts/:0:0,custom/:30:0.5'); + expect(out['daily/']).toEqual({ halflifeDays: 7, coefficient: 1.5 }); + expect(out['concepts/']).toEqual({ halflifeDays: 0, coefficient: 0 }); + expect(out['custom/']).toEqual({ halflifeDays: 30, coefficient: 0.5 }); + }); + + test('throws on missing field', () => { + expect(() => parseRecencyDecayEnv('daily/:7')).toThrow(RecencyDecayParseError); + expect(() => parseRecencyDecayEnv('daily/')).toThrow(RecencyDecayParseError); + }); + + test('throws on negative halflife', () => { + expect(() => parseRecencyDecayEnv('daily/:-1:1.5')).toThrow(RecencyDecayParseError); + }); + + test('throws on negative coefficient', () => { + expect(() => parseRecencyDecayEnv('daily/:7:-0.1')).toThrow(RecencyDecayParseError); + }); + + test('throws on non-numeric values', () => { + expect(() => parseRecencyDecayEnv('daily/:abc:1.5')).toThrow(RecencyDecayParseError); + }); + + test('throws on empty prefix', () => { + expect(() => parseRecencyDecayEnv(':7:1.5')).toThrow(RecencyDecayParseError); + }); +}); + +describe('parseRecencyDecayYaml', () => { + test('null / undefined / empty → empty map', () => { + expect(parseRecencyDecayYaml(null)).toEqual({}); + expect(parseRecencyDecayYaml(undefined)).toEqual({}); + expect(parseRecencyDecayYaml({})).toEqual({}); + }); + + test('valid recency block', () => { + const out = parseRecencyDecayYaml({ + recency: { + 'daily/': { halflifeDays: 14, coefficient: 1.5 }, + 'concepts/': { halflifeDays: 0, coefficient: 0 }, + }, + }); + expect(out['daily/']).toEqual({ halflifeDays: 14, coefficient: 1.5 }); + expect(out['concepts/']).toEqual({ halflifeDays: 0, coefficient: 0 }); + }); + + test('throws on bad halflifeDays', () => { + expect(() => + parseRecencyDecayYaml({ recency: { 'daily/': { halflifeDays: -1, coefficient: 1.0 } } }), + ).toThrow(RecencyDecayParseError); + }); + + test('throws on non-object entry', () => { + expect(() => + parseRecencyDecayYaml({ recency: { 'daily/': 'invalid' } }), + ).toThrow(RecencyDecayParseError); + }); +}); + +describe('resolveRecencyDecayMap merge precedence', () => { + test('defaults baseline', () => { + const m = resolveRecencyDecayMap({}); + expect(m['concepts/']).toEqual({ halflifeDays: 0, coefficient: 0 }); + expect(m['daily/']).toEqual({ halflifeDays: 14, coefficient: 1.5 }); + }); + + test('env overrides defaults', () => { + const m = resolveRecencyDecayMap({ envValue: 'daily/:30:0.5' }); + expect(m['daily/']).toEqual({ halflifeDays: 30, coefficient: 0.5 }); + }); + + test('yaml + env: env wins', () => { + const m = resolveRecencyDecayMap({ + yaml: { recency: { 'daily/': { halflifeDays: 7, coefficient: 2.0 } } }, + envValue: 'daily/:30:0.5', + }); + expect(m['daily/']).toEqual({ halflifeDays: 30, coefficient: 0.5 }); + }); + + test('caller wins over env', () => { + const m = resolveRecencyDecayMap({ + envValue: 'daily/:30:0.5', + caller: { 'daily/': { halflifeDays: 1, coefficient: 5.0 } }, + }); + expect(m['daily/']).toEqual({ halflifeDays: 1, coefficient: 5.0 }); + }); +}); + +describe('buildRecencyComponentSql', () => { + const mini = { + 'concepts/': { halflifeDays: 0, coefficient: 0 }, + 'daily/': { halflifeDays: 14, coefficient: 1.5 }, + 'media/': { halflifeDays: 90, coefficient: 0.5 }, + }; + + test('emits CASE expression with longest-prefix-first ordering', () => { + const longerFirst = { + 'media/articles/': { halflifeDays: 60, coefficient: 0.5 }, + 'media/': { halflifeDays: 90, coefficient: 0.4 }, + }; + const sql = buildRecencyComponentSql({ + slugColumn: 'p.slug', + dateExpr: 'p.updated_at', + decayMap: longerFirst, + fallback: DEFAULT_FALLBACK, + }); + const idxLong = sql.indexOf("'media/articles/%'"); + const idxShort = sql.indexOf("'media/%'"); + expect(idxLong).toBeGreaterThan(0); + expect(idxShort).toBeGreaterThan(0); + expect(idxLong).toBeLessThan(idxShort); + }); + + test('evergreen short-circuit emits literal 0', () => { + const sql = buildRecencyComponentSql({ + slugColumn: 'p.slug', + dateExpr: 'p.updated_at', + decayMap: mini, + fallback: DEFAULT_FALLBACK, + }); + expect(sql).toContain("WHEN p.slug LIKE 'concepts/%' THEN 0"); + }); + + test('non-zero branches include EXTRACT(EPOCH ...)', () => { + const sql = buildRecencyComponentSql({ + slugColumn: 'p.slug', + dateExpr: 'p.updated_at', + decayMap: mini, + fallback: DEFAULT_FALLBACK, + }); + expect(sql).toContain('EXTRACT(EPOCH FROM (NOW() - p.updated_at)) / 86400.0'); + expect(sql).toContain('1.5 * 14.0 / (14.0 + EXTRACT(EPOCH'); + }); + + test('NowExpr.fixed is escaped (single-quote doubling) and timestamptz-cast', () => { + const sql = buildRecencyComponentSql({ + slugColumn: 'p.slug', + dateExpr: 'p.updated_at', + decayMap: { 'daily/': { halflifeDays: 7, coefficient: 1.0 } }, + fallback: DEFAULT_FALLBACK, + now: { kind: 'fixed', isoUtc: "2026-05-04T00:00:00Z" }, + }); + expect(sql).toContain("'2026-05-04T00:00:00Z'::timestamptz"); + expect(sql).not.toContain('NOW()'); + }); + + test('NowExpr.fixed with embedded single quote is doubled (injection defense)', () => { + const sql = buildRecencyComponentSql({ + slugColumn: 'p.slug', + dateExpr: 'p.updated_at', + decayMap: { 'daily/': { halflifeDays: 7, coefficient: 1.0 } }, + fallback: DEFAULT_FALLBACK, + now: { kind: 'fixed', isoUtc: "2026'; DROP TABLE pages;--" }, + }); + // The malicious quote must be doubled to ''. + expect(sql).toContain("''"); + expect(sql).not.toContain("DROP TABLE'"); + }); + + test('empty decayMap → only fallback ELSE branch', () => { + const sql = buildRecencyComponentSql({ + slugColumn: 'p.slug', + dateExpr: 'p.updated_at', + decayMap: {}, + fallback: { halflifeDays: 30, coefficient: 1.0 }, + }); + expect(sql).not.toContain('CASE'); + expect(sql).toContain('1 * 30.0 / (30.0 +'); + }); +}); + +describe('DEFAULT_RECENCY_DECAY composition', () => { + test('does not contain fork-specific names (no openclaw/, no wintermute/)', () => { + const keys = Object.keys(DEFAULT_RECENCY_DECAY); + for (const k of keys) { + expect(k.includes('openclaw')).toBe(false); + expect(k.includes('wintermute')).toBe(false); + } + }); + + test('concepts/ is evergreen (halflifeDays = 0)', () => { + expect(DEFAULT_RECENCY_DECAY['concepts/']?.halflifeDays).toBe(0); + }); + + test('daily/ has aggressive decay', () => { + expect(DEFAULT_RECENCY_DECAY['daily/']?.halflifeDays).toBeLessThan(30); + expect(DEFAULT_RECENCY_DECAY['daily/']?.coefficient).toBeGreaterThan(1); + }); +}); diff --git a/test/recompute-emotional-weight.test.ts b/test/recompute-emotional-weight.test.ts new file mode 100644 index 000000000..f2ada9565 --- /dev/null +++ b/test/recompute-emotional-weight.test.ts @@ -0,0 +1,133 @@ +import { describe, expect, test } from 'bun:test'; +import { runPhaseRecomputeEmotionalWeight } from '../src/core/cycle/recompute-emotional-weight.ts'; +import type { + EmotionalWeightInputRow, + EmotionalWeightWriteRow, +} from '../src/core/types.ts'; + +/** + * Unit-level coverage for the v0.29 recompute_emotional_weight phase. + * The full e2e (against PGLite) is in test/e2e/cycle-recompute-emotional-weight-pglite.test.ts. + */ + +interface FakeEngine { + batchLoadEmotionalInputs(slugs?: string[]): Promise; + setEmotionalWeightBatch(rows: EmotionalWeightWriteRow[]): Promise; + getConfig(key: string): Promise; +} + +function makeEngine(rows: EmotionalWeightInputRow[], configMap: Record = {}): FakeEngine & { + written: EmotionalWeightWriteRow[]; + loadCalls: (string[] | undefined)[]; +} { + const written: EmotionalWeightWriteRow[] = []; + const loadCalls: (string[] | undefined)[] = []; + return { + written, + loadCalls, + async batchLoadEmotionalInputs(slugs?: string[]) { + loadCalls.push(slugs); + if (!slugs) return rows; + const sset = new Set(slugs); + return rows.filter(r => sset.has(r.slug)); + }, + async setEmotionalWeightBatch(rs: EmotionalWeightWriteRow[]) { + written.push(...rs); + return rs.length; + }, + async getConfig(key: string) { + return configMap[key] ?? null; + }, + }; +} + +describe('runPhaseRecomputeEmotionalWeight', () => { + test('zero affected slugs short-circuits — no DB read, no write', async () => { + const engine = makeEngine([]); + const r = await runPhaseRecomputeEmotionalWeight(engine as any, { affectedSlugs: [] }); + expect(r.status).toBe('ok'); + expect(r.pages_recomputed).toBe(0); + expect(engine.loadCalls.length).toBe(0); // skipped batch read + expect(engine.written.length).toBe(0); + }); + + test('full mode walks every page', async () => { + const rows: EmotionalWeightInputRow[] = [ + { slug: 'a', source_id: 'default', tags: ['wedding'], takes: [] }, + { slug: 'b', source_id: 'default', tags: [], takes: [] }, + ]; + const engine = makeEngine(rows); + const r = await runPhaseRecomputeEmotionalWeight(engine as any, {}); + expect(r.status).toBe('ok'); + expect(r.pages_recomputed).toBe(2); + expect(engine.written.length).toBe(2); + // Both rows present, with weights from the formula. + const byslug = Object.fromEntries(engine.written.map(w => [w.slug, w.weight])); + expect(byslug.a).toBeCloseTo(0.5, 5); // wedding tag + expect(byslug.b).toBe(0); + }); + + test('incremental mode passes slugs through to engine read', async () => { + const rows: EmotionalWeightInputRow[] = [ + { slug: 'a', source_id: 'default', tags: ['wedding'], takes: [] }, + { slug: 'b', source_id: 'default', tags: ['family'], takes: [] }, + { slug: 'c', source_id: 'default', tags: [], takes: [] }, + ]; + const engine = makeEngine(rows); + const r = await runPhaseRecomputeEmotionalWeight(engine as any, { affectedSlugs: ['a', 'c'] }); + expect(r.status).toBe('ok'); + expect(engine.loadCalls[0]).toEqual(['a', 'c']); + // Filter on the fake engine returns only a + c → both written. + expect(engine.written.map(w => w.slug).sort()).toEqual(['a', 'c']); + }); + + test('multi-source: writes preserve source_id (no fan-out)', async () => { + const rows: EmotionalWeightInputRow[] = [ + { slug: 'shared', source_id: 'src-a', tags: ['wedding'], takes: [] }, + { slug: 'shared', source_id: 'src-b', tags: [], takes: [] }, + ]; + const engine = makeEngine(rows); + await runPhaseRecomputeEmotionalWeight(engine as any, {}); + const sources = engine.written.map(w => w.source_id).sort(); + expect(sources).toEqual(['src-a', 'src-b']); + const wA = engine.written.find(w => w.source_id === 'src-a'); + const wB = engine.written.find(w => w.source_id === 'src-b'); + expect(wA!.weight).toBeCloseTo(0.5, 5); + expect(wB!.weight).toBe(0); + }); + + test('dry-run computes weights but skips the UPDATE', async () => { + const rows: EmotionalWeightInputRow[] = [ + { slug: 'a', source_id: 'default', tags: ['wedding'], takes: [] }, + ]; + const engine = makeEngine(rows); + const r = await runPhaseRecomputeEmotionalWeight(engine as any, { dryRun: true }); + expect(r.status).toBe('ok'); + expect(r.details.dry_run).toBe(true); + expect(r.pages_recomputed).toBe(1); // would-write count + expect(engine.written.length).toBe(0); // but nothing actually written + }); + + test('config override of high_emotion_tags is honored', async () => { + const rows: EmotionalWeightInputRow[] = [ + { slug: 'p', source_id: 'default', tags: ['hardware-failure'], takes: [] }, + ]; + const engine = makeEngine(rows, { + 'emotional_weight.high_tags': JSON.stringify(['hardware-failure']), + }); + await runPhaseRecomputeEmotionalWeight(engine as any, {}); + expect(engine.written[0].weight).toBeCloseTo(0.5, 5); + }); + + test('engine throw bubbles into a fail PhaseResult, not an unhandled exception', async () => { + const engine: FakeEngine = { + batchLoadEmotionalInputs: async () => { throw new Error('db down'); }, + setEmotionalWeightBatch: async () => 0, + getConfig: async () => null, + }; + const r = await runPhaseRecomputeEmotionalWeight(engine as any, {}); + expect(r.status).toBe('fail'); + expect(r.error?.code).toBe('RECOMPUTE_EMOTIONAL_WEIGHT_FAIL'); + expect(r.error?.message).toContain('db down'); + }); +}); diff --git a/test/salience.test.ts b/test/salience.test.ts new file mode 100644 index 000000000..d9dd97954 --- /dev/null +++ b/test/salience.test.ts @@ -0,0 +1,42 @@ +import { describe, expect, test } from 'bun:test'; +import { computeAnomaliesFromBuckets } from '../src/core/cycle/anomaly.ts'; + +/** + * Unit-level salience checks. The full Garry-test fixture lives in + * test/e2e/salience-pglite.test.ts (PGLite, no DATABASE_URL needed). + * + * These are the pure-function checks for the CLI args parser and a smoke + * for the engine method shape via a minimal fake. Most of the salience + * logic is SQL — see e2e for behavior validation. + */ + +describe('v0.29 — salience SQL shape pinned via type contract', () => { + // The salience score formula is in postgres-engine.ts and pglite-engine.ts + // SQL strings. These are smoke-tested against the engines in + // test/e2e/salience-pglite.test.ts and (optionally) the Postgres parity test. + // Here we just confirm the SalienceResult fields are present on the type + // so any future renames break compilation, not runtime. + test('SalienceResult is a stable contract', async () => { + const mod = await import('../src/core/types.ts'); + // If any of these fields is dropped, tsc fails before tests run. + const sample = {} as import('../src/core/types.ts').SalienceResult; + void sample.slug; + void sample.source_id; + void sample.title; + void sample.type; + void sample.updated_at; + void sample.emotional_weight; + void sample.take_count; + void sample.take_avg_weight; + void sample.score; + expect(typeof mod).toBe('object'); + }); +}); + +describe('v0.29 — anomaly cohort buckets connect to the engine', () => { + test('computeAnomaliesFromBuckets is exported and pure', () => { + // Smoke that the import path engines use is wired. The behavior is + // covered exhaustively in test/anomalies.test.ts. + expect(typeof computeAnomaliesFromBuckets).toBe('function'); + }); +}); diff --git a/test/transcripts.test.ts b/test/transcripts.test.ts new file mode 100644 index 000000000..80af2a9ac --- /dev/null +++ b/test/transcripts.test.ts @@ -0,0 +1,159 @@ +import { describe, expect, test, beforeAll, afterAll, beforeEach } from 'bun:test'; +import { mkdtempSync, writeFileSync, rmSync, mkdirSync, utimesSync } from 'node:fs'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; +import { listRecentTranscripts } from '../src/core/transcripts.ts'; + +/** + * v0.29 — listRecentTranscripts unit coverage. + * + * Uses a hermetic temp dir as the corpus dir. Engine calls are mocked + * minimally — we only use engine.getConfig to resolve corpus paths. + */ + +let tmpRoot: string; +let corpusDir: string; +const configMap = new Map(); + +const fakeEngine = { + async getConfig(key: string): Promise { + return configMap.get(key) ?? null; + }, +} as unknown as Parameters[0]; + +function setMtime(path: string, mtimeMs: number) { + utimesSync(path, mtimeMs / 1000, mtimeMs / 1000); +} + +beforeAll(() => { + tmpRoot = mkdtempSync(join(tmpdir(), 'gbrain-transcripts-test-')); + corpusDir = join(tmpRoot, 'sessions'); + mkdirSync(corpusDir, { recursive: true }); +}); + +afterAll(() => { + if (tmpRoot) rmSync(tmpRoot, { recursive: true, force: true }); +}); + +beforeEach(() => { + configMap.clear(); + configMap.set('dream.synthesize.session_corpus_dir', corpusDir); +}); + +describe('listRecentTranscripts', () => { + test('returns [] when no corpus dir is configured', async () => { + configMap.clear(); + const result = await listRecentTranscripts(fakeEngine); + expect(result).toEqual([]); + }); + + test('returns [] when the corpus dir is empty', async () => { + // Empty dir, no files yet. + const subdir = join(tmpRoot, 'empty-dir'); + mkdirSync(subdir, { recursive: true }); + configMap.set('dream.synthesize.session_corpus_dir', subdir); + const result = await listRecentTranscripts(fakeEngine); + expect(result).toEqual([]); + }); + + test('returns transcripts within mtime window, newest first', async () => { + const a = join(corpusDir, '2026-04-25-session-a.txt'); + const b = join(corpusDir, '2026-04-26-session-b.txt'); + writeFileSync(a, 'A content first line\nmore content here.\n'); + writeFileSync(b, 'B content first line\nmore content there.\n'); + // Set b newer than a. + setMtime(a, Date.now() - 86400000); // 1 day ago + setMtime(b, Date.now() - 3600000); // 1 hour ago + + const result = await listRecentTranscripts(fakeEngine, { days: 7 }); + expect(result.length).toBe(2); + expect(result[0].path).toBe('2026-04-26-session-b.txt'); + expect(result[1].path).toBe('2026-04-25-session-a.txt'); + }); + + test('mtime window excludes older files', async () => { + const old = join(corpusDir, '2020-01-01-old.txt'); + writeFileSync(old, 'old content\n'); + setMtime(old, Date.now() - 365 * 86400000); // 1 year ago + + const result = await listRecentTranscripts(fakeEngine, { days: 7 }); + expect(result.find(r => r.path === '2020-01-01-old.txt')).toBeUndefined(); + }); + + test('skips dream-generated outputs', async () => { + const dream = join(corpusDir, '2026-04-26-dream-out.txt'); + // Identity marker: ---\n + frontmatter with dream_generated: true. + writeFileSync(dream, '---\ntitle: A reflection\ndream_generated: true\n---\n\nbody\n'); + setMtime(dream, Date.now() - 60_000); + + const result = await listRecentTranscripts(fakeEngine, { days: 7 }); + expect(result.find(r => r.path === '2026-04-26-dream-out.txt')).toBeUndefined(); + }); + + test('summary=true returns first non-empty line + ~250 trailing chars', async () => { + const file = join(corpusDir, '2026-04-26-summary.txt'); + const body = 'First line of the transcript\n' + 'x'.repeat(500); + writeFileSync(file, body); + setMtime(file, Date.now()); + + const result = await listRecentTranscripts(fakeEngine, { days: 7, summary: true }); + const row = result.find(r => r.path === '2026-04-26-summary.txt'); + expect(row).toBeDefined(); + expect(row!.summary.startsWith('First line of the transcript')).toBe(true); + // The summary should be much shorter than the full body. + expect(row!.summary.length).toBeLessThan(body.length); + expect(row!.summary.length).toBeLessThan(400); + }); + + test('summary=false returns full content capped at 100 KB', async () => { + const file = join(corpusDir, '2026-04-26-full.txt'); + const body = 'big body\n' + 'y'.repeat(200_000); + writeFileSync(file, body); + setMtime(file, Date.now()); + + const result = await listRecentTranscripts(fakeEngine, { days: 7, summary: false }); + const row = result.find(r => r.path === '2026-04-26-full.txt'); + expect(row).toBeDefined(); + expect(row!.summary.length).toBeLessThanOrEqual(100 * 1024); + expect(row!.length).toBe(body.length); // length always = full file size + }); + + test('extracts date from YYYY-MM-DD prefix when present', async () => { + const file = join(corpusDir, '2026-05-01-dated.txt'); + writeFileSync(file, 'content\n'); + setMtime(file, Date.now()); + + const result = await listRecentTranscripts(fakeEngine, { days: 7 }); + const row = result.find(r => r.path === '2026-05-01-dated.txt'); + expect(row).toBeDefined(); + expect(row!.date).toBe('2026-05-01'); + }); + + test('returns null date when filename has no date prefix', async () => { + const file = join(corpusDir, 'random-name.txt'); + writeFileSync(file, 'content\n'); + setMtime(file, Date.now()); + + const result = await listRecentTranscripts(fakeEngine, { days: 7 }); + const row = result.find(r => r.path === 'random-name.txt'); + expect(row).toBeDefined(); + expect(row!.date).toBeNull(); + }); + + test('limit caps the number of returned transcripts', async () => { + // Add 5 files. + for (let i = 0; i < 5; i++) { + const file = join(corpusDir, `multi-${i}.txt`); + writeFileSync(file, `content ${i}\n`); + setMtime(file, Date.now()); + } + const result = await listRecentTranscripts(fakeEngine, { days: 7, limit: 2 }); + expect(result.length).toBe(2); + }); + + test('non-existent corpus dir is skipped silently', async () => { + configMap.set('dream.synthesize.session_corpus_dir', '/nope/does/not/exist'); + const result = await listRecentTranscripts(fakeEngine); + expect(result).toEqual([]); + }); +}); diff --git a/test/v0_29-tool-surfaces.test.ts b/test/v0_29-tool-surfaces.test.ts new file mode 100644 index 000000000..423a1984b --- /dev/null +++ b/test/v0_29-tool-surfaces.test.ts @@ -0,0 +1,111 @@ +/** + * v0.29 — tool-surface contracts. + * + * Verifies two filter contracts that touch the v0.29 ops but live outside + * the v0.29 source files (in serve-http.ts and brain-allowlist.ts): + * + * 1. `localOnly: true` on `get_recent_transcripts` is what hides it from + * the HTTP MCP tool-list. serve-http.ts:745 does + * `operations.filter(op => !op.localOnly)`. The v0.29 trust gate + * (in-handler `ctx.remote === true` reject) is defense-in-depth on top + * of this; if the filter ever drops the flag, the in-handler check is + * the last line. We assert both halves of the contract. + * + * 2. `buildBrainTools` (subagent registry) surfaces salience + anomalies + * as `brain_get_recent_salience` / `brain_find_anomalies` and EXCLUDES + * `brain_get_recent_transcripts`. The exclusion is intentional — + * subagent calls always run with `ctx.remote === true`, and the v0.29 + * trust gate would always reject. Listing it would be a footgun + * (subagent calls op, gets permission_denied, looks like a bug). + * + * Both filters are pure-function checks; no DB / engine / network needed. + */ + +import { describe, expect, test } from 'bun:test'; +import { operations } from '../src/core/operations.ts'; +import { buildBrainTools } from '../src/core/minions/tools/brain-allowlist.ts'; +import type { BrainEngine } from '../src/core/engine.ts'; +import type { GBrainConfig } from '../src/core/config.ts'; + +describe('v0.29 — serve-http localOnly filter', () => { + // serve-http.ts:745 is the canonical filter expression. + const mcpVisible = operations.filter(op => !op.localOnly); + + test('get_recent_transcripts is hidden from the HTTP MCP tool list', () => { + const names = mcpVisible.map(o => o.name); + expect(names).not.toContain('get_recent_transcripts'); + }); + + test('get_recent_salience and find_anomalies stay visible', () => { + const names = mcpVisible.map(o => o.name); + expect(names).toContain('get_recent_salience'); + expect(names).toContain('find_anomalies'); + }); + + test('get_recent_transcripts carries the localOnly: true flag', () => { + const op = operations.find(o => o.name === 'get_recent_transcripts'); + expect(op).toBeDefined(); + expect(op!.localOnly).toBe(true); + }); + + test('get_recent_salience and find_anomalies do NOT carry localOnly', () => { + // Read-only ops that subagents need; localOnly would block them from MCP. + const sal = operations.find(o => o.name === 'get_recent_salience'); + const ano = operations.find(o => o.name === 'find_anomalies'); + expect(sal!.localOnly).toBeFalsy(); + expect(ano!.localOnly).toBeFalsy(); + }); + + test('all three v0.29 ops carry scope: read', () => { + // v0.26.0 contract: every op must annotate scope. Read-only is correct + // for all three (no DB writes, no fs writes). + for (const name of ['get_recent_salience', 'find_anomalies', 'get_recent_transcripts']) { + const op = operations.find(o => o.name === name); + expect(op!.scope).toBe('read'); + } + }); +}); + +describe('v0.29 — buildBrainTools subagent surfacing', () => { + // buildBrainTools doesn't issue any SQL at registry-build time — it only + // reads `engine.kind` for the put_page namespace-wrap branch. A minimal + // fake-engine literal keeps the test pure (no PGLite WASM cold-start, no + // connect/disconnect lifecycle, no test-isolation R3/R4 violations). + // Cast through `unknown` because the BrainEngine surface is large and + // we only touch one property. + const fakeEngine = { kind: 'pglite' } as unknown as BrainEngine; + const config: GBrainConfig = { engine: 'pglite' } as GBrainConfig; + + test('subagent registry includes brain_get_recent_salience', () => { + const tools = buildBrainTools({ subagentId: 1, engine: fakeEngine, config }); + const names = tools.map(t => t.name); + expect(names).toContain('brain_get_recent_salience'); + }); + + test('subagent registry includes brain_find_anomalies', () => { + const tools = buildBrainTools({ subagentId: 1, engine: fakeEngine, config }); + const names = tools.map(t => t.name); + expect(names).toContain('brain_find_anomalies'); + }); + + test('subagent registry EXCLUDES brain_get_recent_transcripts (codex C3 footgun gate)', () => { + // All subagent calls run with ctx.remote === true; the v0.29 trust gate + // would always reject. Listing the op would be a footgun: subagent + // calls it, gets permission_denied, looks like a bug. The cycle's + // synthesize phase reaches transcripts via discoverTranscripts directly, + // not via the op. + const tools = buildBrainTools({ subagentId: 1, engine: fakeEngine, config }); + const names = tools.map(t => t.name); + expect(names).not.toContain('brain_get_recent_transcripts'); + }); + + test('the v0.29 ops carry their description verbatim into the registry', () => { + const tools = buildBrainTools({ subagentId: 1, engine: fakeEngine, config }); + const sal = tools.find(t => t.name === 'brain_get_recent_salience'); + const ano = tools.find(t => t.name === 'brain_find_anomalies'); + const opSal = operations.find(o => o.name === 'get_recent_salience'); + const opAno = operations.find(o => o.name === 'find_anomalies'); + expect(sal!.description).toBe(opSal!.description); + expect(ano!.description).toBe(opAno!.description); + }); +});