v0.29.0 + v0.29.1 feat: salience + anomaly detection — brain surfaces what's hot without being asked (#730)

* v0.29 foundation: emotional_weight column + formula + anomaly stats

Migration v34 adds pages.emotional_weight REAL DEFAULT 0.0 (column-only,
no index — salience query orders by computed score, not raw weight).
Embedded DDL (schema.sql + pglite-schema.ts + schema-embedded.ts)
mirrors the column so fresh installs don't need migration replay.

types.ts gains: PageFilters.sort enum + PAGE_SORT_SQL whitelist (engines
hardcoded ORDER BY updated_at DESC; threading lands in the next commit);
SalienceOpts/SalienceResult, AnomaliesOpts/AnomalyResult,
EmotionalWeightInputRow/EmotionalWeightWriteRow contracts.

cycle/emotional-weight.ts: pure-function score in [0..1] from tags +
takes (anglocentric default seed list; user-overridable via config key
emotional_weight.high_tags). cycle/anomaly.ts: meanStddev + cohort
threshold helpers with zero-stddev fallback (count > mean + 1) so rare
cohorts don't produce NaN sigmas.

Test coverage: migrate v34 structural assertions + 14-case formula
unit + 13-case anomaly stats unit. Codex review fixes baked in:
formula clamped to [0,1]; per-take weight clamped to [0,1] before
averaging; zero-stddev fallback finite, never NaN.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* v0.29 engine: batch emotional-weight methods + listPages sort

BrainEngine adds 4 methods, both engines implement:

- batchLoadEmotionalInputs(slugs?): CTE-shaped read with per-table
  pre-aggregates. A page with N tags + M takes never produces N×M rows
  (codex C4#4) — page_tags + page_takes CTEs aggregate independently,
  then LEFT JOIN to pages.

- setEmotionalWeightBatch(rows): UPDATE FROM unnest($1::text[],
  $2::text[], $3::real[]) composite-keyed on (slug, source_id). Multi-
  source brains can't fan out (codex C4#3) — pages.slug is unique only
  within source_id. Same shape that v0.18 link batches use.

- getRecentSalience: time boundary computed in JS, bound as TIMESTAMPTZ.
  SQL identical across engines (codex C5/D5 — avoids dialect drift on
  $1::interval binding which has zero current uses on PGLite).

- findAnomalies: tag + type cohort baselines via generate_series-
  densified daily-count CTEs (codex C4#6). Sparse-day rare cohorts get
  correct (mean, stddev) instead of biased upward by zero-omission.
  Year cohort deferred to v0.30.

listPages threads the new PageFilters.sort enum through both engines.
Was hardcoded ORDER BY updated_at DESC; now PAGE_SORT_SQL whitelist
maps the 4 enum values to literal SQL fragments — no injection surface.
postgres.js uses sql.unsafe; PGLite splices the fragment directly.

Regression tests (PGLite, no DATABASE_URL needed):

- multi-source-emotional-weight: same slug under two source_ids,
  setEmotionalWeightBatch on one of them, asserts the other survives
  untouched. Direct codex C4#3 guard.

- list-pages-regression (IRON RULE): old call shape (type, tag, limit)
  still returns updated_desc default; new sort=updated_asc reverses;
  sort=created_desc orders by created_at; sort=slug alphabetical;
  unsupported sort enum falls back to default (defense in depth).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* v0.29 cycle: new recompute_emotional_weight phase

Adds a 9th cycle phase between extract and embed. Sees the union of
syncPagesAffected + synthesizeWrittenSlugs for incremental mode (so
synthesize-written pages get their weight computed too — codex C2 caught
that the prior plan threaded only sync). Full mode (no incremental
anchors) walks every page; users hit this path on first upgrade via
gbrain dream --phase recompute_emotional_weight.

Phase orchestrator (cycle/recompute-emotional-weight.ts) is two SQL
round-trips total regardless of brain size:
  1. batchLoadEmotionalInputs(slugs?) → per-page tag/take inputs.
  2. computeEmotionalWeight in memory (pure function).
  3. setEmotionalWeightBatch(rows) → composite-keyed UPDATE FROM unnest.

Empty affectedSlugs short-circuits (no DB read, no write). Dry-run
computes weights and reports the would-write count without touching
the DB. Engine throw bubbles into status:fail with code
RECOMPUTE_EMOTIONAL_WEIGHT_FAIL — cycle continues to the next phase.

Plumbing:
- CyclePhase type adds 'recompute_emotional_weight'.
- ALL_PHASES + NEEDS_LOCK_PHASES include it.
- CycleReport.totals adds pages_emotional_weight_recomputed (additive,
  schema_version stays "1").
- runCycle's totals rollup + status derivation honor the new field.
- synthesize.ts emits writtenSlugs in details so cycle.ts can union
  with syncPagesAffected for incremental backfill.

Tests: 7-case unit (fake-engine), 3-case PGLite e2e (full mode + dry-
run + ALL_PHASES position), 1000-page perf budget (<5s on PGLite).

Codex C2 → A: clean separation. Phase doesn't modify runExtractCore;
runs on its own seam after the existing 8 phases plus synthesize.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* v0.29 ops: get_recent_salience + find_anomalies + get_recent_transcripts

Three new MCP operations + a transcripts library:

- get_recent_salience: pages ranked by emotional + activity salience.
  Subagent-allow-listed. params: days (default 14), limit (default 20,
  capped 100), slugPrefix (renamed from `kind` per codex C4#10 to
  avoid collision with PageKind/TakeKind).

- find_anomalies: cohort-level activity outliers (tag + type).
  Subagent-allow-listed. Year cohort deferred to v0.30.

- get_recent_transcripts: raw .txt transcripts from the dream-cycle
  corpus dirs. LOCAL-ONLY: rejects ctx.remote === true with
  permission_denied (codex C3). NOT in the subagent allow-list — all
  subagent calls run with remote=true, would always reject (footgun if
  visible). Cycle's synthesize phase calls discoverTranscripts
  directly, so subagents that need transcripts go through the library
  function, not the op.

Tool descriptions extracted to src/core/operations-descriptions.ts so
they're pinnable in tests and stable for the Tier-2 LLM routing eval.
Redirects on query/search/list_pages: personal/emotional questions
should reach the new ops, not semantic search. Anti-flattery hint on
query: "Do NOT assume words like crazy, notable, or big mean
impressive — they often mean difficult or emotionally charged."

list_pages gains updated_after (string ISO) and sort enum params,
surfacing the engine threading from the prior commit.

src/core/transcripts.ts: filesystem walk shared by the gated MCP op
and the (commit 5) CLI command. Reuses discoverTranscripts corpus-dir
resolution + isDreamOutput from cycle/transcript-discovery.ts. Trust
gate lives in the op handler, not the library — the library is
trusted by both the gated op and the local CLI.

Allow-list: 11 → 13 (add salience + anomalies; transcripts excluded
per codex C3, with a comment explaining why).

Tests: 21-case description pin (catches accidental edits that change
LLM-facing surface); 11-case transcripts unit covering trust gate,
mtime window, dream-output skip, summary truncation, no corpus_dir;
2-case salience type-contract smoke (full Garry-test fixture in commit
6's e2e suite).

Codex C1: routing-eval fixtures (skills/<x>/routing-eval.jsonl)
deliberately NOT shipped — routing-eval.ts is substring-match on
resolver triggers, not MCP tool routing. Real coverage lands as
test/e2e/salience-llm-routing.test.ts in commit 6.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* v0.29 CLI: gbrain salience / anomalies / transcripts

Three new CLI commands wired into src/cli.ts dispatch + CLI_ONLY set +
help text:

- gbrain salience [--days N] [--limit N] [--kind PREFIX] [--json]
- gbrain anomalies [--since YYYY-MM-DD] [--lookback-days N] [--sigma N] [--json]
- gbrain transcripts recent [--days N] [--full] [--json]

Each command file mirrors src/commands/orphans.ts shape: pure data fn
+ JSON formatter + human formatter. Calls into engine.getRecentSalience
/ findAnomalies (already shipped) and src/core/transcripts.ts.

salience and anomalies show ranked rows with per-cohort
mean/stddev/sigma. transcripts honors `--full` (caps at 100KB/file)
vs default summary (first non-empty line + ~250 chars). All three
emit JSON with --json for agent consumption.

`--kind` is accepted as a slug-prefix shorthand on `gbrain salience`
even though the underlying op param is `slugPrefix` (kept the CLI
flag short; the MCP-facing param uses the more-explicit name to
align with PageKind/TakeKind/slugPrefix vocabulary).

CLI_ONLY set in src/cli.ts gains the three new command names so
they don't get forwarded to MCP-only routing.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* v0.29 e2e: Garry-test fixtures + Postgres parity + LLM routing eval

PGLite e2e (no DATABASE_URL needed):

- salience-pglite: the Garry test. 7 wedding-tagged pages updated today
  + 100 background pages backdated across 30 days via raw SQL UPDATE
  (codex C4#7 — engine.putPage stamps updated_at = now(), so seeding
  via the engine alone can't reproduce historical recency windows).
  Asserts wedding pages outrank random-tag noise in the 7-day window;
  slugPrefix filter narrows correctly; days=0 boundary case; limit cap.

- anomalies-pglite: same fixture shape (7 wedding pages today, 100
  background backdated). findAnomalies with sigma=3 returns the
  wedding-tag cohort with sigma_observed > 3 vs near-zero baseline;
  page_slugs sample carries the wedding pages; date with no activity
  returns []; high sigma threshold suppresses borderline cohorts
  (zero-stddev fallback stays finite — no NaN sigma).

Postgres-gated e2e:

- engine-parity-salience: PGLite ↔ Postgres parity for getRecentSalience
  and findAnomalies. Same fixture into both engines; top-result and
  cohort-set match. Closes the v0.22.0-style parity gap for the new
  v0.29 SQL idioms (EXTRACT(EPOCH ...), generate_series, CTE chain).

Tier-2 LLM routing eval (ANTHROPIC_API_KEY-gated):

- salience-llm-routing: calls Claude with v0.29 tool descriptions and
  12 personal-query phrasings ("anything crazy lately", "what's been
  going on with me", etc.). Asserts the chosen tool is in the v0.29
  set, not query() / search(). ~$0.10 per CI run on Haiku. Tests the
  ACTUAL ship criterion — replaces the discarded fake-coverage
  routing-eval.jsonl fixtures (codex C1 → B).

This is the only test that proves the description edits drive routing.
Without it, we'd ship description changes and only learn from
production behavior.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* v0.29.0: ship-prep — VERSION + CHANGELOG + CLAUDE Key Files

VERSION + package.json bump 0.28.0 → 0.29.0.

CHANGELOG.md adds a v0.29.0 release-summary in the GStack/Garry voice
plus the "To take advantage of v0.29.0" block. Headline two-liner:
"The brain tells you what's hot without being asked. Salience +
anomaly detection ship. Search rewards hypotheses; salience surfaces
them." Numbers-that-matter table covers engine surface delta, MCP op
delta, allow-list delta, cycle-phase delta, schema migration, list_pages
param surface, and test count. Itemized changes section lists the
schema migration + new cycle phase + new MCP ops + redirect
descriptions + subagent allow-list rules + new tests + a contributor
note clarifying that routing-eval.ts is not the right surface for
testing MCP tool routing (use the Tier-2 LLM eval pattern instead).

CLAUDE.md Key Files updated for the v0.29 surface:

- src/core/engine.ts: notes the 4 new methods + PageFilters.sort threading.
- src/core/migrate.ts: v34 (pages_emotional_weight) entry.
- src/core/cycle.ts: 8 → 9 phases, recompute_emotional_weight inserted
  between patterns and embed; totals.pages_emotional_weight_recomputed.
- src/core/cycle/emotional-weight.ts (NEW): formula + override path.
- src/core/cycle/anomaly.ts (NEW): stats helpers + zero-stddev fallback.
- src/core/cycle/recompute-emotional-weight.ts (NEW): phase orchestrator.
- src/core/transcripts.ts (NEW): library shared by gated MCP op + CLI.
- src/core/operations-descriptions.ts (NEW): pinned tool descriptions.
- src/core/minions/tools/brain-allowlist.ts: 11 → 13 entries; comment
  on why get_recent_transcripts is excluded.
- src/commands/salience.ts / anomalies.ts / transcripts.ts (NEW): CLI surface.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* v0.29.1 feat: recency + salience as two orthogonal options on query op (#696)

* feat: recency boost for search (v0.27.0) — temporal intent auto-detection, date filters, configurable decay

New search pipeline stage: keyword + vector → RRF → cosine re-score → backlink boost → recency boost → dedup

- applyRecencyBoost: hyperbolic decay, two strengths (moderate 30-day halflife, aggressive 7-day halflife)
- Auto-enabled when intent.ts detects temporal/event queries (detail='high')
- Manual override via SearchOpts.recencyBoost (0/1/2)
- Date filtering: afterDate/beforeDate on all three search paths (keyword, keywordChunks, vector)
- getPageTimestamps on both Postgres and PGLite engines
- 15 tests passing (boost math + intent classification)

* v0.29.1 schema: pages.{effective_date, effective_date_source, import_filename, salience_touched_at} + expression index

Migration v38 adds 4 nullable columns to pages and an expression index on
COALESCE(effective_date, updated_at) to support the new since/until date
filters. All additive — no behavior change in the default search path; only
consulted when callers opt into the new salience='on' / recency='on' axes
or pass since/until.

  effective_date         — content date (event_date / date / published /
                           filename-date / fallback). Read by recency boost
                           and date-filter paths only. Auto-link doesn't
                           touch it (immune to updated_at churn).
  effective_date_source  — sentinel for the doctor's effective_date_health
                           check ('event_date' | 'date' | 'published' |
                           'filename' | 'fallback').
  import_filename        — basename without extension, captured at import.
                           Used for filename-date precedence on daily/,
                           meetings/. Older rows leave it NULL.
  salience_touched_at    — bumped by recompute_emotional_weight when
                           emotional_weight changes. Salience window uses
                           GREATEST(updated_at, salience_touched_at) so
                           newly-salient old pages enter the recent salience
                           query.

Index strategy: a partial index on effective_date alone wouldn't help the
COALESCE expression in since/until filters (planner can't use it for the
negative side). The expression index ((COALESCE(effective_date, updated_at)))
is what actually accelerates the filter.

Postgres uses CONCURRENTLY + v14-style pg_index.indisvalid pre-drop guard
for prior failed CONCURRENTLY runs; PGLite uses plain CREATE INDEX. Mirror
of v34's pattern.

src/schema.sql + src/core/pglite-schema.ts updated for fresh installs;
src/core/schema-embedded.ts regenerated via bun run build:schema.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* v0.29.1: computeEffectiveDate helper + putPage integration

Pure helper computing a page's effective_date from frontmatter precedence:
  1. event_date (meeting/event pages)
  2. date (dated essays)
  3. published (writing/)
  4. filename-date (leading YYYY-MM-DD in basename)
  5. updated_at (fallback)
  6. created_at (last resort)

Per-prefix override: for daily/ and meetings/ slugs, filename-date jumps
to position 1 — the filename is the user's primary signal there.

Returns {date, source}. The source label powers the doctor's
effective_date_health check to detect "fell back to updated_at" rows that
look populated but are functionally a NULL.

Range validation: parsed value must be in [1990-01-01, NOW + 1 year].
Out-of-range values drop to the next chain element.

Wired into importFromContent + importFromFile. The put_page MCP op derives
filename from slug-tail when no caller-supplied filename is available.

putPage SQL on both engines extended to write the new columns. ON CONFLICT
uses COALESCE(EXCLUDED.x, pages.x) so callers that don't know about the
new columns (auto-link, code reindex) preserve existing values rather than
blanking them. SELECT projection extended to return them; rowToPage threads
them through.

21 unit tests covering: precedence chain default order, per-prefix override,
parse failure fall-through, range validation [1990, NOW+1y], parseDateLoose
shape variants. All pass; typecheck clean.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* v0.29.1: backfill orchestrator + library function for existing pages

src/core/backfill-effective-date.ts is the shared library function. Walks
pages in keyset-paginated batches (id > last_id ORDER BY id LIMIT 1000),
runs computeEffectiveDate per row, UPDATEs effective_date +
effective_date_source. Resumable via the `backfill.effective_date.last_id`
checkpoint key in the config table — a killed process can re-run and pick
up without re-doing rows. Idempotent: a full re-walk produces the same
writes.

Postgres-only: SET LOCAL statement_timeout = '600s' per batch. Doesn't
refuse the migration on low session settings (codex pass-2 #16).

src/commands/migrations/v0_29_1.ts is the orchestrator (4 phases mirroring
v0_12_2). Phase A schema (gbrain init --migrate-only), Phase B backfill
(via the library function), Phase C verify (count NULL effective_date),
Phase D record (handled by runner). The library function is reusable from
the gbrain reindex-frontmatter CLI command in the next commit.

import_filename stays NULL for backfilled rows — pre-v0.29.1 imports
didn't capture it. computeEffectiveDate uses the slug-tail when filename
is NULL; daily/2024-03-15 backfilled gets effective_date from the slug.

Registered in src/commands/migrations/index.ts.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* v0.29.1: gbrain reindex-frontmatter CLI command

Recovery / explicit-rebuild path for pages.effective_date. Used when:
  - User edited frontmatter dates after import
  - Post-upgrade backfill orchestrator finished but the user wants to
    re-walk a subset (e.g. just meetings/) after fixing some frontmatter
  - Precedence rules change between releases

Thin wrapper over backfillEffectiveDate from commit 3 — same code path
the v0_29_1 orchestrator uses; one source of truth.

Flags mirror reindex-code:
  --source <id>      Scope to one sources row (placeholder; library
                     library doesn't filter by source today, tracked v0.30+)
  --slug-prefix P    Scope to slugs starting with P (e.g. 'meetings/')
  --dry-run          Print what WOULD change, no DB writes
  --yes              Skip confirmation prompt (required for non-TTY non-JSON)
  --json             Machine-readable result envelope
  --force            Re-apply even when computed value matches existing

Wired into src/cli.ts. CLI handles its own engine lifecycle (creates +
disconnects).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* v0.29.1: recency-decay map + buildRecencyComponentSql (pure, unused)

src/core/search/recency-decay.ts mirrors source-boost.ts in shape but
drives RECENCY ONLY (per D9 codex resolution). Salience is a separate
orthogonal axis; this map does not feed it.

DEFAULT_RECENCY_DECAY: 10 generic prefixes (no fork-specific names).
  - concepts/      evergreen (halflifeDays=0)
  - originals/     180d × 0.5 (long-tail decay; new essays nudged)
  - writing/       365d × 0.4
  - daily/         14d × 1.5  (aggressive — freshness IS the signal)
  - meetings/      60d × 1.0
  - chat/          7d × 1.0
  - media/x/       7d × 1.5
  - media/articles/ 90d × 0.5
  - people/companies/ 365d × 0.3
  - deals/         180d × 0.5

DEFAULT_FALLBACK: 90d × 0.5 for unmatched slugs.

Override priority: defaults < gbrain.yml recency: < env (GBRAIN_RECENCY_DECAY)
< per-call SearchOpts.recency_decay.

parseRecencyDecayEnv format: comma-separated prefix:halflifeDays:coefficient
triples. Refuses LOUD on parse error (RecencyDecayParseError) — codex
pass-2 #M3 finding. No silent fallback like source-boost's parser.

parseRecencyDecayYaml takes already-parsed YAML; throws on bad shape.

buildRecencyComponentSql in sql-ranking.ts emits a CASE expression with
longest-prefix-first ordering, evergreen short-circuit (literal 0 when
halflifeDays=0 or coefficient=0), and EXTRACT(EPOCH ...) for non-zero
branches. Output: ((CASE WHEN p.slug LIKE 'daily/%' THEN 1.5 * 14.0 /
(14.0 + EXTRACT(EPOCH FROM (NOW() - <dateExpr>))/86400.0) ... END))

Typed NowExpr enum prevents SQL injection (codex pass-1 #5). Tests pass
{ kind: 'fixed', isoUtc } for deterministic output; production NOW().
The 'fixed' branch escapes single quotes via escapeSqlLiteral.

25 unit tests covering: env parser shape, env error cases, yaml parser
shape, merge precedence (defaults < yaml < env < caller), CASE longest-
prefix-first ordering, evergreen short-circuit, NowExpr fixed/now,
single-quote injection defense, empty decayMap fallback path, default
map composition (no fork names, concepts/ evergreen, daily/ aggressive).

Pure module. Zero consumers in this commit; commit 6 wires it into
getRecentSalience, commit 10 wires it into the post-fusion stage.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* v0.29.1: refactor getRecentSalience to consume buildRecencyComponentSql

Both engines (Postgres + PGLite) now build the salience formula's third
term via buildRecencyComponentSql instead of inlining 1.0 / (1 + days_old).
Parameters: empty decayMap + fallback { halflifeDays: 1, coefficient: 1.0 }.
Math expands to 1 * 1.0 / (1.0 + days_old) = 1 / (1 + days_old) — same
numeric output as v0.29.0.

This is a no-behavior-change refactor preparing for commit 7's recency_bias
param. recency_bias='flat' (default) reproduces v0.29.0 exactly; 'on'
swaps in DEFAULT_RECENCY_DECAY for per-prefix decay.

Single source of truth for the recency math: same builder feeds the
salience query AND (in commit 10) the post-fusion applyRecencyBoost stage.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* v0.29.1: get_recent_salience gains recency_bias param (default 'flat')

SalienceOpts.recency_bias: 'flat' | 'on' added; default 'flat' preserves
v0.29.0 ranking verbatim. Pass 'on' to opt into per-prefix decay map
(concepts/originals/writing/ evergreen; daily/, media/x/, chat/ aggressive
decay).

When recency_bias='on', the salience query reads
COALESCE(p.effective_date, p.updated_at) instead of bare p.updated_at, so
the recency component is immune to auto-link updated_at churn — old
concepts/ pages just-touched by auto-link don't suddenly look fresh.

Both engines (Postgres + PGLite) wire the param through. resolveRecencyDecayMap()
honors gbrain.yml + GBRAIN_RECENCY_DECAY env at runtime.

MCP op surface: get_recent_salience gains the param with a load-bearing
description teaching the agent when to use 'on' vs 'flat' (current state →
on; mattering across all time → flat).

No silent v0.29.0 behavior change — opt-in only (per D11 codex resolution).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* v0.29.1: recompute_emotional_weight writes salience_touched_at; window picks up newly-salient pages

setEmotionalWeightBatch on both engines now bumps salience_touched_at to
NOW() ONLY when the new emotional_weight differs from the existing one
(IS DISTINCT FROM, NULL-safe). No-op writes (same weight) leave the
column alone — preserves "actual change" semantics.

getRecentSalience window changes from
  WHERE p.updated_at >= boundary
to
  WHERE GREATEST(p.updated_at, COALESCE(p.salience_touched_at, p.updated_at)) >= boundary

Closes codex pass-1 finding #4: pages whose emotional_weight just changed
in the dream cycle (because tags or takes shifted) but whose updated_at
is older than the salience window now correctly enter the recent-salience
results. Without this, "Garry just added a take to a 6-month-old page"
stayed invisible to get_recent_salience until the next content edit.

COALESCE(salience_touched_at, p.updated_at) handles pre-v0.29.1 rows
where salience_touched_at is NULL — they fall back to p.updated_at and
behave identically to v0.29.0.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* v0.29.1: merge intent.ts → query-intent.ts; emit 3 suggestions per query

D1 + D4 + D6 + D8: single regex-pass classifier returning
{intent, suggestedDetail, suggestedSalience, suggestedRecency}.

intent + suggestedDetail are v0.29.0 behavior verbatim (legacy intent.ts
deleted; classifyQueryIntent + autoDetectDetail compat shims preserved).

NEW for v0.29.1 — two orthogonal recency-axis suggestions:

  suggestedSalience: 'off' | 'on' | 'strong'
  suggestedRecency:  'off' | 'on' | 'strong'

Resolution rules (per D6 narrow temporal-bound exception):
  - CANONICAL patterns (who is X / what is Y / code / graph) → both off
  - UNLESS an EXPLICIT_TEMPORAL_BOUND also matches (today / right now /
    this week / since X / last N days), in which case temporal-bound wins
  - STRONG_RECENCY (today / right now / this morning / just now) → strong
  - RECENCY_ON (latest / recent / this week / meeting prep / catch up
    / remind me / status update) → on
  - SALIENCE_ON (catch up / remind me / status update / prep me /
    what's going on / what matters) → on
  - default → off for both axes (v0.29.1 prime-directive: pure opt-in)

Salience and recency are TRULY orthogonal (per D9). A query like
"latest news on AI" → recency='on' but salience='off' (the user wants
fresh, not emotionally-weighted). "What's going on with widget-co" →
both on. "Who is X right now" → both 'strong'/'on' (temporal bound
beats canonical 'who is').

intent.ts deleted; test/intent.test.ts renamed → test/query-intent-legacy.test.ts
(unchanged behavior coverage). New test/query-intent.test.ts adds 21
cases covering all three axes' interactions: canonical wins on bare
'who is', temporal bound overrides, "catch me up" matches with up to 15
chars between, "today" → strong, intent vs recency independence.

Updated callers:
  - src/core/search/hybrid.ts (autoDetectDetail import)
  - test/recency-boost.test.ts (classifyQueryIntent import)
  - test/benchmark-search-quality.ts (autoDetectDetail import)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* v0.29.1: applySalienceBoost + applyRecencyBoost + runPostFusionStages wrapper

D9 + codex pass-1 #2 + #3 + pass-2 #4: salience and recency are TRULY
ORTHOGONAL post-fusion stages, both running from ALL THREE hybridSearch
return paths (keyword-only, embed-failure-fallback, full-hybrid).

NEW src/core/search/hybrid.ts exports:
  - applySalienceBoost(results, scores, strength)
      score *= 1 + k * log(1 + score) where k = 0.15 (on) or 0.30 (strong)
      No time component. Pure mattering signal.
  - applyRecencyBoost(results, dates, strength, decayMap, fallback, nowMs?)
      Per-prefix decay factor: 1 + strengthMul * coefficient * halflife / (halflife + days_old)
      strengthMul: 1.0 (on) or 1.5 (strong)
      Evergreen prefixes (halflifeDays=0) skipped (factor 1.0).
      Pure recency signal. Independent of mattering.
  - runPostFusionStages(engine, results, opts)
      Wraps backlink + salience + recency. Called from EACH return path so
      keyless installs and embed failures get the same boost surface as
      the full hybrid path.

NEW engine methods (composite-keyed for multi-source isolation):
  - getEffectiveDates(refs: Array<{slug, source_id}>): Map<key, Date>
      Returns COALESCE(effective_date, updated_at, created_at). Key format:
      `${source_id}::${slug}`. Mirror of getBacklinkCounts shape.
  - getSalienceScores(refs: Array<{slug, source_id}>): Map<key, number>
      Returns emotional_weight × 5 + ln(1 + take_count). Composite key.

Deprecated (kept for back-compat through v0.29.x):
  - SearchOpts.afterDate / beforeDate (alias for since/until)
  - SearchOpts.recencyBoost: 0|1|2 (alias for recency: 'off'|'on'|'strong')
  - getPageTimestamps (use getEffectiveDates instead)

NEW SearchOpts fields:
  - salience: 'off' | 'on' | 'strong'
  - recency:  'off' | 'on' | 'strong'
  - since:    string (ISO-8601 or relative, replaces afterDate)
  - until:    string (replaces beforeDate)

Resolution: caller-explicit > legacy alias (recencyBoost) > heuristic
(classifyQuery's suggestedSalience / suggestedRecency).

Deleted: src/core/search/recency.ts (PR #618's, replaced) +
test/recency-boost.test.ts (its scope is replaced by query-intent.test.ts +
future post-fusion tests).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: Wintermute <wintermute@garrytan.com>

* v0.29.1: query op gains salience + recency + since + until params; PGLite since/until parity

Combines commits 12 + 13 of the plan.

Query op surface (src/core/operations.ts):
  - salience: 'off' | 'on' | 'strong' (with load-bearing description)
  - recency:  'off' | 'on' | 'strong'
  - since:    string (ISO-8601 or relative; replaces deprecated afterDate)
  - until:    string (replaces deprecated beforeDate)

Tool descriptions teach the calling agent:
  - salience axis = mattering, no time component
  - recency axis = age decay, no mattering signal
  - omit either to let gbrain auto-detect from query text via classifyQuery

hybrid.ts maps since/until → afterDate/beforeDate at the engine call
boundary so PR #618's existing engine plumbing keeps working without
rename. Codex pass-1 #10 finding closed.

PGLite engine (codex pass-1 #10): since/until parity added to all three
search methods (searchKeyword, searchKeywordChunks, searchVector). SQL
filter against COALESCE(p.effective_date, p.updated_at, p.created_at)
so date filtering matches user content-date intent (a meeting was on
event_date, not when it got reimported). Filter is applied INSIDE the
HNSW inner CTE in searchVector so HNSW's candidate pool already
excludes out-of-range pages — preserves pagination contract.

This also closes existing cross-engine drift: pre-v0.29.1 Postgres had
afterDate/beforeDate from PR #618; PGLite had nothing.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* v0.29.1: migration v39 — eval_candidates capture columns for replay reproducibility

D11 codex pass-2 resolution: extend eval_candidates with 7 new nullable
columns so `gbrain eval replay` can reproduce captured runs of agent-explicit
salience + recency choices.

Without these columns, replays of the new axis params drift. The live
behavior depends on the resolved {salience, recency} values; v0.29.0's
schema doesn't capture them.

  as_of_ts            TIMESTAMPTZ  — brain's logical NOW at capture
                                     (replay uses this instead of wall-clock)
  salience_param      TEXT         — what the caller passed (NULL if omitted)
  recency_param       TEXT         — same
  salience_resolved   TEXT         — final value applied
  recency_resolved    TEXT         — same
  salience_source     TEXT         — 'caller' or 'auto_heuristic'
  recency_source      TEXT         — same

All nullable + additive. Pre-v0.29.1 rows stay valid. NDJSON
schema_version STAYS at 1 — consumers ignore unknown fields (codex
pass-1 #C2 dissolves; no cross-repo coordination needed).

ADD COLUMN with no DEFAULT is metadata-only on PG 11+ and PGLite —
instant on tables of any size.

src/schema.sql + src/core/pglite-schema.ts mirror the additions for fresh
installs; src/core/schema-embedded.ts regenerated. eval_capture.ts
populates the new fields in commit 16 (docs + ship).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* v0.29.1: doctor checks — effective_date_health + salience_health

effective_date_health: sample-1000 scan detects three classes of
problems (codex pass-1 #5 resolution via the effective_date_source
sentinel column added in commit 1):

  fallback_with_fm_date  — page fell back to updated_at even though
                           frontmatter has parseable event_date / date /
                           published. The "wrong but populated" residual
                           that earlier review iterations missed.
  future_dated            — effective_date > NOW() + 1 year (corrupt
                            or typo'd century).
  pre_1990                — effective_date < 1990-01-01 (epoch math gone
                            wrong, bad parse).

Sample of last 1000 pages by default — fast on 200K-page brains. Fix
hint: gbrain reindex-frontmatter.

salience_health: detects pages with active takes whose emotional_weight
is still 0 (recompute_emotional_weight phase hasn't run since the
take landed). Reports the brain's non-zero emotional_weight count as
an informational baseline. Fix hint: gbrain dream --phase
recompute_emotional_weight.

Both checks gracefully skip on pre-v0.29.1 brains (column doesn't
exist → 42703) without surfacing as warnings.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* v0.29.1: docs + skills convention + CHANGELOG + version bump

- VERSION 0.29.0 → 0.29.1
- package.json version bump
- CHANGELOG.md: full release-summary + itemized + "To take advantage"
  block per the project's voice rules. Two-line headline + concrete
  pathology framing (existing callers unchanged; new axes opt-in;
  agent in charge per the prime directive).
- skills/conventions/salience-and-recency.md: agent-readable decision
  rules. "Current state → on. Canonical truth → off." plus the narrow
  temporal-bound exception. Cross-cutting convention propagates to
  brain skills via RESOLVER.md.
- skills/migrations/v0.29.1.md: agent-readable upgrade instructions.
  Verify steps + behavior-change reference + recovery commands.

The build-time tool-description generator from D2 (extract decision
tables from skills/conventions/salience-and-recency.md, embed into
operations.ts at build time) is deferred to a follow-up commit. The
tool descriptions on the query op + get_recent_salience are inline in
operations.ts for v0.29.1; the auto-gen + CI staleness gate land in
v0.29.2 if drift becomes a problem in practice.

148 unit tests pass across the v0.29.1 surface (effective-date,
recency-decay, query-intent, migrate, salience, recompute-emotional-weight).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: Wintermute <wintermute@garrytan.com>

---------

Co-authored-by: Wintermute <wintermute@garrytan.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* v0.29 master-rebase fixups: renumber + drift cleanup

- v0.29.1 migrations renumber v38/v39 → v41/v42 (master shipped takes_table at
  v37 + access_tokens_permissions at v38; v0.27.1 took v39). My v0.29.0
  emotional_weight slots in at v40; v0.29.1's pages_recency_columns lands at
  v41 and eval_candidates_recency_capture at v42.
- src/core/utils.ts comment refs updated v37 → v40 (emotional_weight) and
  v38 → v41 (effective_date/etc).
- test/brain-allowlist.test.ts: size assertion 11 → 13 + the new
  get_recent_salience / find_anomalies positive checks + the explicit
  get_recent_transcripts negative check (v0.29 added the salience pair to
  the allow-list; transcripts are deliberately excluded because all
  subagent calls have remote=true and the v0.29 trust gate rejects them —
  visibility would be a footgun).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* v0.29 CI fixups: privacy allow-list + cycle phase count + migration plan

Three CI test failures on PR #730, all caused by master-side state the
v0.29 cherry-picks didn't yet account for:

1. scripts/check-privacy.sh allow-lists test/recency-decay.test.ts
   The v0.29.1 recency-decay test asserts that DEFAULT_RECENCY_DECAY's
   keys do NOT include fork-specific path prefixes. Because the assertion
   has to name the banned tokens to assert their absence, the privacy
   guard flagged the literal occurrence. Same exception class as
   CHANGELOG.md, CLAUDE.md, and scripts/check-privacy.sh itself —
   meta-rule enforcement requires mentioning what the rule forbids.

2. test/core/cycle.serial.test.ts: 9 → 10 phases.
   The yieldBetweenPhases test was written for v0.26.5 (9 phases incl.
   purge). v0.29 added a 10th phase (recompute_emotional_weight)
   between patterns and embed; the test's expected hookCalls and
   report.phases.length needed bumping.

3. test/apply-migrations.test.ts: append '0.29.1' to skippedFuture lists.
   v0.29.1 added a new entry to src/commands/migrations/index.ts; the
   buildPlan test snapshots the exact ordered list of versions, so it
   needs the new entry in both the fresh-install case and the Codex H9
   regression case.

All three verified locally:
  - bash scripts/check-privacy.sh → exit 0
  - bun test test/apply-migrations.test.ts → 18/18 pass
  - bun test test/core/cycle.serial.test.ts → 28/28 pass

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* v0.29 CI fixup: regenerate llms-full.txt to match CLAUDE.md state

build-llms test asserts the committed llms.txt + llms-full.txt match
what the generator produces from the current source tree. CLAUDE.md
got new v0.29 Key Files entries (recompute_emotional_weight phase,
emotional-weight formula, anomaly stats, transcripts library, salience
ops, etc.) without a corresponding regen. `bun run build:llms` brings
llms-full.txt back in sync; llms.txt is byte-for-byte identical so
only the larger inline bundle changed.

Verified locally: bun test test/build-llms.test.ts → 7/7 pass.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* v0.29 e2e: cover tool-surfaces + MCP dispatch path

Two gaps were uncovered when reviewing v0.29 coverage against the new
contracts the cherry-picks landed onto master.

1. test/v0_29-tool-surfaces.test.ts (unit, 9 cases)

   Existing tests pin the description constants module and the
   BRAIN_TOOL_ALLOWLIST set membership, but nothing checked the two
   filters that ACT on those constants:

   - serve-http.ts:745 filters operations by !op.localOnly to build the
     HTTP MCP tool list. Without a test, anyone removing `localOnly: true`
     from get_recent_transcripts would silently expose it to remote
     callers — defense-in-depth on top of the in-handler ctx.remote check
     would be the only guard. Now pinned: get_recent_transcripts is
     hidden, salience + anomalies stay visible.

   - buildBrainTools surfaces the v0.29 ops as `brain_get_recent_salience`
     and `brain_find_anomalies`, and EXCLUDES `brain_get_recent_transcripts`
     (codex C3 footgun gate — all subagent calls are remote=true, the op
     would always reject). Now pinned.

   Both filters are pure functions; no DB / engine.connect needed.

2. test/e2e/v0_29-mcp-dispatch-pglite.test.ts (e2e, 5 cases)

   Existing v0.29 e2e tests call engine methods directly. None went
   through the full dispatchToolCall pipeline that stdio MCP and HTTP
   MCP both use. The new file covers:

   - get_recent_salience returns ranked rows via dispatch (top result
     is the wedding-tagged page from the seeded fixture).
   - find_anomalies returns the AnomalyResult shape via dispatch.
   - get_recent_transcripts rejects with permission_denied when
     ctx.remote === true (the in-handler trust gate is the last line if
     localOnly ever drops).
   - get_recent_transcripts succeeds with ctx.remote === false (CLI
     path) and returns [] when no corpus dir is configured.
   - Unknown tool name returns the standard isError + "Unknown tool"
     envelope (regression guard for dispatch shape).

Verified locally — all 14 cases pass:
  bun test test/v0_29-tool-surfaces.test.ts                          → 9 pass
  bun test test/e2e/v0_29-mcp-dispatch-pglite.test.ts                → 5 pass

Re-ran the full v0.29 PGLite e2e suite to confirm no regressions:
  salience-pglite.test.ts                       5 pass
  anomalies-pglite.test.ts                      4 pass
  cycle-recompute-emotional-weight-pglite.test  3 pass
  list-pages-regression.test.ts                 6 pass
  multi-source-emotional-weight-pglite.test     4 pass
  backfill-perf-pglite.test.ts                  1 pass
  v0_29-mcp-dispatch-pglite.test.ts             5 pass
  -----
  Total: 28 pass / 0 fail
  Postgres parity test (DATABASE_URL gated)     7 skip (correct)
  LLM routing eval (ANTHROPIC_API_KEY gated)   12 skip (correct)
  bun run typecheck                             clean

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* v0.29 CI fixup: drop unused PGLiteEngine in tool-surfaces test

scripts/check-test-isolation.sh's R3 + R4 lints flagged the new
test/v0_29-tool-surfaces.test.ts for instantiating PGLiteEngine outside
a beforeAll() block (R3) and lacking the matching afterAll(disconnect)
(R4). The intent of those rules is to prevent engine leaks across the
shard process — every PGLiteEngine must follow the canonical
beforeAll(connect+initSchema) / afterAll(disconnect) pattern.

The fix here is upstream of the rule, not a workaround: this test never
needed an engine. buildBrainTools doesn't issue any SQL at registry-build
time — it only reads `engine.kind` for the put_page namespace-wrap
branch. A `{ kind: 'pglite' } as unknown as BrainEngine` fake-engine
literal keeps the test pure-function: no WASM cold-start, no connect
lifecycle, no test-isolation rule fired.

Verified locally:
  bash scripts/check-test-isolation.sh → OK (257 non-serial unit files)
  bun test test/v0_29-tool-surfaces.test.ts → 9 pass
  bun run typecheck → clean

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-authored-by: Wintermute <wintermute@garrytan.com>
This commit is contained in:
Garry Tan
2026-05-07 21:52:58 -07:00
committed by GitHub
co-authored by Claude Opus 4.7 Wintermute
parent bca993e09f
commit b8e0a0eada
67 changed files with 7120 additions and 196 deletions
+330
View File
@@ -2,6 +2,336 @@
All notable changes to GBrain will be documented in this file.
## [0.29.1] - 2026-05-05
**Recency and salience as two orthogonal options. Agent in charge.**
**Two ranking knobs, smart heuristic, no default behavior change for existing callers.**
v0.29 made the brain tell you what's hot. v0.29.1 lets the agent ask for
recency or salience independently — two orthogonal axes on the regular
`query` op, both opt-in, both with smart auto-detection from query text.
"What's going on with widget-co" auto-fires both. "Who is widget-ceo"
keeps both off. The agent overrides per query.
The two axes:
- **`salience: 'off' | 'on' | 'strong'`** — boost pages with high
`emotional_weight` + many active takes. NO time component. Use for
"what matters about X."
- **`recency: 'off' | 'on' | 'strong'`** — per-prefix age decay. NO
mattering signal. `concepts/`, `originals/`, `writing/` stay
evergreen; `daily/`, `media/x/`, `chat/` decay aggressively. Use for
"what's new on X."
Plus `since` / `until` date filters (replacing PR #618's `afterDate` /
`beforeDate` with proper PGLite parity), a new `pages.effective_date`
column populated from frontmatter precedence (immune to auto-link
`updated_at` churn), and `gbrain reindex-frontmatter` for explicit
recompute. Existing callers (no new params) get UNCHANGED behavior.
### What this means for you
A v0.29.0 caller upgrading to v0.29.1 with no code changes gets
identical query results. The new axes are pure opt-in. The agent
reads the new tool descriptions on every `tools/list` poll and learns
when to pass each value.
Pass `salience='on'` for meeting prep, conversation recall, "what's
going on with X." Pass `recency='on'` for "latest" / "this week" /
"recent updates." Pass `recency='strong'` for "today" / "right now."
Omit and gbrain auto-detects via the layered classifier in
`src/core/search/query-intent.ts` (canonical patterns win over
current-state EXCEPT when explicit temporal bounds like "today" /
"this week" / "since X" are present).
### Itemized changes
**Schema** (additive only, NDJSON schema_version stays at 1):
- Migration v38 adds 4 nullable columns to `pages`: `effective_date`,
`effective_date_source`, `import_filename`, `salience_touched_at`.
- Migration v39 adds 7 nullable columns to `eval_candidates` for
agent-explicit recency capture (replay reproducibility per D11).
- Expression index `pages_coalesce_date_idx` for `since`/`until` filters.
**Engine methods** (composite-keyed for multi-source isolation):
- `getEffectiveDates(refs)` returns `COALESCE(effective_date,
updated_at, created_at)`. Map keyed by `${source_id}::${slug}`.
- `getSalienceScores(refs)` returns `emotional_weight × 5 + ln(1 +
take_count)`. Same composite key.
**Search pipeline**:
- New `runPostFusionStages` wrapper consolidates backlink + salience +
recency. Called from ALL THREE `hybridSearch` return paths so
keyless installs and embed failures get the same boost surface.
- `applySalienceBoost` — pure mattering. `applyRecencyBoost` — pure
age decay. Truly orthogonal.
- `buildRecencyComponentSql` shared SQL builder with typed `NowExpr`
enum (no SQL injection).
**Query op**: gains `salience`, `recency`, `since`, `until` with
load-bearing tool descriptions. `get_recent_salience` gains
`recency_bias: 'flat' | 'on'` (default `'flat'` = v0.29.0 verbatim).
**Back-compat**: `afterDate`/`beforeDate`/`recencyBoost` from PR #618
remain as deprecated aliases. Stderr warning fires once per process.
Removed in v0.30.
**Heuristic**: `query-intent.ts` replaces `intent.ts`. Single regex
pass returning `{intent, suggestedDetail, suggestedSalience,
suggestedRecency}`. Canonical-wins + narrow temporal-bound exception.
English-only in v0.29.1.
**Doctor**: `effective_date_health` + `salience_health` checks. Both
gracefully skip on pre-v0.29.1 brains.
**CLI**: `gbrain reindex-frontmatter` — recovery / explicit-rebuild
path mirroring `gbrain reindex-code`.
**Tests**: `test/effective-date.test.ts` (21 cases),
`test/recency-decay.test.ts` (25 cases), `test/query-intent.test.ts`
(21 cases).
### To take advantage of v0.29.1
`gbrain upgrade` runs the full migration chain automatically. Verify:
1. **Confirm upgrade**:
```bash
gbrain --version # 0.29.1
```
2. **Recompute emotional weights** (one-time after upgrade):
```bash
gbrain dream --phase recompute_emotional_weight
```
3. **Verify health checks**:
```bash
gbrain doctor --json | jq '.checks[] | select(.name == "salience_health" or .name == "effective_date_health")'
```
4. **Try the new axes**:
```bash
gbrain query "what's been going on with X" --explain --json | jq '._resolved'
# expected: salience='on', recency='on'
gbrain query "who is X" --explain --json | jq '._resolved'
# expected: salience='off', recency='off'
```
5. **If anything looks wrong**`gbrain doctor --json` output and
`~/.gbrain/upgrade-errors.jsonl` (if present) on a Github issue:
https://github.com/garrytan/gbrain/issues
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: Wintermute <wintermute@garrytan.com>
## [0.29.0] - 2026-05-03
**The brain tells you what's hot without being asked.**
**Salience + anomaly detection ship. Search rewards hypotheses; salience surfaces them.**
Search rewards pre-formed hypotheses. To find the wedding cluster you
already had to know to type "wedding". v0.29 inverts that: three new MCP
ops surface what is unusual and emotionally charged in your brain without
needing a search term. Ask "anything crazy happening lately?" and the
agent reaches for `get_recent_salience` instead of running `query("crazy")`
and missing the cluster of pages all sharing one tag.
Three primitives. `get_recent_salience` ranks pages touched in the window
by a deterministic emotional + activity score (no LLM call). `find_anomalies`
detects cohort-level activity bursts vs a 30-day baseline densified with
`generate_series` zero-fill (so rare cohorts stop looking "normally
active"). `get_recent_transcripts` returns one-line summaries of the raw
`.txt` transcripts from the dream-cycle corpus dirs, gated to local CLI
only — MCP and HTTP cannot reach raw conversation text.
Plus a new dream-cycle phase, `recompute_emotional_weight`, that batches
weight computation in two SQL round-trips total — `WITH page_tags AS ...
WITH page_takes AS ...` so the page × N tags × M takes cartesian product
never happens; `UPDATE pages FROM unnest(...) USING (slug, source_id)` so
multi-source brains can't accidentally fan out across sources. 1000 pages
backfill in ~1.4s on PGLite; 50K pages should land under 60s on Postgres.
### The numbers that matter
Surface area added vs v0.28. Numbers from `git diff master..HEAD --stat`
on this branch:
| Surface | Before | After | Δ |
|---|---|---|---|
| Engine methods on BrainEngine | 50 | 54 | +4 (`batchLoadEmotionalInputs`, `setEmotionalWeightBatch`, `getRecentSalience`, `findAnomalies`) |
| MCP operations | 44 | 47 | +3 (`get_recent_salience`, `find_anomalies`, `get_recent_transcripts`) |
| Subagent tool allow-list | 11 | 13 | +2 (transcripts intentionally excluded — local-only via remote=false gate) |
| Cycle phases | 8 | 9 | +`recompute_emotional_weight` between extract/synthesize and embed |
| New SQL columns | — | 1 | `pages.emotional_weight REAL DEFAULT 0.0` (no index — score is computed) |
| Schema migrations | v39 | v40 | +1 (column-only, ADD COLUMN IF NOT EXISTS, instant) |
| `list_pages` params | 3 | 5 | +`updated_after`, +`sort` enum (engine ORDER BY threading; was hardcoded DESC) |
| New unit + e2e tests | — | 75+ | 14 emotional-weight, 13 anomalies, 8 transcripts, 21 descriptions, 7 phase, 5 salience-pglite, 4 anomalies-pglite, 4 multi-source, 3 cycle e2e, 6 list_pages, 1 perf, +12 LLM routing eval (Tier 2) |
**What this means for you:** ask "what's been going on with me?" and the
agent finds the cluster on the first tool call instead of the fifth.
A cluster of pages sharing a tag, all touched the same day, surface at the top of
`gbrain salience --days 7`. A burst of 15 pages tagged `family` shows up
in `gbrain anomalies` as a 3σ outlier vs a 0.3/day baseline. The brain
stops being a search engine and starts being an aide who notices.
## To take advantage of v0.29.0
`gbrain upgrade` should do this automatically. If it didn't, or if
`gbrain doctor` warns about an incomplete migration:
1. **Run the migration** (mechanical schema-only ALTER TABLE):
```bash
gbrain apply-migrations --yes
```
2. **Backfill emotional_weight on every page** (one-time, deterministic;
~1s per 1000 pages on PGLite, ~60s for 50K pages on Postgres):
```bash
gbrain dream --phase recompute_emotional_weight
```
3. **Try the Garry test** to verify routing changed:
```bash
gbrain salience --days 14
gbrain anomalies
gbrain transcripts recent --days 7
```
The salience output should rank pages with high-emotion tags (family,
wedding, loss, mental-health) above pages with the same recency but
no emotional content.
4. **(Optional) Tune the high-emotion tag list** if you keep a brain
that's mostly work-life. The default list is anglocentric +
personal-life-biased; override with the tags that drive *your*
emotional weight:
```bash
gbrain config set emotional_weight.high_tags '["family","health","grief","custom-tag"]'
gbrain dream --phase recompute_emotional_weight
```
Tag matching is case-insensitive. The override goes through the same
formula, just with your tag set.
5. **If any step fails or salience returns nothing,** file an issue:
https://github.com/garrytan/gbrain/issues with:
- output of `gbrain doctor`
- output of `gbrain salience --json --days 30`
- contents of `~/.gbrain/upgrade-errors.jsonl` if it exists
### Itemized changes
#### Schema (migration v40)
- **`pages.emotional_weight REAL NOT NULL DEFAULT 0.0`** — column-only,
no index. Default 0.0 so freshly imported pages don't pollute salience
ranking before the cycle phase populates real values.
- **No `idx_pages_emotional_weight`** — the salience query orders by a
computed score (`emotional_weight × 5 + ln(1+takes) + recency_decay`),
not raw weight. Adding the index later requires a separate migration.
#### New cycle phase: `recompute_emotional_weight`
- Runs **after** extract + synthesize, **before** embed/orphans. Sees fresh
tag + take state for every page touched in the cycle.
- Two SQL round-trips total regardless of brain size:
1. CTE-shaped `batchLoadEmotionalInputs` with per-table aggregates that
avoid the page × N tags × M takes cartesian product.
2. `setEmotionalWeightBatch` with `UPDATE FROM unnest($1::text[],
$2::text[], $3::real[])` keyed on `(slug, source_id)` so multi-source
brains can't fan out.
- Selectable via `gbrain dream --phase recompute_emotional_weight` for
targeted backfills (initial upgrade path).
- Incremental mode in routine cycles: union of `syncPagesAffected` +
`synthesizeWrittenSlugs`, so only the pages touched this cycle get
recomputed. Full mode walks every page.
#### New MCP ops + CLI
- **`get_recent_salience`** / `gbrain salience [--days N] [--limit N]
[--kind PREFIX] [--json]` — Pages ranked by `(emotional_weight × 5) +
ln(1 + active_take_count) + recency_decay`. Time boundary computed in
JS and bound as TIMESTAMPTZ so the SQL is identical across PGLite +
Postgres.
- **`find_anomalies`** / `gbrain anomalies [--since YYYY-MM-DD]
[--lookback-days N] [--sigma N] [--json]` — Cohort-level activity
outliers. Two cohort kinds in v1: tag, type. Year cohort deferred to
v0.30 pending proper frontmatter date detection. Baseline densified with
`generate_series` zero-fill so rare cohorts get correct `(mean, stddev)`
instead of biased upward by sparse-day omission. Zero-stddev fallback:
cohort fires when `count > mean + 1` (no NaN sigma).
- **`get_recent_transcripts`** / `gbrain transcripts recent [--days N]
[--full] [--json]` — Reads `.txt` files from `dream.synthesize.session_corpus_dir`
+ `dream.synthesize.meeting_transcripts_dir`. Skips dream-generated
outputs via `isDreamOutput` (v0.23.2 self-consumption guard). **Local-only:**
rejects `ctx.remote === true` callers with `permission_denied`. Not in
the subagent allow-list (subagent calls always run with `remote=true`,
so it would always reject — a footgun if visible).
#### Tool description redirects (zero-cost routing nudges)
- `query`, `search`, `list_pages` descriptions now redirect personal /
emotional / "what's recent" intents to the new ops. Descriptions
extracted to `src/core/operations-descriptions.ts` so they're pinnable
in tests.
- `query` description warns the LLM not to assume words like "crazy",
"notable", or "big" mean impressive — they often mean difficult or
emotionally charged.
- `list_pages` gains `updated_after` (string ISO) and `sort` enum
(`updated_desc | updated_asc | created_desc | slug`, default
`updated_desc`). Engines threaded — they previously hardcoded
`ORDER BY updated_at DESC`.
#### Subagent allow-list
- `get_recent_salience` + `find_anomalies` added (read-only, no
schema-shaping needed).
- `get_recent_transcripts` deliberately excluded — see codex C3 finding
reflected in `BRAIN_TOOL_ALLOWLIST` comments.
#### Tests
- 14 unit tests for the formula (`test/emotional-weight.test.ts`).
- 13 unit tests for the anomaly stats helpers + zero-stddev fallback
(`test/anomalies.test.ts`).
- 21 unit tests pinning the description constants + allow-list invariants
(`test/operations-descriptions.test.ts`).
- 7 unit tests for the cycle phase orchestration with a fake engine
(`test/recompute-emotional-weight.test.ts`).
- 8 unit tests for `listRecentTranscripts` covering trust gate, mtime
window, summary truncation, dream-output skip, no corpus_dir
(`test/transcripts.test.ts`).
- E2E PGLite (no DATABASE_URL needed):
- `salience-pglite.test.ts` (Garry test — 7 wedding pages outrank 100 random)
- `anomalies-pglite.test.ts` (cohort burst > 3σ vs zero baseline)
- `multi-source-emotional-weight-pglite.test.ts` (codex C4#3 regression
guard for `(slug, source_id)` composite key)
- `cycle-recompute-emotional-weight-pglite.test.ts` (phase wiring +
dry-run)
- `list-pages-regression.test.ts` (IRON RULE — old call shape still
works; new sort + updated_after threaded)
- `backfill-perf-pglite.test.ts` (1000-page fixture under 5s budget)
- E2E Postgres-gated:
- `engine-parity-salience.test.ts` (PGLite ↔ Postgres top-result and
cohort parity)
- Tier-2 LLM routing eval (`ANTHROPIC_API_KEY` gated):
- `salience-llm-routing.test.ts` — calls Claude with v0.29 tool
descriptions and 12 personal-query phrasings, asserts routing lands
in `{get_recent_salience, find_anomalies, get_recent_transcripts}`.
~$0.10/CI run. **Tests the actual ship criterion** — replaces the
discarded substring-match routing-eval fixtures (codex correctly
flagged those as fake coverage).
#### For contributors
- v0.23 routing-eval framework (`routing-eval.ts`) is **not** the right
surface for testing MCP tool-description routing. It's a substring
matcher over `skills/<name>/triggers:` frontmatter — useful for skill
resolver coverage, not LLM tool selection. Use the Tier-2 LLM eval
pattern in `test/e2e/salience-llm-routing.test.ts` for any future
feature whose value prop depends on the LLM choosing the right tool.
- All v0.29 SQL was reviewed for cross-engine parity. Where postgres.js
and PGLite handle parameter binding differently (e.g., `$1::interval`
vs computing the boundary in JS), v0.29 always picks the parity-safe
path. See engine-parity-salience.test.ts for the smoke.
## [0.28.12] - 2026-05-07
**gbrain hits 97.60% retrieval recall on the public LongMemEval benchmark.
+13 -5
View File
@@ -27,7 +27,7 @@ mount, CEO-class with multiple team brains) and
## Architecture
Contract-first: `src/core/operations.ts` defines ~41 shared operations (adds `find_orphans` in v0.12.3). CLI and MCP
Contract-first: `src/core/operations.ts` defines ~47 shared operations (v0.29 adds `get_recent_salience`, `find_anomalies`, `get_recent_transcripts`). CLI and MCP
server are both generated from this single source. Engine factory (`src/core/engine-factory.ts`)
dynamically imports the configured engine (`'pglite'` or `'postgres'`). Skills are fat
markdown files (tool-agnostic, work with both CLI and plugin contexts).
@@ -41,7 +41,7 @@ strict behavior when unset.
## Key files
- `src/core/operations.ts` — Contract-first operation definitions (the foundation). Also exports upload validators: `validateUploadPath`, `validatePageSlug`, `validateFilename`, plus `matchesSlugAllowList(slug, prefixes)` (v0.23 glob matcher: `<prefix>/*` matches recursive children; bare `<prefix>` matches exact only). `OperationContext.remote` flags untrusted callers; `OperationContext.allowedSlugPrefixes` (v0.23) is the trusted-workspace allow-list set by the dream cycle. `put_page` enforces: when `viaSubagent` and `allowedSlugPrefixes` is set, slug must match the allow-list; else the legacy `wiki/agents/<id>/...` namespace check applies. Auto-link enabled for trusted-workspace writes (skipped only when `remote=true && !trustedWorkspace`). As of v0.26.0, every `Operation` also carries `scope?: 'read' | 'write' | 'admin'` + `localOnly?: boolean`. All ops are annotated; `sync_brain`, `file_upload`, `file_list`, and `file_url` are `admin + localOnly` (rejected over HTTP). `OperationContext.auth?: AuthInfo` is threaded through HTTP dispatch for scope enforcement in `serve-http.ts` before the op runs. **v0.26.9 (D12 + F7b):** `OperationContext.remote` is now a REQUIRED field in the TypeScript type — the compiler is the first defense against transports that forget to set it. Four trust-boundary call sites (`put_page` allowlist, file_upload trust-narrowing, submit_job protected-name guard, auto-link skip) flipped from falsy-default (`!ctx.remote`) to fail-closed semantics (`ctx.remote === false` for "trusted-only" sites and `ctx.remote !== false` for "untrust unless explicit-false"). Anything that isn't strictly `false` is now treated as remote. Closed an HTTP MCP shell-job RCE: a `read+write`-scoped OAuth token could submit `shell` jobs because the HTTP request handler's literal context skipped `remote: true` and `submit_job`'s protected-name guard saw a falsy undefined. Stdio MCP set the field correctly via dispatch.ts; HTTP inlined a parallel context-builder for several releases and lost it.
- `src/core/engine.ts` — Pluggable engine interface (BrainEngine). `clampSearchLimit(limit, default, cap)` takes an explicit cap so per-operation caps can be tighter than `MAX_SEARCH_LIMIT`. Exports `LinkBatchInput` / `TimelineBatchInput` for the v0.12.1 bulk-insert API (`addLinksBatch` / `addTimelineEntriesBatch`). As of v0.13.1, `BrainEngine` has a `readonly kind: 'postgres' | 'pglite'` discriminator so migrations (`src/core/migrate.ts`) and other consumers can branch on engine without `instanceof` + dynamic imports.
- `src/core/engine.ts` — Pluggable engine interface (BrainEngine). `clampSearchLimit(limit, default, cap)` takes an explicit cap so per-operation caps can be tighter than `MAX_SEARCH_LIMIT`. Exports `LinkBatchInput` / `TimelineBatchInput` for the v0.12.1 bulk-insert API (`addLinksBatch` / `addTimelineEntriesBatch`). As of v0.13.1, `BrainEngine` has a `readonly kind: 'postgres' | 'pglite'` discriminator so migrations (`src/core/migrate.ts`) and other consumers can branch on engine without `instanceof` + dynamic imports. **v0.29:** four new methods — `batchLoadEmotionalInputs(slugs?)` (CTE-shaped read with per-table aggregates so a page × N tags × M takes never produces N×M rows), `setEmotionalWeightBatch(rows)` (`UPDATE FROM unnest($1::text[], $2::text[], $3::real[])` composite-keyed on `(slug, source_id)` for multi-source safety), `getRecentSalience(opts)`, `findAnomalies(opts)`. `PageFilters` extended with `sort?: 'updated_desc' | 'updated_asc' | 'created_desc' | 'slug'` + `PAGE_SORT_SQL` whitelist consumed by both engines (was hardcoded `ORDER BY updated_at DESC`).
- `src/core/engine-factory.ts` — Engine factory with dynamic imports (`'pglite'` | `'postgres'`)
- `src/core/pglite-engine.ts` — PGLite (embedded Postgres 17.5 via WASM) implementation, all 40 BrainEngine methods. `addLinksBatch` / `addTimelineEntriesBatch` use multi-row `unnest()` with manual `$N` placeholders. As of v0.13.1, `connect()` wraps `PGlite.create()` in a try/catch that emits an actionable error naming the macOS 26.3 WASM bug (#223) and pointing at `gbrain doctor`; the lock is released on failure so the next process can retry cleanly. v0.22.0: `searchKeyword` and `searchKeywordChunks` multiply `ts_rank` by the source-factor CASE expression at the chunk-grain level; `searchVector` becomes a two-stage CTE — inner CTE keeps `ORDER BY cc.embedding <=> vec` so HNSW stays usable, outer SELECT re-ranks by `raw_score * source_factor`. Inner LIMIT scales with offset to preserve pagination contract. As of v0.22.6.1, `initSchema()` calls `applyForwardReferenceBootstrap()` BEFORE replaying SCHEMA_SQL — probes for the specific forward-referenced state the embedded schema blob needs (`pages.source_id`, `links.link_source`, `links.origin_page_id`, `content_chunks.symbol_name`, `content_chunks.language`, `sources` FK target table) and adds only what's missing. Closes the upgrade-wedge bug class that bit users 10+ times across 6 schema versions over 2 years (#239/#243/#266/#357/#366/#374/#375/#378/#395/#396). No-op on fresh installs and modern brains.
- `src/core/pglite-schema.ts` — PGLite-specific DDL (pgvector, pg_trgm, triggers)
@@ -129,7 +129,7 @@ strict behavior when unset.
- `src/core/minions/wait-for-completion.ts` (v0.15) — poll-until-terminal helper for CLI callers. `TimeoutError` does NOT cancel the job; `AbortSignal` exits without throwing. Default `pollMs`: 1000 on Postgres, 250 on PGLite inline.
- `src/core/minions/transcript.ts` (v0.15) — renders `subagent_messages` + `subagent_tool_executions` to markdown. Tool rows splice under their owning assistant `tool_use` by `tool_use_id`. UTF-8-safe truncation; unknown block types fall through to fenced JSON.
- `src/core/minions/plugin-loader.ts` (v0.15) — `GBRAIN_PLUGIN_PATH` discovery. Absolute paths only, left-wins collision, `gbrain.plugin.json` with `plugin_version: "gbrain-plugin-v1"`, plugins ship DEFS only (no new tools), `allowed_tools:` validated at load time against the derived registry.
- `src/core/minions/tools/brain-allowlist.ts` (v0.15, extended v0.23) — derives subagent tool registry from `src/core/operations.ts`. 11-name allow-list. By default `put_page` schema is namespace-wrapped per subagent (`^wiki/agents/<subagentId>/.+`). **v0.23 trusted-workspace path:** when `BuildBrainToolsOpts.allowedSlugPrefixes` is set, the put_page schema instead describes the prefix list to the model and the OperationContext is threaded with `allowedSlugPrefixes`. Trust comes from `PROTECTED_JOB_NAMES` gating subagent submission — MCP cannot reach this field. Only cycle.ts (synthesize/patterns) and direct CLI submitters set it.
- `src/core/minions/tools/brain-allowlist.ts` (v0.15, extended v0.23, v0.29) — derives subagent tool registry from `src/core/operations.ts`. 13-name allow-list as of v0.29 (was 11). By default `put_page` schema is namespace-wrapped per subagent (`^wiki/agents/<subagentId>/.+`). **v0.23 trusted-workspace path:** when `BuildBrainToolsOpts.allowedSlugPrefixes` is set, the put_page schema instead describes the prefix list to the model and the OperationContext is threaded with `allowedSlugPrefixes`. Trust comes from `PROTECTED_JOB_NAMES` gating subagent submission — MCP cannot reach this field. Only cycle.ts (synthesize/patterns) and direct CLI submitters set it. **v0.29:** `get_recent_salience` + `find_anomalies` added to the allow-list. `get_recent_transcripts` deliberately NOT added — all subagent calls run with `ctx.remote === true`, and the v0.29 trust gate rejects remote callers, so adding it would always reject (footgun). The cycle synthesize phase already calls `discoverTranscripts` directly.
- `src/mcp/tool-defs.ts` (v0.15) — extracted `buildToolDefs(ops)` helper. MCP server + subagent tool registry both call it; byte-for-byte equivalence pinned by `test/mcp-tool-defs.test.ts`.
- `src/core/minions/attachments.ts` — Attachment validation (path traversal, null byte, oversize, base64, duplicate detection)
- `src/commands/agent.ts` (v0.16) — `gbrain agent run <prompt> [flags]` CLI. Submits `subagent` (or N children + 1 aggregator) under `{allowProtectedSubmit: true}`. Single-entry `--fanout-manifest` short-circuits. Children get `on_child_fail: 'continue'` + `max_stalled: 3`. `--follow` is the default on TTY; streams logs + polls `waitForCompletion` in parallel. Ctrl-C detaches, does not cancel.
@@ -148,17 +148,25 @@ strict behavior when unset.
- `src/commands/migrations/` — TS migration registry (compiled into the binary; no filesystem walk of `skills/migrations/*.md` needed at runtime). `index.ts` lists migrations in semver order. `v0_11_0.ts` = Minions adoption orchestrator (8 phases). `v0_12_0.ts` = Knowledge Graph auto-wire orchestrator (5 phases: schema → config check → backfill links → backfill timeline → verify). `phaseASchema` has a 600s timeout (bumped from 60s in v0.12.1 for duplicate-heavy brains). `v0_12_2.ts` = JSONB double-encode repair orchestrator (4 phases: schema → repair-jsonb → verify → record). `v0_14_0.ts` = shell-jobs + autopilot cooperative (2 phases: schema ALTER minion_jobs.max_stalled SET DEFAULT 3 — superseded by v0.14.3's schema-level DEFAULT 5 + UPDATE backfill; pending-host-work ping for skills/migrations/v0.14.0.md). All orchestrators are idempotent and resumable from `partial` status. As of v0.14.2 (Bug 3), the RUNNER owns all ledger writes — orchestrators return `OrchestratorResult` and `apply-migrations.ts` persists a canonical `{version, status, phases}` shape after return. Orchestrators no longer call `appendCompletedMigration` directly. `statusForVersion` prefers `complete` over `partial` (never regresses). 3 consecutive partials → wedged → `--force-retry <version>` writes a `'retry'` reset marker. v0.14.3 (fix wave) ships schema-only migrations v14 (`pages_updated_at_index`) + v15 (`minion_jobs_max_stalled_default_5` with UPDATE backfill) via the `MIGRATIONS` array in `src/core/migrate.ts` — no orchestrator phases needed.
- `src/commands/repair-jsonb.ts``gbrain repair-jsonb [--dry-run] [--json]`: rewrites `jsonb_typeof='string'` rows in place across 5 affected columns (pages.frontmatter, raw_data.data, ingest_log.pages_updated, files.metadata, page_versions.frontmatter). Fixes v0.12.0 double-encode bug on Postgres; PGLite no-ops. Idempotent.
- `src/commands/orphans.ts``gbrain orphans [--json] [--count] [--include-pseudo]`: surfaces pages with zero inbound wikilinks, grouped by domain. Auto-generated/raw/pseudo pages filtered by default. Also exposed as `find_orphans` MCP operation. Shipped in v0.12.3 (contributed by @knee5).
- `src/commands/salience.ts` (v0.29) — `gbrain salience [--days N] [--limit N] [--kind PREFIX] [--json]`: pages ranked by emotional + activity salience over a recency window. Mirrors orphans.ts shape (pure data fn + JSON formatter + human formatter). Calls `engine.getRecentSalience(opts)`. Score formula: `(emotional_weight × 5) + ln(1 + active_take_count) + 1/(1 + days_since_update)`.
- `src/commands/anomalies.ts` (v0.29) — `gbrain anomalies [--since YYYY-MM-DD] [--lookback-days N] [--sigma N] [--json]`: cohort-level activity outliers. Calls `engine.findAnomalies(opts)`. Two cohort kinds in v1: tag, type. Year cohort deferred to v0.30.
- `src/commands/transcripts.ts` (v0.29) — `gbrain transcripts recent [--days N] [--full] [--json]`: recent raw `.txt` transcripts from the dream-cycle corpus dirs. Imports `listRecentTranscripts` from `src/core/transcripts.ts` (the same library the gated `get_recent_transcripts` MCP op uses). Local-only by construction — the CLI always runs with `ctx.remote=false`.
- `src/commands/integrity.ts``gbrain integrity check|auto|review|extract`: bare-tweet detection, dead-link detection, three-bucket repair (auto-repair / review-queue / skip). `scanIntegrity()` is the shared library function called from `gbrain doctor` (sampled at limit=500) and `cmdCheck` (full scan). v0.22.8: batch-load fast path on Postgres uses `SELECT DISTINCT ON (slug)` in a single SQL query to fix the PgBouncer round-trip timeout (60s → ~6s) while preserving `engine.getAllSlugs()`'s `Set<string>` semantics on multi-source brains. Gated by `engine.kind === 'postgres'` at the call site so PGLite never enters batch; fallback `catch` logs at `GBRAIN_DEBUG=1` so real Postgres errors are diagnosable.
- `src/commands/doctor.ts``gbrain doctor [--json] [--fast] [--fix] [--dry-run] [--index-audit]`: health checks. v0.12.3 added `jsonb_integrity` + `markdown_body_completeness` reliability checks. v0.14.1: `--fix` delegates inlined cross-cutting rules to `> **Convention:** see [path](path).` callouts (pipes DRY violations into `src/core/dry-fix.ts`); `--fix --dry-run` previews without writing. v0.14.2: `schema_version` check fails loudly when `version=0` (migrations never ran — the #218 `bun install -g` signature) and routes users to `gbrain apply-migrations --yes`; new opt-in `--index-audit` flag (Postgres-only) reports zero-scan indexes from `pg_stat_user_indexes` (informational only, no auto-drop). v0.15.2: every DB check is wrapped in a progress phase; `markdown_body_completeness` runs under a 1s heartbeat timer so 10+ min scans are observable on 50K-page brains. v0.19.1 added `queue_health` (Postgres-only) with two subchecks: stalled-forever active jobs (started_at > 1h) and waiting-depth-per-name > threshold (default 10, override via `GBRAIN_QUEUE_WAITING_THRESHOLD`). Worker-heartbeat subcheck intentionally deferred to follow-up B7 because it needs a `minion_workers` table to produce ground-truth signal. Fix hints point at `gbrain repair-jsonb`, `gbrain sync --force`, `gbrain apply-migrations`, and `gbrain jobs get/cancel <id>`. v0.22.12 (#500): `sync_failures` check shows `[CODE=N, ...]` breakdown for both unacked entries (warn) and acked-historical entries (ok), surfacing systemic failure modes (`SLUG_MISMATCH=2685`) instead of a bare count. v0.26.7 (#612): `rls_event_trigger` check (post-install drift detector for migration v35's auto-RLS event trigger). Lives outside the `// 5. RLS` slice that the structural doctor.test.ts guards anchor on, so the existing test guards stay intact. Healthy `evtenabled` set is `('O','A')` only — `R` is replica-only and would not fire in normal sessions; `D` is disabled. Fix hint is `gbrain apply-migrations --force-retry 35`.
- `src/core/migrate.ts` — schema-migration runner. Owns the `MIGRATIONS` array (source of truth for schema DDL). v0.14.2 extended the `Migration` interface with `sqlFor?: { postgres?, pglite? }` (engine-specific SQL overrides `sql`) and `transaction?: boolean` (set to false for `CREATE INDEX CONCURRENTLY`, which Postgres refuses inside a transaction; ignored on PGLite since it has no concurrent writers). Migration v14 (fix wave) uses a handler branching on `engine.kind` to run CONCURRENTLY on Postgres (with a pre-drop of any invalid remnant via `pg_index.indisvalid`) and plain `CREATE INDEX` on PGLite. v15 bumps `minion_jobs.max_stalled` default 1→5 and backfills existing non-terminal rows. v0.22.6.1: migration v24 (`rls_backfill_missing_tables`) uses `sqlFor: { pglite: '' }` to no-op on PGLite — PGLite has no RLS engine and is single-tenant by definition, and the v24 ALTERs target subagent tables that don't exist in pglite-schema.ts. Closes #395 (contributed by @jdcastro2). **v30 (v0.23):** creates `dream_verdicts (file_path TEXT, content_hash TEXT, worth_processing BOOL, reasons JSONB, judged_at TIMESTAMPTZ, PK(file_path, content_hash))`. RLS-enabled when running as a BYPASSRLS role. The synthesize phase reads/writes this table to avoid re-judging on backfill re-runs. **v35 (v0.26.7):** auto-RLS event trigger + one-time backfill. `auto_rls_on_create_table` fires on `ddl_command_end` for `WHEN TAG IN ('CREATE TABLE','CREATE TABLE AS','SELECT INTO')` and runs `ALTER TABLE … ENABLE ROW LEVEL SECURITY` on every new `public.*` table — no FORCE (matches v24/v29/schema.sql posture so non-BYPASSRLS apps can still read their own tables). The same migration backfills RLS on every existing `public.*` base table whose comment doesn't match the doctor regex (`^GBRAIN:RLS_EXEMPT\s+reason=\S.{3,}`). Per-table failure aborts the offending CREATE TABLE (event triggers fire inside the DDL transaction); no EXCEPTION wrap — that would convert loud rollback into silent permissive default. PGLite no-op via `sqlFor.pglite: ''`. Breaking change: operators with intentionally-RLS-off public tables must add the GBRAIN:RLS_EXEMPT comment BEFORE upgrade or the backfill will flip them on.
- `src/core/migrate.ts` — schema-migration runner. Owns the `MIGRATIONS` array (source of truth for schema DDL). **v40 (v0.29):** `pages_emotional_weight` adds `pages.emotional_weight REAL NOT NULL DEFAULT 0.0`. Column-only (no index). On Postgres 11+ and PGLite, `ADD COLUMN` with a constant DEFAULT is metadata-only — instant on tables of any size. v0.14.2 extended the `Migration` interface with `sqlFor?: { postgres?, pglite? }` (engine-specific SQL overrides `sql`) and `transaction?: boolean` (set to false for `CREATE INDEX CONCURRENTLY`, which Postgres refuses inside a transaction; ignored on PGLite since it has no concurrent writers). Migration v14 (fix wave) uses a handler branching on `engine.kind` to run CONCURRENTLY on Postgres (with a pre-drop of any invalid remnant via `pg_index.indisvalid`) and plain `CREATE INDEX` on PGLite. v15 bumps `minion_jobs.max_stalled` default 1→5 and backfills existing non-terminal rows. v0.22.6.1: migration v24 (`rls_backfill_missing_tables`) uses `sqlFor: { pglite: '' }` to no-op on PGLite — PGLite has no RLS engine and is single-tenant by definition, and the v24 ALTERs target subagent tables that don't exist in pglite-schema.ts. Closes #395 (contributed by @jdcastro2). **v30 (v0.23):** creates `dream_verdicts (file_path TEXT, content_hash TEXT, worth_processing BOOL, reasons JSONB, judged_at TIMESTAMPTZ, PK(file_path, content_hash))`. RLS-enabled when running as a BYPASSRLS role. The synthesize phase reads/writes this table to avoid re-judging on backfill re-runs. **v35 (v0.26.7):** auto-RLS event trigger + one-time backfill. `auto_rls_on_create_table` fires on `ddl_command_end` for `WHEN TAG IN ('CREATE TABLE','CREATE TABLE AS','SELECT INTO')` and runs `ALTER TABLE … ENABLE ROW LEVEL SECURITY` on every new `public.*` table — no FORCE (matches v24/v29/schema.sql posture so non-BYPASSRLS apps can still read their own tables). The same migration backfills RLS on every existing `public.*` base table whose comment doesn't match the doctor regex (`^GBRAIN:RLS_EXEMPT\s+reason=\S.{3,}`). Per-table failure aborts the offending CREATE TABLE (event triggers fire inside the DDL transaction); no EXCEPTION wrap — that would convert loud rollback into silent permissive default. PGLite no-op via `sqlFor.pglite: ''`. Breaking change: operators with intentionally-RLS-off public tables must add the GBRAIN:RLS_EXEMPT comment BEFORE upgrade or the backfill will flip them on.
- `src/core/progress.ts` — Shared bulk-action progress reporter. Writes to stderr. Modes: `auto` (TTY: `\r`-rewriting; non-TTY: plain lines), `human`, `json` (JSONL), `quiet`. Rate-gated by `minIntervalMs` and `minItems`. `startHeartbeat(reporter, note)` helper for single long queries. `child()` composes phase paths. Singleton SIGINT/SIGTERM coordinator emits `abort` events for every live phase. EPIPE defense on both sync throws and stream `'error'` events. Zero dependencies. Introduced in v0.15.2.
- `src/core/cli-options.ts` — Global CLI flag parser. `parseGlobalFlags(argv)` returns `{cliOpts, rest}` with `--quiet` / `--progress-json` / `--progress-interval=<ms>` stripped. `getCliOptions()` / `setCliOptions()` expose a module-level singleton so commands reach the resolved flags without parameter threading. `cliOptsToProgressOptions()` maps to reporter options. `childGlobalFlags()` returns the flag suffix to append to `execSync('gbrain ...')` calls in migration orchestrators. `OperationContext.cliOpts` extends shared-op dispatch for MCP callers.
- `src/core/db-lock.ts` (v0.22.13) — generic `tryAcquireDbLock(engine, lockId, ttlMinutes)` over the existing `gbrain_cycle_locks` table. Parameterized lock id so different scopes can nest cleanly: `gbrain-cycle` for the broad cycle (held by `cycle.ts`) and `gbrain-sync` (`SYNC_LOCK_ID` constant) for `performSync`'s narrower writer window. Same UPSERT-with-TTL semantics as the prior cycle-only helper, just generalized. Survives PgBouncer transaction pooling (unlike session-scoped `pg_try_advisory_lock`); crashed holders auto-release once their TTL expires.
- `src/core/sync-concurrency.ts` (v0.22.13) — single source of truth for the parallel-sync policy. Exports `autoConcurrency(engine, fileCount, override?)` (PGLite always serial; explicit override clamped to >=1; auto path returns `DEFAULT_PARALLEL_WORKERS=4` when `fileCount > AUTO_CONCURRENCY_FILE_THRESHOLD=100`), `shouldRunParallel(workers, fileCount, explicit)` (Q1: explicit `--workers` bypasses the >50-file floor), and `parseWorkers(s)` (rejects `'0'`, `'-3'`, `'foo'`, `'1.5'`, trailing chars — replaces the prior parseInt-with-no-validation in both `sync.ts` and `import.ts`). Used by `performSync`, `performFullSync`, `runImport`, and the Minion `sync` handler so the three sites can no longer drift.
- `src/commands/sync.ts``gbrain sync` CLI + the `performSync` / `performFullSync` library entrypoints (consumed by the autopilot cycle and the Minion sync handler). v0.22.13 (PR #490): `performSync` wraps its body in a `gbrain-sync` writer lock so two concurrent syncs (manual + autopilot, two terminals, two Conductor workspaces) cannot both write `last_commit` and let the last writer win. Head-drift gate after the import phase re-checks `git rev-parse HEAD`; if HEAD moved (someone ran `git checkout` / `git pull` mid-sync), the bookmark refuses to advance. Vanished files now record a failedFiles entry instead of silent-skip — the silent-skip-then-advance pathology that survived prior hardening passes is dead. Worker engines wrap in try/finally so disconnect always fires (panic-path leak fix). Both PGLite-detection sites use `engine.kind === 'pglite'`. CLI accepts `--workers N` (alias `--concurrency N`), validated via `parseWorkers`. Explicit `--workers` bypasses the auto-path file-count floor; auto path defers to `autoConcurrency()`. Banner moved to stderr.
- `src/core/cycle.ts` — v0.17 brain maintenance cycle primitive (extended to **8 phases in v0.23**). `runCycle(engine: BrainEngine | null, opts: CycleOpts): Promise<CycleReport>` composes phases in semantically-driven order: **lint → backlinks → sync → synthesize → extract → patterns → embed → orphans**. v0.23's `synthesize` phase runs after sync (cross-references see fresh brain) and before extract (auto-link materializes its writes); `patterns` runs after extract so it reads a fresh graph (codex finding #7 — subagent put_page sets `ctx.remote=true` and skips auto-link/timeline by default; extract is the canonical materialization). Three callers: `gbrain dream` CLI, `gbrain autopilot` daemon's inline path, and the Minions `autopilot-cycle` handler. Coordination via `gbrain_cycle_locks` DB table + `~/.gbrain/cycle.lock` file lock with PID-liveness for PGLite. `CycleReport.schema_version: "1"` is stable; totals additively grew in v0.23 (`transcripts_processed`, `synth_pages_written`, `patterns_written`). `yieldBetweenPhases` runs between phases. **v0.23 added `yieldDuringPhase`** for in-phase keepalive — synthesize/patterns call it during long waits to renew the cycle-lock TTL. Engine nullable; lock-skip on read-only phase selections. v0.22.1 (#403): `CycleOpts.signal?: AbortSignal` propagates the worker's abort signal; `checkAborted()` fires between every phase. v0.22.1 (#417): `runPhaseSync` returns `pagesAffected` via `SyncPhaseResult`; `runCycle` captures it and threads to `runPhaseExtract` as the 4th arg. v0.22.1 (Codex F2): `runPhaseSync` takes `willRunExtractPhase: boolean` and sets `noExtract: phases.includes('extract')` so `gbrain dream --phase sync` doesn't silently lose extraction. v0.22.5 (#475): `resolveSourceForDir(engine, brainDir)` threads `sourceId` to `performSync()` so sync reads the per-source `sources.last_commit` anchor instead of the drift-prone global `config.sync.last_commit` key.
- `src/core/cycle.ts` — v0.17 brain maintenance cycle primitive (extended to **9 phases in v0.29**). `runCycle(engine: BrainEngine | null, opts: CycleOpts): Promise<CycleReport>` composes phases in semantically-driven order: **lint → backlinks → sync → synthesize → extract → patterns → recompute_emotional_weight → embed → orphans**. v0.29 adds the `recompute_emotional_weight` phase between patterns and embed; it sees the union of `syncPagesAffected` + `synthesizeWrittenSlugs` for incremental mode, or all pages when neither anchor is set (full backfill via `gbrain dream --phase recompute_emotional_weight`). v0.29 also extends `CycleReport.totals` with `pages_emotional_weight_recomputed` (additive, schema_version stays "1"). v0.23's `synthesize` phase runs after sync (cross-references see fresh brain) and before extract (auto-link materializes its writes); `patterns` runs after extract so it reads a fresh graph (codex finding #7 — subagent put_page sets `ctx.remote=true` and skips auto-link/timeline by default; extract is the canonical materialization). Three callers: `gbrain dream` CLI, `gbrain autopilot` daemon's inline path, and the Minions `autopilot-cycle` handler. Coordination via `gbrain_cycle_locks` DB table + `~/.gbrain/cycle.lock` file lock with PID-liveness for PGLite. `CycleReport.schema_version: "1"` is stable; totals additively grew in v0.23 (`transcripts_processed`, `synth_pages_written`, `patterns_written`). `yieldBetweenPhases` runs between phases. **v0.23 added `yieldDuringPhase`** for in-phase keepalive — synthesize/patterns call it during long waits to renew the cycle-lock TTL. Engine nullable; lock-skip on read-only phase selections. v0.22.1 (#403): `CycleOpts.signal?: AbortSignal` propagates the worker's abort signal; `checkAborted()` fires between every phase. v0.22.1 (#417): `runPhaseSync` returns `pagesAffected` via `SyncPhaseResult`; `runCycle` captures it and threads to `runPhaseExtract` as the 4th arg. v0.22.1 (Codex F2): `runPhaseSync` takes `willRunExtractPhase: boolean` and sets `noExtract: phases.includes('extract')` so `gbrain dream --phase sync` doesn't silently lose extraction. v0.22.5 (#475): `resolveSourceForDir(engine, brainDir)` threads `sourceId` to `performSync()` so sync reads the per-source `sources.last_commit` anchor instead of the drift-prone global `config.sync.last_commit` key.
- `src/core/cycle/synthesize.ts` (v0.23) — Synthesize phase: conversation-transcript-to-brain pipeline. Reads from `dream.synthesize.session_corpus_dir`, runs cheap Haiku verdict (cached in `dream_verdicts`), then fans out one Sonnet subagent per worth-processing transcript with `allowed_slug_prefixes` (sourced from `skills/_brain-filing-rules.json` `dream_synthesize_paths.globs`). Orchestrator collects slugs from `subagent_tool_executions` (NOT `pages.updated_at` — codex finding #2) and reverse-renders DB → markdown via `serializeMarkdown`. Cooldown via `dream.synthesize.last_completion_ts`, written ONLY on success. Idempotency key `dream:synth:<file_path>:<content_hash>`. Auto-commit deferred to v1.1 (codex #5). `--dry-run` runs Haiku, skips Sonnet (codex #8). Subagent never gets fs-write access. **v0.23.2:** `renderPageToMarkdown` (now exported) stamps `dream_generated: true` and `dream_cycle_date` into every reverse-write's frontmatter; `writeSummaryPage` does the same on the dream-cycle summary index. The marker is the explicit identity surface checked by `isDreamOutput` in `transcript-discovery.ts` — replaces the v0.23.1 content-prefix heuristic that could miss real output (`serializeMarkdown` doesn't embed slugs in body) and false-positive on user transcripts citing brain pages. `judgeSignificance` and `JudgeClient` are exported; `judgeSignificance` accepts a `verdictModel` parameter (default `claude-haiku-4-5-20251001`) loaded from `dream.synthesize.verdict_model` via `loadSynthConfig`.
- `src/core/cycle/patterns.ts` (v0.23) — Patterns phase: cross-session theme detection over reflections within `dream.patterns.lookback_days` (default 30). Names a pattern only when ≥`dream.patterns.min_evidence` (default 3) reflections support it. Single Sonnet subagent; same allow-list path as synthesize. Runs AFTER `extract` so the graph is fresh.
- `src/core/cycle/emotional-weight.ts` (v0.29) — Pure function `computeEmotionalWeight({tags, takes}, {highEmotionTags?, userHolder?})`. Deterministic 0..1 score: tag-emotion boost (max 0.5, case-insensitive match against `HIGH_EMOTION_TAGS` seed list), take density (0.1/take, capped at 0.3), take avg weight (0..0.1), user-holder ratio (0..0.1 over active takes; default holder = 'garry'). Total clamped to [0..1]. Anglocentric / personal-life-biased seed list intentional; users override via config key `emotional_weight.high_tags` (JSON array). `userHolder` overridable via `emotional_weight.user_holder`.
- `src/core/cycle/anomaly.ts` (v0.29) — Pure stats helpers for `find_anomalies`. `meanStddev` returns sample stddev (n-1 denominator) and (0,0) for empty input. `computeAnomaliesFromBuckets(baseline, today, sigma, limit)` takes densified daily-count buckets + today's counts per cohort, returns `AnomalyResult[]`. Zero-stddev fallback: cohort fires when `count > mean + 1`, with `sigma_observed = count - mean` as a finite sort proxy (no NaN). Brand-new cohorts (no baseline) have `mean=0, stddev=0` so the fallback fires at count >= 2. Sorted by `sigma_observed` desc, top `limit` (default 20). `page_slugs` capped at 50 per cohort.
- `src/core/cycle/recompute-emotional-weight.ts` (v0.29) — Cycle phase orchestrator. Two SQL round-trips total: `engine.batchLoadEmotionalInputs(slugs?)``computeEmotionalWeight` (per-row pure function) → `engine.setEmotionalWeightBatch(rows)`. Reads config keys `emotional_weight.high_tags` (JSON array, falls back to default seed list on parse error) and `emotional_weight.user_holder`. Empty `affectedSlugs` array short-circuits with zero-work success. dry-run mode reports the would-write count without touching the DB. Engine throw bubbles into `status: 'fail'` with code `RECOMPUTE_EMOTIONAL_WEIGHT_FAIL` so the cycle continues.
- `src/core/transcripts.ts` (v0.29) — `listRecentTranscripts(engine, opts)` library reused by both the `gbrain transcripts recent` CLI and the `get_recent_transcripts` MCP op. Reads `dream.synthesize.session_corpus_dir` + `dream.synthesize.meeting_transcripts_dir` config keys (same as `discoverTranscripts`); walks for `.txt` files within `days`; applies `isDreamOutput` guard from `transcript-discovery.ts` (skips dream-generated files); returns `{path, date, mtime, length, summary}[]` sorted newest-first. Summary mode (default true) returns first non-empty line + ~250 trailing chars. Full mode caps at 100KB/file. Missing/non-existent corpus dirs return `[]`, not error. **Trust gate lives in the op handler, not here**: the op throws `permission_denied` for `ctx.remote === true`; this library is a trusted library function used by both the gated op and the local CLI.
- `src/core/operations-descriptions.ts` (v0.29) — Constants module for tool descriptions. Pinned via `test/operations-descriptions.test.ts`. Houses `GET_RECENT_SALIENCE_DESCRIPTION`, `FIND_ANOMALIES_DESCRIPTION`, `GET_RECENT_TRANSCRIPTS_DESCRIPTION` plus the redirect-edited `LIST_PAGES_DESCRIPTION`, `QUERY_DESCRIPTION`, `SEARCH_DESCRIPTION`. Stable surface for the Tier-2 LLM routing eval — extracting them keeps the test from binding to whatever was in `operations.ts` at test-run time.
- `src/core/cycle/transcript-discovery.ts` (v0.23) — Pure filesystem walk for synthesize. `discoverTranscripts(opts)` filters `.txt` files by date range, min_chars, and word-boundary regex `excludePatterns` (Q-3: `medical` matches "medical advice" but NOT "comedical"; power users may pass full regex). `readSingleTranscript(path)` is the `gbrain dream --input <file>` ad-hoc path. **v0.23.2 self-consumption guard:** `DREAM_OUTPUT_MARKER_RE` (anchored at frontmatter open `---\n`, optional BOM + CRLF tolerance, scans first 2000 chars for `dream_generated: true` with case-insensitive value and word boundary on `true`) drives `isDreamOutput(content, bypass=false)`. Both `discoverTranscripts` and `readSingleTranscript` skip matching files and emit a `[dream] skipped <basename>: dream_generated marker` stderr log (no more silent skips). `bypassGuard?: boolean` on `DiscoverOpts` and `readSingleTranscript`'s opts disables the guard for the explicit `--unsafe-bypass-dream-guard` escape hatch only — never auto-applied for `--input`. Replaces v0.23.1's `DREAM_OUTPUT_SLUGS` content-prefix list.
- `src/commands/dream.ts` — v0.17 `gbrain dream` CLI; ~80-line thin alias over `runCycle`. brainDir resolution requires explicit `--dir` OR `sync.repo_path` config. Flags: `--dry-run`, `--json`, `--phase <name>`, `--pull`, `--dir <path>`. **v0.23 added** `--input <file>` (ad-hoc transcript, implies `--phase synthesize`), `--date YYYY-MM-DD`, `--from <d> --to <d>` (backfill range). Conflict detection: `--input` + `--date` exits 2. ISO date validation. `--dry-run` runs Haiku significance verdict but skips Sonnet synthesis (codex finding #8 — NOT zero LLM calls). Exit code 1 on status=failed. **v0.23.2 added** `--unsafe-bypass-dream-guard` (long-form intentional, plumbed through `runCycle.synthBypassDreamGuard``SynthesizePhaseOpts.bypassDreamGuard``discoverTranscripts({bypassGuard})` and `readSingleTranscript({bypassGuard})`). Loud stderr warning fires at synthesize-phase entry when set. Never auto-applied for `--input` so any caller can't silently re-trigger the loop bug.
- `src/commands/friction.ts` + `src/core/friction.ts` (v0.23) — `gbrain friction {log,render,list,summary}` reporter. Append-only JSONL under `$GBRAIN_HOME/friction/<run-id>.jsonl`. Schema is a flat extension of `StructuredAgentError` (D20). Render groups by severity → phase, defaults to `--redact` for md output (strips `$HOME`/`$CWD` to placeholders so reports paste safely in PRs). Run-id resolves from `--run-id` > `$GBRAIN_FRICTION_RUN_ID` > `standalone.jsonl`. Skills the claw-test exercises gain a `_friction-protocol.md` callout so agents know when to log friction.
+1 -1
View File
@@ -1 +1 @@
0.28.12
0.29.1
+13 -5
View File
@@ -127,7 +127,7 @@ mount, CEO-class with multiple team brains) and
## Architecture
Contract-first: `src/core/operations.ts` defines ~41 shared operations (adds `find_orphans` in v0.12.3). CLI and MCP
Contract-first: `src/core/operations.ts` defines ~47 shared operations (v0.29 adds `get_recent_salience`, `find_anomalies`, `get_recent_transcripts`). CLI and MCP
server are both generated from this single source. Engine factory (`src/core/engine-factory.ts`)
dynamically imports the configured engine (`'pglite'` or `'postgres'`). Skills are fat
markdown files (tool-agnostic, work with both CLI and plugin contexts).
@@ -141,7 +141,7 @@ strict behavior when unset.
## Key files
- `src/core/operations.ts` — Contract-first operation definitions (the foundation). Also exports upload validators: `validateUploadPath`, `validatePageSlug`, `validateFilename`, plus `matchesSlugAllowList(slug, prefixes)` (v0.23 glob matcher: `<prefix>/*` matches recursive children; bare `<prefix>` matches exact only). `OperationContext.remote` flags untrusted callers; `OperationContext.allowedSlugPrefixes` (v0.23) is the trusted-workspace allow-list set by the dream cycle. `put_page` enforces: when `viaSubagent` and `allowedSlugPrefixes` is set, slug must match the allow-list; else the legacy `wiki/agents/<id>/...` namespace check applies. Auto-link enabled for trusted-workspace writes (skipped only when `remote=true && !trustedWorkspace`). As of v0.26.0, every `Operation` also carries `scope?: 'read' | 'write' | 'admin'` + `localOnly?: boolean`. All ops are annotated; `sync_brain`, `file_upload`, `file_list`, and `file_url` are `admin + localOnly` (rejected over HTTP). `OperationContext.auth?: AuthInfo` is threaded through HTTP dispatch for scope enforcement in `serve-http.ts` before the op runs. **v0.26.9 (D12 + F7b):** `OperationContext.remote` is now a REQUIRED field in the TypeScript type — the compiler is the first defense against transports that forget to set it. Four trust-boundary call sites (`put_page` allowlist, file_upload trust-narrowing, submit_job protected-name guard, auto-link skip) flipped from falsy-default (`!ctx.remote`) to fail-closed semantics (`ctx.remote === false` for "trusted-only" sites and `ctx.remote !== false` for "untrust unless explicit-false"). Anything that isn't strictly `false` is now treated as remote. Closed an HTTP MCP shell-job RCE: a `read+write`-scoped OAuth token could submit `shell` jobs because the HTTP request handler's literal context skipped `remote: true` and `submit_job`'s protected-name guard saw a falsy undefined. Stdio MCP set the field correctly via dispatch.ts; HTTP inlined a parallel context-builder for several releases and lost it.
- `src/core/engine.ts` — Pluggable engine interface (BrainEngine). `clampSearchLimit(limit, default, cap)` takes an explicit cap so per-operation caps can be tighter than `MAX_SEARCH_LIMIT`. Exports `LinkBatchInput` / `TimelineBatchInput` for the v0.12.1 bulk-insert API (`addLinksBatch` / `addTimelineEntriesBatch`). As of v0.13.1, `BrainEngine` has a `readonly kind: 'postgres' | 'pglite'` discriminator so migrations (`src/core/migrate.ts`) and other consumers can branch on engine without `instanceof` + dynamic imports.
- `src/core/engine.ts` — Pluggable engine interface (BrainEngine). `clampSearchLimit(limit, default, cap)` takes an explicit cap so per-operation caps can be tighter than `MAX_SEARCH_LIMIT`. Exports `LinkBatchInput` / `TimelineBatchInput` for the v0.12.1 bulk-insert API (`addLinksBatch` / `addTimelineEntriesBatch`). As of v0.13.1, `BrainEngine` has a `readonly kind: 'postgres' | 'pglite'` discriminator so migrations (`src/core/migrate.ts`) and other consumers can branch on engine without `instanceof` + dynamic imports. **v0.29:** four new methods — `batchLoadEmotionalInputs(slugs?)` (CTE-shaped read with per-table aggregates so a page × N tags × M takes never produces N×M rows), `setEmotionalWeightBatch(rows)` (`UPDATE FROM unnest($1::text[], $2::text[], $3::real[])` composite-keyed on `(slug, source_id)` for multi-source safety), `getRecentSalience(opts)`, `findAnomalies(opts)`. `PageFilters` extended with `sort?: 'updated_desc' | 'updated_asc' | 'created_desc' | 'slug'` + `PAGE_SORT_SQL` whitelist consumed by both engines (was hardcoded `ORDER BY updated_at DESC`).
- `src/core/engine-factory.ts` — Engine factory with dynamic imports (`'pglite'` | `'postgres'`)
- `src/core/pglite-engine.ts` — PGLite (embedded Postgres 17.5 via WASM) implementation, all 40 BrainEngine methods. `addLinksBatch` / `addTimelineEntriesBatch` use multi-row `unnest()` with manual `$N` placeholders. As of v0.13.1, `connect()` wraps `PGlite.create()` in a try/catch that emits an actionable error naming the macOS 26.3 WASM bug (#223) and pointing at `gbrain doctor`; the lock is released on failure so the next process can retry cleanly. v0.22.0: `searchKeyword` and `searchKeywordChunks` multiply `ts_rank` by the source-factor CASE expression at the chunk-grain level; `searchVector` becomes a two-stage CTE — inner CTE keeps `ORDER BY cc.embedding <=> vec` so HNSW stays usable, outer SELECT re-ranks by `raw_score * source_factor`. Inner LIMIT scales with offset to preserve pagination contract. As of v0.22.6.1, `initSchema()` calls `applyForwardReferenceBootstrap()` BEFORE replaying SCHEMA_SQL — probes for the specific forward-referenced state the embedded schema blob needs (`pages.source_id`, `links.link_source`, `links.origin_page_id`, `content_chunks.symbol_name`, `content_chunks.language`, `sources` FK target table) and adds only what's missing. Closes the upgrade-wedge bug class that bit users 10+ times across 6 schema versions over 2 years (#239/#243/#266/#357/#366/#374/#375/#378/#395/#396). No-op on fresh installs and modern brains.
- `src/core/pglite-schema.ts` — PGLite-specific DDL (pgvector, pg_trgm, triggers)
@@ -229,7 +229,7 @@ strict behavior when unset.
- `src/core/minions/wait-for-completion.ts` (v0.15) — poll-until-terminal helper for CLI callers. `TimeoutError` does NOT cancel the job; `AbortSignal` exits without throwing. Default `pollMs`: 1000 on Postgres, 250 on PGLite inline.
- `src/core/minions/transcript.ts` (v0.15) — renders `subagent_messages` + `subagent_tool_executions` to markdown. Tool rows splice under their owning assistant `tool_use` by `tool_use_id`. UTF-8-safe truncation; unknown block types fall through to fenced JSON.
- `src/core/minions/plugin-loader.ts` (v0.15) — `GBRAIN_PLUGIN_PATH` discovery. Absolute paths only, left-wins collision, `gbrain.plugin.json` with `plugin_version: "gbrain-plugin-v1"`, plugins ship DEFS only (no new tools), `allowed_tools:` validated at load time against the derived registry.
- `src/core/minions/tools/brain-allowlist.ts` (v0.15, extended v0.23) — derives subagent tool registry from `src/core/operations.ts`. 11-name allow-list. By default `put_page` schema is namespace-wrapped per subagent (`^wiki/agents/<subagentId>/.+`). **v0.23 trusted-workspace path:** when `BuildBrainToolsOpts.allowedSlugPrefixes` is set, the put_page schema instead describes the prefix list to the model and the OperationContext is threaded with `allowedSlugPrefixes`. Trust comes from `PROTECTED_JOB_NAMES` gating subagent submission — MCP cannot reach this field. Only cycle.ts (synthesize/patterns) and direct CLI submitters set it.
- `src/core/minions/tools/brain-allowlist.ts` (v0.15, extended v0.23, v0.29) — derives subagent tool registry from `src/core/operations.ts`. 13-name allow-list as of v0.29 (was 11). By default `put_page` schema is namespace-wrapped per subagent (`^wiki/agents/<subagentId>/.+`). **v0.23 trusted-workspace path:** when `BuildBrainToolsOpts.allowedSlugPrefixes` is set, the put_page schema instead describes the prefix list to the model and the OperationContext is threaded with `allowedSlugPrefixes`. Trust comes from `PROTECTED_JOB_NAMES` gating subagent submission — MCP cannot reach this field. Only cycle.ts (synthesize/patterns) and direct CLI submitters set it. **v0.29:** `get_recent_salience` + `find_anomalies` added to the allow-list. `get_recent_transcripts` deliberately NOT added — all subagent calls run with `ctx.remote === true`, and the v0.29 trust gate rejects remote callers, so adding it would always reject (footgun). The cycle synthesize phase already calls `discoverTranscripts` directly.
- `src/mcp/tool-defs.ts` (v0.15) — extracted `buildToolDefs(ops)` helper. MCP server + subagent tool registry both call it; byte-for-byte equivalence pinned by `test/mcp-tool-defs.test.ts`.
- `src/core/minions/attachments.ts` — Attachment validation (path traversal, null byte, oversize, base64, duplicate detection)
- `src/commands/agent.ts` (v0.16) — `gbrain agent run <prompt> [flags]` CLI. Submits `subagent` (or N children + 1 aggregator) under `{allowProtectedSubmit: true}`. Single-entry `--fanout-manifest` short-circuits. Children get `on_child_fail: 'continue'` + `max_stalled: 3`. `--follow` is the default on TTY; streams logs + polls `waitForCompletion` in parallel. Ctrl-C detaches, does not cancel.
@@ -248,17 +248,25 @@ strict behavior when unset.
- `src/commands/migrations/` — TS migration registry (compiled into the binary; no filesystem walk of `skills/migrations/*.md` needed at runtime). `index.ts` lists migrations in semver order. `v0_11_0.ts` = Minions adoption orchestrator (8 phases). `v0_12_0.ts` = Knowledge Graph auto-wire orchestrator (5 phases: schema → config check → backfill links → backfill timeline → verify). `phaseASchema` has a 600s timeout (bumped from 60s in v0.12.1 for duplicate-heavy brains). `v0_12_2.ts` = JSONB double-encode repair orchestrator (4 phases: schema → repair-jsonb → verify → record). `v0_14_0.ts` = shell-jobs + autopilot cooperative (2 phases: schema ALTER minion_jobs.max_stalled SET DEFAULT 3 — superseded by v0.14.3's schema-level DEFAULT 5 + UPDATE backfill; pending-host-work ping for skills/migrations/v0.14.0.md). All orchestrators are idempotent and resumable from `partial` status. As of v0.14.2 (Bug 3), the RUNNER owns all ledger writes — orchestrators return `OrchestratorResult` and `apply-migrations.ts` persists a canonical `{version, status, phases}` shape after return. Orchestrators no longer call `appendCompletedMigration` directly. `statusForVersion` prefers `complete` over `partial` (never regresses). 3 consecutive partials → wedged → `--force-retry <version>` writes a `'retry'` reset marker. v0.14.3 (fix wave) ships schema-only migrations v14 (`pages_updated_at_index`) + v15 (`minion_jobs_max_stalled_default_5` with UPDATE backfill) via the `MIGRATIONS` array in `src/core/migrate.ts` — no orchestrator phases needed.
- `src/commands/repair-jsonb.ts` — `gbrain repair-jsonb [--dry-run] [--json]`: rewrites `jsonb_typeof='string'` rows in place across 5 affected columns (pages.frontmatter, raw_data.data, ingest_log.pages_updated, files.metadata, page_versions.frontmatter). Fixes v0.12.0 double-encode bug on Postgres; PGLite no-ops. Idempotent.
- `src/commands/orphans.ts` — `gbrain orphans [--json] [--count] [--include-pseudo]`: surfaces pages with zero inbound wikilinks, grouped by domain. Auto-generated/raw/pseudo pages filtered by default. Also exposed as `find_orphans` MCP operation. Shipped in v0.12.3 (contributed by @knee5).
- `src/commands/salience.ts` (v0.29) — `gbrain salience [--days N] [--limit N] [--kind PREFIX] [--json]`: pages ranked by emotional + activity salience over a recency window. Mirrors orphans.ts shape (pure data fn + JSON formatter + human formatter). Calls `engine.getRecentSalience(opts)`. Score formula: `(emotional_weight × 5) + ln(1 + active_take_count) + 1/(1 + days_since_update)`.
- `src/commands/anomalies.ts` (v0.29) — `gbrain anomalies [--since YYYY-MM-DD] [--lookback-days N] [--sigma N] [--json]`: cohort-level activity outliers. Calls `engine.findAnomalies(opts)`. Two cohort kinds in v1: tag, type. Year cohort deferred to v0.30.
- `src/commands/transcripts.ts` (v0.29) — `gbrain transcripts recent [--days N] [--full] [--json]`: recent raw `.txt` transcripts from the dream-cycle corpus dirs. Imports `listRecentTranscripts` from `src/core/transcripts.ts` (the same library the gated `get_recent_transcripts` MCP op uses). Local-only by construction — the CLI always runs with `ctx.remote=false`.
- `src/commands/integrity.ts` — `gbrain integrity check|auto|review|extract`: bare-tweet detection, dead-link detection, three-bucket repair (auto-repair / review-queue / skip). `scanIntegrity()` is the shared library function called from `gbrain doctor` (sampled at limit=500) and `cmdCheck` (full scan). v0.22.8: batch-load fast path on Postgres uses `SELECT DISTINCT ON (slug)` in a single SQL query to fix the PgBouncer round-trip timeout (60s → ~6s) while preserving `engine.getAllSlugs()`'s `Set<string>` semantics on multi-source brains. Gated by `engine.kind === 'postgres'` at the call site so PGLite never enters batch; fallback `catch` logs at `GBRAIN_DEBUG=1` so real Postgres errors are diagnosable.
- `src/commands/doctor.ts` — `gbrain doctor [--json] [--fast] [--fix] [--dry-run] [--index-audit]`: health checks. v0.12.3 added `jsonb_integrity` + `markdown_body_completeness` reliability checks. v0.14.1: `--fix` delegates inlined cross-cutting rules to `> **Convention:** see [path](path).` callouts (pipes DRY violations into `src/core/dry-fix.ts`); `--fix --dry-run` previews without writing. v0.14.2: `schema_version` check fails loudly when `version=0` (migrations never ran — the #218 `bun install -g` signature) and routes users to `gbrain apply-migrations --yes`; new opt-in `--index-audit` flag (Postgres-only) reports zero-scan indexes from `pg_stat_user_indexes` (informational only, no auto-drop). v0.15.2: every DB check is wrapped in a progress phase; `markdown_body_completeness` runs under a 1s heartbeat timer so 10+ min scans are observable on 50K-page brains. v0.19.1 added `queue_health` (Postgres-only) with two subchecks: stalled-forever active jobs (started_at > 1h) and waiting-depth-per-name > threshold (default 10, override via `GBRAIN_QUEUE_WAITING_THRESHOLD`). Worker-heartbeat subcheck intentionally deferred to follow-up B7 because it needs a `minion_workers` table to produce ground-truth signal. Fix hints point at `gbrain repair-jsonb`, `gbrain sync --force`, `gbrain apply-migrations`, and `gbrain jobs get/cancel <id>`. v0.22.12 (#500): `sync_failures` check shows `[CODE=N, ...]` breakdown for both unacked entries (warn) and acked-historical entries (ok), surfacing systemic failure modes (`SLUG_MISMATCH=2685`) instead of a bare count. v0.26.7 (#612): `rls_event_trigger` check (post-install drift detector for migration v35's auto-RLS event trigger). Lives outside the `// 5. RLS` slice that the structural doctor.test.ts guards anchor on, so the existing test guards stay intact. Healthy `evtenabled` set is `('O','A')` only — `R` is replica-only and would not fire in normal sessions; `D` is disabled. Fix hint is `gbrain apply-migrations --force-retry 35`.
- `src/core/migrate.ts` — schema-migration runner. Owns the `MIGRATIONS` array (source of truth for schema DDL). v0.14.2 extended the `Migration` interface with `sqlFor?: { postgres?, pglite? }` (engine-specific SQL overrides `sql`) and `transaction?: boolean` (set to false for `CREATE INDEX CONCURRENTLY`, which Postgres refuses inside a transaction; ignored on PGLite since it has no concurrent writers). Migration v14 (fix wave) uses a handler branching on `engine.kind` to run CONCURRENTLY on Postgres (with a pre-drop of any invalid remnant via `pg_index.indisvalid`) and plain `CREATE INDEX` on PGLite. v15 bumps `minion_jobs.max_stalled` default 1→5 and backfills existing non-terminal rows. v0.22.6.1: migration v24 (`rls_backfill_missing_tables`) uses `sqlFor: { pglite: '' }` to no-op on PGLite — PGLite has no RLS engine and is single-tenant by definition, and the v24 ALTERs target subagent tables that don't exist in pglite-schema.ts. Closes #395 (contributed by @jdcastro2). **v30 (v0.23):** creates `dream_verdicts (file_path TEXT, content_hash TEXT, worth_processing BOOL, reasons JSONB, judged_at TIMESTAMPTZ, PK(file_path, content_hash))`. RLS-enabled when running as a BYPASSRLS role. The synthesize phase reads/writes this table to avoid re-judging on backfill re-runs. **v35 (v0.26.7):** auto-RLS event trigger + one-time backfill. `auto_rls_on_create_table` fires on `ddl_command_end` for `WHEN TAG IN ('CREATE TABLE','CREATE TABLE AS','SELECT INTO')` and runs `ALTER TABLE … ENABLE ROW LEVEL SECURITY` on every new `public.*` table — no FORCE (matches v24/v29/schema.sql posture so non-BYPASSRLS apps can still read their own tables). The same migration backfills RLS on every existing `public.*` base table whose comment doesn't match the doctor regex (`^GBRAIN:RLS_EXEMPT\s+reason=\S.{3,}`). Per-table failure aborts the offending CREATE TABLE (event triggers fire inside the DDL transaction); no EXCEPTION wrap — that would convert loud rollback into silent permissive default. PGLite no-op via `sqlFor.pglite: ''`. Breaking change: operators with intentionally-RLS-off public tables must add the GBRAIN:RLS_EXEMPT comment BEFORE upgrade or the backfill will flip them on.
- `src/core/migrate.ts` — schema-migration runner. Owns the `MIGRATIONS` array (source of truth for schema DDL). **v40 (v0.29):** `pages_emotional_weight` adds `pages.emotional_weight REAL NOT NULL DEFAULT 0.0`. Column-only (no index). On Postgres 11+ and PGLite, `ADD COLUMN` with a constant DEFAULT is metadata-only — instant on tables of any size. v0.14.2 extended the `Migration` interface with `sqlFor?: { postgres?, pglite? }` (engine-specific SQL overrides `sql`) and `transaction?: boolean` (set to false for `CREATE INDEX CONCURRENTLY`, which Postgres refuses inside a transaction; ignored on PGLite since it has no concurrent writers). Migration v14 (fix wave) uses a handler branching on `engine.kind` to run CONCURRENTLY on Postgres (with a pre-drop of any invalid remnant via `pg_index.indisvalid`) and plain `CREATE INDEX` on PGLite. v15 bumps `minion_jobs.max_stalled` default 1→5 and backfills existing non-terminal rows. v0.22.6.1: migration v24 (`rls_backfill_missing_tables`) uses `sqlFor: { pglite: '' }` to no-op on PGLite — PGLite has no RLS engine and is single-tenant by definition, and the v24 ALTERs target subagent tables that don't exist in pglite-schema.ts. Closes #395 (contributed by @jdcastro2). **v30 (v0.23):** creates `dream_verdicts (file_path TEXT, content_hash TEXT, worth_processing BOOL, reasons JSONB, judged_at TIMESTAMPTZ, PK(file_path, content_hash))`. RLS-enabled when running as a BYPASSRLS role. The synthesize phase reads/writes this table to avoid re-judging on backfill re-runs. **v35 (v0.26.7):** auto-RLS event trigger + one-time backfill. `auto_rls_on_create_table` fires on `ddl_command_end` for `WHEN TAG IN ('CREATE TABLE','CREATE TABLE AS','SELECT INTO')` and runs `ALTER TABLE … ENABLE ROW LEVEL SECURITY` on every new `public.*` table — no FORCE (matches v24/v29/schema.sql posture so non-BYPASSRLS apps can still read their own tables). The same migration backfills RLS on every existing `public.*` base table whose comment doesn't match the doctor regex (`^GBRAIN:RLS_EXEMPT\s+reason=\S.{3,}`). Per-table failure aborts the offending CREATE TABLE (event triggers fire inside the DDL transaction); no EXCEPTION wrap — that would convert loud rollback into silent permissive default. PGLite no-op via `sqlFor.pglite: ''`. Breaking change: operators with intentionally-RLS-off public tables must add the GBRAIN:RLS_EXEMPT comment BEFORE upgrade or the backfill will flip them on.
- `src/core/progress.ts` — Shared bulk-action progress reporter. Writes to stderr. Modes: `auto` (TTY: `\r`-rewriting; non-TTY: plain lines), `human`, `json` (JSONL), `quiet`. Rate-gated by `minIntervalMs` and `minItems`. `startHeartbeat(reporter, note)` helper for single long queries. `child()` composes phase paths. Singleton SIGINT/SIGTERM coordinator emits `abort` events for every live phase. EPIPE defense on both sync throws and stream `'error'` events. Zero dependencies. Introduced in v0.15.2.
- `src/core/cli-options.ts` — Global CLI flag parser. `parseGlobalFlags(argv)` returns `{cliOpts, rest}` with `--quiet` / `--progress-json` / `--progress-interval=<ms>` stripped. `getCliOptions()` / `setCliOptions()` expose a module-level singleton so commands reach the resolved flags without parameter threading. `cliOptsToProgressOptions()` maps to reporter options. `childGlobalFlags()` returns the flag suffix to append to `execSync('gbrain ...')` calls in migration orchestrators. `OperationContext.cliOpts` extends shared-op dispatch for MCP callers.
- `src/core/db-lock.ts` (v0.22.13) — generic `tryAcquireDbLock(engine, lockId, ttlMinutes)` over the existing `gbrain_cycle_locks` table. Parameterized lock id so different scopes can nest cleanly: `gbrain-cycle` for the broad cycle (held by `cycle.ts`) and `gbrain-sync` (`SYNC_LOCK_ID` constant) for `performSync`'s narrower writer window. Same UPSERT-with-TTL semantics as the prior cycle-only helper, just generalized. Survives PgBouncer transaction pooling (unlike session-scoped `pg_try_advisory_lock`); crashed holders auto-release once their TTL expires.
- `src/core/sync-concurrency.ts` (v0.22.13) — single source of truth for the parallel-sync policy. Exports `autoConcurrency(engine, fileCount, override?)` (PGLite always serial; explicit override clamped to >=1; auto path returns `DEFAULT_PARALLEL_WORKERS=4` when `fileCount > AUTO_CONCURRENCY_FILE_THRESHOLD=100`), `shouldRunParallel(workers, fileCount, explicit)` (Q1: explicit `--workers` bypasses the >50-file floor), and `parseWorkers(s)` (rejects `'0'`, `'-3'`, `'foo'`, `'1.5'`, trailing chars — replaces the prior parseInt-with-no-validation in both `sync.ts` and `import.ts`). Used by `performSync`, `performFullSync`, `runImport`, and the Minion `sync` handler so the three sites can no longer drift.
- `src/commands/sync.ts` — `gbrain sync` CLI + the `performSync` / `performFullSync` library entrypoints (consumed by the autopilot cycle and the Minion sync handler). v0.22.13 (PR #490): `performSync` wraps its body in a `gbrain-sync` writer lock so two concurrent syncs (manual + autopilot, two terminals, two Conductor workspaces) cannot both write `last_commit` and let the last writer win. Head-drift gate after the import phase re-checks `git rev-parse HEAD`; if HEAD moved (someone ran `git checkout` / `git pull` mid-sync), the bookmark refuses to advance. Vanished files now record a failedFiles entry instead of silent-skip — the silent-skip-then-advance pathology that survived prior hardening passes is dead. Worker engines wrap in try/finally so disconnect always fires (panic-path leak fix). Both PGLite-detection sites use `engine.kind === 'pglite'`. CLI accepts `--workers N` (alias `--concurrency N`), validated via `parseWorkers`. Explicit `--workers` bypasses the auto-path file-count floor; auto path defers to `autoConcurrency()`. Banner moved to stderr.
- `src/core/cycle.ts` — v0.17 brain maintenance cycle primitive (extended to **8 phases in v0.23**). `runCycle(engine: BrainEngine | null, opts: CycleOpts): Promise<CycleReport>` composes phases in semantically-driven order: **lint → backlinks → sync → synthesize → extract → patterns → embed → orphans**. v0.23's `synthesize` phase runs after sync (cross-references see fresh brain) and before extract (auto-link materializes its writes); `patterns` runs after extract so it reads a fresh graph (codex finding #7 — subagent put_page sets `ctx.remote=true` and skips auto-link/timeline by default; extract is the canonical materialization). Three callers: `gbrain dream` CLI, `gbrain autopilot` daemon's inline path, and the Minions `autopilot-cycle` handler. Coordination via `gbrain_cycle_locks` DB table + `~/.gbrain/cycle.lock` file lock with PID-liveness for PGLite. `CycleReport.schema_version: "1"` is stable; totals additively grew in v0.23 (`transcripts_processed`, `synth_pages_written`, `patterns_written`). `yieldBetweenPhases` runs between phases. **v0.23 added `yieldDuringPhase`** for in-phase keepalive — synthesize/patterns call it during long waits to renew the cycle-lock TTL. Engine nullable; lock-skip on read-only phase selections. v0.22.1 (#403): `CycleOpts.signal?: AbortSignal` propagates the worker's abort signal; `checkAborted()` fires between every phase. v0.22.1 (#417): `runPhaseSync` returns `pagesAffected` via `SyncPhaseResult`; `runCycle` captures it and threads to `runPhaseExtract` as the 4th arg. v0.22.1 (Codex F2): `runPhaseSync` takes `willRunExtractPhase: boolean` and sets `noExtract: phases.includes('extract')` so `gbrain dream --phase sync` doesn't silently lose extraction. v0.22.5 (#475): `resolveSourceForDir(engine, brainDir)` threads `sourceId` to `performSync()` so sync reads the per-source `sources.last_commit` anchor instead of the drift-prone global `config.sync.last_commit` key.
- `src/core/cycle.ts` — v0.17 brain maintenance cycle primitive (extended to **9 phases in v0.29**). `runCycle(engine: BrainEngine | null, opts: CycleOpts): Promise<CycleReport>` composes phases in semantically-driven order: **lint → backlinks → sync → synthesize → extract → patterns → recompute_emotional_weight → embed → orphans**. v0.29 adds the `recompute_emotional_weight` phase between patterns and embed; it sees the union of `syncPagesAffected` + `synthesizeWrittenSlugs` for incremental mode, or all pages when neither anchor is set (full backfill via `gbrain dream --phase recompute_emotional_weight`). v0.29 also extends `CycleReport.totals` with `pages_emotional_weight_recomputed` (additive, schema_version stays "1"). v0.23's `synthesize` phase runs after sync (cross-references see fresh brain) and before extract (auto-link materializes its writes); `patterns` runs after extract so it reads a fresh graph (codex finding #7 — subagent put_page sets `ctx.remote=true` and skips auto-link/timeline by default; extract is the canonical materialization). Three callers: `gbrain dream` CLI, `gbrain autopilot` daemon's inline path, and the Minions `autopilot-cycle` handler. Coordination via `gbrain_cycle_locks` DB table + `~/.gbrain/cycle.lock` file lock with PID-liveness for PGLite. `CycleReport.schema_version: "1"` is stable; totals additively grew in v0.23 (`transcripts_processed`, `synth_pages_written`, `patterns_written`). `yieldBetweenPhases` runs between phases. **v0.23 added `yieldDuringPhase`** for in-phase keepalive — synthesize/patterns call it during long waits to renew the cycle-lock TTL. Engine nullable; lock-skip on read-only phase selections. v0.22.1 (#403): `CycleOpts.signal?: AbortSignal` propagates the worker's abort signal; `checkAborted()` fires between every phase. v0.22.1 (#417): `runPhaseSync` returns `pagesAffected` via `SyncPhaseResult`; `runCycle` captures it and threads to `runPhaseExtract` as the 4th arg. v0.22.1 (Codex F2): `runPhaseSync` takes `willRunExtractPhase: boolean` and sets `noExtract: phases.includes('extract')` so `gbrain dream --phase sync` doesn't silently lose extraction. v0.22.5 (#475): `resolveSourceForDir(engine, brainDir)` threads `sourceId` to `performSync()` so sync reads the per-source `sources.last_commit` anchor instead of the drift-prone global `config.sync.last_commit` key.
- `src/core/cycle/synthesize.ts` (v0.23) — Synthesize phase: conversation-transcript-to-brain pipeline. Reads from `dream.synthesize.session_corpus_dir`, runs cheap Haiku verdict (cached in `dream_verdicts`), then fans out one Sonnet subagent per worth-processing transcript with `allowed_slug_prefixes` (sourced from `skills/_brain-filing-rules.json` `dream_synthesize_paths.globs`). Orchestrator collects slugs from `subagent_tool_executions` (NOT `pages.updated_at` — codex finding #2) and reverse-renders DB → markdown via `serializeMarkdown`. Cooldown via `dream.synthesize.last_completion_ts`, written ONLY on success. Idempotency key `dream:synth:<file_path>:<content_hash>`. Auto-commit deferred to v1.1 (codex #5). `--dry-run` runs Haiku, skips Sonnet (codex #8). Subagent never gets fs-write access. **v0.23.2:** `renderPageToMarkdown` (now exported) stamps `dream_generated: true` and `dream_cycle_date` into every reverse-write's frontmatter; `writeSummaryPage` does the same on the dream-cycle summary index. The marker is the explicit identity surface checked by `isDreamOutput` in `transcript-discovery.ts` — replaces the v0.23.1 content-prefix heuristic that could miss real output (`serializeMarkdown` doesn't embed slugs in body) and false-positive on user transcripts citing brain pages. `judgeSignificance` and `JudgeClient` are exported; `judgeSignificance` accepts a `verdictModel` parameter (default `claude-haiku-4-5-20251001`) loaded from `dream.synthesize.verdict_model` via `loadSynthConfig`.
- `src/core/cycle/patterns.ts` (v0.23) — Patterns phase: cross-session theme detection over reflections within `dream.patterns.lookback_days` (default 30). Names a pattern only when ≥`dream.patterns.min_evidence` (default 3) reflections support it. Single Sonnet subagent; same allow-list path as synthesize. Runs AFTER `extract` so the graph is fresh.
- `src/core/cycle/emotional-weight.ts` (v0.29) — Pure function `computeEmotionalWeight({tags, takes}, {highEmotionTags?, userHolder?})`. Deterministic 0..1 score: tag-emotion boost (max 0.5, case-insensitive match against `HIGH_EMOTION_TAGS` seed list), take density (0.1/take, capped at 0.3), take avg weight (0..0.1), user-holder ratio (0..0.1 over active takes; default holder = 'garry'). Total clamped to [0..1]. Anglocentric / personal-life-biased seed list intentional; users override via config key `emotional_weight.high_tags` (JSON array). `userHolder` overridable via `emotional_weight.user_holder`.
- `src/core/cycle/anomaly.ts` (v0.29) — Pure stats helpers for `find_anomalies`. `meanStddev` returns sample stddev (n-1 denominator) and (0,0) for empty input. `computeAnomaliesFromBuckets(baseline, today, sigma, limit)` takes densified daily-count buckets + today's counts per cohort, returns `AnomalyResult[]`. Zero-stddev fallback: cohort fires when `count > mean + 1`, with `sigma_observed = count - mean` as a finite sort proxy (no NaN). Brand-new cohorts (no baseline) have `mean=0, stddev=0` so the fallback fires at count >= 2. Sorted by `sigma_observed` desc, top `limit` (default 20). `page_slugs` capped at 50 per cohort.
- `src/core/cycle/recompute-emotional-weight.ts` (v0.29) — Cycle phase orchestrator. Two SQL round-trips total: `engine.batchLoadEmotionalInputs(slugs?)` → `computeEmotionalWeight` (per-row pure function) → `engine.setEmotionalWeightBatch(rows)`. Reads config keys `emotional_weight.high_tags` (JSON array, falls back to default seed list on parse error) and `emotional_weight.user_holder`. Empty `affectedSlugs` array short-circuits with zero-work success. dry-run mode reports the would-write count without touching the DB. Engine throw bubbles into `status: 'fail'` with code `RECOMPUTE_EMOTIONAL_WEIGHT_FAIL` so the cycle continues.
- `src/core/transcripts.ts` (v0.29) — `listRecentTranscripts(engine, opts)` library reused by both the `gbrain transcripts recent` CLI and the `get_recent_transcripts` MCP op. Reads `dream.synthesize.session_corpus_dir` + `dream.synthesize.meeting_transcripts_dir` config keys (same as `discoverTranscripts`); walks for `.txt` files within `days`; applies `isDreamOutput` guard from `transcript-discovery.ts` (skips dream-generated files); returns `{path, date, mtime, length, summary}[]` sorted newest-first. Summary mode (default true) returns first non-empty line + ~250 trailing chars. Full mode caps at 100KB/file. Missing/non-existent corpus dirs return `[]`, not error. **Trust gate lives in the op handler, not here**: the op throws `permission_denied` for `ctx.remote === true`; this library is a trusted library function used by both the gated op and the local CLI.
- `src/core/operations-descriptions.ts` (v0.29) — Constants module for tool descriptions. Pinned via `test/operations-descriptions.test.ts`. Houses `GET_RECENT_SALIENCE_DESCRIPTION`, `FIND_ANOMALIES_DESCRIPTION`, `GET_RECENT_TRANSCRIPTS_DESCRIPTION` plus the redirect-edited `LIST_PAGES_DESCRIPTION`, `QUERY_DESCRIPTION`, `SEARCH_DESCRIPTION`. Stable surface for the Tier-2 LLM routing eval — extracting them keeps the test from binding to whatever was in `operations.ts` at test-run time.
- `src/core/cycle/transcript-discovery.ts` (v0.23) — Pure filesystem walk for synthesize. `discoverTranscripts(opts)` filters `.txt` files by date range, min_chars, and word-boundary regex `excludePatterns` (Q-3: `medical` matches "medical advice" but NOT "comedical"; power users may pass full regex). `readSingleTranscript(path)` is the `gbrain dream --input <file>` ad-hoc path. **v0.23.2 self-consumption guard:** `DREAM_OUTPUT_MARKER_RE` (anchored at frontmatter open `---\n`, optional BOM + CRLF tolerance, scans first 2000 chars for `dream_generated: true` with case-insensitive value and word boundary on `true`) drives `isDreamOutput(content, bypass=false)`. Both `discoverTranscripts` and `readSingleTranscript` skip matching files and emit a `[dream] skipped <basename>: dream_generated marker` stderr log (no more silent skips). `bypassGuard?: boolean` on `DiscoverOpts` and `readSingleTranscript`'s opts disables the guard for the explicit `--unsafe-bypass-dream-guard` escape hatch only — never auto-applied for `--input`. Replaces v0.23.1's `DREAM_OUTPUT_SLUGS` content-prefix list.
- `src/commands/dream.ts` — v0.17 `gbrain dream` CLI; ~80-line thin alias over `runCycle`. brainDir resolution requires explicit `--dir` OR `sync.repo_path` config. Flags: `--dry-run`, `--json`, `--phase <name>`, `--pull`, `--dir <path>`. **v0.23 added** `--input <file>` (ad-hoc transcript, implies `--phase synthesize`), `--date YYYY-MM-DD`, `--from <d> --to <d>` (backfill range). Conflict detection: `--input` + `--date` exits 2. ISO date validation. `--dry-run` runs Haiku significance verdict but skips Sonnet synthesis (codex finding #8 — NOT zero LLM calls). Exit code 1 on status=failed. **v0.23.2 added** `--unsafe-bypass-dream-guard` (long-form intentional, plumbed through `runCycle.synthBypassDreamGuard` → `SynthesizePhaseOpts.bypassDreamGuard` → `discoverTranscripts({bypassGuard})` and `readSingleTranscript({bypassGuard})`). Loud stderr warning fires at synthesize-phase entry when set. Never auto-applied for `--input` so any caller can't silently re-trigger the loop bug.
- `src/commands/friction.ts` + `src/core/friction.ts` (v0.23) — `gbrain friction {log,render,list,summary}` reporter. Append-only JSONL under `$GBRAIN_HOME/friction/<run-id>.jsonl`. Schema is a flat extension of `StructuredAgentError` (D20). Render groups by severity → phase, defaults to `--redact` for md output (strips `$HOME`/`$CWD` to placeholders so reports paste safely in PRs). Run-id resolves from `--run-id` > `$GBRAIN_FRICTION_RUN_ID` > `standalone.jsonl`. Skills the claw-test exercises gain a `_friction-protocol.md` callout so agents know when to log friction.
+1 -1
View File
@@ -1,6 +1,6 @@
{
"name": "gbrain",
"version": "0.28.12",
"version": "0.29.1",
"description": "Postgres-native personal knowledge brain with hybrid RAG search",
"type": "module",
"main": "src/core/index.ts",
+7
View File
@@ -121,6 +121,13 @@ ALLOW_LIST=(
# walkthrough; it explains the privacy-guard extension to the
# operating agent and references the banned literals while doing so.
'skills/migrations/v0.25.1.md'
# v0.29.1: the recency-decay default-map test asserts that
# DEFAULT_RECENCY_DECAY's keys do NOT include fork-specific path
# prefixes. The test must name the banned tokens to assert their
# absence — same exception status as scripts/check-privacy.sh,
# CHANGELOG.md, and CLAUDE.md (meta-rule enforcement requires
# mentioning what the rule forbids).
'test/recency-decay.test.ts'
)
is_allowed() {
+131
View File
@@ -0,0 +1,131 @@
# Salience + Recency on `gbrain query` (v0.29.1)
YOU ARE IN CHARGE of the `salience` and `recency` parameters on gbrain's
`query` op. They are TWO ORTHOGONAL axes — use either, both, or neither.
If you OMIT a parameter, gbrain auto-detects from query text via a
regex heuristic. The default for queries that don't match any pattern
is `'off'`. Prefer to pass values EXPLICITLY when you know what the
user wants.
## What each axis means
- `salience`**mattering**. Boosts pages with high `emotional_weight`
and many active takes. NO time component. Use when the user wants
the most important / most-discussed pages on a topic, regardless of
when they were updated.
- `recency`**age**. Boosts pages with recent `effective_date`. NO
mattering signal. Per-prefix decay (`concepts/`, `originals/`,
`writing/` are evergreen; `daily/`, `media/x/`, `chat/` decay
aggressively). Use when freshness is the signal.
## When to pass `salience='on'`
The "mattering" axis. The user wants what matters in this brain on
the topic, not the canonical encyclopedia entry.
- `"prep me for the widget-ceo meeting"` (meeting prep)
- `"catch me up on acme"` (conversation recall)
- `"what's going on with widget-co"` (current state matters)
- `"remind me about the deal"` (recall takes / opinions)
- `"what's been happening lately"`
- `"status update on X"`
Pair with `recency='on'` when current-state matters. Just `salience='on'`
alone gives you "what matters about X regardless of when."
## When to pass `recency='on'`
The "freshness" axis. The user wants recent content, with or without
mattering.
- `"latest news on AI"` (recent, no mattering needed)
- `"what's new this week"`
- `"recent updates on widget-co"`
- `"this week's announcements"`
Use `'strong'` when the user explicitly asks for the most recent:
- `"what happened today"`
- `"right now what's going on"`
- `"this morning"`
## When to pass BOTH `'off'`
The "canonical truth" axis. The user wants the authoritative answer.
- `"who is widget-ceo"` (entity lookup)
- `"what is widget-co"` (definitional)
- `"history of acme"` (historical research)
- `"explain how recursion works"` (concept query)
- `"tell me about widget-co"` (canonical recall)
- Code lookups: function/class names, syntax like `Foo::bar()` or `obj.method`
- Graph traversal: backlinks, inbound/outbound edges
- Anything not matching above
## Heuristic when unsure
> Current state → on. Canonical truth → off.
If you can't classify confidently, OMIT the param and let gbrain's
auto-detect handle it. The heuristic defaults to `off` for everything
that doesn't clearly match a current-state pattern. The `--explain`
output shows `_resolved.salience_source` and `_resolved.recency_source`
('caller' vs. 'auto_heuristic') so you can see what fired and why.
You can override at any time. gbrain is smart but not infallible. You
have context gbrain doesn't.
## Narrow temporal-bound exception
Even when a query matches canonical patterns, an explicit temporal
bound (`today`, `this week`, `right now`, `since X`, `last N days`)
overrides the canonical-wins rule:
- `"who is widget-ceo right now"` → recency = `'strong'`, salience = `'on'`
(the temporal bound wins over "who is")
- `"who is widget-ceo"` → recency = `'off'`, salience = `'off'` (no bound)
## English-only
The auto-detect heuristic is English-only in v0.29.1. Non-English
queries fall through to the default `off` for both axes. Pass
`salience` and `recency` explicitly for non-English queries.
## Tuning the recency formula
Defaults are in `src/core/search/recency-decay.ts`. Override per-brain
via `gbrain.yml`:
```yaml
recency:
daily/:
halflifeDays: 7
coefficient: 2.0
custom-prefix/:
halflifeDays: 30
coefficient: 0.5
```
Or per-process via env: `GBRAIN_RECENCY_DECAY="prefix:halflife:coefficient,..."`.
The parser fails LOUD on bad syntax (no silent fallback).
## Date filtering with `since` / `until`
Independent of the axes. Filter to pages whose `effective_date` is
within a range:
- `since: '7d'` — last 7 days
- `since: '2024-06-01'` — ISO-8601
- `until: '2024-06-30'` — ends at end-of-day
`since`/`until` work with OR without `salience`/`recency`. Pure filter,
no boost.
## See also
- `docs/recency.md` — full reference
- `gbrain query --explain` — see resolved values + factor contributions
- `get_recent_salience` op gains `recency_bias: 'flat' | 'on'` — opt
into per-prefix decay on the dedicated salience query
+106
View File
@@ -0,0 +1,106 @@
# v0.29.1 migration — Salience + Recency as two orthogonal options
`gbrain upgrade` runs `gbrain apply-migrations` automatically. Most users
won't need to do anything else. If you hit issues or want to verify the
upgrade succeeded, run the steps below.
## What changed
This release adds two new optional ranking axes to the `query` MCP op:
- `salience: 'off' | 'on' | 'strong'` — boost pages with high
`emotional_weight` + many active takes. NO time component.
- `recency: 'off' | 'on' | 'strong'` — boost pages with recent
`effective_date`, per-prefix decay. NO mattering signal.
They're TRULY ORTHOGONAL — either, both, or neither. The agent decides
per query, with `--explain` output showing the resolved values and
sources.
Existing v0.29.0 callers (no new params) get UNCHANGED behavior.
## What the agent should do
Read `skills/conventions/salience-and-recency.md` (auto-installed by
the skillpack). Then in your `gbrain query` calls:
1. Pass `salience='on'` for "what matters about X" queries (catch up,
meeting prep, conversation recall).
2. Pass `recency='on'` for "what's new on X" queries (latest, this
week, today's news).
3. Pass both for "what's been going on with X" queries.
4. Omit both for canonical / definitional / code / graph queries
(`who is X`, `what is X`, etc.) — gbrain's heuristic defaults
to `off`.
## Verification
```bash
# 1. Confirm upgrade
gbrain --version # 0.29.1
gbrain doctor --json | jq '.checks[] | select(.name | startswith("schema_version"))'
# 2. Recompute emotional weights (one-time after upgrade)
gbrain dream --phase recompute_emotional_weight
# 3. Verify health checks
gbrain doctor --json | jq '.checks[] | select(.name | startswith("salience_health") or startswith("effective_date_health"))'
# 4. Try the new axes
gbrain query "what's been going on with X" --explain --json | jq '._resolved'
# expected: { salience: "on", recency: "on", salience_source: "auto_heuristic", recency_source: "auto_heuristic" }
gbrain query "who is X" --explain --json | jq '._resolved'
# expected: { salience: "off", recency: "off", ... }
# 5. Date filter
gbrain query "acme" --since 7d --until 2024-06-30 --json
```
## If something looks wrong
```bash
# Re-apply migrations manually
gbrain apply-migrations --yes
# Force re-run the v0.29.1 backfill (computeEffectiveDate on every page)
gbrain reindex-frontmatter --yes --force
# Doctor for any warnings
gbrain doctor --json
```
If issues persist, file at https://github.com/garrytan/gbrain/issues
with the doctor output and contents of `~/.gbrain/upgrade-errors.jsonl`
(if it exists).
## Schema additions (idempotent, additive only)
Migration v38 adds 4 nullable columns to `pages`:
- `effective_date` — content-date computed from frontmatter precedence
- `effective_date_source` — sentinel for the doctor check
- `import_filename` — basename captured at import for filename-date precedence
- `salience_touched_at` — bumped by recompute_emotional_weight on changes
Migration v39 adds 7 nullable columns to `eval_candidates`:
- `as_of_ts`, `salience_param`, `recency_param`, `salience_resolved`,
`recency_resolved`, `salience_source`, `recency_source`
Plus the `pages_coalesce_date_idx` expression index for since/until filters.
NDJSON `schema_version` STAYS at 1; consumers ignore unknown fields.
No cross-repo coordination required.
## Behavior changes
- v0.29.0 `get_recent_salience` formula: UNCHANGED for callers who don't
pass `recency_bias='on'`. Pass `recency_bias='on'` to opt into per-prefix
decay (concepts/originals/writing/ evergreen; daily/, media/x/ aggressive).
- v0.29.0 SearchOpts: `afterDate`, `beforeDate`, `recencyBoost: 0|1|2`
remain as DEPRECATED ALIASES for `since`, `until`, `recency`. They
emit a stderr warning once per process. Removed in v0.30.
- `detail='high'` source-boost bypass — UNCHANGED in v0.29.1. The
known temporal-query swamp is documented; pass `salience='on'` to
compensate via salience boost.
+30 -1
View File
@@ -24,7 +24,7 @@ for (const op of operations) {
}
// CLI-only commands that bypass the operation layer
const CLI_ONLY = new Set(['init', 'upgrade', 'post-upgrade', 'check-update', 'integrations', 'publish', 'check-backlinks', 'lint', 'report', 'import', 'export', 'files', 'embed', 'serve', 'call', 'config', 'doctor', 'migrate', 'eval', 'sync', 'extract', 'features', 'autopilot', 'graph-query', 'jobs', 'agent', 'apply-migrations', 'skillpack-check', 'skillpack', 'resolvers', 'integrity', 'repair-jsonb', 'orphans', 'sources', 'mounts', 'dream', 'check-resolvable', 'routing-eval', 'skillify', 'smoke-test', 'providers', 'storage', 'repos', 'code-def', 'code-refs', 'reindex-code', 'code-callers', 'code-callees', 'frontmatter', 'auth', 'friction', 'claw-test', 'book-mirror', 'takes', 'think']);
const CLI_ONLY = new Set(['init', 'upgrade', 'post-upgrade', 'check-update', 'integrations', 'publish', 'check-backlinks', 'lint', 'report', 'import', 'export', 'files', 'embed', 'serve', 'call', 'config', 'doctor', 'migrate', 'eval', 'sync', 'extract', 'features', 'autopilot', 'graph-query', 'jobs', 'agent', 'apply-migrations', 'skillpack-check', 'skillpack', 'resolvers', 'integrity', 'repair-jsonb', 'orphans', 'sources', 'mounts', 'dream', 'check-resolvable', 'routing-eval', 'skillify', 'smoke-test', 'providers', 'storage', 'repos', 'code-def', 'code-refs', 'reindex-code', 'reindex-frontmatter', 'code-callers', 'code-callees', 'frontmatter', 'auth', 'friction', 'claw-test', 'book-mirror', 'takes', 'think', 'salience', 'anomalies', 'transcripts']);
async function main() {
// Parse global flags (--quiet / --progress-json / --progress-interval)
@@ -639,6 +639,22 @@ async function handleCliOnly(command: string, args: string[]) {
await runOrphans(engine, args);
break;
}
// v0.29 — Salience + Anomaly Detection
case 'salience': {
const { runSalience } = await import('./commands/salience.ts');
await runSalience(engine, args);
break;
}
case 'anomalies': {
const { runAnomalies } = await import('./commands/anomalies.ts');
await runAnomalies(engine, args);
break;
}
case 'transcripts': {
const { runTranscripts } = await import('./commands/transcripts.ts');
await runTranscripts(engine, args);
break;
}
case 'takes': {
const { runTakes } = await import('./commands/takes.ts');
await runTakes(engine, args);
@@ -683,6 +699,16 @@ async function handleCliOnly(command: string, args: string[]) {
await runReindexCodeCli(engine, args);
break;
}
case 'reindex-frontmatter': {
// v0.29.1: recovery / explicit-rebuild path for pages.effective_date.
// Mirror of reindex-code shape. Wraps the shared library function in
// src/core/backfill-effective-date.ts (same code path the v0.29.1
// migration orchestrator uses). The orchestrator runs once on
// upgrade; this command is for after-the-fact frontmatter edits.
const { reindexFrontmatterCli } = await import('./commands/reindex-frontmatter.ts');
await reindexFrontmatterCli(args);
return; // reindexFrontmatterCli handles its own engine lifecycle
}
case 'code-callers': {
// v0.20.0 Cathedral II Layer 10 (C4): "who calls <symbol>?"
const { runCodeCallers } = await import('./commands/code-callers.ts');
@@ -896,6 +922,9 @@ TOOLS
check-backlinks <check|fix> [dir] Find/fix missing back-links across brain
lint <dir|file> [--fix] Catch LLM artifacts, placeholder dates, bad frontmatter
orphans [--json] [--count] Find pages with no inbound wikilinks
salience [--days N] [--kind P] v0.29: pages ranked by emotional + activity salience
anomalies [--since D] [--sigma N] v0.29: cohort-based statistical anomalies (tag, type)
transcripts recent [--days N] v0.29: recent raw .txt transcripts (local-only)
dream [--dry-run] [--json] Run the overnight maintenance cycle once (cron-friendly).
See also: autopilot --install (continuous daemon).
check-resolvable [--json] [--fix] Validate skill tree (reachability/MECE/DRY)
+96
View File
@@ -0,0 +1,96 @@
/**
* gbrain anomalies — Statistical anomalies in recent page activity.
*
* Deterministic: zero LLM calls. Computes baseline (mean, stddev) of pages
* touched per cohort × day over `lookback_days`, with `generate_series`
* zero-fill so rare cohorts don't get sparse-day biased baselines. Reports
* cohorts whose target-day count exceeds `mean + sigma * stddev`.
*
* Cohort kinds: tag, type. Year cohort deferred to v0.30.
*
* Usage:
* gbrain anomalies # since=today, lookback=30d, sigma=3
* gbrain anomalies --since 2026-04-28
* gbrain anomalies --sigma 2 --lookback-days 60
* gbrain anomalies --json
*/
import type { BrainEngine } from '../core/engine.ts';
interface RunOpts {
since?: string;
lookbackDays?: number;
sigma?: number;
json?: boolean;
}
function parseArgs(args: string[]): RunOpts | { help: true } {
const opts: RunOpts = {};
for (let i = 0; i < args.length; i++) {
const a = args[i];
if (a === '--help' || a === '-h') return { help: true };
if (a === '--json') { opts.json = true; continue; }
if (a === '--since') {
const v = args[++i];
if (v && /^\d{4}-\d{2}-\d{2}$/.test(v)) opts.since = v;
continue;
}
if (a === '--lookback-days') {
const n = parseInt(args[++i] ?? '', 10);
if (Number.isFinite(n) && n >= 1) opts.lookbackDays = n;
continue;
}
if (a === '--sigma') {
const n = parseFloat(args[++i] ?? '');
if (Number.isFinite(n) && n > 0) opts.sigma = n;
continue;
}
}
return opts;
}
const HELP = `Usage: gbrain anomalies [options]
Statistical anomalies in recent page activity, grouped by cohort (tag, type).
Options:
--since YYYY-MM-DD Target day (default: today UTC)
--lookback-days N Baseline window (default 30)
--sigma N Threshold multiplier (default 3.0)
--json JSON output for agents
--help, -h Show this help
`;
export async function runAnomalies(engine: BrainEngine, args: string[]): Promise<void> {
const parsed = parseArgs(args);
if ('help' in parsed) {
console.log(HELP);
return;
}
const rows = await engine.findAnomalies({
since: parsed.since,
lookback_days: parsed.lookbackDays,
sigma: parsed.sigma,
});
if (parsed.json) {
console.log(JSON.stringify(rows, null, 2));
return;
}
if (rows.length === 0) {
console.log('(no anomalies for this window)');
return;
}
console.log(`${rows.length} anomalous cohort(s) for ${parsed.since ?? new Date().toISOString().slice(0, 10)}:\n`);
rows.forEach(r => {
const baselineMean = r.baseline_mean.toFixed(2);
const baselineStd = r.baseline_stddev.toFixed(2);
const sigma = r.sigma_observed.toFixed(2);
console.log(
`[${r.cohort_kind}=${r.cohort_value}] ` +
`count=${r.count}, baseline mean=${baselineMean}±${baselineStd}, sigma=${sigma}`
);
const slugSample = r.page_slugs.slice(0, 5).join(', ');
const more = r.page_slugs.length > 5 ? `, +${r.page_slugs.length - 5} more` : '';
if (slugSample) console.log(` pages: ${slugSample}${more}`);
});
}
+108
View File
@@ -949,6 +949,114 @@ export async function runDoctor(engine: BrainEngine | null, args: string[], dbSo
}
}
// 11a-2. effective_date_health (v0.29.1).
//
// Detects pages where computeEffectiveDate fell back to updated_at even
// though parseable frontmatter dates are present (codex pass-1 #5
// resolution: the sentinel column lets us catch "wrong but populated"
// rows that look healthy at first glance).
//
// Sample 1000 random rows by default to keep the check fast on 200K-page
// brains. The expression index pages_coalesce_date_idx makes the future-
// date and pre-1990 scans cheap; the parseable-fm-date scan reads
// frontmatter JSONB and is the slow path.
progress.heartbeat('effective_date_health');
try {
const result = await engine.executeRaw<{ kind: string; count: string }>(
`WITH sample AS (
SELECT slug, frontmatter, effective_date, effective_date_source
FROM pages
ORDER BY id DESC
LIMIT 1000
)
SELECT 'fallback_with_fm_date' AS kind, COUNT(*)::text AS count
FROM sample
WHERE effective_date_source = 'fallback'
AND (frontmatter ? 'event_date' OR frontmatter ? 'date' OR frontmatter ? 'published')
UNION ALL
SELECT 'future_dated', COUNT(*)::text FROM sample
WHERE effective_date IS NOT NULL AND effective_date > NOW() + INTERVAL '1 year'
UNION ALL
SELECT 'pre_1990', COUNT(*)::text FROM sample
WHERE effective_date IS NOT NULL AND effective_date < TIMESTAMPTZ '1990-01-01'`,
);
const counts = new Map(result.map(r => [r.kind, Number(r.count)]));
const fallbackWithFm = counts.get('fallback_with_fm_date') ?? 0;
const future = counts.get('future_dated') ?? 0;
const pre1990 = counts.get('pre_1990') ?? 0;
if (fallbackWithFm > 0 || future > 0 || pre1990 > 0) {
const parts: string[] = [];
if (fallbackWithFm > 0) parts.push(`${fallbackWithFm} fell back to updated_at despite parseable frontmatter date`);
if (future > 0) parts.push(`${future} dated > NOW() + 1y`);
if (pre1990 > 0) parts.push(`${pre1990} pre-1990`);
checks.push({
name: 'effective_date_health',
status: 'warn',
message: `${parts.join('; ')} (sample of last 1000 pages). Run \`gbrain reindex-frontmatter\` to recompute.`,
});
} else {
checks.push({
name: 'effective_date_health',
status: 'ok',
message: 'Sample of last 1000 pages clean (no fallback-with-parseable-fm-date, no future-dated, no pre-1990)',
});
}
} catch (err) {
const code = (err as { code?: string } | null)?.code;
if (code === '42703') {
// column doesn't exist — pre-v0.29.1 brain
checks.push({ name: 'effective_date_health', status: 'ok', message: 'Skipped (effective_date column unavailable — run gbrain apply-migrations)' });
} else {
checks.push({ name: 'effective_date_health', status: 'warn', message: `Could not read pages: ${(err as Error)?.message ?? String(err)}` });
}
}
// 11a-3. salience_health (v0.29.1).
//
// Detects pages with active takes (so emotional_weight should be > 0)
// whose recompute_emotional_weight phase hasn't yet run, plus the
// brain-average emotional_weight as an informational signal.
progress.heartbeat('salience_health');
try {
const result = await engine.executeRaw<{ kind: string; n: string }>(
`SELECT 'zero_weight_with_takes' AS kind, COUNT(DISTINCT p.id)::text AS n
FROM pages p
JOIN takes t ON t.page_id = p.id AND t.active = TRUE
WHERE COALESCE(p.emotional_weight, 0) = 0
UNION ALL
SELECT 'nonzero_weight', COUNT(*)::text FROM pages WHERE COALESCE(emotional_weight, 0) > 0`,
);
const counts = new Map(result.map(r => [r.kind, Number(r.n)]));
const zeroWithTakes = counts.get('zero_weight_with_takes') ?? 0;
const nonzero = counts.get('nonzero_weight') ?? 0;
if (zeroWithTakes > 0) {
checks.push({
name: 'salience_health',
status: 'warn',
message: `${zeroWithTakes} pages with active takes have emotional_weight=0. Run \`gbrain dream --phase recompute_emotional_weight\` to populate. Brain has ${nonzero} pages with non-zero emotional_weight.`,
});
} else if (nonzero === 0) {
checks.push({
name: 'salience_health',
status: 'ok',
message: 'Skipped (no pages have emotional_weight > 0; either fresh install or recompute hasn\'t run yet)',
});
} else {
checks.push({
name: 'salience_health',
status: 'ok',
message: `${nonzero} pages have non-zero emotional_weight; no take/weight mismatches detected`,
});
}
} catch (err) {
const code = (err as { code?: string } | null)?.code;
if (code === '42703' || code === '42P01') {
checks.push({ name: 'salience_health', status: 'ok', message: 'Skipped (emotional_weight or takes table unavailable — pre-v0.29 brain)' });
} else {
checks.push({ name: 'salience_health', status: 'warn', message: `Could not read pages: ${(err as Error)?.message ?? String(err)}` });
}
}
// 11b. Queue health (v0.19.1 queue-resilience wave).
// Postgres-only because PGLite has no multi-process worker surface. Two
// subchecks, both cheap (single SELECT each, status-index-covered):
+2
View File
@@ -23,6 +23,7 @@ import { v0_18_1 } from './v0_18_1.ts';
import { v0_21_0 } from './v0_21_0.ts';
import { v0_22_4 } from './v0_22_4.ts';
import { v0_28_0 } from './v0_28_0.ts';
import { v0_29_1 } from './v0_29_1.ts';
export const migrations: Migration[] = [
v0_11_0,
@@ -37,6 +38,7 @@ export const migrations: Migration[] = [
v0_21_0,
v0_22_4,
v0_28_0,
v0_29_1,
];
/** Look up a migration by exact version string. */
+160
View File
@@ -0,0 +1,160 @@
/**
* v0.29.1 migration orchestrator — backfill effective_date for existing
* pages.
*
* Migration v38 added pages.effective_date / effective_date_source /
* import_filename / salience_touched_at as nullable columns. Fresh imports
* post-v0.29.1 populate effective_date via the importer's
* `computeEffectiveDate`. Pre-v0.29.1 rows have NULL until this orchestrator
* walks them.
*
* Phases (all idempotent, resumable):
* A. Schema — `gbrain init --migrate-only` ensures v38 ran.
* B. Backfill — keyset-paginated UPDATE via `backfillEffectiveDate`.
* Resumable via the `backfill.effective_date.last_id`
* checkpoint key in the config table. Statement timeout
* set per-batch (Postgres only).
* C. Verify — count remaining NULL effective_date rows; warn if > 0.
* D. Record — handled by the runner.
*/
import { execSync } from 'child_process';
import type { Migration, OrchestratorOpts, OrchestratorResult, OrchestratorPhaseResult } from './types.ts';
import { childGlobalFlags } from '../../core/cli-options.ts';
// ── Phase A — Schema ────────────────────────────────────────
function phaseASchema(opts: OrchestratorOpts): OrchestratorPhaseResult {
if (opts.dryRun) return { name: 'schema', status: 'skipped', detail: 'dry-run' };
try {
execSync('gbrain init --migrate-only' + childGlobalFlags(), {
stdio: 'inherit',
timeout: 600_000, // 10 min — duplicate-heavy installs can be slow
env: process.env,
});
return { name: 'schema', status: 'complete' };
} catch (e) {
return { name: 'schema', status: 'failed', detail: e instanceof Error ? e.message : String(e) };
}
}
// ── Phase B — Backfill effective_date ───────────────────────
async function phaseBBackfill(opts: OrchestratorOpts): Promise<OrchestratorPhaseResult> {
if (opts.dryRun) return { name: 'backfill_effective_date', status: 'skipped', detail: 'dry-run' };
try {
const { createEngine } = await import('../../core/engine-factory.ts');
const { loadConfig, toEngineConfig } = await import('../../core/config.ts');
const { backfillEffectiveDate } = await import('../../core/backfill-effective-date.ts');
const cfg = loadConfig();
if (!cfg) throw new Error('No gbrain config; run `gbrain init` first.');
const engine = await createEngine(toEngineConfig(cfg));
let totalExamined = 0;
let totalUpdated = 0;
const result = await backfillEffectiveDate(engine, {
onBatch: ({ batch, lastId, rowsTouched, cumulative }) => {
totalExamined = cumulative;
totalUpdated += rowsTouched;
if (batch % 10 === 0) {
process.stderr.write(` [backfill] batch ${batch} | last_id=${lastId} | examined=${cumulative} | updated_so_far=${totalUpdated}\n`);
}
},
});
return {
name: 'backfill_effective_date',
status: 'complete',
detail: `examined=${result.examined} updated=${result.updated} fallback=${result.fallback} dur=${result.durationSec.toFixed(1)}s`,
};
} catch (e) {
return { name: 'backfill_effective_date', status: 'failed', detail: e instanceof Error ? e.message : String(e) };
}
}
// ── Phase C — Verify ────────────────────────────────────────
async function phaseCVerify(opts: OrchestratorOpts): Promise<OrchestratorPhaseResult> {
if (opts.dryRun) return { name: 'verify', status: 'skipped', detail: 'dry-run' };
try {
const { createEngine } = await import('../../core/engine-factory.ts');
const { loadConfig, toEngineConfig } = await import('../../core/config.ts');
const cfg = loadConfig();
if (!cfg) throw new Error('No gbrain config; run `gbrain init` first.');
const engine = await createEngine(toEngineConfig(cfg));
// Count rows where effective_date is still NULL but frontmatter HAS a
// parseable date — those are the rows the backfill should have touched
// but didn't. (Rows that fall through to 'fallback' have non-null
// effective_date already; this catches genuine misses.)
const rows = await engine.executeRaw<{ count: string }>(
`SELECT COUNT(*)::text AS count FROM pages WHERE effective_date IS NULL`,
);
const remaining = Number(rows[0]?.count ?? 0);
if (remaining > 0) {
return {
name: 'verify',
status: 'failed',
detail: `${remaining} pages still have NULL effective_date (backfill incomplete)`,
};
}
return { name: 'verify', status: 'complete', detail: '0 pages with NULL effective_date' };
} catch (e) {
return { name: 'verify', status: 'failed', detail: e instanceof Error ? e.message : String(e) };
}
}
// ── Orchestrator ────────────────────────────────────────────
async function orchestrator(opts: OrchestratorOpts): Promise<OrchestratorResult> {
console.log('');
console.log('=== v0.29.1 — backfill effective_date for existing pages ===');
if (opts.dryRun) console.log(' (dry-run; no side effects)');
console.log('');
const phases: OrchestratorPhaseResult[] = [];
const a = phaseASchema(opts);
phases.push(a);
if (a.status === 'failed') return finalize(phases, 'failed');
const b = await phaseBBackfill(opts);
phases.push(b);
if (b.status === 'failed') return finalize(phases, 'partial');
const c = await phaseCVerify(opts);
phases.push(c);
const status: 'complete' | 'partial' | 'failed' =
c.status === 'failed' ? 'partial' : 'complete';
return finalize(phases, status);
}
function finalize(phases: OrchestratorPhaseResult[], status: 'complete' | 'partial' | 'failed'): OrchestratorResult {
return { version: '0.29.1', status, phases };
}
export const v0_29_1: Migration = {
version: '0.29.1',
featurePitch: {
headline: 'Recency + salience as two opt-in axes — agent in charge of when to use each',
description:
'gbrain v0.29.1 adds two new optional ranking axes to the query op: salience ' +
'(emotional_weight + take_count, the "this matters" signal) and recency (per-prefix ' +
'age decay, the "this is recent" signal). Truly orthogonal — use either, both, or ' +
"neither. The query op's tool description teaches your agent when each makes sense " +
'("current state → on; canonical truth → off") and the agent can override per query. ' +
'A new pages.effective_date column is computed at import from frontmatter precedence ' +
'(event_date / date / published / filename) and is immune to auto-link updated_at ' +
'churn. Existing callers (no new params) get UNCHANGED behavior. Run ' +
"`gbrain dream --phase recompute_emotional_weight` once after upgrading.",
},
orchestrator,
};
export const __testing = {
phaseASchema,
phaseBBackfill,
phaseCVerify,
};
+186
View File
@@ -0,0 +1,186 @@
/**
* v0.29.1 — `gbrain reindex-frontmatter`.
*
* Recovery / explicit-rebuild path for `pages.effective_date`. Useful when:
* - The user edited frontmatter dates after import and wants the effective_date
* column refreshed without a full `gbrain sync`.
* - The post-upgrade backfill orchestrator finished but the user wants to
* re-walk a subset (e.g. just `meetings/`) after fixing some frontmatter.
* - The precedence rules change between releases and the user wants to
* re-apply on existing rows.
*
* Thin wrapper over the shared library function in
* `src/core/backfill-effective-date.ts` (same code path the migration
* orchestrator uses; one source of truth for the backfill logic).
*
* Flags mirror `reindex-code`:
* --source <id> Scope to one sources row. Omit = all pages.
* --slug-prefix P Scope to slugs starting with P (e.g. 'meetings/').
* --dry-run Print what WOULD change, no DB writes.
* --yes Skip the confirmation prompt (required for non-TTY non-JSON).
* --json Machine-readable result envelope.
* --force Re-apply even when computed value matches existing
* (bypasses no-op-on-equal guard).
*/
import type { BrainEngine } from '../core/engine.ts';
import { backfillEffectiveDate } from '../core/backfill-effective-date.ts';
import { createInterface } from 'readline';
export interface ReindexFrontmatterOpts {
sourceId?: string;
slugPrefix?: string;
dryRun?: boolean;
yes?: boolean;
json?: boolean;
force?: boolean;
}
export interface ReindexFrontmatterResult {
status: 'ok' | 'dry_run' | 'cancelled';
examined: number;
updated: number;
fallback: number;
durationSec: number;
source_filter?: string;
slug_prefix?: string;
}
async function countAffected(
engine: BrainEngine,
slugPrefix: string | undefined,
sourceId: string | undefined,
): Promise<number> {
const where: string[] = [];
const params: unknown[] = [];
if (slugPrefix) {
params.push(slugPrefix.replace(/[\\%_]/g, (c) => '\\' + c) + '%');
where.push(`slug LIKE $${params.length} ESCAPE '\\\\'`);
}
if (sourceId) {
params.push(sourceId);
where.push(`source_id = $${params.length}`);
}
const sql = `SELECT COUNT(*)::text AS n FROM pages${where.length ? ' WHERE ' + where.join(' AND ') : ''}`;
const rows = await engine.executeRaw<{ n: string }>(sql, params);
return Number(rows[0]?.n ?? 0);
}
async function confirm(prompt: string): Promise<boolean> {
if (!process.stdin.isTTY) return false; // No TTY = require --yes
const rl = createInterface({ input: process.stdin, output: process.stderr });
return new Promise(resolve => {
rl.question(prompt + ' [y/N] ', (ans: string) => {
rl.close();
resolve(ans.trim().toLowerCase() === 'y');
});
});
}
export async function runReindexFrontmatter(
engine: BrainEngine,
opts: ReindexFrontmatterOpts,
): Promise<ReindexFrontmatterResult> {
const total = await countAffected(engine, opts.slugPrefix, opts.sourceId);
if (opts.dryRun) {
// Library function with dryRun=true counts would-update without writing.
const r = await backfillEffectiveDate(engine, {
slugPrefix: opts.slugPrefix,
dryRun: true,
force: opts.force,
// Note: the library doesn't support sourceId filter today; documented
// as a v0.30+ enhancement. CLI surfaces the param so the future
// refinement is non-breaking.
maxRows: total > 0 ? total : undefined,
});
return {
status: 'dry_run',
examined: r.examined,
updated: r.updated,
fallback: r.fallback,
durationSec: r.durationSec,
slug_prefix: opts.slugPrefix,
source_filter: opts.sourceId,
};
}
// Confirm in TTY non-yes flow.
if (!opts.yes && !opts.json && total > 100) {
const ok = await confirm(`Reindex effective_date on ${total} page(s)? Force=${opts.force ? 'yes' : 'no'}.`);
if (!ok) {
return {
status: 'cancelled',
examined: 0, updated: 0, fallback: 0, durationSec: 0,
slug_prefix: opts.slugPrefix,
source_filter: opts.sourceId,
};
}
}
const r = await backfillEffectiveDate(engine, {
slugPrefix: opts.slugPrefix,
force: opts.force,
fresh: true, // CLI is explicit; ignore checkpoint from prior orchestrator runs
onBatch: ({ batch, lastId, rowsTouched, cumulative }) => {
if (!opts.json && batch % 5 === 0) {
process.stderr.write(` [reindex] batch ${batch} | last_id=${lastId} | examined=${cumulative} | updated=${rowsTouched}\n`);
}
},
});
return {
status: 'ok',
examined: r.examined,
updated: r.updated,
fallback: r.fallback,
durationSec: r.durationSec,
slug_prefix: opts.slugPrefix,
source_filter: opts.sourceId,
};
}
/** CLI entrypoint. Argv shape matches reindex-code for consistency. */
export async function reindexFrontmatterCli(args: string[]): Promise<void> {
const opts: ReindexFrontmatterOpts = {};
for (let i = 0; i < args.length; i++) {
const a = args[i];
if (a === '--source') opts.sourceId = args[++i];
else if (a === '--slug-prefix') opts.slugPrefix = args[++i];
else if (a === '--dry-run') opts.dryRun = true;
else if (a === '--yes' || a === '-y') opts.yes = true;
else if (a === '--json') opts.json = true;
else if (a === '--force') opts.force = true;
else {
console.error(`Unknown arg: ${a}`);
process.exit(2);
}
}
const { createEngine } = await import('../core/engine-factory.ts');
const { loadConfig, toEngineConfig } = await import('../core/config.ts');
const cfg = loadConfig();
if (!cfg) {
console.error('No gbrain config; run `gbrain init` first.');
process.exit(1);
}
const engine = await createEngine(toEngineConfig(cfg));
try {
const result = await runReindexFrontmatter(engine, opts);
if (opts.json) {
console.log(JSON.stringify(result, null, 2));
} else {
const noun = result.status === 'dry_run' ? 'would update' : 'updated';
console.error(
`\nReindex ${result.status}: examined=${result.examined} ${noun}=${result.updated} ` +
`fallback=${result.fallback} dur=${result.durationSec.toFixed(1)}s`,
);
}
if (result.status === 'cancelled') process.exit(1);
} finally {
if ('disconnect' in engine && typeof engine.disconnect === 'function') {
await engine.disconnect();
}
}
}
+97
View File
@@ -0,0 +1,97 @@
/**
* gbrain salience — Pages recently touched, ranked by emotional + activity salience.
*
* Deterministic: zero LLM calls. The score blends `emotional_weight`
* (computed during the dream cycle's recompute_emotional_weight phase),
* the count of active takes, and a recency-decay term. See the engine method
* `getRecentSalience` for the SQL.
*
* Usage:
* gbrain salience # top 20 over last 14 days
* gbrain salience --days 7 # narrower window
* gbrain salience --kind personal # filter to slug-prefix
* gbrain salience --json # JSON for agents
*/
import type { BrainEngine } from '../core/engine.ts';
interface RunOpts {
days?: number;
limit?: number;
slugPrefix?: string;
json?: boolean;
}
function parseArgs(args: string[]): RunOpts | { help: true } {
const opts: RunOpts = {};
for (let i = 0; i < args.length; i++) {
const a = args[i];
if (a === '--help' || a === '-h') return { help: true };
if (a === '--json') { opts.json = true; continue; }
if (a === '--days') {
const n = parseInt(args[++i] ?? '', 10);
if (Number.isFinite(n) && n >= 0) opts.days = n;
continue;
}
if (a === '--limit') {
const n = parseInt(args[++i] ?? '', 10);
if (Number.isFinite(n) && n > 0) opts.limit = n;
continue;
}
if (a === '--kind' || a === '--slug-prefix') {
const v = args[++i];
if (v) opts.slugPrefix = v;
continue;
}
}
return opts;
}
const HELP = `Usage: gbrain salience [options]
Pages recently touched, ranked by emotional + activity salience. Surfaces
what's unusual without needing a search term — the inverse of /query.
Options:
--days N Window in days (default 14)
--limit N Max results (default 20, capped at 100)
--kind PREFIX Slug-prefix filter (e.g. personal, wiki/people)
--slug-prefix P Same as --kind
--json JSON output for agents
--help, -h Show this help
`;
export async function runSalience(engine: BrainEngine, args: string[]): Promise<void> {
const parsed = parseArgs(args);
if ('help' in parsed) {
console.log(HELP);
return;
}
const rows = await engine.getRecentSalience({
days: parsed.days,
limit: parsed.limit,
slugPrefix: parsed.slugPrefix,
});
if (parsed.json) {
console.log(JSON.stringify(rows, null, 2));
return;
}
if (rows.length === 0) {
console.log('(no pages touched in the salience window)');
return;
}
// Human format: rank | score | emotion | takes | slug — title
const header = `${pad('#', 3)} ${pad('score', 7)} ${pad('emo', 5)} ${pad('takes', 6)} slug — title`;
console.log(header);
console.log('-'.repeat(Math.min(80, header.length)));
rows.forEach((r, i) => {
const score = r.score.toFixed(3);
const emo = r.emotional_weight.toFixed(2);
const takes = String(r.take_count);
console.log(`${pad(String(i + 1), 3)} ${pad(score, 7)} ${pad(emo, 5)} ${pad(takes, 6)} ${r.slug}${r.title}`);
});
}
function pad(s: string, n: number): string {
return s.length >= n ? s : s + ' '.repeat(n - s.length);
}
+94
View File
@@ -0,0 +1,94 @@
/**
* gbrain transcripts — Recent raw conversation transcripts.
*
* Local-only: this command reads `.txt` files from the dream-cycle corpus
* directories. It exists as a CLI surface so humans can trigger the same
* read path the v0.29 `get_recent_transcripts` MCP op uses (which is itself
* gated on remote=false; subagents and MCP/HTTP callers cannot reach it).
*
* Usage:
* gbrain transcripts recent # last 7 days, summaries
* gbrain transcripts recent --days 14
* gbrain transcripts recent --full # full content (capped at 100KB/file)
* gbrain transcripts recent --json
*/
import type { BrainEngine } from '../core/engine.ts';
interface RunOpts {
days?: number;
full?: boolean;
limit?: number;
json?: boolean;
}
function parseArgs(args: string[]): RunOpts | { help: true } {
const opts: RunOpts = {};
for (let i = 0; i < args.length; i++) {
const a = args[i];
if (a === '--help' || a === '-h') return { help: true };
if (a === '--json') { opts.json = true; continue; }
if (a === '--full') { opts.full = true; continue; }
if (a === '--days') {
const n = parseInt(args[++i] ?? '', 10);
if (Number.isFinite(n) && n >= 0) opts.days = n;
continue;
}
if (a === '--limit') {
const n = parseInt(args[++i] ?? '', 10);
if (Number.isFinite(n) && n > 0) opts.limit = n;
continue;
}
}
return opts;
}
const HELP = `Usage: gbrain transcripts recent [options]
Recent raw conversation transcripts (NOT polished reflections). Reads from
the dream-cycle corpus dirs (dream.synthesize.session_corpus_dir and
dream.synthesize.meeting_transcripts_dir).
Options:
--days N Window in days (default 7)
--limit N Max transcripts (default 50)
--full Return full content (default: ~300-char summary). Capped 100KB/file.
--json JSON output for agents
--help, -h Show this help
Note: dream-generated outputs (frontmatter dream_generated: true) are skipped.
`;
export async function runTranscripts(engine: BrainEngine, args: string[]): Promise<void> {
const sub = args[0];
if (sub !== 'recent') {
console.log(HELP);
if (sub && sub !== '--help' && sub !== '-h') process.exitCode = 2;
return;
}
const parsed = parseArgs(args.slice(1));
if ('help' in parsed) {
console.log(HELP);
return;
}
const { listRecentTranscripts } = await import('../core/transcripts.ts');
const rows = await listRecentTranscripts(engine, {
days: parsed.days,
summary: !parsed.full,
limit: parsed.limit,
});
if (parsed.json) {
console.log(JSON.stringify(rows, null, 2));
return;
}
if (rows.length === 0) {
console.log('(no recent transcripts in the corpus dir)');
return;
}
rows.forEach(r => {
const date = r.date ?? r.mtime.slice(0, 10);
console.log(`\n--- ${date} | ${r.path} | ${r.length} bytes ---`);
console.log(r.summary);
});
}
+261
View File
@@ -0,0 +1,261 @@
/**
* v0.29.1 — Backfill effective_date / effective_date_source for existing
* pages.
*
* Migration v38 added the columns; they're NULL for rows imported before
* v0.29.1. This walks every page in keyset-paginated batches, runs the
* `computeEffectiveDate` precedence chain, and UPDATEs in place.
*
* Resumable: stores `last_processed_id` in the `config` table after each
* batch. A killed process can re-run and pick up where it left off without
* re-doing rows. Idempotent: even a full re-walk produces the same writes.
*
* Postgres only sets `SET LOCAL statement_timeout = '600s'` per batch (does
* NOT refuse the migration on low session settings — codex pass-2 #16).
*
* Pure library function — same code path used by the v0_29_1 orchestrator
* AND the `gbrain reindex-frontmatter` CLI command (added in commit 4).
*
* Note: the `import_filename` column stays NULL on backfilled rows. We
* don't have the original filename for pre-v0.29.1 imports (codex pass-1
* finding #6). For `daily/`/`meetings/` slugs whose filename-derived date
* IS in the slug tail, computeEffectiveDate falls through to the slug-tail
* heuristic via `slug.split('/').pop()` in importFromContent's caller path
* — but the orchestrator passes the slug-tail explicitly here so backfilled
* rows behave the same as fresh imports for those prefixes.
*/
import type { BrainEngine } from './engine.ts';
import { computeEffectiveDate } from './effective-date.ts';
import type { EffectiveDateSource } from './types.ts';
const BATCH_SIZE = 1000;
const CHECKPOINT_KEY = 'backfill.effective_date.last_id';
export interface BackfillOpts {
/** Limit total rows touched (testing). Undefined = no cap. */
maxRows?: number;
/** Restart from id=0 even if a checkpoint exists. */
fresh?: boolean;
/** Don't write; report what would happen. */
dryRun?: boolean;
/** Per-batch progress callback. */
onBatch?: (info: { batch: number; lastId: number; rowsTouched: number; cumulative: number }) => void;
/**
* Optional slug-prefix filter (e.g. 'meetings/') so the CLI command can
* scope to a subset. Undefined = no filter.
*/
slugPrefix?: string;
/**
* When true, recompute even if existing effective_date matches what
* the chain would produce. Default false (no-op-on-equal saves writes).
*/
force?: boolean;
}
export interface BackfillResult {
/** Total rows examined across all batches. */
examined: number;
/** Rows where effective_date was actually written (changed or newly computed). */
updated: number;
/** Rows that fell through the chain to 'fallback' (matches updated_at/created_at). */
fallback: number;
/** Final last_processed_id (for resume / debugging). */
lastId: number;
/** Total wall-clock seconds. */
durationSec: number;
}
interface PageRow {
id: number;
slug: string;
frontmatter: unknown;
import_filename: string | null;
effective_date: string | null;
effective_date_source: EffectiveDateSource | null;
created_at: string;
updated_at: string;
}
function parseFrontmatter(raw: unknown): Record<string, unknown> {
if (raw == null) return {};
if (typeof raw === 'string') {
try { return JSON.parse(raw) as Record<string, unknown>; }
catch { return {}; }
}
if (typeof raw === 'object') return raw as Record<string, unknown>;
return {};
}
async function getCheckpoint(engine: BrainEngine, fresh: boolean): Promise<number> {
if (fresh) return 0;
try {
const rows = await engine.executeRaw<{ value: string }>(
`SELECT value FROM config WHERE key = $1 LIMIT 1`,
[CHECKPOINT_KEY],
);
if (rows.length === 0) return 0;
const n = Number(rows[0].value);
return Number.isFinite(n) && n >= 0 ? n : 0;
} catch {
return 0;
}
}
async function setCheckpoint(engine: BrainEngine, lastId: number): Promise<void> {
try {
await engine.executeRaw(
`INSERT INTO config (key, value) VALUES ($1, $2)
ON CONFLICT (key) DO UPDATE SET value = EXCLUDED.value`,
[CHECKPOINT_KEY, String(lastId)],
);
} catch {
// Best effort. Failure to checkpoint just means re-walk on next run;
// doesn't corrupt state.
}
}
async function clearCheckpoint(engine: BrainEngine): Promise<void> {
try {
await engine.executeRaw(`DELETE FROM config WHERE key = $1`, [CHECKPOINT_KEY]);
} catch {
// Same — best effort.
}
}
export async function backfillEffectiveDate(
engine: BrainEngine,
opts: BackfillOpts = {},
): Promise<BackfillResult> {
const start = Date.now();
const slugPrefix = opts.slugPrefix?.replace(/[\\%_]/g, (c) => '\\' + c) ?? null;
let lastId = await getCheckpoint(engine, opts.fresh ?? false);
let examined = 0;
let updated = 0;
let fallback = 0;
let batchNum = 0;
// Per-engine statement_timeout boost. Postgres can wedge on a slow
// batch otherwise; PGLite ignores SET LOCAL outside transactions but
// doesn't have the timeout problem in the first place (single writer).
const isPostgres = engine.kind === 'postgres';
while (true) {
if (opts.maxRows && examined >= opts.maxRows) break;
const limit = opts.maxRows
? Math.min(BATCH_SIZE, opts.maxRows - examined)
: BATCH_SIZE;
// Keyset pagination: WHERE id > last_id ORDER BY id LIMIT N. Single-direction
// walk; safe under concurrent inserts (new rows show up at the tail).
const slugFilter = slugPrefix
? `AND slug LIKE $2 ESCAPE '\\\\'`
: '';
const params: unknown[] = [lastId];
if (slugPrefix) params.push(slugPrefix + '%');
params.push(limit);
const limitParam = `$${params.length}`;
const rows = await engine.executeRaw<PageRow>(
`SELECT id, slug, frontmatter, import_filename, effective_date, effective_date_source, created_at, updated_at
FROM pages
WHERE id > $1 ${slugFilter}
ORDER BY id
LIMIT ${limitParam}`,
params,
);
if (rows.length === 0) break;
examined += rows.length;
let touched = 0;
if (!opts.dryRun) {
// Compute effective_date for each row, then UPDATE in a batch wrapped
// in its own transaction (so SET LOCAL statement_timeout scopes to it).
// postgres.js's `transaction` would be cleaner but we're using executeRaw
// for engine portability; explicit BEGIN/COMMIT does the same on both.
if (isPostgres) {
await engine.executeRaw(`BEGIN`);
await engine.executeRaw(`SET LOCAL statement_timeout = '600s'`);
}
try {
for (const r of rows) {
const fm = parseFrontmatter(r.frontmatter);
const filename = r.import_filename
|| (r.slug.includes('/') ? r.slug.split('/').pop()! : r.slug);
const computed = computeEffectiveDate({
slug: r.slug,
frontmatter: fm,
filename,
updatedAt: new Date(r.updated_at),
createdAt: new Date(r.created_at),
});
// No-op-on-equal: skip the UPDATE if existing matches (saves write
// amplification on re-runs). `force: true` bypasses.
const existingMs = r.effective_date ? new Date(r.effective_date).getTime() : null;
const computedMs = computed.date ? computed.date.getTime() : null;
const datesMatch = existingMs === computedMs;
const sourcesMatch = (r.effective_date_source ?? null) === (computed.source ?? null);
if (!opts.force && datesMatch && sourcesMatch) continue;
await engine.executeRaw(
`UPDATE pages SET effective_date = $1::timestamptz, effective_date_source = $2 WHERE id = $3`,
[computed.date ? computed.date.toISOString() : null, computed.source, r.id],
);
touched++;
if (computed.source === 'fallback') fallback++;
}
if (isPostgres) await engine.executeRaw(`COMMIT`);
} catch (e) {
if (isPostgres) {
try { await engine.executeRaw(`ROLLBACK`); } catch { /* ignore */ }
}
throw e;
}
} else {
// Dry run: still count what WOULD change.
for (const r of rows) {
const fm = parseFrontmatter(r.frontmatter);
const filename = r.import_filename
|| (r.slug.includes('/') ? r.slug.split('/').pop()! : r.slug);
const computed = computeEffectiveDate({
slug: r.slug,
frontmatter: fm,
filename,
updatedAt: new Date(r.updated_at),
createdAt: new Date(r.created_at),
});
const existingMs = r.effective_date ? new Date(r.effective_date).getTime() : null;
const computedMs = computed.date ? computed.date.getTime() : null;
if (existingMs !== computedMs || (r.effective_date_source ?? null) !== (computed.source ?? null)) {
touched++;
}
if (computed.source === 'fallback') fallback++;
}
}
updated += touched;
lastId = rows[rows.length - 1].id;
batchNum++;
if (!opts.dryRun) await setCheckpoint(engine, lastId);
opts.onBatch?.({ batch: batchNum, lastId, rowsTouched: touched, cumulative: examined });
}
// Walk done; clear the checkpoint so the next manual run starts fresh.
if (!opts.dryRun) await clearCheckpoint(engine);
return {
examined,
updated,
fallback,
lastId,
durationSec: (Date.now() - start) / 1000,
};
}
+67 -7
View File
@@ -22,8 +22,9 @@
* │ Phase 6: patterns (v0.23: cross-session themes; │
* │ MUST be after extract so │
* │ graph state is fresh) │
* │ Phase 7: embed --stale (DB writes)
* │ Phase 8: orphans (DB read, report only)
* │ Phase 7: recompute_emotional_weight (v0.29: DB writes)
* │ Phase 8: embed --stale (DB writes)
* │ Phase 9: orphans (DB read, report only) │
* └───────────────────────────────────────────────────────────┘
*
* COORDINATION:
@@ -52,7 +53,7 @@ import { getCliOptions, cliOptsToProgressOptions } from './cli-options.ts';
// ─── Types ─────────────────────────────────────────────────────────
export type CyclePhase = 'lint' | 'backlinks' | 'sync' | 'synthesize' | 'extract' | 'patterns' | 'embed' | 'orphans' | 'purge';
export type CyclePhase = 'lint' | 'backlinks' | 'sync' | 'synthesize' | 'extract' | 'patterns' | 'recompute_emotional_weight' | 'embed' | 'orphans' | 'purge';
export const ALL_PHASES: CyclePhase[] = [
'lint',
@@ -61,6 +62,9 @@ export const ALL_PHASES: CyclePhase[] = [
'synthesize',
'extract',
'patterns',
// v0.29 — runs AFTER extract + synthesize so it sees the union of
// sync-touched + synthesize-written pages with fresh tag + take state.
'recompute_emotional_weight',
'embed',
'orphans',
// v0.26.5: hard-deletes soft-deleted pages and expired archived sources past
@@ -83,6 +87,8 @@ const NEEDS_LOCK_PHASES: ReadonlySet<CyclePhase> = new Set([
'synthesize',
'extract',
'patterns',
// v0.29 — writes pages.emotional_weight column.
'recompute_emotional_weight',
'embed',
'purge',
]);
@@ -144,6 +150,8 @@ export interface CycleReport {
synth_pages_written: number;
/** v0.23: number of pattern pages written/updated by patterns phase. */
patterns_written: number;
/** v0.29: number of pages whose emotional_weight was (re)computed. */
pages_emotional_weight_recomputed: number;
/** v0.26.5: number of source rows hard-deleted by the purge phase. */
purged_sources_count: number;
/** v0.26.5: number of page rows hard-deleted by the purge phase. */
@@ -888,8 +896,11 @@ export async function runCycle(
}
// ── Phase 3: sync ───────────────────────────────────────────
// Track which slugs sync touched so extract can run incrementally.
// Track which slugs sync touched so extract can run incrementally,
// and which slugs synthesize wrote so recompute_emotional_weight can
// pick up the union of (sync synthesize) for v0.29 incremental mode.
let syncPagesAffected: string[] | undefined;
let synthesizeWrittenSlugs: string[] | undefined;
if (phases.includes('sync')) {
checkAborted(opts.signal);
if (!engine) {
@@ -937,6 +948,11 @@ export async function runCycle(
}));
result.duration_ms = duration_ms;
phaseResults.push(result);
// v0.29: capture synthesize-written slugs so the recompute_emotional_weight
// phase can union them with sync's pagesAffected for incremental mode.
if (result.details && Array.isArray(result.details.written_slugs)) {
synthesizeWrittenSlugs = result.details.written_slugs as string[];
}
progress.finish();
}
await safeYield(opts.yieldBetweenPhases);
@@ -995,7 +1011,47 @@ export async function runCycle(
await safeYield(opts.yieldBetweenPhases);
}
// ── Phase 7: embed ──────────────────────────────────────────
// ── Phase 7: recompute_emotional_weight (v0.29) ─────────────
// Runs AFTER extract + synthesize so it sees fresh tags + takes for
// every page touched in this cycle. Incremental mode uses union(sync,
// synthesize); full mode walks every page in the brain.
if (phases.includes('recompute_emotional_weight')) {
checkAborted(opts.signal);
if (!engine) {
phaseResults.push({
phase: 'recompute_emotional_weight',
status: 'skipped',
duration_ms: 0,
summary: 'no database connected',
details: { reason: 'no_database' },
});
} else {
progress.start('cycle.recompute_emotional_weight');
const { runPhaseRecomputeEmotionalWeight } = await import('./cycle/recompute-emotional-weight.ts');
// Determine incremental vs full mode. If sync OR synthesize ran in this
// cycle, do incremental over their union. If neither phase ran (e.g.,
// user passed `--phase recompute_emotional_weight`), do full walk.
const incremental: string[] | undefined =
(syncPagesAffected || synthesizeWrittenSlugs)
? Array.from(new Set([
...(syncPagesAffected ?? []),
...(synthesizeWrittenSlugs ?? []),
]))
: undefined;
const { result, duration_ms } = await timePhase(() =>
runPhaseRecomputeEmotionalWeight(engine, {
dryRun,
affectedSlugs: incremental,
}),
);
result.duration_ms = duration_ms;
phaseResults.push(result);
progress.finish();
}
await safeYield(opts.yieldBetweenPhases);
}
// ── Phase 8: embed ──────────────────────────────────────────
if (phases.includes('embed')) {
checkAborted(opts.signal);
if (!engine) {
@@ -1016,7 +1072,7 @@ export async function runCycle(
await safeYield(opts.yieldBetweenPhases);
}
// ── Phase 8: orphans ────────────────────────────────────────
// ── Phase 9: orphans ────────────────────────────────────────
if (phases.includes('orphans')) {
checkAborted(opts.signal);
if (!engine) {
@@ -1094,6 +1150,7 @@ function emptyTotals(): CycleReport['totals'] {
transcripts_processed: 0,
synth_pages_written: 0,
patterns_written: 0,
pages_emotional_weight_recomputed: 0,
purged_sources_count: 0,
purged_pages_count: 0,
};
@@ -1123,6 +1180,8 @@ function extractTotals(phases: PhaseResult[]): CycleReport['totals'] {
t.synth_pages_written = Number(p.details.pages_written ?? 0);
} else if (p.phase === 'patterns' && p.details) {
t.patterns_written = Number(p.details.patterns_written ?? 0);
} else if (p.phase === 'recompute_emotional_weight' && p.details) {
t.pages_emotional_weight_recomputed = Number(p.details.pages_recomputed ?? 0);
} else if (p.phase === 'purge' && p.details) {
t.purged_sources_count = Number(p.details.purged_sources_count ?? 0);
t.purged_pages_count = Number(p.details.purged_pages_count ?? 0);
@@ -1144,6 +1203,7 @@ function deriveStatus(phases: PhaseResult[], totals: CycleReport['totals']): Cyc
totals.backlinks_added > 0 ||
totals.pages_synced > 0 ||
totals.pages_extracted > 0 ||
totals.pages_embedded > 0;
totals.pages_embedded > 0 ||
totals.pages_emotional_weight_recomputed > 0;
return anyWork ? 'ok' : 'clean';
}
+128
View File
@@ -0,0 +1,128 @@
/**
* v0.29 — Anomaly detection: statistical helpers for the `find_anomalies` op.
*
* Pure functions over densified daily-count buckets. The engine layer runs the
* SQL (CTE-shaped, with `generate_series` zero-fill so rare cohorts don't get
* sparse-day biased baselines per codex C4#6) and hands the results here. This
* keeps `findAnomalies` mostly testable without a database.
*
* Cohort kinds: tag, type. Year cohort is deferred to v0.30 pending proper
* frontmatter date-field detection.
*/
import type { AnomalyResult } from '../types.ts';
/** One row of the densified daily-count series for a single cohort key. */
export interface CohortDayRow {
cohort_kind: 'tag' | 'type';
cohort_value: string;
/** ISO date (YYYY-MM-DD). */
day: string;
/** Distinct pages touched in this cohort on `day`. Zero if no activity. */
count: number;
}
/** "Today" current-window count per cohort plus the page slugs that drove it. */
export interface CohortTodayRow {
cohort_kind: 'tag' | 'type';
cohort_value: string;
count: number;
page_slugs: string[];
}
/**
* Mean and (sample) stddev of a number array. Returns `(0, 0)` for empty
* input. Uses the sample stddev (n-1 denominator) so a single-sample baseline
* doesn't claim zero variance.
*/
export function meanStddev(samples: number[]): { mean: number; stddev: number } {
if (samples.length === 0) return { mean: 0, stddev: 0 };
const sum = samples.reduce((a, b) => a + b, 0);
const mean = sum / samples.length;
if (samples.length === 1) return { mean, stddev: 0 };
const sqSum = samples.reduce((a, b) => a + (b - mean) * (b - mean), 0);
const variance = sqSum / (samples.length - 1);
return { mean, stddev: Math.sqrt(variance) };
}
/**
* Compute anomaly results from densified baseline buckets + today's counts.
*
* For each cohort:
* 1. Compute (mean, stddev) over the baseline daily counts.
* 2. If stddev > 0: anomalous when `today.count > mean + sigma*stddev`.
* sigma_observed = (today.count - mean) / stddev.
* 3. If stddev == 0: small-sample fallback — anomalous when
* `today.count > mean + 1`. sigma_observed treated as a finite proxy
* `today.count - mean` so callers still get a usable sort key.
*
* Cohorts with no baseline rows AND no today rows are skipped. Cohorts
* appearing only in `today` (a brand-new cohort) get a baseline_mean of 0
* — they're surfaced as anomalies whenever today.count >= 2 (mean+1 fallback).
*
* Returns top `limit` rows sorted by `sigma_observed` descending. Each row
* caps `page_slugs` at 50 entries.
*
* @param baseline densified rows over the lookback window, grouped by cohort × day.
* @param today rows for the target day, grouped by cohort.
* @param sigma threshold multiplier (default 3.0).
* @param limit max anomalies to return (default 20).
*/
export function computeAnomaliesFromBuckets(
baseline: CohortDayRow[],
today: CohortTodayRow[],
sigma: number,
limit: number = 20,
): AnomalyResult[] {
// Group baseline samples by (cohort_kind, cohort_value).
const baselineByCohort = new Map<string, number[]>();
for (const row of baseline) {
const key = cohortKey(row.cohort_kind, row.cohort_value);
const list = baselineByCohort.get(key);
if (list) {
list.push(row.count);
} else {
baselineByCohort.set(key, [row.count]);
}
}
const out: AnomalyResult[] = [];
for (const t of today) {
const key = cohortKey(t.cohort_kind, t.cohort_value);
const samples = baselineByCohort.get(key) ?? [];
const { mean, stddev } = meanStddev(samples);
let isAnomaly: boolean;
let sigmaObserved: number;
if (stddev > 0) {
const threshold = mean + sigma * stddev;
isAnomaly = t.count > threshold;
sigmaObserved = (t.count - mean) / stddev;
} else {
// Zero-stddev fallback (or empty baseline). Sigma is undefined; we use
// (count - mean) as a finite sort proxy and require count > mean + 1
// to avoid surfacing every 1-page-touched cohort as anomalous.
isAnomaly = t.count > mean + 1;
sigmaObserved = t.count - mean;
}
if (!isAnomaly) continue;
out.push({
cohort_kind: t.cohort_kind,
cohort_value: t.cohort_value,
count: t.count,
baseline_mean: mean,
baseline_stddev: stddev,
sigma_observed: sigmaObserved,
page_slugs: t.page_slugs.slice(0, 50),
});
}
out.sort((a, b) => b.sigma_observed - a.sigma_observed);
return out.slice(0, limit);
}
function cohortKey(kind: string, value: string): string {
// \x1f (unit separator) — a byte that can't appear in tags or PageType values.
return `${kind}\x1f${value}`;
}
+142
View File
@@ -0,0 +1,142 @@
/**
* v0.29 — Emotional weight: deterministic 0..1 score for each page, computed
* from tags + active takes during the dream cycle's recompute_emotional_weight
* phase. Feeds the salience query (`get_recent_salience`) so pages with high
* emotional weight outrank busy-but-shallow ones in "what's been going on?"
* style queries.
*
* Pure function, no DB. The cycle phase loads inputs in batch via
* `engine.batchLoadEmotionalInputs` and writes results in batch via
* `engine.setEmotionalWeightBatch`.
*
* Tunable: the `HIGH_EMOTION_TAGS` seed list below is the default. Users
* override via the `emotional_weight.high_tags` config key (array of strings).
* See `loadHighEmotionTags` for the resolution path.
*/
/**
* Default high-emotion tag seed list. Pages with any tag in this set get the
* tag-emotion boost in the formula below. Override via config key
* `emotional_weight.high_tags` to add domain-specific tags (e.g. health
* conditions, family member names, project names tied to grief / loss).
*
* Anglocentric and personal-life-biased on purpose: this is the v1 default
* for someone who keeps a personal brain. Override unconditionally at install
* time if your brain is mostly work-life.
*/
export const HIGH_EMOTION_TAGS: ReadonlySet<string> = new Set([
'family',
'marriage',
'wedding',
'loss',
'death',
'grief',
'relationship',
'love',
'mental-health',
'health',
'illness',
'birth',
'children',
'kids',
'parents',
]);
/**
* Holder name treated as "the user" for the Garry-as-holder ratio. Configurable
* via the `emotional_weight.user_holder` config key (defaults to 'garry' to
* match the v0.28 schema's takes table convention).
*/
export const DEFAULT_USER_HOLDER = 'garry';
export interface EmotionalWeightTake {
holder: string;
weight: number;
kind: string;
active: boolean;
}
export interface EmotionalWeightInput {
tags: readonly string[];
takes: readonly EmotionalWeightTake[];
}
export interface EmotionalWeightOpts {
/** Override the default HIGH_EMOTION_TAGS set. Tag matching is case-insensitive. */
highEmotionTags?: ReadonlySet<string>;
/** Override the default user holder name (used in the Garry-as-holder ratio). */
userHolder?: string;
}
/**
* Compute emotional weight in [0..1] from a page's tags + active takes.
*
* Formula (sum capped at 1.0):
* 1) Tag emotion boost max 0.5 (any matching high-emotion tag)
* 2) Take density max 0.3 (0.1 per active take, capped)
* 3) Take avg weight max 0.1 (avg of take.weight, scaled)
* 4) User-holder ratio max 0.1 (active takes by user / total active)
*
* Why these numbers:
* - Tag emotion is the strongest signal (0.5 cap) because tags are an explicit
* user act of categorization. A page tagged `wedding` is *about* something
* emotionally weighty by construction.
* - Take density (0.3) covers the case of pages with no emotion-tag but lots
* of opinions / hot-take attention (Garry's "I have a bunch of takes about
* this person/company" signal).
* - Avg weight (0.1) captures take confidence; high-confidence takes amplify
* density.
* - User-holder ratio (0.1) preserves the personal-vs-other distinction.
* A page where Garry has takes outweighs one where only third-party holders
* are recorded.
*
* Returns exactly 0.0 for empty inputs (no tags, no takes) so default-row
* behavior survives the formula.
*/
export function computeEmotionalWeight(
input: EmotionalWeightInput,
opts: EmotionalWeightOpts = {},
): number {
const tagSet = opts.highEmotionTags ?? HIGH_EMOTION_TAGS;
const userHolder = (opts.userHolder ?? DEFAULT_USER_HOLDER).toLowerCase();
const tags = input.tags ?? [];
const allTakes = input.takes ?? [];
const takes = allTakes.filter((t) => t.active);
// 1) Tag emotion boost — case-insensitive match.
let tagBoost = 0;
for (const t of tags) {
if (tagSet.has(t.toLowerCase())) {
tagBoost = 0.5;
break;
}
}
// 2) Take density: 0.1 per active take, capped at 0.3.
const takeDensity = Math.min(takes.length * 0.1, 0.3);
// 3) Take avg weight, scaled into 0..0.1.
let takeAvgWeight = 0;
if (takes.length > 0) {
const sum = takes.reduce((acc, t) => acc + clamp01(t.weight), 0);
takeAvgWeight = (sum / takes.length) * 0.1;
}
// 4) User-holder ratio over active takes, scaled into 0..0.1.
let userHolderRatio = 0;
if (takes.length > 0) {
const userTakes = takes.filter((t) => t.holder?.toLowerCase() === userHolder).length;
userHolderRatio = (userTakes / takes.length) * 0.1;
}
const total = tagBoost + takeDensity + takeAvgWeight + userHolderRatio;
return Math.max(0, Math.min(1, total));
}
function clamp01(n: number): number {
if (!Number.isFinite(n)) return 0;
if (n < 0) return 0;
if (n > 1) return 1;
return n;
}
@@ -0,0 +1,134 @@
/**
* v0.29 — Recompute emotional weight phase. Runs AFTER extract + synthesize so
* it sees the union of (sync-touched, synthesize-written) pages with fresh
* tag + take state. Pure deterministic computation; no LLM calls.
*
* Two SQL round-trips total regardless of brain size (codex C4#3, C4#4):
* 1. batchLoadEmotionalInputs — single CTE-shaped read with per-table
* pre-aggregates so a page × N tags × M takes never produces N×M rows.
* 2. setEmotionalWeightBatch — composite-keyed (slug, source_id) UPDATE
* FROM unnest so multi-source brains can't get cross-source fan-out.
*
* In incremental mode (`affectedSlugs` non-empty), only those pages are
* touched. In full mode (`affectedSlugs` undefined or null) every page in
* the brain is recomputed — this is the path users hit on first upgrade
* via `gbrain dream --phase recompute_emotional_weight`.
*
* Target wall-clock budget: <5s on a 1000-page fixture; <60s on a 50K-page
* real brain. Catastrophic-exception path returns a 'fail' PhaseResult so
* the cycle continues to the next phase.
*/
import type { BrainEngine } from '../engine.ts';
import type { PhaseResult, PhaseError } from '../cycle.ts';
import { computeEmotionalWeight } from './emotional-weight.ts';
import type { GBrainConfig } from '../config.ts';
export interface RecomputeEmotionalWeightOpts {
/** When false, the phase reads + computes but skips the UPDATE. */
dryRun?: boolean;
/**
* Slugs to recompute. Undefined / empty array = full brain recompute.
* Caller passes `union(syncPagesAffected, synthesizeWrittenSlugs)` for
* the incremental path.
*/
affectedSlugs?: string[];
/** GBrain config for high_emotion_tags + user_holder overrides. */
config?: GBrainConfig;
}
export interface RecomputeEmotionalWeightResult extends PhaseResult {
/** Number of pages whose emotional_weight was (re)computed. */
pages_recomputed: number;
}
export async function runPhaseRecomputeEmotionalWeight(
engine: BrainEngine,
opts: RecomputeEmotionalWeightOpts,
): Promise<RecomputeEmotionalWeightResult> {
const start = Date.now();
try {
// Resolve override tag list + user-holder from config (optional).
const overrideTags = await engine.getConfig('emotional_weight.high_tags');
const userHolder = await engine.getConfig('emotional_weight.user_holder');
let highEmotionTags: ReadonlySet<string> | undefined;
if (overrideTags) {
try {
const parsed = JSON.parse(overrideTags) as unknown;
if (Array.isArray(parsed) && parsed.every(t => typeof t === 'string')) {
highEmotionTags = new Set(parsed.map(t => t.toLowerCase()));
}
} catch {
// Bad JSON — fall back to default seed list. The doctor check
// (added separately) will surface the parse error.
}
}
// Incremental path: empty array means "no changes touched" — record
// a zero-work success and return without touching the DB.
if (Array.isArray(opts.affectedSlugs) && opts.affectedSlugs.length === 0) {
return result('ok', 'recompute_emotional_weight (incremental, 0 slugs)', 0, {
mode: 'incremental',
pages_recomputed: 0,
}, start);
}
const inputs = await engine.batchLoadEmotionalInputs(opts.affectedSlugs);
const writes = inputs.map(row => ({
slug: row.slug,
source_id: row.source_id,
weight: computeEmotionalWeight(
{ tags: row.tags, takes: row.takes },
{ highEmotionTags, userHolder: userHolder ?? undefined },
),
}));
if (opts.dryRun) {
return result('ok', `recompute_emotional_weight (dry-run, ${writes.length} pages)`, writes.length, {
mode: opts.affectedSlugs ? 'incremental' : 'full',
pages_recomputed: writes.length,
dry_run: true,
}, start);
}
const updated = await engine.setEmotionalWeightBatch(writes);
return result('ok', `recompute_emotional_weight (${updated} pages)`, updated, {
mode: opts.affectedSlugs ? 'incremental' : 'full',
pages_recomputed: updated,
}, start);
} catch (e) {
const msg = e instanceof Error ? e.message : String(e);
const err: PhaseError = {
class: 'InternalError',
code: 'RECOMPUTE_EMOTIONAL_WEIGHT_FAIL',
message: msg || 'recompute_emotional_weight phase threw',
};
return {
phase: 'recompute_emotional_weight',
status: 'fail',
duration_ms: Date.now() - start,
summary: 'recompute_emotional_weight failed',
details: { error: err },
error: err,
pages_recomputed: 0,
};
}
}
function result(
status: 'ok',
summary: string,
pagesRecomputed: number,
details: Record<string, unknown>,
start: number,
): RecomputeEmotionalWeightResult {
return {
phase: 'recompute_emotional_weight',
status,
duration_ms: Date.now() - start,
summary,
details,
pages_recomputed: pagesRecomputed,
};
}
+4
View File
@@ -243,6 +243,10 @@ export async function runPhaseSynthesize(
transcripts_discovered: transcripts.length,
transcripts_processed: worthProcessing.length,
pages_written: writtenSlugs.length,
// v0.29: emit the slug list so the recompute_emotional_weight phase can
// union with sync's pagesAffected and recompute weights for every page
// synthesize wrote in this cycle.
written_slugs: writtenSlugs,
reverse_write_count: reverseWriteCount,
child_outcomes: childOutcomes,
summary_slug: summarySlug,
+158
View File
@@ -0,0 +1,158 @@
/**
* v0.29.1 — Compute a page's effective_date from frontmatter precedence.
*
* The "effective date" is the answer to "when was this page about?" It's
* NOT updated_at (which churns from auto-link) and NOT created_at (which
* is the row insert time). It's the user's stated content date.
*
* Precedence chain (default order):
* 1. frontmatter.event_date — meeting / event pages
* 2. frontmatter.date — dated essays
* 3. frontmatter.published — writing/
* 4. filename-date — leading YYYY-MM-DD in basename
* 5. updated_at — fallback
* 6. created_at — last resort (only if updated_at NULL)
*
* Per-prefix override: for `daily/` and `meetings/` slug prefixes, the
* filename-date jumps to position 1 — the filename is the user's primary
* signal there ("daily/2024-03-15.md" the FILE date matters more than any
* frontmatter the user pasted).
*
* Returns BOTH the parsed Date and the source label so the doctor's
* `effective_date_health` check can detect "fell back to updated_at" rows
* that look populated but are functionally equivalent to a NULL.
*
* Range validation: parsed value must be in [1990-01-01, NOW + 1 year].
* Out-of-range values are dropped (the chain falls through to the next
* element). NaN / unparseable strings drop the same way.
*
* Pure function. No DB. Tested in test/effective-date.test.ts.
*/
import type { EffectiveDateSource } from './types.ts';
export interface EffectiveDateResult {
date: Date | null;
source: EffectiveDateSource | null;
}
export interface ComputeEffectiveDateOpts {
slug: string;
frontmatter: Record<string, unknown>;
/** Basename without extension, e.g. "2024-03-15-acme-call". May be null/empty. */
filename?: string | null;
updatedAt: Date;
createdAt: Date;
}
/**
* Slug prefixes where the filename date wins over frontmatter dates. The
* user's primary signal in these directories is the filename, not arbitrary
* frontmatter the importer might have copied.
*
* Hardcoded in v0.29.1 (commit 2). v0.29.1 commit 5 introduces the
* recency-decay map; we could move this list there if we wanted user-tunable
* filename-first prefixes, but the daily/ + meetings/ defaults are stable
* enough that hardcoding is correct.
*/
const FILENAME_FIRST_PREFIXES = ['daily/', 'meetings/'];
const MIN_DATE_MS = Date.UTC(1990, 0, 1);
const FILENAME_DATE_RE = /^(\d{4}-\d{2}-\d{2})/;
function maxDateMs(): number {
// NOW + 1 year, computed at call time so tests with a mocked Date.now()
// see a moving boundary. Pages dated > 1 year in the future are almost
// always corrupt (epoch math gone wrong, typoed century, bad parse).
return Date.now() + 365 * 24 * 60 * 60 * 1000;
}
/** Parse a frontmatter value as a Date. Accepts Date instances, ISO strings, YYYY-MM-DD. Returns null on any failure. */
export function parseDateLoose(value: unknown): Date | null {
if (value == null) return null;
if (value instanceof Date) {
return Number.isFinite(value.getTime()) ? value : null;
}
if (typeof value === 'string') {
const trimmed = value.trim();
if (trimmed === '') return null;
const ms = Date.parse(trimmed);
if (!Number.isFinite(ms)) return null;
return new Date(ms);
}
if (typeof value === 'number') {
// Plausibility: numbers are usually ms since epoch but YAML can yield
// bare integers (year? month? day?) — accept only if the resulting Date
// falls inside the valid window. validateInRange catches the rest.
return Number.isFinite(value) ? new Date(value) : null;
}
return null;
}
function validateInRange(d: Date | null): Date | null {
if (d === null) return null;
const ms = d.getTime();
if (!Number.isFinite(ms)) return null;
if (ms < MIN_DATE_MS) return null;
if (ms > maxDateMs()) return null;
return d;
}
function extractFilenameDate(filename: string | null | undefined): Date | null {
if (!filename) return null;
const m = filename.match(FILENAME_DATE_RE);
if (!m) return null;
return validateInRange(parseDateLoose(m[1]));
}
function hasFilenameFirstPrefix(slug: string): boolean {
for (const p of FILENAME_FIRST_PREFIXES) {
if (slug.startsWith(p)) return true;
}
return false;
}
/**
* Run the precedence chain. Returns the first valid (in-range) date and its
* source label. Falls all the way through to updated_at / created_at as
* 'fallback' when nothing in frontmatter or filename parses.
*/
export function computeEffectiveDate(opts: ComputeEffectiveDateOpts): EffectiveDateResult {
const { slug, frontmatter, filename, updatedAt, createdAt } = opts;
const filenameFirst = hasFilenameFirstPrefix(slug);
const fmEvent = validateInRange(parseDateLoose(frontmatter.event_date));
const fmDate = validateInRange(parseDateLoose(frontmatter.date));
const fmPublished = validateInRange(parseDateLoose(frontmatter.published));
const filenameDate = extractFilenameDate(filename);
// Build the ordered candidate list. For filename-first prefixes
// (daily/, meetings/) the filename moves to the head of the chain.
const candidates: Array<{ date: Date | null; source: EffectiveDateSource }> = filenameFirst
? [
{ date: filenameDate, source: 'filename' },
{ date: fmEvent, source: 'event_date' },
{ date: fmDate, source: 'date' },
{ date: fmPublished, source: 'published' },
]
: [
{ date: fmEvent, source: 'event_date' },
{ date: fmDate, source: 'date' },
{ date: fmPublished, source: 'published' },
{ date: filenameDate, source: 'filename' },
];
for (const c of candidates) {
if (c.date !== null) return { date: c.date, source: c.source };
}
// Fallback chain: updated_at, then created_at. Both are guaranteed
// non-null by the schema; the validation here is defensive against bad
// test fixtures.
const upd = validateInRange(updatedAt);
if (upd !== null) return { date: upd, source: 'fallback' };
const cre = validateInRange(createdAt);
if (cre !== null) return { date: cre, source: 'fallback' };
return { date: null, source: null };
}
+86
View File
@@ -12,6 +12,8 @@ import type {
CodeEdgeInput, CodeEdgeResult,
EvalCandidate, EvalCandidateInput,
EvalCaptureFailure, EvalCaptureFailureReason,
SalienceOpts, SalienceResult, AnomaliesOpts, AnomalyResult,
EmotionalWeightInputRow, EmotionalWeightWriteRow,
} from './types.ts';
/**
@@ -411,6 +413,35 @@ export interface BrainEngine {
* Slugs with zero inbound links are present in the map with value 0.
*/
getBacklinkCounts(slugs: string[]): Promise<Map<string, number>>;
/**
* v0.27.0: for a list of slugs, return their updated_at timestamps (or created_at fallback).
* Used by hybrid search recency boost. Single SQL query, not N+1.
* Slugs with no timestamp get no entry in the map.
*
* @deprecated v0.29.1: prefer getEffectiveDates (composite-keyed, multi-source-safe).
* Kept for back-compat with PR #618 callers.
*/
getPageTimestamps(slugs: string[]): Promise<Map<string, Date>>;
/**
* v0.29.1: for a list of (slug, source_id) refs, return COALESCE(effective_date,
* updated_at) per ref. Single SQL query. Composite-keyed map (key format:
* `${source_id}::${slug}`) so multi-source brains don't conflate pages with
* the same slug across sources (codex pass-1 finding #3).
*
* Drives the new applyRecencyBoost post-fusion stage. Returns NULL for refs
* with no row; map omits them.
*/
getEffectiveDates(refs: Array<{slug: string; source_id: string}>): Promise<Map<string, Date>>;
/**
* v0.29.1: for a list of (slug, source_id) refs, return the salience score
* (emotional_weight × 5 + ln(1 + take_count)) per ref. Single SQL query.
* Composite-keyed (`${source_id}::${slug}`) like getEffectiveDates.
*
* Drives the new applySalienceBoost post-fusion stage. Pages with no row
* (or zero emotional_weight + zero takes) get score = 0; the boost stage
* skips them.
*/
getSalienceScores(refs: Array<{slug: string; source_id: string}>): Promise<Map<string, number>>;
/**
* Return every page with no inbound links (from any source).
* Domain comes from the frontmatter `domain` field (null if unset).
@@ -642,4 +673,59 @@ export interface BrainEngine {
logEvalCaptureFailure(reason: EvalCaptureFailureReason): Promise<void>;
/** Read capture failures within an optional time window. Used by `gbrain doctor`. */
listEvalCaptureFailures(filter?: { since?: Date }): Promise<EvalCaptureFailure[]>;
// ============================================================
// v0.29 — Salience + Anomaly Detection
// ============================================================
// The brain surfaces what's unusual and emotionally charged without being
// asked. Cost: ~zero at query time (deterministic SQL), with backfill done
// during the new `recompute_emotional_weight` cycle phase.
/**
* Batch-load tag + take inputs for the emotional-weight formula. One CTE-shaped
* query: `pages` LEFT JOIN aggregated `tags` and aggregated `takes` (each
* pre-aggregated in its own CTE so the page × N tags × M takes cartesian
* product is avoided).
*
* If `slugs` is undefined, returns inputs for every page in the brain
* (full-mode backfill). If provided, returns only matching slugs (incremental
* recompute after sync / synthesize touched specific pages).
*
* Multi-source-aware: each row carries its `source_id` so the matching
* `setEmotionalWeightBatch` UPDATE can composite-key correctly.
*/
batchLoadEmotionalInputs(slugs?: string[]): Promise<EmotionalWeightInputRow[]>;
/**
* Apply pre-computed emotional weights in a single UPDATE. Composite-keyed
* on `(slug, source_id)` because `pages.slug` is only unique within a
* source — a slug-only UPDATE would fan out across sources, the same bug
* that the v0.18.0 link batches fixed for cross-source edges.
*
* Returns the count of rows actually updated. Pages whose `(slug, source_id)`
* tuple doesn't exist (race with delete) are silently skipped.
*/
setEmotionalWeightBatch(rows: EmotionalWeightWriteRow[]): Promise<number>;
/**
* Salience query: pages recently touched, ranked by a deterministic
* `(emotional_weight * 5) + ln(1 + take_count) + recency_decay` score.
*
* The handler computes the time boundary in JS (`now - days * 86400000`)
* and binds it as TIMESTAMPTZ so the SQL is identical across PGLite +
* Postgres (eng review D5 — avoids dialect drift on `interval` binding).
*/
getRecentSalience(opts: SalienceOpts): Promise<SalienceResult[]>;
/**
* Anomaly detection: cohorts (tag, type) with unusually-high page activity
* on a target day vs baseline mean+stddev over the previous N days. Year
* cohort is deferred to v0.30 (slug-regex year extraction is fragile).
*
* Baseline densifies the day series via `generate_series` zero-fill so
* sparse-day rare cohorts don't look "normally active" — a sparse-day cohort
* with one touch in 30 days has a low baseline mean and high sigma at 7 touches,
* not a misleading mean of 1.
*/
findAnomalies(opts: AnomaliesOpts): Promise<AnomalyResult[]>;
}
+35 -2
View File
@@ -11,6 +11,7 @@ import { extractCodeRefs, imageOfCandidates } from './link-extraction.ts';
import { embedBatch, embedMultimodal } from './embedding.ts';
import { slugifyPath, slugifyCodePath, isCodeFilePath } from './sync.ts';
import type { ChunkInput, PageInput, PageType } from './types.ts';
import { computeEffectiveDate } from './effective-date.ts';
/**
* v0.20.0 Cathedral II Layer 8 D2 — markdown fence extraction helper.
@@ -185,7 +186,15 @@ export async function importFromContent(
engine: BrainEngine,
slug: string,
content: string,
opts: { noEmbed?: boolean } = {},
opts: {
noEmbed?: boolean;
/**
* v0.29.1: basename without extension for filename-date precedence on
* `daily/`, `meetings/` slugs. importFromFile threads this from the
* disk path; the put_page MCP op derives it from the slug tail.
*/
filename?: string;
} = {},
): Promise<ImportResult> {
// Reject oversized payloads before any parsing, chunking, or embedding happens.
// Uses Buffer.byteLength to count UTF-8 bytes the same way disk size would,
@@ -263,6 +272,23 @@ export async function importFromContent(
await engine.transaction(async (tx) => {
if (existing) await tx.createVersion(slug);
// v0.29.1 — compute effective_date from frontmatter precedence chain.
// Filename comes from importFromFile path (basename) or the slug tail
// (put_page MCP op fallback). updatedAt/createdAt use the existing
// page's timestamps when present; otherwise NOW() (the row about to
// be created). The result drives the recency boost and since/until
// filters when callers opt in; nothing in the default search path
// consults it.
const filenameForChain = opts.filename ?? slug.split('/').pop() ?? slug;
const nowDate = new Date();
const { date: effectiveDate, source: effectiveDateSource } = computeEffectiveDate({
slug,
frontmatter: parsed.frontmatter,
filename: filenameForChain,
updatedAt: existing?.updated_at ?? nowDate,
createdAt: existing?.created_at ?? nowDate,
});
await tx.putPage(slug, {
type: parsed.type,
title: parsed.title,
@@ -270,6 +296,9 @@ export async function importFromContent(
timeline: parsed.timeline || '',
frontmatter: parsed.frontmatter,
content_hash: hash,
effective_date: effectiveDate,
effective_date_source: effectiveDateSource,
import_filename: filenameForChain,
});
// Tag reconciliation: remove stale, add current
@@ -386,7 +415,11 @@ export async function importFromFile(
// Pass the path-derived slug explicitly so that any future change to
// parseMarkdown's precedence rules cannot re-introduce this bug.
return importFromContent(engine, expectedSlug, content, opts);
// v0.29.1: thread the basename (without extension) for filename-date
// precedence in computeEffectiveDate. e.g. `daily/2024-03-15.md` →
// filename `2024-03-15`.
const fileBasename = basename(relativePath, '.md');
return importFromContent(engine, expectedSlug, content, { ...opts, filename: fileBasename });
}
/**
+137
View File
@@ -1798,6 +1798,143 @@ export const MIGRATIONS: Migration[] = [
}
},
},
{
version: 40,
name: 'pages_emotional_weight',
// v0.29 — Salience + Anomaly Detection.
//
// Adds the `emotional_weight` column to pages. Populated by the new
// `recompute_emotional_weight` cycle phase from tags + takes (deterministic;
// no LLM). Default 0.0 so freshly imported pages don't pollute salience
// ranking before the cycle has run; users run `gbrain dream --phase
// recompute_emotional_weight` once after upgrading to backfill.
//
// No index: the salience query orders by a computed score (emotional_weight,
// take_count, recency-decay), not by raw emotional_weight. Add an index
// later only if a query orders by the raw column directly.
//
// Postgres ADD COLUMN with a constant DEFAULT is metadata-only on PG 11+
// and PGLite (PG 17.5 via WASM) — instant on tables of any size.
sql: `
ALTER TABLE pages
ADD COLUMN IF NOT EXISTS emotional_weight REAL NOT NULL DEFAULT 0.0;
`,
},
{
version: 41,
name: 'pages_recency_columns',
sql: '',
// v0.29.1 — Salience-and-Recency, additive opt-in.
//
// Four new pages columns (all nullable, additive only, no behavior change
// in the default search path; only consulted when a caller opts into
// `salience='on'` / `recency='on'` or the new `since`/`until` filter):
//
// effective_date — content date (event_date / date / published /
// filename-date / fallback). Read by the new
// recency boost and date-filter paths only.
// Auto-link doesn't touch it (immune to
// updated_at churn).
// effective_date_source — sentinel for the doctor's effective_date_health
// check ('event_date' | 'date' | 'published' |
// 'filename' | 'fallback'). The 'fallback' value
// is what surfaces "page that fell back to
// updated_at when frontmatter was unparseable".
// import_filename — basename without extension, captured at import.
// computeEffectiveDate uses it for filename-date
// precedence (daily/, meetings/ prefixes). Older
// rows leave it NULL; backfill falls through.
// salience_touched_at — bumped by recompute_emotional_weight when
// emotional_weight changes. Salience window
// uses GREATEST(updated_at, salience_touched_at)
// so newly-salient old pages enter the recent
// salience query.
//
// Plus an expression index used by since/until filters that read
// COALESCE(effective_date, updated_at). Partial-index claim from earlier
// plan iterations was wrong (codex pass-2 #15) — the planner won't use a
// partial index for the negative side of a COALESCE; expression index does.
//
// CONCURRENTLY + pre-drop guard (mirror of v34) on Postgres; plain CREATE
// INDEX on PGLite via the handler branching on engine.kind.
handler: async (engine) => {
// 1. ADD COLUMN x4. ALTER TABLE ADD COLUMN IF NOT EXISTS is idempotent.
// No defaults, all nullable, all metadata-only on PG 11+ and PGLite.
await engine.runMigration(38, `
ALTER TABLE pages ADD COLUMN IF NOT EXISTS effective_date TIMESTAMPTZ;
ALTER TABLE pages ADD COLUMN IF NOT EXISTS effective_date_source TEXT;
ALTER TABLE pages ADD COLUMN IF NOT EXISTS import_filename TEXT;
ALTER TABLE pages ADD COLUMN IF NOT EXISTS salience_touched_at TIMESTAMPTZ;
`);
// 2. Expression index for since/until date-range filters.
if (engine.kind === 'postgres') {
// Pre-drop any invalid index from a prior CONCURRENTLY failure.
await engine.runMigration(38, `
DO $$ BEGIN
IF EXISTS (
SELECT 1 FROM pg_index i
JOIN pg_class c ON c.oid = i.indexrelid
WHERE c.relname = 'pages_coalesce_date_idx' AND NOT i.indisvalid
) THEN
EXECUTE 'DROP INDEX CONCURRENTLY IF EXISTS pages_coalesce_date_idx';
END IF;
END $$;
`);
await engine.runMigration(38, `
CREATE INDEX CONCURRENTLY IF NOT EXISTS pages_coalesce_date_idx
ON pages ((COALESCE(effective_date, updated_at)));
`);
} else {
await engine.runMigration(38, `
CREATE INDEX IF NOT EXISTS pages_coalesce_date_idx
ON pages ((COALESCE(effective_date, updated_at)));
`);
}
},
// CONCURRENTLY on Postgres requires no surrounding transaction.
transaction: false,
},
{
version: 42,
name: 'eval_candidates_recency_capture',
// v0.29.1 — capture agent-explicit recency + salience choices for replay
// reproducibility (D11 codex resolution).
//
// Without these fields, `gbrain eval replay` cannot reproduce a captured
// run: the live behavior depends on the resolved {salience, recency}
// values, which are absent from v0.29.0's eval_candidates schema. Replays
// of agent-explicit choices drift the same way as_of_ts replays drifted
// before being captured.
//
// All columns are nullable + additive. Pre-v0.29.1 rows stay valid. The
// NDJSON `schema_version` STAYS at 1 — the new fields are optional, and
// gbrain-evals consumers that don't know about them ignore them
// (standard permissive deserialization). No cross-repo coordination
// required (codex pass-1 #C2 dissolved).
//
// as_of_ts — brain's logical NOW at capture (replay uses
// this instead of wall-clock so old captures
// reproduce identically against today's brain).
// salience_param — what the caller passed (or NULL if omitted).
// recency_param — same for recency.
// salience_resolved — final value applied ('off' / 'on' / 'strong').
// recency_resolved — same for recency.
// salience_source — 'caller' or 'auto_heuristic'.
// recency_source — same for recency.
//
// ADD COLUMN with no DEFAULT is metadata-only on PG 11+ and PGLite —
// instant on tables of any size.
sql: `
ALTER TABLE eval_candidates ADD COLUMN IF NOT EXISTS as_of_ts TIMESTAMPTZ;
ALTER TABLE eval_candidates ADD COLUMN IF NOT EXISTS salience_param TEXT;
ALTER TABLE eval_candidates ADD COLUMN IF NOT EXISTS recency_param TEXT;
ALTER TABLE eval_candidates ADD COLUMN IF NOT EXISTS salience_resolved TEXT;
ALTER TABLE eval_candidates ADD COLUMN IF NOT EXISTS recency_resolved TEXT;
ALTER TABLE eval_candidates ADD COLUMN IF NOT EXISTS salience_source TEXT;
ALTER TABLE eval_candidates ADD COLUMN IF NOT EXISTS recency_source TEXT;
`,
},
];
export const LATEST_VERSION = MIGRATIONS.length > 0
@@ -56,6 +56,13 @@ export const BRAIN_TOOL_ALLOWLIST: ReadonlySet<string> = new Set([
'resolve_slugs',
'get_ingest_log',
'put_page',
// v0.29 — Salience + Anomaly Detection. Both read-only. `get_recent_transcripts`
// is intentionally NOT included: subagent calls always have ctx.remote=true,
// and the v0.29 trust gate rejects remote callers — adding it here would be
// a footgun (subagent calls op, gets permission_denied, looks like a bug).
// The cycle synthesize phase already calls discoverTranscripts directly.
'get_recent_salience',
'find_anomalies',
]);
/** Matches Anthropic's tool-name constraint. No dots. */
+65
View File
@@ -0,0 +1,65 @@
/**
* v0.29 — Tool descriptions, extracted to a constants module so that:
* 1. The exact LLM-facing strings are pinnable in tests
* (`test/operations-descriptions.test.ts`).
* 2. Routing changes ship as data, not buried-in-handler edits.
* 3. The `salience-llm-routing.test.ts` Tier-2 eval has a stable surface
* to load tool definitions from.
*
* Description style:
* - Lead with what the tool does in one short sentence.
* - Include explicit triggers ("Use this when the user asks ...") that
* the LLM tool-selection prompt can match.
* - For redirect hints (query/search → salience), be blunt:
* "Do NOT run a semantic search for these."
*/
// ──────────────────────────────────────────────────────────────────────────────
// New v0.29 ops
// ──────────────────────────────────────────────────────────────────────────────
export const GET_RECENT_SALIENCE_DESCRIPTION =
"Returns pages recently touched and ranked by emotional + activity salience " +
"(deterministic 0..1 emotional_weight + take density + recency decay). " +
"Use this when the user asks what's been going on, what's notable, what's hot, " +
"anything crazy happening, or for any open-ended 'current state' question " +
"about themselves or their work. Do NOT run a semantic search for these — " +
"salience surfaces what's unusual without needing a search term.";
export const FIND_ANOMALIES_DESCRIPTION =
"Returns statistical anomalies in recent page activity, grouped by cohort " +
"(tag or type). Use this for questions about what stood out, what's unusual, " +
"or what changed recently. Returns explanatory cohorts (e.g. '15 pages tagged " +
"wedding touched on 2026-04-28, baseline 0.3/day') so you can speak about " +
"patterns the user wouldn't have searched for. Cohort kinds: tag, type. " +
"Year cohort is deferred to a later release.";
export const GET_RECENT_TRANSCRIPTS_DESCRIPTION =
"Returns one-line summaries of recent raw conversation transcripts (NOT polished " +
"reflections). Use this FIRST for questions about 'what's going on with me', " +
"'what have I been thinking about', or anything personal/emotional. Raw " +
"transcripts are the canonical source for the user's own state — polished pages " +
"summarize and flatten. Local-only: rejects remote (MCP/HTTP) callers with a " +
"clear permission_denied; call via the gbrain CLI.";
// ──────────────────────────────────────────────────────────────────────────────
// Redirect hints appended to existing op descriptions
// ──────────────────────────────────────────────────────────────────────────────
export const LIST_PAGES_DESCRIPTION =
"List pages with optional filters. " +
"For 'what's recent / what did I touch this week' questions, use list_pages " +
"with sort=updated_desc instead of semantic search.";
export const QUERY_DESCRIPTION =
"Hybrid search with vector + keyword + multi-query expansion. " +
"For personal/emotional questions ('what's going on with me', 'anything notable', " +
"'how am I feeling'), prefer get_recent_salience, find_anomalies, or " +
"get_recent_transcripts. Semantic search returns polished pages and misses " +
"recent activity bursts. Do NOT assume words like 'crazy', 'notable', or 'big' " +
"mean impressive — they often mean difficult or emotionally charged.";
export const SEARCH_DESCRIPTION =
"Keyword search using full-text search. For personal/emotional questions, " +
"prefer get_recent_salience or find_anomalies — they surface activity bursts " +
"without needing a search term.";
+177 -3
View File
@@ -17,6 +17,14 @@ import { captureEvalCandidate, isEvalCaptureEnabled, isEvalScrubEnabled } from '
import type { HybridSearchMeta } from './types.ts';
import { extractPageLinks, isAutoLinkEnabled, isAutoTimelineEnabled, parseTimelineEntries, makeResolver, type UnresolvedFrontmatterRef } from './link-extraction.ts';
import * as db from './db.ts';
import {
GET_RECENT_SALIENCE_DESCRIPTION,
FIND_ANOMALIES_DESCRIPTION,
GET_RECENT_TRANSCRIPTS_DESCRIPTION,
LIST_PAGES_DESCRIPTION,
QUERY_DESCRIPTION,
SEARCH_DESCRIPTION,
} from './operations-descriptions.ts';
// --- Types ---
@@ -726,21 +734,43 @@ const purge_deleted_pages: Operation = {
cliHints: { name: 'purge-deleted' },
};
const LIST_PAGES_SORT_VALUES = ['updated_desc', 'updated_asc', 'created_desc', 'slug'] as const;
type ListPagesSort = typeof LIST_PAGES_SORT_VALUES[number];
const list_pages: Operation = {
name: 'list_pages',
description: 'List pages with optional filters. Soft-deleted pages are hidden by default; pass include_deleted: true to surface them with deleted_at populated.',
description: LIST_PAGES_DESCRIPTION,
params: {
type: { type: 'string', description: 'Filter by page type' },
tag: { type: 'string', description: 'Filter by tag' },
limit: { type: 'number', description: 'Max results (default 50)' },
// v0.29 — surface filter that already exists on PageFilters.
updated_after: {
type: 'string',
description: 'ISO date (YYYY-MM-DD) or full timestamp. Returns pages with updated_at > value.',
},
sort: {
type: 'string',
enum: [...LIST_PAGES_SORT_VALUES],
description: 'Sort order. Default updated_desc (matches pre-v0.29). Options: updated_desc, updated_asc, created_desc, slug.',
},
include_deleted: { type: 'boolean', description: 'v0.26.5: include soft-deleted pages (default: false). Used by restore workflows and operator diagnostics.' },
},
handler: async (ctx, p) => {
// Whitelist the sort enum at the handler before passing to the engine.
// Engines also whitelist via PAGE_SORT_SQL but defending here keeps
// unsupported strings from reaching the SQL layer.
const rawSort = p.sort as string | undefined;
const sort = rawSort && (LIST_PAGES_SORT_VALUES as readonly string[]).includes(rawSort)
? (rawSort as ListPagesSort)
: undefined;
const pages = await ctx.engine.listPages({
type: p.type as any,
tag: p.tag as string,
limit: clampSearchLimit(p.limit as number | undefined, 50, 100),
includeDeleted: (p.include_deleted as boolean) === true,
updated_after: typeof p.updated_after === 'string' ? p.updated_after : undefined,
sort,
});
return pages.map(pg => ({
slug: pg.slug,
@@ -758,7 +788,7 @@ const list_pages: Operation = {
const search: Operation = {
name: 'search',
description: 'Keyword search using full-text search',
description: SEARCH_DESCRIPTION,
params: {
query: { type: 'string', required: true },
limit: { type: 'number', description: 'Max results (default 20)' },
@@ -804,7 +834,7 @@ const search: Operation = {
const query: Operation = {
name: 'query',
description: 'Hybrid search with vector + keyword + multi-query expansion',
description: QUERY_DESCRIPTION,
params: {
// v0.27.1: `query` is no longer strictly required — `--image <path>`
// is the alternative entry point for image-similarity search. The CLI
@@ -827,6 +857,37 @@ const query: Operation = {
// v0.20.0 Cathedral II Layer 7 (A2) / Layer 10 C3: two-pass structural expansion.
near_symbol: { type: 'string', description: 'Anchor retrieval at this qualified symbol name (e.g., BrainEngine.searchKeyword). Enables A2 two-pass.' },
walk_depth: { type: 'number', description: 'Structural walk depth 1-2. Default 0 (off). Expands anchors through code_edges with 1/(1+hop) decay.' },
// v0.29.1 — orthogonal recency + salience axes. YOU (the agent) decide.
salience: {
type: 'string',
enum: ['off', 'on', 'strong'],
description:
"v0.29.1 salience boost — emotional_weight + take_count, NO time component.\n" +
" 'off' — default for entity / canonical / definitional queries\n" +
" 'on' — surface emotionally-weighted + take-rich pages\n" +
" 'strong' — aggressive mattering tilt\n" +
"Omit and gbrain auto-detects from query text. Independent of `recency`.",
},
recency: {
type: 'string',
enum: ['off', 'on', 'strong'],
description:
"v0.29.1 recency boost — per-prefix age decay, NO mattering signal.\n" +
" 'off' — default for canonical truth\n" +
" 'on' — daily/, media/x/, chat/ decay aggressively; concepts/, originals/, writing/ stay evergreen\n" +
" 'strong' — multiplies the recency factor by 1.5 (use for 'today' / 'right now')\n" +
"Omit and gbrain auto-detects. Independent of `salience` (orthogonal axes).",
},
since: {
type: 'string',
description:
"v0.29.1 — filter to pages whose effective_date is >= this. ISO-8601 (YYYY-MM-DD or full timestamp) OR relative ('7d', '2w', '1y'). Replaces deprecated `afterDate`.",
},
until: {
type: 'string',
description:
"v0.29.1 — filter to effective_date <= this. Same format as `since`. Replaces deprecated `beforeDate`. YYYY-MM-DD lands at end-of-day.",
},
},
handler: async (ctx, p) => {
const startedAt = Date.now();
@@ -870,6 +931,11 @@ const query: Operation = {
symbolKind: (p.symbol_kind as string) || undefined,
nearSymbol: (p.near_symbol as string) || undefined,
walkDepth: typeof p.walk_depth === 'number' ? (p.walk_depth as number) : undefined,
// v0.29.1 — agent-explicit recency + salience. Omitted = heuristic defaults.
salience: p.salience as 'off' | 'on' | 'strong' | undefined,
recency: p.recency as 'off' | 'on' | 'strong' | undefined,
since: typeof p.since === 'string' ? p.since : undefined,
until: typeof p.until === 'string' ? p.until : undefined,
onMeta: (m) => { capturedMeta = m; },
});
const latency_ms = Date.now() - startedAt;
@@ -1753,6 +1819,112 @@ const find_orphans: Operation = {
cliHints: { name: 'orphans', hidden: true },
};
// --- v0.29: Salience + Anomaly Detection ---
const get_recent_salience: Operation = {
name: 'get_recent_salience',
description: GET_RECENT_SALIENCE_DESCRIPTION,
scope: 'read',
params: {
days: { type: 'number', description: 'Window in days. Default 14.' },
limit: { type: 'number', description: 'Max results (default 20, capped at 100).' },
slugPrefix: {
type: 'string',
description: "Optional slug-prefix filter, e.g. 'personal' or 'wiki/people'.",
},
recency_bias: {
type: 'string',
enum: ['flat', 'on'],
description:
"v0.29.1: how to weight recency in the salience score.\n" +
" 'flat' (DEFAULT) — v0.29.0 behavior. Every page gets 1/(1+days_old).\n" +
" Stable, predictable; what most callers want.\n" +
" 'on' — Per-prefix decay map. concepts/originals/writing/\n" +
" become evergreen (recency component = 0); daily/,\n" +
" media/x/, chat/ decay aggressively. Use when the\n" +
" user explicitly biases for recency-aware salience\n" +
" ('what's been salient lately' vs 'what matters\n" +
" in this brain regardless of when').",
},
},
handler: async (ctx, p) => {
const recencyBias = p.recency_bias === 'on' ? 'on' : 'flat';
return ctx.engine.getRecentSalience({
days: typeof p.days === 'number' ? p.days : undefined,
limit: typeof p.limit === 'number' ? p.limit : undefined,
slugPrefix: typeof p.slugPrefix === 'string' ? p.slugPrefix : undefined,
recency_bias: recencyBias,
});
},
cliHints: { name: 'salience' },
};
const find_anomalies: Operation = {
name: 'find_anomalies',
description: FIND_ANOMALIES_DESCRIPTION,
scope: 'read',
params: {
since: {
type: 'string',
description: 'ISO date YYYY-MM-DD. Default = today (UTC).',
},
lookback_days: {
type: 'number',
description: 'Days of history for the baseline. Default 30.',
},
sigma: {
type: 'number',
description: 'Sigma threshold. Default 3.0.',
},
},
handler: async (ctx, p) => {
return ctx.engine.findAnomalies({
since: typeof p.since === 'string' ? p.since : undefined,
lookback_days: typeof p.lookback_days === 'number' ? p.lookback_days : undefined,
sigma: typeof p.sigma === 'number' ? p.sigma : undefined,
});
},
cliHints: { name: 'anomalies' },
};
const get_recent_transcripts: Operation = {
name: 'get_recent_transcripts',
description: GET_RECENT_TRANSCRIPTS_DESCRIPTION,
scope: 'read',
// Local-only: rejects HTTP-borne MCP traffic at tool-list time
// (serve-http.ts filters on `localOnly`) AND at runtime via the in-handler
// ctx.remote check. Defense in depth: hidden + rejected.
localOnly: true,
params: {
days: { type: 'number', description: 'Window in days. Default 7.' },
summary: {
type: 'boolean',
description: 'When true (default), return first ~300 chars per transcript. When false, full content (capped at 100 KB per file).',
},
limit: { type: 'number', description: 'Max transcripts (default 50).' },
},
handler: async (ctx, p) => {
// Trust gate (eng review D2 + codex C3): MCP / HTTP callers (`remote=true`)
// are blocked. Local CLI callers (`remote=false`) and the trusted-workspace
// dream cycle pass through. This op is intentionally NOT in the subagent
// allow-list (subagents always run with remote=true; they would always be
// rejected, which is a footgun if the op is visible).
if (ctx.remote === true) {
throw new OperationError(
'permission_denied',
'get_recent_transcripts is local-only — call via the gbrain CLI.',
);
}
const { listRecentTranscripts } = await import('./transcripts.ts');
return listRecentTranscripts(ctx.engine, {
days: typeof p.days === 'number' ? p.days : undefined,
summary: typeof p.summary === 'boolean' ? p.summary : undefined,
limit: typeof p.limit === 'number' ? p.limit : undefined,
});
},
cliHints: { name: 'transcripts', hidden: true },
};
// --- v0.28: whoami + sources management ---
const whoami: Operation = {
@@ -1992,6 +2164,8 @@ export const operations: Operation[] = [
takes_list, takes_search, think,
// v0.28: whoami + scoped sources management
whoami, sources_add, sources_list, sources_remove, sources_status,
// v0.29: Salience + anomalies + recent transcripts
get_recent_salience, find_anomalies, get_recent_transcripts,
];
export const operationsByName = Object.fromEntries(
+346 -13
View File
@@ -28,11 +28,14 @@ import type {
EngineConfig,
EvalCandidate, EvalCandidateInput,
EvalCaptureFailure, EvalCaptureFailureReason,
SalienceOpts, SalienceResult, AnomaliesOpts, AnomalyResult,
EmotionalWeightInputRow, EmotionalWeightWriteRow,
} from './types.ts';
import { validateSlug, contentHash, rowToPage, rowToChunk, rowToSearchResult, takeRowToTake } from './utils.ts';
import { GBrainError } from './types.ts';
import { GBrainError, PAGE_SORT_SQL } from './types.ts';
import { computeAnomaliesFromBuckets } from './cycle/anomaly.ts';
import { resolveBoostMap, resolveHardExcludes } from './search/source-boost.ts';
import { buildSourceFactorCase, buildHardExcludeClause, buildVisibilityClause } from './search/sql-ranking.ts';
import { buildSourceFactorCase, buildHardExcludeClause, buildVisibilityClause, buildRecencyComponentSql } from './search/sql-ranking.ts';
type PGLiteDB = PGlite;
@@ -461,15 +464,20 @@ export class PGLiteEngine implements BrainEngine {
const hash = page.content_hash || contentHash(page);
const frontmatter = page.frontmatter || {};
// v0.18.0 Step 2: source_id relies on the schema DEFAULT 'default' so
// existing callers still target the default source without threading
// a parameter. ON CONFLICT target becomes (source_id, slug) since the
// global UNIQUE(slug) was dropped in migration v17. Step 5+ will
// surface an explicit sourceId param on putPage for multi-source sync.
// v0.18.0 Step 2: source_id relies on the schema DEFAULT 'default'.
// ON CONFLICT target is (source_id, slug); global UNIQUE(slug) dropped in v17.
const pageKind = page.page_kind || 'markdown';
// v0.29.1 — additive opt-in columns. COALESCE(EXCLUDED.x, pages.x)
// preserves existing values when caller omits them (auto-link path,
// code reindex, etc.). Mirrors postgres-engine.ts.
const effectiveDate = page.effective_date instanceof Date
? page.effective_date.toISOString()
: (page.effective_date ?? null);
const effectiveDateSource = page.effective_date_source ?? null;
const importFilename = page.import_filename ?? null;
const { rows } = await this.db.query(
`INSERT INTO pages (slug, type, page_kind, title, compiled_truth, timeline, frontmatter, content_hash, updated_at)
VALUES ($1, $2, $3, $4, $5, $6, $7::jsonb, $8, now())
`INSERT INTO pages (slug, type, page_kind, title, compiled_truth, timeline, frontmatter, content_hash, updated_at, effective_date, effective_date_source, import_filename)
VALUES ($1, $2, $3, $4, $5, $6, $7::jsonb, $8, now(), $9::timestamptz, $10, $11)
ON CONFLICT (source_id, slug) DO UPDATE SET
type = EXCLUDED.type,
page_kind = EXCLUDED.page_kind,
@@ -478,9 +486,12 @@ export class PGLiteEngine implements BrainEngine {
timeline = EXCLUDED.timeline,
frontmatter = EXCLUDED.frontmatter,
content_hash = EXCLUDED.content_hash,
updated_at = now()
RETURNING id, slug, type, title, compiled_truth, timeline, frontmatter, content_hash, created_at, updated_at`,
[slug, page.type, pageKind, page.title, page.compiled_truth, page.timeline || '', JSON.stringify(frontmatter), hash]
updated_at = now(),
effective_date = COALESCE(EXCLUDED.effective_date, pages.effective_date),
effective_date_source = COALESCE(EXCLUDED.effective_date_source, pages.effective_date_source),
import_filename = COALESCE(EXCLUDED.import_filename, pages.import_filename)
RETURNING id, slug, type, title, compiled_truth, timeline, frontmatter, content_hash, created_at, updated_at, effective_date, effective_date_source, import_filename`,
[slug, page.type, pageKind, page.title, page.compiled_truth, page.timeline || '', JSON.stringify(frontmatter), hash, effectiveDate, effectiveDateSource, importFilename]
);
return rowToPage(rows[0] as Record<string, unknown>);
}
@@ -573,9 +584,13 @@ export class PGLiteEngine implements BrainEngine {
params.push(limit, offset);
const limitSql = `LIMIT $${params.length - 1} OFFSET $${params.length}`;
// v0.29: ORDER BY threading via PAGE_SORT_SQL whitelist (no SQL injection).
const sortKey = filters?.sort && PAGE_SORT_SQL[filters.sort] ? filters.sort : 'updated_desc';
const orderBy = PAGE_SORT_SQL[sortKey];
const { rows } = await this.db.query(
`SELECT p.* FROM pages p ${tagJoin} ${whereSql}
ORDER BY p.updated_at DESC ${limitSql}`,
ORDER BY ${orderBy} ${limitSql}`,
params
);
@@ -647,6 +662,18 @@ export class PGLiteEngine implements BrainEngine {
params.push(opts.symbolKind);
extraFilter += ` AND cc.symbol_type = $${params.length}`;
}
// v0.29.1 — since/until date filter (Postgres parity, codex pass-1 #10).
// Reads against COALESCE(effective_date, updated_at) so date filtering
// matches user intent (a meeting was on its event_date, not when it
// got reimported). Same param shape as Postgres engine.
if (opts?.afterDate) {
params.push(opts.afterDate);
extraFilter += ` AND COALESCE(p.effective_date, p.updated_at, p.created_at) > $${params.length}::timestamptz`;
}
if (opts?.beforeDate) {
params.push(opts.beforeDate);
extraFilter += ` AND COALESCE(p.effective_date, p.updated_at, p.created_at) < $${params.length}::timestamptz`;
}
// v0.26.5: visibility filter (soft-deleted + archived-source).
const visibilityClause = buildVisibilityClause('p', 's');
@@ -723,6 +750,15 @@ export class PGLiteEngine implements BrainEngine {
params.push(opts.symbolKind);
extraFilter += ` AND cc.symbol_type = $${params.length}`;
}
// v0.29.1 since/until parity (codex pass-1 #10).
if (opts?.afterDate) {
params.push(opts.afterDate);
extraFilter += ` AND COALESCE(p.effective_date, p.updated_at, p.created_at) > $${params.length}::timestamptz`;
}
if (opts?.beforeDate) {
params.push(opts.beforeDate);
extraFilter += ` AND COALESCE(p.effective_date, p.updated_at, p.created_at) < $${params.length}::timestamptz`;
}
// v0.26.5: visibility filter for the chunk-grain anchor primitive.
const visibilityClause = buildVisibilityClause('p', 's');
@@ -783,6 +819,17 @@ export class PGLiteEngine implements BrainEngine {
params.push(opts.symbolKind);
extraFilter += ` AND cc.symbol_type = $${params.length}`;
}
// v0.29.1 since/until parity (codex pass-1 #10). Filter applied INSIDE
// the inner CTE so HNSW's candidate pool already excludes out-of-range
// pages — preserves pagination contract.
if (opts?.afterDate) {
params.push(opts.afterDate);
extraFilter += ` AND COALESCE(p.effective_date, p.updated_at, p.created_at) > $${params.length}::timestamptz`;
}
if (opts?.beforeDate) {
params.push(opts.beforeDate);
extraFilter += ` AND COALESCE(p.effective_date, p.updated_at, p.created_at) < $${params.length}::timestamptz`;
}
// v0.26.5: visibility filter applied in the inner CTE so HNSW sees the
// same candidate count it always did. See postgres-engine.ts for rationale.
@@ -1314,6 +1361,58 @@ export class PGLiteEngine implements BrainEngine {
return result;
}
async getPageTimestamps(slugs: string[]): Promise<Map<string, Date>> {
if (slugs.length === 0) return new Map();
const { rows } = await this.db.query(
`SELECT slug, COALESCE(updated_at, created_at) as ts
FROM pages WHERE slug = ANY($1::text[])`,
[slugs]
);
return new Map(rows.map((r: any) => [r.slug as string, new Date(r.ts as string)]));
}
async getEffectiveDates(refs: Array<{slug: string; source_id: string}>): Promise<Map<string, Date>> {
if (refs.length === 0) return new Map();
const slugs = refs.map(r => r.slug);
const sourceIds = refs.map(r => r.source_id);
const { rows } = await this.db.query(
`SELECT p.slug, p.source_id, COALESCE(p.effective_date, p.updated_at, p.created_at) AS ts
FROM pages p
JOIN unnest($1::text[], $2::text[]) AS u(slug, source_id)
ON p.slug = u.slug AND p.source_id = u.source_id`,
[slugs, sourceIds],
);
const out = new Map<string, Date>();
for (const r of rows as Array<{slug: string; source_id: string; ts: string | Date}>) {
const key = `${r.source_id}::${r.slug}`;
out.set(key, r.ts instanceof Date ? r.ts : new Date(r.ts));
}
return out;
}
async getSalienceScores(refs: Array<{slug: string; source_id: string}>): Promise<Map<string, number>> {
if (refs.length === 0) return new Map();
const slugs = refs.map(r => r.slug);
const sourceIds = refs.map(r => r.source_id);
const { rows } = await this.db.query(
`SELECT p.slug, p.source_id,
(COALESCE(p.emotional_weight, 0) * 5
+ ln(1 + COUNT(DISTINCT t.id))) AS score
FROM pages p
JOIN unnest($1::text[], $2::text[]) AS u(slug, source_id)
ON p.slug = u.slug AND p.source_id = u.source_id
LEFT JOIN takes t ON t.page_id = p.id AND t.active = TRUE
GROUP BY p.id`,
[slugs, sourceIds],
);
const out = new Map<string, number>();
for (const r of rows as Array<{slug: string; source_id: string; score: number | string}>) {
const key = `${r.source_id}::${r.slug}`;
out.set(key, Number(r.score));
}
return out;
}
async findOrphanPages(): Promise<Array<{ slug: string; title: string; domain: string | null }>> {
const { rows } = await this.db.query(
`SELECT
@@ -2314,6 +2413,240 @@ export class PGLiteEngine implements BrainEngine {
);
return rows as unknown as EvalCaptureFailure[];
}
// ============================================================
// v0.29 — Salience + Anomaly Detection
// ============================================================
async batchLoadEmotionalInputs(slugs?: string[]): Promise<EmotionalWeightInputRow[]> {
// Two CTEs avoid the N×M cartesian product (codex C4#4).
const baseSql = `
WITH page_tags AS (
SELECT page_id, array_agg(DISTINCT tag) AS tags
FROM tags GROUP BY page_id
),
page_takes AS (
SELECT page_id, json_agg(json_build_object(
'holder', holder, 'weight', weight, 'kind', kind, 'active', active
)) AS takes
FROM takes WHERE active = TRUE GROUP BY page_id
)
SELECT p.slug, p.source_id,
COALESCE(pt.tags, ARRAY[]::text[]) AS tags,
COALESCE(pk.takes, '[]'::json) AS takes
FROM pages p
LEFT JOIN page_tags pt ON pt.page_id = p.id
LEFT JOIN page_takes pk ON pk.page_id = p.id
`;
const { rows } = slugs
? await this.db.query(`${baseSql} WHERE p.slug = ANY($1::text[])`, [slugs])
: await this.db.query(baseSql);
return (rows as Record<string, unknown>[]).map(r => ({
slug: String(r.slug),
source_id: String(r.source_id),
tags: (r.tags as string[]) ?? [],
takes: (r.takes as EmotionalWeightInputRow['takes']) ?? [],
}));
}
async setEmotionalWeightBatch(rows: EmotionalWeightWriteRow[]): Promise<number> {
if (rows.length === 0) return 0;
const slugs = rows.map(r => r.slug);
const sourceIds = rows.map(r => r.source_id);
const weights = rows.map(r => r.weight);
// Composite-keyed UPDATE FROM unnest (codex C4#3).
// v0.29.1: bump salience_touched_at when emotional_weight actually changes
// so the salience query window picks up newly-salient old pages. Mirror
// of postgres-engine.ts.
const result = await this.db.query(
`UPDATE pages
SET emotional_weight = u.weight,
salience_touched_at = CASE
WHEN pages.emotional_weight IS DISTINCT FROM u.weight THEN now()
ELSE pages.salience_touched_at
END
FROM unnest($1::text[], $2::text[], $3::real[])
AS u(slug, source_id, weight)
WHERE pages.slug = u.slug AND pages.source_id = u.source_id
RETURNING 1`,
[slugs, sourceIds, weights]
);
return result.rows.length;
}
async getRecentSalience(opts: SalienceOpts): Promise<SalienceResult[]> {
const days = Math.max(0, opts.days ?? 14);
const limit = clampSearchLimit(opts.limit, 20, 100);
const slugPrefix = opts.slugPrefix;
const boundaryIso = new Date(Date.now() - days * 86400000).toISOString();
const params: unknown[] = [boundaryIso];
let prefixCondition = '';
if (slugPrefix) {
const escaped = slugPrefix.replace(/[\\%_]/g, (c) => '\\' + c) + '%';
params.push(escaped);
prefixCondition = `AND p.slug LIKE $${params.length} ESCAPE '\\'`;
}
params.push(limit);
const limitParam = `$${params.length}`;
// v0.29.1: third score term via buildRecencyComponentSql. Default
// 'flat' = v0.29.0 behavior. 'on' opts into per-prefix decay.
const recencyBias = opts.recency_bias ?? 'flat';
let recencySql: string;
if (recencyBias === 'on') {
const { resolveRecencyDecayMap, DEFAULT_FALLBACK } = await import('./search/recency-decay.ts');
recencySql = buildRecencyComponentSql({
slugColumn: 'p.slug',
dateExpr: 'COALESCE(p.effective_date, p.updated_at)',
decayMap: resolveRecencyDecayMap(),
fallback: DEFAULT_FALLBACK,
});
} else {
recencySql = buildRecencyComponentSql({
slugColumn: 'p.slug',
dateExpr: 'p.updated_at',
decayMap: {},
fallback: { halflifeDays: 1, coefficient: 1.0 },
});
}
const { rows } = await this.db.query(
`SELECT p.slug, p.source_id, p.title, p.type, p.updated_at, p.emotional_weight,
COUNT(DISTINCT t.id) AS take_count,
COALESCE(AVG(t.weight), 0) AS take_avg_weight,
(p.emotional_weight * 5)
+ ln(1 + COUNT(DISTINCT t.id))
+ ${recencySql}
AS score
FROM pages p
LEFT JOIN takes t ON t.page_id = p.id AND t.active = TRUE
WHERE GREATEST(p.updated_at, COALESCE(p.salience_touched_at, p.updated_at)) >= $1::timestamptz
${prefixCondition}
GROUP BY p.id
ORDER BY score DESC
LIMIT ${limitParam}`,
params
);
return (rows as Record<string, unknown>[]).map(r => ({
slug: String(r.slug),
source_id: String(r.source_id),
title: String(r.title ?? ''),
type: r.type as SalienceResult['type'],
updated_at: r.updated_at as Date,
emotional_weight: Number(r.emotional_weight ?? 0),
take_count: Number(r.take_count ?? 0),
take_avg_weight: Number(r.take_avg_weight ?? 0),
score: Number(r.score ?? 0),
}));
}
async findAnomalies(opts: AnomaliesOpts): Promise<AnomalyResult[]> {
const sigma = opts.sigma ?? 3.0;
const lookbackDays = Math.max(1, opts.lookback_days ?? 30);
const sinceIso = (opts.since ?? new Date().toISOString().slice(0, 10));
const sinceDate = new Date(sinceIso + 'T00:00:00Z');
const sinceEnd = new Date(sinceDate.getTime() + 86400000);
const baselineStart = new Date(sinceDate.getTime() - lookbackDays * 86400000);
const tagBaselineRes = await this.db.query(
`WITH days AS (
SELECT day::date FROM generate_series(
$1::date, $2::date - 1, '1 day'::interval
) AS day
),
cohort_keys AS (
SELECT DISTINCT t.tag FROM tags t JOIN pages p ON p.id = t.page_id
WHERE p.updated_at >= $1::timestamptz AND p.updated_at < $2::timestamptz
),
touched AS (
SELECT t.tag,
date_trunc('day', p.updated_at)::date AS day,
COUNT(DISTINCT p.id) AS cnt
FROM tags t JOIN pages p ON p.id = t.page_id
WHERE p.updated_at >= $1::timestamptz AND p.updated_at < $2::timestamptz
GROUP BY 1, 2
)
SELECT cd.tag AS cohort_value, d.day::text AS day, COALESCE(t.cnt, 0)::int AS count
FROM cohort_keys cd CROSS JOIN days d
LEFT JOIN touched t ON t.tag = cd.tag AND t.day = d.day`,
[baselineStart.toISOString(), sinceDate.toISOString()]
);
const typeBaselineRes = await this.db.query(
`WITH days AS (
SELECT day::date FROM generate_series(
$1::date, $2::date - 1, '1 day'::interval
) AS day
),
cohort_keys AS (
SELECT DISTINCT p.type FROM pages p
WHERE p.updated_at >= $1::timestamptz AND p.updated_at < $2::timestamptz
),
touched AS (
SELECT p.type,
date_trunc('day', p.updated_at)::date AS day,
COUNT(DISTINCT p.id) AS cnt
FROM pages p
WHERE p.updated_at >= $1::timestamptz AND p.updated_at < $2::timestamptz
GROUP BY 1, 2
)
SELECT cd.type AS cohort_value, d.day::text AS day, COALESCE(t.cnt, 0)::int AS count
FROM cohort_keys cd CROSS JOIN days d
LEFT JOIN touched t ON t.type = cd.type AND t.day = d.day`,
[baselineStart.toISOString(), sinceDate.toISOString()]
);
const tagTodayRes = await this.db.query(
`SELECT t.tag AS cohort_value,
COUNT(DISTINCT p.id)::int AS count,
array_agg(DISTINCT p.slug) AS slugs
FROM tags t JOIN pages p ON p.id = t.page_id
WHERE p.updated_at >= $1::timestamptz AND p.updated_at < $2::timestamptz
GROUP BY 1`,
[sinceIso, sinceEnd.toISOString()]
);
const typeTodayRes = await this.db.query(
`SELECT p.type AS cohort_value,
COUNT(DISTINCT p.id)::int AS count,
array_agg(DISTINCT p.slug) AS slugs
FROM pages p
WHERE p.updated_at >= $1::timestamptz AND p.updated_at < $2::timestamptz
GROUP BY 1`,
[sinceIso, sinceEnd.toISOString()]
);
const baseline = [
...(tagBaselineRes.rows as Record<string, unknown>[]).map(r => ({
cohort_kind: 'tag' as const,
cohort_value: String(r.cohort_value),
day: String(r.day),
count: Number(r.count),
})),
...(typeBaselineRes.rows as Record<string, unknown>[]).map(r => ({
cohort_kind: 'type' as const,
cohort_value: String(r.cohort_value),
day: String(r.day),
count: Number(r.count),
})),
];
const today = [
...(tagTodayRes.rows as Record<string, unknown>[]).map(r => ({
cohort_kind: 'tag' as const,
cohort_value: String(r.cohort_value),
count: Number(r.count),
page_slugs: (r.slugs as string[]) ?? [],
})),
...(typeTodayRes.rows as Record<string, unknown>[]).map(r => ({
cohort_kind: 'type' as const,
cohort_value: String(r.cohort_value),
count: Number(r.count),
page_slugs: (r.slugs as string[]) ?? [],
})),
];
return computeAnomaliesFromBuckets(baseline, today, sigma);
}
}
function rowToCodeEdge(row: Record<string, unknown>): import('./types.ts').CodeEdgeResult {
+20 -1
View File
@@ -69,10 +69,18 @@ CREATE TABLE IF NOT EXISTS pages (
timeline TEXT NOT NULL DEFAULT '',
frontmatter JSONB NOT NULL DEFAULT '{}',
content_hash TEXT,
-- v0.29: deterministic 0..1 score (tag emotion + take density + user-as-holder ratio).
-- Populated by the recompute_emotional_weight cycle phase.
emotional_weight REAL NOT NULL DEFAULT 0.0,
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
updated_at TIMESTAMPTZ NOT NULL DEFAULT now(),
-- v0.26.5: soft-delete + recovery window (mirrors src/schema.sql).
deleted_at TIMESTAMPTZ,
-- v0.29.1: salience-and-recency, additive opt-in (mirrors src/schema.sql).
effective_date TIMESTAMPTZ,
effective_date_source TEXT,
import_filename TEXT,
salience_touched_at TIMESTAMPTZ,
CONSTRAINT pages_source_slug_key UNIQUE (source_id, slug)
);
@@ -83,6 +91,9 @@ CREATE INDEX IF NOT EXISTS idx_pages_source_id ON pages(source_id);
-- v0.26.5: partial index supports the autopilot purge sweep (mirrors src/schema.sql).
CREATE INDEX IF NOT EXISTS pages_deleted_at_purge_idx
ON pages (deleted_at) WHERE deleted_at IS NOT NULL;
-- v0.29.1: expression index for since/until date-range filters.
CREATE INDEX IF NOT EXISTS pages_coalesce_date_idx
ON pages ((COALESCE(effective_date, updated_at)));
-- ============================================================
-- content_chunks: chunked content with embeddings
@@ -444,7 +455,15 @@ CREATE TABLE IF NOT EXISTS eval_candidates (
remote BOOLEAN NOT NULL,
job_id INTEGER,
subagent_id INTEGER,
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
-- v0.29.1 — agent-explicit recency + salience capture for replay (mirrors src/schema.sql).
as_of_ts TIMESTAMPTZ,
salience_param TEXT,
recency_param TEXT,
salience_resolved TEXT,
recency_resolved TEXT,
salience_source TEXT,
recency_source TEXT
);
CREATE INDEX IF NOT EXISTS idx_eval_candidates_created_at ON eval_candidates(created_at DESC);
+389 -9
View File
@@ -26,12 +26,15 @@ import type {
EngineConfig,
EvalCandidate, EvalCandidateInput,
EvalCaptureFailure, EvalCaptureFailureReason,
SalienceOpts, SalienceResult, AnomaliesOpts, AnomalyResult,
EmotionalWeightInputRow, EmotionalWeightWriteRow,
} from './types.ts';
import { GBrainError } from './types.ts';
import { GBrainError, PAGE_SORT_SQL } from './types.ts';
import { computeAnomaliesFromBuckets } from './cycle/anomaly.ts';
import * as db from './db.ts';
import { validateSlug, contentHash, rowToPage, rowToChunk, rowToSearchResult, parseEmbedding, tryParseEmbedding, takeRowToTake } from './utils.ts';
import { resolveBoostMap, resolveHardExcludes } from './search/source-boost.ts';
import { buildSourceFactorCase, buildHardExcludeClause, buildVisibilityClause } from './search/sql-ranking.ts';
import { buildSourceFactorCase, buildHardExcludeClause, buildVisibilityClause, buildRecencyComponentSql } from './search/sql-ranking.ts';
function escapeSqlStringLiteral(value: string): string {
return value.replace(/'/g, "''");
@@ -429,12 +432,19 @@ export class PostgresEngine implements BrainEngine {
// v0.18.0 Step 2: source_id relies on schema DEFAULT 'default'. ON
// CONFLICT target becomes (source_id, slug) since global UNIQUE(slug)
// was dropped in migration v17. See pglite-engine.ts for matching
// notes; multi-source sync (Step 5) will surface an explicit sourceId.
// was dropped in migration v17.
const pageKind = page.page_kind || 'markdown';
// v0.29.1 — effective_date / effective_date_source / import_filename are
// additive opt-in inputs from the importer (computeEffectiveDate). When
// omitted, the ON CONFLICT path preserves any existing value via
// COALESCE(EXCLUDED.x, pages.x) so a putPage that doesn't know about
// these columns (auto-link, code reindex, etc.) doesn't blank them out.
const effectiveDate = page.effective_date ?? null;
const effectiveDateSource = page.effective_date_source ?? null;
const importFilename = page.import_filename ?? null;
const rows = await sql`
INSERT INTO pages (slug, type, page_kind, title, compiled_truth, timeline, frontmatter, content_hash, updated_at)
VALUES (${slug}, ${page.type}, ${pageKind}, ${page.title}, ${page.compiled_truth}, ${page.timeline || ''}, ${sql.json(frontmatter as Parameters<typeof sql.json>[0])}, ${hash}, now())
INSERT INTO pages (slug, type, page_kind, title, compiled_truth, timeline, frontmatter, content_hash, updated_at, effective_date, effective_date_source, import_filename)
VALUES (${slug}, ${page.type}, ${pageKind}, ${page.title}, ${page.compiled_truth}, ${page.timeline || ''}, ${sql.json(frontmatter as Parameters<typeof sql.json>[0])}, ${hash}, now(), ${effectiveDate}, ${effectiveDateSource}, ${importFilename})
ON CONFLICT (source_id, slug) DO UPDATE SET
type = EXCLUDED.type,
page_kind = EXCLUDED.page_kind,
@@ -443,8 +453,11 @@ export class PostgresEngine implements BrainEngine {
timeline = EXCLUDED.timeline,
frontmatter = EXCLUDED.frontmatter,
content_hash = EXCLUDED.content_hash,
updated_at = now()
RETURNING id, slug, type, title, compiled_truth, timeline, frontmatter, content_hash, created_at, updated_at
updated_at = now(),
effective_date = COALESCE(EXCLUDED.effective_date, pages.effective_date),
effective_date_source = COALESCE(EXCLUDED.effective_date_source, pages.effective_date_source),
import_filename = COALESCE(EXCLUDED.import_filename, pages.import_filename)
RETURNING id, slug, type, title, compiled_truth, timeline, frontmatter, content_hash, created_at, updated_at, effective_date, effective_date_source, import_filename
`;
return rowToPage(rows[0]);
}
@@ -522,11 +535,16 @@ export class PostgresEngine implements BrainEngine {
? sql``
: sql`AND p.deleted_at IS NULL`;
// v0.29: ORDER BY threading via PAGE_SORT_SQL whitelist (no SQL injection).
// postgres.js sql.unsafe lets us splice the literal fragment safely.
const sortKey = filters?.sort && PAGE_SORT_SQL[filters.sort] ? filters.sort : 'updated_desc';
const orderBy = sql.unsafe(PAGE_SORT_SQL[sortKey]);
const rows = await sql`
SELECT p.* FROM pages p
${tagJoin}
WHERE 1=1 ${typeCondition} ${tagCondition} ${updatedCondition} ${slugCondition} ${deletedCondition}
ORDER BY p.updated_at DESC LIMIT ${limit} OFFSET ${offset}
ORDER BY ${orderBy} LIMIT ${limit} OFFSET ${offset}
`;
return rows.map(rowToPage);
@@ -614,6 +632,17 @@ export class PostgresEngine implements BrainEngine {
params.push(symbolKind);
symbolKindClause = `AND cc.symbol_type = $${params.length}`;
}
// v0.27.0: date filtering support
let afterDateClause = '';
if (opts?.afterDate) {
params.push(opts.afterDate);
afterDateClause = `AND COALESCE(p.updated_at, p.created_at) > $${params.length}::timestamptz`;
}
let beforeDateClause = '';
if (opts?.beforeDate) {
params.push(opts.beforeDate);
beforeDateClause = `AND COALESCE(p.updated_at, p.created_at) < $${params.length}::timestamptz`;
}
params.push(innerLimit);
const innerLimitParam = `$${params.length}`;
params.push(limit);
@@ -642,6 +671,8 @@ export class PostgresEngine implements BrainEngine {
${detailLow ? `AND cc.chunk_source = 'compiled_truth'` : ''}
${languageClause}
${symbolKindClause}
${afterDateClause}
${beforeDateClause}
${hardExcludeClause}
${visibilityClause}
-- v0.27.1: hide image rows from text-keyword search so OCR text
@@ -727,6 +758,17 @@ export class PostgresEngine implements BrainEngine {
params.push(symbolKind);
symbolKindClause = `AND cc.symbol_type = $${params.length}`;
}
// v0.27.0: date filtering support
let afterDateClause = '';
if (opts?.afterDate) {
params.push(opts.afterDate);
afterDateClause = `AND COALESCE(p.updated_at, p.created_at) > $${params.length}::timestamptz`;
}
let beforeDateClause = '';
if (opts?.beforeDate) {
params.push(opts.beforeDate);
beforeDateClause = `AND COALESCE(p.updated_at, p.created_at) < $${params.length}::timestamptz`;
}
params.push(limit);
const limitParam = `$${params.length}`;
params.push(offset);
@@ -750,6 +792,8 @@ export class PostgresEngine implements BrainEngine {
${detailLow ? `AND cc.chunk_source = 'compiled_truth'` : ''}
${languageClause}
${symbolKindClause}
${afterDateClause}
${beforeDateClause}
${hardExcludeClause}
${visibilityClause}
ORDER BY score DESC
@@ -815,6 +859,17 @@ export class PostgresEngine implements BrainEngine {
params.push(symbolKind);
symbolKindClause = `AND cc.symbol_type = $${params.length}`;
}
// v0.27.0: date filtering support
let afterDateClause = '';
if (opts?.afterDate) {
params.push(opts.afterDate);
afterDateClause = `AND COALESCE(p.updated_at, p.created_at) > $${params.length}::timestamptz`;
}
let beforeDateClause = '';
if (opts?.beforeDate) {
params.push(opts.beforeDate);
beforeDateClause = `AND COALESCE(p.updated_at, p.created_at) < $${params.length}::timestamptz`;
}
params.push(innerLimit);
const innerLimitParam = `$${params.length}`;
params.push(limit);
@@ -847,6 +902,8 @@ export class PostgresEngine implements BrainEngine {
${excludeSlugsClause}
${languageClause}
${symbolKindClause}
${afterDateClause}
${beforeDateClause}
${hardExcludeClause}
${visibilityClause}
ORDER BY cc.${col} <=> $1::vector
@@ -1366,6 +1423,66 @@ export class PostgresEngine implements BrainEngine {
return result;
}
async getPageTimestamps(slugs: string[]): Promise<Map<string, Date>> {
if (slugs.length === 0) return new Map();
const sql = this.sql;
const rows = await sql`
SELECT slug, COALESCE(updated_at, created_at) as ts
FROM pages WHERE slug = ANY(${slugs}::text[])
`;
return new Map(rows.map(r => [r.slug as string, new Date(r.ts as string)]));
}
async getEffectiveDates(refs: Array<{slug: string; source_id: string}>): Promise<Map<string, Date>> {
if (refs.length === 0) return new Map();
const sql = this.sql;
const slugs = refs.map(r => r.slug);
const sourceIds = refs.map(r => r.source_id);
// Composite-keyed: a page is unique by (source_id, slug). unnest the
// two arrays in lockstep so multi-source brains don't fan out across
// sources (codex pass-1 finding #3).
const rows = await sql`
SELECT p.slug, p.source_id, COALESCE(p.effective_date, p.updated_at, p.created_at) AS ts
FROM pages p
JOIN unnest(${slugs}::text[], ${sourceIds}::text[]) AS u(slug, source_id)
ON p.slug = u.slug AND p.source_id = u.source_id
`;
const out = new Map<string, Date>();
for (const raw of rows as unknown as Array<Record<string, unknown>>) {
const r = raw as { slug: string; source_id: string; ts: string | Date };
const key = `${r.source_id}::${r.slug}`;
out.set(key, r.ts instanceof Date ? r.ts : new Date(r.ts));
}
return out;
}
async getSalienceScores(refs: Array<{slug: string; source_id: string}>): Promise<Map<string, number>> {
if (refs.length === 0) return new Map();
const sql = this.sql;
const slugs = refs.map(r => r.slug);
const sourceIds = refs.map(r => r.source_id);
// Salience = emotional_weight × 5 + ln(1 + take_count). Pure mattering
// signal — NO time component (per D9: salience and recency are
// orthogonal axes). Composite-keyed for multi-source isolation.
const rows = await sql`
SELECT p.slug, p.source_id,
(COALESCE(p.emotional_weight, 0) * 5
+ ln(1 + COUNT(DISTINCT t.id))) AS score
FROM pages p
JOIN unnest(${slugs}::text[], ${sourceIds}::text[]) AS u(slug, source_id)
ON p.slug = u.slug AND p.source_id = u.source_id
LEFT JOIN takes t ON t.page_id = p.id AND t.active = TRUE
GROUP BY p.id
`;
const out = new Map<string, number>();
for (const raw of rows as unknown as Array<Record<string, unknown>>) {
const r = raw as { slug: string; source_id: string; score: number | string };
const key = `${r.source_id}::${r.slug}`;
out.set(key, Number(r.score));
}
return out;
}
async findOrphanPages(): Promise<Array<{ slug: string; title: string; domain: string | null }>> {
const sql = this.sql;
const rows = await sql`
@@ -2341,6 +2458,269 @@ export class PostgresEngine implements BrainEngine {
`;
return rows as unknown as EvalCaptureFailure[];
}
// ============================================================
// v0.29 — Salience + Anomaly Detection
// ============================================================
async batchLoadEmotionalInputs(slugs?: string[]): Promise<EmotionalWeightInputRow[]> {
const sql = this.sql;
// Two CTEs avoid the N×M cartesian product (codex C4#4): a page with N tags
// and M takes joined directly would emit N×M rows and corrupt aggregates.
// Per-table aggregation keeps each table's grouping correct.
const rows = slugs
? await sql`
WITH page_tags AS (
SELECT page_id, array_agg(DISTINCT tag) AS tags
FROM tags GROUP BY page_id
),
page_takes AS (
SELECT page_id, json_agg(json_build_object(
'holder', holder, 'weight', weight, 'kind', kind, 'active', active
)) AS takes
FROM takes WHERE active = TRUE GROUP BY page_id
)
SELECT p.slug, p.source_id,
COALESCE(pt.tags, ARRAY[]::text[]) AS tags,
COALESCE(pk.takes, '[]'::json) AS takes
FROM pages p
LEFT JOIN page_tags pt ON pt.page_id = p.id
LEFT JOIN page_takes pk ON pk.page_id = p.id
WHERE p.slug = ANY(${slugs}::text[])
`
: await sql`
WITH page_tags AS (
SELECT page_id, array_agg(DISTINCT tag) AS tags
FROM tags GROUP BY page_id
),
page_takes AS (
SELECT page_id, json_agg(json_build_object(
'holder', holder, 'weight', weight, 'kind', kind, 'active', active
)) AS takes
FROM takes WHERE active = TRUE GROUP BY page_id
)
SELECT p.slug, p.source_id,
COALESCE(pt.tags, ARRAY[]::text[]) AS tags,
COALESCE(pk.takes, '[]'::json) AS takes
FROM pages p
LEFT JOIN page_tags pt ON pt.page_id = p.id
LEFT JOIN page_takes pk ON pk.page_id = p.id
`;
return rows.map((r: Record<string, unknown>) => ({
slug: String(r.slug),
source_id: String(r.source_id),
tags: (r.tags as string[]) ?? [],
takes: (r.takes as EmotionalWeightInputRow['takes']) ?? [],
}));
}
async setEmotionalWeightBatch(rows: EmotionalWeightWriteRow[]): Promise<number> {
if (rows.length === 0) return 0;
const sql = this.sql;
const slugs = rows.map(r => r.slug);
const sourceIds = rows.map(r => r.source_id);
const weights = rows.map(r => r.weight);
// Composite-keyed UPDATE FROM unnest (codex C4#3): pages.slug is unique
// only within a source, so a slug-only join would fan out across sources.
//
// v0.29.1: bump salience_touched_at to NOW() ONLY when emotional_weight
// actually changes. The salience query window then includes the page in
// GREATEST(updated_at, salience_touched_at) >= boundary, so a previously
// calm page that just became salient surfaces in the recent salience
// results without a content edit. No-op writes (same weight) leave
// salience_touched_at alone — preserves "actual change" semantics.
const result = await sql`
UPDATE pages
SET emotional_weight = u.weight,
salience_touched_at = CASE
WHEN pages.emotional_weight IS DISTINCT FROM u.weight THEN now()
ELSE pages.salience_touched_at
END
FROM unnest(${slugs}::text[], ${sourceIds}::text[], ${weights}::real[])
AS u(slug, source_id, weight)
WHERE pages.slug = u.slug AND pages.source_id = u.source_id
RETURNING 1
`;
return result.length;
}
async getRecentSalience(opts: SalienceOpts): Promise<SalienceResult[]> {
const sql = this.sql;
const days = Math.max(0, opts.days ?? 14);
const limit = clampSearchLimit(opts.limit, 20, 100);
const slugPrefix = opts.slugPrefix;
// Compute the boundary in JS so the SQL is identical across engines (eng review D5).
const boundaryIso = new Date(Date.now() - days * 86400000).toISOString();
// Escape LIKE meta for the optional prefix match.
const prefixCondition = slugPrefix
? sql`AND p.slug LIKE ${slugPrefix.replace(/[\\%_]/g, (c) => '\\' + c) + '%'} ESCAPE '\\'`
: sql``;
// v0.29.1: third score term via buildRecencyComponentSql. Default
// 'flat' = v0.29.0 behavior (1 / (1 + days_old)). 'on' opts into the
// per-prefix decay map (concepts/ evergreen, daily/ aggressive, etc.).
const recencyBias = opts.recency_bias ?? 'flat';
let recencySql: string;
if (recencyBias === 'on') {
const { resolveRecencyDecayMap, DEFAULT_FALLBACK } = await import('./search/recency-decay.ts');
recencySql = buildRecencyComponentSql({
slugColumn: 'p.slug',
dateExpr: 'COALESCE(p.effective_date, p.updated_at)',
decayMap: resolveRecencyDecayMap(),
fallback: DEFAULT_FALLBACK,
});
} else {
recencySql = buildRecencyComponentSql({
slugColumn: 'p.slug',
dateExpr: 'p.updated_at',
decayMap: {},
fallback: { halflifeDays: 1, coefficient: 1.0 },
});
}
const rows = await sql`
SELECT p.slug, p.source_id, p.title, p.type, p.updated_at, p.emotional_weight,
COUNT(DISTINCT t.id) AS take_count,
COALESCE(AVG(t.weight), 0) AS take_avg_weight,
(p.emotional_weight * 5)
+ ln(1 + COUNT(DISTINCT t.id))
+ ${sql.unsafe(recencySql)}
AS score
FROM pages p
LEFT JOIN takes t ON t.page_id = p.id AND t.active = TRUE
WHERE GREATEST(p.updated_at, COALESCE(p.salience_touched_at, p.updated_at)) >= ${boundaryIso}::timestamptz
${prefixCondition}
GROUP BY p.id
ORDER BY score DESC
LIMIT ${limit}
`;
return rows.map((r: Record<string, unknown>) => ({
slug: String(r.slug),
source_id: String(r.source_id),
title: String(r.title ?? ''),
type: r.type as SalienceResult['type'],
updated_at: r.updated_at as Date,
emotional_weight: Number(r.emotional_weight ?? 0),
take_count: Number(r.take_count ?? 0),
take_avg_weight: Number(r.take_avg_weight ?? 0),
score: Number(r.score ?? 0),
}));
}
async findAnomalies(opts: AnomaliesOpts): Promise<AnomalyResult[]> {
const sql = this.sql;
const sigma = opts.sigma ?? 3.0;
const lookbackDays = Math.max(1, opts.lookback_days ?? 30);
// Boundaries: today's window is [since, since+1day); baseline is [since-lookback, since).
const sinceIso = (opts.since ?? new Date().toISOString().slice(0, 10)); // YYYY-MM-DD
const sinceDate = new Date(sinceIso + 'T00:00:00Z');
const sinceEnd = new Date(sinceDate.getTime() + 86400000);
const baselineStart = new Date(sinceDate.getTime() - lookbackDays * 86400000);
// Tag cohort baseline with day densification + zero-fill (codex C4#6).
const tagBaseline = await sql`
WITH days AS (
SELECT day::date FROM generate_series(
${baselineStart.toISOString()}::date,
${sinceDate.toISOString()}::date - 1,
'1 day'::interval
) AS day
),
cohort_keys AS (
SELECT DISTINCT t.tag FROM tags t JOIN pages p ON p.id = t.page_id
WHERE p.updated_at >= ${baselineStart.toISOString()}::timestamptz
AND p.updated_at < ${sinceDate.toISOString()}::timestamptz
),
touched AS (
SELECT t.tag,
date_trunc('day', p.updated_at)::date AS day,
COUNT(DISTINCT p.id) AS cnt
FROM tags t JOIN pages p ON p.id = t.page_id
WHERE p.updated_at >= ${baselineStart.toISOString()}::timestamptz
AND p.updated_at < ${sinceDate.toISOString()}::timestamptz
GROUP BY 1, 2
)
SELECT cd.tag AS cohort_value, d.day::text AS day, COALESCE(t.cnt, 0)::int AS count
FROM cohort_keys cd CROSS JOIN days d
LEFT JOIN touched t ON t.tag = cd.tag AND t.day = d.day
`;
const typeBaseline = await sql`
WITH days AS (
SELECT day::date FROM generate_series(
${baselineStart.toISOString()}::date,
${sinceDate.toISOString()}::date - 1,
'1 day'::interval
) AS day
),
cohort_keys AS (
SELECT DISTINCT p.type FROM pages p
WHERE p.updated_at >= ${baselineStart.toISOString()}::timestamptz
AND p.updated_at < ${sinceDate.toISOString()}::timestamptz
),
touched AS (
SELECT p.type,
date_trunc('day', p.updated_at)::date AS day,
COUNT(DISTINCT p.id) AS cnt
FROM pages p
WHERE p.updated_at >= ${baselineStart.toISOString()}::timestamptz
AND p.updated_at < ${sinceDate.toISOString()}::timestamptz
GROUP BY 1, 2
)
SELECT cd.type AS cohort_value, d.day::text AS day, COALESCE(t.cnt, 0)::int AS count
FROM cohort_keys cd CROSS JOIN days d
LEFT JOIN touched t ON t.type = cd.type AND t.day = d.day
`;
// Today's window — current counts + slugs per cohort.
const tagToday = await sql`
SELECT t.tag AS cohort_value,
COUNT(DISTINCT p.id)::int AS count,
array_agg(DISTINCT p.slug) AS slugs
FROM tags t JOIN pages p ON p.id = t.page_id
WHERE p.updated_at >= ${sinceIso}::timestamptz
AND p.updated_at < ${sinceEnd.toISOString()}::timestamptz
GROUP BY 1
`;
const typeToday = await sql`
SELECT p.type AS cohort_value,
COUNT(DISTINCT p.id)::int AS count,
array_agg(DISTINCT p.slug) AS slugs
FROM pages p
WHERE p.updated_at >= ${sinceIso}::timestamptz
AND p.updated_at < ${sinceEnd.toISOString()}::timestamptz
GROUP BY 1
`;
const baseline = [
...tagBaseline.map((r: Record<string, unknown>) => ({
cohort_kind: 'tag' as const,
cohort_value: String(r.cohort_value),
day: String(r.day),
count: Number(r.count),
})),
...typeBaseline.map((r: Record<string, unknown>) => ({
cohort_kind: 'type' as const,
cohort_value: String(r.cohort_value),
day: String(r.day),
count: Number(r.count),
})),
];
const today = [
...tagToday.map((r: Record<string, unknown>) => ({
cohort_kind: 'tag' as const,
cohort_value: String(r.cohort_value),
count: Number(r.count),
page_slugs: (r.slugs as string[]) ?? [],
})),
...typeToday.map((r: Record<string, unknown>) => ({
cohort_kind: 'type' as const,
cohort_value: String(r.cohort_value),
count: Number(r.count),
page_slugs: (r.slugs as string[]) ?? [],
})),
];
return computeAnomaliesFromBuckets(baseline, today, sigma);
}
}
function pgRowToCodeEdge(row: Record<string, unknown>): import('./types.ts').CodeEdgeResult {
+31 -1
View File
@@ -81,6 +81,10 @@ CREATE TABLE IF NOT EXISTS pages (
timeline TEXT NOT NULL DEFAULT '',
frontmatter JSONB NOT NULL DEFAULT '{}',
content_hash TEXT,
-- v0.29: deterministic 0..1 score (tag emotion + take density + Garry-as-holder ratio).
-- Populated by the \`recompute_emotional_weight\` cycle phase. Default 0.0 so freshly
-- imported pages don't pollute salience ranking before the cycle has run.
emotional_weight REAL NOT NULL DEFAULT 0.0,
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
updated_at TIMESTAMPTZ NOT NULL DEFAULT now(),
-- v0.26.5: soft-delete + recovery window. \`delete_page\` sets deleted_at = now()
@@ -88,6 +92,17 @@ CREATE TABLE IF NOT EXISTS pages (
-- where deleted_at < now() - 72h. Search and \`get_page\` filter
-- \`WHERE deleted_at IS NULL\` by default; \`include_deleted: true\` opts in.
deleted_at TIMESTAMPTZ,
-- v0.29.1: salience-and-recency, additive opt-in. All NULL by default;
-- only consulted when a caller passes \`salience='on'\` / \`recency='on'\` or
-- the new \`since\`/\`until\` filter. effective_date_source is a sentinel for
-- the doctor's effective_date_health check (values: 'event_date' | 'date'
-- | 'published' | 'filename' | 'fallback'). salience_touched_at is bumped
-- by recompute_emotional_weight when emotional_weight changes so the
-- salience window picks up newly-salient old pages.
effective_date TIMESTAMPTZ,
effective_date_source TEXT,
import_filename TEXT,
salience_touched_at TIMESTAMPTZ,
CONSTRAINT pages_source_slug_key UNIQUE (source_id, slug)
);
@@ -105,6 +120,12 @@ CREATE INDEX IF NOT EXISTS idx_pages_source_id ON pages(source_id);
-- stays low. Don't add a regular \`(deleted_at)\` index without measuring.
CREATE INDEX IF NOT EXISTS pages_deleted_at_purge_idx
ON pages (deleted_at) WHERE deleted_at IS NOT NULL;
-- v0.29.1: expression index used by since/until date-range filters that read
-- COALESCE(effective_date, updated_at). A partial index on effective_date
-- alone would NOT help — the planner can't use it for the negative side of
-- the COALESCE. Expression index is what actually accelerates the filter.
CREATE INDEX IF NOT EXISTS pages_coalesce_date_idx
ON pages ((COALESCE(effective_date, updated_at)));
-- ============================================================
-- content_chunks: chunked content with embeddings
@@ -746,7 +767,16 @@ CREATE TABLE IF NOT EXISTS eval_candidates (
remote BOOLEAN NOT NULL,
job_id INTEGER,
subagent_id INTEGER,
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
-- v0.29.1 — agent-explicit recency + salience capture for replay reproducibility.
-- All nullable + additive. NDJSON schema_version stays at 1; consumers ignore unknown fields.
as_of_ts TIMESTAMPTZ,
salience_param TEXT,
recency_param TEXT,
salience_resolved TEXT,
recency_resolved TEXT,
salience_source TEXT,
recency_source TEXT
);
CREATE INDEX IF NOT EXISTS idx_eval_candidates_created_at ON eval_candidates(created_at DESC);
+185 -22
View File
@@ -14,7 +14,7 @@ import { MAX_SEARCH_LIMIT, clampSearchLimit } from '../engine.ts';
import type { SearchResult, SearchOpts, HybridSearchMeta } from '../types.ts';
import { embed } from '../embedding.ts';
import { dedupResults } from './dedup.ts';
import { autoDetectDetail } from './intent.ts';
import { autoDetectDetail, classifyQuery } from './query-intent.ts';
import { expandAnchors, hydrateChunks } from './two-pass.ts';
const RRF_K = 60;
@@ -45,6 +45,147 @@ export function applyBacklinkBoost(results: SearchResult[], counts: Map<string,
}
}
/**
* v0.29.1 — apply salience boost (emotional_weight + take_count, NO time
* component). Mirror of applyBacklinkBoost. Mutate-in-place; caller re-sorts.
*
* `scores` is keyed by `${source_id}::${slug}` (composite) so multi-source
* brains don't conflate same-slug pages across sources (codex pass-1 #3).
*
* strength: 'on' (k=0.15) or 'strong' (k=0.30); 'off' callers should not
* invoke this function. Logarithmic compression keeps the factor in
* [1.0, ~1.6] so a strong boost can't catastrophically flip rankings.
*/
export function applySalienceBoost(
results: SearchResult[],
scores: Map<string, number>,
strength: 'on' | 'strong',
): void {
const k = strength === 'strong' ? 0.30 : 0.15;
for (const r of results) {
const key = `${r.source_id ?? 'default'}::${r.slug}`;
const score = scores.get(key);
if (!score || score <= 0) continue;
r.score *= (1.0 + k * Math.log(1 + score));
}
}
/**
* v0.29.1 — apply per-prefix recency boost. Mutate-in-place; caller re-sorts.
*
* `dates` is keyed by `${source_id}::${slug}`. The boost factor for each
* page comes from the per-prefix decay map: `1 + coefficient × halflife /
* (halflife + days_old)`. Evergreen prefixes (halflifeDays=0) contribute 0
* (factor stays 1.0).
*
* strength: 'on' multiplies the coefficient by 1.0; 'strong' multiplies by
* 1.5 (more aggressive recency tilt). Pages with no date entry in the map
* are skipped (factor 1.0).
*/
export function applyRecencyBoost(
results: SearchResult[],
dates: Map<string, Date>,
strength: 'on' | 'strong',
decayMap: import('./recency-decay.ts').RecencyDecayMap,
fallback: import('./recency-decay.ts').RecencyDecayConfig,
nowMs: number = Date.now(),
): void {
const strengthMul = strength === 'strong' ? 1.5 : 1.0;
// Sort prefixes longest-first so 'media/articles/' matches before 'media/'.
const prefixes = Object.keys(decayMap).sort((a, b) => b.length - a.length);
for (const r of results) {
const key = `${r.source_id ?? 'default'}::${r.slug}`;
const d = dates.get(key);
if (!d) continue;
const daysOld = Math.max(0, (nowMs - d.getTime()) / 86_400_000);
// Find first matching prefix.
let cfg: import('./recency-decay.ts').RecencyDecayConfig = fallback;
for (const p of prefixes) {
if (r.slug.startsWith(p)) {
cfg = decayMap[p];
break;
}
}
if (cfg.halflifeDays === 0 || cfg.coefficient === 0) continue; // evergreen
const recencyComponent = cfg.coefficient * cfg.halflifeDays / (cfg.halflifeDays + daysOld);
const factor = 1.0 + strengthMul * recencyComponent;
r.score *= factor;
}
}
/**
* v0.29.1 — runPostFusionStages: wrap backlink + salience + recency in a
* single stage that fires from EVERY hybridSearch return path (codex
* pass-1 #2 + pass-2 #4: keyword-only, embed-fail-fallback, full-hybrid).
* Without this wrapper, salience='on' silently does nothing on keyless
* installs that fall back to keyword-only.
*
* Mutates `results` in place; caller re-sorts.
*/
export interface PostFusionOpts {
applyBacklinks: boolean;
salience: 'off' | 'on' | 'strong';
recency: 'off' | 'on' | 'strong';
decayMap?: import('./recency-decay.ts').RecencyDecayMap;
fallback?: import('./recency-decay.ts').RecencyDecayConfig;
}
export async function runPostFusionStages(
engine: import('../engine.ts').BrainEngine,
results: SearchResult[],
opts: PostFusionOpts,
): Promise<void> {
if (results.length === 0) return;
// Backlink stage (existing behavior, preserved).
if (opts.applyBacklinks) {
try {
const slugs = Array.from(new Set(results.map(r => r.slug)));
const counts = await engine.getBacklinkCounts(slugs);
applyBacklinkBoost(results, counts);
} catch {
// Non-fatal; preserves the existing pre-v0.29.1 contract.
}
}
// Composite refs for the orthogonal axes (multi-source isolation).
const refs = Array.from(
new Map(
results.map(r => [`${r.source_id ?? 'default'}::${r.slug}`, { slug: r.slug, source_id: r.source_id ?? 'default' }]),
).values(),
);
// Salience stage (mattering, no time).
if (opts.salience !== 'off') {
try {
const scores = await engine.getSalienceScores(refs);
applySalienceBoost(results, scores, opts.salience);
} catch {
// Non-fatal.
}
}
// Recency stage (per-prefix decay, no mattering).
if (opts.recency !== 'off') {
try {
const dates = await engine.getEffectiveDates(refs);
const { DEFAULT_RECENCY_DECAY, DEFAULT_FALLBACK } = await import('./recency-decay.ts');
applyRecencyBoost(
results,
dates,
opts.recency,
opts.decayMap ?? DEFAULT_RECENCY_DECAY,
opts.fallback ?? DEFAULT_FALLBACK,
);
} catch {
// Non-fatal.
}
}
}
export interface HybridSearchOpts extends SearchOpts {
expansion?: boolean;
expandFn?: (query: string) => Promise<string[]>;
@@ -86,6 +227,11 @@ export async function hybridSearch(
// per-engine searchKeyword / searchVector apply the filters at SQL level.
language: opts?.language,
symbolKind: opts?.symbolKind,
// v0.29.1: since/until take precedence over deprecated afterDate/beforeDate.
// The engine still consumes the legacy field names; this aliasing keeps
// PR #618 callers compiling while the new names are the public surface.
afterDate: opts?.since ?? opts?.afterDate,
beforeDate: opts?.until ?? opts?.beforeDate,
};
// Track what actually ran for the optional onMeta callback (v0.25.0).
// Caller leaves onMeta undefined → these flags are computed but never
@@ -111,20 +257,30 @@ export async function hybridSearch(
// Run keyword search (always available, no API key needed)
const keywordResults = await engine.searchKeyword(query, searchOpts);
// v0.29.1: resolve salience/recency from caller (back-compat aliases for
// PR #618's `recencyBoost` numeric scale) or fall back to the heuristic.
// The wrapper fires from ALL THREE return paths (codex pass-1 #2 + pass-2 #4).
const suggestions = classifyQuery(query);
// Back-compat: recencyBoost: 1|2 → 'on'|'strong'; 0 → 'off'.
const legacyRecency: 'off' | 'on' | 'strong' | undefined =
opts?.recencyBoost === 2 ? 'strong' :
opts?.recencyBoost === 1 ? 'on' :
opts?.recencyBoost === 0 ? 'off' :
undefined;
const salienceMode: 'off' | 'on' | 'strong' = opts?.salience ?? suggestions.suggestedSalience;
const recencyMode: 'off' | 'on' | 'strong' = opts?.recency ?? legacyRecency ?? suggestions.suggestedRecency;
const postFusionOpts = {
applyBacklinks: true,
salience: salienceMode,
recency: recencyMode,
};
// Skip vector search entirely if the gateway has no embedding provider configured (Codex C3).
const { isAvailable } = await import('../ai/gateway.ts');
if (!isAvailable('embedding')) {
// Apply backlink boost in keyword-only path too. One getBacklinkCounts query
// per search request; not N+1.
if (keywordResults.length > 0) {
try {
const slugs = Array.from(new Set(keywordResults.map(r => r.slug)));
const counts = await engine.getBacklinkCounts(slugs);
applyBacklinkBoost(keywordResults, counts);
keywordResults.sort((a, b) => b.score - a.score);
} catch {
// Boost failure is non-fatal: keep unboosted ranking.
}
await runPostFusionStages(engine, keywordResults, postFusionOpts);
keywordResults.sort((a, b) => b.score - a.score);
}
emitMeta({ vector_enabled: false, detail_resolved: detailResolved, expansion_applied: false });
return dedupResults(keywordResults).slice(offset, offset + limit);
@@ -160,6 +316,13 @@ export async function hybridSearch(
if (vectorLists.length === 0) {
// Embed/vector failed silently; record that vector did not run.
// v0.29.1 codex pass-2 #4: this is the third return path. Apply
// post-fusion stages here too — without it, salience='on' silently
// does nothing on embed failures.
if (keywordResults.length > 0) {
await runPostFusionStages(engine, keywordResults, postFusionOpts);
keywordResults.sort((a, b) => b.score - a.score);
}
emitMeta({ vector_enabled: false, detail_resolved: detailResolved, expansion_applied: expansionApplied });
return dedupResults(keywordResults).slice(offset, offset + limit);
}
@@ -174,18 +337,13 @@ export async function hybridSearch(
fused = await cosineReScore(engine, fused, queryEmbedding);
}
// Apply backlink boost AFTER cosine re-score so the boost survives normalization,
// and BEFORE dedup so it influences which chunks per page survive deduplication.
// One DB query for the whole result set (not N+1).
// v0.29.1: post-fusion stages (backlink + salience + recency) run via
// runPostFusionStages so all three early-return paths share the same
// boost surface. Salience and recency are independent axes — either,
// both, or neither fires depending on resolved modes.
if (fused.length > 0) {
try {
const slugs = Array.from(new Set(fused.map(r => r.slug)));
const counts = await engine.getBacklinkCounts(slugs);
applyBacklinkBoost(fused, counts);
fused.sort((a, b) => b.score - a.score);
} catch {
// Boost failure is non-fatal: keep blended cosine ranking.
}
await runPostFusionStages(engine, fused, postFusionOpts);
fused.sort((a, b) => b.score - a.score);
}
// v0.20.0 Cathedral II Layer 7 (A2): two-pass structural expansion.
@@ -232,6 +390,11 @@ export async function hybridSearch(
}
}
// v0.27.0 PR #618 recency boost was here; v0.29.1 unifies it into
// runPostFusionStages above so all three return paths get the same
// treatment. PR #618's recencyBoost: 0|1|2 still works via back-compat
// aliasing in the postFusionOpts resolver near line ~256.
// Dedup
const deduped = dedupResults(fused, dedupOpts);
-108
View File
@@ -1,108 +0,0 @@
/**
* Query Intent Classifier
*
* Zero-latency heuristic classifier that detects query intent from text patterns.
* Maps intent to the appropriate detail level for hybrid search.
*
* No LLM call, no API cost, no latency. Pattern matching on query text.
*/
export type QueryIntent = 'entity' | 'temporal' | 'event' | 'general';
// Temporal patterns: questions about when things happened, meeting history
const TEMPORAL_PATTERNS = [
/\bwhen\b/i,
/\blast\s+(met|meeting|call|conversation|chat|talked|spoke|seen|heard|time)\b/i,
/\brecent(ly)?\b/i,
/\bhistory\b/i,
/\btimeline\b/i,
/\bmeeting\s+notes?\b/i,
/\bwhat('s| is| was)\s+new\b/i,
/\blatest\b/i,
/\bupdate(s)?\s+(on|from|about)\b/i,
/\bhow\s+long\s+(ago|since)\b/i,
/\b\d{4}[-/]\d{2}\b/i, // date pattern like 2024-03
/\blast\s+(week|month|quarter|year)\b/i,
];
// Event patterns: specific events, announcements, launches
const EVENT_PATTERNS = [
/\bannounce[ds]?(ment)?\b/i,
/\blaunch(ed|es|ing)?\b/i,
/\braised?\s+\$?\d/i,
/\bfund(ing|raise)\b/i,
/\bIPO\b/i,
/\bacquisition\b/i,
/\bmerge[drs]?\b/i,
/\bnews\b/i,
/\bhappened?\b/i,
];
// Entity patterns: identity questions, overviews
const ENTITY_PATTERNS = [
/\bwho\s+is\b/i,
/\bwhat\s+(is|does|are)\b/i,
/\btell\s+me\s+about\b/i,
/\bdescribe\b/i,
/\bsummar(y|ize)\b/i,
/\boverview\b/i,
/\bbackground\b/i,
/\bprofile\b/i,
/\bwhat\s+do\s+(you|we)\s+know\b/i,
];
// Full-context patterns: requests for everything
const FULL_CONTEXT_PATTERNS = [
/\beverything\b/i,
/\ball\s+(about|info|information|details)\b/i,
/\bfull\s+(history|context|picture|story|details)\b/i,
/\bcomprehensive\b/i,
/\bdeep\s+dive\b/i,
/\bgive\s+me\s+everything\b/i,
];
/**
* Classify query intent from text patterns.
* Returns the detected intent type.
*/
export function classifyQueryIntent(query: string): QueryIntent {
// Full context requests → treat as temporal (return everything)
if (FULL_CONTEXT_PATTERNS.some(p => p.test(query))) return 'temporal';
// Check temporal patterns first (highest priority for detail=high)
if (TEMPORAL_PATTERNS.some(p => p.test(query))) return 'temporal';
// Check event patterns
if (EVENT_PATTERNS.some(p => p.test(query))) return 'event';
// Check entity patterns
if (ENTITY_PATTERNS.some(p => p.test(query))) return 'entity';
// Default: general query
return 'general';
}
/**
* Map query intent to detail level.
*
* entity → 'low' (compiled truth only, user wants the assessment)
* temporal → 'high' (need timeline, user wants dates/events)
* event → 'high' (need timeline, user wants specific events)
* general → undefined (use default medium, let the boost handle it)
*/
export function intentToDetail(intent: QueryIntent): 'low' | 'medium' | 'high' | undefined {
switch (intent) {
case 'entity': return 'low';
case 'temporal': return 'high';
case 'event': return 'high';
case 'general': return undefined; // use default
}
}
/**
* Auto-detect detail level from query text.
* Returns undefined if no strong signal detected (uses default).
*/
export function autoDetectDetail(query: string): 'low' | 'medium' | 'high' | undefined {
return intentToDetail(classifyQueryIntent(query));
}
+253
View File
@@ -0,0 +1,253 @@
/**
* v0.29.1 — merged query-intent classifier.
*
* Replaces v0.29.0's `intent.ts` (which only emitted a detail suggestion).
* After D1 + D4 the codebase needs ONE classifier that returns three
* suggestions from a single regex pass:
*
* - intent: original v0.29.0 type ('entity' | 'temporal' | 'event' | 'general')
* - suggestedDetail: v0.29.0 mapping (entity→low, temporal/event→high)
* - suggestedSalience: NEW for v0.29.1 — 'off' | 'on' | 'strong'
* - suggestedRecency: NEW for v0.29.1 — 'off' | 'on' | 'strong'
*
* Salience and recency are TRULY ORTHOGONAL (per D9):
* - salience boosts pages with high emotional_weight + take_count (mattering)
* - recency boosts pages with recent effective_date (per-prefix decay)
* Both can fire, neither can fire, or just one.
*
* The classifier follows "current state → on. canonical truth → off." with
* a NARROW exception per D6: explicit temporal bounds (today / this week /
* right now / since X / last N days) override canonical-pattern wins. So
* "who is X right now" → suggestedRecency='on' even though "who is" is a
* canonical pattern.
*
* Pure module. No DB, no LLM, no async. Tested in test/query-intent.test.ts.
*/
export type QueryIntent = 'entity' | 'temporal' | 'event' | 'general';
export type SalienceMode = 'off' | 'on' | 'strong';
export type RecencyMode = 'off' | 'on' | 'strong';
export interface QuerySuggestions {
intent: QueryIntent;
/** v0.29.0 detail mapping. entity→low, temporal/event→high, general→undefined. */
suggestedDetail: 'low' | 'medium' | 'high' | undefined;
/** v0.29.1 — emotional_weight + take_count boost. */
suggestedSalience: SalienceMode;
/** v0.29.1 — per-prefix age-decay boost. */
suggestedRecency: RecencyMode;
}
// ─────────────────────────────────────────────────────────
// Pattern banks (organized by axis they signal)
// ─────────────────────────────────────────────────────────
// Original v0.29.0 intent patterns. Drive .intent + .suggestedDetail.
const TEMPORAL_PATTERNS = [
/\bwhen\b/i,
/\blast\s+(met|meeting|call|conversation|chat|talked|spoke|seen|heard|time)\b/i,
/\brecent(ly)?\b/i,
/\bhistory\b/i,
/\btimeline\b/i,
/\bmeeting\s+notes?\b/i,
/\bwhat('s| is| was)\s+new\b/i,
/\blatest\b/i,
/\bupdate(s)?\s+(on|from|about)\b/i,
/\bhow\s+long\s+(ago|since)\b/i,
/\b\d{4}[-/]\d{2}\b/i,
/\blast\s+(week|month|quarter|year)\b/i,
];
const EVENT_PATTERNS = [
/\bannounce[ds]?(ment)?\b/i,
/\blaunch(ed|es|ing)?\b/i,
/\braised?\s+\$?\d/i,
/\bfund(ing|raise)\b/i,
/\bIPO\b/i,
/\bacquisition\b/i,
/\bmerge[drs]?\b/i,
/\bnews\b/i,
/\bhappened?\b/i,
];
const ENTITY_PATTERNS = [
/\bwho\s+is\b/i,
/\bwhat\s+(is|does|are)\b/i,
/\btell\s+me\s+about\b/i,
/\bdescribe\b/i,
/\bsummar(y|ize)\b/i,
/\boverview\b/i,
/\bbackground\b/i,
/\bprofile\b/i,
/\bwhat\s+do\s+(you|we)\s+know\b/i,
];
const FULL_CONTEXT_PATTERNS = [
/\beverything\b/i,
/\ball\s+(about|info|information|details)\b/i,
/\bfull\s+(history|context|picture|story|details)\b/i,
/\bcomprehensive\b/i,
/\bdeep\s+dive\b/i,
/\bgive\s+me\s+everything\b/i,
];
// v0.29.1 — recency-axis patterns
//
// Canonical patterns: queries asking for the authoritative / definitional
// answer. These signal recency='off' even when other axes match — UNLESS
// an explicit temporal bound is present (per D6 narrow exception).
const CANONICAL_PATTERNS = [
/\bwho\s+is\b/i,
/\bwhat\s+(is|are|does|means?)\b/i,
/\bdefin(e|ition|ing)\b/i,
/\bexplain\s+(what|how|why)\b/i,
/\b(history|origin|background)\s+of\b/i,
/\bconcept\s+of\b/i,
/\boverview\s+of\b/i,
/\btell\s+me\s+about\b/i,
/\bcompiled\s+truth\b/i,
/::|->|\.\w+\(/,
/\b(function|class|method|module)\s+\w+/i,
/\b(graph|traversal|backlinks?|inbound|outbound)\b/i,
];
// Aggressive recency: "today", "right now", "this morning", "just now".
const STRONG_RECENCY_PATTERNS = [
/\btoday\b/i,
/\bright\s+now\b/i,
/\bthis\s+morning\b/i,
/\bjust\s+now\b/i,
];
// Moderate recency: "what's going on", "latest", "recent", "this week",
// meeting prep, conversation recall, status updates.
const RECENCY_ON_PATTERNS = [
/\bwhat'?s\s+(going\s+on|happening|new|latest|up)\b/i,
/\b(latest|recent(ly)?|currently)\b/i,
/\b(this|last|past)\s+(week|month|few\s+days|couple\s+days)\b/i,
/\bmeeting\s+(prep|with|for|notes?|brief)\b/i,
/\bbefore\s+(my|the|our)\s+(meeting|call|sync|chat)\b/i,
/\bprep(are)?\s+(for|me)\b/i,
/\bcatch(es|ing)?\b[\s\w]{0,15}\bup\b/i, // "catch up", "catch me up", "catching X up"
/\bremind\s+me\s+(what|about|of)\b/i,
/\b(update|status|progress)\s+(on|with|from)\b/i,
];
// Per D6: explicit temporal bounds override canonical-wins. "Who is X today"
// → recency='on' (temporal bound wins). "Who is X" alone → recency='off'.
const EXPLICIT_TEMPORAL_BOUND_PATTERNS = [
/\btoday\b/i,
/\bright\s+now\b/i,
/\bthis\s+morning\b/i,
/\bthis\s+week\b/i,
/\bsince\s+(launch|last|the|\d)/i,
/\blast\s+\d+\s+(day|days|week|weeks|month|months)\b/i,
];
// v0.29.1 — salience-axis patterns
//
// Salience suggests "what matters in this brain right now" — when the user
// is asking about people/companies/deals in the current context, they
// usually want the emotionally-weighted + take-rich pages to surface.
// Salience patterns are a subset of recency-on patterns (meeting prep,
// catch-up, update language) plus people-centric phrasings.
const SALIENCE_ON_PATTERNS = [
/\bwhat'?s\s+(going\s+on|happening|been\s+going|been\s+up)\b/i,
/\bcatch(es|ing)?\b[\s\w]{0,15}\bup\b/i,
/\bremind\s+me\s+(what|about|of)\b/i,
/\bprep(are)?\s+(for|me)\b/i,
/\bbefore\s+(my|the|our)\s+(meeting|call|sync|chat)\b/i,
/\bmeeting\s+(prep|with|for|brief)\b/i,
/\b(update|status|progress)\s+(on|with|from)\b/i,
/\bwhat\s+matters\b/i,
/\bwhat'?s\s+important\b/i,
];
// ─────────────────────────────────────────────────────────
// Classifier
// ─────────────────────────────────────────────────────────
function matches(patterns: RegExp[], q: string): boolean {
for (const re of patterns) if (re.test(q)) return true;
return false;
}
/**
* Classify a query and return all three axis suggestions.
*
* Resolution rules:
* - intent: original v0.29.0 priority (full-context > temporal > event > entity > general)
* - suggestedDetail: intent → detail mapping (entity=low, temporal/event=high)
* - suggestedRecency: STRONG_RECENCY > RECENCY_ON; CANONICAL wins UNLESS
* EXPLICIT_TEMPORAL_BOUND also matches; default 'off'
* - suggestedSalience: SALIENCE_ON; CANONICAL wins UNLESS
* EXPLICIT_TEMPORAL_BOUND; default 'off'
*
* Note: salience and recency are independent. A "what's going on with X"
* query gets BOTH on; "who is X" gets BOTH off; "today's news" gets
* recency='strong' but salience='off' (the user wants newest, not
* emotionally-weighted).
*/
export function classifyQuery(query: string): QuerySuggestions {
const intent = classifyQueryIntent(query);
const suggestedDetail = intentToDetail(intent);
const hasCanonical = matches(CANONICAL_PATTERNS, query);
const hasTemporalBound = matches(EXPLICIT_TEMPORAL_BOUND_PATTERNS, query);
const hasStrongRecency = matches(STRONG_RECENCY_PATTERNS, query);
const hasRecencyOn = matches(RECENCY_ON_PATTERNS, query);
const hasSalienceOn = matches(SALIENCE_ON_PATTERNS, query);
// Recency axis
let suggestedRecency: RecencyMode;
if (hasCanonical && !hasTemporalBound) {
suggestedRecency = 'off';
} else if (hasStrongRecency) {
suggestedRecency = 'strong';
} else if (hasRecencyOn) {
suggestedRecency = 'on';
} else {
suggestedRecency = 'off';
}
// Salience axis (orthogonal)
let suggestedSalience: SalienceMode;
if (hasCanonical && !hasTemporalBound) {
suggestedSalience = 'off';
} else if (hasSalienceOn) {
suggestedSalience = 'on';
} else {
suggestedSalience = 'off';
}
return { intent, suggestedDetail, suggestedSalience, suggestedRecency };
}
// ─────────────────────────────────────────────────────────
// v0.29.0 compatibility shims
// ─────────────────────────────────────────────────────────
/** v0.29.0 intent type. Preserved verbatim for back-compat. */
export function classifyQueryIntent(query: string): QueryIntent {
if (matches(FULL_CONTEXT_PATTERNS, query)) return 'temporal';
if (matches(TEMPORAL_PATTERNS, query)) return 'temporal';
if (matches(EVENT_PATTERNS, query)) return 'event';
if (matches(ENTITY_PATTERNS, query)) return 'entity';
return 'general';
}
/** v0.29.0 mapping. */
export function intentToDetail(intent: QueryIntent): 'low' | 'medium' | 'high' | undefined {
switch (intent) {
case 'entity': return 'low';
case 'temporal': return 'high';
case 'event': return 'high';
case 'general': return undefined;
}
}
/** v0.29.0 helper. Routes through classifyQuery internally. */
export function autoDetectDetail(query: string): 'low' | 'medium' | 'high' | undefined {
return classifyQuery(query).suggestedDetail;
}
+201
View File
@@ -0,0 +1,201 @@
/**
* v0.29.1 — Per-prefix recency decay map.
*
* Drives the recency boost ONLY (per D9 codex resolution). Salience is a
* separate orthogonal axis based on emotional_weight + take_count and
* does NOT consume this map. The two axes compose multiplicatively in
* runPostFusionStages when both opt in.
*
* Keyed by slug prefix. Longest-prefix-match wins (sorted at lookup time
* inside sql-ranking.ts). Defaults are GENERIC prefixes only (no fork-
* specific names like 'openclaw/chat/' — that's a privacy violation per
* CLAUDE.md and tracked in iteration-1 codex finding C-CX-3).
*
* Override priority (later wins):
* 1. DEFAULT_RECENCY_DECAY (this file)
* 2. gbrain.yml `recency:` section
* 3. GBRAIN_RECENCY_DECAY env var (prefix:halflifeDays:coefficient,...)
* 4. Per-call SearchOpts.recency_decay (tests + library consumers; not
* exposed on MCP)
*
* Per-prefix interpretation:
* - halflifeDays = 0 → evergreen, no decay (recency component = 0)
* - halflifeDays > 0 → hyperbolic decay; coefficient × halflife / (halflife + days_old)
* - At days_old=0: recency component = coefficient (max boost)
* - At days_old=halflife: recency component = coefficient / 2
*
* Pure module. No side effects. Tested in test/recency-decay.test.ts.
*/
export interface RecencyDecayConfig {
/** Days at which the recency component is halved. 0 = no decay (evergreen). */
halflifeDays: number;
/** Max recency boost contribution at days_old = 0. Must be >= 0. */
coefficient: number;
}
export type RecencyDecayMap = Record<string, RecencyDecayConfig>;
export const DEFAULT_RECENCY_DECAY: RecencyDecayMap = {
// Evergreen (curated, opinion, knowledge artifacts) — no decay.
// concepts/ is the canonical evergreen tier; originals/ + writing/ get
// long-tail decay so freshly-published essays do see a small nudge.
'concepts/': { halflifeDays: 0, coefficient: 0 },
'originals/': { halflifeDays: 180, coefficient: 0.5 },
'writing/': { halflifeDays: 365, coefficient: 0.4 },
// Time-bound personal records — strongest decay, biggest coefficient.
// The user is asking "what was on my plate this week" / "what did we
// discuss in our 1:1"; freshness IS the signal.
'daily/': { halflifeDays: 14, coefficient: 1.5 },
'meetings/': { halflifeDays: 60, coefficient: 1.0 },
// Bulk feeds — generic prefixes only. Real fork names go in user
// gbrain.yml, never in shipped defaults.
'chat/': { halflifeDays: 7, coefficient: 1.0 },
'media/x/': { halflifeDays: 7, coefficient: 1.5 },
'media/articles/': { halflifeDays: 90, coefficient: 0.5 },
// Entities — slow decay (a deal from 2 years ago is still relevant
// to a current portfolio query; less so to "what's new lately").
'people/': { halflifeDays: 365, coefficient: 0.3 },
'companies/': { halflifeDays: 365, coefficient: 0.3 },
'deals/': { halflifeDays: 180, coefficient: 0.5 },
};
/** Fallback applied to slugs that don't match any default or override prefix. */
export const DEFAULT_FALLBACK: RecencyDecayConfig = {
halflifeDays: 90,
coefficient: 0.5,
};
/** Sentinel error thrown by parsers; CLI catches it and exits with a useful message. */
export class RecencyDecayParseError extends Error {
constructor(message: string, public readonly source: 'env' | 'yaml' | 'caller') {
super(message);
this.name = 'RecencyDecayParseError';
}
}
/**
* Parse the GBRAIN_RECENCY_DECAY env var.
* Format: comma-separated `prefix:halflifeDays:coefficient` triples.
* Example: "daily/:7:2.0,concepts/:0:0,custom/:30:1.0"
*
* Refuses on parse error (codex M-CX-3 / iteration-2 review). The source-boost
* env parser silently skipped malformed entries; that pattern bit users for
* years. Recency parser fails LOUD so misconfigurations surface at startup
* instead of silently degrading rankings.
*/
export function parseRecencyDecayEnv(env: string | undefined): RecencyDecayMap {
if (!env) return {};
const out: RecencyDecayMap = {};
const triples = env.split(',').map(s => s.trim()).filter(Boolean);
for (const triple of triples) {
// Prefix can't contain `:` because the field separator is `:`. We split
// on the FIRST and SECOND `:` from the right so the prefix may safely
// contain `/` etc. but NOT colons.
const lastIdx = triple.lastIndexOf(':');
if (lastIdx <= 0) {
throw new RecencyDecayParseError(
`Invalid GBRAIN_RECENCY_DECAY entry "${triple}": expected prefix:halflife:coefficient`,
'env',
);
}
const beforeLast = triple.slice(0, lastIdx);
const middleIdx = beforeLast.lastIndexOf(':');
if (middleIdx <= 0) {
throw new RecencyDecayParseError(
`Invalid GBRAIN_RECENCY_DECAY entry "${triple}": expected prefix:halflife:coefficient`,
'env',
);
}
const prefix = triple.slice(0, middleIdx).trim();
const halflifeRaw = triple.slice(middleIdx + 1, lastIdx).trim();
const coefficientRaw = triple.slice(lastIdx + 1).trim();
const halflife = Number.parseFloat(halflifeRaw);
const coefficient = Number.parseFloat(coefficientRaw);
if (!prefix) {
throw new RecencyDecayParseError(`Empty prefix in GBRAIN_RECENCY_DECAY entry "${triple}"`, 'env');
}
if (!Number.isFinite(halflife) || halflife < 0) {
throw new RecencyDecayParseError(
`Invalid halflifeDays "${halflifeRaw}" in GBRAIN_RECENCY_DECAY (must be number >= 0; 0 = evergreen)`,
'env',
);
}
if (!Number.isFinite(coefficient) || coefficient < 0) {
throw new RecencyDecayParseError(
`Invalid coefficient "${coefficientRaw}" in GBRAIN_RECENCY_DECAY (must be number >= 0)`,
'env',
);
}
out[prefix] = { halflifeDays: halflife, coefficient };
}
return out;
}
/**
* Parse a `recency:` section from a parsed gbrain.yml. The shape is:
* recency:
* daily/: { halflifeDays: 14, coefficient: 1.5 }
* concepts/: { halflifeDays: 0, coefficient: 0 }
*
* `parsed` is the already-parsed YAML object. This is a pure transform.
* Caller is responsible for reading + parsing the YAML file.
*/
export function parseRecencyDecayYaml(parsed: unknown): RecencyDecayMap {
if (parsed == null) return {};
if (typeof parsed !== 'object' || Array.isArray(parsed)) return {};
const obj = parsed as Record<string, unknown>;
const recency = obj.recency;
if (recency == null) return {};
if (typeof recency !== 'object' || Array.isArray(recency)) {
throw new RecencyDecayParseError(`gbrain.yml recency: must be a map, got ${typeof recency}`, 'yaml');
}
const out: RecencyDecayMap = {};
for (const [prefix, raw] of Object.entries(recency as Record<string, unknown>)) {
if (typeof raw !== 'object' || raw === null || Array.isArray(raw)) {
throw new RecencyDecayParseError(
`gbrain.yml recency."${prefix}" must be an object with halflifeDays + coefficient`,
'yaml',
);
}
const cfg = raw as Record<string, unknown>;
const halflife = Number(cfg.halflifeDays);
const coefficient = Number(cfg.coefficient);
if (!Number.isFinite(halflife) || halflife < 0) {
throw new RecencyDecayParseError(
`gbrain.yml recency."${prefix}".halflifeDays invalid (must be number >= 0)`,
'yaml',
);
}
if (!Number.isFinite(coefficient) || coefficient < 0) {
throw new RecencyDecayParseError(
`gbrain.yml recency."${prefix}".coefficient invalid (must be number >= 0)`,
'yaml',
);
}
out[prefix] = { halflifeDays: halflife, coefficient };
}
return out;
}
/**
* Merge defaults + yaml + env + caller-supplied overrides into the effective
* decay map. Later sources win. Empty entries are dropped.
*/
export function resolveRecencyDecayMap(opts: {
yaml?: unknown;
envValue?: string;
caller?: RecencyDecayMap;
} = {}): RecencyDecayMap {
const fromYaml = opts.yaml !== undefined ? parseRecencyDecayYaml(opts.yaml) : {};
const fromEnv = parseRecencyDecayEnv(opts.envValue ?? process.env.GBRAIN_RECENCY_DECAY);
return {
...DEFAULT_RECENCY_DECAY,
...fromYaml,
...fromEnv,
...(opts.caller ?? {}),
};
}
+82
View File
@@ -129,5 +129,87 @@ export function buildVisibilityClause(pageAlias: string, sourceAlias: string): s
return `AND ${pageAlias}.deleted_at IS NULL AND NOT ${sourceAlias}.archived`;
}
// ============================================================
// v0.29.1 — Recency component SQL builder
// ============================================================
/**
* Typed expression for "what NOW() should be" in the SQL. Tests pass
* `{ kind: 'fixed', isoUtc }` for deterministic output regardless of wall
* clock. Production callers leave it default (`{ kind: 'now' }`).
*
* The builder constructs the SQL literal internally via escapeSqlLiteral
* for the 'fixed' branch — caller-supplied strings NEVER flow into raw SQL,
* preventing the injection vector codex pass-1 #5 flagged.
*/
export type NowExpr = { kind: 'now' } | { kind: 'fixed'; isoUtc: string };
function nowExprToSql(now: NowExpr): string {
if (now.kind === 'now') return 'NOW()';
return `'${escapeSqlLiteral(now.isoUtc)}'::timestamptz`;
}
/**
* Build the per-row recency component SQL fragment.
*
* For each prefix in the decay map, emit one CASE branch:
* - halflifeDays = 0 (or coefficient = 0) → literal 0 (evergreen short-circuit)
* - halflifeDays > 0 → coefficient * halflife / (halflife + days_old)
*
* Prefixes sorted longest-first so 'media/articles/' matches before 'media/'
* (mirror of buildSourceFactorCase's ordering).
*
* Output is a single SQL expression suitable for SELECT / ORDER BY.
*
* @param slugColumn — qualified column reference (engine-supplied, trusted)
* @param dateExpr — qualified expression for the page's effective date
* (typically `COALESCE(p.effective_date, p.updated_at)`)
* @param decayMap — per-prefix configurations (resolved from defaults +
* yaml + env + caller)
* @param fallback — applied to slugs matching no prefix
* @param now — typed NOW() expression (default `{ kind: 'now' }`)
*/
export function buildRecencyComponentSql(opts: {
slugColumn: string;
dateExpr: string;
decayMap: import('./recency-decay.ts').RecencyDecayMap;
fallback: import('./recency-decay.ts').RecencyDecayConfig;
now?: NowExpr;
}): string {
const { slugColumn, dateExpr, decayMap, fallback } = opts;
const now = opts.now ?? { kind: 'now' };
const nowSql = nowExprToSql(now);
const daysOldSql = `EXTRACT(EPOCH FROM (${nowSql} - ${dateExpr})) / 86400.0`;
const prefixes = Object.keys(decayMap).sort((a, b) => b.length - a.length);
const branches: string[] = [];
for (const prefix of prefixes) {
const cfg = decayMap[prefix];
const literal = buildLikePrefixLiteral(prefix);
if (cfg.halflifeDays === 0 || cfg.coefficient === 0) {
branches.push(`WHEN ${slugColumn} LIKE ${literal} THEN 0`);
} else {
const h = cfg.halflifeDays;
const c = cfg.coefficient;
branches.push(
`WHEN ${slugColumn} LIKE ${literal} THEN ${c} * ${h}.0 / (${h}.0 + ${daysOldSql})`,
);
}
}
let elseSql: string;
if (fallback.halflifeDays === 0 || fallback.coefficient === 0) {
elseSql = '0';
} else {
const h = fallback.halflifeDays;
const c = fallback.coefficient;
elseSql = `${c} * ${h}.0 / (${h}.0 + ${daysOldSql})`;
}
if (branches.length === 0) return `(${elseSql})`;
return `(CASE ${branches.join(' ')} ELSE ${elseSql} END)`;
}
// Exported for unit tests
export const __test__ = { escapeLikePattern, escapeSqlLiteral, buildLikePrefixLiteral };
+140
View File
@@ -0,0 +1,140 @@
/**
* v0.29 — Recent transcripts: read raw `.txt` transcript files from the dream
* cycle's corpus directories and return one-line summaries (or full content)
* filtered by mtime.
*
* Reuses the same corpus-dir resolution + dream-output guard as the v0.23
* synthesize phase. Specifically does NOT depend on or call into the dream
* cycle — this is a simple read-only filesystem walk for human / CLI / MCP-
* via-local-CLI consumption.
*
* Trust: the calling op (`get_recent_transcripts`) gates on `ctx.remote === false`
* so MCP/HTTP can't reach this function with attacker-controlled inputs. CLI
* callers are trusted; the cycle calls `discoverTranscripts` directly.
*/
import { readFileSync, readdirSync, statSync } from 'node:fs';
import { join, basename } from 'node:path';
import type { BrainEngine } from './engine.ts';
import { isDreamOutput } from './cycle/transcript-discovery.ts';
export interface RecentTranscriptOpts {
/** Window in days. Default 7. */
days?: number;
/** When true (default), return ~300-char summary. When false, full content (capped at 100 KB). */
summary?: boolean;
/** Max transcripts (default 50). */
limit?: number;
}
export interface RecentTranscript {
/** Filename basename (no directory). */
path: string;
/** Inferred date if filename matches `YYYY-MM-DD...`, else null. */
date: string | null;
/** Modified time (ISO). */
mtime: string;
/** Full file size in bytes (regardless of summary mode). */
length: number;
/**
* When summary=true: first non-empty line + next ~250 chars. Cheap, deterministic.
* When summary=false: file content capped at 100 KB.
*/
summary: string;
}
const DATE_RE = /^(\d{4}-\d{2}-\d{2})/;
const FULL_READ_CAP = 100 * 1024;
const SUMMARY_HEAD_CHARS = 250;
/**
* Walk the corpus directories configured for the dream cycle, filter to `.txt`
* files modified within `days`, skip dream-generated outputs, and return
* summaries sorted newest first.
*
* Returns [] (not error) when no corpus dir is configured or the dir is empty.
*/
export async function listRecentTranscripts(
engine: BrainEngine,
opts: RecentTranscriptOpts = {},
): Promise<RecentTranscript[]> {
const days = Math.max(0, opts.days ?? 7);
const summary = opts.summary !== false;
const limit = Math.max(1, Math.min(opts.limit ?? 50, 500));
const dirs: string[] = [];
const sessionDir = await engine.getConfig('dream.synthesize.session_corpus_dir');
const meetingDir = await engine.getConfig('dream.synthesize.meeting_transcripts_dir');
if (sessionDir) dirs.push(sessionDir);
if (meetingDir) dirs.push(meetingDir);
if (dirs.length === 0) return [];
const cutoffMs = Date.now() - days * 86400000;
const candidates: { path: string; mtimeMs: number; size: number }[] = [];
for (const dir of dirs) {
let entries: string[];
try {
entries = readdirSync(dir);
} catch {
// Missing dir or permission error → skip silently. The op deliberately
// doesn't surface filesystem-level diagnostics; users running into this
// path should `gbrain doctor` to debug.
continue;
}
for (const name of entries) {
if (!name.endsWith('.txt')) continue;
const full = join(dir, name);
let st;
try {
st = statSync(full);
} catch {
continue;
}
if (!st.isFile()) continue;
if (st.mtimeMs < cutoffMs) continue;
candidates.push({ path: full, mtimeMs: st.mtimeMs, size: st.size });
}
}
// Newest first.
candidates.sort((a, b) => b.mtimeMs - a.mtimeMs);
const out: RecentTranscript[] = [];
for (const c of candidates) {
if (out.length >= limit) break;
let raw: string;
try {
raw = readFileSync(c.path, 'utf-8');
} catch {
continue;
}
// Skip dream-generated outputs (would re-feed the synthesize loop).
if (isDreamOutput(raw)) continue;
const name = basename(c.path);
const dateMatch = DATE_RE.exec(name);
out.push({
path: name,
date: dateMatch ? dateMatch[1] : null,
mtime: new Date(c.mtimeMs).toISOString(),
length: c.size,
summary: summary ? buildSummary(raw) : raw.slice(0, FULL_READ_CAP),
});
}
return out;
}
/**
* First non-empty line + next ~250 chars (cap on the summary body).
* Strips leading whitespace; preserves internal newlines truncated by the cap.
*/
function buildSummary(raw: string): string {
const trimmed = raw.replace(/^[\s]+/, '');
// First non-empty line.
const firstLineEnd = trimmed.search(/\r?\n/);
const firstLine = firstLineEnd === -1 ? trimmed : trimmed.slice(0, firstLineEnd);
const after = firstLineEnd === -1 ? '' : trimmed.slice(firstLineEnd + 1, firstLineEnd + 1 + SUMMARY_HEAD_CHARS);
if (!after) return firstLine;
return `${firstLine}\n${after}`.trim();
}
+187
View File
@@ -56,6 +56,8 @@ export interface Page {
timeline: string;
frontmatter: Record<string, unknown>;
content_hash?: string;
/** v0.29 — deterministic 0..1 score; populated by the recompute_emotional_weight cycle phase. */
emotional_weight?: number;
created_at: Date;
updated_at: Date;
/**
@@ -64,8 +66,44 @@ export interface Page {
* The autopilot purge phase hard-deletes rows where `deleted_at < now() - 72h`.
*/
deleted_at?: Date | null;
/**
* v0.29.1: content date computed from frontmatter precedence chain
* (event_date / date / published / filename / fallback). Populated by
* `computeEffectiveDate`; immune to auto-link updated_at churn. Read by
* the recency boost and since/until filter; nothing in the default search
* path consults it.
*/
effective_date?: Date | null;
/**
* v0.29.1: which precedence step won (`event_date | date | published |
* filename | fallback`). Powers the doctor's `effective_date_health` check
* to detect pages that fell back to updated_at because frontmatter was
* unparseable.
*/
effective_date_source?: EffectiveDateSource | null;
/**
* v0.29.1: basename without extension captured at import (e.g.
* "2024-03-15-acme-call"). Used by computeEffectiveDate for filename-date
* precedence on `daily/` and `meetings/` prefixes. NULL for older rows
* imported pre-v0.29.1.
*/
import_filename?: string | null;
/**
* v0.29.1: bumped by `recompute_emotional_weight` when the page's
* emotional_weight changes. The salience query window uses
* `GREATEST(updated_at, salience_touched_at)` so newly-salient old pages
* surface in `get_recent_salience`.
*/
salience_touched_at?: Date | null;
}
export type EffectiveDateSource =
| 'event_date'
| 'date'
| 'published'
| 'filename'
| 'fallback';
// `image` (v0.27.1): multimodal ingestion path, parallel to markdown + code.
export type PageKind = 'markdown' | 'code' | 'image';
@@ -83,6 +121,17 @@ export interface PageInput {
* `query --lang` filtering.
*/
page_kind?: PageKind;
/**
* v0.29.1: content date from frontmatter precedence (computed by importer
* via `computeEffectiveDate`). When omitted, putPage leaves the column
* unchanged on conflict (preserves any existing value); on insert the
* column is NULL. NULL is fine — recency paths COALESCE to updated_at.
*/
effective_date?: Date | null;
/** v0.29.1: paired with effective_date; NULL when effective_date is NULL. */
effective_date_source?: EffectiveDateSource | null;
/** v0.29.1: basename without extension captured at import. */
import_filename?: string | null;
}
export interface PageFilters {
@@ -107,6 +156,13 @@ export interface PageFilters {
* the 72h window before the autopilot purge phase hard-deletes them.
*/
includeDeleted?: boolean;
/**
* v0.29: ORDER BY enum. Default `updated_desc` matches pre-v0.29 behavior
* (engines hardcoded `ORDER BY updated_at DESC`). New options: `updated_asc`,
* `created_desc`, `slug` (alphabetical, useful for stable pagination).
* Whitelisted enum — no SQL-injection risk; engines map to literal SQL fragments.
*/
sort?: 'updated_desc' | 'updated_asc' | 'created_desc' | 'slug';
}
/** v0.26.5 — opts for getPage / softDeletePage / restorePage. */
@@ -117,6 +173,102 @@ export interface GetPageOpts {
includeDeleted?: boolean;
}
/** v0.29: literal ORDER BY fragments for the PageFilters.sort enum. Whitelisted. */
export const PAGE_SORT_SQL: Record<NonNullable<PageFilters['sort']>, string> = {
updated_desc: 'p.updated_at DESC',
updated_asc: 'p.updated_at ASC',
created_desc: 'p.created_at DESC',
slug: 'p.slug ASC',
};
/**
* v0.29 — Salience: pages ranked by emotional + activity salience over a recency window.
* See `src/core/cycle/emotional-weight.ts` for the score formula and
* `engine.getRecentSalience` for the SQL.
*/
export interface SalienceOpts {
/** Window in days. Default 14. */
days?: number;
/** Max rows to return (clamped at 100). Default 20. */
limit?: number;
/** Optional slug-prefix filter (e.g., `personal`, `wiki/people`). */
slugPrefix?: string;
/**
* v0.29.1 — recency-decay treatment for the salience formula's third term.
* - 'flat' (default): v0.29.0 behavior, `1.0 / (1 + days_old)` for every page
* - 'on': per-prefix decay from DEFAULT_RECENCY_DECAY (concepts/originals
* evergreen; daily/, media/x/ aggressive). Use when the agent wants
* "recency-biased salience" — what's been mattering AND fresh.
* Default preserves v0.29.0 ranking; 'on' is opt-in.
*/
recency_bias?: 'flat' | 'on';
}
export interface SalienceResult {
slug: string;
source_id: string;
title: string;
type: PageType;
updated_at: Date;
emotional_weight: number;
take_count: number;
take_avg_weight: number;
score: number;
}
/**
* v0.29 — Anomaly detection: cohorts (tag, type) with unusually-high activity in a window.
* Cohort baseline is computed over `lookback_days` excluding `since`; current count is
* the number of distinct pages touched on `since`. A cohort is anomalous when its
* current count exceeds `mean + sigma * stddev`. Year cohort deferred to v0.30.
*/
export interface AnomaliesOpts {
/** ISO date (YYYY-MM-DD). Default = today (UTC). */
since?: string;
/** Days of history for the baseline. Default 30. */
lookback_days?: number;
/** Sigma threshold. Default 3.0. */
sigma?: number;
}
export interface AnomalyResult {
cohort_kind: 'tag' | 'type';
cohort_value: string;
count: number;
baseline_mean: number;
baseline_stddev: number;
sigma_observed: number;
page_slugs: string[];
}
/**
* v0.29 — Per-page tag + take inputs to the emotional-weight formula.
* Returned in batch by `engine.batchLoadEmotionalInputs` so the cycle phase
* computes weights for many pages with two SQL round-trips total.
*/
export interface EmotionalWeightInputRow {
slug: string;
source_id: string;
tags: string[];
takes: {
holder: string;
weight: number;
kind: string;
active: boolean;
}[];
}
/**
* v0.29 — Multi-source-safe write batch. Composite-keyed on `(slug, source_id)`
* because `pages.slug` is only unique within a source. Slug-only UPDATE would
* fan out across sources.
*/
export interface EmotionalWeightWriteRow {
slug: string;
source_id: string;
weight: number;
}
// Chunks
export interface Chunk {
id: number;
@@ -270,6 +422,41 @@ export interface SearchOpts {
* is unaffected — modality filtering on the keyword path is independent.
*/
embeddingColumn?: 'embedding' | 'embedding_image';
/**
* @deprecated v0.29.1: use `since` instead. Removed in v0.30.
* v0.27.0: filter results to pages updated/created after this date. ISO-8601 string.
*/
afterDate?: string;
/**
* @deprecated v0.29.1: use `until` instead. Removed in v0.30.
* v0.27.0: filter results to pages updated/created before this date. ISO-8601 string.
*/
beforeDate?: string;
/**
* @deprecated v0.29.1: use `recency` ('off' | 'on' | 'strong') instead. Removed in v0.30.
* v0.27.0: recency boost strength. 0 = off, 1 = moderate, 2 = aggressive.
*/
recencyBoost?: 0 | 1 | 2;
/**
* v0.29.1: salience boost on emotional_weight + take_count. Independent of recency.
* 'off' (default) disables; 'on' applies a moderate boost; 'strong' more aggressive.
*/
salience?: 'off' | 'on' | 'strong';
/**
* v0.29.1: recency boost on per-prefix age decay. Independent of salience.
* 'off' (default) disables; 'on' applies the per-prefix decay map; 'strong' multiplies by 1.5.
*/
recency?: 'off' | 'on' | 'strong';
/**
* v0.29.1: ISO-8601 date OR relative duration ('7d', '2w', '1y'). Filter to
* pages whose effective_date >= this time. Replaces afterDate (kept as alias).
*/
since?: string;
/**
* v0.29.1: same shape as `since`. Filter to effective_date <= this time.
* Boundary semantics: end-of-day for plain YYYY-MM-DD.
*/
until?: string;
}
/**
+21 -6
View File
@@ -43,13 +43,21 @@ export function contentHash(page: PageInput): string {
.digest('hex');
}
function readOptionalDate(raw: unknown): Date | null | undefined {
// Three-state read for columns that may or may not be in the SELECT
// projection: undefined (not selected), null (selected, NULL value),
// Date (selected, populated). Mirrors the v0.26.5 deleted_at pattern.
if (raw === undefined) return undefined;
if (raw === null) return null;
return new Date(raw as string);
}
export function rowToPage(row: Record<string, unknown>): Page {
// v0.26.5: deleted_at is optional in the SELECT projection. When the column
// isn't selected (legacy callers), keep the field absent on the returned object.
const deletedAtRaw = row.deleted_at;
const deletedAt = deletedAtRaw == null
? (deletedAtRaw === null ? null : undefined)
: new Date(deletedAtRaw as string);
const deletedAt = readOptionalDate(row.deleted_at);
const effectiveDate = readOptionalDate(row.effective_date);
const salienceTouchedAt = readOptionalDate(row.salience_touched_at);
const effectiveDateSource = row.effective_date_source as Page['effective_date_source'] | undefined;
const importFilename = row.import_filename as string | null | undefined;
return {
id: row.id as number,
slug: row.slug as string,
@@ -59,9 +67,16 @@ export function rowToPage(row: Record<string, unknown>): Page {
timeline: row.timeline as string,
frontmatter: (typeof row.frontmatter === 'string' ? JSON.parse(row.frontmatter) : row.frontmatter) as Record<string, unknown>,
content_hash: row.content_hash as string | undefined,
// v0.29 (column added in migration v40). Old brains pre-migration return undefined.
emotional_weight: row.emotional_weight == null ? undefined : Number(row.emotional_weight),
created_at: new Date(row.created_at as string),
updated_at: new Date(row.updated_at as string),
...(deletedAt !== undefined && { deleted_at: deletedAt }),
// v0.29.1 (columns added in migration v41). Optional in SELECT projection.
...(effectiveDate !== undefined && { effective_date: effectiveDate }),
...(effectiveDateSource !== undefined && { effective_date_source: effectiveDateSource }),
...(importFilename !== undefined && { import_filename: importFilename }),
...(salienceTouchedAt !== undefined && { salience_touched_at: salienceTouchedAt }),
};
}
+31 -1
View File
@@ -77,6 +77,10 @@ CREATE TABLE IF NOT EXISTS pages (
timeline TEXT NOT NULL DEFAULT '',
frontmatter JSONB NOT NULL DEFAULT '{}',
content_hash TEXT,
-- v0.29: deterministic 0..1 score (tag emotion + take density + Garry-as-holder ratio).
-- Populated by the `recompute_emotional_weight` cycle phase. Default 0.0 so freshly
-- imported pages don't pollute salience ranking before the cycle has run.
emotional_weight REAL NOT NULL DEFAULT 0.0,
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
updated_at TIMESTAMPTZ NOT NULL DEFAULT now(),
-- v0.26.5: soft-delete + recovery window. `delete_page` sets deleted_at = now()
@@ -84,6 +88,17 @@ CREATE TABLE IF NOT EXISTS pages (
-- where deleted_at < now() - 72h. Search and `get_page` filter
-- `WHERE deleted_at IS NULL` by default; `include_deleted: true` opts in.
deleted_at TIMESTAMPTZ,
-- v0.29.1: salience-and-recency, additive opt-in. All NULL by default;
-- only consulted when a caller passes `salience='on'` / `recency='on'` or
-- the new `since`/`until` filter. effective_date_source is a sentinel for
-- the doctor's effective_date_health check (values: 'event_date' | 'date'
-- | 'published' | 'filename' | 'fallback'). salience_touched_at is bumped
-- by recompute_emotional_weight when emotional_weight changes so the
-- salience window picks up newly-salient old pages.
effective_date TIMESTAMPTZ,
effective_date_source TEXT,
import_filename TEXT,
salience_touched_at TIMESTAMPTZ,
CONSTRAINT pages_source_slug_key UNIQUE (source_id, slug)
);
@@ -101,6 +116,12 @@ CREATE INDEX IF NOT EXISTS idx_pages_source_id ON pages(source_id);
-- stays low. Don't add a regular `(deleted_at)` index without measuring.
CREATE INDEX IF NOT EXISTS pages_deleted_at_purge_idx
ON pages (deleted_at) WHERE deleted_at IS NOT NULL;
-- v0.29.1: expression index used by since/until date-range filters that read
-- COALESCE(effective_date, updated_at). A partial index on effective_date
-- alone would NOT help — the planner can't use it for the negative side of
-- the COALESCE. Expression index is what actually accelerates the filter.
CREATE INDEX IF NOT EXISTS pages_coalesce_date_idx
ON pages ((COALESCE(effective_date, updated_at)));
-- ============================================================
-- content_chunks: chunked content with embeddings
@@ -742,7 +763,16 @@ CREATE TABLE IF NOT EXISTS eval_candidates (
remote BOOLEAN NOT NULL,
job_id INTEGER,
subagent_id INTEGER,
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
-- v0.29.1 — agent-explicit recency + salience capture for replay reproducibility.
-- All nullable + additive. NDJSON schema_version stays at 1; consumers ignore unknown fields.
as_of_ts TIMESTAMPTZ,
salience_param TEXT,
recency_param TEXT,
salience_resolved TEXT,
recency_resolved TEXT,
salience_source TEXT,
recency_source TEXT
);
CREATE INDEX IF NOT EXISTS idx_eval_candidates_created_at ON eval_candidates(created_at DESC);
+144
View File
@@ -0,0 +1,144 @@
import { describe, expect, test } from 'bun:test';
import {
meanStddev,
computeAnomaliesFromBuckets,
type CohortDayRow,
type CohortTodayRow,
} from '../src/core/cycle/anomaly.ts';
describe('meanStddev', () => {
test('empty returns (0, 0)', () => {
expect(meanStddev([])).toEqual({ mean: 0, stddev: 0 });
});
test('single sample returns mean=value, stddev=0', () => {
expect(meanStddev([7])).toEqual({ mean: 7, stddev: 0 });
});
test('all-equal returns stddev=0', () => {
const r = meanStddev([3, 3, 3, 3]);
expect(r.mean).toBe(3);
expect(r.stddev).toBe(0);
});
test('sample stddev (n-1 denominator)', () => {
// stddev of [2,4,4,4,5,5,7,9] with sample-stddev = 2.0
const r = meanStddev([2, 4, 4, 4, 5, 5, 7, 9]);
expect(r.mean).toBe(5);
expect(r.stddev).toBeCloseTo(2.138, 2);
});
});
describe('computeAnomaliesFromBuckets', () => {
function densify(values: number[], cohort_value: string, kind: 'tag' | 'type' = 'tag'): CohortDayRow[] {
return values.map((count, i) => ({
cohort_kind: kind,
cohort_value,
day: `2026-04-${String(i + 1).padStart(2, '0')}`,
count,
}));
}
test('clear anomaly: 7-touch day on a 0..1 baseline triggers tag cohort', () => {
const baseline: CohortDayRow[] = densify([0, 0, 1, 0, 0, 0, 1, 0, 0, 0], 'wedding');
const today: CohortTodayRow[] = [
{ cohort_kind: 'tag', cohort_value: 'wedding', count: 7, page_slugs: ['p1','p2','p3','p4','p5','p6','p7'] },
];
const r = computeAnomaliesFromBuckets(baseline, today, 3.0);
expect(r.length).toBe(1);
expect(r[0].cohort_value).toBe('wedding');
expect(r[0].count).toBe(7);
expect(r[0].sigma_observed).toBeGreaterThan(3);
});
test('zero-stddev baseline + count > mean+1 triggers fallback (no NaN)', () => {
// baseline is all 1s (stddev=0), today is 5 → count > mean+1 → anomaly
const baseline: CohortDayRow[] = densify(Array.from({length: 10}, () => 1), 'work');
const today: CohortTodayRow[] = [
{ cohort_kind: 'tag', cohort_value: 'work', count: 5, page_slugs: ['a','b','c','d','e'] },
];
const r = computeAnomaliesFromBuckets(baseline, today, 3.0);
expect(r.length).toBe(1);
expect(r[0].baseline_stddev).toBe(0);
expect(Number.isFinite(r[0].sigma_observed)).toBe(true);
expect(r[0].sigma_observed).toBe(4); // 5 - 1
});
test('zero-stddev baseline + count <= mean+1 does not fire', () => {
const baseline: CohortDayRow[] = densify(Array.from({length: 10}, () => 1), 'work');
const today: CohortTodayRow[] = [
{ cohort_kind: 'tag', cohort_value: 'work', count: 2, page_slugs: ['a','b'] },
];
expect(computeAnomaliesFromBuckets(baseline, today, 3.0)).toEqual([]);
});
test('non-anomalous current count returns empty', () => {
const baseline: CohortDayRow[] = densify([5, 4, 6, 5, 4, 5, 6, 5, 4, 6], 'daily');
const today: CohortTodayRow[] = [
{ cohort_kind: 'tag', cohort_value: 'daily', count: 6, page_slugs: ['a'] },
];
expect(computeAnomaliesFromBuckets(baseline, today, 3.0)).toEqual([]);
});
test('brand-new cohort (no baseline) requires count >= 2', () => {
// No baseline rows for "newtag"; today.count=2 → mean=0, stddev=0, threshold=mean+1=1, 2>1 ✓
const today: CohortTodayRow[] = [
{ cohort_kind: 'tag', cohort_value: 'newtag', count: 2, page_slugs: ['a','b'] },
{ cohort_kind: 'tag', cohort_value: 'singleton', count: 1, page_slugs: ['x'] },
];
const r = computeAnomaliesFromBuckets([], today, 3.0);
expect(r.length).toBe(1);
expect(r[0].cohort_value).toBe('newtag');
});
test('top results sorted by sigma_observed desc', () => {
const baseline: CohortDayRow[] = [
...densify([0,0,0,0,0,0,0,0,0,0], 'low'),
...densify([2,2,2,2,2,2,2,2,2,2], 'medium'),
];
const today: CohortTodayRow[] = [
{ cohort_kind: 'tag', cohort_value: 'low', count: 3, page_slugs: ['a','b','c'] },
{ cohort_kind: 'tag', cohort_value: 'medium', count: 4, page_slugs: ['x','y','z','w'] },
];
const r = computeAnomaliesFromBuckets(baseline, today, 0.5);
// both should fire; "low" has bigger sigma_observed (mean=0) so it's first.
expect(r[0].cohort_value).toBe('low');
});
test('limit caps result count', () => {
const today: CohortTodayRow[] = Array.from({length: 50}, (_, i) => ({
cohort_kind: 'tag' as const,
cohort_value: `tag${i}`,
count: 5,
page_slugs: [`p${i}`],
}));
const r = computeAnomaliesFromBuckets([], today, 3.0, 10);
expect(r.length).toBe(10);
});
test('page_slugs are capped at 50 per cohort', () => {
const slugs = Array.from({length: 100}, (_, i) => `p${i}`);
const today: CohortTodayRow[] = [
{ cohort_kind: 'tag', cohort_value: 'huge', count: 100, page_slugs: slugs },
];
const r = computeAnomaliesFromBuckets([], today, 3.0);
expect(r[0].page_slugs.length).toBe(50);
});
test('cohort_kind=tag and =type are tracked independently', () => {
// Same name "wedding" as both a tag and a type — should not collide.
const baseline: CohortDayRow[] = [
...densify([0,0,0,0,0], 'wedding', 'tag'),
...densify([5,5,5,5,5], 'wedding', 'type'),
];
const today: CohortTodayRow[] = [
{ cohort_kind: 'tag', cohort_value: 'wedding', count: 7, page_slugs: ['t1'] },
{ cohort_kind: 'type', cohort_value: 'wedding', count: 7, page_slugs: ['t2'] },
];
const r = computeAnomaliesFromBuckets(baseline, today, 0.5);
// Tag cohort should fire (mean=0); type cohort might or might not depending on stddev.
const tagEntry = r.find(x => x.cohort_kind === 'tag');
expect(tagEntry).toBeDefined();
expect(tagEntry!.baseline_mean).toBe(0);
});
});
+2 -2
View File
@@ -108,7 +108,7 @@ describe('buildPlan — diff against completed + installed VERSION', () => {
// autopilot cooperative, v0.16.0 = subagent runtime, v0.18.0 = multi-
// source brains, v0.18.1 = RLS hardening, v0.21.0 = Cathedral II
// (renumbered from v0.20.0 after master shipped v0.20.x in parallel).
expect(plan.skippedFuture.map(m => m.version)).toEqual(['0.12.0', '0.12.2', '0.13.0', '0.13.1', '0.14.0', '0.16.0', '0.18.0', '0.18.1', '0.21.0', '0.22.4', '0.28.0']);
expect(plan.skippedFuture.map(m => m.version)).toEqual(['0.12.0', '0.12.2', '0.13.0', '0.13.1', '0.14.0', '0.16.0', '0.18.0', '0.18.1', '0.21.0', '0.22.4', '0.28.0', '0.29.1']);
});
test('already applied → v0.11.0 lands in `applied` bucket, not pending', () => {
@@ -148,7 +148,7 @@ describe('buildPlan — diff against completed + installed VERSION', () => {
// were added later; installed=0.12.0 means they belong in skippedFuture,
// not pending. v0.11.0 and v0.12.0 stay pending despite being ≤ installed —
// that is the H9 invariant.
expect(plan.skippedFuture.map(m => m.version)).toEqual(['0.12.2', '0.13.0', '0.13.1', '0.14.0', '0.16.0', '0.18.0', '0.18.1', '0.21.0', '0.22.4', '0.28.0']);
expect(plan.skippedFuture.map(m => m.version)).toEqual(['0.12.2', '0.13.0', '0.13.1', '0.14.0', '0.16.0', '0.18.0', '0.18.1', '0.21.0', '0.22.4', '0.28.0', '0.29.1']);
});
test('--migration filter narrows to one version', () => {
+1 -1
View File
@@ -13,7 +13,7 @@ import { PGLiteEngine } from '../src/core/pglite-engine.ts';
import { rrfFusion } from '../src/core/search/hybrid.ts';
import { dedupResults } from '../src/core/search/dedup.ts';
import { precisionAtK, recallAtK, mrr, ndcgAtK } from '../src/core/search/eval.ts';
import { autoDetectDetail } from '../src/core/search/intent.ts';
import { autoDetectDetail } from '../src/core/search/query-intent.ts';
import type { SearchResult, ChunkInput } from '../src/core/types.ts';
const RRF_K = 60;
+8 -2
View File
@@ -44,13 +44,19 @@ describe('BRAIN_TOOL_ALLOWLIST', () => {
expect(missing).toEqual([]);
});
test('contains the read-only 10 + put_page', () => {
expect(BRAIN_TOOL_ALLOWLIST.size).toBe(11);
test('contains the v0.15 read-only 10 + put_page + v0.29 salience pair', () => {
// v0.29 added get_recent_salience + find_anomalies (read-only).
// get_recent_transcripts is deliberately excluded — subagent calls always
// have ctx.remote=true, and the v0.29 trust gate rejects remote callers.
expect(BRAIN_TOOL_ALLOWLIST.size).toBe(13);
expect(BRAIN_TOOL_ALLOWLIST.has('query')).toBe(true);
expect(BRAIN_TOOL_ALLOWLIST.has('search')).toBe(true);
expect(BRAIN_TOOL_ALLOWLIST.has('get_page')).toBe(true);
expect(BRAIN_TOOL_ALLOWLIST.has('list_pages')).toBe(true);
expect(BRAIN_TOOL_ALLOWLIST.has('put_page')).toBe(true);
expect(BRAIN_TOOL_ALLOWLIST.has('get_recent_salience')).toBe(true);
expect(BRAIN_TOOL_ALLOWLIST.has('find_anomalies')).toBe(true);
expect(BRAIN_TOOL_ALLOWLIST.has('get_recent_transcripts')).toBe(false);
});
test('does NOT contain destructive ops', () => {
+5 -4
View File
@@ -377,8 +377,9 @@ describe('runCycle — yieldBetweenPhases hook', () => {
hookCalls++;
},
});
// v0.26.5: 9 phases (added `purge`) → 9 yield calls (one after each).
expect(hookCalls).toBe(9);
// v0.26.5: 9 phases (added `purge`).
// v0.29: 10 phases (added `recompute_emotional_weight`) → 10 yield calls.
expect(hookCalls).toBe(10);
});
test('hook exceptions do not abort the cycle', async () => {
@@ -388,8 +389,8 @@ describe('runCycle — yieldBetweenPhases hook', () => {
throw new Error('synthetic hook error');
},
});
// Cycle still completed all phases (v0.26.5: 9 with the new purge phase).
expect(report.phases.length).toBe(9);
// Cycle still completed all phases (v0.29: 10 with recompute_emotional_weight).
expect(report.phases.length).toBe(10);
});
});
+105
View File
@@ -0,0 +1,105 @@
/**
* v0.29 E2E find_anomalies against PGLite.
*
* Same fixture shape as the Garry test: 7 wedding-tagged pages touched today
* + 100 background pages spread across 30 days. Anomaly detection should
* fire on the wedding tag cohort because its baseline is near-zero.
*
* Also covers the brand-new-cohort case (no baseline rows; small-sample
* fallback fires when count >= 2) and the no-anomaly case (steady cohort,
* no spike).
*/
import { describe, test, expect, beforeAll, afterAll } from 'bun:test';
import { PGLiteEngine } from '../../src/core/pglite-engine.ts';
let engine: PGLiteEngine;
const TODAY = new Date().toISOString().slice(0, 10);
beforeAll(async () => {
engine = new PGLiteEngine();
await engine.connect({ engine: 'pglite' } as never);
await engine.initSchema();
// 7 wedding-tagged pages, all updated today (default putPage stamp).
for (let i = 0; i < 7; i++) {
const slug = `personal/wedding/photos-${i}`;
await engine.putPage(slug, {
type: 'note',
title: `Wedding photo ${i}`,
compiled_truth: `Photos from the wedding day, group ${i}.`,
});
await engine.addTag(slug, 'wedding');
}
// 100 background pages, tagged with rotating "steady" tags, backdated.
const RANDOM_TAGS = ['hardware', 'product', 'meeting'];
for (let i = 0; i < 100; i++) {
const slug = `notes/random-${i}`;
await engine.putPage(slug, {
type: 'note',
title: `Random note ${i}`,
compiled_truth: `Body text for note ${i}.`,
});
await engine.addTag(slug, RANDOM_TAGS[i % RANDOM_TAGS.length]);
}
// Spread the random pages randomly across the last 30 days
// (excluding today, so the wedding cohort is the only "today" spike).
await engine.executeRaw(
`UPDATE pages
SET updated_at = now() - interval '1 day' - (random() * interval '29 days')
WHERE slug LIKE 'notes/random-%'`
);
});
afterAll(async () => {
if (engine) await engine.disconnect();
});
describe('v0.29 E2E — findAnomalies (Garry test)', () => {
test('wedding-tag cohort fires as anomaly with sigma > 3 vs zero baseline', async () => {
const rows = await engine.findAnomalies({ since: TODAY, lookback_days: 30, sigma: 3.0 });
const wedding = rows.find(r => r.cohort_kind === 'tag' && r.cohort_value === 'wedding');
expect(wedding).toBeDefined();
expect(wedding!.count).toBe(7);
// Baseline should be ~0 because none of the wedding pages were backdated.
expect(wedding!.baseline_mean).toBeLessThan(1);
expect(wedding!.sigma_observed).toBeGreaterThan(3);
});
test('returned page_slugs sample contains wedding pages', async () => {
const rows = await engine.findAnomalies({ since: TODAY, lookback_days: 30 });
const wedding = rows.find(r => r.cohort_kind === 'tag' && r.cohort_value === 'wedding');
expect(wedding!.page_slugs.length).toBe(7);
expect(wedding!.page_slugs.every(s => s.startsWith('personal/wedding/'))).toBe(true);
});
test('a date with no activity returns []', async () => {
// Look at a date earlier than any seeded page — every cohort has count=0,
// none should fire as anomalous.
const rows = await engine.findAnomalies({
since: '2024-01-15',
lookback_days: 30,
sigma: 3.0,
});
expect(rows).toEqual([]);
});
test('high sigma threshold suppresses borderline cohorts', async () => {
const lowRows = await engine.findAnomalies({ since: TODAY, lookback_days: 30, sigma: 0.5 });
const highRows = await engine.findAnomalies({ since: TODAY, lookback_days: 30, sigma: 100 });
// sigma=100 should suppress every cohort (would need a literally
// impossible spike to fire). The wedding cohort fires at sigma 3 so
// there's enough headroom for sigma=0.5 ⊇ sigma=100.
expect(lowRows.length).toBeGreaterThanOrEqual(highRows.length);
// Wedding-tag (count=7, baseline mean ~0, stddev ~0) shouldn't pass
// sigma=100 because the small-sample fallback uses count > mean+1, not
// sigma scaling.
const weddingHigh = highRows.find(r => r.cohort_kind === 'tag' && r.cohort_value === 'wedding');
if (weddingHigh) {
// It can still fire because the sample-stddev fallback uses count > mean + 1,
// not sigma * stddev. Confirm the sigma_observed is finite (no NaN).
expect(Number.isFinite(weddingHigh.sigma_observed)).toBe(true);
}
});
});
+60
View File
@@ -0,0 +1,60 @@
/**
* v0.29 backfill perf regression guard (codex C4#3+#4).
*
* The first plan revision did per-page reads + per-page writes (N+1) which
* would multi-minute on real brains. The shipped path is two SQL round-trips
* total (CTE-shaped batch read + UPDATE FROM unnest batch write).
*
* This test seeds 1000 pages with random tags + 0-3 takes each, runs the
* recompute_emotional_weight phase against PGLite in-memory, and asserts
* wall-clock < 5s on the same fixture pattern. Goal is to catch a regression
* to N+1 a fast machine on PGLite in-memory should finish in well under
* a second; the 5s budget is generous for slow CI.
*/
import { describe, test, expect, beforeAll, afterAll } from 'bun:test';
import { PGLiteEngine } from '../../src/core/pglite-engine.ts';
import { runPhaseRecomputeEmotionalWeight } from '../../src/core/cycle/recompute-emotional-weight.ts';
let engine: PGLiteEngine;
const TAG_POOL = [
'wedding', 'family', 'work', 'product', 'hardware',
'meeting', 'idea', 'concept', 'people', 'health',
];
beforeAll(async () => {
engine = new PGLiteEngine();
await engine.connect({ engine: 'pglite' } as never);
await engine.initSchema();
// Seed 1000 pages, each with 1-3 tags from the pool.
for (let i = 0; i < 1000; i++) {
const slug = `notes/perf-${i}`;
await engine.putPage(slug, {
type: 'note',
title: `Page ${i}`,
compiled_truth: 'body',
});
const tagCount = 1 + (i % 3);
for (let t = 0; t < tagCount; t++) {
await engine.addTag(slug, TAG_POOL[(i + t) % TAG_POOL.length]);
}
}
}, 60_000);
afterAll(async () => {
if (engine) await engine.disconnect();
});
describe('v0.29 — recompute_emotional_weight perf on a 1000-page fixture', () => {
test('full-mode backfill completes in under 5 seconds', async () => {
const start = Date.now();
const result = await runPhaseRecomputeEmotionalWeight(engine, {});
const elapsedMs = Date.now() - start;
expect(result.status).toBe('ok');
expect(result.pages_recomputed).toBeGreaterThanOrEqual(1000);
expect(elapsedMs).toBeLessThan(5_000);
}, 30_000);
});
@@ -0,0 +1,111 @@
/**
* v0.29 E2E recompute_emotional_weight cycle phase wiring against PGLite.
*
* Asserts:
* - Phase appears in ALL_PHASES and runs as part of a default cycle.
* - Full mode (no incremental anchors) walks every page in the brain.
* - Selectable via `--phase recompute_emotional_weight` (single phase run).
* - dry-run skips the UPDATE but still reports the would-write count.
*/
import { describe, test, expect, beforeAll, afterAll } from 'bun:test';
import { PGLiteEngine } from '../../src/core/pglite-engine.ts';
import { runCycle, ALL_PHASES } from '../../src/core/cycle.ts';
import { mkdtempSync, rmSync, mkdirSync } from 'node:fs';
import { tmpdir } from 'node:os';
import { join } from 'node:path';
let engine: PGLiteEngine;
let brainDir: string;
beforeAll(async () => {
engine = new PGLiteEngine();
await engine.connect({ engine: 'pglite' } as never);
await engine.initSchema();
// Brain dir for filesystem phases (lint/backlinks/sync). They'll skip
// gracefully when there's nothing to read.
brainDir = mkdtempSync(join(tmpdir(), 'gbrain-cycle-test-'));
mkdirSync(join(brainDir, 'wiki'), { recursive: true });
// Seed two pages: one with a high-emotion tag, one without.
await engine.putPage('personal/wedding/photos', {
type: 'note',
title: 'Wedding photos',
compiled_truth: 'Some wedding photos.',
});
await engine.addTag('personal/wedding/photos', 'wedding');
await engine.putPage('notes/random', {
type: 'note',
title: 'Random note',
compiled_truth: 'Just a note.',
});
await engine.addTag('notes/random', 'product');
});
afterAll(async () => {
if (engine) await engine.disconnect();
if (brainDir) rmSync(brainDir, { recursive: true, force: true });
});
describe('v0.29 — recompute_emotional_weight phase is registered', () => {
test('appears in ALL_PHASES between patterns and embed', () => {
const idx = ALL_PHASES.indexOf('recompute_emotional_weight');
const patternsIdx = ALL_PHASES.indexOf('patterns');
const embedIdx = ALL_PHASES.indexOf('embed');
expect(idx).toBeGreaterThan(-1);
expect(idx).toBeGreaterThan(patternsIdx);
expect(idx).toBeLessThan(embedIdx);
});
});
describe('v0.29 — recompute_emotional_weight phase runs end-to-end', () => {
test('--phase recompute_emotional_weight populates the column for every page (full mode)', async () => {
const report = await runCycle(engine, {
brainDir,
phases: ['recompute_emotional_weight'],
});
expect(report.status).not.toBe('failed');
const phaseResult = report.phases.find(p => p.phase === 'recompute_emotional_weight');
expect(phaseResult).toBeDefined();
expect(phaseResult!.status).toBe('ok');
expect(phaseResult!.details.mode).toBe('full');
expect(Number(phaseResult!.details.pages_recomputed)).toBeGreaterThanOrEqual(2);
// Verify both pages got their weights populated.
const wedding = await engine.executeRaw<{ emotional_weight: number }>(
`SELECT emotional_weight FROM pages WHERE slug = 'personal/wedding/photos'`
);
const random = await engine.executeRaw<{ emotional_weight: number }>(
`SELECT emotional_weight FROM pages WHERE slug = 'notes/random'`
);
expect(Number(wedding[0].emotional_weight)).toBeCloseTo(0.5, 5);
expect(Number(random[0].emotional_weight)).toBe(0);
// Totals roll up the new field.
expect(report.totals.pages_emotional_weight_recomputed).toBeGreaterThanOrEqual(2);
});
test('dry-run skips the UPDATE but reports a would-write count', async () => {
// Reset weights to a sentinel so we can detect a write.
await engine.executeRaw(`UPDATE pages SET emotional_weight = 0.99`);
const report = await runCycle(engine, {
brainDir,
phases: ['recompute_emotional_weight'],
dryRun: true,
});
const phaseResult = report.phases.find(p => p.phase === 'recompute_emotional_weight');
expect(phaseResult).toBeDefined();
expect(phaseResult!.status).toBe('ok');
expect(phaseResult!.details.dry_run).toBe(true);
expect(Number(phaseResult!.details.pages_recomputed)).toBeGreaterThanOrEqual(2);
// Sentinel survives because dry-run never writes.
const after = await engine.executeRaw<{ emotional_weight: number }>(
`SELECT emotional_weight FROM pages WHERE slug = 'personal/wedding/photos'`
);
expect(Number(after[0].emotional_weight)).toBeCloseTo(0.99, 5);
});
});
+124
View File
@@ -0,0 +1,124 @@
/**
* v0.29 Engine parity: salience + anomalies on PGLite vs Postgres.
*
* Codex flagged in the v0.22.0 source-boost review that engine-shape
* differences (postgres.js vs PGLite SQL idioms) can silently diverge
* results. The same risk applies to the new v0.29 ops:
* - getRecentSalience uses EXTRACT(EPOCH FROM ...), ln(), GROUP BY p.id.
* - findAnomalies uses generate_series + date_trunc + array_agg.
*
* This test seeds identical fixtures into both engines, runs the v0.29
* ops, and asserts the result sets line up.
*
* DATABASE_URL gated skips gracefully when not set.
*/
import { describe, test, expect, beforeAll, afterAll } from 'bun:test';
import { PGLiteEngine } from '../../src/core/pglite-engine.ts';
import { hasDatabase, setupDB, teardownDB } from './helpers.ts';
import type { BrainEngine } from '../../src/core/engine.ts';
const SKIP_PG = !hasDatabase();
const describeBoth = SKIP_PG ? describe.skip : describe;
const TODAY = new Date().toISOString().slice(0, 10);
async function seedFixture(engine: BrainEngine): Promise<void> {
// 5 wedding-tagged pages, all updated today.
for (let i = 0; i < 5; i++) {
const slug = `personal/wedding/photos-${i}`;
await engine.putPage(slug, {
type: 'note',
title: `Wedding photo ${i}`,
compiled_truth: 'photos',
});
await engine.addTag(slug, 'wedding');
}
// 30 background pages backdated across 30 days.
for (let i = 0; i < 30; i++) {
const slug = `notes/random-${i}`;
await engine.putPage(slug, {
type: 'note',
title: `Random ${i}`,
compiled_truth: 'body',
});
await engine.addTag(slug, ['hardware', 'product', 'idea'][i % 3]);
}
await engine.executeRaw(
`UPDATE pages
SET updated_at = now() - interval '1 day' - (random() * interval '29 days')
WHERE slug LIKE 'notes/random-%'`
);
}
describeBoth('v0.29 engine parity — getRecentSalience', () => {
let pglite: PGLiteEngine;
let postgres: BrainEngine;
beforeAll(async () => {
pglite = new PGLiteEngine();
await pglite.connect({ engine: 'pglite' } as never);
await pglite.initSchema();
await seedFixture(pglite);
postgres = await setupDB();
await seedFixture(postgres);
}, 60_000);
afterAll(async () => {
if (pglite) await pglite.disconnect();
await teardownDB();
});
test('top result is a wedding page on both engines', async () => {
const pgliteRows = await pglite.getRecentSalience({ days: 7, limit: 5 });
const postgresRows = await postgres.getRecentSalience({ days: 7, limit: 5 });
expect(pgliteRows.length).toBeGreaterThan(0);
expect(postgresRows.length).toBeGreaterThan(0);
expect(pgliteRows[0].slug.startsWith('personal/wedding/')).toBe(true);
expect(postgresRows[0].slug.startsWith('personal/wedding/')).toBe(true);
});
test('same set of wedding slugs returned in the top 5 on both engines', async () => {
const pgliteRows = await pglite.getRecentSalience({ days: 7, limit: 10 });
const postgresRows = await postgres.getRecentSalience({ days: 7, limit: 10 });
const pgliteWedding = new Set(pgliteRows.filter(r => r.slug.startsWith('personal/wedding/')).map(r => r.slug));
const postgresWedding = new Set(postgresRows.filter(r => r.slug.startsWith('personal/wedding/')).map(r => r.slug));
expect(pgliteWedding.size).toBe(postgresWedding.size);
for (const s of pgliteWedding) expect(postgresWedding.has(s)).toBe(true);
});
});
describeBoth('v0.29 engine parity — findAnomalies', () => {
let pglite: PGLiteEngine;
let postgres: BrainEngine;
beforeAll(async () => {
pglite = new PGLiteEngine();
await pglite.connect({ engine: 'pglite' } as never);
await pglite.initSchema();
await seedFixture(pglite);
postgres = await setupDB();
await seedFixture(postgres);
}, 60_000);
afterAll(async () => {
if (pglite) await pglite.disconnect();
await teardownDB();
});
test('wedding tag cohort fires on both engines with similar counts', async () => {
const pgliteRows = await pglite.findAnomalies({ since: TODAY, lookback_days: 30, sigma: 2 });
const postgresRows = await postgres.findAnomalies({ since: TODAY, lookback_days: 30, sigma: 2 });
const pgliteWedding = pgliteRows.find(r => r.cohort_kind === 'tag' && r.cohort_value === 'wedding');
const postgresWedding = postgresRows.find(r => r.cohort_kind === 'tag' && r.cohort_value === 'wedding');
expect(pgliteWedding).toBeDefined();
expect(postgresWedding).toBeDefined();
expect(pgliteWedding!.count).toBe(5);
expect(postgresWedding!.count).toBe(5);
// baseline mean should be very small (random-tag pages don't carry "wedding").
expect(pgliteWedding!.baseline_mean).toBeLessThan(1);
expect(postgresWedding!.baseline_mean).toBeLessThan(1);
});
});
+94
View File
@@ -0,0 +1,94 @@
/**
* v0.29 IRON RULE list_pages regression coverage.
*
* Adding optional params (`updated_after`, `sort`) to a long-shipped op
* must not change behavior for callers that only pass the pre-v0.29 shape
* (`type`, `tag`, `limit`). This test asserts the old shape produces the
* pre-v0.29 default order and that the new `sort` enum threads through both
* engine implementations (codex C4#9 engines hardcoded ORDER BY DESC).
*/
import { describe, test, expect, beforeAll, afterAll } from 'bun:test';
import { PGLiteEngine } from '../../src/core/pglite-engine.ts';
let engine: PGLiteEngine;
beforeAll(async () => {
engine = new PGLiteEngine();
await engine.connect({ engine: 'pglite' } as never);
await engine.initSchema();
// Seed 5 pages with deterministic timestamps so order is provable.
for (let i = 0; i < 5; i++) {
await engine.putPage(`alpha/page-${i}`, {
type: 'note',
title: `Page ${i}`,
compiled_truth: 'body',
});
}
// Backdate updated_at so we can test ordering meaningfully.
await engine.executeRaw(
`UPDATE pages SET updated_at = '2026-01-01'::timestamptz + (id * interval '1 day')`
);
});
afterAll(async () => {
if (engine) await engine.disconnect();
});
describe('v0.29 IRON RULE — list_pages back-compat (pre-v0.29 shape)', () => {
test('old call shape (type, tag, limit) returns updated_desc order by default', async () => {
const rows = await engine.listPages({ limit: 10 });
expect(rows.length).toBeGreaterThanOrEqual(5);
// Pre-v0.29 default = ORDER BY updated_at DESC. Our seeding made id=5 the
// newest, so it must appear first.
for (let i = 0; i < rows.length - 1; i++) {
expect(rows[i].updated_at.getTime() >= rows[i + 1].updated_at.getTime()).toBe(true);
}
});
test('updated_after filter narrows to recent rows', async () => {
const rows = await engine.listPages({
limit: 10,
updated_after: '2026-01-04',
});
// Only pages with updated_at > 2026-01-04 — that's id=4 + id=5 in the seed.
expect(rows.length).toBeGreaterThanOrEqual(1);
for (const r of rows) {
expect(r.updated_at.getTime()).toBeGreaterThan(new Date('2026-01-04').getTime());
}
});
});
describe('v0.29 — list_pages sort enum threads through the engine', () => {
test('sort=updated_asc reverses default order', async () => {
const desc = await engine.listPages({ limit: 10, sort: 'updated_desc' });
const asc = await engine.listPages({ limit: 10, sort: 'updated_asc' });
expect(asc.length).toBe(desc.length);
expect(asc[0].slug).toBe(desc[desc.length - 1].slug);
});
test('sort=created_desc orders by created_at, not updated_at', async () => {
const rows = await engine.listPages({ limit: 10, sort: 'created_desc' });
expect(rows.length).toBeGreaterThanOrEqual(5);
// In our seed, id=5 was inserted last → newest created_at → first row.
for (let i = 0; i < rows.length - 1; i++) {
expect(rows[i].created_at.getTime() >= rows[i + 1].created_at.getTime()).toBe(true);
}
});
test('sort=slug returns alphabetical order', async () => {
const rows = await engine.listPages({ limit: 10, sort: 'slug' });
expect(rows.length).toBeGreaterThanOrEqual(5);
const sorted = [...rows].sort((a, b) => a.slug.localeCompare(b.slug));
expect(rows.map(r => r.slug)).toEqual(sorted.map(r => r.slug));
});
test('unsupported sort value falls back to default (does not crash)', async () => {
// An invalid string would be filtered by the handler-side whitelist;
// call the engine directly with a junk value to verify defense-in-depth.
const rows = await engine.listPages({ limit: 10, sort: 'whatever' as any });
// Engine PAGE_SORT_SQL[unknown] is undefined → falls back to default desc.
expect(rows.length).toBeGreaterThan(0);
});
});
@@ -0,0 +1,95 @@
/**
* v0.29 E2E multi-source UPDATE safety (codex C4#3).
*
* pages.slug is unique only within a source_id. A slug-only UPDATE would
* fan out across sources and corrupt other sources' rows. This test seeds
* pages with the SAME slug under two different source_ids, runs
* setEmotionalWeightBatch for one of them, and asserts the other source's
* row is untouched.
*
* Regression guard: if a future maintainer drops the source_id from the
* UPDATE WHERE clause, this test fires.
*/
import { describe, test, expect, beforeAll, afterAll } from 'bun:test';
import { PGLiteEngine } from '../../src/core/pglite-engine.ts';
let engine: PGLiteEngine;
beforeAll(async () => {
engine = new PGLiteEngine();
await engine.connect({ engine: 'pglite' } as never);
await engine.initSchema();
// Register a second source so we can put pages with the same slug
// across two source_ids. (default source is auto-seeded on schema init.)
await engine.executeRaw(
`INSERT INTO sources (id, name) VALUES ('src-b', 'Test Source B')`
);
// Same slug under both sources.
await engine.executeRaw(
`INSERT INTO pages (source_id, slug, type, title, compiled_truth)
VALUES ('default', 'shared/page', 'note', 'Default copy', 'A')`
);
await engine.executeRaw(
`INSERT INTO pages (source_id, slug, type, title, compiled_truth)
VALUES ('src-b', 'shared/page', 'note', 'src-b copy', 'B')`
);
});
afterAll(async () => {
if (engine) await engine.disconnect();
});
describe('v0.29 E2E — setEmotionalWeightBatch is multi-source safe', () => {
test('UPDATE on (slug=shared, source_id=default) leaves src-b untouched', async () => {
const updated = await engine.setEmotionalWeightBatch([
{ slug: 'shared/page', source_id: 'default', weight: 0.42 },
]);
expect(updated).toBe(1); // exactly one row touched
// Verify both rows independently.
const defRow = await engine.executeRaw<{ slug: string; source_id: string; emotional_weight: number }>(
`SELECT slug, source_id, emotional_weight FROM pages
WHERE slug = 'shared/page' AND source_id = 'default'`
);
const srcBRow = await engine.executeRaw<{ slug: string; source_id: string; emotional_weight: number }>(
`SELECT slug, source_id, emotional_weight FROM pages
WHERE slug = 'shared/page' AND source_id = 'src-b'`
);
expect(defRow.length).toBe(1);
expect(srcBRow.length).toBe(1);
expect(Number(defRow[0].emotional_weight)).toBeCloseTo(0.42, 5);
// src-b row stays at default 0.0.
expect(Number(srcBRow[0].emotional_weight)).toBe(0);
});
test('two updates in one batch hit the right sources', async () => {
const updated = await engine.setEmotionalWeightBatch([
{ slug: 'shared/page', source_id: 'default', weight: 0.10 },
{ slug: 'shared/page', source_id: 'src-b', weight: 0.20 },
]);
expect(updated).toBe(2);
const rows = await engine.executeRaw<{ source_id: string; emotional_weight: number }>(
`SELECT source_id, emotional_weight FROM pages WHERE slug = 'shared/page' ORDER BY source_id`
);
expect(rows.length).toBe(2);
const byid = Object.fromEntries(rows.map(r => [r.source_id, Number(r.emotional_weight)]));
expect(byid.default).toBeCloseTo(0.10, 5);
expect(byid['src-b']).toBeCloseTo(0.20, 5);
});
test('non-existent (slug, source_id) tuple is silently skipped (no error)', async () => {
const updated = await engine.setEmotionalWeightBatch([
{ slug: 'shared/page', source_id: 'default', weight: 0.50 }, // exists
{ slug: 'nope/missing', source_id: 'default', weight: 0.99 }, // doesn't exist
{ slug: 'shared/page', source_id: 'src-zzz', weight: 0.99 }, // wrong source_id
]);
// Only the existing tuple is updated.
expect(updated).toBe(1);
});
test('empty batch returns 0', async () => {
const updated = await engine.setEmotionalWeightBatch([]);
expect(updated).toBe(0);
});
});
+160
View File
@@ -0,0 +1,160 @@
/**
* v0.29 LLM routing eval (Tier-2, ANTHROPIC_API_KEY gated).
*
* The whole point of v0.29 is the agent reaches for get_recent_salience
* (or find_anomalies / get_recent_transcripts) instead of running query()
* when the user asks "what's been going on with me?". This test confirms
* the description edits actually drive that routing without it, we ship
* description changes and only learn from production behavior.
*
* Implementation: builds a tools list with the v0.29 op definitions, calls
* Claude with a series of personal-query phrasings, asserts the chosen
* tool is in the v0.29 set. Cost ~$0.10/CI run on Haiku.
*
* Skips gracefully when ANTHROPIC_API_KEY is missing.
*
* Replaces the discarded `skills/{salience,anomalies,transcripts}/routing-eval.jsonl`
* fixtures (codex C1) which would have shipped fake coverage
* `routing-eval.ts` evaluates skill resolver triggers via substring match,
* not MCP tool routing.
*/
import { describe, test, expect } from 'bun:test';
import {
GET_RECENT_SALIENCE_DESCRIPTION,
FIND_ANOMALIES_DESCRIPTION,
GET_RECENT_TRANSCRIPTS_DESCRIPTION,
QUERY_DESCRIPTION,
SEARCH_DESCRIPTION,
} from '../../src/core/operations-descriptions.ts';
const SKIP = !process.env.ANTHROPIC_API_KEY;
const describeIfKey = SKIP ? describe.skip : describe;
interface ToolDef {
name: string;
description: string;
input_schema: { type: 'object'; properties: Record<string, unknown> };
}
const TOOLS: ToolDef[] = [
{
name: 'get_recent_salience',
description: GET_RECENT_SALIENCE_DESCRIPTION,
input_schema: {
type: 'object',
properties: {
days: { type: 'number' },
limit: { type: 'number' },
slugPrefix: { type: 'string' },
},
},
},
{
name: 'find_anomalies',
description: FIND_ANOMALIES_DESCRIPTION,
input_schema: {
type: 'object',
properties: {
since: { type: 'string' },
lookback_days: { type: 'number' },
sigma: { type: 'number' },
},
},
},
{
name: 'get_recent_transcripts',
description: GET_RECENT_TRANSCRIPTS_DESCRIPTION,
input_schema: {
type: 'object',
properties: {
days: { type: 'number' },
summary: { type: 'boolean' },
limit: { type: 'number' },
},
},
},
{
name: 'query',
description: QUERY_DESCRIPTION,
input_schema: {
type: 'object',
properties: {
query: { type: 'string' },
limit: { type: 'number' },
},
},
},
{
name: 'search',
description: SEARCH_DESCRIPTION,
input_schema: {
type: 'object',
properties: {
query: { type: 'string' },
limit: { type: 'number' },
},
},
},
];
const V029_TOOLS = new Set(['get_recent_salience', 'find_anomalies', 'get_recent_transcripts']);
const PERSONAL_QUERY_PHRASINGS = [
'anything crazy happening in my brain lately?',
"what's been going on with me?",
"how have I been?",
"anything notable in my brain?",
"what's been on my mind?",
"what stood out this week?",
"what's hot in my notes?",
"anything weird going on lately?",
"any unusual patterns?",
"what have I been thinking about?",
"what did I talk about yesterday?",
"what's notable in the brain right now?",
];
interface AnthropicResponse {
content: Array<{ type: string; name?: string }>;
stop_reason: string;
}
async function callClaudeWithTools(prompt: string): Promise<{ tool: string | null; raw: AnthropicResponse }> {
const apiKey = process.env.ANTHROPIC_API_KEY!;
const res = await fetch('https://api.anthropic.com/v1/messages', {
method: 'POST',
headers: {
'x-api-key': apiKey,
'anthropic-version': '2023-06-01',
'content-type': 'application/json',
},
body: JSON.stringify({
model: 'claude-haiku-4-5-20251001',
max_tokens: 256,
tools: TOOLS,
messages: [
{
role: 'user',
content: prompt,
},
],
}),
});
if (!res.ok) {
throw new Error(`Anthropic API ${res.status}: ${await res.text()}`);
}
const json = (await res.json()) as AnthropicResponse;
const useBlock = (json.content ?? []).find(b => b.type === 'tool_use');
return { tool: useBlock?.name ?? null, raw: json };
}
describeIfKey('v0.29 — LLM routes personal queries to v0.29 ops, not query() / search()', () => {
for (const prompt of PERSONAL_QUERY_PHRASINGS) {
test(`routes "${prompt}" to a v0.29 tool`, async () => {
const { tool } = await callClaudeWithTools(prompt);
expect(tool).not.toBeNull();
expect(V029_TOOLS.has(tool!)).toBe(true);
}, 30_000);
}
});
+115
View File
@@ -0,0 +1,115 @@
/**
* v0.29 E2E the "Garry test" for salience.
*
* Seeds a fixture: 7 pages tagged `wedding`, all touched today, plus 100
* background pages with random tags spread across 30 days. Asserts that
* `getRecentSalience({days:7})` returns the wedding pages at the top.
*
* Uses raw SQL UPDATE to backdate `updated_at` on the background pages
* (codex C4#7) `engine.putPage` always stamps `updated_at = now()` so
* seeding via the engine alone can't reproduce historical recency windows.
*
* Runs against PGLite in-memory; no DATABASE_URL required.
*/
import { describe, test, expect, beforeAll, afterAll } from 'bun:test';
import { PGLiteEngine } from '../../src/core/pglite-engine.ts';
let engine: PGLiteEngine;
beforeAll(async () => {
engine = new PGLiteEngine();
await engine.connect({ engine: 'pglite' } as never);
await engine.initSchema();
// ── Seed: 7 wedding pages (all touched today, default updated_at).
for (let i = 0; i < 7; i++) {
const slug = `personal/wedding/photos-${i}`;
await engine.putPage(slug, {
type: 'note',
title: `Wedding photo ${i}`,
compiled_truth: `Photos from the wedding day, group ${i}.`,
});
await engine.addTag(slug, 'wedding');
}
// ── Seed: 100 background pages, tagged with miscellaneous tags.
const RANDOM_TAGS = ['hardware', 'product', 'meeting', 'idea', 'people', 'concept'];
for (let i = 0; i < 100; i++) {
const slug = `notes/random-${i}`;
await engine.putPage(slug, {
type: 'note',
title: `Random note ${i}`,
compiled_truth: `Body text for note ${i}.`,
});
await engine.addTag(slug, RANDOM_TAGS[i % RANDOM_TAGS.length]);
}
// ── Backdate background pages across the last 30 days via raw SQL
// (codex C4#7 — putPage stamps updated_at = now(), so we can't get
// historical timestamps without bypassing the engine path).
await engine.executeRaw(
`UPDATE pages
SET updated_at = now() - (random() * interval '30 days')
WHERE slug LIKE 'notes/random-%'`
);
// Recompute emotional_weight for the wedding pages so they get the
// tag-emotion boost in the salience formula.
const inputs = await engine.batchLoadEmotionalInputs();
const { computeEmotionalWeight } = await import('../../src/core/cycle/emotional-weight.ts');
const writes = inputs.map(r => ({
slug: r.slug,
source_id: r.source_id,
weight: computeEmotionalWeight({ tags: r.tags, takes: r.takes }),
}));
await engine.setEmotionalWeightBatch(writes);
});
afterAll(async () => {
if (engine) await engine.disconnect();
});
describe('v0.29 E2E — getRecentSalience (Garry test)', () => {
test('wedding pages outrank random-tag noise in the 7-day window', async () => {
const rows = await engine.getRecentSalience({ days: 7, limit: 20 });
expect(rows.length).toBeGreaterThan(0);
// The top result should be a wedding page (max emotional_weight = 0.5).
const top = rows[0];
expect(top.slug).toMatch(/^personal\/wedding\//);
expect(top.emotional_weight).toBeGreaterThan(0);
// All 7 wedding pages should appear in the top 10. Compare against the
// result-set, not the rank order — score ties on emotional_weight + the
// recency-decay term may shuffle within the wedding cohort.
const top10 = rows.slice(0, 10).map(r => r.slug);
const weddingHits = top10.filter(s => s.startsWith('personal/wedding/'));
expect(weddingHits.length).toBeGreaterThanOrEqual(7);
});
test('slugPrefix filter narrows to the named directory', async () => {
const rows = await engine.getRecentSalience({ days: 30, slugPrefix: 'personal/wedding/' });
expect(rows.length).toBe(7);
for (const r of rows) {
expect(r.slug.startsWith('personal/wedding/')).toBe(true);
}
});
test('days=0 returns no rows (boundary case)', async () => {
// boundary = now 0 = now, so only pages updated > now are matched.
// updated_at = now() inserts are inclusive, so allow at most a few rows
// that match the equality boundary; assert window is at least narrow.
const rows = await engine.getRecentSalience({ days: 0, limit: 1000 });
expect(rows.length).toBeLessThanOrEqual(7); // only wedding pages from this run
});
test('limit cap is respected', async () => {
const rows = await engine.getRecentSalience({ days: 365, limit: 5 });
expect(rows.length).toBe(5);
});
test('empty-window slugPrefix returns []', async () => {
const rows = await engine.getRecentSalience({ days: 7, slugPrefix: 'nope/does-not-exist/' });
expect(rows).toEqual([]);
});
});
+149
View File
@@ -0,0 +1,149 @@
/**
* v0.29 E2E MCP dispatch path for the three new ops.
*
* Existing v0.29 e2e tests call engine methods directly. This file goes
* through the full `dispatchToolCall` pipeline same code path that
* stdio MCP and HTTP MCP use so we get coverage for:
*
* 1. validateParams (params shape contract per op definition)
* 2. buildOperationContext (ctx.remote, ctx.engine, ctx.config wiring)
* 3. handler invocation + JSON serialization (ToolResult shape)
* 4. Error path: OperationError isError + JSON envelope
* 5. Trust gate: ctx.remote === true on get_recent_transcripts must
* reach the handler and produce a permission_denied error.
*
* Runs against PGLite in-memory. No DATABASE_URL, no API keys.
*/
import { describe, test, expect, beforeAll, afterAll } from 'bun:test';
import { PGLiteEngine } from '../../src/core/pglite-engine.ts';
import { dispatchToolCall } from '../../src/mcp/dispatch.ts';
let engine: PGLiteEngine;
beforeAll(async () => {
engine = new PGLiteEngine();
await engine.connect({ engine: 'pglite' } as never);
await engine.initSchema();
// Seed enough fixture for the salience + anomalies ops to return non-empty
// rows. 5 wedding pages (today), 5 random-tag pages backdated 14 days.
for (let i = 0; i < 5; i++) {
const slug = `personal/wedding/photo-${i}`;
await engine.putPage(slug, {
type: 'note',
title: `Wedding photo ${i}`,
compiled_truth: `Photos from the day, batch ${i}.`,
});
await engine.addTag(slug, 'wedding');
}
for (let i = 0; i < 5; i++) {
const slug = `notes/bg-${i}`;
await engine.putPage(slug, {
type: 'note',
title: `Background ${i}`,
compiled_truth: `Body ${i}.`,
});
await engine.addTag(slug, 'product');
}
await engine.executeRaw(
`UPDATE pages SET updated_at = now() - interval '14 days'
WHERE slug LIKE 'notes/bg-%'`,
);
// Populate emotional_weight so the salience query produces an ordering.
const inputs = await engine.batchLoadEmotionalInputs();
const { computeEmotionalWeight } = await import(
'../../src/core/cycle/emotional-weight.ts'
);
const rows = inputs.map(r => ({
slug: r.slug,
source_id: r.source_id,
weight: computeEmotionalWeight({ tags: r.tags, takes: r.takes }),
}));
await engine.setEmotionalWeightBatch(rows);
});
afterAll(async () => {
if (engine) await engine.disconnect();
});
describe('v0.29 E2E — dispatchToolCall for the three new ops', () => {
test('get_recent_salience returns ranked rows via the MCP dispatch path', async () => {
const result = await dispatchToolCall(engine, 'get_recent_salience', {
days: 7,
limit: 10,
}, { remote: true });
expect(result.isError).toBeFalsy();
expect(result.content[0].type).toBe('text');
const rows = JSON.parse(result.content[0].text);
expect(Array.isArray(rows)).toBe(true);
expect(rows.length).toBeGreaterThan(0);
// Wedding pages should be at or near the top (max tag-emotion boost).
expect(rows[0].slug).toMatch(/^personal\/wedding\//);
});
test('find_anomalies returns cohort outliers via the MCP dispatch path', async () => {
const result = await dispatchToolCall(engine, 'find_anomalies', {
lookback_days: 30,
sigma: 1.5, // lower threshold so the small fixture tips the cohort
}, { remote: true });
expect(result.isError).toBeFalsy();
const rows = JSON.parse(result.content[0].text);
expect(Array.isArray(rows)).toBe(true);
// Schema check on one cohort if anything fired (small fixture may not
// always trip 1.5σ; this is a smoke contract for the response shape).
if (rows.length > 0) {
const row = rows[0];
expect(row).toHaveProperty('cohort_kind');
expect(row).toHaveProperty('cohort_value');
expect(row).toHaveProperty('count');
expect(row).toHaveProperty('baseline_mean');
expect(row).toHaveProperty('baseline_stddev');
expect(row).toHaveProperty('sigma_observed');
expect(Array.isArray(row.page_slugs)).toBe(true);
}
});
test('get_recent_transcripts rejects with permission_denied when ctx.remote === true', async () => {
// Defense-in-depth: even though serve-http filters localOnly: true ops
// out of the MCP tool list, the in-handler ctx.remote check is the
// last line. dispatchToolCall defaults remote=true, which is what
// every MCP transport sets, so the reject must fire here.
const result = await dispatchToolCall(engine, 'get_recent_transcripts', {
days: 7,
}, { remote: true });
expect(result.isError).toBe(true);
const err = JSON.parse(result.content[0].text);
// OperationError.toJSON() serializes the code as `error:`, not `code:`.
expect(err.error).toBe('permission_denied');
expect(err.message.toLowerCase()).toContain('local-only');
});
test('get_recent_transcripts succeeds when ctx.remote === false (CLI path)', async () => {
// The local-CLI path explicitly sets remote: false. Op should run
// (returning [] is fine — no corpus dir is configured in this test
// fixture; the test just asserts the trust gate didn't reject).
const result = await dispatchToolCall(engine, 'get_recent_transcripts', {
days: 7,
}, { remote: false });
expect(result.isError).toBeFalsy();
const rows = JSON.parse(result.content[0].text);
expect(Array.isArray(rows)).toBe(true);
// No corpus_dir configured for this test brain → empty array, not error.
expect(rows).toEqual([]);
});
test('unknown tool returns Unknown tool error envelope (regression guard)', async () => {
// Generic dispatch shape contract — protects against typos in op
// names accidentally short-circuiting elsewhere in the dispatcher.
const result = await dispatchToolCall(engine, 'get_recent_definitely_not_a_real_op', {}, { remote: true });
expect(result.isError).toBe(true);
expect(result.content[0].text).toMatch(/Unknown tool/);
});
});
+170
View File
@@ -0,0 +1,170 @@
/**
* v0.29.1 Tests for computeEffectiveDate (precedence chain + per-prefix
* override + range validation + parse-failure fall-through).
*
* The function is pure (no DB), so these are fast unit tests.
*/
import { describe, test, expect } from 'bun:test';
import { computeEffectiveDate, parseDateLoose } from '../src/core/effective-date.ts';
const baseUpdated = new Date('2026-05-04T12:00:00Z');
const baseCreated = new Date('2026-05-01T12:00:00Z');
function run(opts: {
slug?: string;
fm?: Record<string, unknown>;
filename?: string | null;
updatedAt?: Date;
createdAt?: Date;
}) {
return computeEffectiveDate({
slug: opts.slug ?? 'wiki/example',
frontmatter: opts.fm ?? {},
filename: opts.filename ?? null,
updatedAt: opts.updatedAt ?? baseUpdated,
createdAt: opts.createdAt ?? baseCreated,
});
}
describe('parseDateLoose', () => {
test('Date instance passthrough', () => {
const d = new Date('2024-03-15');
expect(parseDateLoose(d)?.getTime()).toBe(d.getTime());
});
test('ISO string parses', () => {
const d = parseDateLoose('2024-03-15T00:00:00Z');
expect(d?.toISOString()).toBe('2024-03-15T00:00:00.000Z');
});
test('YYYY-MM-DD string parses', () => {
const d = parseDateLoose('2024-03-15');
expect(d?.toISOString().startsWith('2024-03-15')).toBe(true);
});
test('null/undefined → null', () => {
expect(parseDateLoose(null)).toBeNull();
expect(parseDateLoose(undefined)).toBeNull();
});
test('invalid Date → null', () => {
expect(parseDateLoose(new Date('not a date'))).toBeNull();
});
test('unparseable string → null', () => {
expect(parseDateLoose('tomorrow')).toBeNull();
expect(parseDateLoose('garbage')).toBeNull();
expect(parseDateLoose('')).toBeNull();
});
});
describe('computeEffectiveDate precedence chain (default order)', () => {
test('event_date wins when present', () => {
const r = run({ fm: { event_date: '2024-03-15', date: '2024-04-01', published: '2024-05-01' } });
expect(r.source).toBe('event_date');
expect(r.date?.toISOString().startsWith('2024-03-15')).toBe(true);
});
test('date wins when event_date absent', () => {
const r = run({ fm: { date: '2024-04-01', published: '2024-05-01' } });
expect(r.source).toBe('date');
expect(r.date?.toISOString().startsWith('2024-04-01')).toBe(true);
});
test('published wins when event_date + date absent', () => {
const r = run({ fm: { published: '2024-05-01' } });
expect(r.source).toBe('published');
expect(r.date?.toISOString().startsWith('2024-05-01')).toBe(true);
});
test('filename wins when no frontmatter dates', () => {
const r = run({ filename: '2024-06-15-some-meeting' });
expect(r.source).toBe('filename');
expect(r.date?.toISOString().startsWith('2024-06-15')).toBe(true);
});
test('fallback to updated_at when chain exhausted', () => {
const r = run({});
expect(r.source).toBe('fallback');
expect(r.date?.toISOString()).toBe(baseUpdated.toISOString());
});
});
describe('computeEffectiveDate per-prefix override (daily/, meetings/)', () => {
test('daily/ filename wins over event_date', () => {
const r = run({
slug: 'daily/2024-03-15',
fm: { event_date: '2024-04-01' },
filename: '2024-03-15',
});
expect(r.source).toBe('filename');
expect(r.date?.toISOString().startsWith('2024-03-15')).toBe(true);
});
test('meetings/ filename wins over date', () => {
const r = run({
slug: 'meetings/2024-06-15-acme-call',
fm: { date: '2024-07-01' },
filename: '2024-06-15-acme-call',
});
expect(r.source).toBe('filename');
expect(r.date?.toISOString().startsWith('2024-06-15')).toBe(true);
});
test('daily/ falls through to event_date when filename has no date', () => {
const r = run({
slug: 'daily/notes',
fm: { event_date: '2024-04-01' },
filename: 'notes-some-text',
});
expect(r.source).toBe('event_date');
expect(r.date?.toISOString().startsWith('2024-04-01')).toBe(true);
});
test('non-prefixed slug uses default precedence (event_date over filename)', () => {
const r = run({
slug: 'wiki/people/widget-ceo',
fm: { event_date: '2024-04-01' },
filename: '2024-06-15-widget-ceo',
});
expect(r.source).toBe('event_date');
expect(r.date?.toISOString().startsWith('2024-04-01')).toBe(true);
});
});
describe('computeEffectiveDate parse failure fall-through', () => {
test('event_date "tomorrow" falls through to date', () => {
const r = run({ fm: { event_date: 'tomorrow', date: '2024-04-01' } });
expect(r.source).toBe('date');
expect(r.date?.toISOString().startsWith('2024-04-01')).toBe(true);
});
test('all frontmatter dates unparseable → filename wins', () => {
const r = run({
fm: { event_date: 'garbage', date: 'tomorrow', published: 'last week' },
filename: '2024-06-15-something',
});
expect(r.source).toBe('filename');
expect(r.date?.toISOString().startsWith('2024-06-15')).toBe(true);
});
test('filename without date prefix → fallback', () => {
const r = run({ filename: 'no-date-here' });
expect(r.source).toBe('fallback');
expect(r.date?.toISOString()).toBe(baseUpdated.toISOString());
});
});
describe('computeEffectiveDate range validation [1990, NOW + 1y]', () => {
test('pre-1990 frontmatter date drops to next chain element', () => {
const r = run({ fm: { event_date: '1985-01-01', date: '2024-04-01' } });
expect(r.source).toBe('date');
});
test('far-future frontmatter date drops to next chain element', () => {
// NOW is 2026-05-04 in test fixtures; 2030 is > NOW + 1y
const r = run({ fm: { event_date: '2030-01-01', date: '2024-04-01' } });
expect(r.source).toBe('date');
});
test('out-of-range filename date drops to fallback', () => {
const r = run({ filename: '1850-01-01-ancient' });
expect(r.source).toBe('fallback');
});
});
+114
View File
@@ -0,0 +1,114 @@
import { describe, expect, test } from 'bun:test';
import {
computeEmotionalWeight,
HIGH_EMOTION_TAGS,
DEFAULT_USER_HOLDER,
} from '../src/core/cycle/emotional-weight.ts';
describe('computeEmotionalWeight', () => {
test('empty inputs return 0', () => {
expect(computeEmotionalWeight({ tags: [], takes: [] })).toBe(0);
});
test('high-emotion tag alone gives 0.5', () => {
const w = computeEmotionalWeight({ tags: ['wedding'], takes: [] });
expect(w).toBeCloseTo(0.5, 5);
});
test('tag matching is case-insensitive', () => {
const w = computeEmotionalWeight({ tags: ['WEDDING'], takes: [] });
expect(w).toBeCloseTo(0.5, 5);
});
test('non-emotion tag gives no boost', () => {
const w = computeEmotionalWeight({ tags: ['hardware', 'product'], takes: [] });
expect(w).toBe(0);
});
test('multiple high-emotion tags do not stack (cap at 0.5)', () => {
const w = computeEmotionalWeight({ tags: ['family', 'wedding', 'love'], takes: [] });
expect(w).toBeCloseTo(0.5, 5);
});
test('takes-only: density caps at 0.3 with 3+ takes', () => {
const takes = Array.from({ length: 5 }, () => ({
holder: 'other',
weight: 0,
kind: 'fact',
active: true,
}));
// density = min(5*0.1, 0.3) = 0.3; avg-weight = 0; holder-ratio = 0 (none match user).
expect(computeEmotionalWeight({ tags: [], takes })).toBeCloseTo(0.3, 5);
});
test('takes-only: avg-weight contribution scales 0..0.1', () => {
const takes = [
{ holder: 'other', weight: 1.0, kind: 'take', active: true },
];
// density = 0.1; avg-weight = 1.0 * 0.1 = 0.1; holder-ratio = 0.
expect(computeEmotionalWeight({ tags: [], takes })).toBeCloseTo(0.2, 5);
});
test('user-holder takes give the 0.1 ratio bump', () => {
const takes = [
{ holder: DEFAULT_USER_HOLDER, weight: 0, kind: 'take', active: true },
];
// density = 0.1; avg-weight = 0; holder-ratio = 1.0 * 0.1 = 0.1.
expect(computeEmotionalWeight({ tags: [], takes })).toBeCloseTo(0.2, 5);
});
test('user-holder ratio is mixed-holder aware', () => {
const takes = [
{ holder: DEFAULT_USER_HOLDER, weight: 0, kind: 'take', active: true },
{ holder: 'other', weight: 0, kind: 'take', active: true },
];
// 1 of 2 active takes = 0.5 ratio * 0.1 = 0.05; density = 0.2; avg = 0.
expect(computeEmotionalWeight({ tags: [], takes })).toBeCloseTo(0.25, 5);
});
test('inactive takes are excluded from density + ratio + avg', () => {
const takes = [
{ holder: DEFAULT_USER_HOLDER, weight: 1, kind: 'take', active: false },
{ holder: DEFAULT_USER_HOLDER, weight: 1, kind: 'take', active: false },
];
expect(computeEmotionalWeight({ tags: [], takes })).toBe(0);
});
test('output is bounded to [0..1]', () => {
// High-emotion tag + max takes + max weights + all-user-holder = 0.5+0.3+0.1+0.1 = 1.0
const takes = Array.from({ length: 10 }, () => ({
holder: DEFAULT_USER_HOLDER,
weight: 1.0,
kind: 'take',
active: true,
}));
const w = computeEmotionalWeight({ tags: ['wedding'], takes });
expect(w).toBeCloseTo(1.0, 5);
expect(w).toBeLessThanOrEqual(1);
});
test('over-1 take weights are clamped before averaging', () => {
// weight=2.0 should clamp to 1.0 in the avg path.
const takes = [
{ holder: 'other', weight: 2.0, kind: 'take', active: true },
];
expect(computeEmotionalWeight({ tags: [], takes })).toBeCloseTo(0.2, 5);
});
test('custom highEmotionTags override default', () => {
const customTags: ReadonlySet<string> = new Set(['hardware-failure']);
const w = computeEmotionalWeight(
{ tags: ['hardware-failure'], takes: [] },
{ highEmotionTags: customTags },
);
expect(w).toBeCloseTo(0.5, 5);
// Default seed list still excludes hardware-failure so without override:
expect(computeEmotionalWeight({ tags: ['hardware-failure'], takes: [] })).toBe(0);
});
test('HIGH_EMOTION_TAGS includes the v1 seed list', () => {
expect(HIGH_EMOTION_TAGS.has('wedding')).toBe(true);
expect(HIGH_EMOTION_TAGS.has('family')).toBe(true);
expect(HIGH_EMOTION_TAGS.has('mental-health')).toBe(true);
});
});
+34
View File
@@ -1146,6 +1146,40 @@ describe('migration v31 — eval_capture_tables', () => {
});
});
describe('migration v40 — pages_emotional_weight (v0.29)', () => {
// v0.29 ships off master. Master is at v39 (multimodal_dual_column_v0_27_1);
// v0.29 lands at v40. Idempotent ADD COLUMN IF NOT EXISTS, so brains that
// applied this at any prior number on a feature branch see v40 as new and
// run cleanly.
test('exists with the expected name', () => {
const v40 = MIGRATIONS.find(m => m.version === 40);
expect(v40).toBeDefined();
expect(v40?.name).toBe('pages_emotional_weight');
});
test('adds emotional_weight REAL NOT NULL DEFAULT 0.0 to pages', () => {
const v40 = MIGRATIONS.find(m => m.version === 40);
const sql = v40!.sql || '';
expect(sql).toContain('ALTER TABLE pages');
expect(sql).toContain('ADD COLUMN IF NOT EXISTS emotional_weight');
expect(sql).toContain('REAL');
expect(sql).toContain('NOT NULL DEFAULT 0.0');
});
test('does NOT create an idx_pages_emotional_weight index (eng review D6)', () => {
// Salience query orders by computed score, not raw weight; the index
// would never be used. Adding it later requires a separate migration.
const v40 = MIGRATIONS.find(m => m.version === 40);
const sql = v40!.sql || '';
expect(sql).not.toContain('idx_pages_emotional_weight');
expect(sql).not.toContain('CREATE INDEX');
});
test('LATEST_VERSION caught up to 40', () => {
expect(LATEST_VERSION).toBeGreaterThanOrEqual(40);
});
});
// ─────────────────────────────────────────────────────────────────
// PR #363 regression guards — session timeouts via startup parameters
// resolveSessionTimeouts — GBRAIN_*_TIMEOUT env overrides
+143
View File
@@ -0,0 +1,143 @@
import { describe, expect, test } from 'bun:test';
import {
GET_RECENT_SALIENCE_DESCRIPTION,
FIND_ANOMALIES_DESCRIPTION,
GET_RECENT_TRANSCRIPTS_DESCRIPTION,
LIST_PAGES_DESCRIPTION,
QUERY_DESCRIPTION,
SEARCH_DESCRIPTION,
} from '../src/core/operations-descriptions.ts';
import { operations, operationsByName } from '../src/core/operations.ts';
import { BRAIN_TOOL_ALLOWLIST } from '../src/core/minions/tools/brain-allowlist.ts';
/**
* Tool descriptions are LLM-facing strings that drive routing. v0.29 adds
* three new ops + redirects on three existing ones. These tests pin the
* key phrases that the routing decision depends on so accidental edits
* (description rewrites, AI cleanup, voice changes) fail CI.
*/
describe('v0.29 — get_recent_salience description', () => {
test('matches the operation registration', () => {
expect(operationsByName['get_recent_salience'].description).toBe(GET_RECENT_SALIENCE_DESCRIPTION);
});
test('contains the explicit "Use this when" trigger phrase', () => {
expect(GET_RECENT_SALIENCE_DESCRIPTION).toContain("Use this when the user asks");
});
test('lists the personal-query trigger keywords', () => {
expect(GET_RECENT_SALIENCE_DESCRIPTION).toContain("what's been going on");
expect(GET_RECENT_SALIENCE_DESCRIPTION).toContain("anything crazy happening");
expect(GET_RECENT_SALIENCE_DESCRIPTION).toContain("notable");
});
test('explicitly bans semantic search for the same intent', () => {
expect(GET_RECENT_SALIENCE_DESCRIPTION).toContain("Do NOT run a semantic search for these");
});
});
describe('v0.29 — find_anomalies description', () => {
test('matches the operation registration', () => {
expect(operationsByName['find_anomalies'].description).toBe(FIND_ANOMALIES_DESCRIPTION);
});
test('mentions the cohort framing', () => {
expect(FIND_ANOMALIES_DESCRIPTION).toContain("grouped by cohort");
expect(FIND_ANOMALIES_DESCRIPTION).toContain("(tag or type)");
});
test('lists the unusual / what-stood-out trigger phrases', () => {
expect(FIND_ANOMALIES_DESCRIPTION).toContain("stood out");
expect(FIND_ANOMALIES_DESCRIPTION).toContain("unusual");
});
test('does not promise year-cohort behavior (deferred to v0.30)', () => {
// v0.29 ships tag + type only. The phrase below confirms the description
// does not lie about coverage — surfacing year would route the LLM to
// call the op for date-bucket questions it can't actually serve.
expect(FIND_ANOMALIES_DESCRIPTION).toContain("Cohort kinds: tag, type");
});
});
describe('v0.29 — get_recent_transcripts description', () => {
test('matches the operation registration', () => {
expect(operationsByName['get_recent_transcripts'].description).toBe(GET_RECENT_TRANSCRIPTS_DESCRIPTION);
});
test('mandates priority over query/search for personal questions', () => {
expect(GET_RECENT_TRANSCRIPTS_DESCRIPTION).toContain("FIRST");
});
test('explains raw vs polished distinction', () => {
expect(GET_RECENT_TRANSCRIPTS_DESCRIPTION).toContain("NOT polished");
expect(GET_RECENT_TRANSCRIPTS_DESCRIPTION).toContain("canonical source");
});
test('discloses the local-only constraint to the LLM', () => {
expect(GET_RECENT_TRANSCRIPTS_DESCRIPTION).toContain("Local-only");
expect(GET_RECENT_TRANSCRIPTS_DESCRIPTION).toContain("permission_denied");
});
});
describe('v0.29 — redirect hints on existing ops', () => {
test('list_pages mentions sort=updated_desc as the recency-question answer', () => {
expect(operationsByName['list_pages'].description).toBe(LIST_PAGES_DESCRIPTION);
expect(LIST_PAGES_DESCRIPTION).toContain("sort=updated_desc");
expect(LIST_PAGES_DESCRIPTION).toContain("what did I touch this week");
});
test('query redirects personal/emotional queries to the v0.29 ops', () => {
expect(operationsByName['query'].description).toBe(QUERY_DESCRIPTION);
expect(QUERY_DESCRIPTION).toContain("get_recent_salience");
expect(QUERY_DESCRIPTION).toContain("find_anomalies");
expect(QUERY_DESCRIPTION).toContain("get_recent_transcripts");
});
test('query warns the LLM not to assume "crazy" means impressive', () => {
expect(QUERY_DESCRIPTION).toContain("Do NOT assume");
expect(QUERY_DESCRIPTION).toContain("difficult or emotionally charged");
});
test('search has the shorter redirect hint', () => {
expect(operationsByName['search'].description).toBe(SEARCH_DESCRIPTION);
expect(SEARCH_DESCRIPTION).toContain("get_recent_salience");
});
});
describe('v0.29 — subagent allow-list', () => {
test('includes get_recent_salience and find_anomalies', () => {
expect(BRAIN_TOOL_ALLOWLIST.has('get_recent_salience')).toBe(true);
expect(BRAIN_TOOL_ALLOWLIST.has('find_anomalies')).toBe(true);
});
test('excludes get_recent_transcripts (codex C3 — would be a remote=true footgun)', () => {
// The op throws permission_denied for remote=true callers, and all subagent
// calls run with remote=true. Including it in the allow-list would mean
// every subagent call to it returns an error — looks like a bug.
expect(BRAIN_TOOL_ALLOWLIST.has('get_recent_transcripts')).toBe(false);
});
test('all v0.29 ops in the allow-list resolve to a registered Operation', () => {
// brain-allowlist invariant: every name maps to an entry in operations.ts
// OPERATIONS array. This guard catches rename drift.
for (const name of ['get_recent_salience', 'find_anomalies']) {
expect(operationsByName[name]).toBeDefined();
}
});
test('list_pages has new sort + updated_after params surfaced to MCP', () => {
const op = operationsByName['list_pages'];
expect(op.params.sort).toBeDefined();
expect(op.params.updated_after).toBeDefined();
});
});
describe('v0.29 — operations array carries the three new ops', () => {
test('all three are registered (one allow-listed pair + one local-only)', () => {
const names = operations.map(o => o.name);
expect(names).toContain('get_recent_salience');
expect(names).toContain('find_anomalies');
expect(names).toContain('get_recent_transcripts');
});
});
@@ -3,7 +3,7 @@
*/
import { describe, test, expect } from 'bun:test';
import { classifyQueryIntent, autoDetectDetail } from '../src/core/search/intent.ts';
import { classifyQueryIntent, autoDetectDetail } from '../src/core/search/query-intent.ts';
describe('classifyQueryIntent', () => {
describe('entity queries', () => {
+157
View File
@@ -0,0 +1,157 @@
/**
* v0.29.1 merged query-intent classifier tests.
*
* Covers the new classifyQuery(query) returning {intent, suggestedDetail,
* suggestedSalience, suggestedRecency}. Legacy intent.ts behavior is
* preserved in test/query-intent-legacy.test.ts (which imports the
* classifyQueryIntent + autoDetectDetail compat shims).
*
* Pure regex; no DB.
*/
import { describe, test, expect } from 'bun:test';
import { classifyQuery } from '../src/core/search/query-intent.ts';
describe('classifyQuery — entity / canonical queries → both axes off', () => {
test('"who is widget-ceo" → recency=off, salience=off', () => {
const r = classifyQuery('who is widget-ceo');
expect(r.intent).toBe('entity');
expect(r.suggestedRecency).toBe('off');
expect(r.suggestedSalience).toBe('off');
expect(r.suggestedDetail).toBe('low');
});
test('"what is recursion" → both off', () => {
const r = classifyQuery('what is recursion');
expect(r.suggestedRecency).toBe('off');
expect(r.suggestedSalience).toBe('off');
});
test('"tell me about widget-co" → both off', () => {
const r = classifyQuery('tell me about widget-co');
expect(r.suggestedRecency).toBe('off');
expect(r.suggestedSalience).toBe('off');
});
test('"history of X" → both off (canonical)', () => {
const r = classifyQuery('history of acme corp');
expect(r.suggestedRecency).toBe('off');
expect(r.suggestedSalience).toBe('off');
});
test('code lookup syntax → both off', () => {
const r = classifyQuery('Foo::bar() returns null');
expect(r.suggestedRecency).toBe('off');
expect(r.suggestedSalience).toBe('off');
});
test('graph traversal language → both off', () => {
const r = classifyQuery('show me backlinks to widget-co');
expect(r.suggestedRecency).toBe('off');
expect(r.suggestedSalience).toBe('off');
});
});
describe('classifyQuery — current-state queries → both axes on', () => {
test('"what\'s going on with widget-co" → both on', () => {
const r = classifyQuery("what's going on with widget-co");
expect(r.suggestedRecency).toBe('on');
expect(r.suggestedSalience).toBe('on');
});
test('"catch me up on acme" → both on', () => {
const r = classifyQuery('catch me up on acme');
expect(r.suggestedRecency).toBe('on');
expect(r.suggestedSalience).toBe('on');
});
test('"prep me for the widget-ceo meeting" → both on', () => {
const r = classifyQuery('prep me for the widget-ceo meeting');
expect(r.suggestedRecency).toBe('on');
expect(r.suggestedSalience).toBe('on');
});
test('"before my meeting with X" → both on', () => {
const r = classifyQuery('before my meeting with widget-ceo');
expect(r.suggestedRecency).toBe('on');
expect(r.suggestedSalience).toBe('on');
});
test('"remind me about acme" → both on', () => {
const r = classifyQuery('remind me about acme');
expect(r.suggestedRecency).toBe('on');
expect(r.suggestedSalience).toBe('on');
});
});
describe('classifyQuery — recency-only patterns (no salience signal)', () => {
test('"latest news on AI" → recency=on, salience=off', () => {
const r = classifyQuery('latest news on AI');
expect(r.suggestedRecency).toBe('on');
expect(r.suggestedSalience).toBe('off');
});
test('"this week\'s updates" → recency=on, salience=off', () => {
const r = classifyQuery("this week's updates");
expect(r.suggestedRecency).toBe('on');
// "updates" + "on/with/from" pattern needed for salience
expect(r.suggestedSalience).toBe('off');
});
});
describe('classifyQuery — strong recency ("today" / "right now")', () => {
test('"what happened today" → recency=strong', () => {
const r = classifyQuery('what happened today');
expect(r.suggestedRecency).toBe('strong');
});
test('"right now what is the status" → strong', () => {
const r = classifyQuery('right now what is the status of the deal');
// "what is" canonical fires; but "right now" is a temporal bound
expect(r.suggestedRecency).toBe('strong');
});
});
describe('classifyQuery — D6 narrow temporal-bound exception', () => {
test('"who is widget-ceo right now" → recency=strong (temporal bound wins)', () => {
const r = classifyQuery('who is widget-ceo right now');
expect(r.suggestedRecency).toBe('strong');
});
test('"who is widget-ceo today" → recency=strong', () => {
const r = classifyQuery('who is widget-ceo today');
expect(r.suggestedRecency).toBe('strong');
});
test('"who is widget-ceo" (no bound) → recency=off (canonical wins)', () => {
const r = classifyQuery('who is widget-ceo');
expect(r.suggestedRecency).toBe('off');
});
test('"what is widget-co\'s status this week" → recency=on (temporal bound wins)', () => {
const r = classifyQuery("what is widget-co's status this week");
expect(r.suggestedRecency).toBe('on');
});
});
describe('classifyQuery — orthogonality of axes', () => {
test('default plain query → both off', () => {
const r = classifyQuery('the quick brown fox');
expect(r.suggestedRecency).toBe('off');
expect(r.suggestedSalience).toBe('off');
expect(r.intent).toBe('general');
expect(r.suggestedDetail).toBeUndefined();
});
test('intent vs recency are independent axes', () => {
// "when did widget-co IPO": both 'when' (temporal) and 'IPO' (event)
// match v0.29.0 patterns. classifyQueryIntent's priority is
// temporal > event so .intent = 'temporal'. But recency depends on
// CANONICAL/RECENCY_ON patterns, not on .intent — neither set
// matches here, so suggestedRecency = 'off'.
const r = classifyQuery('when did widget-co IPO');
expect(r.intent).toBe('temporal');
expect(r.suggestedRecency).toBe('off');
expect(r.suggestedSalience).toBe('off');
});
});
+221
View File
@@ -0,0 +1,221 @@
/**
* v0.29.1 recency-decay map + buildRecencyComponentSql tests.
*
* Pure functions, no DB. Fast unit tests. Cover the full env / yaml / merge
* resolution chain plus the SQL CASE shape (longest-prefix-match, evergreen
* short-circuit, injection-safe NowExpr).
*/
import { describe, test, expect } from 'bun:test';
import {
DEFAULT_RECENCY_DECAY,
DEFAULT_FALLBACK,
RecencyDecayParseError,
parseRecencyDecayEnv,
parseRecencyDecayYaml,
resolveRecencyDecayMap,
} from '../src/core/search/recency-decay.ts';
import { buildRecencyComponentSql } from '../src/core/search/sql-ranking.ts';
describe('parseRecencyDecayEnv', () => {
test('empty / undefined → empty map', () => {
expect(parseRecencyDecayEnv(undefined)).toEqual({});
expect(parseRecencyDecayEnv('')).toEqual({});
});
test('single triple', () => {
expect(parseRecencyDecayEnv('daily/:7:1.5')).toEqual({
'daily/': { halflifeDays: 7, coefficient: 1.5 },
});
});
test('multiple triples comma-separated', () => {
const out = parseRecencyDecayEnv('daily/:7:1.5,concepts/:0:0,custom/:30:0.5');
expect(out['daily/']).toEqual({ halflifeDays: 7, coefficient: 1.5 });
expect(out['concepts/']).toEqual({ halflifeDays: 0, coefficient: 0 });
expect(out['custom/']).toEqual({ halflifeDays: 30, coefficient: 0.5 });
});
test('throws on missing field', () => {
expect(() => parseRecencyDecayEnv('daily/:7')).toThrow(RecencyDecayParseError);
expect(() => parseRecencyDecayEnv('daily/')).toThrow(RecencyDecayParseError);
});
test('throws on negative halflife', () => {
expect(() => parseRecencyDecayEnv('daily/:-1:1.5')).toThrow(RecencyDecayParseError);
});
test('throws on negative coefficient', () => {
expect(() => parseRecencyDecayEnv('daily/:7:-0.1')).toThrow(RecencyDecayParseError);
});
test('throws on non-numeric values', () => {
expect(() => parseRecencyDecayEnv('daily/:abc:1.5')).toThrow(RecencyDecayParseError);
});
test('throws on empty prefix', () => {
expect(() => parseRecencyDecayEnv(':7:1.5')).toThrow(RecencyDecayParseError);
});
});
describe('parseRecencyDecayYaml', () => {
test('null / undefined / empty → empty map', () => {
expect(parseRecencyDecayYaml(null)).toEqual({});
expect(parseRecencyDecayYaml(undefined)).toEqual({});
expect(parseRecencyDecayYaml({})).toEqual({});
});
test('valid recency block', () => {
const out = parseRecencyDecayYaml({
recency: {
'daily/': { halflifeDays: 14, coefficient: 1.5 },
'concepts/': { halflifeDays: 0, coefficient: 0 },
},
});
expect(out['daily/']).toEqual({ halflifeDays: 14, coefficient: 1.5 });
expect(out['concepts/']).toEqual({ halflifeDays: 0, coefficient: 0 });
});
test('throws on bad halflifeDays', () => {
expect(() =>
parseRecencyDecayYaml({ recency: { 'daily/': { halflifeDays: -1, coefficient: 1.0 } } }),
).toThrow(RecencyDecayParseError);
});
test('throws on non-object entry', () => {
expect(() =>
parseRecencyDecayYaml({ recency: { 'daily/': 'invalid' } }),
).toThrow(RecencyDecayParseError);
});
});
describe('resolveRecencyDecayMap merge precedence', () => {
test('defaults baseline', () => {
const m = resolveRecencyDecayMap({});
expect(m['concepts/']).toEqual({ halflifeDays: 0, coefficient: 0 });
expect(m['daily/']).toEqual({ halflifeDays: 14, coefficient: 1.5 });
});
test('env overrides defaults', () => {
const m = resolveRecencyDecayMap({ envValue: 'daily/:30:0.5' });
expect(m['daily/']).toEqual({ halflifeDays: 30, coefficient: 0.5 });
});
test('yaml + env: env wins', () => {
const m = resolveRecencyDecayMap({
yaml: { recency: { 'daily/': { halflifeDays: 7, coefficient: 2.0 } } },
envValue: 'daily/:30:0.5',
});
expect(m['daily/']).toEqual({ halflifeDays: 30, coefficient: 0.5 });
});
test('caller wins over env', () => {
const m = resolveRecencyDecayMap({
envValue: 'daily/:30:0.5',
caller: { 'daily/': { halflifeDays: 1, coefficient: 5.0 } },
});
expect(m['daily/']).toEqual({ halflifeDays: 1, coefficient: 5.0 });
});
});
describe('buildRecencyComponentSql', () => {
const mini = {
'concepts/': { halflifeDays: 0, coefficient: 0 },
'daily/': { halflifeDays: 14, coefficient: 1.5 },
'media/': { halflifeDays: 90, coefficient: 0.5 },
};
test('emits CASE expression with longest-prefix-first ordering', () => {
const longerFirst = {
'media/articles/': { halflifeDays: 60, coefficient: 0.5 },
'media/': { halflifeDays: 90, coefficient: 0.4 },
};
const sql = buildRecencyComponentSql({
slugColumn: 'p.slug',
dateExpr: 'p.updated_at',
decayMap: longerFirst,
fallback: DEFAULT_FALLBACK,
});
const idxLong = sql.indexOf("'media/articles/%'");
const idxShort = sql.indexOf("'media/%'");
expect(idxLong).toBeGreaterThan(0);
expect(idxShort).toBeGreaterThan(0);
expect(idxLong).toBeLessThan(idxShort);
});
test('evergreen short-circuit emits literal 0', () => {
const sql = buildRecencyComponentSql({
slugColumn: 'p.slug',
dateExpr: 'p.updated_at',
decayMap: mini,
fallback: DEFAULT_FALLBACK,
});
expect(sql).toContain("WHEN p.slug LIKE 'concepts/%' THEN 0");
});
test('non-zero branches include EXTRACT(EPOCH ...)', () => {
const sql = buildRecencyComponentSql({
slugColumn: 'p.slug',
dateExpr: 'p.updated_at',
decayMap: mini,
fallback: DEFAULT_FALLBACK,
});
expect(sql).toContain('EXTRACT(EPOCH FROM (NOW() - p.updated_at)) / 86400.0');
expect(sql).toContain('1.5 * 14.0 / (14.0 + EXTRACT(EPOCH');
});
test('NowExpr.fixed is escaped (single-quote doubling) and timestamptz-cast', () => {
const sql = buildRecencyComponentSql({
slugColumn: 'p.slug',
dateExpr: 'p.updated_at',
decayMap: { 'daily/': { halflifeDays: 7, coefficient: 1.0 } },
fallback: DEFAULT_FALLBACK,
now: { kind: 'fixed', isoUtc: "2026-05-04T00:00:00Z" },
});
expect(sql).toContain("'2026-05-04T00:00:00Z'::timestamptz");
expect(sql).not.toContain('NOW()');
});
test('NowExpr.fixed with embedded single quote is doubled (injection defense)', () => {
const sql = buildRecencyComponentSql({
slugColumn: 'p.slug',
dateExpr: 'p.updated_at',
decayMap: { 'daily/': { halflifeDays: 7, coefficient: 1.0 } },
fallback: DEFAULT_FALLBACK,
now: { kind: 'fixed', isoUtc: "2026'; DROP TABLE pages;--" },
});
// The malicious quote must be doubled to ''.
expect(sql).toContain("''");
expect(sql).not.toContain("DROP TABLE'");
});
test('empty decayMap → only fallback ELSE branch', () => {
const sql = buildRecencyComponentSql({
slugColumn: 'p.slug',
dateExpr: 'p.updated_at',
decayMap: {},
fallback: { halflifeDays: 30, coefficient: 1.0 },
});
expect(sql).not.toContain('CASE');
expect(sql).toContain('1 * 30.0 / (30.0 +');
});
});
describe('DEFAULT_RECENCY_DECAY composition', () => {
test('does not contain fork-specific names (no openclaw/, no wintermute/)', () => {
const keys = Object.keys(DEFAULT_RECENCY_DECAY);
for (const k of keys) {
expect(k.includes('openclaw')).toBe(false);
expect(k.includes('wintermute')).toBe(false);
}
});
test('concepts/ is evergreen (halflifeDays = 0)', () => {
expect(DEFAULT_RECENCY_DECAY['concepts/']?.halflifeDays).toBe(0);
});
test('daily/ has aggressive decay', () => {
expect(DEFAULT_RECENCY_DECAY['daily/']?.halflifeDays).toBeLessThan(30);
expect(DEFAULT_RECENCY_DECAY['daily/']?.coefficient).toBeGreaterThan(1);
});
});
+133
View File
@@ -0,0 +1,133 @@
import { describe, expect, test } from 'bun:test';
import { runPhaseRecomputeEmotionalWeight } from '../src/core/cycle/recompute-emotional-weight.ts';
import type {
EmotionalWeightInputRow,
EmotionalWeightWriteRow,
} from '../src/core/types.ts';
/**
* Unit-level coverage for the v0.29 recompute_emotional_weight phase.
* The full e2e (against PGLite) is in test/e2e/cycle-recompute-emotional-weight-pglite.test.ts.
*/
interface FakeEngine {
batchLoadEmotionalInputs(slugs?: string[]): Promise<EmotionalWeightInputRow[]>;
setEmotionalWeightBatch(rows: EmotionalWeightWriteRow[]): Promise<number>;
getConfig(key: string): Promise<string | null>;
}
function makeEngine(rows: EmotionalWeightInputRow[], configMap: Record<string, string | null> = {}): FakeEngine & {
written: EmotionalWeightWriteRow[];
loadCalls: (string[] | undefined)[];
} {
const written: EmotionalWeightWriteRow[] = [];
const loadCalls: (string[] | undefined)[] = [];
return {
written,
loadCalls,
async batchLoadEmotionalInputs(slugs?: string[]) {
loadCalls.push(slugs);
if (!slugs) return rows;
const sset = new Set(slugs);
return rows.filter(r => sset.has(r.slug));
},
async setEmotionalWeightBatch(rs: EmotionalWeightWriteRow[]) {
written.push(...rs);
return rs.length;
},
async getConfig(key: string) {
return configMap[key] ?? null;
},
};
}
describe('runPhaseRecomputeEmotionalWeight', () => {
test('zero affected slugs short-circuits — no DB read, no write', async () => {
const engine = makeEngine([]);
const r = await runPhaseRecomputeEmotionalWeight(engine as any, { affectedSlugs: [] });
expect(r.status).toBe('ok');
expect(r.pages_recomputed).toBe(0);
expect(engine.loadCalls.length).toBe(0); // skipped batch read
expect(engine.written.length).toBe(0);
});
test('full mode walks every page', async () => {
const rows: EmotionalWeightInputRow[] = [
{ slug: 'a', source_id: 'default', tags: ['wedding'], takes: [] },
{ slug: 'b', source_id: 'default', tags: [], takes: [] },
];
const engine = makeEngine(rows);
const r = await runPhaseRecomputeEmotionalWeight(engine as any, {});
expect(r.status).toBe('ok');
expect(r.pages_recomputed).toBe(2);
expect(engine.written.length).toBe(2);
// Both rows present, with weights from the formula.
const byslug = Object.fromEntries(engine.written.map(w => [w.slug, w.weight]));
expect(byslug.a).toBeCloseTo(0.5, 5); // wedding tag
expect(byslug.b).toBe(0);
});
test('incremental mode passes slugs through to engine read', async () => {
const rows: EmotionalWeightInputRow[] = [
{ slug: 'a', source_id: 'default', tags: ['wedding'], takes: [] },
{ slug: 'b', source_id: 'default', tags: ['family'], takes: [] },
{ slug: 'c', source_id: 'default', tags: [], takes: [] },
];
const engine = makeEngine(rows);
const r = await runPhaseRecomputeEmotionalWeight(engine as any, { affectedSlugs: ['a', 'c'] });
expect(r.status).toBe('ok');
expect(engine.loadCalls[0]).toEqual(['a', 'c']);
// Filter on the fake engine returns only a + c → both written.
expect(engine.written.map(w => w.slug).sort()).toEqual(['a', 'c']);
});
test('multi-source: writes preserve source_id (no fan-out)', async () => {
const rows: EmotionalWeightInputRow[] = [
{ slug: 'shared', source_id: 'src-a', tags: ['wedding'], takes: [] },
{ slug: 'shared', source_id: 'src-b', tags: [], takes: [] },
];
const engine = makeEngine(rows);
await runPhaseRecomputeEmotionalWeight(engine as any, {});
const sources = engine.written.map(w => w.source_id).sort();
expect(sources).toEqual(['src-a', 'src-b']);
const wA = engine.written.find(w => w.source_id === 'src-a');
const wB = engine.written.find(w => w.source_id === 'src-b');
expect(wA!.weight).toBeCloseTo(0.5, 5);
expect(wB!.weight).toBe(0);
});
test('dry-run computes weights but skips the UPDATE', async () => {
const rows: EmotionalWeightInputRow[] = [
{ slug: 'a', source_id: 'default', tags: ['wedding'], takes: [] },
];
const engine = makeEngine(rows);
const r = await runPhaseRecomputeEmotionalWeight(engine as any, { dryRun: true });
expect(r.status).toBe('ok');
expect(r.details.dry_run).toBe(true);
expect(r.pages_recomputed).toBe(1); // would-write count
expect(engine.written.length).toBe(0); // but nothing actually written
});
test('config override of high_emotion_tags is honored', async () => {
const rows: EmotionalWeightInputRow[] = [
{ slug: 'p', source_id: 'default', tags: ['hardware-failure'], takes: [] },
];
const engine = makeEngine(rows, {
'emotional_weight.high_tags': JSON.stringify(['hardware-failure']),
});
await runPhaseRecomputeEmotionalWeight(engine as any, {});
expect(engine.written[0].weight).toBeCloseTo(0.5, 5);
});
test('engine throw bubbles into a fail PhaseResult, not an unhandled exception', async () => {
const engine: FakeEngine = {
batchLoadEmotionalInputs: async () => { throw new Error('db down'); },
setEmotionalWeightBatch: async () => 0,
getConfig: async () => null,
};
const r = await runPhaseRecomputeEmotionalWeight(engine as any, {});
expect(r.status).toBe('fail');
expect(r.error?.code).toBe('RECOMPUTE_EMOTIONAL_WEIGHT_FAIL');
expect(r.error?.message).toContain('db down');
});
});
+42
View File
@@ -0,0 +1,42 @@
import { describe, expect, test } from 'bun:test';
import { computeAnomaliesFromBuckets } from '../src/core/cycle/anomaly.ts';
/**
* Unit-level salience checks. The full Garry-test fixture lives in
* test/e2e/salience-pglite.test.ts (PGLite, no DATABASE_URL needed).
*
* These are the pure-function checks for the CLI args parser and a smoke
* for the engine method shape via a minimal fake. Most of the salience
* logic is SQL see e2e for behavior validation.
*/
describe('v0.29 — salience SQL shape pinned via type contract', () => {
// The salience score formula is in postgres-engine.ts and pglite-engine.ts
// SQL strings. These are smoke-tested against the engines in
// test/e2e/salience-pglite.test.ts and (optionally) the Postgres parity test.
// Here we just confirm the SalienceResult fields are present on the type
// so any future renames break compilation, not runtime.
test('SalienceResult is a stable contract', async () => {
const mod = await import('../src/core/types.ts');
// If any of these fields is dropped, tsc fails before tests run.
const sample = {} as import('../src/core/types.ts').SalienceResult;
void sample.slug;
void sample.source_id;
void sample.title;
void sample.type;
void sample.updated_at;
void sample.emotional_weight;
void sample.take_count;
void sample.take_avg_weight;
void sample.score;
expect(typeof mod).toBe('object');
});
});
describe('v0.29 — anomaly cohort buckets connect to the engine', () => {
test('computeAnomaliesFromBuckets is exported and pure', () => {
// Smoke that the import path engines use is wired. The behavior is
// covered exhaustively in test/anomalies.test.ts.
expect(typeof computeAnomaliesFromBuckets).toBe('function');
});
});
+159
View File
@@ -0,0 +1,159 @@
import { describe, expect, test, beforeAll, afterAll, beforeEach } from 'bun:test';
import { mkdtempSync, writeFileSync, rmSync, mkdirSync, utimesSync } from 'node:fs';
import { tmpdir } from 'node:os';
import { join } from 'node:path';
import { listRecentTranscripts } from '../src/core/transcripts.ts';
/**
* v0.29 listRecentTranscripts unit coverage.
*
* Uses a hermetic temp dir as the corpus dir. Engine calls are mocked
* minimally we only use engine.getConfig to resolve corpus paths.
*/
let tmpRoot: string;
let corpusDir: string;
const configMap = new Map<string, string | null>();
const fakeEngine = {
async getConfig(key: string): Promise<string | null> {
return configMap.get(key) ?? null;
},
} as unknown as Parameters<typeof listRecentTranscripts>[0];
function setMtime(path: string, mtimeMs: number) {
utimesSync(path, mtimeMs / 1000, mtimeMs / 1000);
}
beforeAll(() => {
tmpRoot = mkdtempSync(join(tmpdir(), 'gbrain-transcripts-test-'));
corpusDir = join(tmpRoot, 'sessions');
mkdirSync(corpusDir, { recursive: true });
});
afterAll(() => {
if (tmpRoot) rmSync(tmpRoot, { recursive: true, force: true });
});
beforeEach(() => {
configMap.clear();
configMap.set('dream.synthesize.session_corpus_dir', corpusDir);
});
describe('listRecentTranscripts', () => {
test('returns [] when no corpus dir is configured', async () => {
configMap.clear();
const result = await listRecentTranscripts(fakeEngine);
expect(result).toEqual([]);
});
test('returns [] when the corpus dir is empty', async () => {
// Empty dir, no files yet.
const subdir = join(tmpRoot, 'empty-dir');
mkdirSync(subdir, { recursive: true });
configMap.set('dream.synthesize.session_corpus_dir', subdir);
const result = await listRecentTranscripts(fakeEngine);
expect(result).toEqual([]);
});
test('returns transcripts within mtime window, newest first', async () => {
const a = join(corpusDir, '2026-04-25-session-a.txt');
const b = join(corpusDir, '2026-04-26-session-b.txt');
writeFileSync(a, 'A content first line\nmore content here.\n');
writeFileSync(b, 'B content first line\nmore content there.\n');
// Set b newer than a.
setMtime(a, Date.now() - 86400000); // 1 day ago
setMtime(b, Date.now() - 3600000); // 1 hour ago
const result = await listRecentTranscripts(fakeEngine, { days: 7 });
expect(result.length).toBe(2);
expect(result[0].path).toBe('2026-04-26-session-b.txt');
expect(result[1].path).toBe('2026-04-25-session-a.txt');
});
test('mtime window excludes older files', async () => {
const old = join(corpusDir, '2020-01-01-old.txt');
writeFileSync(old, 'old content\n');
setMtime(old, Date.now() - 365 * 86400000); // 1 year ago
const result = await listRecentTranscripts(fakeEngine, { days: 7 });
expect(result.find(r => r.path === '2020-01-01-old.txt')).toBeUndefined();
});
test('skips dream-generated outputs', async () => {
const dream = join(corpusDir, '2026-04-26-dream-out.txt');
// Identity marker: ---\n + frontmatter with dream_generated: true.
writeFileSync(dream, '---\ntitle: A reflection\ndream_generated: true\n---\n\nbody\n');
setMtime(dream, Date.now() - 60_000);
const result = await listRecentTranscripts(fakeEngine, { days: 7 });
expect(result.find(r => r.path === '2026-04-26-dream-out.txt')).toBeUndefined();
});
test('summary=true returns first non-empty line + ~250 trailing chars', async () => {
const file = join(corpusDir, '2026-04-26-summary.txt');
const body = 'First line of the transcript\n' + 'x'.repeat(500);
writeFileSync(file, body);
setMtime(file, Date.now());
const result = await listRecentTranscripts(fakeEngine, { days: 7, summary: true });
const row = result.find(r => r.path === '2026-04-26-summary.txt');
expect(row).toBeDefined();
expect(row!.summary.startsWith('First line of the transcript')).toBe(true);
// The summary should be much shorter than the full body.
expect(row!.summary.length).toBeLessThan(body.length);
expect(row!.summary.length).toBeLessThan(400);
});
test('summary=false returns full content capped at 100 KB', async () => {
const file = join(corpusDir, '2026-04-26-full.txt');
const body = 'big body\n' + 'y'.repeat(200_000);
writeFileSync(file, body);
setMtime(file, Date.now());
const result = await listRecentTranscripts(fakeEngine, { days: 7, summary: false });
const row = result.find(r => r.path === '2026-04-26-full.txt');
expect(row).toBeDefined();
expect(row!.summary.length).toBeLessThanOrEqual(100 * 1024);
expect(row!.length).toBe(body.length); // length always = full file size
});
test('extracts date from YYYY-MM-DD prefix when present', async () => {
const file = join(corpusDir, '2026-05-01-dated.txt');
writeFileSync(file, 'content\n');
setMtime(file, Date.now());
const result = await listRecentTranscripts(fakeEngine, { days: 7 });
const row = result.find(r => r.path === '2026-05-01-dated.txt');
expect(row).toBeDefined();
expect(row!.date).toBe('2026-05-01');
});
test('returns null date when filename has no date prefix', async () => {
const file = join(corpusDir, 'random-name.txt');
writeFileSync(file, 'content\n');
setMtime(file, Date.now());
const result = await listRecentTranscripts(fakeEngine, { days: 7 });
const row = result.find(r => r.path === 'random-name.txt');
expect(row).toBeDefined();
expect(row!.date).toBeNull();
});
test('limit caps the number of returned transcripts', async () => {
// Add 5 files.
for (let i = 0; i < 5; i++) {
const file = join(corpusDir, `multi-${i}.txt`);
writeFileSync(file, `content ${i}\n`);
setMtime(file, Date.now());
}
const result = await listRecentTranscripts(fakeEngine, { days: 7, limit: 2 });
expect(result.length).toBe(2);
});
test('non-existent corpus dir is skipped silently', async () => {
configMap.set('dream.synthesize.session_corpus_dir', '/nope/does/not/exist');
const result = await listRecentTranscripts(fakeEngine);
expect(result).toEqual([]);
});
});
+111
View File
@@ -0,0 +1,111 @@
/**
* v0.29 tool-surface contracts.
*
* Verifies two filter contracts that touch the v0.29 ops but live outside
* the v0.29 source files (in serve-http.ts and brain-allowlist.ts):
*
* 1. `localOnly: true` on `get_recent_transcripts` is what hides it from
* the HTTP MCP tool-list. serve-http.ts:745 does
* `operations.filter(op => !op.localOnly)`. The v0.29 trust gate
* (in-handler `ctx.remote === true` reject) is defense-in-depth on top
* of this; if the filter ever drops the flag, the in-handler check is
* the last line. We assert both halves of the contract.
*
* 2. `buildBrainTools` (subagent registry) surfaces salience + anomalies
* as `brain_get_recent_salience` / `brain_find_anomalies` and EXCLUDES
* `brain_get_recent_transcripts`. The exclusion is intentional
* subagent calls always run with `ctx.remote === true`, and the v0.29
* trust gate would always reject. Listing it would be a footgun
* (subagent calls op, gets permission_denied, looks like a bug).
*
* Both filters are pure-function checks; no DB / engine / network needed.
*/
import { describe, expect, test } from 'bun:test';
import { operations } from '../src/core/operations.ts';
import { buildBrainTools } from '../src/core/minions/tools/brain-allowlist.ts';
import type { BrainEngine } from '../src/core/engine.ts';
import type { GBrainConfig } from '../src/core/config.ts';
describe('v0.29 — serve-http localOnly filter', () => {
// serve-http.ts:745 is the canonical filter expression.
const mcpVisible = operations.filter(op => !op.localOnly);
test('get_recent_transcripts is hidden from the HTTP MCP tool list', () => {
const names = mcpVisible.map(o => o.name);
expect(names).not.toContain('get_recent_transcripts');
});
test('get_recent_salience and find_anomalies stay visible', () => {
const names = mcpVisible.map(o => o.name);
expect(names).toContain('get_recent_salience');
expect(names).toContain('find_anomalies');
});
test('get_recent_transcripts carries the localOnly: true flag', () => {
const op = operations.find(o => o.name === 'get_recent_transcripts');
expect(op).toBeDefined();
expect(op!.localOnly).toBe(true);
});
test('get_recent_salience and find_anomalies do NOT carry localOnly', () => {
// Read-only ops that subagents need; localOnly would block them from MCP.
const sal = operations.find(o => o.name === 'get_recent_salience');
const ano = operations.find(o => o.name === 'find_anomalies');
expect(sal!.localOnly).toBeFalsy();
expect(ano!.localOnly).toBeFalsy();
});
test('all three v0.29 ops carry scope: read', () => {
// v0.26.0 contract: every op must annotate scope. Read-only is correct
// for all three (no DB writes, no fs writes).
for (const name of ['get_recent_salience', 'find_anomalies', 'get_recent_transcripts']) {
const op = operations.find(o => o.name === name);
expect(op!.scope).toBe('read');
}
});
});
describe('v0.29 — buildBrainTools subagent surfacing', () => {
// buildBrainTools doesn't issue any SQL at registry-build time — it only
// reads `engine.kind` for the put_page namespace-wrap branch. A minimal
// fake-engine literal keeps the test pure (no PGLite WASM cold-start, no
// connect/disconnect lifecycle, no test-isolation R3/R4 violations).
// Cast through `unknown` because the BrainEngine surface is large and
// we only touch one property.
const fakeEngine = { kind: 'pglite' } as unknown as BrainEngine;
const config: GBrainConfig = { engine: 'pglite' } as GBrainConfig;
test('subagent registry includes brain_get_recent_salience', () => {
const tools = buildBrainTools({ subagentId: 1, engine: fakeEngine, config });
const names = tools.map(t => t.name);
expect(names).toContain('brain_get_recent_salience');
});
test('subagent registry includes brain_find_anomalies', () => {
const tools = buildBrainTools({ subagentId: 1, engine: fakeEngine, config });
const names = tools.map(t => t.name);
expect(names).toContain('brain_find_anomalies');
});
test('subagent registry EXCLUDES brain_get_recent_transcripts (codex C3 footgun gate)', () => {
// All subagent calls run with ctx.remote === true; the v0.29 trust gate
// would always reject. Listing the op would be a footgun: subagent
// calls it, gets permission_denied, looks like a bug. The cycle's
// synthesize phase reaches transcripts via discoverTranscripts directly,
// not via the op.
const tools = buildBrainTools({ subagentId: 1, engine: fakeEngine, config });
const names = tools.map(t => t.name);
expect(names).not.toContain('brain_get_recent_transcripts');
});
test('the v0.29 ops carry their description verbatim into the registry', () => {
const tools = buildBrainTools({ subagentId: 1, engine: fakeEngine, config });
const sal = tools.find(t => t.name === 'brain_get_recent_salience');
const ano = tools.find(t => t.name === 'brain_find_anomalies');
const opSal = operations.find(o => o.name === 'get_recent_salience');
const opAno = operations.find(o => o.name === 'find_anomalies');
expect(sal!.description).toBe(opSal!.description);
expect(ano!.description).toBe(opAno!.description);
});
});