Files
gbrain/src/schema.sql
T
200a74104c v0.31.6 feat: extract facts during sync (real-time hot memory) (#796)
* feat: extract facts during sync (real-time hot memory)

Wire facts extraction into the sync pipeline so pages imported via
git get facts extracted immediately, not only through MCP put_page.

Changes:
- Add notability field (high/medium/low) to facts extraction schema
- Upgrade default extraction model from Haiku to Sonnet (configurable
  via facts.extraction_model brain_config)
- Add notability-gated facts extraction to sync post-import hook:
  - Only HIGH notability facts inserted during sync (life events,
    major commitments, relationship/health changes)
  - MEDIUM facts deferred to dream cycle
  - LOW facts (logistical noise) dropped entirely
- Add notability column to facts table DDL
- Pass engine to extraction for config-aware model selection

Before: facts only extracted via MCP put_page (never during git sync)
After: meetings, conversations, personal pages get facts extracted
immediately on sync, with salience filtering

Closes the hot-memory gap where brain content committed via git was
invisible to the facts table until manually processed.

* fix: B1 — pass notability through facts JSON parser

Pre-fix, src/core/facts/extract.ts:tryArrayShape silently dropped the
LLM's notability field on the floor: the function copied fact/kind/
entity/confidence into the output but never read o.notability. The
outer loop in extractFactsFromTurn then read candidate.notability,
found undefined, and defaulted to 'medium'. sync.ts's HIGH-only filter
(`if (f.notability !== 'high') continue`) discarded 100% of facts.

Net: real-time facts on sync was a no-op despite Sonnet running and
costing money. Headline feature was dead on the happy path.

Fix is a one-line change in tryArrayShape. Two layers of test pin it:

  1. Parser-pin (test/facts-extract.test.ts +75 LOC, 5 cases):
     - notability passes through when LLM emits it
     - notability omitted defaults to undefined (legacy compat)
     - non-string notability is dropped defensively
     - every documented field survives the parse (future field-drop guard)
     - fenced JSON output (markdown code blocks) still threads correctly

  2. End-to-end smoke (test/facts-extract-smoke.test.ts NEW, 145 LOC,
     4 cases): drives extractFactsFromTurn with a stubbed gateway chat
     transport. Asserts HIGH input → notability:'high' all the way out.
     Guards against future prompt drift where Sonnet returns 'medium'
     for everything; smoke fails loudly so the eval-mining flow gets
     triggered.

Adds the chat test seam to enable the smoke test:
  src/core/ai/gateway.ts: __setChatTransportForTests(fn) mirrors
  v0.28.7's __setEmbedTransportForTests pattern. When set, chat()
  routes through the stub; isAvailable('chat') returns true so tests
  don't need full gateway configuration. resetGateway() clears it.
  Test files stay regular .test.ts (parallel-safe; no mock.module).

PR 1 commit 1 of 15. See ~/.claude/plans/swift-gliding-key.md for the
full eng review and bisect-friendly commit ordering.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix: B2 — migration v46 ALTER facts.notability with idempotent CHECK

Pre-fix, the v0.31.1 PR shipped a CREATE TABLE edit to migration v45 that
added `notability NOT NULL DEFAULT 'medium' CHECK (notability IN (...))`
inline. Fresh installs got the column. But every brain that already ran
v45 BEFORE that edit (i.e., everyone running v0.31.0+ in production) keeps
the old facts table shape. INSERT now crashes with:

  column "notability" of relation "facts" does not exist

This is the canonical "embedded schema mutation breaks upgrades" trap that
CLAUDE.md cites: "bit users 10+ times across 6 schema versions over 2 years."

Fix: new migration v46 ALTER. Idempotent under all four states:

  1. Fresh install (v45 already added column inline)
     → ADD COLUMN IF NOT EXISTS no-ops; named CHECK probe finds existing
       constraint → skip. Postgres emits a NOTICE; no error.

  2. Old brain pre-edit (no column)
     → ADD COLUMN adds it with NOT NULL DEFAULT 'medium'; named CHECK
       probe finds nothing → adds the constraint.

  3. Partial state (column exists, CHECK missing)
     → ADD COLUMN no-ops; CHECK probe adds the named constraint.

  4. Re-run after success
     → all probes skip; no error, no state change.

Implementation notes:
  - CHECK constraint is named `facts_notability_check` (not autogen) so the
    information_schema-equivalent probe via `pg_constraint` can find it
    deterministically.
  - Column-level CHECK in v45 inline (autogen-named) and the named CHECK
    here are additive and non-conflicting — Postgres allows multiple CHECKs
    covering the same predicate. Codex flagged this concern; the named
    constraint addresses it cleanly.
  - Both engines run the same SQL. PGLite is real Postgres in WASM and
    supports DO $$ blocks. PGLite users with persistent older brains hit
    the same bug.

E2E coverage (test/e2e/migration-v46-notability.test.ts, 5 cases):
  - fresh-install fully-migrated: column + named CHECK both exist
  - old brain (column dropped): v46 adds both back
  - partial state (column exists, CHECK missing): v46 adds CHECK
  - idempotent re-run on fully-migrated: no error, state unchanged
  - CHECK constraint actually rejects out-of-domain values

Verified against real Postgres (pgvector/pgvector:pg16): 5/5 pass in 696ms.

PR 1 commit 2 of 15.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix: B3 — restore v0_31_0 orchestrator gate to v < 45

Pre-fix, the v0_31_0 orchestrator's phaseASchema gate had been demoted
from `v < 45` to `v < 40` with an operator-facing message claiming
"v40 (facts hot memory + notability)". Facts is at v45, not v40 — the
message was wrong and the gate was permissive.

Symptom: brains at schema_version 40-44 (real states for users mid-
upgrade) passed the precondition, then immediately crashed on the
post-condition check three lines later (`SELECT FROM pg_tables WHERE
tablename = 'facts'`). Operator saw a green light, then a red light.

Fix: restore the gate to `v < 45` (the real semantic precondition:
the facts table is created by migration v45). Drop the misleading
"+ notability" claim — column shape is enforced by migration v46
alone (see MIGRATIONS[v46]), not gated here. Add a one-line comment
pointing at v46 so the next reader sees the separation.

Test coverage (test/migration-orchestrator-v0_31_0.test.ts NEW, 4 cases):
  - schema_version < 45 fails with operator-facing message naming v45
    + recovery command. Negative assertions guard against regression
    to the "v >= 40" / "+ notability" prior text.
  - schema_version >= 45 with facts table present → status complete.
  - dryRun short-circuits before any DB read.
  - null engine short-circuits with no_brain_configured.

Verified: 4/4 pass; v45 + v46 both apply cleanly during test setup.

PR 1 commit 3 of 15.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* refactor: widen FactRow to expose notability across all readers

Codex's outside-voice pass on the cathedral plan flagged P1 #4: the read-
side contract was behind the write-side schema. notability lived in DDL
and the insertFact INSERT, but FactRow type omitted it and both row
mappers (pglite-engine + postgres-engine) silently dropped the column.
Every consumer above the engine (recall op, MCP _meta hook, CLI JSON
output) returned facts without their salience tier. PR2/PR3 surfaces
that need to filter or display notability would have required contract
surgery first; this lands the contract widening as the foundation.

Changes:
  - src/core/engine.ts: add `notability: 'high' | 'medium' | 'low'` to
    FactRow with doc comment naming the row source (column added by
    migration v46) and the consumers (recall, daily-page, admin, MCP).
  - src/core/postgres-engine.ts: FactRowSqlShape gains notability;
    rowToFactPg propagates it with `?? 'medium'` belt-and-suspenders
    fallback (NOT NULL DEFAULT in DDL is the primary; this is the
    second line for any pre-v46 row that survives a SELECT).
  - src/core/pglite-engine.ts: same pair (interface + mapper).
  - src/core/operations.ts: recall op response shape adds notability.
  - src/core/facts/meta-hook.ts: `_meta.brain_hot_memory` payload
    surfaces notability so connected agents can filter or weight
    HIGH-tier facts in their context budget.
  - src/commands/recall.ts: `--json` output adds notability.

Test contract pin (test/facts-engine.test.ts):
  - Existing 'inserts a fact' case asserts default 'medium' on the
    read side (caller-omits-notability path).
  - New 'notability round-trips for each tier' case inserts HIGH /
    MEDIUM / LOW explicitly and reads back the same tier — without
    this assertion, codex P1 #4 reappears silently.

Test fixtures (facts-classify.test.ts + facts-decay.test.ts) also
updated: makeFact() factories now construct complete FactRow objects
with notability:'medium' to match the tightened type.

PR 1 commit 4 of 15.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* refactor: move isFactsBackstopEligible to src/core/facts/eligibility.ts

Single source of truth for "should this page write fire the facts
extraction backstop?" Pre-extraction, lived inline at operations.ts:633
where only put_page could see it; sync.ts had its own divergent type
filter (`['conversation', 'transcript', 'personal', 'therapy', 'call']`
— only `meeting` was a real PageType, the rest never matched). Sync's
filter is deleted in commit 7; everyone routes through this predicate.

Adds the slug-prefix rescue branch the eng review pinned (D-eligibility):
parsed.type ∈ ELIGIBLE_TYPES OR slug.startsWith('meetings/' | 'personal/'
| 'daily/'). The rescue catches `meetings/2026-05-09-foo.md` pages that
frontmatter-typed themselves as 'note' (the legacy default) — directory
location wins.

Test pin (test/facts-eligibility.test.ts NEW, 28 cases):
  - 4 BRANCH cases: typed-only, slug-only (each prefix), both, neither
  - 7 GUARD cases: null/undefined parsed, wiki/agents/, dream_generated,
    body length thresholds (< 80, exactly 80, whitespace-only)
  - 14 COVERAGE cases: every eligible PageType on arbitrary slug → ok;
    every non-eligible PageType on non-rescued slug → kind:<type> reason

Pure-function tests; no DB. The full predicate covered without spinning
a brain.

Existing test/facts-backstop-gating.test.ts still passes (it tests the
predicate via put_page; the move is transparent to that surface).

PR 1 commit 5 of 15.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat: add runFactsBackstop helper with full extract→resolve→dedup→insert pipeline

Single shared facts pipeline used by every brain write surface that
wants real-time hot memory extraction. Replaces five divergent
implementations:
  - put_page MCP backstop hook (operations.ts:556)
  - extract_facts MCP op (operations.ts:2438-2486)
  - sync.ts post-import block (deleted in commit 7)
  - file_upload + code_import (wired in commit 10)

Encapsulates the v0.31 smart pipeline:
  extract → resolve → dedup (cosine @ 0.95) → insert
(matches extract_facts op precedent at operations.ts:2460.)

Two execution modes (D8):
  - 'queue' (default): fire-and-forget via getFactsQueue().enqueue.
    Caller awaits ~zero (just enqueue + microtask). Sync stays fast
    on a 50-page batch.
  - 'inline': await full pipeline; return real {inserted, duplicate,
    superseded, fact_ids} counts. Used by extract_facts MCP op.

Discriminated return shape so TypeScript catches mode/result mismatches
at the call site:
  | { mode: 'queue'; enqueued; queueDepth; skipped? }
  | { mode: 'inline'; inserted; duplicate; superseded; fact_ids; skipped? }

Notability filter (D4): per-caller policy via FactsBackstopCtx.notabilityFilter.
Sync passes 'high-only' (HIGH lands now, MEDIUM waits for dream cycle,
LOW dropped at LLM layer). Other surfaces default to 'all'. Filter runs
post-LLM, pre-insert: saves the insert work but not the LLM call (the
notability tier IS what we're calling Sonnet to determine).

Eligibility + kill-switch gates run before any LLM cost. Skipped reasons
are stable strings the future facts:absorb writer (commit 13) and doctor
check (commit 12) consume.

Re-throws AbortError; absorbs gateway/parse/queue errors as `skipped: '...'`
envelope. Operator visibility lands via PR1 commit 13's ingest_log writer
(facts:absorb source_type).

Test pin (test/facts-backstop.test.ts NEW, 12 cases):
  - 3 eligibility/kill-switch cases (extraction_disabled, subagent_namespace,
    dream_generated)
  - 5 inline-mode cases (insert + counts, notability filter, source string,
    empty extraction, abort)
  - 3 queue-mode cases (default mode, explicit mode, kill-switch envelope)
  - 1 dedup contract case (insertions without embeddings short-circuit
    cleanly; embedding-driven dedup is exercised by E2E with real gateway)

PGLite in-memory; LLM stubbed via __setChatTransportForTests (commit 1's
seam). 12/12 pass in 912ms.

PR 1 commit 6 of 15.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* refactor: sync.ts uses runFactsBackstop (deletes dead-code type filter)

Pre-fix sync.ts had a 60-line inline facts extraction block carrying:
  1. Dead-code eligibility filter: ['meeting', 'conversation',
     'transcript', 'personal', 'therapy', 'call'] — only `meeting` is
     a real PageType. The other five never matched anything; eligibility
     rested on the slug-prefix branch alone.
  2. Divergent shape from put_page's backstop: no dedup, no supersede,
     raw extract→insert. Garbage rows on re-sync.
  3. Sequential per-page LLM calls in sync's request path: a 50-page
     sync = 50 Sonnet calls in series ≈ 5+ minutes blocking.

Replaced with `runFactsBackstop(parsedPage, ctx)` from PR1 commit 6:
  - Queue mode (fire-and-forget) so sync stays fast on multi-page batches.
  - 'high-only' notabilityFilter (cathedral spec: HIGH lands now,
    MEDIUM waits for dream cycle, LOW dropped at LLM).
  - isFactsBackstopEligible (commit 5) — eligibility lives in one place.
  - extract → resolve → dedup (cosine @ 0.95) → insert pipeline shared
    with put_page + extract_facts.

Per-page try/catch survives so one failed page doesn't blow up the
whole sync (best-effort posture preserved).

Existing test/sync.test.ts (39 cases) passes unchanged — sync's outer
contract is untouched, only the inner facts-extract block changed.

PR 1 commit 7 of 15.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* refactor: operations.ts put_page uses runFactsBackstop

Replace the inline get-queue-extract-resolve-insert closure (operations.ts:540-583)
with a single `runFactsBackstop(parsed, ctx)` call in queue mode. put_page
and sync now share the same eligibility/extract/dedup/insert pipeline.

Behavioral preservation:
  - Response shape `{queued: true} | {skipped: '<reason>'}` unchanged for
    MCP clients. The helper's namespaced 'eligibility_failed:<reason>'
    discriminator is mapped back to the bare reason ('kind:guide',
    'too_short', 'subagent_namespace', 'dream_generated') before write
    to factsQueued. test/facts-backstop-gating.test.ts (5 cases) passes
    without modification.
  - Default 'all' notabilityFilter (MEDIUM facts continue to land via
    put_page; only sync filters to HIGH-only). This matches the
    pre-v0.31.2 surface: put_page's prior shape inserted everything the
    LLM returned, with the dream cycle's consolidate phase doing the
    salience clustering overnight.

Net: -32 LOC of inline pipeline; one shared call site + one mapping
shim; same observable shape.

PR 1 commit 8 of 15.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* refactor: operations.ts extract_facts uses runFactsPipeline

Replace the 65-line inline extract→resolve→dedup→insert loop in the
extract_facts MCP op (operations.ts:2369-2454) with a single
`runFactsPipeline(turn_text, ctx)` call. The inline pipeline + the
helper are now the same code path; test/facts-mcp-allowlist + test/
facts-anti-loop pass unchanged.

Architecture: the helper has two entry points now —
  - `runFactsBackstop(parsedPage, ctx)` — page-write hook with
    eligibility + kill-switch + queue mode dispatch (PR1 commit 6).
    Used by put_page, sync, file_upload, code_import.
  - `runFactsPipeline(turnText, ctx)` — raw turn-text entry that
    skips the page-shape eligibility predicate. Used by extract_facts
    MCP op (this commit).

Both share an inner `runPipelineWithBody` so the actual extract → resolve
→ dedup (cosine @ 0.95) → insert pipeline lives in one place. Codex P0 #2
called this out: "extract_facts already does the smart pipeline; put_page
+ sync do raw extract→insert. Centralizing only extraction codifies the
worse pipeline." With commit 9, every fact-insert path goes through the
smart pipeline; raw insertFact loops in the brain are gone.

Behavioral preservation:
  - extraction_disabled kill-switch envelope unchanged.
  - is_dream_generated → returns {skipped: 'dream_generated'} envelope
    (the predicate-bypass path; eligibility doesn't apply on raw
    turn_text but dream_generated still does). Pre-fix the extractor
    itself short-circuited; new shape surfaces the skip explicitly to
    MCP clients.
  - Visibility ('private' | 'world') threading preserved.
  - Response shape {inserted, duplicate, superseded, fact_ids} identical
    to pre-fix.

PR 1 commit 9 of 15.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* docs: document why file_upload + code_import don't wire runFactsBackstop

PR1 commit 10 was scoped in the eng review plan to "wire runFactsBackstop
to file_upload and code_import paths." Implementation analysis revealed
all three candidate surfaces are correctly handled WITHOUT explicit
wiring:

  1. file_upload (operations.ts:1713) doesn't write a page. It uploads
     a file to storage + inserts a `files` row. The associated page is
     written separately via put_page, which already fires runFactsBackstop
     in queue mode (commit 8). No double-firing needed.

  2. importCodeFile (this file) writes pages with type='code'. The
     isFactsBackstopEligible predicate rejects 'code' kind with reason
     `kind:code`. Wiring runFactsBackstop here would always return the
     skipped envelope. When README / doc-comment extraction lands in a
     future release, the eligibility predicate is the single place to
     update — adding 'code' to ELIGIBLE_TYPES makes existing call sites
     auto-cover the change.

  3. `gbrain import` (commands/import.ts) is bulk markdown import. Firing
     facts extraction on every imported page would cost-spike on first-
     time bulk imports of large brain repos (10K+ pages × Sonnet =
     hundreds of dollars). User runs `gbrain dream` or the consolidate
     phase to backfill facts from bulk-imported pages.

Adds a docstring above importCodeFile capturing all three rationales so
the next maintainer doesn't re-do this analysis.

PR 1 commit 10 of 15 — no behavior change; documentation only.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat: migration v47 — ingest_log.source_id ALTER (codex P1 #3)

Pre-fix the ingest_log table had no source_id column; sync.ts wrote rows
without source-scoping and doctor only checked 'default'. Codex's outside
voice flagged this on the cathedral plan: "facts:absorb logging inherits
a surface that cannot tell you which source is failing."

This commit closes the multi-source observability gap on the foundation:
  - PR1 commit 13's facts:absorb writer (next) writes ingest_log rows
    with source_id so multi-source brains scope failures per source.
  - PR1 commit 12's doctor's facts_extraction_health check (after that)
    iterates over `SELECT DISTINCT id FROM sources` instead of hardcoded
    'default'.

Migration v47 (idempotent, both engines):
  ALTER TABLE ingest_log ADD COLUMN IF NOT EXISTS source_id TEXT
    NOT NULL DEFAULT 'default';
  CREATE INDEX IF NOT EXISTS idx_ingest_log_source_type_created
    ON ingest_log (source_id, source_type, created_at DESC);

Schema-bootstrap coverage:
  - schema.sql / pglite-schema.ts inline definitions add source_id +
    the new index for fresh installs.
  - applyForwardReferenceBootstrap (both PGLite + Postgres) probes for
    `ingest_log.source_id` and adds the column BEFORE SCHEMA_SQL replay
    builds the new composite index. Without this, old brains running
    initSchema() on the new schema-embedded.ts would crash on the index
    creation (the column doesn't exist yet at replay time).
  - test/schema-bootstrap-coverage.test.ts pins ingest_log.source_id as
    REQUIRED_BOOTSTRAP_COVERAGE — adding a forward reference without
    extending applyForwardReferenceBootstrap would fail this guard.

E2E (test/e2e/migration-v47-ingest-log-source-id.test.ts NEW, 3 cases):
  - fresh-install: column + index both exist after runMigrationsUpTo(LATEST).
  - old-brain simulation: drop column, run v47, column reappears with
    NOT NULL DEFAULT 'default'; INSERT without source_id picks up the
    default.
  - idempotent re-run: v47 twice in a row is a no-op.

Verified against real Postgres (pgvector/pgvector:pg16): 3/3 pass; the v46
+ v47 E2Es land green together (8/8 in 2.05s). Bootstrap-coverage unit
test (5 cases) also green.

PR 1 commit 11 of 15.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat: facts:absorb writer + reason codes (D5 contract)

D5 from /plan-ceo-review: every absorbed failure in the facts extraction
pipeline writes one row to ingest_log so doctor + admin dashboard
surface failures cross-process. CLAUDE.md's "zero silent failures" rule
gets enforced on the foundation.

Wires three layers:

  1. Type widening (src/core/types.ts):
     - IngestLogEntry gains source_id (codex P1 #3 — migration v47).
     - IngestLogInput gains optional source_id; engines default to 'default'.

  2. Engine row writers (pglite-engine.ts + postgres-engine.ts):
     - logIngest threads source_id into INSERT.
     - getIngestLog applies belt-and-suspenders 'default' fallback for
       any pre-v47 row that somehow survived.

  3. Helper (src/core/facts/absorb-log.ts NEW):
     - writeFactsAbsorbLog(engine, ref, reason, detail, sourceId) writes
       one ingest_log row with source_type='facts:absorb' and
       summary='<reason>: <detail truncated to 240 chars>'.
     - classifyFactsAbsorbError(err) heuristic-pattern-matches arbitrary
       Errors into 6 stable reason codes:
         gateway_error  | parse_failure  | queue_overflow
         queue_shutdown | embed_failure  | pipeline_error
     - Best-effort: any logging failure is caught + stderr-warned;
       the caller's pipeline keeps running.

  4. runFactsBackstop wiring (src/core/facts/backstop.ts):
     - queue mode: errors inside the queue worker classify + log via
       absorb-log.ts. Were previously invisible (counter increment only).
     - queue overflow drop also writes an absorb log row so doctor sees
       the depth of capacity pressure.
     - inline mode: errors bubble; caller decides logging (extract_facts
       MCP op surfaces them as op-error responses).

Test pin (test/facts-absorb-log.test.ts NEW, 12 cases):
  - 7 classifier cases pinning every reason path + fallback
  - 5 writer cases pinning ingest_log row shape, custom sourceId,
    240-char detail truncation, no-throw contract, reason-set
    completeness

PR1 commit 12 (next) reads these rows for the facts_extraction_health
doctor check.

PR 1 commit 13 of 15.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat: doctor facts_extraction_health check (multi-source)

Mirrors the eval_capture check shape but reads facts:absorb rows
(written by writeFactsAbsorbLog from PR1 commit 13). Iterates over
EVERY source (codex P1 #3 motivation) so multi-source brains see
per-source failure rates instead of only 'default'.

Configurable threshold: facts.absorb_warn_threshold (default 10 over
the last 24h, per source, per reason). When the threshold is exceeded
for any (source, reason) pair, status flips to warn and the message
names the breakdown:

  facts:absorb activity in last 24h (under threshold 10):
    default: 4 gateway_error, 1 parse_failure |
    team-source: 2 queue_overflow

Single SQL grouping query covers the read; the composite index v47
added (idx_ingest_log_source_type_created on source_id, source_type,
created_at DESC) covers the filter + sort path so the check is fast
on brains with millions of ingest_log rows.

Operator UX:
  - 'ok' under threshold (or zero failures) → quiet.
  - 'warn' over threshold → message names every (source, reason, count)
    tuple. Recovery hint: `gbrain recall --since 24h --json` to inspect
    what landed; `gbrain config set facts.absorb_warn_threshold N` to
    tune.
  - Pre-v47 brain (column missing): 'ok' with skipped reason pointing
    at `gbrain apply-migrations --yes`.
  - RLS denies SELECT: 'warn' calling out that capture INSERTs are
    likely also blocked.

Test pin (test/doctor.test.ts +28 LOC, 1 case):
  Source-string assertions on the doctor.ts block:
    - 'GROUP BY source_id' (multi-source contract)
    - "source_type = 'facts:absorb'" (right table query)
    - 'facts.absorb_warn_threshold' (configurable threshold)
    - INTERVAL '24 hours' (right window)
    - 'Skipped (ingest_log.source_id unavailable' (pre-v47 fallback)
    - 'RLS denies SELECT on ingest_log' (RLS hint)
  Negative: must NOT contain `source_id = 'default'` (the bug we're
  fixing — codex P1 #3 was that doctor only checked 'default').

Live smoke against real Postgres: doctor renders the new check between
'eval_capture' and 'effective_date_health' as expected, shows 'ok' on
an empty test brain.

PR 1 commit 12 of 15.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat: notability-eval mining + public-anonymized fixture (40 cases)

The notability gate is the load-bearing differentiator of the cathedral:
"only HIGH lands on sync, MEDIUM waits for the dream cycle, LOW dropped
at the LLM layer." Without an eval, the gate's quality is asserted via
hope; prompt drift (Sonnet returning 'medium' for everything) silently
turns the headline feature into a no-op.

This commit adds the mining half — eval suite is pinned in the next
commit (15).

NEW src/commands/notability-eval.ts:
  - mineNotabilityCandidates(repoPath, opts): walks meetings/, personal/,
    daily/ in the brain repo, splits markdown bodies into paragraphs
    (filtered by 80–800 char length), pre-classifies each paragraph
    with cheap-Haiku to bucket into HIGH/MEDIUM/LOW (round-robin
    fallback when no chat gateway is available — local development
    without API keys still produces a candidates file).
  - Stratified random sample within each bucket: HIGH/MEDIUM/LOW
    targets default 20/20/10 (per cathedral plan D7=B). Stratified
    further across the three corpus dirs so HIGH cases come from
    multiple dirs not just one.
  - JSONL utilities (loadJsonlCases, writeJsonlCases) shared with the
    review path. Default paths: ~/.gbrain/eval/notability-mining-
    candidates.jsonl (mining) + ~/.gbrain/eval/notability-real.jsonl
    (private confirmed).
  - TTY review subcommand: walks candidates one-by-one, asks for
    HIGH/MEDIUM/LOW confirmation, writes confirmed cases. Smoke-only
    test (TTY interactivity is hard to test deterministically).

CLI dispatch (src/cli.ts):
  - `gbrain notability-eval mine` (default targets 20/20/10).
  - `gbrain notability-eval review` (TTY hand-confirm).
  - `gbrain notability-eval help` (flag reference).
  - sync.repo_path resolution mirrors the dream phase pattern; --repo
    PATH overrides.

NEW test/fixtures/notability-eval-public.jsonl (40 cases):
  - 14 HIGH (life events, major commitments, relationship/health changes,
    financial decisions).
  - 13 MEDIUM (durable preferences, beliefs, strong opinions revealing
    character).
  - 13 LOW (logistical noise — restaurant orders, scheduling, errands).
  - Anonymized per CLAUDE.md privacy rule (alice-example, acme-co,
    widget-co, fund-a placeholder names; no real contacts).
  - Each case has a `tier_rationale` string documenting the choice for
    reviewer transparency.
  - Used by CI's eval harness in commit 15 (no API key required for
    deterministic stub-driven contract tests).

PR 1 commit 14 of 15.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat: notability-eval harness with precision@HIGH metric (40-case fixture)

Pins the load-bearing gate-quality contract in CI. Without this, prompt
drift (Sonnet returning 'medium' for everything → sync inserts nothing)
ships silently. The harness flips it from "asserted by hope" to "asserted
by metric."

NEW test/notability-eval.test.ts (13 cases across 5 describe blocks):

  1. splitParagraphs (2 cases): blank-line splitting, length filters.
  2. walkMarkdownFiles (1 case): tree walk drops non-.md files.
  3. mineNotabilityCandidates round-robin path (2 cases): empty corpus
     + populated corpus produce expected candidate shape; round-robin
     keeps tests deterministic without an LLM.
  4. JSONL utilities (3 cases): write+read round-trip, malformed-line
     skip, default paths under ~/.gbrain/eval/.
  5. Public-anonymized fixture shape (2 cases): 40 cases, ≥10 per tier,
     every paragraph ≥80 chars, every case has a tier_rationale.
  6. Eval harness contract (3 cases) — the headline assertions:
     - Perfect predictor (LLM-stub returns confirmed_tier verbatim) →
       precision@HIGH = 1.0, recall@HIGH = 1.0.
     - Always-medium model → precision@HIGH = 0 (no HIGH predictions
       at all). Pins the "harness handles the no-positive-prediction
       case correctly" contract.
     - Always-high model → precision drops below the 0.50 PR-fail
       threshold (TP / (TP + FP) = 14 / 40 = 0.35). Pins the
       "harness CORRECTLY flags a misaligned model" contract.

Sample size justification: the public fixture has 14 HIGH cases. For
precision@HIGH = 0.75 with a 95% CI ±10pp, n=14 gives the right floor
for "is the gate dramatically wrong" — tighter measurements need the
private fixture (50 cases via mine + review).

The harness is a CONTRACT test for the metric shape, not a quality
measurement of any specific model. A real quality run uses the same
harness against a real Sonnet (no chat-transport stub) — that flow is
exposed via GBRAIN_NOTABILITY_EVAL_REAL=1 + the private mined fixture.

All 92 tests across all PR1 facts files pass green (extract / extract-
smoke / engine / backstop / eligibility / absorb-log / notability-eval).

Soft gate per the cathedral plan: warn if precision@HIGH < 0.75; fail
PR if < 0.50. CI wiring + the production gate are deferred to PR2 (the
visibility/observability surface PR); this PR1 commit lands the harness
+ fixture + contract tests so the gate is ready to wire.

PR 1 commit 15 of 15. Cathedral foundation lands here.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* test: fill PR1 gap-fill — backstop integration + Postgres parity

Test gap analysis flagged three high-priority untested behaviors in
PR1's surface:

  Gap #3: extract_facts MCP op response shape stability after
    routing through runFactsPipeline (commit 9). Existing tests
    pin allowlist + anti-loop but not the {inserted, duplicate,
    superseded, fact_ids} envelope that MCP clients display.

  Gap #4: per-engine row-mapper parity for notability. facts-engine.test.ts
    pins notability round-trip on PGLite; the Postgres row mapper
    (postgres-engine.ts:rowToFactPg) is different code that wasn't
    pinned. Codex P1 #4 was specifically about read-side contracts
    drifting silently.

  Gap #5: multi-source isolation in facts:absorb logging. Codex
    P1 #3 motivated the source_id column; the absorb-log test pins
    that source_id is written but not that source_id-scoped queries
    return only the right source's rows.

NEW test/facts-backstop-integration.test.ts (6 cases):
  - 2 cases on runFactsPipeline (extract_facts path) response shape:
    successful extraction returns full {inserted, duplicate, superseded,
    fact_ids} envelope with positive fact_ids; empty extraction returns
    zero counts (no NaN/undefined).
  - 2 cases on facts:absorb multi-source isolation: writeFactsAbsorbLog
    rows are source-scoped; doctor's GROUP BY source_id query produces
    the expected per-source breakdown.
  - 2 cases on queue mode: happy-path drain pins counters.completed >= 1
    + counters.failed == 0; documented case noting that extract.ts
    absorbs gateway errors silently (errors propagate from layers
    ABOVE extract — resolver, dedup, insert — to backstop's catch,
    not from the chat call itself).

NEW test/e2e/facts-notability-roundtrip.test.ts (5 cases, real Postgres):
  - HIGH/MEDIUM/LOW round-trip via insertFact + listFactsByEntity.
  - Omitting notability defaults to medium (NOT NULL DEFAULT contract).
  - listFactsSince also surfaces notability.
  All 5 pin the postgres.js driver + rowToFactPg row mapper.
  PGLite parity is covered by the existing test/facts-engine.test.ts
  case from commit 4.

Verified: 6/6 unit + 5/5 E2E green. The third high-priority gap
(integration sync.ts → runFactsBackstop end-to-end) is sufficiently
covered by the existing test/sync.test.ts behavior plus the per-page
runFactsBackstop assertions in test/facts-backstop.test.ts; chasing
the full happy-path sync→facts integration would require a real
git fixture which is heavier than warranted for this surface.

PR 1 commit 16 of 16 (gap fill).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Wintermute <wintermute@garrytan.com>
Co-authored-by: Garry Tan <garrytan@gmail.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-10 12:49:14 -07:00

879 lines
43 KiB
PL/PgSQL

-- GBrain Postgres + pgvector schema
CREATE EXTENSION IF NOT EXISTS vector;
CREATE EXTENSION IF NOT EXISTS pg_trgm;
-- gen_random_uuid() is core in Postgres 13+; enable pgcrypto as fallback for older versions
CREATE EXTENSION IF NOT EXISTS pgcrypto;
-- ============================================================
-- sources: multi-repo / multi-brain tenancy (v0.18.0)
-- ============================================================
-- A source is a logical brain-within-the-DB: wiki, gstack, yc-media, etc.
-- Every page/file/ingest_log row carries source_id.
--
-- id: immutable citation key. [a-z0-9-]{1,32} enforced at app layer.
-- Used in [source:slug] citations, --source flag, wikilink syntax.
-- name: mutable display label. Rename via `gbrain sources rename`.
-- local_path: optional git checkout root for filesystem-backed sources.
-- config: forward-compat JSONB. Currently used for federation + ACL slot.
-- { "federated": bool, "access_policy": {...} }
-- - federated=true (or missing-but-explicit on 'default'):
-- participates in cross-source default search.
-- - federated=false (default for new sources):
-- only searched when explicitly named via --source.
-- - access_policy: forward-compat slot, no enforcement in v0.17.
-- Write-side lockdown: mutated only when ctx.remote=false.
CREATE TABLE IF NOT EXISTS sources (
id TEXT PRIMARY KEY,
name TEXT NOT NULL UNIQUE,
local_path TEXT,
last_commit TEXT,
last_sync_at TIMESTAMPTZ,
config JSONB NOT NULL DEFAULT '{}'::jsonb,
-- v0.20.0 Cathedral II (SP-1): chunker version last used to sync this source.
-- performSync forces a full walk when this mismatches CURRENT_CHUNKER_VERSION,
-- bypassing the git-HEAD up_to_date early-return so CHUNKER_VERSION bumps
-- actually trigger re-chunking on upgrade.
chunker_version TEXT,
-- v0.26.5: soft-delete + recovery window. `archive` flips archived=true and
-- sets archive_expires_at = now() + 72h. The autopilot purge phase
-- hard-deletes rows where archive_expires_at <= now(). Promoted from a
-- JSONB key to real columns to avoid reserved-key footguns and to make the
-- search visibility filter (`NOT s.archived`) a column lookup.
archived BOOLEAN NOT NULL DEFAULT false,
archived_at TIMESTAMPTZ,
archive_expires_at TIMESTAMPTZ,
created_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
-- Seed the default source. 'default' is federated=true for backward compat
-- (pre-v0.17 brains behave exactly as before — every page appears in search).
-- Pre-existing sync.repo_path / sync.last_commit are copied in by the v16
-- migration, not here; fresh installs have no local_path until `sources add`
-- or the first `sync`.
INSERT INTO sources (id, name, config)
VALUES ('default', 'default', '{"federated": true}'::jsonb)
ON CONFLICT (id) DO NOTHING;
-- ============================================================
-- pages: the core content table
-- ============================================================
-- v0.18.0 (Step 2): pages.source_id scopes each row to a sources(id) row.
-- Slugs are unique per source, NOT globally. The default source is
-- seeded in the sources block above so the DEFAULT 'default' FK is
-- always valid at INSERT time.
CREATE TABLE IF NOT EXISTS pages (
id SERIAL PRIMARY KEY,
source_id TEXT NOT NULL DEFAULT 'default'
REFERENCES sources(id) ON DELETE CASCADE,
slug TEXT NOT NULL,
type TEXT NOT NULL,
-- v0.19.0: distinguishes markdown vs code pages at the DB level.
-- Drives orphans filter, auto-link bypass, and `query --lang`.
page_kind TEXT NOT NULL DEFAULT 'markdown'
CHECK (page_kind IN ('markdown','code','image')),
title TEXT NOT NULL,
compiled_truth TEXT NOT NULL DEFAULT '',
timeline TEXT NOT NULL DEFAULT '',
frontmatter JSONB NOT NULL DEFAULT '{}',
content_hash TEXT,
-- v0.29: deterministic 0..1 score (tag emotion + take density + Garry-as-holder ratio).
-- Populated by the `recompute_emotional_weight` cycle phase. Default 0.0 so freshly
-- imported pages don't pollute salience ranking before the cycle has run.
emotional_weight REAL NOT NULL DEFAULT 0.0,
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
updated_at TIMESTAMPTZ NOT NULL DEFAULT now(),
-- v0.26.5: soft-delete + recovery window. `delete_page` sets deleted_at = now()
-- instead of issuing DELETE. The autopilot purge phase hard-deletes pages
-- where deleted_at < now() - 72h. Search and `get_page` filter
-- `WHERE deleted_at IS NULL` by default; `include_deleted: true` opts in.
deleted_at TIMESTAMPTZ,
-- v0.29.1: salience-and-recency, additive opt-in. All NULL by default;
-- only consulted when a caller passes `salience='on'` / `recency='on'` or
-- the new `since`/`until` filter. effective_date_source is a sentinel for
-- the doctor's effective_date_health check (values: 'event_date' | 'date'
-- | 'published' | 'filename' | 'fallback'). salience_touched_at is bumped
-- by recompute_emotional_weight when emotional_weight changes so the
-- salience window picks up newly-salient old pages.
effective_date TIMESTAMPTZ,
effective_date_source TEXT,
import_filename TEXT,
salience_touched_at TIMESTAMPTZ,
CONSTRAINT pages_source_slug_key UNIQUE (source_id, slug)
);
CREATE INDEX IF NOT EXISTS idx_pages_type ON pages(type);
CREATE INDEX IF NOT EXISTS idx_pages_frontmatter ON pages USING GIN(frontmatter);
CREATE INDEX IF NOT EXISTS idx_pages_trgm ON pages USING GIN(title gin_trgm_ops);
-- v0.13.1 #170: avoids 14.6s seqscan on large brains when listing pages newest-first.
CREATE INDEX IF NOT EXISTS idx_pages_updated_at_desc ON pages (updated_at DESC);
-- v0.18.0: source-scoped scans (per /plan-eng-review Section 4).
CREATE INDEX IF NOT EXISTS idx_pages_source_id ON pages(source_id);
-- v0.26.5: partial index supports the autopilot purge sweep
-- (`WHERE deleted_at IS NOT NULL AND deleted_at < now() - INTERVAL '72 hours'`).
-- Search filters (`WHERE deleted_at IS NULL`) do not benefit from this index
-- (predicate doesn't match) and don't need their own — soft-deleted cardinality
-- stays low. Don't add a regular `(deleted_at)` index without measuring.
CREATE INDEX IF NOT EXISTS pages_deleted_at_purge_idx
ON pages (deleted_at) WHERE deleted_at IS NOT NULL;
-- v0.29.1: expression index used by since/until date-range filters that read
-- COALESCE(effective_date, updated_at). A partial index on effective_date
-- alone would NOT help — the planner can't use it for the negative side of
-- the COALESCE. Expression index is what actually accelerates the filter.
CREATE INDEX IF NOT EXISTS pages_coalesce_date_idx
ON pages ((COALESCE(effective_date, updated_at)));
-- ============================================================
-- content_chunks: chunked content with embeddings
-- ============================================================
CREATE TABLE IF NOT EXISTS content_chunks (
id SERIAL PRIMARY KEY,
page_id INTEGER NOT NULL REFERENCES pages(id) ON DELETE CASCADE,
chunk_index INTEGER NOT NULL,
chunk_text TEXT NOT NULL,
chunk_source TEXT NOT NULL DEFAULT 'compiled_truth',
embedding vector(1536),
model TEXT NOT NULL DEFAULT 'text-embedding-3-large',
token_count INTEGER,
embedded_at TIMESTAMPTZ,
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
-- v0.19.0: code chunk metadata. Nullable — markdown chunks leave these NULL.
-- Powers `query --lang`, `code-def <symbol>`, and `code-refs <symbol>`.
language TEXT,
symbol_name TEXT,
symbol_type TEXT,
start_line INTEGER,
end_line INTEGER,
-- v0.20.0 Cathedral II: qualified symbol identity + parent scope + doc-comment
-- + chunk-grain FTS. All nullable — markdown chunks leave these NULL.
parent_symbol_path TEXT[],
doc_comment TEXT,
symbol_name_qualified TEXT,
search_vector TSVECTOR,
-- v0.27.1 multimodal. modality discriminates text vs image rows for search
-- filtering. embedding_image holds 1024-dim Voyage multimodal vectors;
-- mixed-provider brains (e.g. OpenAI 1536 text + Voyage 1024 images) keep
-- both columns populated with distinct dim spaces.
modality TEXT NOT NULL DEFAULT 'text',
embedding_image vector(1024)
);
CREATE UNIQUE INDEX IF NOT EXISTS idx_chunks_page_index ON content_chunks(page_id, chunk_index);
CREATE INDEX IF NOT EXISTS idx_chunks_page ON content_chunks(page_id);
CREATE INDEX IF NOT EXISTS idx_chunks_embedding ON content_chunks USING hnsw (embedding vector_cosine_ops);
-- v0.19.0: partial indexes — only code chunks populate these columns.
CREATE INDEX IF NOT EXISTS idx_chunks_symbol_name ON content_chunks(symbol_name) WHERE symbol_name IS NOT NULL;
CREATE INDEX IF NOT EXISTS idx_chunks_language ON content_chunks(language) WHERE language IS NOT NULL;
-- v0.27.1: partial HNSW for multimodal images. Footprint stays proportional
-- to image-chunk count, not table size.
CREATE INDEX IF NOT EXISTS idx_chunks_embedding_image
ON content_chunks USING hnsw (embedding_image vector_cosine_ops)
WHERE embedding_image IS NOT NULL;
-- v0.20.0 Cathedral II: GIN index on the new chunk-grain FTS vector.
CREATE INDEX IF NOT EXISTS idx_chunks_search_vector ON content_chunks USING GIN(search_vector);
CREATE INDEX IF NOT EXISTS idx_chunks_symbol_qualified
ON content_chunks(symbol_name_qualified) WHERE symbol_name_qualified IS NOT NULL;
-- v0.20.0 Cathedral II: chunk-grain FTS trigger.
-- Weight 'A' on doc_comment + symbol_name_qualified; weight 'B' on chunk_text.
-- NL queries ("how do we handle errors") rank doc-comment hits above body text.
-- BEFORE INSERT OR UPDATE OF specific columns — only refires when those change,
-- not on every chunk update (e.g., embedding refresh doesn't trigger rebuild).
CREATE OR REPLACE FUNCTION update_chunk_search_vector() RETURNS TRIGGER AS $fn$
BEGIN
NEW.search_vector :=
setweight(to_tsvector('english', COALESCE(NEW.doc_comment, '')), 'A') ||
setweight(to_tsvector('english', COALESCE(NEW.symbol_name_qualified, '')), 'A') ||
setweight(to_tsvector('english', COALESCE(NEW.chunk_text, '')), 'B');
RETURN NEW;
END;
$fn$ LANGUAGE plpgsql;
DROP TRIGGER IF EXISTS chunk_search_vector_trigger ON content_chunks;
CREATE TRIGGER chunk_search_vector_trigger
BEFORE INSERT OR UPDATE OF chunk_text, doc_comment, symbol_name_qualified
ON content_chunks
FOR EACH ROW EXECUTE FUNCTION update_chunk_search_vector();
-- ============================================================
-- code_edges_chunk + code_edges_symbol: v0.20.0 Cathedral II structural edges
-- ============================================================
-- Two-table design (codex F4 + SP-7):
-- - code_edges_chunk: resolved edges (both endpoints = known chunk IDs)
-- - code_edges_symbol: unresolved refs (target known by qualified name,
-- defining chunk not yet imported)
-- Readers UNION both tables; no promotion step.
-- Source scoping: from_chunk_id -> content_chunks -> pages.source_id
-- determines the source. Resolution logic MUST scope on source (codex SP-3);
-- only --all-sources callers bypass this. UNIQUE keys don't include source_id
-- because from_chunk_id already pins it.
CREATE TABLE IF NOT EXISTS code_edges_chunk (
id SERIAL PRIMARY KEY,
from_chunk_id INTEGER NOT NULL REFERENCES content_chunks(id) ON DELETE CASCADE,
to_chunk_id INTEGER NOT NULL REFERENCES content_chunks(id) ON DELETE CASCADE,
from_symbol_qualified TEXT NOT NULL,
to_symbol_qualified TEXT NOT NULL,
edge_type TEXT NOT NULL,
edge_metadata JSONB NOT NULL DEFAULT '{}',
source_id TEXT REFERENCES sources(id) ON DELETE CASCADE,
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
CONSTRAINT code_edges_chunk_unique UNIQUE (from_chunk_id, to_chunk_id, edge_type)
);
CREATE INDEX IF NOT EXISTS idx_code_edges_chunk_from
ON code_edges_chunk(from_chunk_id, edge_type);
CREATE INDEX IF NOT EXISTS idx_code_edges_chunk_to
ON code_edges_chunk(to_chunk_id, edge_type);
CREATE INDEX IF NOT EXISTS idx_code_edges_chunk_to_symbol
ON code_edges_chunk(to_symbol_qualified, edge_type);
CREATE TABLE IF NOT EXISTS code_edges_symbol (
id SERIAL PRIMARY KEY,
from_chunk_id INTEGER NOT NULL REFERENCES content_chunks(id) ON DELETE CASCADE,
from_symbol_qualified TEXT NOT NULL,
to_symbol_qualified TEXT NOT NULL,
edge_type TEXT NOT NULL,
edge_metadata JSONB NOT NULL DEFAULT '{}',
source_id TEXT REFERENCES sources(id) ON DELETE CASCADE,
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
CONSTRAINT code_edges_symbol_unique UNIQUE (from_chunk_id, to_symbol_qualified, edge_type)
);
CREATE INDEX IF NOT EXISTS idx_code_edges_symbol_from
ON code_edges_symbol(from_chunk_id, edge_type);
CREATE INDEX IF NOT EXISTS idx_code_edges_symbol_to
ON code_edges_symbol(to_symbol_qualified, edge_type);
-- ============================================================
-- links: cross-references between pages
-- ============================================================
-- Provenance model (v0.13):
-- link_source — 'markdown' | 'frontmatter' | 'manual' | NULL
-- (NULL = legacy row written before v0.13; unknown source)
-- origin_page_id — for link_source='frontmatter', the page whose YAML
-- frontmatter created this edge; scopes reconciliation
-- origin_field — the frontmatter field name (e.g. 'key_people')
--
-- The unique constraint includes link_source + origin_page_id so a manual edge
-- and a frontmatter-derived edge with the same (from, to, type) tuple coexist.
-- Reconciliation on put_page filters by (link_source='frontmatter' AND
-- origin_page_id = written_page) — never touches other pages' edges.
CREATE TABLE IF NOT EXISTS links (
id SERIAL PRIMARY KEY,
from_page_id INTEGER NOT NULL REFERENCES pages(id) ON DELETE CASCADE,
to_page_id INTEGER NOT NULL REFERENCES pages(id) ON DELETE CASCADE,
link_type TEXT NOT NULL DEFAULT '',
context TEXT NOT NULL DEFAULT '',
link_source TEXT CHECK (link_source IS NULL OR link_source IN ('markdown', 'frontmatter', 'manual')),
origin_page_id INTEGER REFERENCES pages(id) ON DELETE SET NULL,
origin_field TEXT,
-- v0.18.0 Step 4: 'qualified' when the link was written as
-- [[source:slug]] (target source pinned). 'unqualified' when written
-- as bare [[slug]] and resolved via local-first fallback at
-- extraction time. NULL for legacy/manual/frontmatter edges.
resolution_type TEXT CHECK (resolution_type IS NULL OR resolution_type IN ('qualified', 'unqualified')),
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
-- NULLS NOT DISTINCT (PG15+) so two rows with link_source IS NULL or
-- origin_page_id IS NULL collide as expected. Without this, every row with
-- NULL origin_page_id (markdown/manual edges) would be treated as unique.
CONSTRAINT links_from_to_type_source_origin_unique
UNIQUE NULLS NOT DISTINCT (from_page_id, to_page_id, link_type, link_source, origin_page_id)
);
CREATE INDEX IF NOT EXISTS idx_links_from ON links(from_page_id);
CREATE INDEX IF NOT EXISTS idx_links_to ON links(to_page_id);
CREATE INDEX IF NOT EXISTS idx_links_source ON links(link_source);
CREATE INDEX IF NOT EXISTS idx_links_origin ON links(origin_page_id);
-- ============================================================
-- tags
-- ============================================================
CREATE TABLE IF NOT EXISTS tags (
id SERIAL PRIMARY KEY,
page_id INTEGER NOT NULL REFERENCES pages(id) ON DELETE CASCADE,
tag TEXT NOT NULL,
UNIQUE(page_id, tag)
);
CREATE INDEX IF NOT EXISTS idx_tags_tag ON tags(tag);
CREATE INDEX IF NOT EXISTS idx_tags_page_id ON tags(page_id);
-- ============================================================
-- raw_data: sidecar data (replaces .raw/ JSON files)
-- ============================================================
CREATE TABLE IF NOT EXISTS raw_data (
id SERIAL PRIMARY KEY,
page_id INTEGER NOT NULL REFERENCES pages(id) ON DELETE CASCADE,
source TEXT NOT NULL,
data JSONB NOT NULL,
fetched_at TIMESTAMPTZ NOT NULL DEFAULT now(),
UNIQUE(page_id, source)
);
CREATE INDEX IF NOT EXISTS idx_raw_data_page ON raw_data(page_id);
-- ============================================================
-- timeline_entries: structured timeline
-- ============================================================
CREATE TABLE IF NOT EXISTS timeline_entries (
id SERIAL PRIMARY KEY,
page_id INTEGER NOT NULL REFERENCES pages(id) ON DELETE CASCADE,
date DATE NOT NULL,
source TEXT NOT NULL DEFAULT '',
summary TEXT NOT NULL,
detail TEXT NOT NULL DEFAULT '',
created_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
CREATE INDEX IF NOT EXISTS idx_timeline_page ON timeline_entries(page_id);
CREATE INDEX IF NOT EXISTS idx_timeline_date ON timeline_entries(date);
-- Dedup constraint: same (page, date, summary) treated as same event
CREATE UNIQUE INDEX IF NOT EXISTS idx_timeline_dedup ON timeline_entries(page_id, date, summary);
-- ============================================================
-- page_versions: snapshot history for compiled_truth
-- ============================================================
CREATE TABLE IF NOT EXISTS page_versions (
id SERIAL PRIMARY KEY,
page_id INTEGER NOT NULL REFERENCES pages(id) ON DELETE CASCADE,
compiled_truth TEXT NOT NULL,
frontmatter JSONB NOT NULL DEFAULT '{}',
snapshot_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
CREATE INDEX IF NOT EXISTS idx_versions_page ON page_versions(page_id);
-- ============================================================
-- ingest_log
-- ============================================================
-- v0.31.2 (codex P1 #3): source_id added so facts:absorb logging
-- (runFactsBackstop / writeFactsAbsorbLog) and doctor's
-- facts_extraction_health check can scope failure counts per source.
-- Migration v47 ALTERs existing brains; this inline definition covers
-- fresh installs.
CREATE TABLE IF NOT EXISTS ingest_log (
id SERIAL PRIMARY KEY,
source_id TEXT NOT NULL DEFAULT 'default',
source_type TEXT NOT NULL,
source_ref TEXT NOT NULL,
pages_updated JSONB NOT NULL DEFAULT '[]',
summary TEXT NOT NULL DEFAULT '',
created_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
CREATE INDEX IF NOT EXISTS idx_ingest_log_source_type_created
ON ingest_log (source_id, source_type, created_at DESC);
-- ============================================================
-- config: brain-level settings
-- ============================================================
CREATE TABLE IF NOT EXISTS config (
key TEXT PRIMARY KEY,
value TEXT NOT NULL
);
INSERT INTO config (key, value) VALUES
('version', '1'),
('embedding_model', 'text-embedding-3-large'),
('embedding_dimensions', '1536'),
('chunk_strategy', 'semantic')
ON CONFLICT (key) DO NOTHING;
-- ============================================================
-- access_tokens: bearer tokens for remote MCP access
-- ============================================================
CREATE TABLE IF NOT EXISTS access_tokens (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
name TEXT NOT NULL,
token_hash TEXT NOT NULL UNIQUE,
scopes TEXT[],
created_at TIMESTAMPTZ DEFAULT now(),
last_used_at TIMESTAMPTZ,
revoked_at TIMESTAMPTZ
);
CREATE INDEX IF NOT EXISTS idx_access_tokens_hash ON access_tokens (token_hash) WHERE revoked_at IS NULL;
-- ============================================================
-- mcp_request_log: usage logging for remote MCP requests
-- ============================================================
CREATE TABLE IF NOT EXISTS mcp_request_log (
id SERIAL PRIMARY KEY,
token_name TEXT,
agent_name TEXT,
operation TEXT NOT NULL,
latency_ms INTEGER,
status TEXT NOT NULL DEFAULT 'success',
params JSONB,
error_message TEXT,
created_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
-- ============================================================
-- OAuth 2.1: clients, tokens, authorization codes
-- ============================================================
CREATE TABLE IF NOT EXISTS oauth_clients (
client_id TEXT PRIMARY KEY,
client_secret_hash TEXT,
client_name TEXT NOT NULL,
redirect_uris TEXT[],
grant_types TEXT[] DEFAULT '{"client_credentials"}',
scope TEXT,
token_endpoint_auth_method TEXT,
client_id_issued_at BIGINT,
client_secret_expires_at BIGINT,
token_ttl INTEGER,
deleted_at TIMESTAMPTZ,
created_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
CREATE TABLE IF NOT EXISTS oauth_tokens (
token_hash TEXT PRIMARY KEY,
token_type TEXT NOT NULL,
client_id TEXT NOT NULL REFERENCES oauth_clients(client_id) ON DELETE CASCADE,
scopes TEXT[],
expires_at BIGINT,
resource TEXT,
created_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
CREATE INDEX IF NOT EXISTS idx_oauth_tokens_expiry ON oauth_tokens(expires_at);
CREATE INDEX IF NOT EXISTS idx_oauth_tokens_client ON oauth_tokens(client_id);
CREATE TABLE IF NOT EXISTS oauth_codes (
code_hash TEXT PRIMARY KEY,
client_id TEXT NOT NULL REFERENCES oauth_clients(client_id) ON DELETE CASCADE,
scopes TEXT[],
code_challenge TEXT NOT NULL,
code_challenge_method TEXT NOT NULL DEFAULT 'S256',
redirect_uri TEXT NOT NULL,
state TEXT,
resource TEXT,
expires_at BIGINT NOT NULL,
created_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
-- Composite indexes for admin dashboard request log queries
CREATE INDEX IF NOT EXISTS idx_mcp_log_time_agent ON mcp_request_log(created_at, token_name);
CREATE INDEX IF NOT EXISTS idx_mcp_log_agent_time ON mcp_request_log(agent_name, created_at DESC);
-- ============================================================
-- files: binary attachments stored in Supabase Storage
-- ============================================================
-- v0.18.0 Step 7: files gains source_id + page_id alongside the
-- legacy page_slug (kept for backward compat until a later release).
-- The file_migration_ledger below drives the storage object rewrite.
-- page_slug FK had ON UPDATE CASCADE — removed because slugs are no
-- longer global (composite UNIQUE) so CASCADE on-update is ambiguous.
-- ON DELETE SET NULL is preserved via both page_slug and page_id.
CREATE TABLE IF NOT EXISTS files (
id SERIAL PRIMARY KEY,
source_id TEXT NOT NULL DEFAULT 'default'
REFERENCES sources(id) ON DELETE CASCADE,
page_slug TEXT,
page_id INTEGER REFERENCES pages(id) ON DELETE SET NULL,
filename TEXT NOT NULL,
storage_path TEXT NOT NULL,
mime_type TEXT,
size_bytes BIGINT,
content_hash TEXT NOT NULL,
metadata JSONB NOT NULL DEFAULT '{}',
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
UNIQUE(storage_path)
);
-- Migration: drop storage_url if it exists (renamed to storage_path only)
ALTER TABLE files DROP COLUMN IF EXISTS storage_url;
CREATE INDEX IF NOT EXISTS idx_files_page ON files(page_slug);
CREATE INDEX IF NOT EXISTS idx_files_page_id ON files(page_id);
CREATE INDEX IF NOT EXISTS idx_files_source_id ON files(source_id);
CREATE INDEX IF NOT EXISTS idx_files_hash ON files(content_hash);
-- ============================================================
-- file_migration_ledger (v0.18.0 Step 7)
-- Drives the storage-object rewrite performed by the v0_18_0
-- orchestrator's phase B. Keyed on file_id so two sources can share
-- an old path during migration without PK collision (Codex second-
-- pass caught this).
-- Status state machine: pending → copy_done → db_updated → complete
-- ============================================================
CREATE TABLE IF NOT EXISTS file_migration_ledger (
file_id INTEGER PRIMARY KEY REFERENCES files(id) ON DELETE CASCADE,
storage_path_old TEXT NOT NULL,
storage_path_new TEXT NOT NULL,
status TEXT NOT NULL DEFAULT 'pending',
error TEXT,
updated_at TIMESTAMPTZ NOT NULL DEFAULT now(),
CONSTRAINT chk_ledger_status CHECK (status IN ('pending','copy_done','db_updated','complete','failed'))
);
CREATE INDEX IF NOT EXISTS idx_file_migration_ledger_status
ON file_migration_ledger(status) WHERE status != 'complete';
-- ============================================================
-- Trigger-based search_vector (spans pages + timeline_entries)
-- ============================================================
ALTER TABLE pages ADD COLUMN IF NOT EXISTS search_vector tsvector;
CREATE INDEX IF NOT EXISTS idx_pages_search ON pages USING GIN(search_vector);
-- Function to rebuild search_vector for a page
CREATE OR REPLACE FUNCTION update_page_search_vector() RETURNS trigger AS $$
DECLARE
timeline_text TEXT;
BEGIN
-- Gather timeline_entries text for this page
SELECT coalesce(string_agg(summary || ' ' || detail, ' '), '')
INTO timeline_text
FROM timeline_entries
WHERE page_id = NEW.id;
-- Build weighted tsvector
NEW.search_vector :=
setweight(to_tsvector('english', coalesce(NEW.title, '')), 'A') ||
setweight(to_tsvector('english', coalesce(NEW.compiled_truth, '')), 'B') ||
setweight(to_tsvector('english', coalesce(NEW.timeline, '')), 'C') ||
setweight(to_tsvector('english', coalesce(timeline_text, '')), 'C');
RETURN NEW;
END;
$$ LANGUAGE plpgsql;
DROP TRIGGER IF EXISTS trg_pages_search_vector ON pages;
CREATE TRIGGER trg_pages_search_vector
BEFORE INSERT OR UPDATE ON pages
FOR EACH ROW
EXECUTE FUNCTION update_page_search_vector();
-- Note: timeline_entries trigger removed (v0.10.1).
-- Structured timeline_entries power temporal queries (graph layer).
-- The markdown timeline section in pages.timeline still feeds search_vector via
-- the trg_pages_search_vector trigger above. Removing the timeline_entries
-- trigger avoids double-weighting the same content in search and prevents
-- mutation-induced reordering during timeline-extract pagination.
DROP TRIGGER IF EXISTS trg_timeline_search_vector ON timeline_entries;
DROP FUNCTION IF EXISTS update_page_search_vector_from_timeline();
-- ============================================================
-- Minion Jobs: BullMQ-inspired Postgres-native job queue
-- ============================================================
CREATE TABLE IF NOT EXISTS minion_jobs (
id SERIAL PRIMARY KEY,
name TEXT NOT NULL,
queue TEXT NOT NULL DEFAULT 'default',
status TEXT NOT NULL DEFAULT 'waiting',
priority INTEGER NOT NULL DEFAULT 0,
data JSONB NOT NULL DEFAULT '{}',
max_attempts INTEGER NOT NULL DEFAULT 3,
attempts_made INTEGER NOT NULL DEFAULT 0,
attempts_started INTEGER NOT NULL DEFAULT 0,
backoff_type TEXT NOT NULL DEFAULT 'exponential',
backoff_delay INTEGER NOT NULL DEFAULT 1000,
backoff_jitter REAL NOT NULL DEFAULT 0.2,
stalled_counter INTEGER NOT NULL DEFAULT 0,
max_stalled INTEGER NOT NULL DEFAULT 5,
lock_token TEXT,
lock_until TIMESTAMPTZ,
delay_until TIMESTAMPTZ,
parent_job_id INTEGER REFERENCES minion_jobs(id) ON DELETE SET NULL,
on_child_fail TEXT NOT NULL DEFAULT 'fail_parent',
tokens_input INTEGER NOT NULL DEFAULT 0,
tokens_output INTEGER NOT NULL DEFAULT 0,
tokens_cache_read INTEGER NOT NULL DEFAULT 0,
result JSONB,
progress JSONB,
error_text TEXT,
stacktrace JSONB DEFAULT '[]',
depth INTEGER NOT NULL DEFAULT 0,
max_children INTEGER,
timeout_ms INTEGER,
timeout_at TIMESTAMPTZ,
remove_on_complete BOOLEAN NOT NULL DEFAULT FALSE,
remove_on_fail BOOLEAN NOT NULL DEFAULT FALSE,
idempotency_key TEXT,
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
started_at TIMESTAMPTZ,
finished_at TIMESTAMPTZ,
updated_at TIMESTAMPTZ NOT NULL DEFAULT now(),
CONSTRAINT chk_status CHECK (status IN ('waiting','active','completed','failed','delayed','dead','cancelled','waiting-children','paused')),
CONSTRAINT chk_backoff_type CHECK (backoff_type IN ('fixed','exponential')),
CONSTRAINT chk_on_child_fail CHECK (on_child_fail IN ('fail_parent','remove_dep','ignore','continue')),
CONSTRAINT chk_jitter_range CHECK (backoff_jitter >= 0.0 AND backoff_jitter <= 1.0),
CONSTRAINT chk_attempts_order CHECK (attempts_made <= attempts_started),
CONSTRAINT chk_nonnegative CHECK (attempts_made >= 0 AND attempts_started >= 0 AND stalled_counter >= 0 AND max_attempts >= 1 AND max_stalled >= 0),
CONSTRAINT chk_depth_nonnegative CHECK (depth >= 0),
CONSTRAINT chk_max_children_positive CHECK (max_children IS NULL OR max_children > 0),
CONSTRAINT chk_timeout_positive CHECK (timeout_ms IS NULL OR timeout_ms > 0)
);
CREATE INDEX IF NOT EXISTS idx_minion_jobs_claim ON minion_jobs (queue, priority ASC, created_at ASC) WHERE status = 'waiting';
CREATE INDEX IF NOT EXISTS idx_minion_jobs_status ON minion_jobs(status);
CREATE INDEX IF NOT EXISTS idx_minion_jobs_stalled ON minion_jobs (lock_until) WHERE status = 'active';
CREATE INDEX IF NOT EXISTS idx_minion_jobs_delayed ON minion_jobs (delay_until) WHERE status = 'delayed';
CREATE INDEX IF NOT EXISTS idx_minion_jobs_parent ON minion_jobs(parent_job_id);
CREATE INDEX IF NOT EXISTS idx_minion_jobs_timeout ON minion_jobs (timeout_at) WHERE status = 'active' AND timeout_at IS NOT NULL;
CREATE INDEX IF NOT EXISTS idx_minion_jobs_parent_status ON minion_jobs (parent_job_id, status) WHERE parent_job_id IS NOT NULL;
CREATE UNIQUE INDEX IF NOT EXISTS uniq_minion_jobs_idempotency ON minion_jobs (idempotency_key) WHERE idempotency_key IS NOT NULL;
-- Inbox table for sidechannel messaging
CREATE TABLE IF NOT EXISTS minion_inbox (
id SERIAL PRIMARY KEY,
job_id INTEGER NOT NULL REFERENCES minion_jobs(id) ON DELETE CASCADE,
sender TEXT NOT NULL,
payload JSONB NOT NULL,
sent_at TIMESTAMPTZ NOT NULL DEFAULT now(),
read_at TIMESTAMPTZ
);
CREATE INDEX IF NOT EXISTS idx_minion_inbox_unread ON minion_inbox (job_id) WHERE read_at IS NULL;
CREATE INDEX IF NOT EXISTS idx_minion_inbox_child_done ON minion_inbox (job_id, sent_at) WHERE payload->>'type' = 'child_done';
-- Attachments table: per-job binary blobs (manifests, agent outputs, files)
CREATE TABLE IF NOT EXISTS minion_attachments (
id SERIAL PRIMARY KEY,
job_id INTEGER NOT NULL REFERENCES minion_jobs(id) ON DELETE CASCADE,
filename TEXT NOT NULL,
content_type TEXT NOT NULL,
content BYTEA,
storage_uri TEXT,
size_bytes INTEGER NOT NULL,
sha256 TEXT NOT NULL,
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
CONSTRAINT uniq_minion_attachments_job_filename UNIQUE (job_id, filename),
CONSTRAINT chk_attachment_storage CHECK (content IS NOT NULL OR storage_uri IS NOT NULL),
CONSTRAINT chk_attachment_size CHECK (size_bytes >= 0)
);
CREATE INDEX IF NOT EXISTS idx_minion_attachments_job ON minion_attachments (job_id);
ALTER TABLE minion_attachments ALTER COLUMN content SET STORAGE EXTERNAL;
-- ============================================================
-- Subagent runtime (v0.16.0) — durable LLM loops
-- ============================================================
-- Anthropic-native message blocks, one row per Messages API message. Parallel
-- tool_use blocks in one assistant message live in content_blocks JSONB,
-- not across rows.
CREATE TABLE IF NOT EXISTS subagent_messages (
id BIGSERIAL PRIMARY KEY,
job_id BIGINT NOT NULL REFERENCES minion_jobs(id) ON DELETE CASCADE,
message_idx INTEGER NOT NULL,
role TEXT NOT NULL,
-- v0.27+ stores provider-neutral ChatBlock[] when schema_version=2; legacy
-- Anthropic-shape blocks when schema_version=1 (pre-v0.27 jobs replay).
content_blocks JSONB NOT NULL,
schema_version INTEGER NOT NULL DEFAULT 1,
-- Recipe id of the provider that produced this turn (e.g. 'anthropic',
-- 'openai', 'deepseek'). NULL on legacy v1 rows; set on v2.
provider_id TEXT,
tokens_in INTEGER,
tokens_out INTEGER,
tokens_cache_read INTEGER,
tokens_cache_create INTEGER,
model TEXT,
ended_at TIMESTAMPTZ NOT NULL DEFAULT now(),
CONSTRAINT uniq_subagent_messages_idx UNIQUE (job_id, message_idx),
CONSTRAINT chk_subagent_messages_role CHECK (role IN ('user','assistant'))
);
CREATE INDEX IF NOT EXISTS idx_subagent_messages_job ON subagent_messages (job_id, message_idx);
CREATE INDEX IF NOT EXISTS idx_subagent_messages_provider ON subagent_messages (job_id, provider_id);
-- Two-phase tool execution ledger. Before tool call: INSERT status='pending'.
-- After success: UPDATE to 'complete' + output. On failure: 'failed' + error.
-- Replay re-runs 'pending' rows only if the tool is idempotent.
CREATE TABLE IF NOT EXISTS subagent_tool_executions (
id BIGSERIAL PRIMARY KEY,
job_id BIGINT NOT NULL REFERENCES minion_jobs(id) ON DELETE CASCADE,
message_idx INTEGER NOT NULL,
tool_use_id TEXT NOT NULL,
tool_name TEXT NOT NULL,
input JSONB NOT NULL,
status TEXT NOT NULL,
output JSONB,
error TEXT,
schema_version INTEGER NOT NULL DEFAULT 1,
provider_id TEXT,
started_at TIMESTAMPTZ NOT NULL DEFAULT now(),
ended_at TIMESTAMPTZ,
CONSTRAINT uniq_subagent_tools_use_id UNIQUE (job_id, tool_use_id),
CONSTRAINT chk_subagent_tools_status CHECK (status IN ('pending','complete','failed'))
);
CREATE INDEX IF NOT EXISTS idx_subagent_tools_job ON subagent_tool_executions (job_id, status);
-- Rate-lease table — concurrency cap on outbound providers (e.g.
-- anthropic:messages). Acquire: INSERT if active < max_concurrent under
-- advisory lock. Release: DELETE. Stale leases (expires_at past) auto-prune
-- on next acquire so crashed workers can't strand capacity.
CREATE TABLE IF NOT EXISTS subagent_rate_leases (
id BIGSERIAL PRIMARY KEY,
key TEXT NOT NULL,
owner_job_id BIGINT NOT NULL REFERENCES minion_jobs(id) ON DELETE CASCADE,
acquired_at TIMESTAMPTZ NOT NULL DEFAULT now(),
expires_at TIMESTAMPTZ NOT NULL
);
CREATE INDEX IF NOT EXISTS idx_rate_leases_key_expires ON subagent_rate_leases (key, expires_at);
-- ============================================================
-- Dream-cycle significance verdict cache — v0.21 synthesize phase
-- ============================================================
-- Caches the cheap Haiku "is this transcript worth processing?" verdict
-- per (file_path, content_hash) so backfill re-runs skip already-judged
-- files. Distinct from raw_data (which is page-scoped); transcripts
-- aren't pages.
CREATE TABLE IF NOT EXISTS dream_verdicts (
file_path TEXT NOT NULL,
content_hash TEXT NOT NULL,
worth_processing BOOLEAN NOT NULL,
reasons JSONB,
judged_at TIMESTAMPTZ NOT NULL DEFAULT now(),
PRIMARY KEY (file_path, content_hash)
);
-- ============================================================
-- Cycle coordination lock — v0.17 runCycle primitive
-- ============================================================
-- One row per active cycle. Any caller (autopilot daemon, Minions
-- autopilot-cycle handler, gbrain dream CLI) tries to acquire this
-- row before running a DB-write phase. Holders refresh ttl_expires_at
-- between phases; crashed holders auto-release once TTL expires.
-- Works through PgBouncer transaction pooling, unlike session-scoped
-- pg_try_advisory_lock.
CREATE TABLE IF NOT EXISTS gbrain_cycle_locks (
id TEXT PRIMARY KEY,
holder_pid INT NOT NULL,
holder_host TEXT,
acquired_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
ttl_expires_at TIMESTAMPTZ NOT NULL
);
CREATE INDEX IF NOT EXISTS idx_cycle_locks_ttl ON gbrain_cycle_locks(ttl_expires_at);
-- ============================================================
-- Eval capture (v0.25.0 — BrainBench-Real substrate)
-- ============================================================
-- eval_candidates: captured query/search calls from the op-layer wrapper
-- in src/core/operations.ts. PII is scrubbed before insert by
-- src/core/eval-capture-scrub.ts. query is CHECK-capped at 50KB.
-- eval_capture_failures: cross-process audit of insert failures, surfaced
-- by `gbrain doctor` (in-process counters can't bridge MCP server + doctor
-- CLI process boundaries).
CREATE TABLE IF NOT EXISTS eval_candidates (
id SERIAL PRIMARY KEY,
tool_name TEXT NOT NULL CHECK (tool_name IN ('query', 'search')),
query TEXT NOT NULL CHECK (length(query) <= 51200),
retrieved_slugs TEXT[] NOT NULL DEFAULT '{}',
retrieved_chunk_ids INTEGER[] NOT NULL DEFAULT '{}',
source_ids TEXT[] NOT NULL DEFAULT '{}',
expand_enabled BOOLEAN,
detail TEXT CHECK (detail IS NULL OR detail IN ('low', 'medium', 'high')),
detail_resolved TEXT CHECK (detail_resolved IS NULL OR detail_resolved IN ('low', 'medium', 'high')),
vector_enabled BOOLEAN NOT NULL,
expansion_applied BOOLEAN NOT NULL,
latency_ms INTEGER NOT NULL,
remote BOOLEAN NOT NULL,
job_id INTEGER,
subagent_id INTEGER,
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
-- v0.29.1 — agent-explicit recency + salience capture for replay reproducibility.
-- All nullable + additive. NDJSON schema_version stays at 1; consumers ignore unknown fields.
as_of_ts TIMESTAMPTZ,
salience_param TEXT,
recency_param TEXT,
salience_resolved TEXT,
recency_resolved TEXT,
salience_source TEXT,
recency_source TEXT
);
CREATE INDEX IF NOT EXISTS idx_eval_candidates_created_at ON eval_candidates(created_at DESC);
CREATE TABLE IF NOT EXISTS eval_capture_failures (
id SERIAL PRIMARY KEY,
ts TIMESTAMPTZ NOT NULL DEFAULT NOW(),
reason TEXT NOT NULL CHECK (reason IN ('db_down', 'rls_reject', 'check_violation', 'scrubber_exception', 'other'))
);
CREATE INDEX IF NOT EXISTS idx_eval_capture_failures_ts ON eval_capture_failures(ts DESC);
-- eval_takes_quality_runs (v0.32 — EXP-5): DB-authoritative receipts for the
-- takes-quality eval CLI. 4-sha unique key (corpus, prompt, models, rubric)
-- so re-running the same run is a no-op (ON CONFLICT DO NOTHING) and a
-- future rubric tweak segregates trend rows cleanly. receipt_json carries
-- the full receipt blob so `replay` can reconstruct when the disk artifact
-- is missing. Mirrored in src/core/pglite-schema.ts + migration v49.
CREATE TABLE IF NOT EXISTS eval_takes_quality_runs (
id BIGSERIAL PRIMARY KEY,
receipt_sha8_corpus TEXT NOT NULL,
receipt_sha8_prompt TEXT NOT NULL,
receipt_sha8_models TEXT NOT NULL,
receipt_sha8_rubric TEXT NOT NULL,
rubric_version TEXT NOT NULL,
verdict TEXT NOT NULL CHECK (verdict IN ('pass','fail','inconclusive')),
overall_score REAL NOT NULL,
dim_scores JSONB NOT NULL,
cost_usd REAL NOT NULL,
receipt_json JSONB NOT NULL,
receipt_disk_path TEXT,
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
UNIQUE (receipt_sha8_corpus, receipt_sha8_prompt, receipt_sha8_models, receipt_sha8_rubric)
);
CREATE INDEX IF NOT EXISTS eval_takes_quality_runs_trend_idx
ON eval_takes_quality_runs (rubric_version, created_at DESC);
-- NOTIFY trigger for real-time job events (Postgres only, not PGLite)
CREATE OR REPLACE FUNCTION notify_minion_job_change() RETURNS trigger AS $$
BEGIN
PERFORM pg_notify('minion_jobs', json_build_object(
'id', NEW.id, 'status', NEW.status, 'name', NEW.name,
'queue', NEW.queue, 'prev_status', COALESCE(OLD.status, 'new')
)::text);
RETURN NEW;
END;
$$ LANGUAGE plpgsql;
DROP TRIGGER IF EXISTS minion_job_notify ON minion_jobs;
CREATE TRIGGER minion_job_notify AFTER INSERT OR UPDATE OF status ON minion_jobs
FOR EACH ROW EXECUTE FUNCTION notify_minion_job_change();
-- ============================================================
-- Row Level Security: block anon access, postgres role bypasses
-- ============================================================
-- The postgres role (used by gbrain via pooler) has BYPASSRLS.
-- Enabling RLS with no policies means the anon key can't read anything.
-- Only enable if the current role actually has BYPASSRLS privilege,
-- otherwise we'd lock ourselves out.
DO $$
DECLARE
has_bypass BOOLEAN;
BEGIN
SELECT rolbypassrls INTO has_bypass FROM pg_roles WHERE rolname = current_user;
IF has_bypass THEN
ALTER TABLE pages ENABLE ROW LEVEL SECURITY;
ALTER TABLE content_chunks ENABLE ROW LEVEL SECURITY;
ALTER TABLE links ENABLE ROW LEVEL SECURITY;
ALTER TABLE tags ENABLE ROW LEVEL SECURITY;
ALTER TABLE raw_data ENABLE ROW LEVEL SECURITY;
ALTER TABLE timeline_entries ENABLE ROW LEVEL SECURITY;
ALTER TABLE page_versions ENABLE ROW LEVEL SECURITY;
ALTER TABLE ingest_log ENABLE ROW LEVEL SECURITY;
ALTER TABLE config ENABLE ROW LEVEL SECURITY;
ALTER TABLE files ENABLE ROW LEVEL SECURITY;
ALTER TABLE minion_jobs ENABLE ROW LEVEL SECURITY;
ALTER TABLE sources ENABLE ROW LEVEL SECURITY;
ALTER TABLE file_migration_ledger ENABLE ROW LEVEL SECURITY;
ALTER TABLE access_tokens ENABLE ROW LEVEL SECURITY;
ALTER TABLE mcp_request_log ENABLE ROW LEVEL SECURITY;
ALTER TABLE minion_inbox ENABLE ROW LEVEL SECURITY;
ALTER TABLE minion_attachments ENABLE ROW LEVEL SECURITY;
ALTER TABLE subagent_messages ENABLE ROW LEVEL SECURITY;
ALTER TABLE subagent_tool_executions ENABLE ROW LEVEL SECURITY;
ALTER TABLE subagent_rate_leases ENABLE ROW LEVEL SECURITY;
ALTER TABLE gbrain_cycle_locks ENABLE ROW LEVEL SECURITY;
ALTER TABLE dream_verdicts ENABLE ROW LEVEL SECURITY;
ALTER TABLE eval_candidates ENABLE ROW LEVEL SECURITY;
ALTER TABLE eval_capture_failures ENABLE ROW LEVEL SECURITY;
ALTER TABLE eval_takes_quality_runs ENABLE ROW LEVEL SECURITY;
-- v0.26 OAuth 2.1 tables
ALTER TABLE oauth_clients ENABLE ROW LEVEL SECURITY;
ALTER TABLE oauth_tokens ENABLE ROW LEVEL SECURITY;
ALTER TABLE oauth_codes ENABLE ROW LEVEL SECURITY;
RAISE NOTICE 'RLS enabled on all tables (role % has BYPASSRLS)', current_user;
ELSE
RAISE WARNING 'Skipping RLS: role % does not have BYPASSRLS privilege. Run as postgres role to enable.', current_user;
END IF;
END $$;