mirror of
https://github.com/garrytan/gbrain.git
synced 2026-07-27 22:15:33 +00:00
* fix(sync): honor --dry-run in full-sync path + expose embedded count
Precondition for v0.17 brain maintenance cycle (runCycle primitive).
The full-sync path (performFullSync) previously called runImport() even
when opts.dryRun was true, silently writing to the DB and advancing
sync.last_commit. `gbrain sync --dry-run` on a fresh brain (or with
--full) would mutate state without warning.
Fix:
- performFullSync now early-returns a `dry_run` SyncResult when
opts.dryRun is set. Walks the repo via collectMarkdownFiles +
isSyncable to count what WOULD be imported. No writes, no git
state advance.
- SyncResult gains an `embedded: number` field (required). Tracks
pages re-embedded during the sync's auto-embed step. Existing
return sites set 0; the synced + first_sync paths set real counts
(best-estimate until commit 2 sharpens runEmbedCore's return type).
- first_sync path now returns real added + chunksCreated counts
from runImport instead of hardcoded zeros.
- printSyncResult shows embedded count in human output.
Tests (test/sync.test.ts, new `performSync dry-run never writes`
block, PGLite + temp git repo, no DATABASE_URL required):
- first-sync --dry-run: no pages, no sync.last_commit
- incremental --dry-run after real sync: bookmark unchanged
- --full --dry-run: no reimport, bookmark unchanged
- SyncResult.embedded is a number
Codex outside-voice caught this. Would have shipped silent DB writes
on dry-run for anyone using `gbrain sync --dry-run --full`.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(embed): add dry-run mode + return EmbedResult with counts
Precondition for v0.17 brain maintenance cycle (runCycle primitive).
runEmbedCore previously returned Promise<void> and had no dry-run mode.
That made it impossible for runCycle to (a) report accurate embedded
counts or (b) honor --dry-run without also skipping the entire embed
phase (which would have required runCycle to know embed's internal
semantics — a layering violation).
Changes:
- EmbedOpts gains `dryRun?: boolean`. When set, embedPage and
embedAll enumerate stale chunks (or would-be-created chunks for
unchunked pages, via local chunkText without engine.upsertChunks)
but never call embedBatch and never write to the engine.
- runEmbedCore: Promise<void> -> Promise<EmbedResult>. Result shape:
{ embedded, skipped, would_embed, total_chunks, pages_processed,
dryRun }.
embedded = chunks newly embedded (0 in dryRun).
would_embed = chunks that WOULD be embedded (0 in non-dryRun).
skipped = chunks with pre-existing embeddings.
- runEmbed CLI wrapper honors --dry-run flag and returns the result
through. `gbrain embed --stale --dry-run` is now a safe preview.
- Callers ignoring the return value (sync auto-embed, autopilot
inline fallback, jobs.ts handlers, CLI) keep compiling — the new
return type is additive for `await` callers.
Tests (test/embed.test.ts, new `runEmbedCore --dry-run` block, uses
the existing mock.module embedBatch pattern, no API key required):
- dry-run --all: zero embedBatch calls, zero upsertChunks calls,
would_embed matches stale chunk total
- dry-run --stale correctly splits stale vs already-embedded counts
- dry-run --slugs on a single page tallies per-chunk counts
- non-dry-run regression guard: embedded count matches across
concurrent workers
Codex outside-voice flagged the Promise<void> return as a blocker for
accurate CycleReport.totals.pages_embedded.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* refactor(orphans): engine-injected queries, drop db.getConnection() global
Precondition for v0.17 brain maintenance cycle (runCycle primitive).
findOrphans + queryOrphanPages previously reached into the postgres-js
singleton via db.getConnection(), which (a) didn't compose with
runCycle's explicit-engine contract and (b) was wrong for PGLite test
fixtures and for any caller not using the default global connection.
Codex outside-voice flagged this as a blocker.
Changes:
- BrainEngine interface gains findOrphanPages() — returns pages with
no inbound links via the same NOT EXISTS anti-join. Implemented on
both postgres-engine (sql tag) and pglite-engine (db.query).
- findOrphans signature: findOrphans(engine, { includePseudo }).
Engine is required. Uses engine.findOrphanPages() and
engine.getStats().page_count instead of raw SQL + global counts.
- queryOrphanPages signature: queryOrphanPages(engine). Delegates to
engine.findOrphanPages().
- src/commands/orphans.ts drops the `import * as db` — no more
global-state coupling.
- Callers updated: src/core/operations.ts find_orphans handler now
passes ctx.engine through; runOrphans CLI entry uses its engine arg.
- No signature change needed in cli.ts (it was already passing engine
via CLI_ONLY dispatch).
Tests (test/orphans.test.ts, new `findOrphans (engine-injected)`
describe block, PGLite in-memory, no DATABASE_URL required):
- links correctly scope orphans (alice links to bob -> bob not
an orphan; alice is)
- includePseudo:true surfaces _atlas-style pages
- queryOrphanPages delegates to passed engine
- empty brain returns {orphans: [], total_pages: 0} without crashing
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(cycle): add runCycle primitive in src/core/cycle.ts
The brain maintenance cycle as a single function. Six phases in
semantically-driven order (fix files → sync → extract → embed →
report orphans). Pure composition of existing library calls — no
execSync, no subprocess anti-patterns, no regex-parsed output.
┌───────────────────────────────────────────────────┐
│ runCycle(engine, opts) → CycleReport │
│ Phase 1: lint --fix (fs writes) │
│ Phase 2: backlinks --fix (fs writes) │
│ Phase 3: sync (DB picks up 1+2) │
│ Phase 4: extract (DB picks up links) │
│ Phase 5: embed --stale (DB writes) │
│ Phase 6: orphans (DB read, report) │
└───────────────────────────────────────────────────┘
Why the commit-4 primitive:
- CEO + Eng + Codex reviews all converged on "extract one cycle
function, wire both dream and autopilot through it." Two CLIs,
one definition of what the brain does overnight.
- Phase order was wrong in PR #309's original dream.ts (sync
before lint+backlinks lost the "fix files, then index them"
semantic).
- This commit is the bisectable foundation; commit 5 (dream)
and commit 6 (autopilot+jobs) just call into it.
Coordination — the codex-flagged blocker:
Session-scoped pg_try_advisory_lock does not survive PgBouncer
transaction pooling (the v0.15.4 fix made pooled connections the
default). Replaced with a DB lock table (gbrain_cycle_locks) that
works through every pooler:
- Acquire: INSERT ... ON CONFLICT DO UPDATE ... WHERE ttl < NOW()
- Refresh: UPDATE ttl_expires_at between phases via hook
- Release: DELETE in finally{}
- TTL: 30 min; crashed holders auto-release
PGLite / engine=null path uses a file lock at ~/.gbrain/cycle.lock
with PID liveness check. kill(pid, 0) with EPERM treated as alive
(so init/launchd-pid holders aren't mis-classified as stale).
Lock-skip: only phases that mutate state (lint, backlinks, sync,
extract, embed) trigger lock acquisition. orphans is read-only.
Single-phase --phase orphans runs never block on a held lock.
Engine-null mode preserved: filesystem phases run, DB phases skip
with {status:'skipped', reason:'no_database'}. Matches current
dream's capability that would have been lost if runCycle required
a connected engine.
Contract details:
- CycleReport has schema_version:"1" (stable, additive) so agents
consuming --json can rely on the shape
- status: 'ok' | 'clean' | 'partial' | 'skipped' | 'failed'.
'clean' = ran successfully with zero activity; agents trivially
detect a healthy brain.
- PhaseResult.error: { class, code, message, hint?, docs_url? }
(Stripe-API-tier structured failure info) when status='fail'
- yieldBetweenPhases hook: awaited between EVERY phase and before
return, runs even after phase failure, exceptions logged but
non-fatal. Required so the Minions autopilot-cycle handler can
renew its job lock between phases (prevents the v0.14 stall-death
regression codex flagged).
- git pull explicit: opts.pull defaults to false (cron-safe).
Autopilot daemon callers opt in if user configured it.
- extract phase doesn't have a dry-run mode in the underlying
library function, so runCycle honestly skips extract when
dryRun=true (status:'skipped', reason:'no_dry_run_support').
Schema migration v16: gbrain_cycle_locks table + idx_cycle_locks_ttl.
Also appended to src/schema.sql and src/core/pglite-schema.ts for
fresh installs. schema-embedded.ts regenerated via build:schema.
Tests (test/core/cycle.test.ts, PGLite in-memory + mocked library
functions, no DATABASE_URL required):
- dryRun × phases matrix: dryRun:true reaches lint/backlinks/sync/
embed; extract is honestly skipped
- Phase selection: default runs all 6 in order; --phase lint runs
only lint; --phase orphans runs only orphans
- Lock semantics: acquire + release on mutating phases, skip
entirely for read-only selections
- cycle_already_running: seeded live-holder lock → status:skipped,
zero phase runs; TTL-expired holder → auto-claimed
- Engine null: filesystem phases run, DB phases skip
- File lock (engine=null) blocks when PID 1 holds lock with fresh
mtime — exercises the PID liveness branch including EPERM
- Status derivation: 'ok' vs 'clean' vs 'partial' vs 'skipped'
- yieldBetweenPhases called N times, hook exceptions non-fatal
Next: commit 5 rewrites dream.ts as a thin CLI alias over runCycle,
commit 6 migrates autopilot daemon + jobs.ts handler to delegate to
runCycle too.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(dream): add gbrain dream CLI as a thin alias over runCycle
`gbrain dream` is the README brand-promise command: "the agent runs
while I sleep, the dream cycle ... I wake up and the brain is smarter."
Cron-friendly, JSON-reportable, phase-selectable. Same maintenance
cycle as `gbrain autopilot`, just scheduled differently — both
converge on runCycle (added in commit 4) so there's one source of
truth for what happens overnight.
Contract:
gbrain dream # full 6-phase cycle
gbrain dream --dry-run # preview, no writes
gbrain dream --json # CycleReport JSON (agent-readable)
gbrain dream --phase <name> # single-phase run
gbrain dream --pull # git pull before syncing
gbrain dream --dir /path/to/brain # explicit brain location
Cron: 0 2 * * * gbrain dream --json >> /var/log/gbrain-dream.log
Behavior details:
- Brain-dir resolution: requires explicit --dir OR sync.repo_path
in engine config. No more walk-up-cwd-for-.git footgun that
PR #309's original dream.ts had (would lint unrelated git repos).
- engine=null mode preserved via cli.ts's try/catch around
connectEngine — filesystem phases (lint, backlinks) still run
without a DB, DB phases report skipped/no_database in the output.
- status=clean prints "Brain is healthy. N phase(s) checked in Ns."
status=skipped prints the reason (cycle_already_running, etc.).
Partial/failed prints the phase-by-phase detail.
- Exit code 1 when status=failed (cron spots real problems).
'partial' is not a failure — warnings shouldn't page you.
- --help text cross-references `autopilot --install` for users
who want continuous maintenance as a daemon.
CLI registration (src/cli.ts):
- 'dream' added to CLI_ONLY
- handleCliOnly has a pre-engine branch mirroring doctor's pattern:
try connectEngine() → ok path; catch → runDream(null, args) so
filesystem phases still run when DB is down
- Help text updated with one-line dream entry and autopilot cross-ref
Tests (test/dream.test.ts, real PGLite + real library calls, no mocks
to avoid `mock.module` leakage across test files):
- brainDir resolution: explicit --dir wins, engine config fallback,
missing + nonexistent errors
- phase selection: --phase lint|orphans produces single-phase report
- phase validation: --phase garbage exits 1
- output: --json parses as CycleReport with schema_version:"1"
- human output mentions "Brain is healthy" on clean status
- dry-run: cycle runs but DB stays untouched
- exit code: clean/ok/partial do not call process.exit
Also (test/core/cycle.test.ts): refactored to use beforeAll/afterAll
with one shared PGLite engine per describe + truncateCycleLocks
between tests. Cuts test time from ~11s to ~4s; avoids the 15-migration
penalty per test that was causing parallel-suite timeout flakes.
Co-Authored-By: Wintermute <wintermute@garrytan.com>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat: v0.17.0 — autopilot + jobs delegate to runCycle (unifies the cycle)
Autopilot daemon (`--inline` path) and Minions `autopilot-cycle`
handler both now delegate to `runCycle` (introduced in commit 4).
Three callers, one cycle definition:
1. `gbrain dream` — one-shot cron cycle
2. `gbrain autopilot` daemon inline path — scheduled cycles
3. `autopilot-cycle` Minions handler — durable queue with retry
All three share:
- Same 6 phases in same order (lint → backlinks → sync → extract →
embed → orphans)
- Same DB lock table coordination (`gbrain_cycle_locks`)
- Same yieldBetweenPhases discipline (prevents v0.14 stall-death)
- Same structured CycleReport output
Autopilot inline path gains lint + orphan sweep that the old path
skipped. Minions autopilot-cycle handler also gains lint + orphans.
Users who run `gbrain autopilot --install` see 6-phase reports in
`gbrain jobs get <id>` starting on next interval. No config change
required.
Changes:
- `src/commands/autopilot.ts`: inline fallback path (~20 lines)
replaces the ~22-line sync+extract+embed sequence with a single
runCycle call. Uses pull:true (matches pre-v0.17 autopilot
behavior). Uses setImmediate yield hook. Status/failure reporting
derives from CycleReport.status. `--help` cross-references `gbrain
dream` for one-shot use.
- `src/commands/jobs.ts:579` (`autopilot-cycle` handler): replaces
the 4-step try/catch sequence with a runCycle call. Returns
`{ partial, status, report }` so `gbrain jobs get <id>` shows the
full structured CycleReport. Preserves partial-failure semantic
(one phase failing does NOT throw; next cycle still runs).
yieldBetweenPhases yields the event loop between phases for the
worker's lock-renewal timer.
Release scaffolding:
- VERSION: 0.16.0 → 0.17.0
- CHANGELOG.md: v0.17.0 entry in GStack voice — headline, numbers
table, "what this means" paragraph, "To take advantage" block
per CLAUDE.md post-ship rules. Itemized changes below the fold.
Credit to @Wintermute for the original PR #309 thesis.
- skills/migrations/v0.17.0.md: documents what changed for
upgrading users. No mechanical action required — schema migration
v16 (cycle locks table) + handler delegation both apply
automatically. Includes opt-out paths for users who don't want
their daemon modifying files (use `dream --phase orphans` in cron
and skip autopilot-install, or other explicit configs).
- CLAUDE.md: new entries for `src/core/cycle.ts` and
`src/commands/dream.ts` with contract details.
Tests: no new test file needed for this commit — the cycle primitive
is extensively tested in test/core/cycle.test.ts (18 cases), dream
in test/dream.test.ts (11), and autopilot's delegation is mechanical
(calls runCycle with specific opts). The handler contract is covered
implicitly: if runCycle returns a CycleReport, the handler wraps it
in `{ partial, status, report }` — nothing else to assert.
Verified:
- `bun test test/autopilot-install.test.ts test/autopilot-resolve-cli.test.ts test/core/cycle.test.ts test/dream.test.ts` → 37 pass, 0 fail
Completes the v0.17.0 feature: 6 bisectable commits on one branch
(garrytan/v0.17-dream-cycle), ready to push as one PR.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* test(e2e): add runCycle + dream E2E coverage against real Postgres
Gap from the v0.17 commit series: PR #321 shipped unit-level tests
for runCycle (test/core/cycle.test.ts) and dream (test/dream.test.ts)
but no E2E coverage that exercises the real Postgres paths. Filling
that in before merge.
test/e2e/cycle.test.ts (6 cases):
- schema migration v16 created gbrain_cycle_locks + index
- dry-run full cycle: zero DB writes + lock table empty after
- live cycle: pages + chunks materialize, sync.last_commit set
- concurrent cycle blocked by lock → status:'skipped'
- TTL-expired lock auto-claimed (crashed-holder recovery)
- --phase orphans skips lock entirely (read-only optimization)
test/e2e/dream.test.ts (3 cases):
- dream --dry-run --json emits valid CycleReport + DB stays empty
- dream (no --dry-run) syncs pages into real DB
- dream --phase orphans doesn't touch the cycle-lock table
Both files mock embedBatch via mock.module so the embed phase never
calls OpenAI even when the full 6-phase cycle runs (zero API cost,
zero flakiness from network calls).
Verified locally:
- `docker run pgvector/pgvector:pg16` on port 5434
- `DATABASE_URL=... bun test test/e2e/cycle.test.ts test/e2e/dream.test.ts` → 9 pass, 0 fail
- Full E2E suite (`bun run test:e2e`): 16 files, 150 tests, 0 fail
- Container torn down after: `docker stop + rm gbrain-test-pg`
Per CLAUDE.md E2E test DB lifecycle. These tests skip gracefully when
DATABASE_URL isn't set (via hasDatabase() helper + describe.skip).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-authored-by: Wintermute <wintermute@garrytan.com>
473 lines
21 KiB
PL/PgSQL
473 lines
21 KiB
PL/PgSQL
-- GBrain Postgres + pgvector schema
|
|
|
|
CREATE EXTENSION IF NOT EXISTS vector;
|
|
CREATE EXTENSION IF NOT EXISTS pg_trgm;
|
|
-- gen_random_uuid() is core in Postgres 13+; enable pgcrypto as fallback for older versions
|
|
CREATE EXTENSION IF NOT EXISTS pgcrypto;
|
|
|
|
-- ============================================================
|
|
-- pages: the core content table
|
|
-- ============================================================
|
|
CREATE TABLE IF NOT EXISTS pages (
|
|
id SERIAL PRIMARY KEY,
|
|
slug TEXT NOT NULL UNIQUE,
|
|
type TEXT NOT NULL,
|
|
title TEXT NOT NULL,
|
|
compiled_truth TEXT NOT NULL DEFAULT '',
|
|
timeline TEXT NOT NULL DEFAULT '',
|
|
frontmatter JSONB NOT NULL DEFAULT '{}',
|
|
content_hash TEXT,
|
|
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
|
|
updated_at TIMESTAMPTZ NOT NULL DEFAULT now()
|
|
);
|
|
|
|
CREATE INDEX IF NOT EXISTS idx_pages_type ON pages(type);
|
|
CREATE INDEX IF NOT EXISTS idx_pages_frontmatter ON pages USING GIN(frontmatter);
|
|
CREATE INDEX IF NOT EXISTS idx_pages_trgm ON pages USING GIN(title gin_trgm_ops);
|
|
-- v0.13.1 #170: avoids 14.6s seqscan on large brains when listing pages newest-first.
|
|
CREATE INDEX IF NOT EXISTS idx_pages_updated_at_desc ON pages (updated_at DESC);
|
|
|
|
-- ============================================================
|
|
-- content_chunks: chunked content with embeddings
|
|
-- ============================================================
|
|
CREATE TABLE IF NOT EXISTS content_chunks (
|
|
id SERIAL PRIMARY KEY,
|
|
page_id INTEGER NOT NULL REFERENCES pages(id) ON DELETE CASCADE,
|
|
chunk_index INTEGER NOT NULL,
|
|
chunk_text TEXT NOT NULL,
|
|
chunk_source TEXT NOT NULL DEFAULT 'compiled_truth',
|
|
embedding vector(1536),
|
|
model TEXT NOT NULL DEFAULT 'text-embedding-3-large',
|
|
token_count INTEGER,
|
|
embedded_at TIMESTAMPTZ,
|
|
created_at TIMESTAMPTZ NOT NULL DEFAULT now()
|
|
);
|
|
|
|
CREATE UNIQUE INDEX IF NOT EXISTS idx_chunks_page_index ON content_chunks(page_id, chunk_index);
|
|
CREATE INDEX IF NOT EXISTS idx_chunks_page ON content_chunks(page_id);
|
|
CREATE INDEX IF NOT EXISTS idx_chunks_embedding ON content_chunks USING hnsw (embedding vector_cosine_ops);
|
|
|
|
-- ============================================================
|
|
-- links: cross-references between pages
|
|
-- ============================================================
|
|
-- Provenance model (v0.13):
|
|
-- link_source — 'markdown' | 'frontmatter' | 'manual' | NULL
|
|
-- (NULL = legacy row written before v0.13; unknown source)
|
|
-- origin_page_id — for link_source='frontmatter', the page whose YAML
|
|
-- frontmatter created this edge; scopes reconciliation
|
|
-- origin_field — the frontmatter field name (e.g. 'key_people')
|
|
--
|
|
-- The unique constraint includes link_source + origin_page_id so a manual edge
|
|
-- and a frontmatter-derived edge with the same (from, to, type) tuple coexist.
|
|
-- Reconciliation on put_page filters by (link_source='frontmatter' AND
|
|
-- origin_page_id = written_page) — never touches other pages' edges.
|
|
CREATE TABLE IF NOT EXISTS links (
|
|
id SERIAL PRIMARY KEY,
|
|
from_page_id INTEGER NOT NULL REFERENCES pages(id) ON DELETE CASCADE,
|
|
to_page_id INTEGER NOT NULL REFERENCES pages(id) ON DELETE CASCADE,
|
|
link_type TEXT NOT NULL DEFAULT '',
|
|
context TEXT NOT NULL DEFAULT '',
|
|
link_source TEXT CHECK (link_source IS NULL OR link_source IN ('markdown', 'frontmatter', 'manual')),
|
|
origin_page_id INTEGER REFERENCES pages(id) ON DELETE SET NULL,
|
|
origin_field TEXT,
|
|
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
|
|
-- NULLS NOT DISTINCT (PG15+) so two rows with link_source IS NULL or
|
|
-- origin_page_id IS NULL collide as expected. Without this, every row with
|
|
-- NULL origin_page_id (markdown/manual edges) would be treated as unique.
|
|
CONSTRAINT links_from_to_type_source_origin_unique
|
|
UNIQUE NULLS NOT DISTINCT (from_page_id, to_page_id, link_type, link_source, origin_page_id)
|
|
);
|
|
|
|
CREATE INDEX IF NOT EXISTS idx_links_from ON links(from_page_id);
|
|
CREATE INDEX IF NOT EXISTS idx_links_to ON links(to_page_id);
|
|
CREATE INDEX IF NOT EXISTS idx_links_source ON links(link_source);
|
|
CREATE INDEX IF NOT EXISTS idx_links_origin ON links(origin_page_id);
|
|
|
|
-- ============================================================
|
|
-- tags
|
|
-- ============================================================
|
|
CREATE TABLE IF NOT EXISTS tags (
|
|
id SERIAL PRIMARY KEY,
|
|
page_id INTEGER NOT NULL REFERENCES pages(id) ON DELETE CASCADE,
|
|
tag TEXT NOT NULL,
|
|
UNIQUE(page_id, tag)
|
|
);
|
|
|
|
CREATE INDEX IF NOT EXISTS idx_tags_tag ON tags(tag);
|
|
CREATE INDEX IF NOT EXISTS idx_tags_page_id ON tags(page_id);
|
|
|
|
-- ============================================================
|
|
-- raw_data: sidecar data (replaces .raw/ JSON files)
|
|
-- ============================================================
|
|
CREATE TABLE IF NOT EXISTS raw_data (
|
|
id SERIAL PRIMARY KEY,
|
|
page_id INTEGER NOT NULL REFERENCES pages(id) ON DELETE CASCADE,
|
|
source TEXT NOT NULL,
|
|
data JSONB NOT NULL,
|
|
fetched_at TIMESTAMPTZ NOT NULL DEFAULT now(),
|
|
UNIQUE(page_id, source)
|
|
);
|
|
|
|
CREATE INDEX IF NOT EXISTS idx_raw_data_page ON raw_data(page_id);
|
|
|
|
-- ============================================================
|
|
-- timeline_entries: structured timeline
|
|
-- ============================================================
|
|
CREATE TABLE IF NOT EXISTS timeline_entries (
|
|
id SERIAL PRIMARY KEY,
|
|
page_id INTEGER NOT NULL REFERENCES pages(id) ON DELETE CASCADE,
|
|
date DATE NOT NULL,
|
|
source TEXT NOT NULL DEFAULT '',
|
|
summary TEXT NOT NULL,
|
|
detail TEXT NOT NULL DEFAULT '',
|
|
created_at TIMESTAMPTZ NOT NULL DEFAULT now()
|
|
);
|
|
|
|
CREATE INDEX IF NOT EXISTS idx_timeline_page ON timeline_entries(page_id);
|
|
CREATE INDEX IF NOT EXISTS idx_timeline_date ON timeline_entries(date);
|
|
-- Dedup constraint: same (page, date, summary) treated as same event
|
|
CREATE UNIQUE INDEX IF NOT EXISTS idx_timeline_dedup ON timeline_entries(page_id, date, summary);
|
|
|
|
-- ============================================================
|
|
-- page_versions: snapshot history for compiled_truth
|
|
-- ============================================================
|
|
CREATE TABLE IF NOT EXISTS page_versions (
|
|
id SERIAL PRIMARY KEY,
|
|
page_id INTEGER NOT NULL REFERENCES pages(id) ON DELETE CASCADE,
|
|
compiled_truth TEXT NOT NULL,
|
|
frontmatter JSONB NOT NULL DEFAULT '{}',
|
|
snapshot_at TIMESTAMPTZ NOT NULL DEFAULT now()
|
|
);
|
|
|
|
CREATE INDEX IF NOT EXISTS idx_versions_page ON page_versions(page_id);
|
|
|
|
-- ============================================================
|
|
-- ingest_log
|
|
-- ============================================================
|
|
CREATE TABLE IF NOT EXISTS ingest_log (
|
|
id SERIAL PRIMARY KEY,
|
|
source_type TEXT NOT NULL,
|
|
source_ref TEXT NOT NULL,
|
|
pages_updated JSONB NOT NULL DEFAULT '[]',
|
|
summary TEXT NOT NULL DEFAULT '',
|
|
created_at TIMESTAMPTZ NOT NULL DEFAULT now()
|
|
);
|
|
|
|
-- ============================================================
|
|
-- config: brain-level settings
|
|
-- ============================================================
|
|
CREATE TABLE IF NOT EXISTS config (
|
|
key TEXT PRIMARY KEY,
|
|
value TEXT NOT NULL
|
|
);
|
|
|
|
INSERT INTO config (key, value) VALUES
|
|
('version', '1'),
|
|
('embedding_model', 'text-embedding-3-large'),
|
|
('embedding_dimensions', '1536'),
|
|
('chunk_strategy', 'semantic')
|
|
ON CONFLICT (key) DO NOTHING;
|
|
|
|
-- ============================================================
|
|
-- access_tokens: bearer tokens for remote MCP access
|
|
-- ============================================================
|
|
CREATE TABLE IF NOT EXISTS access_tokens (
|
|
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
|
|
name TEXT NOT NULL,
|
|
token_hash TEXT NOT NULL UNIQUE,
|
|
scopes TEXT[],
|
|
created_at TIMESTAMPTZ DEFAULT now(),
|
|
last_used_at TIMESTAMPTZ,
|
|
revoked_at TIMESTAMPTZ
|
|
);
|
|
|
|
CREATE INDEX IF NOT EXISTS idx_access_tokens_hash ON access_tokens (token_hash) WHERE revoked_at IS NULL;
|
|
|
|
-- ============================================================
|
|
-- mcp_request_log: usage logging for remote MCP requests
|
|
-- ============================================================
|
|
CREATE TABLE IF NOT EXISTS mcp_request_log (
|
|
id SERIAL PRIMARY KEY,
|
|
token_name TEXT,
|
|
operation TEXT NOT NULL,
|
|
latency_ms INTEGER,
|
|
status TEXT NOT NULL DEFAULT 'success',
|
|
created_at TIMESTAMPTZ NOT NULL DEFAULT now()
|
|
);
|
|
|
|
-- ============================================================
|
|
-- files: binary attachments stored in Supabase Storage
|
|
-- ============================================================
|
|
CREATE TABLE IF NOT EXISTS files (
|
|
id SERIAL PRIMARY KEY,
|
|
page_slug TEXT REFERENCES pages(slug) ON DELETE SET NULL ON UPDATE CASCADE,
|
|
filename TEXT NOT NULL,
|
|
storage_path TEXT NOT NULL,
|
|
mime_type TEXT,
|
|
size_bytes BIGINT,
|
|
content_hash TEXT NOT NULL,
|
|
metadata JSONB NOT NULL DEFAULT '{}',
|
|
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
|
|
UNIQUE(storage_path)
|
|
);
|
|
|
|
-- Migration: drop storage_url if it exists (renamed to storage_path only)
|
|
ALTER TABLE files DROP COLUMN IF EXISTS storage_url;
|
|
|
|
CREATE INDEX IF NOT EXISTS idx_files_page ON files(page_slug);
|
|
CREATE INDEX IF NOT EXISTS idx_files_hash ON files(content_hash);
|
|
|
|
-- ============================================================
|
|
-- Trigger-based search_vector (spans pages + timeline_entries)
|
|
-- ============================================================
|
|
ALTER TABLE pages ADD COLUMN IF NOT EXISTS search_vector tsvector;
|
|
|
|
CREATE INDEX IF NOT EXISTS idx_pages_search ON pages USING GIN(search_vector);
|
|
|
|
-- Function to rebuild search_vector for a page
|
|
CREATE OR REPLACE FUNCTION update_page_search_vector() RETURNS trigger AS $$
|
|
DECLARE
|
|
timeline_text TEXT;
|
|
BEGIN
|
|
-- Gather timeline_entries text for this page
|
|
SELECT coalesce(string_agg(summary || ' ' || detail, ' '), '')
|
|
INTO timeline_text
|
|
FROM timeline_entries
|
|
WHERE page_id = NEW.id;
|
|
|
|
-- Build weighted tsvector
|
|
NEW.search_vector :=
|
|
setweight(to_tsvector('english', coalesce(NEW.title, '')), 'A') ||
|
|
setweight(to_tsvector('english', coalesce(NEW.compiled_truth, '')), 'B') ||
|
|
setweight(to_tsvector('english', coalesce(NEW.timeline, '')), 'C') ||
|
|
setweight(to_tsvector('english', coalesce(timeline_text, '')), 'C');
|
|
|
|
RETURN NEW;
|
|
END;
|
|
$$ LANGUAGE plpgsql;
|
|
|
|
DROP TRIGGER IF EXISTS trg_pages_search_vector ON pages;
|
|
CREATE TRIGGER trg_pages_search_vector
|
|
BEFORE INSERT OR UPDATE ON pages
|
|
FOR EACH ROW
|
|
EXECUTE FUNCTION update_page_search_vector();
|
|
|
|
-- Note: timeline_entries trigger removed (v0.10.1).
|
|
-- Structured timeline_entries power temporal queries (graph layer).
|
|
-- The markdown timeline section in pages.timeline still feeds search_vector via
|
|
-- the trg_pages_search_vector trigger above. Removing the timeline_entries
|
|
-- trigger avoids double-weighting the same content in search and prevents
|
|
-- mutation-induced reordering during timeline-extract pagination.
|
|
DROP TRIGGER IF EXISTS trg_timeline_search_vector ON timeline_entries;
|
|
DROP FUNCTION IF EXISTS update_page_search_vector_from_timeline();
|
|
|
|
-- ============================================================
|
|
-- Minion Jobs: BullMQ-inspired Postgres-native job queue
|
|
-- ============================================================
|
|
CREATE TABLE IF NOT EXISTS minion_jobs (
|
|
id SERIAL PRIMARY KEY,
|
|
name TEXT NOT NULL,
|
|
queue TEXT NOT NULL DEFAULT 'default',
|
|
status TEXT NOT NULL DEFAULT 'waiting',
|
|
priority INTEGER NOT NULL DEFAULT 0,
|
|
data JSONB NOT NULL DEFAULT '{}',
|
|
max_attempts INTEGER NOT NULL DEFAULT 3,
|
|
attempts_made INTEGER NOT NULL DEFAULT 0,
|
|
attempts_started INTEGER NOT NULL DEFAULT 0,
|
|
backoff_type TEXT NOT NULL DEFAULT 'exponential',
|
|
backoff_delay INTEGER NOT NULL DEFAULT 1000,
|
|
backoff_jitter REAL NOT NULL DEFAULT 0.2,
|
|
stalled_counter INTEGER NOT NULL DEFAULT 0,
|
|
max_stalled INTEGER NOT NULL DEFAULT 5,
|
|
lock_token TEXT,
|
|
lock_until TIMESTAMPTZ,
|
|
delay_until TIMESTAMPTZ,
|
|
parent_job_id INTEGER REFERENCES minion_jobs(id) ON DELETE SET NULL,
|
|
on_child_fail TEXT NOT NULL DEFAULT 'fail_parent',
|
|
tokens_input INTEGER NOT NULL DEFAULT 0,
|
|
tokens_output INTEGER NOT NULL DEFAULT 0,
|
|
tokens_cache_read INTEGER NOT NULL DEFAULT 0,
|
|
result JSONB,
|
|
progress JSONB,
|
|
error_text TEXT,
|
|
stacktrace JSONB DEFAULT '[]',
|
|
depth INTEGER NOT NULL DEFAULT 0,
|
|
max_children INTEGER,
|
|
timeout_ms INTEGER,
|
|
timeout_at TIMESTAMPTZ,
|
|
remove_on_complete BOOLEAN NOT NULL DEFAULT FALSE,
|
|
remove_on_fail BOOLEAN NOT NULL DEFAULT FALSE,
|
|
idempotency_key TEXT,
|
|
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
|
|
started_at TIMESTAMPTZ,
|
|
finished_at TIMESTAMPTZ,
|
|
updated_at TIMESTAMPTZ NOT NULL DEFAULT now(),
|
|
CONSTRAINT chk_status CHECK (status IN ('waiting','active','completed','failed','delayed','dead','cancelled','waiting-children','paused')),
|
|
CONSTRAINT chk_backoff_type CHECK (backoff_type IN ('fixed','exponential')),
|
|
CONSTRAINT chk_on_child_fail CHECK (on_child_fail IN ('fail_parent','remove_dep','ignore','continue')),
|
|
CONSTRAINT chk_jitter_range CHECK (backoff_jitter >= 0.0 AND backoff_jitter <= 1.0),
|
|
CONSTRAINT chk_attempts_order CHECK (attempts_made <= attempts_started),
|
|
CONSTRAINT chk_nonnegative CHECK (attempts_made >= 0 AND attempts_started >= 0 AND stalled_counter >= 0 AND max_attempts >= 1 AND max_stalled >= 0),
|
|
CONSTRAINT chk_depth_nonnegative CHECK (depth >= 0),
|
|
CONSTRAINT chk_max_children_positive CHECK (max_children IS NULL OR max_children > 0),
|
|
CONSTRAINT chk_timeout_positive CHECK (timeout_ms IS NULL OR timeout_ms > 0)
|
|
);
|
|
|
|
CREATE INDEX IF NOT EXISTS idx_minion_jobs_claim ON minion_jobs (queue, priority ASC, created_at ASC) WHERE status = 'waiting';
|
|
CREATE INDEX IF NOT EXISTS idx_minion_jobs_status ON minion_jobs(status);
|
|
CREATE INDEX IF NOT EXISTS idx_minion_jobs_stalled ON minion_jobs (lock_until) WHERE status = 'active';
|
|
CREATE INDEX IF NOT EXISTS idx_minion_jobs_delayed ON minion_jobs (delay_until) WHERE status = 'delayed';
|
|
CREATE INDEX IF NOT EXISTS idx_minion_jobs_parent ON minion_jobs(parent_job_id);
|
|
CREATE INDEX IF NOT EXISTS idx_minion_jobs_timeout ON minion_jobs (timeout_at) WHERE status = 'active' AND timeout_at IS NOT NULL;
|
|
CREATE INDEX IF NOT EXISTS idx_minion_jobs_parent_status ON minion_jobs (parent_job_id, status) WHERE parent_job_id IS NOT NULL;
|
|
CREATE UNIQUE INDEX IF NOT EXISTS uniq_minion_jobs_idempotency ON minion_jobs (idempotency_key) WHERE idempotency_key IS NOT NULL;
|
|
|
|
-- Inbox table for sidechannel messaging
|
|
CREATE TABLE IF NOT EXISTS minion_inbox (
|
|
id SERIAL PRIMARY KEY,
|
|
job_id INTEGER NOT NULL REFERENCES minion_jobs(id) ON DELETE CASCADE,
|
|
sender TEXT NOT NULL,
|
|
payload JSONB NOT NULL,
|
|
sent_at TIMESTAMPTZ NOT NULL DEFAULT now(),
|
|
read_at TIMESTAMPTZ
|
|
);
|
|
CREATE INDEX IF NOT EXISTS idx_minion_inbox_unread ON minion_inbox (job_id) WHERE read_at IS NULL;
|
|
CREATE INDEX IF NOT EXISTS idx_minion_inbox_child_done ON minion_inbox (job_id, sent_at) WHERE payload->>'type' = 'child_done';
|
|
|
|
-- Attachments table: per-job binary blobs (manifests, agent outputs, files)
|
|
CREATE TABLE IF NOT EXISTS minion_attachments (
|
|
id SERIAL PRIMARY KEY,
|
|
job_id INTEGER NOT NULL REFERENCES minion_jobs(id) ON DELETE CASCADE,
|
|
filename TEXT NOT NULL,
|
|
content_type TEXT NOT NULL,
|
|
content BYTEA,
|
|
storage_uri TEXT,
|
|
size_bytes INTEGER NOT NULL,
|
|
sha256 TEXT NOT NULL,
|
|
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
|
|
CONSTRAINT uniq_minion_attachments_job_filename UNIQUE (job_id, filename),
|
|
CONSTRAINT chk_attachment_storage CHECK (content IS NOT NULL OR storage_uri IS NOT NULL),
|
|
CONSTRAINT chk_attachment_size CHECK (size_bytes >= 0)
|
|
);
|
|
CREATE INDEX IF NOT EXISTS idx_minion_attachments_job ON minion_attachments (job_id);
|
|
ALTER TABLE minion_attachments ALTER COLUMN content SET STORAGE EXTERNAL;
|
|
|
|
-- ============================================================
|
|
-- Subagent runtime (v0.16.0) — durable LLM loops
|
|
-- ============================================================
|
|
-- Anthropic-native message blocks, one row per Messages API message. Parallel
|
|
-- tool_use blocks in one assistant message live in content_blocks JSONB,
|
|
-- not across rows.
|
|
CREATE TABLE IF NOT EXISTS subagent_messages (
|
|
id BIGSERIAL PRIMARY KEY,
|
|
job_id BIGINT NOT NULL REFERENCES minion_jobs(id) ON DELETE CASCADE,
|
|
message_idx INTEGER NOT NULL,
|
|
role TEXT NOT NULL,
|
|
content_blocks JSONB NOT NULL,
|
|
tokens_in INTEGER,
|
|
tokens_out INTEGER,
|
|
tokens_cache_read INTEGER,
|
|
tokens_cache_create INTEGER,
|
|
model TEXT,
|
|
ended_at TIMESTAMPTZ NOT NULL DEFAULT now(),
|
|
CONSTRAINT uniq_subagent_messages_idx UNIQUE (job_id, message_idx),
|
|
CONSTRAINT chk_subagent_messages_role CHECK (role IN ('user','assistant'))
|
|
);
|
|
CREATE INDEX IF NOT EXISTS idx_subagent_messages_job ON subagent_messages (job_id, message_idx);
|
|
|
|
-- Two-phase tool execution ledger. Before tool call: INSERT status='pending'.
|
|
-- After success: UPDATE to 'complete' + output. On failure: 'failed' + error.
|
|
-- Replay re-runs 'pending' rows only if the tool is idempotent.
|
|
CREATE TABLE IF NOT EXISTS subagent_tool_executions (
|
|
id BIGSERIAL PRIMARY KEY,
|
|
job_id BIGINT NOT NULL REFERENCES minion_jobs(id) ON DELETE CASCADE,
|
|
message_idx INTEGER NOT NULL,
|
|
tool_use_id TEXT NOT NULL,
|
|
tool_name TEXT NOT NULL,
|
|
input JSONB NOT NULL,
|
|
status TEXT NOT NULL,
|
|
output JSONB,
|
|
error TEXT,
|
|
started_at TIMESTAMPTZ NOT NULL DEFAULT now(),
|
|
ended_at TIMESTAMPTZ,
|
|
CONSTRAINT uniq_subagent_tools_use_id UNIQUE (job_id, tool_use_id),
|
|
CONSTRAINT chk_subagent_tools_status CHECK (status IN ('pending','complete','failed'))
|
|
);
|
|
CREATE INDEX IF NOT EXISTS idx_subagent_tools_job ON subagent_tool_executions (job_id, status);
|
|
|
|
-- Rate-lease table — concurrency cap on outbound providers (e.g.
|
|
-- anthropic:messages). Acquire: INSERT if active < max_concurrent under
|
|
-- advisory lock. Release: DELETE. Stale leases (expires_at past) auto-prune
|
|
-- on next acquire so crashed workers can't strand capacity.
|
|
CREATE TABLE IF NOT EXISTS subagent_rate_leases (
|
|
id BIGSERIAL PRIMARY KEY,
|
|
key TEXT NOT NULL,
|
|
owner_job_id BIGINT NOT NULL REFERENCES minion_jobs(id) ON DELETE CASCADE,
|
|
acquired_at TIMESTAMPTZ NOT NULL DEFAULT now(),
|
|
expires_at TIMESTAMPTZ NOT NULL
|
|
);
|
|
CREATE INDEX IF NOT EXISTS idx_rate_leases_key_expires ON subagent_rate_leases (key, expires_at);
|
|
|
|
-- ============================================================
|
|
-- Cycle coordination lock — v0.17 runCycle primitive
|
|
-- ============================================================
|
|
-- One row per active cycle. Any caller (autopilot daemon, Minions
|
|
-- autopilot-cycle handler, gbrain dream CLI) tries to acquire this
|
|
-- row before running a DB-write phase. Holders refresh ttl_expires_at
|
|
-- between phases; crashed holders auto-release once TTL expires.
|
|
-- Works through PgBouncer transaction pooling, unlike session-scoped
|
|
-- pg_try_advisory_lock.
|
|
CREATE TABLE IF NOT EXISTS gbrain_cycle_locks (
|
|
id TEXT PRIMARY KEY,
|
|
holder_pid INT NOT NULL,
|
|
holder_host TEXT,
|
|
acquired_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
|
|
ttl_expires_at TIMESTAMPTZ NOT NULL
|
|
);
|
|
CREATE INDEX IF NOT EXISTS idx_cycle_locks_ttl ON gbrain_cycle_locks(ttl_expires_at);
|
|
|
|
-- NOTIFY trigger for real-time job events (Postgres only, not PGLite)
|
|
CREATE OR REPLACE FUNCTION notify_minion_job_change() RETURNS trigger AS $$
|
|
BEGIN
|
|
PERFORM pg_notify('minion_jobs', json_build_object(
|
|
'id', NEW.id, 'status', NEW.status, 'name', NEW.name,
|
|
'queue', NEW.queue, 'prev_status', COALESCE(OLD.status, 'new')
|
|
)::text);
|
|
RETURN NEW;
|
|
END;
|
|
$$ LANGUAGE plpgsql;
|
|
|
|
DROP TRIGGER IF EXISTS minion_job_notify ON minion_jobs;
|
|
CREATE TRIGGER minion_job_notify AFTER INSERT OR UPDATE OF status ON minion_jobs
|
|
FOR EACH ROW EXECUTE FUNCTION notify_minion_job_change();
|
|
|
|
-- ============================================================
|
|
-- Row Level Security: block anon access, postgres role bypasses
|
|
-- ============================================================
|
|
-- The postgres role (used by gbrain via pooler) has BYPASSRLS.
|
|
-- Enabling RLS with no policies means the anon key can't read anything.
|
|
-- Only enable if the current role actually has BYPASSRLS privilege,
|
|
-- otherwise we'd lock ourselves out.
|
|
DO $$
|
|
DECLARE
|
|
has_bypass BOOLEAN;
|
|
BEGIN
|
|
SELECT rolbypassrls INTO has_bypass FROM pg_roles WHERE rolname = current_user;
|
|
IF has_bypass THEN
|
|
ALTER TABLE pages ENABLE ROW LEVEL SECURITY;
|
|
ALTER TABLE content_chunks ENABLE ROW LEVEL SECURITY;
|
|
ALTER TABLE links ENABLE ROW LEVEL SECURITY;
|
|
ALTER TABLE tags ENABLE ROW LEVEL SECURITY;
|
|
ALTER TABLE raw_data ENABLE ROW LEVEL SECURITY;
|
|
ALTER TABLE timeline_entries ENABLE ROW LEVEL SECURITY;
|
|
ALTER TABLE page_versions ENABLE ROW LEVEL SECURITY;
|
|
ALTER TABLE ingest_log ENABLE ROW LEVEL SECURITY;
|
|
ALTER TABLE config ENABLE ROW LEVEL SECURITY;
|
|
ALTER TABLE files ENABLE ROW LEVEL SECURITY;
|
|
ALTER TABLE minion_jobs ENABLE ROW LEVEL SECURITY;
|
|
RAISE NOTICE 'RLS enabled on all tables (role % has BYPASSRLS)', current_user;
|
|
ELSE
|
|
RAISE WARNING 'Skipping RLS: role % does not have BYPASSRLS privilege. Run as postgres role to enable.', current_user;
|
|
END IF;
|
|
END $$;
|