Files
gbrain/src/core/cycle.ts
T
55ca4984b2 feat: v0.17.0 — gbrain dream + runCycle primitive (one cycle, two CLIs) (#321)
* fix(sync): honor --dry-run in full-sync path + expose embedded count

Precondition for v0.17 brain maintenance cycle (runCycle primitive).

The full-sync path (performFullSync) previously called runImport() even
when opts.dryRun was true, silently writing to the DB and advancing
sync.last_commit. `gbrain sync --dry-run` on a fresh brain (or with
--full) would mutate state without warning.

Fix:
  - performFullSync now early-returns a `dry_run` SyncResult when
    opts.dryRun is set. Walks the repo via collectMarkdownFiles +
    isSyncable to count what WOULD be imported. No writes, no git
    state advance.
  - SyncResult gains an `embedded: number` field (required). Tracks
    pages re-embedded during the sync's auto-embed step. Existing
    return sites set 0; the synced + first_sync paths set real counts
    (best-estimate until commit 2 sharpens runEmbedCore's return type).
  - first_sync path now returns real added + chunksCreated counts
    from runImport instead of hardcoded zeros.
  - printSyncResult shows embedded count in human output.

Tests (test/sync.test.ts, new `performSync dry-run never writes`
block, PGLite + temp git repo, no DATABASE_URL required):
  - first-sync --dry-run: no pages, no sync.last_commit
  - incremental --dry-run after real sync: bookmark unchanged
  - --full --dry-run: no reimport, bookmark unchanged
  - SyncResult.embedded is a number

Codex outside-voice caught this. Would have shipped silent DB writes
on dry-run for anyone using `gbrain sync --dry-run --full`.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(embed): add dry-run mode + return EmbedResult with counts

Precondition for v0.17 brain maintenance cycle (runCycle primitive).

runEmbedCore previously returned Promise<void> and had no dry-run mode.
That made it impossible for runCycle to (a) report accurate embedded
counts or (b) honor --dry-run without also skipping the entire embed
phase (which would have required runCycle to know embed's internal
semantics — a layering violation).

Changes:
  - EmbedOpts gains `dryRun?: boolean`. When set, embedPage and
    embedAll enumerate stale chunks (or would-be-created chunks for
    unchunked pages, via local chunkText without engine.upsertChunks)
    but never call embedBatch and never write to the engine.
  - runEmbedCore: Promise<void> -> Promise<EmbedResult>. Result shape:
    { embedded, skipped, would_embed, total_chunks, pages_processed,
      dryRun }.
    embedded = chunks newly embedded (0 in dryRun).
    would_embed = chunks that WOULD be embedded (0 in non-dryRun).
    skipped = chunks with pre-existing embeddings.
  - runEmbed CLI wrapper honors --dry-run flag and returns the result
    through. `gbrain embed --stale --dry-run` is now a safe preview.
  - Callers ignoring the return value (sync auto-embed, autopilot
    inline fallback, jobs.ts handlers, CLI) keep compiling — the new
    return type is additive for `await` callers.

Tests (test/embed.test.ts, new `runEmbedCore --dry-run` block, uses
the existing mock.module embedBatch pattern, no API key required):
  - dry-run --all: zero embedBatch calls, zero upsertChunks calls,
    would_embed matches stale chunk total
  - dry-run --stale correctly splits stale vs already-embedded counts
  - dry-run --slugs on a single page tallies per-chunk counts
  - non-dry-run regression guard: embedded count matches across
    concurrent workers

Codex outside-voice flagged the Promise<void> return as a blocker for
accurate CycleReport.totals.pages_embedded.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* refactor(orphans): engine-injected queries, drop db.getConnection() global

Precondition for v0.17 brain maintenance cycle (runCycle primitive).

findOrphans + queryOrphanPages previously reached into the postgres-js
singleton via db.getConnection(), which (a) didn't compose with
runCycle's explicit-engine contract and (b) was wrong for PGLite test
fixtures and for any caller not using the default global connection.
Codex outside-voice flagged this as a blocker.

Changes:
  - BrainEngine interface gains findOrphanPages() — returns pages with
    no inbound links via the same NOT EXISTS anti-join. Implemented on
    both postgres-engine (sql tag) and pglite-engine (db.query).
  - findOrphans signature: findOrphans(engine, { includePseudo }).
    Engine is required. Uses engine.findOrphanPages() and
    engine.getStats().page_count instead of raw SQL + global counts.
  - queryOrphanPages signature: queryOrphanPages(engine). Delegates to
    engine.findOrphanPages().
  - src/commands/orphans.ts drops the `import * as db` — no more
    global-state coupling.
  - Callers updated: src/core/operations.ts find_orphans handler now
    passes ctx.engine through; runOrphans CLI entry uses its engine arg.
  - No signature change needed in cli.ts (it was already passing engine
    via CLI_ONLY dispatch).

Tests (test/orphans.test.ts, new `findOrphans (engine-injected)`
describe block, PGLite in-memory, no DATABASE_URL required):
  - links correctly scope orphans (alice links to bob -> bob not
    an orphan; alice is)
  - includePseudo:true surfaces _atlas-style pages
  - queryOrphanPages delegates to passed engine
  - empty brain returns {orphans: [], total_pages: 0} without crashing

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(cycle): add runCycle primitive in src/core/cycle.ts

The brain maintenance cycle as a single function. Six phases in
semantically-driven order (fix files → sync → extract → embed →
report orphans). Pure composition of existing library calls — no
execSync, no subprocess anti-patterns, no regex-parsed output.

    ┌───────────────────────────────────────────────────┐
    │ runCycle(engine, opts) → CycleReport              │
    │   Phase 1: lint --fix         (fs writes)         │
    │   Phase 2: backlinks --fix    (fs writes)         │
    │   Phase 3: sync               (DB picks up 1+2)   │
    │   Phase 4: extract            (DB picks up links) │
    │   Phase 5: embed --stale      (DB writes)         │
    │   Phase 6: orphans            (DB read, report)   │
    └───────────────────────────────────────────────────┘

Why the commit-4 primitive:

  - CEO + Eng + Codex reviews all converged on "extract one cycle
    function, wire both dream and autopilot through it." Two CLIs,
    one definition of what the brain does overnight.
  - Phase order was wrong in PR #309's original dream.ts (sync
    before lint+backlinks lost the "fix files, then index them"
    semantic).
  - This commit is the bisectable foundation; commit 5 (dream)
    and commit 6 (autopilot+jobs) just call into it.

Coordination — the codex-flagged blocker:

Session-scoped pg_try_advisory_lock does not survive PgBouncer
transaction pooling (the v0.15.4 fix made pooled connections the
default). Replaced with a DB lock table (gbrain_cycle_locks) that
works through every pooler:

  - Acquire: INSERT ... ON CONFLICT DO UPDATE ... WHERE ttl < NOW()
  - Refresh: UPDATE ttl_expires_at between phases via hook
  - Release: DELETE in finally{}
  - TTL: 30 min; crashed holders auto-release

PGLite / engine=null path uses a file lock at ~/.gbrain/cycle.lock
with PID liveness check. kill(pid, 0) with EPERM treated as alive
(so init/launchd-pid holders aren't mis-classified as stale).

Lock-skip: only phases that mutate state (lint, backlinks, sync,
extract, embed) trigger lock acquisition. orphans is read-only.
Single-phase --phase orphans runs never block on a held lock.

Engine-null mode preserved: filesystem phases run, DB phases skip
with {status:'skipped', reason:'no_database'}. Matches current
dream's capability that would have been lost if runCycle required
a connected engine.

Contract details:

  - CycleReport has schema_version:"1" (stable, additive) so agents
    consuming --json can rely on the shape
  - status: 'ok' | 'clean' | 'partial' | 'skipped' | 'failed'.
    'clean' = ran successfully with zero activity; agents trivially
    detect a healthy brain.
  - PhaseResult.error: { class, code, message, hint?, docs_url? }
    (Stripe-API-tier structured failure info) when status='fail'
  - yieldBetweenPhases hook: awaited between EVERY phase and before
    return, runs even after phase failure, exceptions logged but
    non-fatal. Required so the Minions autopilot-cycle handler can
    renew its job lock between phases (prevents the v0.14 stall-death
    regression codex flagged).
  - git pull explicit: opts.pull defaults to false (cron-safe).
    Autopilot daemon callers opt in if user configured it.
  - extract phase doesn't have a dry-run mode in the underlying
    library function, so runCycle honestly skips extract when
    dryRun=true (status:'skipped', reason:'no_dry_run_support').

Schema migration v16: gbrain_cycle_locks table + idx_cycle_locks_ttl.
Also appended to src/schema.sql and src/core/pglite-schema.ts for
fresh installs. schema-embedded.ts regenerated via build:schema.

Tests (test/core/cycle.test.ts, PGLite in-memory + mocked library
functions, no DATABASE_URL required):

  - dryRun × phases matrix: dryRun:true reaches lint/backlinks/sync/
    embed; extract is honestly skipped
  - Phase selection: default runs all 6 in order; --phase lint runs
    only lint; --phase orphans runs only orphans
  - Lock semantics: acquire + release on mutating phases, skip
    entirely for read-only selections
  - cycle_already_running: seeded live-holder lock → status:skipped,
    zero phase runs; TTL-expired holder → auto-claimed
  - Engine null: filesystem phases run, DB phases skip
  - File lock (engine=null) blocks when PID 1 holds lock with fresh
    mtime — exercises the PID liveness branch including EPERM
  - Status derivation: 'ok' vs 'clean' vs 'partial' vs 'skipped'
  - yieldBetweenPhases called N times, hook exceptions non-fatal

Next: commit 5 rewrites dream.ts as a thin CLI alias over runCycle,
commit 6 migrates autopilot daemon + jobs.ts handler to delegate to
runCycle too.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(dream): add gbrain dream CLI as a thin alias over runCycle

`gbrain dream` is the README brand-promise command: "the agent runs
while I sleep, the dream cycle ... I wake up and the brain is smarter."
Cron-friendly, JSON-reportable, phase-selectable. Same maintenance
cycle as `gbrain autopilot`, just scheduled differently — both
converge on runCycle (added in commit 4) so there's one source of
truth for what happens overnight.

Contract:
  gbrain dream                       # full 6-phase cycle
  gbrain dream --dry-run             # preview, no writes
  gbrain dream --json                # CycleReport JSON (agent-readable)
  gbrain dream --phase <name>        # single-phase run
  gbrain dream --pull                # git pull before syncing
  gbrain dream --dir /path/to/brain  # explicit brain location

Cron: 0 2 * * * gbrain dream --json >> /var/log/gbrain-dream.log

Behavior details:
  - Brain-dir resolution: requires explicit --dir OR sync.repo_path
    in engine config. No more walk-up-cwd-for-.git footgun that
    PR #309's original dream.ts had (would lint unrelated git repos).
  - engine=null mode preserved via cli.ts's try/catch around
    connectEngine — filesystem phases (lint, backlinks) still run
    without a DB, DB phases report skipped/no_database in the output.
  - status=clean prints "Brain is healthy. N phase(s) checked in Ns."
    status=skipped prints the reason (cycle_already_running, etc.).
    Partial/failed prints the phase-by-phase detail.
  - Exit code 1 when status=failed (cron spots real problems).
    'partial' is not a failure — warnings shouldn't page you.
  - --help text cross-references `autopilot --install` for users
    who want continuous maintenance as a daemon.

CLI registration (src/cli.ts):
  - 'dream' added to CLI_ONLY
  - handleCliOnly has a pre-engine branch mirroring doctor's pattern:
    try connectEngine() → ok path; catch → runDream(null, args) so
    filesystem phases still run when DB is down
  - Help text updated with one-line dream entry and autopilot cross-ref

Tests (test/dream.test.ts, real PGLite + real library calls, no mocks
to avoid `mock.module` leakage across test files):
  - brainDir resolution: explicit --dir wins, engine config fallback,
    missing + nonexistent errors
  - phase selection: --phase lint|orphans produces single-phase report
  - phase validation: --phase garbage exits 1
  - output: --json parses as CycleReport with schema_version:"1"
  - human output mentions "Brain is healthy" on clean status
  - dry-run: cycle runs but DB stays untouched
  - exit code: clean/ok/partial do not call process.exit

Also (test/core/cycle.test.ts): refactored to use beforeAll/afterAll
with one shared PGLite engine per describe + truncateCycleLocks
between tests. Cuts test time from ~11s to ~4s; avoids the 15-migration
penalty per test that was causing parallel-suite timeout flakes.

Co-Authored-By: Wintermute <wintermute@garrytan.com>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat: v0.17.0 — autopilot + jobs delegate to runCycle (unifies the cycle)

Autopilot daemon (`--inline` path) and Minions `autopilot-cycle`
handler both now delegate to `runCycle` (introduced in commit 4).
Three callers, one cycle definition:

  1. `gbrain dream`                        — one-shot cron cycle
  2. `gbrain autopilot` daemon inline path — scheduled cycles
  3. `autopilot-cycle` Minions handler     — durable queue with retry

All three share:
  - Same 6 phases in same order (lint → backlinks → sync → extract →
    embed → orphans)
  - Same DB lock table coordination (`gbrain_cycle_locks`)
  - Same yieldBetweenPhases discipline (prevents v0.14 stall-death)
  - Same structured CycleReport output

Autopilot inline path gains lint + orphan sweep that the old path
skipped. Minions autopilot-cycle handler also gains lint + orphans.
Users who run `gbrain autopilot --install` see 6-phase reports in
`gbrain jobs get <id>` starting on next interval. No config change
required.

Changes:
  - `src/commands/autopilot.ts`: inline fallback path (~20 lines)
    replaces the ~22-line sync+extract+embed sequence with a single
    runCycle call. Uses pull:true (matches pre-v0.17 autopilot
    behavior). Uses setImmediate yield hook. Status/failure reporting
    derives from CycleReport.status. `--help` cross-references `gbrain
    dream` for one-shot use.
  - `src/commands/jobs.ts:579` (`autopilot-cycle` handler): replaces
    the 4-step try/catch sequence with a runCycle call. Returns
    `{ partial, status, report }` so `gbrain jobs get <id>` shows the
    full structured CycleReport. Preserves partial-failure semantic
    (one phase failing does NOT throw; next cycle still runs).
    yieldBetweenPhases yields the event loop between phases for the
    worker's lock-renewal timer.

Release scaffolding:
  - VERSION: 0.16.0 → 0.17.0
  - CHANGELOG.md: v0.17.0 entry in GStack voice — headline, numbers
    table, "what this means" paragraph, "To take advantage" block
    per CLAUDE.md post-ship rules. Itemized changes below the fold.
    Credit to @Wintermute for the original PR #309 thesis.
  - skills/migrations/v0.17.0.md: documents what changed for
    upgrading users. No mechanical action required — schema migration
    v16 (cycle locks table) + handler delegation both apply
    automatically. Includes opt-out paths for users who don't want
    their daemon modifying files (use `dream --phase orphans` in cron
    and skip autopilot-install, or other explicit configs).
  - CLAUDE.md: new entries for `src/core/cycle.ts` and
    `src/commands/dream.ts` with contract details.

Tests: no new test file needed for this commit — the cycle primitive
is extensively tested in test/core/cycle.test.ts (18 cases), dream
in test/dream.test.ts (11), and autopilot's delegation is mechanical
(calls runCycle with specific opts). The handler contract is covered
implicitly: if runCycle returns a CycleReport, the handler wraps it
in `{ partial, status, report }` — nothing else to assert.

Verified:
  - `bun test test/autopilot-install.test.ts test/autopilot-resolve-cli.test.ts test/core/cycle.test.ts test/dream.test.ts` → 37 pass, 0 fail

Completes the v0.17.0 feature: 6 bisectable commits on one branch
(garrytan/v0.17-dream-cycle), ready to push as one PR.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* test(e2e): add runCycle + dream E2E coverage against real Postgres

Gap from the v0.17 commit series: PR #321 shipped unit-level tests
for runCycle (test/core/cycle.test.ts) and dream (test/dream.test.ts)
but no E2E coverage that exercises the real Postgres paths. Filling
that in before merge.

  test/e2e/cycle.test.ts (6 cases):
    - schema migration v16 created gbrain_cycle_locks + index
    - dry-run full cycle: zero DB writes + lock table empty after
    - live cycle: pages + chunks materialize, sync.last_commit set
    - concurrent cycle blocked by lock → status:'skipped'
    - TTL-expired lock auto-claimed (crashed-holder recovery)
    - --phase orphans skips lock entirely (read-only optimization)

  test/e2e/dream.test.ts (3 cases):
    - dream --dry-run --json emits valid CycleReport + DB stays empty
    - dream (no --dry-run) syncs pages into real DB
    - dream --phase orphans doesn't touch the cycle-lock table

Both files mock embedBatch via mock.module so the embed phase never
calls OpenAI even when the full 6-phase cycle runs (zero API cost,
zero flakiness from network calls).

Verified locally:
  - `docker run pgvector/pgvector:pg16` on port 5434
  - `DATABASE_URL=... bun test test/e2e/cycle.test.ts test/e2e/dream.test.ts` → 9 pass, 0 fail
  - Full E2E suite (`bun run test:e2e`): 16 files, 150 tests, 0 fail
  - Container torn down after: `docker stop + rm gbrain-test-pg`

Per CLAUDE.md E2E test DB lifecycle. These tests skip gracefully when
DATABASE_URL isn't set (via hasDatabase() helper + describe.skip).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-authored-by: Wintermute <wintermute@garrytan.com>
2026-04-22 08:23:24 -07:00

818 lines
28 KiB
TypeScript

/**
* src/core/cycle.ts — The brain maintenance cycle primitive.
*
* Composes lint, backlinks, sync, extract, embed, and orphans into
* one honest unit of work. Called from:
* - `gbrain dream` (CLI alias; one-shot cron-triggered cycle)
* - `gbrain autopilot` (daemon; scheduled on an interval)
* - Minions `autopilot-cycle` handler (durable queue; retry + observability)
*
* All three converge on runCycle() so there's one source of truth for
* what "overnight maintenance" means.
*
* PHASE ORDER (semantically driven — fix files first, then index):
*
* ┌───────────────────────────────────────────────────────────┐
* │ Phase 1: lint --fix (filesystem writes, no DB) │
* │ Phase 2: backlinks --fix (filesystem writes, no DB) │
* │ Phase 3: sync (DB picks up phases 1+2) │
* │ Phase 4: extract (DB picks up links from sync) │
* │ Phase 5: embed --stale (DB writes) │
* │ Phase 6: orphans (DB read, report only) │
* └───────────────────────────────────────────────────────────┘
*
* COORDINATION:
*
* Postgres: a row in gbrain_cycle_locks with a TTL (30 min). Refreshed
* between phases via yieldBetweenPhases. Works through PgBouncer
* transaction pooling (session-scoped pg_try_advisory_lock does not).
*
* PGLite / engine=null: a file lock at ~/.gbrain/cycle.lock holding
* the PID + mtime. Same 30-min TTL semantics.
*
* LOCK-SKIP:
*
* Filesystem-only or read-only phase selections (lint, backlinks,
* orphans) skip the lock. Only DB-write phases (sync, extract, embed)
* trigger lock acquisition.
*/
import { existsSync, readFileSync, writeFileSync, unlinkSync, mkdirSync, statSync } from 'fs';
import { join } from 'path';
import { homedir, hostname } from 'os';
import type { BrainEngine } from './engine.ts';
import { createProgress, type ProgressReporter } from './progress.ts';
import { getCliOptions, cliOptsToProgressOptions } from './cli-options.ts';
// ─── Types ─────────────────────────────────────────────────────────
export type CyclePhase = 'lint' | 'backlinks' | 'sync' | 'extract' | 'embed' | 'orphans';
export const ALL_PHASES: CyclePhase[] = [
'lint',
'backlinks',
'sync',
'extract',
'embed',
'orphans',
];
/**
* Phases that mutate state (filesystem or DB) and therefore should
* coordinate via the cycle lock. Only orphans is truly read-only
* and skips the lock.
*/
const NEEDS_LOCK_PHASES: ReadonlySet<CyclePhase> = new Set([
'lint',
'backlinks',
'sync',
'extract',
'embed',
]);
export type PhaseStatus = 'ok' | 'warn' | 'fail' | 'skipped';
export interface PhaseError {
/** Error class for machine branching — e.g., 'DatabaseConnection', 'Timeout', 'LLMError', 'FilesystemError', 'InternalError'. */
class: string;
/** System error code or short identifier, e.g., 'ECONNREFUSED', 'ETIMEDOUT', 'UNKNOWN'. */
code: string;
/** Human-readable single-line message. */
message: string;
/** Optional suggestion of what to try next. */
hint?: string;
/** Optional link to a troubleshooting doc. */
docs_url?: string;
}
export interface PhaseResult {
phase: CyclePhase;
status: PhaseStatus;
duration_ms: number;
summary: string;
details: Record<string, unknown>;
error?: PhaseError;
}
export type CycleStatus = 'ok' | 'clean' | 'partial' | 'skipped' | 'failed';
export interface CycleReport {
/** Additive schema. Bumped on breaking changes. */
schema_version: '1';
timestamp: string;
duration_ms: number;
/**
* Overall status derived from phase results:
* - 'clean' : ran successfully, zero fixes/writes across every phase
* - 'ok' : ran successfully, some work was done
* - 'partial' : at least one phase warned or failed, others ran
* - 'skipped' : cycle did not run (lock held by another holder)
* - 'failed' : lock acquired but all attempted phases failed
*/
status: CycleStatus;
/** Present when status = 'skipped'. E.g., 'cycle_already_running' or 'no_database'. */
reason?: string;
brain_dir: string | null;
phases: PhaseResult[];
totals: {
lint_fixes: number;
backlinks_added: number;
pages_synced: number;
pages_extracted: number;
pages_embedded: number;
orphans_found: number;
};
}
export interface CycleOpts {
/** If true, no writes to filesystem or DB. All phases honor this. */
dryRun?: boolean;
/** Defaults to ALL_PHASES. Pass a subset for --phase lint etc. */
phases?: CyclePhase[];
/** Brain directory (git repo). Required for filesystem phases. */
brainDir: string;
/** Whether sync should run `git pull`. Default false (cron-safe). */
pull?: boolean;
/**
* Called between phases AND before runCycle returns. Awaited even
* after phase failure. Hook exceptions are logged, never fatal.
* Minions handlers pass a function that yields + renews the job lock
* + refreshes the cycle-lock-table TTL.
*/
yieldBetweenPhases?: () => Promise<void>;
}
// ─── Lock primitives ───────────────────────────────────────────────
const CYCLE_LOCK_ID = 'gbrain-cycle';
const LOCK_TTL_MS = 30 * 60 * 1000; // 30 minutes
const LOCK_FILE_PATH_DEFAULT = join(homedir(), '.gbrain', 'cycle.lock');
interface LockHandle {
release: () => Promise<void>;
refresh: () => Promise<void>;
}
/**
* Acquire the Postgres-backed cycle lock.
* Returns a LockHandle on success, or null if another live holder has it.
*
* Uses INSERT ... ON CONFLICT (id) DO UPDATE ... WHERE ttl_expires_at < NOW()
* RETURNING *. An empty RETURNING means the existing row is still live.
* Crashed holders auto-release: when their TTL expires, the next
* acquirer's UPDATE branch fires and takes over.
*/
async function acquirePostgresLock(engine: BrainEngine): Promise<LockHandle | null> {
const pid = process.pid;
const host = hostname();
// Engine-agnostic: BrainEngine exposes findOrphanPages etc., but not raw SQL.
// We reach through the engine's internal connection for this lock operation.
// Both engines expose `sql` (postgres-js tag) or `db.query` (PGLite).
const maybePG = engine as unknown as { sql?: (...args: unknown[]) => Promise<unknown> };
const maybePGLite = engine as unknown as { db?: { query: (sql: string, params?: unknown[]) => Promise<{ rows: unknown[] }> } };
if (engine.kind === 'postgres' && maybePG.sql) {
const sql = maybePG.sql as any;
const rows: Array<{ id: string }> = await sql`
INSERT INTO gbrain_cycle_locks (id, holder_pid, holder_host, acquired_at, ttl_expires_at)
VALUES (${CYCLE_LOCK_ID}, ${pid}, ${host}, NOW(), NOW() + INTERVAL '30 minutes')
ON CONFLICT (id) DO UPDATE
SET holder_pid = ${pid},
holder_host = ${host},
acquired_at = NOW(),
ttl_expires_at = NOW() + INTERVAL '30 minutes'
WHERE gbrain_cycle_locks.ttl_expires_at < NOW()
RETURNING id
`;
if (rows.length === 0) return null; // live holder
return {
refresh: async () => {
await sql`
UPDATE gbrain_cycle_locks
SET ttl_expires_at = NOW() + INTERVAL '30 minutes'
WHERE id = ${CYCLE_LOCK_ID} AND holder_pid = ${pid}
`;
},
release: async () => {
await sql`
DELETE FROM gbrain_cycle_locks
WHERE id = ${CYCLE_LOCK_ID} AND holder_pid = ${pid}
`;
},
};
}
if (engine.kind === 'pglite' && maybePGLite.db) {
// PGLite is single-writer; the DB row is belt-and-braces on top of the
// file lock. Callers always hold the file lock first, so this UPSERT
// is race-free against other processes.
const db = maybePGLite.db;
const { rows } = await db.query(
`INSERT INTO gbrain_cycle_locks (id, holder_pid, holder_host, acquired_at, ttl_expires_at)
VALUES ($1, $2, $3, NOW(), NOW() + INTERVAL '30 minutes')
ON CONFLICT (id) DO UPDATE
SET holder_pid = $2,
holder_host = $3,
acquired_at = NOW(),
ttl_expires_at = NOW() + INTERVAL '30 minutes'
WHERE gbrain_cycle_locks.ttl_expires_at < NOW()
RETURNING id`,
[CYCLE_LOCK_ID, pid, host],
);
if (rows.length === 0) return null;
return {
refresh: async () => {
await db.query(
`UPDATE gbrain_cycle_locks
SET ttl_expires_at = NOW() + INTERVAL '30 minutes'
WHERE id = $1 AND holder_pid = $2`,
[CYCLE_LOCK_ID, pid],
);
},
release: async () => {
await db.query(
`DELETE FROM gbrain_cycle_locks WHERE id = $1 AND holder_pid = $2`,
[CYCLE_LOCK_ID, pid],
);
},
};
}
throw new Error(`Unknown engine kind: ${engine.kind}`);
}
/**
* Acquire the file-based cycle lock (used when engine === null).
* Returns a LockHandle on success, or null if a live holder has it.
*
* The file contains `{pid}\n{iso-timestamp}`. Staleness = mtime older
* than LOCK_TTL_MS OR the PID is no longer alive on this host.
*/
function acquireFileLock(lockPath = LOCK_FILE_PATH_DEFAULT): LockHandle | null {
mkdirSync(join(lockPath, '..'), { recursive: true });
const pid = process.pid;
if (existsSync(lockPath)) {
// Check TTL.
try {
const st = statSync(lockPath);
const ageMs = Date.now() - st.mtimeMs;
const existingContent = readFileSync(lockPath, 'utf-8').trim();
const existingPid = parseInt(existingContent.split('\n')[0] || '0', 10);
// PID liveness check (same host only). kill(pid, 0) distinguishes:
// - success → process exists, caller can signal it
// - error ESRCH → no such process (truly dead)
// - error EPERM → process exists but caller can't signal it
// (e.g., PID 1/init on unix) → still alive
// Any error code OTHER than ESRCH means the PID is alive.
let pidAlive = false;
if (existingPid > 0 && existingPid !== pid) {
try {
process.kill(existingPid, 0);
pidAlive = true;
} catch (e) {
const code = (e as NodeJS.ErrnoException).code;
pidAlive = code !== 'ESRCH';
}
} else if (existingPid === pid) {
// Our own stale lock (same pid, previous run) — treat as stale.
pidAlive = false;
}
if (pidAlive && ageMs < LOCK_TTL_MS) {
return null; // live holder
}
// Stale lock — fall through to overwrite.
} catch {
// Any read/stat error: treat as stale.
}
}
writeFileSync(lockPath, `${pid}\n${new Date().toISOString()}\n`);
return {
refresh: async () => {
try {
writeFileSync(lockPath, `${pid}\n${new Date().toISOString()}\n`);
} catch {
/* non-fatal — a next-run stale check will notice */
}
},
release: async () => {
try {
const content = readFileSync(lockPath, 'utf-8').trim();
const heldPid = parseInt(content.split('\n')[0] || '0', 10);
if (heldPid === pid) unlinkSync(lockPath);
} catch {
/* already gone */
}
},
};
}
// ─── Helpers ───────────────────────────────────────────────────────
function makeErrorFromException(e: unknown, fallbackClass = 'InternalError'): PhaseError {
const err = e instanceof Error ? e : new Error(String(e));
// Node errors often have .code (e.g., 'ECONNREFUSED').
const code = (err as NodeJS.ErrnoException).code || 'UNKNOWN';
let className = fallbackClass;
if (code === 'ECONNREFUSED' || code === 'ENOTFOUND') className = 'DatabaseConnection';
if (code === 'ETIMEDOUT') className = 'Timeout';
if (/OpenAI|embed/i.test(err.message)) className = 'LLMError';
if (/ENOENT|EACCES|EISDIR|ENOTDIR/.test(code)) className = 'FilesystemError';
return {
class: className,
code,
message: err.message.slice(0, 200),
};
}
async function timePhase<T>(fn: () => Promise<T>): Promise<{ result: T; duration_ms: number }> {
const start = performance.now();
const result = await fn();
return { result, duration_ms: Math.round(performance.now() - start) };
}
async function safeYield(hook?: () => Promise<void>) {
if (!hook) return;
try {
await hook();
} catch (e) {
console.warn(`[cycle] yieldBetweenPhases hook error (non-fatal): ${e instanceof Error ? e.message : String(e)}`);
}
}
// ─── Phase runners ─────────────────────────────────────────────────
async function runPhaseLint(brainDir: string, dryRun: boolean): Promise<PhaseResult> {
try {
const { runLintCore } = await import('../commands/lint.ts');
const result = await runLintCore({ target: brainDir, fix: true, dryRun });
const issues = result.total_issues ?? 0;
const fixed = result.total_fixed ?? 0;
const remaining = Math.max(0, issues - fixed);
// 'ok' when nothing noteworthy remains:
// - no issues at all, or
// - non-dry-run and everything fixable was fixed.
// 'warn' when issues remain after the run.
const status: PhaseStatus =
issues === 0 || (!dryRun && remaining === 0) ? 'ok' : 'warn';
return {
phase: 'lint',
status,
duration_ms: 0, // set by caller
summary: dryRun
? `${issues} issue(s) found (dry-run, no writes)`
: `${fixed} fix(es) applied, ${remaining} remaining`,
details: { issues, fixed, pages_scanned: result.pages_scanned, dryRun },
};
} catch (e) {
return {
phase: 'lint',
status: 'fail',
duration_ms: 0,
summary: 'lint phase failed',
details: {},
error: makeErrorFromException(e),
};
}
}
async function runPhaseBacklinks(brainDir: string, dryRun: boolean): Promise<PhaseResult> {
try {
// Library function path — the v0.15 backlinks.ts exports
// runBacklinksCore when --fix is requested.
const { runBacklinksCore } = await import('../commands/backlinks.ts');
const result = await runBacklinksCore({
action: 'fix',
dir: brainDir,
dryRun,
});
const gaps = result.gaps_found ?? 0;
const added = result.fixed ?? 0;
const remaining = Math.max(0, gaps - added);
const status: PhaseStatus =
gaps === 0 || (!dryRun && remaining === 0) ? 'ok' : 'warn';
return {
phase: 'backlinks',
status,
duration_ms: 0,
summary: dryRun
? `${gaps} missing back-link(s) (dry-run)`
: `${added} back-link(s) added, ${remaining} remaining`,
details: { gaps, added, pages_affected: result.pages_affected, dryRun },
};
} catch (e) {
return {
phase: 'backlinks',
status: 'fail',
duration_ms: 0,
summary: 'backlinks phase failed',
details: {},
error: makeErrorFromException(e),
};
}
}
async function runPhaseSync(
engine: BrainEngine,
brainDir: string,
dryRun: boolean,
pull: boolean,
): Promise<PhaseResult> {
try {
const { performSync } = await import('../commands/sync.ts');
const result = await performSync(engine, {
repoPath: brainDir,
dryRun,
noPull: !pull,
noEmbed: true, // embed is a separate phase
});
const syncedCount = result.added + result.modified;
return {
phase: 'sync',
status: result.status === 'blocked_by_failures' ? 'warn' : 'ok',
duration_ms: 0,
summary: dryRun
? `${syncedCount} page(s) would sync, ${result.deleted} would delete`
: `+${result.added} added, ~${result.modified} modified, -${result.deleted} deleted`,
details: {
added: result.added,
modified: result.modified,
deleted: result.deleted,
renamed: result.renamed,
chunksCreated: result.chunksCreated,
failedFiles: result.failedFiles ?? 0,
syncStatus: result.status,
dryRun,
},
};
} catch (e) {
return {
phase: 'sync',
status: 'fail',
duration_ms: 0,
summary: 'sync phase failed',
details: {},
error: makeErrorFromException(e),
};
}
}
async function runPhaseExtract(
engine: BrainEngine,
brainDir: string,
dryRun: boolean,
): Promise<PhaseResult> {
try {
const { runExtractCore } = await import('../commands/extract.ts');
// Extract is read-mostly against the filesystem + write to links table.
// Honor dryRun by skipping with a 'skipped' entry: extract doesn't have
// a clean dry-run mode today and runCycle should be honest about it.
if (dryRun) {
return {
phase: 'extract',
status: 'skipped',
duration_ms: 0,
summary: 'dry-run: extract phase skipped (no dry-run mode yet)',
details: { dryRun: true, reason: 'no_dry_run_support' },
};
}
const result = await runExtractCore(engine, { mode: 'all', dir: brainDir });
const linksCreated = result?.links_created ?? 0;
const timelineCreated = result?.timeline_entries_created ?? 0;
return {
phase: 'extract',
status: 'ok',
duration_ms: 0,
summary: `${linksCreated} link(s), ${timelineCreated} timeline entries`,
details: { linksCreated, timelineCreated, pages_processed: result?.pages_processed ?? 0 },
};
} catch (e) {
return {
phase: 'extract',
status: 'fail',
duration_ms: 0,
summary: 'extract phase failed',
details: {},
error: makeErrorFromException(e),
};
}
}
async function runPhaseEmbed(engine: BrainEngine, dryRun: boolean): Promise<PhaseResult> {
try {
const { runEmbedCore } = await import('../commands/embed.ts');
const result = await runEmbedCore(engine, { stale: true, dryRun });
const embeddedCount = dryRun ? result.would_embed : result.embedded;
return {
phase: 'embed',
status: 'ok',
duration_ms: 0,
summary: dryRun
? `${result.would_embed} chunk(s) would be embedded (dry-run)`
: `${result.embedded} chunk(s) newly embedded (${result.skipped} already had embeddings)`,
details: {
embedded: result.embedded,
skipped: result.skipped,
would_embed: result.would_embed,
total_chunks: result.total_chunks,
pages_processed: result.pages_processed,
dryRun,
// Convenience field used by CycleReport.totals.pages_embedded.
// In dry-run, this counts pages with stale chunks that would
// have been processed (same semantic as a real run).
pages_embedded_count: dryRun ? result.pages_processed : embeddedCount > 0 ? result.pages_processed : 0,
},
};
} catch (e) {
return {
phase: 'embed',
status: 'fail',
duration_ms: 0,
summary: 'embed phase failed',
details: {},
error: makeErrorFromException(e),
};
}
}
async function runPhaseOrphans(engine: BrainEngine): Promise<PhaseResult> {
try {
const { findOrphans } = await import('../commands/orphans.ts');
const result = await findOrphans(engine);
const count = result.total_orphans;
return {
phase: 'orphans',
status: count > 20 ? 'warn' : 'ok',
duration_ms: 0,
summary: `${count} orphan page(s) out of ${result.total_pages} total`,
details: {
total_orphans: count,
total_pages: result.total_pages,
excluded: result.excluded,
},
};
} catch (e) {
return {
phase: 'orphans',
status: 'fail',
duration_ms: 0,
summary: 'orphans phase failed',
details: {},
error: makeErrorFromException(e),
};
}
}
// ─── Main ──────────────────────────────────────────────────────────
/**
* Run the brain maintenance cycle.
*
* Engine may be null: filesystem phases (lint, backlinks) still run;
* DB-dependent phases skip with status='skipped', reason='no_database'.
*
* Acquires the cycle lock for any DB-write phase selection. Non-DB-write
* selections (e.g., --phase lint) skip the lock as an optimization so
* single-phase runs are always responsive even if another cycle is live.
*/
export async function runCycle(
engine: BrainEngine | null,
opts: CycleOpts,
): Promise<CycleReport> {
const start = performance.now();
const phases = opts.phases ?? ALL_PHASES;
const dryRun = !!opts.dryRun;
const pull = !!opts.pull;
const timestamp = new Date().toISOString();
const phaseResults: PhaseResult[] = [];
const progress = createProgress(cliOptsToProgressOptions(getCliOptions()));
// Decide if we need the cycle lock: any state-mutating phase in the selection.
const needsLock = phases.some(p => NEEDS_LOCK_PHASES.has(p));
let lock: LockHandle | null = null;
if (needsLock) {
if (engine) {
try {
lock = await acquirePostgresLock(engine);
} catch (e) {
// Lock acquisition failed catastrophically (e.g., migration missing).
// Return a failed report rather than silently running without a lock.
return {
schema_version: '1',
timestamp,
duration_ms: Math.round(performance.now() - start),
status: 'failed',
reason: 'lock_acquisition_error',
brain_dir: opts.brainDir,
phases: [
{
phase: 'sync',
status: 'fail',
duration_ms: 0,
summary: 'could not acquire cycle lock',
details: {},
error: makeErrorFromException(e, 'DatabaseConnection'),
},
],
totals: emptyTotals(),
};
}
} else {
lock = acquireFileLock();
}
if (lock === null) {
return {
schema_version: '1',
timestamp,
duration_ms: Math.round(performance.now() - start),
status: 'skipped',
reason: 'cycle_already_running',
brain_dir: opts.brainDir,
phases: [],
totals: emptyTotals(),
};
}
}
try {
// ── Phase 1: lint ────────────────────────────────────────────
if (phases.includes('lint')) {
progress.start('cycle.lint');
const { result, duration_ms } = await timePhase(() => runPhaseLint(opts.brainDir, dryRun));
result.duration_ms = duration_ms;
phaseResults.push(result);
progress.finish();
await safeYield(opts.yieldBetweenPhases);
}
// ── Phase 2: backlinks ──────────────────────────────────────
if (phases.includes('backlinks')) {
progress.start('cycle.backlinks');
const { result, duration_ms } = await timePhase(() => runPhaseBacklinks(opts.brainDir, dryRun));
result.duration_ms = duration_ms;
phaseResults.push(result);
progress.finish();
await safeYield(opts.yieldBetweenPhases);
}
// ── Phase 3: sync ───────────────────────────────────────────
if (phases.includes('sync')) {
if (!engine) {
phaseResults.push({
phase: 'sync',
status: 'skipped',
duration_ms: 0,
summary: 'no database connected',
details: { reason: 'no_database' },
});
} else {
progress.start('cycle.sync');
const { result, duration_ms } = await timePhase(() => runPhaseSync(engine, opts.brainDir, dryRun, pull));
result.duration_ms = duration_ms;
phaseResults.push(result);
progress.finish();
}
await safeYield(opts.yieldBetweenPhases);
}
// ── Phase 4: extract ────────────────────────────────────────
if (phases.includes('extract')) {
if (!engine) {
phaseResults.push({
phase: 'extract',
status: 'skipped',
duration_ms: 0,
summary: 'no database connected',
details: { reason: 'no_database' },
});
} else {
progress.start('cycle.extract');
const { result, duration_ms } = await timePhase(() => runPhaseExtract(engine, opts.brainDir, dryRun));
result.duration_ms = duration_ms;
phaseResults.push(result);
progress.finish();
}
await safeYield(opts.yieldBetweenPhases);
}
// ── Phase 5: embed ──────────────────────────────────────────
if (phases.includes('embed')) {
if (!engine) {
phaseResults.push({
phase: 'embed',
status: 'skipped',
duration_ms: 0,
summary: 'no database connected',
details: { reason: 'no_database' },
});
} else {
progress.start('cycle.embed');
const { result, duration_ms } = await timePhase(() => runPhaseEmbed(engine, dryRun));
result.duration_ms = duration_ms;
phaseResults.push(result);
progress.finish();
}
await safeYield(opts.yieldBetweenPhases);
}
// ── Phase 6: orphans ────────────────────────────────────────
if (phases.includes('orphans')) {
if (!engine) {
phaseResults.push({
phase: 'orphans',
status: 'skipped',
duration_ms: 0,
summary: 'no database connected',
details: { reason: 'no_database' },
});
} else {
progress.start('cycle.orphans');
const { result, duration_ms } = await timePhase(() => runPhaseOrphans(engine));
result.duration_ms = duration_ms;
phaseResults.push(result);
progress.finish();
}
await safeYield(opts.yieldBetweenPhases);
}
} finally {
if (lock) {
try { await lock.release(); } catch { /* best-effort */ }
}
}
const duration_ms = Math.round(performance.now() - start);
const totals = extractTotals(phaseResults);
const status = deriveStatus(phaseResults, totals);
return {
schema_version: '1',
timestamp,
duration_ms,
status,
brain_dir: opts.brainDir,
phases: phaseResults,
totals,
};
}
// ─── Totals + status derivation ────────────────────────────────────
function emptyTotals(): CycleReport['totals'] {
return {
lint_fixes: 0,
backlinks_added: 0,
pages_synced: 0,
pages_extracted: 0,
pages_embedded: 0,
orphans_found: 0,
};
}
function extractTotals(phases: PhaseResult[]): CycleReport['totals'] {
const t = emptyTotals();
for (const p of phases) {
if (p.phase === 'lint' && p.details) {
t.lint_fixes = Number(p.details.fixed ?? 0);
} else if (p.phase === 'backlinks' && p.details) {
t.backlinks_added = Number(p.details.added ?? 0);
} else if (p.phase === 'sync' && p.details) {
t.pages_synced = Number(p.details.added ?? 0) + Number(p.details.modified ?? 0);
} else if (p.phase === 'extract' && p.details) {
t.pages_extracted = Number(p.details.linksCreated ?? 0);
} else if (p.phase === 'embed' && p.details) {
// In dry-run, use would_embed as the "activity" measure; else embedded.
const dryRun = p.details.dryRun === true;
t.pages_embedded = dryRun
? Number(p.details.would_embed ?? 0)
: Number(p.details.embedded ?? 0);
} else if (p.phase === 'orphans' && p.details) {
t.orphans_found = Number(p.details.total_orphans ?? 0);
}
}
return t;
}
function deriveStatus(phases: PhaseResult[], totals: CycleReport['totals']): CycleStatus {
if (phases.length === 0) return 'failed';
const anyFailed = phases.some(p => p.status === 'fail');
const allFailed = phases.every(p => p.status === 'fail');
const anyWarn = phases.some(p => p.status === 'warn');
if (allFailed) return 'failed';
if (anyFailed || anyWarn) return 'partial';
// All phases 'ok' or 'skipped'. Distinguish clean (no activity) from ok (work done).
const anyWork =
totals.lint_fixes > 0 ||
totals.backlinks_added > 0 ||
totals.pages_synced > 0 ||
totals.pages_extracted > 0 ||
totals.pages_embedded > 0;
return anyWork ? 'ok' : 'clean';
}