admin/gbrain
1
0
Fork 0
mirror of https://github.com/garrytan/gbrain.git synced 2026-07-29 19:01:39 +00:00
Code Issues Packages Projects Releases Wiki Activity

fix: 8 root-cause fixes from /investigate (v0.14.2) (#259)

Browse Source 
* fix: 8 root-cause fixes from /investigate wave

Consolidated bundle of bug fixes from /investigate on the 8 deferred bugs.
Each fix was designed to go at the structural gap, not the symptom. Codex
verified 20 load-bearing claims on the plan; 12 triggered plan revisions.

Bug 2  — GBRAIN_POOL_SIZE env knob + init finally blocks (no auto-detect).
         Covers both the singleton pool (db.ts) and instance pool (import.ts:140).
Bug 3  — Centralize migration ledger writes in apply-migrations runner.
         Removed appendCompletedMigration from v0_11_0, v0_12_0, v0_12_2,
         v0_13_0, v0_13_1. Added 3-partial wedge cap + --force-retry reset.
         'complete wins' preserved; no partial can regress a completed migration.
Bug 5  — v0.14.0 migration registered. src/commands/migrations/v0_14_0.ts
         ships Phase A (ALTER minion_jobs.max_stalled SET DEFAULT 3) + Phase B
         (pending-host-work ping for shell-jobs adoption).
Bug 6/10 — jsonb_agg(DISTINCT ...) in legacy traverseGraph (both engines).
         Presentation-level dedup; schema still preserves provenance rows.
Bug 7  — doctor --fast reads DB URL source via getDbUrlSource() in config.ts.
         Precise message: 'Skipping DB checks (--fast mode, URL present from env)'
         replaces the misleading 'No database configured'.
Bug 8  — max_stalled default bumped 1→3 in schema-embedded.ts, pglite-schema.ts,
         schema.sql (new installs). v0_14_0 Phase A ALTER for existing installs.
         autopilot-cycle handler yields to event loop between phases so the
         worker's lock-renewal timer fires on huge brains. (Deep AbortSignal
         threading through runEmbedCore/runExtractCore/runBacklinksCore/performSync
         deferred to v0.15 queue polish.)
Bug 9  — Gate sync.last_commit on no-failures across all three sync paths
         (incremental, full via runImport, gbrain import git continuity).
         recordSyncFailures() helper + ~/.gbrain/sync-failures.jsonl with
         dedup key path+commit+error-hash. New flags: --skip-failed (ack) +
         --retry-failed (re-attempt). Doctor surfaces unacknowledged failures.
Bug 11 — brain_score breakdown fields on BrainHealth (embed_coverage_score,
         link_density_score, timeline_coverage_score, no_orphans_score,
         no_dead_links_score); sum equals brain_score by construction.
         dead_links now on the type (resolves featuresTeaserForDoctor drift).
         orphan_pages kept as 'islanded' (no inbound AND no outbound) and
         docs updated to match — explicit semantic instead of doc drift.

New tests: test/traverse-graph-dedup.test.ts, test/sync-failures.test.ts,
test/brain-score-breakdown.test.ts, test/migration-resume.test.ts,
test/migrations-v0_14_0.test.ts. Extended: migrate, doctor, apply-migrations.

All 1696 unit tests pass locally. postgres-jsonb E2E regression unchanged
(none of these touch the JSONB write surface).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* docs: v0.14.2 CHANGELOG + CLAUDE.md; align migration-flow E2E with runner-owned ledger

CHANGELOG: v0.14.2 entry in the standard release-summary format
(two-line headline + lead + numbers table + "what this means" +
"To take advantage of v0.14.2" self-repair block + itemized
changes grouped by reliability / observability / graph correctness /
new migration / tests / deferred-to-v0.15).

CLAUDE.md: new "Key commands added in v0.14.2" section covers
--skip-failed, --retry-failed, --force-retry, GBRAIN_POOL_SIZE env,
and the new doctor checks (sync_failures, brain_score breakdown).
Migration orchestrator docs updated to describe v0_14_0.ts + the
runner-owned ledger contract from Bug 3.

test/e2e/migration-flow.test.ts: three assertions updated to match
the Bug 3 contract — orchestrators no longer append to completed.jsonl
directly, so direct-orchestrator E2E calls leave the ledger empty.
Preferences assertions remain (that's still the orchestrator's side
of the contract). Runner's ledger write is covered by the unit suite
(test/apply-migrations.test.ts + test/migration-resume.test.ts).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
Garry Tan
2026-04-20 23:14:38 +08:00
committed by GitHub
co-authored by Claude Opus 4.6
parent ebfbd5e6f7
commit b5fa3d044a
37 changed files with 1804 additions and 210 deletions
Download Patch File Download Diff File Expand all files Collapse all files
src/cli.ts
+5 -2
View File
@@ -332,8 +332,11 @@ async function handleCliOnly(command: string, args: string[]) {
// Doctor runs filesystem checks first (no DB needed), then DB checks.
// --fast skips DB checks entirely.
const { runDoctor } = await import('./commands/doctor.ts');
const { getDbUrlSource } = await import('./core/config.ts');
if (args.includes('--fast')) {
await runDoctor(null, args);
// Pass the DB URL source so doctor can tell "no config at all" from
// "user chose --fast while config is present".
await runDoctor(null, args, getDbUrlSource());
} else {
try {
const eng = await connectEngine();
@@ -341,7 +344,7 @@ async function handleCliOnly(command: string, args: string[]) {
await eng.disconnect();
} catch {
// DB unavailable — still run filesystem checks
await runDoctor(null, args);
await runDoctor(null, args, getDbUrlSource());
}
}
return;
src/commands/apply-migrations.ts
+113 -4
View File
@@ -14,9 +14,12 @@
import { VERSION } from '../version.ts';
import { loadConfig } from '../core/config.ts';
import { loadCompletedMigrations, type CompletedMigrationEntry } from '../core/preferences.ts';
import { loadCompletedMigrations, appendCompletedMigration, type CompletedMigrationEntry } from '../core/preferences.ts';
import { migrations, compareVersions, type Migration, type OrchestratorOpts } from './migrations/index.ts';
/** Bug 3 — max consecutive partials before we wedge a migration. */
const MAX_CONSECUTIVE_PARTIALS = 3;
interface ApplyMigrationsArgs {
list: boolean;
dryRun: boolean;
@@ -26,6 +29,8 @@ interface ApplyMigrationsArgs {
specificMigration?: string;
hostDir?: string;
noAutopilotInstall: boolean;
/** Bug 3 — explicit reset for a wedged migration. Writes a 'retry' marker. */
forceRetry?: string;
help: boolean;
}
@@ -49,6 +54,7 @@ function parseArgs(args: string[]): ApplyMigrationsArgs {
specificMigration: val('--migration'),
hostDir: val('--host-dir'),
noAutopilotInstall: has('--no-autopilot-install'),
forceRetry: val('--force-retry'),
help: has('--help') || has('-h'),
};
}
@@ -63,6 +69,10 @@ Usage:
gbrain apply-migrations --list Show applied + pending migrations.
gbrain apply-migrations --migration vX.Y.Z
Force-run a specific migration by version.
gbrain apply-migrations --force-retry vX.Y.Z
Clear a wedged migration (3+ consecutive
partials). Writes a 'retry' marker so the
next run treats it as fresh.
Flags:
--mode <always|pain_triggered|off> Set minion_mode without prompting.
@@ -94,14 +104,38 @@ function indexCompleted(entries: CompletedMigrationEntry[]): CompletedIndex {
: { byVersion: new Map() };
}
/** Returns the resolved status for a migration based on its entries. */
/**
* Returns the resolved status for a migration based on its entries.
*
* Semantics (Bug 3 — keep "complete wins" safety):
* - If any entry is `complete`, the version is complete. Terminal state.
* - Otherwise, if the latest entry is `retry`, the version is pending
* (user requested a fresh attempt).
* - Otherwise, if any entry is `partial`, the version is partial.
* - Otherwise, pending.
*
* `complete` never regresses. A later accidental `partial` append cannot
* undo a completed migration.
*/
function statusForVersion(
version: string,
idx: CompletedIndex,
): 'complete' | 'partial' | 'pending' {
): 'complete' | 'partial' | 'pending' | 'wedged' {
const entries = idx.byVersion.get(version) ?? [];
if (entries.length === 0) return 'pending';
if (entries.some(e => e.status === 'complete')) return 'complete';
const latest = entries[entries.length - 1];
if (latest.status === 'retry') return 'pending';
// Bug 3 attempt cap — count consecutive partials from the end (stopping
// at any 'retry' or 'complete'). If we hit MAX_CONSECUTIVE_PARTIALS,
// the migration is wedged and needs explicit --force-retry to try again.
let consecutive = 0;
for (let i = entries.length - 1; i >= 0; i--) {
const e = entries[i];
if (e.status === 'partial') consecutive++;
else break;
}
if (consecutive >= MAX_CONSECUTIVE_PARTIALS) return 'wedged';
if (entries.some(e => e.status === 'partial')) return 'partial';
return 'pending';
}
@@ -111,6 +145,7 @@ interface Plan {
partial: Migration[];
pending: Migration[];
skippedFuture: Migration[];
wedged: Migration[];
}
/**
@@ -127,7 +162,7 @@ interface Plan {
* skip v0.11.0 when running v0.11.1. Compare against completed.jsonl.
*/
function buildPlan(idx: CompletedIndex, installed: string, filterVersion?: string): Plan {
const plan: Plan = { applied: [], partial: [], pending: [], skippedFuture: [] };
const plan: Plan = { applied: [], partial: [], pending: [], skippedFuture: [], wedged: [] };
for (const m of migrations) {
if (filterVersion && m.version !== filterVersion) continue;
if (compareVersions(m.version, installed) > 0) {
@@ -137,6 +172,7 @@ function buildPlan(idx: CompletedIndex, installed: string, filterVersion?: strin
const status = statusForVersion(m.version, idx);
if (status === 'complete') plan.applied.push(m);
else if (status === 'partial') plan.partial.push(m);
else if (status === 'wedged') plan.wedged.push(m);
else plan.pending.push(m);
}
return plan;
@@ -149,6 +185,7 @@ function printList(plan: Plan, installed: string): void {
const rows: Array<{ status: string; m: Migration }> = [
...plan.applied.map(m => ({ status: 'applied', m })),
...plan.partial.map(m => ({ status: 'partial', m })),
...plan.wedged.map(m => ({ status: 'wedged', m })),
...plan.pending.map(m => ({ status: 'pending', m })),
...plan.skippedFuture.map(m => ({ status: 'future', m })),
];
@@ -227,10 +264,37 @@ export async function runApplyMigrations(args: string[]): Promise<void> {
return;
}
// Bug 3 — --force-retry: write an explicit reset marker for a wedged
// migration, then return. User re-runs `gbrain apply-migrations --yes`
// to actually re-attempt.
if (cli.forceRetry) {
const target = migrations.find(m => m.version === cli.forceRetry);
if (!target) {
console.error(`No migration registered with version "${cli.forceRetry}". Run \`gbrain apply-migrations --list\`.`);
process.exit(2);
}
appendCompletedMigration({ version: cli.forceRetry, status: 'retry' });
console.log(`Wrote 'retry' marker for v${cli.forceRetry}. Run \`gbrain apply-migrations --yes\` to re-attempt.`);
return;
}
const completed = loadCompletedMigrations();
const idx = indexCompleted(completed);
const plan = buildPlan(idx, installed, cli.specificMigration);
// Bug 3 — surface wedged migrations as a loud, actionable error.
if (plan.wedged.length > 0) {
for (const m of plan.wedged) {
console.error(
`\nMigration v${m.version} is WEDGED (${MAX_CONSECUTIVE_PARTIALS}+ consecutive partials with no completion). ` +
`Check ~/.gbrain/upgrade-errors.jsonl for the last failure reasons, fix the underlying issue, then run:\n` +
` gbrain apply-migrations --force-retry ${m.version}\n` +
`Then re-run \`gbrain apply-migrations --yes\`.`,
);
}
// Don't exit — applied/partial/pending are still worth reporting and running.
}
if (cli.specificMigration && plan.applied.length + plan.partial.length + plan.pending.length + plan.skippedFuture.length === 0) {
console.error(`No migration registered with version "${cli.specificMigration}". Run \`gbrain apply-migrations --list\` to see registered versions.`);
process.exit(2);
@@ -248,6 +312,11 @@ export async function runApplyMigrations(args: string[]): Promise<void> {
// Run each orchestrator in registry order. An orchestrator failure aborts
// the rest of the chain; fixing the failure and re-running picks up where
// we left off (per-phase idempotency markers + resume from "partial").
//
// Bug 3 — the RUNNER owns the ledger write now. Orchestrators return their
// result; we persist it here with a canonical shape. If the write fails,
// surface the error and DO NOT proceed to the next migration (a silent
// ledger drop was the root cause of the original infinite-retry symptom).
let failed = false;
for (const m of toRun) {
console.log(`\n=== Applying migration v${m.version}: ${m.featurePitch.headline} ===`);
@@ -255,9 +324,45 @@ export async function runApplyMigrations(args: string[]): Promise<void> {
const result = await m.orchestrator(orchestratorOptsFrom(cli));
if (result.status === 'failed') {
console.error(`Migration v${m.version} reported status=failed.`);
// Record the attempt as 'partial' (not 'complete') so the cap counts
// it. Don't let a failed orchestrator look like it never ran.
try {
appendCompletedMigration({
version: m.version,
status: 'partial',
phases: result.phases,
files_rewritten: result.files_rewritten,
autopilot_installed: result.autopilot_installed,
install_target: result.install_target,
apply_migrations_pending: result.pending_host_work ? result.pending_host_work > 0 : undefined,
});
} catch (e) {
console.error(`Also: could not persist failure record: ${e instanceof Error ? e.message : String(e)}`);
}
failed = true;
break;
}
// Persist the terminal outcome. appendCompletedMigration no-ops when
// the last entry for this version is already 'complete' (idempotency
// guard), so repeated clean runs don't spam the ledger.
try {
appendCompletedMigration({
version: m.version,
status: result.status, // 'complete' | 'partial'
phases: result.phases,
files_rewritten: result.files_rewritten,
autopilot_installed: result.autopilot_installed,
install_target: result.install_target,
apply_migrations_pending: result.pending_host_work ? result.pending_host_work > 0 : undefined,
});
} catch (e) {
const msg = e instanceof Error ? e.message : String(e);
console.error(`Failed to persist ledger entry for v${m.version}: ${msg}. Stopping to prevent silent drift.`);
failed = true;
break;
}
if (result.status === 'partial') {
console.log(`Migration v${m.version} finished as PARTIAL. Re-run \`gbrain apply-migrations --yes\` after resolving any pending host-work items.`);
} else {
@@ -266,6 +371,10 @@ export async function runApplyMigrations(args: string[]): Promise<void> {
} catch (e) {
const msg = e instanceof Error ? e.message : String(e);
console.error(`Migration v${m.version} threw: ${msg}`);
// Same partial-on-throw treatment so the cap counts runaway failures.
try {
appendCompletedMigration({ version: m.version, status: 'partial' });
} catch { /* swallow ledger-write failure on throw path */ }
failed = true;
break;
}
src/commands/doctor.ts
+72 -3
View File
@@ -4,6 +4,7 @@ import { LATEST_VERSION } from '../core/migrate.ts';
import { checkResolvable } from '../core/check-resolvable.ts';
import { autoFixDryViolations, type AutoFixReport, type FixOutcome } from '../core/dry-fix.ts';
import { loadCompletedMigrations } from '../core/preferences.ts';
import type { DbUrlSource } from '../core/config.ts';
import { join } from 'path';
import { existsSync, readFileSync, readdirSync } from 'fs';
@@ -18,8 +19,13 @@ export interface Check {
* Run doctor with filesystem-first, DB-second architecture.
* Filesystem checks (resolver, conformance) run without engine.
* DB checks run only if engine is provided.
*
* `dbSource` is passed only from the `--fast` and DB-unavailable paths in
* cli.ts so we can emit a precise "why no DB check" message. When null, the
* user has no DB configured anywhere; otherwise the caller chose --fast or
* we failed to connect despite a configured URL.
*/
export async function runDoctor(engine: BrainEngine | null, args: string[]) {
export async function runDoctor(engine: BrainEngine | null, args: string[], dbSource?: DbUrlSource) {
const jsonOutput = args.includes('--json');
const fastMode = args.includes('--fast');
const doFix = args.includes('--fix');
@@ -136,11 +142,54 @@ export async function runDoctor(engine: BrainEngine | null, args: string[]) {
// Read/parse failure is itself best-effort; skip silently.
}
// 3c. Sync failure trail (Bug 9). sync.ts gates the `sync.last_commit`
// bookmark when per-file parse errors happen, and appends each failure
// to ~/.gbrain/sync-failures.jsonl with the commit hash + exact error.
// Without this doctor check, users see "sync blocked" and have no
// surface showing which files to fix.
try {
const { unacknowledgedSyncFailures, loadSyncFailures } = await import('../core/sync.ts');
const unacked = unacknowledgedSyncFailures();
const all = loadSyncFailures();
if (unacked.length > 0) {
const preview = unacked.slice(0, 3).map(f => `${f.path} (${f.error.slice(0, 60)})`).join('; ');
checks.push({
name: 'sync_failures',
status: 'warn',
message:
`${unacked.length} unacknowledged sync failure(s). ${preview}` +
`${unacked.length > 3 ? `, and ${unacked.length - 3} more` : ''}. ` +
`Fix the file(s) and re-run 'gbrain sync', or use 'gbrain sync --skip-failed' to acknowledge.`,
});
} else if (all.length > 0) {
// Acknowledged-only: informational, not a warning.
checks.push({
name: 'sync_failures',
status: 'ok',
message: `${all.length} historical sync failure(s), all acknowledged.`,
});
}
} catch {
// Best-effort. A broken JSONL should not stop doctor.
}
// --- DB checks (skip if --fast or no engine) ---
if (fastMode || !engine) {
if (!engine) {
checks.push({ name: 'connection', status: 'warn', message: 'No database configured (filesystem checks only)' });
// Pick the precise message. When dbSource is provided, we know
// whether a URL exists (env or config-file) — the caller simply
// skipped the connection. When null, there really is no config
// anywhere.
let msg: string;
if (fastMode && dbSource) {
msg = `Skipping DB checks (--fast mode, URL present from ${dbSource})`;
} else if (!fastMode && dbSource) {
msg = `Could not connect to configured DB (URL from ${dbSource}); filesystem checks only`;
} else {
msg = 'No database configured (filesystem checks only). Set GBRAIN_DATABASE_URL or run `gbrain init`.';
}
checks.push({ name: 'connection', status: 'warn', message: msg });
}
const earlyFail1 = outputResults(checks, jsonOutput);
process.exit(earlyFail1 ? 1 : 0);
@@ -229,7 +278,6 @@ export async function runDoctor(engine: BrainEngine | null, args: string[]) {
}
// 8. Graph health (link + timeline coverage on entity pages).
// dead_links removed in v0.10.1: ON DELETE CASCADE on link FKs makes it always 0.
try {
const health = await engine.getHealth();
const linkPct = ((health.link_coverage ?? 0) * 100).toFixed(0);
@@ -243,6 +291,27 @@ export async function runDoctor(engine: BrainEngine | null, args: string[]) {
message: `Entity link coverage ${linkPct}%, timeline ${timelinePct}%. Run: gbrain link-extract && gbrain timeline-extract`,
});
}
// Bug 11 — brain_score breakdown. When the total is < 100, show which
// components contributed the deficit so users know what to fix.
// Uses distinct *_score field names (not overloading link_coverage /
// timeline_coverage, which are entity-scoped).
if (health.brain_score < 100) {
const parts = [
`embed ${health.embed_coverage_score}/35`,
`links ${health.link_density_score}/25`,
`timeline ${health.timeline_coverage_score}/15`,
`orphans ${health.no_orphans_score}/15`,
`dead-links ${health.no_dead_links_score}/10`,
];
checks.push({
name: 'brain_score',
status: health.brain_score >= 70 ? 'ok' : 'warn',
message: `Brain score ${health.brain_score}/100 (${parts.join(', ')})`,
});
} else {
checks.push({ name: 'brain_score', status: 'ok', message: `Brain score 100/100` });
}
} catch {
checks.push({ name: 'graph_coverage', status: 'warn', message: 'Could not check graph coverage' });
}
src/commands/import.ts
+50 -8
View File
@@ -17,7 +17,16 @@ function defaultWorkers(): number {
return Math.min(byPool, byCpu, byMem);
}
export async function runImport(engine: BrainEngine, args: string[]) {
/** Bug 9 — surface per-file failures so callers (performFullSync) can gate state advances. */
export interface RunImportResult {
imported: number;
skipped: number;
errors: number;
chunksCreated: number;
failures: Array<{ path: string; error: string }>;
}
export async function runImport(engine: BrainEngine, args: string[], opts: { commit?: string } = {}): Promise<RunImportResult> {
const noEmbed = args.includes('--no-embed');
const fresh = args.includes('--fresh');
const jsonOutput = args.includes('--json');
@@ -69,6 +78,7 @@ export async function runImport(engine: BrainEngine, args: string[]) {
let chunksCreated = 0;
const importedSlugs: string[] = [];
const errorCounts: Record<string, number> = {};
const failures: Array<{ path: string; error: string }> = []; // Bug 9
const startTime = Date.now();
function logProgress() {
@@ -91,6 +101,8 @@ export async function runImport(engine: BrainEngine, args: string[]) {
skipped++;
if (result.error && result.error !== 'unchanged') {
console.error(` Skipped ${relativePath}: ${result.error}`);
// Bug 9 — non-"unchanged" skips carry a real error reason.
failures.push({ path: relativePath, error: result.error });
}
}
} catch (e: unknown) {
@@ -104,6 +116,7 @@ export async function runImport(engine: BrainEngine, args: string[]) {
}
errors++;
skipped++;
failures.push({ path: relativePath, error: msg });
}
processed++;
if (processed % 100 === 0 || processed === files.length) {
@@ -135,10 +148,15 @@ export async function runImport(engine: BrainEngine, args: string[]) {
}
} else {
const { PostgresEngine } = await import('../core/postgres-engine.ts');
const { resolvePoolSize } = await import('../core/db.ts');
// Default per-worker pool is 2 (small, parallel import case). Users on
// constrained poolers (e.g. Supabase port 6543) can cap below this via
// GBRAIN_POOL_SIZE=1.
const workerPoolSize = Math.min(2, resolvePoolSize(2));
const workerEngines = await Promise.all(
Array.from({ length: actualWorkers }, async () => {
const eng = new PostgresEngine();
await eng.connect({ database_url: config!.database_url!, poolSize: 2 });
await eng.connect({ database_url: config!.database_url!, poolSize: workerPoolSize });
return eng;
})
);
@@ -198,17 +216,41 @@ export async function runImport(engine: BrainEngine, args: string[]) {
summary: `Imported ${imported} pages, ${skipped} skipped, ${chunksCreated} chunks`,
});
// Import → sync continuity: write sync checkpoint if this is a git repo
// Import → sync continuity: write sync checkpoint if this is a git repo.
// Bug 9 — gate last_commit on "no failures" so import doesn't silently
// stomp on the sync bookmark when parsing broke. We still write
// last_run + repo_path either way (those are progress indicators).
let gitHead: string | null = null;
try {
if (existsSync(join(dir, '.git'))) {
const head = execFileSync('git', ['-C', dir, 'rev-parse', 'HEAD'], { encoding: 'utf-8' }).trim();
await engine.setConfig('sync.last_commit', head);
await engine.setConfig('sync.last_run', new Date().toISOString());
await engine.setConfig('sync.repo_path', dir);
gitHead = execFileSync('git', ['-C', dir, 'rev-parse', 'HEAD'], { encoding: 'utf-8' }).trim();
}
} catch {
// Not a git repo or git not available, skip checkpoint
// Not a git repo or git not available
}
if (gitHead) {
// Record failures into the central JSONL so doctor can surface them.
// Use gitHead as the commit so a later sync can tell "same broken
// state as last time" from "new broken state."
if (failures.length > 0) {
const { recordSyncFailures } = await import('../core/sync.ts');
recordSyncFailures(failures, gitHead);
}
if (failures.length === 0) {
await engine.setConfig('sync.last_commit', gitHead);
} else {
console.error(
`\nImport completed with ${failures.length} failure(s). ` +
`sync.last_commit NOT advanced — re-run 'gbrain sync' to retry, or ` +
`'gbrain sync --skip-failed' to acknowledge and move past them.`,
);
}
await engine.setConfig('sync.last_run', new Date().toISOString());
await engine.setConfig('sync.repo_path', dir);
}
return { imported, skipped, errors, chunksCreated, failures };
}
export function collectMarkdownFiles(dir: string): string[] {
src/commands/init.ts
+81 -75
View File
@@ -107,36 +107,39 @@ async function initPGLite(opts: { jsonOutput: boolean; apiKey: string | null; cu
console.log(`Setting up local brain with PGLite (no server needed)...`);
const engine = await createEngine({ engine: 'pglite' });
await engine.connect({ database_path: dbPath, engine: 'pglite' });
await engine.initSchema();
try {
await engine.connect({ database_path: dbPath, engine: 'pglite' });
await engine.initSchema();
const config: GBrainConfig = {
engine: 'pglite',
database_path: dbPath,
...(opts.apiKey ? { openai_api_key: opts.apiKey } : {}),
};
saveConfig(config);
const config: GBrainConfig = {
engine: 'pglite',
database_path: dbPath,
...(opts.apiKey ? { openai_api_key: opts.apiKey } : {}),
};
saveConfig(config);
const stats = await engine.getStats();
await engine.disconnect();
const stats = await engine.getStats();
if (opts.jsonOutput) {
console.log(JSON.stringify({ status: 'success', engine: 'pglite', path: dbPath, pages: stats.page_count }));
} else {
console.log(`\nBrain ready at ${dbPath}`);
console.log(`${stats.page_count} pages. Engine: PGLite (local Postgres).`);
if (stats.page_count > 0) {
console.log('');
console.log('Existing brain detected. To wire up the v0.10.3 knowledge graph:');
console.log(' gbrain extract links --source db (typed link backfill)');
console.log(' gbrain extract timeline --source db (structured timeline backfill)');
console.log(' gbrain stats (verify links > 0)');
if (opts.jsonOutput) {
console.log(JSON.stringify({ status: 'success', engine: 'pglite', path: dbPath, pages: stats.page_count }));
} else {
console.log('Next: gbrain import <dir>');
console.log(`\nBrain ready at ${dbPath}`);
console.log(`${stats.page_count} pages. Engine: PGLite (local Postgres).`);
if (stats.page_count > 0) {
console.log('');
console.log('Existing brain detected. To wire up the v0.10.3 knowledge graph:');
console.log(' gbrain extract links --source db (typed link backfill)');
console.log(' gbrain extract timeline --source db (structured timeline backfill)');
console.log(' gbrain stats (verify links > 0)');
} else {
console.log('Next: gbrain import <dir>');
}
console.log('');
console.log('When you outgrow local: gbrain migrate --to supabase');
reportModStatus();
}
console.log('');
console.log('When you outgrow local: gbrain migrate --to supabase');
reportModStatus();
} finally {
try { await engine.disconnect(); } catch { /* best-effort */ }
}
}
@@ -157,64 +160,67 @@ async function initPostgres(opts: { databaseUrl: string; jsonOutput: boolean; ap
console.log('Connecting to database...');
const engine = await createEngine({ engine: 'postgres' });
try {
await engine.connect({ database_url: databaseUrl });
} catch (e: unknown) {
const msg = e instanceof Error ? e.message : String(e);
if (databaseUrl.includes('supabase.co') && (msg.includes('ECONNREFUSED') || msg.includes('ETIMEDOUT'))) {
console.error('Connection failed. Supabase direct connections (db.*.supabase.co:5432) are IPv6 only.');
console.error('Use the Session pooler connection string instead (port 6543).');
}
throw e;
}
// Check and auto-create pgvector extension
try {
const conn = (engine as any).sql || (await import('../core/db.ts')).getConnection();
const ext = await conn`SELECT extname FROM pg_extension WHERE extname = 'vector'`;
if (ext.length === 0) {
console.log('pgvector extension not found. Attempting to create...');
try {
await conn`CREATE EXTENSION IF NOT EXISTS vector`;
console.log('pgvector extension created successfully.');
} catch {
console.error('Could not auto-create pgvector extension. Run manually in SQL Editor:');
console.error(' CREATE EXTENSION vector;');
await engine.disconnect();
process.exit(1);
try {
await engine.connect({ database_url: databaseUrl });
} catch (e: unknown) {
const msg = e instanceof Error ? e.message : String(e);
if (databaseUrl.includes('supabase.co') && (msg.includes('ECONNREFUSED') || msg.includes('ETIMEDOUT'))) {
console.error('Connection failed. Supabase direct connections (db.*.supabase.co:5432) are IPv6 only.');
console.error('Use the Session pooler connection string instead (port 6543).');
}
throw e;
}
} catch {
// Non-fatal
}
console.log('Running schema migration...');
await engine.initSchema();
// Check and auto-create pgvector extension
try {
const conn = (engine as any).sql || (await import('../core/db.ts')).getConnection();
const ext = await conn`SELECT extname FROM pg_extension WHERE extname = 'vector'`;
if (ext.length === 0) {
console.log('pgvector extension not found. Attempting to create...');
try {
await conn`CREATE EXTENSION IF NOT EXISTS vector`;
console.log('pgvector extension created successfully.');
} catch {
console.error('Could not auto-create pgvector extension. Run manually in SQL Editor:');
console.error(' CREATE EXTENSION vector;');
// Throw so the outer finally runs engine.disconnect() before we die.
throw new Error('pgvector extension missing');
}
}
} catch {
// Non-fatal
}
const config: GBrainConfig = {
engine: 'postgres',
database_url: databaseUrl,
...(opts.apiKey ? { openai_api_key: opts.apiKey } : {}),
};
saveConfig(config);
console.log('Config saved to ~/.gbrain/config.json');
console.log('Running schema migration...');
await engine.initSchema();
const stats = await engine.getStats();
await engine.disconnect();
const config: GBrainConfig = {
engine: 'postgres',
database_url: databaseUrl,
...(opts.apiKey ? { openai_api_key: opts.apiKey } : {}),
};
saveConfig(config);
console.log('Config saved to ~/.gbrain/config.json');
if (opts.jsonOutput) {
console.log(JSON.stringify({ status: 'success', engine: 'postgres', pages: stats.page_count }));
} else {
console.log(`\nBrain ready. ${stats.page_count} pages. Engine: Postgres (Supabase).`);
if (stats.page_count > 0) {
console.log('');
console.log('Existing brain detected. To wire up the v0.10.3 knowledge graph:');
console.log(' gbrain extract links --source db (typed link backfill)');
console.log(' gbrain extract timeline --source db (structured timeline backfill)');
console.log(' gbrain stats (verify links > 0)');
const stats = await engine.getStats();
if (opts.jsonOutput) {
console.log(JSON.stringify({ status: 'success', engine: 'postgres', pages: stats.page_count }));
} else {
console.log('Next: gbrain import <dir>');
console.log(`\nBrain ready. ${stats.page_count} pages. Engine: Postgres (Supabase).`);
if (stats.page_count > 0) {
console.log('');
console.log('Existing brain detected. To wire up the v0.10.3 knowledge graph:');
console.log(' gbrain extract links --source db (typed link backfill)');
console.log(' gbrain extract timeline --source db (structured timeline backfill)');
console.log(' gbrain stats (verify links > 0)');
} else {
console.log('Next: gbrain import <dir>');
}
reportModStatus();
}
reportModStatus();
} finally {
try { await engine.disconnect(); } catch { /* best-effort */ }
}
}
src/commands/jobs.ts
+16
View File
@@ -511,14 +511,30 @@ export async function registerBuiltinHandlers(worker: MinionWorker, engine: Brai
const steps: Record<string, unknown> = {};
const failed: string[] = [];
// Bug 8 — Between phases, yield to the event loop. The worker's lock
// renewal runs on a timer (src/core/minions/worker.ts); without a
// periodic yield, long CPU-bound phases starve the renewal callback
// and the job gets killed by the stalled-sweeper. A single
// `await new Promise(r => setImmediate(r))` gives the timer a chance
// to fire. The per-phase body is async+await already, so each phase
// internally yields on its own I/O boundaries — this is a belt for
// the gap between phases.
//
// Follow-up (deferred to v0.15): thread ctx.signal / ctx.shutdownSignal
// through each core fn so mid-phase cancellation works on huge brains.
const yieldToLoop = () => new Promise<void>(r => setImmediate(r));
try { steps.sync = await performSync(engine, { repoPath, noEmbed: true }); }
catch (e) { steps.sync = { error: e instanceof Error ? e.message : String(e) }; failed.push('sync'); }
await yieldToLoop();
try { steps.extract = await runExtractCore(engine, { mode: 'all', dir: repoPath }); }
catch (e) { steps.extract = { error: e instanceof Error ? e.message : String(e) }; failed.push('extract'); }
await yieldToLoop();
try { await runEmbedCore(engine, { stale: true }); steps.embed = { embedded: true }; }
catch (e) { steps.embed = { error: e instanceof Error ? e.message : String(e) }; failed.push('embed'); }
await yieldToLoop();
try { steps.backlinks = await runBacklinksCore({ action: 'fix', dir: repoPath }); }
catch (e) { steps.backlinks = { error: e instanceof Error ? e.message : String(e) }; failed.push('backlinks'); }
src/commands/migrations/index.ts
+2
View File
@@ -16,6 +16,7 @@ import { v0_12_0 } from './v0_12_0.ts';
import { v0_12_2 } from './v0_12_2.ts';
import { v0_13_0 } from './v0_13_0.ts';
import { v0_13_1 } from './v0_13_1.ts';
import { v0_14_0 } from './v0_14_0.ts';
export const migrations: Migration[] = [
v0_11_0,
@@ -23,6 +24,7 @@ export const migrations: Migration[] = [
v0_12_2,
v0_13_0,
v0_13_1,
v0_14_0,
];
/** Look up a migration by exact version string. */
src/commands/migrations/v0_11_0.ts
+6 -16
View File
@@ -24,7 +24,8 @@ import { existsSync, readFileSync, writeFileSync, mkdirSync, appendFileSync, lst
import { join, resolve, dirname } from 'path';
import { execSync } from 'child_process';
import type { Migration, OrchestratorOpts, OrchestratorResult, OrchestratorPhaseResult } from './types.ts';
import { savePreferences, loadPreferences, appendCompletedMigration } from '../../core/preferences.ts';
import { savePreferences, loadPreferences } from '../../core/preferences.ts';
// Bug 3 — appendCompletedMigration moved to the runner (apply-migrations.ts).
import { promptLine } from '../../core/cli-util.ts';
import { VERSION } from '../../version.ts';
@@ -441,22 +442,11 @@ async function orchestrator(opts: OrchestratorOpts): Promise<OrchestratorResult>
const f = phaseFInstall(opts);
phases.push(f);
// Phase G: record in completed.jsonl. Status depends on whether any
// host work remains pending AND whether the install phase succeeded.
// Bug 3 — Phase G (record in completed.jsonl) moved to the runner. The
// runner in apply-migrations.ts persists the result after orchestrator
// returns, so we just decide the status here.
const status: 'complete' | 'partial' = (pending_host_work > 0) ? 'partial' : 'complete';
if (!opts.dryRun) {
appendCompletedMigration({
version: '0.11.0',
status,
mode,
files_rewritten,
autopilot_installed: f.status === 'complete',
install_target: undefined, // install target is decided inside autopilot --install
...(status === 'partial' ? { apply_migrations_pending: true } : {}),
});
}
phases.push({ name: 'record', status: opts.dryRun ? 'skipped' : 'complete', detail: `status=${status}` });
phases.push({ name: 'record', status: opts.dryRun ? 'skipped' : 'complete', detail: `status=${status} (ledger write in runner)` });
// Post-run: print pending-host-work summary if anything needs host action.
if (pending_host_work > 0) {
src/commands/migrations/v0_12_0.ts
+2 -8
View File
@@ -32,7 +32,7 @@
import { execSync } from 'child_process';
import type { Migration, OrchestratorOpts, OrchestratorResult, OrchestratorPhaseResult } from './types.ts';
import { appendCompletedMigration } from '../../core/preferences.ts';
// Bug 3 — ledger writes moved to the runner (apply-migrations.ts).
// ── Phase A — Schema ────────────────────────────────────────
@@ -225,13 +225,7 @@ async function orchestrator(opts: OrchestratorOpts): Promise<OrchestratorResult>
}
function finalizeResult(phases: OrchestratorPhaseResult[], status: 'complete' | 'partial' | 'failed'): OrchestratorResult {
if (status !== 'failed') {
try {
appendCompletedMigration({ version: '0.12.0', status: status as 'complete' | 'partial' });
} catch {
// Recording is best-effort.
}
}
// Ledger write lives in the runner now (Bug 3).
return {
version: '0.12.0',
status,
src/commands/migrations/v0_12_2.ts
+2 -8
View File
@@ -22,7 +22,7 @@
import { execSync } from 'child_process';
import type { Migration, OrchestratorOpts, OrchestratorResult, OrchestratorPhaseResult } from './types.ts';
import { appendCompletedMigration } from '../../core/preferences.ts';
// Bug 3 — ledger writes moved to the runner (apply-migrations.ts).
// ── Phase A — Schema ────────────────────────────────────────
@@ -104,13 +104,7 @@ async function orchestrator(opts: OrchestratorOpts): Promise<OrchestratorResult>
}
function finalizeResult(phases: OrchestratorPhaseResult[], status: 'complete' | 'partial' | 'failed'): OrchestratorResult {
if (status !== 'failed') {
try {
appendCompletedMigration({ version: '0.12.2', status: status as 'complete' | 'partial' });
} catch {
// Recording is best-effort.
}
}
// Ledger write lives in the runner now (Bug 3).
return {
version: '0.12.2',
status,
src/commands/migrations/v0_13_0.ts
+3 -8
View File
@@ -27,7 +27,8 @@
import { execSync } from 'child_process';
import type { Migration, OrchestratorOpts, OrchestratorResult, OrchestratorPhaseResult } from './types.ts';
import { appendCompletedMigration } from '../../core/preferences.ts';
// Bug 3 — ledger writes moved to the runner (apply-migrations.ts). The
// orchestrator returns its result and the runner persists it.
// ── Phase A — Schema ────────────────────────────────────────
//
@@ -136,13 +137,7 @@ async function orchestrator(opts: OrchestratorOpts): Promise<OrchestratorResult>
}
function finalizeResult(phases: OrchestratorPhaseResult[], status: 'complete' | 'partial' | 'failed'): OrchestratorResult {
if (status !== 'failed') {
try {
appendCompletedMigration({ version: '0.13.0', status: status as 'complete' | 'partial' });
} catch {
// Recording is best-effort.
}
}
// Ledger write lives in the runner now (Bug 3).
return {
version: '0.13.0',
status,
src/commands/migrations/v0_13_1.ts
+2 -19
View File
@@ -42,7 +42,7 @@ import type { Migration, OrchestratorOpts, OrchestratorResult, OrchestratorPhase
import { loadConfig, toEngineConfig } from '../../core/config.ts';
import { createEngine } from '../../core/engine-factory.ts';
import type { BrainEngine } from '../../core/engine.ts';
import { appendCompletedMigration } from '../../core/preferences.ts';
// Bug 3 — ledger writes moved to the runner (apply-migrations.ts).
const ROLLBACK_DIR = join(homedir(), '.gbrain', 'migrations');
const ROLLBACK_FILE = join(ROLLBACK_DIR, 'v0_13_1-rollback.jsonl');
@@ -233,24 +233,7 @@ async function orchestrator(opts: OrchestratorOpts): Promise<OrchestratorResult>
const anyFailed = phases.some(p => p.status === 'failed');
const status: OrchestratorResult['status'] = anyFailed ? 'partial' : 'complete';
if (!opts.dryRun && status === 'complete') {
try {
appendCompletedMigration({
version: '0.13.1',
completed_at: new Date().toISOString(),
status: 'complete',
phases: phases.map(p => ({ name: p.name, status: p.status })),
files_rewritten: filesRewritten,
});
} catch (e) {
// Recording failure is non-fatal; migration still ran.
phases.push({
name: 'record',
status: 'failed',
detail: e instanceof Error ? e.message : String(e),
});
}
}
// Bug 3 — ledger write lives in the runner now.
return {
version: '0.13.1',
src/commands/migrations/v0_14_0.ts
+180
View File
@@ -0,0 +1,180 @@
/**
* v0.14.0 migration — shell-jobs adoption + autopilot cooperative fix.
*
* Ships two phases:
*
* A. Schema: `ALTER TABLE minion_jobs ALTER COLUMN max_stalled SET DEFAULT 3`.
* New installs already get the bumped default from schema-embedded.ts +
* pglite-schema.ts. This ALTER is for existing brains where the table
* was created under v0.13.x (default 1). Idempotent — running twice is
* a no-op because the default is a table-level attribute, not per-row.
* Existing rows keep their stored max_stalled value; only rows created
* after the ALTER pick up the new default.
*
* B. Pending-host-work ping: emit one entry to
* ~/.gbrain/migrations/pending-host-work.jsonl so the host agent knows
* to read skills/migrations/v0.14.0.md (shell-jobs adoption, autopilot
* cooperative handler wiring, GBRAIN_POOL_SIZE doc). Idempotent — the
* write checks for an existing entry before appending.
*
* Ledger writes live in the runner (Bug 3). This orchestrator returns its
* result; apply-migrations.ts persists.
*/
import { existsSync, readFileSync, mkdirSync, appendFileSync } from 'fs';
import { homedir } from 'os';
import { join } from 'path';
import type { Migration, OrchestratorOpts, OrchestratorResult, OrchestratorPhaseResult } from './types.ts';
import { loadConfig, toEngineConfig } from '../../core/config.ts';
import { createEngine } from '../../core/engine-factory.ts';
import type { BrainEngine } from '../../core/engine.ts';
// Resolve HOME at CALL time, not module-load time — Bun caches os.homedir()
// and ignores later HOME mutations, which breaks test isolation and scripted
// installs. Match the preferences.ts pattern.
function resolveHome(): string { return process.env.HOME || homedir(); }
function pendingHostWorkDir(): string { return join(resolveHome(), '.gbrain', 'migrations'); }
function pendingHostWorkPath(): string { return join(pendingHostWorkDir(), 'pending-host-work.jsonl'); }
// ---------------------------------------------------------------------------
// Phase A — schema: bump minion_jobs.max_stalled default 1 → 3
// ---------------------------------------------------------------------------
async function phaseASchema(opts: OrchestratorOpts): Promise<{ result: OrchestratorPhaseResult; engine: BrainEngine | null }> {
if (opts.dryRun) {
return { result: { name: 'schema', status: 'skipped', detail: 'dry-run' }, engine: null };
}
try {
const config = loadConfig();
if (!config) {
return {
result: { name: 'schema', status: 'skipped', detail: 'no brain configured (run gbrain init first)' },
engine: null,
};
}
const engine = await createEngine(toEngineConfig(config));
await engine.connect(toEngineConfig(config));
try {
// Both Postgres and PGLite accept this ALTER. Idempotent at the
// table level — setting the default to 3 twice is fine.
await engine.executeRaw('ALTER TABLE minion_jobs ALTER COLUMN max_stalled SET DEFAULT 3');
} catch (e) {
// If minion_jobs doesn't exist yet (brand new install), the schema
// file already has the new default, so this is moot. Skip instead of
// fail.
const msg = e instanceof Error ? e.message : String(e);
if (/does not exist|no such table|relation .* does not exist/i.test(msg)) {
return {
result: { name: 'schema', status: 'skipped', detail: 'minion_jobs not yet created (fresh install)' },
engine,
};
}
throw e;
}
return { result: { name: 'schema', status: 'complete' }, engine };
} catch (e) {
return {
result: { name: 'schema', status: 'failed', detail: e instanceof Error ? e.message : String(e) },
engine: null,
};
}
}
// ---------------------------------------------------------------------------
// Phase B — emit pending-host-work entry for the v0.14.0 skill
// ---------------------------------------------------------------------------
interface PendingHostWorkEntry {
migration: string;
ts: string;
skill: string;
reason: string;
}
function existingEntryForVersion(version: string): boolean {
const p = pendingHostWorkPath();
if (!existsSync(p)) return false;
try {
const raw = readFileSync(p, 'utf-8');
for (const line of raw.split('\n')) {
const trimmed = line.trim();
if (!trimmed) continue;
try {
const obj = JSON.parse(trimmed) as PendingHostWorkEntry;
if (obj.migration === version) return true;
} catch { /* skip malformed */ }
}
} catch { /* read error */ }
return false;
}
function phaseBHostWork(opts: OrchestratorOpts): OrchestratorPhaseResult {
if (opts.dryRun) {
return { name: 'host-work', status: 'skipped', detail: 'dry-run' };
}
try {
if (existingEntryForVersion('0.14.0')) {
return { name: 'host-work', status: 'skipped', detail: 'already recorded' };
}
mkdirSync(pendingHostWorkDir(), { recursive: true });
const entry: PendingHostWorkEntry = {
migration: '0.14.0',
ts: new Date().toISOString(),
skill: 'skills/migrations/v0.14.0.md',
reason: 'shell-jobs adoption + autopilot cooperative wiring',
};
appendFileSync(pendingHostWorkPath(), JSON.stringify(entry) + '\n');
return { name: 'host-work', status: 'complete', detail: pendingHostWorkPath() };
} catch (e) {
return { name: 'host-work', status: 'failed', detail: e instanceof Error ? e.message : String(e) };
}
}
// ---------------------------------------------------------------------------
// Orchestrator
// ---------------------------------------------------------------------------
async function orchestrator(opts: OrchestratorOpts): Promise<OrchestratorResult> {
const phases: OrchestratorPhaseResult[] = [];
const { result: schemaRes, engine } = await phaseASchema(opts);
phases.push(schemaRes);
try {
const hostRes = phaseBHostWork(opts);
phases.push(hostRes);
} finally {
if (engine) {
try { await engine.disconnect(); } catch { /* best-effort */ }
}
}
const anyFailed = phases.some(p => p.status === 'failed');
const status: OrchestratorResult['status'] = anyFailed ? 'partial' : 'complete';
return {
version: '0.14.0',
status,
phases,
pending_host_work: phases.some(p => p.name === 'host-work' && p.status === 'complete') ? 1 : 0,
};
}
// ---------------------------------------------------------------------------
// Export
// ---------------------------------------------------------------------------
export const v0_14_0: Migration = {
version: '0.14.0',
featurePitch: {
headline: 'Shell jobs + autopilot cooperative handler + max_stalled default bump.',
description:
'v0.14.0 unlocks `shell` as a Minion job type (gated by GBRAIN_ALLOW_SHELL_JOBS=1 ' +
'on the worker). The autopilot-cycle handler now yields to the event loop ' +
'between phases so lock renewal fires on huge brains. The minion_jobs.max_stalled ' +
'default is bumped 1→3 so one lock-lost tick no longer dead-letters a job. ' +
'Host-specific skill doc: skills/migrations/v0.14.0.md.',
},
orchestrator,
};
src/commands/sync.ts
+105 -4
View File
@@ -3,11 +3,18 @@ import { execFileSync } from 'child_process';
import { join, relative } from 'path';
import type { BrainEngine } from '../core/engine.ts';
import { importFile } from '../core/import-file.ts';
import { buildSyncManifest, isSyncable, pathToSlug } from '../core/sync.ts';
import {
buildSyncManifest,
isSyncable,
pathToSlug,
recordSyncFailures,
unacknowledgedSyncFailures,
acknowledgeSyncFailures,
} from '../core/sync.ts';
import type { SyncManifest } from '../core/sync.ts';
export interface SyncResult {
status: 'up_to_date' | 'synced' | 'first_sync' | 'dry_run';
status: 'up_to_date' | 'synced' | 'first_sync' | 'dry_run' | 'blocked_by_failures';
fromCommit: string | null;
toCommit: string;
added: number;
@@ -16,6 +23,7 @@ export interface SyncResult {
renamed: number;
chunksCreated: number;
pagesAffected: string[];
failedFiles?: number; // count of parse failures (Bug 9)
}
export interface SyncOpts {
@@ -25,6 +33,10 @@ export interface SyncOpts {
noPull?: boolean;
noEmbed?: boolean;
noExtract?: boolean;
/** Bug 9 — acknowledge + skip past current failure set (CLI --skip-failed). */
skipFailed?: boolean;
/** Bug 9 — re-attempt unacknowledged failures explicitly (CLI --retry-failed). */
retryFailed?: boolean;
}
function git(repoPath: string, ...args: string[]): string {
@@ -213,6 +225,7 @@ export async function performSync(engine: BrainEngine, opts: SyncOpts): Promise<
// ep_poll whenever the diff crosses the old > 10 threshold that used to
// trigger the outer wrap. Per-file atomicity is also the right granularity:
// one file's failure should not roll back the others' successful imports.
const failedFiles: Array<{ path: string; error: string; line?: number }> = [];
for (const path of [...filtered.added, ...filtered.modified]) {
const filePath = join(repoPath, path);
if (!existsSync(filePath)) continue;
@@ -221,15 +234,55 @@ export async function performSync(engine: BrainEngine, opts: SyncOpts): Promise<
if (result.status === 'imported') {
chunksCreated += result.chunks;
pagesAffected.push(result.slug);
} else if (result.status === 'skipped' && (result as any).error) {
// importFile returned a non-throw skip with a reason
failedFiles.push({ path, error: String((result as any).error) });
}
} catch (e: unknown) {
const msg = e instanceof Error ? e.message : String(e);
console.error(` Warning: skipped ${path}: ${msg}`);
failedFiles.push({ path, error: msg });
}
}
const elapsed = Date.now() - start;
// Bug 9 — gate the sync bookmark on success. If any per-file parse
// failed, record it to ~/.gbrain/sync-failures.jsonl and DO NOT advance
// sync.last_commit. The next sync re-walks the same diff and re-attempts
// the failed files. Escape hatches: --skip-failed acknowledges the
// current set, --retry-failed re-parses before running the normal sync.
if (failedFiles.length > 0) {
recordSyncFailures(failedFiles, headCommit);
if (!opts.skipFailed) {
console.error(
`\nSync blocked: ${failedFiles.length} file(s) failed to parse. ` +
`Fix the YAML frontmatter in the files above and re-run, or use ` +
`'gbrain sync --skip-failed' to acknowledge and move on.`,
);
// Update last_run + repo_path (progress on infra) but NOT last_commit.
await engine.setConfig('sync.last_run', new Date().toISOString());
await engine.setConfig('sync.repo_path', repoPath);
return {
status: 'blocked_by_failures',
fromCommit: lastCommit,
toCommit: headCommit,
added: filtered.added.length,
modified: filtered.modified.length,
deleted: filtered.deleted.length,
renamed: filtered.renamed.length,
chunksCreated,
pagesAffected,
failedFiles: failedFiles.length,
};
}
// --skip-failed: acknowledge the now-recorded set and proceed.
const acked = acknowledgeSyncFailures();
if (acked > 0) {
console.error(` Acknowledged ${acked} failure(s) and advancing past them.`);
}
}
// Update sync state AFTER all changes succeed
await engine.setConfig('sync.last_commit', headCommit);
await engine.setConfig('sync.last_run', new Date().toISOString());
@@ -288,7 +341,34 @@ async function performFullSync(
const { runImport } = await import('./import.ts');
const importArgs = [repoPath];
if (opts.noEmbed) importArgs.push('--no-embed');
await runImport(engine, importArgs);
const result = await runImport(engine, importArgs, { commit: headCommit });
// Bug 9 — gate the full-sync bookmark on success. runImport already
// writes its own sync.last_commit conditionally (import.ts), but
// performFullSync is called on first-sync + force-full paths where
// the sync module owns the last_commit write. Respect the same gate.
if (result.failures.length > 0) {
recordSyncFailures(result.failures, headCommit);
if (!opts.skipFailed) {
console.error(
`\nFull sync blocked: ${result.failures.length} file(s) failed. ` +
`Fix the YAML in those files and re-run, or use '--skip-failed'.`,
);
await engine.setConfig('sync.last_run', new Date().toISOString());
await engine.setConfig('sync.repo_path', repoPath);
return {
status: 'blocked_by_failures',
fromCommit: null,
toCommit: headCommit,
added: 0, modified: 0, deleted: 0, renamed: 0,
chunksCreated: result.chunksCreated,
pagesAffected: [],
failedFiles: result.failures.length,
};
}
const acked = acknowledgeSyncFailures();
if (acked > 0) console.error(` Acknowledged ${acked} failure(s) and advancing past them.`);
}
// Persist sync state so next sync is incremental (C1 fix: was missing)
await engine.setConfig('sync.last_commit', headCommit);
@@ -322,8 +402,24 @@ export async function runSync(engine: BrainEngine, args: string[]) {
const full = args.includes('--full');
const noPull = args.includes('--no-pull');
const noEmbed = args.includes('--no-embed');
const skipFailed = args.includes('--skip-failed');
const retryFailed = args.includes('--retry-failed');
const opts: SyncOpts = { repoPath, dryRun, full, noPull, noEmbed };
const opts: SyncOpts = { repoPath, dryRun, full, noPull, noEmbed, skipFailed, retryFailed };
// Bug 9 — --retry-failed: before running normal sync, clear acknowledgment
// flags so the sync picks them up as fresh work. The actual re-attempt
// happens inside the regular incremental/full loop because once the commit
// pointer is behind the failures, the diff naturally revisits them.
if (retryFailed) {
const failures = unacknowledgedSyncFailures();
if (failures.length === 0) {
console.log('No unacknowledged sync failures to retry.');
} else {
console.log(`Retrying ${failures.length} previously-failed file(s)...`);
// Don't acknowledge them yet — they must succeed to clear.
}
}
if (!watch) {
const result = await performSync(engine, opts);
@@ -371,5 +467,10 @@ function printSyncResult(result: SyncResult) {
break;
case 'dry_run':
break; // already printed in performSync
case 'blocked_by_failures':
console.log(`Sync BLOCKED at ${result.toCommit.slice(0, 8)}: ${result.failedFiles ?? 0} file(s) failed to parse.`);
console.log(` See ~/.gbrain/sync-failures.jsonl for details, or run 'gbrain doctor'.`);
console.log(` Fix the files then re-run 'gbrain sync', or 'gbrain sync --skip-failed' to move on.`);
break;
}
}
src/core/config.ts
+37 -1
View File
@@ -1,8 +1,24 @@
import { readFileSync, writeFileSync, mkdirSync, chmodSync } from 'fs';
import { readFileSync, writeFileSync, mkdirSync, chmodSync, existsSync } from 'fs';
import { join } from 'path';
import { homedir } from 'os';
import type { EngineConfig } from './types.ts';
/**
* Where is the active DB URL coming from? Pure introspection, no connection
* attempt. Used by `gbrain doctor --fast` so the user gets a precise message
* instead of the misleading "No database configured" when GBRAIN_DATABASE_URL
* (or DATABASE_URL) is actually set.
*
* Precedence matches loadConfig(): env vars win over config-file URL. Returns
* null only when NO source provides a URL at all.
*/
export type DbUrlSource =
| 'env:GBRAIN_DATABASE_URL'
| 'env:DATABASE_URL'
| 'config-file'
| 'config-file-path' // PGLite: config file present, no URL but database_path set
| null;
// Lazy-evaluated to avoid calling homedir() at module scope (breaks in serverless/bundled environments)
function getConfigDir() { return join(homedir(), '.gbrain'); }
function getConfigPath() { return join(getConfigDir(), 'config.json'); }
@@ -70,3 +86,23 @@ export function configDir(): string {
export function configPath(): string {
return join(configDir(), 'config.json');
}
/**
* Introspect where the active DB URL would come from if we tried to connect.
* Never throws, never connects. Env vars take precedence (matches loadConfig).
*/
export function getDbUrlSource(): DbUrlSource {
if (process.env.GBRAIN_DATABASE_URL) return 'env:GBRAIN_DATABASE_URL';
if (process.env.DATABASE_URL) return 'env:DATABASE_URL';
if (!existsSync(configPath())) return null;
try {
const raw = readFileSync(configPath(), 'utf-8');
const parsed = JSON.parse(raw) as Partial<GBrainConfig>;
if (parsed.database_url) return 'config-file';
if (parsed.database_path) return 'config-file-path';
return null;
} catch {
// Config file exists but is unreadable/malformed — treat as null source.
return null;
}
}
src/core/db.ts
+19 -1
View File
@@ -5,6 +5,24 @@ import { SCHEMA_SQL } from './schema-embedded.ts';
let sql: ReturnType<typeof postgres> | null = null;
let connectedUrl: string | null = null;
/**
* Default pool size for Postgres connections. Users on the Supabase transaction
* pooler (port 6543) or any multi-tenant pooler can lower this to avoid
* MaxClients errors when `gbrain upgrade` spawns subprocesses that each open
* their own pool. Set `GBRAIN_POOL_SIZE=2` (or similar) before the command.
*/
const DEFAULT_POOL_SIZE_FALLBACK = 10;
export function resolvePoolSize(explicit?: number): number {
if (typeof explicit === 'number' && explicit > 0) return explicit;
const raw = process.env.GBRAIN_POOL_SIZE;
if (raw) {
const parsed = parseInt(raw, 10);
if (Number.isFinite(parsed) && parsed > 0) return parsed;
}
return DEFAULT_POOL_SIZE_FALLBACK;
}
export function getConnection(): ReturnType<typeof postgres> {
if (!sql) {
throw new GBrainError(
@@ -36,7 +54,7 @@ export async function connect(config: EngineConfig): Promise<void> {
try {
sql = postgres(url, {
max: 10,
max: resolvePoolSize(),
idle_timeout: 20,
connect_timeout: 10,
types: {
src/core/pglite-engine.ts
+22 -5
View File
@@ -481,7 +481,12 @@ export class PGLiteEngine implements BrainEngine {
)
SELECT DISTINCT g.slug, g.title, g.type, g.depth,
coalesce(
(SELECT jsonb_agg(jsonb_build_object('to_slug', p3.slug, 'link_type', l2.link_type))
-- jsonb_agg(DISTINCT ...) collapses duplicate (to_slug, link_type)
-- edges that originate from different provenance (markdown body
-- vs frontmatter vs auto-extracted). Presentation-only dedup;
-- the links table still preserves every provenance row. See
-- plan Bug 6/10.
(SELECT jsonb_agg(DISTINCT jsonb_build_object('to_slug', p3.slug, 'link_type', l2.link_type))
FROM links l2
JOIN pages p3 ON p3.id = l2.to_page_id
WHERE l2.from_page_id = g.id),
@@ -850,6 +855,8 @@ export class PGLiteEngine implements BrainEngine {
(SELECT count(*) FROM pages p
WHERE p.updated_at < (SELECT MAX(te.created_at) FROM timeline_entries te WHERE te.page_id = p.id)
) as stale_pages,
-- Bug 11 — orphan = islanded (no inbound AND no outbound).
-- See BrainHealth.orphan_pages docstring; docs updated to match this.
(SELECT count(*) FROM pages p
WHERE NOT EXISTS (SELECT 1 FROM links l WHERE l.to_page_id = p.id)
AND NOT EXISTS (SELECT 1 FROM links l WHERE l.from_page_id = p.id)
@@ -890,10 +897,14 @@ export class PGLiteEngine implements BrainEngine {
const timelineCoverageDensity = pageCount > 0 ? Math.min(pagesWithTimeline / pageCount, 1) : 0;
const noOrphans = pageCount > 0 ? 1 - (orphanPages / pageCount) : 1;
const noDeadLinks = pageCount > 0 ? 1 - Math.min(deadLinks / pageCount, 1) : 1;
const brainScore = pageCount === 0 ? 0 : Math.round(
(embedCoverage * 0.35 + linkDensity * 0.25 + timelineCoverageDensity * 0.15 +
noOrphans * 0.15 + noDeadLinks * 0.10) * 100
);
// Bug 11 — per-component points. Sum equals brainScore by construction
// so `doctor` can render a breakdown that adds up to the total.
const embedCoverageScore = pageCount === 0 ? 0 : Math.round(embedCoverage * 35);
const linkDensityScore = pageCount === 0 ? 0 : Math.round(linkDensity * 25);
const timelineCoverageScore = pageCount === 0 ? 0 : Math.round(timelineCoverageDensity * 15);
const noOrphansScore = pageCount === 0 ? 0 : Math.round(noOrphans * 15);
const noDeadLinksScore = pageCount === 0 ? 0 : Math.round(noDeadLinks * 10);
const brainScore = embedCoverageScore + linkDensityScore + timelineCoverageScore + noOrphansScore + noDeadLinksScore;
return {
page_count: pageCount,
@@ -902,12 +913,18 @@ export class PGLiteEngine implements BrainEngine {
orphan_pages: orphanPages,
missing_embeddings: Number(r.missing_embeddings),
brain_score: brainScore,
dead_links: deadLinks,
link_coverage: Number(r.link_coverage),
timeline_coverage: Number(r.timeline_coverage),
most_connected: (connected as { slug: string; link_count: number }[]).map(c => ({
slug: c.slug,
link_count: Number(c.link_count),
})),
embed_coverage_score: embedCoverageScore,
link_density_score: linkDensityScore,
timeline_coverage_score: timelineCoverageScore,
no_orphans_score: noOrphansScore,
no_dead_links_score: noDeadLinksScore,
};
}
src/core/pglite-schema.ts
+1 -1
View File
@@ -185,7 +185,7 @@ CREATE TABLE IF NOT EXISTS minion_jobs (
backoff_delay INTEGER NOT NULL DEFAULT 1000,
backoff_jitter REAL NOT NULL DEFAULT 0.2,
stalled_counter INTEGER NOT NULL DEFAULT 0,
max_stalled INTEGER NOT NULL DEFAULT 1,
max_stalled INTEGER NOT NULL DEFAULT 3,
lock_token TEXT,
lock_until TIMESTAMPTZ,
delay_until TIMESTAMPTZ,
src/core/postgres-engine.ts
+33 -11
View File
@@ -31,11 +31,14 @@ export class PostgresEngine implements BrainEngine {
// Lifecycle
async connect(config: EngineConfig & { poolSize?: number }): Promise<void> {
if (config.poolSize) {
// Instance-level connection for worker isolation
// Instance-level connection for worker isolation. resolvePoolSize lets
// GBRAIN_POOL_SIZE cap below the caller's requested size when set — the
// env var is a user escape hatch, so it wins.
const url = config.database_url;
if (!url) throw new GBrainError('No database URL', 'database_url is missing', 'Provide --url');
const size = Math.min(config.poolSize, db.resolvePoolSize(config.poolSize));
this._sql = postgres(url, {
max: config.poolSize,
max: size,
idle_timeout: 20,
connect_timeout: 10,
types: { bigint: postgres.BigInt },
@@ -540,7 +543,14 @@ export class PostgresEngine implements BrainEngine {
)
SELECT DISTINCT g.slug, g.title, g.type, g.depth,
coalesce(
(SELECT jsonb_agg(jsonb_build_object('to_slug', p3.slug, 'link_type', l2.link_type))
-- jsonb_agg(DISTINCT ...) collapses duplicate (to_slug, link_type)
-- edges that originate from different provenance (markdown body
-- vs frontmatter vs auto-extracted). The underlying links table
-- preserves every row with its origin_page_id / link_source —
-- the dedup is presentation-only for the legacy traverseGraph
-- aggregation. traversePaths has its own in-memory dedup at a
-- different layer. See plan Bug 6/10.
(SELECT jsonb_agg(DISTINCT jsonb_build_object('to_slug', p3.slug, 'link_type', l2.link_type))
FROM links l2
JOIN pages p3 ON p3.id = l2.to_page_id
WHERE l2.from_page_id = g.id),
@@ -893,9 +903,12 @@ export class PostgresEngine implements BrainEngine {
async getHealth(): Promise<BrainHealth> {
const sql = this.sql;
// dead_links omitted (always 0 under ON DELETE CASCADE on link FKs).
// orphan_pages now matches PGLite definition: no inbound links (regardless of outbound).
// stale_pages aligned to PGLite definition (page updated_at < latest timeline entry).
// Bug 11 doc-drift fix — orphan_pages means "islanded" (no inbound AND
// no outbound links), aligning both engines with the user-facing
// definition. The type comment previously said "no inbound" but the
// SQL required both — docs now match code so users can trust the
// number. A hub page that links out to many but has no back-references
// is working as intended, not an orphan.
const [h] = await sql`
WITH entity_pages AS (
SELECT id, slug FROM pages WHERE type IN ('person', 'company')
@@ -943,13 +956,16 @@ export class PostgresEngine implements BrainEngine {
// brain_score: 0-100 weighted average
const linkDensity = pageCount > 0 ? Math.min(linkCount / pageCount, 1) : 0;
const timelineCoverage = pageCount > 0 ? Math.min(pagesWithTimeline / pageCount, 1) : 0;
const timelineCoverageWhole = pageCount > 0 ? Math.min(pagesWithTimeline / pageCount, 1) : 0;
const noOrphans = pageCount > 0 ? 1 - (orphanPages / pageCount) : 1;
const noDeadLinks = pageCount > 0 ? 1 - Math.min(deadLinks / pageCount, 1) : 1;
const brainScore = pageCount === 0 ? 0 : Math.round(
(embedCoverage * 0.35 + linkDensity * 0.25 + timelineCoverage * 0.15 +
noOrphans * 0.15 + noDeadLinks * 0.10) * 100
);
// Per-component points. Sum equals brainScore by construction.
const embedCoverageScore = pageCount === 0 ? 0 : Math.round(embedCoverage * 35);
const linkDensityScore = pageCount === 0 ? 0 : Math.round(linkDensity * 25);
const timelineCoverageScore = pageCount === 0 ? 0 : Math.round(timelineCoverageWhole * 15);
const noOrphansScore = pageCount === 0 ? 0 : Math.round(noOrphans * 15);
const noDeadLinksScore = pageCount === 0 ? 0 : Math.round(noDeadLinks * 10);
const brainScore = embedCoverageScore + linkDensityScore + timelineCoverageScore + noOrphansScore + noDeadLinksScore;
return {
page_count: pageCount,
@@ -958,12 +974,18 @@ export class PostgresEngine implements BrainEngine {
orphan_pages: orphanPages,
missing_embeddings: Number(h.missing_embeddings),
brain_score: brainScore,
dead_links: deadLinks,
link_coverage: Number(h.link_coverage),
timeline_coverage: Number(h.timeline_coverage),
most_connected: (connected as { slug: string; link_count: number }[]).map(c => ({
slug: c.slug,
link_count: Number(c.link_count),
})),
embed_coverage_score: embedCoverageScore,
link_density_score: linkDensityScore,
timeline_coverage_score: timelineCoverageScore,
no_orphans_score: noOrphansScore,
no_dead_links_score: noDeadLinksScore,
};
}
src/core/preferences.ts
+26 -3
View File
@@ -33,12 +33,23 @@ export interface Preferences {
export interface CompletedMigrationEntry {
version: string;
ts?: string;
status: 'complete' | 'partial';
/**
* - `complete` — orchestrator finished cleanly. Terminal state; future
* runs no-op this version unless `retry` is appended.
* - `partial` — orchestrator ran but reported missed phases; re-run is
* expected. Attempt cap (3 consecutive partials without a `complete`
* or `retry` between them) triggers the "wedged" skip in the runner.
* - `retry` — explicit reset marker written by `--force-retry`.
* Clears a wedge without faking success; the next upgrade treats the
* version as fresh again.
*/
status: 'complete' | 'partial' | 'retry';
mode?: MinionMode;
files_rewritten?: number;
autopilot_installed?: boolean;
install_target?: string;
apply_migrations_pending?: boolean;
phases?: Array<{ name: string; status: string; detail?: string }>;
[key: string]: unknown;
}
@@ -103,8 +114,20 @@ export function savePreferences(prefs: Preferences): void {
*/
export function appendCompletedMigration(entry: CompletedMigrationEntry): void {
if (!entry.version) throw new Error('appendCompletedMigration: version required');
if (entry.status !== 'complete' && entry.status !== 'partial') {
throw new Error(`appendCompletedMigration: status must be 'complete' or 'partial', got "${entry.status}"`);
if (entry.status !== 'complete' && entry.status !== 'partial' && entry.status !== 'retry') {
throw new Error(`appendCompletedMigration: status must be 'complete', 'partial', or 'retry', got "${entry.status}"`);
}
// Bug 3 — idempotency guard. If the most recent existing entry for this
// version is already 'complete' and we're about to write another
// 'complete', skip. This protects against accidental double-writes
// during the Bug 3 runner-owned-ledger transition (old orchestrator
// code paths and new runner path shouldn't both write).
if (entry.status === 'complete') {
const existing = loadCompletedMigrations();
const prior = existing.filter(e => e.version === entry.version);
if (prior.length > 0 && prior[prior.length - 1].status === 'complete') {
return; // no-op — already terminal
}
}
const full: CompletedMigrationEntry = {
ts: new Date().toISOString(),
src/core/schema-embedded.ts
+1 -1
View File
@@ -280,7 +280,7 @@ CREATE TABLE IF NOT EXISTS minion_jobs (
backoff_delay INTEGER NOT NULL DEFAULT 1000,
backoff_jitter REAL NOT NULL DEFAULT 0.2,
stalled_counter INTEGER NOT NULL DEFAULT 0,
max_stalled INTEGER NOT NULL DEFAULT 1,
max_stalled INTEGER NOT NULL DEFAULT 3,
lock_token TEXT,
lock_until TIMESTAMPTZ,
delay_until TIMESTAMPTZ,
src/core/sync.ts
+124
View File
@@ -133,3 +133,127 @@ export function pathToSlug(filePath: string, repoPrefix?: string): string {
if (repoPrefix) slug = `${repoPrefix}/${slug}`;
return slug.toLowerCase();
}
// ─────────────────────────────────────────────────────────────────
// Sync failure tracking — Bug 9
// ─────────────────────────────────────────────────────────────────
//
// When a sync run catches a per-file parse error (YAML with unquoted
// colons, malformed frontmatter, etc.), we record it here instead of just
// logging and moving on. Three goals:
// 1. Gate the sync.last_commit bookmark advance in all three sync paths
// (incremental, full/runImport, `gbrain import` git continuity).
// 2. Give users a visible record of what failed, with the commit hash
// they can use to re-attempt after fixing the source file.
// 3. Let `gbrain sync --skip-failed` acknowledge a known-bad set so
// repos with many broken files aren't permanently stuck.
import { existsSync as _existsSync, readFileSync as _readFileSync, appendFileSync as _appendFileSync, mkdirSync as _mkdirSync } from 'fs';
import { join as _joinPath } from 'path';
import { homedir as _homedir } from 'os';
import { createHash as _createHash } from 'crypto';
export interface SyncFailure {
path: string;
error: string;
commit: string;
line?: number;
ts: string;
acknowledged?: boolean;
acknowledged_at?: string;
}
function _failuresDir(): string {
return _joinPath(_homedir(), '.gbrain');
}
export function syncFailuresPath(): string {
return _joinPath(_failuresDir(), 'sync-failures.jsonl');
}
function _hashError(msg: string): string {
return _createHash('sha256').update(msg).digest('hex').slice(0, 12);
}
function _dedupKey(f: { path: string; commit: string; error: string }): string {
return `${f.path}|${f.commit}|${_hashError(f.error)}`;
}
/**
* Read the failures JSONL, skipping malformed lines with a warning to stderr.
* Returns empty array if the file doesn't exist.
*/
export function loadSyncFailures(): SyncFailure[] {
const path = syncFailuresPath();
if (!_existsSync(path)) return [];
const raw = _readFileSync(path, 'utf-8');
const out: SyncFailure[] = [];
for (const line of raw.split('\n')) {
const trimmed = line.trim();
if (!trimmed) continue;
try {
out.push(JSON.parse(trimmed) as SyncFailure);
} catch {
console.warn(`[sync-failures] skipping malformed line: ${trimmed.slice(0, 120)}`);
}
}
return out;
}
/**
* Append failure entries to the JSONL. Dedups by (path, commit, error-hash) —
* the same file failing with the same error on the same commit writes ONCE
* to the log, not once per sync run.
*/
export function recordSyncFailures(
failures: Array<{ path: string; error: string; line?: number }>,
commit: string,
): void {
if (failures.length === 0) return;
const existing = loadSyncFailures();
const seen = new Set(existing.map(f => _dedupKey(f)));
_mkdirSync(_failuresDir(), { recursive: true });
const now = new Date().toISOString();
for (const f of failures) {
const entry: SyncFailure = {
path: f.path,
error: f.error,
commit,
line: f.line,
ts: now,
};
if (seen.has(_dedupKey(entry))) continue;
_appendFileSync(syncFailuresPath(), JSON.stringify(entry) + '\n');
seen.add(_dedupKey(entry));
}
}
/**
* Mark all unacknowledged failures as acknowledged. Used by
* `gbrain sync --skip-failed`. Returns the number newly acknowledged.
*
* We do not delete — acknowledged entries stay as historical record so
* doctor can still show them under a "previously skipped" bucket.
*/
export function acknowledgeSyncFailures(): number {
const entries = loadSyncFailures();
if (entries.length === 0) return 0;
const now = new Date().toISOString();
let changed = 0;
const updated = entries.map(e => {
if (e.acknowledged) return e;
changed++;
return { ...e, acknowledged: true, acknowledged_at: now };
});
if (changed === 0) return 0;
_mkdirSync(_failuresDir(), { recursive: true });
const fd = require('fs').writeFileSync;
fd(syncFailuresPath(), updated.map(e => JSON.stringify(e)).join('\n') + '\n');
return changed;
}
/** Return only unacknowledged failures. */
export function unacknowledgedSyncFailures(): SyncFailure[] {
return loadSyncFailures().filter(f => !f.acknowledged);
}
src/core/types.ts
+31 -2
View File
@@ -181,17 +181,46 @@ export interface BrainHealth {
page_count: number;
embed_coverage: number;
stale_pages: number;
/** Pages with zero inbound links. Definition aligned across PGLite and Postgres. */
/**
* Islanded pages — zero inbound AND zero outbound links. A hub page
* that has references out but no back-references is NOT an orphan under
* this definition (it's working as intended as an index). The metric
* aims at "pages I forgot to connect to anything", not the stricter
* graph-theory "no inbound" definition. Both engines share this
* semantics after Bug 11 doc-drift fix.
*/
orphan_pages: number;
missing_embeddings: number;
/** Composite quality score (0-10). Computed from coverage, staleness, orphans. */
/**
* Composite quality score, 0-100. Weighted sum of five components: embed
* coverage, link density, timeline coverage, orphan avoidance, dead-link
* avoidance. See the per-component *_score fields below for breakdown.
*/
brain_score: number;
/**
* Number of links whose to_page_id no longer resolves to a page. Under
* `ON DELETE CASCADE` this is always 0, but malformed data or direct SQL
* DELETEs can produce dangling references.
*/
dead_links: number;
/** Fraction of entity pages (person/company) with >= 1 inbound link. */
link_coverage: number;
/** Fraction of entity pages (person/company) with >= 1 structured timeline entry. */
timeline_coverage: number;
/** Top 5 entities by total link count (in + out). */
most_connected: Array<{ slug: string; link_count: number }>;
/**
* Per-component contribution to brain_score. Sum equals brain_score by
* construction. Displayed by `gbrain doctor` when brain_score < 100.
* Field names are distinct from the entity-scoped link_coverage /
* timeline_coverage above to avoid semantic collision (these reflect
* whole-brain measures used in the score formula).
*/
embed_coverage_score: number; // 0-35
link_density_score: number; // 0-25
timeline_coverage_score: number; // 0-15
no_orphans_score: number; // 0-15
no_dead_links_score: number; // 0-10
}
// Ingest log
src/schema.sql
+1 -1
View File
@@ -276,7 +276,7 @@ CREATE TABLE IF NOT EXISTS minion_jobs (
backoff_delay INTEGER NOT NULL DEFAULT 1000,
backoff_jitter REAL NOT NULL DEFAULT 0.2,
stalled_counter INTEGER NOT NULL DEFAULT 0,
max_stalled INTEGER NOT NULL DEFAULT 1,
max_stalled INTEGER NOT NULL DEFAULT 3,
lock_token TEXT,
lock_until TIMESTAMPTZ,
delay_until TIMESTAMPTZ,