mirror of
https://github.com/garrytan/gbrain.git
synced 2026-07-30 19:49:14 +00:00
fix: 8 root-cause fixes from /investigate (v0.14.2) (#259)
* fix: 8 root-cause fixes from /investigate wave
Consolidated bundle of bug fixes from /investigate on the 8 deferred bugs.
Each fix was designed to go at the structural gap, not the symptom. Codex
verified 20 load-bearing claims on the plan; 12 triggered plan revisions.
Bug 2 — GBRAIN_POOL_SIZE env knob + init finally blocks (no auto-detect).
Covers both the singleton pool (db.ts) and instance pool (import.ts:140).
Bug 3 — Centralize migration ledger writes in apply-migrations runner.
Removed appendCompletedMigration from v0_11_0, v0_12_0, v0_12_2,
v0_13_0, v0_13_1. Added 3-partial wedge cap + --force-retry reset.
'complete wins' preserved; no partial can regress a completed migration.
Bug 5 — v0.14.0 migration registered. src/commands/migrations/v0_14_0.ts
ships Phase A (ALTER minion_jobs.max_stalled SET DEFAULT 3) + Phase B
(pending-host-work ping for shell-jobs adoption).
Bug 6/10 — jsonb_agg(DISTINCT ...) in legacy traverseGraph (both engines).
Presentation-level dedup; schema still preserves provenance rows.
Bug 7 — doctor --fast reads DB URL source via getDbUrlSource() in config.ts.
Precise message: 'Skipping DB checks (--fast mode, URL present from env)'
replaces the misleading 'No database configured'.
Bug 8 — max_stalled default bumped 1→3 in schema-embedded.ts, pglite-schema.ts,
schema.sql (new installs). v0_14_0 Phase A ALTER for existing installs.
autopilot-cycle handler yields to event loop between phases so the
worker's lock-renewal timer fires on huge brains. (Deep AbortSignal
threading through runEmbedCore/runExtractCore/runBacklinksCore/performSync
deferred to v0.15 queue polish.)
Bug 9 — Gate sync.last_commit on no-failures across all three sync paths
(incremental, full via runImport, gbrain import git continuity).
recordSyncFailures() helper + ~/.gbrain/sync-failures.jsonl with
dedup key path+commit+error-hash. New flags: --skip-failed (ack) +
--retry-failed (re-attempt). Doctor surfaces unacknowledged failures.
Bug 11 — brain_score breakdown fields on BrainHealth (embed_coverage_score,
link_density_score, timeline_coverage_score, no_orphans_score,
no_dead_links_score); sum equals brain_score by construction.
dead_links now on the type (resolves featuresTeaserForDoctor drift).
orphan_pages kept as 'islanded' (no inbound AND no outbound) and
docs updated to match — explicit semantic instead of doc drift.
New tests: test/traverse-graph-dedup.test.ts, test/sync-failures.test.ts,
test/brain-score-breakdown.test.ts, test/migration-resume.test.ts,
test/migrations-v0_14_0.test.ts. Extended: migrate, doctor, apply-migrations.
All 1696 unit tests pass locally. postgres-jsonb E2E regression unchanged
(none of these touch the JSONB write surface).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* docs: v0.14.2 CHANGELOG + CLAUDE.md; align migration-flow E2E with runner-owned ledger
CHANGELOG: v0.14.2 entry in the standard release-summary format
(two-line headline + lead + numbers table + "what this means" +
"To take advantage of v0.14.2" self-repair block + itemized
changes grouped by reliability / observability / graph correctness /
new migration / tests / deferred-to-v0.15).
CLAUDE.md: new "Key commands added in v0.14.2" section covers
--skip-failed, --retry-failed, --force-retry, GBRAIN_POOL_SIZE env,
and the new doctor checks (sync_failures, brain_score breakdown).
Migration orchestrator docs updated to describe v0_14_0.ts + the
runner-owned ledger contract from Bug 3.
test/e2e/migration-flow.test.ts: three assertions updated to match
the Bug 3 contract — orchestrators no longer append to completed.jsonl
directly, so direct-orchestrator E2E calls leave the ledger empty.
Preferences assertions remain (that's still the orchestrator's side
of the contract). Runner's ledger write is covered by the unit suite
(test/apply-migrations.test.ts + test/migration-resume.test.ts).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 4.6
parent
ebfbd5e6f7
commit
b5fa3d044a
+37
-1
@@ -1,8 +1,24 @@
|
||||
import { readFileSync, writeFileSync, mkdirSync, chmodSync } from 'fs';
|
||||
import { readFileSync, writeFileSync, mkdirSync, chmodSync, existsSync } from 'fs';
|
||||
import { join } from 'path';
|
||||
import { homedir } from 'os';
|
||||
import type { EngineConfig } from './types.ts';
|
||||
|
||||
/**
|
||||
* Where is the active DB URL coming from? Pure introspection, no connection
|
||||
* attempt. Used by `gbrain doctor --fast` so the user gets a precise message
|
||||
* instead of the misleading "No database configured" when GBRAIN_DATABASE_URL
|
||||
* (or DATABASE_URL) is actually set.
|
||||
*
|
||||
* Precedence matches loadConfig(): env vars win over config-file URL. Returns
|
||||
* null only when NO source provides a URL at all.
|
||||
*/
|
||||
export type DbUrlSource =
|
||||
| 'env:GBRAIN_DATABASE_URL'
|
||||
| 'env:DATABASE_URL'
|
||||
| 'config-file'
|
||||
| 'config-file-path' // PGLite: config file present, no URL but database_path set
|
||||
| null;
|
||||
|
||||
// Lazy-evaluated to avoid calling homedir() at module scope (breaks in serverless/bundled environments)
|
||||
function getConfigDir() { return join(homedir(), '.gbrain'); }
|
||||
function getConfigPath() { return join(getConfigDir(), 'config.json'); }
|
||||
@@ -70,3 +86,23 @@ export function configDir(): string {
|
||||
export function configPath(): string {
|
||||
return join(configDir(), 'config.json');
|
||||
}
|
||||
|
||||
/**
|
||||
* Introspect where the active DB URL would come from if we tried to connect.
|
||||
* Never throws, never connects. Env vars take precedence (matches loadConfig).
|
||||
*/
|
||||
export function getDbUrlSource(): DbUrlSource {
|
||||
if (process.env.GBRAIN_DATABASE_URL) return 'env:GBRAIN_DATABASE_URL';
|
||||
if (process.env.DATABASE_URL) return 'env:DATABASE_URL';
|
||||
if (!existsSync(configPath())) return null;
|
||||
try {
|
||||
const raw = readFileSync(configPath(), 'utf-8');
|
||||
const parsed = JSON.parse(raw) as Partial<GBrainConfig>;
|
||||
if (parsed.database_url) return 'config-file';
|
||||
if (parsed.database_path) return 'config-file-path';
|
||||
return null;
|
||||
} catch {
|
||||
// Config file exists but is unreadable/malformed — treat as null source.
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
+19
-1
@@ -5,6 +5,24 @@ import { SCHEMA_SQL } from './schema-embedded.ts';
|
||||
let sql: ReturnType<typeof postgres> | null = null;
|
||||
let connectedUrl: string | null = null;
|
||||
|
||||
/**
|
||||
* Default pool size for Postgres connections. Users on the Supabase transaction
|
||||
* pooler (port 6543) or any multi-tenant pooler can lower this to avoid
|
||||
* MaxClients errors when `gbrain upgrade` spawns subprocesses that each open
|
||||
* their own pool. Set `GBRAIN_POOL_SIZE=2` (or similar) before the command.
|
||||
*/
|
||||
const DEFAULT_POOL_SIZE_FALLBACK = 10;
|
||||
|
||||
export function resolvePoolSize(explicit?: number): number {
|
||||
if (typeof explicit === 'number' && explicit > 0) return explicit;
|
||||
const raw = process.env.GBRAIN_POOL_SIZE;
|
||||
if (raw) {
|
||||
const parsed = parseInt(raw, 10);
|
||||
if (Number.isFinite(parsed) && parsed > 0) return parsed;
|
||||
}
|
||||
return DEFAULT_POOL_SIZE_FALLBACK;
|
||||
}
|
||||
|
||||
export function getConnection(): ReturnType<typeof postgres> {
|
||||
if (!sql) {
|
||||
throw new GBrainError(
|
||||
@@ -36,7 +54,7 @@ export async function connect(config: EngineConfig): Promise<void> {
|
||||
|
||||
try {
|
||||
sql = postgres(url, {
|
||||
max: 10,
|
||||
max: resolvePoolSize(),
|
||||
idle_timeout: 20,
|
||||
connect_timeout: 10,
|
||||
types: {
|
||||
|
||||
@@ -481,7 +481,12 @@ export class PGLiteEngine implements BrainEngine {
|
||||
)
|
||||
SELECT DISTINCT g.slug, g.title, g.type, g.depth,
|
||||
coalesce(
|
||||
(SELECT jsonb_agg(jsonb_build_object('to_slug', p3.slug, 'link_type', l2.link_type))
|
||||
-- jsonb_agg(DISTINCT ...) collapses duplicate (to_slug, link_type)
|
||||
-- edges that originate from different provenance (markdown body
|
||||
-- vs frontmatter vs auto-extracted). Presentation-only dedup;
|
||||
-- the links table still preserves every provenance row. See
|
||||
-- plan Bug 6/10.
|
||||
(SELECT jsonb_agg(DISTINCT jsonb_build_object('to_slug', p3.slug, 'link_type', l2.link_type))
|
||||
FROM links l2
|
||||
JOIN pages p3 ON p3.id = l2.to_page_id
|
||||
WHERE l2.from_page_id = g.id),
|
||||
@@ -850,6 +855,8 @@ export class PGLiteEngine implements BrainEngine {
|
||||
(SELECT count(*) FROM pages p
|
||||
WHERE p.updated_at < (SELECT MAX(te.created_at) FROM timeline_entries te WHERE te.page_id = p.id)
|
||||
) as stale_pages,
|
||||
-- Bug 11 — orphan = islanded (no inbound AND no outbound).
|
||||
-- See BrainHealth.orphan_pages docstring; docs updated to match this.
|
||||
(SELECT count(*) FROM pages p
|
||||
WHERE NOT EXISTS (SELECT 1 FROM links l WHERE l.to_page_id = p.id)
|
||||
AND NOT EXISTS (SELECT 1 FROM links l WHERE l.from_page_id = p.id)
|
||||
@@ -890,10 +897,14 @@ export class PGLiteEngine implements BrainEngine {
|
||||
const timelineCoverageDensity = pageCount > 0 ? Math.min(pagesWithTimeline / pageCount, 1) : 0;
|
||||
const noOrphans = pageCount > 0 ? 1 - (orphanPages / pageCount) : 1;
|
||||
const noDeadLinks = pageCount > 0 ? 1 - Math.min(deadLinks / pageCount, 1) : 1;
|
||||
const brainScore = pageCount === 0 ? 0 : Math.round(
|
||||
(embedCoverage * 0.35 + linkDensity * 0.25 + timelineCoverageDensity * 0.15 +
|
||||
noOrphans * 0.15 + noDeadLinks * 0.10) * 100
|
||||
);
|
||||
// Bug 11 — per-component points. Sum equals brainScore by construction
|
||||
// so `doctor` can render a breakdown that adds up to the total.
|
||||
const embedCoverageScore = pageCount === 0 ? 0 : Math.round(embedCoverage * 35);
|
||||
const linkDensityScore = pageCount === 0 ? 0 : Math.round(linkDensity * 25);
|
||||
const timelineCoverageScore = pageCount === 0 ? 0 : Math.round(timelineCoverageDensity * 15);
|
||||
const noOrphansScore = pageCount === 0 ? 0 : Math.round(noOrphans * 15);
|
||||
const noDeadLinksScore = pageCount === 0 ? 0 : Math.round(noDeadLinks * 10);
|
||||
const brainScore = embedCoverageScore + linkDensityScore + timelineCoverageScore + noOrphansScore + noDeadLinksScore;
|
||||
|
||||
return {
|
||||
page_count: pageCount,
|
||||
@@ -902,12 +913,18 @@ export class PGLiteEngine implements BrainEngine {
|
||||
orphan_pages: orphanPages,
|
||||
missing_embeddings: Number(r.missing_embeddings),
|
||||
brain_score: brainScore,
|
||||
dead_links: deadLinks,
|
||||
link_coverage: Number(r.link_coverage),
|
||||
timeline_coverage: Number(r.timeline_coverage),
|
||||
most_connected: (connected as { slug: string; link_count: number }[]).map(c => ({
|
||||
slug: c.slug,
|
||||
link_count: Number(c.link_count),
|
||||
})),
|
||||
embed_coverage_score: embedCoverageScore,
|
||||
link_density_score: linkDensityScore,
|
||||
timeline_coverage_score: timelineCoverageScore,
|
||||
no_orphans_score: noOrphansScore,
|
||||
no_dead_links_score: noDeadLinksScore,
|
||||
};
|
||||
}
|
||||
|
||||
|
||||
@@ -185,7 +185,7 @@ CREATE TABLE IF NOT EXISTS minion_jobs (
|
||||
backoff_delay INTEGER NOT NULL DEFAULT 1000,
|
||||
backoff_jitter REAL NOT NULL DEFAULT 0.2,
|
||||
stalled_counter INTEGER NOT NULL DEFAULT 0,
|
||||
max_stalled INTEGER NOT NULL DEFAULT 1,
|
||||
max_stalled INTEGER NOT NULL DEFAULT 3,
|
||||
lock_token TEXT,
|
||||
lock_until TIMESTAMPTZ,
|
||||
delay_until TIMESTAMPTZ,
|
||||
|
||||
+33
-11
@@ -31,11 +31,14 @@ export class PostgresEngine implements BrainEngine {
|
||||
// Lifecycle
|
||||
async connect(config: EngineConfig & { poolSize?: number }): Promise<void> {
|
||||
if (config.poolSize) {
|
||||
// Instance-level connection for worker isolation
|
||||
// Instance-level connection for worker isolation. resolvePoolSize lets
|
||||
// GBRAIN_POOL_SIZE cap below the caller's requested size when set — the
|
||||
// env var is a user escape hatch, so it wins.
|
||||
const url = config.database_url;
|
||||
if (!url) throw new GBrainError('No database URL', 'database_url is missing', 'Provide --url');
|
||||
const size = Math.min(config.poolSize, db.resolvePoolSize(config.poolSize));
|
||||
this._sql = postgres(url, {
|
||||
max: config.poolSize,
|
||||
max: size,
|
||||
idle_timeout: 20,
|
||||
connect_timeout: 10,
|
||||
types: { bigint: postgres.BigInt },
|
||||
@@ -540,7 +543,14 @@ export class PostgresEngine implements BrainEngine {
|
||||
)
|
||||
SELECT DISTINCT g.slug, g.title, g.type, g.depth,
|
||||
coalesce(
|
||||
(SELECT jsonb_agg(jsonb_build_object('to_slug', p3.slug, 'link_type', l2.link_type))
|
||||
-- jsonb_agg(DISTINCT ...) collapses duplicate (to_slug, link_type)
|
||||
-- edges that originate from different provenance (markdown body
|
||||
-- vs frontmatter vs auto-extracted). The underlying links table
|
||||
-- preserves every row with its origin_page_id / link_source —
|
||||
-- the dedup is presentation-only for the legacy traverseGraph
|
||||
-- aggregation. traversePaths has its own in-memory dedup at a
|
||||
-- different layer. See plan Bug 6/10.
|
||||
(SELECT jsonb_agg(DISTINCT jsonb_build_object('to_slug', p3.slug, 'link_type', l2.link_type))
|
||||
FROM links l2
|
||||
JOIN pages p3 ON p3.id = l2.to_page_id
|
||||
WHERE l2.from_page_id = g.id),
|
||||
@@ -893,9 +903,12 @@ export class PostgresEngine implements BrainEngine {
|
||||
|
||||
async getHealth(): Promise<BrainHealth> {
|
||||
const sql = this.sql;
|
||||
// dead_links omitted (always 0 under ON DELETE CASCADE on link FKs).
|
||||
// orphan_pages now matches PGLite definition: no inbound links (regardless of outbound).
|
||||
// stale_pages aligned to PGLite definition (page updated_at < latest timeline entry).
|
||||
// Bug 11 doc-drift fix — orphan_pages means "islanded" (no inbound AND
|
||||
// no outbound links), aligning both engines with the user-facing
|
||||
// definition. The type comment previously said "no inbound" but the
|
||||
// SQL required both — docs now match code so users can trust the
|
||||
// number. A hub page that links out to many but has no back-references
|
||||
// is working as intended, not an orphan.
|
||||
const [h] = await sql`
|
||||
WITH entity_pages AS (
|
||||
SELECT id, slug FROM pages WHERE type IN ('person', 'company')
|
||||
@@ -943,13 +956,16 @@ export class PostgresEngine implements BrainEngine {
|
||||
|
||||
// brain_score: 0-100 weighted average
|
||||
const linkDensity = pageCount > 0 ? Math.min(linkCount / pageCount, 1) : 0;
|
||||
const timelineCoverage = pageCount > 0 ? Math.min(pagesWithTimeline / pageCount, 1) : 0;
|
||||
const timelineCoverageWhole = pageCount > 0 ? Math.min(pagesWithTimeline / pageCount, 1) : 0;
|
||||
const noOrphans = pageCount > 0 ? 1 - (orphanPages / pageCount) : 1;
|
||||
const noDeadLinks = pageCount > 0 ? 1 - Math.min(deadLinks / pageCount, 1) : 1;
|
||||
const brainScore = pageCount === 0 ? 0 : Math.round(
|
||||
(embedCoverage * 0.35 + linkDensity * 0.25 + timelineCoverage * 0.15 +
|
||||
noOrphans * 0.15 + noDeadLinks * 0.10) * 100
|
||||
);
|
||||
// Per-component points. Sum equals brainScore by construction.
|
||||
const embedCoverageScore = pageCount === 0 ? 0 : Math.round(embedCoverage * 35);
|
||||
const linkDensityScore = pageCount === 0 ? 0 : Math.round(linkDensity * 25);
|
||||
const timelineCoverageScore = pageCount === 0 ? 0 : Math.round(timelineCoverageWhole * 15);
|
||||
const noOrphansScore = pageCount === 0 ? 0 : Math.round(noOrphans * 15);
|
||||
const noDeadLinksScore = pageCount === 0 ? 0 : Math.round(noDeadLinks * 10);
|
||||
const brainScore = embedCoverageScore + linkDensityScore + timelineCoverageScore + noOrphansScore + noDeadLinksScore;
|
||||
|
||||
return {
|
||||
page_count: pageCount,
|
||||
@@ -958,12 +974,18 @@ export class PostgresEngine implements BrainEngine {
|
||||
orphan_pages: orphanPages,
|
||||
missing_embeddings: Number(h.missing_embeddings),
|
||||
brain_score: brainScore,
|
||||
dead_links: deadLinks,
|
||||
link_coverage: Number(h.link_coverage),
|
||||
timeline_coverage: Number(h.timeline_coverage),
|
||||
most_connected: (connected as { slug: string; link_count: number }[]).map(c => ({
|
||||
slug: c.slug,
|
||||
link_count: Number(c.link_count),
|
||||
})),
|
||||
embed_coverage_score: embedCoverageScore,
|
||||
link_density_score: linkDensityScore,
|
||||
timeline_coverage_score: timelineCoverageScore,
|
||||
no_orphans_score: noOrphansScore,
|
||||
no_dead_links_score: noDeadLinksScore,
|
||||
};
|
||||
}
|
||||
|
||||
|
||||
+26
-3
@@ -33,12 +33,23 @@ export interface Preferences {
|
||||
export interface CompletedMigrationEntry {
|
||||
version: string;
|
||||
ts?: string;
|
||||
status: 'complete' | 'partial';
|
||||
/**
|
||||
* - `complete` — orchestrator finished cleanly. Terminal state; future
|
||||
* runs no-op this version unless `retry` is appended.
|
||||
* - `partial` — orchestrator ran but reported missed phases; re-run is
|
||||
* expected. Attempt cap (3 consecutive partials without a `complete`
|
||||
* or `retry` between them) triggers the "wedged" skip in the runner.
|
||||
* - `retry` — explicit reset marker written by `--force-retry`.
|
||||
* Clears a wedge without faking success; the next upgrade treats the
|
||||
* version as fresh again.
|
||||
*/
|
||||
status: 'complete' | 'partial' | 'retry';
|
||||
mode?: MinionMode;
|
||||
files_rewritten?: number;
|
||||
autopilot_installed?: boolean;
|
||||
install_target?: string;
|
||||
apply_migrations_pending?: boolean;
|
||||
phases?: Array<{ name: string; status: string; detail?: string }>;
|
||||
[key: string]: unknown;
|
||||
}
|
||||
|
||||
@@ -103,8 +114,20 @@ export function savePreferences(prefs: Preferences): void {
|
||||
*/
|
||||
export function appendCompletedMigration(entry: CompletedMigrationEntry): void {
|
||||
if (!entry.version) throw new Error('appendCompletedMigration: version required');
|
||||
if (entry.status !== 'complete' && entry.status !== 'partial') {
|
||||
throw new Error(`appendCompletedMigration: status must be 'complete' or 'partial', got "${entry.status}"`);
|
||||
if (entry.status !== 'complete' && entry.status !== 'partial' && entry.status !== 'retry') {
|
||||
throw new Error(`appendCompletedMigration: status must be 'complete', 'partial', or 'retry', got "${entry.status}"`);
|
||||
}
|
||||
// Bug 3 — idempotency guard. If the most recent existing entry for this
|
||||
// version is already 'complete' and we're about to write another
|
||||
// 'complete', skip. This protects against accidental double-writes
|
||||
// during the Bug 3 runner-owned-ledger transition (old orchestrator
|
||||
// code paths and new runner path shouldn't both write).
|
||||
if (entry.status === 'complete') {
|
||||
const existing = loadCompletedMigrations();
|
||||
const prior = existing.filter(e => e.version === entry.version);
|
||||
if (prior.length > 0 && prior[prior.length - 1].status === 'complete') {
|
||||
return; // no-op — already terminal
|
||||
}
|
||||
}
|
||||
const full: CompletedMigrationEntry = {
|
||||
ts: new Date().toISOString(),
|
||||
|
||||
@@ -280,7 +280,7 @@ CREATE TABLE IF NOT EXISTS minion_jobs (
|
||||
backoff_delay INTEGER NOT NULL DEFAULT 1000,
|
||||
backoff_jitter REAL NOT NULL DEFAULT 0.2,
|
||||
stalled_counter INTEGER NOT NULL DEFAULT 0,
|
||||
max_stalled INTEGER NOT NULL DEFAULT 1,
|
||||
max_stalled INTEGER NOT NULL DEFAULT 3,
|
||||
lock_token TEXT,
|
||||
lock_until TIMESTAMPTZ,
|
||||
delay_until TIMESTAMPTZ,
|
||||
|
||||
@@ -133,3 +133,127 @@ export function pathToSlug(filePath: string, repoPrefix?: string): string {
|
||||
if (repoPrefix) slug = `${repoPrefix}/${slug}`;
|
||||
return slug.toLowerCase();
|
||||
}
|
||||
|
||||
// ─────────────────────────────────────────────────────────────────
|
||||
// Sync failure tracking — Bug 9
|
||||
// ─────────────────────────────────────────────────────────────────
|
||||
//
|
||||
// When a sync run catches a per-file parse error (YAML with unquoted
|
||||
// colons, malformed frontmatter, etc.), we record it here instead of just
|
||||
// logging and moving on. Three goals:
|
||||
// 1. Gate the sync.last_commit bookmark advance in all three sync paths
|
||||
// (incremental, full/runImport, `gbrain import` git continuity).
|
||||
// 2. Give users a visible record of what failed, with the commit hash
|
||||
// they can use to re-attempt after fixing the source file.
|
||||
// 3. Let `gbrain sync --skip-failed` acknowledge a known-bad set so
|
||||
// repos with many broken files aren't permanently stuck.
|
||||
|
||||
import { existsSync as _existsSync, readFileSync as _readFileSync, appendFileSync as _appendFileSync, mkdirSync as _mkdirSync } from 'fs';
|
||||
import { join as _joinPath } from 'path';
|
||||
import { homedir as _homedir } from 'os';
|
||||
import { createHash as _createHash } from 'crypto';
|
||||
|
||||
export interface SyncFailure {
|
||||
path: string;
|
||||
error: string;
|
||||
commit: string;
|
||||
line?: number;
|
||||
ts: string;
|
||||
acknowledged?: boolean;
|
||||
acknowledged_at?: string;
|
||||
}
|
||||
|
||||
function _failuresDir(): string {
|
||||
return _joinPath(_homedir(), '.gbrain');
|
||||
}
|
||||
|
||||
export function syncFailuresPath(): string {
|
||||
return _joinPath(_failuresDir(), 'sync-failures.jsonl');
|
||||
}
|
||||
|
||||
function _hashError(msg: string): string {
|
||||
return _createHash('sha256').update(msg).digest('hex').slice(0, 12);
|
||||
}
|
||||
|
||||
function _dedupKey(f: { path: string; commit: string; error: string }): string {
|
||||
return `${f.path}|${f.commit}|${_hashError(f.error)}`;
|
||||
}
|
||||
|
||||
/**
|
||||
* Read the failures JSONL, skipping malformed lines with a warning to stderr.
|
||||
* Returns empty array if the file doesn't exist.
|
||||
*/
|
||||
export function loadSyncFailures(): SyncFailure[] {
|
||||
const path = syncFailuresPath();
|
||||
if (!_existsSync(path)) return [];
|
||||
const raw = _readFileSync(path, 'utf-8');
|
||||
const out: SyncFailure[] = [];
|
||||
for (const line of raw.split('\n')) {
|
||||
const trimmed = line.trim();
|
||||
if (!trimmed) continue;
|
||||
try {
|
||||
out.push(JSON.parse(trimmed) as SyncFailure);
|
||||
} catch {
|
||||
console.warn(`[sync-failures] skipping malformed line: ${trimmed.slice(0, 120)}`);
|
||||
}
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
/**
|
||||
* Append failure entries to the JSONL. Dedups by (path, commit, error-hash) —
|
||||
* the same file failing with the same error on the same commit writes ONCE
|
||||
* to the log, not once per sync run.
|
||||
*/
|
||||
export function recordSyncFailures(
|
||||
failures: Array<{ path: string; error: string; line?: number }>,
|
||||
commit: string,
|
||||
): void {
|
||||
if (failures.length === 0) return;
|
||||
const existing = loadSyncFailures();
|
||||
const seen = new Set(existing.map(f => _dedupKey(f)));
|
||||
|
||||
_mkdirSync(_failuresDir(), { recursive: true });
|
||||
const now = new Date().toISOString();
|
||||
for (const f of failures) {
|
||||
const entry: SyncFailure = {
|
||||
path: f.path,
|
||||
error: f.error,
|
||||
commit,
|
||||
line: f.line,
|
||||
ts: now,
|
||||
};
|
||||
if (seen.has(_dedupKey(entry))) continue;
|
||||
_appendFileSync(syncFailuresPath(), JSON.stringify(entry) + '\n');
|
||||
seen.add(_dedupKey(entry));
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Mark all unacknowledged failures as acknowledged. Used by
|
||||
* `gbrain sync --skip-failed`. Returns the number newly acknowledged.
|
||||
*
|
||||
* We do not delete — acknowledged entries stay as historical record so
|
||||
* doctor can still show them under a "previously skipped" bucket.
|
||||
*/
|
||||
export function acknowledgeSyncFailures(): number {
|
||||
const entries = loadSyncFailures();
|
||||
if (entries.length === 0) return 0;
|
||||
const now = new Date().toISOString();
|
||||
let changed = 0;
|
||||
const updated = entries.map(e => {
|
||||
if (e.acknowledged) return e;
|
||||
changed++;
|
||||
return { ...e, acknowledged: true, acknowledged_at: now };
|
||||
});
|
||||
if (changed === 0) return 0;
|
||||
_mkdirSync(_failuresDir(), { recursive: true });
|
||||
const fd = require('fs').writeFileSync;
|
||||
fd(syncFailuresPath(), updated.map(e => JSON.stringify(e)).join('\n') + '\n');
|
||||
return changed;
|
||||
}
|
||||
|
||||
/** Return only unacknowledged failures. */
|
||||
export function unacknowledgedSyncFailures(): SyncFailure[] {
|
||||
return loadSyncFailures().filter(f => !f.acknowledged);
|
||||
}
|
||||
|
||||
+31
-2
@@ -181,17 +181,46 @@ export interface BrainHealth {
|
||||
page_count: number;
|
||||
embed_coverage: number;
|
||||
stale_pages: number;
|
||||
/** Pages with zero inbound links. Definition aligned across PGLite and Postgres. */
|
||||
/**
|
||||
* Islanded pages — zero inbound AND zero outbound links. A hub page
|
||||
* that has references out but no back-references is NOT an orphan under
|
||||
* this definition (it's working as intended as an index). The metric
|
||||
* aims at "pages I forgot to connect to anything", not the stricter
|
||||
* graph-theory "no inbound" definition. Both engines share this
|
||||
* semantics after Bug 11 doc-drift fix.
|
||||
*/
|
||||
orphan_pages: number;
|
||||
missing_embeddings: number;
|
||||
/** Composite quality score (0-10). Computed from coverage, staleness, orphans. */
|
||||
/**
|
||||
* Composite quality score, 0-100. Weighted sum of five components: embed
|
||||
* coverage, link density, timeline coverage, orphan avoidance, dead-link
|
||||
* avoidance. See the per-component *_score fields below for breakdown.
|
||||
*/
|
||||
brain_score: number;
|
||||
/**
|
||||
* Number of links whose to_page_id no longer resolves to a page. Under
|
||||
* `ON DELETE CASCADE` this is always 0, but malformed data or direct SQL
|
||||
* DELETEs can produce dangling references.
|
||||
*/
|
||||
dead_links: number;
|
||||
/** Fraction of entity pages (person/company) with >= 1 inbound link. */
|
||||
link_coverage: number;
|
||||
/** Fraction of entity pages (person/company) with >= 1 structured timeline entry. */
|
||||
timeline_coverage: number;
|
||||
/** Top 5 entities by total link count (in + out). */
|
||||
most_connected: Array<{ slug: string; link_count: number }>;
|
||||
/**
|
||||
* Per-component contribution to brain_score. Sum equals brain_score by
|
||||
* construction. Displayed by `gbrain doctor` when brain_score < 100.
|
||||
* Field names are distinct from the entity-scoped link_coverage /
|
||||
* timeline_coverage above to avoid semantic collision (these reflect
|
||||
* whole-brain measures used in the score formula).
|
||||
*/
|
||||
embed_coverage_score: number; // 0-35
|
||||
link_density_score: number; // 0-25
|
||||
timeline_coverage_score: number; // 0-15
|
||||
no_orphans_score: number; // 0-15
|
||||
no_dead_links_score: number; // 0-10
|
||||
}
|
||||
|
||||
// Ingest log
|
||||
|
||||
Reference in New Issue
Block a user