mirror of
https://github.com/garrytan/gbrain.git
synced 2026-07-27 22:15:33 +00:00
* fix(frontmatter): prune vendor dirs at descent + bounded wall-clock with partial-state surfacing Two production-grade fixes for the v0.38.2.0 wave (supersedes PR #1287). Root cause Fix 1 (the bug that hung gbrain doctor on 216K-page brains): both brain-writer.ts:walkDir and frontmatter.ts:collectFiles recursed into every subdirectory without calling pruneDir, the canonical descent-time pruner used by sync/extract/transcript-discovery since v0.35.5.0. On brains that double as code workspaces, the walkers stat'd hundreds of thousands of entries under node_modules / .git / .obsidian / *.raw / ops that isSyncable filtered out at the leaf — paying the IO cost for nothing. Wiring pruneDir at descent (with the v0.37.7.0 #1169 submodule-gitfile check) eliminates the bulk of the wall-clock pain. Fix 2 (codex outside-voice C1): AbortSignal.timeout cannot interrupt the synchronous walker — readdirSync / lstatSync / readFileSync block the event loop, so timer callbacks never fire mid-walk. The load-bearing wall-clock bound is now a deadline check inside scanOneSource's visit callback (Date.now() > opts.deadline). AbortSignal still works at source boundaries. Shape changes (codex C2 + C4): - ScanOpts: + deadline?: number, + dbPageCountForSource hook, + visitDir test seam - PerSourceReport: + status: 'scanned' | 'partial' | 'skipped', + files_scanned, + db_page_count - AuditReport: + partial: boolean, + aborted_at_source: string | null - ok = grandTotal === 0 && !partial (a clean prefix from a timed-out scan no longer falsely reports clean) walkDir + collectFiles now exported with an optional visitDir callback for the regression suite. Production callers don't pass it. Tests: - test/brain-writer-walk-prune.test.ts (new, 12 cases): visitDir-based descent-time pruning assertions for both walkers. Pins the property output-based tests can't catch (isSyncable rejects vendor files at the leaf — so a test checking only output passes under the original bug). - test/brain-writer-partial-scan.test.ts (new, 5 cases): deadline + partial state + ok-after-abort + numerator/denominator coverage. Uses deadline, NOT AbortSignal, since codex C1 proved abort can't interrupt sync. - test/brain-writer.test.ts: existing "abort mid-scan" test refit to the new partial-state contract (per_source has 'skipped' entries instead of being empty — gives doctor visibility into which sources weren't checked). - test/migrations-v0_22_4.test.ts: AuditReport fixture extended with the new required fields. Plan + cross-model review: ~/.claude/plans/system-instruction-you-are-working-hidden-lollipop.md Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(doctor): wire deadline + partial-state into frontmatter_integrity check Adopts the v0.38.2.0 ScanBrainSources surface in doctor's frontmatter_integrity check. - AbortSignal.timeout(fmTimeoutMs) for between-source bound. - deadline = Date.now() + fmTimeoutMs (the load-bearing mid-walk bound — codex C1 caught that AbortSignal alone can't fire inside the sync walker). - GBRAIN_DOCTOR_FM_TIMEOUT_MS env override (default 30000ms; invalid values fall back to default rather than crash). - Per-source DB denominator via SELECT COUNT(*) FROM pages WHERE source_id = $1 AND deleted_at IS NULL (codex C3: deleted_at filter so soft-deleted pages don't inflate the count). - Honest partial-render: "PARTIAL — scanned ~N files (source has ~M pages in DB), K issue(s) so far" instead of "scanned ~N of M pages" (codex C3 — the two populations are overlapping but not identical sets). - "NOT SCANNED (timeout — run gbrain frontmatter validate <id>)" per skipped source so the user knows which sources didn't get checked. - Catch block simplified to "unexpected error only" (codex D4 — the AbortError special case from PR #1287 was unreachable in a sync walker). Tests: test/doctor-frontmatter-partial.test.ts (new, 11 cases) — structural source-grep pins on every load-bearing render string plus the simplified- catch contract. Behavioral coverage is deferred to the heavy script (tests/heavy/frontmatter_scan_wallclock.sh, T6) because runDoctor calls process.exit unconditionally and can't be driven from bun:test directly; refactoring runDoctor to return rather than exit is a separate TODO. Plan + cross-model review: ~/.claude/plans/system-instruction-you-are-working-hidden-lollipop.md Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs: v0.38.2.0 release notes, Phase 2 design sketch, heavy wall-clock smoke - CHANGELOG.md: ELI10-lead-first release entry per CLAUDE.md voice rules. Names the user-visible behavior change, the per-source partial render, the performance numbers table, the "things to watch" caveats. Credits @garrytan-agents for PR #1287's diagnosis. - VERSION + package.json: 0.37.11.0 -> 0.38.2.0. - docs/architecture/frontmatter-scan-incremental.md: Phase 2 design sketch for DB-backed scan state. Schema, migration shape, writer paths (sync-side UPSERT + incremental scan + autopilot cycle phase), doctor reader, sequencing concerns, two-phase rollout plan. Starting point for the follow-up PR — sub-second steady-state doctor needs incremental state, but the schema migration carries its own contract surface (forward-reference bootstrap, schema-drift E2E, PGLite-vs-Postgres parity) that deserves its own focused PR. - tests/heavy/frontmatter_scan_wallclock.sh (new, manual / nightly per tests/heavy/README.md): seeds a synthetic 60K-file brain (10K real + 50K under node_modules/) and asserts gbrain doctor completes in <15s with frontmatter_integrity: ok. Codex C7 caught that the original plan's 1500-file budget was too small to be a meaningful guard — at that scale the test passes BEFORE AND AFTER the fix, proving nothing. 60K is the minimum that catches the descent-into-vendor-trees regression. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix: adversarial review followups — CLI hint, deadline-vs-await race, between-source breadcrumb Codex adversarial review caught 4 real bugs in the v0.38.2.0 wave. All four fixed before ship. #1 (user-facing): `gbrain frontmatter validate` takes a filesystem PATH, not a source id. Pre-fix the NOT SCANNED hint pointed users at `gbrain frontmatter validate src-a` — which would fail with "no such directory", breaking the very remediation this PR ships to give them. Fix: render `src.source_path` instead. #2 (correctness): between sources, `await dbPageCountForSource(src.id)` ran unchecked. A slow query could blow past the deadline, then scanOneSource was still called and returned `status='partial'` with `files_scanned=0` — misleading ("partial scan" when actually zero files were scanned). Fix: add a post-await deadline re-check; mark source + remainder as 'skipped' if the budget already burned. #3 (UX): when the outer-loop deadline check fired BETWEEN sources, `aborted_at_source` stayed null and the doctor message said "PARTIAL SCAN" with no source name. Fix: stamp `aborted_at_source` with the source we were about to start. #4 (correctness): the COUNT query had no per-call deadline. A wedged Postgres pool could make a single COUNT hang past the budget and defeat the wall-clock guarantee. Fix: Promise.race against the remaining deadline; on timeout, resolve null and the post-await re-check (#2) marks the source skipped. Tests: 3 new regression cases in brain-writer-partial-scan.test.ts pinning the fixed contracts (skipped-vs-partial under slow COUNT, hanging COUNT within deadline, aborted_at_source before any source starts). 8648 pass / 0 fail across the full suite. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs(README): refresh production-brain stats — 8.2x pages, 5.6x people, 7.4x companies Pre-update line (months stale): "17,888 pages, 4,383 people, 723 companies, 21 cron jobs running autonomously, built in 12 days." Fresh counts from ~/git/brain (the wintermute production brain): - pages: 17,888 → 146,646 (8.2x) - people: 4,383 → 24,585 (5.6x) - companies: 723 → 5,339 (7.4x) - cron jobs running: 21 → 66 (113 total, 66 enabled per ~/git/wintermute/workspace/ops/cron-snapshot.json) Dropped "built in 12 days" — at 146K pages the initial-velocity claim is stale narrative that no longer matches the current scale story. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
150 lines
5.9 KiB
TypeScript
150 lines
5.9 KiB
TypeScript
import { describe, expect, test } from 'bun:test';
|
|
import { v0_22_4 } from '../src/commands/migrations/v0_22_4.ts';
|
|
import { migrations, getMigration } from '../src/commands/migrations/index.ts';
|
|
|
|
describe('v0.22.4 migration (B11)', () => {
|
|
test('exports a Migration with the right version', () => {
|
|
expect(v0_22_4.version).toBe('0.22.4');
|
|
expect(typeof v0_22_4.orchestrator).toBe('function');
|
|
});
|
|
|
|
test('registered in migrations array in order', () => {
|
|
const versions = migrations.map(m => m.version);
|
|
expect(versions).toContain('0.22.4');
|
|
// v0.22.4 must come after v0.21.0 (semver order is the contract).
|
|
expect(versions.indexOf('0.22.4')).toBeGreaterThan(versions.indexOf('0.21.0'));
|
|
});
|
|
|
|
test('getMigration("0.22.4") returns the same module', () => {
|
|
const found = getMigration('0.22.4');
|
|
expect(found).not.toBeNull();
|
|
expect(found!.version).toBe('0.22.4');
|
|
});
|
|
|
|
test('featurePitch includes a non-trivial headline + description', () => {
|
|
expect(v0_22_4.featurePitch.headline.length).toBeGreaterThan(20);
|
|
expect(v0_22_4.featurePitch.description?.length ?? 0).toBeGreaterThan(50);
|
|
expect(v0_22_4.featurePitch.headline.toLowerCase()).toContain('frontmatter');
|
|
});
|
|
|
|
test('dry-run orchestrator returns complete with all phases skipped', async () => {
|
|
const result = await v0_22_4.orchestrator({
|
|
yes: true,
|
|
dryRun: true,
|
|
noAutopilotInstall: true,
|
|
});
|
|
expect(result.version).toBe('0.22.4');
|
|
expect(result.phases.length).toBe(3);
|
|
for (const p of result.phases) {
|
|
// schema/audit/emit-todo all return 'skipped' on dry-run.
|
|
expect(['skipped', 'complete']).toContain(p.status);
|
|
}
|
|
// Phase A returns 'skipped' on dry-run; B and C also skip. So overall is complete.
|
|
expect(['complete', 'partial']).toContain(result.status);
|
|
});
|
|
|
|
test('phaseASchema is a no-op (returns complete with the no-changes hint)', async () => {
|
|
const { __testing } = await import('../src/commands/migrations/v0_22_4.ts');
|
|
const result = __testing.phaseASchema({ yes: true, dryRun: false, noAutopilotInstall: true });
|
|
expect(result.name).toBe('schema');
|
|
expect(result.status).toBe('complete');
|
|
expect(result.detail).toContain('no schema changes');
|
|
});
|
|
|
|
test('exports paths used for audit + pending-host-work outputs', async () => {
|
|
const { __testing } = await import('../src/commands/migrations/v0_22_4.ts');
|
|
expect(__testing.auditReportPath()).toMatch(/v0\.22\.4-audit\.json$/);
|
|
expect(__testing.pendingHostWorkPath()).toMatch(/pending-host-work\.jsonl$/);
|
|
});
|
|
|
|
test('dotted migration filename references — emit-todo entries point at v0.22.4.md', async () => {
|
|
// The runtime convention is dotted (v0.22.4.md), not underscored.
|
|
// Source-grep guards the contract without spinning up a real audit.
|
|
const fs = await import('fs');
|
|
const src = fs.readFileSync('src/commands/migrations/v0_22_4.ts', 'utf8');
|
|
expect(src).toContain("'skills/migrations/v0.22.4.md'");
|
|
expect(src).not.toMatch(/skills\/migrations\/v0_22_4\.md/);
|
|
});
|
|
|
|
test('phaseCEmitTodo writes per-source entries with the right shape', async () => {
|
|
const { __testing } = await import('../src/commands/migrations/v0_22_4.ts');
|
|
const fs = await import('fs');
|
|
const path = await import('path');
|
|
const os = await import('os');
|
|
const tmpHome = fs.mkdtempSync(path.join(os.tmpdir(), 'gbrain-migration-test-'));
|
|
const origHome = process.env.HOME;
|
|
process.env.HOME = tmpHome;
|
|
try {
|
|
const fakeReport = {
|
|
ok: false,
|
|
total: 12,
|
|
errors_by_code: { NESTED_QUOTES: 8, NULL_BYTES: 4 },
|
|
per_source: [
|
|
{
|
|
source_id: 'wiki',
|
|
source_path: '/tmp/fake-wiki',
|
|
total: 8,
|
|
errors_by_code: { NESTED_QUOTES: 8 },
|
|
sample: [],
|
|
ignoredMissingOpen: 0,
|
|
status: 'scanned' as const,
|
|
files_scanned: 8,
|
|
},
|
|
{
|
|
source_id: 'archive',
|
|
source_path: '/tmp/fake-archive',
|
|
total: 4,
|
|
errors_by_code: { NULL_BYTES: 4 },
|
|
sample: [],
|
|
ignoredMissingOpen: 0,
|
|
status: 'scanned' as const,
|
|
files_scanned: 4,
|
|
},
|
|
{
|
|
source_id: 'clean-source',
|
|
source_path: '/tmp/fake-clean',
|
|
total: 0,
|
|
errors_by_code: {},
|
|
sample: [],
|
|
ignoredMissingOpen: 0,
|
|
status: 'scanned' as const,
|
|
files_scanned: 10,
|
|
},
|
|
],
|
|
scanned_at: new Date().toISOString(),
|
|
partial: false,
|
|
aborted_at_source: null,
|
|
};
|
|
const r = __testing.phaseCEmitTodo(
|
|
{ yes: true, dryRun: false, noAutopilotInstall: true },
|
|
fakeReport,
|
|
);
|
|
expect(r.status).toBe('complete');
|
|
const jsonl = fs.readFileSync(__testing.pendingHostWorkPath(), 'utf8');
|
|
const lines = jsonl.split('\n').filter(Boolean);
|
|
// Two sources had issues; clean-source should NOT produce an entry.
|
|
expect(lines.length).toBe(2);
|
|
const entries = lines.map(l => JSON.parse(l));
|
|
const ids = entries.map(e => e.source_id).sort();
|
|
expect(ids).toEqual(['archive', 'wiki']);
|
|
// Idempotency: re-running emit doesn't duplicate.
|
|
__testing.phaseCEmitTodo(
|
|
{ yes: true, dryRun: false, noAutopilotInstall: true },
|
|
fakeReport,
|
|
);
|
|
const lines2 = fs.readFileSync(__testing.pendingHostWorkPath(), 'utf8').split('\n').filter(Boolean);
|
|
expect(lines2.length).toBe(2);
|
|
// Schema check on entries.
|
|
for (const e of entries) {
|
|
expect(e.migration).toBe('0.22.4');
|
|
expect(e.skill).toBe('skills/migrations/v0.22.4.md');
|
|
expect(e.command).toContain('gbrain frontmatter validate');
|
|
expect(e.command).toContain('--fix');
|
|
}
|
|
} finally {
|
|
process.env.HOME = origHome;
|
|
fs.rmSync(tmpHome, { recursive: true, force: true });
|
|
}
|
|
});
|
|
});
|