fix(frontmatter): centralize --fix backups under ~/.gbrain/backups/

`gbrain frontmatter validate --fix` and `gbrain frontmatter generate
--fix` wrote `<file>.bak` siblings into the source tree. Users running
gbrain over a brain repo found .bak files scattered through people/,
companies/, etc. that broke gitignore expectations and showed up in
`git status` after every fix pass.

Backups now land under `~/.gbrain/backups/frontmatter/<run-id>/<rel>.bak`
with an iso-week-sorted run-id so a multi-fix session keeps the same
parent directory. Backup directory + per-file structure mirrored from
the original file's relative path. The .bak safety contract is intact
for both git and non-git brain repos.

Also adds `--include-catch-all` opt-in to `frontmatter generate` so the
default catch-all rule (`type: note`) is no longer applied to arbitrary
workspace documents that happen to live under a brain root.

Closes #902. Cherry-picked from PR #903.

Co-Authored-By: 100yenadmin <100yenadmin@users.noreply.github.com>
This commit is contained in:
Garry Tan
2026-05-18 13:32:06 -07:00
co-authored by 100yenadmin
parent f46cac966d
commit bd60cdf68b
7 changed files with 209 additions and 63 deletions
+46 -35
View File
@@ -3,8 +3,9 @@
* *
* Subcommands: * Subcommands:
* gbrain frontmatter validate <path> [--json] [--fix] [--dry-run] * gbrain frontmatter validate <path> [--json] [--fix] [--dry-run]
* Validate one file or recursively a directory. --fix writes .bak then * Validate one file or recursively a directory. --fix writes centralized
* rewrites in place. --dry-run previews without writing. * backups under ~/.gbrain/backups/frontmatter/... then rewrites in place.
* --dry-run previews without writing.
* *
* gbrain frontmatter audit [--source <id>] [--json] * gbrain frontmatter audit [--source <id>] [--json]
* Read-only scan across all registered sources (or one with --source). * Read-only scan across all registered sources (or one with --source).
@@ -14,7 +15,7 @@
* validate. Pass an explicit path to validate a non-source-registered tree. * validate. Pass an explicit path to validate a non-source-registered tree.
*/ */
import { readFileSync, writeFileSync, existsSync, lstatSync, readdirSync, copyFileSync } from 'fs'; import { readFileSync, writeFileSync, existsSync, lstatSync, readdirSync } from 'fs';
import { join, relative, resolve } from 'path'; import { join, relative, resolve } from 'path';
import type { BrainEngine } from '../core/engine.ts'; import type { BrainEngine } from '../core/engine.ts';
import { loadConfig, toEngineConfig } from '../core/config.ts'; import { loadConfig, toEngineConfig } from '../core/config.ts';
@@ -22,6 +23,8 @@ import { createEngine } from '../core/engine-factory.ts';
import { parseMarkdown, type ParseValidationCode } from '../core/markdown.ts'; import { parseMarkdown, type ParseValidationCode } from '../core/markdown.ts';
import { import {
autoFixFrontmatter, autoFixFrontmatter,
createFrontmatterBackup,
makeFrontmatterBackupRunId,
scanBrainSources, scanBrainSources,
type AuditReport, type AuditReport,
type AuditFix, type AuditFix,
@@ -79,7 +82,7 @@ function printHelp() {
Usage: Usage:
gbrain frontmatter validate <path> [--json] [--fix] [--dry-run] gbrain frontmatter validate <path> [--json] [--fix] [--dry-run]
gbrain frontmatter generate <path> [--fix] [--dry-run] [--json] gbrain frontmatter generate <path> [--fix] [--dry-run] [--json] [--include-catch-all]
gbrain frontmatter audit [--source <id>] [--json] gbrain frontmatter audit [--source <id>] [--json]
gbrain frontmatter install-hook [--source <id>] [--force] [--uninstall] gbrain frontmatter install-hook [--source <id>] [--force] [--uninstall]
@@ -90,9 +93,10 @@ validate
NULL_BYTES, NESTED_QUOTES, EMPTY_FRONTMATTER NULL_BYTES, NESTED_QUOTES, EMPTY_FRONTMATTER
--fix Auto-repair the fixable subset (NULL_BYTES, MISSING_CLOSE, --fix Auto-repair the fixable subset (NULL_BYTES, MISSING_CLOSE,
NESTED_QUOTES, SLUG_MISMATCH). Writes <file>.bak before any NESTED_QUOTES, SLUG_MISMATCH). Writes a backup under
in-place rewrite. .bak is the safety contract; works for both ~/.gbrain/backups/frontmatter/... before any in-place rewrite.
git and non-git brain repos. Backups work for both git and non-git brain repos without
littering the source tree.
--dry-run Preview --fix without writing. --dry-run Preview --fix without writing.
--json Emit a JSON envelope on stdout. --json Emit a JSON envelope on stdout.
@@ -102,7 +106,10 @@ generate
the filesystem path and file content. Zero LLM calls, fully deterministic. the filesystem path and file content. Zero LLM calls, fully deterministic.
Without --fix: dry-run preview showing what would be generated. Without --fix: dry-run preview showing what would be generated.
With --fix: writes frontmatter to files (with .bak safety backups). With --fix: writes frontmatter to files with centralized safety backups.
Unknown/catch-all files are skipped by default so GBrain does not stamp
meaningless "type: note" metadata onto arbitrary workspace documents. Pass
--include-catch-all to opt into the legacy catch-all note behavior.
Rules are defined in src/core/frontmatter-inference.ts DIRECTORY_RULES. Rules are defined in src/core/frontmatter-inference.ts DIRECTORY_RULES.
Add new directory conventions by adding rules to the table. Add new directory conventions by adding rules to the table.
@@ -112,9 +119,12 @@ generate
gbrain frontmatter generate /path/to/brain --fix # write all gbrain frontmatter generate /path/to/brain --fix # write all
gbrain frontmatter generate /path/to/brain/people/ --fix # just people/ gbrain frontmatter generate /path/to/brain/people/ --fix # just people/
--fix Write generated frontmatter to files (.bak safety backups). --fix Write generated frontmatter to files with centralized backups.
--dry-run Preview without writing (default when --fix is omitted). --dry-run Preview without writing (default when --fix is omitted).
--json Emit JSON output. --json Emit JSON output.
--include-catch-all
Also write the default catch-all rule ("type: note") for paths
that do not match a more specific directory rule.
audit audit
Read-only scan across all registered sources (or one with --source <id>). Read-only scan across all registered sources (or one with --source <id>).
@@ -140,6 +150,7 @@ interface FileValidation {
path: string; path: string;
errors: { code: ParseValidationCode; message: string; line?: number }[]; errors: { code: ParseValidationCode; message: string; line?: number }[];
fixesApplied?: AuditFix[]; fixesApplied?: AuditFix[];
backupPath?: string;
} }
async function runValidate(rest: string[]): Promise<void> { async function runValidate(rest: string[]): Promise<void> {
@@ -166,6 +177,7 @@ async function runValidate(rest: string[]): Promise<void> {
const files = collectFiles(resolved); const files = collectFiles(resolved);
const results: FileValidation[] = []; const results: FileValidation[] = [];
const backupRunId = makeFrontmatterBackupRunId();
for (const file of files) { for (const file of files) {
const content = readFileSync(file, 'utf8'); const content = readFileSync(file, 'utf8');
@@ -181,7 +193,7 @@ async function runValidate(rest: string[]): Promise<void> {
const { content: fixed, fixes } = autoFixFrontmatter(content, { filePath: file }); const { content: fixed, fixes } = autoFixFrontmatter(content, { filePath: file });
result.fixesApplied = fixes; result.fixesApplied = fixes;
if (fixes.length > 0 && !flags.dryRun) { if (fixes.length > 0 && !flags.dryRun) {
copyFileSync(file, file + '.bak'); result.backupPath = createFrontmatterBackup(file, { sourcePath: resolved, runId: backupRunId });
writeFileSync(file, fixed, 'utf8'); writeFileSync(file, fixed, 'utf8');
} }
} }
@@ -225,7 +237,7 @@ async function runValidate(rest: string[]): Promise<void> {
} }
} }
if (flags.fix && !flags.dryRun) { if (flags.fix && !flags.dryRun) {
console.log(`\nWrote .bak backups for ${filesFixed} file(s).`); console.log(`\nWrote centralized backups for ${filesFixed} file(s) under ~/.gbrain/backups/frontmatter/.`);
} }
} }
} }
@@ -299,7 +311,10 @@ function printAuditHumanReport(report: AuditReport): void {
console.log('No registered sources to audit. Run `gbrain sources list` to inspect.'); console.log('No registered sources to audit. Run `gbrain sources list` to inspect.');
return; return;
} }
console.log(`Frontmatter audit — ${report.total} issue(s) across ${report.per_source.length} source(s) (scanned at ${report.scanned_at})`); console.log(`Frontmatter audit — ${report.total} malformed issue(s) across ${report.per_source.length} source(s) (scanned at ${report.scanned_at})`);
if (report.ignored_missing_open) {
console.log(`Missing frontmatter ignored: ${report.ignored_missing_open} file(s). Use \`frontmatter validate\` for strict per-file checks or \`frontmatter generate\` to add meaningful metadata.`);
}
for (const src of report.per_source) { for (const src of report.per_source) {
console.log(`\n[${src.source_id}] ${src.source_path}`); console.log(`\n[${src.source_id}] ${src.source_path}`);
if (src.total === 0) { if (src.total === 0) {
@@ -332,6 +347,7 @@ async function runGenerate(args: string[]): Promise<void> {
const doFix = args.includes('--fix'); const doFix = args.includes('--fix');
const dryRun = args.includes('--dry-run'); const dryRun = args.includes('--dry-run');
const jsonOut = args.includes('--json'); const jsonOut = args.includes('--json');
const includeCatchAll = args.includes('--include-catch-all') || args.includes('--allow-catch-all');
if (!targetPath) { if (!targetPath) {
console.error('error: gbrain frontmatter generate requires a <path> argument'); console.error('error: gbrain frontmatter generate requires a <path> argument');
@@ -342,7 +358,7 @@ async function runGenerate(args: string[]): Promise<void> {
const { inferFrontmatter, serializeFrontmatter } = await import('../core/frontmatter-inference.ts'); const { inferFrontmatter, serializeFrontmatter } = await import('../core/frontmatter-inference.ts');
const { resolve, relative, join, basename } = await import('path'); const { resolve, relative, join, basename } = await import('path');
const { readFileSync, writeFileSync, copyFileSync, statSync, readdirSync, lstatSync } = await import('fs'); const { readFileSync, writeFileSync, statSync, lstatSync } = await import('fs');
const rootPath = resolve(targetPath); const rootPath = resolve(targetPath);
const isDir = statSync(rootPath).isDirectory(); const isDir = statSync(rootPath).isDirectory();
@@ -376,12 +392,14 @@ async function runGenerate(args: string[]): Promise<void> {
const results: GenerateResult[] = []; const results: GenerateResult[] = [];
let scanned = 0; let scanned = 0;
let skipped = 0; let skipped = 0;
let skippedCatchAll = 0;
let generated = 0; let generated = 0;
let written = 0; let written = 0;
const backupRunId = makeFrontmatterBackupRunId();
function processFile(absPath: string, relPath: string) { function processFile(absPath: string, relPath: string) {
scanned++; scanned++;
if (!absPath.endsWith('.md')) return; if (!isSyncable(relPath, { strategy: 'markdown' })) return;
// Skip symlinks // Skip symlinks
try { if (lstatSync(absPath).isSymbolicLink()) return; } catch { return; } try { if (lstatSync(absPath).isSymbolicLink()) return; } catch { return; }
@@ -394,6 +412,10 @@ async function runGenerate(args: string[]): Promise<void> {
skipped++; skipped++;
return; return;
} }
if (!includeCatchAll && inferred.matchedRule === '(default)') {
skippedCatchAll++;
return;
}
generated++; generated++;
results.push({ results.push({
@@ -407,32 +429,17 @@ async function runGenerate(args: string[]): Promise<void> {
if (doFix && !dryRun) { if (doFix && !dryRun) {
const fm = serializeFrontmatter(inferred); const fm = serializeFrontmatter(inferred);
const newContent = fm + '\n' + content; const newContent = fm + '\n' + content;
// Safety: write .bak first // Safety: write a centralized backup first.
copyFileSync(absPath, absPath + '.bak'); createFrontmatterBackup(absPath, { sourcePath: brainRoot, runId: backupRunId });
writeFileSync(absPath, newContent, 'utf-8'); writeFileSync(absPath, newContent, 'utf-8');
written++; written++;
} }
} }
function walkDir(dir: string, rootForRel: string) {
let entries: string[];
try { entries = readdirSync(dir); } catch { return; }
for (const entry of entries) {
if (entry === '.git' || entry === 'node_modules' || entry === '.obsidian') continue;
const abs = join(dir, entry);
try {
const stat = statSync(abs);
if (stat.isDirectory()) {
walkDir(abs, rootForRel);
} else if (stat.isFile() && entry.endsWith('.md')) {
processFile(abs, relative(rootForRel, abs));
}
} catch { /* skip unreadable */ }
}
}
if (isDir) { if (isDir) {
walkDir(rootPath, brainRoot); for (const absPath of collectFiles(rootPath)) {
processFile(absPath, relative(brainRoot, absPath));
}
} else { } else {
const relPath = relative(brainRoot, rootPath) || basename(rootPath); const relPath = relative(brainRoot, rootPath) || basename(rootPath);
processFile(rootPath, relPath); processFile(rootPath, relPath);
@@ -443,6 +450,7 @@ async function runGenerate(args: string[]): Promise<void> {
console.log(JSON.stringify({ console.log(JSON.stringify({
scanned, scanned,
skipped, skipped,
skippedCatchAll,
generated, generated,
written, written,
dryRun: !doFix || dryRun, dryRun: !doFix || dryRun,
@@ -457,9 +465,12 @@ async function runGenerate(args: string[]): Promise<void> {
console.log(`\nFrontmatter generation (${mode})`); console.log(`\nFrontmatter generation (${mode})`);
console.log(` Scanned: ${scanned} files`); console.log(` Scanned: ${scanned} files`);
console.log(` Already have frontmatter: ${skipped}`); console.log(` Already have frontmatter: ${skipped}`);
if (skippedCatchAll > 0) {
console.log(` Skipped catch-all/unknown: ${skippedCatchAll} (pass --include-catch-all to write type: note)`);
}
console.log(` Would generate: ${generated}`); console.log(` Would generate: ${generated}`);
if (doFix && !dryRun) { if (doFix && !dryRun) {
console.log(` Written: ${written} (with .bak backups)`); console.log(` Written: ${written} (with centralized backups)`);
} }
// Show sample by type // Show sample by type
+6 -3
View File
@@ -10,7 +10,9 @@
* one entry per source-with-issues to ~/.gbrain/migrations/pending-host-work.jsonl. * one entry per source-with-issues to ~/.gbrain/migrations/pending-host-work.jsonl.
* It NEVER mutates brain content. The agent reads skills/migrations/v0.22.4.md * It NEVER mutates brain content. The agent reads skills/migrations/v0.22.4.md
* after upgrade and runs `gbrain frontmatter validate <source-path> --fix` with * after upgrade and runs `gbrain frontmatter validate <source-path> --fix` with
* explicit user consent. * explicit user consent. Missing frontmatter is not queued by default; plain
* Markdown imports are valid, and generated frontmatter should add real signal
* rather than blanket `type: note`.
* *
* Phases (all idempotent): * Phases (all idempotent):
* A. Schema — no-op (no DB changes in v0.22.4). * A. Schema — no-op (no DB changes in v0.22.4).
@@ -212,8 +214,9 @@ export const v0_22_4: Migration = {
'pre-commit hook helper, and a new frontmatter-guard skill. The migration is audit-only ' + 'pre-commit hook helper, and a new frontmatter-guard skill. The migration is audit-only ' +
'(it never mutates your brain) — it scans every registered source, writes a per-source ' + '(it never mutates your brain) — it scans every registered source, writes a per-source ' +
'report to ~/.gbrain/migrations/v0.22.4-audit.json, and queues a TODO with the exact fix ' + 'report to ~/.gbrain/migrations/v0.22.4-audit.json, and queues a TODO with the exact fix ' +
'command. Run `gbrain frontmatter validate <source-path> --fix` to repair (creates .bak ' + 'command for malformed frontmatter only. Missing frontmatter is treated as optional metadata ' +
'backups). Resolves all 7 check-resolvable warnings on master; ships frontmatter-guard.', 'coverage for broad document sources. Run `gbrain frontmatter validate <source-path> --fix` ' +
'to repair (creates centralized backups under ~/.gbrain/backups/frontmatter). Ships frontmatter-guard.',
}, },
orchestrator, orchestrator,
}; };
+75 -14
View File
@@ -9,15 +9,19 @@
* validation stack. * validation stack.
* *
* Path-guard contract: writeBrainPage refuses to write outside the source * Path-guard contract: writeBrainPage refuses to write outside the source
* path. .bak backups are the safety contract (works for both git and non-git * path. Pre-write backups are the safety contract (works for both git and
* brain repos; the existing src/core/dry-fix.ts:getWorkingTreeStatus rejects * non-git brain repos; the existing src/core/dry-fix.ts:getWorkingTreeStatus
* non-git repos as unsafe, which is the wrong shape for brain rewrites). * rejects non-git repos as unsafe, which is the wrong shape for brain
* rewrites). Backups live under ~/.gbrain/backups/frontmatter/... instead of
* beside source files so bulk repair never litters the user's workspace.
*/ */
import { existsSync, readFileSync, readdirSync, statSync, copyFileSync, writeFileSync, mkdirSync, lstatSync } from 'fs'; import { createHash } from 'crypto';
import { join, relative, resolve, dirname } from 'path'; import { existsSync, readFileSync, readdirSync, copyFileSync, writeFileSync, mkdirSync, lstatSync } from 'fs';
import { join, relative, resolve, dirname, basename, isAbsolute } from 'path';
import type { BrainEngine } from './engine.ts'; import type { BrainEngine } from './engine.ts';
import type { ProgressReporter } from './progress.ts'; import type { ProgressReporter } from './progress.ts';
import { gbrainPath } from './config.ts';
import { import {
parseMarkdown, parseMarkdown,
type ParseValidationCode, type ParseValidationCode,
@@ -38,6 +42,7 @@ export interface PerSourceReport {
total: number; total: number;
errors_by_code: Partial<Record<ParseValidationCode, number>>; errors_by_code: Partial<Record<ParseValidationCode, number>>;
sample: { path: string; codes: ParseValidationCode[] }[]; sample: { path: string; codes: ParseValidationCode[] }[];
ignoredMissingOpen: number;
} }
export interface AuditReport { export interface AuditReport {
@@ -46,10 +51,45 @@ export interface AuditReport {
errors_by_code: Partial<Record<ParseValidationCode, number>>; errors_by_code: Partial<Record<ParseValidationCode, number>>;
per_source: PerSourceReport[]; per_source: PerSourceReport[];
scanned_at: string; scanned_at: string;
ignored_missing_open?: number;
} }
const SAMPLE_PER_SOURCE = 20; const SAMPLE_PER_SOURCE = 20;
// ---------------------------------------------------------------------------
// Frontmatter backups
// ---------------------------------------------------------------------------
export function makeFrontmatterBackupRunId(date = new Date()): string {
return date.toISOString().replace(/[:.]/g, '-');
}
export interface FrontmatterBackupOpts {
sourcePath?: string;
backupRoot?: string;
runId?: string;
}
function sourceKey(sourcePath: string): string {
return createHash('sha256').update(resolve(sourcePath)).digest('hex').slice(0, 12);
}
export function defaultFrontmatterBackupRoot(runId = makeFrontmatterBackupRunId()): string {
return gbrainPath('backups', 'frontmatter', runId);
}
export function createFrontmatterBackup(filePath: string, opts: FrontmatterBackupOpts = {}): string {
const resolvedFile = resolve(filePath);
const resolvedSource = resolve(opts.sourcePath ?? dirname(resolvedFile));
const rel = relative(resolvedSource, resolvedFile);
const safeRel = rel && !rel.startsWith('..') && !isAbsolute(rel) ? rel : basename(resolvedFile);
const root = opts.backupRoot ?? defaultFrontmatterBackupRoot(opts.runId);
const backupPath = join(root, sourceKey(resolvedSource), safeRel + '.bak');
mkdirSync(dirname(backupPath), { recursive: true });
copyFileSync(resolvedFile, backupPath);
return backupPath;
}
// --------------------------------------------------------------------------- // ---------------------------------------------------------------------------
// autoFixFrontmatter // autoFixFrontmatter
// --------------------------------------------------------------------------- // ---------------------------------------------------------------------------
@@ -178,7 +218,7 @@ export function autoFixFrontmatter(
} }
// --------------------------------------------------------------------------- // ---------------------------------------------------------------------------
// writeBrainPage — path-guarded write with .bak backup // writeBrainPage — path-guarded write with centralized backup
// --------------------------------------------------------------------------- // ---------------------------------------------------------------------------
export class BrainWriterError extends Error { export class BrainWriterError extends Error {
@@ -193,15 +233,16 @@ export class BrainWriterError extends Error {
} }
/** /**
* Path-guarded brain page writer. Always writes `<filePath>.bak` before any * Path-guarded brain page writer. Always writes a backup under
* in-place mutation (the contract that replaces git-tree-clean for non-git * ~/.gbrain/backups/frontmatter/... before any in-place mutation (the contract
* brain repos). Throws BrainWriterError if filePath is not under sourcePath. * that replaces git-tree-clean for non-git brain repos). Throws
* BrainWriterError if filePath is not under sourcePath.
*/ */
export function writeBrainPage( export function writeBrainPage(
filePath: string, filePath: string,
content: string, content: string,
opts: { sourcePath: string; autoFix?: boolean }, opts: { sourcePath: string; autoFix?: boolean; backupRoot?: string; backupRunId?: string },
): { fixes: AuditFix[] } { ): { fixes: AuditFix[]; backupPath?: string } {
const resolvedSource = resolve(opts.sourcePath); const resolvedSource = resolve(opts.sourcePath);
const resolvedTarget = resolve(filePath); const resolvedTarget = resolve(filePath);
if (resolvedTarget !== resolvedSource && !resolvedTarget.startsWith(resolvedSource + '/')) { if (resolvedTarget !== resolvedSource && !resolvedTarget.startsWith(resolvedSource + '/')) {
@@ -220,13 +261,18 @@ export function writeBrainPage(
fixes = result.fixes; fixes = result.fixes;
} }
let backupPath: string | undefined;
if (existsSync(filePath)) { if (existsSync(filePath)) {
copyFileSync(filePath, filePath + '.bak'); backupPath = createFrontmatterBackup(filePath, {
sourcePath: opts.sourcePath,
backupRoot: opts.backupRoot,
runId: opts.backupRunId,
});
} else { } else {
mkdirSync(dirname(filePath), { recursive: true }); mkdirSync(dirname(filePath), { recursive: true });
} }
writeFileSync(filePath, toWrite, 'utf8'); writeFileSync(filePath, toWrite, 'utf8');
return { fixes }; return { fixes, backupPath };
} }
// --------------------------------------------------------------------------- // ---------------------------------------------------------------------------
@@ -242,6 +288,10 @@ export interface ScanOpts {
/** Limit scan to one source. When omitted, all registered sources with a /** Limit scan to one source. When omitted, all registered sources with a
* local_path are scanned. */ * local_path are scanned. */
sourceId?: string; sourceId?: string;
/** Missing frontmatter is optional metadata coverage for broad document
* sources. Set true for curated page repos that require every file to carry
* YAML frontmatter. */
strictMissingOpen?: boolean;
onProgress?: ProgressReporter; onProgress?: ProgressReporter;
signal?: AbortSignal; signal?: AbortSignal;
} }
@@ -254,6 +304,7 @@ export async function scanBrainSources(
const totals: Partial<Record<ParseValidationCode, number>> = {}; const totals: Partial<Record<ParseValidationCode, number>> = {};
const perSource: PerSourceReport[] = []; const perSource: PerSourceReport[] = [];
let grandTotal = 0; let grandTotal = 0;
let ignoredMissingOpen = 0;
for (const src of sources) { for (const src of sources) {
if (opts.signal?.aborted) break; if (opts.signal?.aborted) break;
@@ -267,12 +318,14 @@ export async function scanBrainSources(
total: 0, total: 0,
errors_by_code: {}, errors_by_code: {},
sample: [], sample: [],
ignoredMissingOpen: 0,
}); });
continue; continue;
} }
const report = scanOneSource(src.id, src.local_path, opts); const report = scanOneSource(src.id, src.local_path, opts);
perSource.push(report); perSource.push(report);
grandTotal += report.total; grandTotal += report.total;
ignoredMissingOpen += report.ignoredMissingOpen;
for (const [code, n] of Object.entries(report.errors_by_code)) { for (const [code, n] of Object.entries(report.errors_by_code)) {
const k = code as ParseValidationCode; const k = code as ParseValidationCode;
totals[k] = (totals[k] ?? 0) + (n as number); totals[k] = (totals[k] ?? 0) + (n as number);
@@ -285,6 +338,7 @@ export async function scanBrainSources(
errors_by_code: totals, errors_by_code: totals,
per_source: perSource, per_source: perSource,
scanned_at: new Date().toISOString(), scanned_at: new Date().toISOString(),
ignored_missing_open: ignoredMissingOpen || undefined,
}; };
} }
@@ -298,6 +352,7 @@ function scanOneSource(
const rootResolved = resolve(sourcePath); const rootResolved = resolve(sourcePath);
let scanned = 0; let scanned = 0;
let total = 0; let total = 0;
let ignoredMissingOpen = 0;
walkDir(rootResolved, (absPath) => { walkDir(rootResolved, (absPath) => {
if (opts.signal?.aborted) return false; if (opts.signal?.aborted) return false;
@@ -312,7 +367,12 @@ function scanOneSource(
} }
const expectedSlug = slugifyPath(relPath); const expectedSlug = slugifyPath(relPath);
const parsed = parseMarkdown(content, relPath, { validate: true, expectedSlug }); const parsed = parseMarkdown(content, relPath, { validate: true, expectedSlug });
const errs = parsed.errors ?? []; const errs = (parsed.errors ?? []).filter((e) => {
if (e.code !== 'MISSING_OPEN') return true;
if (opts.strictMissingOpen) return true;
ignoredMissingOpen++;
return false;
});
if (errs.length > 0) { if (errs.length > 0) {
total += errs.length; total += errs.length;
const codes: ParseValidationCode[] = []; const codes: ParseValidationCode[] = [];
@@ -340,6 +400,7 @@ function scanOneSource(
total, total,
errors_by_code: errorsByCode, errors_by_code: errorsByCode,
sample, sample,
ignoredMissingOpen,
}; };
} }
+21 -2
View File
@@ -28,7 +28,8 @@
* - **Deterministic.** Same file always produces the same frontmatter. No LLM calls, no network. * - **Deterministic.** Same file always produces the same frontmatter. No LLM calls, no network.
* - **Extensible via rules.** The `DIRECTORY_RULES` table maps path patterns to type + source + tags. * - **Extensible via rules.** The `DIRECTORY_RULES` table maps path patterns to type + source + tags.
* Adding a new directory convention = adding one rule. * Adding a new directory convention = adding one rule.
* - **Safe.** `.bak` files on write, `--dry-run` by default in CLI, idempotent. * - **Safe.** centralized backups on write, `--dry-run` by default in CLI,
* idempotent.
* *
* ## How it fits in the pipeline * ## How it fits in the pipeline
* *
@@ -177,6 +178,22 @@ export const DIRECTORY_RULES: DirectoryRule[] = [
titleStrategy: 'filename', titleStrategy: 'filename',
}, },
// Documentation/workspace repos. These make generated frontmatter useful
// instead of flattening everything to the catch-all `note` type.
{ pathPrefix: 'docs/runbooks/', type: 'guide', source: 'docs', tags: ['runbook'], titleStrategy: 'heading' },
{ pathPrefix: 'runbooks/', type: 'guide', source: 'docs', tags: ['runbook'], titleStrategy: 'heading' },
{ pathPrefix: 'docs/guides/', type: 'guide', source: 'docs', tags: ['guide'], titleStrategy: 'heading' },
{ pathPrefix: 'docs/policies/', type: 'guide', source: 'docs', tags: ['policy'], titleStrategy: 'heading' },
{ pathPrefix: 'docs/projects/', type: 'project', source: 'docs', tags: ['project'], titleStrategy: 'heading' },
{ pathPrefix: 'docs/audits/', type: 'analysis', source: 'docs', tags: ['audit'], titleStrategy: 'heading' },
{ pathPrefix: 'docs/research/', type: 'analysis', source: 'docs', tags: ['research'], titleStrategy: 'heading' },
{ pathPrefix: 'docs/evaos/', type: 'architecture', source: 'docs', tags: ['evaos'], titleStrategy: 'heading' },
{ pathPrefix: 'docs/architecture/', type: 'architecture', source: 'docs', titleStrategy: 'heading' },
{ pathPrefix: 'docs/providers/', type: 'source', source: 'docs', tags: ['provider'], titleStrategy: 'heading' },
{ pathPrefix: 'security/', type: 'guide', source: 'docs', tags: ['security'], titleStrategy: 'heading' },
{ pathPrefix: 'support/', type: 'source', source: 'docs', tags: ['support'], titleStrategy: 'heading' },
{ pathPrefix: 'notes/', type: 'note', titleStrategy: 'heading' },
// Personal sections // Personal sections
{ {
pathPrefix: 'personal/therapy/', pathPrefix: 'personal/therapy/',
@@ -233,7 +250,9 @@ export const DIRECTORY_RULES: DirectoryRule[] = [
{ pathPrefix: 'meetings/', type: 'meeting', titleStrategy: 'heading', datePattern: 'filename' }, { pathPrefix: 'meetings/', type: 'meeting', titleStrategy: 'heading', datePattern: 'filename' },
{ pathPrefix: 'media/', type: 'media', titleStrategy: 'heading' }, { pathPrefix: 'media/', type: 'media', titleStrategy: 'heading' },
// Catch-all for any remaining files // Catch-all for any remaining files. The CLI does not write this rule by
// default; users must pass --include-catch-all so `type: note` means an
// intentional fallback instead of "the script did not know what this was".
{ pathPrefix: '', type: 'note', titleStrategy: 'heading' }, { pathPrefix: '', type: 'note', titleStrategy: 'heading' },
]; ];
+38 -8
View File
@@ -7,10 +7,11 @@ import { spawnSync } from 'child_process';
const fence = '---'; const fence = '---';
const CLI = ['run', 'src/cli.ts', 'frontmatter']; const CLI = ['run', 'src/cli.ts', 'frontmatter'];
function runCli(args: string[]): { stdout: string; stderr: string; code: number } { function runCli(args: string[], env: Record<string, string> = {}): { stdout: string; stderr: string; code: number } {
const result = spawnSync('bun', [...CLI, ...args], { const result = spawnSync(process.execPath, [...CLI, ...args], {
encoding: 'utf8', encoding: 'utf8',
cwd: process.cwd(), cwd: process.cwd(),
env: { ...process.env, ...env },
}); });
return { return {
stdout: result.stdout ?? '', stdout: result.stdout ?? '',
@@ -75,24 +76,32 @@ describe('gbrain frontmatter CLI (B4)', () => {
expect(code).toBe(0); expect(code).toBe(0);
}); });
test('validate --fix writes .bak and rewrites in place', () => { test('validate --fix writes centralized backup and rewrites in place', () => {
const f = join(tmp, 'broken.md'); const f = join(tmp, 'broken.md');
const gbrainHome = join(tmp, 'home');
const original = `${fence}\ntype: concept\ntitle: "P "I" L"\n${fence}\n\nbody`; const original = `${fence}\ntype: concept\ntitle: "P "I" L"\n${fence}\n\nbody`;
writeFileSync(f, original); writeFileSync(f, original);
const { code } = runCli(['validate', f, '--fix']); const { stdout, code } = runCli(['validate', f, '--fix'], { GBRAIN_HOME: gbrainHome });
expect(code).toBe(0); expect(code).toBe(0);
expect(existsSync(f + '.bak')).toBe(true); expect(existsSync(f + '.bak')).toBe(false);
expect(readFileSync(f + '.bak', 'utf8')).toBe(original); expect(stdout).toContain('centralized backups');
const backupDir = join(gbrainHome, '.gbrain', 'backups', 'frontmatter');
const backup = spawnSync('find', [backupDir, '-type', 'f', '-name', 'broken.md.bak'], { encoding: 'utf8' });
const backupPath = backup.stdout.trim().split('\n').filter(Boolean)[0];
expect(backupPath).toBeTruthy();
expect(readFileSync(backupPath, 'utf8')).toBe(original);
expect(readFileSync(f, 'utf8')).toMatch(/^title: '.*'\s*$/m); expect(readFileSync(f, 'utf8')).toMatch(/^title: '.*'\s*$/m);
}); });
test('validate --fix succeeds on a non-git path (no dirty-tree guard)', () => { test('validate --fix succeeds on a non-git path (no dirty-tree guard)', () => {
// tmp is not a git repo; --fix must still work. // tmp is not a git repo; --fix must still work.
const f = join(tmp, 'broken.md'); const f = join(tmp, 'broken.md');
const gbrainHome = join(tmp, 'home');
writeFileSync(f, `${fence}\ntype: concept\ntitle: "A "B" C"\n${fence}\n\nbody`); writeFileSync(f, `${fence}\ntype: concept\ntitle: "A "B" C"\n${fence}\n\nbody`);
const { code } = runCli(['validate', f, '--fix']); const { code } = runCli(['validate', f, '--fix'], { GBRAIN_HOME: gbrainHome });
expect(code).toBe(0); expect(code).toBe(0);
expect(existsSync(f + '.bak')).toBe(true); expect(existsSync(f + '.bak')).toBe(false);
expect(existsSync(join(gbrainHome, '.gbrain', 'backups', 'frontmatter'))).toBe(true);
}); });
test('validate scans a directory recursively, skips non-.md files', () => { test('validate scans a directory recursively, skips non-.md files', () => {
@@ -107,6 +116,27 @@ describe('gbrain frontmatter CLI (B4)', () => {
expect(env.total_files).toBe(2); expect(env.total_files).toBe(2);
}); });
test('generate --fix skips catch-all note writes unless explicitly included', () => {
const gbrainHome = join(tmp, 'home');
writeFileSync(join(tmp, 'random.md'), '# Random\n\nbody');
writeFileSync(join(tmp, 'notes.md'), '# Also Random\n\nbody');
const skipped = runCli(['generate', tmp, '--fix', '--json'], { GBRAIN_HOME: gbrainHome });
expect(skipped.code).toBe(0);
const skippedEnv = JSON.parse(skipped.stdout);
expect(skippedEnv.generated).toBe(0);
expect(skippedEnv.skippedCatchAll).toBe(2);
expect(readFileSync(join(tmp, 'random.md'), 'utf8')).toBe('# Random\n\nbody');
const included = runCli(['generate', tmp, '--fix', '--json', '--include-catch-all'], { GBRAIN_HOME: gbrainHome });
expect(included.code).toBe(0);
const includedEnv = JSON.parse(included.stdout);
expect(includedEnv.generated).toBe(2);
expect(readFileSync(join(tmp, 'random.md'), 'utf8')).toContain('type: note');
expect(existsSync(join(gbrainHome, '.gbrain', 'backups', 'frontmatter'))).toBe(true);
expect(existsSync(join(tmp, 'random.md.bak'))).toBe(false);
});
test('validate missing path errors clearly', () => { test('validate missing path errors clearly', () => {
const { stderr, code } = runCli(['validate', join(tmp, 'does-not-exist.md')]); const { stderr, code } = runCli(['validate', join(tmp, 'does-not-exist.md')]);
expect(code).toBe(1); expect(code).toBe(1);
+20 -1
View File
@@ -176,6 +176,25 @@ describe('inferFrontmatter', () => {
expect(result.source).toBe('calendar'); expect(result.source).toBe('calendar');
}); });
test('docs/runbooks: infers guide instead of catch-all note', () => {
const result = inferFrontmatter(
'docs/runbooks/cron-management-runbook.md',
'# Cron Management Runbook\n\nUse this when changing schedules.',
);
expect(result.type).toBe('guide');
expect(result.tags).toContain('runbook');
expect(result.title).toBe('Cron Management Runbook');
});
test('docs/projects: infers project type', () => {
const result = inferFrontmatter(
'docs/projects/eva-brain.md',
'# Eva Brain\n\nProject notes.',
);
expect(result.type).toBe('project');
expect(result.tags).toContain('project');
});
test('companies/ directory: type company', () => { test('companies/ directory: type company', () => {
const result = inferFrontmatter( const result = inferFrontmatter(
'companies/stripe.md', 'companies/stripe.md',
@@ -185,7 +204,7 @@ describe('inferFrontmatter', () => {
expect(result.title).toBe('Stripe'); expect(result.title).toBe('Stripe');
}); });
test('unknown directory: defaults to note type with heading title', () => { test('unknown directory: catch-all remains note when explicitly used by callers', () => {
const result = inferFrontmatter( const result = inferFrontmatter(
'random/some-file.md', 'random/some-file.md',
'# My Random Notes\n\nStuff here', '# My Random Notes\n\nStuff here',
+3
View File
@@ -86,6 +86,7 @@ describe('v0.22.4 migration (B11)', () => {
total: 8, total: 8,
errors_by_code: { NESTED_QUOTES: 8 }, errors_by_code: { NESTED_QUOTES: 8 },
sample: [], sample: [],
ignoredMissingOpen: 0,
}, },
{ {
source_id: 'archive', source_id: 'archive',
@@ -93,6 +94,7 @@ describe('v0.22.4 migration (B11)', () => {
total: 4, total: 4,
errors_by_code: { NULL_BYTES: 4 }, errors_by_code: { NULL_BYTES: 4 },
sample: [], sample: [],
ignoredMissingOpen: 0,
}, },
{ {
source_id: 'clean-source', source_id: 'clean-source',
@@ -100,6 +102,7 @@ describe('v0.22.4 migration (B11)', () => {
total: 0, total: 0,
errors_by_code: {}, errors_by_code: {},
sample: [], sample: [],
ignoredMissingOpen: 0,
}, },
], ],
scanned_at: new Date().toISOString(), scanned_at: new Date().toISOString(),