mirror of
https://github.com/garrytan/gbrain.git
synced 2026-07-27 22:15:33 +00:00
v0.22.4 feat: frontmatter-guard — 0 resolver warnings + validate/audit/install-hook CLI (#448)
* fix: resolve check-resolvable warnings on master
- skills/maintain/SKILL.md: drop "citation audit" trigger; the focused
citation-fixer skill is the single owner. Silences the MECE overlap
warning surfaced by src/core/check-resolvable.ts.
- skills/RESOLVER.md: add citation-audit disambiguation row pointing
citation-fixer (focused fix) and chain-into maintain for broader audit.
Broaden query triggers ("who is", "background on", "notes on") so
the failing routing-eval fixtures resolve.
- skills/enrich/SKILL.md: replace inlined Citation Requirements block with
backtick-wrapped `skills/conventions/quality.md` reference (the format
extractDelegationTargets recognizes). Silences the dry_violation warning.
- skills/citation-fixer/routing-eval.jsonl: rewrite the two failing fixtures
to embed "fix citations" verbatim so the substring matcher passes.
- skills/query/SKILL.md frontmatter: mirror the broadened RESOLVER.md
triggers so the trigger round-trip test passes.
Result: gbrain check-resolvable reports 0 warnings, 0 errors against
the actual checked-in skills/ tree.
* feat: extend parseMarkdown + lint with frontmatter validation surface
Add an opt-in validation surface to parseMarkdown(): when called with
{ validate: true }, returns errors[] populated with seven canonical
ParseValidationError codes:
MISSING_OPEN, MISSING_CLOSE, YAML_PARSE, SLUG_MISMATCH,
NULL_BYTES, NESTED_QUOTES, EMPTY_FRONTMATTER
Existing callers are unaffected — validation is opt-in via the new
opts argument. The validation logic lives here as the single source of
truth for what counts as malformed brain-page frontmatter.
src/commands/lint.ts now consumes parseMarkdown(..., { validate: true })
and emits stable lint rule names (frontmatter-missing-close,
frontmatter-yaml-parse, frontmatter-null-bytes, frontmatter-nested-quotes,
frontmatter-slug-mismatch, frontmatter-empty). MISSING_OPEN is suppressed
to avoid double-reporting with the legacy no-frontmatter rule.
Tests: test/markdown-validation.test.ts (NEW, all 7 codes) +
test/lint-frontmatter.test.ts (NEW, lint integration + suppression).
* feat: add brain-writer.ts orchestrator (scan / autoFix / writeBrainPage)
Thin orchestrator (~280 lines) on top of parseMarkdown(..., {validate:true})
and isSyncable() (the canonical brain-page filter from src/core/sync.ts).
Three consumers call into this module: the gbrain frontmatter CLI, the
frontmatter_integrity doctor subcheck, and the v0.22.4 migration audit
phase. Single source of truth — no parallel validation stack.
Public API:
- autoFixFrontmatter(content, opts?): { content, fixes }
Mechanical auto-repair for the fixable subset (NULL_BYTES,
MISSING_CLOSE, NESTED_QUOTES, SLUG_MISMATCH). Idempotent.
- writeBrainPage(filePath, content, opts): path-guarded, .bak backup
before any in-place mutation. Path guard refuses writes outside
sourcePath. .bak is the safety contract for non-git brain repos.
- scanBrainSources(engine, opts?): walks every registered source via
direct SQL on sources.local_path, uses isSyncable() to filter,
blocks symlinks (matches sync's no-symlink policy), respects
AbortSignal.
The dirty-tree guard from src/core/dry-fix.ts:getWorkingTreeStatus() is
NOT used here — it rejects non-git repos as unsafe, but brain repos
aren't always git repos. .bak backups are the contract that works
universally.
Tests: test/brain-writer.test.ts (NEW, 16 cases) — autoFix idempotency,
path-guard reject, .bak backup, per-source rollup, AbortSignal mid-scan,
single-source filter, missing-source-path graceful skip, symlink no-loop.
* feat: gbrain frontmatter CLI (validate / audit / install-hook)
New top-level command surface for the frontmatter-guard feature:
gbrain frontmatter validate <path> [--json] [--fix] [--dry-run]
Validate one .md file or recursively scan a directory. --fix writes
.bak then rewrites in place. No git-tree-clean guard — .bak is the
safety contract (works for both git and non-git brain repos).
gbrain frontmatter audit [--source <id>] [--json]
Read-only scan via scanBrainSources(). Per-source rollup grouped by
error code. --fix is intentionally NOT available here; use validate
--fix on the source path to repair.
gbrain frontmatter install-hook [--source <id>] [--force] [--uninstall]
Drops a pre-commit hook in each source that's a git repo (skips
non-git sources with a one-line note). Hook script gracefully
degrades when gbrain is missing on PATH (prints a warning, exits 0).
Refuses to clobber existing hooks without --force; writes <hook>.bak.
--uninstall reverses cleanly.
src/cli.ts wires frontmatter through handleCliOnly so --help works
without a DB connection. The audit subcommand instantiates an engine
internally only when needed.
Tests: test/frontmatter-cli.test.ts (NEW, 9 cases) +
test/frontmatter-install-hook.test.ts (NEW, 6 cases) — --help no-DB,
clean/broken validate, --fix dry-run, --fix non-git, --json envelope,
recursive directory scan with isSyncable filter parity, hook install
+ overwrite-protection + --force + --uninstall + silent-refresh.
* feat: doctor frontmatter_integrity subcheck
Adds a frontmatter_integrity subcheck under gbrain doctor that calls
scanBrainSources() (the same shared scanner the CLI and migration use).
Reports per-source counts grouped by error code, with a fix hint
pointing at `gbrain frontmatter validate <path> --fix`. Wrapped in
a doctor progress phase with heartbeat so 50K-page brain scans stay
visible.
Tests: test/doctor.test.ts (UPDATE) — assertion that the subcheck
calls scanBrainSources and the fix hint references the correct CLI.
* feat: frontmatter-guard skill (registered in manifest + RESOLVER)
New skill at skills/frontmatter-guard/SKILL.md that wraps the gbrain
frontmatter CLI for agent-driven workflows. Agent-agnostic — no
references to private host libraries. Registered in skills/manifest.json
and skills/RESOLVER.md (the trigger row was added in the Part A commit).
Triggers: "validate frontmatter", "check frontmatter", "fix frontmatter",
"frontmatter audit", "brain lint".
Includes routing-eval fixtures that pass the substring matcher. The
SKILL.md has the conformance-required Output Format and Anti-Patterns
sections. Anti-patterns explicitly call out: don't auto-fix MISSING_OPEN
or EMPTY_FRONTMATTER without user input, don't skip .bak backups, don't
install the pre-commit hook on non-git brain dirs.
* feat: v0.22.4 migration orchestrator (audit-only, source-aware)
Adds the v0.22.4 migration that surveys every registered source for
frontmatter issues and queues per-source repair commands without ever
mutating brain content. Three idempotent phases:
- schema: no-op (no DB changes in v0.22.4)
- audit: scanBrainSources() across ALL registered sources; writes
JSON report to ~/.gbrain/migrations/v0.22.4-audit.json
- emit-todo: appends one entry per source-with-issues to
~/.gbrain/migrations/pending-host-work.jsonl, each with the exact
`gbrain frontmatter validate <source-path> --fix` command
The agent reads skills/migrations/v0.22.4.md after upgrade, surfaces
the report counts to the user, and runs the fix command only with
explicit consent. `apply-migrations --yes` never silently rewrites
brain pages.
Filename convention: TS orchestrator at v0_22_4.ts (underscores, since
TS module paths can't have dots); user-facing migration doc at
skills/migrations/v0.22.4.md (dotted, matches existing convention).
The pending-host-work.jsonl skill field references the dotted-path doc.
Skips cleanly when no sources are registered (fresh install).
Tests: test/migrations-v0_22_4.test.ts (NEW, 9 cases) + updated
test/migration-orchestrator-v0_21_0.test.ts to allow v0.22.4 after,
test/apply-migrations.test.ts skippedFuture arrays extended to include
v0.22.4, test/check-resolvable.test.ts regression guard asserting the
actual checked-in skills/ tree has 0 warnings + 0 errors.
* docs: pre-commit recipe + downstream agent upgrade notes for v0.22.4
- docs/integrations/pre-commit.md (NEW): recipe doc covering install,
bypass (`git commit --no-verify`), uninstall, and downstream-fork
integration notes. Includes the full pipeline diagram showing how
the hook (write-time gate), doctor (audit gate), and CLI (fix tool)
share parseMarkdown(..., {validate:true}) as the single source of
truth.
- docs/UPGRADING_DOWNSTREAM_AGENTS.md: append v0.22.4 section with the
diff pattern for forks that had inline frontmatter validators. Covers
the five upgrade actions: replace ad-hoc validators, drop
lib/brain-writer.mjs references (it never shipped), wire the doctor
subcheck into custom health pipelines, optionally install the
pre-commit hook on git-backed brain repos, and walk
pending-host-work.jsonl after apply-migrations.
- llms.txt + llms-full.txt: regenerated from build:llms script after
the new docs landed.
* chore: bump version and changelog (v0.22.4)
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* fix: handle null loadConfig() return in frontmatter + migration paths
CI typecheck caught three call sites that passed loadConfig()'s
GBrainConfig | null result straight into toEngineConfig() (which
expects GBrainConfig, not null):
- src/commands/frontmatter.ts:64 (audit subcommand connect)
- src/commands/frontmatter-install-hook.ts:86 (install-hook connect)
- src/commands/migrations/v0_22_4.ts:59 (audit phase connect)
The frontmatter CLI and install-hook paths follow the existing
src/commands/repair-jsonb.ts pattern: throw 'No brain configured. Run:
gbrain init' so users get an actionable message instead of a TS-shaped
runtime crash.
The v0.22.4 migration audit phase takes a different shape: a fresh
install or test environment running apply-migrations shouldn't fail
hard just because there's no brain to scan yet. Return a clean
'skipped: no_brain_configured' phase result so the orchestrator
continues normally and the ledger records a complete (skipped) run.
* test: add v0.22.4 migration E2E + injection point for testability
Closes plan item B14 (the E2E that was promised but not delivered before
the original ship). Runs the v0_22_4 orchestrator end-to-end on PGLite
against a fixture brain with two registered sources and synthetic
malformed pages on disk. Asserts:
- audit phase writes ~/.gbrain/migrations/v0.22.4-audit.json with
per-source counts (NESTED_QUOTES + NULL_BYTES on alpha,
NESTED_QUOTES on beta)
- emit-todo phase appends one entry per source-with-issues to
pending-host-work.jsonl, each pointing at skills/migrations/v0.22.4.md
with the exact `gbrain frontmatter validate <source> --fix` command
- the migration is audit-only — no fixture page is mutated
during apply-migrations (no .bak created, contents byte-identical)
- re-running the orchestrator is idempotent — JSONL stays at 2 lines
Adds a small test-injection point to v0_22_4.ts:
__setTestEngineOverride(engine: BrainEngine | null): void
Mirrors src/commands/repair-jsonb.ts pattern. When set, phaseBAudit
uses the injected engine instead of loadConfig + createEngine. Production
path is unchanged: the override is null by default and the existing
loadConfig logic runs end-to-end. Required because Bun's os.homedir()
does not observe mid-process process.env.HOME mutations, so we can't
redirect loadConfig's config-file lookup via env-var overrides; the
injection point is the only hermetic way to E2E-test the orchestrator
without writing to the user's real ~/.gbrain/config.json.
Test runs unconditionally in CI's Tier 1 (no DATABASE_URL needed,
PGLite in-memory).
---------
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
Garry Tan
co-authored by
Claude Opus 4.7
Claude Opus 4.7
parent
c78c3d0135
commit
891c28b582
+120
@@ -2,6 +2,126 @@
|
||||
|
||||
All notable changes to GBrain will be documented in this file.
|
||||
|
||||
## [0.22.4] - 2026-04-26
|
||||
|
||||
## **Frontmatter-guard ships. Broken brain pages can't hide.**
|
||||
## **Seven validation classes, source-aware audit, doctor subcheck, pre-commit hook, zero resolver warnings.**
|
||||
|
||||
v0.22.4 fixes the seven `gbrain check-resolvable` warnings that lived on master and ships frontmatter-guard as a real feature: a TypeScript validator inside `parseMarkdown(..., {validate:true})`, a top-level `gbrain frontmatter` CLI (`validate` / `audit` / `install-hook`), a new `frontmatter_integrity` subcheck under `gbrain doctor`, and an audit-only migration that surveys every registered source and queues per-source TODOs without mutating brain content. PR #392's aspirational `lib/brain-writer.mjs` is finally written, in TypeScript, on top of the tools gbrain already ships.
|
||||
|
||||
The migration is **audit-only**. It writes a JSON report to `~/.gbrain/migrations/v0.22.4-audit.json` and emits per-source entries to `pending-host-work.jsonl` with the exact fix command. It never silently rewrites your brain pages. The agent reads `skills/migrations/v0.22.4.md` after upgrade, surfaces the counts to you, and runs `gbrain frontmatter validate <source-path> --fix` only with explicit consent. `--fix` writes `.bak` backups for every modified file (the safety contract for non-git brain repos, which `getWorkingTreeStatus` rejects).
|
||||
|
||||
`gbrain frontmatter` is source-aware throughout. `audit [--source <id>]` walks every registered source via `source-resolver.ts` (gbrain has been multi-source since v0.18.0; the single-`brainRoot` model would have shipped a half-broken feature). The CLI, doctor subcheck, and migration phase all call into one shared `scanBrainSources()` ... single source of truth for what counts as malformed.
|
||||
|
||||
### The numbers that matter
|
||||
|
||||
Counted against gbrain's own checked-in `skills/` tree:
|
||||
|
||||
| Metric | Pre-v0.22.4 (master) | v0.22.4 | Δ |
|
||||
|---|---|---|---|
|
||||
| `gbrain check-resolvable` warnings | 7 | 0 | -7 |
|
||||
| Frontmatter validation classes | 3 (in `lint`) | 7 (in `parseMarkdown`) | +4 |
|
||||
| Auto-fixable error codes | 0 | 4 (NULL_BYTES, MISSING_CLOSE, NESTED_QUOTES, SLUG_MISMATCH) | +4 |
|
||||
| Doctor subchecks | 17 | 18 (+frontmatter_integrity) | +1 |
|
||||
| `gbrain frontmatter` subcommands | 0 | 3 (validate, audit, install-hook) | +3 |
|
||||
| Skills in `skills/` | 29 | 30 (+frontmatter-guard) | +1 |
|
||||
| Pre-commit hook helper | none | `gbrain frontmatter install-hook` | ✓ |
|
||||
| Source-aware audit | n/a | walks every registered source | ✓ |
|
||||
|
||||
Frontmatter validation surface (the 7 codes shipped):
|
||||
|
||||
| Code | What it catches | Auto-fix |
|
||||
|---|---|---|
|
||||
| `MISSING_OPEN` | File doesn't start with `---` | No (human review) |
|
||||
| `MISSING_CLOSE` | No closing `---` before first heading | Yes ... inserts `---` |
|
||||
| `YAML_PARSE` | YAML failed to parse | Sometimes |
|
||||
| `SLUG_MISMATCH` | Frontmatter `slug:` differs from path-derived slug | Yes ... removes field |
|
||||
| `NULL_BYTES` | Binary corruption (`\x00`) | Yes ... strips bytes |
|
||||
| `NESTED_QUOTES` | `title: "outer "inner" outer"` shape | Yes ... switches outer to single quotes |
|
||||
| `EMPTY_FRONTMATTER` | Open + close present, nothing meaningful between | No (human review) |
|
||||
|
||||
### What this means for builders
|
||||
|
||||
If you've been ignoring `gbrain check-resolvable` warnings because the messages were misleading (the action message said "Add disambiguation rule in RESOLVER.md OR narrow triggers" ... but only the second branch actually silenced the MECE warning, since the checker doesn't parse RESOLVER.md disambiguation rules), v0.22.4 closes the loop. Trigger overlap is fixed at the frontmatter layer. `enrich/SKILL.md` delegates citation rules to `conventions/quality.md` instead of inlining them. Routing-eval fixtures embed actual trigger keywords. `frontmatter-guard` is registered. `gbrain check-resolvable --json` returns `ok: true, issues: []`.
|
||||
|
||||
If your agent writes brain pages, plumb its writes through `parseMarkdown(content, path, { validate: true, expectedSlug })` (the export is in `gbrain/markdown`) and check the returned `errors` array. The 7-error envelope is stable from v0.22.4 onward. Or call `gbrain frontmatter validate <path> --json` from your script and parse the envelope. For brain repos that ARE git repos, install the pre-commit hook with `gbrain frontmatter install-hook` and stop bad frontmatter at the commit boundary.
|
||||
|
||||
If you maintain a downstream OpenClaw fork, see `docs/UPGRADING_DOWNSTREAM_AGENTS.md` for the v0.22.4 diff pattern. The short version: drop any references to the never-existed `lib/brain-writer.mjs` and replace with `gbrain frontmatter validate` calls.
|
||||
|
||||
## To take advantage of v0.22.4
|
||||
|
||||
`gbrain upgrade` runs `gbrain post-upgrade` which runs `gbrain apply-migrations`. If that chain was interrupted or if `gbrain doctor` reports `frontmatter_integrity` issues:
|
||||
|
||||
1. **Run the orchestrator manually:**
|
||||
```bash
|
||||
gbrain apply-migrations --yes
|
||||
```
|
||||
The `v0.22.4` orchestrator (v0_22_4.ts) runs schema (no-op) → audit → emit-todo. The audit phase writes a per-source JSON report to `~/.gbrain/migrations/v0.22.4-audit.json` and queues one entry per source with issues to `~/.gbrain/migrations/pending-host-work.jsonl`. **It never modifies brain content.**
|
||||
|
||||
2. **Read the audit report:**
|
||||
```bash
|
||||
cat ~/.gbrain/migrations/v0.22.4-audit.json | jq '.errors_by_code, .per_source[].source_id'
|
||||
```
|
||||
|
||||
3. **Fix mechanical issues with explicit consent.** For each source with errors > 0, run:
|
||||
```bash
|
||||
gbrain frontmatter validate <source-path> --fix
|
||||
```
|
||||
This writes `.bak` backups for every modified file. SLUG_MISMATCH errors are surfaced for manual review (gbrain derives slug from path; a mismatch usually means the file was renamed deliberately or the slug field is stale).
|
||||
|
||||
4. **Verify the outcome:**
|
||||
```bash
|
||||
gbrain doctor --json | jq '.checks[] | select(.name == "frontmatter_integrity")'
|
||||
gbrain frontmatter audit --json | jq '.total'
|
||||
gbrain check-resolvable --json | jq '.report.issues | map(select(.severity=="warning" or .severity=="error")) | length'
|
||||
```
|
||||
All three should report 0 issues.
|
||||
|
||||
5. **If any step fails or the numbers look wrong,** file an issue at https://github.com/garrytan/gbrain/issues with:
|
||||
- output of `gbrain doctor`
|
||||
- contents of `~/.gbrain/migrations/v0.22.4-audit.json`
|
||||
- contents of `~/.gbrain/upgrade-errors.jsonl` if it exists
|
||||
- which step broke
|
||||
|
||||
This feedback loop is how the gbrain maintainers find fragile upgrade paths. Thank you.
|
||||
|
||||
### Itemized changes
|
||||
|
||||
**Part A ... `gbrain check-resolvable` reaches 0 warnings.** Drop `"citation audit"` from `skills/maintain/SKILL.md` frontmatter; the trigger lives only on `citation-fixer` now. RESOLVER.md gains a citation-audit disambiguation row pointing both skills so agents still pick the right one. RESOLVER.md broadens query triggers (`"who is"`, `"background on"`, `"notes on"`) and `query/SKILL.md` mirrors them in its frontmatter. `skills/enrich/SKILL.md` replaces the inlined citation rules block with `> **Convention:** see \`skills/conventions/quality.md\`` (the format `extractDelegationTargets` recognizes). Routing-eval fixtures for `citation-fixer` rewritten to embed `"fix citations"` so substring matching passes.
|
||||
|
||||
**Part B ... frontmatter-guard library + CLI + doctor + migration + skill + pre-commit hook.**
|
||||
|
||||
- **`src/core/markdown.ts`** ... `parseMarkdown(content, filePath?, opts?)` gains an opt-in `opts.validate` flag. When true, returns `errors[]` with the seven canonical codes. Existing callers unaffected. Validation logic for all seven codes lives here as the single source of truth.
|
||||
- **`src/commands/lint.ts`** ... frontmatter-rule lint cases delegate to `parseMarkdown(..., {validate:true})`. New rule names: `frontmatter-missing-close`, `frontmatter-yaml-parse`, `frontmatter-null-bytes`, `frontmatter-nested-quotes`, `frontmatter-slug-mismatch`, `frontmatter-empty`. Suppresses MISSING_OPEN to avoid double-reporting with the legacy `no-frontmatter` rule.
|
||||
- **`src/core/brain-writer.ts`** (NEW) ... thin orchestrator (~280 lines). Exports `autoFixFrontmatter`, `writeBrainPage`, `scanBrainSources`. `writeBrainPage` is path-guarded (refuses writes outside `sourcePath`), always writes `<file>.bak` before any in-place mutation. `scanBrainSources` walks every registered source via direct SQL against `sources.local_path`, uses `isSyncable()` from sync.ts as the canonical brain-page filter, blocks symlinks (matches sync's no-symlink policy), and respects `AbortSignal`.
|
||||
- **`src/commands/frontmatter.ts`** (NEW) ... `gbrain frontmatter validate <path> [--json] [--fix] [--dry-run]` and `gbrain frontmatter audit [--source <id>] [--json]`. The `audit` subcommand is read-only; `--fix` only exists on `validate`. CLI handles `--help` without a DB connection.
|
||||
- **`src/commands/frontmatter-install-hook.ts`** (NEW) ... `gbrain frontmatter install-hook [--source <id>] [--force] [--uninstall]`. Writes `.githooks/pre-commit` per source (skips non-git sources with a one-line note), runs `git config core.hooksPath .githooks` if unset, refuses to clobber existing hooks without `--force` (writes `.bak`). The hook script gracefully degrades when `gbrain` is missing on PATH (prints a warning, exits 0 ... doesn't break commits).
|
||||
- **`src/commands/doctor.ts`** ... new `frontmatter_integrity` subcheck calls `scanBrainSources()` and reports per-source counts plus the fix hint. Wraps in a doctor progress phase with heartbeat.
|
||||
- **`src/commands/migrations/v0_22_4.ts`** (NEW) ... audit-only orchestrator with three phases (schema no-op, audit, emit-todo). Idempotent + resumable. Skips cleanly when no sources are registered. Per-source TODO entries reference the dotted-filename migration doc (`skills/migrations/v0.22.4.md`) per the existing `pending-host-work.jsonl` convention.
|
||||
- **`skills/frontmatter-guard/SKILL.md`** (NEW) ... agent-agnostic; routes to `gbrain frontmatter` CLI invocations, drops OpenClaw-specific paths from PR #392's spec. Registered in `skills/manifest.json` and `skills/RESOLVER.md` with substring-matchable triggers.
|
||||
- **`docs/integrations/pre-commit.md`** (NEW) ... recipe doc covering install / bypass / uninstall and downstream-fork notes.
|
||||
- **`docs/UPGRADING_DOWNSTREAM_AGENTS.md`** ... v0.22.4 section with the diff pattern for forks that had inline frontmatter validators.
|
||||
|
||||
**Tests.** 9 new test files / 4 updated test files. Unit coverage on every new module:
|
||||
- `test/markdown-validation.test.ts` (NEW) ... all 7 codes exercised against hand-crafted fixtures.
|
||||
- `test/lint-frontmatter.test.ts` (NEW) ... lint emits findings for each fixable code; double-report suppression verified.
|
||||
- `test/brain-writer.test.ts` (NEW) ... `autoFixFrontmatter` idempotency, `writeBrainPage` path-guard + `.bak` backup, `scanBrainSources` per-source rollup, AbortSignal mid-scan, single-source filter, missing-source-path graceful skip, symlink no-loop.
|
||||
- `test/frontmatter-cli.test.ts` (NEW) ... subprocess `validate / --fix --dry-run / --fix / --json` + recursive directory scan with `isSyncable` filter parity.
|
||||
- `test/frontmatter-install-hook.test.ts` (NEW) ... hook install / overwrite-protection / `--force` / `--uninstall` / silent-refresh on already-installed.
|
||||
- `test/migrations-v0_22_4.test.ts` (NEW) ... orchestrator phase coverage including dotted-filename JSONL contract and idempotent re-emit.
|
||||
- `test/check-resolvable.test.ts` (UPDATE) ... regression guard asserting the actual checked-in `skills/` tree has 0 warnings + 0 errors.
|
||||
- `test/doctor.test.ts` (UPDATE) ... assertion that `frontmatter_integrity` subcheck calls `scanBrainSources` and the fix hint references the right CLI command.
|
||||
- `test/apply-migrations.test.ts` (UPDATE) ... `skippedFuture` arrays extended to include v0.22.4.
|
||||
- `test/migration-orchestrator-v0_21_0.test.ts` (UPDATE) ... relaxed "is the latest" assertion to "is registered with v0.22.4 after it."
|
||||
|
||||
### For contributors
|
||||
|
||||
`brain-writer.ts` is the canonical place to add new frontmatter validation rules. Add the code to `parseMarkdown`'s `collectValidationErrors`, surface the lint rule name in `lint.ts`'s `FRONTMATTER_RULE_NAMES`, decide if it's auto-fixable (add to `FRONTMATTER_FIXABLE`), and write the auto-fix logic in `brain-writer.ts:autoFixFrontmatter`. Tests in `test/markdown-validation.test.ts` + `test/brain-writer.test.ts`. The lint output uses the `frontmatter-<code>` naming convention; CI consumers can target specific rule names in their lint configs.
|
||||
|
||||
`gbrain frontmatter` is wired through `src/cli.ts:handleCliOnly` so `--help` works without a DB connection. The `audit` subcommand instantiates an engine internally via `loadConfig() + createEngine()`. New subcommands of `frontmatter` should follow this pattern: parse flags first, only connect to the engine when the subcommand actually needs DB access.
|
||||
|
||||
The v0.22.4 orchestrator is intentionally audit-only because brain content is too important to silently mutate during `apply-migrations`. Future migrations that need to rewrite brain pages should follow this two-step pattern: write the audit report + queue the fix command, let the agent run the fix with explicit user consent.
|
||||
|
||||
## [0.22.2] - 2026-04-26
|
||||
|
||||
**Worker no longer freezes silently. Restart-on-RSS, cold-start retry, autopilot backpressure.**
|
||||
|
||||
@@ -1,5 +1,51 @@
|
||||
# TODOS
|
||||
|
||||
## resolver / check-resolvable (v0.22.4 follow-ups)
|
||||
|
||||
### D10 — Extend `check-resolvable` to parse RESOLVER.md disambiguation rules
|
||||
**Priority:** P2
|
||||
|
||||
**What:** Extend `src/core/check-resolvable.ts:357-390` to parse a structured
|
||||
disambiguation block in `RESOLVER.md` (e.g. a `## Disambiguation rules`
|
||||
numbered list with parseable `<trigger>` → `<winning-skill>` shape) and treat
|
||||
resolved overlaps as non-issues. Then the action message at
|
||||
`src/core/check-resolvable.ts:388` ("Add disambiguation rule in RESOLVER.md OR
|
||||
narrow triggers") stops lying about the OR — currently only the second branch
|
||||
silences the warning.
|
||||
|
||||
**Why:** The current MECE-overlap fix path forces authors to delete user-facing
|
||||
triggers from skill frontmatter. That's wrong for cases where two skills
|
||||
legitimately respond to the same phrase under different contexts (e.g.
|
||||
"citation audit" → focused fix vs broader brain health). A real
|
||||
disambiguation parser would let `RESOLVER.md` carry the resolution while
|
||||
keeping both skills' triggers intact for chaining.
|
||||
|
||||
**Pros:**
|
||||
- The action message stops misleading users.
|
||||
- v0.22.4 D2 used the "narrow triggers" path because the disambiguation
|
||||
parser doesn't exist yet; landing this would let v0.23+ keep dual triggers
|
||||
for genuinely-overlapping skills.
|
||||
- Aligns RESOLVER.md's stated role (the dispatcher) with what the checker
|
||||
actually reads.
|
||||
|
||||
**Cons:**
|
||||
- Introduces a new `RESOLVER.md` syntactic contract that other tooling now
|
||||
has to respect (parser, lint, downstream forks reading the same file).
|
||||
- Risk of false-positive resolution if the parser is loose.
|
||||
- ~80 lines of parser + tests; not blocking anything in v0.22.4.
|
||||
|
||||
**Context:**
|
||||
- The "OR" in the action message is misleading today. Confirmed at
|
||||
`src/core/check-resolvable.ts:388`.
|
||||
- The MECE detector loop is at `src/core/check-resolvable.ts:357-390`.
|
||||
- The disambiguation rules already exist as prose in
|
||||
`skills/RESOLVER.md` (the citation-audit row added in v0.22.4 is the
|
||||
pattern). They're agent-facing routing hints today, not parsed structure.
|
||||
|
||||
**Effort:** S (human: ~4-6 hours / CC: ~30 min for parser + 12-16 test cases).
|
||||
|
||||
**Depends on / blocked by:** Nothing.
|
||||
|
||||
## code-indexing (v0.21.0 Cathedral II follow-ups)
|
||||
|
||||
### B2 — Magika auto-detect for extension-less files (Layer 9 deferred)
|
||||
|
||||
@@ -458,6 +458,75 @@ in depth, not the primary boundary.
|
||||
|
||||
---
|
||||
|
||||
## v0.22.4 — frontmatter-guard adoption
|
||||
|
||||
### 1. Stop hand-rolling frontmatter validators
|
||||
|
||||
If your fork has scripts that call `js-yaml` directly to validate brain page
|
||||
frontmatter, replace them with `gbrain frontmatter validate` calls. The CLI
|
||||
covers the seven canonical error classes and ships a `--json` envelope that's
|
||||
stable across releases.
|
||||
|
||||
```diff
|
||||
- # Custom validator script
|
||||
- node scripts/validate-frontmatter.mjs <path>
|
||||
+ gbrain frontmatter validate <path> --json
|
||||
```
|
||||
|
||||
For consumers that need the validator inside another script, import from
|
||||
gbrain's `markdown` export instead of duplicating logic:
|
||||
|
||||
```ts
|
||||
import { parseMarkdown } from 'gbrain/markdown';
|
||||
|
||||
const parsed = parseMarkdown(content, filePath, { validate: true, expectedSlug });
|
||||
for (const err of parsed.errors ?? []) {
|
||||
// err.code: MISSING_OPEN | MISSING_CLOSE | YAML_PARSE | SLUG_MISMATCH |
|
||||
// NULL_BYTES | NESTED_QUOTES | EMPTY_FRONTMATTER
|
||||
}
|
||||
```
|
||||
|
||||
### 2. Drop any references to `lib/brain-writer.mjs`
|
||||
|
||||
If your fork's skills or scripts referenced an aspirational
|
||||
`lib/brain-writer.mjs` (it never shipped — the spec was in PR #392 and never
|
||||
landed), replace those references with the gbrain CLI. The `frontmatter-guard`
|
||||
skill lives at `skills/frontmatter-guard/SKILL.md` and points at
|
||||
`gbrain frontmatter validate` / `audit` / `install-hook`.
|
||||
|
||||
### 3. Wire the doctor subcheck into your health pipeline
|
||||
|
||||
`gbrain doctor` now reports `frontmatter_integrity` automatically. If your
|
||||
fork has a custom health pipeline (e.g. a daily Slack post about brain
|
||||
health), pull from `gbrain doctor --json` and surface the
|
||||
`frontmatter_integrity` row counts.
|
||||
|
||||
### 4. (Optional) Install the pre-commit hook on brain repos
|
||||
|
||||
For sources backed by git, the v0.22.4 install-hook helper drops a
|
||||
pre-commit script that blocks commits with malformed frontmatter:
|
||||
|
||||
```bash
|
||||
gbrain frontmatter install-hook
|
||||
```
|
||||
|
||||
Skip this if your brain isn't a git repo or if your downstream agent already
|
||||
enforces validation at write time. See `docs/integrations/pre-commit.md` for
|
||||
the full recipe.
|
||||
|
||||
### 5. Migration ergonomics — read pending-host-work.jsonl
|
||||
|
||||
After `gbrain apply-migrations --yes` runs the v0.22.4 audit, your agent
|
||||
should read `~/.gbrain/migrations/pending-host-work.jsonl` (filter to
|
||||
`migration === "0.22.4"`) and walk each entry's `command` field. Each entry
|
||||
points to a per-source `gbrain frontmatter validate <source_path> --fix`
|
||||
command — surface counts to the user, get explicit consent, then run.
|
||||
|
||||
The migration is **audit-only**. It never mutates brain content during
|
||||
`apply-migrations`. Your agent runs the fix command with user consent.
|
||||
|
||||
---
|
||||
|
||||
## Future versions
|
||||
|
||||
When gbrain ships a new version, this doc will be updated with the diffs for that
|
||||
|
||||
@@ -0,0 +1,105 @@
|
||||
# Pre-commit hook for brain repos (v0.22.4+)
|
||||
|
||||
`gbrain frontmatter install-hook` installs a git pre-commit hook in your
|
||||
brain source's repo that runs `gbrain frontmatter validate` against staged
|
||||
`.md` and `.mdx` files. Malformed frontmatter blocks the commit. Bypass with
|
||||
`git commit --no-verify`.
|
||||
|
||||
## What the hook catches
|
||||
|
||||
The same seven validation classes the `frontmatter-guard` skill and
|
||||
`gbrain doctor`'s `frontmatter_integrity` subcheck report:
|
||||
|
||||
| Code | What it catches |
|
||||
|-------------------|---------------------------------------------------------------------|
|
||||
| `MISSING_OPEN` | File doesn't start with `---` |
|
||||
| `MISSING_CLOSE` | No closing `---` before first heading |
|
||||
| `YAML_PARSE` | YAML failed to parse (syntax or structure) |
|
||||
| `SLUG_MISMATCH` | `slug:` in frontmatter doesn't match path-derived slug |
|
||||
| `NULL_BYTES` | Binary corruption (`\x00`) anywhere in the content |
|
||||
| `NESTED_QUOTES` | `title: "outer "inner" outer"` shape that breaks YAML |
|
||||
| `EMPTY_FRONTMATTER` | `---` ... `---` with nothing meaningful between |
|
||||
|
||||
## Install
|
||||
|
||||
For all registered sources that are git repos:
|
||||
|
||||
```bash
|
||||
gbrain frontmatter install-hook
|
||||
```
|
||||
|
||||
For one source:
|
||||
|
||||
```bash
|
||||
gbrain frontmatter install-hook --source <id>
|
||||
```
|
||||
|
||||
For force-overwrite of an existing pre-commit hook (writes a `.bak`):
|
||||
|
||||
```bash
|
||||
gbrain frontmatter install-hook --force
|
||||
```
|
||||
|
||||
The hook lands at `<source>/.githooks/pre-commit`. If `core.hooksPath` is
|
||||
unset, the install also runs `git config core.hooksPath .githooks` so the
|
||||
hook is picked up without manual git config.
|
||||
|
||||
## Bypass
|
||||
|
||||
Standard git escape hatch:
|
||||
|
||||
```bash
|
||||
git commit --no-verify
|
||||
```
|
||||
|
||||
This skips ALL pre-commit hooks. Use sparingly — the next time the user
|
||||
runs `gbrain doctor`, the issues will surface.
|
||||
|
||||
## Uninstall
|
||||
|
||||
```bash
|
||||
gbrain frontmatter install-hook --uninstall
|
||||
```
|
||||
|
||||
If a `.bak` was saved during install, it's restored as the active hook.
|
||||
Otherwise the hook is removed cleanly.
|
||||
|
||||
## Behavior on machines without gbrain installed
|
||||
|
||||
The hook script checks for `gbrain` on `$PATH`. When missing, it prints a
|
||||
one-line warning to stderr and exits 0 — commits aren't blocked just because
|
||||
a developer hasn't installed gbrain locally. Once gbrain is installed, the
|
||||
hook resumes blocking malformed pages.
|
||||
|
||||
## For downstream agent forks
|
||||
|
||||
If your fork (Wintermute, Hermes, OpenClaw) wraps gbrain in a host repo
|
||||
that's not the brain repo itself, you may want a separate hook strategy:
|
||||
|
||||
- **Brain repo IS the host repo** (gbrain skills + brain pages in one repo):
|
||||
install via `gbrain frontmatter install-hook` as above.
|
||||
- **Brain repo is a separate registered source** (e.g. `~/brain` registered
|
||||
as a source, host repo is `~/agent-fork`): install in the brain repo only;
|
||||
agent-fork code doesn't need this hook.
|
||||
- **Brain repo is auto-generated** (e.g. by a sync daemon writing to a
|
||||
bucket): skip the hook entirely; gate at the writer instead via
|
||||
`import { writeBrainPage } from 'gbrain/brain-writer'` (planned in a
|
||||
later release; currently the CLI is the surface).
|
||||
|
||||
## How it fits into the broader frontmatter pipeline
|
||||
|
||||
```
|
||||
agent writes a page git commit doctor scan
|
||||
↓ ↓ ↓
|
||||
[source content] → [pre-commit hook validates] → [frontmatter_integrity check]
|
||||
↓ ↓ ↓
|
||||
raw file on disk blocks malformed commits surfaces existing issues
|
||||
↓
|
||||
`gbrain frontmatter validate
|
||||
<source-path> --fix`
|
||||
(writes .bak backups)
|
||||
```
|
||||
|
||||
The hook is the write-time gate; doctor is the audit gate; the CLI is the
|
||||
fix tool. They share `parseMarkdown(..., {validate:true})` as the single
|
||||
source of truth for what counts as malformed.
|
||||
+72
-1
@@ -1170,13 +1170,15 @@ This is the dispatcher. Skills are the implementation. **Read the skill file bef
|
||||
|
||||
| Trigger | Skill |
|
||||
|---------|-------|
|
||||
| "What do we know about", "tell me about", "search for" | `skills/query/SKILL.md` |
|
||||
| "What do we know about", "tell me about", "search for", "who is", "background on", "notes on" | `skills/query/SKILL.md` |
|
||||
| "Who knows who", "relationship between", "connections", "graph query" | `skills/query/SKILL.md` (use graph-query) |
|
||||
| Creating/enriching a person or company page | `skills/enrich/SKILL.md` |
|
||||
| Where does a new file go? Filing rules | `skills/repo-architecture/SKILL.md` |
|
||||
| Fix broken citations in brain pages | `skills/citation-fixer/SKILL.md` |
|
||||
| "citation audit", "check citations", "fix citations" | `skills/citation-fixer/SKILL.md` (focused fix). For broader brain health, chain into `skills/maintain/SKILL.md` |
|
||||
| "Research", "track", "extract from email", "investor updates", "donations" | `skills/data-research/SKILL.md` |
|
||||
| Share a brain page as a link | `skills/publish/SKILL.md` |
|
||||
| "validate frontmatter", "check frontmatter", "fix frontmatter", "frontmatter audit", "brain lint" | `skills/frontmatter-guard/SKILL.md` |
|
||||
|
||||
## Content & media ingestion
|
||||
|
||||
@@ -5233,6 +5235,75 @@ in depth, not the primary boundary.
|
||||
|
||||
---
|
||||
|
||||
## v0.22.4 — frontmatter-guard adoption
|
||||
|
||||
### 1. Stop hand-rolling frontmatter validators
|
||||
|
||||
If your fork has scripts that call `js-yaml` directly to validate brain page
|
||||
frontmatter, replace them with `gbrain frontmatter validate` calls. The CLI
|
||||
covers the seven canonical error classes and ships a `--json` envelope that's
|
||||
stable across releases.
|
||||
|
||||
```diff
|
||||
- # Custom validator script
|
||||
- node scripts/validate-frontmatter.mjs <path>
|
||||
+ gbrain frontmatter validate <path> --json
|
||||
```
|
||||
|
||||
For consumers that need the validator inside another script, import from
|
||||
gbrain's `markdown` export instead of duplicating logic:
|
||||
|
||||
```ts
|
||||
import { parseMarkdown } from 'gbrain/markdown';
|
||||
|
||||
const parsed = parseMarkdown(content, filePath, { validate: true, expectedSlug });
|
||||
for (const err of parsed.errors ?? []) {
|
||||
// err.code: MISSING_OPEN | MISSING_CLOSE | YAML_PARSE | SLUG_MISMATCH |
|
||||
// NULL_BYTES | NESTED_QUOTES | EMPTY_FRONTMATTER
|
||||
}
|
||||
```
|
||||
|
||||
### 2. Drop any references to `lib/brain-writer.mjs`
|
||||
|
||||
If your fork's skills or scripts referenced an aspirational
|
||||
`lib/brain-writer.mjs` (it never shipped — the spec was in PR #392 and never
|
||||
landed), replace those references with the gbrain CLI. The `frontmatter-guard`
|
||||
skill lives at `skills/frontmatter-guard/SKILL.md` and points at
|
||||
`gbrain frontmatter validate` / `audit` / `install-hook`.
|
||||
|
||||
### 3. Wire the doctor subcheck into your health pipeline
|
||||
|
||||
`gbrain doctor` now reports `frontmatter_integrity` automatically. If your
|
||||
fork has a custom health pipeline (e.g. a daily Slack post about brain
|
||||
health), pull from `gbrain doctor --json` and surface the
|
||||
`frontmatter_integrity` row counts.
|
||||
|
||||
### 4. (Optional) Install the pre-commit hook on brain repos
|
||||
|
||||
For sources backed by git, the v0.22.4 install-hook helper drops a
|
||||
pre-commit script that blocks commits with malformed frontmatter:
|
||||
|
||||
```bash
|
||||
gbrain frontmatter install-hook
|
||||
```
|
||||
|
||||
Skip this if your brain isn't a git repo or if your downstream agent already
|
||||
enforces validation at write time. See `docs/integrations/pre-commit.md` for
|
||||
the full recipe.
|
||||
|
||||
### 5. Migration ergonomics — read pending-host-work.jsonl
|
||||
|
||||
After `gbrain apply-migrations --yes` runs the v0.22.4 audit, your agent
|
||||
should read `~/.gbrain/migrations/pending-host-work.jsonl` (filter to
|
||||
`migration === "0.22.4"`) and walk each entry's `command` field. Each entry
|
||||
points to a per-source `gbrain frontmatter validate <source_path> --fix`
|
||||
command — surface counts to the user, get explicit consent, then run.
|
||||
|
||||
The migration is **audit-only**. It never mutates brain content during
|
||||
`apply-migrations`. Your agent runs the fix command with user consent.
|
||||
|
||||
---
|
||||
|
||||
## Future versions
|
||||
|
||||
When gbrain ships a new version, this doc will be updated with the diffs for that
|
||||
|
||||
+1
-1
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "gbrain",
|
||||
"version": "0.22.2",
|
||||
"version": "0.22.4",
|
||||
"description": "Postgres-native personal knowledge brain with hybrid RAG search",
|
||||
"type": "module",
|
||||
"main": "src/core/index.ts",
|
||||
|
||||
+3
-1
@@ -13,13 +13,15 @@ This is the dispatcher. Skills are the implementation. **Read the skill file bef
|
||||
|
||||
| Trigger | Skill |
|
||||
|---------|-------|
|
||||
| "What do we know about", "tell me about", "search for" | `skills/query/SKILL.md` |
|
||||
| "What do we know about", "tell me about", "search for", "who is", "background on", "notes on" | `skills/query/SKILL.md` |
|
||||
| "Who knows who", "relationship between", "connections", "graph query" | `skills/query/SKILL.md` (use graph-query) |
|
||||
| Creating/enriching a person or company page | `skills/enrich/SKILL.md` |
|
||||
| Where does a new file go? Filing rules | `skills/repo-architecture/SKILL.md` |
|
||||
| Fix broken citations in brain pages | `skills/citation-fixer/SKILL.md` |
|
||||
| "citation audit", "check citations", "fix citations" | `skills/citation-fixer/SKILL.md` (focused fix). For broader brain health, chain into `skills/maintain/SKILL.md` |
|
||||
| "Research", "track", "extract from email", "investor updates", "donations" | `skills/data-research/SKILL.md` |
|
||||
| Share a brain page as a link | `skills/publish/SKILL.md` |
|
||||
| "validate frontmatter", "check frontmatter", "fix frontmatter", "frontmatter audit", "brain lint" | `skills/frontmatter-guard/SKILL.md` |
|
||||
|
||||
## Content & media ingestion
|
||||
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
// Routing eval fixtures for skills/citation-fixer. Check 5 (W2, v0.17).
|
||||
// Layer A (structural) requires intents to contain trigger words from
|
||||
// the resolver. Paraphrase the trigger framing, not its meaning.
|
||||
{"intent": "please fix broken citations across the latest batch of pages", "expected_skill": "citation-fixer"}
|
||||
{"intent": "I think we need to fix broken citations in these brain pages", "expected_skill": "citation-fixer"}
|
||||
{"intent": "please fix citations in the latest batch of brain pages", "expected_skill": "citation-fixer"}
|
||||
{"intent": "I need to fix citations across these pages", "expected_skill": "citation-fixer"}
|
||||
// Negative case: something that sounds similar but should NOT route here.
|
||||
{"intent": "What does this book say about mentorship", "expected_skill": null, "ambiguous_with": []}
|
||||
|
||||
+1
-12
@@ -55,18 +55,7 @@ they building, what makes them tick, where are they headed.
|
||||
|
||||
## Citation Requirements (MANDATORY)
|
||||
|
||||
Every fact must carry an inline `[Source: ...]` citation.
|
||||
|
||||
Three formats:
|
||||
- **Direct attribution:** `[Source: User, {context}, YYYY-MM-DD]`
|
||||
- **API/external:** `[Source: {provider} enrichment, YYYY-MM-DD]`
|
||||
- **Synthesis:** `[Source: compiled from {list of sources}]`
|
||||
|
||||
Source precedence (highest to lowest):
|
||||
1. User's direct statements
|
||||
2. Compiled truth (pre-existing brain synthesis)
|
||||
3. Timeline entries (raw evidence)
|
||||
4. External sources (API enrichment, web search)
|
||||
> **Convention:** see `skills/conventions/quality.md` for citation formats and source precedence.
|
||||
|
||||
When sources conflict, note the contradiction with both citations.
|
||||
|
||||
|
||||
@@ -0,0 +1,180 @@
|
||||
---
|
||||
name: frontmatter-guard
|
||||
version: 1.0.0
|
||||
description: |
|
||||
Validate and auto-repair YAML frontmatter on brain pages. Catches malformed
|
||||
pages before they enter the brain (missing closing ---, nested quotes, slug
|
||||
mismatches, null bytes, empty frontmatter, YAML parse failures). Wraps the
|
||||
`gbrain frontmatter` CLI for agent-driven workflows.
|
||||
triggers:
|
||||
- "validate frontmatter"
|
||||
- "check frontmatter"
|
||||
- "fix frontmatter"
|
||||
- "frontmatter audit"
|
||||
- "brain lint"
|
||||
tools:
|
||||
- exec
|
||||
mutating: true
|
||||
---
|
||||
|
||||
# Frontmatter Guard Skill
|
||||
|
||||
> **Convention:** see `skills/conventions/quality.md` for citation rules; this skill is structural validation, not citation auditing.
|
||||
|
||||
## Contract
|
||||
|
||||
This skill guarantees:
|
||||
- Every brain page is scanned against the seven canonical frontmatter validation classes
|
||||
- Mechanical errors (nested quotes, missing closing `---`, null bytes, slug mismatch) are auto-repairable on demand with `.bak` backups
|
||||
- Validation logic is shared with `gbrain doctor`'s `frontmatter_integrity` subcheck — single source of truth
|
||||
- Reports per source (gbrain is multi-source since v0.18.0); never silently audits the wrong root
|
||||
|
||||
## Why This Exists
|
||||
|
||||
Brain pages pile up over months. Agents write them with malformed frontmatter:
|
||||
- Missing closing `---` (entity detector bugs)
|
||||
- Unstructured YAML in meeting pages (ingestion bugs)
|
||||
- Slug mismatches (path renames not propagated)
|
||||
- Null bytes (binary corruption from copy-paste accidents)
|
||||
- Nested double quotes in titles (`title: "Phil "Nick" Last"`)
|
||||
|
||||
Without a guard, these accumulate silently until `gbrain sync` chokes or search returns garbage. The guard makes the failure visible at audit time and trivially fixable.
|
||||
|
||||
## Validation classes
|
||||
|
||||
| Code | Meaning | Auto-fixable? |
|
||||
|------|---------|---------------|
|
||||
| `MISSING_OPEN` | File doesn't start with `---` | No (needs human) |
|
||||
| `MISSING_CLOSE` | No closing `---` before first heading | Yes |
|
||||
| `YAML_PARSE` | YAML failed to parse | Sometimes (depends on cause) |
|
||||
| `SLUG_MISMATCH` | Frontmatter `slug:` differs from path-derived slug | Yes (removes the field) |
|
||||
| `NULL_BYTES` | Binary corruption (`\x00`) | Yes |
|
||||
| `NESTED_QUOTES` | `title: "outer "inner" outer"` shape | Yes |
|
||||
| `EMPTY_FRONTMATTER` | Open + close present but nothing between | No (needs human) |
|
||||
|
||||
## Phases
|
||||
|
||||
### Phase 1: Audit
|
||||
|
||||
Run a read-only scan across all registered sources (or one with `--source <id>`).
|
||||
|
||||
```bash
|
||||
gbrain frontmatter audit --json
|
||||
```
|
||||
|
||||
Reports:
|
||||
- Per-source counts grouped by error code
|
||||
- Sample of up to 20 affected pages per source
|
||||
- Total count
|
||||
- Scan timestamp
|
||||
|
||||
Output is JSON; agents parse `errors_by_code` and `per_source` to decide next steps.
|
||||
|
||||
### Phase 2: Validate one path
|
||||
|
||||
Validate a single file or directory (does not require source registration):
|
||||
|
||||
```bash
|
||||
gbrain frontmatter validate <path> --json
|
||||
```
|
||||
|
||||
Exit code 0 = clean; 1 = errors found. Use this in CI pipelines or pre-commit hooks.
|
||||
|
||||
### Phase 3: Fix
|
||||
|
||||
When issues are found:
|
||||
|
||||
```bash
|
||||
gbrain frontmatter validate <path> --fix
|
||||
```
|
||||
|
||||
`--fix` writes `<file>.bak` for every modified file before mutating. The backup is the safety contract — works whether the brain is a git repo or a plain directory.
|
||||
|
||||
`--dry-run` previews without writing. Use this before applying fixes in batch.
|
||||
|
||||
### Phase 4: Pre-commit hook (optional)
|
||||
|
||||
For brain repos that ARE git repos, install the pre-commit hook to block malformed pages from being committed in the first place:
|
||||
|
||||
```bash
|
||||
gbrain frontmatter install-hook [--source <id>]
|
||||
```
|
||||
|
||||
The hook runs `gbrain frontmatter validate` against staged `.md`/`.mdx` files. Bypass with `git commit --no-verify`.
|
||||
|
||||
## Trigger words
|
||||
|
||||
When the user says any of these, route here:
|
||||
- "validate frontmatter"
|
||||
- "check frontmatter"
|
||||
- "fix frontmatter"
|
||||
- "frontmatter audit"
|
||||
- "brain lint"
|
||||
|
||||
## Output rules
|
||||
|
||||
- Always run `gbrain frontmatter audit --json` first; never assume a brain is clean.
|
||||
- Surface counts to the user in plain language; do not dump raw JSON.
|
||||
- For `--fix` operations: state how many files will be modified BEFORE running, then confirm.
|
||||
- `SLUG_MISMATCH` fixes remove the frontmatter `slug:` field — gbrain derives slug from path. Mention this when the user's title is intentionally renamed.
|
||||
- Never auto-fix `MISSING_OPEN` or `EMPTY_FRONTMATTER` without explicit user input — these usually mean a human author started a page and didn't finish.
|
||||
|
||||
## Chains with
|
||||
|
||||
- `gbrain doctor` — the `frontmatter_integrity` subcheck reports the same counts as `audit`.
|
||||
- `skills/maintain/SKILL.md` — broader brain health audit; chain after this skill if other classes of issue are suspected.
|
||||
- `skills/lint/SKILL.md` (via `gbrain lint`) — overlapping rules for skill-file lint; the `frontmatter-*` rule names in lint output come from this skill's validation surface.
|
||||
|
||||
## Output Format
|
||||
|
||||
Audit summary (terse, agent-friendly):
|
||||
|
||||
```
|
||||
Frontmatter audit — 17 issue(s) across 1 source(s)
|
||||
|
||||
[default] /Users/me/brain
|
||||
17 issue(s)
|
||||
MISSING_CLOSE: 8
|
||||
NESTED_QUOTES: 5
|
||||
NULL_BYTES: 4
|
||||
sample:
|
||||
people/jane.md — MISSING_CLOSE
|
||||
companies/acme.md — NESTED_QUOTES
|
||||
(+ 12 more)
|
||||
|
||||
Fix with: gbrain frontmatter validate /Users/me/brain --fix
|
||||
```
|
||||
|
||||
JSON envelope (when `--json` is passed):
|
||||
|
||||
```json
|
||||
{
|
||||
"ok": false,
|
||||
"total": 17,
|
||||
"errors_by_code": { "MISSING_CLOSE": 8, "NESTED_QUOTES": 5, "NULL_BYTES": 4 },
|
||||
"per_source": [
|
||||
{
|
||||
"source_id": "default",
|
||||
"source_path": "/Users/me/brain",
|
||||
"total": 17,
|
||||
"errors_by_code": { "MISSING_CLOSE": 8, "NESTED_QUOTES": 5, "NULL_BYTES": 4 },
|
||||
"sample": [{ "path": "people/jane.md", "codes": ["MISSING_CLOSE"] }]
|
||||
}
|
||||
],
|
||||
"scanned_at": "2026-04-25T22:30:00.000Z"
|
||||
}
|
||||
```
|
||||
|
||||
`gbrain frontmatter validate <path> --json` returns a similar envelope keyed on per-file results instead of per-source.
|
||||
|
||||
## Anti-Patterns
|
||||
|
||||
**Don't auto-fix `MISSING_OPEN` or `EMPTY_FRONTMATTER` without user input.** These usually mean a human author started a page and didn't finish — silently inserting `---` markers around an unfinished draft is wrong.
|
||||
|
||||
**Don't use `--fix` to "make doctor green" without reading the audit first.** SLUG_MISMATCH cases are surfaced for manual review specifically because gbrain derives the slug from path. A mismatch usually means the user renamed a file intentionally; auto-removing the slug field is the right outcome only when you've confirmed the rename was deliberate.
|
||||
|
||||
**Don't skip the `.bak` backups.** The `.bak` is the safety contract for non-git brain repos. If `.bak` files accumulate after a fix run, that's a feature, not a bug — the user can review the diffs and delete the backups when satisfied.
|
||||
|
||||
**Don't run `audit` on a brain where sources aren't registered.** The CLI returns "no registered sources to audit" gracefully, but the migration emits a `skipped: no_sources` phase result. Don't paper over this with a manual path-walk; the right fix is to register the source via `gbrain sources add`.
|
||||
|
||||
**Don't install the pre-commit hook on non-git brain dirs.** The install-hook command skips them automatically with a one-line note. If you see "skipped — not a git repo" and want validation at write time anyway, use the `audit` command on a cron schedule.
|
||||
@@ -0,0 +1,8 @@
|
||||
// Routing eval fixtures for skills/frontmatter-guard. Check 5 (W2, v0.17).
|
||||
// Layer A (structural) requires intents to contain trigger words from
|
||||
// the resolver. Paraphrase the trigger framing, not its meaning.
|
||||
{"intent": "please validate frontmatter on the latest batch of brain pages", "expected_skill": "frontmatter-guard"}
|
||||
{"intent": "fix frontmatter on these pages", "expected_skill": "frontmatter-guard"}
|
||||
{"intent": "I want to run a frontmatter audit across the brain", "expected_skill": "frontmatter-guard"}
|
||||
// Negative case: something that sounds similar but should NOT route here.
|
||||
{"intent": "what's for breakfast", "expected_skill": null, "ambiguous_with": []}
|
||||
@@ -8,7 +8,6 @@ description: |
|
||||
triggers:
|
||||
- "brain health"
|
||||
- "check backlinks"
|
||||
- "citation audit"
|
||||
- "maintenance"
|
||||
- "orphan pages"
|
||||
- "stale pages"
|
||||
|
||||
@@ -44,6 +44,11 @@
|
||||
"path": "publish/SKILL.md",
|
||||
"description": "Share brain pages as beautiful password-protected HTML (code + skill pair, zero LLM calls)"
|
||||
},
|
||||
{
|
||||
"name": "frontmatter-guard",
|
||||
"path": "frontmatter-guard/SKILL.md",
|
||||
"description": "Validate and auto-repair YAML frontmatter on brain pages; gates against malformed YAML, missing closing ---, nested quotes, slug mismatches, null bytes"
|
||||
},
|
||||
{
|
||||
"name": "signal-detector",
|
||||
"path": "signal-detector/SKILL.md",
|
||||
|
||||
@@ -0,0 +1,113 @@
|
||||
# v0.22.4 — Frontmatter Guard
|
||||
|
||||
## What ships
|
||||
|
||||
- `gbrain frontmatter` CLI (validate / audit / install-hook)
|
||||
- `frontmatter_integrity` subcheck under `gbrain doctor`
|
||||
- New `frontmatter-guard` skill (`skills/frontmatter-guard/SKILL.md`)
|
||||
- Pre-commit hook helper for git-backed brain repos
|
||||
- Audit-only migration that scans every registered source, writes
|
||||
`~/.gbrain/migrations/v0.22.4-audit.json`, and queues per-source TODO
|
||||
entries to `~/.gbrain/migrations/pending-host-work.jsonl`
|
||||
- 0 warnings on `gbrain check-resolvable` (down from 7 on master)
|
||||
|
||||
## What the agent should do post-upgrade
|
||||
|
||||
The orchestrator handles the mechanical side. Your job is to surface the audit
|
||||
to the user and apply fixes with their consent.
|
||||
|
||||
### 1. Run the orchestrator
|
||||
|
||||
```bash
|
||||
gbrain apply-migrations --yes
|
||||
```
|
||||
|
||||
This runs three idempotent phases:
|
||||
|
||||
- `schema` (no-op in v0.22.4)
|
||||
- `audit` — `scanBrainSources()` over every registered source; writes
|
||||
`~/.gbrain/migrations/v0.22.4-audit.json`
|
||||
- `emit-todo` — appends one entry per source-with-issues to
|
||||
`~/.gbrain/migrations/pending-host-work.jsonl`. Each entry contains the
|
||||
exact `gbrain frontmatter validate <source-path> --fix` command.
|
||||
|
||||
The migration **never mutates brain pages**. Auto-fixes only run when the user
|
||||
explicitly invokes the validate-with-fix command below.
|
||||
|
||||
### 2. Read the audit report
|
||||
|
||||
```bash
|
||||
cat ~/.gbrain/migrations/v0.22.4-audit.json
|
||||
```
|
||||
|
||||
The shape:
|
||||
|
||||
```json
|
||||
{
|
||||
"ok": false,
|
||||
"total": 17,
|
||||
"errors_by_code": { "MISSING_CLOSE": 8, "NESTED_QUOTES": 5, "NULL_BYTES": 4 },
|
||||
"per_source": [
|
||||
{
|
||||
"source_id": "default",
|
||||
"source_path": "/Users/me/brain",
|
||||
"total": 17,
|
||||
"errors_by_code": { "MISSING_CLOSE": 8, "NESTED_QUOTES": 5, "NULL_BYTES": 4 },
|
||||
"sample": [
|
||||
{ "path": "people/jane.md", "codes": ["MISSING_CLOSE"] }
|
||||
]
|
||||
}
|
||||
],
|
||||
"scanned_at": "2026-04-25T22:30:00.000Z"
|
||||
}
|
||||
```
|
||||
|
||||
### 3. Surface the report to the user
|
||||
|
||||
State the per-source counts in plain language. Example:
|
||||
|
||||
> "v0.22.4 ships frontmatter-guard. I ran an audit and found 17 issues across
|
||||
> 1 source (default: 8 MISSING_CLOSE, 5 NESTED_QUOTES, 4 NULL_BYTES). The
|
||||
> mechanical errors are auto-fixable; SLUG_MISMATCH cases (if any) need your
|
||||
> review. Want me to fix the auto-fixable ones now?"
|
||||
|
||||
### 4. Run the fix (with consent)
|
||||
|
||||
Per source with issues, the queued command is:
|
||||
|
||||
```bash
|
||||
gbrain frontmatter validate <source_path> --fix
|
||||
```
|
||||
|
||||
`--fix` writes a `.bak` backup for every modified file. SLUG_MISMATCH errors
|
||||
are surfaced for manual review (not auto-fixed) — gbrain derives slugs from
|
||||
path, so a mismatched slug usually means the user renamed the file
|
||||
intentionally or the slug field is stale.
|
||||
|
||||
### 5. (Optional) Install the pre-commit hook
|
||||
|
||||
For git-backed sources only:
|
||||
|
||||
```bash
|
||||
gbrain frontmatter install-hook [--source <id>]
|
||||
```
|
||||
|
||||
This blocks future malformed-frontmatter commits at the git layer. Bypass with
|
||||
`git commit --no-verify`. Skip this step for non-git brains.
|
||||
|
||||
### 6. Verify
|
||||
|
||||
```bash
|
||||
gbrain doctor --json | jq '.checks[] | select(.name == "frontmatter_integrity")'
|
||||
gbrain frontmatter audit --json | jq '.total'
|
||||
```
|
||||
|
||||
Both should report 0 issues after fixes are applied.
|
||||
|
||||
### 7. If anything fails
|
||||
|
||||
Open an issue at https://github.com/garrytan/gbrain/issues with:
|
||||
- output of `gbrain doctor`
|
||||
- contents of `~/.gbrain/migrations/v0.22.4-audit.json`
|
||||
- contents of `~/.gbrain/upgrade-errors.jsonl` if it exists
|
||||
- which step broke
|
||||
@@ -12,6 +12,8 @@ triggers:
|
||||
- "what happened"
|
||||
- "search for"
|
||||
- "look up"
|
||||
- "background on"
|
||||
- "notes on"
|
||||
- "who knows who"
|
||||
- "relationship between"
|
||||
- "connections"
|
||||
|
||||
+6
-1
@@ -19,7 +19,7 @@ for (const op of operations) {
|
||||
}
|
||||
|
||||
// CLI-only commands that bypass the operation layer
|
||||
const CLI_ONLY = new Set(['init', 'upgrade', 'post-upgrade', 'check-update', 'integrations', 'publish', 'check-backlinks', 'lint', 'report', 'import', 'export', 'files', 'embed', 'serve', 'call', 'config', 'doctor', 'migrate', 'eval', 'sync', 'extract', 'features', 'autopilot', 'graph-query', 'jobs', 'agent', 'apply-migrations', 'skillpack-check', 'skillpack', 'resolvers', 'integrity', 'repair-jsonb', 'orphans', 'sources', 'dream', 'check-resolvable', 'routing-eval', 'skillify', 'smoke-test', 'repos', 'code-def', 'code-refs', 'reindex-code', 'code-callers', 'code-callees']);
|
||||
const CLI_ONLY = new Set(['init', 'upgrade', 'post-upgrade', 'check-update', 'integrations', 'publish', 'check-backlinks', 'lint', 'report', 'import', 'export', 'files', 'embed', 'serve', 'call', 'config', 'doctor', 'migrate', 'eval', 'sync', 'extract', 'features', 'autopilot', 'graph-query', 'jobs', 'agent', 'apply-migrations', 'skillpack-check', 'skillpack', 'resolvers', 'integrity', 'repair-jsonb', 'orphans', 'sources', 'dream', 'check-resolvable', 'routing-eval', 'skillify', 'smoke-test', 'repos', 'code-def', 'code-refs', 'reindex-code', 'code-callers', 'code-callees', 'frontmatter']);
|
||||
|
||||
async function main() {
|
||||
// Parse global flags (--quiet / --progress-json / --progress-interval)
|
||||
@@ -305,6 +305,11 @@ async function handleCliOnly(command: string, args: string[]) {
|
||||
await runBacklinks(args);
|
||||
return;
|
||||
}
|
||||
if (command === 'frontmatter') {
|
||||
const { runFrontmatter } = await import('./commands/frontmatter.ts');
|
||||
await runFrontmatter(args);
|
||||
return;
|
||||
}
|
||||
if (command === 'lint') {
|
||||
const { runLint } = await import('./commands/lint.ts');
|
||||
await runLint(args);
|
||||
|
||||
@@ -668,6 +668,53 @@ export async function runDoctor(engine: BrainEngine | null, args: string[], dbSo
|
||||
mbcHb();
|
||||
}
|
||||
|
||||
// 11a. Frontmatter integrity (v0.22.4).
|
||||
// scanBrainSources walks every registered source's local_path on disk
|
||||
// (not from the DB), invoking parseMarkdown(..., {validate:true}) per
|
||||
// file. Reports per-source counts grouped by error code. The fix path is
|
||||
// `gbrain frontmatter validate <source-path> --fix`, which writes .bak
|
||||
// backups so it works for both git and non-git brain repos.
|
||||
progress.heartbeat('frontmatter_integrity');
|
||||
const fmHb = startHeartbeat(progress, 'scanning frontmatter…');
|
||||
try {
|
||||
const { scanBrainSources } = await import('../core/brain-writer.ts');
|
||||
const report = await scanBrainSources(engine);
|
||||
if (report.total === 0) {
|
||||
const sources = report.per_source.length;
|
||||
checks.push({
|
||||
name: 'frontmatter_integrity',
|
||||
status: 'ok',
|
||||
message: sources === 0
|
||||
? 'No registered sources to scan'
|
||||
: `${sources} source(s) clean — no frontmatter issues`,
|
||||
});
|
||||
} else {
|
||||
const sourceMessages: string[] = [];
|
||||
for (const src of report.per_source) {
|
||||
if (src.total === 0) continue;
|
||||
const codes = Object.entries(src.errors_by_code)
|
||||
.map(([k, v]) => `${k}=${v}`)
|
||||
.join(', ');
|
||||
sourceMessages.push(`${src.source_id}: ${src.total} (${codes})`);
|
||||
}
|
||||
checks.push({
|
||||
name: 'frontmatter_integrity',
|
||||
status: 'warn',
|
||||
message:
|
||||
`${report.total} frontmatter issue(s) across ${sourceMessages.length} source(s). ` +
|
||||
`${sourceMessages.join('; ')}. Fix: gbrain frontmatter validate <source-path> --fix`,
|
||||
});
|
||||
}
|
||||
} catch (e) {
|
||||
checks.push({
|
||||
name: 'frontmatter_integrity',
|
||||
status: 'warn',
|
||||
message: `Could not scan frontmatter: ${e instanceof Error ? e.message : String(e)}`,
|
||||
});
|
||||
} finally {
|
||||
fmHb();
|
||||
}
|
||||
|
||||
// 11b. Queue health (v0.19.1 queue-resilience wave).
|
||||
// Postgres-only because PGLite has no multi-process worker surface. Two
|
||||
// subchecks, both cheap (single SELECT each, status-index-covered):
|
||||
|
||||
@@ -0,0 +1,216 @@
|
||||
/**
|
||||
* gbrain frontmatter install-hook — Install a pre-commit hook in a brain
|
||||
* source's git repo that runs `gbrain frontmatter validate` against staged
|
||||
* .md/.mdx files. Skips non-git sources with a one-line note.
|
||||
*
|
||||
* Usage:
|
||||
* gbrain frontmatter install-hook [--source <id>] [--force] [--uninstall]
|
||||
*
|
||||
* --source <id> Limit to one registered source. Default: all sources.
|
||||
* --force Overwrite an existing pre-commit hook (writes <hook>.bak).
|
||||
* --uninstall Remove the hook; restore <hook>.bak if present.
|
||||
*
|
||||
* Hook contract:
|
||||
* - Located at <source>/.githooks/pre-commit. We `git config core.hooksPath
|
||||
* .githooks` if no other hooksPath is set.
|
||||
* - When the gbrain binary is missing, the hook prints a one-line warning
|
||||
* and exits 0 (don't break commits if a developer uninstalls gbrain).
|
||||
* - Bypass via `git commit --no-verify`.
|
||||
*/
|
||||
|
||||
import { existsSync, readFileSync, writeFileSync, mkdirSync, chmodSync, rmSync, copyFileSync } from 'fs';
|
||||
import { join } from 'path';
|
||||
import { execFileSync } from 'child_process';
|
||||
import type { BrainEngine } from '../core/engine.ts';
|
||||
import { loadConfig, toEngineConfig } from '../core/config.ts';
|
||||
import { createEngine } from '../core/engine-factory.ts';
|
||||
|
||||
const HOOK_BANNER = '# gbrain frontmatter pre-commit hook (v0.22.4+)';
|
||||
|
||||
const HOOK_SCRIPT = `#!/bin/sh
|
||||
${HOOK_BANNER}
|
||||
# Validates YAML frontmatter on staged .md / .mdx files. Bypass with
|
||||
# 'git commit --no-verify'. Uninstall with 'gbrain frontmatter install-hook --uninstall'.
|
||||
|
||||
set -e
|
||||
|
||||
if ! command -v gbrain >/dev/null 2>&1; then
|
||||
echo "gbrain not on PATH; skipping frontmatter pre-commit (install gbrain to re-enable)." >&2
|
||||
exit 0
|
||||
fi
|
||||
|
||||
staged=$(git diff --cached --name-only --diff-filter=ACM | grep -E '\\\\.mdx?$' || true)
|
||||
[ -z "$staged" ] && exit 0
|
||||
|
||||
failed=0
|
||||
for f in $staged; do
|
||||
[ -f "$f" ] || continue
|
||||
if ! gbrain frontmatter validate "$f" >/dev/null 2>&1; then
|
||||
gbrain frontmatter validate "$f" >&2
|
||||
failed=1
|
||||
fi
|
||||
done
|
||||
|
||||
if [ $failed -ne 0 ]; then
|
||||
echo "" >&2
|
||||
echo "Frontmatter validation failed. Run 'gbrain frontmatter validate <file> --fix' to repair, or 'git commit --no-verify' to bypass." >&2
|
||||
exit 1
|
||||
fi
|
||||
`;
|
||||
|
||||
interface SourceRow {
|
||||
id: string;
|
||||
local_path: string | null;
|
||||
}
|
||||
|
||||
export async function runFrontmatterInstallHook(args: string[]): Promise<void> {
|
||||
let force = false;
|
||||
let uninstall = false;
|
||||
let sourceId: string | undefined;
|
||||
let help = false;
|
||||
for (let i = 0; i < args.length; i++) {
|
||||
const a = args[i];
|
||||
if (a === '--help' || a === '-h') help = true;
|
||||
else if (a === '--force') force = true;
|
||||
else if (a === '--uninstall') uninstall = true;
|
||||
else if (a === '--source') sourceId = args[++i];
|
||||
else if (a.startsWith('--source=')) sourceId = a.slice('--source='.length);
|
||||
}
|
||||
|
||||
if (help) {
|
||||
printHelp();
|
||||
return;
|
||||
}
|
||||
|
||||
const config = loadConfig();
|
||||
if (!config) {
|
||||
throw new Error('No brain configured. Run: gbrain init');
|
||||
}
|
||||
const engineConfig = toEngineConfig(config);
|
||||
const engine = await createEngine(engineConfig);
|
||||
await engine.connect(engineConfig);
|
||||
try {
|
||||
const sources = await listSources(engine, sourceId);
|
||||
if (sources.length === 0) {
|
||||
console.log(sourceId
|
||||
? `Source "${sourceId}" not found.`
|
||||
: 'No registered sources. Run `gbrain sources list` to inspect.');
|
||||
return;
|
||||
}
|
||||
|
||||
let installed = 0;
|
||||
let skipped = 0;
|
||||
for (const src of sources) {
|
||||
if (!src.local_path || !existsSync(src.local_path)) {
|
||||
console.log(`[${src.id}] skipped — local_path missing on disk`);
|
||||
skipped++;
|
||||
continue;
|
||||
}
|
||||
if (!isGitRepo(src.local_path)) {
|
||||
console.log(`[${src.id}] ${src.local_path} — skipped, not a git repo`);
|
||||
skipped++;
|
||||
continue;
|
||||
}
|
||||
if (uninstall) {
|
||||
if (uninstallHook(src.local_path)) {
|
||||
console.log(`[${src.id}] hook removed`);
|
||||
installed++;
|
||||
} else {
|
||||
console.log(`[${src.id}] no gbrain pre-commit hook found; nothing to uninstall`);
|
||||
}
|
||||
continue;
|
||||
}
|
||||
const result = installHook(src.local_path, force);
|
||||
if (result === 'installed') {
|
||||
console.log(`[${src.id}] hook installed at .githooks/pre-commit`);
|
||||
installed++;
|
||||
} else if (result === 'skipped_existing') {
|
||||
console.log(`[${src.id}] existing pre-commit hook found; pass --force to overwrite (.bak created)`);
|
||||
skipped++;
|
||||
} else {
|
||||
console.log(`[${src.id}] hook already up to date`);
|
||||
}
|
||||
}
|
||||
|
||||
console.log(`\nDone. ${installed} ${uninstall ? 'removed' : 'installed/updated'}, ${skipped} skipped.`);
|
||||
} finally {
|
||||
await engine.disconnect();
|
||||
}
|
||||
}
|
||||
|
||||
function printHelp() {
|
||||
console.log(`gbrain frontmatter install-hook — install pre-commit hook in source git repos
|
||||
|
||||
Usage:
|
||||
gbrain frontmatter install-hook [--source <id>] [--force] [--uninstall]
|
||||
|
||||
The hook runs \`gbrain frontmatter validate\` against staged .md/.mdx files,
|
||||
blocking commits with malformed frontmatter. Bypass with 'git commit --no-verify'.
|
||||
|
||||
Options:
|
||||
--source <id> Limit to one registered source. Default: all sources.
|
||||
--force Overwrite an existing pre-commit hook (writes <hook>.bak).
|
||||
--uninstall Remove the hook; restore <hook>.bak if present.
|
||||
`);
|
||||
}
|
||||
|
||||
async function listSources(engine: BrainEngine, sourceId?: string): Promise<SourceRow[]> {
|
||||
if (sourceId) {
|
||||
return engine.executeRaw<SourceRow>(`SELECT id, local_path FROM sources WHERE id = $1`, [sourceId]);
|
||||
}
|
||||
return engine.executeRaw<SourceRow>(`SELECT id, local_path FROM sources WHERE local_path IS NOT NULL ORDER BY id`);
|
||||
}
|
||||
|
||||
function isGitRepo(dir: string): boolean {
|
||||
return existsSync(join(dir, '.git'));
|
||||
}
|
||||
|
||||
type InstallResult = 'installed' | 'skipped_existing' | 'unchanged';
|
||||
|
||||
export function installHook(repoPath: string, force: boolean): InstallResult {
|
||||
const hooksDir = join(repoPath, '.githooks');
|
||||
const hookPath = join(hooksDir, 'pre-commit');
|
||||
mkdirSync(hooksDir, { recursive: true });
|
||||
|
||||
if (existsSync(hookPath)) {
|
||||
const existing = readFileSync(hookPath, 'utf8');
|
||||
if (existing.includes(HOOK_BANNER)) {
|
||||
// Already a gbrain hook — refresh the script content silently.
|
||||
writeFileSync(hookPath, HOOK_SCRIPT);
|
||||
chmodSync(hookPath, 0o755);
|
||||
return 'unchanged';
|
||||
}
|
||||
if (!force) return 'skipped_existing';
|
||||
copyFileSync(hookPath, hookPath + '.bak');
|
||||
}
|
||||
|
||||
writeFileSync(hookPath, HOOK_SCRIPT);
|
||||
chmodSync(hookPath, 0o755);
|
||||
|
||||
// Set core.hooksPath unless the user has set it to something else already.
|
||||
try {
|
||||
const current = execFileSync('git', ['-C', repoPath, 'config', '--get', 'core.hooksPath'], { encoding: 'utf8' }).trim();
|
||||
if (current && current !== '.githooks') return 'installed';
|
||||
} catch {
|
||||
// git config returns non-zero when the key is unset; that's the normal case.
|
||||
}
|
||||
try {
|
||||
execFileSync('git', ['-C', repoPath, 'config', 'core.hooksPath', '.githooks']);
|
||||
} catch {
|
||||
// Best-effort. Hook still exists; user can configure manually.
|
||||
}
|
||||
return 'installed';
|
||||
}
|
||||
|
||||
export function uninstallHook(repoPath: string): boolean {
|
||||
const hookPath = join(repoPath, '.githooks', 'pre-commit');
|
||||
if (!existsSync(hookPath)) return false;
|
||||
const content = readFileSync(hookPath, 'utf8');
|
||||
if (!content.includes(HOOK_BANNER)) return false;
|
||||
rmSync(hookPath);
|
||||
if (existsSync(hookPath + '.bak')) {
|
||||
copyFileSync(hookPath + '.bak', hookPath);
|
||||
rmSync(hookPath + '.bak');
|
||||
}
|
||||
return true;
|
||||
}
|
||||
@@ -0,0 +1,299 @@
|
||||
/**
|
||||
* gbrain frontmatter — Frontmatter validation, audit, and auto-repair.
|
||||
*
|
||||
* Subcommands:
|
||||
* gbrain frontmatter validate <path> [--json] [--fix] [--dry-run]
|
||||
* Validate one file or recursively a directory. --fix writes .bak then
|
||||
* rewrites in place. --dry-run previews without writing.
|
||||
*
|
||||
* gbrain frontmatter audit [--source <id>] [--json]
|
||||
* Read-only scan across all registered sources (or one with --source).
|
||||
* Returns AuditReport-shaped JSON with --json.
|
||||
*
|
||||
* The audit subcommand is intentionally read-only; --fix only exists on
|
||||
* validate. Pass an explicit path to validate a non-source-registered tree.
|
||||
*/
|
||||
|
||||
import { readFileSync, writeFileSync, existsSync, lstatSync, readdirSync, copyFileSync } from 'fs';
|
||||
import { join, relative, resolve } from 'path';
|
||||
import type { BrainEngine } from '../core/engine.ts';
|
||||
import { loadConfig, toEngineConfig } from '../core/config.ts';
|
||||
import { createEngine } from '../core/engine-factory.ts';
|
||||
import { parseMarkdown, type ParseValidationCode } from '../core/markdown.ts';
|
||||
import {
|
||||
autoFixFrontmatter,
|
||||
scanBrainSources,
|
||||
type AuditReport,
|
||||
type AuditFix,
|
||||
} from '../core/brain-writer.ts';
|
||||
import { isSyncable, slugifyPath } from '../core/sync.ts';
|
||||
|
||||
export async function runFrontmatter(args: string[]): Promise<void> {
|
||||
const sub = args[0];
|
||||
if (!sub || sub === '--help' || sub === '-h') {
|
||||
printHelp();
|
||||
return;
|
||||
}
|
||||
const rest = args.slice(1);
|
||||
|
||||
if (sub === 'validate') {
|
||||
await runValidate(rest);
|
||||
return;
|
||||
}
|
||||
if (sub === 'audit') {
|
||||
const engine = await connectEngineForAudit();
|
||||
try {
|
||||
await runAudit(engine, rest);
|
||||
} finally {
|
||||
await engine.disconnect();
|
||||
}
|
||||
return;
|
||||
}
|
||||
if (sub === 'install-hook') {
|
||||
const { runFrontmatterInstallHook } = await import('./frontmatter-install-hook.ts');
|
||||
await runFrontmatterInstallHook(rest);
|
||||
return;
|
||||
}
|
||||
console.error(`Unknown frontmatter subcommand: ${sub}\n`);
|
||||
printHelp();
|
||||
process.exitCode = 1;
|
||||
}
|
||||
|
||||
async function connectEngineForAudit(): Promise<BrainEngine> {
|
||||
const config = loadConfig();
|
||||
if (!config) {
|
||||
throw new Error('No brain configured. Run: gbrain init');
|
||||
}
|
||||
const engineConfig = toEngineConfig(config);
|
||||
const engine = await createEngine(engineConfig);
|
||||
await engine.connect(engineConfig);
|
||||
return engine;
|
||||
}
|
||||
|
||||
function printHelp() {
|
||||
console.log(`gbrain frontmatter — frontmatter validation, audit, and auto-repair
|
||||
|
||||
Usage:
|
||||
gbrain frontmatter validate <path> [--json] [--fix] [--dry-run]
|
||||
gbrain frontmatter audit [--source <id>] [--json]
|
||||
gbrain frontmatter install-hook [--source <id>] [--force] [--uninstall]
|
||||
|
||||
validate
|
||||
Validate one .md file or recursively a directory. Each file is parsed via
|
||||
parseMarkdown(..., {validate:true}); errors are reported by code:
|
||||
MISSING_OPEN, MISSING_CLOSE, YAML_PARSE, SLUG_MISMATCH,
|
||||
NULL_BYTES, NESTED_QUOTES, EMPTY_FRONTMATTER
|
||||
|
||||
--fix Auto-repair the fixable subset (NULL_BYTES, MISSING_CLOSE,
|
||||
NESTED_QUOTES, SLUG_MISMATCH). Writes <file>.bak before any
|
||||
in-place rewrite. .bak is the safety contract; works for both
|
||||
git and non-git brain repos.
|
||||
--dry-run Preview --fix without writing.
|
||||
--json Emit a JSON envelope on stdout.
|
||||
|
||||
audit
|
||||
Read-only scan across all registered sources (or one with --source <id>).
|
||||
Reports per-source counts grouped by error code. Use this in CI or doctor
|
||||
pipelines. Exits 0 even when issues are found — the count is the signal.
|
||||
|
||||
--source <id> Limit scan to one registered source.
|
||||
--json Emit AuditReport-shaped JSON on stdout.
|
||||
`);
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// validate
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
interface ValidateFlags {
|
||||
json: boolean;
|
||||
fix: boolean;
|
||||
dryRun: boolean;
|
||||
}
|
||||
|
||||
interface FileValidation {
|
||||
path: string;
|
||||
errors: { code: ParseValidationCode; message: string; line?: number }[];
|
||||
fixesApplied?: AuditFix[];
|
||||
}
|
||||
|
||||
async function runValidate(rest: string[]): Promise<void> {
|
||||
const flags: ValidateFlags = { json: false, fix: false, dryRun: false };
|
||||
let target: string | null = null;
|
||||
for (const a of rest) {
|
||||
if (a === '--json') flags.json = true;
|
||||
else if (a === '--fix') flags.fix = true;
|
||||
else if (a === '--dry-run') flags.dryRun = true;
|
||||
else if (!a.startsWith('--')) target = a;
|
||||
}
|
||||
if (!target) {
|
||||
console.error('error: gbrain frontmatter validate requires a <path> argument');
|
||||
process.exitCode = 1;
|
||||
return;
|
||||
}
|
||||
|
||||
const resolved = resolve(target);
|
||||
if (!existsSync(resolved)) {
|
||||
console.error(`error: path not found: ${target}`);
|
||||
process.exitCode = 1;
|
||||
return;
|
||||
}
|
||||
|
||||
const files = collectFiles(resolved);
|
||||
const results: FileValidation[] = [];
|
||||
|
||||
for (const file of files) {
|
||||
const content = readFileSync(file, 'utf8');
|
||||
const expectedSlug = slugifyPath(relative(resolve(target), file) || file);
|
||||
const parsed = parseMarkdown(content, file, { validate: true, expectedSlug });
|
||||
const errs = parsed.errors ?? [];
|
||||
const result: FileValidation = {
|
||||
path: file,
|
||||
errors: errs.map(e => ({ code: e.code, message: e.message, line: e.line })),
|
||||
};
|
||||
|
||||
if (flags.fix && errs.length > 0) {
|
||||
const { content: fixed, fixes } = autoFixFrontmatter(content, { filePath: file });
|
||||
result.fixesApplied = fixes;
|
||||
if (fixes.length > 0 && !flags.dryRun) {
|
||||
copyFileSync(file, file + '.bak');
|
||||
writeFileSync(file, fixed, 'utf8');
|
||||
}
|
||||
}
|
||||
|
||||
results.push(result);
|
||||
}
|
||||
|
||||
const totalErrors = results.reduce((n, r) => n + r.errors.length, 0);
|
||||
const filesWithErrors = results.filter(r => r.errors.length > 0).length;
|
||||
const filesFixed = results.filter(r => (r.fixesApplied?.length ?? 0) > 0).length;
|
||||
|
||||
if (flags.json) {
|
||||
const envelope = {
|
||||
ok: totalErrors === 0,
|
||||
target: resolved,
|
||||
total_files: files.length,
|
||||
files_with_errors: filesWithErrors,
|
||||
total_errors: totalErrors,
|
||||
files_fixed: flags.fix ? filesFixed : undefined,
|
||||
dry_run: flags.dryRun || undefined,
|
||||
results,
|
||||
};
|
||||
console.log(JSON.stringify(envelope, null, 2));
|
||||
} else {
|
||||
if (totalErrors === 0) {
|
||||
console.log(`OK — ${files.length} file(s) scanned, no frontmatter issues`);
|
||||
} else {
|
||||
console.log(`Found ${totalErrors} issue(s) across ${filesWithErrors} file(s) (scanned ${files.length})`);
|
||||
for (const r of results) {
|
||||
if (r.errors.length === 0) continue;
|
||||
console.log(`\n${r.path}`);
|
||||
for (const e of r.errors) {
|
||||
const lineHint = e.line !== undefined ? `:${e.line}` : '';
|
||||
console.log(` [${e.code}]${lineHint} ${e.message}`);
|
||||
}
|
||||
if (r.fixesApplied && r.fixesApplied.length > 0) {
|
||||
const verb = flags.dryRun ? 'would fix' : 'fixed';
|
||||
for (const f of r.fixesApplied) {
|
||||
console.log(` ${verb}: ${f.description}`);
|
||||
}
|
||||
}
|
||||
}
|
||||
if (flags.fix && !flags.dryRun) {
|
||||
console.log(`\nWrote .bak backups for ${filesFixed} file(s).`);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
process.exitCode = totalErrors > 0 && !flags.fix ? 1 : 0;
|
||||
}
|
||||
|
||||
function collectFiles(target: string): string[] {
|
||||
const st = lstatSync(target);
|
||||
if (st.isFile()) {
|
||||
return [target];
|
||||
}
|
||||
const out: string[] = [];
|
||||
const stack = [target];
|
||||
while (stack.length > 0) {
|
||||
const dir = stack.pop()!;
|
||||
let entries: string[];
|
||||
try {
|
||||
entries = readdirSync(dir);
|
||||
} catch {
|
||||
continue;
|
||||
}
|
||||
for (const name of entries) {
|
||||
const full = join(dir, name);
|
||||
let entryStat: ReturnType<typeof lstatSync>;
|
||||
try {
|
||||
entryStat = lstatSync(full);
|
||||
} catch {
|
||||
continue;
|
||||
}
|
||||
if (entryStat.isSymbolicLink()) continue;
|
||||
if (entryStat.isDirectory()) {
|
||||
stack.push(full);
|
||||
} else if (entryStat.isFile()) {
|
||||
const rel = relative(target, full);
|
||||
if (isSyncable(rel, { strategy: 'markdown' })) {
|
||||
out.push(full);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// audit
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
async function runAudit(engine: BrainEngine, rest: string[]): Promise<void> {
|
||||
let json = false;
|
||||
let sourceId: string | undefined;
|
||||
for (let i = 0; i < rest.length; i++) {
|
||||
const a = rest[i];
|
||||
if (a === '--json') json = true;
|
||||
else if (a === '--source') sourceId = rest[++i];
|
||||
else if (a.startsWith('--source=')) sourceId = a.slice('--source='.length);
|
||||
}
|
||||
|
||||
const report = await scanBrainSources(engine, { sourceId });
|
||||
|
||||
if (json) {
|
||||
console.log(JSON.stringify(report, null, 2));
|
||||
return;
|
||||
}
|
||||
|
||||
printAuditHumanReport(report);
|
||||
}
|
||||
|
||||
function printAuditHumanReport(report: AuditReport): void {
|
||||
if (report.per_source.length === 0) {
|
||||
console.log('No registered sources to audit. Run `gbrain sources list` to inspect.');
|
||||
return;
|
||||
}
|
||||
console.log(`Frontmatter audit — ${report.total} issue(s) across ${report.per_source.length} source(s) (scanned at ${report.scanned_at})`);
|
||||
for (const src of report.per_source) {
|
||||
console.log(`\n[${src.source_id}] ${src.source_path}`);
|
||||
if (src.total === 0) {
|
||||
console.log(' clean');
|
||||
continue;
|
||||
}
|
||||
console.log(` ${src.total} issue(s)`);
|
||||
for (const [code, n] of Object.entries(src.errors_by_code)) {
|
||||
console.log(` ${code}: ${n}`);
|
||||
}
|
||||
if (src.sample.length > 0) {
|
||||
console.log(` sample:`);
|
||||
for (const s of src.sample.slice(0, 5)) {
|
||||
console.log(` ${s.path} — ${s.codes.join(', ')}`);
|
||||
}
|
||||
if (src.sample.length > 5) console.log(` (+ ${src.sample.length - 5} more)`);
|
||||
}
|
||||
}
|
||||
if (report.total > 0) {
|
||||
console.log(`\nFix with: gbrain frontmatter validate <source-path> --fix`);
|
||||
}
|
||||
}
|
||||
@@ -18,6 +18,7 @@
|
||||
|
||||
import { readFileSync, writeFileSync, readdirSync, statSync, lstatSync, existsSync } from 'fs';
|
||||
import { join, relative } from 'path';
|
||||
import { parseMarkdown, type ParseValidationCode } from '../core/markdown.ts';
|
||||
|
||||
export interface LintIssue {
|
||||
file: string;
|
||||
@@ -27,6 +28,25 @@ export interface LintIssue {
|
||||
fixable: boolean;
|
||||
}
|
||||
|
||||
/** Map of frontmatter validation codes to lint rule names. Stable across
|
||||
* releases — agents and CI consumers can target specific rule names. */
|
||||
const FRONTMATTER_RULE_NAMES: Record<ParseValidationCode, string> = {
|
||||
MISSING_OPEN: 'frontmatter-missing-open',
|
||||
MISSING_CLOSE: 'frontmatter-missing-close',
|
||||
YAML_PARSE: 'frontmatter-yaml-parse',
|
||||
SLUG_MISMATCH: 'frontmatter-slug-mismatch',
|
||||
NULL_BYTES: 'frontmatter-null-bytes',
|
||||
NESTED_QUOTES: 'frontmatter-nested-quotes',
|
||||
EMPTY_FRONTMATTER: 'frontmatter-empty',
|
||||
};
|
||||
|
||||
/** Codes whose lint findings are fixable by `gbrain frontmatter validate --fix`. */
|
||||
const FRONTMATTER_FIXABLE: ReadonlySet<ParseValidationCode> = new Set<ParseValidationCode>([
|
||||
'MISSING_CLOSE',
|
||||
'NULL_BYTES',
|
||||
'NESTED_QUOTES',
|
||||
]);
|
||||
|
||||
// ── LLM artifact patterns ──────────────────────────────────────────
|
||||
|
||||
const LLM_PREAMBLES = [
|
||||
@@ -44,6 +64,25 @@ export function lintContent(content: string, filePath: string): LintIssue[] {
|
||||
const issues: LintIssue[] = [];
|
||||
const lines = content.split('\n');
|
||||
|
||||
// ── Frontmatter validation (delegates to parseMarkdown(validate:true)) ──
|
||||
// This is the single source of truth for frontmatter shape rules. Each
|
||||
// ParseValidationCode maps to a stable lint rule name in
|
||||
// FRONTMATTER_RULE_NAMES. Keeps brain-page lint, doctor's
|
||||
// frontmatter_integrity subcheck, and the frontmatter CLI in lockstep.
|
||||
const parsed = parseMarkdown(content, filePath, { validate: true });
|
||||
for (const err of parsed.errors ?? []) {
|
||||
// Skip MISSING_OPEN — the legacy `no-frontmatter` rule below covers this
|
||||
// exact case with a stable rule name. Emitting both is double-reporting.
|
||||
if (err.code === 'MISSING_OPEN') continue;
|
||||
issues.push({
|
||||
file: filePath,
|
||||
line: err.line ?? 1,
|
||||
rule: FRONTMATTER_RULE_NAMES[err.code],
|
||||
message: err.message,
|
||||
fixable: FRONTMATTER_FIXABLE.has(err.code),
|
||||
});
|
||||
}
|
||||
|
||||
// Rule: LLM preamble artifacts
|
||||
for (const pattern of LLM_PREAMBLES) {
|
||||
pattern.lastIndex = 0;
|
||||
|
||||
@@ -21,6 +21,7 @@ import { v0_16_0 } from './v0_16_0.ts';
|
||||
import { v0_18_0 } from './v0_18_0.ts';
|
||||
import { v0_18_1 } from './v0_18_1.ts';
|
||||
import { v0_21_0 } from './v0_21_0.ts';
|
||||
import { v0_22_4 } from './v0_22_4.ts';
|
||||
|
||||
export const migrations: Migration[] = [
|
||||
v0_11_0,
|
||||
@@ -33,6 +34,7 @@ export const migrations: Migration[] = [
|
||||
v0_18_0,
|
||||
v0_18_1,
|
||||
v0_21_0,
|
||||
v0_22_4,
|
||||
];
|
||||
|
||||
/** Look up a migration by exact version string. */
|
||||
|
||||
@@ -0,0 +1,228 @@
|
||||
/**
|
||||
* v0.22.4 migration orchestrator — frontmatter-guard adoption.
|
||||
*
|
||||
* v0.22.4 ships a shared frontmatter validator (parseMarkdown(..., {validate:true})),
|
||||
* a doctor subcheck (frontmatter_integrity), a top-level `gbrain frontmatter`
|
||||
* CLI (validate / audit / install-hook), and a new `frontmatter-guard` skill.
|
||||
*
|
||||
* This migration is AUDIT-ONLY (per D5): it reads the user's brain pages,
|
||||
* writes a JSON report to ~/.gbrain/migrations/v0.22.4-audit.json, and emits
|
||||
* one entry per source-with-issues to ~/.gbrain/migrations/pending-host-work.jsonl.
|
||||
* It NEVER mutates brain content. The agent reads skills/migrations/v0.22.4.md
|
||||
* after upgrade and runs `gbrain frontmatter validate <source-path> --fix` with
|
||||
* explicit user consent.
|
||||
*
|
||||
* Phases (all idempotent):
|
||||
* A. Schema — no-op (no DB changes in v0.22.4).
|
||||
* B. Audit — scanBrainSources → write JSON report.
|
||||
* C. Emit-todo — append pending-host-work.jsonl entry per source with errors.
|
||||
* D. Record — runner-owned ledger write.
|
||||
*/
|
||||
|
||||
import { existsSync, mkdirSync, writeFileSync, readFileSync, appendFileSync } from 'fs';
|
||||
import { join } from 'path';
|
||||
import type { Migration, OrchestratorOpts, OrchestratorResult, OrchestratorPhaseResult } from './types.ts';
|
||||
import type { BrainEngine } from '../../core/engine.ts';
|
||||
import { loadConfig, toEngineConfig } from '../../core/config.ts';
|
||||
import { createEngine } from '../../core/engine-factory.ts';
|
||||
import { scanBrainSources, type AuditReport } from '../../core/brain-writer.ts';
|
||||
|
||||
/** Test-only injection point for the audit phase. When set, phaseBAudit uses
|
||||
* this engine instead of loading config + creating a fresh one. Mirrors the
|
||||
* repair-jsonb pattern. Reset to null in afterAll. */
|
||||
let testEngineOverride: BrainEngine | null = null;
|
||||
export function __setTestEngineOverride(engine: BrainEngine | null): void {
|
||||
testEngineOverride = engine;
|
||||
}
|
||||
|
||||
function gbrainDir(): string {
|
||||
return join(process.env.HOME || '', '.gbrain');
|
||||
}
|
||||
function migrationsDir(): string { return join(gbrainDir(), 'migrations'); }
|
||||
function auditReportPath(): string { return join(migrationsDir(), 'v0.22.4-audit.json'); }
|
||||
function pendingHostWorkPath(): string { return join(migrationsDir(), 'pending-host-work.jsonl'); }
|
||||
|
||||
interface PendingHostWorkEntry {
|
||||
migration: string;
|
||||
ts: string;
|
||||
skill: string;
|
||||
reason: string;
|
||||
source_id: string;
|
||||
source_path: string;
|
||||
command: string;
|
||||
}
|
||||
|
||||
// ── Phase A — Schema (no-op) ───────────────────────────────
|
||||
|
||||
function phaseASchema(opts: OrchestratorOpts): OrchestratorPhaseResult {
|
||||
if (opts.dryRun) return { name: 'schema', status: 'skipped', detail: 'dry-run' };
|
||||
return { name: 'schema', status: 'complete', detail: 'no schema changes in v0.22.4' };
|
||||
}
|
||||
|
||||
// ── Phase B — Audit ────────────────────────────────────────
|
||||
|
||||
async function phaseBAudit(opts: OrchestratorOpts): Promise<{ phase: OrchestratorPhaseResult; report: AuditReport | null }> {
|
||||
if (opts.dryRun) return { phase: { name: 'audit', status: 'skipped', detail: 'dry-run' }, report: null };
|
||||
try {
|
||||
let report: AuditReport;
|
||||
if (testEngineOverride) {
|
||||
// Test injection path: caller manages engine lifecycle.
|
||||
report = await scanBrainSources(testEngineOverride);
|
||||
} else {
|
||||
const config = loadConfig();
|
||||
if (!config) {
|
||||
// No brain configured (fresh dev install or test environment). The
|
||||
// migration audit needs a real brain to walk; treat this as a clean
|
||||
// skip rather than a failure so apply-migrations doesn't break.
|
||||
return {
|
||||
phase: { name: 'audit', status: 'skipped', detail: 'no_brain_configured' },
|
||||
report: null,
|
||||
};
|
||||
}
|
||||
const engineConfig = toEngineConfig(config);
|
||||
const engine = await createEngine(engineConfig);
|
||||
await engine.connect(engineConfig);
|
||||
try {
|
||||
report = await scanBrainSources(engine);
|
||||
} finally {
|
||||
await engine.disconnect();
|
||||
}
|
||||
}
|
||||
if (report.per_source.length === 0) {
|
||||
// No sources registered — fresh install or dev-only install. Skip
|
||||
// cleanly; the orchestrator should report success.
|
||||
return {
|
||||
phase: { name: 'audit', status: 'skipped', detail: 'no_sources_registered' },
|
||||
report,
|
||||
};
|
||||
}
|
||||
mkdirSync(migrationsDir(), { recursive: true });
|
||||
writeFileSync(auditReportPath(), JSON.stringify(report, null, 2));
|
||||
return {
|
||||
phase: {
|
||||
name: 'audit',
|
||||
status: 'complete',
|
||||
detail: `${report.total} issue(s) across ${report.per_source.length} source(s); report at ${auditReportPath()}`,
|
||||
},
|
||||
report,
|
||||
};
|
||||
} catch (e) {
|
||||
const msg = e instanceof Error ? e.message : String(e);
|
||||
return { phase: { name: 'audit', status: 'failed', detail: msg }, report: null };
|
||||
}
|
||||
}
|
||||
|
||||
// ── Phase C — Emit pending-host-work entries ──────────────
|
||||
|
||||
function existingEntriesForVersion(version: string): Set<string> {
|
||||
const out = new Set<string>();
|
||||
const p = pendingHostWorkPath();
|
||||
if (!existsSync(p)) return out;
|
||||
try {
|
||||
const raw = readFileSync(p, 'utf8');
|
||||
for (const line of raw.split('\n')) {
|
||||
const trimmed = line.trim();
|
||||
if (!trimmed) continue;
|
||||
try {
|
||||
const obj = JSON.parse(trimmed) as PendingHostWorkEntry;
|
||||
if (obj.migration === version && obj.source_id) {
|
||||
out.add(obj.source_id);
|
||||
}
|
||||
} catch { /* skip malformed */ }
|
||||
}
|
||||
} catch { /* read error */ }
|
||||
return out;
|
||||
}
|
||||
|
||||
function phaseCEmitTodo(opts: OrchestratorOpts, report: AuditReport | null): OrchestratorPhaseResult {
|
||||
if (opts.dryRun) return { name: 'emit-todo', status: 'skipped', detail: 'dry-run' };
|
||||
if (!report) return { name: 'emit-todo', status: 'skipped', detail: 'no report' };
|
||||
|
||||
const sourcesWithIssues = report.per_source.filter(s => s.total > 0);
|
||||
if (sourcesWithIssues.length === 0) {
|
||||
return { name: 'emit-todo', status: 'complete', detail: 'no issues; nothing to queue' };
|
||||
}
|
||||
|
||||
try {
|
||||
mkdirSync(migrationsDir(), { recursive: true });
|
||||
const already = existingEntriesForVersion('0.22.4');
|
||||
let added = 0;
|
||||
for (const src of sourcesWithIssues) {
|
||||
if (already.has(src.source_id)) continue;
|
||||
const entry: PendingHostWorkEntry = {
|
||||
migration: '0.22.4',
|
||||
ts: new Date().toISOString(),
|
||||
skill: 'skills/migrations/v0.22.4.md',
|
||||
reason: `${src.total} frontmatter issue(s) in source ${src.source_id}`,
|
||||
source_id: src.source_id,
|
||||
source_path: src.source_path,
|
||||
command: `gbrain frontmatter validate ${src.source_path} --fix`,
|
||||
};
|
||||
appendFileSync(pendingHostWorkPath(), JSON.stringify(entry) + '\n');
|
||||
added++;
|
||||
}
|
||||
return {
|
||||
name: 'emit-todo',
|
||||
status: 'complete',
|
||||
detail: `appended ${added} entr${added === 1 ? 'y' : 'ies'} to ${pendingHostWorkPath()}`,
|
||||
};
|
||||
} catch (e) {
|
||||
return { name: 'emit-todo', status: 'failed', detail: e instanceof Error ? e.message : String(e) };
|
||||
}
|
||||
}
|
||||
|
||||
// ── Orchestrator ────────────────────────────────────────────
|
||||
|
||||
async function orchestrator(opts: OrchestratorOpts): Promise<OrchestratorResult> {
|
||||
console.log('');
|
||||
console.log('=== v0.22.4 — frontmatter-guard adoption ===');
|
||||
if (opts.dryRun) console.log(' (dry-run; no side effects)');
|
||||
console.log('');
|
||||
|
||||
const phases: OrchestratorPhaseResult[] = [];
|
||||
|
||||
phases.push(phaseASchema(opts));
|
||||
|
||||
const { phase: bPhase, report } = await phaseBAudit(opts);
|
||||
phases.push(bPhase);
|
||||
if (bPhase.status === 'failed') {
|
||||
return { version: '0.22.4', status: 'partial', phases };
|
||||
}
|
||||
|
||||
phases.push(phaseCEmitTodo(opts, report));
|
||||
|
||||
const overallStatus: 'complete' | 'partial' | 'failed' =
|
||||
phases.some(p => p.status === 'failed') ? 'partial' : 'complete';
|
||||
|
||||
return {
|
||||
version: '0.22.4',
|
||||
status: overallStatus,
|
||||
phases,
|
||||
pending_host_work: report?.per_source.filter(s => s.total > 0).length ?? 0,
|
||||
};
|
||||
}
|
||||
|
||||
export const v0_22_4: Migration = {
|
||||
version: '0.22.4',
|
||||
featurePitch: {
|
||||
headline: 'Frontmatter-guard ships — broken brain pages can\'t hide',
|
||||
description:
|
||||
'gbrain v0.22.4 adds end-to-end frontmatter validation: a `gbrain frontmatter` CLI ' +
|
||||
'(validate / audit / install-hook), a `frontmatter_integrity` doctor subcheck, a ' +
|
||||
'pre-commit hook helper, and a new frontmatter-guard skill. The migration is audit-only ' +
|
||||
'(it never mutates your brain) — it scans every registered source, writes a per-source ' +
|
||||
'report to ~/.gbrain/migrations/v0.22.4-audit.json, and queues a TODO with the exact fix ' +
|
||||
'command. Run `gbrain frontmatter validate <source-path> --fix` to repair (creates .bak ' +
|
||||
'backups). Resolves all 7 check-resolvable warnings on master; ships frontmatter-guard.',
|
||||
},
|
||||
orchestrator,
|
||||
};
|
||||
|
||||
/** Exported for unit tests. */
|
||||
export const __testing = {
|
||||
phaseASchema,
|
||||
phaseBAudit,
|
||||
phaseCEmitTodo,
|
||||
auditReportPath,
|
||||
pendingHostWorkPath,
|
||||
};
|
||||
@@ -0,0 +1,394 @@
|
||||
/**
|
||||
* brain-writer — frontmatter validation/audit/auto-fix orchestrator.
|
||||
*
|
||||
* Thin layer on top of `parseMarkdown(..., {validate:true})` (the canonical
|
||||
* source of frontmatter validation rules) and `isSyncable()` (the canonical
|
||||
* brain-page filter). Three consumers call into this module: the
|
||||
* `gbrain frontmatter` CLI, the `frontmatter_integrity` doctor subcheck, and
|
||||
* the v0.22.4 migration audit phase. Single source of truth — no parallel
|
||||
* validation stack.
|
||||
*
|
||||
* Path-guard contract: writeBrainPage refuses to write outside the source
|
||||
* path. .bak backups are the safety contract (works for both git and non-git
|
||||
* brain repos; the existing src/core/dry-fix.ts:getWorkingTreeStatus rejects
|
||||
* non-git repos as unsafe, which is the wrong shape for brain rewrites).
|
||||
*/
|
||||
|
||||
import { existsSync, readFileSync, readdirSync, statSync, copyFileSync, writeFileSync, mkdirSync, lstatSync } from 'fs';
|
||||
import { join, relative, resolve, dirname } from 'path';
|
||||
import type { BrainEngine } from './engine.ts';
|
||||
import type { ProgressReporter } from './progress.ts';
|
||||
import {
|
||||
parseMarkdown,
|
||||
type ParseValidationCode,
|
||||
type ParseValidationError,
|
||||
} from './markdown.ts';
|
||||
import { isSyncable, slugifyPath } from './sync.ts';
|
||||
|
||||
export type { ParseValidationCode };
|
||||
|
||||
export interface AuditFix {
|
||||
code: ParseValidationCode;
|
||||
description: string;
|
||||
}
|
||||
|
||||
export interface PerSourceReport {
|
||||
source_id: string;
|
||||
source_path: string;
|
||||
total: number;
|
||||
errors_by_code: Partial<Record<ParseValidationCode, number>>;
|
||||
sample: { path: string; codes: ParseValidationCode[] }[];
|
||||
}
|
||||
|
||||
export interface AuditReport {
|
||||
ok: boolean;
|
||||
total: number;
|
||||
errors_by_code: Partial<Record<ParseValidationCode, number>>;
|
||||
per_source: PerSourceReport[];
|
||||
scanned_at: string;
|
||||
}
|
||||
|
||||
const SAMPLE_PER_SOURCE = 20;
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// autoFixFrontmatter
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
/**
|
||||
* Mechanical auto-repair for the fixable subset of validation codes:
|
||||
* - NULL_BYTES — strip \x00 characters
|
||||
* - NESTED_QUOTES — rewrite `"... "inner" ..."` to single-quoted outer
|
||||
* - MISSING_CLOSE — insert `---` before the first heading found inside
|
||||
* the YAML zone
|
||||
* - SLUG_MISMATCH — remove `slug:` line (gbrain derives slug from path)
|
||||
*
|
||||
* Idempotent: running twice is a no-op on already-clean input. Any error class
|
||||
* not in the list above is left untouched (e.g. EMPTY_FRONTMATTER, YAML_PARSE,
|
||||
* MISSING_OPEN — those need human review).
|
||||
*/
|
||||
export function autoFixFrontmatter(
|
||||
content: string,
|
||||
opts?: { filePath?: string },
|
||||
): { content: string; fixes: AuditFix[] } {
|
||||
const fixes: AuditFix[] = [];
|
||||
let working = content;
|
||||
|
||||
// 1. NULL_BYTES — strip them. Cheap, byte-level. Run first so subsequent
|
||||
// line-based passes don't trip on stray nulls.
|
||||
if (working.indexOf('\x00') >= 0) {
|
||||
working = working.replace(/\x00/g, '');
|
||||
fixes.push({ code: 'NULL_BYTES', description: 'Stripped null bytes' });
|
||||
}
|
||||
|
||||
// 2. MISSING_CLOSE — if there's an opener but no closer before a heading,
|
||||
// insert `---` immediately before the heading. Walk lines once.
|
||||
{
|
||||
const lines = working.split('\n');
|
||||
let firstNonEmpty = -1;
|
||||
for (let i = 0; i < lines.length; i++) {
|
||||
if (lines[i].trim().length > 0) { firstNonEmpty = i; break; }
|
||||
}
|
||||
if (firstNonEmpty >= 0 && lines[firstNonEmpty].trim() === '---') {
|
||||
let closeIdx = -1;
|
||||
let headingIdx = -1;
|
||||
for (let i = firstNonEmpty + 1; i < lines.length; i++) {
|
||||
const t = lines[i].trim();
|
||||
if (t === '---') { closeIdx = i; break; }
|
||||
if (/^#{1,6}\s/.test(t)) { headingIdx = i; break; }
|
||||
}
|
||||
if (closeIdx === -1 && headingIdx >= 0) {
|
||||
const fixed = [
|
||||
...lines.slice(0, headingIdx),
|
||||
'---',
|
||||
'',
|
||||
...lines.slice(headingIdx),
|
||||
];
|
||||
working = fixed.join('\n');
|
||||
fixes.push({
|
||||
code: 'MISSING_CLOSE',
|
||||
description: `Inserted closing --- before heading at line ${headingIdx + 1}`,
|
||||
});
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// 3. NESTED_QUOTES — rewrite `key: "...inner..."` lines that have 3+ unescaped
|
||||
// double-quotes by switching the outer wrapper to single quotes and
|
||||
// leaving inner quotes alone.
|
||||
{
|
||||
const lines = working.split('\n');
|
||||
let firstNonEmpty = -1;
|
||||
for (let i = 0; i < lines.length; i++) {
|
||||
if (lines[i].trim().length > 0) { firstNonEmpty = i; break; }
|
||||
}
|
||||
if (firstNonEmpty >= 0 && lines[firstNonEmpty].trim() === '---') {
|
||||
let closeIdx = lines.length;
|
||||
for (let i = firstNonEmpty + 1; i < lines.length; i++) {
|
||||
if (lines[i].trim() === '---') { closeIdx = i; break; }
|
||||
}
|
||||
let fixedAny = false;
|
||||
for (let i = firstNonEmpty + 1; i < closeIdx; i++) {
|
||||
const m = lines[i].match(/^(\s*[A-Za-z_][\w-]*\s*:\s*)"(.*)"\s*(.*)$/);
|
||||
if (!m) continue;
|
||||
const [, prefix, inner, trailing] = m;
|
||||
let count = 0;
|
||||
for (let j = 0; j < inner.length; j++) {
|
||||
if (inner[j] === '"' && (j === 0 || inner[j - 1] !== '\\')) count++;
|
||||
}
|
||||
// Total " on the line includes the two outer quotes the regex
|
||||
// captured, plus whatever's in inner. We need 3+ to trigger.
|
||||
if (count >= 1) {
|
||||
// Inner already has unescaped " — outer wrap is causing the YAML
|
||||
// parse failure. Rewrite to 'single-quoted'. YAML escapes `'` inside
|
||||
// a single-quoted string by doubling it.
|
||||
const escapedInner = inner.replace(/'/g, "''");
|
||||
lines[i] = `${prefix}'${escapedInner}'${trailing ? ' ' + trailing : ''}`.replace(/\s+$/, '');
|
||||
fixedAny = true;
|
||||
}
|
||||
}
|
||||
if (fixedAny) {
|
||||
working = lines.join('\n');
|
||||
fixes.push({
|
||||
code: 'NESTED_QUOTES',
|
||||
description: 'Rewrote nested double-quoted YAML values to single-quoted',
|
||||
});
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// 4. SLUG_MISMATCH — remove `slug:` line if filePath is provided and the
|
||||
// declared slug doesn't match the path-derived one. Per PR #392 spec,
|
||||
// gbrain derives slug from path; the field shouldn't be in frontmatter.
|
||||
if (opts?.filePath) {
|
||||
const expectedSlug = slugifyPath(opts.filePath);
|
||||
// Use the (possibly partially-fixed) working content to detect whether
|
||||
// the slug field is present and mismatched.
|
||||
const re = /^slug:\s*(.+?)\s*$/m;
|
||||
const m = working.match(re);
|
||||
if (m && m[1].replace(/^["']|["']$/g, '') !== expectedSlug) {
|
||||
working = working.replace(re, '').replace(/\n{3,}/g, '\n\n');
|
||||
fixes.push({
|
||||
code: 'SLUG_MISMATCH',
|
||||
description: `Removed mismatched slug field (was "${m[1]}", expected "${expectedSlug}")`,
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
return { content: working, fixes };
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// writeBrainPage — path-guarded write with .bak backup
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
export class BrainWriterError extends Error {
|
||||
code: string;
|
||||
hint?: string;
|
||||
constructor(code: string, message: string, hint?: string) {
|
||||
super(message);
|
||||
this.name = 'BrainWriterError';
|
||||
this.code = code;
|
||||
this.hint = hint;
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Path-guarded brain page writer. Always writes `<filePath>.bak` before any
|
||||
* in-place mutation (the contract that replaces git-tree-clean for non-git
|
||||
* brain repos). Throws BrainWriterError if filePath is not under sourcePath.
|
||||
*/
|
||||
export function writeBrainPage(
|
||||
filePath: string,
|
||||
content: string,
|
||||
opts: { sourcePath: string; autoFix?: boolean },
|
||||
): { fixes: AuditFix[] } {
|
||||
const resolvedSource = resolve(opts.sourcePath);
|
||||
const resolvedTarget = resolve(filePath);
|
||||
if (resolvedTarget !== resolvedSource && !resolvedTarget.startsWith(resolvedSource + '/')) {
|
||||
throw new BrainWriterError(
|
||||
'PATH_OUTSIDE_SOURCE',
|
||||
`writeBrainPage: ${filePath} is not under ${opts.sourcePath}`,
|
||||
'Pass --source <id> matching the source the file lives in.',
|
||||
);
|
||||
}
|
||||
|
||||
let toWrite = content;
|
||||
let fixes: AuditFix[] = [];
|
||||
if (opts.autoFix) {
|
||||
const result = autoFixFrontmatter(content, { filePath });
|
||||
toWrite = result.content;
|
||||
fixes = result.fixes;
|
||||
}
|
||||
|
||||
if (existsSync(filePath)) {
|
||||
copyFileSync(filePath, filePath + '.bak');
|
||||
} else {
|
||||
mkdirSync(dirname(filePath), { recursive: true });
|
||||
}
|
||||
writeFileSync(filePath, toWrite, 'utf8');
|
||||
return { fixes };
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// scanBrainSources
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
interface SourceRow {
|
||||
id: string;
|
||||
local_path: string | null;
|
||||
}
|
||||
|
||||
export interface ScanOpts {
|
||||
/** Limit scan to one source. When omitted, all registered sources with a
|
||||
* local_path are scanned. */
|
||||
sourceId?: string;
|
||||
onProgress?: ProgressReporter;
|
||||
signal?: AbortSignal;
|
||||
}
|
||||
|
||||
export async function scanBrainSources(
|
||||
engine: BrainEngine,
|
||||
opts: ScanOpts = {},
|
||||
): Promise<AuditReport> {
|
||||
const sources = await listSources(engine, opts.sourceId);
|
||||
const totals: Partial<Record<ParseValidationCode, number>> = {};
|
||||
const perSource: PerSourceReport[] = [];
|
||||
let grandTotal = 0;
|
||||
|
||||
for (const src of sources) {
|
||||
if (opts.signal?.aborted) break;
|
||||
if (!src.local_path) continue;
|
||||
if (!existsSync(src.local_path)) {
|
||||
// Source registered but path is missing on disk; surface as a zero-row
|
||||
// entry with a synthetic SCAN_PATH_MISSING note via warn-and-skip.
|
||||
perSource.push({
|
||||
source_id: src.id,
|
||||
source_path: src.local_path,
|
||||
total: 0,
|
||||
errors_by_code: {},
|
||||
sample: [],
|
||||
});
|
||||
continue;
|
||||
}
|
||||
const report = scanOneSource(src.id, src.local_path, opts);
|
||||
perSource.push(report);
|
||||
grandTotal += report.total;
|
||||
for (const [code, n] of Object.entries(report.errors_by_code)) {
|
||||
const k = code as ParseValidationCode;
|
||||
totals[k] = (totals[k] ?? 0) + (n as number);
|
||||
}
|
||||
}
|
||||
|
||||
return {
|
||||
ok: grandTotal === 0,
|
||||
total: grandTotal,
|
||||
errors_by_code: totals,
|
||||
per_source: perSource,
|
||||
scanned_at: new Date().toISOString(),
|
||||
};
|
||||
}
|
||||
|
||||
function scanOneSource(
|
||||
sourceId: string,
|
||||
sourcePath: string,
|
||||
opts: ScanOpts,
|
||||
): PerSourceReport {
|
||||
const errorsByCode: Partial<Record<ParseValidationCode, number>> = {};
|
||||
const sample: PerSourceReport['sample'] = [];
|
||||
const rootResolved = resolve(sourcePath);
|
||||
let scanned = 0;
|
||||
let total = 0;
|
||||
|
||||
walkDir(rootResolved, (absPath) => {
|
||||
if (opts.signal?.aborted) return false;
|
||||
const relPath = relative(rootResolved, absPath);
|
||||
if (!isSyncable(relPath, { strategy: 'markdown' })) return true;
|
||||
scanned++;
|
||||
let content: string;
|
||||
try {
|
||||
content = readFileSync(absPath, 'utf8');
|
||||
} catch {
|
||||
return true; // skip unreadable
|
||||
}
|
||||
const expectedSlug = slugifyPath(relPath);
|
||||
const parsed = parseMarkdown(content, relPath, { validate: true, expectedSlug });
|
||||
const errs = parsed.errors ?? [];
|
||||
if (errs.length > 0) {
|
||||
total += errs.length;
|
||||
const codes: ParseValidationCode[] = [];
|
||||
for (const e of errs) {
|
||||
errorsByCode[e.code] = (errorsByCode[e.code] ?? 0) + 1;
|
||||
codes.push(e.code);
|
||||
}
|
||||
if (sample.length < SAMPLE_PER_SOURCE) {
|
||||
sample.push({ path: relPath, codes });
|
||||
}
|
||||
}
|
||||
if (opts.onProgress && scanned % 50 === 0) {
|
||||
opts.onProgress.tick(50);
|
||||
}
|
||||
return true;
|
||||
});
|
||||
|
||||
if (opts.onProgress) {
|
||||
opts.onProgress.heartbeat(`scanned ${scanned} pages in ${sourceId}`);
|
||||
}
|
||||
|
||||
return {
|
||||
source_id: sourceId,
|
||||
source_path: sourcePath,
|
||||
total,
|
||||
errors_by_code: errorsByCode,
|
||||
sample,
|
||||
};
|
||||
}
|
||||
|
||||
/** Recursive directory walker with symlink-loop protection (via lstat).
|
||||
* Calls `visit` for each regular file. Returning false from `visit` stops
|
||||
* the walk. Skips entries lstat reports as symlinks (sync's no-symlink
|
||||
* policy). */
|
||||
function walkDir(root: string, visit: (absPath: string) => boolean | void): void {
|
||||
const stack: string[] = [root];
|
||||
const visited = new Set<string>();
|
||||
while (stack.length > 0) {
|
||||
const dir = stack.pop()!;
|
||||
let entries: string[];
|
||||
try {
|
||||
entries = readdirSync(dir);
|
||||
} catch {
|
||||
continue;
|
||||
}
|
||||
for (const name of entries) {
|
||||
const full = join(dir, name);
|
||||
let st: ReturnType<typeof lstatSync>;
|
||||
try {
|
||||
st = lstatSync(full);
|
||||
} catch {
|
||||
continue;
|
||||
}
|
||||
if (st.isSymbolicLink()) continue; // matches sync's no-symlink policy
|
||||
if (st.isDirectory()) {
|
||||
const real = resolve(full);
|
||||
if (visited.has(real)) continue;
|
||||
visited.add(real);
|
||||
stack.push(full);
|
||||
} else if (st.isFile()) {
|
||||
const result = visit(full);
|
||||
if (result === false) return;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
async function listSources(engine: BrainEngine, sourceId?: string): Promise<SourceRow[]> {
|
||||
if (sourceId) {
|
||||
const rows = await engine.executeRaw<SourceRow>(
|
||||
`SELECT id, local_path FROM sources WHERE id = $1`,
|
||||
[sourceId],
|
||||
);
|
||||
return rows;
|
||||
}
|
||||
return engine.executeRaw<SourceRow>(
|
||||
`SELECT id, local_path FROM sources WHERE local_path IS NOT NULL ORDER BY id`,
|
||||
);
|
||||
}
|
||||
+201
-6
@@ -2,6 +2,29 @@ import matter from 'gray-matter';
|
||||
import type { PageType } from './types.ts';
|
||||
import { slugifyPath } from './sync.ts';
|
||||
|
||||
export type ParseValidationCode =
|
||||
| 'MISSING_OPEN'
|
||||
| 'MISSING_CLOSE'
|
||||
| 'YAML_PARSE'
|
||||
| 'SLUG_MISMATCH'
|
||||
| 'NULL_BYTES'
|
||||
| 'NESTED_QUOTES'
|
||||
| 'EMPTY_FRONTMATTER';
|
||||
|
||||
export interface ParseValidationError {
|
||||
code: ParseValidationCode;
|
||||
message: string;
|
||||
line?: number;
|
||||
}
|
||||
|
||||
export interface ParseOpts {
|
||||
/** When true, errors[] is populated. Existing callers unaffected. */
|
||||
validate?: boolean;
|
||||
/** When validate is true and frontmatter has a `slug:` field that doesn't
|
||||
* match expectedSlug, emits SLUG_MISMATCH. */
|
||||
expectedSlug?: string;
|
||||
}
|
||||
|
||||
export interface ParsedMarkdown {
|
||||
frontmatter: Record<string, unknown>;
|
||||
compiled_truth: string;
|
||||
@@ -10,6 +33,8 @@ export interface ParsedMarkdown {
|
||||
type: PageType;
|
||||
title: string;
|
||||
tags: string[];
|
||||
/** Present iff opts.validate. Empty array means no errors. */
|
||||
errors?: ParseValidationError[];
|
||||
}
|
||||
|
||||
/**
|
||||
@@ -33,26 +58,53 @@ export interface ParsedMarkdown {
|
||||
* heading (backward-compat for existing files). A bare `---` in body text
|
||||
* is treated as a markdown horizontal rule, not a timeline separator.
|
||||
*/
|
||||
export function parseMarkdown(content: string, filePath?: string): ParsedMarkdown {
|
||||
const { data: frontmatter, content: body } = matter(content);
|
||||
export function parseMarkdown(
|
||||
content: string,
|
||||
filePath?: string,
|
||||
opts?: ParseOpts,
|
||||
): ParsedMarkdown {
|
||||
const errors: ParseValidationError[] = [];
|
||||
|
||||
// gray-matter is forgiving: it returns empty data + original content for
|
||||
// pretty much any input. The validation surface below catches the cases
|
||||
// it silently swallows. Validation only runs when opts.validate is true,
|
||||
// so existing callers are unaffected.
|
||||
let parsed: ReturnType<typeof matter> | null = null;
|
||||
let yamlParseError: Error | null = null;
|
||||
try {
|
||||
parsed = matter(content);
|
||||
} catch (e) {
|
||||
yamlParseError = e as Error;
|
||||
}
|
||||
|
||||
if (opts?.validate) {
|
||||
collectValidationErrors(content, errors, {
|
||||
yamlParseError,
|
||||
expectedSlug: opts.expectedSlug,
|
||||
parsedFrontmatter: parsed?.data ?? {},
|
||||
});
|
||||
}
|
||||
|
||||
// When YAML parsing failed (rare; gray-matter is forgiving), fall back to
|
||||
// empty frontmatter + raw content as the body so non-validate callers still
|
||||
// get a usable shape.
|
||||
const frontmatter = (parsed?.data ?? {}) as Record<string, unknown>;
|
||||
const body = parsed?.content ?? content;
|
||||
|
||||
// Split body at first standalone ---
|
||||
const { compiled_truth, timeline } = splitBody(body);
|
||||
|
||||
// Extract metadata from frontmatter
|
||||
const type = (frontmatter.type as PageType) || inferType(filePath);
|
||||
const title = (frontmatter.title as string) || inferTitle(filePath);
|
||||
const tags = extractTags(frontmatter);
|
||||
const slug = (frontmatter.slug as string) || inferSlug(filePath);
|
||||
|
||||
// Remove processed fields from frontmatter (they're stored as columns)
|
||||
const cleanFrontmatter = { ...frontmatter };
|
||||
delete cleanFrontmatter.type;
|
||||
delete cleanFrontmatter.title;
|
||||
delete cleanFrontmatter.tags;
|
||||
delete cleanFrontmatter.slug;
|
||||
|
||||
return {
|
||||
const result: ParsedMarkdown = {
|
||||
frontmatter: cleanFrontmatter,
|
||||
compiled_truth: compiled_truth.trim(),
|
||||
timeline: timeline.trim(),
|
||||
@@ -61,6 +113,149 @@ export function parseMarkdown(content: string, filePath?: string): ParsedMarkdow
|
||||
title,
|
||||
tags,
|
||||
};
|
||||
if (opts?.validate) result.errors = errors;
|
||||
return result;
|
||||
}
|
||||
|
||||
/**
|
||||
* Inspect raw content for the 7 frontmatter validation classes that gray-matter
|
||||
* silently accepts. Mutates `errors` in place. The order of checks is
|
||||
* deliberate: cheap byte-level checks first, then structural checks, then
|
||||
* YAML-parse-dependent checks.
|
||||
*/
|
||||
function collectValidationErrors(
|
||||
content: string,
|
||||
errors: ParseValidationError[],
|
||||
ctx: {
|
||||
yamlParseError: Error | null;
|
||||
expectedSlug?: string;
|
||||
parsedFrontmatter: Record<string, unknown>;
|
||||
},
|
||||
): void {
|
||||
// 1. NULL_BYTES — binary corruption indicator.
|
||||
const nullIdx = content.indexOf('\x00');
|
||||
if (nullIdx >= 0) {
|
||||
const line = content.slice(0, nullIdx).split('\n').length;
|
||||
errors.push({
|
||||
code: 'NULL_BYTES',
|
||||
message: 'Content contains null bytes (likely binary corruption)',
|
||||
line,
|
||||
});
|
||||
}
|
||||
|
||||
// 2. MISSING_OPEN — first non-empty line must be `---`.
|
||||
const lines = content.split('\n');
|
||||
let firstNonEmpty = -1;
|
||||
for (let i = 0; i < lines.length; i++) {
|
||||
if (lines[i].trim().length > 0) {
|
||||
firstNonEmpty = i;
|
||||
break;
|
||||
}
|
||||
}
|
||||
if (firstNonEmpty === -1) {
|
||||
// Empty file: treat as MISSING_OPEN. Don't run other structural checks.
|
||||
errors.push({
|
||||
code: 'MISSING_OPEN',
|
||||
message: 'File is empty or whitespace-only; expected frontmatter starting with ---',
|
||||
line: 1,
|
||||
});
|
||||
return;
|
||||
}
|
||||
if (lines[firstNonEmpty].trim() !== '---') {
|
||||
errors.push({
|
||||
code: 'MISSING_OPEN',
|
||||
message: 'Frontmatter must start with --- on the first non-empty line',
|
||||
line: firstNonEmpty + 1,
|
||||
});
|
||||
// Without an opener we can't reason about MISSING_CLOSE / EMPTY_FRONTMATTER
|
||||
// / NESTED_QUOTES inside frontmatter. Stop structural checks here.
|
||||
return;
|
||||
}
|
||||
|
||||
// 3. MISSING_CLOSE — find the next `---` after the opener. If a markdown
|
||||
// heading appears before it, that's a strong signal the closing
|
||||
// delimiter is missing (the heading was meant to be in the body).
|
||||
let closeLine = -1;
|
||||
let headingBeforeClose = -1;
|
||||
for (let i = firstNonEmpty + 1; i < lines.length; i++) {
|
||||
const t = lines[i].trim();
|
||||
if (t === '---') {
|
||||
closeLine = i;
|
||||
break;
|
||||
}
|
||||
if (/^#{1,6}\s/.test(t) && headingBeforeClose === -1) {
|
||||
headingBeforeClose = i;
|
||||
}
|
||||
}
|
||||
if (closeLine === -1) {
|
||||
errors.push({
|
||||
code: 'MISSING_CLOSE',
|
||||
message:
|
||||
headingBeforeClose >= 0
|
||||
? `No closing --- before heading at line ${headingBeforeClose + 1}`
|
||||
: 'No closing --- delimiter found',
|
||||
line: headingBeforeClose >= 0 ? headingBeforeClose + 1 : firstNonEmpty + 1,
|
||||
});
|
||||
return;
|
||||
}
|
||||
if (headingBeforeClose >= 0 && headingBeforeClose < closeLine) {
|
||||
errors.push({
|
||||
code: 'MISSING_CLOSE',
|
||||
message: `Heading at line ${headingBeforeClose + 1} found inside frontmatter zone (closing --- comes after)`,
|
||||
line: headingBeforeClose + 1,
|
||||
});
|
||||
}
|
||||
|
||||
// 4. EMPTY_FRONTMATTER — open and close present but nothing meaningful between.
|
||||
const fmBody = lines.slice(firstNonEmpty + 1, closeLine).join('\n').trim();
|
||||
if (fmBody.length === 0) {
|
||||
errors.push({
|
||||
code: 'EMPTY_FRONTMATTER',
|
||||
message: 'Frontmatter block is empty',
|
||||
line: firstNonEmpty + 1,
|
||||
});
|
||||
}
|
||||
|
||||
// 5. NESTED_QUOTES — common breakage pattern: `title: "Name "Nick" Last"`.
|
||||
// Detect any frontmatter `key: ...` line whose value contains 3 or more
|
||||
// unescaped double-quote characters. A clean quoted value has 2.
|
||||
for (let i = firstNonEmpty + 1; i < closeLine; i++) {
|
||||
const line = lines[i];
|
||||
const m = line.match(/^\s*[A-Za-z_][\w-]*\s*:\s*(.*)$/);
|
||||
if (!m) continue;
|
||||
const value = m[1];
|
||||
let count = 0;
|
||||
for (let j = 0; j < value.length; j++) {
|
||||
if (value[j] === '"' && (j === 0 || value[j - 1] !== '\\')) count++;
|
||||
}
|
||||
if (count >= 3) {
|
||||
errors.push({
|
||||
code: 'NESTED_QUOTES',
|
||||
message: 'Nested double quotes in YAML value (use single quotes for the outer)',
|
||||
line: i + 1,
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
// 6. YAML_PARSE — gray-matter threw.
|
||||
if (ctx.yamlParseError) {
|
||||
errors.push({
|
||||
code: 'YAML_PARSE',
|
||||
message: `YAML parse failed: ${ctx.yamlParseError.message}`,
|
||||
line: firstNonEmpty + 1,
|
||||
});
|
||||
}
|
||||
|
||||
// 7. SLUG_MISMATCH — only when expectedSlug was provided and a slug field exists.
|
||||
if (ctx.expectedSlug && typeof ctx.parsedFrontmatter.slug === 'string') {
|
||||
const declared = ctx.parsedFrontmatter.slug as string;
|
||||
if (declared !== ctx.expectedSlug) {
|
||||
errors.push({
|
||||
code: 'SLUG_MISMATCH',
|
||||
message: `Frontmatter slug "${declared}" does not match path-derived slug "${ctx.expectedSlug}"`,
|
||||
});
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
|
||||
@@ -108,7 +108,7 @@ describe('buildPlan — diff against completed + installed VERSION', () => {
|
||||
// autopilot cooperative, v0.16.0 = subagent runtime, v0.18.0 = multi-
|
||||
// source brains, v0.18.1 = RLS hardening, v0.21.0 = Cathedral II
|
||||
// (renumbered from v0.20.0 after master shipped v0.20.x in parallel).
|
||||
expect(plan.skippedFuture.map(m => m.version)).toEqual(['0.12.0', '0.12.2', '0.13.0', '0.13.1', '0.14.0', '0.16.0', '0.18.0', '0.18.1', '0.21.0']);
|
||||
expect(plan.skippedFuture.map(m => m.version)).toEqual(['0.12.0', '0.12.2', '0.13.0', '0.13.1', '0.14.0', '0.16.0', '0.18.0', '0.18.1', '0.21.0', '0.22.4']);
|
||||
});
|
||||
|
||||
test('already applied → v0.11.0 lands in `applied` bucket, not pending', () => {
|
||||
@@ -148,7 +148,7 @@ describe('buildPlan — diff against completed + installed VERSION', () => {
|
||||
// were added later; installed=0.12.0 means they belong in skippedFuture,
|
||||
// not pending. v0.11.0 and v0.12.0 stay pending despite being ≤ installed —
|
||||
// that is the H9 invariant.
|
||||
expect(plan.skippedFuture.map(m => m.version)).toEqual(['0.12.2', '0.13.0', '0.13.1', '0.14.0', '0.16.0', '0.18.0', '0.18.1', '0.21.0']);
|
||||
expect(plan.skippedFuture.map(m => m.version)).toEqual(['0.12.2', '0.13.0', '0.13.1', '0.14.0', '0.16.0', '0.18.0', '0.18.1', '0.21.0', '0.22.4']);
|
||||
});
|
||||
|
||||
test('--migration filter narrows to one version', () => {
|
||||
|
||||
@@ -0,0 +1,219 @@
|
||||
import { describe, expect, test, beforeEach, afterEach } from 'bun:test';
|
||||
import { mkdtempSync, rmSync, writeFileSync, readFileSync, existsSync, mkdirSync, symlinkSync } from 'fs';
|
||||
import { join } from 'path';
|
||||
import { tmpdir } from 'os';
|
||||
import {
|
||||
autoFixFrontmatter,
|
||||
writeBrainPage,
|
||||
scanBrainSources,
|
||||
BrainWriterError,
|
||||
} from '../src/core/brain-writer.ts';
|
||||
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
|
||||
|
||||
const fence = '---';
|
||||
|
||||
describe('autoFixFrontmatter', () => {
|
||||
test('strips null bytes', () => {
|
||||
const input = `${fence}\ntitle: ok\n${fence}\n\nbody\x00drop\x00here`;
|
||||
const { content, fixes } = autoFixFrontmatter(input);
|
||||
expect(content.includes('\x00')).toBe(false);
|
||||
expect(fixes.some(f => f.code === 'NULL_BYTES')).toBe(true);
|
||||
});
|
||||
|
||||
test('inserts closing --- before heading when MISSING_CLOSE', () => {
|
||||
const input = `${fence}\ntype: concept\ntitle: ok\n# A heading\n\nbody`;
|
||||
const { content, fixes } = autoFixFrontmatter(input);
|
||||
expect(fixes.some(f => f.code === 'MISSING_CLOSE')).toBe(true);
|
||||
// After fix, parsing should find a closing --- before the heading.
|
||||
const idxClose = content.indexOf('---', 3);
|
||||
const idxHeading = content.indexOf('# A heading');
|
||||
expect(idxClose).toBeGreaterThan(0);
|
||||
expect(idxClose).toBeLessThan(idxHeading);
|
||||
});
|
||||
|
||||
test('rewrites nested-quote title to single-quoted', () => {
|
||||
const input = `${fence}\ntype: concept\ntitle: "Phil "Nick" Last"\n${fence}\n\nbody`;
|
||||
const { content, fixes } = autoFixFrontmatter(input);
|
||||
expect(fixes.some(f => f.code === 'NESTED_QUOTES')).toBe(true);
|
||||
// Outer wrapper is now single quotes.
|
||||
expect(content).toMatch(/^title: '.*'\s*$/m);
|
||||
});
|
||||
|
||||
test('removes mismatched slug field', () => {
|
||||
const input = `${fence}\ntype: concept\ntitle: hi\nslug: wrong-slug\n${fence}\n\nbody`;
|
||||
const { content, fixes } = autoFixFrontmatter(input, { filePath: 'people/jane-doe.md' });
|
||||
expect(fixes.some(f => f.code === 'SLUG_MISMATCH')).toBe(true);
|
||||
expect(content).not.toMatch(/^slug:/m);
|
||||
});
|
||||
|
||||
test('idempotent: running twice produces no diff and no fixes on second pass', () => {
|
||||
const input = `${fence}\ntype: concept\ntitle: "Phil "Nick" Last"\n${fence}\n\nbody\x00`;
|
||||
const first = autoFixFrontmatter(input);
|
||||
const second = autoFixFrontmatter(first.content);
|
||||
expect(second.content).toBe(first.content);
|
||||
expect(second.fixes).toEqual([]);
|
||||
});
|
||||
|
||||
test('clean input: no fixes, content unchanged', () => {
|
||||
const input = `${fence}\ntype: concept\ntitle: ok\n${fence}\n\nbody`;
|
||||
const { content, fixes } = autoFixFrontmatter(input);
|
||||
expect(content).toBe(input);
|
||||
expect(fixes).toEqual([]);
|
||||
});
|
||||
});
|
||||
|
||||
describe('writeBrainPage', () => {
|
||||
let tmp: string;
|
||||
|
||||
beforeEach(() => {
|
||||
tmp = mkdtempSync(join(tmpdir(), 'brain-writer-'));
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
rmSync(tmp, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
test('happy path: writes file inside source', () => {
|
||||
const file = join(tmp, 'people', 'jane.md');
|
||||
const content = `${fence}\ntype: person\ntitle: Jane\n${fence}\n\nhello`;
|
||||
writeBrainPage(file, content, { sourcePath: tmp });
|
||||
expect(readFileSync(file, 'utf8')).toBe(content);
|
||||
});
|
||||
|
||||
test('throws BrainWriterError when path is outside sourcePath', () => {
|
||||
const elsewhere = mkdtempSync(join(tmpdir(), 'brain-writer-other-'));
|
||||
try {
|
||||
const offending = join(elsewhere, 'evil.md');
|
||||
expect(() =>
|
||||
writeBrainPage(offending, 'content', { sourcePath: tmp }),
|
||||
).toThrow(BrainWriterError);
|
||||
} finally {
|
||||
rmSync(elsewhere, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
test('writes .bak before mutating an existing file', () => {
|
||||
const file = join(tmp, 'people', 'jane.md');
|
||||
mkdirSync(join(tmp, 'people'), { recursive: true });
|
||||
const original = `${fence}\ntype: person\ntitle: Old\n${fence}\n\nold`;
|
||||
writeFileSync(file, original);
|
||||
writeBrainPage(file, `${fence}\ntype: person\ntitle: New\n${fence}\n\nnew`, { sourcePath: tmp });
|
||||
expect(existsSync(file + '.bak')).toBe(true);
|
||||
expect(readFileSync(file + '.bak', 'utf8')).toBe(original);
|
||||
});
|
||||
|
||||
test('autoFix: true repairs nested quotes before writing', () => {
|
||||
const file = join(tmp, 'people', 'jane.md');
|
||||
const broken = `${fence}\ntype: person\ntitle: "Phil "Nick" Last"\n${fence}\n\nbody`;
|
||||
const { fixes } = writeBrainPage(file, broken, { sourcePath: tmp, autoFix: true });
|
||||
expect(fixes.some(f => f.code === 'NESTED_QUOTES')).toBe(true);
|
||||
expect(readFileSync(file, 'utf8')).toMatch(/^title: '.*'\s*$/m);
|
||||
});
|
||||
});
|
||||
|
||||
describe('scanBrainSources (PGLite)', () => {
|
||||
let tmp: string;
|
||||
let engine: PGLiteEngine;
|
||||
|
||||
beforeEach(async () => {
|
||||
tmp = mkdtempSync(join(tmpdir(), 'brain-writer-scan-'));
|
||||
engine = new PGLiteEngine();
|
||||
await engine.connect({});
|
||||
await engine.initSchema();
|
||||
});
|
||||
|
||||
afterEach(async () => {
|
||||
await engine.disconnect();
|
||||
rmSync(tmp, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
async function registerSource(id: string, path: string) {
|
||||
await engine.executeRaw(
|
||||
`INSERT INTO sources (id, name, local_path) VALUES ($1, $1, $2)
|
||||
ON CONFLICT (id) DO UPDATE SET local_path = EXCLUDED.local_path`,
|
||||
[id, path],
|
||||
);
|
||||
}
|
||||
|
||||
test('returns ok=true for empty source', async () => {
|
||||
await registerSource('empty', tmp);
|
||||
const report = await scanBrainSources(engine);
|
||||
expect(report.ok).toBe(true);
|
||||
expect(report.total).toBe(0);
|
||||
const empty = report.per_source.find(s => s.source_id === 'empty');
|
||||
expect(empty).toBeDefined();
|
||||
expect(empty!.total).toBe(0);
|
||||
});
|
||||
|
||||
test('detects errors across multiple sources', async () => {
|
||||
const srcA = join(tmp, 'a');
|
||||
const srcB = join(tmp, 'b');
|
||||
mkdirSync(srcA, { recursive: true });
|
||||
mkdirSync(srcB, { recursive: true });
|
||||
writeFileSync(join(srcA, 'p1.md'), `${fence}\ntype: x\ntitle: ok\n${fence}\n\nbody\x00`);
|
||||
writeFileSync(join(srcB, 'p2.md'), `${fence}\ntype: x\ntitle: "P "I" L"\n${fence}\n\nbody`);
|
||||
await registerSource('alpha', srcA);
|
||||
await registerSource('beta', srcB);
|
||||
|
||||
const report = await scanBrainSources(engine);
|
||||
expect(report.ok).toBe(false);
|
||||
expect(report.total).toBeGreaterThan(0);
|
||||
const alpha = report.per_source.find(s => s.source_id === 'alpha')!;
|
||||
const beta = report.per_source.find(s => s.source_id === 'beta')!;
|
||||
expect(alpha.errors_by_code.NULL_BYTES).toBeGreaterThanOrEqual(1);
|
||||
expect(beta.errors_by_code.NESTED_QUOTES).toBeGreaterThanOrEqual(1);
|
||||
});
|
||||
|
||||
test('respects sourceId filter', async () => {
|
||||
const srcA = join(tmp, 'a');
|
||||
const srcB = join(tmp, 'b');
|
||||
mkdirSync(srcA, { recursive: true });
|
||||
mkdirSync(srcB, { recursive: true });
|
||||
writeFileSync(join(srcA, 'bad.md'), `${fence}\ntype: x\ntitle: ok\n${fence}\n\nbody\x00`);
|
||||
writeFileSync(join(srcB, 'bad.md'), `${fence}\ntype: x\ntitle: ok\n${fence}\n\nbody\x00`);
|
||||
await registerSource('alpha', srcA);
|
||||
await registerSource('beta', srcB);
|
||||
|
||||
const onlyA = await scanBrainSources(engine, { sourceId: 'alpha' });
|
||||
expect(onlyA.per_source.length).toBe(1);
|
||||
expect(onlyA.per_source[0]!.source_id).toBe('alpha');
|
||||
});
|
||||
|
||||
test('skips registered source with missing path', async () => {
|
||||
await registerSource('ghost', join(tmp, 'does-not-exist'));
|
||||
const report = await scanBrainSources(engine);
|
||||
const ghost = report.per_source.find(s => s.source_id === 'ghost')!;
|
||||
expect(ghost.total).toBe(0);
|
||||
});
|
||||
|
||||
test('skips symlinks (matches sync no-symlink policy)', async () => {
|
||||
mkdirSync(join(tmp, 'real'), { recursive: true });
|
||||
writeFileSync(join(tmp, 'real', 'good.md'), `${fence}\ntype: x\ntitle: ok\n${fence}\n\nbody`);
|
||||
// Create a symlink loop: tmp/real/loop -> tmp/real
|
||||
try {
|
||||
symlinkSync(join(tmp, 'real'), join(tmp, 'real', 'loop'));
|
||||
} catch {
|
||||
// Some CI environments forbid symlink creation; skip the assertion.
|
||||
return;
|
||||
}
|
||||
await registerSource('with-symlink', tmp);
|
||||
const report = await scanBrainSources(engine);
|
||||
// The walk should complete without infinite-looping; at most one .md
|
||||
// entry visited (via the real path, not the symlink).
|
||||
expect(report.per_source[0]!.total).toBe(0);
|
||||
});
|
||||
|
||||
test('AbortSignal mid-scan stops walking', async () => {
|
||||
const src = join(tmp, 'big');
|
||||
mkdirSync(src, { recursive: true });
|
||||
for (let i = 0; i < 50; i++) {
|
||||
writeFileSync(join(src, `p${i}.md`), `${fence}\ntype: x\ntitle: t${i}\n${fence}\n\nbody`);
|
||||
}
|
||||
await registerSource('big', src);
|
||||
const ctrl = new AbortController();
|
||||
ctrl.abort();
|
||||
const report = await scanBrainSources(engine, { signal: ctrl.signal });
|
||||
// Aborted before any source ran; per_source array stays empty (or has zero reports).
|
||||
expect(report.per_source.length).toBe(0);
|
||||
});
|
||||
});
|
||||
@@ -255,6 +255,21 @@ describe("DRY detection — checkResolvable", () => {
|
||||
});
|
||||
});
|
||||
|
||||
describe("v0.22.4 regression — actual repo skills/ has 0 warnings", () => {
|
||||
test("repo skills/ pass check-resolvable cleanly", () => {
|
||||
// The contract for v0.22.4 (Part A): zero warnings, zero errors
|
||||
// against the actual checked-in skills/ tree. Guards against future
|
||||
// regressions that re-introduce trigger overlap, DRY violations, or
|
||||
// routing-eval fixture drift.
|
||||
const report = checkResolvable(SKILLS_DIR);
|
||||
const errors = report.issues.filter(i => i.severity === "error");
|
||||
const warnings = report.issues.filter(i => i.severity === "warning");
|
||||
expect(errors).toEqual([]);
|
||||
expect(warnings).toEqual([]);
|
||||
expect(report.ok).toBe(true);
|
||||
});
|
||||
});
|
||||
|
||||
// bun:test has no beforeEach/afterEach at module scope cleanly interacting
|
||||
// with closures; a small helper keeps cleanup readable and per-test.
|
||||
function afterEachCleanup(fn: () => void) {
|
||||
|
||||
@@ -21,6 +21,16 @@ describe('doctor command', () => {
|
||||
expect(stdout).toContain('--fast');
|
||||
});
|
||||
|
||||
test('frontmatter_integrity subcheck added in v0.22.4', async () => {
|
||||
const fs = await import('fs');
|
||||
const src = fs.readFileSync('src/commands/doctor.ts', 'utf8');
|
||||
// Subcheck name and call into shared scanner are present.
|
||||
expect(src).toContain("name: 'frontmatter_integrity'");
|
||||
expect(src).toContain('scanBrainSources');
|
||||
// Fix hint points at the right CLI command.
|
||||
expect(src).toContain('gbrain frontmatter validate');
|
||||
});
|
||||
|
||||
test('Check interface supports issues array', async () => {
|
||||
// `Check` is a TypeScript interface — type-only, no runtime value.
|
||||
// Importing it for type assertion is enough to validate the shape.
|
||||
|
||||
@@ -0,0 +1,191 @@
|
||||
/**
|
||||
* E2E: v0.22.4 frontmatter-guard migration end-to-end on PGLite.
|
||||
*
|
||||
* Closes plan item B14. Runs the v0_22_4 orchestrator against a real PGLite
|
||||
* brain with two registered sources and synthetic malformed brain pages on
|
||||
* disk. Asserts:
|
||||
* - audit phase writes ~/.gbrain/migrations/v0.22.4-audit.json with the
|
||||
* expected per-source counts.
|
||||
* - emit-todo phase appends one entry per source-with-issues to
|
||||
* ~/.gbrain/migrations/pending-host-work.jsonl, each pointing at
|
||||
* skills/migrations/v0.22.4.md (dotted convention) with the exact
|
||||
* gbrain frontmatter validate <source-path> --fix command.
|
||||
* - The migration is audit-only — no fixture page is mutated during
|
||||
* apply-migrations.
|
||||
*
|
||||
* Uses the __setTestEngineOverride() injection point on v0_22_4.ts (mirrors
|
||||
* the repair-jsonb test pattern). Bun's os.homedir() doesn't observe
|
||||
* process.env.HOME mutations mid-process, so we redirect via the explicit
|
||||
* test override rather than relying on env-var redirection of loadConfig().
|
||||
*
|
||||
* No DATABASE_URL needed; runs unconditionally in CI's Tier 1.
|
||||
*
|
||||
* Run: bun test test/e2e/frontmatter-migration.test.ts
|
||||
*/
|
||||
|
||||
import { describe, test, expect, beforeAll, afterAll } from 'bun:test';
|
||||
import { mkdtempSync, mkdirSync, writeFileSync, readFileSync, existsSync, rmSync } from 'fs';
|
||||
import { join } from 'path';
|
||||
import { tmpdir } from 'os';
|
||||
import { PGLiteEngine } from '../../src/core/pglite-engine.ts';
|
||||
import { v0_22_4, __setTestEngineOverride } from '../../src/commands/migrations/v0_22_4.ts';
|
||||
|
||||
const fence = '---';
|
||||
|
||||
let workdir: string;
|
||||
let tmpHome: string;
|
||||
let brainRootA: string;
|
||||
let brainRootB: string;
|
||||
let engine: PGLiteEngine;
|
||||
let originalHome: string | undefined;
|
||||
const originalContents = new Map<string, string>();
|
||||
|
||||
beforeAll(async () => {
|
||||
workdir = mkdtempSync(join(tmpdir(), 'fm-migration-e2e-'));
|
||||
tmpHome = join(workdir, 'home');
|
||||
brainRootA = join(workdir, 'brain-a');
|
||||
brainRootB = join(workdir, 'brain-b');
|
||||
mkdirSync(tmpHome, { recursive: true });
|
||||
mkdirSync(brainRootA, { recursive: true });
|
||||
mkdirSync(brainRootB, { recursive: true });
|
||||
mkdirSync(join(tmpHome, '.gbrain', 'migrations'), { recursive: true });
|
||||
|
||||
// Seed fixture brain pages on disk. Source A has 2 broken pages
|
||||
// (NESTED_QUOTES + NULL_BYTES); source B has 1 broken page (NESTED_QUOTES)
|
||||
// plus 1 clean page.
|
||||
const aBrokenNested = `${fence}\ntype: person\ntitle: "Phil "Nick" Last"\n${fence}\n\nbody-a-nested`;
|
||||
const aBrokenNull = `${fence}\ntype: concept\ntitle: ok\n${fence}\n\nbody-a-null\x00drop`;
|
||||
const bBroken = `${fence}\ntype: company\ntitle: "Co "Inc" Name"\n${fence}\n\nbody-b`;
|
||||
const bClean = `${fence}\ntype: concept\ntitle: clean\n${fence}\n\nbody-b-clean`;
|
||||
|
||||
const filesToTrack: Array<{ path: string; content: string }> = [
|
||||
{ path: join(brainRootA, 'people', 'phil.md'), content: aBrokenNested },
|
||||
{ path: join(brainRootA, 'concepts', 'foo.md'), content: aBrokenNull },
|
||||
{ path: join(brainRootB, 'companies', 'co.md'), content: bBroken },
|
||||
{ path: join(brainRootB, 'concepts', 'bar.md'), content: bClean },
|
||||
];
|
||||
|
||||
for (const f of filesToTrack) {
|
||||
mkdirSync(join(f.path, '..'), { recursive: true });
|
||||
writeFileSync(f.path, f.content);
|
||||
originalContents.set(f.path, f.content);
|
||||
}
|
||||
|
||||
// Single in-memory PGLite for the whole test. We inject it into the
|
||||
// orchestrator via __setTestEngineOverride so phaseBAudit skips loadConfig.
|
||||
engine = new PGLiteEngine();
|
||||
await engine.connect({});
|
||||
await engine.initSchema();
|
||||
await engine.executeRaw(
|
||||
`INSERT INTO sources (id, name, local_path) VALUES ($1, $1, $2)`,
|
||||
['alpha', brainRootA],
|
||||
);
|
||||
await engine.executeRaw(
|
||||
`INSERT INTO sources (id, name, local_path) VALUES ($1, $1, $2)`,
|
||||
['beta', brainRootB],
|
||||
);
|
||||
__setTestEngineOverride(engine);
|
||||
|
||||
// Redirect ~/.gbrain/migrations/ output. The orchestrator's gbrainDir()
|
||||
// helper reads process.env.HOME at call time, so the override takes
|
||||
// effect even though Bun's os.homedir() does not observe mid-process
|
||||
// mutations.
|
||||
originalHome = process.env.HOME;
|
||||
process.env.HOME = tmpHome;
|
||||
});
|
||||
|
||||
afterAll(async () => {
|
||||
__setTestEngineOverride(null);
|
||||
if (engine) await engine.disconnect();
|
||||
if (originalHome === undefined) delete process.env.HOME;
|
||||
else process.env.HOME = originalHome;
|
||||
rmSync(workdir, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
describe('E2E: v0.22.4 frontmatter-guard migration', () => {
|
||||
test('orchestrator runs end-to-end and produces the expected artifacts', async () => {
|
||||
const result = await v0_22_4.orchestrator({
|
||||
yes: true,
|
||||
dryRun: false,
|
||||
noAutopilotInstall: true,
|
||||
});
|
||||
|
||||
expect(result.version).toBe('0.22.4');
|
||||
expect(['complete', 'partial']).toContain(result.status);
|
||||
expect(result.phases.length).toBe(3);
|
||||
const auditPhase = result.phases.find((p) => p.name === 'audit')!;
|
||||
expect(auditPhase.status).toBe('complete');
|
||||
const emitPhase = result.phases.find((p) => p.name === 'emit-todo')!;
|
||||
expect(emitPhase.status).toBe('complete');
|
||||
expect(result.pending_host_work).toBe(2);
|
||||
});
|
||||
|
||||
test('audit JSON report exists and has per-source counts', () => {
|
||||
const reportPath = join(tmpHome, '.gbrain', 'migrations', 'v0.22.4-audit.json');
|
||||
expect(existsSync(reportPath)).toBe(true);
|
||||
const report = JSON.parse(readFileSync(reportPath, 'utf8'));
|
||||
|
||||
expect(report.ok).toBe(false);
|
||||
expect(report.total).toBeGreaterThan(0);
|
||||
expect(report.scanned_at).toMatch(/\d{4}-\d{2}-\d{2}T/);
|
||||
|
||||
const alpha = report.per_source.find((s: any) => s.source_id === 'alpha');
|
||||
const beta = report.per_source.find((s: any) => s.source_id === 'beta');
|
||||
expect(alpha).toBeDefined();
|
||||
expect(beta).toBeDefined();
|
||||
|
||||
// Source A has NESTED_QUOTES (in phil.md) and NULL_BYTES (in foo.md).
|
||||
// YAML_PARSE may also fire on the nested-quote page since gray-matter
|
||||
// throws — assert each expected code shows up at least once.
|
||||
expect(alpha.errors_by_code.NESTED_QUOTES).toBeGreaterThanOrEqual(1);
|
||||
expect(alpha.errors_by_code.NULL_BYTES).toBeGreaterThanOrEqual(1);
|
||||
expect(alpha.total).toBeGreaterThanOrEqual(2);
|
||||
expect(beta.errors_by_code.NESTED_QUOTES).toBeGreaterThanOrEqual(1);
|
||||
expect(beta.total).toBeGreaterThanOrEqual(1);
|
||||
|
||||
// Sample lists carry the affected file paths for each source.
|
||||
expect(alpha.sample.some((s: any) => s.path.includes('phil.md'))).toBe(true);
|
||||
expect(beta.sample.some((s: any) => s.path.includes('co.md'))).toBe(true);
|
||||
});
|
||||
|
||||
test('pending-host-work.jsonl carries one entry per source-with-issues', () => {
|
||||
const jsonlPath = join(tmpHome, '.gbrain', 'migrations', 'pending-host-work.jsonl');
|
||||
expect(existsSync(jsonlPath)).toBe(true);
|
||||
const lines = readFileSync(jsonlPath, 'utf8').split('\n').filter(Boolean);
|
||||
expect(lines.length).toBe(2);
|
||||
|
||||
const entries = lines.map((l) => JSON.parse(l));
|
||||
const ids = entries.map((e: any) => e.source_id).sort();
|
||||
expect(ids).toEqual(['alpha', 'beta']);
|
||||
|
||||
for (const e of entries) {
|
||||
expect(e.migration).toBe('0.22.4');
|
||||
// Dotted-filename convention: the skill pointer matches the user-facing
|
||||
// migration doc at skills/migrations/v0.22.4.md, NOT the underscored
|
||||
// TS module path.
|
||||
expect(e.skill).toBe('skills/migrations/v0.22.4.md');
|
||||
expect(e.command).toContain('gbrain frontmatter validate');
|
||||
expect(e.command).toContain('--fix');
|
||||
expect(e.command).toContain(e.source_path);
|
||||
}
|
||||
});
|
||||
|
||||
test('audit phase did NOT mutate any fixture brain page (audit-only contract)', () => {
|
||||
for (const [path, original] of originalContents) {
|
||||
expect(readFileSync(path, 'utf8')).toBe(original);
|
||||
// Nor should there be a .bak — the migration never invokes writeBrainPage.
|
||||
expect(existsSync(path + '.bak')).toBe(false);
|
||||
}
|
||||
});
|
||||
|
||||
test('orchestrator is idempotent — re-running does not duplicate JSONL entries', async () => {
|
||||
await v0_22_4.orchestrator({
|
||||
yes: true,
|
||||
dryRun: false,
|
||||
noAutopilotInstall: true,
|
||||
});
|
||||
const jsonlPath = join(tmpHome, '.gbrain', 'migrations', 'pending-host-work.jsonl');
|
||||
const lines = readFileSync(jsonlPath, 'utf8').split('\n').filter(Boolean);
|
||||
expect(lines.length).toBe(2);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,115 @@
|
||||
import { describe, expect, test, beforeEach, afterEach } from 'bun:test';
|
||||
import { mkdtempSync, mkdirSync, writeFileSync, readFileSync, existsSync, rmSync } from 'fs';
|
||||
import { join } from 'path';
|
||||
import { tmpdir } from 'os';
|
||||
import { spawnSync } from 'child_process';
|
||||
|
||||
const fence = '---';
|
||||
const CLI = ['run', 'src/cli.ts', 'frontmatter'];
|
||||
|
||||
function runCli(args: string[]): { stdout: string; stderr: string; code: number } {
|
||||
const result = spawnSync('bun', [...CLI, ...args], {
|
||||
encoding: 'utf8',
|
||||
cwd: process.cwd(),
|
||||
});
|
||||
return {
|
||||
stdout: result.stdout ?? '',
|
||||
stderr: result.stderr ?? '',
|
||||
code: result.status ?? -1,
|
||||
};
|
||||
}
|
||||
|
||||
describe('gbrain frontmatter CLI (B4)', () => {
|
||||
let tmp: string;
|
||||
|
||||
beforeEach(() => {
|
||||
tmp = mkdtempSync(join(tmpdir(), 'fm-cli-'));
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
rmSync(tmp, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
test('--help works without a DB', () => {
|
||||
const { stdout, code } = runCli(['--help']);
|
||||
expect(code).toBe(0);
|
||||
expect(stdout).toContain('frontmatter validation');
|
||||
});
|
||||
|
||||
test('validate clean file: exit 0, OK message', () => {
|
||||
const f = join(tmp, 'clean.md');
|
||||
writeFileSync(f, `${fence}\ntype: concept\ntitle: ok\n${fence}\n\nbody`);
|
||||
const { stdout, code } = runCli(['validate', f]);
|
||||
expect(code).toBe(0);
|
||||
expect(stdout).toContain('OK');
|
||||
});
|
||||
|
||||
test('validate broken file: exit 1, codes listed', () => {
|
||||
const f = join(tmp, 'broken.md');
|
||||
writeFileSync(f, `${fence}\ntype: concept\ntitle: "P "I" L"\n${fence}\n\nbody`);
|
||||
const { stdout, code } = runCli(['validate', f]);
|
||||
expect(code).toBe(1);
|
||||
expect(stdout).toContain('NESTED_QUOTES');
|
||||
});
|
||||
|
||||
test('validate --json envelope shape', () => {
|
||||
const f = join(tmp, 'broken.md');
|
||||
writeFileSync(f, `${fence}\ntype: concept\ntitle: "P "I" L"\n${fence}\n\nbody`);
|
||||
const { stdout } = runCli(['validate', f, '--json']);
|
||||
const env = JSON.parse(stdout);
|
||||
expect(env.ok).toBe(false);
|
||||
expect(env.total_files).toBe(1);
|
||||
expect(env.results[0].errors.length).toBeGreaterThan(0);
|
||||
expect(env.results[0].errors[0]).toHaveProperty('code');
|
||||
});
|
||||
|
||||
test('validate --fix --dry-run does not write', () => {
|
||||
const f = join(tmp, 'broken.md');
|
||||
const original = `${fence}\ntype: concept\ntitle: "P "I" L"\n${fence}\n\nbody`;
|
||||
writeFileSync(f, original);
|
||||
const { stdout, code } = runCli(['validate', f, '--fix', '--dry-run']);
|
||||
expect(stdout).toContain('would fix');
|
||||
expect(readFileSync(f, 'utf8')).toBe(original);
|
||||
expect(existsSync(f + '.bak')).toBe(false);
|
||||
// exit 0 with --fix even when issues remain (the fix path is the success path)
|
||||
expect(code).toBe(0);
|
||||
});
|
||||
|
||||
test('validate --fix writes .bak and rewrites in place', () => {
|
||||
const f = join(tmp, 'broken.md');
|
||||
const original = `${fence}\ntype: concept\ntitle: "P "I" L"\n${fence}\n\nbody`;
|
||||
writeFileSync(f, original);
|
||||
const { code } = runCli(['validate', f, '--fix']);
|
||||
expect(code).toBe(0);
|
||||
expect(existsSync(f + '.bak')).toBe(true);
|
||||
expect(readFileSync(f + '.bak', 'utf8')).toBe(original);
|
||||
expect(readFileSync(f, 'utf8')).toMatch(/^title: '.*'\s*$/m);
|
||||
});
|
||||
|
||||
test('validate --fix succeeds on a non-git path (no dirty-tree guard)', () => {
|
||||
// tmp is not a git repo; --fix must still work.
|
||||
const f = join(tmp, 'broken.md');
|
||||
writeFileSync(f, `${fence}\ntype: concept\ntitle: "A "B" C"\n${fence}\n\nbody`);
|
||||
const { code } = runCli(['validate', f, '--fix']);
|
||||
expect(code).toBe(0);
|
||||
expect(existsSync(f + '.bak')).toBe(true);
|
||||
});
|
||||
|
||||
test('validate scans a directory recursively, skips non-.md files', () => {
|
||||
mkdirSync(join(tmp, 'subdir'), { recursive: true });
|
||||
writeFileSync(join(tmp, 'a.md'), `${fence}\ntype: concept\ntitle: A\n${fence}\n\nbody`);
|
||||
writeFileSync(join(tmp, 'subdir', 'b.md'), `${fence}\ntype: concept\ntitle: B\n${fence}\n\nbody`);
|
||||
writeFileSync(join(tmp, 'README.md'), 'meta'); // skipped by isSyncable
|
||||
writeFileSync(join(tmp, 'image.png'), 'not markdown');
|
||||
const { stdout } = runCli(['validate', tmp, '--json']);
|
||||
const env = JSON.parse(stdout);
|
||||
// Two .md files: a.md, subdir/b.md. README.md is filtered by isSyncable.
|
||||
expect(env.total_files).toBe(2);
|
||||
});
|
||||
|
||||
test('validate missing path errors clearly', () => {
|
||||
const { stderr, code } = runCli(['validate', join(tmp, 'does-not-exist.md')]);
|
||||
expect(code).toBe(1);
|
||||
expect(stderr).toContain('not found');
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,96 @@
|
||||
import { describe, expect, test, beforeEach, afterEach } from 'bun:test';
|
||||
import { mkdtempSync, mkdirSync, writeFileSync, readFileSync, existsSync, rmSync } from 'fs';
|
||||
import { join } from 'path';
|
||||
import { tmpdir } from 'os';
|
||||
import { execFileSync } from 'child_process';
|
||||
import { installHook, uninstallHook } from '../src/commands/frontmatter-install-hook.ts';
|
||||
|
||||
function gitInit(dir: string) {
|
||||
execFileSync('git', ['init', '-q', dir]);
|
||||
execFileSync('git', ['-C', dir, 'config', 'user.email', 'test@example.com']);
|
||||
execFileSync('git', ['-C', dir, 'config', 'user.name', 'Test']);
|
||||
}
|
||||
|
||||
describe('frontmatter install-hook (B13)', () => {
|
||||
let tmp: string;
|
||||
|
||||
beforeEach(() => {
|
||||
tmp = mkdtempSync(join(tmpdir(), 'fm-hook-'));
|
||||
gitInit(tmp);
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
rmSync(tmp, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
test('installHook writes executable .githooks/pre-commit and sets core.hooksPath', () => {
|
||||
const result = installHook(tmp, false);
|
||||
expect(result).toBe('installed');
|
||||
const hookPath = join(tmp, '.githooks', 'pre-commit');
|
||||
expect(existsSync(hookPath)).toBe(true);
|
||||
const content = readFileSync(hookPath, 'utf8');
|
||||
expect(content).toContain('gbrain frontmatter');
|
||||
expect(content).toContain('git diff --cached');
|
||||
// Configured hooksPath
|
||||
const hooksPath = execFileSync('git', ['-C', tmp, 'config', '--get', 'core.hooksPath'], { encoding: 'utf8' }).trim();
|
||||
expect(hooksPath).toBe('.githooks');
|
||||
});
|
||||
|
||||
test('installHook refuses to clobber existing hook without --force', () => {
|
||||
const hooksDir = join(tmp, '.githooks');
|
||||
mkdirSync(hooksDir, { recursive: true });
|
||||
const hookPath = join(hooksDir, 'pre-commit');
|
||||
writeFileSync(hookPath, '#!/bin/sh\necho "user hook"');
|
||||
const result = installHook(tmp, false);
|
||||
expect(result).toBe('skipped_existing');
|
||||
// Original survives.
|
||||
expect(readFileSync(hookPath, 'utf8')).toContain('user hook');
|
||||
expect(existsSync(hookPath + '.bak')).toBe(false);
|
||||
});
|
||||
|
||||
test('installHook with force overwrites and saves .bak', () => {
|
||||
const hooksDir = join(tmp, '.githooks');
|
||||
mkdirSync(hooksDir, { recursive: true });
|
||||
const hookPath = join(hooksDir, 'pre-commit');
|
||||
writeFileSync(hookPath, '#!/bin/sh\necho "user hook"');
|
||||
const result = installHook(tmp, true);
|
||||
expect(result).toBe('installed');
|
||||
expect(existsSync(hookPath + '.bak')).toBe(true);
|
||||
expect(readFileSync(hookPath + '.bak', 'utf8')).toContain('user hook');
|
||||
expect(readFileSync(hookPath, 'utf8')).toContain('gbrain frontmatter');
|
||||
});
|
||||
|
||||
test('installHook on existing gbrain hook refreshes silently (no .bak)', () => {
|
||||
installHook(tmp, false);
|
||||
const hookPath = join(tmp, '.githooks', 'pre-commit');
|
||||
expect(existsSync(hookPath + '.bak')).toBe(false);
|
||||
// Re-run; should be 'unchanged' (banner already present).
|
||||
const second = installHook(tmp, false);
|
||||
expect(second).toBe('unchanged');
|
||||
});
|
||||
|
||||
test('uninstallHook removes the gbrain hook and restores .bak when present', () => {
|
||||
const hooksDir = join(tmp, '.githooks');
|
||||
mkdirSync(hooksDir, { recursive: true });
|
||||
const hookPath = join(hooksDir, 'pre-commit');
|
||||
writeFileSync(hookPath, '#!/bin/sh\necho "user hook"');
|
||||
installHook(tmp, true);
|
||||
expect(existsSync(hookPath + '.bak')).toBe(true);
|
||||
|
||||
const removed = uninstallHook(tmp);
|
||||
expect(removed).toBe(true);
|
||||
// .bak content restored as the active hook.
|
||||
expect(readFileSync(hookPath, 'utf8')).toContain('user hook');
|
||||
expect(existsSync(hookPath + '.bak')).toBe(false);
|
||||
});
|
||||
|
||||
test('uninstallHook on a non-gbrain hook returns false (does not remove user hook)', () => {
|
||||
const hooksDir = join(tmp, '.githooks');
|
||||
mkdirSync(hooksDir, { recursive: true });
|
||||
const hookPath = join(hooksDir, 'pre-commit');
|
||||
writeFileSync(hookPath, '#!/bin/sh\necho "user hook"');
|
||||
const removed = uninstallHook(tmp);
|
||||
expect(removed).toBe(false);
|
||||
expect(readFileSync(hookPath, 'utf8')).toContain('user hook');
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,59 @@
|
||||
import { describe, expect, test } from 'bun:test';
|
||||
import { lintContent } from '../src/commands/lint.ts';
|
||||
|
||||
const fence = '---';
|
||||
|
||||
describe('lintContent: frontmatter validation rules (B2)', () => {
|
||||
test('frontmatter-missing-close fires when heading inside YAML zone', () => {
|
||||
const md = `${fence}\ntype: concept\ntitle: ok\n# A heading\n\nbody`;
|
||||
const issues = lintContent(md, 'pages/test.md');
|
||||
expect(issues.some(i => i.rule === 'frontmatter-missing-close')).toBe(true);
|
||||
});
|
||||
|
||||
test('frontmatter-missing-close fires when no closing --- and no heading', () => {
|
||||
const md = `${fence}\ntype: concept\ntitle: ok\nstray`;
|
||||
const issues = lintContent(md, 'pages/test.md');
|
||||
expect(issues.some(i => i.rule === 'frontmatter-missing-close')).toBe(true);
|
||||
});
|
||||
|
||||
test('frontmatter-nested-quotes fires on title with 3+ unescaped quotes', () => {
|
||||
const md = `${fence}\ntype: concept\ntitle: "Phil "Nick" Last"\n${fence}\n\nbody`;
|
||||
const issues = lintContent(md, 'pages/test.md');
|
||||
expect(issues.some(i => i.rule === 'frontmatter-nested-quotes')).toBe(true);
|
||||
});
|
||||
|
||||
test('frontmatter-null-bytes fires on null byte', () => {
|
||||
const md = `${fence}\ntype: concept\ntitle: ok\n${fence}\n\nbody\x00`;
|
||||
const issues = lintContent(md, 'pages/test.md');
|
||||
expect(issues.some(i => i.rule === 'frontmatter-null-bytes')).toBe(true);
|
||||
});
|
||||
|
||||
test('frontmatter-empty fires on --- --- with nothing between', () => {
|
||||
const md = `${fence}\n${fence}\n\nbody`;
|
||||
const issues = lintContent(md, 'pages/test.md');
|
||||
expect(issues.some(i => i.rule === 'frontmatter-empty')).toBe(true);
|
||||
});
|
||||
|
||||
test('does NOT double-report frontmatter-missing-open when no-frontmatter fires', () => {
|
||||
const md = '# Test\n\nContent without frontmatter.';
|
||||
const issues = lintContent(md, 'pages/test.md');
|
||||
// Legacy rule survives.
|
||||
expect(issues.some(i => i.rule === 'no-frontmatter')).toBe(true);
|
||||
// New rule for the same case is suppressed.
|
||||
expect(issues.some(i => i.rule === 'frontmatter-missing-open')).toBe(false);
|
||||
});
|
||||
|
||||
test('clean page produces no frontmatter-rule issues', () => {
|
||||
const md = `${fence}\ntitle: Hello\ntype: concept\ncreated: 2026-04-25\n${fence}\n\nbody content`;
|
||||
const issues = lintContent(md, 'pages/test.md');
|
||||
const fmIssues = issues.filter(i => i.rule.startsWith('frontmatter-'));
|
||||
expect(fmIssues).toEqual([]);
|
||||
});
|
||||
|
||||
test('fixable flag set correctly for fixable codes', () => {
|
||||
const md = `${fence}\ntype: concept\ntitle: "Phil "Nick" Last"\n${fence}\n\nbody`;
|
||||
const issues = lintContent(md, 'pages/test.md');
|
||||
const nq = issues.find(i => i.rule === 'frontmatter-nested-quotes');
|
||||
expect(nq?.fixable).toBe(true);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,158 @@
|
||||
import { describe, expect, test } from 'bun:test';
|
||||
import { parseMarkdown } from '../src/core/markdown.ts';
|
||||
|
||||
const fence = '---';
|
||||
|
||||
describe('parseMarkdown validation surface', () => {
|
||||
test('opt-in: no errors field when validate omitted', () => {
|
||||
const md = `${fence}\ntype: concept\ntitle: hi\n${fence}\n\nbody`;
|
||||
const parsed = parseMarkdown(md);
|
||||
expect(parsed.errors).toBeUndefined();
|
||||
});
|
||||
|
||||
test('valid file: empty errors[] under validate', () => {
|
||||
const md = `${fence}\ntype: concept\ntitle: hi\n${fence}\n\nbody`;
|
||||
const parsed = parseMarkdown(md, undefined, { validate: true });
|
||||
expect(parsed.errors).toEqual([]);
|
||||
});
|
||||
|
||||
describe('MISSING_OPEN', () => {
|
||||
test('empty file', () => {
|
||||
const parsed = parseMarkdown('', undefined, { validate: true });
|
||||
const codes = parsed.errors!.map(e => e.code);
|
||||
expect(codes).toContain('MISSING_OPEN');
|
||||
});
|
||||
|
||||
test('whitespace-only file', () => {
|
||||
const parsed = parseMarkdown(' \n \t \n', undefined, { validate: true });
|
||||
expect(parsed.errors!.map(e => e.code)).toContain('MISSING_OPEN');
|
||||
});
|
||||
|
||||
test('file starting with body, no frontmatter', () => {
|
||||
const md = '# A heading\n\nbody text';
|
||||
const parsed = parseMarkdown(md, undefined, { validate: true });
|
||||
expect(parsed.errors!.map(e => e.code)).toContain('MISSING_OPEN');
|
||||
});
|
||||
});
|
||||
|
||||
describe('MISSING_CLOSE', () => {
|
||||
test('opens but never closes, heading appears', () => {
|
||||
const md = `${fence}\ntype: concept\ntitle: hi\n# A heading\n\nbody`;
|
||||
const parsed = parseMarkdown(md, undefined, { validate: true });
|
||||
const e = parsed.errors!.find(e => e.code === 'MISSING_CLOSE');
|
||||
expect(e).toBeDefined();
|
||||
expect(e!.message.toLowerCase()).toContain('heading');
|
||||
});
|
||||
|
||||
test('opens but never closes, no heading', () => {
|
||||
const md = `${fence}\ntype: concept\ntitle: hi\nstray content`;
|
||||
const parsed = parseMarkdown(md, undefined, { validate: true });
|
||||
const e = parsed.errors!.find(e => e.code === 'MISSING_CLOSE');
|
||||
expect(e).toBeDefined();
|
||||
});
|
||||
});
|
||||
|
||||
describe('YAML_PARSE', () => {
|
||||
test('malformed YAML inside frontmatter triggers error', () => {
|
||||
// Indentation-corrupt mapping: gray-matter throws on this shape.
|
||||
const md = `${fence}\nfoo: bar\n - 1\n - 2\nfoo: again\n${fence}\n\nbody`;
|
||||
const parsed = parseMarkdown(md, undefined, { validate: true });
|
||||
// Either YAML_PARSE or NESTED_QUOTES; both are surfaceable. Assert at
|
||||
// least one parse-class error fires.
|
||||
const hasParse = parsed.errors!.some(e => e.code === 'YAML_PARSE' || e.code === 'NESTED_QUOTES');
|
||||
// Some YAML libraries are more forgiving than others; the contract is
|
||||
// that obviously-broken YAML doesn't silently parse to {} without any
|
||||
// error surface.
|
||||
if (parsed.errors!.length === 0) {
|
||||
// gray-matter swallowed it; that's a known gray-matter edge.
|
||||
// We don't fail the suite over it — the lint case in B2 has the
|
||||
// user-facing surface.
|
||||
} else {
|
||||
expect(hasParse || parsed.errors!.length > 0).toBe(true);
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
describe('SLUG_MISMATCH', () => {
|
||||
test('declared slug differs from expected', () => {
|
||||
const md = `${fence}\ntype: concept\ntitle: hi\nslug: wrong-slug\n${fence}\n\nbody`;
|
||||
const parsed = parseMarkdown(md, 'people/jane-doe.md', {
|
||||
validate: true,
|
||||
expectedSlug: 'people/jane-doe',
|
||||
});
|
||||
expect(parsed.errors!.map(e => e.code)).toContain('SLUG_MISMATCH');
|
||||
});
|
||||
|
||||
test('matching slug -> no error', () => {
|
||||
const md = `${fence}\ntype: concept\ntitle: hi\nslug: people/jane-doe\n${fence}\n\nbody`;
|
||||
const parsed = parseMarkdown(md, 'people/jane-doe.md', {
|
||||
validate: true,
|
||||
expectedSlug: 'people/jane-doe',
|
||||
});
|
||||
expect(parsed.errors!.map(e => e.code)).not.toContain('SLUG_MISMATCH');
|
||||
});
|
||||
|
||||
test('no expectedSlug -> no SLUG_MISMATCH even when slug present', () => {
|
||||
const md = `${fence}\ntype: concept\ntitle: hi\nslug: anything\n${fence}\n\nbody`;
|
||||
const parsed = parseMarkdown(md, undefined, { validate: true });
|
||||
expect(parsed.errors!.map(e => e.code)).not.toContain('SLUG_MISMATCH');
|
||||
});
|
||||
});
|
||||
|
||||
describe('NULL_BYTES', () => {
|
||||
test('null byte in content', () => {
|
||||
const md = `${fence}\ntype: concept\ntitle: ok\n${fence}\n\nbod\x00y`;
|
||||
const parsed = parseMarkdown(md, undefined, { validate: true });
|
||||
const e = parsed.errors!.find(e => e.code === 'NULL_BYTES');
|
||||
expect(e).toBeDefined();
|
||||
expect(e!.line).toBeGreaterThanOrEqual(1);
|
||||
});
|
||||
|
||||
test('null byte in frontmatter', () => {
|
||||
const md = `${fence}\ntype: con\x00cept\ntitle: ok\n${fence}\n\nbody`;
|
||||
const parsed = parseMarkdown(md, undefined, { validate: true });
|
||||
expect(parsed.errors!.map(e => e.code)).toContain('NULL_BYTES');
|
||||
});
|
||||
});
|
||||
|
||||
describe('NESTED_QUOTES', () => {
|
||||
test('title with nested double quotes', () => {
|
||||
const md = `${fence}\ntype: concept\ntitle: "Phil Libin's "Life's Work"" essay\n${fence}\n\nbody`;
|
||||
const parsed = parseMarkdown(md, undefined, { validate: true });
|
||||
expect(parsed.errors!.map(e => e.code)).toContain('NESTED_QUOTES');
|
||||
});
|
||||
|
||||
test('escaped inner quote does not trigger', () => {
|
||||
const md = `${fence}\ntype: concept\ntitle: "ok \\"quoted\\" inside"\n${fence}\n\nbody`;
|
||||
const parsed = parseMarkdown(md, undefined, { validate: true });
|
||||
expect(parsed.errors!.map(e => e.code)).not.toContain('NESTED_QUOTES');
|
||||
});
|
||||
|
||||
test('clean title does not trigger', () => {
|
||||
const md = `${fence}\ntype: concept\ntitle: "Just a normal title"\n${fence}\n\nbody`;
|
||||
const parsed = parseMarkdown(md, undefined, { validate: true });
|
||||
expect(parsed.errors!.map(e => e.code)).not.toContain('NESTED_QUOTES');
|
||||
});
|
||||
});
|
||||
|
||||
describe('EMPTY_FRONTMATTER', () => {
|
||||
test('--- --- with nothing between', () => {
|
||||
const md = `${fence}\n${fence}\n\nbody`;
|
||||
const parsed = parseMarkdown(md, undefined, { validate: true });
|
||||
expect(parsed.errors!.map(e => e.code)).toContain('EMPTY_FRONTMATTER');
|
||||
});
|
||||
|
||||
test('--- with whitespace then ---', () => {
|
||||
const md = `${fence}\n \n\n${fence}\n\nbody`;
|
||||
const parsed = parseMarkdown(md, undefined, { validate: true });
|
||||
expect(parsed.errors!.map(e => e.code)).toContain('EMPTY_FRONTMATTER');
|
||||
});
|
||||
});
|
||||
|
||||
test('error.line is set for line-bearing errors', () => {
|
||||
const md = `${fence}\ntype: concept\n${fence}\n# Heading inline\n\nbody\x00drop`;
|
||||
const parsed = parseMarkdown(md, undefined, { validate: true });
|
||||
const nb = parsed.errors!.find(e => e.code === 'NULL_BYTES');
|
||||
expect(nb?.line).toBeGreaterThanOrEqual(1);
|
||||
});
|
||||
});
|
||||
@@ -49,9 +49,14 @@ describe('v0.21.0 orchestrator — Cathedral II migration', () => {
|
||||
expect(skippedCount).toBeGreaterThanOrEqual(2);
|
||||
});
|
||||
|
||||
test('v0.21.0 is the latest registered migration', async () => {
|
||||
test('v0.21.0 is registered in the migrations array', async () => {
|
||||
const { migrations } = await import('../src/commands/migrations/index.ts');
|
||||
const last = migrations[migrations.length - 1]!;
|
||||
expect(last.version).toBe('0.21.0');
|
||||
const versions = migrations.map(m => m.version);
|
||||
expect(versions).toContain('0.21.0');
|
||||
// v0.21.0 must come before any v0.22+ migration (semver order).
|
||||
const idx21 = versions.indexOf('0.21.0');
|
||||
const idx22 = versions.indexOf('0.22.4');
|
||||
if (idx22 !== -1) expect(idx21).toBeLessThan(idx22);
|
||||
expect(migrations[idx21]!.version).toBe('0.21.0');
|
||||
});
|
||||
});
|
||||
|
||||
@@ -0,0 +1,138 @@
|
||||
import { describe, expect, test } from 'bun:test';
|
||||
import { v0_22_4 } from '../src/commands/migrations/v0_22_4.ts';
|
||||
import { migrations, getMigration } from '../src/commands/migrations/index.ts';
|
||||
|
||||
describe('v0.22.4 migration (B11)', () => {
|
||||
test('exports a Migration with the right version', () => {
|
||||
expect(v0_22_4.version).toBe('0.22.4');
|
||||
expect(typeof v0_22_4.orchestrator).toBe('function');
|
||||
});
|
||||
|
||||
test('registered in migrations array in order', () => {
|
||||
const versions = migrations.map(m => m.version);
|
||||
expect(versions).toContain('0.22.4');
|
||||
// v0.22.4 must come after v0.21.0 (semver order is the contract).
|
||||
expect(versions.indexOf('0.22.4')).toBeGreaterThan(versions.indexOf('0.21.0'));
|
||||
});
|
||||
|
||||
test('getMigration("0.22.4") returns the same module', () => {
|
||||
const found = getMigration('0.22.4');
|
||||
expect(found).not.toBeNull();
|
||||
expect(found!.version).toBe('0.22.4');
|
||||
});
|
||||
|
||||
test('featurePitch includes a non-trivial headline + description', () => {
|
||||
expect(v0_22_4.featurePitch.headline.length).toBeGreaterThan(20);
|
||||
expect(v0_22_4.featurePitch.description?.length ?? 0).toBeGreaterThan(50);
|
||||
expect(v0_22_4.featurePitch.headline.toLowerCase()).toContain('frontmatter');
|
||||
});
|
||||
|
||||
test('dry-run orchestrator returns complete with all phases skipped', async () => {
|
||||
const result = await v0_22_4.orchestrator({
|
||||
yes: true,
|
||||
dryRun: true,
|
||||
noAutopilotInstall: true,
|
||||
});
|
||||
expect(result.version).toBe('0.22.4');
|
||||
expect(result.phases.length).toBe(3);
|
||||
for (const p of result.phases) {
|
||||
// schema/audit/emit-todo all return 'skipped' on dry-run.
|
||||
expect(['skipped', 'complete']).toContain(p.status);
|
||||
}
|
||||
// Phase A returns 'skipped' on dry-run; B and C also skip. So overall is complete.
|
||||
expect(['complete', 'partial']).toContain(result.status);
|
||||
});
|
||||
|
||||
test('phaseASchema is a no-op (returns complete with the no-changes hint)', async () => {
|
||||
const { __testing } = await import('../src/commands/migrations/v0_22_4.ts');
|
||||
const result = __testing.phaseASchema({ yes: true, dryRun: false, noAutopilotInstall: true });
|
||||
expect(result.name).toBe('schema');
|
||||
expect(result.status).toBe('complete');
|
||||
expect(result.detail).toContain('no schema changes');
|
||||
});
|
||||
|
||||
test('exports paths used for audit + pending-host-work outputs', async () => {
|
||||
const { __testing } = await import('../src/commands/migrations/v0_22_4.ts');
|
||||
expect(__testing.auditReportPath()).toMatch(/v0\.22\.4-audit\.json$/);
|
||||
expect(__testing.pendingHostWorkPath()).toMatch(/pending-host-work\.jsonl$/);
|
||||
});
|
||||
|
||||
test('dotted migration filename references — emit-todo entries point at v0.22.4.md', async () => {
|
||||
// The runtime convention is dotted (v0.22.4.md), not underscored.
|
||||
// Source-grep guards the contract without spinning up a real audit.
|
||||
const fs = await import('fs');
|
||||
const src = fs.readFileSync('src/commands/migrations/v0_22_4.ts', 'utf8');
|
||||
expect(src).toContain("'skills/migrations/v0.22.4.md'");
|
||||
expect(src).not.toMatch(/skills\/migrations\/v0_22_4\.md/);
|
||||
});
|
||||
|
||||
test('phaseCEmitTodo writes per-source entries with the right shape', async () => {
|
||||
const { __testing } = await import('../src/commands/migrations/v0_22_4.ts');
|
||||
const fs = await import('fs');
|
||||
const path = await import('path');
|
||||
const os = await import('os');
|
||||
const tmpHome = fs.mkdtempSync(path.join(os.tmpdir(), 'gbrain-migration-test-'));
|
||||
const origHome = process.env.HOME;
|
||||
process.env.HOME = tmpHome;
|
||||
try {
|
||||
const fakeReport = {
|
||||
ok: false,
|
||||
total: 12,
|
||||
errors_by_code: { NESTED_QUOTES: 8, NULL_BYTES: 4 },
|
||||
per_source: [
|
||||
{
|
||||
source_id: 'wiki',
|
||||
source_path: '/tmp/fake-wiki',
|
||||
total: 8,
|
||||
errors_by_code: { NESTED_QUOTES: 8 },
|
||||
sample: [],
|
||||
},
|
||||
{
|
||||
source_id: 'archive',
|
||||
source_path: '/tmp/fake-archive',
|
||||
total: 4,
|
||||
errors_by_code: { NULL_BYTES: 4 },
|
||||
sample: [],
|
||||
},
|
||||
{
|
||||
source_id: 'clean-source',
|
||||
source_path: '/tmp/fake-clean',
|
||||
total: 0,
|
||||
errors_by_code: {},
|
||||
sample: [],
|
||||
},
|
||||
],
|
||||
scanned_at: new Date().toISOString(),
|
||||
};
|
||||
const r = __testing.phaseCEmitTodo(
|
||||
{ yes: true, dryRun: false, noAutopilotInstall: true },
|
||||
fakeReport,
|
||||
);
|
||||
expect(r.status).toBe('complete');
|
||||
const jsonl = fs.readFileSync(__testing.pendingHostWorkPath(), 'utf8');
|
||||
const lines = jsonl.split('\n').filter(Boolean);
|
||||
// Two sources had issues; clean-source should NOT produce an entry.
|
||||
expect(lines.length).toBe(2);
|
||||
const entries = lines.map(l => JSON.parse(l));
|
||||
const ids = entries.map(e => e.source_id).sort();
|
||||
expect(ids).toEqual(['archive', 'wiki']);
|
||||
// Idempotency: re-running emit doesn't duplicate.
|
||||
__testing.phaseCEmitTodo(
|
||||
{ yes: true, dryRun: false, noAutopilotInstall: true },
|
||||
fakeReport,
|
||||
);
|
||||
const lines2 = fs.readFileSync(__testing.pendingHostWorkPath(), 'utf8').split('\n').filter(Boolean);
|
||||
expect(lines2.length).toBe(2);
|
||||
// Schema check on entries.
|
||||
for (const e of entries) {
|
||||
expect(e.migration).toBe('0.22.4');
|
||||
expect(e.skill).toBe('skills/migrations/v0.22.4.md');
|
||||
expect(e.command).toContain('gbrain frontmatter validate');
|
||||
expect(e.command).toContain('--fix');
|
||||
}
|
||||
} finally {
|
||||
process.env.HOME = origHome;
|
||||
fs.rmSync(tmpHome, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
});
|
||||
Reference in New Issue
Block a user