mirror of
https://github.com/garrytan/gbrain.git
synced 2026-07-27 22:15:33 +00:00
* v0.37.1.0 feat: skill_brain_first doctor check + auto-fix + declarative opt-out Cathedral wave superseding PR #1206. Doctor now scans every SKILL.md for external-lookup tools (web_search / web_fetch / exa / perplexity / happenstance / crustdata / captain_api / firecrawl) and warns when the skill has no brain- first compliance signal. gbrain doctor --fix auto-inserts the canonical > **Convention:** see [conventions/brain-first.md](...) callout via the dry-fix.ts MISSING_RULE_PATTERNS extension (sharing safety gates with the existing REPLACE patterns). Motivated by the 2026-05-19 tweet-shield incident: cross-modal eval flagged Garry's Palantir tweet as risky because no model knew he built it, but the brain already had "designed the entire Finance product UI" and "150+ PSDs from April-December 2006." Static check catches authorship; v0.37+ runtime gate (filed in TODOS.md) closes the dispatch side. Key design decisions locked via /plan-eng-review + codex outside-voice review: - A1: frontmatter ships only brain_first: exempt (no required/n/a enum) - A2: snapshot+diff audit at ~/.gbrain/audit/skill-brain-first-YYYY-Www.jsonl with transition-only writes (stable brains = 0 lines/run) - A3: scaffold template pre-inserts callout; skillify check fails (exit 1) on external + no callout + no exempt - A4: position-relative gate is BODY-ONLY (frontmatter tools: [web_search] declaration doesn't false-flag the skill) - Q1: single pure analyzeSkillBrainFirst() helper consumed by 3 surfaces - CMT1: no upgrade migration — doctor surfaces hint, --fix applies via dry-fix safety gates (user stays in loop) - CMT2: dropped tools+writes_pages auto-exemption (was hiding mixed-class skills like idea-ingest/meeting-ingestion/data-research) Trio: VERSION + package.json + CHANGELOG aligned at 0.37.1.0. 56 unit cases + 12 E2E cases pass. 170 related existing tests pass unchanged. Self-dogfood: gbrain doctor against this repo's skills/ reports skill_brain_first: ok across 43 skills (compliant or exempt). functional-area-resolver and strategic-reading skills gained brain_first: exempt to validate the declarative opt-out in production code (both name perplexity in dispatcher prose without calling it). Co-Authored-By: garrytan-agents <noreply@github.com> Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs: update CLAUDE.md for v0.37.1.0 skill_brain_first wave Added Key Files entries for the four new modules: - src/core/skill-frontmatter.ts (shared parser) - src/core/skill-brain-first.ts (analyzer + FORMERLY_HARDCODED_EXEMPT) - src/core/skill-fix-gates.ts (extracted safety primitives) - src/core/audit-skill-brain-first.ts (snapshot+diff JSONL) Extended existing entries: - src/core/filing-audit.ts: rewired to shared parser - src/core/dry-fix.ts: MISSING_RULE_PATTERNS INSERT pattern type - src/commands/doctor.ts: skill_brain_first check + tweet-shield framing - src/commands/skillify-check.ts: required item 12 + scaffold pre-insert Added test inventory entries: - test/skill-brain-first.test.ts (56 unit cases) - test/e2e/skill-brain-first.test.ts (12 E2E cases) Regenerated llms-full.txt via bun run build:llms. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(ci): skill_brain_first guard uses doctor --fast to skip engine connect CI run #76881161092 failed because scripts/check-skill-brain-first.sh invoked plain `gbrain doctor --json`, which routes through connectEngine(). With no ~/.gbrain/config.json present (CI's case — runner is bun-only, no brain init), connectEngine() exits 1 with "No brain configured." and emits zero stdout. The python parser sees an empty file and returns parse_error, failing the verify gate. Fix: pass --fast to doctor. --fast routes through runDoctor(null, ...) which runs the filesystem-only check set (resolver_health, skill_conformance, skill_brain_first) and emits the standard single-line JSON envelope the parser expects. skill_brain_first is filesystem-only by design (scans SKILL.md, no DB touch), so --fast is the correct knob, not a workaround. Verified by reproducing the CI failure mode locally with GBRAIN_HOME=/tmp/empty-... — gate now passes both with and without a configured brain. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore: rebump v0.37.1.0 → v0.37.3.0 (queue collision with #1214) PR #1214 (brainstorm + lsd) claimed v0.37.1.0 concurrently with #1215. Skipping 0.37.2.0 leaves a buffer for #1214's adjacent slot. Trio (VERSION + package.json + CHANGELOG header + inline "To take advantage of v0.37.3.0" block) aligned at 0.37.3.0. No behavior changes — version metadata only. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: garrytan-agents <noreply@github.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
621 lines
25 KiB
TypeScript
621 lines
25 KiB
TypeScript
/**
|
||
* Unit suite for the v0.36.x skill_brain_first analyzer (T9 from
|
||
* /plan-eng-review).
|
||
*
|
||
* Drives the fixture corpus at `test/fixtures/brain-first-skills/*`.
|
||
* Absorbs PR #1206's 10 inline-string cases (IRON-RULE regression
|
||
* preservation: every behavior the PR pinned must still pass here).
|
||
*
|
||
* Coverage targets:
|
||
* - parseSkillFrontmatter: canonical / typo variants / array fields
|
||
* - analyzeSkillBrainFirst: all 6 BrainFirstReason values × edge cases
|
||
* - Position-relative gate body-only semantics (F6 regression)
|
||
* - Canonical callout regex shape variations (F7 regression)
|
||
* - FORMERLY_HARDCODED_EXEMPT membership preserved
|
||
* - buildBrainFirstSummaryLine output shape (doctor + skillify consume it)
|
||
* - Snapshot+diff audit logic (A2 contract: zero writes on no-transition runs)
|
||
*
|
||
* Hermetic: no DATABASE_URL, no network, no real audit dir. The audit
|
||
* snapshot tests redirect `GBRAIN_AUDIT_DIR` to a tempdir.
|
||
*/
|
||
|
||
import { describe, expect, test } from 'bun:test';
|
||
import { existsSync, mkdirSync, readdirSync, readFileSync, rmSync } from 'fs';
|
||
import { join } from 'path';
|
||
import { tmpdir } from 'os';
|
||
|
||
import { withEnv } from './helpers/with-env.ts';
|
||
import {
|
||
parseSkillFrontmatter,
|
||
formatBrainFirstTypoHint,
|
||
} from '../src/core/skill-frontmatter.ts';
|
||
import {
|
||
analyzeSkillBrainFirst,
|
||
buildBrainFirstSummaryLine,
|
||
findFirstBrainRefOffset,
|
||
findFirstExternalRefOffset,
|
||
stripFrontmatter,
|
||
CONVENTION_CALLOUT_RE,
|
||
PHASE_HEADING_RE,
|
||
EXTERNAL_LOOKUP_PATTERNS,
|
||
FORMERLY_HARDCODED_EXEMPT,
|
||
} from '../src/core/skill-brain-first.ts';
|
||
import {
|
||
diffAgainstSnapshot,
|
||
loadSnapshot,
|
||
writeSnapshotAtomically,
|
||
computeBrainFirstAuditFilename,
|
||
logBrainFirstEvent,
|
||
readRecentBrainFirstEvents,
|
||
appendAuditEventsForTransitions,
|
||
_resetWarnedSetForTests,
|
||
} from '../src/core/audit-skill-brain-first.ts';
|
||
|
||
const FIXTURE_DIR = join(import.meta.dir, 'fixtures', 'brain-first-skills');
|
||
|
||
function loadFixture(name: string): { content: string; skillName: string } {
|
||
const skillMd = join(FIXTURE_DIR, name, 'SKILL.md');
|
||
return { content: readFileSync(skillMd, 'utf-8'), skillName: name };
|
||
}
|
||
|
||
// ---------------------------------------------------------------------------
|
||
// parseSkillFrontmatter
|
||
// ---------------------------------------------------------------------------
|
||
|
||
describe('parseSkillFrontmatter', () => {
|
||
test('returns null when no frontmatter fence is present', () => {
|
||
expect(parseSkillFrontmatter('# Just a heading\nNo frontmatter here.')).toBeNull();
|
||
});
|
||
|
||
test('parses name, mutating, writes_pages, writes_to', () => {
|
||
const content = [
|
||
'---',
|
||
'name: thing',
|
||
'mutating: true',
|
||
'writes_pages: true',
|
||
'writes_to:',
|
||
' - people/',
|
||
' - companies/',
|
||
'---',
|
||
'',
|
||
'# thing',
|
||
].join('\n');
|
||
const fm = parseSkillFrontmatter(content);
|
||
expect(fm).not.toBeNull();
|
||
expect(fm!.name).toBe('thing');
|
||
expect(fm!.mutating).toBe(true);
|
||
expect(fm!.writes_pages).toBe(true);
|
||
expect(fm!.writes_to).toEqual(['people/', 'companies/']);
|
||
});
|
||
|
||
test('parses inline array forms for writes_to / tools / triggers', () => {
|
||
const content = [
|
||
'---',
|
||
'name: x',
|
||
'writes_to: [people/, companies/]',
|
||
'tools: [search, query, put_page]',
|
||
'triggers: [foo, bar]',
|
||
'---',
|
||
].join('\n');
|
||
const fm = parseSkillFrontmatter(content);
|
||
expect(fm!.writes_to).toEqual(['people/', 'companies/']);
|
||
expect(fm!.tools).toEqual(['search', 'query', 'put_page']);
|
||
expect(fm!.triggers).toEqual(['foo', 'bar']);
|
||
});
|
||
|
||
test('canonical brain_first: exempt populates the typed field', () => {
|
||
const content = '---\nname: x\nbrain_first: exempt\n---\n';
|
||
const fm = parseSkillFrontmatter(content);
|
||
expect(fm!.brain_first).toBe('exempt');
|
||
expect(fm!.brain_first_typo).toBeUndefined();
|
||
});
|
||
|
||
test('kebab-case brain-first triggers noncanonical_key typo', () => {
|
||
const content = '---\nname: x\nbrain-first: exempt\n---\n';
|
||
const fm = parseSkillFrontmatter(content);
|
||
expect(fm!.brain_first).toBeUndefined();
|
||
expect(fm!.brain_first_typo).toBeDefined();
|
||
expect(fm!.brain_first_typo!.reason).toBe('noncanonical_key');
|
||
expect(fm!.brain_first_typo!.key).toBe('brain-first');
|
||
});
|
||
|
||
test('CamelCase BrainFirst triggers noncanonical_key typo', () => {
|
||
const content = '---\nname: x\nBrainFirst: exempt\n---\n';
|
||
const fm = parseSkillFrontmatter(content);
|
||
expect(fm!.brain_first).toBeUndefined();
|
||
expect(fm!.brain_first_typo!.reason).toBe('noncanonical_key');
|
||
});
|
||
|
||
test('quoted value triggers quoted_value typo', () => {
|
||
const content = "---\nname: x\nbrain_first: 'exempt'\n---\n";
|
||
const fm = parseSkillFrontmatter(content);
|
||
expect(fm!.brain_first).toBeUndefined();
|
||
expect(fm!.brain_first_typo!.reason).toBe('quoted_value');
|
||
});
|
||
|
||
test('capitalized value triggers capitalized_value typo', () => {
|
||
const content = '---\nname: x\nbrain_first: Exempt\n---\n';
|
||
const fm = parseSkillFrontmatter(content);
|
||
expect(fm!.brain_first).toBeUndefined();
|
||
expect(fm!.brain_first_typo!.reason).toBe('capitalized_value');
|
||
});
|
||
|
||
test('unknown value triggers unknown_value typo', () => {
|
||
const content = '---\nname: x\nbrain_first: required\n---\n';
|
||
const fm = parseSkillFrontmatter(content);
|
||
expect(fm!.brain_first).toBeUndefined();
|
||
expect(fm!.brain_first_typo!.reason).toBe('unknown_value');
|
||
});
|
||
|
||
test('formatBrainFirstTypoHint produces paste-ready strings for every reason', () => {
|
||
expect(
|
||
formatBrainFirstTypoHint({ key: 'brain-first', value: 'exempt', reason: 'noncanonical_key' }),
|
||
).toContain('snake_case');
|
||
expect(
|
||
formatBrainFirstTypoHint({ key: 'brain_first', value: "'exempt'", reason: 'quoted_value' }),
|
||
).toContain('drop the quotes');
|
||
expect(
|
||
formatBrainFirstTypoHint({ key: 'brain_first', value: 'Exempt', reason: 'capitalized_value' }),
|
||
).toContain('lowercase');
|
||
expect(
|
||
formatBrainFirstTypoHint({ key: 'brain_first', value: 'required', reason: 'unknown_value' }),
|
||
).toContain("only 'brain_first: exempt'");
|
||
expect(formatBrainFirstTypoHint(undefined)).toBeNull();
|
||
});
|
||
});
|
||
|
||
// ---------------------------------------------------------------------------
|
||
// stripFrontmatter + offset helpers
|
||
// ---------------------------------------------------------------------------
|
||
|
||
describe('stripFrontmatter', () => {
|
||
test('removes leading YAML fence', () => {
|
||
const content = '---\nname: x\n---\n# Body\nrest';
|
||
expect(stripFrontmatter(content)).toBe('# Body\nrest');
|
||
});
|
||
|
||
test('leaves content unchanged when no fence is present', () => {
|
||
expect(stripFrontmatter('# No fence')).toBe('# No fence');
|
||
});
|
||
|
||
test('CRITICAL F6 — frontmatter exclusion prevents tools: [web_search] false-positive', () => {
|
||
const content = [
|
||
'---',
|
||
'name: x',
|
||
'tools: [web_search]',
|
||
'---',
|
||
'',
|
||
'# x',
|
||
'',
|
||
'Body says gbrain search comes first.',
|
||
'Then perplexity for follow-up.',
|
||
].join('\n');
|
||
const body = stripFrontmatter(content);
|
||
// `web_search` appears ONLY in frontmatter — body has no external pattern
|
||
// before the gbrain reference.
|
||
expect(findFirstBrainRefOffset(body)).toBeGreaterThanOrEqual(0);
|
||
// Body should NOT contain `web_search` (it was in the stripped frontmatter).
|
||
expect(body.includes('web_search')).toBe(false);
|
||
});
|
||
});
|
||
|
||
describe('offset helpers', () => {
|
||
test('findFirstBrainRefOffset finds earliest gbrain ref', () => {
|
||
const body = 'Some preamble. gbrain search and later gbrain query.';
|
||
const o = findFirstBrainRefOffset(body);
|
||
expect(o).toBe(body.indexOf('gbrain search'));
|
||
});
|
||
|
||
test('findFirstExternalRefOffset finds earliest external pattern', () => {
|
||
const body = 'First call perplexity, then later web_search.';
|
||
const o = findFirstExternalRefOffset(body);
|
||
expect(o).toBe(body.indexOf('perplexity'));
|
||
});
|
||
|
||
test('returns -1 when no match', () => {
|
||
expect(findFirstBrainRefOffset('no brain ref here')).toBe(-1);
|
||
expect(findFirstExternalRefOffset('no external ref here')).toBe(-1);
|
||
});
|
||
});
|
||
|
||
// ---------------------------------------------------------------------------
|
||
// CONVENTION_CALLOUT_RE (F7 — path-syntax-agnostic)
|
||
// ---------------------------------------------------------------------------
|
||
|
||
describe('CONVENTION_CALLOUT_RE', () => {
|
||
test('matches plain-text path form (brain-ops shape)', () => {
|
||
const line = '> **Convention:** See skills/conventions/brain-first.md for the 5-step lookup protocol.';
|
||
expect(CONVENTION_CALLOUT_RE.test(line)).toBe(true);
|
||
});
|
||
|
||
test('matches relative plain-text form (perplexity-research shape)', () => {
|
||
const line = '> **Convention:** see conventions/brain-first.md for the lookup chain.';
|
||
expect(CONVENTION_CALLOUT_RE.test(line)).toBe(true);
|
||
});
|
||
|
||
test('matches markdown-link form (auto-fix output shape)', () => {
|
||
const line = '> **Convention:** see [conventions/brain-first.md](../conventions/brain-first.md) for the lookup chain.';
|
||
expect(CONVENTION_CALLOUT_RE.test(line)).toBe(true);
|
||
});
|
||
|
||
test('does NOT match a non-blockquote mention of brain-first', () => {
|
||
const line = 'In a paragraph: see conventions/brain-first.md for the chain.';
|
||
expect(CONVENTION_CALLOUT_RE.test(line)).toBe(false);
|
||
});
|
||
|
||
test('does NOT match an unrelated Convention callout', () => {
|
||
const line = '> **Convention:** see conventions/quality.md for citation format.';
|
||
expect(CONVENTION_CALLOUT_RE.test(line)).toBe(false);
|
||
});
|
||
});
|
||
|
||
// ---------------------------------------------------------------------------
|
||
// PHASE_HEADING_RE
|
||
// ---------------------------------------------------------------------------
|
||
|
||
describe('PHASE_HEADING_RE', () => {
|
||
test('matches ## Phase 1: Brain-First Lookup', () => {
|
||
expect(PHASE_HEADING_RE.test('## Phase 1: Brain-First Lookup')).toBe(true);
|
||
});
|
||
|
||
test('matches ### Step 0: Brain Context', () => {
|
||
expect(PHASE_HEADING_RE.test('### Step 0: Brain Context')).toBe(true);
|
||
});
|
||
|
||
test('does NOT match # Phase 1 (H1, not H2+)', () => {
|
||
expect(PHASE_HEADING_RE.test('# Phase 1: Brain Lookup')).toBe(false);
|
||
});
|
||
|
||
test('does NOT match Phase 2 etc.', () => {
|
||
expect(PHASE_HEADING_RE.test('## Phase 2: Synthesis')).toBe(false);
|
||
});
|
||
});
|
||
|
||
// ---------------------------------------------------------------------------
|
||
// EXTERNAL_LOOKUP_PATTERNS coverage matrix
|
||
// ---------------------------------------------------------------------------
|
||
|
||
describe('EXTERNAL_LOOKUP_PATTERNS', () => {
|
||
test('all 8 patterns are present and named', () => {
|
||
const names = EXTERNAL_LOOKUP_PATTERNS.map(p => p.name).sort();
|
||
expect(names).toEqual([
|
||
'captain_api',
|
||
'crustdata',
|
||
'exa',
|
||
'firecrawl',
|
||
'happenstance',
|
||
'perplexity',
|
||
'web_fetch',
|
||
'web_search',
|
||
]);
|
||
});
|
||
|
||
test('captain_api regex handles all four shapes', () => {
|
||
const re = EXTERNAL_LOOKUP_PATTERNS.find(p => p.name === 'captain_api')!.re;
|
||
expect(re.test('use captain api')).toBe(true);
|
||
expect(re.test('use captain_api')).toBe(true);
|
||
expect(re.test('use captain-api')).toBe(true);
|
||
expect(re.test('use captainapi')).toBe(true);
|
||
expect(re.test('captain is not an api ref')).toBe(false);
|
||
});
|
||
|
||
test('exa requires a separator (avoids matching exam, exalt)', () => {
|
||
const re = EXTERNAL_LOOKUP_PATTERNS.find(p => p.name === 'exa')!.re;
|
||
expect(re.test('use exa.search')).toBe(true);
|
||
expect(re.test('use exa_lookup')).toBe(true);
|
||
expect(re.test('use exa-api')).toBe(true);
|
||
expect(re.test('do not use exam')).toBe(false);
|
||
expect(re.test('exalt the brain')).toBe(false);
|
||
});
|
||
});
|
||
|
||
// ---------------------------------------------------------------------------
|
||
// FORMERLY_HARDCODED_EXEMPT preservation
|
||
// ---------------------------------------------------------------------------
|
||
|
||
describe('FORMERLY_HARDCODED_EXEMPT', () => {
|
||
test('preserves all 40+ entries from PR #1206 allowlist', () => {
|
||
// Spot-check across the three categories from PR #1206.
|
||
expect(FORMERLY_HARDCODED_EXEMPT.has('brain-ops')).toBe(true);
|
||
expect(FORMERLY_HARDCODED_EXEMPT.has('gbrain')).toBe(true);
|
||
expect(FORMERLY_HARDCODED_EXEMPT.has('exa')).toBe(true);
|
||
expect(FORMERLY_HARDCODED_EXEMPT.has('perplexity-research')).toBe(false); // not in PR allowlist
|
||
expect(FORMERLY_HARDCODED_EXEMPT.has('browser')).toBe(true);
|
||
expect(FORMERLY_HARDCODED_EXEMPT.has('cron-scheduler')).toBe(true);
|
||
expect(FORMERLY_HARDCODED_EXEMPT.has('ask-user')).toBe(true);
|
||
});
|
||
|
||
test('contains at least 40 entries (preserves PR #1206 intent)', () => {
|
||
expect(FORMERLY_HARDCODED_EXEMPT.size).toBeGreaterThanOrEqual(40);
|
||
});
|
||
});
|
||
|
||
// ---------------------------------------------------------------------------
|
||
// analyzeSkillBrainFirst — fixture corpus drive (the IRON-RULE regression set)
|
||
// ---------------------------------------------------------------------------
|
||
|
||
describe('analyzeSkillBrainFirst (fixture corpus)', () => {
|
||
test('compliant-callout → ok via compliant_callout reason', () => {
|
||
const { content, skillName } = loadFixture('compliant-callout');
|
||
const fm = parseSkillFrontmatter(content);
|
||
const result = analyzeSkillBrainFirst(content, skillName, fm);
|
||
expect(result.status).toBe('ok');
|
||
expect(result.reason).toBe('compliant_callout');
|
||
expect(result.external_patterns_matched.length).toBeGreaterThan(0);
|
||
});
|
||
|
||
test('compliant-phase → ok via compliant_phase reason', () => {
|
||
const { content, skillName } = loadFixture('compliant-phase');
|
||
const fm = parseSkillFrontmatter(content);
|
||
const result = analyzeSkillBrainFirst(content, skillName, fm);
|
||
expect(result.status).toBe('ok');
|
||
expect(result.reason).toBe('compliant_phase');
|
||
});
|
||
|
||
test('compliant-position → ok via compliant_position reason', () => {
|
||
const { content, skillName } = loadFixture('compliant-position');
|
||
const fm = parseSkillFrontmatter(content);
|
||
const result = analyzeSkillBrainFirst(content, skillName, fm);
|
||
expect(result.status).toBe('ok');
|
||
expect(result.reason).toBe('compliant_position');
|
||
});
|
||
|
||
test('missing-brain-first → warn via missing_brain_first', () => {
|
||
const { content, skillName } = loadFixture('missing-brain-first');
|
||
const fm = parseSkillFrontmatter(content);
|
||
const result = analyzeSkillBrainFirst(content, skillName, fm);
|
||
expect(result.status).toBe('warn');
|
||
expect(result.reason).toBe('missing_brain_first');
|
||
expect(result.external_patterns_matched).toContain('web_search');
|
||
expect(result.external_patterns_matched).toContain('perplexity');
|
||
});
|
||
|
||
test('exempt-frontmatter → ok via exempt_explicit', () => {
|
||
const { content, skillName } = loadFixture('exempt-frontmatter');
|
||
const fm = parseSkillFrontmatter(content);
|
||
const result = analyzeSkillBrainFirst(content, skillName, fm);
|
||
expect(result.status).toBe('ok');
|
||
expect(result.reason).toBe('exempt_explicit');
|
||
});
|
||
|
||
test('no-external → ok via exempt_no_external', () => {
|
||
const { content, skillName } = loadFixture('no-external');
|
||
const fm = parseSkillFrontmatter(content);
|
||
const result = analyzeSkillBrainFirst(content, skillName, fm);
|
||
expect(result.status).toBe('ok');
|
||
expect(result.reason).toBe('exempt_no_external');
|
||
expect(result.external_patterns_matched).toEqual([]);
|
||
});
|
||
|
||
test('multi-pattern → warn with all 3 external patterns matched', () => {
|
||
const { content, skillName } = loadFixture('multi-pattern');
|
||
const fm = parseSkillFrontmatter(content);
|
||
const result = analyzeSkillBrainFirst(content, skillName, fm);
|
||
expect(result.status).toBe('warn');
|
||
expect(result.reason).toBe('missing_brain_first');
|
||
expect(result.external_patterns_matched).toContain('exa');
|
||
expect(result.external_patterns_matched).toContain('perplexity');
|
||
expect(result.external_patterns_matched).toContain('crustdata');
|
||
});
|
||
|
||
test('typo-frontmatter → warn with typo_hint surfaced', () => {
|
||
const { content, skillName } = loadFixture('typo-frontmatter');
|
||
const fm = parseSkillFrontmatter(content);
|
||
const result = analyzeSkillBrainFirst(content, skillName, fm);
|
||
expect(result.status).toBe('warn');
|
||
expect(result.typo_hint).toBeDefined();
|
||
expect(result.typo_hint).toContain('brain-first');
|
||
expect(result.typo_hint).toContain('snake_case');
|
||
});
|
||
|
||
test('negation-prose → ok via compliant_position (brain ref appears first)', () => {
|
||
const { content, skillName } = loadFixture('negation-prose');
|
||
const fm = parseSkillFrontmatter(content);
|
||
const result = analyzeSkillBrainFirst(content, skillName, fm);
|
||
expect(result.status).toBe('ok');
|
||
expect(result.reason).toBe('compliant_position');
|
||
});
|
||
});
|
||
|
||
// ---------------------------------------------------------------------------
|
||
// analyzeSkillBrainFirst — direct-input regression (PR #1206 absorption)
|
||
// ---------------------------------------------------------------------------
|
||
|
||
describe('analyzeSkillBrainFirst (PR #1206 regression preservation)', () => {
|
||
test('PR test case: good skill with Step 0 brain context passes', () => {
|
||
// Direct copy of PR #1206 `good-skill` test case shape.
|
||
const content = `---\nname: good-skill\n---\n# Good Skill\n\n## Step 0: Brain Context\nSearch the brain first with \`gbrain search\` for relevant context.\n\n## Step 1: External Lookup\nUse web_search to find additional information.\n`;
|
||
const result = analyzeSkillBrainFirst(content, 'good-skill', parseSkillFrontmatter(content));
|
||
expect(result.status).toBe('ok');
|
||
});
|
||
|
||
test('PR test case: bad skill with web_search but no brain-first → warn', () => {
|
||
const content = `---\nname: bad-skill\n---\n# Bad Skill\n\n## Step 1: Research\nUse web_search to find information about the entity.\nThen use Perplexity for deeper research.\n`;
|
||
const result = analyzeSkillBrainFirst(content, 'bad-skill', parseSkillFrontmatter(content));
|
||
expect(result.status).toBe('warn');
|
||
});
|
||
|
||
test('PR test case: skill with brain search reference is not flagged', () => {
|
||
const content = `---\nname: bf\n---\n# Brain-First\n\nFirst, search the brain for existing context.\nThen use web_search for anything missing. Use gbrain search.\n`;
|
||
const result = analyzeSkillBrainFirst(content, 'bf', parseSkillFrontmatter(content));
|
||
expect(result.status).toBe('ok');
|
||
});
|
||
|
||
test('PR test case: skill without external lookups not flagged', () => {
|
||
const content = `---\nname: internal\n---\n# Internal Skill\nThis skill only operates on local files. No external lookups.\n`;
|
||
const result = analyzeSkillBrainFirst(content, 'internal', parseSkillFrontmatter(content));
|
||
expect(result.status).toBe('ok');
|
||
expect(result.reason).toBe('exempt_no_external');
|
||
});
|
||
});
|
||
|
||
// ---------------------------------------------------------------------------
|
||
// buildBrainFirstSummaryLine
|
||
// ---------------------------------------------------------------------------
|
||
|
||
describe('buildBrainFirstSummaryLine', () => {
|
||
test('ok results render with reason tag', () => {
|
||
const line = buildBrainFirstSummaryLine({
|
||
skill: 'x',
|
||
status: 'ok',
|
||
reason: 'compliant_callout',
|
||
external_patterns_matched: [],
|
||
formerly_hardcoded_exempt: false,
|
||
});
|
||
expect(line).toBe('x: ok (compliant_callout)');
|
||
});
|
||
|
||
test('warn results include external patterns + hint', () => {
|
||
const line = buildBrainFirstSummaryLine({
|
||
skill: 'bad',
|
||
status: 'warn',
|
||
reason: 'missing_brain_first',
|
||
external_patterns_matched: ['perplexity', 'exa'],
|
||
formerly_hardcoded_exempt: false,
|
||
});
|
||
expect(line).toContain('bad');
|
||
expect(line).toContain('perplexity, exa');
|
||
});
|
||
|
||
test('warn results with formerly_hardcoded_exempt include the PR #1206 hint', () => {
|
||
const line = buildBrainFirstSummaryLine({
|
||
skill: 'browser',
|
||
status: 'warn',
|
||
reason: 'missing_brain_first',
|
||
external_patterns_matched: ['web_fetch'],
|
||
formerly_hardcoded_exempt: true,
|
||
});
|
||
expect(line).toContain('hardcoded-exempt in PR #1206');
|
||
expect(line).toContain('brain_first: exempt');
|
||
});
|
||
|
||
test('warn results carry typo hint through to message', () => {
|
||
const line = buildBrainFirstSummaryLine({
|
||
skill: 'typo',
|
||
status: 'warn',
|
||
reason: 'missing_brain_first',
|
||
external_patterns_matched: ['perplexity'],
|
||
typo_hint: 'Found brain-first: exempt — did you mean brain_first?',
|
||
formerly_hardcoded_exempt: false,
|
||
});
|
||
expect(line).toContain('typo');
|
||
expect(line).toContain('brain-first');
|
||
});
|
||
});
|
||
|
||
// ---------------------------------------------------------------------------
|
||
// Snapshot+diff audit (A2 contract — zero writes on no-transition runs)
|
||
// ---------------------------------------------------------------------------
|
||
|
||
/**
|
||
* Helper: provision an isolated audit tempdir for one test body and tear
|
||
* it down via try/finally. Wraps the body in `withEnv()` so the
|
||
* GBRAIN_AUDIT_DIR mutation is scoped to this test only — cross-test-
|
||
* safe (no leak to other tests in the same shard) per the test-
|
||
* isolation lint (R1).
|
||
*/
|
||
async function withAuditDir<T>(fn: (auditDir: string) => Promise<T> | T): Promise<T> {
|
||
_resetWarnedSetForTests();
|
||
const auditDir = join(
|
||
tmpdir(),
|
||
`brain-first-audit-${Date.now()}-${Math.random().toString(36).slice(2, 8)}`,
|
||
);
|
||
mkdirSync(auditDir, { recursive: true });
|
||
try {
|
||
return await withEnv({ GBRAIN_AUDIT_DIR: auditDir }, () => fn(auditDir));
|
||
} finally {
|
||
try { rmSync(auditDir, { recursive: true, force: true }); } catch { /* ignore */ }
|
||
}
|
||
}
|
||
|
||
describe('audit-skill-brain-first (snapshot+diff)', () => {
|
||
test('loadSnapshot returns present:false when file missing', async () => {
|
||
await withAuditDir(() => {
|
||
const r = loadSnapshot();
|
||
expect(r.present).toBe(false);
|
||
expect(r.violators.size).toBe(0);
|
||
});
|
||
});
|
||
|
||
test('writeSnapshotAtomically + loadSnapshot round-trip', async () => {
|
||
await withAuditDir(() => {
|
||
writeSnapshotAtomically(new Set(['a', 'b', 'c']));
|
||
const r = loadSnapshot();
|
||
expect(r.present).toBe(true);
|
||
expect(r.violators.has('a')).toBe(true);
|
||
expect(r.violators.has('b')).toBe(true);
|
||
expect(r.violators.has('c')).toBe(true);
|
||
});
|
||
});
|
||
|
||
test('diffAgainstSnapshot detects added/removed/unchanged', () => {
|
||
// Pure function — no audit dir needed.
|
||
const prev = new Set(['a', 'b', 'c']);
|
||
const curr = new Set(['b', 'c', 'd']);
|
||
const diff = diffAgainstSnapshot(curr, prev);
|
||
expect(diff.added).toEqual(['d']);
|
||
expect(diff.removed).toEqual(['a']);
|
||
expect(diff.unchanged).toEqual(['b', 'c']);
|
||
});
|
||
|
||
test('diff result is sorted for determinism', () => {
|
||
const prev = new Set(['c', 'a', 'd']);
|
||
const curr = new Set(['a', 'b', 'e']);
|
||
const diff = diffAgainstSnapshot(curr, prev);
|
||
expect(diff.added).toEqual(['b', 'e']);
|
||
expect(diff.removed).toEqual(['c', 'd']);
|
||
expect(diff.unchanged).toEqual(['a']);
|
||
});
|
||
|
||
test('corrupt snapshot JSON treated as missing with warn-once', async () => {
|
||
await withAuditDir(auditDir => {
|
||
const file = join(auditDir, 'skill-brain-first-snapshot.json');
|
||
require('fs').writeFileSync(file, 'not-json-at-all');
|
||
const r = loadSnapshot();
|
||
expect(r.present).toBe(false);
|
||
expect(r.violators.size).toBe(0);
|
||
});
|
||
});
|
||
|
||
test('appendAuditEventsForTransitions writes one line per added/removed', async () => {
|
||
await withAuditDir(() => {
|
||
const diff = { added: ['skill-a'], removed: ['skill-b'], unchanged: ['skill-c'] };
|
||
const patterns = new Map([['skill-a', ['web_search']]]);
|
||
appendAuditEventsForTransitions(diff, patterns, 'test-run-1');
|
||
const events = readRecentBrainFirstEvents(7);
|
||
expect(events.length).toBe(2);
|
||
const detected = events.find(e => e.event === 'detected');
|
||
const resolved = events.find(e => e.event === 'resolved');
|
||
expect(detected?.skill).toBe('skill-a');
|
||
expect(detected?.external_patterns).toEqual(['web_search']);
|
||
expect(resolved?.skill).toBe('skill-b');
|
||
});
|
||
});
|
||
|
||
test('no-transition diff produces zero audit writes (A2 contract)', async () => {
|
||
await withAuditDir(() => {
|
||
const diff = { added: [], removed: [], unchanged: ['skill-a', 'skill-b'] };
|
||
appendAuditEventsForTransitions(diff, new Map(), 'test-run-2');
|
||
const events = readRecentBrainFirstEvents(7);
|
||
expect(events.length).toBe(0);
|
||
});
|
||
});
|
||
|
||
test('logBrainFirstEvent writes a fixed event', async () => {
|
||
await withAuditDir(() => {
|
||
logBrainFirstEvent({ event: 'fixed', skill: 'browser' });
|
||
const events = readRecentBrainFirstEvents(7);
|
||
expect(events.length).toBe(1);
|
||
expect(events[0].event).toBe('fixed');
|
||
expect(events[0].skill).toBe('browser');
|
||
expect(events[0].code).toBe('SKILL_BRAIN_FIRST');
|
||
expect(events[0].severity).toBe('info');
|
||
});
|
||
});
|
||
|
||
test('computeBrainFirstAuditFilename produces ISO-week format', () => {
|
||
const name = computeBrainFirstAuditFilename(new Date('2026-05-19T10:00:00Z'));
|
||
expect(name).toMatch(/^skill-brain-first-2026-W\d{2}\.jsonl$/);
|
||
});
|
||
});
|