mirror of
https://github.com/garrytan/gbrain.git
synced 2026-07-27 22:15:33 +00:00
* feat: add content-sanity assessor + embed-skip helper + audit JSONL primitives Four new core modules (pure, no engine I/O): - src/core/content-sanity.ts — assessor with 6 hand-vetted junk patterns (Cloudflare attention-required, just-a-moment, ray-id; access-denied; captcha-required; bare error-page titles). Bytes measured against compiled_truth + timeline (parseMarkdown body split, not file bytes). ContentSanityBlockError tagged with PAGE_JUNK_PATTERN code so classifyErrorCode hits via regex without a new ImportResult field. - src/core/content-sanity-literals.ts — operator literal-substring loader for ~/.gbrain/junk-substrings.txt. Comment directives for name + applies_to. ENOENT returns empty list (fail-soft); no regex parsing so no ReDoS surface. - src/core/embed-skip.ts — single source of truth for the embed-skip predicate. JS isEmbedSkipped() + filterOutEmbedSkipped() for in-memory callers; EMBED_SKIP_FILTER_FRAGMENT raw SQL string for engine-layer filters. buildEmbedSkipMarker() emits the canonical frontmatter shape. Both Postgres and PGLite use the same JSONB '?' existence operator. - src/core/audit/content-sanity-audit.ts — ISO-week JSONL at ~/.gbrain/audit/content-sanity-YYYY-Www.jsonl. Built on v0.40.4.0 audit-writer primitive. One stream for hard-block + soft-block + warn events with event_type discriminator. summarizeContentSanityEvents rolls up by type + source + pattern hits for doctor consumption. 99 unit tests across 4 new test files (207 assertions) covering boundaries, every built-in pattern, bytes-parity assertion, operator literals (regex meta-chars stay literal), audit JSONL round-trip + reader. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(embed): apply embed-skip filter at all 5 stale-chunk sites Embed sweep must skip pages with frontmatter.embed_skip set so soft-blocked pages don't get re-embedded. Five wiring sites all use the shared helper: 1. src/commands/embed.ts — --stale CLI path (delegates to embedAllStale) 2. src/commands/embed.ts — --all CLI path (JS-side filterOutEmbedSkipped on the listPages result; Codex r2 #11 caught this previously-missed surface that re-embedded soft-blocked pages on every model swap) 3. src/core/embed-stale.ts:90 — Minion helper (inherits via engine) 4. src/core/postgres-engine.ts — listStaleChunks + countStaleChunks gain 'NOT (COALESCE(p.frontmatter, ''{}''::jsonb) ? ''embed_skip'')' filter at the SQL layer. Always JOINs pages now (pre-fix bare path skipped the JOIN; D4 + D8 require it for the filter). 5. src/core/pglite-engine.ts — mirror of postgres-engine; PGLite is Postgres 17.5 in WASM so the same JSONB '?' operator works. Cross-site invariant pinned by test/embed-skip.test.ts (20 cases on the JS predicate + SQL fragment semantics). When v0.41+ promotes embed_skip to a schema column, all 5 sites get updated in one helper file. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(ingest): wire content-sanity gate into importFromContent narrow waist Hard-block via thrown ContentSanityBlockError; soft-block via frontmatter marker + chunk deletion on transition (D9 invariant). Single throw point means every wrapper site (CLI, MCP put_page, sync) inherits correct exit/error semantics through existing exception flow — no per-wrapper status-vocabulary changes (Codex r2 #2). import-file.ts: - Gate runs AFTER parseMarkdown so assessor sees compiled_truth + timeline + title + frontmatter (Codex r2 #5+#7). - Kill-switch (GBRAIN_NO_SANITY=1) checked via direct process.env AS WELL AS effective config — loadConfig() returns null on bare installs (no ~/.gbrain/config.json, no DATABASE_URL) so the config-only path missed the kill-switch. Caught by test/import-file-content-sanity.test.ts. - Hard-block: throws ContentSanityBlockError. Existing import.ts catch increments errors; sync.ts:929 catch records failure with classified code. - Soft-block: sets parsed.frontmatter.embed_skip via buildEmbedSkipMarker before hash compute (so hash differs from prior version → real write). Chunking block guards on isEmbedSkipped → chunks stays empty → existing tx.deleteChunks fires (D9 transition invariant). - Audit JSONL records every assessment (hard / soft / warn + bypass-mode). sync.ts: - classifyErrorCode gains /PAGE_JUNK_PATTERN/ → 'PAGE_JUNK_PATTERN' regex. No PAGE_OVERSIZED code because oversize is now a soft state — page lands. config.ts: - New content_sanity.* field on GBrainConfig (4 keys: bytes_warn, bytes_block, junk_patterns_enabled, disabled). - loadConfig() reads GBRAIN_PAGE_WARN_BYTES, GBRAIN_PAGE_BLOCK_BYTES, GBRAIN_NO_JUNK_PATTERNS, GBRAIN_NO_SANITY env vars sparse-merged. - loadConfigWithEngine merges DB-plane content_sanity.* keys per-key sparse-merge so 'gbrain config set content_sanity.bytes_block N' takes effect uniformly (Codex r2 #6 D1 acceptance). - KNOWN_CONFIG_KEYS + KNOWN_CONFIG_KEY_PREFIXES include the new keys. cli.ts: - runImport now honors result.errors > 0 for non-zero exit. Pre-fix the CLI awaited runImport but discarded the result, so hard-blocked imports exited 0 silently (Codex r2 #3). 9 PGLite-backed unit tests pin: hard-block throws, error message contains PAGE_JUNK_PATTERN, blocked page does NOT land in DB, soft-block writes page with embed_skip set, soft-block deletes pre-existing chunks (D9 transition), kill-switch bypass works. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat: lint rules + doctor checks + 'gbrain sources audit' CLI Three operator surfaces backed by the shared content-sanity assessor: lint.ts (2 new rules): - huge-page: bytes (compiled_truth + timeline post-parse) exceeds warn or block threshold. Message names the actual byte count. - scraper-junk: built-in junk pattern OR operator literal matched. - Lint runs parseMarkdown to extract body for bytes-parity with doctor (D2 — both surfaces measure body-only, not file-with-frontmatter). - runLintCore resolves effective config once per run: file/env (sync via loadConfig) + DB-lift when ~/.gbrain/ is reachable (D1). CI without ~/.gbrain/ falls through immediately. Engine probe wrapped in try/catch so lint never blocks on engine state. - Operator literals loaded once per lint run; passed through to every page's lintContent call. doctor.ts (3 new checks + 1 flag): - oversized_pages: indexed-free table scan via octet_length(compiled_truth) + octet_length(COALESCE(timeline, '')) (Codex r2 #13: octet_length is bytes, length is chars). Status warn on 1+ rows; oversize is now a soft state so no 'fail'. - scraper_junk_pages: capped 1000 most-recent default + --content-audit opt-in for full scan (D10 mirrors --index-audit precedent from v0.14.3). Applies assessor per-page on title + 2KB body slice + frontmatter. - content_sanity_audit_recent: reads ~/.gbrain/audit/content-sanity-*.jsonl for last 7 days, aggregates by event_type + source. Warn at 10+ events, fail at 100+. Doctor message names the multi-host limitation explicitly (Codex r1 #14): 'audit reflects events on this host only; multi-host operators should share GBRAIN_AUDIT_DIR'. sources.ts (new audit subcommand): - gbrain sources audit <id> [--json] [--include-warns] - Reads sources.local_path, walks disk (via pruneDir for node_modules / .git / dotfiles), runs assessContentSanity per .md file. - Reports size distribution (p50, p99, max) + would-hard-block count + would-soft-block count + junk-pattern hit map. - Read-only: NO DB writes, NO file mutations. Operator runs this BEFORE a sync to catch junk early, or AFTER landing v0.40.9.0 to audit historical inventory. 13 unit tests on lint rules; D1 config-lift behavior pinned by lift in runLintCore + manual override via opts.contentSanity for tests. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore: bump version and changelog (v0.40.9.0) v0.40.9.0 — content sanity defense: junk-pattern throw + oversize-skip-embed. Plus TODOS.md entries for the 9 deferred v0.41+ follow-ups: - chunk-level embed-quarantine (Codex r1 #3 — page-level granularity wrong) - source-repo remediation CLI (gbrain sources prune-junk) - threshold validation post-deploy on real corpora - brain-score no_junk_pages_score component - pages soft-delete --where CLI (paired with prune-junk) - post-v0.45 operator-regex extensibility (needs real ReDoS story) - post-v0.45 HTML-density rule (needs fenced-code handling) - bytes-parity E2E across lint + doctor - 5-path narrow-waist E2E pin tests + doctor integration tests Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs: update CLAUDE.md for v0.40.9.0 content-sanity wave Add v0.40.9.0 Key Files entries for the content-sanity defense modules: content-sanity.ts (assessor), content-sanity-literals.ts (operator loader), embed-skip.ts (5-site shared predicate), audit/content-sanity-audit.ts (JSONL writer). Extend doctor.ts, lint.ts, embed.ts, import-file.ts, and sources.ts entries with the v0.40.9.0 surfaces (3 new doctor checks, 2 new lint rules, embed-skip filter at 5 sites, importFromContent gate, sources audit subcommand). Regenerate llms-full.txt per the CLAUDE.md edit rule. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore: rebump v0.40.9.0 → v0.40.10.0 (queue collision with #1350) PR #1350 also claimed v0.40.9.0. Advancing this PR to v0.40.10.0 so CI's version-gate doesn't reject on overlap. No functional change — same shipped content, just a different version slot. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(brain-writer): +1ms overshoot on COUNT-race timer to defeat CI boundary flake PR #1351 ship CI hit a single test failure (one in 2552): (fail) scanBrainSources partial-scan state > hanging COUNT does not exceed deadline — Promise.race timeout fires [579.01ms] Run: https://github.com/garrytan/gbrain/actions/runs/77611667786 Cause: heavily-loaded CI runners (8 parallel shards × 4 concurrent test files = ~32 concurrent bun processes) occasionally let the setTimeout race callback resolve a microsecond BEFORE the wall-clock boundary, leaving Date.now() one tick below deadline. The post-await deadline check at brain-writer.ts:512 uses Date.now() >= deadline; on that tick the check evaluated false and scanOneSource ran src-a anyway. Test then asserted firstSource.status === 'skipped' and got 'scanned'. Fix: add 1ms overshoot to the race-timer schedule: setTimeout(..., remainingMs + 1) Guarantees the timer fires past the deadline by at least one millisecond regardless of runner timer drift. Cost: 1ms additional wall-clock latency on hung COUNT queries — operationally negligible. Verified: stress-tested 5/5 passing locally. The bug class is identical to the one the existing test comment block (lines 180-187) documents (`>=` not `>` at line 512); this +1ms is the belt to that suspenders. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
106 lines
3.9 KiB
TypeScript
106 lines
3.9 KiB
TypeScript
import { describe, test, expect } from 'bun:test';
|
|
import {
|
|
isEmbedSkipped,
|
|
filterOutEmbedSkipped,
|
|
buildEmbedSkipMarker,
|
|
EMBED_SKIP_KEY,
|
|
EMBED_SKIP_FILTER_FRAGMENT,
|
|
} from '../src/core/embed-skip.ts';
|
|
|
|
describe('isEmbedSkipped', () => {
|
|
test('false on null', () => {
|
|
expect(isEmbedSkipped(null)).toBe(false);
|
|
});
|
|
test('false on undefined', () => {
|
|
expect(isEmbedSkipped(undefined)).toBe(false);
|
|
});
|
|
test('false on empty object', () => {
|
|
expect(isEmbedSkipped({})).toBe(false);
|
|
});
|
|
test('false when key is undefined', () => {
|
|
expect(isEmbedSkipped({ other_key: true })).toBe(false);
|
|
});
|
|
test('false when key value is null', () => {
|
|
// Explicit null = "not skipped" (key existence != truthy).
|
|
expect(isEmbedSkipped({ embed_skip: null })).toBe(false);
|
|
});
|
|
test('true on full marker object (canonical write shape)', () => {
|
|
expect(isEmbedSkipped({ embed_skip: { reason: 'oversized', bytes: 100, assessed_at: 'iso' } })).toBe(true);
|
|
});
|
|
test('true on bare boolean (future flexibility)', () => {
|
|
expect(isEmbedSkipped({ embed_skip: true })).toBe(true);
|
|
});
|
|
test('true on any non-null/undefined value (key-existence semantics)', () => {
|
|
// Mirrors the SQL fragment's JSONB `?` existence operator —
|
|
// contents are diagnostic, not functional.
|
|
expect(isEmbedSkipped({ embed_skip: 'string-marker' })).toBe(true);
|
|
expect(isEmbedSkipped({ embed_skip: 0 })).toBe(true);
|
|
});
|
|
test('EMBED_SKIP_KEY constant is stable contract', () => {
|
|
expect(EMBED_SKIP_KEY).toBe('embed_skip');
|
|
});
|
|
});
|
|
|
|
describe('filterOutEmbedSkipped', () => {
|
|
test('empty array passes through', () => {
|
|
expect(filterOutEmbedSkipped([])).toEqual([]);
|
|
});
|
|
test('keeps pages without frontmatter', () => {
|
|
const pages = [{ id: 1 }, { id: 2, frontmatter: null }];
|
|
expect(filterOutEmbedSkipped(pages).length).toBe(2);
|
|
});
|
|
test('excludes pages with embed_skip set', () => {
|
|
const pages = [
|
|
{ id: 1, frontmatter: {} },
|
|
{ id: 2, frontmatter: { embed_skip: { reason: 'oversized', bytes: 100, assessed_at: '' } } },
|
|
{ id: 3, frontmatter: { other: true } },
|
|
];
|
|
const kept = filterOutEmbedSkipped(pages);
|
|
expect(kept.length).toBe(2);
|
|
expect(kept.map((p) => p.id)).toEqual([1, 3]);
|
|
});
|
|
test('preserves order of kept pages', () => {
|
|
const pages = [
|
|
{ id: 1 },
|
|
{ id: 2, frontmatter: { embed_skip: true } },
|
|
{ id: 3 },
|
|
{ id: 4, frontmatter: { embed_skip: true } },
|
|
{ id: 5 },
|
|
];
|
|
expect(filterOutEmbedSkipped(pages).map((p) => p.id)).toEqual([1, 3, 5]);
|
|
});
|
|
});
|
|
|
|
describe('buildEmbedSkipMarker', () => {
|
|
test('returns canonical marker shape', () => {
|
|
const marker = buildEmbedSkipMarker(123456);
|
|
expect(marker.reason).toBe('oversized');
|
|
expect(marker.bytes).toBe(123456);
|
|
expect(typeof marker.assessed_at).toBe('string');
|
|
expect(() => new Date(marker.assessed_at)).not.toThrow();
|
|
});
|
|
test('uses injected Date for deterministic tests', () => {
|
|
const d = new Date('2026-05-24T07:00:00Z');
|
|
const m = buildEmbedSkipMarker(100, d);
|
|
expect(m.assessed_at).toBe('2026-05-24T07:00:00.000Z');
|
|
});
|
|
});
|
|
|
|
describe('EMBED_SKIP_FILTER_FRAGMENT', () => {
|
|
test('fragment references the canonical key name', () => {
|
|
expect(EMBED_SKIP_FILTER_FRAGMENT).toContain(`'${EMBED_SKIP_KEY}'`);
|
|
});
|
|
test('fragment negates (NOT) so kept rows are without the marker', () => {
|
|
expect(EMBED_SKIP_FILTER_FRAGMENT.trim().startsWith('NOT')).toBe(true);
|
|
});
|
|
test('fragment uses JSONB `?` existence operator (works on Postgres + PGLite)', () => {
|
|
expect(EMBED_SKIP_FILTER_FRAGMENT).toContain(' ? ');
|
|
});
|
|
test('fragment COALESCEs null frontmatter so pages without one are not filtered', () => {
|
|
expect(EMBED_SKIP_FILTER_FRAGMENT).toContain('COALESCE');
|
|
});
|
|
test('fragment assumes pages alias is `p` (engine-call-site contract)', () => {
|
|
expect(EMBED_SKIP_FILTER_FRAGMENT).toContain('p.frontmatter');
|
|
});
|
|
});
|