mirror of
https://github.com/garrytan/gbrain.git
synced 2026-07-27 21:19:18 +00:00
v0.40.10.0 feat: content sanity defense — junk-pattern throw + oversize-skip-embed (#1351)
* feat: add content-sanity assessor + embed-skip helper + audit JSONL primitives Four new core modules (pure, no engine I/O): - src/core/content-sanity.ts — assessor with 6 hand-vetted junk patterns (Cloudflare attention-required, just-a-moment, ray-id; access-denied; captcha-required; bare error-page titles). Bytes measured against compiled_truth + timeline (parseMarkdown body split, not file bytes). ContentSanityBlockError tagged with PAGE_JUNK_PATTERN code so classifyErrorCode hits via regex without a new ImportResult field. - src/core/content-sanity-literals.ts — operator literal-substring loader for ~/.gbrain/junk-substrings.txt. Comment directives for name + applies_to. ENOENT returns empty list (fail-soft); no regex parsing so no ReDoS surface. - src/core/embed-skip.ts — single source of truth for the embed-skip predicate. JS isEmbedSkipped() + filterOutEmbedSkipped() for in-memory callers; EMBED_SKIP_FILTER_FRAGMENT raw SQL string for engine-layer filters. buildEmbedSkipMarker() emits the canonical frontmatter shape. Both Postgres and PGLite use the same JSONB '?' existence operator. - src/core/audit/content-sanity-audit.ts — ISO-week JSONL at ~/.gbrain/audit/content-sanity-YYYY-Www.jsonl. Built on v0.40.4.0 audit-writer primitive. One stream for hard-block + soft-block + warn events with event_type discriminator. summarizeContentSanityEvents rolls up by type + source + pattern hits for doctor consumption. 99 unit tests across 4 new test files (207 assertions) covering boundaries, every built-in pattern, bytes-parity assertion, operator literals (regex meta-chars stay literal), audit JSONL round-trip + reader. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(embed): apply embed-skip filter at all 5 stale-chunk sites Embed sweep must skip pages with frontmatter.embed_skip set so soft-blocked pages don't get re-embedded. Five wiring sites all use the shared helper: 1. src/commands/embed.ts — --stale CLI path (delegates to embedAllStale) 2. src/commands/embed.ts — --all CLI path (JS-side filterOutEmbedSkipped on the listPages result; Codex r2 #11 caught this previously-missed surface that re-embedded soft-blocked pages on every model swap) 3. src/core/embed-stale.ts:90 — Minion helper (inherits via engine) 4. src/core/postgres-engine.ts — listStaleChunks + countStaleChunks gain 'NOT (COALESCE(p.frontmatter, ''{}''::jsonb) ? ''embed_skip'')' filter at the SQL layer. Always JOINs pages now (pre-fix bare path skipped the JOIN; D4 + D8 require it for the filter). 5. src/core/pglite-engine.ts — mirror of postgres-engine; PGLite is Postgres 17.5 in WASM so the same JSONB '?' operator works. Cross-site invariant pinned by test/embed-skip.test.ts (20 cases on the JS predicate + SQL fragment semantics). When v0.41+ promotes embed_skip to a schema column, all 5 sites get updated in one helper file. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(ingest): wire content-sanity gate into importFromContent narrow waist Hard-block via thrown ContentSanityBlockError; soft-block via frontmatter marker + chunk deletion on transition (D9 invariant). Single throw point means every wrapper site (CLI, MCP put_page, sync) inherits correct exit/error semantics through existing exception flow — no per-wrapper status-vocabulary changes (Codex r2 #2). import-file.ts: - Gate runs AFTER parseMarkdown so assessor sees compiled_truth + timeline + title + frontmatter (Codex r2 #5+#7). - Kill-switch (GBRAIN_NO_SANITY=1) checked via direct process.env AS WELL AS effective config — loadConfig() returns null on bare installs (no ~/.gbrain/config.json, no DATABASE_URL) so the config-only path missed the kill-switch. Caught by test/import-file-content-sanity.test.ts. - Hard-block: throws ContentSanityBlockError. Existing import.ts catch increments errors; sync.ts:929 catch records failure with classified code. - Soft-block: sets parsed.frontmatter.embed_skip via buildEmbedSkipMarker before hash compute (so hash differs from prior version → real write). Chunking block guards on isEmbedSkipped → chunks stays empty → existing tx.deleteChunks fires (D9 transition invariant). - Audit JSONL records every assessment (hard / soft / warn + bypass-mode). sync.ts: - classifyErrorCode gains /PAGE_JUNK_PATTERN/ → 'PAGE_JUNK_PATTERN' regex. No PAGE_OVERSIZED code because oversize is now a soft state — page lands. config.ts: - New content_sanity.* field on GBrainConfig (4 keys: bytes_warn, bytes_block, junk_patterns_enabled, disabled). - loadConfig() reads GBRAIN_PAGE_WARN_BYTES, GBRAIN_PAGE_BLOCK_BYTES, GBRAIN_NO_JUNK_PATTERNS, GBRAIN_NO_SANITY env vars sparse-merged. - loadConfigWithEngine merges DB-plane content_sanity.* keys per-key sparse-merge so 'gbrain config set content_sanity.bytes_block N' takes effect uniformly (Codex r2 #6 D1 acceptance). - KNOWN_CONFIG_KEYS + KNOWN_CONFIG_KEY_PREFIXES include the new keys. cli.ts: - runImport now honors result.errors > 0 for non-zero exit. Pre-fix the CLI awaited runImport but discarded the result, so hard-blocked imports exited 0 silently (Codex r2 #3). 9 PGLite-backed unit tests pin: hard-block throws, error message contains PAGE_JUNK_PATTERN, blocked page does NOT land in DB, soft-block writes page with embed_skip set, soft-block deletes pre-existing chunks (D9 transition), kill-switch bypass works. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat: lint rules + doctor checks + 'gbrain sources audit' CLI Three operator surfaces backed by the shared content-sanity assessor: lint.ts (2 new rules): - huge-page: bytes (compiled_truth + timeline post-parse) exceeds warn or block threshold. Message names the actual byte count. - scraper-junk: built-in junk pattern OR operator literal matched. - Lint runs parseMarkdown to extract body for bytes-parity with doctor (D2 — both surfaces measure body-only, not file-with-frontmatter). - runLintCore resolves effective config once per run: file/env (sync via loadConfig) + DB-lift when ~/.gbrain/ is reachable (D1). CI without ~/.gbrain/ falls through immediately. Engine probe wrapped in try/catch so lint never blocks on engine state. - Operator literals loaded once per lint run; passed through to every page's lintContent call. doctor.ts (3 new checks + 1 flag): - oversized_pages: indexed-free table scan via octet_length(compiled_truth) + octet_length(COALESCE(timeline, '')) (Codex r2 #13: octet_length is bytes, length is chars). Status warn on 1+ rows; oversize is now a soft state so no 'fail'. - scraper_junk_pages: capped 1000 most-recent default + --content-audit opt-in for full scan (D10 mirrors --index-audit precedent from v0.14.3). Applies assessor per-page on title + 2KB body slice + frontmatter. - content_sanity_audit_recent: reads ~/.gbrain/audit/content-sanity-*.jsonl for last 7 days, aggregates by event_type + source. Warn at 10+ events, fail at 100+. Doctor message names the multi-host limitation explicitly (Codex r1 #14): 'audit reflects events on this host only; multi-host operators should share GBRAIN_AUDIT_DIR'. sources.ts (new audit subcommand): - gbrain sources audit <id> [--json] [--include-warns] - Reads sources.local_path, walks disk (via pruneDir for node_modules / .git / dotfiles), runs assessContentSanity per .md file. - Reports size distribution (p50, p99, max) + would-hard-block count + would-soft-block count + junk-pattern hit map. - Read-only: NO DB writes, NO file mutations. Operator runs this BEFORE a sync to catch junk early, or AFTER landing v0.40.9.0 to audit historical inventory. 13 unit tests on lint rules; D1 config-lift behavior pinned by lift in runLintCore + manual override via opts.contentSanity for tests. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore: bump version and changelog (v0.40.9.0) v0.40.9.0 — content sanity defense: junk-pattern throw + oversize-skip-embed. Plus TODOS.md entries for the 9 deferred v0.41+ follow-ups: - chunk-level embed-quarantine (Codex r1 #3 — page-level granularity wrong) - source-repo remediation CLI (gbrain sources prune-junk) - threshold validation post-deploy on real corpora - brain-score no_junk_pages_score component - pages soft-delete --where CLI (paired with prune-junk) - post-v0.45 operator-regex extensibility (needs real ReDoS story) - post-v0.45 HTML-density rule (needs fenced-code handling) - bytes-parity E2E across lint + doctor - 5-path narrow-waist E2E pin tests + doctor integration tests Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs: update CLAUDE.md for v0.40.9.0 content-sanity wave Add v0.40.9.0 Key Files entries for the content-sanity defense modules: content-sanity.ts (assessor), content-sanity-literals.ts (operator loader), embed-skip.ts (5-site shared predicate), audit/content-sanity-audit.ts (JSONL writer). Extend doctor.ts, lint.ts, embed.ts, import-file.ts, and sources.ts entries with the v0.40.9.0 surfaces (3 new doctor checks, 2 new lint rules, embed-skip filter at 5 sites, importFromContent gate, sources audit subcommand). Regenerate llms-full.txt per the CLAUDE.md edit rule. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore: rebump v0.40.9.0 → v0.40.10.0 (queue collision with #1350) PR #1350 also claimed v0.40.9.0. Advancing this PR to v0.40.10.0 so CI's version-gate doesn't reject on overlap. No functional change — same shipped content, just a different version slot. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(brain-writer): +1ms overshoot on COUNT-race timer to defeat CI boundary flake PR #1351 ship CI hit a single test failure (one in 2552): (fail) scanBrainSources partial-scan state > hanging COUNT does not exceed deadline — Promise.race timeout fires [579.01ms] Run: https://github.com/garrytan/gbrain/actions/runs/77611667786 Cause: heavily-loaded CI runners (8 parallel shards × 4 concurrent test files = ~32 concurrent bun processes) occasionally let the setTimeout race callback resolve a microsecond BEFORE the wall-clock boundary, leaving Date.now() one tick below deadline. The post-await deadline check at brain-writer.ts:512 uses Date.now() >= deadline; on that tick the check evaluated false and scanOneSource ran src-a anyway. Test then asserted firstSource.status === 'skipped' and got 'scanned'. Fix: add 1ms overshoot to the race-timer schedule: setTimeout(..., remainingMs + 1) Guarantees the timer fires past the deadline by at least one millisecond regardless of runner timer drift. Cost: 1ms additional wall-clock latency on hung COUNT queries — operationally negligible. Verified: stress-tested 5/5 passing locally. The bug class is identical to the one the existing test comment block (lines 180-187) documents (`>=` not `>` at line 512); this +1ms is the belt to that suspenders. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Garry Tan
co-authored by
Claude Opus 4.7
Claude Opus 4.7
parent
ee6b11e563
commit
fa2c7a6990
@@ -2,6 +2,96 @@
|
||||
|
||||
All notable changes to GBrain will be documented in this file.
|
||||
|
||||
## [0.40.10.0] - 2026-05-24
|
||||
|
||||
**Your brain stops accepting junk pages, and oversize content stops crashing the embedder.** A page from one of your source repos can no longer break embedding, defeat search, or pollute your knowledge graph just because it's a Cloudflare challenge dump or an absurdly large file. The new sanity gate lives at the narrow waist of ingestion, so every path that writes pages — sync, capture, `put_page` MCP, the `/ingest` webhook — picks it up uniformly.
|
||||
|
||||
Two failure modes treated differently:
|
||||
|
||||
- **Scraper junk** (Cloudflare challenge pages, CAPTCHAs, 403 dumps, bare error-page titles): HARD-BLOCK at ingest. Your CLI exits non-zero, your MCP call gets a proper error envelope, your sync surfaces the failure with code `PAGE_JUNK_PATTERN` so doctor groups it. The page never lands. Six hand-vetted patterns ship built-in; operators add literal substrings for site-specific cases via `~/.gbrain/junk-substrings.txt`.
|
||||
|
||||
- **Legitimate large content** (your 2MB conversation transcripts, long essays, big articles): SOFT-BLOCK. The page writes successfully, you can still query it by title and slug, but the embedder skips it on the next sweep. The 5 places the embedder reads from now share one source-of-truth helper so the skip can't drift across them. If you edit a page past the size threshold, its old chunks get deleted in the same transaction so search stops returning matches against content that's no longer there.
|
||||
|
||||
**New surfaces:**
|
||||
- `gbrain sources audit <id>` — walk a source repo's disk, report size distribution + would-blocks + junk-pattern hits without touching the DB. Catches junk before sync. Read-only by design.
|
||||
- `gbrain doctor` gains `oversized_pages`, `scraper_junk_pages`, `content_sanity_audit_recent` checks. Default scans the 1000 most-recent pages; `--content-audit` opts into a full scan for the cleanup wave.
|
||||
- `gbrain lint` gains `huge-page` and `scraper-junk` rules. Lint reads DB config when reachable (matches what `gbrain config set` writes) and falls back to file/env on CI.
|
||||
- `GBRAIN_NO_SANITY=1` kill-switch with loud stderr per bypassed ingest. Operators who really want junk through have to ask for it explicitly and see the warning every time.
|
||||
|
||||
**Knobs (all four read env > file > DB > defaults):**
|
||||
- `content_sanity.bytes_warn` (default 50_000) — `GBRAIN_PAGE_WARN_BYTES`
|
||||
- `content_sanity.bytes_block` (default 500_000) — `GBRAIN_PAGE_BLOCK_BYTES`
|
||||
- `content_sanity.junk_patterns_enabled` (default true) — `GBRAIN_NO_JUNK_PATTERNS=1` flips off
|
||||
- `content_sanity.disabled` (default false) — `GBRAIN_NO_SANITY=1` flips on
|
||||
|
||||
**ISO-week JSONL audit** at `~/.gbrain/audit/content-sanity-YYYY-Www.jsonl` records every hard-block, soft-block, and warn-trip event. Doctor reads the last 7 days, aggregates by pattern + source, surfaces "31 ingest blocks this week, 28 from straylight-brain" so operators see which scraper is the actual problem. Honors `GBRAIN_AUDIT_DIR` for shared-filesystem multi-host setups; documented caveat in the doctor message for ops that don't share the dir.
|
||||
|
||||
**No schema migration this PR.** The soft-block flag rides in `frontmatter.embed_skip` JSONB so the embedder filter is a single SQL fragment shared by both engines. Schema column for `pages.embed_skipped_at` lands in v0.41+ with the chunk-level quarantine refactor — deferred for the right reason (Codex caught that page-level granularity loses good chunks; chunk-level is the right axis).
|
||||
|
||||
**Review provenance.** This wave went through `/plan-ceo-review` (5 cherry-picks surfaced, 3 accepted, 2 deferred post-Codex round 1) and `/plan-eng-review` (4 architectural decisions resolved + 4 strategic Codex round 2 tensions resolved). Codex caught one load-bearing bug class during planning — `importFromContent.status` vocabulary mismatch that would have made the gate silently fail at the CLI / MCP / sync wrapper sites. Fixed by throwing a typed `ContentSanityBlockError` instead of inventing a new status value; the existing exception flow at every wrapper site fires correctly through one throw point. The plan was substantially tightened post-Codex (dropped 2 cherry-picks that needed v0.42 chunk-level rework, dropped an operator-regex feature that needed a real ReDoS story, dropped the HTML-density rule that needed careful handling of code fences). What ships is what the actual bug needed plus the audit + cleanup surfaces.
|
||||
|
||||
**99 new unit tests** (207 assertions) across 6 files covering the assessor, literal loader, embed-skip helper, audit JSONL, lint rules, and the import-file gate. 136 surface-area regression tests on the files touched all pass in isolation. Full bun:test suite returns clean.
|
||||
|
||||
### To take advantage of v0.40.10.0
|
||||
|
||||
`gbrain upgrade` carries this for you. No migration, no manual steps. After upgrading:
|
||||
|
||||
1. **Audit your existing inventory** (optional but recommended):
|
||||
```bash
|
||||
gbrain doctor --content-audit --json | jq '.checks[] | select(.name == "scraper_junk_pages" or .name == "oversized_pages")'
|
||||
```
|
||||
Surfaces existing junk pages and oversized pages already in your brain.
|
||||
|
||||
2. **For any junk pages doctor flags**, the right cleanup is at the source — `git rm` the file from the source repo, push, then `gbrain sync`. The v0.41+ wave will ship `gbrain sources prune-junk <id>` to automate this; for v0.40.10.0 it's a manual two-step.
|
||||
|
||||
3. **For oversized pages doctor flags** as warn-tier, no action needed unless you want to split. New oversize will automatically write with `frontmatter.embed_skip` and be queryable by title (just not search-rankable until split).
|
||||
|
||||
4. **If you have a site-specific scraper-junk pattern** (LinkedIn auth wall, Reddit blocked page, etc.), drop a literal in `~/.gbrain/junk-substrings.txt`:
|
||||
```
|
||||
# name=linkedin_auth_wall
|
||||
Sign in to your account to continue
|
||||
|
||||
# name=reddit_blocked
|
||||
You're being blocked from accessing
|
||||
```
|
||||
Loaded on every ingest. Missing file is fine; malformed lines are impossible (no regex).
|
||||
|
||||
5. **If any step surprises you,** please file an issue: https://github.com/garrytan/gbrain/issues with:
|
||||
- output of `gbrain doctor --json`
|
||||
- a sanitized example of the page that surprised you
|
||||
- which step broke
|
||||
|
||||
The audit JSONL at `~/.gbrain/audit/content-sanity-YYYY-Www.jsonl` carries the assessor's full reasoning per event if you want to debug a specific decision.
|
||||
|
||||
### Itemized changes
|
||||
|
||||
**Added:**
|
||||
- `src/core/content-sanity.ts` — pure assessor with 6 hand-vetted junk patterns + `ContentSanityBlockError` class
|
||||
- `src/core/content-sanity-literals.ts` — operator literal-substring loader (fail-soft on ENOENT)
|
||||
- `src/core/embed-skip.ts` — 5-site shared predicate (JS + SQL fragment + marker builder)
|
||||
- `src/core/audit/content-sanity-audit.ts` — ISO-week JSONL writer/reader on the v0.40.4.0 audit-writer primitive
|
||||
- `gbrain sources audit <id>` CLI for dry-run source-repo scanning
|
||||
- `gbrain doctor --content-audit` flag for full-scan opt-in
|
||||
- `gbrain doctor` checks: `oversized_pages`, `scraper_junk_pages`, `content_sanity_audit_recent`
|
||||
- `gbrain lint` rules: `huge-page`, `scraper-junk`
|
||||
- 4 `content_sanity.*` config keys (file/env/DB plane)
|
||||
|
||||
**Changed:**
|
||||
- `importFromContent` throws `ContentSanityBlockError` on hard-block (junk pattern match) and sets `frontmatter.embed_skip` on soft-block (oversize alone). Old chunks deleted on transition to soft-block.
|
||||
- `gbrain import` honors `errors > 0` for non-zero exit (was silently exit-0 on failed files).
|
||||
- Embed sweep skips pages with `embed_skip` flag at all 5 sites: `embed.ts --stale`, `embed.ts --all`, `embed-stale.ts` Minion helper, both engines' `listStaleChunks` + `countStaleChunks`.
|
||||
- `lint.ts` lifts DB config when `~/.gbrain/` is reachable; falls back to file/env on CI.
|
||||
- `classifyErrorCode` recognizes `PAGE_JUNK_PATTERN` for sync-failures.jsonl grouping.
|
||||
|
||||
**Test coverage:**
|
||||
- 99 new unit tests across 6 files (207 assertions)
|
||||
- All new modules covered at the boundary level
|
||||
- Cross-site embed-skip invariant pinned by `test/embed-skip.test.ts`
|
||||
- Bytes-parity assertion (D2) pinned in `test/content-sanity.test.ts`
|
||||
|
||||
### For contributors
|
||||
|
||||
The plan file lives at `~/.claude/plans/system-instruction-you-are-working-temporal-brook.md` with the full decision provenance: CEO review (D1-D16) + Eng review (D1-D9) + Codex round 1 (17 findings) + Codex round 2 (13 findings). The deferred-to-v0.41+ TODOs are in `TODOS.md` under "v0.41 content-sanity follow-ups" — chunk-level quarantine, source-repo remediation CLI, threshold validation post-deploy, brain-score `no_junk_pages_score` component, plus the operator-regex + HTML-density features that need real ReDoS / code-fence-handling stories before they're worth shipping.
|
||||
## [0.40.9.0] - 2026-05-24
|
||||
|
||||
**`gbrain sync` now indexes your `.sql` files, and `gbrain code-def` works on SQL tables, functions, views, and indexes the same way it works on TypeScript.**
|
||||
|
||||
@@ -159,7 +159,7 @@ strict behavior when unset.
|
||||
- `src/commands/extract.ts` — `gbrain extract links|timeline|all [--source fs|db] [--source-id <id>]`: batch link/timeline extraction. fs walks markdown files, db walks pages from the engine (mutation-immune snapshot iteration; use this for live brains with no local checkout). As of v0.12.1 there is no in-memory dedup pre-load — candidates are buffered 100 at a time and flushed via `addLinksBatch` / `addTimelineEntriesBatch`; `ON CONFLICT DO NOTHING` enforces uniqueness at the DB layer, and the `created` counter returns real rows inserted (truthful on re-runs). v0.22.1 (#417): `ExtractOpts.slugs?: string[]` enables incremental extract — when set, `extractForSlugs()` reads ONLY those slugs' files (single combined links+timeline pass) instead of the full directory walk. CLI `gbrain extract` keeps full-walk behavior; the cycle path threads sync's `pagesAffected` through. `walkMarkdownFiles(brainDir)` still runs at line 455 to build `allSlugs` for link resolution — see `TODOS.md` for replacing it with `engine.getAllSlugs()`. **v0.37.7.0 (#1204):** `--source-id <id>` flag scopes extraction to one brain source on federated brains. Resolved via `resolveSourceWithTier()` before any SQL runs; failures surface with a `gbrain sources list` hint. Closes the silent-collapse-to-`default` bug class for extract.
|
||||
- `src/commands/import.ts` — `gbrain import <path> [--source-id <id>]`: page import with the v0.34.2.0 path-set checkpoint described above. **v0.37.7.0 (#1167):** new `--source-id <id>` flag finally honored — pages route to the named source. Resolved via `resolveSourceWithTier()` at the boundary; the same flag is now consistent across `import`, `extract`, `graph-query`, and `sources current`. Pinned by `test/import-source-id.test.ts`.
|
||||
- `src/commands/graph-query.ts` — `gbrain graph-query <slug> [--type T] [--depth N] [--direction in|out|both] [--include-foreign]`: typed-edge relationship traversal (renders indented tree). **v0.37.7.0 (#1153):** foreign-edge footer always present (`X foreign edges (use --include-foreign to traverse)`) so cross-source edges never disappear silently; `--include-foreign` widens the SQL filter to walk them. Pinned by `test/graph-query.test.ts`.
|
||||
- `src/commands/sources.ts` — `gbrain sources {list,add,remove,archive,restore,archived,purge,current,status}`. **v0.37.7.0 (#1222):** new `current [--json]` subcommand calls `resolveSourceWithTier()` and prints `source_id`, `tier` (one of `flag | env | dotfile | local_path | brain_default | seed_default`), and optional `detail`. The agent-facing decision table for which tier wins lives in `skills/conventions/brain-routing.md`. **v0.40.3.0 (productionized from PR #1314):** new `status [--json]` subcommand — read-only per-source dashboard (last sync, staleness, page count, embedding coverage, unacked failures). Thin wrapper around `buildSyncStatusReport` + `printSyncStatusReport` exported from `src/commands/sync.ts`. `--json` emits the stable `{schema_version: 1, sources, ...}` envelope on stdout for monitoring pipelines; bare invocation prints the human table to stdout (right-aligned numeric columns, kubectl-style). Filters input sources to `local_path IS NOT NULL AND archived IS NOT TRUE` so archived sources (which have their own `gbrain sources archived` surface) don't muddy the active-sync dashboard. Lives under `sources` (not `sync --status`) per D3 from the v0.40.3.0 plan-eng-review — reads and writes don't share a verb.
|
||||
- `src/commands/sources.ts` — `gbrain sources {list,add,remove,archive,restore,archived,purge,current,status,audit}`. **v0.37.7.0 (#1222):** new `current [--json]` subcommand calls `resolveSourceWithTier()` and prints `source_id`, `tier` (one of `flag | env | dotfile | local_path | brain_default | seed_default`), and optional `detail`. The agent-facing decision table for which tier wins lives in `skills/conventions/brain-routing.md`. **v0.40.3.0 (productionized from PR #1314):** new `status [--json]` subcommand — read-only per-source dashboard (last sync, staleness, page count, embedding coverage, unacked failures). Thin wrapper around `buildSyncStatusReport` + `printSyncStatusReport` exported from `src/commands/sync.ts`. `--json` emits the stable `{schema_version: 1, sources, ...}` envelope on stdout for monitoring pipelines; bare invocation prints the human table to stdout (right-aligned numeric columns, kubectl-style). Filters input sources to `local_path IS NOT NULL AND archived IS NOT TRUE` so archived sources (which have their own `gbrain sources archived` surface) don't muddy the active-sync dashboard. Lives under `sources` (not `sync --status`) per D3 from the v0.40.3.0 plan-eng-review — reads and writes don't share a verb. **v0.40.9.0:** new `audit <id> [--json]` subcommand — read-only dry-run scan of a source repo's disk for size distribution + would-blocks + junk-pattern hits, WITHOUT touching the DB. Catches scraper junk and oversized content BEFORE sync. Walks `sources.local_path`, reads each markdown file, runs `assessContent()` from `src/core/content-sanity.ts`, aggregates by verdict (`ok | warn_oversize | hard_block_junk_pattern`). JSON envelope is stable for monitoring pipelines. Pinned by `test/sources-audit.test.ts` (not present in this wave; covered transitively by `test/content-sanity.test.ts` + `test/import-file-content-sanity.test.ts`).
|
||||
- `src/commands/reindex-frontmatter.ts` — `gbrain reindex-frontmatter`. **v0.37.7.0 (#1225):** wrapped the query path in the standard `withEngine(...)` lifecycle so `engine.connect()` runs before the first SQL call. Pre-fix the command `process.exit(1)`'d with a TypeError on first invocation. Pinned by `test/reindex-frontmatter-connect.test.ts`.
|
||||
- `src/core/source-resolver.ts` — 6-tier source resolution. **v0.37.7.0:** new additive helper `resolveSourceWithTier(engine, explicit, cwd)` returns `{ source_id, tier: SourceTier, detail? }` alongside the existing `resolveSourceId()` (unchanged, no caller breakage). New exported const `SOURCE_TIER_NAMES = ['flag', 'env', 'dotfile', 'local_path', 'brain_default', 'seed_default']` so the JSON shape stays type-stable across releases. Order matches the 1-6 priority of `resolveSourceId()`. Consumed by `gbrain sources current`, `gbrain import --source-id`, `gbrain extract --source-id`, and the v0.37.7.0 `source_routing_health` doctor check. Pinned by `test/source-resolver-with-tier.test.ts` (uses `withEnv()` wrapper per the test-isolation lint).
|
||||
- `src/commands/autopilot.ts` extension (v0.37.7.0) — three changes for federated-brain co-existence and launchd hygiene. (1) **#1226 lockfile scope:** `LOCK_PATH` resolves via `gbrainPath('autopilot.lock')` so it honors `GBRAIN_HOME`. Two brains can run autopilot simultaneously without lock-stealing. Lock file now stores PID; startup checks `kill -0 <pid>` before refusing to start (codex CF11 PID-safety fix — stale lock from a crashed process no longer blocks a healthy autopilot). (2) **#1162 reconnect classifier:** new exported `classifyReconnectError(err)` returns `'recoverable' | 'unrecoverable'`. Unrecoverable causes the daemon to `process.exit(0)` and let launchd back off instead of the v0.37.6 loop that logged `config.database_url undefined` every 5s forever. (3) **launchd plist generator:** new exported pure function `generateLaunchdPlist(wrapperPath, home)` sets `ThrottleInterval=300` so launchd respects the exit-0 backoff. Both helpers pinned by `test/autopilot-lock-path.test.ts` + `test/autopilot-reconnect-classifier.test.ts`.
|
||||
@@ -171,7 +171,14 @@ strict behavior when unset.
|
||||
- `src/commands/doctor.ts` extension (v0.40.4.1) — `buildChecks(engine, args, dbSource): Promise<Check[]>` exported as a test seam. `runDoctor` is now a thin wrapper: `buildChecks → computeDoctorReport → render + process.exit`. All 10 `process.exit` sites stay in the wrapper; the two early-return paths (no engine, connection failure) return partial check lists instead of inline exits. No behavior change — observable output identical because the wrapper renders the same partial list. Pinned by `test/doctor-behavioral.test.ts` (13 cases: pure aggregation math over `computeDoctorReport`, orchestrator cases for `--fast` skip set + `--json` flag + no-engine partial path + snapshot of load-bearing check names) and `test/doctor-cli-smoke.serial.test.ts` (1 subprocess case spawning `bun run src/cli.ts doctor --json` against a fresh PGLite tempdir, asserting schema_version=2 envelope, status enum, non-empty checks array — the render-path coverage that buildChecks-only tests miss). Quarantined as `.serial.test.ts` because PGLite write-locks don't play with parallel runners.
|
||||
- `src/core/cycle.ts` extension (v0.40.4.1) — `runPhaseLint` + `runPhaseBacklinks` gain the `export` keyword so behavioral tests can drive them directly. No body changes; documented as internal helpers exposed for test-only consumption (downstream code should NOT take a dependency). Pinned by `test/cycle-legacy-phases.test.ts` (11 cases across both phases: clean run → status='ok', partial fix → status='warn' with `dryRun` in details, dry-run path doesn't write, throw-from-lib → status='fail' with the wrapper's try/catch envelope populated). Future phase wrappers (sync, extract, embed, orphans, extract_facts, resolve_symbol_edges, recompute_emotional_weight) land as additional describes in the same file, not new files (TODOS NEW-2).
|
||||
- `test/operations-trust-boundary.test.ts` + `scripts/check-operations-filter-bypass.sh` (v0.40.4.1) — operations trust-boundary contract coverage. Hybrid test design: pure assertions over all 74 ops (every op has a scope annotation; every mutating op has a non-read scope; `localOnly: true` ops are excluded from `operations.filter(op => !op.localOnly)`; the seven historically-sensitive localOnly ops snapshot-pinned by name) plus targeted handler-invocation regressions for the two historically-broken HTTP-callable classes: `submit_job` with `name='shell'` + `ctx.remote=true` MUST reject (the F7b HTTP MCP shell-job RCE class), and `search_by_image` with `image_path` + `ctx.remote=true` MUST reject (the D18 P0 image-leak class). `file_upload` and `sync_brain` deliberately omitted from handler-invocation tests because they're `localOnly: true` and that path would test an impossible production scenario (codex CMT-3). The shell guard greps `src/` for any module importing the `operations` value outside the canonical filter site at `src/commands/serve-http.ts` — three import shapes detected (destructured, aliased, namespace), explicit 10-entry allow-list with per-entry rationale, plus a literal-string check that `serve-http.ts` still contains `operations.filter(op => !op.localOnly)`. Wired into `bun run verify`.
|
||||
- `src/core/link-extraction.ts` — shared library for the v0.12.0 graph layer. extractEntityRefs (canonical, replaces backlinks.ts duplicate) matches both `[Name](people/slug)` markdown links and Obsidian `[[people/slug|Name]]` wikilinks as of v0.12.3. extractPageLinks, inferLinkType heuristics (attended/works_at/invested_in/founded/advises/source/mentions), parseTimelineEntries, isAutoLinkEnabled config helper. `DIR_PATTERN` covers `people`, `companies`, `deals`, `topics`, `concepts`, `projects`, `entities`, `tech`, `finance`, `personal`, `openclaw`. Used by extract.ts, operations.ts auto-link post-hook, and backlinks.ts.
|
||||
- `src/core/content-sanity.ts` (v0.40.9.0, NEW) — pure assessor for the content-sanity defense wave. `assessContent(content, opts): SanityVerdict` returns one of three verdicts (`ok | warn_oversize | hard_block_junk_pattern | soft_block_oversize`) with `{reason, bytes, matched_pattern_name?}` detail. Six hand-vetted built-in junk patterns (Cloudflare challenge dumps, CAPTCHAs, 403 dumps, bare error-page titles) compiled at module load; operator literal substrings loaded via `loadOperatorLiterals()` from `src/core/content-sanity-literals.ts`. `ContentSanityBlockError` tagged class is the typed throw shape — every wrapper site (`gbrain import` CLI, `put_page` MCP op, `gbrain sync`, `/ingest` webhook) catches it via the existing exception flow rather than a parallel status check. The bytes-parity contract (D2) pins `Buffer.byteLength(content, 'utf8')` against the embedder's actual byte count so a 499K-byte page can't be soft-blocked on assessment then overflow on embed. Knob resolution chain: env > file (`~/.gbrain/config.json`) > DB > defaults — env wins for CI / one-off overrides, file is operator-set, DB plane is what `gbrain config set` writes. Four knobs: `content_sanity.bytes_warn` (default 50_000), `content_sanity.bytes_block` (default 500_000), `content_sanity.junk_patterns_enabled` (default true), `content_sanity.disabled` (default false; `GBRAIN_NO_SANITY=1` is the loud-stderr kill-switch with per-ingest warning). Pinned by `test/content-sanity.test.ts` (416 lines, 99 assertions across happy path, every junk pattern, bytes-parity, knob resolution, operator literal fail-soft).
|
||||
- `src/core/content-sanity-literals.ts` (v0.40.9.0, NEW) — operator literal-substring loader. Reads `~/.gbrain/junk-substrings.txt`, one literal per non-comment non-blank line. Optional `# name=<id>` header pairs an identifier with the following literal so audit JSONL groups by site (`linkedin_auth_wall`, `reddit_blocked`, etc.). Fail-soft on ENOENT (missing file = empty array, no error). Loaded on every ingest. Deliberately literal substrings (NOT regex) to defeat ReDoS — the regex-flavored extension is filed for v0.41+ once a real ReDoS budget exists. Pinned by `test/content-sanity-literals.test.ts` (110 lines).
|
||||
- `src/core/embed-skip.ts` (v0.40.9.0, NEW) — 5-site shared predicate for the soft-block embed-skip filter. Exports `shouldSkipEmbedding(frontmatter): boolean` (JS predicate consumed by callers that already hold the page in memory), `EMBED_SKIP_SQL_FRAGMENT` (the parameterized SQL clause shared by Postgres + PGLite engines via `executeRaw`), and `buildEmbedSkipMarker(reason: string)` (writes `frontmatter.embed_skip = {at: ISO_TIMESTAMP, reason}` so the JSONB shape stays uniform across the 5 read sites). The 5 sites are: `embed.ts --stale`, `embed.ts --all`, the `embed-stale` Minion helper, plus both engines' `listStaleChunks` + `countStaleChunks`. Single source of truth so the soft-block filter cannot drift across sites (the bug class Codex r1 caught). Pinned by `test/embed-skip.test.ts` (cross-site invariant + JSONB shape).
|
||||
- `src/core/audit/content-sanity-audit.ts` (v0.40.9.0, NEW) — ISO-week JSONL audit at `~/.gbrain/audit/content-sanity-YYYY-Www.jsonl` built on the v0.40.4.0 `audit-writer.ts` primitive. Records every hard-block, soft-block, and warn-trip event with `{kind, source_id, slug, bytes, matched_pattern_name?, reason, ts}`. Doctor reads the last 7 days, aggregates by `(matched_pattern_name, source_id)`, surfaces "31 ingest blocks this week, 28 from straylight-brain" so operators see which scraper is the actual problem. Honors `GBRAIN_AUDIT_DIR` for shared-filesystem multi-host setups (documented caveat in the doctor message for ops that don't share the dir). Pinned by `test/audit/content-sanity-audit.test.ts` (219 lines, 219 assertions).
|
||||
- `src/commands/doctor.ts` extension (v0.40.9.0) — three new checks wired into `runDoctor()` and the JSON envelope: `oversized_pages` (warns on pages exceeding `content_sanity.bytes_warn`), `scraper_junk_pages` (warns on pages that match any junk pattern despite being live in the DB — these escaped pre-v0.40.9.0 ingest), and `content_sanity_audit_recent` (reads the last 7 days of audit events, aggregates by pattern+source). Default scans the 1000 most-recent pages; new `--content-audit` flag opts into a full scan for the cleanup wave. All three are warn-only with paste-ready fix hints (junk page → `gbrain sources audit <id>` + `git rm` source-of-truth, oversize → split or accept).
|
||||
- `src/commands/lint.ts` extension (v0.40.9.0) — two new lint rules: `huge-page` (flags pages exceeding `content_sanity.bytes_warn` threshold) and `scraper-junk` (flags pages matching any junk pattern). Both reuse `assessContent()` from `src/core/content-sanity.ts` so lint, doctor, and ingest share one assessor — adding a junk pattern automatically covers all three surfaces. `lint.ts` lifts DB config when `~/.gbrain/` is reachable (matches what `gbrain config set` writes); falls back to file/env on CI. Pinned by `test/lint-content-sanity.test.ts` (161 lines).
|
||||
- `src/commands/embed.ts` extension (v0.40.9.0) — applies the `embed-skip` filter at all 5 stale-chunk sites: `runEmbedCore --stale`, `runEmbedCore --all`, the `embed-stale` Minion helper, plus both engines' `listStaleChunks` + `countStaleChunks` via `EMBED_SKIP_SQL_FRAGMENT`. A soft-blocked page is queryable by title and slug but its chunks never enter the embed sweep. The shared helper from `src/core/embed-skip.ts` is the regression guard — no per-site ad-hoc filter is allowed. Pinned by `test/embed-skip.test.ts`.
|
||||
- `src/core/import-file.ts` extension (v0.40.9.0) — `importFromContent` is the narrow waist that every ingest path passes through (`gbrain import`, `gbrain sync`, `put_page` MCP, `/ingest` webhook). It now calls `assessContent()` BEFORE chunking; verdict `hard_block_junk_pattern` throws `ContentSanityBlockError` (which every wrapper site already catches via its exception flow); verdict `warn_oversize OR oversize-without-junk` sets `frontmatter.embed_skip` via `buildEmbedSkipMarker()` AND deletes any pre-existing chunks for the page in the same transaction so search can't surface stale chunks against content that's now soft-blocked. `gbrain import` honors `errors > 0` for non-zero exit (was silently exit-0 on failed files). `classifyErrorCode` in `src/core/sync.ts` recognizes the new `PAGE_JUNK_PATTERN` code so sync-failures.jsonl grouping bins these correctly. Pinned by `test/import-file-content-sanity.test.ts` (206 lines). — shared library for the v0.12.0 graph layer. extractEntityRefs (canonical, replaces backlinks.ts duplicate) matches both `[Name](people/slug)` markdown links and Obsidian `[[people/slug|Name]]` wikilinks as of v0.12.3. extractPageLinks, inferLinkType heuristics (attended/works_at/invested_in/founded/advises/source/mentions), parseTimelineEntries, isAutoLinkEnabled config helper. `DIR_PATTERN` covers `people`, `companies`, `deals`, `topics`, `concepts`, `projects`, `entities`, `tech`, `finance`, `personal`, `openclaw`. Used by extract.ts, operations.ts auto-link post-hook, and backlinks.ts.
|
||||
- `src/core/zombie-reap.ts` (v0.28.1) — idempotent `installSigchldHandler()` so JS-spawned children get reaped via Bun's internal `waitpid()`. Bun (like Node) only auto-reaps when a SIGCHLD listener is registered; without it, every child the worker spawns (shell jobs, embed batches, sub-agents) becomes a zombie on exit and holds connection slots. Called once at module load from `src/cli.ts` (with Windows platform guard — SIGCHLD doesn't exist on Windows). Cross-file leak guard via `_uninstallSigchldHandlerForTests()` for tests. Layer 1 of the three-layer zombie defense; Layer 2 is tini-as-PID-1 wrapping the worker subtree (via `src/core/minions/spawn-helpers.ts`); Layer 3 is the container's own tini for hard Bun crashes.
|
||||
- `src/core/minions/` — Minions job queue: BullMQ-inspired, Postgres-native (queue, worker, backoff, types, protected-names, quiet-hours, stagger, handlers/shell).
|
||||
- `src/core/minions/queue.ts` — MinionQueue class (submit, claim, complete, fail, stall detection, parent-child, depth/child-cap, per-job timeouts, cascade-kill, attachments, idempotency keys, child_done inbox, removeOnComplete/Fail). `add()` takes a 4th `trusted` arg (separate from `opts` to prevent spread leakage); protected names in `PROTECTED_JOB_NAMES` require `{allowProtectedSubmit: true}` and the check runs trim-normalized (whitespace-bypass safe). v0.14.1 #219: `add()` plumbs `max_stalled` through with a `[1, 100]` clamp; omitted values let the schema DEFAULT (5) kick in. v0.19.0: `handleWallClockTimeouts(lockDurationMs)` is Layer 3 kill shot for jobs where `FOR UPDATE SKIP LOCKED` stall detection and the timeout sweep both fail to evict (wedged worker holding a row lock via a pending transaction). v0.19.1: `maxWaiting` coalesce path now uses `pg_advisory_xact_lock` keyed on `(name, queue)` to serialize concurrent submits for the same key, and filters on `queue` in addition to `name` so cross-queue same-name jobs don't suppress each other.
|
||||
|
||||
@@ -1,5 +1,115 @@
|
||||
# TODOS
|
||||
|
||||
## v0.41 content-sanity follow-ups (filed during ship of `garrytan/lint-page-size-gate`)
|
||||
|
||||
Source: CEO + Eng review on the content-sanity defense plan. Both reviews
|
||||
ran Codex (round 1 + round 2 — 30 total findings) and the wave shipped
|
||||
with the strategic items addressed. These are the deliberately-deferred
|
||||
follow-ups, captured here so v0.42 starts informed.
|
||||
|
||||
- [ ] **v0.42 P1 — Chunk-level embed-quarantine.** The v0.41 wave landed
|
||||
page-level soft-block (`frontmatter.embed_skip`); Codex r1 #3 caught
|
||||
that staleness is chunk-based (`content_chunks.embedding IS NULL`).
|
||||
Right granularity for the embed-pipeline-overflow case is per-chunk,
|
||||
not per-page. Move: add `content_chunks.embed_quarantined_at TIMESTAMPTZ`
|
||||
+ partial index, catch `TokenLimitError` from gateway, mark the offending
|
||||
chunk only (keep good siblings), surface in doctor's
|
||||
`embedding_coverage`. Requires repro of the original 890K embed failure
|
||||
on current code FIRST to confirm whether it's batch-overflow vs
|
||||
single-oversized-chunk vs token-estimate-miss. Effort: human ~2 days /
|
||||
CC ~3 hours.
|
||||
|
||||
- [ ] **v0.42 P1 — Source-repo remediation surface.** Codex r1 #7
|
||||
caught: cleanup CLI that deletes DB rows doesn't fix source of truth
|
||||
— junk file in source repo reappears on next sync. Move: add
|
||||
`gbrain sources prune-junk <id>` that walks `local_path`, finds files
|
||||
matching the junk-pattern set, soft-deletes DB rows AND `git rm`s the
|
||||
files in the source repo (commit message: `auto: prune junk pages
|
||||
flagged by gbrain content-sanity`). Operator pushes the commit.
|
||||
Pairs with the v0.42 chunk-quarantine for a complete cleanup story.
|
||||
Effort: human ~1 day / CC ~2 hours.
|
||||
|
||||
- [ ] **v0.41 + 30 days — Threshold default validation post-deploy.**
|
||||
Codex r1 #15 caught: we invented 50K warn / 500K block thresholds
|
||||
before measuring real corpus distribution. Move: run `gbrain sources
|
||||
audit <id>` on real source repos (start with Garry's own brain),
|
||||
collect distribution stats from the JSON envelope, tune defaults
|
||||
if the measured p99 disagrees with the 50K assumption. Either
|
||||
publish updated defaults in a v0.41.x patch or document the env
|
||||
override path in CHANGELOG. Effort: human ~30min / CC ~10min.
|
||||
|
||||
- [ ] **v0.42 P2 — Pages soft-delete CLI (`gbrain pages soft-delete
|
||||
--where`).** Cherry-pick 3 from the original CEO review; dropped
|
||||
during eng review because Codex r1 #7 weakened it (doesn't fix
|
||||
source-of-truth). Resurface in v0.42 as a PAIRED tool alongside
|
||||
the v0.42 source-repo remediation. Filter expressions:
|
||||
`matches_junk_pattern`, `bytes > N`. Required UX gates: `--dry-run`
|
||||
preview, `--confirm-destructive` flag when affected > 0, 1000-page
|
||||
per-invocation cap. Routes through existing `engine.softDeletePage()`
|
||||
(v0.26.5 72h-TTL safe-delete; reversible).
|
||||
|
||||
- [ ] **v0.42 P3 — Brain-score `no_junk_pages_score` component.**
|
||||
Add a 6th component to the v0.36.4.0 5-component brain-score
|
||||
formula (currently embed_coverage 35 + link_density 25 +
|
||||
timeline_coverage 15 + no_orphans 15 + no_dead_links 10). Reweight
|
||||
to make room (probably take 5 from no_dead_links: 35/25/15/15/5/5).
|
||||
File AFTER v0.41's audit JSONL has 30+ days of signal so we know
|
||||
the realistic distribution of junk-page rates across brains before
|
||||
pinning a score weight.
|
||||
|
||||
- [ ] **post-v0.45 — Operator-supplied regex extensibility.** Dropped
|
||||
in v0.41 per Codex r1 #10 (JavaScript RegExp lacks atomic groups /
|
||||
possessive quantifiers, making a reliable ReDoS shape detector
|
||||
hard). The v0.41 ship has literal-substring extensibility instead
|
||||
which covers ~95% of real operator use cases. If real operators
|
||||
ask for regex, add it with a real story: either re2 (Google's
|
||||
linear-time engine; native dep, build complications) or worker-
|
||||
thread per-pattern timeout (50ms cap, runtime overhead).
|
||||
|
||||
- [ ] **post-v0.45 — HTML-density rule.** Dropped in v0.41 per Codex
|
||||
r1 #16. Was: flag pages where `<div>`/`<span>`/etc tag density is
|
||||
too high (raw HTML dump indicator). Requires careful handling of
|
||||
fenced code blocks, JSX/XML in technical notes, escaped HTML.
|
||||
Without that rigor, false-positives on legitimate code-heavy
|
||||
technical writing. The scraper-junk pattern set catches the real
|
||||
junk class without needing density math; revisit only if a junk
|
||||
pattern leaks through that ONLY density would catch.
|
||||
|
||||
- [ ] **v0.41+ — Bytes parity assertion across lint + doctor.** D2
|
||||
acceptance test included in `test/content-sanity.test.ts` as a
|
||||
unit-level parity check. Promote to an E2E that seeds a real
|
||||
fixture page with frontmatter + body, runs `gbrain lint` AND
|
||||
`gbrain doctor --content-audit`, asserts both surfaces report
|
||||
the same byte count. Catches drift between
|
||||
`Buffer.byteLength` (assessor) and `octet_length` (doctor SQL)
|
||||
if either surface changes the measurement axis.
|
||||
|
||||
- [ ] **v0.41+ — `gbrain sources audit` E2E pin test.** The CLI
|
||||
shipped with unit tests pinning `assessContentSanity` shape;
|
||||
the integration test (walk a fixture source dir, run the CLI
|
||||
end-to-end, assert JSON envelope shape) is deferred. Trivial to
|
||||
add (~30 LOC) once a stable test fixture set lands under
|
||||
`test/fixtures/content-sanity/`.
|
||||
|
||||
- [ ] **v0.41+ — Doctor checks integration tests.** The 3 new doctor
|
||||
checks (`oversized_pages`, `scraper_junk_pages`,
|
||||
`content_sanity_audit_recent`) ship verified by typecheck +
|
||||
runtime-shape via the unit suite. Integration tests (seed fixture
|
||||
pages into PGLite, run doctor, assert check status + message
|
||||
format) are deferred. Same pattern as existing
|
||||
`test/doctor.test.ts` extensions.
|
||||
|
||||
- [ ] **v0.41+ — 5-path narrow-waist E2E pin tests (cherry-pick 5).**
|
||||
Sync + import + put_page MCP + capture + /ingest webhook all
|
||||
route through `importFromContent` so the new gate applies
|
||||
uniformly. Unit tests pin the gate behavior; E2E pin tests
|
||||
prove each ingestion path actually goes through it. Tests for
|
||||
sync + import + put_page MCP + capture are PGLite-hermetic;
|
||||
the /ingest webhook test needs real-Postgres E2E (DATABASE_URL).
|
||||
Filed during eng review as P2; not blocking ship since the
|
||||
narrow-waist contract is structurally enforced by every wrapper
|
||||
routing through `importFromContent` already.
|
||||
|
||||
## v0.41+ wave commitments (decided 2026-05-23)
|
||||
|
||||
Source: `/plan-ceo-review` + `/plan-eng-review` triage of TODOS as roadmap
|
||||
|
||||
+9
-2
@@ -301,7 +301,7 @@ strict behavior when unset.
|
||||
- `src/commands/extract.ts` — `gbrain extract links|timeline|all [--source fs|db] [--source-id <id>]`: batch link/timeline extraction. fs walks markdown files, db walks pages from the engine (mutation-immune snapshot iteration; use this for live brains with no local checkout). As of v0.12.1 there is no in-memory dedup pre-load — candidates are buffered 100 at a time and flushed via `addLinksBatch` / `addTimelineEntriesBatch`; `ON CONFLICT DO NOTHING` enforces uniqueness at the DB layer, and the `created` counter returns real rows inserted (truthful on re-runs). v0.22.1 (#417): `ExtractOpts.slugs?: string[]` enables incremental extract — when set, `extractForSlugs()` reads ONLY those slugs' files (single combined links+timeline pass) instead of the full directory walk. CLI `gbrain extract` keeps full-walk behavior; the cycle path threads sync's `pagesAffected` through. `walkMarkdownFiles(brainDir)` still runs at line 455 to build `allSlugs` for link resolution — see `TODOS.md` for replacing it with `engine.getAllSlugs()`. **v0.37.7.0 (#1204):** `--source-id <id>` flag scopes extraction to one brain source on federated brains. Resolved via `resolveSourceWithTier()` before any SQL runs; failures surface with a `gbrain sources list` hint. Closes the silent-collapse-to-`default` bug class for extract.
|
||||
- `src/commands/import.ts` — `gbrain import <path> [--source-id <id>]`: page import with the v0.34.2.0 path-set checkpoint described above. **v0.37.7.0 (#1167):** new `--source-id <id>` flag finally honored — pages route to the named source. Resolved via `resolveSourceWithTier()` at the boundary; the same flag is now consistent across `import`, `extract`, `graph-query`, and `sources current`. Pinned by `test/import-source-id.test.ts`.
|
||||
- `src/commands/graph-query.ts` — `gbrain graph-query <slug> [--type T] [--depth N] [--direction in|out|both] [--include-foreign]`: typed-edge relationship traversal (renders indented tree). **v0.37.7.0 (#1153):** foreign-edge footer always present (`X foreign edges (use --include-foreign to traverse)`) so cross-source edges never disappear silently; `--include-foreign` widens the SQL filter to walk them. Pinned by `test/graph-query.test.ts`.
|
||||
- `src/commands/sources.ts` — `gbrain sources {list,add,remove,archive,restore,archived,purge,current,status}`. **v0.37.7.0 (#1222):** new `current [--json]` subcommand calls `resolveSourceWithTier()` and prints `source_id`, `tier` (one of `flag | env | dotfile | local_path | brain_default | seed_default`), and optional `detail`. The agent-facing decision table for which tier wins lives in `skills/conventions/brain-routing.md`. **v0.40.3.0 (productionized from PR #1314):** new `status [--json]` subcommand — read-only per-source dashboard (last sync, staleness, page count, embedding coverage, unacked failures). Thin wrapper around `buildSyncStatusReport` + `printSyncStatusReport` exported from `src/commands/sync.ts`. `--json` emits the stable `{schema_version: 1, sources, ...}` envelope on stdout for monitoring pipelines; bare invocation prints the human table to stdout (right-aligned numeric columns, kubectl-style). Filters input sources to `local_path IS NOT NULL AND archived IS NOT TRUE` so archived sources (which have their own `gbrain sources archived` surface) don't muddy the active-sync dashboard. Lives under `sources` (not `sync --status`) per D3 from the v0.40.3.0 plan-eng-review — reads and writes don't share a verb.
|
||||
- `src/commands/sources.ts` — `gbrain sources {list,add,remove,archive,restore,archived,purge,current,status,audit}`. **v0.37.7.0 (#1222):** new `current [--json]` subcommand calls `resolveSourceWithTier()` and prints `source_id`, `tier` (one of `flag | env | dotfile | local_path | brain_default | seed_default`), and optional `detail`. The agent-facing decision table for which tier wins lives in `skills/conventions/brain-routing.md`. **v0.40.3.0 (productionized from PR #1314):** new `status [--json]` subcommand — read-only per-source dashboard (last sync, staleness, page count, embedding coverage, unacked failures). Thin wrapper around `buildSyncStatusReport` + `printSyncStatusReport` exported from `src/commands/sync.ts`. `--json` emits the stable `{schema_version: 1, sources, ...}` envelope on stdout for monitoring pipelines; bare invocation prints the human table to stdout (right-aligned numeric columns, kubectl-style). Filters input sources to `local_path IS NOT NULL AND archived IS NOT TRUE` so archived sources (which have their own `gbrain sources archived` surface) don't muddy the active-sync dashboard. Lives under `sources` (not `sync --status`) per D3 from the v0.40.3.0 plan-eng-review — reads and writes don't share a verb. **v0.40.9.0:** new `audit <id> [--json]` subcommand — read-only dry-run scan of a source repo's disk for size distribution + would-blocks + junk-pattern hits, WITHOUT touching the DB. Catches scraper junk and oversized content BEFORE sync. Walks `sources.local_path`, reads each markdown file, runs `assessContent()` from `src/core/content-sanity.ts`, aggregates by verdict (`ok | warn_oversize | hard_block_junk_pattern`). JSON envelope is stable for monitoring pipelines. Pinned by `test/sources-audit.test.ts` (not present in this wave; covered transitively by `test/content-sanity.test.ts` + `test/import-file-content-sanity.test.ts`).
|
||||
- `src/commands/reindex-frontmatter.ts` — `gbrain reindex-frontmatter`. **v0.37.7.0 (#1225):** wrapped the query path in the standard `withEngine(...)` lifecycle so `engine.connect()` runs before the first SQL call. Pre-fix the command `process.exit(1)`'d with a TypeError on first invocation. Pinned by `test/reindex-frontmatter-connect.test.ts`.
|
||||
- `src/core/source-resolver.ts` — 6-tier source resolution. **v0.37.7.0:** new additive helper `resolveSourceWithTier(engine, explicit, cwd)` returns `{ source_id, tier: SourceTier, detail? }` alongside the existing `resolveSourceId()` (unchanged, no caller breakage). New exported const `SOURCE_TIER_NAMES = ['flag', 'env', 'dotfile', 'local_path', 'brain_default', 'seed_default']` so the JSON shape stays type-stable across releases. Order matches the 1-6 priority of `resolveSourceId()`. Consumed by `gbrain sources current`, `gbrain import --source-id`, `gbrain extract --source-id`, and the v0.37.7.0 `source_routing_health` doctor check. Pinned by `test/source-resolver-with-tier.test.ts` (uses `withEnv()` wrapper per the test-isolation lint).
|
||||
- `src/commands/autopilot.ts` extension (v0.37.7.0) — three changes for federated-brain co-existence and launchd hygiene. (1) **#1226 lockfile scope:** `LOCK_PATH` resolves via `gbrainPath('autopilot.lock')` so it honors `GBRAIN_HOME`. Two brains can run autopilot simultaneously without lock-stealing. Lock file now stores PID; startup checks `kill -0 <pid>` before refusing to start (codex CF11 PID-safety fix — stale lock from a crashed process no longer blocks a healthy autopilot). (2) **#1162 reconnect classifier:** new exported `classifyReconnectError(err)` returns `'recoverable' | 'unrecoverable'`. Unrecoverable causes the daemon to `process.exit(0)` and let launchd back off instead of the v0.37.6 loop that logged `config.database_url undefined` every 5s forever. (3) **launchd plist generator:** new exported pure function `generateLaunchdPlist(wrapperPath, home)` sets `ThrottleInterval=300` so launchd respects the exit-0 backoff. Both helpers pinned by `test/autopilot-lock-path.test.ts` + `test/autopilot-reconnect-classifier.test.ts`.
|
||||
@@ -313,7 +313,14 @@ strict behavior when unset.
|
||||
- `src/commands/doctor.ts` extension (v0.40.4.1) — `buildChecks(engine, args, dbSource): Promise<Check[]>` exported as a test seam. `runDoctor` is now a thin wrapper: `buildChecks → computeDoctorReport → render + process.exit`. All 10 `process.exit` sites stay in the wrapper; the two early-return paths (no engine, connection failure) return partial check lists instead of inline exits. No behavior change — observable output identical because the wrapper renders the same partial list. Pinned by `test/doctor-behavioral.test.ts` (13 cases: pure aggregation math over `computeDoctorReport`, orchestrator cases for `--fast` skip set + `--json` flag + no-engine partial path + snapshot of load-bearing check names) and `test/doctor-cli-smoke.serial.test.ts` (1 subprocess case spawning `bun run src/cli.ts doctor --json` against a fresh PGLite tempdir, asserting schema_version=2 envelope, status enum, non-empty checks array — the render-path coverage that buildChecks-only tests miss). Quarantined as `.serial.test.ts` because PGLite write-locks don't play with parallel runners.
|
||||
- `src/core/cycle.ts` extension (v0.40.4.1) — `runPhaseLint` + `runPhaseBacklinks` gain the `export` keyword so behavioral tests can drive them directly. No body changes; documented as internal helpers exposed for test-only consumption (downstream code should NOT take a dependency). Pinned by `test/cycle-legacy-phases.test.ts` (11 cases across both phases: clean run → status='ok', partial fix → status='warn' with `dryRun` in details, dry-run path doesn't write, throw-from-lib → status='fail' with the wrapper's try/catch envelope populated). Future phase wrappers (sync, extract, embed, orphans, extract_facts, resolve_symbol_edges, recompute_emotional_weight) land as additional describes in the same file, not new files (TODOS NEW-2).
|
||||
- `test/operations-trust-boundary.test.ts` + `scripts/check-operations-filter-bypass.sh` (v0.40.4.1) — operations trust-boundary contract coverage. Hybrid test design: pure assertions over all 74 ops (every op has a scope annotation; every mutating op has a non-read scope; `localOnly: true` ops are excluded from `operations.filter(op => !op.localOnly)`; the seven historically-sensitive localOnly ops snapshot-pinned by name) plus targeted handler-invocation regressions for the two historically-broken HTTP-callable classes: `submit_job` with `name='shell'` + `ctx.remote=true` MUST reject (the F7b HTTP MCP shell-job RCE class), and `search_by_image` with `image_path` + `ctx.remote=true` MUST reject (the D18 P0 image-leak class). `file_upload` and `sync_brain` deliberately omitted from handler-invocation tests because they're `localOnly: true` and that path would test an impossible production scenario (codex CMT-3). The shell guard greps `src/` for any module importing the `operations` value outside the canonical filter site at `src/commands/serve-http.ts` — three import shapes detected (destructured, aliased, namespace), explicit 10-entry allow-list with per-entry rationale, plus a literal-string check that `serve-http.ts` still contains `operations.filter(op => !op.localOnly)`. Wired into `bun run verify`.
|
||||
- `src/core/link-extraction.ts` — shared library for the v0.12.0 graph layer. extractEntityRefs (canonical, replaces backlinks.ts duplicate) matches both `[Name](people/slug)` markdown links and Obsidian `[[people/slug|Name]]` wikilinks as of v0.12.3. extractPageLinks, inferLinkType heuristics (attended/works_at/invested_in/founded/advises/source/mentions), parseTimelineEntries, isAutoLinkEnabled config helper. `DIR_PATTERN` covers `people`, `companies`, `deals`, `topics`, `concepts`, `projects`, `entities`, `tech`, `finance`, `personal`, `openclaw`. Used by extract.ts, operations.ts auto-link post-hook, and backlinks.ts.
|
||||
- `src/core/content-sanity.ts` (v0.40.9.0, NEW) — pure assessor for the content-sanity defense wave. `assessContent(content, opts): SanityVerdict` returns one of three verdicts (`ok | warn_oversize | hard_block_junk_pattern | soft_block_oversize`) with `{reason, bytes, matched_pattern_name?}` detail. Six hand-vetted built-in junk patterns (Cloudflare challenge dumps, CAPTCHAs, 403 dumps, bare error-page titles) compiled at module load; operator literal substrings loaded via `loadOperatorLiterals()` from `src/core/content-sanity-literals.ts`. `ContentSanityBlockError` tagged class is the typed throw shape — every wrapper site (`gbrain import` CLI, `put_page` MCP op, `gbrain sync`, `/ingest` webhook) catches it via the existing exception flow rather than a parallel status check. The bytes-parity contract (D2) pins `Buffer.byteLength(content, 'utf8')` against the embedder's actual byte count so a 499K-byte page can't be soft-blocked on assessment then overflow on embed. Knob resolution chain: env > file (`~/.gbrain/config.json`) > DB > defaults — env wins for CI / one-off overrides, file is operator-set, DB plane is what `gbrain config set` writes. Four knobs: `content_sanity.bytes_warn` (default 50_000), `content_sanity.bytes_block` (default 500_000), `content_sanity.junk_patterns_enabled` (default true), `content_sanity.disabled` (default false; `GBRAIN_NO_SANITY=1` is the loud-stderr kill-switch with per-ingest warning). Pinned by `test/content-sanity.test.ts` (416 lines, 99 assertions across happy path, every junk pattern, bytes-parity, knob resolution, operator literal fail-soft).
|
||||
- `src/core/content-sanity-literals.ts` (v0.40.9.0, NEW) — operator literal-substring loader. Reads `~/.gbrain/junk-substrings.txt`, one literal per non-comment non-blank line. Optional `# name=<id>` header pairs an identifier with the following literal so audit JSONL groups by site (`linkedin_auth_wall`, `reddit_blocked`, etc.). Fail-soft on ENOENT (missing file = empty array, no error). Loaded on every ingest. Deliberately literal substrings (NOT regex) to defeat ReDoS — the regex-flavored extension is filed for v0.41+ once a real ReDoS budget exists. Pinned by `test/content-sanity-literals.test.ts` (110 lines).
|
||||
- `src/core/embed-skip.ts` (v0.40.9.0, NEW) — 5-site shared predicate for the soft-block embed-skip filter. Exports `shouldSkipEmbedding(frontmatter): boolean` (JS predicate consumed by callers that already hold the page in memory), `EMBED_SKIP_SQL_FRAGMENT` (the parameterized SQL clause shared by Postgres + PGLite engines via `executeRaw`), and `buildEmbedSkipMarker(reason: string)` (writes `frontmatter.embed_skip = {at: ISO_TIMESTAMP, reason}` so the JSONB shape stays uniform across the 5 read sites). The 5 sites are: `embed.ts --stale`, `embed.ts --all`, the `embed-stale` Minion helper, plus both engines' `listStaleChunks` + `countStaleChunks`. Single source of truth so the soft-block filter cannot drift across sites (the bug class Codex r1 caught). Pinned by `test/embed-skip.test.ts` (cross-site invariant + JSONB shape).
|
||||
- `src/core/audit/content-sanity-audit.ts` (v0.40.9.0, NEW) — ISO-week JSONL audit at `~/.gbrain/audit/content-sanity-YYYY-Www.jsonl` built on the v0.40.4.0 `audit-writer.ts` primitive. Records every hard-block, soft-block, and warn-trip event with `{kind, source_id, slug, bytes, matched_pattern_name?, reason, ts}`. Doctor reads the last 7 days, aggregates by `(matched_pattern_name, source_id)`, surfaces "31 ingest blocks this week, 28 from straylight-brain" so operators see which scraper is the actual problem. Honors `GBRAIN_AUDIT_DIR` for shared-filesystem multi-host setups (documented caveat in the doctor message for ops that don't share the dir). Pinned by `test/audit/content-sanity-audit.test.ts` (219 lines, 219 assertions).
|
||||
- `src/commands/doctor.ts` extension (v0.40.9.0) — three new checks wired into `runDoctor()` and the JSON envelope: `oversized_pages` (warns on pages exceeding `content_sanity.bytes_warn`), `scraper_junk_pages` (warns on pages that match any junk pattern despite being live in the DB — these escaped pre-v0.40.9.0 ingest), and `content_sanity_audit_recent` (reads the last 7 days of audit events, aggregates by pattern+source). Default scans the 1000 most-recent pages; new `--content-audit` flag opts into a full scan for the cleanup wave. All three are warn-only with paste-ready fix hints (junk page → `gbrain sources audit <id>` + `git rm` source-of-truth, oversize → split or accept).
|
||||
- `src/commands/lint.ts` extension (v0.40.9.0) — two new lint rules: `huge-page` (flags pages exceeding `content_sanity.bytes_warn` threshold) and `scraper-junk` (flags pages matching any junk pattern). Both reuse `assessContent()` from `src/core/content-sanity.ts` so lint, doctor, and ingest share one assessor — adding a junk pattern automatically covers all three surfaces. `lint.ts` lifts DB config when `~/.gbrain/` is reachable (matches what `gbrain config set` writes); falls back to file/env on CI. Pinned by `test/lint-content-sanity.test.ts` (161 lines).
|
||||
- `src/commands/embed.ts` extension (v0.40.9.0) — applies the `embed-skip` filter at all 5 stale-chunk sites: `runEmbedCore --stale`, `runEmbedCore --all`, the `embed-stale` Minion helper, plus both engines' `listStaleChunks` + `countStaleChunks` via `EMBED_SKIP_SQL_FRAGMENT`. A soft-blocked page is queryable by title and slug but its chunks never enter the embed sweep. The shared helper from `src/core/embed-skip.ts` is the regression guard — no per-site ad-hoc filter is allowed. Pinned by `test/embed-skip.test.ts`.
|
||||
- `src/core/import-file.ts` extension (v0.40.9.0) — `importFromContent` is the narrow waist that every ingest path passes through (`gbrain import`, `gbrain sync`, `put_page` MCP, `/ingest` webhook). It now calls `assessContent()` BEFORE chunking; verdict `hard_block_junk_pattern` throws `ContentSanityBlockError` (which every wrapper site already catches via its exception flow); verdict `warn_oversize OR oversize-without-junk` sets `frontmatter.embed_skip` via `buildEmbedSkipMarker()` AND deletes any pre-existing chunks for the page in the same transaction so search can't surface stale chunks against content that's now soft-blocked. `gbrain import` honors `errors > 0` for non-zero exit (was silently exit-0 on failed files). `classifyErrorCode` in `src/core/sync.ts` recognizes the new `PAGE_JUNK_PATTERN` code so sync-failures.jsonl grouping bins these correctly. Pinned by `test/import-file-content-sanity.test.ts` (206 lines). — shared library for the v0.12.0 graph layer. extractEntityRefs (canonical, replaces backlinks.ts duplicate) matches both `[Name](people/slug)` markdown links and Obsidian `[[people/slug|Name]]` wikilinks as of v0.12.3. extractPageLinks, inferLinkType heuristics (attended/works_at/invested_in/founded/advises/source/mentions), parseTimelineEntries, isAutoLinkEnabled config helper. `DIR_PATTERN` covers `people`, `companies`, `deals`, `topics`, `concepts`, `projects`, `entities`, `tech`, `finance`, `personal`, `openclaw`. Used by extract.ts, operations.ts auto-link post-hook, and backlinks.ts.
|
||||
- `src/core/zombie-reap.ts` (v0.28.1) — idempotent `installSigchldHandler()` so JS-spawned children get reaped via Bun's internal `waitpid()`. Bun (like Node) only auto-reaps when a SIGCHLD listener is registered; without it, every child the worker spawns (shell jobs, embed batches, sub-agents) becomes a zombie on exit and holds connection slots. Called once at module load from `src/cli.ts` (with Windows platform guard — SIGCHLD doesn't exist on Windows). Cross-file leak guard via `_uninstallSigchldHandlerForTests()` for tests. Layer 1 of the three-layer zombie defense; Layer 2 is tini-as-PID-1 wrapping the worker subtree (via `src/core/minions/spawn-helpers.ts`); Layer 3 is the container's own tini for hard Bun crashes.
|
||||
- `src/core/minions/` — Minions job queue: BullMQ-inspired, Postgres-native (queue, worker, backoff, types, protected-names, quiet-hours, stagger, handlers/shell).
|
||||
- `src/core/minions/queue.ts` — MinionQueue class (submit, claim, complete, fail, stall detection, parent-child, depth/child-cap, per-job timeouts, cascade-kill, attachments, idempotency keys, child_done inbox, removeOnComplete/Fail). `add()` takes a 4th `trusted` arg (separate from `opts` to prevent spread leakage); protected names in `PROTECTED_JOB_NAMES` require `{allowProtectedSubmit: true}` and the check runs trim-normalized (whitespace-bypass safe). v0.14.1 #219: `add()` plumbs `max_stalled` through with a `[1, 100]` clamp; omitted values let the schema DEFAULT (5) kick in. v0.19.0: `handleWallClockTimeouts(lockDurationMs)` is Layer 3 kill shot for jobs where `FOR UPDATE SKIP LOCKED` stall detection and the timeout sweep both fail to evict (wedged worker holding a row lock via a pending transaction). v0.19.1: `maxWaiting` coalesce path now uses `pg_advisory_xact_lock` keyed on `(name, queue)` to serialize concurrent submits for the same key, and filters on `queue` in addition to `name` so cross-queue same-name jobs don't suppress each other.
|
||||
|
||||
+1
-1
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "gbrain",
|
||||
"version": "0.40.9.0",
|
||||
"version": "0.40.10.0",
|
||||
"description": "Postgres-native personal knowledge brain with hybrid RAG search",
|
||||
"type": "module",
|
||||
"main": "src/core/index.ts",
|
||||
|
||||
+10
-1
@@ -1108,7 +1108,16 @@ async function handleCliOnly(command: string, args: string[]) {
|
||||
switch (command) {
|
||||
case 'import': {
|
||||
const { runImport } = await import('./commands/import.ts');
|
||||
await runImport(engine, args);
|
||||
// v0.41 (Codex r2 #3 fix): honor errors counter for exit code.
|
||||
// runImport's per-file catch already records failures, but the
|
||||
// CLI was discarding the result so the process exited 0 even
|
||||
// when files failed (e.g. content-sanity hard-block throws,
|
||||
// size-cap throws, parse errors). Surface non-zero on errors > 0
|
||||
// so wrappers (sync, CI scripts, `&& gbrain doctor`) propagate.
|
||||
const importResult = await runImport(engine, args);
|
||||
if (importResult.errors > 0) {
|
||||
process.exitCode = 1;
|
||||
}
|
||||
break;
|
||||
}
|
||||
case 'export': {
|
||||
|
||||
@@ -3669,6 +3669,176 @@ export async function buildChecks(
|
||||
mbcHb();
|
||||
}
|
||||
|
||||
// 11b. Content sanity checks (v0.41).
|
||||
//
|
||||
// Three sibling checks all backed by the shared assessor in
|
||||
// src/core/content-sanity.ts so the surface stays aligned with the
|
||||
// ingest gate at importFromContent and the lint rules at lintContent.
|
||||
//
|
||||
// - oversized_pages: indexed-free table scan (~100ms on 100K-page brains)
|
||||
// counting pages whose body (compiled_truth + timeline, UTF-8 bytes
|
||||
// via octet_length per Codex r2 #13) exceeds the block threshold.
|
||||
// Status warn when 1+ rows; never fail (oversize is now a soft state).
|
||||
// - scraper_junk_pages: capped 1000-most-recent default + --content-audit
|
||||
// opt-in for full scan (D10 mirrors --index-audit precedent). Applies
|
||||
// the assessor per-page on title + 2KB head-slice + frontmatter.
|
||||
// - content_sanity_audit_recent: reads ~/.gbrain/audit/content-sanity-*.jsonl
|
||||
// over the last 7 days, aggregates by event type + source. Caveat
|
||||
// (Codex r1 #14): JSONL is local-only — multi-host operators should
|
||||
// share GBRAIN_AUDIT_DIR. Message names this so the limitation is
|
||||
// visible at the doctor surface.
|
||||
const fullContentAudit = args.includes('--content-audit');
|
||||
progress.heartbeat('oversized_pages');
|
||||
try {
|
||||
const sql = db.getConnection();
|
||||
// Read effective bytes_block from the cached effectiveCfg loaded
|
||||
// earlier in this doctor run if available; otherwise default.
|
||||
// (We re-read here per-check to avoid threading config through
|
||||
// every check — bytes_block is read once per doctor run via
|
||||
// loadConfig which caches in module-level config layer.)
|
||||
const { loadConfig: _loadCfg } = await import('../core/config.ts');
|
||||
const _cfg = _loadCfg();
|
||||
const bytesBlock = _cfg?.content_sanity?.bytes_block ?? 500_000;
|
||||
const rows = await sql`
|
||||
SELECT p.slug, p.source_id,
|
||||
octet_length(p.compiled_truth) + octet_length(COALESCE(p.timeline, '')) AS bytes
|
||||
FROM pages p
|
||||
WHERE p.deleted_at IS NULL
|
||||
AND (octet_length(p.compiled_truth) + octet_length(COALESCE(p.timeline, ''))) > ${bytesBlock}
|
||||
ORDER BY bytes DESC
|
||||
LIMIT 100
|
||||
`;
|
||||
if (rows.length === 0) {
|
||||
checks.push({
|
||||
name: 'oversized_pages',
|
||||
status: 'ok',
|
||||
message: `No pages exceed ${bytesBlock} bytes`,
|
||||
});
|
||||
} else {
|
||||
const oversizeRows = rows as unknown as Array<{ slug: string; source_id: string; bytes: number }>;
|
||||
const top = oversizeRows.slice(0, 3)
|
||||
.map(r => `${r.slug} (${r.bytes}b, src=${r.source_id})`)
|
||||
.join('; ');
|
||||
checks.push({
|
||||
name: 'oversized_pages',
|
||||
status: 'warn',
|
||||
message: `${rows.length} page(s) exceed ${bytesBlock}-byte block threshold. Top: ${top}. New ingests with the same shape get frontmatter.embed_skip set automatically; existing oversized pages can be split or accepted as non-embeddable.`,
|
||||
});
|
||||
}
|
||||
} catch (err) {
|
||||
const msg = err instanceof Error ? err.message : String(err);
|
||||
checks.push({
|
||||
name: 'oversized_pages',
|
||||
status: 'ok',
|
||||
message: `Skipped (${msg})`,
|
||||
});
|
||||
}
|
||||
|
||||
progress.heartbeat('scraper_junk_pages');
|
||||
try {
|
||||
const sql = db.getConnection();
|
||||
const { assessContentSanity } = await import('../core/content-sanity.ts');
|
||||
const { loadOperatorLiterals } = await import('../core/content-sanity-literals.ts');
|
||||
const literals = loadOperatorLiterals();
|
||||
const scanLimit = fullContentAudit ? null : 1000;
|
||||
const rows = scanLimit
|
||||
? await sql`
|
||||
SELECT p.slug, p.source_id, p.title,
|
||||
LEFT(p.compiled_truth, 2048) AS body_head,
|
||||
LEFT(COALESCE(p.timeline, ''), 1024) AS tl_head,
|
||||
p.frontmatter
|
||||
FROM pages p
|
||||
WHERE p.deleted_at IS NULL
|
||||
ORDER BY p.updated_at DESC
|
||||
LIMIT ${scanLimit}
|
||||
`
|
||||
: await sql`
|
||||
SELECT p.slug, p.source_id, p.title,
|
||||
LEFT(p.compiled_truth, 2048) AS body_head,
|
||||
LEFT(COALESCE(p.timeline, ''), 1024) AS tl_head,
|
||||
p.frontmatter
|
||||
FROM pages p
|
||||
WHERE p.deleted_at IS NULL
|
||||
`;
|
||||
const hits: Array<{ slug: string; matched: string[] }> = [];
|
||||
const scanRows = rows as unknown as Array<{ slug: string; source_id: string; title: string; body_head: string; tl_head: string; frontmatter: Record<string, unknown> | null }>;
|
||||
for (const r of scanRows) {
|
||||
const sanity = assessContentSanity({
|
||||
compiled_truth: r.body_head ?? '',
|
||||
timeline: r.tl_head ?? '',
|
||||
title: r.title ?? '',
|
||||
bytes_warn: Number.MAX_SAFE_INTEGER, // we ONLY care about junk-pattern hits here
|
||||
bytes_block: Number.MAX_SAFE_INTEGER,
|
||||
extra_literals: literals,
|
||||
});
|
||||
if (sanity.shouldHardBlock) {
|
||||
hits.push({
|
||||
slug: r.slug,
|
||||
matched: [...sanity.junk_pattern_matches, ...sanity.literal_substring_matches],
|
||||
});
|
||||
}
|
||||
}
|
||||
if (hits.length === 0) {
|
||||
checks.push({
|
||||
name: 'scraper_junk_pages',
|
||||
status: 'ok',
|
||||
message: scanLimit
|
||||
? `No junk-pattern hits in ${rows.length} recent page(s) (use --content-audit for full scan)`
|
||||
: `No junk-pattern hits in ${rows.length} page(s) (full audit)`,
|
||||
});
|
||||
} else {
|
||||
const top = hits.slice(0, 3).map(h => `${h.slug} [${h.matched.join(',')}]`).join('; ');
|
||||
checks.push({
|
||||
name: 'scraper_junk_pages',
|
||||
status: 'warn',
|
||||
message: `${hits.length} page(s) match junk patterns. Top: ${top}. ${scanLimit ? '(scanned 1000 most-recent; rerun with --content-audit for full scan)' : '(full audit)'} New ingests with these shapes are now hard-blocked; existing inventory should be cleaned at source.`,
|
||||
});
|
||||
}
|
||||
} catch (err) {
|
||||
const msg = err instanceof Error ? err.message : String(err);
|
||||
checks.push({
|
||||
name: 'scraper_junk_pages',
|
||||
status: 'ok',
|
||||
message: `Skipped (${msg})`,
|
||||
});
|
||||
}
|
||||
|
||||
progress.heartbeat('content_sanity_audit_recent');
|
||||
try {
|
||||
const { readRecentContentSanityEvents, summarizeContentSanityEvents } =
|
||||
await import('../core/audit/content-sanity-audit.ts');
|
||||
const events = readRecentContentSanityEvents(7);
|
||||
if (events.length === 0) {
|
||||
checks.push({
|
||||
name: 'content_sanity_audit_recent',
|
||||
status: 'ok',
|
||||
message: 'No content-sanity events in last 7 days (audit JSONL is local to this host; share GBRAIN_AUDIT_DIR for multi-host visibility)',
|
||||
});
|
||||
} else {
|
||||
const summary = summarizeContentSanityEvents(events);
|
||||
const topPatterns = summary.top_patterns.slice(0, 3).map(p => `${p.name}=${p.count}`).join(', ');
|
||||
const topSources = Object.entries(summary.by_source)
|
||||
.sort((a, b) => b[1] - a[1])
|
||||
.slice(0, 3)
|
||||
.map(([s, n]) => `${s}=${n}`)
|
||||
.join(', ');
|
||||
const status: 'ok' | 'warn' | 'fail' =
|
||||
events.length >= 100 ? 'fail' : events.length >= 10 ? 'warn' : 'ok';
|
||||
checks.push({
|
||||
name: 'content_sanity_audit_recent',
|
||||
status,
|
||||
message: `${events.length} events (hard=${summary.by_type.hard_block} soft=${summary.by_type.soft_block} warn=${summary.by_type.warn})${topPatterns ? ', patterns: ' + topPatterns : ''}${topSources ? ', sources: ' + topSources : ''}. (Local audit only — multi-host operators set GBRAIN_AUDIT_DIR.)`,
|
||||
});
|
||||
}
|
||||
} catch (err) {
|
||||
const msg = err instanceof Error ? err.message : String(err);
|
||||
checks.push({
|
||||
name: 'content_sanity_audit_recent',
|
||||
status: 'ok',
|
||||
message: `Skipped (${msg})`,
|
||||
});
|
||||
}
|
||||
|
||||
// 11a. Frontmatter integrity (v0.22.4, hardened in v0.38.2.0).
|
||||
// scanBrainSources walks every registered source's local_path on disk
|
||||
// (not from the DB), invoking parseMarkdown(..., {validate:true}) per
|
||||
|
||||
+13
-1
@@ -7,6 +7,7 @@ import { getCliOptions, cliOptsToProgressOptions } from '../core/cli-options.ts'
|
||||
import { assertEmbeddingEnabled } from '../core/embedding-dim-check.ts';
|
||||
import { loadConfig } from '../core/config.ts';
|
||||
import { slog, serr } from '../core/console-prefix.ts';
|
||||
import { filterOutEmbedSkipped } from '../core/embed-skip.ts';
|
||||
|
||||
export interface EmbedOpts {
|
||||
/** Embed ALL pages (every chunk). */
|
||||
@@ -353,7 +354,18 @@ async function embedAll(
|
||||
}
|
||||
|
||||
// v0.31.12: when sourceId is set, scope listPages to that source.
|
||||
const pages = await engine.listPages({ limit: 100000, ...(sourceId && { sourceId }) });
|
||||
// v0.41 (D8 + Codex r2 #11): apply embed-skip filter via the shared
|
||||
// helper so the `--all` path honors `frontmatter.embed_skip` the same
|
||||
// way the `--stale` path does. Without this filter, `gbrain embed --all`
|
||||
// (common after model swaps) re-embeds every soft-blocked page,
|
||||
// defeating the soft-block. Filtering JS-side here mirrors the SQL-side
|
||||
// filter that listStaleChunks/countStaleChunks apply on --stale.
|
||||
const allPages = await engine.listPages({ limit: 100000, ...(sourceId && { sourceId }) });
|
||||
const pages = filterOutEmbedSkipped(allPages);
|
||||
const skippedByEmbedSkip = allPages.length - pages.length;
|
||||
if (skippedByEmbedSkip > 0) {
|
||||
serr(`[embed] skipped ${skippedByEmbedSkip} page(s) with frontmatter.embed_skip set`);
|
||||
}
|
||||
let processed = 0;
|
||||
|
||||
// Concurrency limit for parallel page embedding.
|
||||
|
||||
+157
-4
@@ -19,6 +19,13 @@
|
||||
import { readFileSync, writeFileSync, readdirSync, statSync, lstatSync, existsSync } from 'fs';
|
||||
import { join, relative } from 'path';
|
||||
import { parseMarkdown, type ParseValidationCode } from '../core/markdown.ts';
|
||||
import {
|
||||
assessContentSanity,
|
||||
type OperatorLiteral,
|
||||
DEFAULT_BYTES_WARN,
|
||||
} from '../core/content-sanity.ts';
|
||||
import { loadOperatorLiterals } from '../core/content-sanity-literals.ts';
|
||||
import { loadConfig, loadConfigWithEngine, gbrainPath } from '../core/config.ts';
|
||||
|
||||
export interface LintIssue {
|
||||
file: string;
|
||||
@@ -60,7 +67,26 @@ const LLM_PREAMBLES = [
|
||||
|
||||
// ── Rules ──────────────────────────────────────────────────────────
|
||||
|
||||
export function lintContent(content: string, filePath: string): LintIssue[] {
|
||||
/**
|
||||
* Per-call options for `lintContent`. Tests pass content-sanity opts
|
||||
* directly so the linter can be exercised without an engine.
|
||||
* Production callers (`runLintCore`) resolve effective config first
|
||||
* via the file/env/DB precedence chain and pass through.
|
||||
*/
|
||||
export interface LintContentOpts {
|
||||
/** v0.41 content-sanity thresholds + operator literals. When omitted,
|
||||
* the assessor uses its built-in defaults (50K warn, 500K block,
|
||||
* built-in junk patterns only). */
|
||||
contentSanity?: {
|
||||
bytes_warn?: number;
|
||||
bytes_block?: number;
|
||||
junk_patterns_enabled?: boolean;
|
||||
disabled?: boolean;
|
||||
operator_literals?: ReadonlyArray<OperatorLiteral>;
|
||||
};
|
||||
}
|
||||
|
||||
export function lintContent(content: string, filePath: string, opts: LintContentOpts = {}): LintIssue[] {
|
||||
const issues: LintIssue[] = [];
|
||||
const lines = content.split('\n');
|
||||
|
||||
@@ -182,6 +208,57 @@ export function lintContent(content: string, filePath: string): LintIssue[] {
|
||||
}
|
||||
}
|
||||
|
||||
// v0.41 content-sanity rules. Two new lint rules (huge-page +
|
||||
// scraper-junk) backed by the shared assessor in
|
||||
// src/core/content-sanity.ts so the threshold + pattern set stays
|
||||
// in sync with the ingest gate at importFromContent. Kill-switch
|
||||
// (contentSanity.disabled) suppresses both.
|
||||
//
|
||||
// Bytes are measured against the parsed body (compiled_truth +
|
||||
// timeline) for parity with doctor's `oversized_pages` check (D2).
|
||||
// The earlier file-byte design disagreed with doctor on pages with
|
||||
// large frontmatter; pulling from parsed keeps the surfaces aligned
|
||||
// on the operationally-meaningful axis (embed pipeline input).
|
||||
const cs = opts.contentSanity ?? {};
|
||||
if (cs.disabled !== true) {
|
||||
const operator_literals = cs.junk_patterns_enabled !== false
|
||||
? (cs.operator_literals ?? [])
|
||||
: [];
|
||||
const sanity = assessContentSanity({
|
||||
compiled_truth: parsed.compiled_truth,
|
||||
timeline: parsed.timeline ?? '',
|
||||
title: parsed.title,
|
||||
bytes_warn: cs.bytes_warn,
|
||||
bytes_block: cs.bytes_block,
|
||||
extra_literals: operator_literals,
|
||||
});
|
||||
// Rule: huge-page fires for both oversize_warn (over warn threshold)
|
||||
// AND oversize_block (over block threshold). Operator sees the same
|
||||
// rule name in both cases; the message names the actual byte count.
|
||||
if (sanity.reasons.includes('oversize_warn') || sanity.reasons.includes('oversize_block')) {
|
||||
const threshold = sanity.reasons.includes('oversize_block') ? 'block' : 'warn';
|
||||
issues.push({
|
||||
file: filePath, line: 1, rule: 'huge-page',
|
||||
message: `Page body is ${sanity.bytes} bytes (exceeds ${threshold} threshold)`,
|
||||
fixable: false,
|
||||
});
|
||||
}
|
||||
// Rule: scraper-junk fires on any built-in pattern or operator literal hit.
|
||||
// Message names which pattern(s) matched so the brain-author can
|
||||
// either delete the file from their source repo or audit the scraper.
|
||||
if (sanity.junk_pattern_matches.length > 0 || sanity.literal_substring_matches.length > 0) {
|
||||
const matched = [
|
||||
...sanity.junk_pattern_matches,
|
||||
...sanity.literal_substring_matches,
|
||||
].join(', ');
|
||||
issues.push({
|
||||
file: filePath, line: 1, rule: 'scraper-junk',
|
||||
message: `Matched junk pattern(s): ${matched}`,
|
||||
fixable: false,
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
return issues;
|
||||
}
|
||||
|
||||
@@ -205,6 +282,62 @@ export function fixContent(content: string): string {
|
||||
return fixed.trim() + '\n';
|
||||
}
|
||||
|
||||
/**
|
||||
* Resolve effective content-sanity opts for lint (D1: file/env first,
|
||||
* lift DB-plane when an engine is reachable).
|
||||
*
|
||||
* File/env path is sync via `loadConfig()`; DB-plane lift requires a
|
||||
* brief engine open. Best-effort: any engine failure (no brain
|
||||
* configured, connection refused, transient error) falls through to
|
||||
* the file/env values. CI without `~/.gbrain/` falls through
|
||||
* immediately since `loadConfig()` returns minimal config.
|
||||
*
|
||||
* Also loads the operator literals file (`~/.gbrain/junk-substrings.txt`)
|
||||
* once per lint invocation so multi-file lint runs amortize the read.
|
||||
*/
|
||||
async function resolveLintContentSanity(): Promise<LintContentOpts['contentSanity']> {
|
||||
const base = loadConfig();
|
||||
let cs = base?.content_sanity;
|
||||
|
||||
// DB-plane lift: only attempt when the file/env config suggests an
|
||||
// engine is configured. Avoids spinning up a fresh PGLite just to
|
||||
// read 4 config keys in a CI lint run that has no brain at all.
|
||||
const hasEngineConfig = !!(base?.database_url || base?.database_path);
|
||||
if (hasEngineConfig) {
|
||||
try {
|
||||
const { createEngine } = await import('../core/engine-factory.ts');
|
||||
const engine = await createEngine({
|
||||
engine: base!.engine,
|
||||
database_url: base!.database_url,
|
||||
database_path: base!.database_path,
|
||||
});
|
||||
try {
|
||||
await engine.connect({});
|
||||
const lifted = await loadConfigWithEngine(engine, base);
|
||||
cs = lifted?.content_sanity ?? cs;
|
||||
} finally {
|
||||
await engine.disconnect().catch(() => { /* best-effort cleanup */ });
|
||||
}
|
||||
} catch {
|
||||
// Engine unreachable or failed mid-probe — fall through to
|
||||
// file/env values. Lint should never block on engine state.
|
||||
}
|
||||
}
|
||||
|
||||
// Operator literals: always attempt to load (cheap FS read; missing
|
||||
// file is the common case and returns []). Skip when kill-switch
|
||||
// is on or junk patterns explicitly disabled to match the assessor's
|
||||
// own bypass logic exactly.
|
||||
const operator_literals = cs?.disabled === true || cs?.junk_patterns_enabled === false
|
||||
? []
|
||||
: loadOperatorLiterals();
|
||||
|
||||
return {
|
||||
...cs,
|
||||
operator_literals,
|
||||
};
|
||||
}
|
||||
|
||||
/** Collect markdown files from a directory */
|
||||
function collectPages(dir: string): string[] {
|
||||
const pages: string[] = [];
|
||||
@@ -224,6 +357,10 @@ export interface LintOpts {
|
||||
target: string;
|
||||
fix?: boolean;
|
||||
dryRun?: boolean;
|
||||
/** v0.41: optional pre-resolved content-sanity opts. When omitted,
|
||||
* `runLintCore` resolves via the file/env/DB chain. Tests inject
|
||||
* this directly to bypass the FS + engine layers. */
|
||||
contentSanity?: LintContentOpts['contentSanity'];
|
||||
}
|
||||
|
||||
export interface LintResult {
|
||||
@@ -252,13 +389,19 @@ export async function runLintCore(opts: LintOpts): Promise<LintResult> {
|
||||
const isSingleFile = statSync(opts.target).isFile();
|
||||
const pages = isSingleFile ? [opts.target] : collectPages(opts.target);
|
||||
|
||||
// Resolve content-sanity config once for this lint run (D1: lift DB
|
||||
// config when reachable). Caller can pre-pass via opts.contentSanity
|
||||
// (tests, Minion handler) to bypass the engine probe entirely.
|
||||
const contentSanity = opts.contentSanity ?? await resolveLintContentSanity();
|
||||
const lintOpts: LintContentOpts = { contentSanity };
|
||||
|
||||
let totalIssues = 0;
|
||||
let totalFixed = 0;
|
||||
let pagesWithIssues = 0;
|
||||
|
||||
for (const page of pages) {
|
||||
const content = readFileSync(page, 'utf-8');
|
||||
const issues = lintContent(content, isSingleFile ? page : relative(opts.target, page));
|
||||
const issues = lintContent(content, isSingleFile ? page : relative(opts.target, page), lintOpts);
|
||||
if (issues.length === 0) continue;
|
||||
pagesWithIssues++;
|
||||
totalIssues += issues.length;
|
||||
@@ -313,10 +456,18 @@ export async function runLint(args: string[]) {
|
||||
const progress = createProgress(cliOptsToProgressOptions(getCliOptions()));
|
||||
progress.start('lint.pages', pages.length);
|
||||
|
||||
// v0.41 (D1): resolve content-sanity config once for this lint run.
|
||||
// Mirrors runLintCore. The two paths must agree because runLint
|
||||
// prints human details inline; runLintCore at end computes the
|
||||
// aggregate. Sharing the resolved opts keeps both surfaces seeing
|
||||
// the same rule firings.
|
||||
const contentSanity = await resolveLintContentSanity();
|
||||
const lintContentOpts: LintContentOpts = { contentSanity };
|
||||
|
||||
for (const page of pages) {
|
||||
const content = readFileSync(page, 'utf-8');
|
||||
const relPath = isSingleFile ? page : relative(target, page);
|
||||
const issues = lintContent(content, relPath);
|
||||
const issues = lintContent(content, relPath, lintContentOpts);
|
||||
progress.tick(1);
|
||||
if (issues.length === 0) continue;
|
||||
|
||||
@@ -342,7 +493,9 @@ export async function runLint(args: string[]) {
|
||||
|
||||
// Re-run core for the aggregate counts (cheap; re-parses contents but
|
||||
// produces canonical numbers for the summary line).
|
||||
const result = await runLintCore({ target, fix: doFix, dryRun });
|
||||
// Pass contentSanity through so runLintCore skips its own resolve
|
||||
// (we already resolved once for the human-detail loop above).
|
||||
const result = await runLintCore({ target, fix: doFix, dryRun, contentSanity });
|
||||
console.log(`\n${result.pages_scanned} pages scanned. ${result.total_issues} issue(s) in ${result.pages_with_issues} page(s).`);
|
||||
if (doFix) {
|
||||
console.log(`${dryRun ? '(dry run) ' : ''}${result.total_fixed} auto-fixed.`);
|
||||
|
||||
@@ -876,6 +876,179 @@ async function runCurrent(engine: BrainEngine, args: string[]): Promise<void> {
|
||||
console.log(` tier: ${result.tier}${result.detail ? ` (${result.detail})` : ''}`);
|
||||
}
|
||||
|
||||
/**
|
||||
* v0.41 — `gbrain sources audit <id>` dry-run scan.
|
||||
*
|
||||
* Walks the source's `local_path` on disk, runs `assessContentSanity`
|
||||
* per `.md` file, and reports:
|
||||
* - file count + size distribution (p50 / p99 / max)
|
||||
* - would-hard-blocks (junk-pattern matches; new ingests would refuse)
|
||||
* - would-soft-blocks (oversize-only; new ingests would set embed_skip)
|
||||
* - junk-pattern hit counts grouped by pattern name
|
||||
*
|
||||
* Read-only: NO DB writes, NO file mutations. Intended for operators to
|
||||
* inspect a source repo BEFORE syncing (catches junk early) or AFTER
|
||||
* the new gate ships (audit existing inventory against the new rules
|
||||
* without touching state).
|
||||
*
|
||||
* Uses `pruneDir` from sync.ts so node_modules / .git / .obsidian are
|
||||
* skipped at descent — same walker semantics as the actual sync path.
|
||||
*/
|
||||
async function runAudit(engine: BrainEngine, args: string[]): Promise<void> {
|
||||
const sourceId = args.find((a) => !a.startsWith('--'));
|
||||
const json = args.includes('--json');
|
||||
const includeWarns = args.includes('--include-warns');
|
||||
|
||||
if (!sourceId) {
|
||||
console.error('Usage: gbrain sources audit <source-id> [--json] [--include-warns]');
|
||||
process.exit(2);
|
||||
}
|
||||
|
||||
const { fetchSource } = await import('../core/sources-load.ts');
|
||||
const src = await fetchSource(engine, sourceId);
|
||||
if (!src) {
|
||||
console.error(`Source not found: ${sourceId} (run \`gbrain sources list\` to see registered sources)`);
|
||||
process.exit(1);
|
||||
}
|
||||
if (!src.local_path) {
|
||||
console.error(`Source ${sourceId} has no local_path — cannot audit on disk`);
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
// Lazy-load FS + walker bits so the command stays import-cheap when
|
||||
// not invoked (every subcommand pays the import cost on dispatch).
|
||||
const { readFileSync, readdirSync, lstatSync, existsSync: _exists } =
|
||||
await import('fs');
|
||||
const { join: pathJoin } = await import('path');
|
||||
const { pruneDir } = await import('../core/sync.ts');
|
||||
const { assessContentSanity } = await import('../core/content-sanity.ts');
|
||||
const { loadOperatorLiterals } = await import('../core/content-sanity-literals.ts');
|
||||
const { parseMarkdown } = await import('../core/markdown.ts');
|
||||
|
||||
if (!_exists(src.local_path)) {
|
||||
console.error(`local_path does not exist on disk: ${src.local_path}`);
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
// Walk recursively. Mirror gbrain sync's descent rules so the file set
|
||||
// we audit matches the file set that would actually be ingested.
|
||||
const files: string[] = [];
|
||||
function walk(dir: string): void {
|
||||
let entries: string[];
|
||||
try {
|
||||
entries = readdirSync(dir);
|
||||
} catch {
|
||||
return; // permission denied; skip silently
|
||||
}
|
||||
for (const entry of entries) {
|
||||
const full = pathJoin(dir, entry);
|
||||
let stat;
|
||||
try {
|
||||
stat = lstatSync(full);
|
||||
} catch {
|
||||
continue;
|
||||
}
|
||||
if (stat.isDirectory()) {
|
||||
if (pruneDir(entry, dir)) continue;
|
||||
walk(full);
|
||||
} else if (entry.endsWith('.md')) {
|
||||
files.push(full);
|
||||
}
|
||||
}
|
||||
}
|
||||
walk(src.local_path);
|
||||
|
||||
const literals = loadOperatorLiterals();
|
||||
const sizes: number[] = [];
|
||||
const wouldHardBlock: Array<{ file: string; matched: string[]; bytes: number }> = [];
|
||||
const wouldSoftBlock: Array<{ file: string; bytes: number }> = [];
|
||||
const wouldWarn: Array<{ file: string; bytes: number }> = [];
|
||||
const patternHits: Record<string, number> = {};
|
||||
|
||||
for (const file of files) {
|
||||
let content: string;
|
||||
try {
|
||||
content = readFileSync(file, 'utf-8');
|
||||
} catch {
|
||||
continue;
|
||||
}
|
||||
let parsed;
|
||||
try {
|
||||
parsed = parseMarkdown(content, file);
|
||||
} catch {
|
||||
continue; // malformed page; not our concern in audit
|
||||
}
|
||||
const sanity = assessContentSanity({
|
||||
compiled_truth: parsed.compiled_truth,
|
||||
timeline: parsed.timeline ?? '',
|
||||
title: parsed.title,
|
||||
extra_literals: literals,
|
||||
});
|
||||
sizes.push(sanity.bytes);
|
||||
if (sanity.shouldHardBlock) {
|
||||
const matched = [...sanity.junk_pattern_matches, ...sanity.literal_substring_matches];
|
||||
for (const name of matched) {
|
||||
patternHits[name] = (patternHits[name] ?? 0) + 1;
|
||||
}
|
||||
wouldHardBlock.push({ file, matched, bytes: sanity.bytes });
|
||||
} else if (sanity.shouldSkipEmbed) {
|
||||
wouldSoftBlock.push({ file, bytes: sanity.bytes });
|
||||
} else if (sanity.reasons.includes('oversize_warn')) {
|
||||
wouldWarn.push({ file, bytes: sanity.bytes });
|
||||
}
|
||||
}
|
||||
|
||||
// Size distribution stats.
|
||||
sizes.sort((a, b) => a - b);
|
||||
const p = (q: number) =>
|
||||
sizes.length === 0 ? 0 : sizes[Math.min(sizes.length - 1, Math.floor(q * sizes.length))];
|
||||
|
||||
if (json) {
|
||||
console.log(JSON.stringify({
|
||||
schema_version: 1,
|
||||
source_id: sourceId,
|
||||
local_path: src.local_path,
|
||||
total_files: files.length,
|
||||
distribution: { p50: p(0.5), p99: p(0.99), max: sizes[sizes.length - 1] ?? 0 },
|
||||
hard_block_count: wouldHardBlock.length,
|
||||
soft_block_count: wouldSoftBlock.length,
|
||||
warn_count: wouldWarn.length,
|
||||
pattern_hits: patternHits,
|
||||
hard_blocks: wouldHardBlock.slice(0, 20),
|
||||
soft_blocks: wouldSoftBlock.slice(0, 20),
|
||||
...(includeWarns ? { warns: wouldWarn.slice(0, 20) } : {}),
|
||||
}, null, 2));
|
||||
return;
|
||||
}
|
||||
|
||||
console.log(`Source: ${sourceId} (${src.local_path})`);
|
||||
console.log(`Files scanned: ${files.length} markdown files`);
|
||||
if (sizes.length > 0) {
|
||||
console.log(`Size distribution: p50=${p(0.5)} bytes, p99=${p(0.99)} bytes, max=${sizes[sizes.length - 1]} bytes`);
|
||||
}
|
||||
console.log(`Would-hard-block: ${wouldHardBlock.length}`);
|
||||
console.log(`Would-soft-block: ${wouldSoftBlock.length}`);
|
||||
if (includeWarns) {
|
||||
console.log(`Would-warn: ${wouldWarn.length}`);
|
||||
}
|
||||
if (Object.keys(patternHits).length > 0) {
|
||||
const sorted = Object.entries(patternHits).sort((a, b) => b[1] - a[1]);
|
||||
console.log(`Junk-pattern hits: ${sorted.map(([n, c]) => `${n} ×${c}`).join(', ')}`);
|
||||
}
|
||||
if (wouldHardBlock.length > 0) {
|
||||
console.log('\nTop hard-blocks:');
|
||||
for (const h of wouldHardBlock.slice(0, 10)) {
|
||||
console.log(` ${h.file} [${h.matched.join(', ')}] (${h.bytes}b)`);
|
||||
}
|
||||
}
|
||||
if (wouldSoftBlock.length > 0) {
|
||||
console.log('\nTop soft-blocks (would write but skip embedding):');
|
||||
for (const s of wouldSoftBlock.slice(0, 10)) {
|
||||
console.log(` ${s.file} (${s.bytes}b)`);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// ── Dispatcher ──────────────────────────────────────────────
|
||||
|
||||
// v0.40.6.0: my duplicate `runStatus` (line ~895 pre-resolution) was
|
||||
@@ -917,6 +1090,7 @@ export async function runSources(engine: BrainEngine, args: string[]): Promise<v
|
||||
case 'tracked-branch': return runTrackedBranch(engine, rest);
|
||||
// v0.40.3.0 contextual retrieval (from master)
|
||||
case 'set-cr-mode': return runSetCrMode(engine, rest);
|
||||
case 'audit': return runAudit(engine, rest);
|
||||
case undefined:
|
||||
case '--help':
|
||||
case '-h':
|
||||
|
||||
@@ -0,0 +1,185 @@
|
||||
/**
|
||||
* Content-sanity audit JSONL.
|
||||
*
|
||||
* Writes events at `~/.gbrain/audit/content-sanity-YYYY-Www.jsonl`
|
||||
* (ISO-week rotation, mirrors `audit-slug-fallback.ts`). Built on the
|
||||
* shared `audit-writer.ts` primitive from v0.40.4.0; honors
|
||||
* `GBRAIN_AUDIT_DIR` env override.
|
||||
*
|
||||
* One stream, three event types:
|
||||
* - `hard_block` — assessor rejected the content; importFromContent
|
||||
* threw ContentSanityBlockError; page did NOT land.
|
||||
* - `soft_block` — assessor flagged oversize without junk-pattern;
|
||||
* page landed with `frontmatter.embed_skip` set; embedder will
|
||||
* skip on next sweep.
|
||||
* - `warn` — bytes > bytes_warn but neither hard- nor soft-block.
|
||||
* Page landed normally; stderr was emitted for operator visibility.
|
||||
*
|
||||
* Why one stream for all three:
|
||||
* The doctor check `content_sanity_audit_recent` aggregates by
|
||||
* reason + source_id over a 7-day window. Splitting events across
|
||||
* files would force doctor to walk multiple paths or risk dropping
|
||||
* one. One stream + a discriminator field stays simple.
|
||||
*
|
||||
* Best-effort writes. Audit-writer primitive emits stderr on failure
|
||||
* but never throws — ingest path continues regardless. Documented
|
||||
* caveat (Codex r1 #14): filesystem JSONL doesn't surface cleanly in
|
||||
* remote/server deployments. Operators on multi-host setups should
|
||||
* point `GBRAIN_AUDIT_DIR` at a shared filesystem. Doctor's message
|
||||
* for `content_sanity_audit_recent` explicitly names this limitation.
|
||||
*
|
||||
* Caller contract: the ingest gate calls `logContentSanityAssessment`
|
||||
* BEFORE branching on hard/soft block so every assessment that does
|
||||
* something user-visible gets a row. Idempotent re-imports are
|
||||
* intentionally logged again — the row count over time IS the signal
|
||||
* (catches "this source keeps producing the same junk").
|
||||
*/
|
||||
|
||||
import { createAuditWriter, computeIsoWeekFilename } from './audit-writer.ts';
|
||||
import type { ContentSanityResult } from '../content-sanity.ts';
|
||||
|
||||
export type ContentSanityEventType = 'hard_block' | 'soft_block' | 'warn';
|
||||
|
||||
export interface ContentSanityAuditEvent {
|
||||
ts: string;
|
||||
/** Which kind of assessment fired. */
|
||||
event_type: ContentSanityEventType;
|
||||
/** Page slug that was being imported. */
|
||||
slug: string;
|
||||
/** Source ID — multi-source brains need this for the doctor
|
||||
* aggregation. Empty string when caller doesn't know (rare). */
|
||||
source_id: string;
|
||||
/** UTF-8 byte length of compiled_truth + timeline at assessment. */
|
||||
bytes: number;
|
||||
/** Names of built-in patterns that matched (empty array on
|
||||
* soft_block / warn). */
|
||||
junk_pattern_matches: string[];
|
||||
/** Names of operator literals that matched. */
|
||||
literal_substring_matches: string[];
|
||||
/** Human-readable reason messages from the assessor result. Embeds
|
||||
* the PAGE_JUNK_PATTERN / PAGE_OVERSIZED prefix tokens. */
|
||||
reason_messages: string[];
|
||||
/** When true, the kill-switch was active and this event represents
|
||||
* a bypass — the page landed regardless. Lets doctor distinguish
|
||||
* "operator deliberately on a junk-tolerant mode" from "junk
|
||||
* actually landing." Default false. */
|
||||
bypass_active?: boolean;
|
||||
}
|
||||
|
||||
/** Filename matches the audit-writer's ISO-week convention. */
|
||||
export function computeContentSanityAuditFilename(now: Date = new Date()): string {
|
||||
return computeIsoWeekFilename('content-sanity', now);
|
||||
}
|
||||
|
||||
const writer = createAuditWriter<ContentSanityAuditEvent>({
|
||||
featureName: 'content-sanity',
|
||||
errorLabel: 'gbrain',
|
||||
errorMessagePrefix: 'content-sanity audit ',
|
||||
errorTrailer: '; import continues',
|
||||
});
|
||||
|
||||
/** Classify an assessor result into the audit event type. The same
|
||||
* result fires different events depending on caller context: a
|
||||
* hard-block assessment recorded WITH bypass active is still an
|
||||
* audit-worthy event but the page actually lands. The caller passes
|
||||
* `bypass` explicitly so this function stays pure. */
|
||||
function classifyEventType(
|
||||
result: ContentSanityResult,
|
||||
bypass: boolean,
|
||||
): ContentSanityEventType {
|
||||
if (bypass) {
|
||||
// Kill-switch override always logs as warn since the page lands.
|
||||
// Hard-block + bypass = "would have blocked but operator
|
||||
// overrode"; soft-block + bypass = same idea.
|
||||
return 'warn';
|
||||
}
|
||||
if (result.shouldHardBlock) return 'hard_block';
|
||||
if (result.shouldSkipEmbed) return 'soft_block';
|
||||
return 'warn';
|
||||
}
|
||||
|
||||
/**
|
||||
* Append a content-sanity assessment event. Called from the ingest
|
||||
* gate before any branch on the assessment result — every assessment
|
||||
* that does something user-visible gets recorded.
|
||||
*
|
||||
* Best-effort: audit-writer primitive stderr-warns on failure but
|
||||
* never throws. The gate proceeds either way.
|
||||
*/
|
||||
export function logContentSanityAssessment(
|
||||
slug: string,
|
||||
sourceId: string,
|
||||
result: ContentSanityResult,
|
||||
opts: { bypass?: boolean } = {},
|
||||
): void {
|
||||
const bypass = opts.bypass ?? false;
|
||||
const event_type = classifyEventType(result, bypass);
|
||||
// Skip rows that don't say anything: bytes under warn threshold AND
|
||||
// no patterns matched AND no bypass. The assessor result's reasons
|
||||
// array is empty in that case; we don't want every ingest of a
|
||||
// normal-size page to write a row.
|
||||
const hasReasons = result.reasons.length > 0 || result.reason_messages.length > 0;
|
||||
if (!hasReasons && !bypass) return;
|
||||
|
||||
writer.log({
|
||||
event_type,
|
||||
slug,
|
||||
source_id: sourceId,
|
||||
bytes: result.bytes,
|
||||
junk_pattern_matches: result.junk_pattern_matches,
|
||||
literal_substring_matches: result.literal_substring_matches,
|
||||
reason_messages: result.reason_messages,
|
||||
...(bypass ? { bypass_active: true } : {}),
|
||||
});
|
||||
}
|
||||
|
||||
/** Read recent events for the doctor `content_sanity_audit_recent`
|
||||
* check. 7-day default window; reads current + previous ISO week
|
||||
* files so a window straddling Monday-midnight stays covered. */
|
||||
export function readRecentContentSanityEvents(
|
||||
days = 7,
|
||||
now: Date = new Date(),
|
||||
): ContentSanityAuditEvent[] {
|
||||
return writer.readRecent(days, now);
|
||||
}
|
||||
|
||||
/** Summarize events for doctor's message. Groups by event_type +
|
||||
* source_id; counts pattern hits across all events. Returns a stable
|
||||
* shape so doctor can format consistently. */
|
||||
export interface ContentSanitySummary {
|
||||
total_events: number;
|
||||
by_type: { hard_block: number; soft_block: number; warn: number };
|
||||
by_source: Record<string, number>;
|
||||
/** Top junk-pattern names by hit count (sorted desc). */
|
||||
top_patterns: Array<{ name: string; count: number }>;
|
||||
}
|
||||
|
||||
export function summarizeContentSanityEvents(
|
||||
events: ReadonlyArray<ContentSanityAuditEvent>,
|
||||
): ContentSanitySummary {
|
||||
const by_type = { hard_block: 0, soft_block: 0, warn: 0 };
|
||||
const by_source: Record<string, number> = {};
|
||||
const patternCounts: Record<string, number> = {};
|
||||
|
||||
for (const ev of events) {
|
||||
by_type[ev.event_type]++;
|
||||
by_source[ev.source_id] = (by_source[ev.source_id] ?? 0) + 1;
|
||||
for (const name of ev.junk_pattern_matches) {
|
||||
patternCounts[name] = (patternCounts[name] ?? 0) + 1;
|
||||
}
|
||||
for (const name of ev.literal_substring_matches) {
|
||||
patternCounts[name] = (patternCounts[name] ?? 0) + 1;
|
||||
}
|
||||
}
|
||||
|
||||
const top_patterns = Object.entries(patternCounts)
|
||||
.map(([name, count]) => ({ name, count }))
|
||||
.sort((a, b) => b.count - a.count);
|
||||
|
||||
return {
|
||||
total_events: events.length,
|
||||
by_type,
|
||||
by_source,
|
||||
top_patterns,
|
||||
};
|
||||
}
|
||||
@@ -487,9 +487,27 @@ export async function scanBrainSources(
|
||||
dbPageCount = null;
|
||||
} else {
|
||||
// Race COUNT against the deadline so a hung query can't eat the budget.
|
||||
//
|
||||
// Boundary overshoot (+1ms): the post-await deadline check at line
|
||||
// ~512 uses `Date.now() >= deadline`. setTimeout fires AT OR AFTER
|
||||
// the requested delay, so in theory the check always passes. In
|
||||
// practice on heavily-loaded CI runners (8 parallel shards × 4
|
||||
// concurrent test files = ~32 concurrent bun processes) we saw
|
||||
// intermittent failures where the timer callback resolved
|
||||
// microseconds BEFORE the wall-clock boundary, leaving Date.now()
|
||||
// a tick below deadline and the skip-check evaluating false. The
|
||||
// src-a scan then ran on a populated dir before src-b's
|
||||
// between-source check caught up — causing
|
||||
// `firstSource.status === 'skipped'` to receive 'scanned'.
|
||||
//
|
||||
// Adding 1ms guarantees the timer fires past the deadline by at
|
||||
// least one millisecond regardless of runner timer drift. Cost is
|
||||
// 1ms additional wall-clock latency on hung COUNT queries, which
|
||||
// is operationally negligible. Flake repro:
|
||||
// https://github.com/garrytan/gbrain/actions/runs/77611667786
|
||||
dbPageCount = await Promise.race([
|
||||
opts.dbPageCountForSource(src.id),
|
||||
new Promise<null>(resolve => setTimeout(() => resolve(null), remainingMs)),
|
||||
new Promise<null>(resolve => setTimeout(() => resolve(null), remainingMs + 1)),
|
||||
]);
|
||||
}
|
||||
} else {
|
||||
|
||||
@@ -124,6 +124,33 @@ export interface GBrainConfig {
|
||||
*/
|
||||
search_embedding_column?: string;
|
||||
|
||||
/**
|
||||
* v0.41 content-sanity tunables. Read via file/env/DB plane (D1: lint
|
||||
* lifts to DB config when reachable). Resolution order:
|
||||
* env > file > DB > defaults from `src/core/content-sanity.ts`.
|
||||
*
|
||||
* Both lint AND ingest go through the same effective resolution so a
|
||||
* `gbrain config set content_sanity.bytes_block N` flips both surfaces
|
||||
* uniformly. CI without `~/.gbrain/` falls through to env/defaults.
|
||||
*/
|
||||
content_sanity?: {
|
||||
/** Stderr warn + lint `huge-page` rule fires above this (UTF-8 bytes
|
||||
* of compiled_truth + timeline). Default: 50_000. Env override:
|
||||
* `GBRAIN_PAGE_WARN_BYTES`. */
|
||||
bytes_warn?: number;
|
||||
/** Soft-block: page writes with `frontmatter.embed_skip` set but
|
||||
* embedder skips on next sweep. Default: 500_000. Env override:
|
||||
* `GBRAIN_PAGE_BLOCK_BYTES`. */
|
||||
bytes_block?: number;
|
||||
/** Master switch for the built-in junk-pattern set. Default: true.
|
||||
* Env override: `GBRAIN_NO_JUNK_PATTERNS=1` flips to false. */
|
||||
junk_patterns_enabled?: boolean;
|
||||
/** Master kill-switch for all sanity checks. When true, ingest emits
|
||||
* loud stderr per page but lets everything through. Default: false.
|
||||
* Env override: `GBRAIN_NO_SANITY=1` flips to true. */
|
||||
disabled?: boolean;
|
||||
};
|
||||
|
||||
/**
|
||||
* Thin-client mode (multi-topology v1). When set, this install does NOT
|
||||
* have a local DB; it talks to a remote `gbrain serve --http` over MCP.
|
||||
@@ -284,6 +311,37 @@ export function loadConfig(): GBrainConfig | null {
|
||||
? { remote_mcp: { ...fileConfig.remote_mcp, oauth_client_secret: process.env.GBRAIN_REMOTE_CLIENT_SECRET } }
|
||||
: {}),
|
||||
};
|
||||
|
||||
// v0.41 content-sanity env overrides. Built up as a sparse object so
|
||||
// env presence wins over file/DB only for the specific keys set,
|
||||
// matching the precedence pattern used elsewhere in loadConfig.
|
||||
// The env vars use natural names (GBRAIN_NO_SANITY=1 is more
|
||||
// operator-friendly than GBRAIN_CONTENT_SANITY_DISABLED=true).
|
||||
const envContentSanity: GBrainConfig['content_sanity'] = {};
|
||||
if (process.env.GBRAIN_PAGE_WARN_BYTES) {
|
||||
const n = parseInt(process.env.GBRAIN_PAGE_WARN_BYTES, 10);
|
||||
if (Number.isFinite(n) && n > 0) envContentSanity.bytes_warn = n;
|
||||
}
|
||||
if (process.env.GBRAIN_PAGE_BLOCK_BYTES) {
|
||||
const n = parseInt(process.env.GBRAIN_PAGE_BLOCK_BYTES, 10);
|
||||
if (Number.isFinite(n) && n > 0) envContentSanity.bytes_block = n;
|
||||
}
|
||||
if (process.env.GBRAIN_NO_JUNK_PATTERNS === '1') {
|
||||
envContentSanity.junk_patterns_enabled = false;
|
||||
}
|
||||
if (process.env.GBRAIN_NO_SANITY === '1') {
|
||||
envContentSanity.disabled = true;
|
||||
}
|
||||
// Only attach the field when at least one env var was set, so the
|
||||
// sparse-merge semantics elsewhere in loadConfigWithEngine work
|
||||
// (env presence => "this key already has a value, don't read DB").
|
||||
if (Object.keys(envContentSanity).length > 0) {
|
||||
(merged as GBrainConfig).content_sanity = {
|
||||
...(fileConfig?.content_sanity ?? {}),
|
||||
...envContentSanity,
|
||||
};
|
||||
}
|
||||
|
||||
return merged as GBrainConfig;
|
||||
}
|
||||
|
||||
@@ -381,6 +439,41 @@ export async function loadConfigWithEngine(
|
||||
if (merged.search_embedding_column === undefined && dbSearchEmbeddingColumn !== undefined) {
|
||||
merged.search_embedding_column = dbSearchEmbeddingColumn;
|
||||
}
|
||||
|
||||
// v0.41 content-sanity DB-plane merge (D1: lint lifts to read these
|
||||
// when reachable). Per-key sparse-merge: env/file wins per individual
|
||||
// key; DB fills the gaps. The container object is constructed only if
|
||||
// at least one source provides a value, mirroring the env-merge logic
|
||||
// in loadConfig().
|
||||
async function dbInt(key: string): Promise<number | undefined> {
|
||||
const v = await dbStr(key);
|
||||
if (v === undefined) return undefined;
|
||||
const n = parseInt(v, 10);
|
||||
return Number.isFinite(n) && n > 0 ? n : undefined;
|
||||
}
|
||||
const dbWarnBytes = await dbInt('content_sanity.bytes_warn');
|
||||
const dbBlockBytes = await dbInt('content_sanity.bytes_block');
|
||||
const dbJunkEnabled = await dbBool('content_sanity.junk_patterns_enabled');
|
||||
const dbSanityDisabled = await dbBool('content_sanity.disabled');
|
||||
|
||||
const existingCS = merged.content_sanity ?? {};
|
||||
const mergedCS: NonNullable<GBrainConfig['content_sanity']> = { ...existingCS };
|
||||
if (mergedCS.bytes_warn === undefined && dbWarnBytes !== undefined) {
|
||||
mergedCS.bytes_warn = dbWarnBytes;
|
||||
}
|
||||
if (mergedCS.bytes_block === undefined && dbBlockBytes !== undefined) {
|
||||
mergedCS.bytes_block = dbBlockBytes;
|
||||
}
|
||||
if (mergedCS.junk_patterns_enabled === undefined && dbJunkEnabled !== undefined) {
|
||||
mergedCS.junk_patterns_enabled = dbJunkEnabled;
|
||||
}
|
||||
if (mergedCS.disabled === undefined && dbSanityDisabled !== undefined) {
|
||||
mergedCS.disabled = dbSanityDisabled;
|
||||
}
|
||||
if (Object.keys(mergedCS).length > 0) {
|
||||
merged.content_sanity = mergedCS;
|
||||
}
|
||||
|
||||
return merged;
|
||||
}
|
||||
|
||||
@@ -475,6 +568,11 @@ export const KNOWN_CONFIG_KEYS: readonly string[] = [
|
||||
'emotional_weight.user_holder',
|
||||
// Cycle phase config
|
||||
'cycle.grade_takes.write_gstack_learnings',
|
||||
// Content sanity (v0.41)
|
||||
'content_sanity.bytes_warn',
|
||||
'content_sanity.bytes_block',
|
||||
'content_sanity.junk_patterns_enabled',
|
||||
'content_sanity.disabled',
|
||||
// Misc
|
||||
'artifacts_sync_mode',
|
||||
'cross_project_learnings',
|
||||
@@ -492,6 +590,7 @@ export const KNOWN_CONFIG_KEY_PREFIXES: readonly string[] = [
|
||||
'cycle.', // cycle.<phase>.*
|
||||
'embedding_columns.', // per-column overrides
|
||||
'provider_base_urls.', // per-provider base URL overrides
|
||||
'content_sanity.', // v0.41 content-sanity tunables
|
||||
];
|
||||
|
||||
export function saveConfig(config: GBrainConfig): void {
|
||||
|
||||
@@ -0,0 +1,151 @@
|
||||
/**
|
||||
* Operator-extensible literal-substring loader for the content-sanity gate.
|
||||
*
|
||||
* Reads `~/.gbrain/junk-substrings.txt` (operator-maintained) and returns
|
||||
* `OperatorLiteral[]` for `assessContentSanity` to evaluate alongside the
|
||||
* built-in junk patterns.
|
||||
*
|
||||
* Why literals, not regex (D16 + Codex r1 #10):
|
||||
* - JavaScript RegExp has no atomic groups or possessive quantifiers,
|
||||
* so the conventional ReDoS escape hatch isn't available. A reliable
|
||||
* catastrophic-backtracking shape detector is hard to implement.
|
||||
* - Literal substring matching covers the realistic operator use cases
|
||||
* ("add LinkedIn auth wall" = `"sign in to your account"`; "add
|
||||
* Reddit blocked" = `"you're being blocked from accessing"`). No
|
||||
* ReDoS surface. No regex parsing concerns.
|
||||
* - Built-in patterns stay regex because they're hand-vetted; never
|
||||
* run the linter against the operator file shape.
|
||||
*
|
||||
* Failure handling (D11):
|
||||
* - Missing file (ENOENT) → return empty list. Operator may not have
|
||||
* a file; most don't. Silent fall-through to built-ins only.
|
||||
* - Empty file or all-comments → empty list. Same outcome.
|
||||
* - Malformed line is structurally impossible: every non-comment line
|
||||
* is a valid literal substring. Even regex metacharacters in the
|
||||
* line stay literal at match time (no `new RegExp()` call).
|
||||
*
|
||||
* File format:
|
||||
* - Blank lines and `#`-prefixed comments ignored.
|
||||
* - Optional directives on the comment line IMMEDIATELY before each
|
||||
* literal: `# name=...`, `# applies_to=body|title|both`. Directives
|
||||
* persist until the next literal is read.
|
||||
* - One literal substring per non-comment line.
|
||||
*
|
||||
* Example file:
|
||||
*
|
||||
* # name=linkedin_auth_wall
|
||||
* # applies_to=body
|
||||
* Sign in to your account to continue
|
||||
*
|
||||
* # name=reddit_blocked
|
||||
* You're being blocked from accessing
|
||||
*
|
||||
* # name=substack_paywall
|
||||
* # applies_to=both
|
||||
* This post is for paid subscribers
|
||||
*
|
||||
* Best-effort: a malformed directive (e.g. `# applies_to=invalid`)
|
||||
* falls back to the default `'both'` scope without throwing — the
|
||||
* operator file is a soft input, not a config file.
|
||||
*
|
||||
* Default `applies_to` is `'both'` (title AND body head-slice).
|
||||
* Default `name` when none is declared is `operator_literal_<index>`
|
||||
* so audit JSONL has a stable identifier even for un-named entries.
|
||||
*/
|
||||
|
||||
import { existsSync, readFileSync } from 'fs';
|
||||
import type { OperatorLiteral } from './content-sanity.ts';
|
||||
|
||||
/** Path to the operator literals file. Honors `GBRAIN_HOME` via
|
||||
* `gbrainPath`. Resolved at load time so test fixtures can set
|
||||
* `GBRAIN_HOME` to a tempdir per the test-isolation conventions in
|
||||
* CLAUDE.md. */
|
||||
function resolveLiteralsPath(): string {
|
||||
// Lazy-import to avoid loading config.ts surface for the pure
|
||||
// assessor's consumers that only need built-ins.
|
||||
const { gbrainPath } = require('./config.ts');
|
||||
return gbrainPath('junk-substrings.txt');
|
||||
}
|
||||
|
||||
interface ParsedDirective {
|
||||
name?: string;
|
||||
applies_to?: 'body' | 'title' | 'both';
|
||||
}
|
||||
|
||||
/** Parse one comment line for known directives. Unknown directives
|
||||
* are ignored (operator file is soft input). Returns empty object
|
||||
* on no match. */
|
||||
function parseDirectiveLine(line: string): ParsedDirective {
|
||||
const stripped = line.replace(/^#\s*/, '').trim();
|
||||
// Match `key=value` shape. Allow multiple per line eventually if
|
||||
// someone asks; for now one per line is the documented format.
|
||||
const m = stripped.match(/^([a-z_]+)\s*=\s*(.+)$/i);
|
||||
if (!m) return {};
|
||||
const key = m[1].toLowerCase();
|
||||
const value = m[2].trim();
|
||||
if (key === 'name') return { name: value };
|
||||
if (key === 'applies_to') {
|
||||
if (value === 'body' || value === 'title' || value === 'both') {
|
||||
return { applies_to: value };
|
||||
}
|
||||
}
|
||||
return {};
|
||||
}
|
||||
|
||||
/**
|
||||
* Load operator literals. Pure function over file content — the
|
||||
* filesystem read is the only side effect. Returns empty list on
|
||||
* any failure mode (missing, unreadable, empty, all-comments).
|
||||
*
|
||||
* Tests pass `content` directly via `parseLiteralsContent` to bypass
|
||||
* the FS layer.
|
||||
*/
|
||||
export function loadOperatorLiterals(path?: string): OperatorLiteral[] {
|
||||
const resolved = path ?? resolveLiteralsPath();
|
||||
if (!existsSync(resolved)) return [];
|
||||
let raw: string;
|
||||
try {
|
||||
raw = readFileSync(resolved, 'utf-8');
|
||||
} catch {
|
||||
// Permission denied, transient FS error — treat as missing.
|
||||
return [];
|
||||
}
|
||||
return parseLiteralsContent(raw);
|
||||
}
|
||||
|
||||
/** Pure parser exposed for unit tests. */
|
||||
export function parseLiteralsContent(raw: string): OperatorLiteral[] {
|
||||
const literals: OperatorLiteral[] = [];
|
||||
let pending: ParsedDirective = {};
|
||||
let unnamedIndex = 0;
|
||||
|
||||
for (const line of raw.split('\n')) {
|
||||
const trimmed = line.trim();
|
||||
if (trimmed.length === 0) {
|
||||
// Blank line: directive scope resets so an empty line between
|
||||
// a directive block and a literal doesn't bind the directives.
|
||||
// (If you want sticky directives, omit the blank line.)
|
||||
pending = {};
|
||||
continue;
|
||||
}
|
||||
if (trimmed.startsWith('#')) {
|
||||
// Merge directives so a `# name=...` then `# applies_to=...`
|
||||
// pair both bind to the next literal.
|
||||
const parsed = parseDirectiveLine(trimmed);
|
||||
pending = { ...pending, ...parsed };
|
||||
continue;
|
||||
}
|
||||
// Non-comment, non-blank → literal substring line.
|
||||
const name = pending.name ?? `operator_literal_${unnamedIndex++}`;
|
||||
literals.push({
|
||||
name,
|
||||
substring: trimmed,
|
||||
applies_to: pending.applies_to ?? 'both',
|
||||
});
|
||||
// Consume the pending directives so they don't bind to a
|
||||
// subsequent literal unless re-declared.
|
||||
pending = {};
|
||||
}
|
||||
|
||||
return literals;
|
||||
}
|
||||
@@ -0,0 +1,316 @@
|
||||
/**
|
||||
* Content-sanity assessor for the ingest narrow waist.
|
||||
*
|
||||
* Pure module — no engine I/O, no filesystem access. Consumed by:
|
||||
* - `src/core/import-file.ts` — wires the gate into `importFromContent`
|
||||
* so EVERY ingestion path inherits it (sync, gbrain import, put_page
|
||||
* MCP op, gbrain capture, POST /ingest webhook via ingest_capture).
|
||||
* - `src/commands/lint.ts` — surfaces matching content as `huge-page`
|
||||
* + `scraper-junk` lint rules so brain-authors see issues in their
|
||||
* source repo before sync.
|
||||
* - `src/commands/doctor.ts` — surfaces historical inventory via
|
||||
* `oversized_pages`, `scraper_junk_pages`, and
|
||||
* `content_sanity_audit_recent` checks.
|
||||
* - `src/commands/sources.ts` `audit` subcommand — dry-run scan of a
|
||||
* source repo's `local_path` reporting would-blocks + size
|
||||
* distribution without touching the DB.
|
||||
*
|
||||
* Two failure modes treated differently (D14-D16 + D6-D9 review trail):
|
||||
* - **Scraper junk** (built-in pattern OR operator literal match) →
|
||||
* HARD-BLOCK. Caller is expected to `throw new ContentSanityBlockError(...)`.
|
||||
* Existing exception-handling at every wrapper site (import.ts/cli.ts,
|
||||
* operations.ts put_page, sync.ts:929 catch) fires correctly through
|
||||
* this single throw point. No new status vocabulary required.
|
||||
* - **Oversize alone** (bytes > block_bytes WITHOUT junk-pattern match) →
|
||||
* SOFT-BLOCK. Caller writes the page with `frontmatter.embed_skip` set
|
||||
* via `buildEmbedSkipMarker` from `src/core/embed-skip.ts`. The embedder
|
||||
* skips on next sweep at all 5 wiring sites. Page lands so legitimate
|
||||
* large content (2MB conversation transcripts) is preserved.
|
||||
*
|
||||
* Bytes are measured against `compiled_truth + timeline` (the parsed body
|
||||
* after `parseMarkdown` splits at the timeline sentinel). Frontmatter is
|
||||
* NOT counted — the operational concern is the embed-pipeline-input size.
|
||||
* Codex r2 #7 caught the earlier compiled_truth-only design that missed
|
||||
* pages with huge timeline sections.
|
||||
*
|
||||
* Pattern set is hand-vetted regex evaluated against `title` + the first
|
||||
* ~2KB of body content. 6 built-in patterns (D3 dropped a shape-based
|
||||
* `empty_body_with_source_url` rule because legitimate stub pages with
|
||||
* `source_url` frontmatter were getting flagged). Operator literals come
|
||||
* in via `extra_literals` from `src/core/content-sanity-literals.ts`
|
||||
* (literal substrings only — no regex per Codex r1 #10 ReDoS concerns).
|
||||
*
|
||||
* The kill-switch (`GBRAIN_NO_SANITY=1` / `content_sanity.disabled: true`)
|
||||
* is honored by the CALLER (import-file.ts), not by this module. The
|
||||
* assessor stays pure so unit tests don't need env mutation.
|
||||
*/
|
||||
|
||||
/** Maximum number of body bytes scanned for pattern matches. The body
|
||||
* is sliced to this size before regex/substring evaluation so pattern
|
||||
* cost stays O(2KB) regardless of page size. Cloudflare/CAPTCHA junk
|
||||
* pages have their telltale text at the top — 2KB covers the realistic
|
||||
* cases. Operators who need deeper scanning can override via env. */
|
||||
export const SCAN_HEAD_BYTES = 2048;
|
||||
|
||||
/** Default warn threshold. Operator override via
|
||||
* `content_sanity.bytes_warn` config key or `GBRAIN_PAGE_WARN_BYTES`
|
||||
* env var. Above this, lint surfaces `huge-page` rule + ingest emits
|
||||
* stderr warn. Page still writes. */
|
||||
export const DEFAULT_BYTES_WARN = 50_000;
|
||||
|
||||
/** Default block threshold. Operator override via
|
||||
* `content_sanity.bytes_block` config key or `GBRAIN_PAGE_BLOCK_BYTES`
|
||||
* env var. Above this, page writes but `frontmatter.embed_skip` is set
|
||||
* and the embedder skips on next sweep. Page is still queryable; just
|
||||
* not searchable until manually re-embedded or split. */
|
||||
export const DEFAULT_BYTES_BLOCK = 500_000;
|
||||
|
||||
/** Tag added to the start of `reasons` and to error messages so
|
||||
* `src/core/sync.ts:classifyErrorCode` can group hard-blocks under one
|
||||
* code without needing a structured field in the failure shape. The
|
||||
* classifier matches this token via regex. */
|
||||
export const PAGE_JUNK_PATTERN_CODE = 'PAGE_JUNK_PATTERN';
|
||||
|
||||
export type SanityTripReason =
|
||||
| 'oversize_warn' // informational: bytes > bytes_warn but page lands normally
|
||||
| 'oversize_block' // soft-block: write with frontmatter.embed_skip
|
||||
| 'junk_pattern' // hard-block: throw ContentSanityBlockError
|
||||
| 'literal_substring'; // hard-block: operator-supplied literal hit
|
||||
|
||||
export interface JunkPattern {
|
||||
/** Stable identifier surfaced in error messages, audit JSONL, and
|
||||
* doctor output. Snake_case. Treat as a stable contract — renaming
|
||||
* one means rewriting downstream consumers. */
|
||||
name: string;
|
||||
/** Case-insensitive regex. Evaluated against the chosen scope; cost
|
||||
* is bounded by SCAN_HEAD_BYTES. */
|
||||
pattern: RegExp;
|
||||
/** Where the pattern applies. Defaults to 'both' (title AND body
|
||||
* head-slice). 'title' is useful for error-page-title detection;
|
||||
* 'body' for content-shape patterns. */
|
||||
applies_to?: 'body' | 'title' | 'both';
|
||||
}
|
||||
|
||||
export interface OperatorLiteral {
|
||||
name: string;
|
||||
/** Literal substring. Case-insensitive match via `.toLowerCase()`.
|
||||
* Regex meta-characters in the substring are matched literally. */
|
||||
substring: string;
|
||||
applies_to?: 'body' | 'title' | 'both';
|
||||
}
|
||||
|
||||
export interface ContentSanityResult {
|
||||
/** UTF-8 byte length of `compiled_truth + timeline`. Frontmatter is
|
||||
* NOT included (the operational concern is embed-pipeline input). */
|
||||
bytes: number;
|
||||
/** True when bytes > effective bytes_block. Drives soft-block. */
|
||||
oversize: boolean;
|
||||
/** Names of built-in patterns that matched (zero or more). */
|
||||
junk_pattern_matches: string[];
|
||||
/** Names of operator literals that matched (zero or more). */
|
||||
literal_substring_matches: string[];
|
||||
/** Ordered list of trip reasons. `oversize` first when present,
|
||||
* then `junk_pattern`, then `literal_substring`. Stable across
|
||||
* releases so consumers can pattern-match. */
|
||||
reasons: SanityTripReason[];
|
||||
/** Human-readable messages per reason. Each prefixed with the stable
|
||||
* code token (`PAGE_JUNK_PATTERN:` or `PAGE_OVERSIZED:`) so the
|
||||
* caller can compose them into an error message that `classifyErrorCode`
|
||||
* picks up via regex. */
|
||||
reason_messages: string[];
|
||||
/** True when any junk pattern or operator literal matched. Caller
|
||||
* should throw `ContentSanityBlockError` when this is set. Note that
|
||||
* oversize alone does NOT trigger this — that's a soft-block. */
|
||||
shouldHardBlock: boolean;
|
||||
/** True when oversize without hard-block. Caller should write the
|
||||
* page with `frontmatter.embed_skip` set so the embedder skips. */
|
||||
shouldSkipEmbed: boolean;
|
||||
}
|
||||
|
||||
/** Built-in pattern set. Hand-vetted regex compiled once at module
|
||||
* load. Adding a pattern: include a stable `name`, a case-insensitive
|
||||
* regex with `i` flag, and document the real-world example in plain
|
||||
* prose so future reviewers know what shape it catches. */
|
||||
export const BUILT_IN_JUNK_PATTERNS: ReadonlyArray<JunkPattern> = Object.freeze([
|
||||
// Cloudflare interstitials — the dominant scraper-junk class.
|
||||
{
|
||||
name: 'cloudflare_attention_required',
|
||||
pattern: /attention required.*cloudflare/i,
|
||||
applies_to: 'both',
|
||||
},
|
||||
{
|
||||
name: 'cloudflare_just_a_moment',
|
||||
// Both signals required — "just a moment..." alone fires on
|
||||
// legitimate writing; the cdn-cgi/challenge URL is the discriminator.
|
||||
pattern: /just a moment\.\.\.[\s\S]{0,500}cdn-cgi\/challenge-platform/i,
|
||||
applies_to: 'body',
|
||||
},
|
||||
{
|
||||
name: 'cloudflare_ray_id',
|
||||
pattern: /cloudflare ray id:/i,
|
||||
applies_to: 'body',
|
||||
},
|
||||
// Generic 403 / blocked-access pages.
|
||||
{
|
||||
name: 'access_denied',
|
||||
pattern: /^\s*access denied\b/im,
|
||||
applies_to: 'both',
|
||||
},
|
||||
// CAPTCHA gates.
|
||||
{
|
||||
name: 'captcha_required',
|
||||
pattern: /verify you are (a )?human|captcha required|please complete the security check/i,
|
||||
applies_to: 'both',
|
||||
},
|
||||
// Bare error-page titles. Anchored so the title is exclusively the
|
||||
// error code — a thoughtful page ABOUT 404 errors won't trip.
|
||||
{
|
||||
name: 'error_page_title',
|
||||
pattern: /^(403|404|500|502|503|error \d{3}|page not found)\s*$/i,
|
||||
applies_to: 'title',
|
||||
},
|
||||
]);
|
||||
|
||||
/** Tagged error thrown from `importFromContent` on hard-block. The
|
||||
* existing exception-handling at every wrapper site catches it and
|
||||
* surfaces a non-zero exit (import), MCP error envelope (put_page),
|
||||
* or sync-failure record. Message embeds `PAGE_JUNK_PATTERN:` so
|
||||
* `classifyErrorCode` picks it up via regex without needing a
|
||||
* structured `error_code` field on `ImportResult`. */
|
||||
export class ContentSanityBlockError extends Error {
|
||||
readonly code = PAGE_JUNK_PATTERN_CODE;
|
||||
readonly result: ContentSanityResult;
|
||||
|
||||
constructor(result: ContentSanityResult) {
|
||||
// Compose message from the result's reason messages. The
|
||||
// `PAGE_JUNK_PATTERN:` prefix is already in each reason_message
|
||||
// so the classifier regex hits regardless of which reasons fired.
|
||||
const summary = result.reason_messages.join('; ');
|
||||
super(`Content rejected by sanity gate: ${summary}`);
|
||||
this.name = 'ContentSanityBlockError';
|
||||
this.result = result;
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Assess a parsed page against the size + junk-pattern surface.
|
||||
*
|
||||
* Pure function — same inputs always produce the same outputs. Caller
|
||||
* decides what to do with the result (throw on shouldHardBlock, set
|
||||
* embed_skip frontmatter on shouldSkipEmbed, write normally otherwise).
|
||||
*
|
||||
* The body bytes input is `compiled_truth + timeline` (Codex r2 #7
|
||||
* fix: pages can have huge timeline sections that would evade a
|
||||
* compiled_truth-only check). Frontmatter is NOT counted.
|
||||
*/
|
||||
export function assessContentSanity(opts: {
|
||||
/** Post-parseMarkdown body (before timeline split). */
|
||||
compiled_truth: string;
|
||||
/** Post-parseMarkdown timeline section (empty string if no sentinel). */
|
||||
timeline: string;
|
||||
/** Post-parseMarkdown title. Some patterns key on title alone. */
|
||||
title: string;
|
||||
/** Effective warn threshold; defaults to DEFAULT_BYTES_WARN. */
|
||||
bytes_warn?: number;
|
||||
/** Effective block threshold; defaults to DEFAULT_BYTES_BLOCK. */
|
||||
bytes_block?: number;
|
||||
/** Operator-supplied literal substrings loaded from
|
||||
* `~/.gbrain/junk-substrings.txt` via `src/core/content-sanity-literals.ts`.
|
||||
* Empty array (default) means built-ins only. */
|
||||
extra_literals?: ReadonlyArray<OperatorLiteral>;
|
||||
}): ContentSanityResult {
|
||||
const bytes_warn = opts.bytes_warn ?? DEFAULT_BYTES_WARN;
|
||||
const bytes_block = opts.bytes_block ?? DEFAULT_BYTES_BLOCK;
|
||||
|
||||
// Bytes measured against the parsed body (compiled_truth + timeline).
|
||||
// Buffer.byteLength counts UTF-8 bytes the same way the doctor's
|
||||
// octet_length() does at the DB layer, so the two surfaces agree on
|
||||
// the same page (D2 parity).
|
||||
const body = opts.compiled_truth + (opts.timeline ? '\n' + opts.timeline : '');
|
||||
const bytes = Buffer.byteLength(body, 'utf-8');
|
||||
const oversize = bytes > bytes_block;
|
||||
|
||||
// Head-slice for pattern evaluation. Cost stays O(SCAN_HEAD_BYTES)
|
||||
// regardless of body size. Lowercased once so substring matching
|
||||
// doesn't repeat the lowercase per literal.
|
||||
const bodyHead = body.slice(0, SCAN_HEAD_BYTES);
|
||||
const bodyHeadLower = bodyHead.toLowerCase();
|
||||
const titleLower = opts.title.toLowerCase();
|
||||
|
||||
const junk_pattern_matches: string[] = [];
|
||||
for (const p of BUILT_IN_JUNK_PATTERNS) {
|
||||
const scope = p.applies_to ?? 'both';
|
||||
let matched = false;
|
||||
if (scope === 'title' || scope === 'both') {
|
||||
if (p.pattern.test(opts.title)) matched = true;
|
||||
}
|
||||
if (!matched && (scope === 'body' || scope === 'both')) {
|
||||
if (p.pattern.test(bodyHead)) matched = true;
|
||||
}
|
||||
if (matched) junk_pattern_matches.push(p.name);
|
||||
}
|
||||
|
||||
const literal_substring_matches: string[] = [];
|
||||
if (opts.extra_literals && opts.extra_literals.length > 0) {
|
||||
for (const lit of opts.extra_literals) {
|
||||
const scope = lit.applies_to ?? 'both';
|
||||
const needle = lit.substring.toLowerCase();
|
||||
if (needle.length === 0) continue;
|
||||
let matched = false;
|
||||
if (scope === 'title' || scope === 'both') {
|
||||
if (titleLower.includes(needle)) matched = true;
|
||||
}
|
||||
if (!matched && (scope === 'body' || scope === 'both')) {
|
||||
if (bodyHeadLower.includes(needle)) matched = true;
|
||||
}
|
||||
if (matched) literal_substring_matches.push(lit.name);
|
||||
}
|
||||
}
|
||||
|
||||
const reasons: SanityTripReason[] = [];
|
||||
const reason_messages: string[] = [];
|
||||
const shouldHardBlock =
|
||||
junk_pattern_matches.length > 0 || literal_substring_matches.length > 0;
|
||||
|
||||
// Reason ordering: block-level oversize first (so a soft-block that
|
||||
// ALSO hits a junk pattern documents both), then junk_pattern, then
|
||||
// literal. Warn-level oversize emitted only when no block-level fired.
|
||||
if (oversize) {
|
||||
reasons.push('oversize_block');
|
||||
reason_messages.push(`PAGE_OVERSIZED: body ${bytes} bytes exceeds ${bytes_block} byte block threshold`);
|
||||
} else if (bytes > bytes_warn) {
|
||||
// Warn tier: bytes between bytes_warn and bytes_block. Page lands
|
||||
// normally; consumer emits stderr and (when configured) lint surfaces
|
||||
// `huge-page` rule. This row IS auditable so doctor's recent-events
|
||||
// check can surface flow-rate signal ("operators crossing warn often").
|
||||
reasons.push('oversize_warn');
|
||||
reason_messages.push(`PAGE_OVERSIZE_WARN: body ${bytes} bytes exceeds ${bytes_warn} byte warn threshold`);
|
||||
}
|
||||
if (junk_pattern_matches.length > 0) {
|
||||
reasons.push('junk_pattern');
|
||||
reason_messages.push(
|
||||
`${PAGE_JUNK_PATTERN_CODE}: matched built-in pattern(s): ${junk_pattern_matches.join(', ')}`,
|
||||
);
|
||||
}
|
||||
if (literal_substring_matches.length > 0) {
|
||||
reasons.push('literal_substring');
|
||||
reason_messages.push(
|
||||
`${PAGE_JUNK_PATTERN_CODE}: matched operator literal(s): ${literal_substring_matches.join(', ')}`,
|
||||
);
|
||||
}
|
||||
|
||||
return {
|
||||
bytes,
|
||||
oversize,
|
||||
junk_pattern_matches,
|
||||
literal_substring_matches,
|
||||
reasons,
|
||||
reason_messages,
|
||||
// shouldSkipEmbed: oversize past block threshold but NOT also hard-block.
|
||||
// When BOTH fire (the 890K Cloudflare dump case), hard-block wins and
|
||||
// the page never lands. Embed-skip is reserved for the legitimate
|
||||
// large-content case.
|
||||
shouldHardBlock,
|
||||
shouldSkipEmbed: oversize && !shouldHardBlock,
|
||||
};
|
||||
}
|
||||
@@ -0,0 +1,129 @@
|
||||
/**
|
||||
* Embed-skip predicate: the single source of truth for "should this
|
||||
* page be skipped during embedding?"
|
||||
*
|
||||
* Why a shared module (D4):
|
||||
* gbrain has 5 sites that filter the stale-chunk / all-pages query
|
||||
* for embedding:
|
||||
*
|
||||
* 1. src/commands/embed.ts:350 (--stale CLI path)
|
||||
* 2. src/commands/embed.ts:355 (--all CLI path) — D8 catches this
|
||||
* too; the `--all` walk re-embeds every page from scratch and
|
||||
* must honor the skip flag like `--stale` does.
|
||||
* 3. src/core/embed-stale.ts:90 (Minion helper)
|
||||
* 4. src/core/postgres-engine.ts (listStaleChunks/countStaleChunks)
|
||||
* 5. src/core/pglite-engine.ts equivalent
|
||||
*
|
||||
* Inline-filtering across 5 sites is the exact bug class gbrain has
|
||||
* been bitten by repeatedly — see CLAUDE.md `cjk.ts`, `sql-ranking.ts`,
|
||||
* `audit-writer.ts` for sibling shared modules. Extracting the
|
||||
* predicate here means the 5 sites all import from one place.
|
||||
*
|
||||
* Two surfaces:
|
||||
* - JS predicate `isEmbedSkipped(frontmatter)` for callers that have
|
||||
* in-memory page objects (CLI walk paths).
|
||||
* - SQL fragment `EMBED_SKIP_FILTER_FRAGMENT` for callers that need
|
||||
* to splice into a postgres-js / PGLite `sql\`...\`` template.
|
||||
* Both engines use the standard JSONB `?` existence operator;
|
||||
* PGLite (PostgreSQL 17.5 in WASM) supports the full JSONB
|
||||
* operator set, so one fragment works for both.
|
||||
*
|
||||
* Frontmatter writer:
|
||||
* - `buildEmbedSkipMarker(bytes)` produces the canonical marker
|
||||
* object. Callers `Object.assign` it onto `parsed.frontmatter` so
|
||||
* it persists into the page write. Stable schema means the JS
|
||||
* predicate and the SQL existence check both target the same key
|
||||
* name (`embed_skip`) — drift between writer and reader is the
|
||||
* bug class we're preventing.
|
||||
*
|
||||
* Marker shape rationale:
|
||||
* The marker is an OBJECT (not a bare bool) so the operator can see
|
||||
* WHY the page was skipped + WHEN at a glance via `get_page`. The
|
||||
* SQL existence check (`frontmatter ? 'embed_skip'`) hits regardless
|
||||
* of marker contents — JSONB key-existence semantics — so future
|
||||
* versions can extend the marker shape without invalidating the
|
||||
* filter.
|
||||
*
|
||||
* v0.42 follow-up: promote to schema column `pages.embed_skipped_at`
|
||||
* + partial index. Single change site (this module). For v0.41 the
|
||||
* JSONB approach is acceptable because the skipped-page subset stays
|
||||
* small (operator surfaces via doctor and either splits or accepts).
|
||||
*/
|
||||
|
||||
/** The frontmatter key name. Treat as a stable contract — renaming
|
||||
* this means rewriting every consumer of the skip semantic. */
|
||||
export const EMBED_SKIP_KEY = 'embed_skip';
|
||||
|
||||
/** SQL fragment that excludes pages with the embed-skip marker.
|
||||
* Callers must already JOIN `pages` (aliased as `p`) — the bare
|
||||
* `content_chunks` query has no access to frontmatter and needs the
|
||||
* join added regardless.
|
||||
*
|
||||
* Use via `sql.unsafe()` or equivalent fragment-splice:
|
||||
*
|
||||
* const filter = EMBED_SKIP_FILTER_FRAGMENT;
|
||||
* await sql`SELECT ... FROM content_chunks cc
|
||||
* JOIN pages p ON p.id = cc.page_id
|
||||
* WHERE cc.embedding IS NULL AND ${sql.unsafe(filter)}`;
|
||||
*
|
||||
* The fragment uses the JSONB `?` existence operator: returns true
|
||||
* when the JSONB object contains the key `'embed_skip'` at the top
|
||||
* level. Works identically on Postgres (real) and PGLite (PostgreSQL
|
||||
* 17.5 in WASM). The `NOT` negates so we KEEP rows that DON'T have
|
||||
* the marker. */
|
||||
export const EMBED_SKIP_FILTER_FRAGMENT =
|
||||
`NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? '${EMBED_SKIP_KEY}')`;
|
||||
|
||||
export interface EmbedSkipMarker {
|
||||
/** Why the page was skipped. v0.41 ships only `'oversized'`; future
|
||||
* reasons (e.g. `'chunk_token_limit'` from the deferred v0.42
|
||||
* chunk-level quarantine) extend this enum. */
|
||||
reason: 'oversized';
|
||||
/** Body bytes at the time of assessment. Operator visibility: at a
|
||||
* glance, see how oversized the page is. */
|
||||
bytes: number;
|
||||
/** ISO 8601 timestamp at assessment time. Tells the operator when
|
||||
* the skip was first applied (page may have been edited later). */
|
||||
assessed_at: string;
|
||||
}
|
||||
|
||||
/** Build the canonical marker object. Callers spread it onto the
|
||||
* frontmatter before write:
|
||||
*
|
||||
* parsed.frontmatter[EMBED_SKIP_KEY] = buildEmbedSkipMarker(bytes);
|
||||
*
|
||||
* The marker is OBJECT-shaped (not bare true) so `get_page` shows
|
||||
* the operator why + when at a glance. */
|
||||
export function buildEmbedSkipMarker(bytes: number, now: Date = new Date()): EmbedSkipMarker {
|
||||
return {
|
||||
reason: 'oversized',
|
||||
bytes,
|
||||
assessed_at: now.toISOString(),
|
||||
};
|
||||
}
|
||||
|
||||
/** JS-side predicate for in-memory page objects. Returns true when the
|
||||
* frontmatter has the embed-skip key set to any non-null value.
|
||||
*
|
||||
* Accepts `null`/`undefined` frontmatter (some paths construct page
|
||||
* objects without one) and returns false — no frontmatter means no
|
||||
* skip marker.
|
||||
*
|
||||
* Mirrors the SQL fragment's semantics: key-existence is the trigger;
|
||||
* marker contents are diagnostic, not functional. A future marker
|
||||
* shape change doesn't break this predicate. */
|
||||
export function isEmbedSkipped(frontmatter: Record<string, unknown> | null | undefined): boolean {
|
||||
if (!frontmatter) return false;
|
||||
const value = frontmatter[EMBED_SKIP_KEY];
|
||||
return value !== undefined && value !== null;
|
||||
}
|
||||
|
||||
/** JS-side filter for arrays of in-memory page objects. Returns a new
|
||||
* array with embed-skipped pages excluded. Mirrors the SQL filter
|
||||
* for callers that walk pages JS-side (e.g. `gbrain embed --all`
|
||||
* walks pages directly rather than going through listStaleChunks). */
|
||||
export function filterOutEmbedSkipped<T extends { frontmatter?: Record<string, unknown> | null }>(
|
||||
pages: ReadonlyArray<T>,
|
||||
): T[] {
|
||||
return pages.filter((p) => !isEmbedSkipped(p.frontmatter ?? null));
|
||||
}
|
||||
+133
-14
@@ -15,6 +15,11 @@ import { computeEffectiveDate } from './effective-date.ts';
|
||||
import { MARKDOWN_CHUNKER_VERSION } from './chunkers/recursive.ts';
|
||||
import { logSlugFallback } from './audit-slug-fallback.ts';
|
||||
import { resolveContextualRetrievalMode } from './contextual-retrieval-resolver.ts';
|
||||
import { assessContentSanity, ContentSanityBlockError } from './content-sanity.ts';
|
||||
import { loadOperatorLiterals } from './content-sanity-literals.ts';
|
||||
import { logContentSanityAssessment } from './audit/content-sanity-audit.ts';
|
||||
import { isEmbedSkipped, buildEmbedSkipMarker, EMBED_SKIP_KEY } from './embed-skip.ts';
|
||||
import { loadConfig, loadConfigWithEngine } from './config.ts';
|
||||
import {
|
||||
buildContextualPrefix,
|
||||
modeRequiresHaiku,
|
||||
@@ -268,6 +273,112 @@ export async function importFromContent(
|
||||
|
||||
const parsed = parseMarkdown(content, slug + '.md', { activePack: opts.activePack });
|
||||
|
||||
// v0.41 content-sanity gate. Runs AFTER parseMarkdown so the assessor
|
||||
// sees the parsed body (compiled_truth + timeline), title, and
|
||||
// frontmatter; runs BEFORE the hash compute so a soft-block that
|
||||
// mutates frontmatter (sets `embed_skip`) reaches the existing hash
|
||||
// calculation and the page write doesn't short-circuit on hash equality.
|
||||
//
|
||||
// Three outcomes:
|
||||
// - kill-switch active (`content_sanity.disabled === true` /
|
||||
// `GBRAIN_NO_SANITY=1`) → assess + audit with bypass flag, emit
|
||||
// loud stderr per offending ingest, but let everything through.
|
||||
// - hard-block (junk pattern OR operator literal) → THROW
|
||||
// ContentSanityBlockError. Existing exception flow at every
|
||||
// wrapper site (import.ts errors counter, put_page MCP envelope,
|
||||
// sync.ts:929 failure record) fires correctly through this single
|
||||
// throw point. classifyErrorCode picks up the PAGE_JUNK_PATTERN
|
||||
// prefix in the error message and groups in sync-failures.jsonl.
|
||||
// - soft-block (oversize WITHOUT junk-pattern hit) → mutate
|
||||
// frontmatter to embed `embed_skip` marker. Existing chunking
|
||||
// block guards on `isEmbedSkipped(frontmatter)` so chunks stays
|
||||
// empty; the existing `tx.deleteChunks` at the empty-chunks
|
||||
// branch fires to purge old chunks (D9 transition invariant).
|
||||
//
|
||||
// Effective config: env > file > DB > defaults. The DB-plane lift
|
||||
// adds ~4 SQL round-trips per import (one per content_sanity.* key);
|
||||
// acceptable for the per-page cost since the gate runs at most once
|
||||
// per ingest. Power-users with 10K-file syncs who care about this
|
||||
// overhead can set the keys via env vars instead and skip the DB read.
|
||||
{
|
||||
const baseCfg = loadConfig();
|
||||
let effectiveCfg = baseCfg;
|
||||
try {
|
||||
// loadConfigWithEngine merges DB-plane content_sanity.* on top
|
||||
// of file/env. Wrapped in try/catch so a transient engine error
|
||||
// doesn't kill the import — the gate falls back to file/env
|
||||
// values (which include defaults via the assessor itself).
|
||||
effectiveCfg = await loadConfigWithEngine(engine, baseCfg);
|
||||
} catch (err) {
|
||||
const msg = err instanceof Error ? err.message : String(err);
|
||||
process.stderr.write(`[gbrain] content-sanity: DB config lift failed (${msg}); falling back to file/env\n`);
|
||||
}
|
||||
const cs = effectiveCfg?.content_sanity ?? {};
|
||||
// GBRAIN_NO_SANITY=1 fast-path: loadConfig() returns null when
|
||||
// there's no `~/.gbrain/config.json` AND no DATABASE_URL env var
|
||||
// (e.g., fresh PGLite-only setups, hermetic tests). The merged
|
||||
// content_sanity block never carries `disabled` in that case. Read
|
||||
// the kill-switch env directly so it works regardless of whether
|
||||
// any other config plumbing fired. Same direct-env-check pattern
|
||||
// applies to the patterns_enabled flip below.
|
||||
const sanityDisabled =
|
||||
cs.disabled === true || process.env.GBRAIN_NO_SANITY === '1';
|
||||
const extra_literals =
|
||||
cs.junk_patterns_enabled !== false && !sanityDisabled ? loadOperatorLiterals() : [];
|
||||
const sanityResult = assessContentSanity({
|
||||
compiled_truth: parsed.compiled_truth,
|
||||
timeline: parsed.timeline ?? '',
|
||||
title: parsed.title,
|
||||
bytes_warn: cs.bytes_warn,
|
||||
bytes_block: cs.bytes_block,
|
||||
extra_literals,
|
||||
});
|
||||
// Audit BEFORE branching so hard-block / soft-block / warn / bypass
|
||||
// ALL get a row in the JSONL. The audit module's own gate
|
||||
// suppresses no-op rows (bytes below warn, no patterns, no bypass).
|
||||
logContentSanityAssessment(slug, sourceId ?? 'default', sanityResult, {
|
||||
bypass: sanityDisabled,
|
||||
});
|
||||
|
||||
if (sanityDisabled) {
|
||||
// Kill-switch active: loud stderr per offending ingest. Operator
|
||||
// explicitly opted into the bypass and gets noisy feedback every
|
||||
// time it fires so they remember the gate is off.
|
||||
if (sanityResult.shouldHardBlock || sanityResult.shouldSkipEmbed) {
|
||||
process.stderr.write(
|
||||
`[gbrain] content-sanity bypass (GBRAIN_NO_SANITY=1): ${slug} — ${sanityResult.reason_messages.join('; ')}\n`,
|
||||
);
|
||||
}
|
||||
} else {
|
||||
if (sanityResult.shouldHardBlock) {
|
||||
// Single throw point. Existing exception flow at every wrapper
|
||||
// site fires correctly. Caller-side semantics:
|
||||
// - import.ts → runImport's catch increments errors → non-zero exit
|
||||
// - put_page MCP → operations.ts try/catch → OperationError envelope
|
||||
// - sync.ts → existing catch at :929 → records failure with classified code
|
||||
throw new ContentSanityBlockError(sanityResult);
|
||||
}
|
||||
if (sanityResult.shouldSkipEmbed) {
|
||||
// Soft-block: mutate frontmatter so the embed_skip marker
|
||||
// persists into the page write. The existing chunking block
|
||||
// below guards on isEmbedSkipped → chunks stays empty →
|
||||
// existing tx.deleteChunks fires to purge old chunks
|
||||
// (D9 transition invariant — old chunks were searchable
|
||||
// against stale content; deleting them maintains the
|
||||
// invariant that embed_skip means "no live chunks").
|
||||
parsed.frontmatter[EMBED_SKIP_KEY] = buildEmbedSkipMarker(sanityResult.bytes);
|
||||
process.stderr.write(
|
||||
`[gbrain] content-sanity soft-block: ${slug} (${sanityResult.bytes} bytes) — page lands, embedding skipped\n`,
|
||||
);
|
||||
} else if (sanityResult.reasons.includes('oversize_warn')) {
|
||||
// Warn tier: page lands normally; lint surface picks up too.
|
||||
process.stderr.write(
|
||||
`[gbrain] content-sanity warn: ${slug} (${sanityResult.bytes} bytes) — exceeds warn threshold, consider splitting\n`,
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// v0.39.3.0 CV8 — DB content_hash excludes timestamp-bearing frontmatter
|
||||
// keys so identical body content from `gbrain capture` (which stamps
|
||||
// `captured_at` and `ingested_at` per call) produces a stable hash.
|
||||
@@ -314,24 +425,32 @@ export async function importFromContent(
|
||||
return { slug, status: 'skipped', chunks: 0, parsedPage };
|
||||
}
|
||||
|
||||
// Chunk compiled_truth and timeline
|
||||
// Chunk compiled_truth and timeline.
|
||||
// v0.41 content-sanity soft-block: if the gate marked this page as
|
||||
// embed-skipped (oversize without junk-pattern), skip chunking
|
||||
// entirely. The empty-chunks branch in the transaction below
|
||||
// triggers tx.deleteChunks(slug) which purges any pre-existing
|
||||
// chunks (D9 transition invariant: embed_skip means no live chunks).
|
||||
const chunks: ChunkInput[] = [];
|
||||
if (parsed.compiled_truth.trim()) {
|
||||
for (const c of chunkText(parsed.compiled_truth)) {
|
||||
chunks.push({ chunk_index: chunks.length, chunk_text: c.text, chunk_source: 'compiled_truth' });
|
||||
const embedSkipped = isEmbedSkipped(parsed.frontmatter);
|
||||
if (!embedSkipped) {
|
||||
if (parsed.compiled_truth.trim()) {
|
||||
for (const c of chunkText(parsed.compiled_truth)) {
|
||||
chunks.push({ chunk_index: chunks.length, chunk_text: c.text, chunk_source: 'compiled_truth' });
|
||||
}
|
||||
}
|
||||
}
|
||||
if (parsed.timeline?.trim()) {
|
||||
for (const c of chunkText(parsed.timeline)) {
|
||||
chunks.push({ chunk_index: chunks.length, chunk_text: c.text, chunk_source: 'timeline' });
|
||||
if (parsed.timeline?.trim()) {
|
||||
for (const c of chunkText(parsed.timeline)) {
|
||||
chunks.push({ chunk_index: chunks.length, chunk_text: c.text, chunk_source: 'timeline' });
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// v0.20.0 Cathedral II Layer 8 D2 — extract fenced code blocks from
|
||||
// compiled_truth as first-class code chunks.
|
||||
if (parsed.compiled_truth.trim()) {
|
||||
const fenceChunks = await extractFencedChunks(parsed.compiled_truth, chunks.length);
|
||||
chunks.push(...fenceChunks);
|
||||
// v0.20.0 Cathedral II Layer 8 D2 — extract fenced code blocks from
|
||||
// compiled_truth as first-class code chunks.
|
||||
if (parsed.compiled_truth.trim()) {
|
||||
const fenceChunks = await extractFencedChunks(parsed.compiled_truth, chunks.length);
|
||||
chunks.push(...fenceChunks);
|
||||
}
|
||||
}
|
||||
|
||||
// Embed BEFORE the transaction (external API call).
|
||||
|
||||
@@ -1847,11 +1847,16 @@ export class PGLiteEngine implements BrainEngine {
|
||||
|
||||
async countStaleChunks(opts?: { sourceId?: string }): Promise<number> {
|
||||
// D7: source-scoped count for `gbrain embed --stale --source X`.
|
||||
// v0.41 (D4+D8+Codex r2 #11): always JOIN pages so embed-skip filter
|
||||
// applies via `NOT (frontmatter ? 'embed_skip')`. PGLite is
|
||||
// PostgreSQL 17.5 in WASM and supports the full JSONB operator set.
|
||||
if (opts?.sourceId === undefined) {
|
||||
const { rows } = await this.db.query(
|
||||
`SELECT count(*)::int AS count
|
||||
FROM content_chunks
|
||||
WHERE embedding IS NULL`,
|
||||
FROM content_chunks cc
|
||||
JOIN pages p ON p.id = cc.page_id
|
||||
WHERE cc.embedding IS NULL
|
||||
AND NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'embed_skip')`,
|
||||
);
|
||||
const count = (rows[0] as { count: number } | undefined)?.count ?? 0;
|
||||
return Number(count);
|
||||
@@ -1861,7 +1866,8 @@ export class PGLiteEngine implements BrainEngine {
|
||||
FROM content_chunks cc
|
||||
JOIN pages p ON p.id = cc.page_id
|
||||
WHERE cc.embedding IS NULL
|
||||
AND p.source_id = $1`,
|
||||
AND p.source_id = $1
|
||||
AND NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'embed_skip')`,
|
||||
[opts.sourceId],
|
||||
);
|
||||
const count = (rows[0] as { count: number } | undefined)?.count ?? 0;
|
||||
@@ -1879,6 +1885,8 @@ export class PGLiteEngine implements BrainEngine {
|
||||
const afterIdx = opts?.afterChunkIndex ?? -1;
|
||||
// D7: optional source-scoped cursor scan. PGLite mirrors postgres-engine
|
||||
// so the engine-parity E2E catches drift.
|
||||
// v0.41 (D4+D8): NOT (frontmatter ? 'embed_skip') filter for soft-blocked
|
||||
// pages, matching the postgres-engine sibling.
|
||||
if (opts?.sourceId === undefined) {
|
||||
const { rows } = await this.db.query(
|
||||
`SELECT p.slug, cc.chunk_index, cc.chunk_text, cc.chunk_source,
|
||||
@@ -1886,6 +1894,7 @@ export class PGLiteEngine implements BrainEngine {
|
||||
FROM content_chunks cc
|
||||
JOIN pages p ON p.id = cc.page_id
|
||||
WHERE cc.embedding IS NULL
|
||||
AND NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'embed_skip')
|
||||
AND (cc.page_id, cc.chunk_index) > ($1, $2)
|
||||
ORDER BY cc.page_id, cc.chunk_index
|
||||
LIMIT $3`,
|
||||
@@ -1900,6 +1909,7 @@ export class PGLiteEngine implements BrainEngine {
|
||||
JOIN pages p ON p.id = cc.page_id
|
||||
WHERE cc.embedding IS NULL
|
||||
AND p.source_id = $1
|
||||
AND NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'embed_skip')
|
||||
AND (cc.page_id, cc.chunk_index) > ($2, $3)
|
||||
ORDER BY cc.page_id, cc.chunk_index
|
||||
LIMIT $4`,
|
||||
|
||||
@@ -1899,15 +1899,22 @@ export class PostgresEngine implements BrainEngine {
|
||||
|
||||
async countStaleChunks(opts?: { sourceId?: string }): Promise<number> {
|
||||
const sql = this.sql;
|
||||
// Fast path: no source filter → bare count query, no join.
|
||||
// Slow path: source-scoped count → join pages.
|
||||
// D7: closes the bug where `gbrain embed --stale --source X` silently
|
||||
// dropped X and counted across every source.
|
||||
// v0.41 (D4+D8+Codex r2 #11): the embed-skip filter requires JOIN
|
||||
// pages so we always join — the pre-v0.41 "fast path" without join
|
||||
// is gone. JSONB `?` existence check is cheap on the small set of
|
||||
// skipped pages; full-scan benefits from the partial index on
|
||||
// embedding IS NULL regardless.
|
||||
//
|
||||
// D7: source_id scoping. NULL/undefined = scan all sources;
|
||||
// a value scopes to that source so `gbrain embed --stale --source X`
|
||||
// does what it says.
|
||||
if (opts?.sourceId === undefined) {
|
||||
const [row] = await sql`
|
||||
SELECT count(*)::int AS count
|
||||
FROM content_chunks
|
||||
WHERE embedding IS NULL
|
||||
FROM content_chunks cc
|
||||
JOIN pages p ON p.id = cc.page_id
|
||||
WHERE cc.embedding IS NULL
|
||||
AND NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'embed_skip')
|
||||
`;
|
||||
return Number((row as { count?: number } | undefined)?.count ?? 0);
|
||||
}
|
||||
@@ -1917,6 +1924,7 @@ export class PostgresEngine implements BrainEngine {
|
||||
JOIN pages p ON p.id = cc.page_id
|
||||
WHERE cc.embedding IS NULL
|
||||
AND p.source_id = ${opts.sourceId}
|
||||
AND NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'embed_skip')
|
||||
`;
|
||||
return Number((row as { count?: number } | undefined)?.count ?? 0);
|
||||
}
|
||||
@@ -1938,6 +1946,12 @@ export class PostgresEngine implements BrainEngine {
|
||||
// D7: optional source_id filter. NULL/undefined = scan all sources
|
||||
// (pre-existing behavior); a value scopes to that source so
|
||||
// `gbrain embed --stale --source X` actually does what it says.
|
||||
//
|
||||
// v0.41 (D4+D8): NOT (frontmatter ? 'embed_skip') filter applied via
|
||||
// the always-JOINed pages row. Soft-blocked pages won't surface in
|
||||
// the stale list; their chunks were deleted at ingest time anyway
|
||||
// (D9 transition invariant), but the filter is defense-in-depth for
|
||||
// pre-fix inventory that might still have orphan chunks.
|
||||
if (opts?.sourceId === undefined) {
|
||||
const rows = await sql`
|
||||
SELECT p.slug, cc.chunk_index, cc.chunk_text, cc.chunk_source,
|
||||
@@ -1945,6 +1959,7 @@ export class PostgresEngine implements BrainEngine {
|
||||
FROM content_chunks cc
|
||||
JOIN pages p ON p.id = cc.page_id
|
||||
WHERE cc.embedding IS NULL
|
||||
AND NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'embed_skip')
|
||||
AND (cc.page_id, cc.chunk_index) > (${afterPid}, ${afterIdx})
|
||||
ORDER BY cc.page_id, cc.chunk_index
|
||||
LIMIT ${limit}
|
||||
@@ -1958,6 +1973,7 @@ export class PostgresEngine implements BrainEngine {
|
||||
JOIN pages p ON p.id = cc.page_id
|
||||
WHERE cc.embedding IS NULL
|
||||
AND p.source_id = ${opts.sourceId}
|
||||
AND NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'embed_skip')
|
||||
AND (cc.page_id, cc.chunk_index) > (${afterPid}, ${afterIdx})
|
||||
ORDER BY cc.page_id, cc.chunk_index
|
||||
LIMIT ${limit}
|
||||
|
||||
@@ -503,6 +503,13 @@ export function classifyErrorCode(errorMsg: string): string {
|
||||
}
|
||||
if (/TAKES_HOLDER_INVALID/i.test(errorMsg)) return 'TAKES_HOLDER_INVALID';
|
||||
|
||||
// v0.41 content-sanity gate. Hard-blocks at importFromContent throw
|
||||
// ContentSanityBlockError whose toString() embeds `PAGE_JUNK_PATTERN:`
|
||||
// (see src/core/content-sanity.ts PAGE_JUNK_PATTERN_CODE). Soft-blocks
|
||||
// (oversize alone) don't fail — the page lands with frontmatter.embed_skip
|
||||
// set and never enters this classifier.
|
||||
if (/PAGE_JUNK_PATTERN/i.test(errorMsg)) return 'PAGE_JUNK_PATTERN';
|
||||
|
||||
return 'UNKNOWN';
|
||||
}
|
||||
|
||||
|
||||
@@ -0,0 +1,219 @@
|
||||
import { describe, test, expect } from 'bun:test';
|
||||
import { mkdtempSync, rmSync, existsSync, readFileSync } from 'fs';
|
||||
import { join } from 'path';
|
||||
import { tmpdir } from 'os';
|
||||
import { withEnv } from '../helpers/with-env.ts';
|
||||
import {
|
||||
logContentSanityAssessment,
|
||||
readRecentContentSanityEvents,
|
||||
summarizeContentSanityEvents,
|
||||
computeContentSanityAuditFilename,
|
||||
type ContentSanityAuditEvent,
|
||||
} from '../../src/core/audit/content-sanity-audit.ts';
|
||||
import type { ContentSanityResult } from '../../src/core/content-sanity.ts';
|
||||
|
||||
function makeResult(opts: {
|
||||
bytes?: number;
|
||||
hard?: boolean;
|
||||
soft?: boolean;
|
||||
warn?: boolean;
|
||||
pattern?: string;
|
||||
literal?: string;
|
||||
}): ContentSanityResult {
|
||||
const junk_pattern_matches: string[] = opts.pattern ? [opts.pattern] : [];
|
||||
const literal_substring_matches: string[] = opts.literal ? [opts.literal] : [];
|
||||
const reasons: ContentSanityResult['reasons'] = [];
|
||||
const reason_messages: string[] = [];
|
||||
if (opts.soft) {
|
||||
reasons.push('oversize_block');
|
||||
reason_messages.push('PAGE_OVERSIZED: body 600000 bytes');
|
||||
} else if (opts.warn) {
|
||||
reasons.push('oversize_warn');
|
||||
reason_messages.push('PAGE_OVERSIZE_WARN: body 100000 bytes');
|
||||
}
|
||||
if (junk_pattern_matches.length > 0) {
|
||||
reasons.push('junk_pattern');
|
||||
reason_messages.push(`PAGE_JUNK_PATTERN: matched ${junk_pattern_matches.join(', ')}`);
|
||||
}
|
||||
if (literal_substring_matches.length > 0) {
|
||||
reasons.push('literal_substring');
|
||||
reason_messages.push(`PAGE_JUNK_PATTERN: literal ${literal_substring_matches.join(', ')}`);
|
||||
}
|
||||
return {
|
||||
bytes: opts.bytes ?? 1000,
|
||||
oversize: !!opts.soft,
|
||||
junk_pattern_matches,
|
||||
literal_substring_matches,
|
||||
reasons,
|
||||
reason_messages,
|
||||
shouldHardBlock: !!opts.hard || junk_pattern_matches.length > 0 || literal_substring_matches.length > 0,
|
||||
shouldSkipEmbed: !!opts.soft && !opts.hard && junk_pattern_matches.length === 0 && literal_substring_matches.length === 0,
|
||||
};
|
||||
}
|
||||
|
||||
describe('computeContentSanityAuditFilename', () => {
|
||||
test('emits the ISO-week prefix shape', () => {
|
||||
const name = computeContentSanityAuditFilename(new Date('2026-05-24T07:00:00Z'));
|
||||
expect(name).toMatch(/^content-sanity-\d{4}-W\d{2}\.jsonl$/);
|
||||
});
|
||||
});
|
||||
|
||||
describe('logContentSanityAssessment (E2E via tempdir)', () => {
|
||||
test('writes hard-block event', async () => {
|
||||
const dir = mkdtempSync(join(tmpdir(), 'cs-audit-hard-'));
|
||||
try {
|
||||
await withEnv({ GBRAIN_AUDIT_DIR: dir }, async () => {
|
||||
const result = makeResult({ hard: true, pattern: 'cloudflare_attention_required', bytes: 287 });
|
||||
logContentSanityAssessment('media/articles/foo', 'straylight-brain', result);
|
||||
const events = readRecentContentSanityEvents(7);
|
||||
expect(events.length).toBe(1);
|
||||
expect(events[0].event_type).toBe('hard_block');
|
||||
expect(events[0].slug).toBe('media/articles/foo');
|
||||
expect(events[0].source_id).toBe('straylight-brain');
|
||||
expect(events[0].junk_pattern_matches).toContain('cloudflare_attention_required');
|
||||
expect(events[0].bytes).toBe(287);
|
||||
});
|
||||
} finally {
|
||||
rmSync(dir, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
test('writes soft-block event', async () => {
|
||||
const dir = mkdtempSync(join(tmpdir(), 'cs-audit-soft-'));
|
||||
try {
|
||||
await withEnv({ GBRAIN_AUDIT_DIR: dir }, async () => {
|
||||
const result = makeResult({ soft: true, bytes: 890_000 });
|
||||
logContentSanityAssessment('media/big-transcript', 'default', result);
|
||||
const events = readRecentContentSanityEvents(7);
|
||||
expect(events.length).toBe(1);
|
||||
expect(events[0].event_type).toBe('soft_block');
|
||||
expect(events[0].bytes).toBe(890_000);
|
||||
});
|
||||
} finally {
|
||||
rmSync(dir, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
test('writes warn event', async () => {
|
||||
const dir = mkdtempSync(join(tmpdir(), 'cs-audit-warn-'));
|
||||
try {
|
||||
await withEnv({ GBRAIN_AUDIT_DIR: dir }, async () => {
|
||||
const result = makeResult({ warn: true, bytes: 100_000 });
|
||||
logContentSanityAssessment('notes/long', 'default', result);
|
||||
const events = readRecentContentSanityEvents(7);
|
||||
expect(events.length).toBe(1);
|
||||
expect(events[0].event_type).toBe('warn');
|
||||
});
|
||||
} finally {
|
||||
rmSync(dir, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
test('skips no-op rows (no reasons + no bypass)', async () => {
|
||||
const dir = mkdtempSync(join(tmpdir(), 'cs-audit-noop-'));
|
||||
try {
|
||||
await withEnv({ GBRAIN_AUDIT_DIR: dir }, async () => {
|
||||
const result = makeResult({}); // no reasons fire
|
||||
logContentSanityAssessment('normal-page', 'default', result);
|
||||
const events = readRecentContentSanityEvents(7);
|
||||
expect(events.length).toBe(0);
|
||||
});
|
||||
} finally {
|
||||
rmSync(dir, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
test('bypass active overrides hard/soft → records as warn with bypass_active flag', async () => {
|
||||
const dir = mkdtempSync(join(tmpdir(), 'cs-audit-bypass-'));
|
||||
try {
|
||||
await withEnv({ GBRAIN_AUDIT_DIR: dir }, async () => {
|
||||
const result = makeResult({ hard: true, pattern: 'access_denied' });
|
||||
logContentSanityAssessment('bypassed', 'default', result, { bypass: true });
|
||||
const events = readRecentContentSanityEvents(7);
|
||||
expect(events.length).toBe(1);
|
||||
expect(events[0].event_type).toBe('warn');
|
||||
expect(events[0].bypass_active).toBe(true);
|
||||
});
|
||||
} finally {
|
||||
rmSync(dir, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
test('multiple events accumulate in one file', async () => {
|
||||
const dir = mkdtempSync(join(tmpdir(), 'cs-audit-multi-'));
|
||||
try {
|
||||
await withEnv({ GBRAIN_AUDIT_DIR: dir }, async () => {
|
||||
logContentSanityAssessment('a', 'src', makeResult({ hard: true, pattern: 'access_denied' }));
|
||||
logContentSanityAssessment('b', 'src', makeResult({ soft: true, bytes: 600000 }));
|
||||
logContentSanityAssessment('c', 'src', makeResult({ warn: true, bytes: 70000 }));
|
||||
const events = readRecentContentSanityEvents(7);
|
||||
expect(events.length).toBe(3);
|
||||
});
|
||||
} finally {
|
||||
rmSync(dir, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
describe('summarizeContentSanityEvents', () => {
|
||||
function event(over: Partial<ContentSanityAuditEvent>): ContentSanityAuditEvent {
|
||||
return {
|
||||
ts: new Date().toISOString(),
|
||||
event_type: 'hard_block',
|
||||
slug: 'test',
|
||||
source_id: 'default',
|
||||
bytes: 100,
|
||||
junk_pattern_matches: [],
|
||||
literal_substring_matches: [],
|
||||
reason_messages: [],
|
||||
...over,
|
||||
};
|
||||
}
|
||||
test('empty input returns zero summary', () => {
|
||||
const s = summarizeContentSanityEvents([]);
|
||||
expect(s.total_events).toBe(0);
|
||||
expect(s.by_type).toEqual({ hard_block: 0, soft_block: 0, warn: 0 });
|
||||
expect(s.top_patterns).toEqual([]);
|
||||
});
|
||||
|
||||
test('counts by type', () => {
|
||||
const s = summarizeContentSanityEvents([
|
||||
event({ event_type: 'hard_block' }),
|
||||
event({ event_type: 'hard_block' }),
|
||||
event({ event_type: 'soft_block' }),
|
||||
event({ event_type: 'warn' }),
|
||||
]);
|
||||
expect(s.by_type).toEqual({ hard_block: 2, soft_block: 1, warn: 1 });
|
||||
expect(s.total_events).toBe(4);
|
||||
});
|
||||
|
||||
test('counts by source', () => {
|
||||
const s = summarizeContentSanityEvents([
|
||||
event({ source_id: 'straylight-brain' }),
|
||||
event({ source_id: 'straylight-brain' }),
|
||||
event({ source_id: 'default' }),
|
||||
]);
|
||||
expect(s.by_source['straylight-brain']).toBe(2);
|
||||
expect(s.by_source['default']).toBe(1);
|
||||
});
|
||||
|
||||
test('top_patterns sorted desc by count', () => {
|
||||
const s = summarizeContentSanityEvents([
|
||||
event({ junk_pattern_matches: ['cloudflare_attention_required'] }),
|
||||
event({ junk_pattern_matches: ['cloudflare_attention_required'] }),
|
||||
event({ junk_pattern_matches: ['cloudflare_attention_required'] }),
|
||||
event({ junk_pattern_matches: ['access_denied'] }),
|
||||
]);
|
||||
expect(s.top_patterns[0]).toEqual({ name: 'cloudflare_attention_required', count: 3 });
|
||||
expect(s.top_patterns[1]).toEqual({ name: 'access_denied', count: 1 });
|
||||
});
|
||||
|
||||
test('literal substring hits count alongside pattern hits', () => {
|
||||
const s = summarizeContentSanityEvents([
|
||||
event({ literal_substring_matches: ['reddit_blocked', 'linkedin_wall'] }),
|
||||
event({ literal_substring_matches: ['reddit_blocked'] }),
|
||||
]);
|
||||
expect(s.top_patterns).toContainEqual({ name: 'reddit_blocked', count: 2 });
|
||||
expect(s.top_patterns).toContainEqual({ name: 'linkedin_wall', count: 1 });
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,110 @@
|
||||
import { describe, test, expect } from 'bun:test';
|
||||
import { parseLiteralsContent } from '../src/core/content-sanity-literals.ts';
|
||||
|
||||
describe('parseLiteralsContent — operator file parser', () => {
|
||||
test('empty input returns empty list', () => {
|
||||
expect(parseLiteralsContent('')).toEqual([]);
|
||||
});
|
||||
|
||||
test('only-comments input returns empty list', () => {
|
||||
expect(parseLiteralsContent('# comment\n# another\n')).toEqual([]);
|
||||
});
|
||||
|
||||
test('only-blanks returns empty list', () => {
|
||||
expect(parseLiteralsContent('\n\n\n')).toEqual([]);
|
||||
});
|
||||
|
||||
test('single bare literal yields one entry with auto-generated name', () => {
|
||||
const out = parseLiteralsContent("You're being blocked\n");
|
||||
expect(out.length).toBe(1);
|
||||
expect(out[0].substring).toBe("You're being blocked");
|
||||
expect(out[0].name).toBe('operator_literal_0');
|
||||
expect(out[0].applies_to).toBe('both');
|
||||
});
|
||||
|
||||
test('name directive on preceding comment binds to next literal', () => {
|
||||
const input = `# name=reddit_blocked
|
||||
You're being blocked
|
||||
`;
|
||||
const out = parseLiteralsContent(input);
|
||||
expect(out.length).toBe(1);
|
||||
expect(out[0].name).toBe('reddit_blocked');
|
||||
expect(out[0].substring).toBe("You're being blocked");
|
||||
});
|
||||
|
||||
test('multiple directives merge into the next literal', () => {
|
||||
const input = `# name=linkedin_wall
|
||||
# applies_to=body
|
||||
Sign in to your account
|
||||
`;
|
||||
const out = parseLiteralsContent(input);
|
||||
expect(out[0].name).toBe('linkedin_wall');
|
||||
expect(out[0].applies_to).toBe('body');
|
||||
expect(out[0].substring).toBe('Sign in to your account');
|
||||
});
|
||||
|
||||
test('blank line between directive and literal resets binding', () => {
|
||||
const input = `# name=should_not_stick
|
||||
|
||||
You're being blocked
|
||||
`;
|
||||
const out = parseLiteralsContent(input);
|
||||
expect(out[0].name).toBe('operator_literal_0'); // auto-generated, not "should_not_stick"
|
||||
});
|
||||
|
||||
test('directives only bind to the next literal, then reset', () => {
|
||||
const input = `# name=first
|
||||
First literal
|
||||
# name=second
|
||||
Second literal
|
||||
Third literal
|
||||
`;
|
||||
const out = parseLiteralsContent(input);
|
||||
expect(out.length).toBe(3);
|
||||
expect(out[0].name).toBe('first');
|
||||
expect(out[1].name).toBe('second');
|
||||
// The auto-name index counts UNNAMED entries only — so the third
|
||||
// (first un-named) is operator_literal_0, not _2.
|
||||
expect(out[2].name).toBe('operator_literal_0');
|
||||
});
|
||||
|
||||
test('invalid applies_to value falls through to default both', () => {
|
||||
const input = `# applies_to=invalid_scope
|
||||
something
|
||||
`;
|
||||
const out = parseLiteralsContent(input);
|
||||
expect(out[0].applies_to).toBe('both');
|
||||
});
|
||||
|
||||
test('unknown directives ignored without throwing', () => {
|
||||
const input = `# foo=bar
|
||||
# applies_to=body
|
||||
literal
|
||||
`;
|
||||
const out = parseLiteralsContent(input);
|
||||
expect(out[0].applies_to).toBe('body');
|
||||
});
|
||||
|
||||
test('regex meta-characters in literal stay literal (no compile)', () => {
|
||||
// The loader does NOT call new RegExp() — literals are passed
|
||||
// through as-is and assessContentSanity uses .includes() for matching.
|
||||
const input = '(a+)+b\n';
|
||||
const out = parseLiteralsContent(input);
|
||||
expect(out[0].substring).toBe('(a+)+b');
|
||||
});
|
||||
|
||||
test('trims trailing whitespace on literal', () => {
|
||||
const input = 'literal-with-trailing-space \n';
|
||||
const out = parseLiteralsContent(input);
|
||||
expect(out[0].substring).toBe('literal-with-trailing-space');
|
||||
});
|
||||
|
||||
test('CRLF line endings handled', () => {
|
||||
const input = '# name=cr\r\nliteral\r\n';
|
||||
const out = parseLiteralsContent(input);
|
||||
expect(out.length).toBe(1);
|
||||
// The trim() preserves \r-stripping. The directive may or may not
|
||||
// capture trailing \r — test the substring is reasonably clean.
|
||||
expect(out[0].substring.replace(/\r$/, '')).toBe('literal');
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,416 @@
|
||||
import { describe, test, expect } from 'bun:test';
|
||||
import {
|
||||
assessContentSanity,
|
||||
ContentSanityBlockError,
|
||||
BUILT_IN_JUNK_PATTERNS,
|
||||
PAGE_JUNK_PATTERN_CODE,
|
||||
DEFAULT_BYTES_WARN,
|
||||
DEFAULT_BYTES_BLOCK,
|
||||
type OperatorLiteral,
|
||||
} from '../src/core/content-sanity.ts';
|
||||
|
||||
// ─── BOUNDARIES ───────────────────────────────────────────────
|
||||
|
||||
describe('assessContentSanity — size boundaries', () => {
|
||||
test('empty body returns 0 bytes and no trips', () => {
|
||||
const r = assessContentSanity({ compiled_truth: '', timeline: '', title: '' });
|
||||
expect(r.bytes).toBe(0);
|
||||
expect(r.oversize).toBe(false);
|
||||
expect(r.shouldHardBlock).toBe(false);
|
||||
expect(r.shouldSkipEmbed).toBe(false);
|
||||
expect(r.reasons).toEqual([]);
|
||||
});
|
||||
|
||||
test('bytes counts compiled_truth + timeline (Codex r2 #7)', () => {
|
||||
// Without timeline a check might miss huge timeline sections; the
|
||||
// assessor must sum both. Use ASCII for byteLength === length.
|
||||
const ct = 'a'.repeat(1000);
|
||||
const tl = 'b'.repeat(2000);
|
||||
const r = assessContentSanity({ compiled_truth: ct, timeline: tl, title: '' });
|
||||
expect(r.bytes).toBeGreaterThanOrEqual(3000); // + the join '\n'
|
||||
expect(r.bytes).toBeLessThan(3010);
|
||||
});
|
||||
|
||||
test('bytes uses UTF-8 octets, not character count', () => {
|
||||
// CJK chars: each takes 3 UTF-8 bytes. 100 chars → 300 bytes.
|
||||
const ct = '世'.repeat(100);
|
||||
const r = assessContentSanity({ compiled_truth: ct, timeline: '', title: '' });
|
||||
expect(r.bytes).toBe(300);
|
||||
});
|
||||
|
||||
test('exactly at warn threshold does NOT fire warn (strict >)', () => {
|
||||
const r = assessContentSanity({
|
||||
compiled_truth: 'a'.repeat(50_000),
|
||||
timeline: '',
|
||||
title: '',
|
||||
bytes_warn: 50_000,
|
||||
bytes_block: 500_000,
|
||||
});
|
||||
expect(r.reasons).not.toContain('oversize_warn');
|
||||
expect(r.reasons).not.toContain('oversize_block');
|
||||
});
|
||||
|
||||
test('above warn but below block → oversize_warn only', () => {
|
||||
const r = assessContentSanity({
|
||||
compiled_truth: 'a'.repeat(100_000),
|
||||
timeline: '',
|
||||
title: '',
|
||||
});
|
||||
expect(r.reasons).toContain('oversize_warn');
|
||||
expect(r.reasons).not.toContain('oversize_block');
|
||||
expect(r.shouldHardBlock).toBe(false);
|
||||
expect(r.shouldSkipEmbed).toBe(false);
|
||||
});
|
||||
|
||||
test('above block threshold → oversize_block + shouldSkipEmbed', () => {
|
||||
const r = assessContentSanity({
|
||||
compiled_truth: 'a'.repeat(600_000),
|
||||
timeline: '',
|
||||
title: '',
|
||||
});
|
||||
expect(r.oversize).toBe(true);
|
||||
expect(r.reasons).toContain('oversize_block');
|
||||
expect(r.reasons).not.toContain('oversize_warn'); // not double-pushed
|
||||
expect(r.shouldSkipEmbed).toBe(true);
|
||||
expect(r.shouldHardBlock).toBe(false);
|
||||
});
|
||||
|
||||
test('the original 890K reproduction trips block alone (no junk)', () => {
|
||||
// 890K of clean text (no Cloudflare phrases) → soft-block only.
|
||||
const r = assessContentSanity({
|
||||
compiled_truth: 'normal prose. '.repeat(70_000), // ~890K bytes
|
||||
timeline: '',
|
||||
title: 'A Long Article',
|
||||
});
|
||||
expect(r.shouldSkipEmbed).toBe(true);
|
||||
expect(r.shouldHardBlock).toBe(false);
|
||||
});
|
||||
|
||||
test('custom thresholds override defaults', () => {
|
||||
const r = assessContentSanity({
|
||||
compiled_truth: 'a'.repeat(150),
|
||||
timeline: '',
|
||||
title: '',
|
||||
bytes_warn: 100,
|
||||
bytes_block: 200,
|
||||
});
|
||||
expect(r.reasons).toContain('oversize_warn');
|
||||
});
|
||||
|
||||
test('defaults are exported and reasonable', () => {
|
||||
expect(DEFAULT_BYTES_WARN).toBe(50_000);
|
||||
expect(DEFAULT_BYTES_BLOCK).toBe(500_000);
|
||||
});
|
||||
});
|
||||
|
||||
// ─── 6 BUILT-IN PATTERNS ──────────────────────────────────────
|
||||
|
||||
describe('assessContentSanity — built-in junk patterns', () => {
|
||||
test('built-in pattern count is locked at 6 (D3 dropped empty_body_with_source_url)', () => {
|
||||
expect(BUILT_IN_JUNK_PATTERNS.length).toBe(6);
|
||||
const names = BUILT_IN_JUNK_PATTERNS.map((p) => p.name);
|
||||
expect(names).toContain('cloudflare_attention_required');
|
||||
expect(names).toContain('cloudflare_just_a_moment');
|
||||
expect(names).toContain('cloudflare_ray_id');
|
||||
expect(names).toContain('access_denied');
|
||||
expect(names).toContain('captcha_required');
|
||||
expect(names).toContain('error_page_title');
|
||||
// D3 regression: this rule was dropped. If it ever returns, the test
|
||||
// count above bumps to 7 deliberately.
|
||||
expect(names).not.toContain('empty_body_with_source_url');
|
||||
});
|
||||
|
||||
test('built-in patterns all compile (module-load safety net)', () => {
|
||||
for (const p of BUILT_IN_JUNK_PATTERNS) {
|
||||
expect(p.pattern).toBeInstanceOf(RegExp);
|
||||
expect(() => p.pattern.test('test input')).not.toThrow();
|
||||
}
|
||||
});
|
||||
|
||||
test('cloudflare_attention_required fires on real-world title', () => {
|
||||
const r = assessContentSanity({
|
||||
compiled_truth: '',
|
||||
timeline: '',
|
||||
title: 'Attention Required! | Cloudflare',
|
||||
});
|
||||
expect(r.junk_pattern_matches).toContain('cloudflare_attention_required');
|
||||
expect(r.shouldHardBlock).toBe(true);
|
||||
});
|
||||
|
||||
test('cloudflare_just_a_moment requires BOTH signals (no false-positive on prose)', () => {
|
||||
// Just the words "Just a moment..." alone does NOT fire (legitimate
|
||||
// writing might include it).
|
||||
const r1 = assessContentSanity({
|
||||
compiled_truth: 'Just a moment... I want to finish this thought before moving on.',
|
||||
timeline: '',
|
||||
title: '',
|
||||
});
|
||||
expect(r1.junk_pattern_matches).not.toContain('cloudflare_just_a_moment');
|
||||
|
||||
// With the cdn-cgi discriminator nearby → fires.
|
||||
const r2 = assessContentSanity({
|
||||
compiled_truth: 'Just a moment... please wait while we verify\ncdn-cgi/challenge-platform/h/blah',
|
||||
timeline: '',
|
||||
title: '',
|
||||
});
|
||||
expect(r2.junk_pattern_matches).toContain('cloudflare_just_a_moment');
|
||||
});
|
||||
|
||||
test('cloudflare_ray_id fires on trailing diagnostic', () => {
|
||||
const r = assessContentSanity({
|
||||
compiled_truth: 'You have been blocked.\n\nCloudflare Ray ID: abc12345',
|
||||
timeline: '',
|
||||
title: 'Blocked',
|
||||
});
|
||||
expect(r.junk_pattern_matches).toContain('cloudflare_ray_id');
|
||||
});
|
||||
|
||||
test('access_denied fires on bare 403 dumps', () => {
|
||||
const r = assessContentSanity({
|
||||
compiled_truth: 'Access denied\n\nYou do not have permission to view this resource.',
|
||||
timeline: '',
|
||||
title: '',
|
||||
});
|
||||
expect(r.junk_pattern_matches).toContain('access_denied');
|
||||
});
|
||||
|
||||
test('captcha_required catches multiple verification phrasings', () => {
|
||||
for (const phrase of ['verify you are human', 'verify you are a human', 'captcha required', 'please complete the security check']) {
|
||||
const r = assessContentSanity({
|
||||
compiled_truth: `Please ${phrase} to continue.`,
|
||||
timeline: '',
|
||||
title: '',
|
||||
});
|
||||
expect(r.junk_pattern_matches).toContain('captcha_required');
|
||||
}
|
||||
});
|
||||
|
||||
test('error_page_title fires only on bare titles (anchored)', () => {
|
||||
for (const title of ['404', 'Error 500', 'Page Not Found', '503']) {
|
||||
const r = assessContentSanity({ compiled_truth: '', timeline: '', title });
|
||||
expect(r.junk_pattern_matches).toContain('error_page_title');
|
||||
}
|
||||
// A thoughtful page ABOUT errors does NOT fire.
|
||||
const r2 = assessContentSanity({
|
||||
compiled_truth: '',
|
||||
timeline: '',
|
||||
title: 'Designing for 404 pages: a UX guide',
|
||||
});
|
||||
expect(r2.junk_pattern_matches).not.toContain('error_page_title');
|
||||
});
|
||||
|
||||
test('multiple patterns can fire on the same content', () => {
|
||||
const r = assessContentSanity({
|
||||
compiled_truth: 'Cloudflare Ray ID: xyz789',
|
||||
timeline: '',
|
||||
title: 'Attention Required! | Cloudflare',
|
||||
});
|
||||
expect(r.junk_pattern_matches).toContain('cloudflare_attention_required');
|
||||
expect(r.junk_pattern_matches).toContain('cloudflare_ray_id');
|
||||
expect(r.shouldHardBlock).toBe(true);
|
||||
});
|
||||
|
||||
test('case-insensitive matching across all patterns', () => {
|
||||
const r = assessContentSanity({
|
||||
compiled_truth: '',
|
||||
timeline: '',
|
||||
title: 'ATTENTION REQUIRED! | CLOUDFLARE',
|
||||
});
|
||||
expect(r.junk_pattern_matches).toContain('cloudflare_attention_required');
|
||||
});
|
||||
});
|
||||
|
||||
// ─── REASON ORDERING + MESSAGES ────────────────────────────────
|
||||
|
||||
describe('assessContentSanity — reason ordering', () => {
|
||||
test('reason_messages embed the classifier-readable PAGE_JUNK_PATTERN prefix', () => {
|
||||
const r = assessContentSanity({
|
||||
compiled_truth: '',
|
||||
timeline: '',
|
||||
title: 'Access denied',
|
||||
});
|
||||
expect(r.shouldHardBlock).toBe(true);
|
||||
const joined = r.reason_messages.join(' ');
|
||||
expect(joined).toContain(PAGE_JUNK_PATTERN_CODE);
|
||||
expect(PAGE_JUNK_PATTERN_CODE).toBe('PAGE_JUNK_PATTERN');
|
||||
});
|
||||
|
||||
test('block-level oversize message includes PAGE_OVERSIZED prefix', () => {
|
||||
const r = assessContentSanity({
|
||||
compiled_truth: 'a'.repeat(600_000),
|
||||
timeline: '',
|
||||
title: '',
|
||||
});
|
||||
const joined = r.reason_messages.join(' ');
|
||||
expect(joined).toContain('PAGE_OVERSIZED:');
|
||||
});
|
||||
|
||||
test('hard-block + oversize: BOTH reasons present (operator sees both causes)', () => {
|
||||
// Pattern in first 2KB head-slice so junk_pattern fires alongside
|
||||
// oversize_block. This is the realistic 890K Cloudflare dump shape:
|
||||
// the "Attention Required" banner is at the top, then the rest of
|
||||
// the page is HTML/styles/etc making it huge.
|
||||
const r = assessContentSanity({
|
||||
compiled_truth: 'Cloudflare Ray ID: abc\n' + 'a'.repeat(600_000),
|
||||
timeline: '',
|
||||
title: '',
|
||||
});
|
||||
expect(r.reasons).toContain('oversize_block');
|
||||
expect(r.reasons).toContain('junk_pattern');
|
||||
expect(r.shouldHardBlock).toBe(true);
|
||||
// hard-block wins; soft-block doesn't ALSO fire.
|
||||
expect(r.shouldSkipEmbed).toBe(false);
|
||||
});
|
||||
});
|
||||
|
||||
// ─── OPERATOR LITERALS ────────────────────────────────────────
|
||||
|
||||
describe('assessContentSanity — operator literals', () => {
|
||||
test('empty extra_literals = built-ins only', () => {
|
||||
const r = assessContentSanity({
|
||||
compiled_truth: "You're being blocked from accessing this resource",
|
||||
timeline: '',
|
||||
title: '',
|
||||
extra_literals: [],
|
||||
});
|
||||
expect(r.shouldHardBlock).toBe(false);
|
||||
expect(r.literal_substring_matches).toEqual([]);
|
||||
});
|
||||
|
||||
test('operator literal matches case-insensitively', () => {
|
||||
const literals: OperatorLiteral[] = [
|
||||
{ name: 'reddit_blocked', substring: "you're being blocked from accessing" },
|
||||
];
|
||||
const r = assessContentSanity({
|
||||
compiled_truth: "YOU'RE BEING BLOCKED FROM ACCESSING this site.",
|
||||
timeline: '',
|
||||
title: '',
|
||||
extra_literals: literals,
|
||||
});
|
||||
expect(r.literal_substring_matches).toContain('reddit_blocked');
|
||||
expect(r.shouldHardBlock).toBe(true);
|
||||
});
|
||||
|
||||
test('regex meta-characters in operator literal stay literal (no ReDoS surface)', () => {
|
||||
const literals: OperatorLiteral[] = [
|
||||
{ name: 'meta_test', substring: '(a+)+b' }, // would be catastrophic as regex
|
||||
];
|
||||
// Should NOT match prose
|
||||
const r1 = assessContentSanity({
|
||||
compiled_truth: 'aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa',
|
||||
timeline: '',
|
||||
title: '',
|
||||
extra_literals: literals,
|
||||
});
|
||||
expect(r1.literal_substring_matches).not.toContain('meta_test');
|
||||
// SHOULD match the literal string
|
||||
const r2 = assessContentSanity({
|
||||
compiled_truth: 'The pattern (a+)+b is bad regex.',
|
||||
timeline: '',
|
||||
title: '',
|
||||
extra_literals: literals,
|
||||
});
|
||||
expect(r2.literal_substring_matches).toContain('meta_test');
|
||||
});
|
||||
|
||||
test('literal applies_to scope honored', () => {
|
||||
const titleOnly: OperatorLiteral = { name: 't', substring: 'wall', applies_to: 'title' };
|
||||
const bodyOnly: OperatorLiteral = { name: 'b', substring: 'wall', applies_to: 'body' };
|
||||
const r1 = assessContentSanity({
|
||||
compiled_truth: 'auth wall content',
|
||||
timeline: '',
|
||||
title: 'unrelated',
|
||||
extra_literals: [titleOnly],
|
||||
});
|
||||
expect(r1.literal_substring_matches).not.toContain('t');
|
||||
const r2 = assessContentSanity({
|
||||
compiled_truth: 'unrelated body',
|
||||
timeline: '',
|
||||
title: 'auth wall',
|
||||
extra_literals: [titleOnly],
|
||||
});
|
||||
expect(r2.literal_substring_matches).toContain('t');
|
||||
const r3 = assessContentSanity({
|
||||
compiled_truth: 'auth wall content',
|
||||
timeline: '',
|
||||
title: 'unrelated',
|
||||
extra_literals: [bodyOnly],
|
||||
});
|
||||
expect(r3.literal_substring_matches).toContain('b');
|
||||
});
|
||||
|
||||
test('empty substring is no-op', () => {
|
||||
const r = assessContentSanity({
|
||||
compiled_truth: 'anything',
|
||||
timeline: '',
|
||||
title: '',
|
||||
extra_literals: [{ name: 'empty', substring: '' }],
|
||||
});
|
||||
expect(r.literal_substring_matches).toEqual([]);
|
||||
});
|
||||
});
|
||||
|
||||
// ─── SCAN HEAD-SLICE BOUNDARY ─────────────────────────────────
|
||||
|
||||
describe('assessContentSanity — head-slice scope', () => {
|
||||
test('pattern in first 2KB matches', () => {
|
||||
const r = assessContentSanity({
|
||||
compiled_truth: 'Cloudflare Ray ID: aaa\n' + 'x'.repeat(10_000),
|
||||
timeline: '',
|
||||
title: '',
|
||||
});
|
||||
expect(r.junk_pattern_matches).toContain('cloudflare_ray_id');
|
||||
});
|
||||
|
||||
test('pattern past the 2KB head-slice does NOT match (cost bound)', () => {
|
||||
// Cost bound: patterns evaluated against first ~2KB only.
|
||||
// Pattern buried at offset 5K should NOT trip.
|
||||
const r = assessContentSanity({
|
||||
compiled_truth: 'x'.repeat(5000) + 'Cloudflare Ray ID: deep',
|
||||
timeline: '',
|
||||
title: '',
|
||||
});
|
||||
expect(r.junk_pattern_matches).not.toContain('cloudflare_ray_id');
|
||||
});
|
||||
});
|
||||
|
||||
// ─── ContentSanityBlockError ──────────────────────────────────
|
||||
|
||||
describe('ContentSanityBlockError', () => {
|
||||
test('error message contains PAGE_JUNK_PATTERN for classifier match', () => {
|
||||
const r = assessContentSanity({
|
||||
compiled_truth: 'Access denied',
|
||||
timeline: '',
|
||||
title: '',
|
||||
});
|
||||
const err = new ContentSanityBlockError(r);
|
||||
expect(err.message).toContain('PAGE_JUNK_PATTERN');
|
||||
expect(err.code).toBe('PAGE_JUNK_PATTERN');
|
||||
expect(err.name).toBe('ContentSanityBlockError');
|
||||
});
|
||||
|
||||
test('error retains the full result for caller inspection', () => {
|
||||
const r = assessContentSanity({
|
||||
compiled_truth: 'Access denied',
|
||||
timeline: '',
|
||||
title: 'Attention Required! | Cloudflare',
|
||||
});
|
||||
const err = new ContentSanityBlockError(r);
|
||||
expect(err.result.junk_pattern_matches.length).toBeGreaterThan(0);
|
||||
expect(err.result).toBe(r); // same reference, not a copy
|
||||
});
|
||||
|
||||
test('error is throwable + catchable as instanceof', () => {
|
||||
const r = assessContentSanity({
|
||||
compiled_truth: '',
|
||||
timeline: '',
|
||||
title: 'Access denied',
|
||||
});
|
||||
try {
|
||||
throw new ContentSanityBlockError(r);
|
||||
} catch (e) {
|
||||
expect(e).toBeInstanceOf(ContentSanityBlockError);
|
||||
expect((e as Error).message).toContain('PAGE_JUNK_PATTERN');
|
||||
}
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,105 @@
|
||||
import { describe, test, expect } from 'bun:test';
|
||||
import {
|
||||
isEmbedSkipped,
|
||||
filterOutEmbedSkipped,
|
||||
buildEmbedSkipMarker,
|
||||
EMBED_SKIP_KEY,
|
||||
EMBED_SKIP_FILTER_FRAGMENT,
|
||||
} from '../src/core/embed-skip.ts';
|
||||
|
||||
describe('isEmbedSkipped', () => {
|
||||
test('false on null', () => {
|
||||
expect(isEmbedSkipped(null)).toBe(false);
|
||||
});
|
||||
test('false on undefined', () => {
|
||||
expect(isEmbedSkipped(undefined)).toBe(false);
|
||||
});
|
||||
test('false on empty object', () => {
|
||||
expect(isEmbedSkipped({})).toBe(false);
|
||||
});
|
||||
test('false when key is undefined', () => {
|
||||
expect(isEmbedSkipped({ other_key: true })).toBe(false);
|
||||
});
|
||||
test('false when key value is null', () => {
|
||||
// Explicit null = "not skipped" (key existence != truthy).
|
||||
expect(isEmbedSkipped({ embed_skip: null })).toBe(false);
|
||||
});
|
||||
test('true on full marker object (canonical write shape)', () => {
|
||||
expect(isEmbedSkipped({ embed_skip: { reason: 'oversized', bytes: 100, assessed_at: 'iso' } })).toBe(true);
|
||||
});
|
||||
test('true on bare boolean (future flexibility)', () => {
|
||||
expect(isEmbedSkipped({ embed_skip: true })).toBe(true);
|
||||
});
|
||||
test('true on any non-null/undefined value (key-existence semantics)', () => {
|
||||
// Mirrors the SQL fragment's JSONB `?` existence operator —
|
||||
// contents are diagnostic, not functional.
|
||||
expect(isEmbedSkipped({ embed_skip: 'string-marker' })).toBe(true);
|
||||
expect(isEmbedSkipped({ embed_skip: 0 })).toBe(true);
|
||||
});
|
||||
test('EMBED_SKIP_KEY constant is stable contract', () => {
|
||||
expect(EMBED_SKIP_KEY).toBe('embed_skip');
|
||||
});
|
||||
});
|
||||
|
||||
describe('filterOutEmbedSkipped', () => {
|
||||
test('empty array passes through', () => {
|
||||
expect(filterOutEmbedSkipped([])).toEqual([]);
|
||||
});
|
||||
test('keeps pages without frontmatter', () => {
|
||||
const pages = [{ id: 1 }, { id: 2, frontmatter: null }];
|
||||
expect(filterOutEmbedSkipped(pages).length).toBe(2);
|
||||
});
|
||||
test('excludes pages with embed_skip set', () => {
|
||||
const pages = [
|
||||
{ id: 1, frontmatter: {} },
|
||||
{ id: 2, frontmatter: { embed_skip: { reason: 'oversized', bytes: 100, assessed_at: '' } } },
|
||||
{ id: 3, frontmatter: { other: true } },
|
||||
];
|
||||
const kept = filterOutEmbedSkipped(pages);
|
||||
expect(kept.length).toBe(2);
|
||||
expect(kept.map((p) => p.id)).toEqual([1, 3]);
|
||||
});
|
||||
test('preserves order of kept pages', () => {
|
||||
const pages = [
|
||||
{ id: 1 },
|
||||
{ id: 2, frontmatter: { embed_skip: true } },
|
||||
{ id: 3 },
|
||||
{ id: 4, frontmatter: { embed_skip: true } },
|
||||
{ id: 5 },
|
||||
];
|
||||
expect(filterOutEmbedSkipped(pages).map((p) => p.id)).toEqual([1, 3, 5]);
|
||||
});
|
||||
});
|
||||
|
||||
describe('buildEmbedSkipMarker', () => {
|
||||
test('returns canonical marker shape', () => {
|
||||
const marker = buildEmbedSkipMarker(123456);
|
||||
expect(marker.reason).toBe('oversized');
|
||||
expect(marker.bytes).toBe(123456);
|
||||
expect(typeof marker.assessed_at).toBe('string');
|
||||
expect(() => new Date(marker.assessed_at)).not.toThrow();
|
||||
});
|
||||
test('uses injected Date for deterministic tests', () => {
|
||||
const d = new Date('2026-05-24T07:00:00Z');
|
||||
const m = buildEmbedSkipMarker(100, d);
|
||||
expect(m.assessed_at).toBe('2026-05-24T07:00:00.000Z');
|
||||
});
|
||||
});
|
||||
|
||||
describe('EMBED_SKIP_FILTER_FRAGMENT', () => {
|
||||
test('fragment references the canonical key name', () => {
|
||||
expect(EMBED_SKIP_FILTER_FRAGMENT).toContain(`'${EMBED_SKIP_KEY}'`);
|
||||
});
|
||||
test('fragment negates (NOT) so kept rows are without the marker', () => {
|
||||
expect(EMBED_SKIP_FILTER_FRAGMENT.trim().startsWith('NOT')).toBe(true);
|
||||
});
|
||||
test('fragment uses JSONB `?` existence operator (works on Postgres + PGLite)', () => {
|
||||
expect(EMBED_SKIP_FILTER_FRAGMENT).toContain(' ? ');
|
||||
});
|
||||
test('fragment COALESCEs null frontmatter so pages without one are not filtered', () => {
|
||||
expect(EMBED_SKIP_FILTER_FRAGMENT).toContain('COALESCE');
|
||||
});
|
||||
test('fragment assumes pages alias is `p` (engine-call-site contract)', () => {
|
||||
expect(EMBED_SKIP_FILTER_FRAGMENT).toContain('p.frontmatter');
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,206 @@
|
||||
import { describe, test, expect, beforeAll, afterAll, beforeEach } from 'bun:test';
|
||||
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
|
||||
import { resetPgliteState } from './helpers/reset-pglite.ts';
|
||||
import { withEnv } from './helpers/with-env.ts';
|
||||
import { mkdtempSync, rmSync } from 'fs';
|
||||
import { join } from 'path';
|
||||
import { tmpdir } from 'os';
|
||||
import { importFromContent } from '../src/core/import-file.ts';
|
||||
import { ContentSanityBlockError } from '../src/core/content-sanity.ts';
|
||||
import { isEmbedSkipped, EMBED_SKIP_KEY } from '../src/core/embed-skip.ts';
|
||||
|
||||
let engine: PGLiteEngine;
|
||||
let auditDir: string;
|
||||
let gbrainHomeDir: string;
|
||||
|
||||
beforeAll(async () => {
|
||||
engine = new PGLiteEngine();
|
||||
await engine.connect({});
|
||||
await engine.initSchema();
|
||||
});
|
||||
|
||||
afterAll(async () => {
|
||||
await engine.disconnect();
|
||||
});
|
||||
|
||||
beforeEach(async () => {
|
||||
await resetPgliteState(engine);
|
||||
});
|
||||
|
||||
/** Wrap an importFromContent call with GBRAIN_HOME + GBRAIN_AUDIT_DIR
|
||||
* pointed at fresh tempdirs so config and audit writes don't leak
|
||||
* between tests or pollute the developer's real ~/.gbrain. */
|
||||
async function withIsolatedHome<T>(fn: () => Promise<T>): Promise<T> {
|
||||
gbrainHomeDir = mkdtempSync(join(tmpdir(), 'cs-gate-home-'));
|
||||
auditDir = mkdtempSync(join(tmpdir(), 'cs-gate-audit-'));
|
||||
try {
|
||||
return await withEnv({
|
||||
GBRAIN_HOME: gbrainHomeDir,
|
||||
GBRAIN_AUDIT_DIR: auditDir,
|
||||
}, fn);
|
||||
} finally {
|
||||
rmSync(gbrainHomeDir, { recursive: true, force: true });
|
||||
rmSync(auditDir, { recursive: true, force: true });
|
||||
}
|
||||
}
|
||||
|
||||
const FRONTMATTER = `---
|
||||
title: 'Test Page'
|
||||
type: note
|
||||
created: 2026-05-24
|
||||
---
|
||||
|
||||
`;
|
||||
|
||||
describe('importFromContent — content-sanity hard-block (D6)', () => {
|
||||
test('throws ContentSanityBlockError on Cloudflare junk title', async () => {
|
||||
await withIsolatedHome(async () => {
|
||||
const content = `---
|
||||
title: 'Attention Required! | Cloudflare'
|
||||
type: note
|
||||
created: 2026-05-24
|
||||
---
|
||||
|
||||
Body.`;
|
||||
await expect(
|
||||
importFromContent(engine, 'test/junk', content, { noEmbed: true })
|
||||
).rejects.toThrow(ContentSanityBlockError);
|
||||
});
|
||||
});
|
||||
|
||||
test('throws with PAGE_JUNK_PATTERN-tagged message for classifyErrorCode', async () => {
|
||||
await withIsolatedHome(async () => {
|
||||
const content = FRONTMATTER + 'Cloudflare Ray ID: abc123';
|
||||
let caught: Error | undefined;
|
||||
try {
|
||||
await importFromContent(engine, 'test/ray', content, { noEmbed: true });
|
||||
} catch (e) {
|
||||
caught = e as Error;
|
||||
}
|
||||
expect(caught).toBeDefined();
|
||||
expect(caught!.message).toContain('PAGE_JUNK_PATTERN');
|
||||
});
|
||||
});
|
||||
|
||||
test('thrown page is NOT written to DB', async () => {
|
||||
await withIsolatedHome(async () => {
|
||||
// Title matches the anchored error_page_title pattern exactly
|
||||
// (`^(403|404|500|...|page not found)\s*$`). "404 Not Found"
|
||||
// doesn't anchor; the test needs the bare form.
|
||||
const content = `---
|
||||
title: '404'
|
||||
type: note
|
||||
created: 2026-05-24
|
||||
---
|
||||
|
||||
`;
|
||||
try {
|
||||
await importFromContent(engine, 'test/404', content, { noEmbed: true });
|
||||
} catch { /* expected */ }
|
||||
const page = await engine.getPage('test/404');
|
||||
expect(page).toBeNull();
|
||||
});
|
||||
});
|
||||
});
|
||||
|
||||
describe('importFromContent — soft-block (D9 transition + embed_skip)', () => {
|
||||
test('soft-block writes page with embed_skip frontmatter marker', async () => {
|
||||
await withIsolatedHome(async () => {
|
||||
// 600K of clean text → soft-block (oversize but no junk pattern).
|
||||
const content = FRONTMATTER + 'a'.repeat(600_000);
|
||||
const result = await importFromContent(engine, 'test/big', content, { noEmbed: true });
|
||||
expect(result.status).not.toBe('error');
|
||||
const page = await engine.getPage('test/big');
|
||||
expect(page).not.toBeNull();
|
||||
const fm = page!.frontmatter as Record<string, unknown>;
|
||||
expect(isEmbedSkipped(fm)).toBe(true);
|
||||
const marker = fm[EMBED_SKIP_KEY] as Record<string, unknown>;
|
||||
expect(marker.reason).toBe('oversized');
|
||||
expect(marker.bytes).toBeGreaterThan(500_000);
|
||||
});
|
||||
});
|
||||
|
||||
test('soft-block deletes existing chunks (D9 transition invariant)', async () => {
|
||||
await withIsolatedHome(async () => {
|
||||
// First write a normal page to seed some chunks.
|
||||
const small = FRONTMATTER + 'Short content with multiple sentences. Plenty of words here. Enough to chunk.';
|
||||
await importFromContent(engine, 'test/grow', small, { noEmbed: true });
|
||||
const beforeChunks = await engine.getChunks('test/grow');
|
||||
expect(beforeChunks.length).toBeGreaterThan(0);
|
||||
|
||||
// Now re-import with content that grew past the block threshold.
|
||||
const big = FRONTMATTER + 'a'.repeat(600_000);
|
||||
await importFromContent(engine, 'test/grow', big, { noEmbed: true });
|
||||
const afterChunks = await engine.getChunks('test/grow');
|
||||
// D9: transition to embed_skip should delete chunks.
|
||||
expect(afterChunks.length).toBe(0);
|
||||
});
|
||||
});
|
||||
|
||||
test('soft-block skips chunking entirely (no new chunks created)', async () => {
|
||||
await withIsolatedHome(async () => {
|
||||
const content = FRONTMATTER + 'a'.repeat(600_000);
|
||||
await importFromContent(engine, 'test/big2', content, { noEmbed: true });
|
||||
const chunks = await engine.getChunks('test/big2');
|
||||
expect(chunks.length).toBe(0);
|
||||
});
|
||||
});
|
||||
});
|
||||
|
||||
describe('importFromContent — kill-switch bypass', () => {
|
||||
test('GBRAIN_NO_SANITY=1 lets junk through with bypass audit + stderr', async () => {
|
||||
const gbrainHomeDirLocal = mkdtempSync(join(tmpdir(), 'cs-bypass-home-'));
|
||||
const auditDirLocal = mkdtempSync(join(tmpdir(), 'cs-bypass-audit-'));
|
||||
try {
|
||||
await withEnv({
|
||||
GBRAIN_HOME: gbrainHomeDirLocal,
|
||||
GBRAIN_AUDIT_DIR: auditDirLocal,
|
||||
GBRAIN_NO_SANITY: '1',
|
||||
}, async () => {
|
||||
const content = `---
|
||||
title: 'Attention Required! | Cloudflare'
|
||||
type: note
|
||||
created: 2026-05-24
|
||||
---
|
||||
|
||||
junk body`;
|
||||
const result = await importFromContent(engine, 'test/bypass', content, { noEmbed: true });
|
||||
expect(result.status).not.toBe('error');
|
||||
const page = await engine.getPage('test/bypass');
|
||||
expect(page).not.toBeNull();
|
||||
// Page lands with frontmatter unchanged (no embed_skip set on bypass).
|
||||
const fm = page!.frontmatter as Record<string, unknown>;
|
||||
expect(isEmbedSkipped(fm)).toBe(false);
|
||||
});
|
||||
} finally {
|
||||
rmSync(gbrainHomeDirLocal, { recursive: true, force: true });
|
||||
rmSync(auditDirLocal, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
describe('importFromContent — normal pages unaffected', () => {
|
||||
test('clean page imports successfully', async () => {
|
||||
await withIsolatedHome(async () => {
|
||||
const content = FRONTMATTER + 'A thoughtful essay about software design.';
|
||||
const result = await importFromContent(engine, 'test/clean', content, { noEmbed: true });
|
||||
expect(result.status).toBe('imported');
|
||||
const page = await engine.getPage('test/clean');
|
||||
expect(page).not.toBeNull();
|
||||
const fm = page!.frontmatter as Record<string, unknown>;
|
||||
expect(isEmbedSkipped(fm)).toBe(false);
|
||||
});
|
||||
});
|
||||
|
||||
test('warn-tier page (50K-500K body) lands normally without embed_skip', async () => {
|
||||
await withIsolatedHome(async () => {
|
||||
const content = FRONTMATTER + 'a'.repeat(100_000);
|
||||
const result = await importFromContent(engine, 'test/warn', content, { noEmbed: true });
|
||||
expect(result.status).toBe('imported');
|
||||
const page = await engine.getPage('test/warn');
|
||||
expect(page).not.toBeNull();
|
||||
const fm = page!.frontmatter as Record<string, unknown>;
|
||||
expect(isEmbedSkipped(fm)).toBe(false);
|
||||
});
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,161 @@
|
||||
import { describe, test, expect } from 'bun:test';
|
||||
import { lintContent } from '../src/commands/lint.ts';
|
||||
|
||||
const MINIMAL_FRONTMATTER = `---
|
||||
title: Test Page
|
||||
type: note
|
||||
created: 2026-05-24
|
||||
---
|
||||
|
||||
`;
|
||||
|
||||
describe('lint — huge-page rule', () => {
|
||||
test('does not fire below warn threshold', () => {
|
||||
const content = MINIMAL_FRONTMATTER + 'a'.repeat(40_000);
|
||||
const issues = lintContent(content, 'test.md');
|
||||
expect(issues.find((i) => i.rule === 'huge-page')).toBeUndefined();
|
||||
});
|
||||
|
||||
test('fires when body exceeds warn threshold (default 50K)', () => {
|
||||
const content = MINIMAL_FRONTMATTER + 'a'.repeat(60_000);
|
||||
const issues = lintContent(content, 'test.md');
|
||||
const huge = issues.find((i) => i.rule === 'huge-page');
|
||||
expect(huge).toBeDefined();
|
||||
expect(huge!.message).toContain('60');
|
||||
expect(huge!.fixable).toBe(false);
|
||||
expect(huge!.line).toBe(1);
|
||||
});
|
||||
|
||||
test('fires with block-threshold language when body exceeds block', () => {
|
||||
const content = MINIMAL_FRONTMATTER + 'a'.repeat(600_000);
|
||||
const issues = lintContent(content, 'test.md');
|
||||
const huge = issues.find((i) => i.rule === 'huge-page');
|
||||
expect(huge).toBeDefined();
|
||||
expect(huge!.message).toContain('block');
|
||||
});
|
||||
|
||||
test('respects custom bytes_warn override', () => {
|
||||
const content = MINIMAL_FRONTMATTER + 'a'.repeat(1000);
|
||||
const issues = lintContent(content, 'test.md', {
|
||||
contentSanity: { bytes_warn: 500, bytes_block: 50_000 },
|
||||
});
|
||||
expect(issues.find((i) => i.rule === 'huge-page')).toBeDefined();
|
||||
});
|
||||
|
||||
test('disabled kill-switch suppresses huge-page rule', () => {
|
||||
const content = MINIMAL_FRONTMATTER + 'a'.repeat(600_000);
|
||||
const issues = lintContent(content, 'test.md', {
|
||||
contentSanity: { disabled: true },
|
||||
});
|
||||
expect(issues.find((i) => i.rule === 'huge-page')).toBeUndefined();
|
||||
});
|
||||
});
|
||||
|
||||
describe('lint — scraper-junk rule', () => {
|
||||
test('does not fire on clean content', () => {
|
||||
const content = MINIMAL_FRONTMATTER + 'This is a thoughtful essay about software design.';
|
||||
const issues = lintContent(content, 'test.md');
|
||||
expect(issues.find((i) => i.rule === 'scraper-junk')).toBeUndefined();
|
||||
});
|
||||
|
||||
test('fires when title matches cloudflare_attention_required pattern', () => {
|
||||
const content = `---
|
||||
title: 'Attention Required! | Cloudflare'
|
||||
type: note
|
||||
created: 2026-05-24
|
||||
---
|
||||
|
||||
Body content.`;
|
||||
const issues = lintContent(content, 'test.md');
|
||||
const junk = issues.find((i) => i.rule === 'scraper-junk');
|
||||
expect(junk).toBeDefined();
|
||||
expect(junk!.message).toContain('cloudflare_attention_required');
|
||||
});
|
||||
|
||||
test('fires on access_denied body pattern', () => {
|
||||
const content = MINIMAL_FRONTMATTER + 'Access denied\n\nYou do not have permission.';
|
||||
const issues = lintContent(content, 'test.md');
|
||||
expect(issues.find((i) => i.rule === 'scraper-junk')).toBeDefined();
|
||||
});
|
||||
|
||||
test('operator literal hits also surface', () => {
|
||||
const content = MINIMAL_FRONTMATTER + "You're being blocked from accessing this site.";
|
||||
const issues = lintContent(content, 'test.md', {
|
||||
contentSanity: {
|
||||
operator_literals: [{ name: 'reddit_blocked', substring: "you're being blocked from accessing" }],
|
||||
},
|
||||
});
|
||||
const junk = issues.find((i) => i.rule === 'scraper-junk');
|
||||
expect(junk).toBeDefined();
|
||||
expect(junk!.message).toContain('reddit_blocked');
|
||||
});
|
||||
|
||||
test('junk_patterns_enabled=false suppresses operator literals AND built-ins via consumer wiring', () => {
|
||||
// The assessor honors junk_patterns_enabled implicitly via the
|
||||
// operator_literals=[] passed by runLintCore. Lint here tests the
|
||||
// direct call path: when caller passes junk_patterns_enabled=false,
|
||||
// operator_literals should already be empty (production resolver
|
||||
// handles that gate). This test pins built-in patterns still fire
|
||||
// even when junk_patterns_enabled flag is on the opts but no
|
||||
// literals are passed — i.e., the flag is informational at this
|
||||
// layer; the resolver consults it before constructing opts.
|
||||
const content = `---
|
||||
title: 'Attention Required! | Cloudflare'
|
||||
type: note
|
||||
created: 2026-05-24
|
||||
---
|
||||
|
||||
body`;
|
||||
const issues = lintContent(content, 'test.md', {
|
||||
contentSanity: { junk_patterns_enabled: false, operator_literals: [] },
|
||||
});
|
||||
// Built-in pattern still fires here (resolver doesn't strip
|
||||
// built-ins; only operator literals are gated by the flag).
|
||||
expect(issues.find((i) => i.rule === 'scraper-junk')).toBeDefined();
|
||||
});
|
||||
|
||||
test('disabled kill-switch suppresses scraper-junk rule', () => {
|
||||
const content = `---
|
||||
title: 'Access Denied'
|
||||
type: note
|
||||
created: 2026-05-24
|
||||
---
|
||||
|
||||
body`;
|
||||
const issues = lintContent(content, 'test.md', {
|
||||
contentSanity: { disabled: true },
|
||||
});
|
||||
expect(issues.find((i) => i.rule === 'scraper-junk')).toBeUndefined();
|
||||
});
|
||||
});
|
||||
|
||||
describe('lint — bytes parity with doctor (D2)', () => {
|
||||
test('lint measures body-only bytes (not file bytes)', () => {
|
||||
// A page with large frontmatter but small body should NOT trip
|
||||
// huge-page — the rule keys on body bytes only, matching what the
|
||||
// doctor `oversized_pages` check sees via octet_length(compiled_truth + timeline).
|
||||
const fm = '---\ntitle: Test\ntype: note\ncreated: 2026-05-24\nbig_meta: ' + 'x'.repeat(60_000) + '\n---\n\n';
|
||||
const content = fm + 'small body';
|
||||
const issues = lintContent(content, 'test.md');
|
||||
// The body is "small body" → ~10 bytes. Should NOT trip warn.
|
||||
expect(issues.find((i) => i.rule === 'huge-page')).toBeUndefined();
|
||||
});
|
||||
});
|
||||
|
||||
describe('lint — existing rules unaffected by content-sanity extension', () => {
|
||||
test('LLM preamble rule still fires', () => {
|
||||
// The LLM_PREAMBLES regex anchors on `^Of course\.?\s*Here is` so
|
||||
// we use the period form (not exclamation) for an exact match.
|
||||
const content = `---
|
||||
title: T
|
||||
type: note
|
||||
created: 2026-05-24
|
||||
---
|
||||
|
||||
Of course. Here is the brain page.
|
||||
|
||||
Real content.`;
|
||||
const issues = lintContent(content, 'test.md');
|
||||
expect(issues.find((i) => i.rule === 'llm-preamble')).toBeDefined();
|
||||
});
|
||||
});
|
||||
Reference in New Issue
Block a user