mirror of
https://github.com/garrytan/gbrain.git
synced 2026-07-28 14:59:47 +00:00
docs: document content-quality gate (quarantine + content_flag) for v0.42.8.0
Add CLAUDE.md Key Files + Commands entries for the #1699 content-quality gate: src/core/quarantine.ts, gbrain quarantine CLI (list/clear/scan), the agent-warning channel (SearchResult.content_flag + get_page), doctor quarantined_pages/flagged_pages checks, the markup-heavy lint rule, sources-audit disposition awareness, and the three new content_sanity config keys. Regenerate llms-full.txt from CLAUDE.md. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 4.8
parent
5903fd920b
commit
6dd4ced187
@@ -216,7 +216,13 @@ strict behavior when unset.
|
||||
- `src/commands/doctor.ts` extension (v0.40.9.0) — three new checks wired into `runDoctor()` and the JSON envelope: `oversized_pages` (warns on pages exceeding `content_sanity.bytes_warn`), `scraper_junk_pages` (warns on pages that match any junk pattern despite being live in the DB — these escaped pre-v0.40.9.0 ingest), and `content_sanity_audit_recent` (reads the last 7 days of audit events, aggregates by pattern+source). Default scans the 1000 most-recent pages; new `--content-audit` flag opts into a full scan for the cleanup wave. All three are warn-only with paste-ready fix hints (junk page → `gbrain sources audit <id>` + `git rm` source-of-truth, oversize → split or accept).
|
||||
- `src/commands/lint.ts` extension (v0.40.9.0) — two new lint rules: `huge-page` (flags pages exceeding `content_sanity.bytes_warn` threshold) and `scraper-junk` (flags pages matching any junk pattern). Both reuse `assessContent()` from `src/core/content-sanity.ts` so lint, doctor, and ingest share one assessor — adding a junk pattern automatically covers all three surfaces. `lint.ts` lifts DB config when `~/.gbrain/` is reachable (matches what `gbrain config set` writes); falls back to file/env on CI. Pinned by `test/lint-content-sanity.test.ts` (161 lines).
|
||||
- `src/commands/embed.ts` extension (v0.40.9.0) — applies the `embed-skip` filter at all 5 stale-chunk sites: `runEmbedCore --stale`, `runEmbedCore --all`, the `embed-stale` Minion helper, plus both engines' `listStaleChunks` + `countStaleChunks` via `EMBED_SKIP_SQL_FRAGMENT`. A soft-blocked page is queryable by title and slug but its chunks never enter the embed sweep. The shared helper from `src/core/embed-skip.ts` is the regression guard — no per-site ad-hoc filter is allowed. Pinned by `test/embed-skip.test.ts`.
|
||||
- `src/core/import-file.ts` extension (v0.40.9.0) — `importFromContent` is the narrow waist that every ingest path passes through (`gbrain import`, `gbrain sync`, `put_page` MCP, `/ingest` webhook). It now calls `assessContent()` BEFORE chunking; verdict `hard_block_junk_pattern` throws `ContentSanityBlockError` (which every wrapper site already catches via its exception flow); verdict `warn_oversize OR oversize-without-junk` sets `frontmatter.embed_skip` via `buildEmbedSkipMarker()` AND deletes any pre-existing chunks for the page in the same transaction so search can't surface stale chunks against content that's now soft-blocked. `gbrain import` honors `errors > 0` for non-zero exit (was silently exit-0 on failed files). `classifyErrorCode` in `src/core/sync.ts` recognizes the new `PAGE_JUNK_PATTERN` code so sync-failures.jsonl grouping bins these correctly. Pinned by `test/import-file-content-sanity.test.ts` (206 lines). — shared library for the v0.12.0 graph layer. extractEntityRefs (canonical, replaces backlinks.ts duplicate) matches both `[Name](people/slug)` markdown links and Obsidian `[[people/slug|Name]]` wikilinks as of v0.12.3. extractPageLinks, inferLinkType heuristics (attended/works_at/invested_in/founded/advises/source/mentions), parseTimelineEntries, isAutoLinkEnabled config helper. `DIR_PATTERN` covers `people`, `companies`, `deals`, `topics`, `concepts`, `projects`, `entities`, `tech`, `finance`, `personal`, `openclaw`. Used by extract.ts, operations.ts auto-link post-hook, and backlinks.ts.
|
||||
- `src/core/import-file.ts` extension (v0.40.9.0) — `importFromContent` is the narrow waist that every ingest path passes through (`gbrain import`, `gbrain sync`, `put_page` MCP, `/ingest` webhook). It now calls `assessContent()` BEFORE chunking; verdict `hard_block_junk_pattern` throws `ContentSanityBlockError` (which every wrapper site already catches via its exception flow); verdict `warn_oversize OR oversize-without-junk` sets `frontmatter.embed_skip` via `buildEmbedSkipMarker()` AND deletes any pre-existing chunks for the page in the same transaction so search can't surface stale chunks against content that's now soft-blocked. `gbrain import` honors `errors > 0` for non-zero exit (was silently exit-0 on failed files). `classifyErrorCode` in `src/core/sync.ts` recognizes the new `PAGE_JUNK_PATTERN` code so sync-failures.jsonl grouping bins these correctly. Pinned by `test/import-file-content-sanity.test.ts` (206 lines). **v0.42.8.0 (#1699 content-quality gate):** `importFromContent` (the narrow waist for `gbrain import` / `gbrain sync` / `put_page` MCP / `/ingest`) now runs a three-tier disposition via `assessContentSanity` from `src/core/content-sanity.ts`. (1) High-confidence junk (built-in Cloudflare/CAPTCHA interstitial patterns + operator literals) → QUARANTINE: stamps the `quarantine` frontmatter marker, writes ZERO chunks, hides the page from search; OR REJECT (throw → sync-failure) when `content_sanity.junk_disposition` is `reject`. (2) Fuzzy markup-heavy (prose-vs-markup ratio above `content_sanity.max_markup_ratio`, warn-tier byte window, code pages exempt) → `content_flag:markup_heavy` marker — the page stays fully searchable, the marker rides search results + get_page to warn the agent. (3) Oversize → existing `embed_skip` soft-block PLUS a `content_flag:oversized` marker. Gate-owned markers (`quarantine`, `content_flag`) are STRIPPED from untrusted (remote MCP, `ctx.remote !== false`) frontmatter so a write-scoped client can't hide pages or forge the warning channel; markers are excluded from `content_hash` so a flagged page doesn't re-embed every sync. Pinned by `test/import-file-content-sanity.test.ts`. — shared library for the v0.12.0 graph layer. extractEntityRefs (canonical, replaces backlinks.ts duplicate) matches both `[Name](people/slug)` markdown links and Obsidian `[[people/slug|Name]]` wikilinks as of v0.12.3. extractPageLinks, inferLinkType heuristics (attended/works_at/invested_in/founded/advises/source/mentions), parseTimelineEntries, isAutoLinkEnabled config helper. `DIR_PATTERN` covers `people`, `companies`, `deals`, `topics`, `concepts`, `projects`, `entities`, `tech`, `finance`, `personal`, `openclaw`. Used by extract.ts, operations.ts auto-link post-hook, and backlinks.ts.
|
||||
- `src/core/quarantine.ts` (v0.42.8.0, NEW, #1699) — the two frontmatter markers the content-quality gate writes, sibling of `src/core/embed-skip.ts` (same marker-as-JSONB-object pattern, same JSONB `?` existence check that works on Postgres AND PGLite; no schema migration — both are frontmatter JSONB keys). `quarantine` (key `QUARANTINE_KEY`) HIDES: set ONLY for high-confidence junk, writes zero chunks, excluded from search via `quarantineFilterFragment(pageAlias)` / `QUARANTINE_FILTER_FRAGMENT` (the `p`-aliased constant), the single source of truth `buildVisibilityClause` calls so the search filter and the marker key can't drift. `content_flag` (key `CONTENT_FLAG_KEY`) WARNS, does NOT hide: set for fuzzy markup-heavy / oversize, the page stays searchable, the marker is READ INTO search/get_page output — deliberately NO SQL filter fragment. Three distinct markers, three reasons (never overloaded): `embed_skip` = oversized-but-clean, `quarantine` = junk hidden, `content_flag` = odd-examine-still-here; a page can carry more than one (oversize → embed_skip + content_flag:oversized) and each clears independently. Exports `buildQuarantineMarker` / `isQuarantined` / `filterOutQuarantined`, `buildContentFlagMarker` / `getContentFlag` / `hasContentFlag`, plus the two key constants. Pinned by `test/quarantine.test.ts`.
|
||||
- `src/core/content-sanity.ts` extension (v0.42.8.0, #1699) — new `assessContentSanity(opts): SanityAssessment` returns the three-tier disposition (`shouldQuarantine` / `shouldFlag` + reason/detail) consumed by `importFromContent` and `gbrain quarantine scan`. Adds the fuzzy prose-vs-markup ratio pass (markup chars / total chars above `max_markup_ratio`, default 0.85; code pages exempt; gated by `prose_check_enabled`, default true) on top of the v0.40.9.0 byte + junk-pattern passes. Three new config knobs: `content_sanity.junk_disposition` (`quarantine` default | `reject`; no env override — a destructive flip belongs in explicit config), `content_sanity.max_markup_ratio` (default 0.85, env `GBRAIN_MAX_MARKUP_RATIO`, clamped `(0,1]`), `content_sanity.prose_check_enabled` (default true). Same env > file > DB > defaults resolution chain as the v0.40.9.0 knobs. Pinned by `test/content-sanity.test.ts`.
|
||||
- `src/commands/quarantine.ts` (v0.42.8.0, NEW, #1699) — `gbrain quarantine <list|clear|scan>` operator surface for the content-quality gate. `list [--json] [--include-flagged]` paginates `listPages` and reports quarantined (HIDDEN) pages, optionally also `content_flag` (FLAGGED, searchable) pages. `clear <slug> [--force] [--no-embed] [--json]` drops both markers and re-imports through the normal pipeline so the page re-chunks + re-embeds and becomes searchable; the gate re-runs on import so genuinely-junk pages re-quarantine (exit 1) unless `--force` sets `GBRAIN_NO_SANITY=1` for that one import. `scan [--limit N] [--apply] [--no-embed] [--json]` re-assesses already-ingested pages through the gate so junk that predates the gate gets marked (unchanged content short-circuits normal sync, so it never re-assesses otherwise); dry-run uses the SAME effective `content_sanity` config thresholds `--apply` will use, idempotent (skips already-marked pages), `--apply` re-imports with `forceRechunk` to set markers + (for quarantine) drop chunks. Dispatched in `cli.ts`. Pinned by `test/quarantine-cli.test.ts`.
|
||||
- `src/core/search/hybrid.ts` + `src/core/search/sql-ranking.ts` + `src/core/operations.ts` + `src/core/types.ts` extensions (v0.42.8.0, #1699) — agent-warning channel. `SearchResult.content_flag?: {reason, detail}` (new optional field in `types.ts`) is stamped post-fusion by `stampContentFlags` (the v0.41.34 `stampEvidence` precedent) in `hybridSearch` AND in the keyword-only `search` MCP op so both retrieval paths surface the marker. `get_page` returns a top-level `content_flag` parallel field via `getContentFlag(page.frontmatter)`. `buildVisibilityClause` (sql-ranking.ts) now ANDs in `QUARANTINE_FILTER_FRAGMENT` so quarantined pages are excluded from all six search call sites (alongside the existing soft-delete + archived-source filters). Pinned by `test/sql-ranking.test.ts` + `test/e2e/quarantine-search-exclusion.test.ts`.
|
||||
- `src/commands/doctor.ts` extension (v0.42.8.0, #1699) — two new checks wired into `runDoctor()` + the JSON envelope: `quarantined_pages` (counts pages carrying the `quarantine` marker via `engine.executeRaw` JSONB `?` existence, works on PGLite + Postgres; warn-only with a `gbrain quarantine list` hint) and `flagged_pages` (counts `content_flag` pages — searchable but odd; warn-only). Both skip gracefully (status ok, "Skipped") on engines/brains where the probe errors. Pinned by `test/doctor.test.ts`.
|
||||
- `src/commands/lint.ts` + `src/commands/sources.ts` extensions (v0.42.8.0, #1699) — `gbrain lint` gains a `markup-heavy` rule (flags pages whose prose-vs-markup ratio exceeds `content_sanity.max_markup_ratio`, reusing `assessContentSanity` so lint/gate/scan share one assessor); pinned by `test/lint-content-sanity.test.ts`. `gbrain sources audit <id>` becomes disposition-aware: its dry-run disk scan now reports would-quarantine / would-reject / would-flag counts driven by the effective `content_sanity.junk_disposition` + markup config, so an operator previews the gate's verdict before sync. The `content-sanity-audit` JSONL (`src/core/audit/content-sanity-audit.ts`) records the new quarantine/flag dispositions.
|
||||
- `src/core/zombie-reap.ts` (v0.28.1) — idempotent `installSigchldHandler()` so JS-spawned children get reaped via Bun's internal `waitpid()`. Bun (like Node) only auto-reaps when a SIGCHLD listener is registered; without it, every child the worker spawns (shell jobs, embed batches, sub-agents) becomes a zombie on exit and holds connection slots. Called once at module load from `src/cli.ts` (with Windows platform guard — SIGCHLD doesn't exist on Windows). Cross-file leak guard via `_uninstallSigchldHandlerForTests()` for tests. Layer 1 of the three-layer zombie defense; Layer 2 is tini-as-PID-1 wrapping the worker subtree (via `src/core/minions/spawn-helpers.ts`); Layer 3 is the container's own tini for hard Bun crashes.
|
||||
- `src/core/minions/` — Minions job queue: BullMQ-inspired, Postgres-native (queue, worker, backoff, types, protected-names, quiet-hours, stagger, handlers/shell).
|
||||
- `src/core/minions/queue.ts` — MinionQueue class (submit, claim, complete, fail, stall detection, parent-child, depth/child-cap, per-job timeouts, cascade-kill, attachments, idempotency keys, child_done inbox, removeOnComplete/Fail). `add()` takes a 4th `trusted` arg (separate from `opts` to prevent spread leakage); protected names in `PROTECTED_JOB_NAMES` require `{allowProtectedSubmit: true}` and the check runs trim-normalized (whitespace-bypass safe). v0.14.1 #219: `add()` plumbs `max_stalled` through with a `[1, 100]` clamp; omitted values let the schema DEFAULT (5) kick in. v0.19.0: `handleWallClockTimeouts(lockDurationMs)` is Layer 3 kill shot for jobs where `FOR UPDATE SKIP LOCKED` stall detection and the timeout sweep both fail to evict (wedged worker holding a row lock via a pending transaction). v0.19.1: `maxWaiting` coalesce path now uses `pg_advisory_xact_lock` keyed on `(name, queue)` to serialize concurrent submits for the same key, and filters on `queue` in addition to `name` so cross-queue same-name jobs don't suppress each other.
|
||||
@@ -648,6 +654,15 @@ schema_pack_writability, schema_pack_mutation_audit).
|
||||
|
||||
Run `gbrain --help` or `gbrain --tools-json` for full command reference.
|
||||
|
||||
Key commands added in v0.42.8.0 (content-quality gate on sync, #1699):
|
||||
- `gbrain quarantine list [--json] [--include-flagged]` — list pages the content-quality gate HID as high-confidence junk (`quarantine` marker). `--include-flagged` also lists `content_flag` pages (searchable but odd: markup-heavy / oversize).
|
||||
- `gbrain quarantine clear <slug> [--force] [--no-embed] [--json]` — drop the markers and re-import so the page re-chunks + re-embeds and becomes searchable. The gate re-runs on import; genuinely-junk pages re-quarantine (exit 1) unless `--force` bypasses the gate for that one import.
|
||||
- `gbrain quarantine scan [--limit N] [--apply] [--no-embed] [--json]` — re-assess already-ingested pages through the gate so junk that predates the gate gets marked (unchanged content short-circuits normal sync, so it never re-assesses otherwise). Dry-run by default (counts would-quarantine / would-flag); `--apply` sets the markers.
|
||||
- The sync gate is automatic: `gbrain import` / `gbrain sync` / `put_page` / `/ingest` now QUARANTINE high-confidence junk (hidden from search, zero chunks) or REJECT it (`content_sanity.junk_disposition: reject`), and FLAG fuzzy markup-heavy / oversize pages (`content_flag`, still searchable, agent warned).
|
||||
- Agent-warning channel: `search` / `query` results and `get_page` now carry an optional `content_flag: {reason, detail}` so an agent knows a returned page looks like boilerplate or is unusually large before trusting it.
|
||||
- `gbrain doctor` gains `quarantined_pages` + `flagged_pages` checks (warn-only, work on PGLite + Postgres). `gbrain lint` gains a `markup-heavy` rule. `gbrain sources audit <id>` previews the gate's disposition (would-quarantine / would-reject / would-flag) before sync.
|
||||
- Config knobs (`gbrain config set ...`): `content_sanity.junk_disposition` (`quarantine` default | `reject`), `content_sanity.max_markup_ratio` (default 0.85, env `GBRAIN_MAX_MARKUP_RATIO`), `content_sanity.prose_check_enabled` (default true).
|
||||
|
||||
Key commands added in v0.7:
|
||||
- `gbrain init` — defaults to PGLite (no Supabase needed), scans repo size, suggests Supabase for 1000+ files
|
||||
- `gbrain migrate --to supabase` / `gbrain migrate --to pglite` — bidirectional engine migration
|
||||
|
||||
+16
-1
@@ -358,7 +358,13 @@ strict behavior when unset.
|
||||
- `src/commands/doctor.ts` extension (v0.40.9.0) — three new checks wired into `runDoctor()` and the JSON envelope: `oversized_pages` (warns on pages exceeding `content_sanity.bytes_warn`), `scraper_junk_pages` (warns on pages that match any junk pattern despite being live in the DB — these escaped pre-v0.40.9.0 ingest), and `content_sanity_audit_recent` (reads the last 7 days of audit events, aggregates by pattern+source). Default scans the 1000 most-recent pages; new `--content-audit` flag opts into a full scan for the cleanup wave. All three are warn-only with paste-ready fix hints (junk page → `gbrain sources audit <id>` + `git rm` source-of-truth, oversize → split or accept).
|
||||
- `src/commands/lint.ts` extension (v0.40.9.0) — two new lint rules: `huge-page` (flags pages exceeding `content_sanity.bytes_warn` threshold) and `scraper-junk` (flags pages matching any junk pattern). Both reuse `assessContent()` from `src/core/content-sanity.ts` so lint, doctor, and ingest share one assessor — adding a junk pattern automatically covers all three surfaces. `lint.ts` lifts DB config when `~/.gbrain/` is reachable (matches what `gbrain config set` writes); falls back to file/env on CI. Pinned by `test/lint-content-sanity.test.ts` (161 lines).
|
||||
- `src/commands/embed.ts` extension (v0.40.9.0) — applies the `embed-skip` filter at all 5 stale-chunk sites: `runEmbedCore --stale`, `runEmbedCore --all`, the `embed-stale` Minion helper, plus both engines' `listStaleChunks` + `countStaleChunks` via `EMBED_SKIP_SQL_FRAGMENT`. A soft-blocked page is queryable by title and slug but its chunks never enter the embed sweep. The shared helper from `src/core/embed-skip.ts` is the regression guard — no per-site ad-hoc filter is allowed. Pinned by `test/embed-skip.test.ts`.
|
||||
- `src/core/import-file.ts` extension (v0.40.9.0) — `importFromContent` is the narrow waist that every ingest path passes through (`gbrain import`, `gbrain sync`, `put_page` MCP, `/ingest` webhook). It now calls `assessContent()` BEFORE chunking; verdict `hard_block_junk_pattern` throws `ContentSanityBlockError` (which every wrapper site already catches via its exception flow); verdict `warn_oversize OR oversize-without-junk` sets `frontmatter.embed_skip` via `buildEmbedSkipMarker()` AND deletes any pre-existing chunks for the page in the same transaction so search can't surface stale chunks against content that's now soft-blocked. `gbrain import` honors `errors > 0` for non-zero exit (was silently exit-0 on failed files). `classifyErrorCode` in `src/core/sync.ts` recognizes the new `PAGE_JUNK_PATTERN` code so sync-failures.jsonl grouping bins these correctly. Pinned by `test/import-file-content-sanity.test.ts` (206 lines). — shared library for the v0.12.0 graph layer. extractEntityRefs (canonical, replaces backlinks.ts duplicate) matches both `[Name](people/slug)` markdown links and Obsidian `[[people/slug|Name]]` wikilinks as of v0.12.3. extractPageLinks, inferLinkType heuristics (attended/works_at/invested_in/founded/advises/source/mentions), parseTimelineEntries, isAutoLinkEnabled config helper. `DIR_PATTERN` covers `people`, `companies`, `deals`, `topics`, `concepts`, `projects`, `entities`, `tech`, `finance`, `personal`, `openclaw`. Used by extract.ts, operations.ts auto-link post-hook, and backlinks.ts.
|
||||
- `src/core/import-file.ts` extension (v0.40.9.0) — `importFromContent` is the narrow waist that every ingest path passes through (`gbrain import`, `gbrain sync`, `put_page` MCP, `/ingest` webhook). It now calls `assessContent()` BEFORE chunking; verdict `hard_block_junk_pattern` throws `ContentSanityBlockError` (which every wrapper site already catches via its exception flow); verdict `warn_oversize OR oversize-without-junk` sets `frontmatter.embed_skip` via `buildEmbedSkipMarker()` AND deletes any pre-existing chunks for the page in the same transaction so search can't surface stale chunks against content that's now soft-blocked. `gbrain import` honors `errors > 0` for non-zero exit (was silently exit-0 on failed files). `classifyErrorCode` in `src/core/sync.ts` recognizes the new `PAGE_JUNK_PATTERN` code so sync-failures.jsonl grouping bins these correctly. Pinned by `test/import-file-content-sanity.test.ts` (206 lines). **v0.42.8.0 (#1699 content-quality gate):** `importFromContent` (the narrow waist for `gbrain import` / `gbrain sync` / `put_page` MCP / `/ingest`) now runs a three-tier disposition via `assessContentSanity` from `src/core/content-sanity.ts`. (1) High-confidence junk (built-in Cloudflare/CAPTCHA interstitial patterns + operator literals) → QUARANTINE: stamps the `quarantine` frontmatter marker, writes ZERO chunks, hides the page from search; OR REJECT (throw → sync-failure) when `content_sanity.junk_disposition` is `reject`. (2) Fuzzy markup-heavy (prose-vs-markup ratio above `content_sanity.max_markup_ratio`, warn-tier byte window, code pages exempt) → `content_flag:markup_heavy` marker — the page stays fully searchable, the marker rides search results + get_page to warn the agent. (3) Oversize → existing `embed_skip` soft-block PLUS a `content_flag:oversized` marker. Gate-owned markers (`quarantine`, `content_flag`) are STRIPPED from untrusted (remote MCP, `ctx.remote !== false`) frontmatter so a write-scoped client can't hide pages or forge the warning channel; markers are excluded from `content_hash` so a flagged page doesn't re-embed every sync. Pinned by `test/import-file-content-sanity.test.ts`. — shared library for the v0.12.0 graph layer. extractEntityRefs (canonical, replaces backlinks.ts duplicate) matches both `[Name](people/slug)` markdown links and Obsidian `[[people/slug|Name]]` wikilinks as of v0.12.3. extractPageLinks, inferLinkType heuristics (attended/works_at/invested_in/founded/advises/source/mentions), parseTimelineEntries, isAutoLinkEnabled config helper. `DIR_PATTERN` covers `people`, `companies`, `deals`, `topics`, `concepts`, `projects`, `entities`, `tech`, `finance`, `personal`, `openclaw`. Used by extract.ts, operations.ts auto-link post-hook, and backlinks.ts.
|
||||
- `src/core/quarantine.ts` (v0.42.8.0, NEW, #1699) — the two frontmatter markers the content-quality gate writes, sibling of `src/core/embed-skip.ts` (same marker-as-JSONB-object pattern, same JSONB `?` existence check that works on Postgres AND PGLite; no schema migration — both are frontmatter JSONB keys). `quarantine` (key `QUARANTINE_KEY`) HIDES: set ONLY for high-confidence junk, writes zero chunks, excluded from search via `quarantineFilterFragment(pageAlias)` / `QUARANTINE_FILTER_FRAGMENT` (the `p`-aliased constant), the single source of truth `buildVisibilityClause` calls so the search filter and the marker key can't drift. `content_flag` (key `CONTENT_FLAG_KEY`) WARNS, does NOT hide: set for fuzzy markup-heavy / oversize, the page stays searchable, the marker is READ INTO search/get_page output — deliberately NO SQL filter fragment. Three distinct markers, three reasons (never overloaded): `embed_skip` = oversized-but-clean, `quarantine` = junk hidden, `content_flag` = odd-examine-still-here; a page can carry more than one (oversize → embed_skip + content_flag:oversized) and each clears independently. Exports `buildQuarantineMarker` / `isQuarantined` / `filterOutQuarantined`, `buildContentFlagMarker` / `getContentFlag` / `hasContentFlag`, plus the two key constants. Pinned by `test/quarantine.test.ts`.
|
||||
- `src/core/content-sanity.ts` extension (v0.42.8.0, #1699) — new `assessContentSanity(opts): SanityAssessment` returns the three-tier disposition (`shouldQuarantine` / `shouldFlag` + reason/detail) consumed by `importFromContent` and `gbrain quarantine scan`. Adds the fuzzy prose-vs-markup ratio pass (markup chars / total chars above `max_markup_ratio`, default 0.85; code pages exempt; gated by `prose_check_enabled`, default true) on top of the v0.40.9.0 byte + junk-pattern passes. Three new config knobs: `content_sanity.junk_disposition` (`quarantine` default | `reject`; no env override — a destructive flip belongs in explicit config), `content_sanity.max_markup_ratio` (default 0.85, env `GBRAIN_MAX_MARKUP_RATIO`, clamped `(0,1]`), `content_sanity.prose_check_enabled` (default true). Same env > file > DB > defaults resolution chain as the v0.40.9.0 knobs. Pinned by `test/content-sanity.test.ts`.
|
||||
- `src/commands/quarantine.ts` (v0.42.8.0, NEW, #1699) — `gbrain quarantine <list|clear|scan>` operator surface for the content-quality gate. `list [--json] [--include-flagged]` paginates `listPages` and reports quarantined (HIDDEN) pages, optionally also `content_flag` (FLAGGED, searchable) pages. `clear <slug> [--force] [--no-embed] [--json]` drops both markers and re-imports through the normal pipeline so the page re-chunks + re-embeds and becomes searchable; the gate re-runs on import so genuinely-junk pages re-quarantine (exit 1) unless `--force` sets `GBRAIN_NO_SANITY=1` for that one import. `scan [--limit N] [--apply] [--no-embed] [--json]` re-assesses already-ingested pages through the gate so junk that predates the gate gets marked (unchanged content short-circuits normal sync, so it never re-assesses otherwise); dry-run uses the SAME effective `content_sanity` config thresholds `--apply` will use, idempotent (skips already-marked pages), `--apply` re-imports with `forceRechunk` to set markers + (for quarantine) drop chunks. Dispatched in `cli.ts`. Pinned by `test/quarantine-cli.test.ts`.
|
||||
- `src/core/search/hybrid.ts` + `src/core/search/sql-ranking.ts` + `src/core/operations.ts` + `src/core/types.ts` extensions (v0.42.8.0, #1699) — agent-warning channel. `SearchResult.content_flag?: {reason, detail}` (new optional field in `types.ts`) is stamped post-fusion by `stampContentFlags` (the v0.41.34 `stampEvidence` precedent) in `hybridSearch` AND in the keyword-only `search` MCP op so both retrieval paths surface the marker. `get_page` returns a top-level `content_flag` parallel field via `getContentFlag(page.frontmatter)`. `buildVisibilityClause` (sql-ranking.ts) now ANDs in `QUARANTINE_FILTER_FRAGMENT` so quarantined pages are excluded from all six search call sites (alongside the existing soft-delete + archived-source filters). Pinned by `test/sql-ranking.test.ts` + `test/e2e/quarantine-search-exclusion.test.ts`.
|
||||
- `src/commands/doctor.ts` extension (v0.42.8.0, #1699) — two new checks wired into `runDoctor()` + the JSON envelope: `quarantined_pages` (counts pages carrying the `quarantine` marker via `engine.executeRaw` JSONB `?` existence, works on PGLite + Postgres; warn-only with a `gbrain quarantine list` hint) and `flagged_pages` (counts `content_flag` pages — searchable but odd; warn-only). Both skip gracefully (status ok, "Skipped") on engines/brains where the probe errors. Pinned by `test/doctor.test.ts`.
|
||||
- `src/commands/lint.ts` + `src/commands/sources.ts` extensions (v0.42.8.0, #1699) — `gbrain lint` gains a `markup-heavy` rule (flags pages whose prose-vs-markup ratio exceeds `content_sanity.max_markup_ratio`, reusing `assessContentSanity` so lint/gate/scan share one assessor); pinned by `test/lint-content-sanity.test.ts`. `gbrain sources audit <id>` becomes disposition-aware: its dry-run disk scan now reports would-quarantine / would-reject / would-flag counts driven by the effective `content_sanity.junk_disposition` + markup config, so an operator previews the gate's verdict before sync. The `content-sanity-audit` JSONL (`src/core/audit/content-sanity-audit.ts`) records the new quarantine/flag dispositions.
|
||||
- `src/core/zombie-reap.ts` (v0.28.1) — idempotent `installSigchldHandler()` so JS-spawned children get reaped via Bun's internal `waitpid()`. Bun (like Node) only auto-reaps when a SIGCHLD listener is registered; without it, every child the worker spawns (shell jobs, embed batches, sub-agents) becomes a zombie on exit and holds connection slots. Called once at module load from `src/cli.ts` (with Windows platform guard — SIGCHLD doesn't exist on Windows). Cross-file leak guard via `_uninstallSigchldHandlerForTests()` for tests. Layer 1 of the three-layer zombie defense; Layer 2 is tini-as-PID-1 wrapping the worker subtree (via `src/core/minions/spawn-helpers.ts`); Layer 3 is the container's own tini for hard Bun crashes.
|
||||
- `src/core/minions/` — Minions job queue: BullMQ-inspired, Postgres-native (queue, worker, backoff, types, protected-names, quiet-hours, stagger, handlers/shell).
|
||||
- `src/core/minions/queue.ts` — MinionQueue class (submit, claim, complete, fail, stall detection, parent-child, depth/child-cap, per-job timeouts, cascade-kill, attachments, idempotency keys, child_done inbox, removeOnComplete/Fail). `add()` takes a 4th `trusted` arg (separate from `opts` to prevent spread leakage); protected names in `PROTECTED_JOB_NAMES` require `{allowProtectedSubmit: true}` and the check runs trim-normalized (whitespace-bypass safe). v0.14.1 #219: `add()` plumbs `max_stalled` through with a `[1, 100]` clamp; omitted values let the schema DEFAULT (5) kick in. v0.19.0: `handleWallClockTimeouts(lockDurationMs)` is Layer 3 kill shot for jobs where `FOR UPDATE SKIP LOCKED` stall detection and the timeout sweep both fail to evict (wedged worker holding a row lock via a pending transaction). v0.19.1: `maxWaiting` coalesce path now uses `pg_advisory_xact_lock` keyed on `(name, queue)` to serialize concurrent submits for the same key, and filters on `queue` in addition to `name` so cross-queue same-name jobs don't suppress each other.
|
||||
@@ -790,6 +796,15 @@ schema_pack_writability, schema_pack_mutation_audit).
|
||||
|
||||
Run `gbrain --help` or `gbrain --tools-json` for full command reference.
|
||||
|
||||
Key commands added in v0.42.8.0 (content-quality gate on sync, #1699):
|
||||
- `gbrain quarantine list [--json] [--include-flagged]` — list pages the content-quality gate HID as high-confidence junk (`quarantine` marker). `--include-flagged` also lists `content_flag` pages (searchable but odd: markup-heavy / oversize).
|
||||
- `gbrain quarantine clear <slug> [--force] [--no-embed] [--json]` — drop the markers and re-import so the page re-chunks + re-embeds and becomes searchable. The gate re-runs on import; genuinely-junk pages re-quarantine (exit 1) unless `--force` bypasses the gate for that one import.
|
||||
- `gbrain quarantine scan [--limit N] [--apply] [--no-embed] [--json]` — re-assess already-ingested pages through the gate so junk that predates the gate gets marked (unchanged content short-circuits normal sync, so it never re-assesses otherwise). Dry-run by default (counts would-quarantine / would-flag); `--apply` sets the markers.
|
||||
- The sync gate is automatic: `gbrain import` / `gbrain sync` / `put_page` / `/ingest` now QUARANTINE high-confidence junk (hidden from search, zero chunks) or REJECT it (`content_sanity.junk_disposition: reject`), and FLAG fuzzy markup-heavy / oversize pages (`content_flag`, still searchable, agent warned).
|
||||
- Agent-warning channel: `search` / `query` results and `get_page` now carry an optional `content_flag: {reason, detail}` so an agent knows a returned page looks like boilerplate or is unusually large before trusting it.
|
||||
- `gbrain doctor` gains `quarantined_pages` + `flagged_pages` checks (warn-only, work on PGLite + Postgres). `gbrain lint` gains a `markup-heavy` rule. `gbrain sources audit <id>` previews the gate's disposition (would-quarantine / would-reject / would-flag) before sync.
|
||||
- Config knobs (`gbrain config set ...`): `content_sanity.junk_disposition` (`quarantine` default | `reject`), `content_sanity.max_markup_ratio` (default 0.85, env `GBRAIN_MAX_MARKUP_RATIO`), `content_sanity.prose_check_enabled` (default true).
|
||||
|
||||
Key commands added in v0.7:
|
||||
- `gbrain init` — defaults to PGLite (no Supabase needed), scans repo size, suggests Supabase for 1000+ files
|
||||
- `gbrain migrate --to supabase` / `gbrain migrate --to pglite` — bidirectional engine migration
|
||||
|
||||
Reference in New Issue
Block a user