Salvage of PR #2875 (which subsumes the base projection from #2873),
rebased onto current master with the release bookkeeping (VERSION /
package.json / CHANGELOG) dropped, plus one hot-path hardening fix.
Salvaged (verified on this head):
- project trusted message_id / thread_id / Message-ID-gated source_subject
through keyword, chunk-keyword, CJK, and vector paths in BOTH engines
- preserve the citation DTO through alias injection, relational
recall/fanout, two-pass hydration, vector fusion/reranking, and
semantic-cache hits
- raw source_subject is never trusted; only allowlisted `subject` may
supply it, and only when a nonblank Message-ID proves the page is an
email; malformed/non-object frontmatter fails closed (no double-decode)
- source visibility / quarantine / deletion rechecked across indirect
retrieval paths (alias hop, relational hydrate, two-pass expansion,
graph walk, cache-hit gate)
- typed JSON cache scope keys (scalar/set/all) — injective encoding, no
forged-key collisions; store-side write gate skips writeback when the
page-generation clock advanced during the producing search
- KNOBS_HASH_VERSION 12 -> 13 so pre-projection cached DTOs miss instead
of replaying the old shape
Fixed on top of the original head (the flagged hot-path defect):
- cacheScopeKey's forged-id rejection was evaluated inline at the cache
lookup/store call sites inside hybridSearchCached, outside any catch —
an invalid scope id broke the whole search instead of skipping the
cache. The key is now computed once, fail-open: invalid scope =>
cache skipped, search unaffected. Pinned by
test/search/hybrid-cache-scope-failopen.serial.test.ts (fails on the
original head, passes here).
Verified on this exact head: typecheck clean; 379 touched unit tests,
3 serial files (one process each), pglite cache-gate/source-isolation
e2e, and real-PostgreSQL engine-parity (26 pass) + source-routing —
all green. jsonb-pattern/params, key-files-current-state,
test-isolation, progress-to-stdout guards clean.
Closes#2962
Co-authored-by: amtagrwl <amtagrwl@users.noreply.github.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* feat: content-quality gate on sync — quarantine junk + flag boilerplate (#1699)
Three-tier disposition at the importFromContent narrow waist:
- High-confidence junk (Cloudflare/CAPTCHA interstitial patterns + operator
literals) -> quarantine (hidden from search, zero chunks) or reject.
- Fuzzy markup-heavy (prose-vs-markup ratio, warn-tier window, code-exempt)
-> content_flag marker, stays searchable, agent warned.
- Oversize -> existing embed_skip soft-block + content_flag:oversized warning.
Agent-warning channel: SearchResult.content_flag (stamped in hybridSearch +
the keyword-only search op) and a top-level content_flag on get_page.
New quarantine.ts markers, gbrain quarantine CLI (list/clear/scan), doctor
quarantined_pages + flagged_pages checks (engine.executeRaw, works on PGLite),
sources-audit disposition awareness, markup-heavy lint rule, config keys.
Security: gate-owned markers stripped from untrusted (remote MCP) frontmatter
so a write-scoped client can't hide pages or inject the warning channel.
Markers excluded from content_hash so flagged pages don't re-embed every sync.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* chore: bump version and changelog (v0.42.8.0)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs: document content-quality gate (quarantine + content_flag) for v0.42.8.0
Add CLAUDE.md Key Files + Commands entries for the #1699 content-quality
gate: src/core/quarantine.ts, gbrain quarantine CLI (list/clear/scan),
the agent-warning channel (SearchResult.content_flag + get_page), doctor
quarantined_pages/flagged_pages checks, the markup-heavy lint rule,
sources-audit disposition awareness, and the three new content_sanity
config keys. Regenerate llms-full.txt from CLAUDE.md.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
