mirror of
https://github.com/garrytan/gbrain.git
synced 2026-07-28 06:23:01 +00:00
56aac51a08
* feat(extract): quarantine lane for auto-extracted entities from untrusted input (#160) extractAndEnrich regex-extracts entity names from arbitrary ingested text and creates people/ + companies/ stub pages. Those writes are now trust- gated end to end: - src/core/extraction-review.ts: new marker module (sibling of quarantine.ts / embed-skip.ts, frontmatter-key pattern, no migration). Untrusted-input stubs carry `provenance: auto-extracted` + `status: unverified`; the shared unverifiedExtractionFragment() is the single SQL source of truth for every consumer. - enrichment-service: enrichEntity/enrichEntities/extractAndEnrich take EnrichmentTrustOptions; only an explicit trusted:true writes authoritative pages (fail-closed, mirrors the OperationContext.remote invariant). Also threads sourceId through the write path. - retrieval: unverified stubs rank as ordinary content — skipped by the compiled-truth fusion boost (stampUnverifiedExtractions pre-fusion on all three hybrid paths + keyword-only opt-out) and by the people// companies/ namespace source-boost (guard inside buildSourceFactorCase, shared by both engines' search SQL). Results carry `unverified: true`. New engine method getUnverifiedExtractionPageIds in BOTH engines. - ops (contract-first): extract_entities (direct write only for ctx.remote === false + --trusted-extraction; everything else quarantines), extraction_pending (read, source-scoped list), extraction_review (owner-only batch promote/reject; promote flips status to verified keeping provenance for audit, reject soft-deletes). - doctor: unverified_extractions check warns on stubs older than N days (default 7) with the exact review commands. Tests: test/extraction-review.test.ts (PGLite: fail-closed matrix incl. remote-unset, fusion boost skip, review queue, doctor, hostile-transcript e2e proving fake entities land quarantined and rank below a verified page of equal lexical relevance) + test/e2e/extraction-review-postgres.test.ts (live Postgres parity, verified against pgvector:pg16). sql-ranking expectations updated to current state. Docs: KEY_FILES + RETRIEVAL + llms rebuild. Closes #160 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(extract): close vector-arm source-boost gap + harden extract_entities (#160 review round) Adversarial review of the quarantine lane found the people//companies/ 1.2x source factor still applied to unverified stubs inside searchVector's pre-LIMIT re-rank (a different multiplier from the fusion-level 2.0x the lane already cancels — and applied early enough to evict legitimate pages from the candidate pool, which nothing downstream can restore). - buildSourceFactorCase gains an optional unverifiedGuardColumn for the bare-slug re-rank form; both engines' hnsw_candidates CTEs now project the guard predicate as `unverified_stub` and the factor CASE checks it first. Wrong "fusion covers the vector arm" comment corrected. - extract_entities resource guards: 200k-char input cap (loud reject), 200-entity cap surfaced as `truncated` + `entities_found`; the library extractAndEnrich gets the same default cap. (OperationContext has no abort signal field — caps are the bound.) - extraction_review promote is now a targeted JSONB-merge UPDATE instead of putPage, so non-carried columns (page_kind, content_hash) can't be reset by the upsert. - extraction_pending applies buildVisibilityClause (archived-source stubs no longer list). - Wording: op description + module header now state the marker-strip assumption plainly (markers are ordinary frontmatter; the boundary against wholesale rewrite is put_page write authz) and document the CREATE-only scope of the lane. Tests: vector-arm factor-1.0 pinned on BOTH engines (PGLite unit + live Postgres e2e, identical basis embeddings → score ratio is the factor); resource-guard test (oversize reject + 300-entity flood capped at 200); guard-column form pinned in the buildSourceFactorCase unit test. search/ suite (340), sql-ranking, searchvector-maxpool, title-retrieval- arm, rrf-source-key, doctor, ops, cli suites all green; JSONB guards clean. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Garry Tan <garrytan@gmail.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com>