From bea2d3e6c9f6b232c9d2b42539f18a6755103547 Mon Sep 17 00:00:00 2001 From: Garry Tan Date: Wed, 3 Jun 2026 06:28:21 -0700 Subject: [PATCH] v0.42.13.0 fix(search): archive/ content findable by default, demoted not hard-excluded (#1777) (#1797) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * fix(search): archive/ findable by default — demote not hard-exclude (#1777) Move archive/ out of DEFAULT_HARD_EXCLUDES into a 0.5 source-boost demote so archived historical content is findable by default, ranked below curated content. Add a hidden_by_search_policy doctor check so the surviving exclude policy (test/, attachments/, .raw/) is auditable. Bump KNOBS_HASH_VERSION 8->9 so the policy change invalidates archive-excluded query_cache rows. Co-Authored-By: Claude Opus 4.8 (1M context) * chore: bump version and changelog (v0.42.13.0) Co-Authored-By: Claude Opus 4.8 (1M context) * docs: correct search-exclude.test.ts annotation for archive demote (#1777) Co-Authored-By: Claude Opus 4.8 (1M context) --------- Co-authored-by: Claude Opus 4.8 (1M context) --- CHANGELOG.md | 58 ++++++++ VERSION | 2 +- docs/architecture/KEY_FILES.md | 2 +- docs/architecture/RETRIEVAL.md | 4 +- package.json | 2 +- src/commands/doctor.ts | 105 ++++++++++++++ src/core/doctor-categories.ts | 1 + src/core/postgres-engine.ts | 5 +- src/core/search/mode.ts | 10 +- src/core/search/source-boost.ts | 14 +- src/core/search/sql-ranking.ts | 11 +- src/core/types.ts | 4 +- test/cross-modal-phase1.test.ts | 5 +- test/doctor-hidden-by-search-policy.test.ts | 153 ++++++++++++++++++++ test/e2e/search-exclude.test.ts | 111 ++++++++++++-- test/search-alias-resolved-boost.test.ts | 4 +- test/search-mode.test.ts | 13 +- test/search/knobs-hash-reranker.test.ts | 8 +- test/sql-ranking.test.ts | 37 ++++- 19 files changed, 511 insertions(+), 38 deletions(-) create mode 100644 test/doctor-hidden-by-search-policy.test.ts diff --git a/CHANGELOG.md b/CHANGELOG.md index 08e57cd7a..2333ff0fa 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,64 @@ All notable changes to GBrain will be documented in this file. +## [0.42.13.0] - 2026-06-02 + +**Pages under `archive/` are findable again.** If you committed a note to your brain under `archive/` (old conversation exports, prior-system logs, notes you filed away), GBrain was embedding it and graphing it but then hiding it from every search. You would search for an exact phrase you knew was in the page, get nothing back, and conclude the page did not exist. It did. A hardcoded list was quietly dropping the whole `archive/` subtree from results unless you knew to pass a special flag. + +The rule should be simple: if it is committed to your brain, it should be findable. So `archive/` is no longer hidden. It is now ranked a bit lower than your curated content (essays, concepts, people) so old archived material does not crowd out your best pages, but it shows up. A genuinely strong match in `archive/` can still rise to the top when the reranker says it is the best answer. + +`test/`, `attachments/`, and `.raw/` stay hidden. Those are real noise. + +You do not need to do anything. Re-run a search that used to come up empty: + +``` +gbrain search "" +``` + +New `gbrain doctor` check `hidden_by_search_policy` reports how many pages are still withheld from default search by the remaining exclude prefixes, so an empty result is never silently a policy decision again: + +``` +gbrain doctor --json # look for the hidden_by_search_policy check +``` + +**What you would see in a search for "widget"** when `concepts/widget-pattern` and `archive/old/widget-2020` both match: + +| Page | Before | After | +|---|---|---| +| `concepts/widget-pattern` | returned (rank 1) | returned (rank 1) | +| `archive/old/widget-2020` | **withheld** | returned (rank 2, demoted) | +| `test/fixtures/widget` | withheld | withheld | + +**Things to watch after upgrade:** the search result cache gets a one-time full refresh on first use (the exclude-policy change is folded into the cache key so stale archive-excluded results can not be served); it refills within an hour. If you run the contradictions probe, archived pages now classify in its `bulk` source tier instead of `other` — archive is bulk-ish historical content, so that is the right bucket. + +## To take advantage of v0.42.13.0 + +`gbrain upgrade` handles this automatically. There is no migration. If a search that should return an archived page still comes up empty after upgrade: + +1. **Confirm the page is in the brain and chunked:** + ```bash + gbrain doctor --json # hidden_by_search_policy lists what's withheld by prefix + ``` + `archive/` should NOT appear in that list. `test/` / `attachments/` / `.raw/` may. +2. **Re-run the search:** + ```bash + gbrain search "" + ``` +3. **If it still misses,** the page may genuinely have no chunks (never embedded). Run `gbrain embed --stale`, then search again. +4. **If something looks wrong,** file an issue: https://github.com/garrytan/gbrain/issues with your `gbrain doctor` output and the query. + +### Itemized changes + +- `archive/` removed from `DEFAULT_HARD_EXCLUDES` and added to `DEFAULT_SOURCE_BOOSTS` at `0.5` (`src/core/search/source-boost.ts`). Findable by default, ranked below curated content. The demote is a prior applied in the SQL/fusion layer; the cross-encoder reranker can still promote a strongly-matching archive page that survives the demote into the rerank candidate window. +- New `gbrain doctor` check `hidden_by_search_policy` (`src/commands/doctor.ts`, wired into both the local and remote/thin-client paths; `src/core/doctor-categories.ts`). Counts chunked pages withheld by each active exclude prefix in one SQL query, reusing the canonical `resolveHardExcludes` + `buildVisibilityClause` + `escapeLikePattern` so the count mirrors what search actually filters. `ok` for intentional default excludes (with a prescriptive, agent-readable message), `warn` only when a non-default `GBRAIN_SEARCH_EXCLUDE` prefix is hiding pages. +- `KNOBS_HASH_VERSION` bumped 8 to 9 (`src/core/search/mode.ts`). The search-exclude policy is not part of the cache key, so the bump is what invalidates archive-excluded `query_cache` rows on upgrade. One-time global cache cold-miss; refills within `cache.ttl_seconds`. +- `escapeLikePattern` is now exported from `src/core/search/sql-ranking.ts` (was test-only) so the doctor check escapes env-supplied prefixes with `ESCAPE '\'` instead of re-implementing it. +- Docs + comment sweep: `docs/architecture/RETRIEVAL.md`, `src/core/types.ts`, `src/core/postgres-engine.ts` no longer describe `archive/` as a default hard-exclude. + +### For contributors + +- Verified on both PGLite (in-memory e2e) and real Postgres (seeded container smoke): archive findable + demoted below curated, `test/`/`.raw/`/`attachments/` still hidden, the new doctor SQL runs clean on both engines. +- Side-effect documented above: adding `archive/: 0.5` to `DEFAULT_SOURCE_BOOSTS` reclassifies archive pages in the contradictions probe's source-tier breakdown (`src/core/eval-contradictions/cross-source.ts`) from `other` to `bulk` (boost < 0.95). Benign; no code change. ## [0.42.12.0] - 2026-06-02 **GBrain now keeps itself current the way your coding agent already keeps gstack current: it rides every invocation.** Until now, a gbrain install would quietly drift. You'd run an old binary against a newer brain schema, `gbrain doctor` would mutter about it, and nobody would act. There was no nudge at the moment you'd actually see it. diff --git a/VERSION b/VERSION index dd4f51a73..f4b397c9d 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -0.42.12.0 \ No newline at end of file +0.42.13.0 diff --git a/docs/architecture/KEY_FILES.md b/docs/architecture/KEY_FILES.md index 1f89c974f..df722f0bc 100644 --- a/docs/architecture/KEY_FILES.md +++ b/docs/architecture/KEY_FILES.md @@ -51,7 +51,7 @@ per-release `**vX.Y.Z:**` narration — CI enforces this - `src/core/search/` — Hybrid search: vector + keyword + RRF + multi-query expansion + dedup. `searchKeyword`/`searchKeywordChunks`/`searchVector` apply source-aware ranking at the SQL layer (curated content like `originals/`, `concepts/`, `writing/` outranks bulk content like `/chat/`, `daily/`, `media/x/`). `searchVector` uses a two-stage CTE so source-boost re-ranking doesn't kill the HNSW index. Hard-exclude prefixes (`test/`, `archive/`, `attachments/`, `.raw/` by default) filter at retrieval, not post-rank. Both gates honor `detail !== 'high'` so temporal queries surface chat pages normally. - `src/core/search/intent.ts` — Query intent classifier (entity/temporal/event/general → auto-selects detail level). - `src/core/search/eval.ts` — Retrieval eval harness: P@k, R@k, MRR, nDCG@k metrics + runEval() orchestrator. -- `src/core/search/source-boost.ts` — Source-type boost map keyed by slug prefix. `DEFAULT_SOURCE_BOOSTS` (originals/ 1.5, concepts/ 1.3, writing/ 1.4, people/companies/deals/ 1.2, daily/ 0.8, media/x/ 0.7, /chat/ 0.5) and `DEFAULT_HARD_EXCLUDES` (test/, archive/, attachments/, .raw/). `parseSourceBoostEnv`/`parseHardExcludesEnv` parse comma-separated `prefix:factor` pairs from `GBRAIN_SOURCE_BOOST`/`GBRAIN_SEARCH_EXCLUDE`. `resolveBoostMap` and `resolveHardExcludes` merge defaults + env + caller `SearchOpts.exclude_slug_prefixes`/`include_slug_prefixes`. +- `src/core/search/source-boost.ts` — Source-type boost map keyed by slug prefix. `DEFAULT_SOURCE_BOOSTS` (originals/ 1.5, concepts/ 1.3, writing/ 1.4, people/companies/deals/ 1.2, daily/ 0.8, media/x/ 0.7, /chat/ 0.5, archive/ 0.5, extracts/ 0.3) and `DEFAULT_HARD_EXCLUDES` (test/, attachments/, .raw/). `archive/` is DEMOTED (findable, ranked below curated), not hard-excluded — archive holds high-signal history users expect to retrieve; the demote is a prior at the SQL/fusion layer and the cross-encoder reranker can still promote a strongly-matching archive page. `parseSourceBoostEnv`/`parseHardExcludesEnv` parse comma-separated `prefix:factor` pairs from `GBRAIN_SOURCE_BOOST`/`GBRAIN_SEARCH_EXCLUDE`. `resolveBoostMap` and `resolveHardExcludes` merge defaults + env + caller `SearchOpts.exclude_slug_prefixes`/`include_slug_prefixes`. The surviving exclude policy is auditable via the `hidden_by_search_policy` doctor check (`src/commands/doctor.ts`, local + remote paths) which counts chunked pages withheld per active exclude prefix, reusing `resolveHardExcludes` + `buildVisibilityClause` + the exported `escapeLikePattern`. - `src/core/search/sql-ranking.ts` — Pure SQL string builders. `buildSourceFactorCase(slugColumn, boostMap, detail)` emits a CASE with longest-prefix-match wins (returns literal `'1.0'` when `detail === 'high'` for temporal-bypass parity with COMPILED_TRUTH_BOOST). `buildHardExcludeClause(slugColumn, prefixes)` emits `NOT (col LIKE 'p1%' OR col LIKE 'p2%')` — OR-chain wrapped in NOT, NOT `NOT LIKE ALL/ANY` (those quantifiers don't express set-exclusion). LIKE meta-character escape covers all three of `%`, `_`, AND `\` (backslash is Postgres LIKE's default escape char). Single-quote doubling on SQL string literals so injection-style inputs are inert text. `buildBestPerPagePoolCte(...)` is the shared per-page max-pool CTE both engines' `searchVector` inject — instead of returning the single best chunk per page from an inner `ORDER BY embedding <=> vec LIMIT N` (which let a page lose to a neighbor on ONE weak chunk while its strong chunk sat just below the inner cut), the CTE pools the BEST chunk score per `(source_id, slug)` composite key so a page surfaces on its strongest evidence; composite key (not bare slug) keeps multi-source brains correct; single source of truth so the two engines can't drift. - `src/core/search/title-match.ts` — pure, zero-I/O title-phrase matcher shared by the production title boost AND NamedThingBench (no drift). `isTitlePhraseMatch(query, title)` returns true when the normalized query is a contiguous token run inside the title with `>= MIN_CONTENT_TOKENS=2` non-stopword tokens, OR an exact full-title match (covers deliberate 1-word chosen names like "Mingtang"). Token-boundary matching (never raw substring, so "art" doesn't match "Bartholomew"); small conservative English stopword set excluded from the content-token floor (guards against promoting generic pages on stopword-y queries); NFKC normalize so CJK / width variants converge. Exports `tokenizeTitle` + `__test__` internals. - `src/core/search/alias-normalize.ts` — ONE normalizer shared by the WRITE path (ingest projects frontmatter `aliases:` into `page_aliases`) and the READ path (search matches query against `page_aliases`), so stored aliases can't silently fail to match queries via divergent normalization (same single-source posture as `cjk.ts`). `normalizeAlias(raw)` does NFKC + lowercase + whitespace-collapse + trim + strip one layer of wrapping quotes/brackets; returns `''` for empty (callers MUST skip empty aliases). `normalizeAliasList(value)` coerces a frontmatter scalar / array / comma-list / garbage into a deduped list of normalized non-empty aliases — used by both the ingest projection and the `reindex --aliases` backfill. diff --git a/docs/architecture/RETRIEVAL.md b/docs/architecture/RETRIEVAL.md index 64ec0aed5..6193aa7ac 100644 --- a/docs/architecture/RETRIEVAL.md +++ b/docs/architecture/RETRIEVAL.md @@ -54,7 +54,9 @@ The cost: +150ms p50 latency, ~$0.025/M tokens. Disabled with `gbrain config set ## Source-aware ranking -Hybrid search applies a source-factor CASE expression at the SQL layer (lives in `src/core/search/sql-ranking.ts`). Curated content like `originals/`, `concepts/`, `writing/` outranks bulk content like `your-openclaw/chat/`, `daily/`, `media/x/`. Hard-exclude prefixes (`test/`, `archive/`, `attachments/`, `.raw/`) filter at retrieval, not post-rank. +Hybrid search applies a source-factor CASE expression at the SQL layer (lives in `src/core/search/sql-ranking.ts`). Curated content like `originals/`, `concepts/`, `writing/` outranks bulk content like `your-openclaw/chat/`, `daily/`, `media/x/`. Hard-exclude prefixes (`test/`, `attachments/`, `.raw/`) filter at retrieval, not post-rank. + +`archive/` is deliberately NOT hard-excluded (issue #1777): it holds high-signal historical content users expect to find, so it is demoted (`0.5x` in `DEFAULT_SOURCE_BOOSTS`), not hidden. The demote is a prior applied in the outer SQL re-rank; the cross-encoder reranker (balanced/tokenmax modes) can still PROMOTE an archive page that survives the demote into the rerank candidate window — it is not an unconditional suppression. `gbrain doctor`'s `hidden_by_search_policy` check reports how many chunked pages remain hidden by the surviving exclude prefixes. The boost map is configurable via `GBRAIN_SOURCE_BOOST` env var or per-call `SearchOpts.exclude_slug_prefixes`. Temporal queries (`detail: 'high'`) bypass the boost so chat pages re-surface for time-sensitive lookups. diff --git a/package.json b/package.json index fac575721..968f841d9 100644 --- a/package.json +++ b/package.json @@ -143,5 +143,5 @@ "bun": ">=1.3.10" }, "license": "MIT", - "version": "0.42.12.0" + "version": "0.42.13.0" } diff --git a/src/commands/doctor.ts b/src/commands/doctor.ts index 042c79cbe..f7b5d637c 100644 --- a/src/commands/doctor.ts +++ b/src/commands/doctor.ts @@ -40,6 +40,12 @@ import { lagFromContentMs } from '../core/source-health.ts'; import { CHUNKER_VERSION } from '../core/chunkers/code.ts'; import { LINK_EXTRACTOR_VERSION_TS } from '../core/link-extraction.ts'; import { isUndefinedColumnError } from '../core/utils.ts'; +// issue #1777: hidden_by_search_policy — count chunked pages withheld from +// default search by the hard-exclude prefix policy. Reuses the canonical +// exclude resolver + LIKE escaper + visibility clause so the doctor count can't +// drift from what search actually filters. +import { resolveHardExcludes, DEFAULT_HARD_EXCLUDES } from '../core/search/source-boost.ts'; +import { escapeLikePattern, buildVisibilityClause } from '../core/search/sql-ranking.ts'; export interface Check { name: string; @@ -734,6 +740,11 @@ export async function doctorReportRemote(engine: BrainEngine): Promise