mirror of
https://github.com/garrytan/gbrain.git
synced 2026-07-27 21:19:18 +00:00
v0.42.13.0 fix(search): archive/ content findable by default, demoted not hard-excluded (#1777) (#1797)
* fix(search): archive/ findable by default — demote not hard-exclude (#1777) Move archive/ out of DEFAULT_HARD_EXCLUDES into a 0.5 source-boost demote so archived historical content is findable by default, ranked below curated content. Add a hidden_by_search_policy doctor check so the surviving exclude policy (test/, attachments/, .raw/) is auditable. Bump KNOBS_HASH_VERSION 8->9 so the policy change invalidates archive-excluded query_cache rows. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore: bump version and changelog (v0.42.13.0) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: correct search-exclude.test.ts annotation for archive demote (#1777) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Garry Tan
co-authored by
Claude Opus 4.8
Claude Opus 4.8
parent
a57d98b813
commit
bea2d3e6c9
@@ -2,6 +2,64 @@
|
||||
|
||||
All notable changes to GBrain will be documented in this file.
|
||||
|
||||
## [0.42.13.0] - 2026-06-02
|
||||
|
||||
**Pages under `archive/` are findable again.** If you committed a note to your brain under `archive/` (old conversation exports, prior-system logs, notes you filed away), GBrain was embedding it and graphing it but then hiding it from every search. You would search for an exact phrase you knew was in the page, get nothing back, and conclude the page did not exist. It did. A hardcoded list was quietly dropping the whole `archive/` subtree from results unless you knew to pass a special flag.
|
||||
|
||||
The rule should be simple: if it is committed to your brain, it should be findable. So `archive/` is no longer hidden. It is now ranked a bit lower than your curated content (essays, concepts, people) so old archived material does not crowd out your best pages, but it shows up. A genuinely strong match in `archive/` can still rise to the top when the reranker says it is the best answer.
|
||||
|
||||
`test/`, `attachments/`, and `.raw/` stay hidden. Those are real noise.
|
||||
|
||||
You do not need to do anything. Re-run a search that used to come up empty:
|
||||
|
||||
```
|
||||
gbrain search "<a phrase you know is in an archived page>"
|
||||
```
|
||||
|
||||
New `gbrain doctor` check `hidden_by_search_policy` reports how many pages are still withheld from default search by the remaining exclude prefixes, so an empty result is never silently a policy decision again:
|
||||
|
||||
```
|
||||
gbrain doctor --json # look for the hidden_by_search_policy check
|
||||
```
|
||||
|
||||
**What you would see in a search for "widget"** when `concepts/widget-pattern` and `archive/old/widget-2020` both match:
|
||||
|
||||
| Page | Before | After |
|
||||
|---|---|---|
|
||||
| `concepts/widget-pattern` | returned (rank 1) | returned (rank 1) |
|
||||
| `archive/old/widget-2020` | **withheld** | returned (rank 2, demoted) |
|
||||
| `test/fixtures/widget` | withheld | withheld |
|
||||
|
||||
**Things to watch after upgrade:** the search result cache gets a one-time full refresh on first use (the exclude-policy change is folded into the cache key so stale archive-excluded results can not be served); it refills within an hour. If you run the contradictions probe, archived pages now classify in its `bulk` source tier instead of `other` — archive is bulk-ish historical content, so that is the right bucket.
|
||||
|
||||
## To take advantage of v0.42.13.0
|
||||
|
||||
`gbrain upgrade` handles this automatically. There is no migration. If a search that should return an archived page still comes up empty after upgrade:
|
||||
|
||||
1. **Confirm the page is in the brain and chunked:**
|
||||
```bash
|
||||
gbrain doctor --json # hidden_by_search_policy lists what's withheld by prefix
|
||||
```
|
||||
`archive/` should NOT appear in that list. `test/` / `attachments/` / `.raw/` may.
|
||||
2. **Re-run the search:**
|
||||
```bash
|
||||
gbrain search "<phrase from the archived page>"
|
||||
```
|
||||
3. **If it still misses,** the page may genuinely have no chunks (never embedded). Run `gbrain embed --stale`, then search again.
|
||||
4. **If something looks wrong,** file an issue: https://github.com/garrytan/gbrain/issues with your `gbrain doctor` output and the query.
|
||||
|
||||
### Itemized changes
|
||||
|
||||
- `archive/` removed from `DEFAULT_HARD_EXCLUDES` and added to `DEFAULT_SOURCE_BOOSTS` at `0.5` (`src/core/search/source-boost.ts`). Findable by default, ranked below curated content. The demote is a prior applied in the SQL/fusion layer; the cross-encoder reranker can still promote a strongly-matching archive page that survives the demote into the rerank candidate window.
|
||||
- New `gbrain doctor` check `hidden_by_search_policy` (`src/commands/doctor.ts`, wired into both the local and remote/thin-client paths; `src/core/doctor-categories.ts`). Counts chunked pages withheld by each active exclude prefix in one SQL query, reusing the canonical `resolveHardExcludes` + `buildVisibilityClause` + `escapeLikePattern` so the count mirrors what search actually filters. `ok` for intentional default excludes (with a prescriptive, agent-readable message), `warn` only when a non-default `GBRAIN_SEARCH_EXCLUDE` prefix is hiding pages.
|
||||
- `KNOBS_HASH_VERSION` bumped 8 to 9 (`src/core/search/mode.ts`). The search-exclude policy is not part of the cache key, so the bump is what invalidates archive-excluded `query_cache` rows on upgrade. One-time global cache cold-miss; refills within `cache.ttl_seconds`.
|
||||
- `escapeLikePattern` is now exported from `src/core/search/sql-ranking.ts` (was test-only) so the doctor check escapes env-supplied prefixes with `ESCAPE '\'` instead of re-implementing it.
|
||||
- Docs + comment sweep: `docs/architecture/RETRIEVAL.md`, `src/core/types.ts`, `src/core/postgres-engine.ts` no longer describe `archive/` as a default hard-exclude.
|
||||
|
||||
### For contributors
|
||||
|
||||
- Verified on both PGLite (in-memory e2e) and real Postgres (seeded container smoke): archive findable + demoted below curated, `test/`/`.raw/`/`attachments/` still hidden, the new doctor SQL runs clean on both engines.
|
||||
- Side-effect documented above: adding `archive/: 0.5` to `DEFAULT_SOURCE_BOOSTS` reclassifies archive pages in the contradictions probe's source-tier breakdown (`src/core/eval-contradictions/cross-source.ts`) from `other` to `bulk` (boost < 0.95). Benign; no code change.
|
||||
## [0.42.12.0] - 2026-06-02
|
||||
|
||||
**GBrain now keeps itself current the way your coding agent already keeps gstack current: it rides every invocation.** Until now, a gbrain install would quietly drift. You'd run an old binary against a newer brain schema, `gbrain doctor` would mutter about it, and nobody would act. There was no nudge at the moment you'd actually see it.
|
||||
|
||||
@@ -51,7 +51,7 @@ per-release `**vX.Y.Z:**` narration — CI enforces this
|
||||
- `src/core/search/` — Hybrid search: vector + keyword + RRF + multi-query expansion + dedup. `searchKeyword`/`searchKeywordChunks`/`searchVector` apply source-aware ranking at the SQL layer (curated content like `originals/`, `concepts/`, `writing/` outranks bulk content like `<fork>/chat/`, `daily/`, `media/x/`). `searchVector` uses a two-stage CTE so source-boost re-ranking doesn't kill the HNSW index. Hard-exclude prefixes (`test/`, `archive/`, `attachments/`, `.raw/` by default) filter at retrieval, not post-rank. Both gates honor `detail !== 'high'` so temporal queries surface chat pages normally.
|
||||
- `src/core/search/intent.ts` — Query intent classifier (entity/temporal/event/general → auto-selects detail level).
|
||||
- `src/core/search/eval.ts` — Retrieval eval harness: P@k, R@k, MRR, nDCG@k metrics + runEval() orchestrator.
|
||||
- `src/core/search/source-boost.ts` — Source-type boost map keyed by slug prefix. `DEFAULT_SOURCE_BOOSTS` (originals/ 1.5, concepts/ 1.3, writing/ 1.4, people/companies/deals/ 1.2, daily/ 0.8, media/x/ 0.7, <fork>/chat/ 0.5) and `DEFAULT_HARD_EXCLUDES` (test/, archive/, attachments/, .raw/). `parseSourceBoostEnv`/`parseHardExcludesEnv` parse comma-separated `prefix:factor` pairs from `GBRAIN_SOURCE_BOOST`/`GBRAIN_SEARCH_EXCLUDE`. `resolveBoostMap` and `resolveHardExcludes` merge defaults + env + caller `SearchOpts.exclude_slug_prefixes`/`include_slug_prefixes`.
|
||||
- `src/core/search/source-boost.ts` — Source-type boost map keyed by slug prefix. `DEFAULT_SOURCE_BOOSTS` (originals/ 1.5, concepts/ 1.3, writing/ 1.4, people/companies/deals/ 1.2, daily/ 0.8, media/x/ 0.7, <fork>/chat/ 0.5, archive/ 0.5, extracts/ 0.3) and `DEFAULT_HARD_EXCLUDES` (test/, attachments/, .raw/). `archive/` is DEMOTED (findable, ranked below curated), not hard-excluded — archive holds high-signal history users expect to retrieve; the demote is a prior at the SQL/fusion layer and the cross-encoder reranker can still promote a strongly-matching archive page. `parseSourceBoostEnv`/`parseHardExcludesEnv` parse comma-separated `prefix:factor` pairs from `GBRAIN_SOURCE_BOOST`/`GBRAIN_SEARCH_EXCLUDE`. `resolveBoostMap` and `resolveHardExcludes` merge defaults + env + caller `SearchOpts.exclude_slug_prefixes`/`include_slug_prefixes`. The surviving exclude policy is auditable via the `hidden_by_search_policy` doctor check (`src/commands/doctor.ts`, local + remote paths) which counts chunked pages withheld per active exclude prefix, reusing `resolveHardExcludes` + `buildVisibilityClause` + the exported `escapeLikePattern`.
|
||||
- `src/core/search/sql-ranking.ts` — Pure SQL string builders. `buildSourceFactorCase(slugColumn, boostMap, detail)` emits a CASE with longest-prefix-match wins (returns literal `'1.0'` when `detail === 'high'` for temporal-bypass parity with COMPILED_TRUTH_BOOST). `buildHardExcludeClause(slugColumn, prefixes)` emits `NOT (col LIKE 'p1%' OR col LIKE 'p2%')` — OR-chain wrapped in NOT, NOT `NOT LIKE ALL/ANY` (those quantifiers don't express set-exclusion). LIKE meta-character escape covers all three of `%`, `_`, AND `\` (backslash is Postgres LIKE's default escape char). Single-quote doubling on SQL string literals so injection-style inputs are inert text. `buildBestPerPagePoolCte(...)` is the shared per-page max-pool CTE both engines' `searchVector` inject — instead of returning the single best chunk per page from an inner `ORDER BY embedding <=> vec LIMIT N` (which let a page lose to a neighbor on ONE weak chunk while its strong chunk sat just below the inner cut), the CTE pools the BEST chunk score per `(source_id, slug)` composite key so a page surfaces on its strongest evidence; composite key (not bare slug) keeps multi-source brains correct; single source of truth so the two engines can't drift.
|
||||
- `src/core/search/title-match.ts` — pure, zero-I/O title-phrase matcher shared by the production title boost AND NamedThingBench (no drift). `isTitlePhraseMatch(query, title)` returns true when the normalized query is a contiguous token run inside the title with `>= MIN_CONTENT_TOKENS=2` non-stopword tokens, OR an exact full-title match (covers deliberate 1-word chosen names like "Mingtang"). Token-boundary matching (never raw substring, so "art" doesn't match "Bartholomew"); small conservative English stopword set excluded from the content-token floor (guards against promoting generic pages on stopword-y queries); NFKC normalize so CJK / width variants converge. Exports `tokenizeTitle` + `__test__` internals.
|
||||
- `src/core/search/alias-normalize.ts` — ONE normalizer shared by the WRITE path (ingest projects frontmatter `aliases:` into `page_aliases`) and the READ path (search matches query against `page_aliases`), so stored aliases can't silently fail to match queries via divergent normalization (same single-source posture as `cjk.ts`). `normalizeAlias(raw)` does NFKC + lowercase + whitespace-collapse + trim + strip one layer of wrapping quotes/brackets; returns `''` for empty (callers MUST skip empty aliases). `normalizeAliasList(value)` coerces a frontmatter scalar / array / comma-list / garbage into a deduped list of normalized non-empty aliases — used by both the ingest projection and the `reindex --aliases` backfill.
|
||||
|
||||
@@ -54,7 +54,9 @@ The cost: +150ms p50 latency, ~$0.025/M tokens. Disabled with `gbrain config set
|
||||
|
||||
## Source-aware ranking
|
||||
|
||||
Hybrid search applies a source-factor CASE expression at the SQL layer (lives in `src/core/search/sql-ranking.ts`). Curated content like `originals/`, `concepts/`, `writing/` outranks bulk content like `your-openclaw/chat/`, `daily/`, `media/x/`. Hard-exclude prefixes (`test/`, `archive/`, `attachments/`, `.raw/`) filter at retrieval, not post-rank.
|
||||
Hybrid search applies a source-factor CASE expression at the SQL layer (lives in `src/core/search/sql-ranking.ts`). Curated content like `originals/`, `concepts/`, `writing/` outranks bulk content like `your-openclaw/chat/`, `daily/`, `media/x/`. Hard-exclude prefixes (`test/`, `attachments/`, `.raw/`) filter at retrieval, not post-rank.
|
||||
|
||||
`archive/` is deliberately NOT hard-excluded (issue #1777): it holds high-signal historical content users expect to find, so it is demoted (`0.5x` in `DEFAULT_SOURCE_BOOSTS`), not hidden. The demote is a prior applied in the outer SQL re-rank; the cross-encoder reranker (balanced/tokenmax modes) can still PROMOTE an archive page that survives the demote into the rerank candidate window — it is not an unconditional suppression. `gbrain doctor`'s `hidden_by_search_policy` check reports how many chunked pages remain hidden by the surviving exclude prefixes.
|
||||
|
||||
The boost map is configurable via `GBRAIN_SOURCE_BOOST` env var or per-call `SearchOpts.exclude_slug_prefixes`. Temporal queries (`detail: 'high'`) bypass the boost so chat pages re-surface for time-sensitive lookups.
|
||||
|
||||
|
||||
+1
-1
@@ -143,5 +143,5 @@
|
||||
"bun": ">=1.3.10"
|
||||
},
|
||||
"license": "MIT",
|
||||
"version": "0.42.12.0"
|
||||
"version": "0.42.13.0"
|
||||
}
|
||||
|
||||
@@ -40,6 +40,12 @@ import { lagFromContentMs } from '../core/source-health.ts';
|
||||
import { CHUNKER_VERSION } from '../core/chunkers/code.ts';
|
||||
import { LINK_EXTRACTOR_VERSION_TS } from '../core/link-extraction.ts';
|
||||
import { isUndefinedColumnError } from '../core/utils.ts';
|
||||
// issue #1777: hidden_by_search_policy — count chunked pages withheld from
|
||||
// default search by the hard-exclude prefix policy. Reuses the canonical
|
||||
// exclude resolver + LIKE escaper + visibility clause so the doctor count can't
|
||||
// drift from what search actually filters.
|
||||
import { resolveHardExcludes, DEFAULT_HARD_EXCLUDES } from '../core/search/source-boost.ts';
|
||||
import { escapeLikePattern, buildVisibilityClause } from '../core/search/sql-ranking.ts';
|
||||
|
||||
export interface Check {
|
||||
name: string;
|
||||
@@ -734,6 +740,11 @@ export async function doctorReportRemote(engine: BrainEngine): Promise<DoctorRep
|
||||
// - synopsis-failures audit JSONL entries from the last 7 days
|
||||
checks.push(await checkContextualRetrievalCoverage(engine));
|
||||
|
||||
// issue #1777 — hidden_by_search_policy: chunked pages withheld from default
|
||||
// search by the hard-exclude prefix policy. Pure SQL COUNT, safe on the
|
||||
// remote/thin-client path.
|
||||
checks.push(await checkHiddenBySearchPolicy(engine));
|
||||
|
||||
// 11a. issue #972 link_resolution_opportunity — same check the local
|
||||
// doctor runs at the equivalent slot in buildChecks. Mirrored for
|
||||
// thin-client parity so `gbrain remote doctor` sees the same hint.
|
||||
@@ -904,6 +915,96 @@ export async function checkContextualRetrievalCoverage(engine: BrainEngine): Pro
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* issue #1777 — hidden_by_search_policy
|
||||
*
|
||||
* Counts CHUNKED pages that are withheld from default search by the
|
||||
* hard-exclude prefix policy (`test/`, `attachments/`, `.raw/`, plus any
|
||||
* `GBRAIN_SEARCH_EXCLUDE` env additions). Makes the surviving exclude policy
|
||||
* auditable so an empty search result is distinguishable from "withheld by
|
||||
* policy" — the deeper bug the archive-demote fix only half-closes.
|
||||
*
|
||||
* HONEST SUPERSET: the count is "chunked pages under an excluded prefix", NOT
|
||||
* "searchable pages". Keyword search additionally filters
|
||||
* `search_vector @@ ... AND modality='text'` and vector search filters text
|
||||
* modality + non-null embedding, so `EXISTS (content_chunks)` over-includes
|
||||
* image-only / non-text pages. Tightening to the exact per-modality predicate
|
||||
* would couple this check to search internals for a number nobody paginates on;
|
||||
* the superset is the right operator signal. The message says "chunked page(s)".
|
||||
*
|
||||
* Status (CV-1a): pages hidden ONLY under DEFAULT excludes → `ok` (intentional
|
||||
* noise; warning would make every healthy brain look unhealthy). Pages hidden
|
||||
* under a NON-default (env-supplied) prefix → `warn`. The message is
|
||||
* agent-prescriptive: move content out of the excluded prefix or pass
|
||||
* `include_slug_prefixes` on the query.
|
||||
*
|
||||
* NOTE: this does NOT verify `archive/` pages are embedded/graphed — after the
|
||||
* #1777 fix `archive/` is no longer excluded, so it never appears here.
|
||||
*/
|
||||
export async function checkHiddenBySearchPolicy(engine: BrainEngine): Promise<Check> {
|
||||
const name = 'hidden_by_search_policy';
|
||||
try {
|
||||
const prefixes = resolveHardExcludes();
|
||||
if (prefixes.length === 0) {
|
||||
return { name, status: 'ok', message: 'No search-exclude prefixes active.' };
|
||||
}
|
||||
|
||||
// ONE query: COUNT(DISTINCT p.id) per prefix in a single pass. Prefixes are
|
||||
// bound params, LIKE-escaped (env-supplied prefixes may contain %/_/\) with
|
||||
// an explicit ESCAPE clause. Candidate gate is EXISTS(content_chunks);
|
||||
// buildVisibilityClause mirrors search's page-level visibility (soft-delete,
|
||||
// archived source, quarantine) and REQUIRES the `sources s` join.
|
||||
const visibility = buildVisibilityClause('p', 's');
|
||||
const filters = prefixes
|
||||
.map((_, i) => `COUNT(DISTINCT p.id) FILTER (WHERE p.slug LIKE $${i + 1} ESCAPE '\\')::int AS c${i}`)
|
||||
.join(',\n ');
|
||||
const params = prefixes.map((pfx) => `${escapeLikePattern(pfx)}%`);
|
||||
const sql =
|
||||
`SELECT
|
||||
${filters}
|
||||
FROM pages p
|
||||
JOIN sources s ON s.id = p.source_id
|
||||
WHERE EXISTS (SELECT 1 FROM content_chunks cc WHERE cc.page_id = p.id)
|
||||
${visibility}`;
|
||||
const rows = await engine.executeRaw<Record<string, number>>(sql, params);
|
||||
const row = rows[0] ?? {};
|
||||
|
||||
const defaults = new Set(DEFAULT_HARD_EXCLUDES);
|
||||
const perPrefix = prefixes
|
||||
.map((pfx, i) => ({ prefix: pfx, count: Number(row[`c${i}`] ?? 0), isDefault: defaults.has(pfx) }))
|
||||
.filter((e) => e.count > 0);
|
||||
|
||||
if (perPrefix.length === 0) {
|
||||
return {
|
||||
name,
|
||||
status: 'ok',
|
||||
message: 'No pages hidden by search-exclude policy.',
|
||||
details: { prefixes, counts: {} },
|
||||
};
|
||||
}
|
||||
|
||||
const counts: Record<string, number> = {};
|
||||
for (const e of perPrefix) counts[e.prefix] = e.count;
|
||||
const breakdown = perPrefix.map((e) => `${e.count} under '${e.prefix}'`).join(', ');
|
||||
const hasNonDefault = perPrefix.some((e) => !e.isDefault);
|
||||
const guidance =
|
||||
'If any hold content you want findable, move them out of the excluded ' +
|
||||
"prefix or pass `include_slug_prefixes` on the query.";
|
||||
return {
|
||||
name,
|
||||
status: hasNonDefault ? 'warn' : 'ok',
|
||||
message: `${breakdown} chunked page(s) are excluded from default search by prefix policy. ${guidance}`,
|
||||
details: { prefixes, counts },
|
||||
};
|
||||
} catch (e) {
|
||||
return {
|
||||
name,
|
||||
status: 'warn',
|
||||
message: `Could not check hidden-by-search-policy: ${e instanceof Error ? e.message : String(e)}`,
|
||||
};
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Issue #972 — link_resolution_opportunity check.
|
||||
*
|
||||
@@ -6335,6 +6436,10 @@ export async function buildChecks(
|
||||
if (engine !== null) {
|
||||
progress.heartbeat('search_mode');
|
||||
checks.push(await checkSearchMode(engine));
|
||||
// issue #1777 — hidden_by_search_policy: chunked pages withheld from default
|
||||
// search by the hard-exclude prefix policy (audit the surviving excludes).
|
||||
progress.heartbeat('hidden_by_search_policy');
|
||||
checks.push(await checkHiddenBySearchPolicy(engine));
|
||||
progress.heartbeat('eval_drift');
|
||||
checks.push(await checkEvalDrift(engine));
|
||||
// v0.35.0.0+ reranker_health — read JSONL audit; warn on auth or volume.
|
||||
|
||||
@@ -82,6 +82,7 @@ export const BRAIN_CHECK_NAMES: ReadonlySet<string> = new Set([
|
||||
'grade_confidence_drift',
|
||||
'graph_coverage',
|
||||
'graph_signals_coverage',
|
||||
'hidden_by_search_policy',
|
||||
'image_assets',
|
||||
'integrity',
|
||||
'jsonb_integrity',
|
||||
|
||||
@@ -1486,8 +1486,9 @@ export class PostgresEngine implements BrainEngine {
|
||||
// by multiplying the chunk-grain ts_rank with a source-factor CASE.
|
||||
// Detail-gated — disabled for `detail='high'` (temporal queries) so
|
||||
// chat surfaces normally for date-framed lookups. Hard-exclude prefixes
|
||||
// (test/, archive/, attachments/, .raw/ by default) filter at the
|
||||
// chunk-rank stage so they never enter the candidate set.
|
||||
// (test/, attachments/, .raw/ by default) filter at the chunk-rank stage
|
||||
// so they never enter the candidate set. (archive/ is demoted, not
|
||||
// excluded — issue #1777.)
|
||||
const boostMap = resolveBoostMap();
|
||||
const sourceFactorCase = buildSourceFactorCase('p.slug', boostMap, opts?.detail);
|
||||
const hardExcludePrefixes = resolveHardExcludes(opts?.exclude_slug_prefixes, opts?.include_slug_prefixes);
|
||||
|
||||
@@ -698,7 +698,15 @@ export function attributeKnob<K extends keyof ModeBundle>(
|
||||
// global cache cold-miss on upgrade — EVERY query_cache row invalidates,
|
||||
// including conservative/no-reranker calls where autocut is a no-op (the hash
|
||||
// is global, not per-mode). Refills within cache.ttl_seconds (3600s default).
|
||||
export const KNOBS_HASH_VERSION = 8;
|
||||
//
|
||||
// bump 8→9 (issue #1777): `archive/` moved from DEFAULT_HARD_EXCLUDES to a 0.5
|
||||
// source-boost demote. The source-boost / hard-exclude policy is NOT part of the
|
||||
// knobs hash, so without a version bump cached rows would keep returning the old
|
||||
// archive-excluded result set for up to cache.ttl_seconds. Bumping forces the fix
|
||||
// to take effect immediately (one-time global cache cold-miss on upgrade; refills
|
||||
// within cache.ttl_seconds). Same cache-key-contamination convention as the
|
||||
// autocut / title_boost / graph_signals bumps above.
|
||||
export const KNOBS_HASH_VERSION = 9;
|
||||
|
||||
/**
|
||||
* v0.36 (D8 / CDX-2) — second-arg context for the cache key. The
|
||||
|
||||
@@ -37,6 +37,14 @@ export const DEFAULT_SOURCE_BOOSTS: Record<string, number> = {
|
||||
'media/x/': 0.7,
|
||||
// Chat transcripts — massive, noisy, swamp keyword queries
|
||||
'openclaw/chat/': 0.5,
|
||||
// Archived historical content — findable by default but ranked below curated
|
||||
// content (issue #1777). NOT hard-excluded: archive/ routinely holds high-signal
|
||||
// history (conversation exports, prior-system logs, older notes) users expect to
|
||||
// retrieve. Demote-not-exclude keeps it findable without letting it dominate. The
|
||||
// 0.5 is a prior applied in the SQL/fusion layer; the cross-encoder reranker can
|
||||
// still PROMOTE a strongly-matching archive page that survives the demote into the
|
||||
// rerank candidate window.
|
||||
'archive/': 0.5,
|
||||
// v0.42 extract_receipt pages — surface when relevant (extraction
|
||||
// questions, audit trail) but never dominate user content. Per plan
|
||||
// D-EXTRACT-42. Receipts stamp `type: extract_receipt` AND
|
||||
@@ -48,10 +56,14 @@ export const DEFAULT_SOURCE_BOOSTS: Record<string, number> = {
|
||||
/**
|
||||
* Hard-excludes — slug prefixes that should never enter search results
|
||||
* (unless explicitly opted-in via include_slug_prefixes).
|
||||
*
|
||||
* These are genuine noise: test fixtures, binary attachments, raw sidecars.
|
||||
* `archive/` is deliberately NOT here (issue #1777) — it holds high-signal
|
||||
* historical content users expect to find, so it is DEMOTED via
|
||||
* DEFAULT_SOURCE_BOOSTS (`archive/`: 0.5) instead of hidden.
|
||||
*/
|
||||
export const DEFAULT_HARD_EXCLUDES: string[] = [
|
||||
'test/',
|
||||
'archive/',
|
||||
'attachments/',
|
||||
'.raw/',
|
||||
];
|
||||
|
||||
@@ -19,8 +19,15 @@
|
||||
|
||||
import { quarantineFilterFragment } from '../quarantine.ts';
|
||||
|
||||
/** Escape `%`, `_`, and `\` so a string can be used as a LIKE prefix literal. */
|
||||
function escapeLikePattern(s: string): string {
|
||||
/**
|
||||
* Escape `%`, `_`, and `\` so a string can be used as a LIKE prefix literal.
|
||||
*
|
||||
* Exported (issue #1777) so callers that build parameterized LIKE clauses with
|
||||
* `ESCAPE '\'` (e.g. the `hidden_by_search_policy` doctor check) reuse this one
|
||||
* escaper instead of re-implementing it. Pair with `ESCAPE '\'` in the SQL so
|
||||
* the backslash this inserts is treated as the escape char, not a literal.
|
||||
*/
|
||||
export function escapeLikePattern(s: string): string {
|
||||
return s.replace(/[%_\\]/g, '\\$&');
|
||||
}
|
||||
|
||||
|
||||
+3
-1
@@ -882,9 +882,11 @@ export interface SearchOpts {
|
||||
types?: PageType[];
|
||||
exclude_slugs?: string[];
|
||||
/**
|
||||
* Slug-prefix excludes — additive over DEFAULT_HARD_EXCLUDES (test/, archive/,
|
||||
* Slug-prefix excludes — additive over DEFAULT_HARD_EXCLUDES (test/,
|
||||
* attachments/, .raw/) and the GBRAIN_SEARCH_EXCLUDE env var. Stacks with
|
||||
* `exclude_slugs` (exact match) — a row is filtered if it matches either set.
|
||||
* NOTE (issue #1777): `archive/` is NOT hard-excluded; it is demoted (0.5x)
|
||||
* via DEFAULT_SOURCE_BOOSTS so archived content stays findable by default.
|
||||
*/
|
||||
exclude_slug_prefixes?: string[];
|
||||
/**
|
||||
|
||||
@@ -136,13 +136,14 @@ describe('D2 — knobsHash differs across cross-modal knob values', () => {
|
||||
return resolveSearchMode({ mode: 'balanced' });
|
||||
}
|
||||
|
||||
test('KNOBS_HASH_VERSION is 6 (v=4 graph_signals + schema-pack; v=5 contextual_retrieval; v=6 alias_resolved; cross-modal still appended)', () => {
|
||||
test('KNOBS_HASH_VERSION is 9 (cross-modal still appended; 8→9 archive-demote #1777)', () => {
|
||||
// v0.35 ladder: 1→2 reranker, 2→3 floor_ratio. v0.36 piggybacks on v=3
|
||||
// with 7 cross-modal knobs + column/provider context. v0.40.4 (salem) +
|
||||
// v0.39 T21 (master) bump to v=4 for graph_signals + schema-pack fields.
|
||||
// v0.40.3.0 D8 bumps to v=5 (sequenced behind salem's v=4 graph-signals).
|
||||
// v0.41.22.0 (type-unification): 5→6 for alias_resolved post-fusion boost.
|
||||
expect(KNOBS_HASH_VERSION).toBe(8);
|
||||
// T2: 6→7 title_boost. v0.42.3.0: 7→8 autocut. issue #1777: 8→9 archive/ demote.
|
||||
expect(KNOBS_HASH_VERSION).toBe(9);
|
||||
});
|
||||
|
||||
test('flipping unified_multimodal changes the hash', () => {
|
||||
|
||||
@@ -0,0 +1,153 @@
|
||||
/**
|
||||
* hidden_by_search_policy doctor check (issue #1777)
|
||||
*
|
||||
* Counts CHUNKED pages withheld from default search by the hard-exclude prefix
|
||||
* policy. Verifies:
|
||||
* - default-only excludes (test/) → status ok with a prescriptive message
|
||||
* - a NON-default env exclude hiding pages → status warn
|
||||
* - multi-chunk page counted once (DISTINCT)
|
||||
* - soft-deleted / quarantined pages NOT counted (visibility mirror)
|
||||
* - archive/ never appears (it's demoted, not excluded)
|
||||
* - SQL error → warn, not throw
|
||||
* - the check name categorizes as a brain check
|
||||
*/
|
||||
|
||||
import { describe, test, expect, beforeAll, afterAll, beforeEach } from 'bun:test';
|
||||
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
|
||||
import { resetPgliteState } from './helpers/reset-pglite.ts';
|
||||
import { withEnv } from './helpers/with-env.ts';
|
||||
import { checkHiddenBySearchPolicy } from '../src/commands/doctor.ts';
|
||||
import { categorizeCheck } from '../src/core/doctor-categories.ts';
|
||||
import { buildQuarantineMarker } from '../src/core/quarantine.ts';
|
||||
import type { BrainEngine } from '../src/core/engine.ts';
|
||||
import type { ChunkInput } from '../src/core/types.ts';
|
||||
|
||||
let engine: PGLiteEngine;
|
||||
|
||||
function basisEmbedding(idx: number, dim = 1536): Float32Array {
|
||||
const emb = new Float32Array(dim);
|
||||
emb[idx % dim] = 1.0;
|
||||
return emb;
|
||||
}
|
||||
|
||||
async function seed(
|
||||
slug: string,
|
||||
opts: { chunks?: number; frontmatter?: Record<string, unknown> } = {},
|
||||
): Promise<void> {
|
||||
const chunks = opts.chunks ?? 1;
|
||||
await engine.putPage(slug, {
|
||||
type: 'note',
|
||||
title: slug,
|
||||
compiled_truth: `body for ${slug}`,
|
||||
timeline: '',
|
||||
...(opts.frontmatter ? { frontmatter: opts.frontmatter } : {}),
|
||||
});
|
||||
if (chunks > 0) {
|
||||
const rows: ChunkInput[] = [];
|
||||
for (let i = 0; i < chunks; i++) {
|
||||
rows.push({
|
||||
chunk_index: i,
|
||||
chunk_text: `chunk ${i} of ${slug}`,
|
||||
chunk_source: 'compiled_truth',
|
||||
embedding: basisEmbedding(100 + i),
|
||||
token_count: 4,
|
||||
});
|
||||
}
|
||||
await engine.upsertChunks(slug, rows);
|
||||
}
|
||||
}
|
||||
|
||||
beforeAll(async () => {
|
||||
engine = new PGLiteEngine();
|
||||
await engine.connect({});
|
||||
await engine.initSchema();
|
||||
}, 60_000);
|
||||
|
||||
afterAll(async () => {
|
||||
await engine.disconnect();
|
||||
});
|
||||
|
||||
beforeEach(async () => {
|
||||
await resetPgliteState(engine);
|
||||
});
|
||||
|
||||
describe('checkHiddenBySearchPolicy', () => {
|
||||
test('default-only excludes → ok, prescriptive message, names test/ not archive/concepts', async () => {
|
||||
await seed('test/fixtures/widget');
|
||||
await seed('concepts/widget-pattern');
|
||||
await seed('archive/old/widget-2020');
|
||||
|
||||
const r = await checkHiddenBySearchPolicy(engine);
|
||||
expect(r.name).toBe('hidden_by_search_policy');
|
||||
expect(r.status).toBe('ok'); // test/ is a DEFAULT exclude → not a warn
|
||||
expect(r.message).toContain("under 'test/'");
|
||||
expect(r.message).toContain('chunked');
|
||||
expect(r.message).not.toContain('searchable'); // honest superset, not "searchable"
|
||||
expect(r.message).toContain('include_slug_prefixes'); // prescriptive guidance
|
||||
// archive/ is demoted, not excluded — must never surface here.
|
||||
expect(r.message).not.toContain('archive/');
|
||||
// concepts/ is not excluded at all.
|
||||
expect(r.message).not.toContain('concepts/');
|
||||
const counts = (r.details?.counts ?? {}) as Record<string, number>;
|
||||
expect(counts['test/']).toBe(1);
|
||||
expect(counts['archive/']).toBeUndefined();
|
||||
expect(counts['concepts/']).toBeUndefined();
|
||||
});
|
||||
|
||||
test('a NON-default env exclude hiding pages → warn', async () => {
|
||||
await withEnv({ GBRAIN_SEARCH_EXCLUDE: 'scratch/' }, async () => {
|
||||
await seed('scratch/notes');
|
||||
await seed('concepts/keeper');
|
||||
const r = await checkHiddenBySearchPolicy(engine);
|
||||
expect(r.status).toBe('warn');
|
||||
expect(r.message).toContain("under 'scratch/'");
|
||||
});
|
||||
});
|
||||
|
||||
test('multi-chunk page counted once (DISTINCT)', async () => {
|
||||
await seed('test/multi', { chunks: 3 });
|
||||
const r = await checkHiddenBySearchPolicy(engine);
|
||||
const counts = (r.details?.counts ?? {}) as Record<string, number>;
|
||||
expect(counts['test/']).toBe(1);
|
||||
});
|
||||
|
||||
test('soft-deleted and quarantined pages are NOT counted', async () => {
|
||||
await seed('test/soft-deleted');
|
||||
await engine.softDeletePage('test/soft-deleted');
|
||||
await seed('test/quarantined', {
|
||||
frontmatter: { quarantine: buildQuarantineMarker('junk_pattern', 'test fixture') },
|
||||
});
|
||||
await seed('concepts/keeper');
|
||||
|
||||
const r = await checkHiddenBySearchPolicy(engine);
|
||||
// Both test/ pages are hidden by the visibility clause, so 0 remain counted.
|
||||
expect(r.status).toBe('ok');
|
||||
expect(r.message).toBe('No pages hidden by search-exclude policy.');
|
||||
const counts = (r.details?.counts ?? {}) as Record<string, number>;
|
||||
expect(counts['test/']).toBeUndefined();
|
||||
});
|
||||
|
||||
test('zero hidden (only curated content) → ok', async () => {
|
||||
await seed('concepts/keeper');
|
||||
const r = await checkHiddenBySearchPolicy(engine);
|
||||
expect(r.status).toBe('ok');
|
||||
expect(r.message).toBe('No pages hidden by search-exclude policy.');
|
||||
});
|
||||
|
||||
test('SQL error → warn, not throw', async () => {
|
||||
const broken = {
|
||||
executeRaw: async () => {
|
||||
throw new Error('boom');
|
||||
},
|
||||
} as unknown as BrainEngine;
|
||||
const r = await checkHiddenBySearchPolicy(broken);
|
||||
expect(r.status).toBe('warn');
|
||||
expect(r.message).toContain('Could not check');
|
||||
});
|
||||
});
|
||||
|
||||
describe('hidden_by_search_policy categorization', () => {
|
||||
test('categorizes as a brain check', () => {
|
||||
expect(categorizeCheck('hidden_by_search_policy')).toBe('brain');
|
||||
});
|
||||
});
|
||||
@@ -1,13 +1,16 @@
|
||||
/**
|
||||
* Hard-Exclude E2E
|
||||
*
|
||||
* Verifies the new exclude_slug_prefixes / include_slug_prefixes plumbing.
|
||||
* test/, archive/, attachments/, .raw/ are hard-excluded by default.
|
||||
* include_slug_prefixes opts back in.
|
||||
* Verifies the exclude_slug_prefixes / include_slug_prefixes plumbing.
|
||||
* test/, attachments/, .raw/ are hard-excluded by default.
|
||||
* archive/ is NOT excluded (issue #1777) — it is demoted (0.5x) so it stays
|
||||
* findable by default but ranks below curated content. include_slug_prefixes
|
||||
* opts back into the genuinely-excluded prefixes.
|
||||
*/
|
||||
|
||||
import { describe, test, expect, beforeAll, afterAll } from 'bun:test';
|
||||
import { PGLiteEngine } from '../../src/core/pglite-engine.ts';
|
||||
import { hybridSearch } from '../../src/core/search/hybrid.ts';
|
||||
import type { ChunkInput } from '../../src/core/types.ts';
|
||||
|
||||
let engine: PGLiteEngine;
|
||||
@@ -70,6 +73,42 @@ beforeAll(async () => {
|
||||
token_count: 9,
|
||||
},
|
||||
] satisfies ChunkInput[]);
|
||||
|
||||
// A genuinely-excluded prefix that stays hidden by default (locks "only
|
||||
// archive moved" — test/, attachments/, .raw/ are unchanged).
|
||||
await engine.putPage('.raw/widget-dump', {
|
||||
type: 'note',
|
||||
title: 'Widget raw dump',
|
||||
compiled_truth: 'widget raw sidecar dump',
|
||||
timeline: '',
|
||||
});
|
||||
await engine.upsertChunks('.raw/widget-dump', [
|
||||
{
|
||||
chunk_index: 0,
|
||||
chunk_text: 'widget raw sidecar dump noise',
|
||||
chunk_source: 'compiled_truth',
|
||||
embedding: basisEmbedding(14),
|
||||
token_count: 5,
|
||||
},
|
||||
] satisfies ChunkInput[]);
|
||||
|
||||
// A term that appears in NO other page, only in an archive page. Proves
|
||||
// archive is reachable even when it is the ONLY match (issue #1777).
|
||||
await engine.putPage('archive/2019/quokkanaut-memo', {
|
||||
type: 'note',
|
||||
title: 'Quokkanaut memo',
|
||||
compiled_truth: 'the quokkanaut project shipped in 2019',
|
||||
timeline: '',
|
||||
});
|
||||
await engine.upsertChunks('archive/2019/quokkanaut-memo', [
|
||||
{
|
||||
chunk_index: 0,
|
||||
chunk_text: 'the quokkanaut project shipped in 2019',
|
||||
chunk_source: 'compiled_truth',
|
||||
embedding: basisEmbedding(15),
|
||||
token_count: 6,
|
||||
},
|
||||
] satisfies ChunkInput[]);
|
||||
}, 60_000);
|
||||
|
||||
afterAll(async () => {
|
||||
@@ -83,10 +122,33 @@ describe('searchKeyword default hard-excludes', () => {
|
||||
expect(slugs).not.toContain('test/fixtures/widget');
|
||||
});
|
||||
|
||||
test('archive/ pages are hidden by default', async () => {
|
||||
test('.raw/ pages stay hidden by default (only archive moved)', async () => {
|
||||
const results = await engine.searchKeyword('widget');
|
||||
const slugs = results.map(r => r.slug);
|
||||
expect(slugs).not.toContain('archive/old-stuff/widget-2020');
|
||||
expect(slugs).not.toContain('.raw/widget-dump');
|
||||
});
|
||||
|
||||
test('archive/ pages ARE returned by default (issue #1777)', async () => {
|
||||
const results = await engine.searchKeyword('widget');
|
||||
const slugs = results.map(r => r.slug);
|
||||
expect(slugs).toContain('archive/old-stuff/widget-2020');
|
||||
});
|
||||
|
||||
test('archive/ is demoted below curated content (keyword ordering)', async () => {
|
||||
const results = await engine.searchKeyword('widget');
|
||||
const slugs = results.map(r => r.slug);
|
||||
const archiveIdx = slugs.indexOf('archive/old-stuff/widget-2020');
|
||||
const conceptsIdx = slugs.indexOf('concepts/widget-pattern');
|
||||
expect(archiveIdx).toBeGreaterThanOrEqual(0); // present
|
||||
expect(conceptsIdx).toBeGreaterThanOrEqual(0);
|
||||
// concepts/ (boost 1.3) ranks ABOVE archive/ (boost 0.5).
|
||||
expect(conceptsIdx).toBeLessThan(archiveIdx);
|
||||
});
|
||||
|
||||
test('archive is reachable even when it is the ONLY match (unique phrase)', async () => {
|
||||
const results = await engine.searchKeyword('quokkanaut');
|
||||
const slugs = results.map(r => r.slug);
|
||||
expect(slugs).toContain('archive/2019/quokkanaut-memo');
|
||||
});
|
||||
|
||||
test('curated content is unaffected', async () => {
|
||||
@@ -103,17 +165,17 @@ describe('searchKeyword include_slug_prefixes opt-back-in', () => {
|
||||
});
|
||||
const slugs = results.map(r => r.slug);
|
||||
expect(slugs).toContain('test/fixtures/widget');
|
||||
// archive/ is still excluded.
|
||||
expect(slugs).not.toContain('archive/old-stuff/widget-2020');
|
||||
// .raw/ is still excluded.
|
||||
expect(slugs).not.toContain('.raw/widget-dump');
|
||||
});
|
||||
|
||||
test('include_slug_prefixes lets caller opt back into both', async () => {
|
||||
test('include_slug_prefixes opts back into multiple excluded prefixes', async () => {
|
||||
const results = await engine.searchKeyword('widget', {
|
||||
include_slug_prefixes: ['test/', 'archive/'],
|
||||
include_slug_prefixes: ['test/', '.raw/'],
|
||||
});
|
||||
const slugs = results.map(r => r.slug);
|
||||
expect(slugs).toContain('test/fixtures/widget');
|
||||
expect(slugs).toContain('archive/old-stuff/widget-2020');
|
||||
expect(slugs).toContain('.raw/widget-dump');
|
||||
});
|
||||
});
|
||||
|
||||
@@ -126,6 +188,13 @@ describe('searchVector hard-excludes', () => {
|
||||
expect(slugs).not.toContain('test/fixtures/widget');
|
||||
});
|
||||
|
||||
test('archive/ pages ARE returned by default in vector search (#1777)', async () => {
|
||||
// basisEmbedding(12) points at archive/old-stuff/widget-2020.
|
||||
const results = await engine.searchVector(basisEmbedding(12));
|
||||
const slugs = results.map(r => r.slug);
|
||||
expect(slugs).toContain('archive/old-stuff/widget-2020');
|
||||
});
|
||||
|
||||
test('include_slug_prefixes lets it back in', async () => {
|
||||
const results = await engine.searchVector(basisEmbedding(11), {
|
||||
include_slug_prefixes: ['test/'],
|
||||
@@ -135,16 +204,32 @@ describe('searchVector hard-excludes', () => {
|
||||
});
|
||||
});
|
||||
|
||||
describe('hybridSearch (the real user/MCP surface)', () => {
|
||||
test('archive/ is returned by default through hybridSearch', async () => {
|
||||
// No embedding provider in this hermetic run, so hybridSearch's vector path
|
||||
// is skipped and it exercises keyword + dedup + post-fusion. archive/ must
|
||||
// still come back by default (it is no longer hard-excluded).
|
||||
const results = await hybridSearch(engine, 'widget', { limit: 20 });
|
||||
const slugs = results.map(r => r.slug);
|
||||
expect(slugs).toContain('archive/old-stuff/widget-2020');
|
||||
// test/ and .raw/ stay hidden through the full pipeline.
|
||||
expect(slugs).not.toContain('test/fixtures/widget');
|
||||
expect(slugs).not.toContain('.raw/widget-dump');
|
||||
});
|
||||
});
|
||||
|
||||
describe('caller-supplied exclude_slug_prefixes (additive)', () => {
|
||||
test('caller can add a custom exclude prefix on top of defaults', async () => {
|
||||
const results = await engine.searchKeyword('widget', {
|
||||
exclude_slug_prefixes: ['concepts/'],
|
||||
});
|
||||
const slugs = results.map(r => r.slug);
|
||||
// concepts/ now also excluded; with all three categories filtered, no
|
||||
// hits remain.
|
||||
// concepts/ now also excluded on top of the defaults (test/, .raw/).
|
||||
expect(slugs).not.toContain('concepts/widget-pattern');
|
||||
expect(slugs).not.toContain('test/fixtures/widget');
|
||||
expect(slugs).not.toContain('archive/old-stuff/widget-2020');
|
||||
expect(slugs).not.toContain('.raw/widget-dump');
|
||||
// archive/ is NOT a default exclude and was not caller-excluded, so it
|
||||
// remains visible (issue #1777).
|
||||
expect(slugs).toContain('archive/old-stuff/widget-2020');
|
||||
});
|
||||
});
|
||||
|
||||
@@ -89,7 +89,7 @@ describe('alias_resolved boost stage', () => {
|
||||
});
|
||||
|
||||
describe('KNOBS_HASH_VERSION', () => {
|
||||
it('bumped to 6 to invalidate caches across v0.42 boost stage addition', () => {
|
||||
expect(KNOBS_HASH_VERSION).toBe(8);
|
||||
it('is 9 (8→9 archive-demote invalidates archive-excluded cache rows, #1777)', () => {
|
||||
expect(KNOBS_HASH_VERSION).toBe(9);
|
||||
});
|
||||
});
|
||||
|
||||
@@ -388,10 +388,11 @@ describe('knobsHash determinism + cross-mode separation (CDX-4)', () => {
|
||||
// embedding spaces. Sequenced behind salem's v=4 graph-signals work.
|
||||
// v0.41.22.0 (type-unification): bumped 5→6 for the new alias_resolved
|
||||
// post-fusion boost stage. T2: bumped 6→7 for title_boost. v0.42.3.0:
|
||||
// bumped 7→8 for autocut (ac=/acj=). A query against a brain with
|
||||
// slug_aliases populated must not be served from a cache row written
|
||||
// before the boost stage existed.
|
||||
expect(KNOBS_HASH_VERSION).toBe(8);
|
||||
// bumped 7→8 for autocut (ac=/acj=). issue #1777: bumped 8→9 for the
|
||||
// archive/ demote (search-exclude policy change isn't in the hash, so the
|
||||
// version bump is what invalidates archive-excluded cache rows). A query
|
||||
// must not be served from a cache row written before the policy change.
|
||||
expect(KNOBS_HASH_VERSION).toBe(9);
|
||||
});
|
||||
|
||||
test('T1 (codex): floor_ratio set vs unset produces DIFFERENT hashes (cache contamination prevention)', () => {
|
||||
@@ -556,8 +557,8 @@ describe('v0.40.4 — graph_signals knob', () => {
|
||||
});
|
||||
|
||||
describe('v0.42.3.0 — autocut knobs', () => {
|
||||
test('KNOBS_HASH_VERSION bumped to 7', () => {
|
||||
expect(KNOBS_HASH_VERSION).toBe(8);
|
||||
test('KNOBS_HASH_VERSION is 9 (8→9 archive-demote, issue #1777)', () => {
|
||||
expect(KNOBS_HASH_VERSION).toBe(9);
|
||||
});
|
||||
|
||||
test('bundle defaults: conservative off, balanced/tokenmax on @0.20', () => {
|
||||
|
||||
@@ -43,7 +43,7 @@ function baseKnobs(): ResolvedSearchKnobs {
|
||||
}
|
||||
|
||||
describe('KNOBS_HASH_VERSION + version invariants', () => {
|
||||
test('version is 6 (1→2 reranker; 2→3 floor_ratio + cross-modal + column; 3→4 graph_signals + schema_pack; 4→5 contextual_retrieval; 5→6 v0.41.22 alias_resolved boost)', () => {
|
||||
test('version is 9 (…; 6→7 title_boost; 7→8 autocut; 8→9 archive-demote #1777)', () => {
|
||||
// v0.35.0.0: 1→2 to fold reranker fields. v0.35.6.0: 2→3 to fold
|
||||
// floor_ratio. v0.36 wave: piggybacks on v=3 with 7 cross-modal knobs
|
||||
// (D2) PLUS column + provider context (D8/CDX-2 cross-column isolation).
|
||||
@@ -54,8 +54,10 @@ describe('KNOBS_HASH_VERSION + version invariants', () => {
|
||||
// sequenced behind salem's v=4 graph-signals.
|
||||
// v0.41.22.0 (type-unification): 5→6 to fold the alias_resolved
|
||||
// post-fusion boost. Cache rows written before the boost stage
|
||||
// cannot leak past the new stage.
|
||||
expect(KNOBS_HASH_VERSION).toBe(8);
|
||||
// cannot leak past the new stage. T2: 6→7 title_boost. v0.42.3.0: 7→8
|
||||
// autocut. issue #1777: 8→9 archive/ demote (search-exclude policy change
|
||||
// isn't in the hash, so the bump invalidates archive-excluded cache rows).
|
||||
expect(KNOBS_HASH_VERSION).toBe(9);
|
||||
});
|
||||
|
||||
test('hash is 16 hex chars regardless of reranker config', () => {
|
||||
|
||||
@@ -3,6 +3,7 @@ import {
|
||||
buildSourceFactorCase,
|
||||
buildHardExcludeClause,
|
||||
buildVisibilityClause,
|
||||
escapeLikePattern as topLevelEscapeLikePattern,
|
||||
__test__,
|
||||
} from '../src/core/search/sql-ranking.ts';
|
||||
import {
|
||||
@@ -241,7 +242,7 @@ describe('resolveHardExcludes', () => {
|
||||
const r = resolveHardExcludes(undefined, ['test/'], undefined);
|
||||
expect(r).not.toContain('test/');
|
||||
// Other defaults still present.
|
||||
expect(r).toContain('archive/');
|
||||
expect(r).toContain('attachments/');
|
||||
});
|
||||
|
||||
test('env GBRAIN_SEARCH_EXCLUDE adds to the union', () => {
|
||||
@@ -255,6 +256,40 @@ describe('resolveHardExcludes', () => {
|
||||
});
|
||||
});
|
||||
|
||||
// issue #1777 — archive/ moved from hard-exclude to a 0.5 source-boost demote.
|
||||
describe('archive demote (issue #1777)', () => {
|
||||
test('archive/ is NOT a default hard-exclude (regression guard)', () => {
|
||||
expect(DEFAULT_HARD_EXCLUDES).not.toContain('archive/');
|
||||
// The genuine-noise prefixes stay excluded.
|
||||
expect(DEFAULT_HARD_EXCLUDES).toContain('test/');
|
||||
expect(DEFAULT_HARD_EXCLUDES).toContain('attachments/');
|
||||
expect(DEFAULT_HARD_EXCLUDES).toContain('.raw/');
|
||||
});
|
||||
|
||||
test('resolveHardExcludes() never includes archive/ by default', () => {
|
||||
expect(resolveHardExcludes()).not.toContain('archive/');
|
||||
});
|
||||
|
||||
test('archive/ is demoted to 0.5 in the boost map', () => {
|
||||
expect(DEFAULT_SOURCE_BOOSTS['archive/']).toBe(0.5);
|
||||
expect(resolveBoostMap()['archive/']).toBe(0.5);
|
||||
});
|
||||
|
||||
test('buildSourceFactorCase emits an archive/ demote branch', () => {
|
||||
const sql = buildSourceFactorCase('p.slug', resolveBoostMap(), undefined);
|
||||
expect(sql).toContain("WHEN p.slug LIKE 'archive/%' THEN 0.5");
|
||||
});
|
||||
|
||||
test('detail=high bypasses the source factor (archive ranks normally)', () => {
|
||||
expect(buildSourceFactorCase('p.slug', resolveBoostMap(), 'high')).toBe('1.0');
|
||||
});
|
||||
|
||||
test('escapeLikePattern is exported at top level (CV-3a contract)', () => {
|
||||
expect(typeof topLevelEscapeLikePattern).toBe('function');
|
||||
expect(topLevelEscapeLikePattern('a_b%c\\d')).toBe('a\\_b\\%c\\\\d');
|
||||
});
|
||||
});
|
||||
|
||||
// v0.26.5 — visibility clause for soft-deleted pages and archived sources.
|
||||
describe('buildVisibilityClause (v0.26.5)', () => {
|
||||
test('emits both predicates joined by AND with a leading AND', () => {
|
||||
|
||||
Reference in New Issue
Block a user