v0.42.3.0 feat(search): autocut — score-discontinuity result-sizing (#1663 wave 1) (#1682)

* feat(search): autocut — score-discontinuity result-sizing on the rerank separatrix

Cut the ranked set at the cross-encoder rerank-score cliff instead of a fixed
top-K. Default-ON in reranked modes (balanced/tokenmax), no-op without a
reranker. New pure src/core/search/autocut.ts; mode.ts knobs + reranker_top_n_in
= searchLimit (no unscored tail); query op autocut param; --explain + glossary.

* test(search): autocut pure-fn, agent-surface, behavioral + precision/recall eval gate

Adds autocut.test.ts, query-op-autocut.test.ts, autocut-integration.serial.test.ts
(IRON-RULE behavioral via rerankerFn seam), autocut-eval.test.ts (in-repo
precision-lift-without-recall-regression gate). Updates existing knobsHash/bundle
pins to v=7 + reranker_top_n_in.

* chore: version + changelog + docs for autocut (v0.41.34.0)

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(search): autocut preserves alias-hop exact matches + cache-HIT meta (codex P1/P2)

P1: applyAliasHop injects the canonical page after reranking (no rerank_score);
autocut would drop it when cutting on the scored set. applyAutocut gains an
optional preserve predicate; hybrid passes r => r.alias_hit === true.
P2: cache-HIT cachedMeta now carries autocut/adaptive_return/mode/embedding_column.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* chore: bump version to v0.42.3.0 (autocut wave)

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* docs: PR titles lead with the version (IRON RULE in CLAUDE.md)

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Garry Tan
2026-06-01 20:16:37 -07:00
committed by GitHub
co-authored by Claude Opus 4.8
parent 7b0d99adb0
commit d9eadfec13
25 changed files with 1418 additions and 20 deletions
+116
View File
@@ -2,6 +2,122 @@
All notable changes to GBrain will be documented in this file.
## [0.42.3.0] - 2026-05-30
**Search now returns the *confident handful* instead of a fixed wall of results
— automatically. When you ask something with one clear answer, you get one
result; when there are genuinely a few, you get those few; you stop getting 20
loosely-related pages just because 20 was the limit. No flag to turn on — it is
the default.**
Here is the problem it fixes. Ask your brain "what's the door code for the
cabin" and the old default would hand back 20 results: the right one on top, then
19 things that merely mention cabins or codes. You (or the agent reading the
results) then wade through the pile. That is fine for "show me everything about
X," but it is noise when the question has a small answer. The new behavior,
called **autocut**, looks at how the relevance scores drop off and cuts the list
where the scores fall off a cliff. One obvious answer comes back as one result.
A real cluster of three comes back as three. A broad question with no clear
winner still returns the full set, so you never lose recall when you actually
want breadth.
The important detail, and why this is trustworthy where a naive version would
not be: autocut cuts on the **reranker's** score, not the raw search score.
gbrain already measured that the raw search score gap looks the same whether the
top hit is right or wrong — it is not a reliable "is this the answer" signal. The
cross-encoder reranker score *is*. So autocut only runs in the search modes where
the reranker runs (the default `balanced` mode and `tokenmax`), and it is a clean
no-op everywhere else. It can never return zero results when matches exist, and
it never runs on a page the reranker did not actually score.
### How to use it
Nothing. It is on by default in the `balanced` and `tokenmax` search modes. The
`conservative` mode (no reranker) is unaffected.
Your agent gets one new lever on the `query` tool — `autocut: false` — to force
the full top-K back when it deliberately wants breadth (broad exploration, "list
everything about X," or when it suspects the top hit is wrong and wants to see
the alternatives). It almost never needs to set it; the default is the smart path.
Per-brain knobs if you want to tune or disable:
```bash
gbrain config set search.autocut false # turn autocut off for this brain
gbrain config set search.autocut_jump 0.30 # require a steeper cliff to cut (default 0.20)
gbrain search modes # see autocut / autocut_jump per-mode
gbrain query "what is X" --explain # shows each result's rerank score + the cut
```
### What a concrete example looks like
Query: "cabin door code" against a brain on the default mode. The reranker scores
the candidates, autocut sees the cliff, and you get:
| Result | Rerank score | Kept? |
|---|---|---|
| `notes/cabin-access` | 0.95 | yes (above the cliff) |
| `notes/cabin-packing-list` | 0.22 | no (below the cliff) |
| `notes/lake-house-wifi` | 0.18 | no |
| ...17 more | <0.2 | no |
One result instead of twenty. Ask "everything about the cabin" instead and the
scores come back flat (no cliff) — autocut declines and you get the full set.
### Things to know about
- **One-time cache cold-start on upgrade.** The query cache key changed
(it now distinguishes autocut-on from autocut-off results), so every cached
search row is invalidated once on upgrade and the cache refills over the next
hour (`search.cache.ttl_seconds`, default 3600s). This is a global one-time
miss spike, including in `conservative` mode where autocut is a no-op — the
cache key is shared. Same pattern as prior search upgrades.
- **`conservative` mode gets no precision change** — it has no reranker, so there
is no trustworthy cliff to cut on. Autocut is a documented no-op there. Use
`balanced` or `tokenmax` to get it.
- **Default search mode reranks a few more candidates per query.** To make
autocut correct, the reranker now scores the full returned set (50 in
`tokenmax`, 25 in `balanced`, up from 30) so there is never a returned-but-
unscored result that autocut might wrongly drop. The extra rerank cost is
rounding error next to the downstream model.
- **This is one wave of a larger retrieval redesign** ([#1663](https://github.com/garrytan/gbrain/issues/1663)).
Still to come: query-shape routing, a structural exact-lookup tier, and
automatic escalation to `think` on low-confidence queries. The issue stays open.
### Itemized changes
- **New `src/core/search/autocut.ts`** — pure score-discontinuity algorithm.
Normalizes the reranker scores, finds the largest gap, and cuts there when the
gap clears a sensitivity threshold (`autocut_jump`, default 0.20). Robust to
unsorted provider output (cuts on a sorted copy, keeps items in input order),
guards against unusable score scales (top ≤ 0, non-finite), and never returns
empty. No-ops when fewer than 2 results carry a reranker score (covers the
reranker's fail-open path).
- **`query` tool gains an `autocut` boolean** — the ceiling override. Description
teaches the agent it is the smart default and `false` is the breadth escape
hatch, and distinguishes it from `adaptive_return` (cuts by score cliff vs.
caps by question intent). The keyword-only `search` tool is unchanged (no
reranker there).
- **`gbrain search modes`** lists `autocut` + `autocut_jump` with per-knob source
attribution; **`--explain`** shows each result's rerank score and an autocut
summary line; the metric glossary documents `autocut.signal` + `autocut.gap_ratio`.
- **Reranker now scores the full returned set** in `balanced` (25) and `tokenmax`
(50) so autocut never drops an unscored tail.
- **Cache-meta fix (found during review):** the cached search path was silently
dropping the `adaptive_return` decision (and would have dropped `autocut`,
`mode`, `embedding_column`) from the metadata it reports. All are now carried
through, so `--explain` and eval-capture report the real decision on cache
writeback and hits.
- `rerank_score` is now a first-class field on search results.
- Cache-key version bumped (7 → 8) to fold in the autocut knobs (stacked on master's title_boost v=7).
- **In-repo eval gate** (`bun run eval:autocut`, also runs in CI) measures the
precision-lift-without-recall-regression claim default-ON rests on, over
labeled qrels fixtures with realistic cross-encoder score distributions — no
API key, no external repo. Current result: mean precision 0.33 → 0.94, mean
recall 1.00 → 0.95, and **zero recall regression on enumeration queries**
(autocut declines on flat curves by construction). Floors are env-overridable.
A live-corpus PrecisionMemBench run remains an optional empirical confirmation,
not a blocker.
## [0.42.2.0] - 2026-05-30
**One command now wires Claude Code, Codex, or Perplexity Computer to a remote
+22
View File
@@ -133,6 +133,7 @@ strict behavior when unset.
- `src/core/search/embedding-column.ts` (v0.36.3.0) — single source of truth for "which `content_chunks.*` column does this query rank against?" Pure functions, no engine I/O: `loadRegistry(cfg)` walks the `embedding_columns` config (DB plane, JSON map keyed by column name with `{provider, dimensions, type}` entries), seeds the OpenAI `embedding` builtin when unset, validates everything before it lands (column-name regex, type ∈ `vector | halfvec`, dims in [1, 8192], provider format) using `Object.create(null)` + `Object.hasOwn` so a key like `constructor` rejects instead of resolving to `Object.prototype.constructor`. `resolveColumn(registry, override?, cfg)` is the boundary call: returns a `ResolvedColumn` descriptor (frozen `{name, provider, dimensions, type}`) honoring per-call override → `search_embedding_column` config → `'embedding'` default. Throws `UnknownEmbeddingColumnError` with the list of registered names on miss. `isCacheSafe(resolved, cfg)` compares the full embedding SPACE (provider + dimensions + name) against cfg's default — a user who repointed the `embedding` builtin at Voyage doesn't accidentally serve OpenAI-shaped cache rows. `validateResolvedColumn(descriptor)` re-validates hand-rolled descriptors that bypass the registry (internal-SDK passthrough path) so the SQL-injection escape hatch through the descriptor field is closed. Consumed by `hybridSearch` (resolves once at the boundary, threads a `ResolvedColumn` into per-engine `searchVector` instead of a raw string), `gateway.embedQuery(text, {embeddingModel, dimensions})` (resolved column's provider drives the query-time embed call), `cosineReScore` (engine pulls vectors from the active column, not the hardcoded `embedding`), and the `query` MCP op (per-call `embedding_column` param). 538 lines of pure resolver, 511 unit cases in `test/search/embedding-column.test.ts` covering the 16 codex-flagged corners (prototype-pollution, descriptor passthrough, env-only Postgres install, empty-brain coverage gate, cache-space comparison).
- `src/core/search/rerank.ts` (v0.35.0.0) — the call-site abstraction. `applyReranker(query, results, opts)` slots between `dedupResults()` and `enforceTokenBudget()` in `src/core/search/hybrid.ts`. Slices `opts.topNIn` (default 30) by current RRF order, sends to `gateway.rerank()`, reorders by `relevanceScore` desc, and appends the un-reranked tail unchanged (recall protection). **Fail-open on every `RerankError.reason`**: any error logs via `logRerankFailure` and returns the input array unchanged. Stamps `rerank_score` onto reordered items so downstream telemetry sees the new ordering signal. `topNOut: null` is the explicit "don't truncate" signal — semantically distinct from `undefined` which means "fall through to mode bundle" (CDX2-F16). Test seam: `opts.rerankerFn` lets tests stub `gateway.rerank` without touching the network.
- `src/core/search/return-policy.ts` (v0.41.33.0, NEW — default OFF) — intent-aware adaptive return-sizing. Pure, dependency-light module that trims the final ranked candidate set to an intent-driven cap instead of always returning the full top-K (noisy) list. `entity` intent (single-answer-ish) gets a tight cap; `temporal`/`event`/`general` (enumeration-ish) get a recall-preserving cap. A `minKeep` failsafe (≥1) guarantees a human never gets a silent blank when candidates exist. WHY a cap, not a score-cliff detector: PrecisionMemBench Phase-1 instrumentation (gbrain-evals) measured that the rank1→rank2 RRF score gap is ~identical whether rank-1 is correct (0.602) or wrong (0.569) — mechanical decay, not a trustworthy separatrix; rank-1 is right in 94% of single-answer cases, so "return a tight set" is the whole win and cliff-cutting just adds noise. Exports `AdaptiveReturnConfig`, `DEFAULT_ADAPTIVE_RETURN` (frozen: `enabled=false`, `entityMax=2`, `otherMax=6`, `minKeep=1`), `AdaptiveReturnDecision` (`{applied, intent, cap, kept, total}`), `AdaptiveReturnInput` (`boolean | Partial<AdaptiveReturnConfig> | undefined`), `adaptiveReturnFromConfig(cfg)` (reads the config plane), `resolveAdaptiveReturn(perCall, fromConfig)` (defaults → config → per-call merge), `adaptiveReturnEnabled(...)` (gate check for the cache skip), and `applyAdaptiveReturn(results, intent, cfg)` (the trim). Config knobs (DB or file plane): `search.adaptive_return` (master switch, boolean), `search.adaptive_return_entity_max`, `search.adaptive_return_other_max`, `search.adaptive_return_min_keep` (each clamped to ≥1). Precision-sensitive agents tune entity=1, other=1 ≈ "return only the top result." Wired into `hybridSearch` (`src/core/search/hybrid.ts`): applied AFTER `applyReranker`, BEFORE the `limit` slice, and ONLY on the first page (`offset===0`) — paginating a confidence-gated set is incoherent, so paginated calls fall through to the fixed limit. Stamps the decision onto `HybridSearchMeta.adaptive_return` for `gbrain search --explain`. `hybridSearchCached` SKIPS the cache when the gate is on (a trimmed set must not be served to a gate-off lookup and vice versa); folding the gate params into `KNOBS_HASH` so adaptive-on calls cache safely is a v0.42+ follow-up before any mode-default flip. `SearchOpts.adaptiveReturn` + `HybridSearchMeta.adaptive_return` declared in `src/core/types.ts`. **Agent-facing surface (v0.41.33.0):** the `query` op (`src/core/operations.ts`) exposes an `adaptive_return` boolean param whose description instructs the agent WHEN to set it (single-answer questions → on; breadth/exploration → off; pass `limit:1` for a hard single-answer cap), threaded into `hybridSearchCached`. This is the path that matters — end users never touch the config knob; their agent decides per query (same pattern as `salience`/`recency`). Pinned by `test/search/return-policy.test.ts` (mechanism) + `test/search/query-op-adaptive-return.test.ts` (agent surface: param exists + description teaches both directions + the never-empty contract).
- `src/core/search/autocut.ts` (v0.42.3.0, NEW — default ON in reranked modes) — Weaviate-style autocut: score-discontinuity result-sizing on the cross-encoder rerank separatrix. Fixes part of #1663 (the "20 vs 1" precision problem). `applyAutocut(results, scoreOf, cfg)` normalizes the reranker scores, finds the largest consecutive gap, and cuts there when it clears `jumpRatio` (default 0.20); robust to unsorted provider output (cuts on a sorted copy, keeps items in INPUT order via a score threshold), guards `top<=0`/non-finite, never returns empty, and no-ops when <2 results carry a finite `rerank_score` (covers the reranker fail-open path). WHY rerank_score and NOT RRF/cosine: gbrain measured (see [[return-policy.ts]]) that the RRF rank1→rank2 gap is ~flat whether rank-1 is right or wrong — not a separatrix; the cross-encoder score IS. So autocut runs ONLY where the reranker ran (the floor reaches `balanced`+`tokenmax`; `conservative` is a documented no-op). Exports `AutocutConfig`, `DEFAULT_AUTOCUT` (frozen: `enabled=true, jumpRatio=0.20, minKeep=1`), `AutocutDecision` (`{applied, signal:'rerank'|'none', cut, kept, total, gapRatio}`), `AutocutInput`, `autocutFromConfig`, `resolveAutocut`, `applyAutocut`. **Cache-key integration (clean path, not the adaptive-return cache-skip hack):** enable+sensitivity flow through `ModeBundle``ResolvedSearchKnobs``knobsHash` exactly like `graph_signals`. `mode.ts` adds `autocut`/`autocut_jump` (conservative false, balanced/tokenmax true@0.20) AND sets `reranker_top_n_in = searchLimit` for reranked modes (D4 — so the reranker scores the full returned set; there is no un-scored tail for autocut to wrongly drop — closes the Codex review's load-bearing recall finding). `KNOBS_HASH_VERSION` 7→8 (master title_boost claimed v=7; autocut appends v=8 — one-time global cache cold-miss on upgrade). Wired into `hybridSearch` AFTER adaptive-return, BEFORE the limit slice, first page only; emits `HybridSearchMeta.autocut`. BOTH the cache-miss `finalMeta` and cache-HIT `cachedMeta` rebuilds carry `autocut`+`adaptive_return`+`mode`+`embedding_column` (Codex P2; also fixes a pre-existing adaptive_return meta-drop). Preserves alias-hop exact matches (Codex P1): `applyAutocut` takes an optional `preserve` predicate; hybrid passes `r => r.alias_hit === true` so a canonical page injected by `applyAliasHop` after reranking (no `rerank_score`) is never cut. Agent surface: `query` op `autocut` boolean (ceiling override — `false` forces full top-K); `SearchOpts.autocut`; `--explain` shows per-result `rerank_score` (cut visible by which results survive), `formatAutocutSummary` renders the decision when search meta is threaded; `gbrain search modes` attribution; metric glossary `autocut.signal`/`autocut.gap_ratio`. Config: `search.autocut`, `search.autocut_jump`. **Default-ON is backed by an in-repo eval gate**`test/search/autocut-eval.test.ts` (also `bun run eval:autocut`) measures precision-lift-without-recall-regression over labeled qrels fixtures with modeled cross-encoder distributions (no API key, no sibling repo; runs in CI): mean precision 0.33→0.94, recall 1.00→0.95, ZERO recall regression on enumeration queries (autocut declines on flat curves by construction). Env-overridable floors. A live-corpus PrecisionMemBench run (gbrain-evals) is an optional empirical confirmation, not a blocker. Pinned by `test/search/autocut.test.ts` (pure-fn), `test/search/query-op-autocut.test.ts` (agent surface), `test/search/autocut-integration.serial.test.ts` (IRON-RULE behavioral via `rerankerFn` DI seam: cliff trims, flat doesn't, no-reranker no-ops, `autocut:false` ceiling, composes with adaptive-return), `test/search/autocut-eval.test.ts` (the precision/recall gate), and the v=8 knobsHash assertions in `test/search-mode.test.ts`. One wave of the #1663 multi-wave redesign; remaining waves (router, tiered tier, CRAG escalation) keep the issue open.
- `src/core/ai/recipes/voyage.ts` — Voyage AI openai-compatible recipe. **v0.28.7 (#680):** declares `chars_per_token=1` + `safety_factor=0.5` so the gateway pre-splits Voyage batches at a 60K-character budget (50% of 120K-token cap with the dense-tokenizer ratio). Closes the v0.27 backfill loop where ~26% of the corpus stayed un-embedded because tiktoken-grounded budgeting silently undercounted Voyage's actual token usage. **v0.28.11 (#719):** declares `multimodal_models: ['voyage-multimodal-3']` so the gateway rejects text-only Voyage models pointed at the multimodal endpoint with a clear `AIConfigError` instead of waiting for Voyage's HTTP 400. **v0.33.1.1 (#962, fixup):** recipe docstring at `:7-16` tightened to name the seven hosted flexible-dim models that accept `output_dimension` explicitly (`voyage-4-large`, `voyage-4`, `voyage-4-lite`, `voyage-3-large`, `voyage-3.5`, `voyage-3.5-lite`, `voyage-code-3`) and call out that `voyage-4-nano` is the open-weight variant listed separately by Voyage as fixed 1024-dim — does NOT accept the parameter. The "all v4 variants are flexible" misread is what caused the original PR to include nano in `VOYAGE_OUTPUT_DIMENSION_MODELS`; the negative regression assertion in `test/ai/gateway.test.ts` (`dimsProviderOptions` returns `undefined` for `voyage-4-nano`) pins the contract. **v0.37.3.0:** `voyage-code-3` is the recommended embedding model for gstack per-worktree code brains (Topology 3 in `docs/architecture/topologies.md`). Registration was already in the `models` list since pre-v0.33; the v0.37.3.0 wave adds discoverability surfaces — decision-tree branch in `docs/integrations/embedding-providers.md`, Topology 3 "Recommended embedding model" subsection, runtime nudge from `gbrain reindex --code` against non-code-tuned models. Recipe-shape regression pinned by `test/ai/voyage-code-3-recipe.test.ts`.
- `src/core/ai/recipes/anthropic.ts` — Anthropic recipe (chat + expansion touchpoints). **v0.31.12:** chat and expansion `models:` lists drop the v0.31.6 phantom `claude-sonnet-4-6-20250929` date suffix — canonical id is `claude-sonnet-4-6`. The wrong-direction alias `claude-sonnet-4-6 → claude-sonnet-4-6-20250929` is removed; a reverse alias `claude-sonnet-4-6-20250929 → claude-sonnet-4-6` keeps stale user configs working (rescues `facts.extraction_model` and `models.dream.synthesize` set by v0.31.6 installs). Recipe-shape regression pinned by `test/anthropic-model-ids.test.ts` (6 cases, verbatim cherry-pick of PR #830 plus the reverse-alias rescue case).
- `src/core/anthropic-pricing.ts` — Single source of truth for Anthropic model pricing (per-MTok input/output). **v0.31.12:** Opus 4.7 corrected from `$15/$75` to `$5/$25` (the old number was from Opus 4 generation, never refreshed when 4.7 shipped); Opus 4.6 also corrected. Consumed by `src/core/budget-meter.ts` and `src/core/cross-modal-eval/runner.ts` — the cross-modal estimator now reads `ANTHROPIC_PRICING` for Anthropic models instead of duplicating the table, killing the v0.31.6 drift bug class.
@@ -1946,6 +1947,27 @@ done
If any SHA differs from what's in the workflow files, update the pin and version comment.
## PR title format — version FIRST (IRON RULE)
**Every PR title MUST start with the version, then the conventional-commit subject:**
```
vMAJOR.MINOR.PATCH.MICRO <type>(<scope>): <summary> (#issue or wave ref)
```
Example (correct): `v0.42.3.0 feat(search): autocut — score-discontinuity result-sizing (#1663 wave 1)`
The version goes at the **BEGINNING**, never the end. This matches the repo's
commit-subject convention (`git log` shows `v0.41.38.0 fix: ...`,
`v0.42.1.0 feat: ...`) so the PR list, the merge commit, and the changelog all
read version-first. A title with the version parenthesized at the end
(`feat(search): autocut ... (v0.42.3.0)`) is WRONG — fix it with
`gh pr edit <N> --title "vX.Y.Z.W <type>: <summary>"`.
This applies to `gh pr create` and every `gh pr edit --title`. When `/ship`
(or any flow) sets a PR title, the version is the first token. Same rule for the
final commit subject that carries the version bump.
## PR descriptions cover the whole branch
Pull request titles and bodies must describe **everything in the PR diff against the
+1 -1
View File
@@ -1 +1 @@
0.42.2.0
0.42.3.0
+18
View File
@@ -150,6 +150,24 @@ Every metric `gbrain eval *` and `gbrain search stats` reports has a plain-Engli
**Range:** 0..unbounded. Warm-cache hits should be <50ms; tokenmax with expansion can exceed 200ms due to the Haiku call.
## Result-Sizing Metrics
### Autocut signal
**Key:** `autocut.signal`
**Plain English:** Which signal autocut used to size the result set. 'rerank' means it found a real score cliff in the cross-encoder rerank scores and cut there; 'none' means no trustworthy cliff (no reranker, <2 scored results, or the gap was too small) so it returned the full list.
**Range:** 'rerank' | 'none'. 'none' is not a failure — it means autocut declined to cut because the signal didn't justify it.
### Autocut gap ratio
**Key:** `autocut.gap_ratio`
**Plain English:** The size of the largest score drop autocut found, as a fraction of the top result's score. A gap of 0.40 means the score fell by 40% of the top score at the steepest point. Autocut cuts there only when this clears the sensitivity threshold (autocut_jump, default 0.20).
**Range:** 0..1, higher = a sharper cliff (more confident cut). Below the autocut_jump threshold → no cut.
---
## Coverage
+22
View File
@@ -275,6 +275,7 @@ strict behavior when unset.
- `src/core/search/embedding-column.ts` (v0.36.3.0) — single source of truth for "which `content_chunks.*` column does this query rank against?" Pure functions, no engine I/O: `loadRegistry(cfg)` walks the `embedding_columns` config (DB plane, JSON map keyed by column name with `{provider, dimensions, type}` entries), seeds the OpenAI `embedding` builtin when unset, validates everything before it lands (column-name regex, type ∈ `vector | halfvec`, dims in [1, 8192], provider format) using `Object.create(null)` + `Object.hasOwn` so a key like `constructor` rejects instead of resolving to `Object.prototype.constructor`. `resolveColumn(registry, override?, cfg)` is the boundary call: returns a `ResolvedColumn` descriptor (frozen `{name, provider, dimensions, type}`) honoring per-call override → `search_embedding_column` config → `'embedding'` default. Throws `UnknownEmbeddingColumnError` with the list of registered names on miss. `isCacheSafe(resolved, cfg)` compares the full embedding SPACE (provider + dimensions + name) against cfg's default — a user who repointed the `embedding` builtin at Voyage doesn't accidentally serve OpenAI-shaped cache rows. `validateResolvedColumn(descriptor)` re-validates hand-rolled descriptors that bypass the registry (internal-SDK passthrough path) so the SQL-injection escape hatch through the descriptor field is closed. Consumed by `hybridSearch` (resolves once at the boundary, threads a `ResolvedColumn` into per-engine `searchVector` instead of a raw string), `gateway.embedQuery(text, {embeddingModel, dimensions})` (resolved column's provider drives the query-time embed call), `cosineReScore` (engine pulls vectors from the active column, not the hardcoded `embedding`), and the `query` MCP op (per-call `embedding_column` param). 538 lines of pure resolver, 511 unit cases in `test/search/embedding-column.test.ts` covering the 16 codex-flagged corners (prototype-pollution, descriptor passthrough, env-only Postgres install, empty-brain coverage gate, cache-space comparison).
- `src/core/search/rerank.ts` (v0.35.0.0) — the call-site abstraction. `applyReranker(query, results, opts)` slots between `dedupResults()` and `enforceTokenBudget()` in `src/core/search/hybrid.ts`. Slices `opts.topNIn` (default 30) by current RRF order, sends to `gateway.rerank()`, reorders by `relevanceScore` desc, and appends the un-reranked tail unchanged (recall protection). **Fail-open on every `RerankError.reason`**: any error logs via `logRerankFailure` and returns the input array unchanged. Stamps `rerank_score` onto reordered items so downstream telemetry sees the new ordering signal. `topNOut: null` is the explicit "don't truncate" signal — semantically distinct from `undefined` which means "fall through to mode bundle" (CDX2-F16). Test seam: `opts.rerankerFn` lets tests stub `gateway.rerank` without touching the network.
- `src/core/search/return-policy.ts` (v0.41.33.0, NEW — default OFF) — intent-aware adaptive return-sizing. Pure, dependency-light module that trims the final ranked candidate set to an intent-driven cap instead of always returning the full top-K (noisy) list. `entity` intent (single-answer-ish) gets a tight cap; `temporal`/`event`/`general` (enumeration-ish) get a recall-preserving cap. A `minKeep` failsafe (≥1) guarantees a human never gets a silent blank when candidates exist. WHY a cap, not a score-cliff detector: PrecisionMemBench Phase-1 instrumentation (gbrain-evals) measured that the rank1→rank2 RRF score gap is ~identical whether rank-1 is correct (0.602) or wrong (0.569) — mechanical decay, not a trustworthy separatrix; rank-1 is right in 94% of single-answer cases, so "return a tight set" is the whole win and cliff-cutting just adds noise. Exports `AdaptiveReturnConfig`, `DEFAULT_ADAPTIVE_RETURN` (frozen: `enabled=false`, `entityMax=2`, `otherMax=6`, `minKeep=1`), `AdaptiveReturnDecision` (`{applied, intent, cap, kept, total}`), `AdaptiveReturnInput` (`boolean | Partial<AdaptiveReturnConfig> | undefined`), `adaptiveReturnFromConfig(cfg)` (reads the config plane), `resolveAdaptiveReturn(perCall, fromConfig)` (defaults → config → per-call merge), `adaptiveReturnEnabled(...)` (gate check for the cache skip), and `applyAdaptiveReturn(results, intent, cfg)` (the trim). Config knobs (DB or file plane): `search.adaptive_return` (master switch, boolean), `search.adaptive_return_entity_max`, `search.adaptive_return_other_max`, `search.adaptive_return_min_keep` (each clamped to ≥1). Precision-sensitive agents tune entity=1, other=1 ≈ "return only the top result." Wired into `hybridSearch` (`src/core/search/hybrid.ts`): applied AFTER `applyReranker`, BEFORE the `limit` slice, and ONLY on the first page (`offset===0`) — paginating a confidence-gated set is incoherent, so paginated calls fall through to the fixed limit. Stamps the decision onto `HybridSearchMeta.adaptive_return` for `gbrain search --explain`. `hybridSearchCached` SKIPS the cache when the gate is on (a trimmed set must not be served to a gate-off lookup and vice versa); folding the gate params into `KNOBS_HASH` so adaptive-on calls cache safely is a v0.42+ follow-up before any mode-default flip. `SearchOpts.adaptiveReturn` + `HybridSearchMeta.adaptive_return` declared in `src/core/types.ts`. **Agent-facing surface (v0.41.33.0):** the `query` op (`src/core/operations.ts`) exposes an `adaptive_return` boolean param whose description instructs the agent WHEN to set it (single-answer questions → on; breadth/exploration → off; pass `limit:1` for a hard single-answer cap), threaded into `hybridSearchCached`. This is the path that matters — end users never touch the config knob; their agent decides per query (same pattern as `salience`/`recency`). Pinned by `test/search/return-policy.test.ts` (mechanism) + `test/search/query-op-adaptive-return.test.ts` (agent surface: param exists + description teaches both directions + the never-empty contract).
- `src/core/search/autocut.ts` (v0.42.3.0, NEW — default ON in reranked modes) — Weaviate-style autocut: score-discontinuity result-sizing on the cross-encoder rerank separatrix. Fixes part of #1663 (the "20 vs 1" precision problem). `applyAutocut(results, scoreOf, cfg)` normalizes the reranker scores, finds the largest consecutive gap, and cuts there when it clears `jumpRatio` (default 0.20); robust to unsorted provider output (cuts on a sorted copy, keeps items in INPUT order via a score threshold), guards `top<=0`/non-finite, never returns empty, and no-ops when <2 results carry a finite `rerank_score` (covers the reranker fail-open path). WHY rerank_score and NOT RRF/cosine: gbrain measured (see [[return-policy.ts]]) that the RRF rank1→rank2 gap is ~flat whether rank-1 is right or wrong — not a separatrix; the cross-encoder score IS. So autocut runs ONLY where the reranker ran (the floor reaches `balanced`+`tokenmax`; `conservative` is a documented no-op). Exports `AutocutConfig`, `DEFAULT_AUTOCUT` (frozen: `enabled=true, jumpRatio=0.20, minKeep=1`), `AutocutDecision` (`{applied, signal:'rerank'|'none', cut, kept, total, gapRatio}`), `AutocutInput`, `autocutFromConfig`, `resolveAutocut`, `applyAutocut`. **Cache-key integration (clean path, not the adaptive-return cache-skip hack):** enable+sensitivity flow through `ModeBundle` → `ResolvedSearchKnobs` → `knobsHash` exactly like `graph_signals`. `mode.ts` adds `autocut`/`autocut_jump` (conservative false, balanced/tokenmax true@0.20) AND sets `reranker_top_n_in = searchLimit` for reranked modes (D4 — so the reranker scores the full returned set; there is no un-scored tail for autocut to wrongly drop — closes the Codex review's load-bearing recall finding). `KNOBS_HASH_VERSION` 7→8 (master title_boost claimed v=7; autocut appends v=8 — one-time global cache cold-miss on upgrade). Wired into `hybridSearch` AFTER adaptive-return, BEFORE the limit slice, first page only; emits `HybridSearchMeta.autocut`. BOTH the cache-miss `finalMeta` and cache-HIT `cachedMeta` rebuilds carry `autocut`+`adaptive_return`+`mode`+`embedding_column` (Codex P2; also fixes a pre-existing adaptive_return meta-drop). Preserves alias-hop exact matches (Codex P1): `applyAutocut` takes an optional `preserve` predicate; hybrid passes `r => r.alias_hit === true` so a canonical page injected by `applyAliasHop` after reranking (no `rerank_score`) is never cut. Agent surface: `query` op `autocut` boolean (ceiling override — `false` forces full top-K); `SearchOpts.autocut`; `--explain` shows per-result `rerank_score` (cut visible by which results survive), `formatAutocutSummary` renders the decision when search meta is threaded; `gbrain search modes` attribution; metric glossary `autocut.signal`/`autocut.gap_ratio`. Config: `search.autocut`, `search.autocut_jump`. **Default-ON is backed by an in-repo eval gate** — `test/search/autocut-eval.test.ts` (also `bun run eval:autocut`) measures precision-lift-without-recall-regression over labeled qrels fixtures with modeled cross-encoder distributions (no API key, no sibling repo; runs in CI): mean precision 0.33→0.94, recall 1.00→0.95, ZERO recall regression on enumeration queries (autocut declines on flat curves by construction). Env-overridable floors. A live-corpus PrecisionMemBench run (gbrain-evals) is an optional empirical confirmation, not a blocker. Pinned by `test/search/autocut.test.ts` (pure-fn), `test/search/query-op-autocut.test.ts` (agent surface), `test/search/autocut-integration.serial.test.ts` (IRON-RULE behavioral via `rerankerFn` DI seam: cliff trims, flat doesn't, no-reranker no-ops, `autocut:false` ceiling, composes with adaptive-return), `test/search/autocut-eval.test.ts` (the precision/recall gate), and the v=8 knobsHash assertions in `test/search-mode.test.ts`. One wave of the #1663 multi-wave redesign; remaining waves (router, tiered tier, CRAG escalation) keep the issue open.
- `src/core/ai/recipes/voyage.ts` — Voyage AI openai-compatible recipe. **v0.28.7 (#680):** declares `chars_per_token=1` + `safety_factor=0.5` so the gateway pre-splits Voyage batches at a 60K-character budget (50% of 120K-token cap with the dense-tokenizer ratio). Closes the v0.27 backfill loop where ~26% of the corpus stayed un-embedded because tiktoken-grounded budgeting silently undercounted Voyage's actual token usage. **v0.28.11 (#719):** declares `multimodal_models: ['voyage-multimodal-3']` so the gateway rejects text-only Voyage models pointed at the multimodal endpoint with a clear `AIConfigError` instead of waiting for Voyage's HTTP 400. **v0.33.1.1 (#962, fixup):** recipe docstring at `:7-16` tightened to name the seven hosted flexible-dim models that accept `output_dimension` explicitly (`voyage-4-large`, `voyage-4`, `voyage-4-lite`, `voyage-3-large`, `voyage-3.5`, `voyage-3.5-lite`, `voyage-code-3`) and call out that `voyage-4-nano` is the open-weight variant listed separately by Voyage as fixed 1024-dim — does NOT accept the parameter. The "all v4 variants are flexible" misread is what caused the original PR to include nano in `VOYAGE_OUTPUT_DIMENSION_MODELS`; the negative regression assertion in `test/ai/gateway.test.ts` (`dimsProviderOptions` returns `undefined` for `voyage-4-nano`) pins the contract. **v0.37.3.0:** `voyage-code-3` is the recommended embedding model for gstack per-worktree code brains (Topology 3 in `docs/architecture/topologies.md`). Registration was already in the `models` list since pre-v0.33; the v0.37.3.0 wave adds discoverability surfaces — decision-tree branch in `docs/integrations/embedding-providers.md`, Topology 3 "Recommended embedding model" subsection, runtime nudge from `gbrain reindex --code` against non-code-tuned models. Recipe-shape regression pinned by `test/ai/voyage-code-3-recipe.test.ts`.
- `src/core/ai/recipes/anthropic.ts` — Anthropic recipe (chat + expansion touchpoints). **v0.31.12:** chat and expansion `models:` lists drop the v0.31.6 phantom `claude-sonnet-4-6-20250929` date suffix — canonical id is `claude-sonnet-4-6`. The wrong-direction alias `claude-sonnet-4-6 → claude-sonnet-4-6-20250929` is removed; a reverse alias `claude-sonnet-4-6-20250929 → claude-sonnet-4-6` keeps stale user configs working (rescues `facts.extraction_model` and `models.dream.synthesize` set by v0.31.6 installs). Recipe-shape regression pinned by `test/anthropic-model-ids.test.ts` (6 cases, verbatim cherry-pick of PR #830 plus the reverse-alias rescue case).
- `src/core/anthropic-pricing.ts` — Single source of truth for Anthropic model pricing (per-MTok input/output). **v0.31.12:** Opus 4.7 corrected from `$15/$75` to `$5/$25` (the old number was from Opus 4 generation, never refreshed when 4.7 shipped); Opus 4.6 also corrected. Consumed by `src/core/budget-meter.ts` and `src/core/cross-modal-eval/runner.ts` — the cross-modal estimator now reads `ANTHROPIC_PRICING` for Anthropic models instead of duplicating the table, killing the v0.31.6 drift bug class.
@@ -2088,6 +2089,27 @@ done
If any SHA differs from what's in the workflow files, update the pin and version comment.
## PR title format — version FIRST (IRON RULE)
**Every PR title MUST start with the version, then the conventional-commit subject:**
```
vMAJOR.MINOR.PATCH.MICRO <type>(<scope>): <summary> (#issue or wave ref)
```
Example (correct): `v0.42.3.0 feat(search): autocut — score-discontinuity result-sizing (#1663 wave 1)`
The version goes at the **BEGINNING**, never the end. This matches the repo's
commit-subject convention (`git log` shows `v0.41.38.0 fix: ...`,
`v0.42.1.0 feat: ...`) so the PR list, the merge commit, and the changelog all
read version-first. A title with the version parenthesized at the end
(`feat(search): autocut ... (v0.42.3.0)`) is WRONG — fix it with
`gh pr edit <N> --title "vX.Y.Z.W <type>: <summary>"`.
This applies to `gh pr create` and every `gh pr edit --title`. When `/ship`
(or any flow) sets a PR title, the version is the first token. Same rule for the
final commit subject that carries the version bump.
## PR descriptions cover the whole branch
Pull request titles and bodies must describe **everything in the PR diff against the
+2 -1
View File
@@ -38,6 +38,7 @@
"build:llms": "bun run scripts/build-llms.ts",
"build:pglite-snapshot": "bun run scripts/build-pglite-snapshot.ts",
"test": "bash scripts/run-unit-parallel.sh",
"eval:autocut": "bun test test/search/autocut-eval.test.ts",
"test:full": "bun run verify && bash scripts/run-unit-parallel.sh && bun run test:slow && ([ -n \"$DATABASE_URL\" ] && bash scripts/run-e2e.sh || echo '[test:full] skipped E2E (no DATABASE_URL); run docker-compose -f docker-compose.ci.yml up + bun run test:e2e to include' 1>&2)",
"verify": "bash scripts/run-verify-parallel.sh",
"check:source-config-leak": "scripts/check-source-config-leak.sh",
@@ -141,5 +142,5 @@
"bun": ">=1.3.10"
},
"license": "MIT",
"version": "0.42.2.0"
"version": "0.42.3.0"
}
+3
View File
@@ -68,6 +68,9 @@ const KNOB_DESCRIPTIONS: Record<keyof ModeBundle, string> = {
// v0.40.3.0 contextual retrieval
contextual_retrieval: 'CR tier (none|title|per_chunk_synopsis) — wraps chunks at embed time',
contextual_retrieval_disabled: 'Soft kill switch — neutralizes CR wrapping for queries + new embeds',
// v0.42.3.0 autocut
autocut: 'Score-discontinuity result-sizing (cuts at the rerank-score cliff; no-op without a reranker)',
autocut_jump: 'Autocut sensitivity: min normalized score gap that counts as a cliff (0..1, 0.20 default)',
};
interface SearchModesReport {
+15
View File
@@ -133,6 +133,20 @@ export const METRIC_GLOSSARY: Readonly<Record<string, Readonly<MetricGlossEntry>
eli10: '99th percentile wall-clock time per search call. The latency that 1% of users see — long-tail experience, not the average.',
range: '0..unbounded. Warm-cache hits should be <50ms; tokenmax with expansion can exceed 200ms due to the Haiku call.',
}),
// ────────────────────────────────────────────────────────────────────────
// Result-sizing metrics (v0.42.3.0 autocut)
// ────────────────────────────────────────────────────────────────────────
'autocut.signal': Object.freeze({
industry_term: 'Autocut signal',
eli10: "Which signal autocut used to size the result set. 'rerank' means it found a real score cliff in the cross-encoder rerank scores and cut there; 'none' means no trustworthy cliff (no reranker, <2 scored results, or the gap was too small) so it returned the full list.",
range: "'rerank' | 'none'. 'none' is not a failure — it means autocut declined to cut because the signal didn't justify it.",
}),
'autocut.gap_ratio': Object.freeze({
industry_term: 'Autocut gap ratio',
eli10: 'The size of the largest score drop autocut found, as a fraction of the top result\'s score. A gap of 0.40 means the score fell by 40% of the top score at the steepest point. Autocut cuts there only when this clears the sensitivity threshold (autocut_jump, default 0.20).',
range: '0..1, higher = a sharper cliff (more confident cut). Below the autocut_jump threshold → no cut.',
}),
});
/**
@@ -210,6 +224,7 @@ export function renderMetricGlossaryMarkdown(): string {
['Set-Similarity / Stability Metrics', ['jaccard@k', 'top1_stability']],
['Statistical-Significance Metrics', ['p_value', 'confidence_interval']],
['Operational / Cost Metrics', ['cache_hit_rate', 'avg_results', 'avg_tokens', 'cost_per_query_usd', 'p99_latency_ms']],
['Result-Sizing Metrics', ['autocut.signal', 'autocut.gap_ratio']],
];
for (const [groupTitle, metrics] of groups) {
+11
View File
@@ -1385,6 +1385,14 @@ const query: Operation = {
" Omit / FALSE for breadth — 'everything about X', 'list all', 'what do I know about Y', exploration, brainstorming, or any time you'd rather see more candidates and judge for yourself. Recall matters more there, so take the full top-K.\n" +
"Safe by construction: it NEVER returns empty when there are matches (you always get at least the top hit), and it only applies to the first page (omit when paginating). Caps come from config (search.adaptive_return_entity_max / _other_max; default 2 / 6) — pass `limit` 1 alongside this for a hard single-answer cap.",
},
autocut: {
type: 'boolean',
description:
"v0.42.3.0 — autocut is the SMART DEFAULT (already ON when the reranker runs, which it does in the default search mode). It returns only the confident cluster by cutting where the relevance score drops off a cliff, so an obvious single answer comes back as 1 result and a genuine handful comes back as that handful — not a fixed wall of 20+.\n" +
" You almost never set this. Pass FALSE only to FORCE the full top-K when you deliberately want breadth — broad exploration, 'show me everything about X', enumeration where you'd rather over-collect and judge for yourself, or when you suspect the top hit is wrong and want to see the alternatives.\n" +
" TRUE is redundant in default mode (it's already on); it only matters to override a brain whose config turned autocut off.\n" +
"Safe by construction: never returns empty when there are matches, only applies to the first page (omit when paginating), and is a no-op when no reranker scored the results (so it can't cut on an untrustworthy signal). Distinct from `adaptive_return`: autocut cuts on the score cliff; adaptive_return caps by question intent. Leave both unset for the smart default.",
},
},
handler: async (ctx, p) => {
const startedAt = Date.now();
@@ -1479,6 +1487,9 @@ const query: Operation = {
// v0.41.33 — agent-explicit adaptive return-sizing. Omitted = off
// (config default applies). hybridSearchCached skips the cache when on.
adaptiveReturn: typeof p.adaptive_return === 'boolean' ? (p.adaptive_return as boolean) : undefined,
// v0.42.3.0 — autocut ceiling override. Omitted = smart default (ON in
// reranked modes). `false` forces the full top-K.
autocut: typeof p.autocut === 'boolean' ? (p.autocut as boolean) : undefined,
});
const latency_ms = Date.now() - startedAt;
+221
View File
@@ -0,0 +1,221 @@
/**
* autocut.ts — score-discontinuity result-sizing on the rerank separatrix (v0.42.3.0).
*
* Weaviate-style "autocut": instead of returning a fixed top-K (noisy), cut the
* ranked list where the score curve breaks. Returns 1 when the answer is obvious,
* the cluster when it's genuinely several, never K just because K was the limit.
* Fixes part of issue #1663 (the "20 vs 1" precision problem), recommendation #2.
*
* WHY this runs on the cross-encoder rerank score and NOTHING else: gbrain
* measured (PrecisionMemBench Phase-1, documented in return-policy.ts) that the
* RRF/cosine rank1→rank2 gap is ~identical whether rank-1 is right (0.602) or
* wrong (0.569) — mechanical decay, not a trustworthy separatrix. The reranker's
* relevance score IS a real cliff. So autocut reads rerank_score; the caller
* gates it on the reranker having actually produced scores (it fails open to RRF
* order on auth/network/timeout — see hybrid.ts), and autocut itself no-ops when
* fewer than 2 items carry a finite score.
*
* Pure + dependency-light so it unit-tests in isolation. Mirrors return-policy.ts's
* resolve-ladder shape; the two are deliberately separate modules (different cut
* signals — score-cliff vs intent-cap) until a third trimmer justifies extraction.
*/
export interface AutocutConfig {
/** Module-default master switch. The EFFECTIVE enable is the mode-bundle knob
* (resolvedMode.autocut) gated on the reranker having scored ≥2 items; the
* caller passes `enabled: true` explicitly once that gate passes. */
enabled: boolean;
/**
* Minimum normalized gap (relative to the top score) that counts as a cliff.
* Eval-derived starting point (calibrated by the PrecisionMemBench run), NOT a
* magic constant — it's a per-mode ModeBundle knob. Clamped to (0, 1].
*/
jumpRatio: number;
/** Failsafe: never return fewer than this when candidates exist (≥1). */
minKeep: number;
}
/**
* Defaults. enabled=true here is the MODULE default; whether autocut actually
* fires is decided by the mode bundle + the reranker-scored-prefix gate in
* hybrid.ts. jumpRatio=0.20 means "a drop of ≥20% of the top score is a cliff."
*/
export const DEFAULT_AUTOCUT: AutocutConfig = Object.freeze({
enabled: true,
jumpRatio: 0.2,
minKeep: 1,
});
export interface AutocutDecision {
applied: boolean;
/** 'rerank' when a real cliff was cut; 'none' when no cut (no signal / no cliff). */
signal: 'rerank' | 'none';
/** Number of items kept (the cut point). */
cut: number;
kept: number;
total: number;
/** The largest normalized gap observed (0 when <2 scored items). */
gapRatio: number;
}
/** Per-call SearchOpts shape: `true`/`false` toggle, or a partial override. */
export type AutocutInput = boolean | Partial<AutocutConfig> | undefined;
/** Read autocut defaults from a loaded config object (DB or file plane).
* Out-of-range values are IGNORED (left unset) so they fall through to the
* mode bundle / module default — mirrors loadOverridesFromConfig. */
export function autocutFromConfig(
cfg: Record<string, unknown> | null | undefined,
): Partial<AutocutConfig> {
const search = (cfg?.search ?? {}) as Record<string, unknown>;
const out: Partial<AutocutConfig> = {};
if (typeof search.autocut === 'boolean') out.enabled = search.autocut;
if (search.autocut_jump !== undefined) {
const n = typeof search.autocut_jump === 'number' ? search.autocut_jump : Number.NaN;
if (Number.isFinite(n) && n > 0 && n <= 1) out.jumpRatio = n;
}
if (search.autocut_min_keep !== undefined) {
const n =
typeof search.autocut_min_keep === 'number' ? Math.floor(search.autocut_min_keep) : Number.NaN;
if (Number.isFinite(n) && n >= 1) out.minKeep = n;
}
return out;
}
/** Merge defaults → config-plane → per-call into a concrete config. */
export function resolveAutocut(
perCall: AutocutInput,
fromConfig?: Partial<AutocutConfig>,
): AutocutConfig {
const base: AutocutConfig = { ...DEFAULT_AUTOCUT, ...(fromConfig ?? {}) };
if (perCall === undefined) return base;
if (perCall === true) return { ...base, enabled: true };
if (perCall === false) return { ...base, enabled: false };
return {
...base,
...perCall,
enabled: perCall.enabled ?? base.enabled,
};
}
function noOp<T>(results: T[]): { kept: T[]; decision: AutocutDecision } {
return {
kept: results,
decision: {
applied: false,
signal: 'none',
cut: results.length,
kept: results.length,
total: results.length,
gapRatio: 0,
},
};
}
/**
* Trim a ranked result list at the largest score discontinuity.
*
* `scoreOf(r)` must return the cross-encoder rerank score (or undefined/non-finite
* for un-scored items). The function is robust to un-sorted provider output: it
* finds the cliff on a sorted copy of the finite scores and keeps every item at or
* above the cut threshold, preserving the input's original order. Never returns
* empty when `results` is non-empty (at-least-minKeep failsafe).
*
* Behavior:
* - cfg.enabled false → no-op (signal 'none')
* - <2 items with a finite score → no-op (no cliff to find; recall preserved)
* - top score <= 0 or non-finite → no-op (score scale unusable)
* - largest normalized gap < jumpRatio → no-op (no real cliff)
* - otherwise → keep items scored >= the cut threshold,
* dropping the lower-scored remainder AND
* any un-scored items (they carry no
* confidence signal)
*/
export function applyAutocut<T>(
results: T[],
scoreOf: (r: T) => number | undefined | null,
cfg: AutocutConfig,
/**
* Optional always-keep predicate. Items where `preserve(r)` is true survive
* the cut regardless of score (and are NOT required to carry a finite score).
* Used to protect structurally-injected high-confidence results that bypass
* reranking — e.g. an exact alias-hop match (`alias_hit === true`) inserted
* after the reranker ran, which therefore has no `rerank_score`. Without this,
* autocut would drop the alias-injected page when it cuts on the scored set.
*/
preserve?: (r: T) => boolean,
): { kept: T[]; decision: AutocutDecision } {
if (!cfg.enabled || results.length < 2) return noOp(results);
// Collect finite scores. Under D4 (rerank the full candidate set) every item is
// scored; we still filter defensively so a fail-open reranker (RRF order, no
// scores) or a partial head degrades to a clean no-op.
const scores: number[] = [];
for (const r of results) {
const s = scoreOf(r);
if (typeof s === 'number' && Number.isFinite(s)) scores.push(s);
}
if (scores.length < 2) return noOp(results);
const top = Math.max(...scores);
if (!Number.isFinite(top) || top <= 0) return noOp(results);
// Sort a copy descending (A2: don't trust upstream order) and normalize.
const sorted = [...scores].sort((a, b) => b - a);
const norm = sorted.map((s) => s / top);
const minKeep = Math.max(1, cfg.minKeep);
// Find the largest consecutive gap. Only consider cut points at or after
// minKeep (so the failsafe is never violated) and before the last element.
let bestGap = -1;
let bestIdx = -1; // cut AFTER sorted[bestIdx] → keep bestIdx+1 items
for (let i = minKeep - 1; i < norm.length - 1; i++) {
const gap = norm[i] - norm[i + 1];
if (gap > bestGap) {
bestGap = gap;
bestIdx = i;
}
}
if (bestIdx < 0 || bestGap < cfg.jumpRatio) {
// No cliff clears the threshold. Report the observed gap for telemetry.
return {
kept: results,
decision: {
applied: false,
signal: 'none',
cut: results.length,
kept: results.length,
total: results.length,
gapRatio: bestGap < 0 ? 0 : bestGap,
},
};
}
// Cut threshold = the score at the cut boundary. Keep every item scored at or
// above it (ties at the boundary stay together — conservative, never-empty),
// PLUS any item the caller marked preserve (alias-injected exact matches that
// bypassed reranking and carry no score).
const threshold = sorted[bestIdx];
const kept = results.filter((r) => {
if (preserve?.(r)) return true;
const s = scoreOf(r);
return typeof s === 'number' && Number.isFinite(s) && s >= threshold;
});
// Failsafe: a degenerate threshold could in theory keep 0 (it cannot here, since
// the top item always passes), but guard anyway.
if (kept.length === 0) return noOp(results);
return {
kept,
decision: {
applied: kept.length < results.length,
signal: kept.length < results.length ? 'rerank' : 'none',
cut: kept.length,
kept: kept.length,
total: results.length,
gapRatio: bestGap,
},
};
}
+29 -3
View File
@@ -28,7 +28,8 @@
* separate JSON formatter needed.
*/
import type { SearchResult } from '../types.ts';
import type { SearchResult, HybridSearchMeta } from '../types.ts';
import type { AutocutDecision } from './autocut.ts';
/**
* Format a single result with per-stage attribution. Returns a string
@@ -86,6 +87,13 @@ export function formatResultExplain(
const arrow = result.reranker_delta > 0 ? '↑' : '↓';
lines.push(` ${arrow} reranker rank ${result.reranker_delta > 0 ? '+' : ''}${result.reranker_delta}`);
}
// v0.42.3.0 — show the cross-encoder rerank score (the signal autocut cuts
// on). Surfacing it per result makes the autocut cliff legible: every kept
// result sits at or above the cut threshold.
if (result.rerank_score !== undefined) {
anyBoost = true;
lines.push(` • rerank score=${fmt(result.rerank_score)}`);
}
if (!anyBoost) {
lines.push(` no boosts applied`);
@@ -95,14 +103,32 @@ export function formatResultExplain(
return lines.join('\n');
}
/**
* v0.42.3.0 — one-line autocut summary for `--explain`. Returns null when
* autocut didn't run (no decision in meta) so callers can omit it cleanly.
*/
export function formatAutocutSummary(decision: AutocutDecision | undefined): string | null {
if (!decision) return null;
if (!decision.applied) {
return `autocut: no cut (signal=${decision.signal}, gap=${fmt(decision.gapRatio)} < threshold) — full ${decision.total} returned`;
}
return `autocut: cut at the rerank cliff (gap=${fmt(decision.gapRatio)}) — kept ${decision.kept}/${decision.total}`;
}
/**
* Format a full result list. Caller passes the SearchResult[] directly;
* the formatter handles enumeration. Returns a single string (multi-line
* with trailing newline so callers can `process.stdout.write(out)`).
*/
export function formatResultsExplain(results: SearchResult[]): string {
export function formatResultsExplain(
results: SearchResult[],
meta?: HybridSearchMeta,
): string {
if (results.length === 0) return 'No results.\n';
return results.map((r, i) => formatResultExplain(r, i + 1)).join('\n\n') + '\n';
const body = results.map((r, i) => formatResultExplain(r, i + 1)).join('\n\n') + '\n';
// v0.42.3.0 — prepend the autocut summary when meta carries a decision.
const autocutLine = formatAutocutSummary(meta?.autocut);
return autocutLine ? `${autocutLine}\n\n${body}` : body;
}
/**
+54 -1
View File
@@ -21,6 +21,7 @@ import {
adaptiveReturnEnabled,
type AdaptiveReturnDecision,
} from './return-policy.ts';
import { applyAutocut, type AutocutDecision } from './autocut.ts';
import { loadConfigWithEngine } from '../config.ts';
import { dedupResults } from './dedup.ts';
import { applyReranker } from './rerank.ts';
@@ -663,6 +664,11 @@ export async function hybridSearch(
// override wins over mode bundle. Without this thread the eval gate
// would be a no-op (both branches resolve to the same mode default).
graph_signals: opts?.graph_signals,
// v0.42.3.0 — autocut per-call enable (boolean ceiling override).
// `false` forces the full top-K; per-call wins over config + bundle.
// Non-boolean AutocutInput shapes (Partial) aren't a v1 per-call surface,
// so only the boolean toggle threads here.
autocut: typeof opts?.autocut === 'boolean' ? opts.autocut : undefined,
},
});
@@ -1250,6 +1256,32 @@ export async function hybridSearch(
adaptiveDecision = r.decision;
}
// v0.42.3.0 — autocut (score-discontinuity result-sizing). The floor:
// default-ON in reranked modes (resolvedMode.autocut, resolved per-call >
// config > bundle like every other knob). Cuts the ranked set at the largest
// cross-encoder rerank-score cliff, BEFORE the limit slice, first page only.
// Runs AFTER adaptive-return so an agent-forced intent cap composes (both are
// trim-only with never-empty failsafes). The reranker scored the full
// returned set (mode.ts D4: top_n_in = searchLimit), so there is no un-scored
// tail to wrongly drop; applyAutocut additionally no-ops when <2 items carry
// a finite rerank_score (covers the fail-open reranker path, where
// applyReranker returns RRF order with no scores). minKeep is the fixed
// never-empty failsafe (1); jumpRatio comes from the resolved mode.
let autocutDecision: AutocutDecision | undefined;
if (resolvedMode.autocut && offset === 0) {
const r = applyAutocut(
returnPool,
(x) => x.rerank_score,
{ enabled: true, jumpRatio: resolvedMode.autocut_jump, minKeep: 1 },
// Preserve alias-hop exact matches: applyAliasHop injects the canonical
// page AFTER reranking, so it has no rerank_score. Without this it would
// be dropped whenever autocut cuts on the scored set (Codex P1).
(x) => x.alias_hit === true,
);
returnPool = r.kept;
autocutDecision = r.decision;
}
const sliced = returnPool.slice(offset, offset + limit);
// v0.32.3 search-lite: budget enforcement at the main return path.
// hybridSearchCached used to be the only place this fired; now bare
@@ -1269,6 +1301,7 @@ export async function hybridSearch(
? { token_budget: budgetMeta }
: {}),
...(adaptiveDecision ? { adaptive_return: adaptiveDecision } : {}),
...(autocutDecision ? { autocut: autocutDecision } : {}),
});
return budgeted;
}
@@ -1328,6 +1361,11 @@ export async function hybridSearchCached(
// override would write to one cache row but read from a different
// one on the next call.
graph_signals: opts?.graph_signals,
// v0.42.3.0 — autocut threaded through the cache resolver so the
// knobsHash `ac=` bit reflects the per-call ceiling override. Without
// this, an `autocut:false` (full top-K) call could be served a trimmed
// autocut-on cache row, or vice versa.
autocut: typeof opts?.autocut === 'boolean' ? opts.autocut : undefined,
},
});
// v0.36 (D8 / CDX-2 + codex /ship #4): resolve column for the cache
@@ -1438,6 +1476,13 @@ export async function hybridSearchCached(
similarity: cacheSimilarity,
age_seconds: cacheAge,
},
// Carry the trimmed-set decision fields from the cached row so cache
// HITS report the same autocut/adaptive/mode/column meta as a fresh
// run (Codex P2 — the cached result set was already trimmed).
...(hit.meta?.mode ? { mode: hit.meta.mode } : {}),
...(hit.meta?.embedding_column ? { embedding_column: hit.meta.embedding_column } : {}),
...(hit.meta?.adaptive_return ? { adaptive_return: hit.meta.adaptive_return } : {}),
...(hit.meta?.autocut ? { autocut: hit.meta.autocut } : {}),
...(opts?.tokenBudget && opts.tokenBudget > 0
? { token_budget: budgetMeta }
: {}),
@@ -1470,13 +1515,21 @@ export async function hybridSearchCached(
// Token budget pass (no-op when not set).
const { results: budgeted, meta: budgetMeta } = enforceTokenBudget(results, opts?.tokenBudget);
// Compose the final meta and emit.
// Compose the final meta and emit. v0.42.3.0 (Codex #5): carry over the
// inner meta's decision fields — pre-fix this manual rebuild silently dropped
// adaptive_return (and would drop autocut), so cached writeback/hit paths and
// eval-capture under-reported the feature. Propagate mode + embedding_column
// too (same drop class).
const finalMeta: HybridSearchMeta = {
vector_enabled: innerMeta?.vector_enabled ?? false,
detail_resolved: innerMeta?.detail_resolved ?? null,
expansion_applied: innerMeta?.expansion_applied ?? false,
intent: innerMeta?.intent,
cache: { status: cacheStatus },
...(innerMeta?.mode ? { mode: innerMeta.mode } : {}),
...(innerMeta?.embedding_column ? { embedding_column: innerMeta.embedding_column } : {}),
...(innerMeta?.adaptive_return ? { adaptive_return: innerMeta.adaptive_return } : {}),
...(innerMeta?.autocut ? { autocut: innerMeta.autocut } : {}),
...(opts?.tokenBudget && opts.tokenBudget > 0
? { token_budget: budgetMeta }
: {}),
+85 -3
View File
@@ -242,6 +242,25 @@ export interface ModeBundle {
* regresses post-deploy.
*/
contextual_retrieval_disabled: boolean;
/**
* v0.42.3.0 — autocut (score-discontinuity result-sizing). Default OFF for
* conservative (no reranker → no trustworthy cliff signal; would no-op
* anyway), ON for balanced + tokenmax. When on AND a reranker scored ≥2
* items, hybridSearch cuts the ranked set at the largest cross-encoder
* rerank-score gap (instead of returning the full top-K). No-op without a
* reranker. Override path: per-call SearchOpts.autocut → `search.autocut`
* config → mode bundle. See src/core/search/autocut.ts.
*/
autocut: boolean;
/**
* v0.42.3.0 — autocut sensitivity: the minimum normalized score gap (as a
* fraction of the top score) that counts as a cliff. Default 0.20. Lower =
* cuts more aggressively (tighter sets); higher = only cuts on dramatic
* cliffs. Eval-derived starting point, calibrated by the PrecisionMemBench
* run. Override: `search.autocut_jump` config → mode bundle.
*/
autocut_jump: number;
}
/**
@@ -287,6 +306,10 @@ export const MODE_BUNDLES: Readonly<Record<SearchMode, Readonly<ModeBundle>>> =
// v0.40.3.0 contextual retrieval — none for conservative (minimum surface).
contextual_retrieval: 'none' as CRMode,
contextual_retrieval_disabled: false,
// v0.42.3.0 — autocut OFF: conservative has no reranker, so no trustworthy
// cliff signal exists (autocut would no-op). Explicit for clarity.
autocut: false,
autocut_jump: 0.2,
}),
balanced: Object.freeze({
cache_enabled: true,
@@ -306,7 +329,11 @@ export const MODE_BUNDLES: Readonly<Record<SearchMode, Readonly<ModeBundle>>> =
// `gbrain config set search.reranker.enabled false`.
reranker_enabled: true,
reranker_model: 'zeroentropyai:zerank-2',
reranker_top_n_in: 30,
// v0.42.3.0 D4: topNIn = searchLimit (25) so the cross-encoder scores
// every result the limit slice will return — no unscored tail for autocut
// to wrongly drop (Codex #2). Was 30; tracking searchLimit is the
// correctness precondition for autocut.
reranker_top_n_in: 25,
reranker_top_n_out: null,
reranker_timeout_ms: 5000,
// v0.35.6.0 — undefined for all three bundles; the per-corpus ablation
@@ -334,6 +361,9 @@ export const MODE_BUNDLES: Readonly<Record<SearchMode, Readonly<ModeBundle>>> =
// per the cost-tier philosophy.
contextual_retrieval: 'title' as CRMode,
contextual_retrieval_disabled: false,
// v0.42.3.0 — autocut ON (reranker fires; cliff signal is trustworthy).
autocut: true,
autocut_jump: 0.2,
}),
tokenmax: Object.freeze({
cache_enabled: true,
@@ -350,7 +380,11 @@ export const MODE_BUNDLES: Readonly<Record<SearchMode, Readonly<ModeBundle>>> =
// tier's $700/mo @ Opus pairing per CLAUDE.md cost matrix.
reranker_enabled: true,
reranker_model: 'zeroentropyai:zerank-2',
reranker_top_n_in: 30,
// v0.42.3.0 D4: topNIn = searchLimit (50) so every returned result is
// cross-encoder scored — closes the Codex #2 recall gap where autocut
// would drop the deliberately-preserved un-reranked tail (results 31-50).
// Was 30. Reranking 50 docs vs 30 is cheap vs the downstream LLM.
reranker_top_n_in: 50,
reranker_top_n_out: null,
reranker_timeout_ms: 5000,
// v0.35.6.0 — undefined for all three bundles; the per-corpus ablation
@@ -375,6 +409,9 @@ export const MODE_BUNDLES: Readonly<Record<SearchMode, Readonly<ModeBundle>>> =
// 10K-page brain; documented in the post-upgrade cost prompt.
contextual_retrieval: 'per_chunk_synopsis' as CRMode,
contextual_retrieval_disabled: false,
// v0.42.3.0 — autocut ON.
autocut: true,
autocut_jump: 0.2,
}),
});
@@ -423,6 +460,9 @@ export interface SearchKeyOverrides {
// v0.40.3.0 contextual retrieval. CRMode override + soft kill switch.
contextual_retrieval?: CRMode;
contextual_retrieval_disabled?: boolean;
// v0.42.3.0 — autocut overrides.
autocut?: boolean;
autocut_jump?: number;
}
/**
@@ -463,6 +503,12 @@ export interface SearchPerCallOpts {
// v0.40.3.0 contextual retrieval per-call overrides.
contextual_retrieval?: CRMode;
contextual_retrieval_disabled?: boolean;
// v0.42.3.0 — autocut per-call overrides. NOTE: the boolean per-call
// autocut toggle from SearchOpts is handled at the hybrid.ts boundary
// (it's an AutocutInput, not a plain bool here); autocut_jump is the
// numeric per-call knob threaded through the bundle.
autocut?: boolean;
autocut_jump?: number;
}
/**
@@ -552,6 +598,9 @@ export function resolveSearchMode(input: ResolveSearchModeInput): ResolvedSearch
// v0.40.3.0 contextual retrieval — resolved via the same pick chain.
contextual_retrieval: pick('contextual_retrieval'),
contextual_retrieval_disabled: pick('contextual_retrieval_disabled'),
// v0.42.3.0 — autocut resolved via the same pick chain.
autocut: pick('autocut'),
autocut_jump: pick('autocut_jump'),
resolved_mode,
mode_valid: valid,
};
@@ -642,7 +691,14 @@ export function attributeKnob<K extends keyof ModeBundle>(
// stage that multiplies title-phrase-matching results. A title-boost-on write
// must NOT be served to a title-boost-off lookup (ranking shifts). Same
// one-time miss-spike pattern; fills within cache.ttl_seconds.
export const KNOBS_HASH_VERSION = 7;
//
// v0.42.3.0 bump 7→8: autocut (score-discontinuity result-sizing) adds `ac`
// + `acj` parts. Default-ON in reranked modes trims the returned set, so an
// autocut-on write must NOT be served to an autocut-off lookup. ONE-TIME
// global cache cold-miss on upgrade — EVERY query_cache row invalidates,
// including conservative/no-reranker calls where autocut is a no-op (the hash
// is global, not per-mode). Refills within cache.ttl_seconds (3600s default).
export const KNOBS_HASH_VERSION = 8;
/**
* v0.36 (D8 / CDX-2) — second-arg context for the cache key. The
@@ -740,6 +796,16 @@ export function knobsHash(
`crd=${knobs.contextual_retrieval_disabled ? 1 : 0}`,
// v=7 addition (append-only) — T2 title-phrase boost (retrieval-maxpool).
`tib=${knobs.title_boost === undefined ? 'none' : knobs.title_boost.toFixed(4)}`,
// v=8 additions (v0.42.3.0, append-only): autocut. An autocut-on write
// (trimmed result set) must not be served to an autocut-off lookup, and a
// sensitivity change (jumpRatio) shifts where the cut lands. Conservative
// (autocut off) hashes differently from balanced/tokenmax (autocut on),
// which is correct — the result sets differ.
`ac=${knobs.autocut ? 1 : 0}`,
// `?? 0.2` mirrors the module's defensive read of other knobs (graph_signals
// etc.) so a partial-knobs caller (tests passing a minimal literal) can't
// crash the hash. Typed callers always carry the field.
`acj=${(knobs.autocut_jump ?? 0.2).toFixed(2)}`,
];
const h = createHash('sha256');
h.update(parts.join('|'));
@@ -894,6 +960,19 @@ export function loadOverridesFromConfig(
out.graph_signals = gs === '1' || gs.toLowerCase() === 'true';
}
// v0.42.3.0 — autocut. `search.autocut` is the master toggle (the ceiling
// override agents use to force the full top-K); `search.autocut_jump` tunes
// sensitivity (clamped to (0, 1] — out-of-range falls through to the bundle).
const ac = get('search.autocut');
if (ac !== undefined) {
out.autocut = ac === '1' || ac.toLowerCase() === 'true';
}
const acj = get('search.autocut_jump');
if (acj !== undefined) {
const n = parseFloat(acj);
if (Number.isFinite(n) && n > 0 && n <= 1) out.autocut_jump = n;
}
return out;
}
@@ -930,6 +1009,9 @@ export const SEARCH_MODE_CONFIG_KEYS: ReadonlyArray<string> = Object.freeze([
// override at the per-key level without flipping the global mode.
'search.contextual_retrieval',
'search.contextual_retrieval_disabled',
// v0.42.3.0 autocut
'search.autocut',
'search.autocut_jump',
]);
/**
+2 -2
View File
@@ -114,9 +114,9 @@ export async function applyReranker(
seen.add(r.index);
const item = head[r.index]!;
// Stamp the reranker score onto the result so downstream callers
// (telemetry, debug) can see the new ordering signal. Doesn't
// (telemetry, debug, autocut) can see the new ordering signal. Doesn't
// replace `score` — that's RRF and other consumers may depend on it.
(item as any).rerank_score = r.relevanceScore;
item.rerank_score = r.relevanceScore;
// v0.40.4 attribution stamp (D12=A) — rank delta. Positive means
// rank improved (moved closer to top). new_index is the next
// push position in reorderedHead; original index was r.index.
+19
View File
@@ -645,6 +645,11 @@ export interface SearchResult {
* Undefined when no reranker fired. The raw reranker relevance score
* is separately stamped as `rerank_score` for back-compat. */
reranker_delta?: number;
/** Raw cross-encoder relevance score stamped by applyReranker on the
* reranked head (undefined when no reranker fired). Distinct from `score`
* (RRF + boosts). v0.42.3.0 autocut cuts on this — the trustworthy
* separatrix — never on RRF/cosine. */
rerank_score?: number;
/**
* v0.42 (T19, plan D6) — multiplier applied by applyAliasResolvedBoost
* (1.0 = unchanged; default 1.05x). Fires when the result's slug is
@@ -763,6 +768,14 @@ export interface SearchOpts {
* See src/core/search/return-policy.ts.
*/
adaptiveReturn?: import('./search/return-policy.ts').AdaptiveReturnInput;
/**
* v0.42.3.0 — autocut (score-discontinuity result-sizing). Default-ON in
* reranked modes (the floor). Pass `false` to force the full top-K for breadth
* / exploration (the ceiling override). Cuts the ranked set at the largest
* cross-encoder rerank-score cliff; no-op without a reranker. Only fires when
* offset===0. See src/core/search/autocut.ts.
*/
autocut?: import('./search/autocut.ts').AutocutInput;
type?: PageType;
/**
* v0.33: multi-type filter. When set, search results are filtered to
@@ -1298,6 +1311,12 @@ export interface HybridSearchMeta {
* Omitted when the gate is off. Surfaced for `gbrain search --explain`.
*/
adaptive_return?: import('./search/return-policy.ts').AdaptiveReturnDecision;
/**
* v0.42.3.0 — autocut decision (signal, cut point, kept/total, gapRatio).
* Omitted when autocut didn't run (no reranker). Surfaced for
* `gbrain search --explain`.
*/
autocut?: import('./search/autocut.ts').AutocutDecision;
/**
* v0.32.x (search-lite): token budget enforcement metadata. Omitted when
* no budget was applied (backward-compatible with pre-search-lite
+1 -1
View File
@@ -142,7 +142,7 @@ describe('D2 — knobsHash differs across cross-modal knob values', () => {
// v0.39 T21 (master) bump to v=4 for graph_signals + schema-pack fields.
// v0.40.3.0 D8 bumps to v=5 (sequenced behind salem's v=4 graph-signals).
// v0.41.22.0 (type-unification): 5→6 for alias_resolved post-fusion boost.
expect(KNOBS_HASH_VERSION).toBe(7);
expect(KNOBS_HASH_VERSION).toBe(8);
});
test('flipping unified_multimodal changes the hash', () => {
+1 -1
View File
@@ -90,6 +90,6 @@ describe('alias_resolved boost stage', () => {
describe('KNOBS_HASH_VERSION', () => {
it('bumped to 6 to invalidate caches across v0.42 boost stage addition', () => {
expect(KNOBS_HASH_VERSION).toBe(7);
expect(KNOBS_HASH_VERSION).toBe(8);
});
});
+84 -6
View File
@@ -74,6 +74,9 @@ describe('SEARCH_MODES + MODE_BUNDLES canonical shape', () => {
graph_signals: false,
...CR_DISABLED_DEFAULT,
contextual_retrieval: 'none',
// v0.42.3.0 — autocut OFF for conservative (no reranker).
autocut: false,
autocut_jump: 0.2,
});
});
@@ -90,7 +93,8 @@ describe('SEARCH_MODES + MODE_BUNDLES canonical shape', () => {
searchLimit: 25,
reranker_enabled: true,
reranker_model: 'zeroentropyai:zerank-2',
reranker_top_n_in: 30,
// v0.42.3.0 D4: topNIn = searchLimit (25), was 30.
reranker_top_n_in: 25,
reranker_top_n_out: null,
reranker_timeout_ms: 5000,
floor_ratio: undefined,
@@ -99,6 +103,9 @@ describe('SEARCH_MODES + MODE_BUNDLES canonical shape', () => {
graph_signals: true,
...CR_DISABLED_DEFAULT,
contextual_retrieval: 'title',
// v0.42.3.0 — autocut ON.
autocut: true,
autocut_jump: 0.2,
});
});
@@ -113,7 +120,8 @@ describe('SEARCH_MODES + MODE_BUNDLES canonical shape', () => {
searchLimit: 50,
reranker_enabled: true,
reranker_model: 'zeroentropyai:zerank-2',
reranker_top_n_in: 30,
// v0.42.3.0 D4: topNIn = searchLimit (50), was 30.
reranker_top_n_in: 50,
reranker_top_n_out: null,
reranker_timeout_ms: 5000,
floor_ratio: undefined,
@@ -122,6 +130,9 @@ describe('SEARCH_MODES + MODE_BUNDLES canonical shape', () => {
graph_signals: true,
...CR_DISABLED_DEFAULT,
contextual_retrieval: 'per_chunk_synopsis',
// v0.42.3.0 — autocut ON.
autocut: true,
autocut_jump: 0.2,
});
});
@@ -376,10 +387,11 @@ describe('knobsHash determinism + cross-mode separation (CDX-4)', () => {
// written when the brain was on balanced (title-only) — different
// embedding spaces. Sequenced behind salem's v=4 graph-signals work.
// v0.41.22.0 (type-unification): bumped 5→6 for the new alias_resolved
// post-fusion boost stage. A query against a brain with slug_aliases
// populated must not be served from a cache row written before the
// boost stage existed.
expect(KNOBS_HASH_VERSION).toBe(7);
// post-fusion boost stage. T2: bumped 6→7 for title_boost. v0.42.3.0:
// bumped 7→8 for autocut (ac=/acj=). A query against a brain with
// slug_aliases populated must not be served from a cache row written
// before the boost stage existed.
expect(KNOBS_HASH_VERSION).toBe(8);
});
test('T1 (codex): floor_ratio set vs unset produces DIFFERENT hashes (cache contamination prevention)', () => {
@@ -542,3 +554,69 @@ describe('v0.40.4 — graph_signals knob', () => {
expect(attr.value).toBe(true);
});
});
describe('v0.42.3.0 — autocut knobs', () => {
test('KNOBS_HASH_VERSION bumped to 7', () => {
expect(KNOBS_HASH_VERSION).toBe(8);
});
test('bundle defaults: conservative off, balanced/tokenmax on @0.20', () => {
expect(MODE_BUNDLES.conservative.autocut).toBe(false);
expect(MODE_BUNDLES.balanced.autocut).toBe(true);
expect(MODE_BUNDLES.tokenmax.autocut).toBe(true);
for (const m of ['conservative', 'balanced', 'tokenmax'] as const) {
expect(MODE_BUNDLES[m].autocut_jump).toBe(0.2);
}
});
test('D4: reranked modes set top_n_in = searchLimit (no unscored tail)', () => {
expect(MODE_BUNDLES.balanced.reranker_top_n_in).toBe(MODE_BUNDLES.balanced.searchLimit);
expect(MODE_BUNDLES.tokenmax.reranker_top_n_in).toBe(MODE_BUNDLES.tokenmax.searchLimit);
expect(MODE_BUNDLES.balanced.reranker_top_n_in).toBe(25);
expect(MODE_BUNDLES.tokenmax.reranker_top_n_in).toBe(50);
});
test('resolveSearchMode threads autocut: per-call > config > bundle', () => {
// per-call wins
expect(resolveSearchMode({ mode: 'balanced', perCall: { autocut: false } }).autocut).toBe(false);
// config override wins over bundle
expect(resolveSearchMode({ mode: 'balanced', overrides: { autocut: false } }).autocut).toBe(false);
// per-call beats config
expect(
resolveSearchMode({ mode: 'balanced', overrides: { autocut: false }, perCall: { autocut: true } }).autocut,
).toBe(true);
// jump knob threads too
expect(resolveSearchMode({ mode: 'balanced', perCall: { autocut_jump: 0.5 } }).autocut_jump).toBe(0.5);
});
test('loadOverridesFromConfig reads search.autocut + search.autocut_jump', () => {
const ov = loadOverridesFromConfig({ 'search.autocut': 'false', 'search.autocut_jump': '0.35' });
expect(ov.autocut).toBe(false);
expect(ov.autocut_jump).toBe(0.35);
});
test('SEARCH_MODE_CONFIG_KEYS includes the autocut keys', () => {
expect(SEARCH_MODE_CONFIG_KEYS).toContain('search.autocut');
expect(SEARCH_MODE_CONFIG_KEYS).toContain('search.autocut_jump');
});
test('knobsHash includes ac= / acj= — autocut-on vs off differ', () => {
const on = knobsHash(resolveSearchMode({ mode: 'balanced' })); // autocut true
const off = knobsHash(resolveSearchMode({ mode: 'balanced', perCall: { autocut: false } }));
expect(on).not.toBe(off);
});
test('knobsHash differs on jump sensitivity', () => {
const a = knobsHash(resolveSearchMode({ mode: 'balanced' }));
const b = knobsHash(resolveSearchMode({ mode: 'balanced', perCall: { autocut_jump: 0.5 } }));
expect(a).not.toBe(b);
});
test('attributeKnob reports autocut source', () => {
const input = { mode: 'balanced', perCall: { autocut: false } };
const resolved = resolveSearchMode(input);
const attr = attributeKnob('autocut', input, resolved);
expect(attr.source).toBe('per-call');
expect(attr.value).toBe(false);
});
});
+197
View File
@@ -0,0 +1,197 @@
/**
* v0.42.3.0 — autocut precision/recall eval gate (in-repo, hermetic).
*
* This is the D2 precondition for default-ON, runnable in CI without the
* sibling gbrain-evals repo and without any API key. It measures the EXACT
* claim default-ON rests on: does cutting at the rerank-score cliff lift
* precision WITHOUT regressing recall — especially on enumeration queries?
*
* WHAT IT MODELS (and what it does NOT): autocut cuts on cross-encoder
* rerank scores. This gate feeds applyAutocut labeled qrels fixtures whose
* per-candidate `rerank_score` follows realistic cross-encoder distributions
* (clean cliffs for single-answer queries, a high cluster + cliff for
* enumeration, flat curves for ambiguous breadth, and an adversarial case
* where the reranker mis-scores a relevant doc below a cliff). It measures
* the precision/recall tradeoff of the CUT DECISION on those distributions.
* It does NOT claim that ZeroEntropy's live scores look like these fixtures
* on a specific brain — that empirical confirmation is the optional
* gbrain-evals PrecisionMemBench run. What this gate DOES guarantee, in CI:
* - autocut lifts mean precision well above the no-autocut baseline,
* - it does NOT regress recall below a floor,
* - it NEVER regresses recall on enumeration/flat queries (structural:
* a flat curve has no cliff, so autocut declines → identical recall).
*
* The gate fails (correctly) if someone over-tunes autocut (e.g. drops
* jumpRatio so low it cuts into clusters) or if the cut math regresses.
* Floors are env-overridable so an intentional ranking change edits the
* threshold with a documented reason.
*/
import { describe, expect, test } from 'bun:test';
import { applyAutocut, DEFAULT_AUTOCUT } from '../../src/core/search/autocut.ts';
type Candidate = { rerank_score: number; relevant: boolean };
type EvalQuery = {
id: string;
kind: 'single' | 'cluster' | 'enumeration' | 'adversarial';
/** Candidates in reranked (descending-score) order. */
candidates: Candidate[];
};
const scoreOf = (c: Candidate) => c.rerank_score;
const LIMIT = 10; // mirrors a typical returned-set cap; doesn't bind these lists
// Realistic cross-encoder distributions. Proportions reflect the empirical
// reality return-policy.ts cites: rank-1 is correct in ~94% of single-answer
// cases, so clean cliffs dominate and adversarial mis-scores are rare.
const FIXTURE: EvalQuery[] = [
// 5 single-answer queries with a clean cliff after the one right answer.
...Array.from({ length: 5 }, (_, i): EvalQuery => ({
id: `single-${i}`,
kind: 'single',
candidates: [
{ rerank_score: 0.95, relevant: true },
{ rerank_score: 0.30, relevant: false },
{ rerank_score: 0.25, relevant: false },
{ rerank_score: 0.20, relevant: false },
{ rerank_score: 0.15, relevant: false },
{ rerank_score: 0.10, relevant: false },
{ rerank_score: 0.08, relevant: false },
{ rerank_score: 0.05, relevant: false },
],
})),
// 2 cluster queries: a tight relevant cluster, then a cliff to noise.
...Array.from({ length: 2 }, (_, i): EvalQuery => ({
id: `cluster-${i}`,
kind: 'cluster',
candidates: [
{ rerank_score: 0.90, relevant: true },
{ rerank_score: 0.88, relevant: true },
{ rerank_score: 0.85, relevant: true },
{ rerank_score: 0.25, relevant: false },
{ rerank_score: 0.20, relevant: false },
{ rerank_score: 0.15, relevant: false },
{ rerank_score: 0.10, relevant: false },
],
})),
// 2 enumeration/broad queries: flat curve, many relevant, NO cliff.
// Autocut must DECLINE here — this is the recall-regression risk D2 gates.
...Array.from({ length: 2 }, (_, i): EvalQuery => ({
id: `enumeration-${i}`,
kind: 'enumeration',
candidates: [
{ rerank_score: 0.60, relevant: true },
{ rerank_score: 0.58, relevant: true },
{ rerank_score: 0.56, relevant: true },
{ rerank_score: 0.54, relevant: true },
{ rerank_score: 0.52, relevant: true },
{ rerank_score: 0.50, relevant: false },
{ rerank_score: 0.48, relevant: false },
],
})),
// 1 adversarial query: the reranker mis-scores a relevant doc BELOW an
// early cliff. Autocut will drop it — this models reranker error, the only
// way autocut can hurt recall. Kept rare to match real distributions.
{
id: 'adversarial-0',
kind: 'adversarial',
candidates: [
{ rerank_score: 0.90, relevant: true },
{ rerank_score: 0.50, relevant: true }, // relevant but mis-scored below the cliff
{ rerank_score: 0.48, relevant: false },
{ rerank_score: 0.46, relevant: false },
{ rerank_score: 0.20, relevant: false },
],
},
];
function precisionRecall(kept: Candidate[], totalRelevant: number): { p: number; r: number } {
const relevantKept = kept.filter((c) => c.relevant).length;
const p = kept.length === 0 ? 0 : relevantKept / kept.length;
const r = totalRelevant === 0 ? 1 : relevantKept / totalRelevant;
return { p, r };
}
function evalQuery(q: EvalQuery, autocutOn: boolean) {
const totalRelevant = q.candidates.filter((c) => c.relevant).length;
const pool = autocutOn
? applyAutocut(q.candidates, scoreOf, { ...DEFAULT_AUTOCUT }).kept
: q.candidates;
const kept = pool.slice(0, LIMIT);
return precisionRecall(kept, totalRelevant);
}
function mean(xs: number[]): number {
return xs.length === 0 ? 0 : xs.reduce((a, b) => a + b, 0) / xs.length;
}
function envFloor(name: string, fallback: number): number {
const v = process.env[name];
if (v === undefined) return fallback;
const n = parseFloat(v);
return Number.isFinite(n) ? n : fallback;
}
describe('autocut eval gate (D2 — precision lift without recall regression)', () => {
const off = FIXTURE.map((q) => evalQuery(q, false));
const on = FIXTURE.map((q) => evalQuery(q, true));
const precisionOff = mean(off.map((x) => x.p));
const precisionOn = mean(on.map((x) => x.p));
const recallOff = mean(off.map((x) => x.r));
const recallOn = mean(on.map((x) => x.r));
// Surface the numbers (the CHANGELOG/eval record reads these).
// eslint-disable-next-line no-console
console.error(
`[autocut-eval] precision ${precisionOff.toFixed(3)}${precisionOn.toFixed(3)} ` +
`(+${(precisionOn - precisionOff).toFixed(3)}) | recall ${recallOff.toFixed(3)}` +
`${recallOn.toFixed(3)} (${(recallOn - recallOff).toFixed(3)})`,
);
// Floors are env-overridable (document the reason in the commit when you move them).
const LIFT_FLOOR = envFloor('GBRAIN_AUTOCUT_EVAL_PRECISION_LIFT_FLOOR', 0.15);
const RECALL_FLOOR = envFloor('GBRAIN_AUTOCUT_EVAL_RECALL_FLOOR', 0.9);
const RECALL_REGRESSION_TOLERANCE = envFloor('GBRAIN_AUTOCUT_EVAL_RECALL_TOLERANCE', 0.1);
test('autocut lifts mean precision well above baseline', () => {
expect(precisionOn - precisionOff).toBeGreaterThanOrEqual(LIFT_FLOOR);
});
test('autocut keeps mean recall above the floor', () => {
expect(recallOn).toBeGreaterThanOrEqual(RECALL_FLOOR);
});
test('recall regression is bounded', () => {
expect(recallOff - recallOn).toBeLessThanOrEqual(RECALL_REGRESSION_TOLERANCE);
});
test('ZERO recall regression on enumeration/flat queries (the D2 core concern)', () => {
// Where recall matters most — broad enumeration with no cliff — autocut
// must decline and return the full set, so recall is identical on/off.
const enums = FIXTURE.filter((q) => q.kind === 'enumeration');
expect(enums.length).toBeGreaterThan(0);
for (const q of enums) {
const offR = evalQuery(q, false).r;
const onR = evalQuery(q, true).r;
expect(onR).toBe(offR);
}
});
test('single-answer queries: precision goes to 1.0 with recall preserved', () => {
const singles = FIXTURE.filter((q) => q.kind === 'single');
for (const q of singles) {
const onPR = evalQuery(q, true);
expect(onPR.p).toBe(1); // only the right answer returned
expect(onPR.r).toBe(1); // and it IS the right answer
}
});
test('cluster queries: the whole relevant cluster survives (no over-cut)', () => {
const clusters = FIXTURE.filter((q) => q.kind === 'cluster');
for (const q of clusters) {
// recall stays 1.0 — autocut cuts AFTER the cluster, not into it.
expect(evalQuery(q, true).r).toBe(1);
}
});
});
@@ -0,0 +1,169 @@
/**
* v0.42.3.0 — autocut end-to-end through hybridSearch (the IRON-RULE
* behavioral regression).
*
* Drives bare hybridSearch against PGLite with a stubbed rerankerFn so we
* pin the behavior the pure-fn tests can't:
* - A cliff-shaped rerank → autocut trims the result set at the cliff.
* - A flat rerank → autocut declines (full set returned).
* - Reranker disabled → autocut is a no-op (no rerank scores to cut on),
* which is the load-bearing gate (no trustworthy signal without a reranker).
* - Per-call autocut:false forces the full top-K even with a cliff (ceiling).
* - Composes with adaptive-return without violating the never-empty floor.
*
* Serial because it mutates gateway global state (configureGateway +
* __setEmbedTransportForTests). No API keys; embedding + reranker stubbed.
*/
import { afterAll, beforeAll, describe, expect, test } from 'bun:test';
import { PGLiteEngine } from '../../src/core/pglite-engine.ts';
import { hybridSearch } from '../../src/core/search/hybrid.ts';
import {
configureGateway,
resetGateway,
__setEmbedTransportForTests,
} from '../../src/core/ai/gateway.ts';
import type { PageInput, SearchOpts } from '../../src/core/types.ts';
import type { RerankInput, RerankResult } from '../../src/core/ai/gateway.ts';
let engine: PGLiteEngine;
const DIMS = 1536;
const FAKE_EMB = Array.from({ length: DIMS }, (_, j) => (j === 0 ? 1 : 0.01));
beforeAll(async () => {
engine = new PGLiteEngine();
await engine.connect({});
await engine.initSchema();
// Seed 5 pages sharing a keyword so the candidate pool is 5 deep.
const pages: Array<[string, PageInput, string]> = [
['notes/a', { type: 'note', title: 'A', compiled_truth: 'alpha keyword one' }, 'alpha keyword one chunk'],
['notes/b', { type: 'note', title: 'B', compiled_truth: 'alpha keyword two' }, 'alpha keyword two chunk'],
['notes/c', { type: 'note', title: 'C', compiled_truth: 'alpha keyword three' }, 'alpha keyword three chunk'],
['notes/d', { type: 'note', title: 'D', compiled_truth: 'alpha keyword four' }, 'alpha keyword four chunk'],
['notes/e', { type: 'note', title: 'E', compiled_truth: 'alpha keyword five' }, 'alpha keyword five chunk'],
];
for (const [slug, page, chunkText] of pages) {
await engine.putPage(slug, page);
await engine.upsertChunks(slug, [
{ chunk_index: 0, chunk_text: chunkText, chunk_source: 'compiled_truth' },
]);
}
configureGateway({
embedding_model: 'openai:text-embedding-3-large',
embedding_dimensions: DIMS,
env: { OPENAI_API_KEY: 'sk-test' },
});
__setEmbedTransportForTests(async (args: any) => ({
embeddings: args.values.map(() => FAKE_EMB),
}) as any);
});
afterAll(async () => {
__setEmbedTransportForTests(null);
resetGateway();
await engine.disconnect();
});
// A reranker that assigns descending scores from a fixed array (by index).
function rerankerWithScores(scores: number[]) {
return async (input: RerankInput): Promise<RerankResult[]> =>
input.documents.map((_, i) => ({ index: i, relevanceScore: scores[i] ?? 0.01 }));
}
// balanced mode (the default) has autocut ON. We pass opts.reranker to stub
// the cross-encoder; resolvedMode.autocut stays true (no search.mode config).
function rerankerOpts(scores: number[]): SearchOpts['reranker'] {
return {
enabled: true,
topNIn: 30,
topNOut: null,
rerankerFn: rerankerWithScores(scores),
};
}
describe('autocut — fires on a real cliff', () => {
test('cliff after rank 2 → result set trimmed to 2', async () => {
const out = await hybridSearch(engine, 'alpha keyword', {
limit: 10,
reranker: rerankerOpts([0.95, 0.9, 0.2, 0.15, 0.1]),
});
expect(out.length).toBe(2);
expect(out.map((r) => r.rerank_score)).toEqual([0.95, 0.9]);
});
test('cliff after rank 1 → single obvious answer', async () => {
const out = await hybridSearch(engine, 'alpha keyword', {
limit: 10,
reranker: rerankerOpts([0.98, 0.12, 0.1, 0.08, 0.05]),
});
expect(out.length).toBe(1);
});
});
describe('autocut — declines on a flat curve', () => {
test('flat rerank scores → full set returned (no trim)', async () => {
const baseline = await hybridSearch(engine, 'alpha keyword', { limit: 10 });
const out = await hybridSearch(engine, 'alpha keyword', {
limit: 10,
reranker: rerankerOpts([0.9, 0.88, 0.86, 0.84, 0.82]),
});
expect(out.length).toBe(baseline.length);
expect(out.length).toBeGreaterThanOrEqual(3); // meaningful pool to NOT trim
});
});
describe('autocut — no-op without a reranker (the load-bearing gate)', () => {
test('reranker disabled → no trim even though autocut is on for the mode', async () => {
const baseline = await hybridSearch(engine, 'alpha keyword', { limit: 10 });
const out = await hybridSearch(engine, 'alpha keyword', {
limit: 10,
reranker: { enabled: false, topNIn: 30, topNOut: null, rerankerFn: rerankerWithScores([0.95, 0.1]) },
});
// No rerank scores were stamped → autocut sees <2 finite scores → no-op.
expect(out.length).toBe(baseline.length);
});
test('reranker fails open (throws) → no trim (fail-open + autocut no-op)', async () => {
const baseline = await hybridSearch(engine, 'alpha keyword', { limit: 10 });
const out = await hybridSearch(engine, 'alpha keyword', {
limit: 10,
reranker: {
enabled: true,
topNIn: 30,
topNOut: null,
rerankerFn: async () => {
throw new Error('upstream down');
},
},
});
expect(out.map((r) => r.slug)).toEqual(baseline.map((r) => r.slug));
});
});
describe('autocut — ceiling override', () => {
test('per-call autocut:false forces full top-K even with a cliff', async () => {
const baseline = await hybridSearch(engine, 'alpha keyword', { limit: 10 });
const out = await hybridSearch(engine, 'alpha keyword', {
limit: 10,
autocut: false,
reranker: rerankerOpts([0.95, 0.9, 0.2, 0.15, 0.1]),
});
// Cliff present, but the override keeps the full reranked set.
expect(out.length).toBe(baseline.length);
});
});
describe('autocut — composes with adaptive-return (never-empty holds)', () => {
test('adaptive-return + autocut both on → non-empty, bounded', async () => {
const out = await hybridSearch(engine, 'alpha keyword', {
limit: 10,
adaptiveReturn: true,
reranker: rerankerOpts([0.95, 0.9, 0.2, 0.15, 0.1]),
});
expect(out.length).toBeGreaterThanOrEqual(1);
expect(out.length).toBeLessThanOrEqual(2); // cliff caps at 2; adaptive may cap further
});
});
+232
View File
@@ -0,0 +1,232 @@
/**
* v0.42.3.0 — autocut pure-function tests.
*
* Pins the score-discontinuity algorithm + the resolve ladder. No engine,
* no network — applyAutocut takes a list + a scoreOf accessor.
*/
import { describe, expect, test } from 'bun:test';
import {
DEFAULT_AUTOCUT,
autocutFromConfig,
resolveAutocut,
applyAutocut,
type AutocutConfig,
} from '../../src/core/search/autocut.ts';
// A tiny result shape: just a score and an id so we can assert which
// items survived the cut.
type R = { id: string; rs?: number };
const scoreOf = (r: R) => r.rs;
const ON: AutocutConfig = { enabled: true, jumpRatio: 0.2, minKeep: 1 };
function mk(scores: Array<number | undefined>): R[] {
return scores.map((rs, i) => ({ id: `r${i}`, rs }));
}
describe('DEFAULT_AUTOCUT', () => {
test('module default is enabled with 0.20 jump + minKeep 1', () => {
expect(DEFAULT_AUTOCUT.enabled).toBe(true);
expect(DEFAULT_AUTOCUT.jumpRatio).toBe(0.2);
expect(DEFAULT_AUTOCUT.minKeep).toBe(1);
});
test('is frozen', () => {
expect(Object.isFrozen(DEFAULT_AUTOCUT)).toBe(true);
});
});
describe('applyAutocut — cuts on a real cliff', () => {
test('clear cliff after rank 2 → keeps 2', () => {
// 1.0, 0.944, 0.222, 0.111 → biggest gap 0.85→0.2 (0.722) clears 0.20.
const r = applyAutocut(mk([0.9, 0.85, 0.2, 0.1]), scoreOf, ON);
expect(r.kept.map((x) => x.id)).toEqual(['r0', 'r1']);
expect(r.decision.applied).toBe(true);
expect(r.decision.signal).toBe('rerank');
expect(r.decision.kept).toBe(2);
expect(r.decision.total).toBe(4);
expect(r.decision.gapRatio).toBeGreaterThan(0.2);
});
test('cliff after rank 1 → keeps the single obvious answer', () => {
const r = applyAutocut(mk([0.95, 0.1, 0.08, 0.05]), scoreOf, ON);
expect(r.kept.map((x) => x.id)).toEqual(['r0']);
expect(r.decision.applied).toBe(true);
});
test('preserves original order among kept items (robust to unsorted input)', () => {
// Provider returned them out of order; autocut finds the cliff on a
// sorted copy and keeps items >= threshold IN INPUT ORDER.
const items = mk([0.85, 0.95, 0.12, 0.9]); // sorted desc: .95 .9 .85 .12
const r = applyAutocut(items, scoreOf, ON);
// Cliff is between .85 and .12 → threshold .85 → keep .85,.95,.9 → r0,r1,r3.
expect(r.kept.map((x) => x.id)).toEqual(['r0', 'r1', 'r3']);
});
});
describe('applyAutocut — declines to cut', () => {
test('flat scores (no cliff) → returns all, signal none', () => {
const r = applyAutocut(mk([0.9, 0.88, 0.86, 0.84]), scoreOf, ON);
expect(r.kept.length).toBe(4);
expect(r.decision.applied).toBe(false);
expect(r.decision.signal).toBe('none');
});
test('all-equal scores → no cut', () => {
const r = applyAutocut(mk([0.7, 0.7, 0.7, 0.7]), scoreOf, ON);
expect(r.kept.length).toBe(4);
expect(r.decision.applied).toBe(false);
});
test('the measured-RRF-flat case: rank1≈rank2 gap → no cut', () => {
// Mirrors return-policy.ts's documented finding: a ~identical top gap is
// NOT a separatrix. With these (correct vs wrong) shapes autocut stays out.
const correct = applyAutocut(mk([0.602, 0.569, 0.55, 0.54]), scoreOf, ON);
const wrong = applyAutocut(mk([0.569, 0.55, 0.54, 0.53]), scoreOf, ON);
expect(correct.decision.applied).toBe(false);
expect(wrong.decision.applied).toBe(false);
});
});
describe('applyAutocut — no-op guards', () => {
test('disabled → returns input unchanged', () => {
const items = mk([0.9, 0.1]);
const r = applyAutocut(items, scoreOf, { ...ON, enabled: false });
expect(r.kept).toBe(items);
expect(r.decision.applied).toBe(false);
});
test('empty input → no-op', () => {
const r = applyAutocut([] as R[], scoreOf, ON);
expect(r.kept).toEqual([]);
expect(r.decision.applied).toBe(false);
});
test('single item → no-op (no cliff possible)', () => {
const items = mk([0.9]);
const r = applyAutocut(items, scoreOf, ON);
expect(r.kept.length).toBe(1);
expect(r.decision.applied).toBe(false);
});
test('<2 finite scores → no-op (fail-open reranker: no scores stamped)', () => {
// All un-scored (reranker failed open → RRF order, no rerank_score).
const items = mk([undefined, undefined, undefined]);
const r = applyAutocut(items, scoreOf, ON);
expect(r.kept.length).toBe(3);
expect(r.decision.signal).toBe('none');
});
test('exactly 1 finite score among many → no-op', () => {
const items = mk([0.9, undefined, undefined]);
const r = applyAutocut(items, scoreOf, ON);
expect(r.kept.length).toBe(3);
expect(r.decision.applied).toBe(false);
});
test('top score <= 0 → no-op (score scale unusable)', () => {
const r = applyAutocut(mk([0, -0.1, -0.5]), scoreOf, ON);
expect(r.decision.applied).toBe(false);
});
test('non-finite scores are ignored', () => {
const items = mk([0.9, Number.NaN, 0.1]);
const r = applyAutocut(items, scoreOf, ON);
// Only 0.9 and 0.1 are finite → 2 scored, cliff → keep r0; NaN item dropped.
expect(r.kept.map((x) => x.id)).toEqual(['r0']);
});
});
describe('applyAutocut — failsafe', () => {
test('never returns empty when input is non-empty', () => {
const r = applyAutocut(mk([0.9, 0.01]), scoreOf, ON);
expect(r.kept.length).toBeGreaterThanOrEqual(1);
});
test('minKeep floor holds even with a cliff after rank 1', () => {
// Cliff says keep 1, but minKeep=2 expands to 2.
const r = applyAutocut(mk([0.95, 0.1, 0.08]), scoreOf, { ...ON, minKeep: 2 });
expect(r.kept.length).toBeGreaterThanOrEqual(2);
});
test('higher jumpRatio means only dramatic cliffs cut', () => {
const scores = mk([0.9, 0.6, 0.5, 0.4]); // top gap normalized ~0.33
const lenient = applyAutocut(scores, scoreOf, { ...ON, jumpRatio: 0.2 });
const strict = applyAutocut(scores, scoreOf, { ...ON, jumpRatio: 0.5 });
expect(lenient.decision.applied).toBe(true);
expect(strict.decision.applied).toBe(false);
});
});
describe('applyAutocut — preserve predicate (Codex P1: alias-injected matches)', () => {
// An alias-hop exact match is injected AFTER reranking, so it has no score.
type AR = { id: string; rs?: number; alias?: boolean };
const arScore = (r: AR) => r.rs;
const isAlias = (r: AR) => r.alias === true;
test('unscored alias item survives a cut that drops scored noise', () => {
const items: AR[] = [
{ id: 'alias', alias: true }, // injected, no rerank_score
{ id: 'top', rs: 0.95 },
{ id: 'noise1', rs: 0.2 },
{ id: 'noise2', rs: 0.1 },
];
const r = applyAutocut(items, arScore, ON, isAlias);
// Cliff after 'top' drops noise1/noise2; 'alias' is preserved despite no score.
expect(r.kept.map((x) => x.id).sort()).toEqual(['alias', 'top']);
expect(r.decision.applied).toBe(true);
});
test('without the predicate, the unscored alias item is dropped on a cut', () => {
const items: AR[] = [
{ id: 'alias', alias: true },
{ id: 'top', rs: 0.95 },
{ id: 'noise', rs: 0.1 },
];
const r = applyAutocut(items, arScore, ON); // no preserve
expect(r.kept.map((x) => x.id)).toEqual(['top']);
});
test('preserve does not force a cut on a flat curve (no-op still returns all)', () => {
const items: AR[] = [
{ id: 'alias', alias: true },
{ id: 'a', rs: 0.6 },
{ id: 'b', rs: 0.58 },
];
const r = applyAutocut(items, arScore, ON, isAlias);
expect(r.decision.applied).toBe(false);
expect(r.kept.length).toBe(3);
});
});
describe('autocutFromConfig', () => {
test('reads search.autocut + search.autocut_jump', () => {
const out = autocutFromConfig({ search: { autocut: false, autocut_jump: 0.4 } });
expect(out.enabled).toBe(false);
expect(out.jumpRatio).toBe(0.4);
});
test('clamps out-of-range jump to fallback (ignored)', () => {
expect(autocutFromConfig({ search: { autocut_jump: 5 } }).jumpRatio).toBeUndefined();
expect(autocutFromConfig({ search: { autocut_jump: 0 } }).jumpRatio).toBeUndefined();
});
test('empty / missing config → empty partial', () => {
expect(autocutFromConfig(null)).toEqual({});
expect(autocutFromConfig({})).toEqual({});
});
});
describe('resolveAutocut — precedence ladder', () => {
test('defaults → config → per-call', () => {
const cfg = resolveAutocut(undefined, { jumpRatio: 0.3 });
expect(cfg.jumpRatio).toBe(0.3);
expect(cfg.enabled).toBe(true); // default
});
test('per-call true/false overrides config enabled', () => {
expect(resolveAutocut(false, { enabled: true }).enabled).toBe(false);
expect(resolveAutocut(true, { enabled: false }).enabled).toBe(true);
});
test('per-call partial overrides specific fields', () => {
const cfg = resolveAutocut({ jumpRatio: 0.5 }, { jumpRatio: 0.3, enabled: false });
expect(cfg.jumpRatio).toBe(0.5);
expect(cfg.enabled).toBe(false); // inherited from config (partial didn't set it)
});
});
+63
View File
@@ -10,7 +10,9 @@ import type { SearchResult } from '../../src/core/types.ts';
import {
formatResultExplain,
formatResultsExplain,
formatAutocutSummary,
} from '../../src/core/search/explain-formatter.ts';
import type { AutocutDecision } from '../../src/core/search/autocut.ts';
function r(slug: string, score: number, extras: Partial<SearchResult> = {}): SearchResult {
return {
@@ -197,3 +199,64 @@ describe('formatResultExplain — number formatting', () => {
expect(out).toContain('score=NaN');
});
});
describe('v0.42.3.0 — autocut in --explain', () => {
test('rerank_score renders per result (the cliff signal)', () => {
const out = formatResultExplain(r('a/b', 1.2, { rerank_score: 0.87 }), 1);
expect(out).toContain('rerank score=0.87');
});
test('no rerank score line when absent', () => {
const out = formatResultExplain(r('a/b', 1.2), 1);
expect(out).not.toContain('rerank score');
});
const applied: AutocutDecision = {
applied: true,
signal: 'rerank',
cut: 2,
kept: 2,
total: 7,
gapRatio: 0.41,
};
const declined: AutocutDecision = {
applied: false,
signal: 'none',
cut: 7,
kept: 7,
total: 7,
gapRatio: 0.05,
};
test('formatAutocutSummary — applied cut', () => {
const s = formatAutocutSummary(applied);
expect(s).toContain('kept 2/7');
expect(s).toContain('0.41');
});
test('formatAutocutSummary — declined', () => {
const s = formatAutocutSummary(declined);
expect(s).toContain('no cut');
expect(s).toContain('full 7');
});
test('formatAutocutSummary — undefined → null (omit cleanly)', () => {
expect(formatAutocutSummary(undefined)).toBeNull();
});
test('formatResultsExplain prepends the autocut summary when meta has a decision', () => {
const out = formatResultsExplain([r('a/b', 1.2, { rerank_score: 0.9 })], {
vector_enabled: true,
detail_resolved: null,
expansion_applied: false,
autocut: applied,
});
expect(out.startsWith('autocut:')).toBe(true);
expect(out).toContain('kept 2/7');
});
test('formatResultsExplain omits the summary when meta has no autocut decision', () => {
const out = formatResultsExplain([r('a/b', 1.2)]);
expect(out.startsWith('autocut:')).toBe(false);
});
});
@@ -167,8 +167,14 @@ describe('hybridSearch — reranker enabled (reorder)', () => {
// Now rerank only the top 2 (swap them); the tail (indices 2..N-1)
// must keep its baseline order.
// v0.42.3.0: autocut is default-ON in balanced mode and would cut this
// artificial 2-item scored head (0.99 vs 0.5 is a cliff) down to 1,
// dropping the un-scored tail. This test isolates RERANKER tail mechanics,
// so disable autocut here — in real balanced mode top_n_in = searchLimit
// (D4), so topNIn < pool with an un-scored tail never happens by default.
const reranked = await hybridSearch(engine, 'alpha keyword', {
limit: 10,
autocut: false,
reranker: {
enabled: true,
topNIn: 2,
+1 -1
View File
@@ -55,7 +55,7 @@ describe('KNOBS_HASH_VERSION + version invariants', () => {
// v0.41.22.0 (type-unification): 5→6 to fold the alias_resolved
// post-fusion boost. Cache rows written before the boost stage
// cannot leak past the new stage.
expect(KNOBS_HASH_VERSION).toBe(7);
expect(KNOBS_HASH_VERSION).toBe(8);
});
test('hash is 16 hex chars regardless of reranker config', () => {
+44
View File
@@ -0,0 +1,44 @@
/**
* v0.42.3.0 — pins the agent-facing autocut surface on the `query` op.
*
* Autocut is the smart default; the param exists ONLY as a ceiling override
* (force the full top-K). The description must teach the agent that they
* almost never set it, and that `false` is the breadth escape hatch. Guards
* against a refactor silently dropping the param or the instruction.
*/
import { describe, expect, test } from 'bun:test';
import { operationsByName } from '../../src/core/operations.ts';
describe('query op — autocut agent surface', () => {
const query = operationsByName['query'];
test('query op exists', () => {
expect(query).toBeDefined();
});
test('autocut is a boolean param on query', () => {
const param = query.params?.autocut as { type?: string; description?: string } | undefined;
expect(param).toBeDefined();
expect(param?.type).toBe('boolean');
});
test('description frames autocut as the default and FALSE as the breadth override', () => {
const desc = ((query.params?.autocut as { description?: string })?.description ?? '').toLowerCase();
// It's a default, not a feature the agent turns on.
expect(desc).toContain('default');
// The actionable direction is FALSE for breadth.
expect(desc).toContain('false');
expect(desc).toContain('breadth');
// Safety contract so the agent trusts it.
expect(desc).toContain('never returns empty');
// Distinguish from adaptive_return so the agent picks the right knob.
expect(desc).toContain('adaptive_return');
});
test('search op (keyword-only) does NOT carry autocut (no reranker there)', () => {
const search = operationsByName['search'];
expect(search).toBeDefined();
expect(search.params?.autocut).toBeUndefined();
});
});