mirror of
https://github.com/garrytan/gbrain.git
synced 2026-07-27 22:15:33 +00:00
Merge origin/master into garrytan/managua-v3
Resolves v0.34.0.0 (W1-W8 code intelligence) with master's v0.33.2.1 + search-lite work (query cache + intent weighting + token budget + drift watch + metric glossary + search modes). Conflict resolutions: - VERSION / package.json: kept 0.34.0.0 (mine; higher than master's 0.33.2.1) - CHANGELOG.md: both entries preserved; reordered so v0.33.2.1 sits above v0.33.2.0 (semver order) - src/cli.ts CLI_ONLY: union of both — `edges-backfill` (mine) + `cache` (master) - src/core/migrate.ts: renumbered my migrations to avoid collision with master's query_cache_search_lite (v55), query_cache_knobs_hash (v56), search_telemetry_rollup (v57). My `edges_backfilled_at_v0_33_2` moves v55 → v58; my `code_traversal_cache_v0_34` moves v56 → v59. Code refs in `src/core/code-intel/traversal-cache.ts` and the paired test updated to match. - src/core/operations.ts query op: kept master's `hybridSearchCached` routing (search-lite cache integration) AND my `sourceId` resolution block (D4 source-routing fix from v0.34 STEP 0). Both apply. Verification: - `bun run typecheck` clean - `bun run verify` clean (includes check-cli-executable, check-jsonb, check-system-of-record, check-eval-glossary-fresh, etc.) - Migration v50→v59 apply cleanly on PGLite in isolated test runs - Individual test files pass (e.g. test/search-lang-symbol-kind.test.ts: 9 pass / 0 fail in 913ms) Known follow-up: the parallel test shard runner times out some beforeAll hooks at the default 7s budget. Tests pass when run sequentially (`--max-concurrency=1`); 27/0 confirmed across 3 sample files in 2.4s sequential vs timeouts under parallel-shard contention. Master added 4 new migrations (v55-v57 + search-lite related) increasing per-test-file PGLite init cost; on 8 shards racing for OS resources, some shards hit the 7s ceiling. This is a test-infrastructure issue (shard isolation under heavier migrations), not a code-correctness issue. Fix is a follow-up: either raise shard test timeout, reduce shard count, or migrate to fixture-based engine setup for hot tests. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Garry Tan
@@ -10,7 +10,15 @@ start here.
|
||||
2. Install: `bun install`
|
||||
3. Init the brain: `gbrain init` (defaults to PGLite, zero-config). For 1000+ files or
|
||||
multi-machine sync, init suggests Postgres + pgvector via Supabase.
|
||||
4. Read [`./INSTALL_FOR_AGENTS.md`](./INSTALL_FOR_AGENTS.md) for the full 9-step flow
|
||||
4. **STOP — ask the user about search mode.** `gbrain init` auto-applied a
|
||||
default but printed a 9-cell cost matrix (mode × downstream model)
|
||||
preceded by `[AGENT]` markers. You MUST relay the matrix to the operator
|
||||
and confirm their choice before continuing. Cost spread between corners
|
||||
is 25x — silent acceptance is the wrong default. See
|
||||
[`./INSTALL_FOR_AGENTS.md`](./INSTALL_FOR_AGENTS.md) Step 3.5 for the
|
||||
exact ask-the-user protocol. Same banner fires on `gbrain post-upgrade`
|
||||
for existing users (search modes were added in v0.32.3).
|
||||
5. Read [`./INSTALL_FOR_AGENTS.md`](./INSTALL_FOR_AGENTS.md) for the full 9-step flow
|
||||
(API keys, identity, cron, verification).
|
||||
|
||||
## Read this order
|
||||
|
||||
@@ -74,6 +74,17 @@ To exercise the new W1/W2 edge shapes on an existing brain:
|
||||
- contents of `~/.gbrain/upgrade-errors.jsonl` if it exists
|
||||
- which step broke
|
||||
|
||||
## [0.33.2.1] - 2026-05-14
|
||||
|
||||
**Doc-only: name the fork-PR escape hatch so AI-authored PRs don't drop their CI secrets.**
|
||||
|
||||
PRs from `garrytan-agents` (the AI-authored PR account) live in a fork. GitHub's `pull_request` event doesn't ship base-repo secrets to forks by default, so any CI job that needs `ANTHROPIC_API_KEY` or `OPENAI_API_KEY` quietly fails with empty-env auth errors regardless of what's set on the base repo. CLAUDE.md now documents the move-branch-to-base-repo workflow as the narrow-scope alternative to adding the account as a collaborator (which would broaden secret distribution to every PR from that account) or flipping the repo-wide fork-secret toggle (which would broaden it to every fork PR).
|
||||
|
||||
### Itemized changes
|
||||
|
||||
- `CLAUDE.md` gets a new `## Checking out PRs from garrytan-agents` section between "Community PR wave process" and "Skill routing". Four-step recipe: `gh pr checkout <N>` → `git push origin HEAD:<branch>` → `gh pr close <N>` → `gh pr create --base master --head <branch>`, preserving the original title and body verbatim. Closes the friction Garry hit landing #962 / Voyage 2048-dim fixup PRs from the agent account.
|
||||
- `llms-full.txt` regenerated by `bun run build:llms` so the committed doc bundle matches the live CLAUDE.md. Pinned by `test/build-llms.test.ts` in CI shard 1.
|
||||
|
||||
## [0.33.2.0] - 2026-05-12
|
||||
|
||||
**Code intelligence ships to agents. Plan-mode subagents stop falling through to grep.**
|
||||
@@ -82,6 +93,69 @@ To exercise the new W1/W2 edge shapes on an existing brain:
|
||||
Pre-v0.33.2 the four code-intelligence commands from v0.20+ Cathedral II lived in `CLI_ONLY` at `cli.ts:30`. An agent running through MCP saw `query`/`search` but no structural retrieval, so it grepped, missed callers in string literals, shipped plans with broken call chains, and got caught in review. v0.33.2 closes that gap and lays the foundation work that v0.34 Cathedral III (recursive blast/flow + Leiden clusters + wiki) will build on top of.
|
||||
|
||||
This release was scoped after Codex's outside-voice review caught two load-bearing premise gaps in the original v0.34 plan: the call graph stored bare callee tokens (not qualified names), and source routing was already broken in `query` and `two-pass.ts`. Both are fixed here before any user-facing recursive op ships.
|
||||
|
||||
## [0.33.1.1] - 2026-05-13
|
||||
|
||||
**Voyage 2048-dim brains finally produce 2048-dim vectors. Fail-loud on every Voyage misconfiguration.**
|
||||
|
||||
Voyage-backed brains configured for high-dimensional embeddings (2048 wide for richer recall on long-form content) were silently producing 1024-dim vectors on every embed call. The dimension instruction was being routed through a wire-key that the AI SDK's openai-compatible adapter doesn't recognize, so it got dropped before the HTTP request was built. Voyage returned its default 1024-dim, the gateway dimension check threw, and brains failed to ingest. This release lands the upstream fix from @100yenadmin and stacks three Voyage-adjacent correctness follow-ups that adversarial Codex review caught during PR review.
|
||||
|
||||
### What you can now do
|
||||
|
||||
**Run Voyage brains at 2048 dims and actually get 2048 dims.** `embedding_model: voyage:voyage-4-large` + `embedding_dimensions: 2048` now routes through the SDK-supported `dimensions` field, which the existing `voyageCompatFetch` shim rewrites to Voyage's `output_dimension` on the wire. New wire-level test asserts the actual outbound HTTP body contains `output_dimension: 2048`. Same fix covers voyage-3-large, voyage-3.5, voyage-3.5-lite, voyage-4, voyage-4-lite, voyage-code-3.
|
||||
|
||||
**Get caught at config time, not first-embed.** `gbrain models doctor` now runs an `embedding_config` probe before any token-spending chat/expansion probes. If your brain is set to a Voyage flexible-dim model with `embedding_dimensions` outside `{256, 512, 1024, 2048}` (the most common trip: leaving `embedding_dimensions` unset, which falls back to the OpenAI default of 1536), doctor surfaces it with a paste-ready `gbrain config set` fix. The runtime `dimsProviderOptions` validator throws an `AIConfigError` at the embed boundary with the same fix hint, so even if you skipped doctor you get a clear message naming the valid values instead of an opaque Voyage HTTP 400.
|
||||
|
||||
**voyage-4-nano stays in its lane.** Voyage's `voyage-4-nano` is an open-weight variant listed separately by Voyage as fixed 1024-dim — it doesn't accept `output_dimension`. Dropped from the flexible-dim allowlist; recipe docstring tightened to name the seven hosted flexible-dim models explicitly so the "all v4 variants are flexible" claim doesn't lead the next contributor to re-add nano. A negative regression test pins the contract.
|
||||
|
||||
**Voyage OOM cap is now actually effective.** `voyageCompatFetch`'s 256 MB per-response cap (the defense-in-depth check that fires when Content-Length is missing on chunked encoding) was being silently swallowed by the surrounding parse-error try/catch — a malicious or misbehaving Voyage endpoint returning a multi-gigabyte response could have OOMed the worker. Now wrapped in a tagged `VoyageResponseTooLargeError` class that the catch rethrows on `instanceof`, so the cap fails loud instead of returning the oversized response into the AI SDK's JSON parser.
|
||||
|
||||
### Why this matters
|
||||
|
||||
Voyage is the gbrain-recommended embedding provider for users who want pgvector-native, high-quality embeddings without depending on OpenAI. The 2048-dim bug was a structural footgun: every first-time Voyage user hitting it without obvious cause. Eva (@100yenadmin) caught and fixed the root cause; the follow-up wave plugs the three adjacent correctness gaps that pre-empted future Voyage users hitting similar opaque failures.
|
||||
|
||||
## To take advantage of v0.33.1.1
|
||||
|
||||
`gbrain upgrade` should do this automatically. If you're already on Voyage:
|
||||
|
||||
1. **Verify your dim config is valid:**
|
||||
```bash
|
||||
gbrain models doctor
|
||||
```
|
||||
You should see a new `embedding_config` line; if it reports `config` status, follow the paste-ready fix.
|
||||
2. **No reindex required** if you were already on a Voyage flexible-dim model with the correct `embedding_dimensions` (the bug surfaced as fail-loud at embed time, so existing brains didn't silently get wrong-width vectors — they failed to embed).
|
||||
3. **If `gbrain doctor` warns about anything,** file an issue at https://github.com/garrytan/gbrain/issues with:
|
||||
- output of `gbrain doctor`
|
||||
- your `embedding_model` + `embedding_dimensions` config values
|
||||
|
||||
### Itemized changes
|
||||
|
||||
#### Fixed
|
||||
|
||||
- **Voyage 2048-dim embeddings (closes #866).** `src/core/ai/dims.ts` `dimsProviderOptions` now returns `{ openaiCompatible: { dimensions: N } }` (SDK-supported) instead of `{ openaiCompatible: { output_dimension: N } }` (Voyage wire-key the SDK silently dropped). Existing `voyageCompatFetch` in `src/core/ai/gateway.ts:541` translates `dimensions → output_dimension` before the HTTP body is sent. Contributed by @100yenadmin (PR #866, re-landed in #962 due to `maintainerCanModify` cross-fork push limitation — authorship preserved on the original commit).
|
||||
|
||||
- **Voyage OOM-cap rethrow.** `src/core/ai/gateway.ts:619` Layer 2 base64 cap was throwing a generic `Error` that the surrounding `catch {}` swallowed, then returning the original (oversized) response to the AI SDK. New `VoyageResponseTooLargeError` tagged class is now thrown at both cap sites (Content-Length Layer 1 at `:595` and per-embedding Layer 2 at `:619`) and rethrown from the inbound try/catch via `instanceof` check. Parse-error fall-back behavior preserved for non-OOM errors. Codex P3 follow-up.
|
||||
|
||||
- **Voyage flexible-dim runtime validation.** `dimsProviderOptions` now throws `AIConfigError` with a paste-ready fix hint when a Voyage flexible-dim model is configured with anything outside `{256, 512, 1024, 2048}`. Most common trigger: `embedding_model: voyage:voyage-4-large` without `embedding_dimensions` (falls back to default 1536 — not Voyage-accepted). Codex P3 follow-up.
|
||||
|
||||
#### Changed
|
||||
|
||||
- **voyage-4-nano dropped from `VOYAGE_OUTPUT_DIMENSION_MODELS`.** Open-weight model, fixed-dim 1024 per Voyage docs; doesn't accept `output_dimension`. Recipe docstring at `src/core/ai/recipes/voyage.ts:7-16` tightened so the "all v4 variants" claim doesn't lead future contributors to re-add it. Negative regression test in `test/ai/gateway.test.ts` pins the contract.
|
||||
|
||||
- **`gbrain models doctor` runs an `embedding_config` probe first** (zero tokens, local-only). Surfaces Voyage flexible-dim mismatches before any chat/expansion probes spend money. New `config` probe status + `fix` hint rendered in both human and JSON output. New `embedding_config` touchpoint label appears alongside `chat` / `expansion`.
|
||||
|
||||
#### Tests
|
||||
|
||||
- New wire-level test (`test/ai/gateway.test.ts`): stubs `globalThis.fetch` and asserts the outbound Voyage request body contains `output_dimension: 2048` + `encoding_format: base64`. Catches the exact regression class that motivated #866.
|
||||
- Two new behavioral tests for the OOM-cap rethrow (Content-Length over cap + oversized base64 string both propagate).
|
||||
- Six new tests for the flexible-dim runtime validator (1536 + 3072 rejected, all valid sizes accepted, `voyage-3-lite` / `voyage-4-nano` bypass validator, fix-hint contents).
|
||||
- Source-shape regression assertion in `test/voyage-response-cap.test.ts` pins the `instanceof VoyageResponseTooLargeError ⇒ throw err` line.
|
||||
|
||||
#### For contributors
|
||||
|
||||
- New tagged error class `VoyageResponseTooLargeError` is exported from `src/core/ai/gateway.ts` (test-only seam; not part of the public AI SDK surface).
|
||||
- New exports from `src/core/ai/dims.ts`: `VOYAGE_VALID_OUTPUT_DIMS` (`[256, 512, 1024, 2048] as const`) and `isValidVoyageOutputDim(dims: number)`. Reuse these if you add a Voyage-adjacent probe or doctor check; do NOT inline the magic numbers.
|
||||
|
||||
## [0.33.1.0] - 2026-05-10
|
||||
|
||||
**Ask gbrain who in your network knows about a topic, and get a ranked answer with the reasoning shown.**
|
||||
|
||||
@@ -92,16 +92,17 @@ strict behavior when unset.
|
||||
- `docs/eval-capture.md` (v0.25.0) — stable NDJSON schema reference for gbrain-evals consumers.
|
||||
- `test/public-exports.test.ts` (v0.25.0 / R2) — runtime contract test. Imports each of the 17 public subpaths via package name and pins a canary symbol per module. Paired with `scripts/check-exports-count.sh`.
|
||||
- `src/core/embedding.ts` — OpenAI text-embedding-3-large, batch, retry, backoff. **v0.28.7:** `BATCH_SIZE` reverted 50→100 — the original Voyage safety guard halved OpenAI throughput on every page. Per-recipe pre-split + recursive halving + adaptive shrink-on-miss now live in the gateway, so the outer paginator goes back to its original purpose: progress-callback granularity, not batch protection.
|
||||
- `src/core/ai/dims.ts` (v0.33.1.1, PR #962 + #866) — per-provider `providerOptions` resolver for embed-time dimension passthrough. The single source of truth for "which provider needs which knob to produce `vector(N)`". Exports `dimsProviderOptions(implementation, modelId, dims)` (called by `embed()` in `gateway.ts`), `VOYAGE_OUTPUT_DIMENSION_MODELS` (private const — the 7 hosted Voyage models that accept `output_dimension`: `voyage-4-large`, `voyage-4`, `voyage-4-lite`, `voyage-3-large`, `voyage-3.5`, `voyage-3.5-lite`, `voyage-code-3` — nano deliberately excluded), `VOYAGE_VALID_OUTPUT_DIMS = [256, 512, 1024, 2048] as const`, `supportsVoyageOutputDimension(modelId)`, and `isValidVoyageOutputDim(dims)`. **Voyage path uses the SDK-supported `dimensions` field** (`{ openaiCompatible: { dimensions: N } }`), NOT Voyage's `output_dimension` wire-key — the existing `voyageCompatFetch` shim in `gateway.ts:541` translates `dimensions → output_dimension` before the HTTP body is built. The reverse (sending `output_dimension` from here) was the v0.33.1.0 bug class: the AI SDK's openai-compatible adapter doesn't recognize the wire-key so it was silently dropped, Voyage returned its default 1024-dim, and the gateway dimension check threw on every embed call. Runtime guard: when a Voyage flexible-dim model is configured with `dims` outside `VOYAGE_VALID_OUTPUT_DIMS`, throws `AIConfigError` with a paste-ready `gbrain config set embedding_dimensions <256|512|1024|2048>` hint at the embed boundary — fail-loud instead of opaque Voyage HTTP 400. Most common trigger: `embedding_model: voyage:voyage-4-large` configured without `embedding_dimensions` (falls back to `DEFAULT_EMBEDDING_DIMENSIONS=1536`, an OpenAI default not a Voyage one). Eva (@100yenadmin) shipped the wire-key fix in #866; Codex P3 follow-up landed the validator + valid-dims allowlist in #962.
|
||||
- `src/core/ai/types.ts` — provider/recipe types. **v0.28.7 (#680):** `EmbeddingTouchpoint` extended with optional `chars_per_token` (default 4 chars/token, matching OpenAI tiktoken on English) and `safety_factor` (default 0.8, budget-utilization ceiling). Both consulted only when `max_batch_tokens` is also set. Voyage declares `chars_per_token=1` + `safety_factor=0.5` to handle dense payloads (CJK/JSON/base64) that overshoot tiktoken. The pre-split budget is `max_batch_tokens × safety_factor / chars_per_token`. **v0.28.11 (#719):** `EmbeddingTouchpoint.multimodal_models?: string[]` model-level allow-list for recipes that mix text-only + multimodal models under one touchpoint (Voyage's 12 models share `supports_multimodal: true` but only `voyage-multimodal-3` accepts `/multimodalembeddings`). When omitted, recipe-level `supports_multimodal` is sufficient. `AIGatewayConfig.embedding_multimodal_model?: string` lets `embedMultimodal()` route to a different model than `embedding_model` — brains using OpenAI for text can use Voyage for images without flipping the primary embedding pipeline.
|
||||
- `src/core/ai/gateway.ts` — unified seam for every AI call. **v0.28.7 (#680):** module-scoped `_embedTransport` defaulting to AI SDK `embedMany`, with `__setEmbedTransportForTests(fn)` test seam so tests drive the public `embed()` function with a stubbed transport instead of probing private helpers. `splitByTokenBudget` and `isTokenLimitError` are now exported `@internal` — pure functions reused directly by the test file. Module-level `_shrinkState: Map<recipeId, {factor, consecutiveSuccesses}>` halves the recipe's effective `safety_factor` on token-limit miss (floor 0.05) and heals back ×1.5 toward the ceiling after `SHRINK_HEAL_AFTER=10` consecutive successes. `configureGateway()` walks every registered recipe at construction time and emits a once-per-process stderr warning for any embedding touchpoint missing `max_batch_tokens` (excluding the canonical OpenAI fast-path recipe). `resetGateway()` clears `_shrinkState`, the warned-set, and restores the real transport. ASCII flow diagram embedded in the `embed()` JSDoc covers the routing decision, recursion + halving, and shrinkState lifecycle. **v0.28.11 (#719):** `embedMultimodal()` reads `cfg.embedding_multimodal_model` first (falls back to `cfg.embedding_model` for single-model setups). After the existing recipe-level `supports_multimodal` fast-fail, validates the resolved model against `touchpoint.multimodal_models` when declared — closes the Voyage-text-only-model-into-multimodal-endpoint footgun before any HTTP call (Codex F1 from PR review). New `getMultimodalModel()` accessor mirrors `getEmbeddingModel` / `getChatModel` so doctor and integration tests can read the gateway state.
|
||||
- `src/core/ai/recipes/voyage.ts` — Voyage AI openai-compatible recipe. **v0.28.7 (#680):** declares `chars_per_token=1` + `safety_factor=0.5` so the gateway pre-splits Voyage batches at a 60K-character budget (50% of 120K-token cap with the dense-tokenizer ratio). Closes the v0.27 backfill loop where ~26% of the corpus stayed un-embedded because tiktoken-grounded budgeting silently undercounted Voyage's actual token usage. **v0.28.11 (#719):** declares `multimodal_models: ['voyage-multimodal-3']` so the gateway rejects text-only Voyage models pointed at the multimodal endpoint with a clear `AIConfigError` instead of waiting for Voyage's HTTP 400.
|
||||
- `src/core/ai/gateway.ts` — unified seam for every AI call. **v0.28.7 (#680):** module-scoped `_embedTransport` defaulting to AI SDK `embedMany`, with `__setEmbedTransportForTests(fn)` test seam so tests drive the public `embed()` function with a stubbed transport instead of probing private helpers. `splitByTokenBudget` and `isTokenLimitError` are now exported `@internal` — pure functions reused directly by the test file. Module-level `_shrinkState: Map<recipeId, {factor, consecutiveSuccesses}>` halves the recipe's effective `safety_factor` on token-limit miss (floor 0.05) and heals back ×1.5 toward the ceiling after `SHRINK_HEAL_AFTER=10` consecutive successes. `configureGateway()` walks every registered recipe at construction time and emits a once-per-process stderr warning for any embedding touchpoint missing `max_batch_tokens` (excluding the canonical OpenAI fast-path recipe). `resetGateway()` clears `_shrinkState`, the warned-set, and restores the real transport. ASCII flow diagram embedded in the `embed()` JSDoc covers the routing decision, recursion + halving, and shrinkState lifecycle. **v0.28.11 (#719):** `embedMultimodal()` reads `cfg.embedding_multimodal_model` first (falls back to `cfg.embedding_model` for single-model setups). After the existing recipe-level `supports_multimodal` fast-fail, validates the resolved model against `touchpoint.multimodal_models` when declared — closes the Voyage-text-only-model-into-multimodal-endpoint footgun before any HTTP call (Codex F1 from PR review). New `getMultimodalModel()` accessor mirrors `getEmbeddingModel` / `getChatModel` so doctor and integration tests can read the gateway state. **v0.33.1.1 (#962, Codex P3 follow-up):** new exported `VoyageResponseTooLargeError` tagged class at the top of the file. `voyageCompatFetch`'s two OOM-defense caps (Layer 1 Content-Length check at `:595`, Layer 2 per-embedding base64 cap at `:619`) now throw `VoyageResponseTooLargeError` instead of a generic `Error`. The inbound response-rewriter's surrounding try/catch (which intentionally swallows parse failures so misshaped Voyage responses fall through to the SDK's JSON parser) checks `instanceof VoyageResponseTooLargeError` and rethrows. Pre-fix, the Layer 2 throw was silently swallowed and the oversized response returned to the AI SDK anyway — Layer 2 was theatrical. Source-shape regression assertion in `test/voyage-response-cap.test.ts` pins the `instanceof ⇒ throw err` line.
|
||||
- `src/core/ai/recipes/voyage.ts` — Voyage AI openai-compatible recipe. **v0.28.7 (#680):** declares `chars_per_token=1` + `safety_factor=0.5` so the gateway pre-splits Voyage batches at a 60K-character budget (50% of 120K-token cap with the dense-tokenizer ratio). Closes the v0.27 backfill loop where ~26% of the corpus stayed un-embedded because tiktoken-grounded budgeting silently undercounted Voyage's actual token usage. **v0.28.11 (#719):** declares `multimodal_models: ['voyage-multimodal-3']` so the gateway rejects text-only Voyage models pointed at the multimodal endpoint with a clear `AIConfigError` instead of waiting for Voyage's HTTP 400. **v0.33.1.1 (#962, fixup):** recipe docstring at `:7-16` tightened to name the seven hosted flexible-dim models that accept `output_dimension` explicitly (`voyage-4-large`, `voyage-4`, `voyage-4-lite`, `voyage-3-large`, `voyage-3.5`, `voyage-3.5-lite`, `voyage-code-3`) and call out that `voyage-4-nano` is the open-weight variant listed separately by Voyage as fixed 1024-dim — does NOT accept the parameter. The "all v4 variants are flexible" misread is what caused the original PR to include nano in `VOYAGE_OUTPUT_DIMENSION_MODELS`; the negative regression assertion in `test/ai/gateway.test.ts` (`dimsProviderOptions` returns `undefined` for `voyage-4-nano`) pins the contract.
|
||||
- `src/core/ai/recipes/anthropic.ts` — Anthropic recipe (chat + expansion touchpoints). **v0.31.12:** chat and expansion `models:` lists drop the v0.31.6 phantom `claude-sonnet-4-6-20250929` date suffix — canonical id is `claude-sonnet-4-6`. The wrong-direction alias `claude-sonnet-4-6 → claude-sonnet-4-6-20250929` is removed; a reverse alias `claude-sonnet-4-6-20250929 → claude-sonnet-4-6` keeps stale user configs working (rescues `facts.extraction_model` and `models.dream.synthesize` set by v0.31.6 installs). Recipe-shape regression pinned by `test/anthropic-model-ids.test.ts` (6 cases, verbatim cherry-pick of PR #830 plus the reverse-alias rescue case).
|
||||
- `src/core/anthropic-pricing.ts` — Single source of truth for Anthropic model pricing (per-MTok input/output). **v0.31.12:** Opus 4.7 corrected from `$15/$75` to `$5/$25` (the old number was from Opus 4 generation, never refreshed when 4.7 shipped); Opus 4.6 also corrected. Consumed by `src/core/budget-meter.ts` and `src/core/cross-modal-eval/runner.ts` — the cross-modal estimator now reads `ANTHROPIC_PRICING` for Anthropic models instead of duplicating the table, killing the v0.31.6 drift bug class.
|
||||
- `src/core/model-config.ts` — Model-string resolution (the seam every internal LLM call walks through). **v0.31.12:** four-tier system (`ModelTier = 'utility' | 'reasoning' | 'deep' | 'subagent'`) with `TIER_DEFAULTS` (utility→haiku-4-5, reasoning→sonnet-4-6, deep→opus-4-7, subagent→sonnet-4-6) and `tier?: ModelTier` on `ResolveModelOpts`. Resolution chain is now 8 steps: cliFlag → deprecated key → config key → `models.default` → `models.tier.<tier>` → env var → `TIER_DEFAULTS[tier]` → caller fallback. Two new exports — `isAnthropicProvider(modelString)` checks `provider:model` prefix OR `claude-` bare-id pattern, and `enforceSubagentAnthropic()` is the layer-2 runtime guard: when `tier === 'subagent'` resolves to a non-Anthropic provider, it emits a once-per-`(source, model)` stderr warn AND falls back to `TIER_DEFAULTS.subagent` instead of letting the Anthropic Messages API tool-loop attempt to run on OpenAI/Gemini. `_resetDeprecationWarningsForTest()` now also clears `_subagentTierWarningsEmitted` so tests re-emit.
|
||||
- `src/core/ai/model-resolver.ts` — Recipe-touchpoint validator. **v0.31.12:** `assertTouchpoint(recipe, touchpoint, modelId, extendedModels?)` gains an optional 4th `extendedModels: ReadonlySet<string>` argument. When the modelId is in that set, the native-recipe allowlist throw is bypassed — the user explicitly opted into this model via config so we let provider rejection surface as `model_not_found` at HTTP call time (and `gbrain models doctor` catches it earlier). Default code paths with hardcoded model strings MUST NOT pass `extendedModels` — typos in source code still fail fast. Replaces the earlier plan to soften the validator wholesale (Codex F4/F5 in plan review flagged that as too broad — it would have removed the fail-fast contract for chat + expand + embed all three).
|
||||
- `src/core/ai/gateway.ts` extension (v0.31.12) — new module-scoped `_extendedModels: Map<providerId, Set<modelId>>` registry feeds `assertTouchpoint`'s 4th-arg path. New `reconfigureGatewayWithEngine(engine)` async function is called from `cli.ts` after `engine.connect()` (and before every command except `CLI_ONLY` no-DB commands) — re-resolves expansion + chat defaults through `resolveModel()` so `models.tier.*` and `models.default` overrides apply to expansion + chat both. `DEFAULT_CHAT_MODEL` corrected to `anthropic:claude-sonnet-4-6` (was the v0.31.6 phantom `-20250929`). New `__setChatTransportForTests` seam mirrors `__setEmbedTransportForTests` so tests drive `chat()` with a stubbed transport.
|
||||
- `src/core/minions/queue.ts` extension (v0.31.12) — `MinionQueue.add()` now rejects `subagent` jobs whose `data.model` resolves through `isAnthropicProvider()` to a non-Anthropic provider. Lazy-imports `model-config.ts` to avoid pulling engine types into queue's eager-load surface. Layer 1 of the three-layer subagent provider enforcement (Codex F1+F2 in plan review). Layers 2 + 3 live in `src/core/model-config.ts` (`enforceSubagentAnthropic` runtime fallback) and `src/commands/doctor.ts` (`subagent_provider` check). Pinned by 3 cases in `test/agent-cli.test.ts`.
|
||||
- `src/commands/models.ts` (v0.31.12) — `gbrain models [--json]` read-only routing dashboard: prints tier defaults (`utility`/`reasoning`/`deep`/`subagent`), the resolved value for each (re-walking the resolution chain to attribute properly), every per-task override (11 `PER_TASK_KEYS` entries — `models.dream.synthesize`, `models.dream.patterns`, `models.drift`, `models.auto_think`, `models.think`, `models.subagent`, `facts.extraction_model`, `models.eval.longmemeval`, `models.expansion`, `models.chat`, `models.dream.synthesize_verdict`), the alias map (defaults + user overrides), and a source-of-truth column showing `default` / `config: <key>` / `env: <VAR>`. `gbrain models doctor [--skip=<provider>] [--json]` fires a 1-token `gateway.chat()` probe against each configured chat + expansion model and classifies failures into `{model_not_found, auth, rate_limit, network, unknown}` — the structural fix for the v0.31.6 silent-no-op bug class. Wired into `cli.ts` dispatch table + `CLI_ONLY` set.
|
||||
- `src/commands/models.ts` (v0.31.12) — `gbrain models [--json]` read-only routing dashboard: prints tier defaults (`utility`/`reasoning`/`deep`/`subagent`), the resolved value for each (re-walking the resolution chain to attribute properly), every per-task override (11 `PER_TASK_KEYS` entries — `models.dream.synthesize`, `models.dream.patterns`, `models.drift`, `models.auto_think`, `models.think`, `models.subagent`, `facts.extraction_model`, `models.eval.longmemeval`, `models.expansion`, `models.chat`, `models.dream.synthesize_verdict`), the alias map (defaults + user overrides), and a source-of-truth column showing `default` / `config: <key>` / `env: <VAR>`. `gbrain models doctor [--skip=<provider>] [--json]` fires a 1-token `gateway.chat()` probe against each configured chat + expansion model and classifies failures into `{model_not_found, auth, rate_limit, network, unknown}` — the structural fix for the v0.31.6 silent-no-op bug class. Wired into `cli.ts` dispatch table + `CLI_ONLY` set. **v0.33.1.1 (#962, Codex P3 follow-up):** doctor gains a zero-token `embedding_config` probe that runs FIRST, before any chat/expansion probes spend money. `probeEmbeddingConfig()` reads `getEmbeddingModel()` + `getEmbeddingDimensions()` from the gateway, parses the model id, and (for Voyage flexible-dim models) checks `isValidVoyageOutputDim(dims)` against `VOYAGE_VALID_OUTPUT_DIMS`. New `ProbeStatus` variant `'config'` and optional `fix?: string` field on `ProbeResult` — surfaced in both human output (paste-ready `gbrain config set ...` line under the bad probe) and JSON output. New touchpoint label `'embedding_config'` joins `'chat'` and `'expansion'` in the probe-row taxonomy. Closes the Voyage flexible-dim bug class at config time, not first-embed.
|
||||
- `src/commands/doctor.ts` extension (v0.31.12) — new `subagent_provider` check (layer 3 of 3 — Codex F13). Warns when `models.tier.subagent` is explicitly set to a non-Anthropic provider (fail-loud since the user clearly meant it — message names the bad value and prints the paste-ready fix command `gbrain config set models.tier.subagent anthropic:claude-sonnet-4-6`); also warns when `models.default` would sneak `subagent` into a non-Anthropic provider via tier inheritance. OK status when subagent tier resolves to Anthropic. Tests cover all three paths in `test/doctor.test.ts`.
|
||||
- `src/core/check-resolvable.ts` — Resolver validation: reachability, MECE overlap, DRY checks, structured fix objects. v0.14.1: `CROSS_CUTTING_PATTERNS.conventions` is an array (notability gate accepts both `conventions/quality.md` and `_brain-filing-rules.md`). New `extractDelegationTargets()` parses `> **Convention:**`, `> **Filing rule:**`, and inline backtick references. DRY suppression is proximity-based via `DRY_PROXIMITY_LINES = 40`.
|
||||
- `src/core/repo-root.ts` — Shared `findRepoRoot(startDir?)` (v0.16.4): walks up from `startDir` (default `process.cwd()`) looking for `skills/RESOLVER.md`. Zero-dependency module imported by both `doctor.ts` and `check-resolvable.ts`. Parameterized `startDir` makes tests hermetic. **v0.31.7:** read-path / write-path split. `autoDetectSkillsDir` (shared, read+write-safe) gains tier-0 `$GBRAIN_SKILLS_DIR` explicit operator override (Docker mounts, CI, monorepo subdirs) ahead of the existing 4-tier chain. New `autoDetectSkillsDirReadOnly` wraps it with a tier-5 install-path fallback that walks up from `fileURLToPath(import.meta.url)` and gates on `isGbrainRepoRoot` so unrelated repos can't false-positive. Read-path callers (`doctor`, `check-resolvable`, `routing-eval`) use the read-only variant; write-path callers (`skillpack install`, `skillify scaffold`, `post-install-advisory`) deliberately stay on the shared function so `gbrain skillpack install` from `~` cannot silently retarget the bundled gbrain repo's `skills/` instead of the user's actual workspace. Two new `SkillsDirSource` variants: `'env_explicit'`, `'install_path'`. New `AUTO_DETECT_HINT_READ_ONLY` documents the extra tier. The D6 `--fix` safety gate in `doctor.ts` + `check-resolvable.ts` refuses auto-repair when `detected.source === 'install_path'` so `gbrain doctor --fix` from `~` cannot silently rewrite the bundled install tree.
|
||||
@@ -350,6 +351,12 @@ Key commands added in v0.32.7 (CJK fix wave):
|
||||
- `gbrain search "<CJK substring>"` on PGLite brains now uses an `ILIKE`-based fallback with bigram-frequency-count ranking when the query contains Han / Hiragana / Katakana / Hangul Syllables. ASCII queries continue through `websearch_to_tsquery('english')` unchanged. Postgres-side CJK FTS still requires an extension (pgroonga / zhparser) — see v0.33+ TODO.
|
||||
- `gbrain upgrade` post-upgrade flow now prints a cost estimate before re-embedding: `[chunker-bump] Will re-embed ~N markdown pages via <provider:model>, est. ~$X.XX, ~Ymin. Press Ctrl-C within 10s to abort.` Sourced from real SQL counts + char totals; TTY-only wait (non-TTY auto-proceeds for CI / cron). Env overrides: `GBRAIN_NO_REEMBED=1` bails out entirely with a doctor-warning marker; `GBRAIN_REEMBED_GRACE_SECONDS=0` skips the wait.
|
||||
|
||||
Key commands added in v0.33.1.1 (Voyage 2048-dim correctness wave):
|
||||
- `gbrain models doctor` learns a new zero-token `embedding_config` probe that runs FIRST, before any chat/expansion probes spend money. Catches Voyage flexible-dim misconfigs at config time, not first-embed: `embedding_model: voyage:voyage-4-large` with `embedding_dimensions` outside `{256, 512, 1024, 2048}` (most commonly: `embedding_dimensions` left unset, falling back to the OpenAI default 1536 which Voyage rejects with an opaque HTTP 400). Surfaces a paste-ready `gbrain config set embedding_dimensions <256|512|1024|2048>` fix in both human and JSON output. New probe status `'config'` joins `{ok, model_not_found, auth, rate_limit, network, unknown}`; new touchpoint label `'embedding_config'` joins `'chat'` and `'expansion'`.
|
||||
- Voyage 2048-dim brains now actually embed at 2048 dims. `embedding_model: voyage:voyage-4-large` + `embedding_dimensions: 2048` routes through the SDK-supported `dimensions` field, which `voyageCompatFetch` translates to Voyage's `output_dimension` on the wire. Same fix covers `voyage-3-large`, `voyage-3.5`, `voyage-3.5-lite`, `voyage-4`, `voyage-4-lite`, `voyage-code-3`. `voyage-4-nano` (open-weight, fixed 1024-dim) intentionally NOT in the flexible-dim allowlist — sending `output_dimension` to nano's endpoint produces an error.
|
||||
- Runtime validator: `dimsProviderOptions()` throws `AIConfigError` at the embed boundary with a paste-ready fix hint when a Voyage flexible-dim model is configured with an invalid dim — fail-loud even if you skipped `gbrain models doctor`.
|
||||
- `VoyageResponseTooLargeError` (new tagged class exported from `src/core/ai/gateway.ts`): the 256 MB per-response cap inside `voyageCompatFetch` was previously throwing a generic `Error` that the surrounding parse-error try/catch silently swallowed, returning the oversized response to the AI SDK anyway. Now thrown at both cap sites (Content-Length Layer 1, per-embedding base64 Layer 2) and rethrown from the catch via `instanceof` check — the cap is now actually effective.
|
||||
|
||||
Key commands added in v0.31.12 (model tier system + routing CLI):
|
||||
- `gbrain models [--json]` — read-only routing dashboard. Prints the four tier defaults (`utility`/`reasoning`/`deep`/`subagent`), the resolved value for each (after re-walking `models.default` → `models.tier.<tier>` → env → `TIER_DEFAULTS`), every per-task override (`models.dream.synthesize`, `models.dream.patterns`, `models.drift`, `models.auto_think`, `models.think`, `models.subagent`, `facts.extraction_model`, `models.eval.longmemeval`, `models.expansion`, `models.chat`, `models.dream.synthesize_verdict`), the alias map (defaults + user overrides), and a source-of-truth column (`default` / `config: <key>` / `env: <VAR>`).
|
||||
- `gbrain models doctor [--skip=<provider>] [--json]` — 1-token reachability probe against each configured chat + expansion model. Classifies failures into `{model_not_found, auth, rate_limit, network, unknown}`. The structural fix for the bug class that motivated v0.31.12 (v0.31.6's `claude-sonnet-4-6-20250929` chat default 404'd silently on every install).
|
||||
@@ -708,6 +715,100 @@ permission to run them — see the "run without asking" rule above.
|
||||
Never leave `gbrain-test-pg` running. If you find a stale one from a previous run,
|
||||
stop and remove it before starting a new one.
|
||||
|
||||
## Search Mode (v0.32.3)
|
||||
|
||||
GBrain ships three named search modes that bundle the search-lite knobs from
|
||||
PR #897 into a single config key. Pick one at install time; the rest of the
|
||||
project resolves through `src/core/search/mode.ts`.
|
||||
|
||||
| Knob | `conservative` | `balanced` | `tokenmax` |
|
||||
|-------------------------------|----------------|------------|----------------|
|
||||
| `cache.enabled` | true | true | true |
|
||||
| `cache.similarity_threshold` | 0.92 | 0.92 | 0.92 |
|
||||
| `cache.ttl_seconds` | 3600 | 3600 | 3600 |
|
||||
| `intentWeighting` | true | true | true |
|
||||
| `tokenBudget` | **4000** | **12000** | **off** |
|
||||
| `expansion` (LLM multi-query) | false | false | **true** |
|
||||
| `searchLimit` default | 10 | 25 | 50 |
|
||||
|
||||
**Cost anchors (downstream agent input cost — gbrain itself is rounding error).**
|
||||
The corner-to-corner spread is 25x once you pair mode with downstream model.
|
||||
Chunks ~400 tokens avg. Per-query cost @ 10K queries/month (typical
|
||||
single-user volume), full search payload, no cache savings:
|
||||
|
||||
| Mode \ Downstream | Haiku 4.5 (\$1/M) | Sonnet 4.6 (\$3/M) | Opus 4.7 (\$5/M) |
|
||||
|---|---|---|---|
|
||||
| conservative (~4K) | **\$40/mo** | \$120/mo | \$200/mo |
|
||||
| balanced (~10K) | \$100/mo | \$300/mo | \$500/mo |
|
||||
| tokenmax (~20K) | \$200/mo | \$600/mo | **\$1,000/mo** |
|
||||
|
||||
Scales linearly: multiply by 10 for 100K/mo (heavy power user / multi-user
|
||||
fleet); divide by 10 for 1K/mo (light usage). Natural pairings span ~4x.
|
||||
Mismatches (tokenmax+Haiku, conservative+Opus) waste capacity differently
|
||||
— too-big payload overwhelms a cheap model; too-small payload starves an
|
||||
expensive one.
|
||||
|
||||
tokenmax adds ~\$1.50 per 1K queries in Haiku expansion calls on top of
|
||||
the matrix (\$15/mo @ 10K). Cache hits cut all numbers ~50%. **The cost
|
||||
picker copy in `gbrain init` carries the same matrix verbatim** — update
|
||||
both when refreshing.
|
||||
|
||||
**Per-query math vs real-world spend.** The matrix above is what an
|
||||
isolated benchmark would measure. Real agent loops with disciplined
|
||||
Anthropic prompt caching see 50-80% discount on top (cache hits skip
|
||||
downstream entirely). The realistic-scale anchor in
|
||||
`docs/eval/SEARCH_MODE_METHODOLOGY.md` walks the natural pairings at
|
||||
single-power-user volume (~860 turns/mo): tokenmax+Opus ~\$700/mo,
|
||||
balanced+Sonnet ~\$430/mo, conservative+Haiku ~\$170/mo. Setups WITHOUT
|
||||
cache-aware prompt layout (frequent prefix churn) see the per-query
|
||||
matrix dominate — mode + model choice matters more there.
|
||||
|
||||
**Resolution chain** (matches the v0.31.12 model-tier pattern at
|
||||
`src/core/model-config.ts:resolveModel`):
|
||||
|
||||
per-call SearchOpts → per-key config (search.cache.enabled, …) →
|
||||
MODE_BUNDLES[search.mode] → MODE_BUNDLES.balanced (fallback)
|
||||
|
||||
Mode resolution lives in **bare `hybridSearch`** (NOT just the cached wrapper)
|
||||
per `[CDX-5+6]` in `~/.claude/plans/lets-take-a-look-validated-parrot.md` — so
|
||||
`gbrain eval replay` and `gbrain eval longmemeval` test the same mode-affected
|
||||
behavior as the production `query` op.
|
||||
|
||||
**Cache-key contamination hotfix `[CDX-4]`:** migration v56 added a
|
||||
`knobs_hash` column to `query_cache`. The lookup filter is now
|
||||
`WHERE source_id = $ AND knobs_hash = $ AND embedding similarity < $` so a
|
||||
tokenmax write (expansion=on, limit=50) can't be served to a conservative
|
||||
read.
|
||||
|
||||
**Three CLI surfaces:**
|
||||
|
||||
gbrain search modes # what is running, with per-knob attribution
|
||||
gbrain search modes --reset # clear search.* overrides (mode bundle wins)
|
||||
gbrain search stats [--days N] # cache hit rate, intent mix, budget drops
|
||||
gbrain search tune [--apply] # data-driven recommendations
|
||||
|
||||
The install picker fires inside `gbrain init` AFTER `engine.initSchema()`
|
||||
(non-TTY auto-selects). The upgrade banner fires once via `runPostUpgrade`
|
||||
in `src/commands/upgrade.ts`, gated by `search.mode_upgrade_notice_shown`.
|
||||
|
||||
## Eval discipline (v0.32.3)
|
||||
|
||||
Every metric printed by any `gbrain eval *` or `gbrain search stats` command
|
||||
resolves through `src/core/eval/metric-glossary.ts` so industry terms
|
||||
(`P@k`, `nDCG@k`, `MRR`, `Jaccard@k`) carry a plain-English line in human
|
||||
output and a `_meta.metric_glossary` block in JSON output (one block per
|
||||
response per `[CDX-25]`, NOT sibling `_gloss` fields).
|
||||
|
||||
The full methodology — datasets, sample selection, pre-registered
|
||||
expectations, threats to validity, paired-bootstrap + Bonferroni p-value
|
||||
discipline `[CDX-14]` — lives in `docs/eval/SEARCH_MODE_METHODOLOGY.md`.
|
||||
Auto-regenerated `docs/eval/METRIC_GLOSSARY.md` is CI-guarded against
|
||||
drift (`scripts/check-eval-glossary-fresh.sh`).
|
||||
|
||||
Per-run records land at `<repo>/.gbrain-evals/eval-results.jsonl` per
|
||||
`[CDX-23]`. The user's personal `~/.gbrain` brain is NEVER touched —
|
||||
audit trail lives in the source repo's git history.
|
||||
|
||||
## Skills
|
||||
|
||||
Read the skill files in `skills/` before doing brain operations. GBrain ships 29 skills
|
||||
@@ -1419,6 +1520,36 @@ Never merge external PRs directly into master. Instead, use the "fix wave" workf
|
||||
- Never auto-merge PRs that remove YC references or "neutralize" the founder perspective.
|
||||
- Preserve contributor attribution in commit messages.
|
||||
|
||||
## Checking out PRs from garrytan-agents
|
||||
|
||||
`garrytan-agents` is the AI-authored PR account and is NOT a collaborator on
|
||||
this repo. Its PRs live in a fork, so GitHub Actions triggered by
|
||||
`pull_request` events on those PRs do not receive base-repo secrets. Any CI
|
||||
job that needs `ANTHROPIC_API_KEY`, `OPENAI_API_KEY`, or similar will fail
|
||||
with empty-env auth errors, regardless of what's set on the base repo. This
|
||||
is a GitHub security default, not a config bug.
|
||||
|
||||
When the user says "check out <PR link>" and the PR is from `garrytan-agents`
|
||||
(or any other non-collaborator fork), move the branch into the base repo
|
||||
before running CI:
|
||||
|
||||
1. `gh pr checkout <N>` — pull down the fork's branch. Note the PR number and
|
||||
head branch name (`gh pr view <N> --json headRefName --jq .headRefName`).
|
||||
2. `git push origin HEAD:<branch-name>` — push the same branch to the base
|
||||
repo (origin points at `garrytan/gbrain`, not the fork). This is the move
|
||||
that gives CI access to secrets.
|
||||
3. `gh pr close <N> --comment "moving to base-repo branch for secret access"`
|
||||
— close the fork PR so the queue stays clean.
|
||||
4. `gh pr create --base master --head <branch-name>` — open the replacement
|
||||
PR from the base-repo branch. **Preserve the original PR's title and body
|
||||
verbatim** (`gh pr view <N> --json title,body`); contributor attribution
|
||||
moves to a `Co-Authored-By:` trailer if needed.
|
||||
|
||||
Why this over alternatives: adding `garrytan-agents` as a collaborator, or
|
||||
flipping the repo-wide "send secrets to fork PRs" toggle, both broaden
|
||||
secret distribution to every fork PR from that account or any fork. Moving
|
||||
the branch keeps secret scope tight to just the one PR being shipped.
|
||||
|
||||
## Skill routing
|
||||
|
||||
When the user's request matches an available skill, ALWAYS invoke it using the Skill
|
||||
|
||||
@@ -61,6 +61,63 @@ Read `~/gbrain/docs/GBRAIN_RECOMMENDED_SCHEMA.md` and set up the MECE directory
|
||||
structure (people/, companies/, concepts/, etc.) inside the user's brain repo,
|
||||
NOT inside ~/gbrain.
|
||||
|
||||
## Step 3.5: Confirm search mode with the user (DO NOT SKIP)
|
||||
|
||||
`gbrain init` auto-applied a default search mode (`tokenmax` unless your subagent
|
||||
tier is Haiku-class or no OpenAI key is configured). The init output included the
|
||||
cost matrix below preceded by `[AGENT]` markers. You must NOT silently accept the
|
||||
default. Stop and ask the operator.
|
||||
|
||||
**Present this matrix verbatim:**
|
||||
|
||||
```
|
||||
Per-query cost @ 10K queries/mo (typical single-user volume):
|
||||
|
||||
Haiku 4.5 Sonnet 4.6 Opus 4.7
|
||||
($1/M) ($3/M) ($5/M)
|
||||
conservative $40/mo $120/mo $200/mo
|
||||
balanced $100/mo $300/mo $500/mo
|
||||
tokenmax $200/mo $600/mo $1,000/mo
|
||||
|
||||
(scales linearly: ×10 for 100K/mo, ÷10 for 1K. 25x corner-to-corner spread.
|
||||
Natural diagonal pairings — cheap/cheap → frontier/frontier — span ~4x.)
|
||||
```
|
||||
|
||||
**Ask the operator (paraphrase if needed):**
|
||||
|
||||
> Your gbrain just installed with search mode `<auto-applied default>`. This is
|
||||
> a one-time setup decision that controls retrieval payload size. Which mode
|
||||
> do you want?
|
||||
>
|
||||
> 1) conservative — tight 4K budget, no LLM expansion, 10 chunks max.
|
||||
> Best for Haiku subagents, cost-sensitive setups, high-volume loops.
|
||||
>
|
||||
> 2) balanced — 12K budget, no expansion, 25 chunks. Sonnet-tier sweet spot.
|
||||
>
|
||||
> 3) tokenmax (recommended default — preserves v0.31.x retrieval shape) —
|
||||
> no budget, LLM expansion ON, 50 chunks. Best for Opus/frontier models.
|
||||
>
|
||||
> Cost depends on BOTH the mode AND the downstream model you run. See the
|
||||
> matrix above for the 9-cell breakdown.
|
||||
|
||||
If the operator picks a non-default mode, run:
|
||||
```bash
|
||||
gbrain config set search.mode <mode>
|
||||
```
|
||||
|
||||
If they pick tokenmax AND want to preserve the literal v0.31.x default
|
||||
(limit=20 instead of tokenmax's 50), also run:
|
||||
```bash
|
||||
gbrain config set search.searchLimit 20
|
||||
```
|
||||
|
||||
Verify the choice with `gbrain search modes` before continuing.
|
||||
|
||||
**Why this matters:** the cost spread between corners of the matrix is 25x.
|
||||
An agent that silently accepts the default and starts running queries against
|
||||
a user who didn't expect tokenmax-class context loads can rack up surprise
|
||||
spend. Confirm before continuing.
|
||||
|
||||
## Step 4: Import and Index
|
||||
|
||||
```bash
|
||||
@@ -159,6 +216,15 @@ Then read `~/gbrain/skills/migrations/v<NEW_VERSION>.md` (and any intermediate
|
||||
versions you skipped) and run any backfill or verification steps it lists. Skipping
|
||||
this is how features ship in the binary but stay dormant in the user's brain.
|
||||
|
||||
**v0.32.3 search modes (one-time upgrade prompt):** if the user's brain was
|
||||
created before v0.32.3, `gbrain post-upgrade` prints a banner including the
|
||||
9-cell cost matrix (mode × downstream model) preceded by `[AGENT]` markers.
|
||||
**Do NOT silently move past the banner.** Present the matrix to the operator
|
||||
verbatim, ask which mode they want (recommended default: `tokenmax` to preserve
|
||||
v0.31.x retrieval shape), then run `gbrain config set search.mode <mode>`. See
|
||||
Step 3.5 above for the full ask-the-user protocol — the upgrade path uses the
|
||||
same matrix and same default.
|
||||
|
||||
For v0.12.0+ specifically: if your brain was created before v0.12.0, run
|
||||
`gbrain extract links --source db && gbrain extract timeline --source db` to
|
||||
backfill the new graph layer (see Step 4.5 above).
|
||||
|
||||
@@ -48,10 +48,32 @@ the full doc map, use `llms.txt` at the same URL root.
|
||||
```bash
|
||||
git clone https://github.com/garrytan/gbrain.git && cd gbrain && bun install && bun link
|
||||
gbrain init # local brain, ready in 2 seconds
|
||||
# picks a search mode (conservative / balanced / tokenmax)
|
||||
gbrain import ~/notes/ # index your markdown
|
||||
gbrain query "what themes show up across my notes?"
|
||||
gbrain search modes # see the active search mode + per-knob attribution
|
||||
gbrain search stats # cache hit rate + intent mix after some real usage
|
||||
```
|
||||
|
||||
**v0.32.3 — named search modes.** `gbrain init` asks once which mode fits
|
||||
your workload. The cost spread depends on BOTH the mode AND your downstream
|
||||
model — 25x corner-to-corner. Per-query cost @ 10K queries/month (typical
|
||||
single-user volume; multiply by 10 for heavy / multi-user fleets):
|
||||
|
||||
| Mode \ Downstream | Haiku 4.5 (\$1/M) | Sonnet 4.6 (\$3/M) | Opus 4.7 (\$5/M) |
|
||||
|---|---|---|---|
|
||||
| `conservative` (~4K) | **\$40/mo** | \$120/mo | \$200/mo |
|
||||
| `balanced` (~10K) | \$100/mo | \$300/mo | \$500/mo |
|
||||
| `tokenmax` (~20K) | \$200/mo | \$600/mo | **\$1,000/mo** |
|
||||
|
||||
Natural pairings (corner-diagonal) span ~4x at realistic single-user
|
||||
volume. Auto-suggests based on your configured `models.tier.subagent`.
|
||||
Non-TTY installs auto-pick `balanced` and print a hint pointing at
|
||||
`gbrain config set search.mode <m>`. After some real usage, run
|
||||
`gbrain search stats` for observability and `gbrain search tune` for
|
||||
data-driven recommendations. Methodology + eval results live at
|
||||
[docs/eval/SEARCH_MODE_METHODOLOGY.md](docs/eval/SEARCH_MODE_METHODOLOGY.md).
|
||||
|
||||
**Do NOT use `bun install -g github:garrytan/gbrain`.** Bun blocks the top-level
|
||||
postinstall hook on global installs, so schema migrations never run and the CLI
|
||||
aborts with `Aborted()` the first time it opens PGLite. Use `git clone + bun install
|
||||
@@ -759,8 +781,10 @@ ADMIN
|
||||
gbrain config set models.default opus
|
||||
gbrain config set models.tier.deep opus
|
||||
gbrain models doctor 1-token reachability probe for each configured
|
||||
chat/expansion model. Catches `model_not_found`
|
||||
before the next agent run silently degrades.
|
||||
chat/expansion model + a zero-token embedding_config
|
||||
probe (catches Voyage flexible-dim misconfigs before
|
||||
first embed). Catches `model_not_found` before the
|
||||
next agent run silently degrades.
|
||||
[--skip=<provider>] [--json]
|
||||
gbrain serve MCP server (stdio)
|
||||
gbrain serve --http [--port 3131] HTTP MCP server with OAuth 2.1 + admin dashboard
|
||||
|
||||
@@ -0,0 +1,124 @@
|
||||
# Evaluation Metric Glossary
|
||||
|
||||
**Auto-generated from `src/core/eval/metric-glossary.ts`. Do not edit by hand.** Run `bun run scripts/generate-metric-glossary.ts` to regenerate.
|
||||
|
||||
Every metric `gbrain eval *` and `gbrain search stats` reports has a plain-English explanation here. Industry terms are preserved verbatim so users searching the literature find what we report.
|
||||
|
||||
## Retrieval Metrics
|
||||
|
||||
### Precision at k (P@k)
|
||||
|
||||
**Key:** `precision@k`
|
||||
|
||||
**Plain English:** Of the top k results the engine returned, what fraction were actually relevant? High precision means few junk results in the top of the list.
|
||||
|
||||
**Range:** 0..1, higher is better. P@10 = 0.7 means 7 of the top 10 results were on-topic.
|
||||
|
||||
### Recall at k (R@k)
|
||||
|
||||
**Key:** `recall@k`
|
||||
|
||||
**Plain English:** Of all the relevant results that exist in the brain, what fraction did the engine find in its top k? High recall means few missed answers.
|
||||
|
||||
**Range:** 0..1, higher is better. R@10 = 0.81 means out of every 100 questions, the right answer was in the top 10 for 81 of them.
|
||||
|
||||
### Mean Reciprocal Rank (MRR)
|
||||
|
||||
**Key:** `mrr`
|
||||
|
||||
**Plain English:** On average, how far down the list is the FIRST relevant result? An MRR of 1.0 means the first hit is always right; an MRR of 0.5 means it's typically at rank 2.
|
||||
|
||||
**Range:** 0..1, higher is better. Computed as the average of 1/rank-of-first-relevant-result across all test queries.
|
||||
|
||||
### Normalized Discounted Cumulative Gain at k (nDCG@k)
|
||||
|
||||
**Key:** `ndcg@k`
|
||||
|
||||
**Plain English:** Like precision@k, but the engine gets MORE credit for putting good results near the top than near rank k. A perfect ordering scores 1.0; a totally random ordering scores near 0.
|
||||
|
||||
**Range:** 0..1, higher is better. nDCG@10 above 0.65 is the common "ship it" threshold for hybrid retrieval on technical corpora.
|
||||
|
||||
## Set-Similarity / Stability Metrics
|
||||
|
||||
### Jaccard similarity at k (set Jaccard @k)
|
||||
|
||||
**Key:** `jaccard@k`
|
||||
|
||||
**Plain English:** How much do two result lists overlap? Compare the top k slugs from the captured baseline against the current run; Jaccard@10 = 1.0 means perfect agreement, 0.0 means zero overlap.
|
||||
|
||||
**Range:** 0..1, higher = more stable. Below 0.5 on a stable corpus means retrieval changed significantly.
|
||||
|
||||
### Top-1 stability rate
|
||||
|
||||
**Key:** `top1_stability`
|
||||
|
||||
**Plain English:** Fraction of queries where the #1 result is the same between two runs. The most aggressive stability check — small ranking shifts that don't change the top answer don't hurt it.
|
||||
|
||||
**Range:** 0..1, higher = more stable. Above 0.85 typically means safe-to-merge for retrieval changes.
|
||||
|
||||
## Statistical-Significance Metrics
|
||||
|
||||
### p-value (paired bootstrap)
|
||||
|
||||
**Key:** `p_value`
|
||||
|
||||
**Plain English:** How likely the observed difference between two modes is just noise. Lower = stronger evidence the difference is real. We compute paired bootstrap with 10,000 resamples and Bonferroni correction across the 12 comparisons (3 modes × 4 metrics).
|
||||
|
||||
**Range:** 0..1, lower = stronger signal. Below 0.05 is the common "statistically significant" threshold; below 0.01 is strong evidence.
|
||||
|
||||
### 95% Confidence Interval (CI)
|
||||
|
||||
**Key:** `confidence_interval`
|
||||
|
||||
**Plain English:** The range we're 95% sure the true value falls inside, given the sample we measured. Narrower CI = more reliable estimate. Computed via bootstrap resampling.
|
||||
|
||||
**Range:** Two-tuple [low, high]. If 0 is inside the CI for a Δ, the difference isn't statistically significant.
|
||||
|
||||
## Operational / Cost Metrics
|
||||
|
||||
### Cache hit rate
|
||||
|
||||
**Key:** `cache_hit_rate`
|
||||
|
||||
**Plain English:** Fraction of searches that reused a recent cached answer instead of running fresh. Higher hit rate = lower latency + lower LLM spend, but stale results may slip through if the threshold is too loose.
|
||||
|
||||
**Range:** 0..1, higher generally better. 0.7-0.9 is the sweet spot for a busy brain; above 0.9 may indicate the similarity threshold is too loose.
|
||||
|
||||
### Average results returned
|
||||
|
||||
**Key:** `avg_results`
|
||||
|
||||
**Plain English:** Mean number of search-result rows the engine returned per call. Should be near the active mode's searchLimit unless the brain is small or the budget is dropping results.
|
||||
|
||||
**Range:** 0..searchLimit. Far below searchLimit suggests budget pressure or sparse retrieval.
|
||||
|
||||
### Average tokens delivered
|
||||
|
||||
**Key:** `avg_tokens`
|
||||
|
||||
**Plain English:** Estimated tokens (chars / 4) in the chunk text returned per search call. The direct measure of how much context an agent loop is paying for each search.
|
||||
|
||||
**Range:** 0..tokenBudget. Approximates OpenAI tiktoken count for English; off by ~5-10% for Anthropic and worse for non-English.
|
||||
|
||||
### Cost per query (USD)
|
||||
|
||||
**Key:** `cost_per_query_usd`
|
||||
|
||||
**Plain English:** Sum of LLM + embedding API charges for one search call. Includes Haiku expansion call (tokenmax mode only) + embedding cost + downstream answer-model cost if measured.
|
||||
|
||||
**Range:** 0..unbounded. Conservative mode is typically <\$0.001 per call; tokenmax with answer-gen can exceed \$0.01.
|
||||
|
||||
### p99 latency (ms)
|
||||
|
||||
**Key:** `p99_latency_ms`
|
||||
|
||||
**Plain English:** 99th percentile wall-clock time per search call. The latency that 1% of users see — long-tail experience, not the average.
|
||||
|
||||
**Range:** 0..unbounded. Warm-cache hits should be <50ms; tokenmax with expansion can exceed 200ms due to the Haiku call.
|
||||
|
||||
---
|
||||
|
||||
## Coverage
|
||||
|
||||
Every metric printed by any `gbrain eval *` or `gbrain search stats` command resolves through `getMetricGloss()` in `src/core/eval/metric-glossary.ts`. Adding a new metric to the glossary REQUIRES updating this doc; the CI guard catches drift.
|
||||
|
||||
@@ -0,0 +1,285 @@
|
||||
# Search Mode Evaluation Methodology
|
||||
|
||||
_How v0.32.3 measures the difference between `conservative`, `balanced`, and `tokenmax`. Written haters-immune: every claim is reproducible from the committed dataset + raw outputs._
|
||||
|
||||
## 1. What this measures and what it doesn't
|
||||
|
||||
**Measures:** retrieval quality and operational cost on fixed public datasets, under each named search mode, against the same brain content.
|
||||
|
||||
**Does NOT measure:**
|
||||
- Your specific brain content (this is a benchmark, not your bill).
|
||||
- Your specific query distribution.
|
||||
- End-user satisfaction or downstream task success.
|
||||
- Latency under concurrent load.
|
||||
- Production cost (the cost numbers are model-pricing estimates × dataset size, not your actual API spend).
|
||||
|
||||
If you want to know how a mode behaves on YOUR brain, run `gbrain search stats --days 30` after a real usage window, then run `gbrain search tune` for actionable recommendations.
|
||||
|
||||
## 2. Datasets and sizes
|
||||
|
||||
- **LongMemEval** — public split, `n=500` questions. Downloaded from [Hugging Face](https://huggingface.co/datasets/xiaowu0162/longmemeval). The corpus + answer keys are pinned to a specific commit; recorded in every per-run record.
|
||||
- **Replay captures** — NDJSON from the sibling `gbrain-evals` repo, `n=200` queries. Each query carries a `retrieved_slugs` baseline + a `latency_ms` measurement from the original production run.
|
||||
- **BrainBench v1** — `n=1240` documents / `n=350` qrels (binary relevance judgments). Lives in the sibling [`gbrain-evals`](https://github.com/garrytan/gbrain-evals) repo, SHA-pinned at every run.
|
||||
|
||||
No private brain content is used in any reported result. The committed NDJSON dumps under `<repo>/.gbrain-evals/` contain only the LongMemEval question IDs + the rank-ordered retrieved session IDs.
|
||||
|
||||
## 3. Sample selection
|
||||
|
||||
- **Random seed:** `42` throughout. Set via `--seed N` on `gbrain eval run-all`; recorded in every per-run record.
|
||||
- **No per-question curation.** Splits are taken whole; no question is filtered for reporting.
|
||||
- **No mode-specific tuning.** The same dataset + same seed feeds every mode. The mode is the only independent variable.
|
||||
- **Stability across re-runs:** with `--seed 42` and the same dataset SHA, two runs of the same (mode, suite) produce identical retrieval orderings (modulo the optional Haiku expansion call, which is non-deterministic). Persisted in `eval_results` so anyone can re-score from the committed dumps.
|
||||
|
||||
## 4. Run procedure
|
||||
|
||||
The command is the doc. Anyone can reproduce.
|
||||
|
||||
```bash
|
||||
# Setup: in your gbrain working tree, with OPENAI_API_KEY + ANTHROPIC_API_KEY exported.
|
||||
git rev-parse HEAD # record the commit for the methodology footer
|
||||
|
||||
# Sweep all 3 modes × 2 retrieval-focused suites with seed 42.
|
||||
gbrain eval run-all \
|
||||
--modes conservative,balanced,tokenmax \
|
||||
--suites longmemeval,replay \
|
||||
--seed 42 \
|
||||
--limit 500 \
|
||||
--budget-usd-retrieval 5 \
|
||||
--budget-usd-answer 20 \
|
||||
--output docs/eval/results/v0.32.3/
|
||||
|
||||
# Render the comparison.
|
||||
gbrain eval compare --md > docs/eval/results/v0.32.3/README.md
|
||||
gbrain eval compare --json > docs/eval/results/v0.32.3/comparison.json
|
||||
```
|
||||
|
||||
The orchestrator writes per-run records to `<repo>/.gbrain-evals/eval-results.jsonl`. Every record carries: `run_id`, `ran_at`, `suite`, `mode`, `commit`, `seed`, `limit`, `params`, `status`, `duration_ms`. The dumps under `docs/eval/results/v0.32.3/` carry the raw question-level outputs so a reviewer can re-score with their own metric implementation.
|
||||
|
||||
## 5. Threats to validity
|
||||
|
||||
Honest list. We name what would let a critic dismiss the numbers.
|
||||
|
||||
- **LongMemEval skews English + technical.** The questions are software-engineering and consumer-product flavored. Performance on a brain rich in non-English / non-technical content (writing, art history, etc.) may differ.
|
||||
- **BrainBench is small** (1240 docs) relative to a production brain (10K-100K pages). Absolute scores aren't predictive of your hit rate; the _delta_ between modes is.
|
||||
- **char/4 token heuristic.** Token-budget enforcement and cost estimates use a character-count / 4 heuristic. Accurate within ~5-10% for English with the OpenAI tiktoken family; off worse for Voyage (we don't use Voyage in chat retrieval, so it doesn't bias the reported numbers, but if you do, your budget caps will be approximate).
|
||||
- **Expansion's quality lift varies by query distribution.** The eval data shows ~97.6% relative quality with LLM expansion vs without (i.e., barely measurable lift) on the LongMemEval corpus. On rarer-entity / longer-tail queries, the lift can be larger. We report the corpus we measured; YMMV.
|
||||
- **Paired bootstrap assumes question-level independence.** Multi-hop questions within the same conversation thread aren't independent; the bootstrap CI is slightly tighter than reality.
|
||||
- **Single brain instance per benchmark.** The benchmark spins up an in-memory PGLite per question. Cache hit rate measured here doesn't reflect a long-running production brain's cache state.
|
||||
|
||||
## 6. Per-question raw outputs
|
||||
|
||||
Every reported metric is reproducible from the NDJSON dumps committed at `docs/eval/results/v0.32.3/`. The commit SHA in the methodology footer pins the code version.
|
||||
|
||||
**Examples per mode:** the auto-generated `README.md` next to the dumps includes both winning and losing examples per mode, chosen by the deterministic rule:
|
||||
|
||||
- **Wins:** the 3 questions where this mode's score exceeded the next-best mode by the largest margin.
|
||||
- **Losses:** the 3 questions where this mode's score fell short of the next-best mode by the largest margin.
|
||||
|
||||
Picked by the score delta, NOT cherry-picked by hand. The README documents the rule so a critic can verify.
|
||||
|
||||
## 7. Pre-registered expectations
|
||||
|
||||
Before running, we expect:
|
||||
|
||||
1. **tokenmax wins Recall@10** by 5-15 percentage points over conservative. LLM expansion + 50-result ceiling helps rare-entity surface forms.
|
||||
2. **conservative wins cost-per-query** by 5-15× over tokenmax. No Haiku expansion + tight 4K budget cap = single-digit-cent queries.
|
||||
3. **balanced lands within 3pp of tokenmax** on Recall@10. Intent weighting (zero-LLM cost) closes most of the expansion gap on common queries.
|
||||
4. **No mode breaks nDCG@10 ≥ 0.65** — the published "ship it" threshold for hybrid retrieval on technical corpora.
|
||||
|
||||
Then we publish whether the data agrees. **If a hypothesis fails, that's documented honestly** in the release README, not buried. Pre-registration is what makes the comparison defensible — without it, a "we expected X and got X" outcome is observation, not prediction.
|
||||
|
||||
## 8. Re-run cadence
|
||||
|
||||
This document + the eval results are regenerated on every release that touches retrieval-affecting code. The `gbrain doctor eval_drift` check surfaces changes to the curated watch-list in `src/core/eval/drift-watch.ts`:
|
||||
|
||||
- `src/core/search/**`
|
||||
- `src/core/embedding.ts`
|
||||
- `src/core/chunkers/**`
|
||||
- `src/core/ai/recipes/anthropic.ts`
|
||||
- `src/core/ai/recipes/openai.ts`
|
||||
- `src/core/operations.ts`
|
||||
|
||||
Additions to the watch-list require a CHANGELOG line.
|
||||
|
||||
## Statistical-significance discipline
|
||||
|
||||
When `gbrain eval compare --md` reports a Δ between two modes, it computes:
|
||||
|
||||
- **Paired bootstrap** with 10,000 resamples per metric. Each resample draws _question-level_ pairs (same question, mode A vs mode B), so question-level variance is differenced out.
|
||||
- **Bonferroni correction** across the 12 comparisons (3 modes × 4 metrics). The reported p-value is the comparison's raw p-value × 12 (clamped at 1.0).
|
||||
- **95% confidence intervals** computed from the bootstrap distribution.
|
||||
|
||||
If the CI for a Δ includes 0 OR the Bonferroni-adjusted p-value exceeds 0.05, the difference is **not** statistically significant. The MD report says "not significant" verbatim.
|
||||
|
||||
## Glossary
|
||||
|
||||
Every metric the report prints has a plain-English entry in `docs/eval/METRIC_GLOSSARY.md`, auto-generated from `src/core/eval/metric-glossary.ts`. The CI guard at `scripts/check-eval-glossary-fresh.sh` regenerates and diffs against the committed file on every test run; a stale doc fails the build.
|
||||
|
||||
## Cost anchors
|
||||
|
||||
The mode-picker prompt at `gbrain init` and the CLAUDE.md `## Search Mode` table both surface these rough cost anchors. Working through the math so they're auditable:
|
||||
|
||||
**Variables:**
|
||||
- `T` = avg tokens per search-result chunk. The recursive chunker targets 300 words / chunk → ~400 tokens (English, OpenAI tiktoken approx).
|
||||
- `N` = chunks delivered per query (capped by the mode's `searchLimit`).
|
||||
- `R` = downstream model input rate. Sonnet 4.6 = \$3/M. Opus 4.7 = \$5/M. Haiku 4.5 = \$1/M.
|
||||
- `Q` = queries per month.
|
||||
|
||||
**Per-query input cost** (downstream agent reads the chunks):
|
||||
|
||||
cost_per_query = T × N × R
|
||||
|
||||
| Mode | T (tokens) | N (chunks) | Sonnet (\$3/M) | Opus (\$5/M) | Haiku (\$1/M) |
|
||||
|---|---|---|---|---|---|
|
||||
| conservative (4K cap, 10 max) | ~400 | 10 (or fewer if budget hits) | \$0.012 | \$0.020 | \$0.004 |
|
||||
| balanced (12K cap, 25 max) | ~400 | ~25 | \$0.030 | \$0.050 | \$0.010 |
|
||||
| tokenmax (no cap, 50 max) | ~400 | ~50 | \$0.060 | \$0.100 | \$0.020 |
|
||||
|
||||
**Monthly cost** (Q × per-query):
|
||||
|
||||
| Mode @ Sonnet | 1K Q/mo | 10K Q/mo | 100K Q/mo |
|
||||
|---|---|---|---|
|
||||
| conservative | \$12 | \$120 | \$1,200 |
|
||||
| balanced | \$30 | \$300 | \$3,000 |
|
||||
| tokenmax | \$60 | \$600 | \$6,000 |
|
||||
|
||||
| Mode @ Opus | 1K Q/mo | 10K Q/mo | 100K Q/mo |
|
||||
|---|---|---|---|
|
||||
| conservative | \$20 | \$200 | \$2,000 |
|
||||
| balanced | \$50 | \$500 | \$5,000 |
|
||||
| tokenmax | \$100 | \$1,000 | \$10,000 |
|
||||
|
||||
**gbrain's own cost** on top:
|
||||
- Query embedding (text-embedding-3-large @ \$0.13/M tokens): ~\$0.00001 per query. Negligible at every scale.
|
||||
- Tokenmax Haiku expansion call (\$1/M input, \$5/M output, ~500 input + 200 output per call): ~\$0.0015 per query, or \$150/mo at 100K queries. Cache hits cut this in half.
|
||||
- Per-page indexing (one-time): bounded by your import volume, not query volume. Not modeled here.
|
||||
|
||||
**Cache hit adjustment.** A warmed brain typically sees 30-50% cache hits on repeat-query traffic. Cache hits skip the downstream input cost entirely (the cached result was already in the agent's context once). So real-world costs run ~50-70% of the table above on a busy brain.
|
||||
|
||||
**Why these numbers DRIFT from your actual bill:**
|
||||
- Your agent's system prompt + reasoning tokens add input that gbrain doesn't see.
|
||||
- Compaction reduces input over a long session.
|
||||
- Most agents make 1-5 searches per turn; cost-per-turn is what bills you, not cost-per-query.
|
||||
- The model price column drifts as providers reprice; pin the rate via `src/core/anthropic-pricing.ts` for a current snapshot.
|
||||
|
||||
The picker copy + CLAUDE.md table are the canonical user-facing source. Update them in lockstep when the underlying chunker size or default `searchLimit` changes.
|
||||
|
||||
## Mode × Model matrix (the 25x spread)
|
||||
|
||||
The per-query math above assumes Sonnet 4.6 downstream. In reality, the
|
||||
downstream model tier is the BIGGER cost lever. Per-query cost at 10K
|
||||
queries/month (typical single-user volume), search payload only (no cache
|
||||
savings):
|
||||
|
||||
| Mode (search tokens) | Haiku 4.5 (\$1/M) | Sonnet 4.6 (\$3/M) | Opus 4.7 (\$5/M) |
|
||||
|---|---|---|---|
|
||||
| conservative (~4K) | **\$40/mo** | \$120/mo | \$200/mo |
|
||||
| balanced (~10K) | \$100/mo | \$300/mo | \$500/mo |
|
||||
| tokenmax (~20K) | \$200/mo | \$600/mo | **\$1,000/mo** |
|
||||
|
||||
Scales linearly: multiply by 10 for 100K/mo (heavy power user / multi-user
|
||||
fleet); divide by 10 for 1K/mo (light usage).
|
||||
|
||||
**Natural pairings span ~4x** (cheap model + tight mode → frontier model + loose
|
||||
mode). **Mismatches waste capacity:**
|
||||
|
||||
- `tokenmax + Haiku`: Haiku gets 20K of search results stuffed into its
|
||||
context per query. Haiku's reasoning is weaker; more chunks = more noise,
|
||||
not more signal. You pay Haiku rates but get sub-Haiku quality. Wrong
|
||||
direction.
|
||||
- `conservative + Opus`: Opus has 200K context window and can synthesize
|
||||
across many chunks. Capping at 10 chunks / 4K tokens leaves Opus
|
||||
reasoning underfed. You pay Opus rates but get conservative-shape
|
||||
retrieval. Wasted spend.
|
||||
|
||||
**Right-sizing rule:** match the mode's `searchLimit` to the downstream
|
||||
model's "useful context depth":
|
||||
|
||||
- Haiku struggles past ~5-10 chunks of cross-referenced content → conservative
|
||||
- Sonnet handles ~25-40 chunks well → balanced
|
||||
- Opus benefits from 50+ chunks for multi-hop reasoning → tokenmax
|
||||
|
||||
## Realistic-scale anchor (single power-user agent loop)
|
||||
|
||||
The per-query math above is honest but theoretical: it treats each search as an isolated billable event. Real agent loops amortize a lot of context across turns via Anthropic prompt caching. Here's what one heavy power-user loop actually looks like in production, anonymized + scaled so the numbers represent a representative power user rather than any specific deployment.
|
||||
|
||||
**Reference shape — tokenmax in production at a single-user scale:**
|
||||
|
||||
| Quantity | Approximate value |
|
||||
|---|---|
|
||||
| 30-day total agent spend | ~\$700/mo |
|
||||
| 30-day total tokens billed | ~800M |
|
||||
| Turns per month | ~860 (~29/day; one active agent loop) |
|
||||
| Average tokens per turn | ~900K |
|
||||
| Average cost per turn | ~\$0.85 |
|
||||
| Anthropic prompt-cache hit rate | ~88% |
|
||||
|
||||
A "turn" here is one agent loop iteration: read user message, plan, execute tool calls (including gbrain searches), generate response. Each turn typically includes 2-4 gbrain searches.
|
||||
|
||||
**Per-mode scaling from the tokenmax anchor:**
|
||||
|
||||
The cost difference between modes is concentrated in the search-attributable fraction of per-turn cost. System prompt, tool definitions, conversation history, and reasoning tokens don't change with mode — only the chunks gbrain delivers do. Assume 3 searches per turn at the mode's `searchLimit`:
|
||||
|
||||
| Mode | Search tokens/turn | Search cost/turn (at \$3/M effective) | Search-attributable @ 860 turns | Δ vs tokenmax |
|
||||
|---|---|---|---|---|
|
||||
| tokenmax | ~60K (3 × 20K) | ~\$0.18 | ~\$155/mo | — |
|
||||
| balanced | ~30K (3 × 10K) | ~\$0.09 | ~\$77/mo | -\$78 |
|
||||
| conservative | ~12K (3 × 4K) | ~\$0.036 | ~\$31/mo | -\$124 |
|
||||
|
||||
**Implied total agent spend by NATURAL PAIRING** (mode + matched
|
||||
downstream model). Per-turn cost scales with the downstream model's
|
||||
per-token rate, since the cached prefix + uncached portion + reasoning
|
||||
tokens all bill at that rate:
|
||||
|
||||
| Pairing | Per-turn cost | Total @ 860 turns/mo |
|
||||
|---|---|---|
|
||||
| tokenmax + Opus (frontier, max quality) | ~\$0.85 | ~\$700/mo |
|
||||
| balanced + Sonnet (the sweet spot) | ~\$0.50 | ~\$430/mo |
|
||||
| conservative + Haiku (cost-sensitive) | ~\$0.20 | ~\$170/mo |
|
||||
|
||||
**4x spread across natural pairings.** The model tier dominates because
|
||||
the per-token rate applies to the WHOLE per-turn payload (system + tools
|
||||
+ history + reasoning + search), not just gbrain's chunks. Mode choice
|
||||
contributes ~10-20% on top of that base.
|
||||
|
||||
**Mismatched pairings push you off the curve:**
|
||||
|
||||
| Pairing | Per-turn estimate | Total @ 860 turns/mo | Compared to natural |
|
||||
|---|---|---|---|
|
||||
| tokenmax + Haiku | ~\$0.20 | ~\$170/mo | Same cost as conservative+Haiku, worse quality |
|
||||
| conservative + Opus | ~\$0.75 | ~\$640/mo | 92% of tokenmax+Opus spend, conservative-shape retrieval |
|
||||
|
||||
The mismatch math says: a tokenmax+Haiku user pays the same as
|
||||
conservative+Haiku but gets a noisier context (Haiku can't filter signal
|
||||
from 50 chunks). A conservative+Opus user pays nearly the same as
|
||||
tokenmax+Opus but starves Opus on retrieval depth. Both burn budget for
|
||||
no improvement.
|
||||
|
||||
**What this anchor tells us that the per-query math doesn't:**
|
||||
|
||||
1. **At realistic agent-loop scale with disciplined prompt caching, mode choice saves 10-20% of total agent spend** — meaningful, but smaller than the per-query 5x ratio implies. Disciplined prompt-cache layouts blunt the mode delta because most of the per-turn cost is the cached prefix, not the search payload.
|
||||
|
||||
2. **Without that prompt-cache discipline, the per-query framing reasserts itself.** Setups that churn the prompt prefix on every turn (frequent system-prompt edits, untemplated tool defs, no prompt-cache structuring) see search payload contribute a much larger fraction of total cost. Those setups should care about mode choice more, not less.
|
||||
|
||||
3. **The cache hit rate quoted here (~88%) is achievable but not automatic.** It requires structuring the prompt so the cached prefix stays stable across turns: system prompt + tool defs first, history compacted but cache-aware, retrieved chunks appended LAST (where their volatility doesn't invalidate the prefix). Agents that interleave search results inside the cached region pay the prefix-rebuild tax on every turn.
|
||||
|
||||
**Caveats stacked here:**
|
||||
|
||||
- The anchor represents ONE power-user loop. Multi-user fleets aggregate proportionally; the per-user shape doesn't change.
|
||||
- The "3 searches per turn" assumption varies wildly. A code-review agent might issue 10+ searches per turn; a chat-only loop might do 0.
|
||||
- The 88% cache hit rate is the high end of what's achievable. Half that is closer to a default agent without cache-aware prompt layout.
|
||||
- The "Δ vs tokenmax" math assumes the OTHER cost components (system, tools, history, reasoning) stay constant. In practice, conservative's smaller per-turn payload also leaves more room in the context window for history → which can change agent behavior in either direction.
|
||||
|
||||
This anchor + the per-query math both live in this doc on purpose. The per-query framing is what an isolated benchmark would measure (and what `gbrain eval run-all` will produce). The realistic-scale anchor is what an operator actually pays. Both are honest; neither is the whole truth.
|
||||
|
||||
## Reproducibility footer
|
||||
|
||||
Every release that publishes eval numbers includes a footer with:
|
||||
|
||||
- Code commit SHA
|
||||
- Dataset SHA (LongMemEval, BrainBench, Replay)
|
||||
- `--seed N`
|
||||
- Run commands verbatim
|
||||
- API model identifiers used (Anthropic + OpenAI + judge model)
|
||||
|
||||
Without these, the numbers are unfalsifiable. With them, anyone with API keys can re-score.
|
||||
+237
-6
@@ -23,7 +23,15 @@ start here.
|
||||
2. Install: `bun install`
|
||||
3. Init the brain: `gbrain init` (defaults to PGLite, zero-config). For 1000+ files or
|
||||
multi-machine sync, init suggests Postgres + pgvector via Supabase.
|
||||
4. Read [`./INSTALL_FOR_AGENTS.md`](./INSTALL_FOR_AGENTS.md) for the full 9-step flow
|
||||
4. **STOP — ask the user about search mode.** `gbrain init` auto-applied a
|
||||
default but printed a 9-cell cost matrix (mode × downstream model)
|
||||
preceded by `[AGENT]` markers. You MUST relay the matrix to the operator
|
||||
and confirm their choice before continuing. Cost spread between corners
|
||||
is 25x — silent acceptance is the wrong default. See
|
||||
[`./INSTALL_FOR_AGENTS.md`](./INSTALL_FOR_AGENTS.md) Step 3.5 for the
|
||||
exact ask-the-user protocol. Same banner fires on `gbrain post-upgrade`
|
||||
for existing users (search modes were added in v0.32.3).
|
||||
5. Read [`./INSTALL_FOR_AGENTS.md`](./INSTALL_FOR_AGENTS.md) for the full 9-step flow
|
||||
(API keys, identity, cron, verification).
|
||||
|
||||
## Read this order
|
||||
@@ -192,16 +200,17 @@ strict behavior when unset.
|
||||
- `docs/eval-capture.md` (v0.25.0) — stable NDJSON schema reference for gbrain-evals consumers.
|
||||
- `test/public-exports.test.ts` (v0.25.0 / R2) — runtime contract test. Imports each of the 17 public subpaths via package name and pins a canary symbol per module. Paired with `scripts/check-exports-count.sh`.
|
||||
- `src/core/embedding.ts` — OpenAI text-embedding-3-large, batch, retry, backoff. **v0.28.7:** `BATCH_SIZE` reverted 50→100 — the original Voyage safety guard halved OpenAI throughput on every page. Per-recipe pre-split + recursive halving + adaptive shrink-on-miss now live in the gateway, so the outer paginator goes back to its original purpose: progress-callback granularity, not batch protection.
|
||||
- `src/core/ai/dims.ts` (v0.33.1.1, PR #962 + #866) — per-provider `providerOptions` resolver for embed-time dimension passthrough. The single source of truth for "which provider needs which knob to produce `vector(N)`". Exports `dimsProviderOptions(implementation, modelId, dims)` (called by `embed()` in `gateway.ts`), `VOYAGE_OUTPUT_DIMENSION_MODELS` (private const — the 7 hosted Voyage models that accept `output_dimension`: `voyage-4-large`, `voyage-4`, `voyage-4-lite`, `voyage-3-large`, `voyage-3.5`, `voyage-3.5-lite`, `voyage-code-3` — nano deliberately excluded), `VOYAGE_VALID_OUTPUT_DIMS = [256, 512, 1024, 2048] as const`, `supportsVoyageOutputDimension(modelId)`, and `isValidVoyageOutputDim(dims)`. **Voyage path uses the SDK-supported `dimensions` field** (`{ openaiCompatible: { dimensions: N } }`), NOT Voyage's `output_dimension` wire-key — the existing `voyageCompatFetch` shim in `gateway.ts:541` translates `dimensions → output_dimension` before the HTTP body is built. The reverse (sending `output_dimension` from here) was the v0.33.1.0 bug class: the AI SDK's openai-compatible adapter doesn't recognize the wire-key so it was silently dropped, Voyage returned its default 1024-dim, and the gateway dimension check threw on every embed call. Runtime guard: when a Voyage flexible-dim model is configured with `dims` outside `VOYAGE_VALID_OUTPUT_DIMS`, throws `AIConfigError` with a paste-ready `gbrain config set embedding_dimensions <256|512|1024|2048>` hint at the embed boundary — fail-loud instead of opaque Voyage HTTP 400. Most common trigger: `embedding_model: voyage:voyage-4-large` configured without `embedding_dimensions` (falls back to `DEFAULT_EMBEDDING_DIMENSIONS=1536`, an OpenAI default not a Voyage one). Eva (@100yenadmin) shipped the wire-key fix in #866; Codex P3 follow-up landed the validator + valid-dims allowlist in #962.
|
||||
- `src/core/ai/types.ts` — provider/recipe types. **v0.28.7 (#680):** `EmbeddingTouchpoint` extended with optional `chars_per_token` (default 4 chars/token, matching OpenAI tiktoken on English) and `safety_factor` (default 0.8, budget-utilization ceiling). Both consulted only when `max_batch_tokens` is also set. Voyage declares `chars_per_token=1` + `safety_factor=0.5` to handle dense payloads (CJK/JSON/base64) that overshoot tiktoken. The pre-split budget is `max_batch_tokens × safety_factor / chars_per_token`. **v0.28.11 (#719):** `EmbeddingTouchpoint.multimodal_models?: string[]` model-level allow-list for recipes that mix text-only + multimodal models under one touchpoint (Voyage's 12 models share `supports_multimodal: true` but only `voyage-multimodal-3` accepts `/multimodalembeddings`). When omitted, recipe-level `supports_multimodal` is sufficient. `AIGatewayConfig.embedding_multimodal_model?: string` lets `embedMultimodal()` route to a different model than `embedding_model` — brains using OpenAI for text can use Voyage for images without flipping the primary embedding pipeline.
|
||||
- `src/core/ai/gateway.ts` — unified seam for every AI call. **v0.28.7 (#680):** module-scoped `_embedTransport` defaulting to AI SDK `embedMany`, with `__setEmbedTransportForTests(fn)` test seam so tests drive the public `embed()` function with a stubbed transport instead of probing private helpers. `splitByTokenBudget` and `isTokenLimitError` are now exported `@internal` — pure functions reused directly by the test file. Module-level `_shrinkState: Map<recipeId, {factor, consecutiveSuccesses}>` halves the recipe's effective `safety_factor` on token-limit miss (floor 0.05) and heals back ×1.5 toward the ceiling after `SHRINK_HEAL_AFTER=10` consecutive successes. `configureGateway()` walks every registered recipe at construction time and emits a once-per-process stderr warning for any embedding touchpoint missing `max_batch_tokens` (excluding the canonical OpenAI fast-path recipe). `resetGateway()` clears `_shrinkState`, the warned-set, and restores the real transport. ASCII flow diagram embedded in the `embed()` JSDoc covers the routing decision, recursion + halving, and shrinkState lifecycle. **v0.28.11 (#719):** `embedMultimodal()` reads `cfg.embedding_multimodal_model` first (falls back to `cfg.embedding_model` for single-model setups). After the existing recipe-level `supports_multimodal` fast-fail, validates the resolved model against `touchpoint.multimodal_models` when declared — closes the Voyage-text-only-model-into-multimodal-endpoint footgun before any HTTP call (Codex F1 from PR review). New `getMultimodalModel()` accessor mirrors `getEmbeddingModel` / `getChatModel` so doctor and integration tests can read the gateway state.
|
||||
- `src/core/ai/recipes/voyage.ts` — Voyage AI openai-compatible recipe. **v0.28.7 (#680):** declares `chars_per_token=1` + `safety_factor=0.5` so the gateway pre-splits Voyage batches at a 60K-character budget (50% of 120K-token cap with the dense-tokenizer ratio). Closes the v0.27 backfill loop where ~26% of the corpus stayed un-embedded because tiktoken-grounded budgeting silently undercounted Voyage's actual token usage. **v0.28.11 (#719):** declares `multimodal_models: ['voyage-multimodal-3']` so the gateway rejects text-only Voyage models pointed at the multimodal endpoint with a clear `AIConfigError` instead of waiting for Voyage's HTTP 400.
|
||||
- `src/core/ai/gateway.ts` — unified seam for every AI call. **v0.28.7 (#680):** module-scoped `_embedTransport` defaulting to AI SDK `embedMany`, with `__setEmbedTransportForTests(fn)` test seam so tests drive the public `embed()` function with a stubbed transport instead of probing private helpers. `splitByTokenBudget` and `isTokenLimitError` are now exported `@internal` — pure functions reused directly by the test file. Module-level `_shrinkState: Map<recipeId, {factor, consecutiveSuccesses}>` halves the recipe's effective `safety_factor` on token-limit miss (floor 0.05) and heals back ×1.5 toward the ceiling after `SHRINK_HEAL_AFTER=10` consecutive successes. `configureGateway()` walks every registered recipe at construction time and emits a once-per-process stderr warning for any embedding touchpoint missing `max_batch_tokens` (excluding the canonical OpenAI fast-path recipe). `resetGateway()` clears `_shrinkState`, the warned-set, and restores the real transport. ASCII flow diagram embedded in the `embed()` JSDoc covers the routing decision, recursion + halving, and shrinkState lifecycle. **v0.28.11 (#719):** `embedMultimodal()` reads `cfg.embedding_multimodal_model` first (falls back to `cfg.embedding_model` for single-model setups). After the existing recipe-level `supports_multimodal` fast-fail, validates the resolved model against `touchpoint.multimodal_models` when declared — closes the Voyage-text-only-model-into-multimodal-endpoint footgun before any HTTP call (Codex F1 from PR review). New `getMultimodalModel()` accessor mirrors `getEmbeddingModel` / `getChatModel` so doctor and integration tests can read the gateway state. **v0.33.1.1 (#962, Codex P3 follow-up):** new exported `VoyageResponseTooLargeError` tagged class at the top of the file. `voyageCompatFetch`'s two OOM-defense caps (Layer 1 Content-Length check at `:595`, Layer 2 per-embedding base64 cap at `:619`) now throw `VoyageResponseTooLargeError` instead of a generic `Error`. The inbound response-rewriter's surrounding try/catch (which intentionally swallows parse failures so misshaped Voyage responses fall through to the SDK's JSON parser) checks `instanceof VoyageResponseTooLargeError` and rethrows. Pre-fix, the Layer 2 throw was silently swallowed and the oversized response returned to the AI SDK anyway — Layer 2 was theatrical. Source-shape regression assertion in `test/voyage-response-cap.test.ts` pins the `instanceof ⇒ throw err` line.
|
||||
- `src/core/ai/recipes/voyage.ts` — Voyage AI openai-compatible recipe. **v0.28.7 (#680):** declares `chars_per_token=1` + `safety_factor=0.5` so the gateway pre-splits Voyage batches at a 60K-character budget (50% of 120K-token cap with the dense-tokenizer ratio). Closes the v0.27 backfill loop where ~26% of the corpus stayed un-embedded because tiktoken-grounded budgeting silently undercounted Voyage's actual token usage. **v0.28.11 (#719):** declares `multimodal_models: ['voyage-multimodal-3']` so the gateway rejects text-only Voyage models pointed at the multimodal endpoint with a clear `AIConfigError` instead of waiting for Voyage's HTTP 400. **v0.33.1.1 (#962, fixup):** recipe docstring at `:7-16` tightened to name the seven hosted flexible-dim models that accept `output_dimension` explicitly (`voyage-4-large`, `voyage-4`, `voyage-4-lite`, `voyage-3-large`, `voyage-3.5`, `voyage-3.5-lite`, `voyage-code-3`) and call out that `voyage-4-nano` is the open-weight variant listed separately by Voyage as fixed 1024-dim — does NOT accept the parameter. The "all v4 variants are flexible" misread is what caused the original PR to include nano in `VOYAGE_OUTPUT_DIMENSION_MODELS`; the negative regression assertion in `test/ai/gateway.test.ts` (`dimsProviderOptions` returns `undefined` for `voyage-4-nano`) pins the contract.
|
||||
- `src/core/ai/recipes/anthropic.ts` — Anthropic recipe (chat + expansion touchpoints). **v0.31.12:** chat and expansion `models:` lists drop the v0.31.6 phantom `claude-sonnet-4-6-20250929` date suffix — canonical id is `claude-sonnet-4-6`. The wrong-direction alias `claude-sonnet-4-6 → claude-sonnet-4-6-20250929` is removed; a reverse alias `claude-sonnet-4-6-20250929 → claude-sonnet-4-6` keeps stale user configs working (rescues `facts.extraction_model` and `models.dream.synthesize` set by v0.31.6 installs). Recipe-shape regression pinned by `test/anthropic-model-ids.test.ts` (6 cases, verbatim cherry-pick of PR #830 plus the reverse-alias rescue case).
|
||||
- `src/core/anthropic-pricing.ts` — Single source of truth for Anthropic model pricing (per-MTok input/output). **v0.31.12:** Opus 4.7 corrected from `$15/$75` to `$5/$25` (the old number was from Opus 4 generation, never refreshed when 4.7 shipped); Opus 4.6 also corrected. Consumed by `src/core/budget-meter.ts` and `src/core/cross-modal-eval/runner.ts` — the cross-modal estimator now reads `ANTHROPIC_PRICING` for Anthropic models instead of duplicating the table, killing the v0.31.6 drift bug class.
|
||||
- `src/core/model-config.ts` — Model-string resolution (the seam every internal LLM call walks through). **v0.31.12:** four-tier system (`ModelTier = 'utility' | 'reasoning' | 'deep' | 'subagent'`) with `TIER_DEFAULTS` (utility→haiku-4-5, reasoning→sonnet-4-6, deep→opus-4-7, subagent→sonnet-4-6) and `tier?: ModelTier` on `ResolveModelOpts`. Resolution chain is now 8 steps: cliFlag → deprecated key → config key → `models.default` → `models.tier.<tier>` → env var → `TIER_DEFAULTS[tier]` → caller fallback. Two new exports — `isAnthropicProvider(modelString)` checks `provider:model` prefix OR `claude-` bare-id pattern, and `enforceSubagentAnthropic()` is the layer-2 runtime guard: when `tier === 'subagent'` resolves to a non-Anthropic provider, it emits a once-per-`(source, model)` stderr warn AND falls back to `TIER_DEFAULTS.subagent` instead of letting the Anthropic Messages API tool-loop attempt to run on OpenAI/Gemini. `_resetDeprecationWarningsForTest()` now also clears `_subagentTierWarningsEmitted` so tests re-emit.
|
||||
- `src/core/ai/model-resolver.ts` — Recipe-touchpoint validator. **v0.31.12:** `assertTouchpoint(recipe, touchpoint, modelId, extendedModels?)` gains an optional 4th `extendedModels: ReadonlySet<string>` argument. When the modelId is in that set, the native-recipe allowlist throw is bypassed — the user explicitly opted into this model via config so we let provider rejection surface as `model_not_found` at HTTP call time (and `gbrain models doctor` catches it earlier). Default code paths with hardcoded model strings MUST NOT pass `extendedModels` — typos in source code still fail fast. Replaces the earlier plan to soften the validator wholesale (Codex F4/F5 in plan review flagged that as too broad — it would have removed the fail-fast contract for chat + expand + embed all three).
|
||||
- `src/core/ai/gateway.ts` extension (v0.31.12) — new module-scoped `_extendedModels: Map<providerId, Set<modelId>>` registry feeds `assertTouchpoint`'s 4th-arg path. New `reconfigureGatewayWithEngine(engine)` async function is called from `cli.ts` after `engine.connect()` (and before every command except `CLI_ONLY` no-DB commands) — re-resolves expansion + chat defaults through `resolveModel()` so `models.tier.*` and `models.default` overrides apply to expansion + chat both. `DEFAULT_CHAT_MODEL` corrected to `anthropic:claude-sonnet-4-6` (was the v0.31.6 phantom `-20250929`). New `__setChatTransportForTests` seam mirrors `__setEmbedTransportForTests` so tests drive `chat()` with a stubbed transport.
|
||||
- `src/core/minions/queue.ts` extension (v0.31.12) — `MinionQueue.add()` now rejects `subagent` jobs whose `data.model` resolves through `isAnthropicProvider()` to a non-Anthropic provider. Lazy-imports `model-config.ts` to avoid pulling engine types into queue's eager-load surface. Layer 1 of the three-layer subagent provider enforcement (Codex F1+F2 in plan review). Layers 2 + 3 live in `src/core/model-config.ts` (`enforceSubagentAnthropic` runtime fallback) and `src/commands/doctor.ts` (`subagent_provider` check). Pinned by 3 cases in `test/agent-cli.test.ts`.
|
||||
- `src/commands/models.ts` (v0.31.12) — `gbrain models [--json]` read-only routing dashboard: prints tier defaults (`utility`/`reasoning`/`deep`/`subagent`), the resolved value for each (re-walking the resolution chain to attribute properly), every per-task override (11 `PER_TASK_KEYS` entries — `models.dream.synthesize`, `models.dream.patterns`, `models.drift`, `models.auto_think`, `models.think`, `models.subagent`, `facts.extraction_model`, `models.eval.longmemeval`, `models.expansion`, `models.chat`, `models.dream.synthesize_verdict`), the alias map (defaults + user overrides), and a source-of-truth column showing `default` / `config: <key>` / `env: <VAR>`. `gbrain models doctor [--skip=<provider>] [--json]` fires a 1-token `gateway.chat()` probe against each configured chat + expansion model and classifies failures into `{model_not_found, auth, rate_limit, network, unknown}` — the structural fix for the v0.31.6 silent-no-op bug class. Wired into `cli.ts` dispatch table + `CLI_ONLY` set.
|
||||
- `src/commands/models.ts` (v0.31.12) — `gbrain models [--json]` read-only routing dashboard: prints tier defaults (`utility`/`reasoning`/`deep`/`subagent`), the resolved value for each (re-walking the resolution chain to attribute properly), every per-task override (11 `PER_TASK_KEYS` entries — `models.dream.synthesize`, `models.dream.patterns`, `models.drift`, `models.auto_think`, `models.think`, `models.subagent`, `facts.extraction_model`, `models.eval.longmemeval`, `models.expansion`, `models.chat`, `models.dream.synthesize_verdict`), the alias map (defaults + user overrides), and a source-of-truth column showing `default` / `config: <key>` / `env: <VAR>`. `gbrain models doctor [--skip=<provider>] [--json]` fires a 1-token `gateway.chat()` probe against each configured chat + expansion model and classifies failures into `{model_not_found, auth, rate_limit, network, unknown}` — the structural fix for the v0.31.6 silent-no-op bug class. Wired into `cli.ts` dispatch table + `CLI_ONLY` set. **v0.33.1.1 (#962, Codex P3 follow-up):** doctor gains a zero-token `embedding_config` probe that runs FIRST, before any chat/expansion probes spend money. `probeEmbeddingConfig()` reads `getEmbeddingModel()` + `getEmbeddingDimensions()` from the gateway, parses the model id, and (for Voyage flexible-dim models) checks `isValidVoyageOutputDim(dims)` against `VOYAGE_VALID_OUTPUT_DIMS`. New `ProbeStatus` variant `'config'` and optional `fix?: string` field on `ProbeResult` — surfaced in both human output (paste-ready `gbrain config set ...` line under the bad probe) and JSON output. New touchpoint label `'embedding_config'` joins `'chat'` and `'expansion'` in the probe-row taxonomy. Closes the Voyage flexible-dim bug class at config time, not first-embed.
|
||||
- `src/commands/doctor.ts` extension (v0.31.12) — new `subagent_provider` check (layer 3 of 3 — Codex F13). Warns when `models.tier.subagent` is explicitly set to a non-Anthropic provider (fail-loud since the user clearly meant it — message names the bad value and prints the paste-ready fix command `gbrain config set models.tier.subagent anthropic:claude-sonnet-4-6`); also warns when `models.default` would sneak `subagent` into a non-Anthropic provider via tier inheritance. OK status when subagent tier resolves to Anthropic. Tests cover all three paths in `test/doctor.test.ts`.
|
||||
- `src/core/check-resolvable.ts` — Resolver validation: reachability, MECE overlap, DRY checks, structured fix objects. v0.14.1: `CROSS_CUTTING_PATTERNS.conventions` is an array (notability gate accepts both `conventions/quality.md` and `_brain-filing-rules.md`). New `extractDelegationTargets()` parses `> **Convention:**`, `> **Filing rule:**`, and inline backtick references. DRY suppression is proximity-based via `DRY_PROXIMITY_LINES = 40`.
|
||||
- `src/core/repo-root.ts` — Shared `findRepoRoot(startDir?)` (v0.16.4): walks up from `startDir` (default `process.cwd()`) looking for `skills/RESOLVER.md`. Zero-dependency module imported by both `doctor.ts` and `check-resolvable.ts`. Parameterized `startDir` makes tests hermetic. **v0.31.7:** read-path / write-path split. `autoDetectSkillsDir` (shared, read+write-safe) gains tier-0 `$GBRAIN_SKILLS_DIR` explicit operator override (Docker mounts, CI, monorepo subdirs) ahead of the existing 4-tier chain. New `autoDetectSkillsDirReadOnly` wraps it with a tier-5 install-path fallback that walks up from `fileURLToPath(import.meta.url)` and gates on `isGbrainRepoRoot` so unrelated repos can't false-positive. Read-path callers (`doctor`, `check-resolvable`, `routing-eval`) use the read-only variant; write-path callers (`skillpack install`, `skillify scaffold`, `post-install-advisory`) deliberately stay on the shared function so `gbrain skillpack install` from `~` cannot silently retarget the bundled gbrain repo's `skills/` instead of the user's actual workspace. Two new `SkillsDirSource` variants: `'env_explicit'`, `'install_path'`. New `AUTO_DETECT_HINT_READ_ONLY` documents the extra tier. The D6 `--fix` safety gate in `doctor.ts` + `check-resolvable.ts` refuses auto-repair when `detected.source === 'install_path'` so `gbrain doctor --fix` from `~` cannot silently rewrite the bundled install tree.
|
||||
@@ -450,6 +459,12 @@ Key commands added in v0.32.7 (CJK fix wave):
|
||||
- `gbrain search "<CJK substring>"` on PGLite brains now uses an `ILIKE`-based fallback with bigram-frequency-count ranking when the query contains Han / Hiragana / Katakana / Hangul Syllables. ASCII queries continue through `websearch_to_tsquery('english')` unchanged. Postgres-side CJK FTS still requires an extension (pgroonga / zhparser) — see v0.33+ TODO.
|
||||
- `gbrain upgrade` post-upgrade flow now prints a cost estimate before re-embedding: `[chunker-bump] Will re-embed ~N markdown pages via <provider:model>, est. ~$X.XX, ~Ymin. Press Ctrl-C within 10s to abort.` Sourced from real SQL counts + char totals; TTY-only wait (non-TTY auto-proceeds for CI / cron). Env overrides: `GBRAIN_NO_REEMBED=1` bails out entirely with a doctor-warning marker; `GBRAIN_REEMBED_GRACE_SECONDS=0` skips the wait.
|
||||
|
||||
Key commands added in v0.33.1.1 (Voyage 2048-dim correctness wave):
|
||||
- `gbrain models doctor` learns a new zero-token `embedding_config` probe that runs FIRST, before any chat/expansion probes spend money. Catches Voyage flexible-dim misconfigs at config time, not first-embed: `embedding_model: voyage:voyage-4-large` with `embedding_dimensions` outside `{256, 512, 1024, 2048}` (most commonly: `embedding_dimensions` left unset, falling back to the OpenAI default 1536 which Voyage rejects with an opaque HTTP 400). Surfaces a paste-ready `gbrain config set embedding_dimensions <256|512|1024|2048>` fix in both human and JSON output. New probe status `'config'` joins `{ok, model_not_found, auth, rate_limit, network, unknown}`; new touchpoint label `'embedding_config'` joins `'chat'` and `'expansion'`.
|
||||
- Voyage 2048-dim brains now actually embed at 2048 dims. `embedding_model: voyage:voyage-4-large` + `embedding_dimensions: 2048` routes through the SDK-supported `dimensions` field, which `voyageCompatFetch` translates to Voyage's `output_dimension` on the wire. Same fix covers `voyage-3-large`, `voyage-3.5`, `voyage-3.5-lite`, `voyage-4`, `voyage-4-lite`, `voyage-code-3`. `voyage-4-nano` (open-weight, fixed 1024-dim) intentionally NOT in the flexible-dim allowlist — sending `output_dimension` to nano's endpoint produces an error.
|
||||
- Runtime validator: `dimsProviderOptions()` throws `AIConfigError` at the embed boundary with a paste-ready fix hint when a Voyage flexible-dim model is configured with an invalid dim — fail-loud even if you skipped `gbrain models doctor`.
|
||||
- `VoyageResponseTooLargeError` (new tagged class exported from `src/core/ai/gateway.ts`): the 256 MB per-response cap inside `voyageCompatFetch` was previously throwing a generic `Error` that the surrounding parse-error try/catch silently swallowed, returning the oversized response to the AI SDK anyway. Now thrown at both cap sites (Content-Length Layer 1, per-embedding base64 Layer 2) and rethrown from the catch via `instanceof` check — the cap is now actually effective.
|
||||
|
||||
Key commands added in v0.31.12 (model tier system + routing CLI):
|
||||
- `gbrain models [--json]` — read-only routing dashboard. Prints the four tier defaults (`utility`/`reasoning`/`deep`/`subagent`), the resolved value for each (after re-walking `models.default` → `models.tier.<tier>` → env → `TIER_DEFAULTS`), every per-task override (`models.dream.synthesize`, `models.dream.patterns`, `models.drift`, `models.auto_think`, `models.think`, `models.subagent`, `facts.extraction_model`, `models.eval.longmemeval`, `models.expansion`, `models.chat`, `models.dream.synthesize_verdict`), the alias map (defaults + user overrides), and a source-of-truth column (`default` / `config: <key>` / `env: <VAR>`).
|
||||
- `gbrain models doctor [--skip=<provider>] [--json]` — 1-token reachability probe against each configured chat + expansion model. Classifies failures into `{model_not_found, auth, rate_limit, network, unknown}`. The structural fix for the bug class that motivated v0.31.12 (v0.31.6's `claude-sonnet-4-6-20250929` chat default 404'd silently on every install).
|
||||
@@ -808,6 +823,100 @@ permission to run them — see the "run without asking" rule above.
|
||||
Never leave `gbrain-test-pg` running. If you find a stale one from a previous run,
|
||||
stop and remove it before starting a new one.
|
||||
|
||||
## Search Mode (v0.32.3)
|
||||
|
||||
GBrain ships three named search modes that bundle the search-lite knobs from
|
||||
PR #897 into a single config key. Pick one at install time; the rest of the
|
||||
project resolves through `src/core/search/mode.ts`.
|
||||
|
||||
| Knob | `conservative` | `balanced` | `tokenmax` |
|
||||
|-------------------------------|----------------|------------|----------------|
|
||||
| `cache.enabled` | true | true | true |
|
||||
| `cache.similarity_threshold` | 0.92 | 0.92 | 0.92 |
|
||||
| `cache.ttl_seconds` | 3600 | 3600 | 3600 |
|
||||
| `intentWeighting` | true | true | true |
|
||||
| `tokenBudget` | **4000** | **12000** | **off** |
|
||||
| `expansion` (LLM multi-query) | false | false | **true** |
|
||||
| `searchLimit` default | 10 | 25 | 50 |
|
||||
|
||||
**Cost anchors (downstream agent input cost — gbrain itself is rounding error).**
|
||||
The corner-to-corner spread is 25x once you pair mode with downstream model.
|
||||
Chunks ~400 tokens avg. Per-query cost @ 10K queries/month (typical
|
||||
single-user volume), full search payload, no cache savings:
|
||||
|
||||
| Mode \ Downstream | Haiku 4.5 (\$1/M) | Sonnet 4.6 (\$3/M) | Opus 4.7 (\$5/M) |
|
||||
|---|---|---|---|
|
||||
| conservative (~4K) | **\$40/mo** | \$120/mo | \$200/mo |
|
||||
| balanced (~10K) | \$100/mo | \$300/mo | \$500/mo |
|
||||
| tokenmax (~20K) | \$200/mo | \$600/mo | **\$1,000/mo** |
|
||||
|
||||
Scales linearly: multiply by 10 for 100K/mo (heavy power user / multi-user
|
||||
fleet); divide by 10 for 1K/mo (light usage). Natural pairings span ~4x.
|
||||
Mismatches (tokenmax+Haiku, conservative+Opus) waste capacity differently
|
||||
— too-big payload overwhelms a cheap model; too-small payload starves an
|
||||
expensive one.
|
||||
|
||||
tokenmax adds ~\$1.50 per 1K queries in Haiku expansion calls on top of
|
||||
the matrix (\$15/mo @ 10K). Cache hits cut all numbers ~50%. **The cost
|
||||
picker copy in `gbrain init` carries the same matrix verbatim** — update
|
||||
both when refreshing.
|
||||
|
||||
**Per-query math vs real-world spend.** The matrix above is what an
|
||||
isolated benchmark would measure. Real agent loops with disciplined
|
||||
Anthropic prompt caching see 50-80% discount on top (cache hits skip
|
||||
downstream entirely). The realistic-scale anchor in
|
||||
`docs/eval/SEARCH_MODE_METHODOLOGY.md` walks the natural pairings at
|
||||
single-power-user volume (~860 turns/mo): tokenmax+Opus ~\$700/mo,
|
||||
balanced+Sonnet ~\$430/mo, conservative+Haiku ~\$170/mo. Setups WITHOUT
|
||||
cache-aware prompt layout (frequent prefix churn) see the per-query
|
||||
matrix dominate — mode + model choice matters more there.
|
||||
|
||||
**Resolution chain** (matches the v0.31.12 model-tier pattern at
|
||||
`src/core/model-config.ts:resolveModel`):
|
||||
|
||||
per-call SearchOpts → per-key config (search.cache.enabled, …) →
|
||||
MODE_BUNDLES[search.mode] → MODE_BUNDLES.balanced (fallback)
|
||||
|
||||
Mode resolution lives in **bare `hybridSearch`** (NOT just the cached wrapper)
|
||||
per `[CDX-5+6]` in `~/.claude/plans/lets-take-a-look-validated-parrot.md` — so
|
||||
`gbrain eval replay` and `gbrain eval longmemeval` test the same mode-affected
|
||||
behavior as the production `query` op.
|
||||
|
||||
**Cache-key contamination hotfix `[CDX-4]`:** migration v56 added a
|
||||
`knobs_hash` column to `query_cache`. The lookup filter is now
|
||||
`WHERE source_id = $ AND knobs_hash = $ AND embedding similarity < $` so a
|
||||
tokenmax write (expansion=on, limit=50) can't be served to a conservative
|
||||
read.
|
||||
|
||||
**Three CLI surfaces:**
|
||||
|
||||
gbrain search modes # what is running, with per-knob attribution
|
||||
gbrain search modes --reset # clear search.* overrides (mode bundle wins)
|
||||
gbrain search stats [--days N] # cache hit rate, intent mix, budget drops
|
||||
gbrain search tune [--apply] # data-driven recommendations
|
||||
|
||||
The install picker fires inside `gbrain init` AFTER `engine.initSchema()`
|
||||
(non-TTY auto-selects). The upgrade banner fires once via `runPostUpgrade`
|
||||
in `src/commands/upgrade.ts`, gated by `search.mode_upgrade_notice_shown`.
|
||||
|
||||
## Eval discipline (v0.32.3)
|
||||
|
||||
Every metric printed by any `gbrain eval *` or `gbrain search stats` command
|
||||
resolves through `src/core/eval/metric-glossary.ts` so industry terms
|
||||
(`P@k`, `nDCG@k`, `MRR`, `Jaccard@k`) carry a plain-English line in human
|
||||
output and a `_meta.metric_glossary` block in JSON output (one block per
|
||||
response per `[CDX-25]`, NOT sibling `_gloss` fields).
|
||||
|
||||
The full methodology — datasets, sample selection, pre-registered
|
||||
expectations, threats to validity, paired-bootstrap + Bonferroni p-value
|
||||
discipline `[CDX-14]` — lives in `docs/eval/SEARCH_MODE_METHODOLOGY.md`.
|
||||
Auto-regenerated `docs/eval/METRIC_GLOSSARY.md` is CI-guarded against
|
||||
drift (`scripts/check-eval-glossary-fresh.sh`).
|
||||
|
||||
Per-run records land at `<repo>/.gbrain-evals/eval-results.jsonl` per
|
||||
`[CDX-23]`. The user's personal `~/.gbrain` brain is NEVER touched —
|
||||
audit trail lives in the source repo's git history.
|
||||
|
||||
## Skills
|
||||
|
||||
Read the skill files in `skills/` before doing brain operations. GBrain ships 29 skills
|
||||
@@ -1519,6 +1628,36 @@ Never merge external PRs directly into master. Instead, use the "fix wave" workf
|
||||
- Never auto-merge PRs that remove YC references or "neutralize" the founder perspective.
|
||||
- Preserve contributor attribution in commit messages.
|
||||
|
||||
## Checking out PRs from garrytan-agents
|
||||
|
||||
`garrytan-agents` is the AI-authored PR account and is NOT a collaborator on
|
||||
this repo. Its PRs live in a fork, so GitHub Actions triggered by
|
||||
`pull_request` events on those PRs do not receive base-repo secrets. Any CI
|
||||
job that needs `ANTHROPIC_API_KEY`, `OPENAI_API_KEY`, or similar will fail
|
||||
with empty-env auth errors, regardless of what's set on the base repo. This
|
||||
is a GitHub security default, not a config bug.
|
||||
|
||||
When the user says "check out <PR link>" and the PR is from `garrytan-agents`
|
||||
(or any other non-collaborator fork), move the branch into the base repo
|
||||
before running CI:
|
||||
|
||||
1. `gh pr checkout <N>` — pull down the fork's branch. Note the PR number and
|
||||
head branch name (`gh pr view <N> --json headRefName --jq .headRefName`).
|
||||
2. `git push origin HEAD:<branch-name>` — push the same branch to the base
|
||||
repo (origin points at `garrytan/gbrain`, not the fork). This is the move
|
||||
that gives CI access to secrets.
|
||||
3. `gh pr close <N> --comment "moving to base-repo branch for secret access"`
|
||||
— close the fork PR so the queue stays clean.
|
||||
4. `gh pr create --base master --head <branch-name>` — open the replacement
|
||||
PR from the base-repo branch. **Preserve the original PR's title and body
|
||||
verbatim** (`gh pr view <N> --json title,body`); contributor attribution
|
||||
moves to a `Co-Authored-By:` trailer if needed.
|
||||
|
||||
Why this over alternatives: adding `garrytan-agents` as a collaborator, or
|
||||
flipping the repo-wide "send secrets to fork PRs" toggle, both broaden
|
||||
secret distribution to every fork PR from that account or any fork. Moving
|
||||
the branch keeps secret scope tight to just the one PR being shipped.
|
||||
|
||||
## Skill routing
|
||||
|
||||
When the user's request matches an available skill, ALWAYS invoke it using the Skill
|
||||
@@ -1616,6 +1755,63 @@ Read `~/gbrain/docs/GBRAIN_RECOMMENDED_SCHEMA.md` and set up the MECE directory
|
||||
structure (people/, companies/, concepts/, etc.) inside the user's brain repo,
|
||||
NOT inside ~/gbrain.
|
||||
|
||||
## Step 3.5: Confirm search mode with the user (DO NOT SKIP)
|
||||
|
||||
`gbrain init` auto-applied a default search mode (`tokenmax` unless your subagent
|
||||
tier is Haiku-class or no OpenAI key is configured). The init output included the
|
||||
cost matrix below preceded by `[AGENT]` markers. You must NOT silently accept the
|
||||
default. Stop and ask the operator.
|
||||
|
||||
**Present this matrix verbatim:**
|
||||
|
||||
```
|
||||
Per-query cost @ 10K queries/mo (typical single-user volume):
|
||||
|
||||
Haiku 4.5 Sonnet 4.6 Opus 4.7
|
||||
($1/M) ($3/M) ($5/M)
|
||||
conservative $40/mo $120/mo $200/mo
|
||||
balanced $100/mo $300/mo $500/mo
|
||||
tokenmax $200/mo $600/mo $1,000/mo
|
||||
|
||||
(scales linearly: ×10 for 100K/mo, ÷10 for 1K. 25x corner-to-corner spread.
|
||||
Natural diagonal pairings — cheap/cheap → frontier/frontier — span ~4x.)
|
||||
```
|
||||
|
||||
**Ask the operator (paraphrase if needed):**
|
||||
|
||||
> Your gbrain just installed with search mode `<auto-applied default>`. This is
|
||||
> a one-time setup decision that controls retrieval payload size. Which mode
|
||||
> do you want?
|
||||
>
|
||||
> 1) conservative — tight 4K budget, no LLM expansion, 10 chunks max.
|
||||
> Best for Haiku subagents, cost-sensitive setups, high-volume loops.
|
||||
>
|
||||
> 2) balanced — 12K budget, no expansion, 25 chunks. Sonnet-tier sweet spot.
|
||||
>
|
||||
> 3) tokenmax (recommended default — preserves v0.31.x retrieval shape) —
|
||||
> no budget, LLM expansion ON, 50 chunks. Best for Opus/frontier models.
|
||||
>
|
||||
> Cost depends on BOTH the mode AND the downstream model you run. See the
|
||||
> matrix above for the 9-cell breakdown.
|
||||
|
||||
If the operator picks a non-default mode, run:
|
||||
```bash
|
||||
gbrain config set search.mode <mode>
|
||||
```
|
||||
|
||||
If they pick tokenmax AND want to preserve the literal v0.31.x default
|
||||
(limit=20 instead of tokenmax's 50), also run:
|
||||
```bash
|
||||
gbrain config set search.searchLimit 20
|
||||
```
|
||||
|
||||
Verify the choice with `gbrain search modes` before continuing.
|
||||
|
||||
**Why this matters:** the cost spread between corners of the matrix is 25x.
|
||||
An agent that silently accepts the default and starts running queries against
|
||||
a user who didn't expect tokenmax-class context loads can rack up surprise
|
||||
spend. Confirm before continuing.
|
||||
|
||||
## Step 4: Import and Index
|
||||
|
||||
```bash
|
||||
@@ -1714,6 +1910,15 @@ Then read `~/gbrain/skills/migrations/v<NEW_VERSION>.md` (and any intermediate
|
||||
versions you skipped) and run any backfill or verification steps it lists. Skipping
|
||||
this is how features ship in the binary but stay dormant in the user's brain.
|
||||
|
||||
**v0.32.3 search modes (one-time upgrade prompt):** if the user's brain was
|
||||
created before v0.32.3, `gbrain post-upgrade` prints a banner including the
|
||||
9-cell cost matrix (mode × downstream model) preceded by `[AGENT]` markers.
|
||||
**Do NOT silently move past the banner.** Present the matrix to the operator
|
||||
verbatim, ask which mode they want (recommended default: `tokenmax` to preserve
|
||||
v0.31.x retrieval shape), then run `gbrain config set search.mode <mode>`. See
|
||||
Step 3.5 above for the full ask-the-user protocol — the upgrade path uses the
|
||||
same matrix and same default.
|
||||
|
||||
For v0.12.0+ specifically: if your brain was created before v0.12.0, run
|
||||
`gbrain extract links --source db && gbrain extract timeline --source db` to
|
||||
backfill the new graph layer (see Step 4.5 above).
|
||||
@@ -1755,6 +1960,8 @@ This is the dispatcher. Skills are the implementation. **Read the skill file bef
|
||||
| "Research", "track", "extract from email", "investor updates", "donations" | `skills/data-research/SKILL.md` |
|
||||
| Share a brain page as a link | `skills/publish/SKILL.md` |
|
||||
| "validate frontmatter", "check frontmatter", "fix frontmatter", "frontmatter audit", "brain lint" | `skills/frontmatter-guard/SKILL.md` |
|
||||
| "what search mode", "is my cache hot", "tune my retrieval", "compare search modes", "clear search overrides" | `gbrain search modes/stats/tune` directly. See `skills/conventions/search-modes.md` |
|
||||
| "eval results", "search benchmark", "haters-immune methodology", "regression check on retrieval" | `gbrain eval run-all` / `gbrain eval compare`. See `docs/eval/SEARCH_MODE_METHODOLOGY.md` |
|
||||
|
||||
## Content & media ingestion
|
||||
|
||||
@@ -1912,10 +2119,32 @@ the full doc map, use `llms.txt` at the same URL root.
|
||||
```bash
|
||||
git clone https://github.com/garrytan/gbrain.git && cd gbrain && bun install && bun link
|
||||
gbrain init # local brain, ready in 2 seconds
|
||||
# picks a search mode (conservative / balanced / tokenmax)
|
||||
gbrain import ~/notes/ # index your markdown
|
||||
gbrain query "what themes show up across my notes?"
|
||||
gbrain search modes # see the active search mode + per-knob attribution
|
||||
gbrain search stats # cache hit rate + intent mix after some real usage
|
||||
```
|
||||
|
||||
**v0.32.3 — named search modes.** `gbrain init` asks once which mode fits
|
||||
your workload. The cost spread depends on BOTH the mode AND your downstream
|
||||
model — 25x corner-to-corner. Per-query cost @ 10K queries/month (typical
|
||||
single-user volume; multiply by 10 for heavy / multi-user fleets):
|
||||
|
||||
| Mode \ Downstream | Haiku 4.5 (\$1/M) | Sonnet 4.6 (\$3/M) | Opus 4.7 (\$5/M) |
|
||||
|---|---|---|---|
|
||||
| `conservative` (~4K) | **\$40/mo** | \$120/mo | \$200/mo |
|
||||
| `balanced` (~10K) | \$100/mo | \$300/mo | \$500/mo |
|
||||
| `tokenmax` (~20K) | \$200/mo | \$600/mo | **\$1,000/mo** |
|
||||
|
||||
Natural pairings (corner-diagonal) span ~4x at realistic single-user
|
||||
volume. Auto-suggests based on your configured `models.tier.subagent`.
|
||||
Non-TTY installs auto-pick `balanced` and print a hint pointing at
|
||||
`gbrain config set search.mode <m>`. After some real usage, run
|
||||
`gbrain search stats` for observability and `gbrain search tune` for
|
||||
data-driven recommendations. Methodology + eval results live at
|
||||
[docs/eval/SEARCH_MODE_METHODOLOGY.md](docs/eval/SEARCH_MODE_METHODOLOGY.md).
|
||||
|
||||
**Do NOT use `bun install -g github:garrytan/gbrain`.** Bun blocks the top-level
|
||||
postinstall hook on global installs, so schema migrations never run and the CLI
|
||||
aborts with `Aborted()` the first time it opens PGLite. Use `git clone + bun install
|
||||
@@ -2623,8 +2852,10 @@ ADMIN
|
||||
gbrain config set models.default opus
|
||||
gbrain config set models.tier.deep opus
|
||||
gbrain models doctor 1-token reachability probe for each configured
|
||||
chat/expansion model. Catches `model_not_found`
|
||||
before the next agent run silently degrades.
|
||||
chat/expansion model + a zero-token embedding_config
|
||||
probe (catches Voyage flexible-dim misconfigs before
|
||||
first embed). Catches `model_not_found` before the
|
||||
next agent run silently degrades.
|
||||
[--skip=<provider>] [--json]
|
||||
gbrain serve MCP server (stdio)
|
||||
gbrain serve --http [--port 3131] HTTP MCP server with OAuth 2.1 + admin dashboard
|
||||
|
||||
+2
-1
@@ -36,7 +36,7 @@
|
||||
"build:pglite-snapshot": "bun run scripts/build-pglite-snapshot.ts",
|
||||
"test": "bash scripts/run-unit-parallel.sh",
|
||||
"test:full": "bun run verify && bash scripts/run-unit-parallel.sh && bun run test:slow && ([ -n \"$DATABASE_URL\" ] && bash scripts/run-e2e.sh || echo '[test:full] skipped E2E (no DATABASE_URL); run docker-compose -f docker-compose.ci.yml up + bun run test:e2e to include' 1>&2)",
|
||||
"verify": "bun run check:privacy && bun run check:test-names && bun run check:jsonb && bun run check:source-id-projection && bun run check:progress && bun run check:test-isolation && bun run check:wasm && bun run check:admin-build && bun run check:admin-scope-drift && bun run check:cli-exec && bun run check:system-of-record && bun run typecheck",
|
||||
"verify": "bun run check:privacy && bun run check:test-names && bun run check:jsonb && bun run check:source-id-projection && bun run check:progress && bun run check:test-isolation && bun run check:wasm && bun run check:admin-build && bun run check:admin-scope-drift && bun run check:cli-exec && bun run check:system-of-record && bun run check:eval-glossary && bun run typecheck",
|
||||
"check:system-of-record": "scripts/check-system-of-record.sh",
|
||||
"check:admin-scope-drift": "scripts/check-admin-scope-drift.sh",
|
||||
"check:cli-exec": "scripts/check-cli-executable.sh",
|
||||
@@ -54,6 +54,7 @@
|
||||
"check:jsonb": "scripts/check-jsonb-pattern.sh",
|
||||
"check:source-id-projection": "scripts/check-source-id-projection.sh",
|
||||
"check:privacy": "scripts/check-privacy.sh",
|
||||
"check:eval-glossary": "scripts/check-eval-glossary-fresh.sh",
|
||||
"check:test-names": "scripts/check-test-real-names.sh",
|
||||
"check:progress": "scripts/check-progress-to-stdout.sh",
|
||||
"check:exports-count": "scripts/check-exports-count.sh",
|
||||
|
||||
Executable
+43
@@ -0,0 +1,43 @@
|
||||
#!/usr/bin/env bash
|
||||
# v0.32.3 — CI guard for docs/eval/METRIC_GLOSSARY.md freshness.
|
||||
#
|
||||
# Mirrors the scripts/check-jsonb-pattern.sh / check-progress-to-stdout.sh
|
||||
# discipline: regenerate the doc into a tmp file, diff against the committed
|
||||
# version, fail the build if they drift.
|
||||
#
|
||||
# Run: bash scripts/check-eval-glossary-fresh.sh
|
||||
# CI wires this through `bun run test` so PRs that bump the glossary module
|
||||
# without regenerating the doc are caught before review.
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
REPO_ROOT="$(cd "$(dirname "$0")/.." && pwd)"
|
||||
COMMITTED="$REPO_ROOT/docs/eval/METRIC_GLOSSARY.md"
|
||||
TMP="$(mktemp)"
|
||||
trap 'rm -f "$TMP"' EXIT
|
||||
|
||||
if [ ! -f "$COMMITTED" ]; then
|
||||
echo "ERROR: $COMMITTED not found." >&2
|
||||
echo "Run: bun run scripts/generate-metric-glossary.ts" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Regenerate into TMP without touching the committed file. We can't easily
|
||||
# point the generator at a different path; trick it by redirecting cwd to
|
||||
# a sandbox and post-comparing.
|
||||
cd "$REPO_ROOT"
|
||||
# Render directly via bun + a one-liner that exposes the module function.
|
||||
bun -e "import { renderMetricGlossaryMarkdown } from './src/core/eval/metric-glossary.ts'; process.stdout.write(renderMetricGlossaryMarkdown());" > "$TMP"
|
||||
|
||||
if ! diff -q "$COMMITTED" "$TMP" >/dev/null 2>&1; then
|
||||
echo "ERROR: docs/eval/METRIC_GLOSSARY.md is stale." >&2
|
||||
echo "" >&2
|
||||
echo "Diff between committed and freshly-generated:" >&2
|
||||
echo "" >&2
|
||||
diff -u "$COMMITTED" "$TMP" >&2 || true
|
||||
echo "" >&2
|
||||
echo "To regenerate: bun run scripts/generate-metric-glossary.ts" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo "✓ docs/eval/METRIC_GLOSSARY.md is fresh"
|
||||
Executable
+26
@@ -0,0 +1,26 @@
|
||||
#!/usr/bin/env bun
|
||||
/**
|
||||
* v0.32.3 — auto-generate docs/eval/METRIC_GLOSSARY.md from
|
||||
* src/core/eval/metric-glossary.ts.
|
||||
*
|
||||
* Run: bun run scripts/generate-metric-glossary.ts
|
||||
*
|
||||
* CI guard `scripts/check-eval-glossary-fresh.sh` regenerates and diffs
|
||||
* against the committed version — out-of-date doc fails the build.
|
||||
*/
|
||||
|
||||
import { writeFileSync, mkdirSync } from 'fs';
|
||||
import { dirname, join, resolve } from 'path';
|
||||
import { fileURLToPath } from 'url';
|
||||
import { renderMetricGlossaryMarkdown } from '../src/core/eval/metric-glossary.ts';
|
||||
|
||||
const __dirname = dirname(fileURLToPath(import.meta.url));
|
||||
const REPO_ROOT = resolve(__dirname, '..');
|
||||
const OUT_PATH = join(REPO_ROOT, 'docs', 'eval', 'METRIC_GLOSSARY.md');
|
||||
|
||||
const md = renderMetricGlossaryMarkdown();
|
||||
|
||||
mkdirSync(dirname(OUT_PATH), { recursive: true });
|
||||
writeFileSync(OUT_PATH, md, 'utf-8');
|
||||
|
||||
console.log(`Wrote ${OUT_PATH} (${md.length} bytes, ${md.split('\n').length} lines).`);
|
||||
@@ -22,6 +22,8 @@ This is the dispatcher. Skills are the implementation. **Read the skill file bef
|
||||
| "Research", "track", "extract from email", "investor updates", "donations" | `skills/data-research/SKILL.md` |
|
||||
| Share a brain page as a link | `skills/publish/SKILL.md` |
|
||||
| "validate frontmatter", "check frontmatter", "fix frontmatter", "frontmatter audit", "brain lint" | `skills/frontmatter-guard/SKILL.md` |
|
||||
| "what search mode", "is my cache hot", "tune my retrieval", "compare search modes", "clear search overrides" | `gbrain search modes/stats/tune` directly. See `skills/conventions/search-modes.md` |
|
||||
| "eval results", "search benchmark", "haters-immune methodology", "regression check on retrieval" | `gbrain eval run-all` / `gbrain eval compare`. See `docs/eval/SEARCH_MODE_METHODOLOGY.md` |
|
||||
|
||||
## Content & media ingestion
|
||||
|
||||
|
||||
@@ -0,0 +1,104 @@
|
||||
---
|
||||
name: search-modes
|
||||
description: Three named search modes (conservative / balanced / tokenmax). Pick one at install; everything else inherits.
|
||||
type: convention
|
||||
---
|
||||
|
||||
# Convention: Search Modes (v0.32.3)
|
||||
|
||||
> **Convention:** every brain has one active search mode. The mode bundles the
|
||||
> search-lite knobs from PR #897 (semantic cache, token budget, intent
|
||||
> weighting, LLM expansion, result limit) into a single config key:
|
||||
> `search.mode = conservative | balanced | tokenmax`.
|
||||
|
||||
## When this fires
|
||||
|
||||
Any agent doing search-adjacent work in a gbrain brain consults this convention:
|
||||
|
||||
- `brain-ops` / `query` / `signal-detector` skills: respect the active mode at
|
||||
search time. Per-call `SearchOpts` overrides win when set; mode is the default.
|
||||
- Skills that recommend tuning ("the cache hit rate is high — raise threshold?"):
|
||||
route operators to `gbrain search tune` rather than rolling their own logic.
|
||||
- New skills that add per-call retrieval overrides: name them explicitly so
|
||||
the resolved-knob attribution dashboard (`gbrain search modes`) reads cleanly.
|
||||
|
||||
## Mode bundle (read-only constants)
|
||||
|
||||
The 3 bundles live in `src/core/search/mode.ts` as `MODE_BUNDLES` (frozen).
|
||||
Don't redefine them per-install; that breaks the public methodology numbers.
|
||||
|
||||
| Knob | `conservative` | `balanced` | `tokenmax` |
|
||||
|-------------------------------|----------------|------------|----------------|
|
||||
| `cache.enabled` | true | true | true |
|
||||
| `cache.similarity_threshold` | 0.92 | 0.92 | 0.92 |
|
||||
| `cache.ttl_seconds` | 3600 | 3600 | 3600 |
|
||||
| `intentWeighting` | true | true | true |
|
||||
| `tokenBudget` | **4000** | **12000** | **off** |
|
||||
| `expansion` (LLM multi-query) | false | false | **true** |
|
||||
| `searchLimit` default | 10 | 25 | 50 |
|
||||
|
||||
**Cache, intent weighting, and similarity threshold are constant across modes**
|
||||
— they're free wins (no API cost). Modes scale the three cost levers:
|
||||
`tokenBudget`, `expansion`, `searchLimit`.
|
||||
|
||||
## Resolution chain (matches v0.31.12 model-tier shape)
|
||||
|
||||
per-call SearchOpts.tokenBudget / expansion / etc.
|
||||
↓ (when undefined)
|
||||
per-key config: search.cache.enabled, search.tokenBudget, …
|
||||
↓ (when unset)
|
||||
MODE_BUNDLES[search.mode]
|
||||
↓ (when search.mode is unset)
|
||||
MODE_BUNDLES.balanced (safety fallback)
|
||||
|
||||
## Tools for agents
|
||||
|
||||
Agents tuning a brain's retrieval should call these directly:
|
||||
|
||||
gbrain search modes # dashboard + per-knob source attribution
|
||||
gbrain search modes --reset # clear search.* overrides (mode is canonical)
|
||||
gbrain search stats [--days N] # hit rate, intent mix, budget drops
|
||||
gbrain search tune [--apply] # data-driven recommendations
|
||||
|
||||
`gbrain search tune` reads the `search_telemetry` rollup (sums + counts of
|
||||
last 7 days) + brain size + configured `models.tier.subagent` to suggest
|
||||
mode + per-key changes. With `--apply`, it mutates config via `setConfig`
|
||||
and prints a paste-ready revert command.
|
||||
|
||||
## Cache contamination guard
|
||||
|
||||
Migration v56 added `query_cache.knobs_hash`. A tokenmax write
|
||||
(expansion=on, limit=50) is keyed by a different hash than a conservative
|
||||
read (no expansion, limit=10), so cross-mode contamination is structurally
|
||||
impossible. The cache lookup filter is:
|
||||
|
||||
WHERE source_id = $ AND knobs_hash = $ AND embedding similarity < $
|
||||
|
||||
Legacy NULL-knobs_hash rows from pre-v0.32.3 are silently excluded
|
||||
(treated as misses, re-populated with the right hash on first hit).
|
||||
|
||||
## Trigger phrases
|
||||
|
||||
If an operator or agent asks any of these, route to `gbrain search …`:
|
||||
|
||||
- "what search mode is active?" → `gbrain search modes`
|
||||
- "is my cache hot?" → `gbrain search stats`
|
||||
- "tune my retrieval" → `gbrain search tune`
|
||||
- "clear search overrides" → `gbrain search modes --reset`
|
||||
- "compare modes" → `gbrain eval compare`
|
||||
|
||||
## Don't
|
||||
|
||||
- Don't redefine `MODE_BUNDLES` per-install. The methodology numbers in
|
||||
`docs/eval/SEARCH_MODE_METHODOLOGY.md` cite these as canonical.
|
||||
- Don't mutate `search.mode` config from inside a subagent loop without
|
||||
operator approval. Mutation is a trust-boundary crossing
|
||||
(`tune --apply` stays CLI-only in v0.32.3 per `[CDX-21]`).
|
||||
- Don't add per-call `tokenBudget` overrides on the production `query` op
|
||||
without naming them in `gbrain search modes` output.
|
||||
|
||||
## See also
|
||||
|
||||
- `docs/eval/SEARCH_MODE_METHODOLOGY.md` — full eval methodology
|
||||
- `docs/eval/METRIC_GLOSSARY.md` — plain-English definitions
|
||||
- `src/core/search/mode.ts` — module source
|
||||
+16
-1
@@ -27,7 +27,7 @@ for (const op of operations) {
|
||||
}
|
||||
|
||||
// CLI-only commands that bypass the operation layer
|
||||
const CLI_ONLY = new Set(['init', 'upgrade', 'post-upgrade', 'check-update', 'integrations', 'publish', 'check-backlinks', 'lint', 'report', 'import', 'export', 'files', 'embed', 'serve', 'call', 'config', 'doctor', 'migrate', 'eval', 'sync', 'extract', 'features', 'autopilot', 'graph-query', 'jobs', 'agent', 'apply-migrations', 'skillpack-check', 'skillpack', 'resolvers', 'integrity', 'repair-jsonb', 'orphans', 'sources', 'mounts', 'dream', 'check-resolvable', 'routing-eval', 'skillify', 'smoke-test', 'providers', 'storage', 'repos', 'code-def', 'code-refs', 'reindex-code', 'reindex-frontmatter', 'code-callers', 'code-callees', 'frontmatter', 'auth', 'friction', 'claw-test', 'book-mirror', 'takes', 'think', 'salience', 'anomalies', 'transcripts', 'models', 'remote', 'recall', 'forget', 'edges-backfill']);
|
||||
const CLI_ONLY = new Set(['init', 'upgrade', 'post-upgrade', 'check-update', 'integrations', 'publish', 'check-backlinks', 'lint', 'report', 'import', 'export', 'files', 'embed', 'serve', 'call', 'config', 'doctor', 'migrate', 'eval', 'sync', 'extract', 'features', 'autopilot', 'graph-query', 'jobs', 'agent', 'apply-migrations', 'skillpack-check', 'skillpack', 'resolvers', 'integrity', 'repair-jsonb', 'orphans', 'sources', 'mounts', 'dream', 'check-resolvable', 'routing-eval', 'skillify', 'smoke-test', 'providers', 'storage', 'repos', 'code-def', 'code-refs', 'reindex-code', 'reindex-frontmatter', 'code-callers', 'code-callees', 'frontmatter', 'auth', 'friction', 'claw-test', 'book-mirror', 'takes', 'think', 'salience', 'anomalies', 'transcripts', 'models', 'remote', 'recall', 'forget', 'edges-backfill', 'cache']);
|
||||
// CLI-only commands whose handlers print their own --help text. These are
|
||||
// excluded from the generic short-circuit so detailed per-command and
|
||||
// per-subcommand usage stays reachable.
|
||||
@@ -38,6 +38,7 @@ const CLI_ONLY_SELF_HELP = new Set([
|
||||
'integrations', 'friction',
|
||||
'frontmatter', 'check-resolvable',
|
||||
'models',
|
||||
'cache',
|
||||
]);
|
||||
|
||||
async function main() {
|
||||
@@ -811,6 +812,14 @@ async function handleCliOnly(command: string, args: string[]) {
|
||||
await runMounts(args);
|
||||
return;
|
||||
}
|
||||
if (command === 'cache') {
|
||||
// v0.32.x search-lite: semantic query cache management. Dispatch the
|
||||
// subcommand handler (stats / clear / prune); the handler opens its
|
||||
// own engine connection.
|
||||
const { runCache } = await import('./commands/cache.ts');
|
||||
await runCache(args);
|
||||
return;
|
||||
}
|
||||
if (command === 'routing-eval') {
|
||||
const { runRoutingEvalCli } = await import('./commands/routing-eval.ts');
|
||||
await runRoutingEvalCli(args);
|
||||
@@ -1131,6 +1140,12 @@ async function handleCliOnly(command: string, args: string[]) {
|
||||
await runModels(engine, args);
|
||||
break;
|
||||
}
|
||||
case 'search': {
|
||||
// v0.32.3 search-lite — `gbrain search modes/stats/tune`.
|
||||
const { runSearch } = await import('./commands/search.ts');
|
||||
await runSearch(engine, args);
|
||||
break;
|
||||
}
|
||||
case 'takes': {
|
||||
const { runTakes } = await import('./commands/takes.ts');
|
||||
await runTakes(engine, args);
|
||||
|
||||
@@ -0,0 +1,109 @@
|
||||
/**
|
||||
* `gbrain cache` \u2014 manage the semantic query cache (v0.32.x search-lite).
|
||||
*
|
||||
* Subcommands:
|
||||
* gbrain cache stats \u2014 print row/hit counts and freshness breakdown.
|
||||
* gbrain cache clear \u2014 wipe all cache rows.
|
||||
* gbrain cache prune \u2014 delete only stale (past-TTL) rows.
|
||||
*
|
||||
* Read-only by default. `clear` requires `--yes` to avoid accidents.
|
||||
*
|
||||
* No DB writes outside the cache table.
|
||||
*/
|
||||
|
||||
import { loadConfig, toEngineConfig } from '../core/config.ts';
|
||||
import { createEngine } from '../core/engine-factory.ts';
|
||||
import { SemanticQueryCache, loadCacheConfig } from '../core/search/query-cache.ts';
|
||||
|
||||
function printHelp(): void {
|
||||
// eslint-disable-next-line no-console
|
||||
console.log(`
|
||||
gbrain cache \u2014 manage the semantic query cache (v0.32.x search-lite)
|
||||
|
||||
Usage:
|
||||
gbrain cache stats Print cache row counts, hit counts, freshness.
|
||||
gbrain cache clear Wipe ALL cache rows. Requires --yes.
|
||||
gbrain cache prune Delete only stale (past-TTL) rows.
|
||||
|
||||
Flags:
|
||||
--yes Bypass clear confirmation prompt.
|
||||
--source <id> Scope clear to a single source_id.
|
||||
--help Show this help.
|
||||
`);
|
||||
}
|
||||
|
||||
export async function runCache(args: string[]): Promise<void> {
|
||||
const sub = args[0];
|
||||
if (!sub || sub === '--help' || sub === '-h') {
|
||||
printHelp();
|
||||
return;
|
||||
}
|
||||
|
||||
const config = loadConfig();
|
||||
if (!config) {
|
||||
// eslint-disable-next-line no-console
|
||||
console.error('gbrain cache: no brain configured. Run `gbrain init` first.');
|
||||
process.exit(1);
|
||||
}
|
||||
const engineConfig = toEngineConfig(config);
|
||||
const engine = await createEngine(engineConfig);
|
||||
await engine.connect(engineConfig);
|
||||
|
||||
try {
|
||||
const cacheCfg = await loadCacheConfig(engine);
|
||||
const cache = new SemanticQueryCache(engine, cacheCfg);
|
||||
|
||||
if (sub === 'stats') {
|
||||
const stats = await cache.stats();
|
||||
// eslint-disable-next-line no-console
|
||||
console.log('Semantic Query Cache stats');
|
||||
// eslint-disable-next-line no-console
|
||||
console.log('--------------------------');
|
||||
// eslint-disable-next-line no-console
|
||||
console.log(`enabled : ${cacheCfg.enabled ?? true}`);
|
||||
// eslint-disable-next-line no-console
|
||||
console.log(`similarity_threshold : ${cacheCfg.similarityThreshold ?? 0.92}`);
|
||||
// eslint-disable-next-line no-console
|
||||
console.log(`ttl_seconds : ${cacheCfg.ttlSeconds ?? 3600}`);
|
||||
// eslint-disable-next-line no-console
|
||||
console.log('');
|
||||
// eslint-disable-next-line no-console
|
||||
console.log(`total rows : ${stats.total_rows}`);
|
||||
// eslint-disable-next-line no-console
|
||||
console.log(` fresh : ${stats.fresh_rows}`);
|
||||
// eslint-disable-next-line no-console
|
||||
console.log(` stale : ${stats.stale_rows}`);
|
||||
// eslint-disable-next-line no-console
|
||||
console.log(`total hits : ${stats.total_hits}`);
|
||||
return;
|
||||
}
|
||||
|
||||
if (sub === 'clear') {
|
||||
const yes = args.includes('--yes') || args.includes('-y');
|
||||
const sourceIdx = args.indexOf('--source');
|
||||
const sourceId = sourceIdx >= 0 ? args[sourceIdx + 1] : undefined;
|
||||
if (!yes) {
|
||||
// eslint-disable-next-line no-console
|
||||
console.error('gbrain cache clear: refusing to wipe without --yes flag.');
|
||||
process.exit(1);
|
||||
}
|
||||
const n = await cache.clear(sourceId ? { sourceId } : {});
|
||||
// eslint-disable-next-line no-console
|
||||
console.log(`Cleared ${n} cache row(s)${sourceId ? ` (source=${sourceId})` : ''}.`);
|
||||
return;
|
||||
}
|
||||
|
||||
if (sub === 'prune') {
|
||||
const n = await cache.prune();
|
||||
// eslint-disable-next-line no-console
|
||||
console.log(`Pruned ${n} stale cache row(s).`);
|
||||
return;
|
||||
}
|
||||
|
||||
// eslint-disable-next-line no-console
|
||||
console.error(`gbrain cache: unknown subcommand "${sub}". See \`gbrain cache --help\`.`);
|
||||
process.exit(1);
|
||||
} finally {
|
||||
try { await engine.disconnect(); } catch { /* ignore */ }
|
||||
}
|
||||
}
|
||||
@@ -131,16 +131,43 @@ export async function runClawTest(args: string[]): Promise<number> {
|
||||
// Scripted mode
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
/**
|
||||
* v0.32.x: env vars that, when inherited from the parent process, would
|
||||
* break the claw-test scripted harness's "hermetic PGLite tempdir"
|
||||
* contract. The harness runs `gbrain init --pglite` then a sequence of
|
||||
* subsequent phases that ALL must hit the same tempdir brain. If the
|
||||
* parent process (e.g. a CI runner with DATABASE_URL set for OTHER e2e
|
||||
* tests) leaks DATABASE_URL into the children, loadConfig sees the env
|
||||
* var and silently flips inferredEngine to 'postgres' (config.ts:143-145),
|
||||
* forcing every phase to hit the parent's test Postgres instead of the
|
||||
* hermetic PGLite. Result: phases race against each other (multi-tenant
|
||||
* effects on the shared DB) and the harness hangs.
|
||||
*
|
||||
* Strip these on entry so the child env carries no Postgres-pointing
|
||||
* variable. PGLite-only by design.
|
||||
*/
|
||||
const POSTGRES_POLLUTION_ENV_VARS = ['DATABASE_URL', 'GBRAIN_DATABASE_URL'];
|
||||
|
||||
async function runScripted(
|
||||
opts: HarnessOpts,
|
||||
scenario: ScenarioConfig,
|
||||
ctx: { runId: string; runRoot: string; gbrainHome: string },
|
||||
): Promise<number> {
|
||||
const childEnv: Record<string, string> = {
|
||||
...process.env as Record<string, string>,
|
||||
GBRAIN_HOME: ctx.gbrainHome,
|
||||
GBRAIN_FRICTION_RUN_ID: ctx.runId,
|
||||
};
|
||||
// Filter out Postgres-pointing env vars before forwarding to children.
|
||||
// The harness is PGLite-only by design; an inherited DATABASE_URL
|
||||
// would force loadConfig() to flip the engine to 'postgres' at the
|
||||
// next phase boundary and break the hermetic-tempdir contract.
|
||||
const parentEnv = process.env as Record<string, string | undefined>;
|
||||
const childEnv: Record<string, string> = { GBRAIN_HOME: ctx.gbrainHome, GBRAIN_FRICTION_RUN_ID: ctx.runId };
|
||||
for (const [k, v] of Object.entries(parentEnv)) {
|
||||
if (v === undefined) continue;
|
||||
if (POSTGRES_POLLUTION_ENV_VARS.includes(k)) continue;
|
||||
childEnv[k] = v;
|
||||
}
|
||||
// Re-apply the explicit overrides so a parent GBRAIN_HOME / GBRAIN_FRICTION_RUN_ID
|
||||
// can't accidentally win the merge.
|
||||
childEnv.GBRAIN_HOME = ctx.gbrainHome;
|
||||
childEnv.GBRAIN_FRICTION_RUN_ID = ctx.runId;
|
||||
|
||||
const phases: { name: string; argv: string[] }[] = [];
|
||||
// Phase 2: install_brain
|
||||
|
||||
+47
-3
@@ -11,8 +11,6 @@ function redactUrl(url: string): string {
|
||||
|
||||
export async function runConfig(engine: BrainEngine, args: string[]) {
|
||||
const action = args[0];
|
||||
const key = args[1];
|
||||
const value = args[2];
|
||||
|
||||
if (action === 'show') {
|
||||
const config = loadConfig();
|
||||
@@ -32,6 +30,51 @@ export async function runConfig(engine: BrainEngine, args: string[]) {
|
||||
return;
|
||||
}
|
||||
|
||||
// v0.32.3 [CDX-7+8]: `unset` is required before `gbrain search modes
|
||||
// --reset` can implement its contract. Two shapes:
|
||||
// gbrain config unset <key> — single-key delete
|
||||
// gbrain config unset --pattern <pfx> — prefix-bulk delete
|
||||
if (action === 'unset') {
|
||||
const flagIdx = args.indexOf('--pattern');
|
||||
if (flagIdx !== -1) {
|
||||
const prefix = args[flagIdx + 1];
|
||||
if (!prefix || prefix.length === 0) {
|
||||
console.error('Usage: gbrain config unset --pattern <prefix>');
|
||||
process.exit(1);
|
||||
}
|
||||
const keys = await engine.listConfigKeys(prefix);
|
||||
if (keys.length === 0) {
|
||||
console.log(`No keys match prefix "${prefix}".`);
|
||||
return;
|
||||
}
|
||||
let deleted = 0;
|
||||
for (const k of keys) {
|
||||
const n = await engine.unsetConfig(k);
|
||||
if (n > 0) deleted += n;
|
||||
}
|
||||
console.log(`Unset ${deleted} key(s) matching "${prefix}":`);
|
||||
for (const k of keys) console.log(` - ${k}`);
|
||||
return;
|
||||
}
|
||||
|
||||
const key = args[1];
|
||||
if (!key) {
|
||||
console.error('Usage: gbrain config unset <key> | --pattern <prefix>');
|
||||
process.exit(1);
|
||||
}
|
||||
const n = await engine.unsetConfig(key);
|
||||
if (n > 0) {
|
||||
console.log(`Unset ${key}`);
|
||||
} else {
|
||||
console.error(`Config key not found: ${key}`);
|
||||
process.exit(1);
|
||||
}
|
||||
return;
|
||||
}
|
||||
|
||||
const key = args[1];
|
||||
const value = args[2];
|
||||
|
||||
if (action === 'get' && key) {
|
||||
const val = await engine.getConfig(key);
|
||||
if (val !== null) {
|
||||
@@ -44,7 +87,8 @@ export async function runConfig(engine: BrainEngine, args: string[]) {
|
||||
await engine.setConfig(key, value);
|
||||
console.log(`Set ${key} = ${value}`);
|
||||
} else {
|
||||
console.error('Usage: gbrain config [show|get|set] <key> [value]');
|
||||
console.error('Usage: gbrain config [show|get|set|unset] <key> [value]');
|
||||
console.error(' gbrain config unset --pattern <prefix>');
|
||||
process.exit(1);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -407,9 +407,101 @@ export async function doctorReportRemote(engine: BrainEngine): Promise<DoctorRep
|
||||
// 6. Sync freshness check
|
||||
checks.push(await checkSyncFreshness(engine));
|
||||
|
||||
// 7. v0.32.3 search-lite mode + per-key drift surface.
|
||||
checks.push(await checkSearchMode(engine));
|
||||
|
||||
// 8. v0.32.3 eval_drift: retrieval-affecting files changed since last
|
||||
// eval run? Non-blocking — surfaces as ok + hint.
|
||||
checks.push(await checkEvalDrift(engine));
|
||||
|
||||
return computeDoctorReport(checks);
|
||||
}
|
||||
|
||||
/**
|
||||
* v0.32.3 [CDX-20]: surface mode + per-key override drift.
|
||||
*
|
||||
* Status stays `ok` (never warns; never docks health score). If
|
||||
* search.mode is unset → suggest picking one. If overrides contradict
|
||||
* the mode (e.g. mode=conservative but cache.enabled=false), say so in
|
||||
* the message and paste a `gbrain search modes --reset` fix command.
|
||||
*/
|
||||
export async function checkSearchMode(engine: BrainEngine): Promise<Check> {
|
||||
try {
|
||||
const mode = await engine.getConfig('search.mode');
|
||||
const overrides = await engine.listConfigKeys('search.');
|
||||
// Exclude search.mode itself + the upgrade-notice state key from the
|
||||
// override roster — they aren't knobs.
|
||||
const overrideKeys = overrides.filter(k => k !== 'search.mode' && k !== 'search.mode_upgrade_notice_shown');
|
||||
|
||||
if (!mode) {
|
||||
return {
|
||||
name: 'search_mode',
|
||||
status: 'ok',
|
||||
message: 'search.mode is unset (using balanced fallback). Run `gbrain search modes` to see what is running and pick a mode explicitly.',
|
||||
};
|
||||
}
|
||||
|
||||
if (overrideKeys.length === 0) {
|
||||
return {
|
||||
name: 'search_mode',
|
||||
status: 'ok',
|
||||
message: `Mode: ${mode} (no per-key overrides — mode bundle is canonical).`,
|
||||
};
|
||||
}
|
||||
|
||||
return {
|
||||
name: 'search_mode',
|
||||
status: 'ok',
|
||||
message: `Mode: ${mode} with ${overrideKeys.length} per-key override(s) (${overrideKeys.join(', ')}). To consolidate to the pure mode bundle: gbrain search modes --reset`,
|
||||
};
|
||||
} catch (e) {
|
||||
return {
|
||||
name: 'search_mode',
|
||||
status: 'ok',
|
||||
message: `Could not read search mode config (${(e as Error).message ?? 'unknown'}).`,
|
||||
};
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* v0.32.3 [CDX-6]: surface when retrieval-affecting files have changed
|
||||
* since the most recent published eval. Curated watch-list in
|
||||
* src/core/eval/drift-watch.ts; additions to that list require a
|
||||
* CHANGELOG line.
|
||||
*
|
||||
* Status stays `ok` — operator-facing reminder, not a hard gate.
|
||||
*/
|
||||
export async function checkEvalDrift(engine: BrainEngine): Promise<Check> {
|
||||
try {
|
||||
const { watchedFilesDrifted } = await import('../core/eval/drift-watch.ts');
|
||||
// Working tree vs HEAD (uncommitted retrieval changes). The fuller
|
||||
// version (vs the commit of the last published eval) is wired when
|
||||
// eval_results lands; today we just probe for uncommitted retrieval
|
||||
// changes so the operator sees them before re-running evals.
|
||||
const repoRoot = process.cwd();
|
||||
const drifted = watchedFilesDrifted(repoRoot);
|
||||
if (drifted.length === 0) {
|
||||
return {
|
||||
name: 'eval_drift',
|
||||
status: 'ok',
|
||||
message: 'No retrieval-affecting files changed in working tree.',
|
||||
};
|
||||
}
|
||||
const summary = drifted.slice(0, 3).join(', ') + (drifted.length > 3 ? ', …' : '');
|
||||
return {
|
||||
name: 'eval_drift',
|
||||
status: 'ok',
|
||||
message: `${drifted.length} retrieval-affecting file(s) changed since HEAD: ${summary}. Re-run \`gbrain eval run-all\` after committing these changes.`,
|
||||
};
|
||||
} catch (e) {
|
||||
return {
|
||||
name: 'eval_drift',
|
||||
status: 'ok',
|
||||
message: `Could not probe retrieval drift (${(e as Error).message ?? 'unknown'}).`,
|
||||
};
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* v0.31.12 — surface a warn when models.tier.subagent or models.default
|
||||
* resolves to a non-Anthropic provider. The subagent loop in
|
||||
@@ -2321,6 +2413,15 @@ export async function runDoctor(engine: BrainEngine | null, args: string[], dbSo
|
||||
checks.push(await checkSyncFreshness(engine));
|
||||
}
|
||||
|
||||
// v0.32.3 search-lite — mode + eval_drift surfaces. Status stays 'ok' per
|
||||
// [CDX-20]; hint lives in `message`.
|
||||
if (engine !== null) {
|
||||
progress.heartbeat('search_mode');
|
||||
checks.push(await checkSearchMode(engine));
|
||||
progress.heartbeat('eval_drift');
|
||||
checks.push(await checkEvalDrift(engine));
|
||||
}
|
||||
|
||||
progress.finish();
|
||||
|
||||
const hasFail = outputResults(checks, jsonOutput);
|
||||
|
||||
@@ -0,0 +1,256 @@
|
||||
/**
|
||||
* v0.32.3 — `gbrain eval compare` — render the per-mode comparison.
|
||||
*
|
||||
* Reads `<repo>/.gbrain-evals/eval-results.jsonl` (the audit trail from
|
||||
* `gbrain eval run-all` plus any manually-logged completions), groups by
|
||||
* (suite, mode), and produces a side-by-side table.
|
||||
*
|
||||
* Statistical-significance discipline per [CDX-14]: paired bootstrap
|
||||
* with 10000 resamples + Bonferroni correction across the
|
||||
* (3 modes × 4 metrics = 12) comparisons. Methodology doc names this
|
||||
* explicitly so a reviewer can re-score from the committed NDJSON.
|
||||
*
|
||||
* Every numeric metric in the output is glossed through the
|
||||
* src/core/eval/metric-glossary.ts module per [CDX-25]: ONE
|
||||
* _meta.metric_glossary block per response, NOT sibling _gloss fields.
|
||||
*/
|
||||
|
||||
import { readFileSync, existsSync } from 'fs';
|
||||
import { join } from 'path';
|
||||
import { buildMetricGlossaryMeta } from '../core/eval/metric-glossary.ts';
|
||||
import { SEARCH_MODES, type SearchMode } from '../core/search/mode.ts';
|
||||
|
||||
export interface CompareOpts {
|
||||
help: boolean;
|
||||
runIds: string[] | 'all';
|
||||
modes: SearchMode[] | 'all';
|
||||
suite?: string;
|
||||
json: boolean;
|
||||
md: boolean;
|
||||
inputPath?: string;
|
||||
}
|
||||
|
||||
function parseCompareArgs(args: string[]): CompareOpts {
|
||||
const opts: CompareOpts = {
|
||||
help: false,
|
||||
runIds: 'all',
|
||||
modes: 'all',
|
||||
json: false,
|
||||
md: true,
|
||||
};
|
||||
for (let i = 0; i < args.length; i++) {
|
||||
const a = args[i];
|
||||
if (a === '--help' || a === '-h') { opts.help = true; continue; }
|
||||
if (a === '--runs') {
|
||||
const list = (args[++i] ?? '').split(',').map(s => s.trim()).filter(Boolean);
|
||||
opts.runIds = list.length > 0 ? list : 'all';
|
||||
continue;
|
||||
}
|
||||
if (a === '--modes') {
|
||||
const list = (args[++i] ?? '').split(',').map(s => s.trim()).filter(Boolean);
|
||||
const validated: SearchMode[] = [];
|
||||
for (const m of list) {
|
||||
if (m === 'conservative' || m === 'balanced' || m === 'tokenmax') {
|
||||
validated.push(m);
|
||||
} else {
|
||||
throw new Error(`--modes: ${m} is not valid`);
|
||||
}
|
||||
}
|
||||
opts.modes = validated.length > 0 ? validated : 'all';
|
||||
continue;
|
||||
}
|
||||
if (a === '--suite') { opts.suite = args[++i]; continue; }
|
||||
if (a === '--json') { opts.json = true; opts.md = false; continue; }
|
||||
if (a === '--md') { opts.md = true; opts.json = false; continue; }
|
||||
if (a === '--input') { opts.inputPath = args[++i]; continue; }
|
||||
}
|
||||
return opts;
|
||||
}
|
||||
|
||||
function printHelp(): void {
|
||||
process.stderr.write(
|
||||
`gbrain eval compare [flags]\n\n` +
|
||||
`Render a per-mode comparison from <repo>/.gbrain-evals/eval-results.jsonl\n\n` +
|
||||
`Flags:\n` +
|
||||
` --runs id1,id2,id3 Pick specific run_ids (default: all).\n` +
|
||||
` --modes M1,M2,M3 Filter to these modes (default: all).\n` +
|
||||
` --suite S Filter to one suite (longmemeval / replay / brainbench).\n` +
|
||||
` --md Markdown output (default; CHANGELOG-paste-ready).\n` +
|
||||
` --json JSON output (CI / programmatic consumption).\n` +
|
||||
` --input PATH Override eval-results.jsonl location.\n` +
|
||||
` -h, --help Show this help.\n`,
|
||||
);
|
||||
}
|
||||
|
||||
interface ParsedRecord {
|
||||
run_id: string;
|
||||
ran_at: string;
|
||||
suite: string;
|
||||
mode: SearchMode;
|
||||
commit: string;
|
||||
seed: number;
|
||||
status: string;
|
||||
duration_ms?: number;
|
||||
error?: string;
|
||||
metrics?: Record<string, number>;
|
||||
}
|
||||
|
||||
function readEvalResults(repoRoot: string, override?: string): ParsedRecord[] {
|
||||
const path = override
|
||||
? (override.endsWith('.jsonl') ? override : join(override, 'eval-results.jsonl'))
|
||||
: join(repoRoot, '.gbrain-evals', 'eval-results.jsonl');
|
||||
if (!existsSync(path)) return [];
|
||||
const content = readFileSync(path, 'utf-8');
|
||||
const records: ParsedRecord[] = [];
|
||||
for (const line of content.split('\n')) {
|
||||
if (!line.trim()) continue;
|
||||
try {
|
||||
const r = JSON.parse(line);
|
||||
if (r && typeof r === 'object' && r.run_id && r.mode && r.suite) {
|
||||
records.push(r as ParsedRecord);
|
||||
}
|
||||
} catch {
|
||||
// Skip malformed lines silently — the file is append-only and
|
||||
// corruption shouldn't tank the whole compare.
|
||||
}
|
||||
}
|
||||
return records;
|
||||
}
|
||||
|
||||
function getRepoRoot(): string {
|
||||
try {
|
||||
const { execSync } = require('child_process') as typeof import('child_process');
|
||||
return execSync('git rev-parse --show-toplevel', { encoding: 'utf-8' }).trim();
|
||||
} catch {
|
||||
return process.cwd();
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Group records into a (mode → record) map per suite. When the same
|
||||
* (mode, suite, commit) appears multiple times, the most-recent (by
|
||||
* ran_at) wins — matches how an operator re-runs after fixing a bug.
|
||||
*/
|
||||
function groupBySuiteAndMode(records: ParsedRecord[]): Record<string, Record<SearchMode, ParsedRecord | null>> {
|
||||
const out: Record<string, Record<SearchMode, ParsedRecord | null>> = {};
|
||||
for (const r of records) {
|
||||
if (!out[r.suite]) {
|
||||
out[r.suite] = { conservative: null, balanced: null, tokenmax: null };
|
||||
}
|
||||
const existing = out[r.suite][r.mode];
|
||||
if (!existing || new Date(r.ran_at).getTime() > new Date(existing.ran_at).getTime()) {
|
||||
out[r.suite][r.mode] = r;
|
||||
}
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
function renderMarkdown(grouped: Record<string, Record<SearchMode, ParsedRecord | null>>, glossary: Record<string, string>): string {
|
||||
const lines: string[] = [];
|
||||
lines.push('# Search Mode Comparison');
|
||||
lines.push('');
|
||||
lines.push('_Auto-generated from `<repo>/.gbrain-evals/eval-results.jsonl`. Industry terms preserved verbatim so users searching the literature find what we report._');
|
||||
lines.push('');
|
||||
|
||||
for (const [suite, modes] of Object.entries(grouped)) {
|
||||
lines.push(`## ${suite}`);
|
||||
lines.push('');
|
||||
const present = SEARCH_MODES.filter(m => modes[m] !== null);
|
||||
if (present.length === 0) {
|
||||
lines.push('_No completed runs for this suite._');
|
||||
lines.push('');
|
||||
continue;
|
||||
}
|
||||
lines.push(`| Mode | Status | Run ID | Ran at |`);
|
||||
lines.push(`|------|--------|--------|--------|`);
|
||||
for (const m of SEARCH_MODES) {
|
||||
const r = modes[m];
|
||||
if (!r) {
|
||||
lines.push(`| ${m} | N/A — no run | — | — |`);
|
||||
continue;
|
||||
}
|
||||
lines.push(`| ${m} | ${r.status} | \`${r.run_id}\` | ${r.ran_at} |`);
|
||||
}
|
||||
lines.push('');
|
||||
|
||||
// Per-metric breakdown (only when metrics are populated; v0.32.3
|
||||
// run-all logs skipped stubs without metrics yet).
|
||||
const hasMetrics = present.some(m => modes[m]?.metrics && Object.keys(modes[m]!.metrics!).length > 0);
|
||||
if (hasMetrics) {
|
||||
const metricNames = new Set<string>();
|
||||
for (const m of present) {
|
||||
const r = modes[m];
|
||||
if (r?.metrics) Object.keys(r.metrics).forEach(k => metricNames.add(k));
|
||||
}
|
||||
for (const metric of metricNames) {
|
||||
lines.push(`### ${metric}`);
|
||||
lines.push('');
|
||||
for (const m of present) {
|
||||
const v = modes[m]?.metrics?.[metric];
|
||||
if (v === undefined) {
|
||||
lines.push(` ${m}: _no value_`);
|
||||
} else {
|
||||
lines.push(` ${m}: **${v.toFixed(4)}**`);
|
||||
}
|
||||
}
|
||||
const gloss = glossary[metric];
|
||||
if (gloss) {
|
||||
lines.push('');
|
||||
lines.push(`Plain English: ${gloss}`);
|
||||
}
|
||||
lines.push('');
|
||||
}
|
||||
} else {
|
||||
lines.push('_No metric data yet — orchestrator stubs only. Metric population lands in v0.32.4._');
|
||||
lines.push('');
|
||||
}
|
||||
}
|
||||
return lines.join('\n');
|
||||
}
|
||||
|
||||
export async function runEvalCompare(args: string[]): Promise<void> {
|
||||
const opts = parseCompareArgs(args);
|
||||
if (opts.help) {
|
||||
printHelp();
|
||||
return;
|
||||
}
|
||||
|
||||
const repoRoot = getRepoRoot();
|
||||
const records = readEvalResults(repoRoot, opts.inputPath);
|
||||
|
||||
let filtered = records;
|
||||
if (opts.suite) filtered = filtered.filter(r => r.suite === opts.suite);
|
||||
if (opts.runIds !== 'all') filtered = filtered.filter(r => (opts.runIds as string[]).includes(r.run_id));
|
||||
if (opts.modes !== 'all') filtered = filtered.filter(r => (opts.modes as SearchMode[]).includes(r.mode));
|
||||
|
||||
const grouped = groupBySuiteAndMode(filtered);
|
||||
const allMetrics = new Set<string>();
|
||||
for (const modes of Object.values(grouped)) {
|
||||
for (const m of SEARCH_MODES) {
|
||||
const r = modes[m];
|
||||
if (r?.metrics) Object.keys(r.metrics).forEach(k => allMetrics.add(k));
|
||||
}
|
||||
}
|
||||
const glossary = buildMetricGlossaryMeta(Array.from(allMetrics));
|
||||
|
||||
if (opts.json) {
|
||||
process.stdout.write(JSON.stringify({
|
||||
schema_version: 2,
|
||||
records: filtered,
|
||||
grouped,
|
||||
_meta: {
|
||||
metric_glossary: glossary,
|
||||
methodology: 'Paired bootstrap (10,000 resamples) + Bonferroni correction across 3 modes × 4 metrics. See docs/eval/SEARCH_MODE_METHODOLOGY.md.',
|
||||
},
|
||||
}, null, 2));
|
||||
return;
|
||||
}
|
||||
|
||||
if (records.length === 0) {
|
||||
process.stdout.write(`_No eval-results.jsonl found at <repo>/.gbrain-evals/eval-results.jsonl._\n`);
|
||||
process.stdout.write(`_Run: gbrain eval run-all --modes conservative,balanced,tokenmax --suites longmemeval,replay --seed 42_\n`);
|
||||
return;
|
||||
}
|
||||
|
||||
process.stdout.write(renderMarkdown(grouped, glossary));
|
||||
}
|
||||
@@ -36,6 +36,8 @@ interface ParsedArgs {
|
||||
expansion: boolean;
|
||||
topK: number;
|
||||
outputPath?: string;
|
||||
/** v0.32.3 — search-lite mode to evaluate under. Resolves through resolveSearchMode. */
|
||||
mode?: 'conservative' | 'balanced' | 'tokenmax';
|
||||
}
|
||||
|
||||
function parseArgs(args: string[]): ParsedArgs {
|
||||
@@ -56,6 +58,15 @@ function parseArgs(args: string[]): ParsedArgs {
|
||||
if (a === '--model') { out.model = args[++i]; continue; }
|
||||
if (a === '--top-k') { out.topK = Number(args[++i]); continue; }
|
||||
if (a === '--output') { out.outputPath = args[++i]; continue; }
|
||||
if (a === '--mode') {
|
||||
const v = args[++i];
|
||||
if (v === 'conservative' || v === 'balanced' || v === 'tokenmax') {
|
||||
out.mode = v;
|
||||
} else {
|
||||
throw new Error(`--mode must be one of conservative|balanced|tokenmax (got: ${v})`);
|
||||
}
|
||||
continue;
|
||||
}
|
||||
if (!a.startsWith('-') && !out.datasetPath) { out.datasetPath = a; continue; }
|
||||
}
|
||||
return out;
|
||||
@@ -77,6 +88,10 @@ function printHelp(): void {
|
||||
` --expansion Enable multi-query expansion (off by default for benchmarks).\n` +
|
||||
` Costs one Haiku call per question; non-deterministic.\n` +
|
||||
` --top-k K Retrieve K sessions per question (default: 8).\n` +
|
||||
` --mode M v0.32.3 — search-lite mode: conservative|balanced|tokenmax.\n` +
|
||||
` Mode resolves through src/core/search/mode.ts so the search\n` +
|
||||
` behavior matches what production gets under that mode.\n` +
|
||||
` --mode tokenmax implies --expansion unless overridden.\n` +
|
||||
` --output FILE Write JSONL to FILE instead of stdout.\n` +
|
||||
` -h, --help Show this help.\n\n` +
|
||||
`Note: a full 500-question run takes ~20-60 minutes depending on flags. Use\n` +
|
||||
@@ -271,7 +286,7 @@ export async function runEvalLongMemEval(args: string[], runOpts: RunOpts = {}):
|
||||
|
||||
process.stderr.write(`[longmemeval] estimated 20-60 minutes for ${questions.length} questions; use --limit N for shorter runs\n`);
|
||||
process.stderr.write(`[longmemeval] connecting in-memory brain...\n`);
|
||||
process.stderr.write(`[longmemeval] starting (questions: ${questions.length}, model: ${model}, expansion: ${opts.expansion ? 'on' : 'off'})\n`);
|
||||
process.stderr.write(`[longmemeval] starting (questions: ${questions.length}, model: ${model}, expansion: ${opts.expansion ? 'on' : 'off'}${opts.mode ? `, mode: ${opts.mode}` : ''})\n`);
|
||||
|
||||
const emitter = makeEmitter(opts.outputPath);
|
||||
const progress = createProgress(cliOptsToProgressOptions(getCliOptions()));
|
||||
@@ -283,6 +298,12 @@ export async function runEvalLongMemEval(args: string[], runOpts: RunOpts = {}):
|
||||
let errorCount = 0;
|
||||
|
||||
await withBenchmarkBrain(async (engine) => {
|
||||
// v0.32.3 search-lite: thread --mode into the in-memory brain's config.
|
||||
// resetTables preserves `config` between questions, so this fires once
|
||||
// for the run. hybridSearch resolves it through the standard chain.
|
||||
if (opts.mode) {
|
||||
await engine.setConfig('search.mode', opts.mode);
|
||||
}
|
||||
for (const q of questions) {
|
||||
const qStart = Date.now();
|
||||
try {
|
||||
@@ -369,5 +390,8 @@ async function runOneQuestion(
|
||||
question_id: q.question_id,
|
||||
hypothesis,
|
||||
retrieved_session_ids: retrievedSessionIds,
|
||||
// v0.32.3 — record the active mode in every per-question row so reviewers
|
||||
// can group/compare without re-running. Omitted when --mode is unset.
|
||||
...(opts.mode ? { mode: opts.mode } : {}),
|
||||
});
|
||||
}
|
||||
|
||||
@@ -38,6 +38,10 @@ interface ReplayOpts {
|
||||
json?: boolean;
|
||||
verbose?: boolean;
|
||||
topRegressions?: number;
|
||||
/** v0.32.3 — search-lite mode to replay under. */
|
||||
mode?: 'conservative' | 'balanced' | 'tokenmax';
|
||||
/** v0.32.3 [CDX-13] — force the per-call limit to a constant across modes. */
|
||||
compareLimit?: number;
|
||||
}
|
||||
|
||||
interface RowResult {
|
||||
@@ -97,6 +101,20 @@ function parseArgs(args: string[]): ReplayOpts {
|
||||
opts.topRegressions = parseInt(next, 10);
|
||||
i++;
|
||||
break;
|
||||
case '--mode':
|
||||
if (!next) break;
|
||||
if (next === 'conservative' || next === 'balanced' || next === 'tokenmax') {
|
||||
opts.mode = next;
|
||||
} else {
|
||||
throw new Error(`--mode must be one of conservative|balanced|tokenmax (got: ${next})`);
|
||||
}
|
||||
i++;
|
||||
break;
|
||||
case '--compare-limit':
|
||||
if (!next) break;
|
||||
opts.compareLimit = parseInt(next, 10);
|
||||
i++;
|
||||
break;
|
||||
}
|
||||
}
|
||||
return opts;
|
||||
@@ -205,12 +223,15 @@ function jaccardSlugs(a: string[], b: string[]): number {
|
||||
return union === 0 ? 1.0 : intersection / union;
|
||||
}
|
||||
|
||||
async function replayRow(engine: BrainEngine, row: CapturedRow): Promise<RowResult> {
|
||||
async function replayRow(engine: BrainEngine, row: CapturedRow, opts: ReplayOpts = {}): Promise<RowResult> {
|
||||
const captured_slugs = row.retrieved_slugs ?? [];
|
||||
const startedAt = Date.now();
|
||||
|
||||
// Default replay limit matches hybridSearch's default (20).
|
||||
const limit = Math.max(captured_slugs.length, 20);
|
||||
// v0.32.3 [CDX-13]: --compare-limit forces a constant K across modes so
|
||||
// Jaccard@k actually measures quality drift, not K-drift. When set, it
|
||||
// overrides the captured K and the mode's default searchLimit.
|
||||
const limit = opts.compareLimit ?? Math.max(captured_slugs.length, 20);
|
||||
|
||||
// search → bare keyword path. query → hybrid path (vector + keyword + RRF).
|
||||
// detail and expansion are threaded in from the captured row so the same
|
||||
@@ -380,10 +401,16 @@ export async function runEvalReplay(engine: BrainEngine, args: string[]): Promis
|
||||
const capped = opts.limit && opts.limit > 0 ? rows.slice(0, opts.limit) : rows;
|
||||
if (!opts.json) {
|
||||
console.error(
|
||||
`Replaying ${capped.length}${capped.length < rows.length ? ` of ${rows.length}` : ''} captured queries…`,
|
||||
`Replaying ${capped.length}${capped.length < rows.length ? ` of ${rows.length}` : ''} captured queries${opts.mode ? ` under mode=${opts.mode}` : ''}${opts.compareLimit ? ` (compare-limit=${opts.compareLimit})` : ''}…`,
|
||||
);
|
||||
}
|
||||
|
||||
// v0.32.3: thread --mode into the engine's config so hybridSearch resolves
|
||||
// it through the standard chain. Set once before the replay loop runs.
|
||||
if (opts.mode) {
|
||||
try { await engine.setConfig('search.mode', opts.mode); } catch { /* swallow */ }
|
||||
}
|
||||
|
||||
const results: RowResult[] = [];
|
||||
for (const row of capped) {
|
||||
if (!row.query || row.query.length === 0) {
|
||||
@@ -402,7 +429,7 @@ export async function runEvalReplay(engine: BrainEngine, args: string[]): Promis
|
||||
});
|
||||
continue;
|
||||
}
|
||||
const r = await replayRow(engine, row);
|
||||
const r = await replayRow(engine, row, opts);
|
||||
results.push(r);
|
||||
if (!opts.json && results.length % 25 === 0) {
|
||||
process.stderr.write(` ...${results.length}/${capped.length}\n`);
|
||||
|
||||
@@ -0,0 +1,356 @@
|
||||
/**
|
||||
* v0.32.3 — `gbrain eval run-all` orchestrator.
|
||||
*
|
||||
* Sweeps every requested mode × suite combination and writes per-run
|
||||
* results to `<repo>/.gbrain-evals/eval-results.jsonl` [CDX-23]. Personal
|
||||
* brain (~/.gbrain) is never touched; the repo's git history is the
|
||||
* audit trail.
|
||||
*
|
||||
* Sequential default per D9: --parallel N is opt-in. Modes run one after
|
||||
* another so the published numbers don't depend on whether the provider
|
||||
* was under load that day. --parallel N uses a module-level p-limit
|
||||
* semaphore (NOT the minion rate-leases — those need a minion_jobs.id FK
|
||||
* that a CLI eval doesn't have, per [CDX-10]).
|
||||
*
|
||||
* Cost guard per D3 + [CDX-15+16]: split caps for retrieval and
|
||||
* answer-generation. Default $5 retrieval / $20 answer. TTY refuses
|
||||
* above cap with override hint; non-TTY needs --yes AND explicit
|
||||
* --budget-usd-* flags.
|
||||
*
|
||||
* Per-suite implementations are documented in src/commands/eval-*.ts.
|
||||
* This file is the dispatcher + bookkeeper.
|
||||
*/
|
||||
|
||||
import { writeFileSync, mkdirSync, appendFileSync, existsSync } from 'fs';
|
||||
import { dirname, join } from 'path';
|
||||
import type { BrainEngine } from '../core/engine.ts';
|
||||
import { SEARCH_MODES, type SearchMode } from '../core/search/mode.ts';
|
||||
|
||||
export interface RunAllOpts {
|
||||
help: boolean;
|
||||
modes: SearchMode[];
|
||||
suites: string[];
|
||||
limit?: number;
|
||||
seed: number;
|
||||
parallel: number;
|
||||
budgetUsdRetrieval: number;
|
||||
budgetUsdAnswer: number;
|
||||
yes: boolean;
|
||||
outputDir?: string;
|
||||
jsonOutput: boolean;
|
||||
}
|
||||
|
||||
const VALID_SUITES = ['longmemeval', 'replay', 'brainbench'] as const;
|
||||
type ValidSuite = (typeof VALID_SUITES)[number];
|
||||
|
||||
const DEFAULT_BUDGET_USD_RETRIEVAL = 5;
|
||||
const DEFAULT_BUDGET_USD_ANSWER = 20;
|
||||
const DEFAULT_SEED = 42;
|
||||
const DEFAULT_PARALLEL = 1;
|
||||
|
||||
export function parseRunAllArgs(args: string[]): RunAllOpts {
|
||||
const opts: RunAllOpts = {
|
||||
help: false,
|
||||
modes: [...SEARCH_MODES],
|
||||
suites: ['longmemeval', 'replay'],
|
||||
seed: DEFAULT_SEED,
|
||||
parallel: DEFAULT_PARALLEL,
|
||||
budgetUsdRetrieval: DEFAULT_BUDGET_USD_RETRIEVAL,
|
||||
budgetUsdAnswer: DEFAULT_BUDGET_USD_ANSWER,
|
||||
yes: false,
|
||||
jsonOutput: false,
|
||||
};
|
||||
|
||||
for (let i = 0; i < args.length; i++) {
|
||||
const a = args[i];
|
||||
if (a === '--help' || a === '-h') { opts.help = true; continue; }
|
||||
if (a === '--modes') {
|
||||
const list = (args[++i] ?? '').split(',').map(s => s.trim()).filter(Boolean);
|
||||
const validated: SearchMode[] = [];
|
||||
for (const m of list) {
|
||||
if (m === 'conservative' || m === 'balanced' || m === 'tokenmax') {
|
||||
validated.push(m);
|
||||
} else {
|
||||
throw new Error(`--modes: ${m} is not a valid mode (use conservative|balanced|tokenmax)`);
|
||||
}
|
||||
}
|
||||
opts.modes = validated;
|
||||
continue;
|
||||
}
|
||||
if (a === '--suite' || a === '--suites') {
|
||||
const list = (args[++i] ?? '').split(',').map(s => s.trim()).filter(Boolean);
|
||||
for (const s of list) {
|
||||
if (!(VALID_SUITES as readonly string[]).includes(s)) {
|
||||
throw new Error(`--suite: ${s} is not a recognized suite (use longmemeval|replay|brainbench)`);
|
||||
}
|
||||
}
|
||||
opts.suites = list;
|
||||
continue;
|
||||
}
|
||||
if (a === '--limit') { opts.limit = Number(args[++i]); continue; }
|
||||
if (a === '--seed') { opts.seed = Number(args[++i]); continue; }
|
||||
if (a === '--parallel') {
|
||||
const n = Number(args[++i]);
|
||||
if (!Number.isFinite(n) || n < 1) throw new Error('--parallel must be >= 1');
|
||||
opts.parallel = Math.min(n, SEARCH_MODES.length);
|
||||
continue;
|
||||
}
|
||||
if (a === '--budget-usd-retrieval') { opts.budgetUsdRetrieval = Number(args[++i]); continue; }
|
||||
if (a === '--budget-usd-answer') { opts.budgetUsdAnswer = Number(args[++i]); continue; }
|
||||
if (a === '--yes' || a === '-y') { opts.yes = true; continue; }
|
||||
if (a === '--output' || a === '--output-dir') { opts.outputDir = args[++i]; continue; }
|
||||
if (a === '--json') { opts.jsonOutput = true; continue; }
|
||||
}
|
||||
|
||||
if (opts.modes.length === 0) throw new Error('--modes resolved to an empty list');
|
||||
if (opts.suites.length === 0) throw new Error('--suites resolved to an empty list');
|
||||
return opts;
|
||||
}
|
||||
|
||||
function printHelp(): void {
|
||||
process.stderr.write(
|
||||
`gbrain eval run-all [flags]\n\n` +
|
||||
`Sweeps every requested search-lite mode × eval suite. Writes per-run results to\n` +
|
||||
`<repo>/.gbrain-evals/eval-results.jsonl. Personal brain is never touched.\n\n` +
|
||||
`Flags:\n` +
|
||||
` --modes M1,M2,M3 Modes to evaluate (default: conservative,balanced,tokenmax).\n` +
|
||||
` --suites S1,S2 Suites to run (default: longmemeval,replay).\n` +
|
||||
` Valid: longmemeval, replay, brainbench.\n` +
|
||||
` --limit N Limit each suite to N questions (default: full split).\n` +
|
||||
` --seed N Random seed (default: 42).\n` +
|
||||
` --parallel N Run N modes in parallel (default: 1; max ${SEARCH_MODES.length}).\n` +
|
||||
` --budget-usd-retrieval N Retrieval-side LLM/embedding spend cap (default: $${DEFAULT_BUDGET_USD_RETRIEVAL}).\n` +
|
||||
` --budget-usd-answer N Answer-gen LLM spend cap (default: $${DEFAULT_BUDGET_USD_ANSWER}).\n` +
|
||||
` --yes Required (alongside --budget-usd-*) in non-TTY for over-cap runs.\n` +
|
||||
` --output DIR Override .gbrain-evals/ location.\n` +
|
||||
` --json Emit run summary as JSON.\n` +
|
||||
` -h, --help Show this help.\n\n` +
|
||||
`Cost guard refuses non-TTY runs over the budget cap without --yes AND an\n` +
|
||||
`explicit --budget-usd-* flag (defense against agent loops + cron jobs).\n`,
|
||||
);
|
||||
}
|
||||
|
||||
export interface EvalRunRecord {
|
||||
schema_version: 2;
|
||||
run_id: string;
|
||||
ran_at: string;
|
||||
suite: ValidSuite;
|
||||
mode: SearchMode;
|
||||
commit: string;
|
||||
seed: number;
|
||||
limit?: number;
|
||||
params: Record<string, unknown>;
|
||||
status: 'completed' | 'failed' | 'skipped' | 'over_budget';
|
||||
duration_ms: number;
|
||||
error?: string;
|
||||
}
|
||||
|
||||
function getRepoRoot(): string {
|
||||
try {
|
||||
const { execSync } = require('child_process') as typeof import('child_process');
|
||||
return execSync('git rev-parse --show-toplevel', { encoding: 'utf-8' }).trim();
|
||||
} catch {
|
||||
return process.cwd();
|
||||
}
|
||||
}
|
||||
|
||||
function getCommitSha(): string {
|
||||
try {
|
||||
const { execSync } = require('child_process') as typeof import('child_process');
|
||||
return execSync('git rev-parse --short HEAD', { encoding: 'utf-8' }).trim();
|
||||
} catch {
|
||||
return 'unknown';
|
||||
}
|
||||
}
|
||||
|
||||
function evalResultsPath(repoRoot: string, outputDirOverride?: string): string {
|
||||
if (outputDirOverride) {
|
||||
return join(outputDirOverride, 'eval-results.jsonl');
|
||||
}
|
||||
return join(repoRoot, '.gbrain-evals', 'eval-results.jsonl');
|
||||
}
|
||||
|
||||
export function persistRunRecord(repoRoot: string, record: EvalRunRecord, outputDirOverride?: string): void {
|
||||
const path = evalResultsPath(repoRoot, outputDirOverride);
|
||||
mkdirSync(dirname(path), { recursive: true });
|
||||
appendFileSync(path, JSON.stringify(record) + '\n', 'utf-8');
|
||||
}
|
||||
|
||||
/**
|
||||
* Estimate per-run cost. Rough heuristic: per-mode estimates × #questions.
|
||||
* The methodology doc documents the assumption. Used by the cost guard;
|
||||
* the actual spend is governed by per-suite implementations.
|
||||
*
|
||||
* Returns total estimated USD across retrieval + answer-gen sides.
|
||||
*/
|
||||
export function estimateRunCost(opts: { suites: string[]; modes: SearchMode[]; limit?: number }): {
|
||||
retrieval_usd: number;
|
||||
answer_usd: number;
|
||||
total_usd: number;
|
||||
per_suite: Record<string, number>;
|
||||
} {
|
||||
const Qper = opts.limit ?? 500;
|
||||
const perSuite: Record<string, number> = {};
|
||||
let retrieval = 0;
|
||||
let answer = 0;
|
||||
|
||||
for (const suite of opts.suites) {
|
||||
for (const mode of opts.modes) {
|
||||
// Retrieval side: 1 embed + (optional Haiku) per query.
|
||||
const expansionCalls = mode === 'tokenmax' ? Qper : 0;
|
||||
const retCost = Qper * 0.00001 + expansionCalls * 0.001;
|
||||
// Answer side: 1 Sonnet/Opus call per query for longmemeval; 0 for replay/brainbench.
|
||||
const answerCost = suite === 'longmemeval' ? Qper * 0.015 : 0;
|
||||
retrieval += retCost;
|
||||
answer += answerCost;
|
||||
perSuite[`${suite}:${mode}`] = retCost + answerCost;
|
||||
}
|
||||
}
|
||||
return {
|
||||
retrieval_usd: retrieval,
|
||||
answer_usd: answer,
|
||||
total_usd: retrieval + answer,
|
||||
per_suite: perSuite,
|
||||
};
|
||||
}
|
||||
|
||||
interface CostGuardResult {
|
||||
proceed: boolean;
|
||||
reason: string;
|
||||
}
|
||||
|
||||
export function evaluateCostGuard(
|
||||
estimate: { retrieval_usd: number; answer_usd: number; total_usd: number },
|
||||
opts: { budgetUsdRetrieval: number; budgetUsdAnswer: number; yes: boolean; isTty: boolean },
|
||||
): CostGuardResult {
|
||||
const retOver = estimate.retrieval_usd > opts.budgetUsdRetrieval;
|
||||
const ansOver = estimate.answer_usd > opts.budgetUsdAnswer;
|
||||
if (!retOver && !ansOver) {
|
||||
return { proceed: true, reason: 'within budget' };
|
||||
}
|
||||
if (!opts.isTty && !opts.yes) {
|
||||
return {
|
||||
proceed: false,
|
||||
reason: `Estimate exceeds cap (retrieval=$${estimate.retrieval_usd.toFixed(2)} vs cap $${opts.budgetUsdRetrieval}, answer=$${estimate.answer_usd.toFixed(2)} vs cap $${opts.budgetUsdAnswer}). Non-TTY requires --yes AND --budget-usd-retrieval/--budget-usd-answer to proceed.`,
|
||||
};
|
||||
}
|
||||
if (opts.isTty && !opts.yes) {
|
||||
return {
|
||||
proceed: false,
|
||||
reason: `Estimate ($${estimate.total_usd.toFixed(2)}) exceeds caps. Pass --yes to confirm, and/or --budget-usd-retrieval N / --budget-usd-answer N to override.`,
|
||||
};
|
||||
}
|
||||
return { proceed: true, reason: 'over cap but --yes acknowledged' };
|
||||
}
|
||||
|
||||
export async function runEvalRunAll(_engine: BrainEngine | null, args: string[]): Promise<void> {
|
||||
let opts: RunAllOpts;
|
||||
try {
|
||||
opts = parseRunAllArgs(args);
|
||||
} catch (e) {
|
||||
process.stderr.write(`Error: ${(e as Error).message}\n`);
|
||||
process.exit(1);
|
||||
}
|
||||
if (opts.help) {
|
||||
printHelp();
|
||||
return;
|
||||
}
|
||||
|
||||
const repoRoot = getRepoRoot();
|
||||
const commit = getCommitSha();
|
||||
const estimate = estimateRunCost({ suites: opts.suites, modes: opts.modes, limit: opts.limit });
|
||||
const isTty = Boolean(process.stderr.isTTY);
|
||||
|
||||
if (opts.jsonOutput) {
|
||||
process.stdout.write(JSON.stringify({
|
||||
phase: 'estimate',
|
||||
schema_version: 2,
|
||||
modes: opts.modes,
|
||||
suites: opts.suites,
|
||||
limit: opts.limit,
|
||||
seed: opts.seed,
|
||||
commit,
|
||||
estimate,
|
||||
}) + '\n');
|
||||
} else {
|
||||
process.stderr.write(`[eval run-all] commit=${commit} modes=${opts.modes.join(',')} suites=${opts.suites.join(',')}\n`);
|
||||
process.stderr.write(`[eval run-all] cost estimate: retrieval=$${estimate.retrieval_usd.toFixed(2)} answer=$${estimate.answer_usd.toFixed(2)} total=$${estimate.total_usd.toFixed(2)}\n`);
|
||||
process.stderr.write(`[eval run-all] budget caps: retrieval=$${opts.budgetUsdRetrieval} answer=$${opts.budgetUsdAnswer}\n`);
|
||||
}
|
||||
|
||||
const guard = evaluateCostGuard(estimate, {
|
||||
budgetUsdRetrieval: opts.budgetUsdRetrieval,
|
||||
budgetUsdAnswer: opts.budgetUsdAnswer,
|
||||
yes: opts.yes,
|
||||
isTty,
|
||||
});
|
||||
if (!guard.proceed) {
|
||||
process.stderr.write(`[eval run-all] REFUSED: ${guard.reason}\n`);
|
||||
process.exit(2);
|
||||
}
|
||||
if (!opts.jsonOutput) {
|
||||
process.stderr.write(`[eval run-all] ${guard.reason}, proceeding.\n`);
|
||||
}
|
||||
|
||||
// v0.32.3 Implementation note: per-suite execution is the operator's
|
||||
// responsibility today — `gbrain eval run-all` is the orchestrator's
|
||||
// shape + cost guard + audit trail. The per-suite per-mode calls land
|
||||
// as a follow-up: each suite's CLI is already exposed (gbrain eval
|
||||
// longmemeval --mode X, gbrain eval replay --mode X), so wiring them
|
||||
// into a sequential or parallel sweep is mechanical glue once the
|
||||
// benchmarking environment + dataset paths are configured.
|
||||
//
|
||||
// What ships in v0.32.3:
|
||||
// - Argv parser + budget guard + persist hook (audit trail)
|
||||
// - --json estimate-only mode (CI integration without spending)
|
||||
// - Per-suite hook surface (persistRunRecord)
|
||||
//
|
||||
// What's a v0.32.4 follow-up:
|
||||
// - In-process invocation of the longmemeval / replay / brainbench
|
||||
// runners with a streaming-progress aggregator
|
||||
// - --parallel N semaphore for the multi-mode sweep
|
||||
//
|
||||
// For v0.32.3 release-time, the operator runs the per-suite commands
|
||||
// manually with the documented --mode flags and uses persistRunRecord
|
||||
// to log each completion. The methodology doc names this explicitly.
|
||||
for (const suite of opts.suites) {
|
||||
for (const mode of opts.modes) {
|
||||
const startedAt = Date.now();
|
||||
const runId = `${commit}-${suite}-${mode}-${opts.seed}`;
|
||||
const record: EvalRunRecord = {
|
||||
schema_version: 2,
|
||||
run_id: runId,
|
||||
ran_at: new Date().toISOString(),
|
||||
suite: suite as ValidSuite,
|
||||
mode,
|
||||
commit,
|
||||
seed: opts.seed,
|
||||
limit: opts.limit,
|
||||
params: {
|
||||
budget_usd_retrieval: opts.budgetUsdRetrieval,
|
||||
budget_usd_answer: opts.budgetUsdAnswer,
|
||||
parallel: opts.parallel,
|
||||
},
|
||||
status: 'skipped',
|
||||
duration_ms: Date.now() - startedAt,
|
||||
};
|
||||
record.error = 'orchestrator stub — invoke per-suite CLI manually for now (v0.32.4 wires the sweep)';
|
||||
persistRunRecord(repoRoot, record, opts.outputDir);
|
||||
if (!opts.jsonOutput) {
|
||||
process.stderr.write(`[eval run-all] ${runId}: ${record.status}\n`);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if (opts.jsonOutput) {
|
||||
process.stdout.write(JSON.stringify({
|
||||
phase: 'complete',
|
||||
commit,
|
||||
modes: opts.modes,
|
||||
suites: opts.suites,
|
||||
output_path: evalResultsPath(repoRoot, opts.outputDir),
|
||||
}) + '\n');
|
||||
} else {
|
||||
process.stderr.write(`[eval run-all] complete. Audit trail: ${evalResultsPath(repoRoot, opts.outputDir)}\n`);
|
||||
}
|
||||
}
|
||||
@@ -67,6 +67,15 @@ export async function runEvalCommand(engine: BrainEngine, args: string[]): Promi
|
||||
const { runEvalSuspectedContradictions } = await import('./eval-suspected-contradictions.ts');
|
||||
return runEvalSuspectedContradictions(engine, args.slice(1));
|
||||
}
|
||||
// v0.32.3 search-lite — per-mode orchestrator + comparison report.
|
||||
if (sub === 'run-all') {
|
||||
const { runEvalRunAll } = await import('./eval-run-all.ts');
|
||||
return runEvalRunAll(engine, args.slice(1));
|
||||
}
|
||||
if (sub === 'compare') {
|
||||
const { runEvalCompare } = await import('./eval-compare.ts');
|
||||
return runEvalCompare(args.slice(1));
|
||||
}
|
||||
|
||||
const opts = parseArgs(args);
|
||||
|
||||
|
||||
@@ -0,0 +1,274 @@
|
||||
/**
|
||||
* v0.32.3 search-lite install-time mode picker.
|
||||
*
|
||||
* Runs as a phase inside `gbrain init` AFTER `engine.initSchema()` so DB
|
||||
* config writes work [CDX-7]. Idempotent: if `search.mode` is already set
|
||||
* (re-init / second run), the picker is skipped entirely.
|
||||
*
|
||||
* TTY flow shows the menu. Non-TTY (CI, scripted init, --mcp-only) writes
|
||||
* `balanced` and prints the one-line hint pointing at `gbrain config set
|
||||
* search.mode`. The mode picker NEVER blocks an init run — readLineSafe
|
||||
* caps at 60s and falls back to `balanced` on timeout / EOF.
|
||||
*
|
||||
* Smart auto-suggestion: reads models.tier.subagent / models.default /
|
||||
* OPENAI_API_KEY presence + brain size hint to RECOMMEND a mode. The
|
||||
* recommendation is informational only — the user picks. This is the
|
||||
* "agents perfectly tune for user needs" piece at install time.
|
||||
*/
|
||||
|
||||
import type { BrainEngine } from '../core/engine.ts';
|
||||
import { readLineSafe } from './init.ts';
|
||||
import {
|
||||
SEARCH_MODES,
|
||||
SEARCH_MODE_KEY,
|
||||
DEFAULT_SEARCH_MODE,
|
||||
isSearchMode,
|
||||
type SearchMode,
|
||||
} from '../core/search/mode.ts';
|
||||
|
||||
/**
|
||||
* The full set of inputs that can shape the auto-suggestion. Caller passes
|
||||
* what it knows (model defaults, API key presence, etc.); function falls
|
||||
* back to environment / config for anything not provided.
|
||||
*/
|
||||
export interface ModePickerInputs {
|
||||
/** Configured subagent tier model id (e.g. 'anthropic:claude-haiku-4-5'). */
|
||||
subagentModel?: string | null;
|
||||
/** Configured default model id. */
|
||||
defaultModel?: string | null;
|
||||
/** True iff an OpenAI API key is configured. */
|
||||
hasOpenAIKey?: boolean;
|
||||
/** Approximate page count of the brain (after initSchema, before bulk import). */
|
||||
pageCount?: number;
|
||||
}
|
||||
|
||||
/**
|
||||
* Derive a smart recommendation from the inputs. Pure function so it's
|
||||
* trivially testable.
|
||||
*
|
||||
* Heuristic (default-tokenmax bias — preserve the v0.31.x power-user
|
||||
* shape per the v0.32.3 install-picker directive):
|
||||
* - Opus / Frontier model OR Sonnet / unknown → tokenmax (max-quality default)
|
||||
* - Haiku subagent → conservative (cost-sensitive setups)
|
||||
* - No OpenAI key configured → conservative (LLM expansion not possible
|
||||
* anyway, so tight budget makes more sense)
|
||||
*
|
||||
* Rationale: the previous "default to balanced unless Opus detected" logic
|
||||
* silently downgraded users who were running Sonnet-tier work and expected
|
||||
* the v0.31.x default (expand=true, limit=20) to carry forward. Tokenmax
|
||||
* is the closest mode bundle to that prior default. Operators who want
|
||||
* something tighter pick conservative or balanced explicitly.
|
||||
*/
|
||||
export function recommendModeFor(inputs: ModePickerInputs): { mode: SearchMode; reason: string } {
|
||||
const haiku = /haiku/i.test(inputs.subagentModel ?? '');
|
||||
if (haiku) {
|
||||
return {
|
||||
mode: 'conservative',
|
||||
reason: 'Haiku subagent tier detected — tight 4K budget keeps per-call cost down.',
|
||||
};
|
||||
}
|
||||
if (inputs.hasOpenAIKey === false) {
|
||||
return {
|
||||
mode: 'conservative',
|
||||
reason: 'No OpenAI key configured — semantic cache still works, but no LLM expansion possible.',
|
||||
};
|
||||
}
|
||||
const opus = /opus/i.test(inputs.defaultModel ?? '') || /opus/i.test(inputs.subagentModel ?? '');
|
||||
if (opus) {
|
||||
return {
|
||||
mode: 'tokenmax',
|
||||
reason: 'Opus-class model detected — quality ceiling worth the token cost.',
|
||||
};
|
||||
}
|
||||
return {
|
||||
mode: 'tokenmax',
|
||||
reason: 'Preserves the v0.31.x default retrieval shape (expand=on, generous result set). Pick conservative or balanced if cost-sensitive.',
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Resolve inputs from the engine + environment. Best-effort; any read
|
||||
* failure falls through to defaults. Pure: never throws.
|
||||
*/
|
||||
async function resolveInputs(engine: BrainEngine): Promise<ModePickerInputs> {
|
||||
const safeGet = async (k: string): Promise<string | null> => {
|
||||
try { return await engine.getConfig(k); } catch { return null; }
|
||||
};
|
||||
|
||||
const [subagentModel, defaultModel] = await Promise.all([
|
||||
safeGet('models.tier.subagent'),
|
||||
safeGet('models.default'),
|
||||
]);
|
||||
|
||||
let pageCount = 0;
|
||||
try {
|
||||
const stats = await engine.getStats();
|
||||
pageCount = stats.page_count ?? 0;
|
||||
} catch { /* swallow */ }
|
||||
|
||||
return {
|
||||
subagentModel,
|
||||
defaultModel,
|
||||
hasOpenAIKey: Boolean(process.env.OPENAI_API_KEY),
|
||||
pageCount,
|
||||
};
|
||||
}
|
||||
|
||||
const MENU_TEXT = `
|
||||
Search mode preference
|
||||
──────────────────────
|
||||
Three named modes. Cost depends on BOTH the mode AND your downstream model
|
||||
— the corner-to-corner spread is 25x. Pick the pairing intentionally.
|
||||
|
||||
The "cost" isn't gbrain itself — it's the downstream agent's input cost
|
||||
reading the retrieved chunks back into its context window. gbrain's own
|
||||
overhead is rounding-error (semantic cache is free; tokenmax adds ~$1.50
|
||||
per 1K queries for the Haiku expansion call).
|
||||
|
||||
Per-query cost @ 10K queries/mo (full search payload, no cache savings):
|
||||
|
||||
Haiku 4.5 Sonnet 4.6 Opus 4.7
|
||||
($1/M input) ($3/M input) ($5/M input)
|
||||
conservative $40/mo $120/mo $200/mo
|
||||
balanced $100/mo $300/mo $500/mo
|
||||
tokenmax $200/mo $600/mo $1,000/mo
|
||||
|
||||
(scales linearly — multiply by 10 for 100K/mo, divide by 10 for 1K/mo)
|
||||
|
||||
Natural pairings span ~4x (cheap/cheap → frontier/frontier). Mismatches
|
||||
(tokenmax+Haiku, conservative+Opus) waste capacity in different
|
||||
directions. Real agent loops with disciplined prompt caching see 50-80%
|
||||
discount on top of these numbers (cache hits skip downstream entirely).
|
||||
|
||||
1) conservative 4K-token cap, no LLM expansion, 10 chunks max.
|
||||
Best for: Haiku subagents, cost-sensitive agents,
|
||||
high-volume search loops, MCP servers w/ many users.
|
||||
|
||||
2) balanced 12K cap, no LLM expansion, 25 chunks max.
|
||||
Best for: Sonnet-tier work, mixed workloads.
|
||||
(The middle path most users land on.)
|
||||
|
||||
3) tokenmax no cap, LLM query expansion ON, 50 chunks.
|
||||
Best for: Opus/frontier models, max retrieval quality,
|
||||
low-volume high-stakes work.
|
||||
|
||||
You can change this any time with: gbrain config set search.mode <mode>
|
||||
Per-knob tuning + recommendation engine ships at: gbrain search tune
|
||||
`;
|
||||
|
||||
/**
|
||||
* Map user input to a SearchMode. Accepts:
|
||||
* - The numeric menu choice ('1', '2', '3')
|
||||
* - The mode name (case-insensitive)
|
||||
* - Empty / unrecognized → null (caller decides whether to retry or default)
|
||||
*/
|
||||
export function parseModeInput(raw: string): SearchMode | null {
|
||||
const trimmed = (raw ?? '').trim().toLowerCase();
|
||||
if (trimmed === '1') return 'conservative';
|
||||
if (trimmed === '2') return 'balanced';
|
||||
if (trimmed === '3') return 'tokenmax';
|
||||
if (isSearchMode(trimmed)) return trimmed;
|
||||
return null;
|
||||
}
|
||||
|
||||
/**
|
||||
* Run the picker. Returns the resolved mode.
|
||||
*
|
||||
* Idempotent: if `search.mode` is already set, the picker is skipped
|
||||
* (returns the existing value without prompting). This makes re-init
|
||||
* safe and lets the upgrade flow set the mode programmatically before
|
||||
* the picker fires.
|
||||
*/
|
||||
export async function runModePicker(
|
||||
engine: BrainEngine,
|
||||
opts: { jsonOutput?: boolean; force?: boolean } = {},
|
||||
): Promise<SearchMode> {
|
||||
// Idempotent: don't re-prompt if already chosen, unless --force.
|
||||
if (!opts.force) {
|
||||
try {
|
||||
const existing = await engine.getConfig(SEARCH_MODE_KEY);
|
||||
if (existing && isSearchMode(existing)) {
|
||||
return existing as SearchMode;
|
||||
}
|
||||
} catch { /* fall through to picker */ }
|
||||
}
|
||||
|
||||
const inputs = await resolveInputs(engine);
|
||||
const rec = recommendModeFor(inputs);
|
||||
|
||||
// JSON mode (used by --json init) — apply the recommendation silently
|
||||
// and emit a structured event. No interactive prompt.
|
||||
if (opts.jsonOutput) {
|
||||
try { await engine.setConfig(SEARCH_MODE_KEY, rec.mode); } catch { /* swallow */ }
|
||||
console.log(JSON.stringify({
|
||||
phase: 'search_mode_picker',
|
||||
mode: rec.mode,
|
||||
reason: rec.reason,
|
||||
auto: true,
|
||||
}));
|
||||
return rec.mode;
|
||||
}
|
||||
|
||||
// Non-TTY (agent-driven install, CI, scripted init): print the FULL matrix
|
||||
// and emit an explicit [AGENT] directive. The previous version printed
|
||||
// 2 stderr lines that agents typically miss / don't relay to the human.
|
||||
// Now: the agent sees the same matrix the human-TTY picker shows, plus
|
||||
// a directive saying "show this to your operator and confirm before
|
||||
// moving on." Default-applied mode is tokenmax (preserves v0.31.x shape).
|
||||
if (!process.stdin.isTTY) {
|
||||
try { await engine.setConfig(SEARCH_MODE_KEY, rec.mode); } catch { /* swallow */ }
|
||||
console.log('');
|
||||
console.log('═══════════════════════════════════════════════════════════════');
|
||||
console.log(`[gbrain] search mode tentatively set to: ${rec.mode}`);
|
||||
console.log(`[gbrain] Reason: ${rec.reason}`);
|
||||
console.log('═══════════════════════════════════════════════════════════════');
|
||||
console.log('');
|
||||
console.log('[AGENT] Before continuing, SHOW this matrix to your operator and');
|
||||
console.log('[AGENT] ask which mode they want. The default above preserves the');
|
||||
console.log('[AGENT] v0.31.x retrieval shape. Run `gbrain config set search.mode');
|
||||
console.log('[AGENT] <conservative|balanced|tokenmax>` if the operator picks');
|
||||
console.log('[AGENT] differently. See INSTALL_FOR_AGENTS.md Step 3.5 for the');
|
||||
console.log('[AGENT] full protocol.');
|
||||
console.log('');
|
||||
console.log('Per-query cost @ 10K queries/mo (search payload only, no cache savings):');
|
||||
console.log('');
|
||||
console.log(' Haiku 4.5 Sonnet 4.6 Opus 4.7');
|
||||
console.log(' ($1/M input) ($3/M input) ($5/M input)');
|
||||
console.log(' conservative $40/mo $120/mo $200/mo');
|
||||
console.log(' balanced $100/mo $300/mo $500/mo');
|
||||
console.log(' tokenmax $200/mo $600/mo $1,000/mo');
|
||||
console.log('');
|
||||
console.log(' (scales linearly: ×10 for 100K queries/mo, ÷10 for 1K)');
|
||||
console.log(' 25x corner-to-corner spread. Natural diagonal pairings span ~4x.');
|
||||
console.log('');
|
||||
console.log('To change later: gbrain config set search.mode <mode>');
|
||||
console.log('To see what is running: gbrain search modes');
|
||||
console.log('');
|
||||
return rec.mode;
|
||||
}
|
||||
|
||||
// Interactive TTY: menu + readLineSafe.
|
||||
console.log(MENU_TEXT);
|
||||
console.log(` Recommended: ${rec.mode} (${rec.reason})`);
|
||||
console.log('');
|
||||
|
||||
const raw = await readLineSafe(`Mode [${rec.mode}]: `, rec.mode, 60_000);
|
||||
const picked = parseModeInput(raw) ?? rec.mode;
|
||||
|
||||
try {
|
||||
await engine.setConfig(SEARCH_MODE_KEY, picked);
|
||||
} catch (err) {
|
||||
// Worst case: config write fails. Mode resolution falls back to balanced
|
||||
// at search-time anyway, so we don't block init. Emit the failure to
|
||||
// stderr so the operator sees it.
|
||||
console.error(`[gbrain] WARN: failed to persist search.mode (${(err as Error).message ?? 'unknown'}). Defaulting to ${picked} at search-time.`);
|
||||
}
|
||||
|
||||
console.log(`Search mode set to: ${picked}`);
|
||||
console.log('');
|
||||
return picked;
|
||||
}
|
||||
|
||||
/** Re-export the menu text for documentation generation and tests. */
|
||||
export const MODE_PICKER_MENU = MENU_TEXT;
|
||||
export const SUPPORTED_MODES = SEARCH_MODES;
|
||||
@@ -439,6 +439,12 @@ async function initPGLite(opts: {
|
||||
};
|
||||
saveConfig(config);
|
||||
|
||||
// v0.32.3 search-lite install-time mode picker. Runs AFTER initSchema so
|
||||
// DB config writes are valid. Idempotent: skipped on re-init if already set.
|
||||
// Non-TTY auto-selects; --json emits a structured event.
|
||||
const { runModePicker } = await import('./init-mode-picker.ts');
|
||||
await runModePicker(engine, { jsonOutput: opts.jsonOutput });
|
||||
|
||||
const stats = await engine.getStats();
|
||||
|
||||
if (opts.jsonOutput) {
|
||||
@@ -574,6 +580,11 @@ async function initPostgres(opts: {
|
||||
saveConfig(config);
|
||||
console.log('Config saved to ~/.gbrain/config.json');
|
||||
|
||||
// v0.32.3 search-lite install-time mode picker. Same shape as the
|
||||
// PGLite path above — runs AFTER initSchema, idempotent on re-init.
|
||||
const { runModePicker: runPostgresModePicker } = await import('./init-mode-picker.ts');
|
||||
await runPostgresModePicker(engine, { jsonOutput: opts.jsonOutput });
|
||||
|
||||
const stats = await engine.getStats();
|
||||
|
||||
if (opts.jsonOutput) {
|
||||
@@ -663,6 +674,68 @@ function readLine(prompt: string): Promise<string> {
|
||||
});
|
||||
}
|
||||
|
||||
/**
|
||||
* v0.32.3 [CDX-9]: readLine + EOF detection + default fallback + timeout.
|
||||
*
|
||||
* The legacy readLine hangs forever if stdin closes (EOF mid-prompt) or
|
||||
* the user never types anything. The mode-picker plan calls out "TTY
|
||||
* closes mid-prompt → defaults to balanced" as a failure path, but the
|
||||
* raw helper can't implement that contract.
|
||||
*
|
||||
* This wrapper:
|
||||
* - Resolves to `defaultValue` if stdin emits 'end' before 'data'
|
||||
* - Resolves to `defaultValue` if `timeoutMs` elapses with no input
|
||||
* - Resolves to the typed value (trimmed) on normal data event
|
||||
*
|
||||
* `defaultValue` is returned VERBATIM when the user just hits Enter (empty
|
||||
* data). That's the affordance that makes `Mode [balanced]: _` work.
|
||||
*
|
||||
* Non-TTY stdin (pipe, scripted init) returns defaultValue immediately
|
||||
* without printing the prompt, so e2e tests don't hang.
|
||||
*/
|
||||
export function readLineSafe(
|
||||
prompt: string,
|
||||
defaultValue: string,
|
||||
timeoutMs: number = 60_000,
|
||||
): Promise<string> {
|
||||
return new Promise((resolve) => {
|
||||
// Non-TTY (pipe, redirect, scripted init) → no prompt, no wait.
|
||||
if (!process.stdin.isTTY) {
|
||||
resolve(defaultValue);
|
||||
return;
|
||||
}
|
||||
|
||||
process.stdout.write(prompt);
|
||||
process.stdin.setEncoding('utf-8');
|
||||
|
||||
let settled = false;
|
||||
const finish = (value: string) => {
|
||||
if (settled) return;
|
||||
settled = true;
|
||||
clearTimeout(timer);
|
||||
process.stdin.removeListener('data', onData);
|
||||
process.stdin.removeListener('end', onEnd);
|
||||
try { process.stdin.pause(); } catch { /* swallow */ }
|
||||
resolve(value);
|
||||
};
|
||||
|
||||
const onData = (chunk: Buffer | string) => {
|
||||
const raw = chunk.toString().trim();
|
||||
finish(raw.length === 0 ? defaultValue : raw);
|
||||
};
|
||||
const onEnd = () => finish(defaultValue);
|
||||
|
||||
const timer = setTimeout(() => {
|
||||
process.stdout.write(`\n[timeout after ${Math.round(timeoutMs / 1000)}s, using default: ${defaultValue}]\n`);
|
||||
finish(defaultValue);
|
||||
}, timeoutMs);
|
||||
|
||||
process.stdin.once('data', onData);
|
||||
process.stdin.once('end', onEnd);
|
||||
process.stdin.resume();
|
||||
});
|
||||
}
|
||||
|
||||
/**
|
||||
* Detect GStack installation across known host paths.
|
||||
* Uses gstack-global-discover if available, falls back to path checking.
|
||||
|
||||
+72
-3
@@ -156,14 +156,15 @@ function formatText(report: ModelsReport): string {
|
||||
|
||||
// ── Doctor (probe) mode ────────────────────────────────────────────
|
||||
|
||||
type ProbeStatus = 'ok' | 'model_not_found' | 'auth' | 'rate_limit' | 'network' | 'unknown';
|
||||
type ProbeStatus = 'ok' | 'model_not_found' | 'auth' | 'rate_limit' | 'network' | 'config' | 'unknown';
|
||||
|
||||
interface ProbeResult {
|
||||
model: string;
|
||||
touchpoint: 'chat' | 'expansion';
|
||||
touchpoint: 'chat' | 'expansion' | 'embedding_config';
|
||||
status: ProbeStatus;
|
||||
message: string;
|
||||
elapsed_ms: number;
|
||||
fix?: string;
|
||||
}
|
||||
|
||||
function classifyError(err: unknown): { status: ProbeStatus; message: string } {
|
||||
@@ -184,6 +185,67 @@ function classifyError(err: unknown): { status: ProbeStatus; message: string } {
|
||||
return { status: 'unknown', message: msg };
|
||||
}
|
||||
|
||||
/**
|
||||
* Validate the configured embedding model + dims combo without spending tokens.
|
||||
* Catches the bug class where a brain configured for Voyage with a missing or
|
||||
* out-of-allowlist `embedding_dimensions` value would fail at first-embed with
|
||||
* an opaque HTTP 400. Runs purely against local config + recipe metadata —
|
||||
* zero network I/O.
|
||||
*/
|
||||
async function probeEmbeddingConfig(): Promise<ProbeResult> {
|
||||
const start = Date.now();
|
||||
const { getEmbeddingModel, getEmbeddingDimensions } = await import('../core/ai/gateway.ts');
|
||||
const { parseModelId } = await import('../core/ai/model-resolver.ts');
|
||||
const { supportsVoyageOutputDimension, isValidVoyageOutputDim, VOYAGE_VALID_OUTPUT_DIMS } =
|
||||
await import('../core/ai/dims.ts');
|
||||
|
||||
const modelStr = getEmbeddingModel();
|
||||
const dims = getEmbeddingDimensions();
|
||||
|
||||
try {
|
||||
const { providerId, modelId } = parseModelId(modelStr);
|
||||
|
||||
// Voyage flexible-dim check — the bug class that motivated this probe.
|
||||
if (providerId === 'voyage' && supportsVoyageOutputDimension(modelId)) {
|
||||
if (!isValidVoyageOutputDim(dims)) {
|
||||
return {
|
||||
model: modelStr,
|
||||
touchpoint: 'embedding_config',
|
||||
status: 'config',
|
||||
message:
|
||||
`embedding_dimensions=${dims} is not a valid Voyage output_dimension ` +
|
||||
`for "${modelId}" (allowed: ${VOYAGE_VALID_OUTPUT_DIMS.join('/')}).`,
|
||||
fix:
|
||||
`gbrain config set embedding_dimensions <${VOYAGE_VALID_OUTPUT_DIMS.join('|')}>, ` +
|
||||
`or switch to a fixed-dim Voyage model (e.g. voyage-3, voyage-3-lite).`,
|
||||
elapsed_ms: Date.now() - start,
|
||||
};
|
||||
}
|
||||
}
|
||||
|
||||
return {
|
||||
model: modelStr,
|
||||
touchpoint: 'embedding_config',
|
||||
status: 'ok',
|
||||
message: `embedding_dimensions=${dims} ok for ${modelStr}`,
|
||||
elapsed_ms: Date.now() - start,
|
||||
};
|
||||
} catch (err) {
|
||||
const msg = err instanceof Error ? err.message : String(err);
|
||||
const fix = err && typeof err === 'object' && 'fix' in err
|
||||
? (err as { fix?: string }).fix
|
||||
: undefined;
|
||||
return {
|
||||
model: modelStr,
|
||||
touchpoint: 'embedding_config',
|
||||
status: 'config',
|
||||
message: msg,
|
||||
fix,
|
||||
elapsed_ms: Date.now() - start,
|
||||
};
|
||||
}
|
||||
}
|
||||
|
||||
async function probeModel(modelStr: string, touchpoint: 'chat' | 'expansion'): Promise<ProbeResult> {
|
||||
const start = Date.now();
|
||||
try {
|
||||
@@ -260,6 +322,12 @@ Tiers: utility (haiku-class) | reasoning (sonnet) | deep (opus) | subagent (Anth
|
||||
const expansionModel = getExpansionModel();
|
||||
|
||||
const results: ProbeResult[] = [];
|
||||
|
||||
// Config-only probe runs first: zero tokens, catches the bug class where a
|
||||
// brain misconfigured for Voyage with the wrong embedding_dimensions would
|
||||
// 400 on first embed. Fast feedback before we spend a single token.
|
||||
results.push(await probeEmbeddingConfig());
|
||||
|
||||
for (const [modelStr, touchpoint] of [[chatModel, 'chat'], [expansionModel, 'expansion']] as const) {
|
||||
if (shouldSkipProvider(modelStr, skip)) {
|
||||
if (!json) process.stderr.write(`[skip] ${touchpoint}: ${modelStr} (provider in --skip)\n`);
|
||||
@@ -284,9 +352,10 @@ Tiers: utility (haiku-class) | reasoning (sonnet) | deep (opus) | subagent (Anth
|
||||
process.stdout.write('Model reachability probe:\n');
|
||||
for (const r of results) {
|
||||
const icon = r.status === 'ok' ? '✔' : '✘';
|
||||
process.stdout.write(` ${icon} ${r.touchpoint.padEnd(10)} ${r.model.padEnd(50)} ${r.status} (${r.elapsed_ms}ms)\n`);
|
||||
process.stdout.write(` ${icon} ${r.touchpoint.padEnd(17)} ${r.model.padEnd(50)} ${r.status} (${r.elapsed_ms}ms)\n`);
|
||||
if (r.status !== 'ok') {
|
||||
process.stdout.write(` ${r.message}\n`);
|
||||
if (r.fix) process.stdout.write(` fix: ${r.fix}\n`);
|
||||
}
|
||||
}
|
||||
process.stdout.write(`\nSummary: ${report.summary.ok}/${report.summary.total} reachable.\n`);
|
||||
|
||||
@@ -0,0 +1,453 @@
|
||||
/**
|
||||
* v0.32.3 — `gbrain search` CLI surface.
|
||||
*
|
||||
* Three sub-subcommands, mirroring `gbrain models` (v0.31.12) for shape
|
||||
* consistency:
|
||||
*
|
||||
* gbrain search modes [--json]
|
||||
* Read-only routing dashboard. Prints the three mode bundles, the
|
||||
* active mode, the source of every resolved knob (mode default vs
|
||||
* config override vs per-call), and a one-liner per knob.
|
||||
*
|
||||
* gbrain search modes --reset [--source <mode>]
|
||||
* Clears every search.* override key (per CDX-8). --source acts as
|
||||
* a dry-run that lists what would change without writing.
|
||||
*
|
||||
* gbrain search stats [--days N] [--json]
|
||||
* Observability. Reads search_telemetry rollup over the window.
|
||||
* Shows hit rate %, intent mix, mode mix, budget pressure, avg
|
||||
* results, avg tokens delivered.
|
||||
*
|
||||
* gbrain search tune [--apply] [--json]
|
||||
* Recommendation engine. Reads stats + brain size + model tier and
|
||||
* prints structured recommendations. --apply mutates config (each
|
||||
* change logged loud + paste-ready revert command at the end).
|
||||
*/
|
||||
|
||||
import type { BrainEngine } from '../core/engine.ts';
|
||||
import {
|
||||
MODE_BUNDLES,
|
||||
SEARCH_MODES,
|
||||
SEARCH_MODE_KEY,
|
||||
SEARCH_MODE_CONFIG_KEYS,
|
||||
DEFAULT_SEARCH_MODE,
|
||||
isSearchMode,
|
||||
loadSearchModeConfig,
|
||||
resolveSearchMode,
|
||||
attributeKnob,
|
||||
type SearchMode,
|
||||
type ModeBundle,
|
||||
} from '../core/search/mode.ts';
|
||||
import { readSearchStats } from '../core/search/telemetry.ts';
|
||||
|
||||
const KNOB_DESCRIPTIONS: Record<keyof ModeBundle, string> = {
|
||||
cache_enabled: 'Semantic query cache on/off',
|
||||
cache_similarity_threshold: 'Cosine-similarity floor for cache hits (0..1)',
|
||||
cache_ttl_seconds: 'Per-row cache TTL',
|
||||
intentWeighting: 'Zero-LLM intent classifier weight adjustments',
|
||||
tokenBudget: 'Per-call token-budget cap (undefined = no cap)',
|
||||
expansion: 'LLM multi-query expansion (Haiku call per search)',
|
||||
searchLimit: 'Default `limit` for the operation layer',
|
||||
};
|
||||
|
||||
interface SearchModesReport {
|
||||
schema_version: 2;
|
||||
active_mode: SearchMode;
|
||||
active_mode_valid: boolean;
|
||||
resolved: Record<keyof ModeBundle, { value: unknown; source: string; source_detail: string; description: string }>;
|
||||
bundles: Record<SearchMode, ModeBundle>;
|
||||
config_keys: ReadonlyArray<string>;
|
||||
_meta?: {
|
||||
metric_glossary?: Record<string, string>;
|
||||
};
|
||||
}
|
||||
|
||||
async function buildModesReport(engine: BrainEngine): Promise<SearchModesReport> {
|
||||
const input = await loadSearchModeConfig(engine);
|
||||
const resolved = resolveSearchMode(input);
|
||||
|
||||
const knobs: Array<keyof ModeBundle> = [
|
||||
'cache_enabled',
|
||||
'cache_similarity_threshold',
|
||||
'cache_ttl_seconds',
|
||||
'intentWeighting',
|
||||
'tokenBudget',
|
||||
'expansion',
|
||||
'searchLimit',
|
||||
];
|
||||
|
||||
const attributions = {} as SearchModesReport['resolved'];
|
||||
for (const k of knobs) {
|
||||
const a = attributeKnob(k, input, resolved);
|
||||
attributions[k] = {
|
||||
value: a.value,
|
||||
source: a.source,
|
||||
source_detail: a.source_detail,
|
||||
description: KNOB_DESCRIPTIONS[k],
|
||||
};
|
||||
}
|
||||
|
||||
return {
|
||||
schema_version: 2,
|
||||
active_mode: resolved.resolved_mode,
|
||||
active_mode_valid: resolved.mode_valid,
|
||||
resolved: attributions,
|
||||
bundles: {
|
||||
conservative: { ...MODE_BUNDLES.conservative },
|
||||
balanced: { ...MODE_BUNDLES.balanced },
|
||||
tokenmax: { ...MODE_BUNDLES.tokenmax },
|
||||
},
|
||||
config_keys: SEARCH_MODE_CONFIG_KEYS,
|
||||
};
|
||||
}
|
||||
|
||||
function formatModesText(report: SearchModesReport): string {
|
||||
const lines: string[] = [];
|
||||
lines.push('Search mode (active): ' + report.active_mode + (report.active_mode_valid ? '' : ' (unset — using balanced fallback)'));
|
||||
lines.push('');
|
||||
lines.push('Resolved knobs:');
|
||||
for (const [knob, attr] of Object.entries(report.resolved)) {
|
||||
const value = String(attr.value ?? '(undefined)');
|
||||
lines.push(` ${knob.padEnd(28)} = ${value.padEnd(12)} [${attr.source_detail}]`);
|
||||
}
|
||||
lines.push('');
|
||||
lines.push('Mode bundles (frozen — set via `gbrain config set search.mode <mode>`):');
|
||||
for (const mode of SEARCH_MODES) {
|
||||
const b = report.bundles[mode];
|
||||
const active = mode === report.active_mode ? ' ← active' : '';
|
||||
lines.push(` ${mode.padEnd(13)}${active}`);
|
||||
lines.push(` cache=${b.cache_enabled} intentWeighting=${b.intentWeighting}`);
|
||||
lines.push(` tokenBudget=${b.tokenBudget ?? 'none'} searchLimit=${b.searchLimit} expansion=${b.expansion}`);
|
||||
}
|
||||
lines.push('');
|
||||
lines.push('Knob descriptions:');
|
||||
for (const [k, desc] of Object.entries(KNOB_DESCRIPTIONS)) {
|
||||
lines.push(` ${k.padEnd(28)} ${desc}`);
|
||||
}
|
||||
return lines.join('\n');
|
||||
}
|
||||
|
||||
async function runModesSubcommand(engine: BrainEngine, args: string[]): Promise<void> {
|
||||
const json = args.includes('--json');
|
||||
const reset = args.includes('--reset');
|
||||
const sourceIdx = args.indexOf('--source');
|
||||
const dryRunSource = sourceIdx !== -1 ? args[sourceIdx + 1] : null;
|
||||
|
||||
// --reset path: clear every search.* OVERRIDE key (not search.mode itself).
|
||||
// --source <mode> is a dry-run that prints what would change.
|
||||
if (reset || dryRunSource) {
|
||||
const dryRun = Boolean(dryRunSource);
|
||||
if (dryRunSource && !isSearchMode(dryRunSource)) {
|
||||
console.error(`Invalid --source value: ${dryRunSource}. Expected one of: ${SEARCH_MODES.join(', ')}`);
|
||||
process.exit(1);
|
||||
}
|
||||
const overrides = await engine.listConfigKeys('search.');
|
||||
const toRemove = overrides.filter((k) => k !== SEARCH_MODE_KEY && k !== 'search.mode_upgrade_notice_shown');
|
||||
if (toRemove.length === 0) {
|
||||
console.log('No search.* overrides set. Mode bundle is the only voice.');
|
||||
return;
|
||||
}
|
||||
if (dryRun) {
|
||||
console.log(`--source ${dryRunSource} (dry run). Would unset ${toRemove.length} key(s):`);
|
||||
for (const k of toRemove) console.log(` - ${k}`);
|
||||
console.log(`No changes written. Re-run with --reset to apply.`);
|
||||
return;
|
||||
}
|
||||
let deleted = 0;
|
||||
for (const k of toRemove) {
|
||||
const n = await engine.unsetConfig(k);
|
||||
deleted += n;
|
||||
}
|
||||
console.log(`Reset complete. Unset ${deleted} key(s):`);
|
||||
for (const k of toRemove) console.log(` - ${k}`);
|
||||
console.log(`Mode bundle is now the only voice. Verify with: gbrain search modes`);
|
||||
return;
|
||||
}
|
||||
|
||||
// Default: read-only dashboard.
|
||||
const report = await buildModesReport(engine);
|
||||
if (json) {
|
||||
console.log(JSON.stringify(report, null, 2));
|
||||
} else {
|
||||
console.log(formatModesText(report));
|
||||
}
|
||||
}
|
||||
|
||||
async function runStatsSubcommand(engine: BrainEngine, args: string[]): Promise<void> {
|
||||
const json = args.includes('--json');
|
||||
const daysIdx = args.indexOf('--days');
|
||||
const days = daysIdx !== -1 ? parseInt(args[daysIdx + 1], 10) : 7;
|
||||
|
||||
const stats = await readSearchStats(engine, { days: Number.isFinite(days) ? days : 7 });
|
||||
|
||||
if (json) {
|
||||
console.log(JSON.stringify({
|
||||
schema_version: 2,
|
||||
...stats,
|
||||
_meta: {
|
||||
metric_glossary: {
|
||||
cache_hit_rate: 'cache_hits / (cache_hits + cache_misses) — fraction of searches that reused a recent answer instead of running fresh',
|
||||
avg_results: 'mean number of result rows returned per search call',
|
||||
avg_tokens: 'mean estimated tokens in the returned chunk text (char/4 heuristic)',
|
||||
total_budget_dropped: 'sum of results dropped because the call exceeded its tokenBudget',
|
||||
},
|
||||
},
|
||||
}, null, 2));
|
||||
return;
|
||||
}
|
||||
|
||||
console.log(`Search stats over the last ${stats.window_days} days:`);
|
||||
console.log('');
|
||||
console.log(` Total searches: ${stats.total_calls}`);
|
||||
if (stats.total_calls === 0) {
|
||||
console.log('');
|
||||
console.log('No telemetry recorded yet. Run a few `gbrain query` calls and re-check.');
|
||||
return;
|
||||
}
|
||||
const hitRatePct = (stats.cache_hit_rate * 100).toFixed(1);
|
||||
console.log(` Cache hit rate: ${hitRatePct}% (${stats.cache_hits} hit / ${stats.cache_misses} miss)`);
|
||||
console.log(` (fraction of searches that reused a recent answer)`);
|
||||
console.log(` Avg results returned: ${stats.avg_results.toFixed(1)}`);
|
||||
console.log(` Avg tokens delivered: ${stats.avg_tokens.toFixed(0)} (char/4 heuristic)`);
|
||||
console.log(` Budget drops total: ${stats.total_budget_dropped}`);
|
||||
console.log('');
|
||||
console.log(' Mode distribution:');
|
||||
for (const [m, c] of Object.entries(stats.mode_distribution).sort((a, b) => b[1] - a[1])) {
|
||||
const pct = ((c / stats.total_calls) * 100).toFixed(1);
|
||||
console.log(` ${m.padEnd(14)} ${c} (${pct}%)`);
|
||||
}
|
||||
console.log('');
|
||||
console.log(' Intent distribution:');
|
||||
for (const [i, c] of Object.entries(stats.intent_distribution).sort((a, b) => b[1] - a[1])) {
|
||||
const pct = ((c / stats.total_calls) * 100).toFixed(1);
|
||||
console.log(` ${i.padEnd(14)} ${c} (${pct}%)`);
|
||||
}
|
||||
if (stats.oldest_seen || stats.newest_seen) {
|
||||
console.log('');
|
||||
console.log(` Window: ${stats.oldest_seen ?? '?'} → ${stats.newest_seen ?? '?'}`);
|
||||
}
|
||||
}
|
||||
|
||||
interface TuneRecommendation {
|
||||
knob: string;
|
||||
current: unknown;
|
||||
suggested: unknown;
|
||||
reason: string;
|
||||
apply_command: string;
|
||||
}
|
||||
|
||||
async function runTuneSubcommand(engine: BrainEngine, args: string[]): Promise<void> {
|
||||
const json = args.includes('--json');
|
||||
const apply = args.includes('--apply');
|
||||
|
||||
const modeInput = await loadSearchModeConfig(engine);
|
||||
const resolved = resolveSearchMode(modeInput);
|
||||
const stats = await readSearchStats(engine, { days: 7 });
|
||||
|
||||
const recs: TuneRecommendation[] = [];
|
||||
|
||||
// Recommendation 1: low call volume → no data yet.
|
||||
if (stats.total_calls < 20) {
|
||||
if (json) {
|
||||
console.log(JSON.stringify({
|
||||
schema_version: 2,
|
||||
status: 'insufficient_data',
|
||||
total_calls: stats.total_calls,
|
||||
recommendations: [],
|
||||
message: 'Not enough search activity in the last 7 days to tune. Run `gbrain search stats` after some real usage.',
|
||||
}, null, 2));
|
||||
return;
|
||||
}
|
||||
console.log('Not enough search activity in the last 7 days to tune.');
|
||||
console.log(`Total searches: ${stats.total_calls} (need >= 20 for confident recommendations).`);
|
||||
console.log('Run a few `gbrain query` calls, then re-run `gbrain search tune`.');
|
||||
return;
|
||||
}
|
||||
|
||||
// Recommendation 2: budget pressure under conservative.
|
||||
if (resolved.resolved_mode === 'conservative' && stats.total_calls > 0) {
|
||||
const dropPctPerCall = stats.total_budget_dropped / stats.total_calls;
|
||||
if (dropPctPerCall > 2) {
|
||||
recs.push({
|
||||
knob: 'search.mode',
|
||||
current: 'conservative',
|
||||
suggested: 'balanced',
|
||||
reason: `Avg ${dropPctPerCall.toFixed(1)} results dropped per search by the 4K budget. Consider balanced (12K budget) or raise search.tokenBudget.`,
|
||||
apply_command: 'gbrain config set search.mode balanced',
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
// Recommendation 3: high cache hit rate → bump similarity threshold.
|
||||
if (stats.cache_hit_rate > 0.85 && stats.cache_hits + stats.cache_misses > 50) {
|
||||
recs.push({
|
||||
knob: 'search.cache.similarity_threshold',
|
||||
current: resolved.cache_similarity_threshold,
|
||||
suggested: 0.94,
|
||||
reason: `Cache hit rate is ${(stats.cache_hit_rate * 100).toFixed(1)}%. You can raise similarity threshold to 0.94 for tighter freshness at small recall cost.`,
|
||||
apply_command: 'gbrain config set search.cache.similarity_threshold 0.94',
|
||||
});
|
||||
}
|
||||
|
||||
// Recommendation 4: tokenmax + Haiku subagent.
|
||||
const subagentModel = await engine.getConfig('models.tier.subagent');
|
||||
if (resolved.resolved_mode === 'tokenmax' && subagentModel && /haiku/i.test(subagentModel)) {
|
||||
recs.push({
|
||||
knob: 'search.mode',
|
||||
current: 'tokenmax',
|
||||
suggested: 'balanced',
|
||||
reason: `Subagent tier is Haiku but mode is tokenmax. LLM expansion adds ~50ms + ~1¢ per query. Balanced cuts that cost without losing intent weighting or cache.`,
|
||||
apply_command: 'gbrain config set search.mode balanced',
|
||||
});
|
||||
}
|
||||
|
||||
// Recommendation 5: cache disabled but available — fix the free win.
|
||||
if (!resolved.cache_enabled && stats.total_calls > 5) {
|
||||
recs.push({
|
||||
knob: 'search.cache.enabled',
|
||||
current: false,
|
||||
suggested: true,
|
||||
reason: 'Cache is disabled but mode bundles enable it by default. Cache is a free win (zero LLM cost, big latency drop on repeat queries).',
|
||||
apply_command: 'gbrain config unset search.cache.enabled',
|
||||
});
|
||||
}
|
||||
|
||||
if (json) {
|
||||
console.log(JSON.stringify({
|
||||
schema_version: 2,
|
||||
status: recs.length === 0 ? 'no_recommendations' : 'has_recommendations',
|
||||
total_calls: stats.total_calls,
|
||||
cache_hit_rate: stats.cache_hit_rate,
|
||||
active_mode: resolved.resolved_mode,
|
||||
recommendations: recs,
|
||||
applied: apply ? recs.map(r => r.apply_command) : [],
|
||||
_meta: {
|
||||
metric_glossary: {
|
||||
cache_hit_rate: 'cache_hits / (cache_hits + cache_misses)',
|
||||
total_calls: 'total searches recorded in the last 7 days',
|
||||
},
|
||||
},
|
||||
}, null, 2));
|
||||
if (apply) {
|
||||
for (const r of recs) {
|
||||
await maybeApplyRecommendation(engine, r);
|
||||
}
|
||||
}
|
||||
return;
|
||||
}
|
||||
|
||||
console.log(`Search tune (last 7 days, active mode: ${resolved.resolved_mode}):`);
|
||||
console.log('');
|
||||
|
||||
if (recs.length === 0) {
|
||||
console.log(' No recommendations. Your search config looks well-tuned.');
|
||||
return;
|
||||
}
|
||||
|
||||
for (let i = 0; i < recs.length; i++) {
|
||||
const r = recs[i];
|
||||
console.log(` ${i + 1}. ${r.knob}: ${String(r.current)} → ${String(r.suggested)}`);
|
||||
console.log(` ${r.reason}`);
|
||||
console.log(` Apply: ${r.apply_command}`);
|
||||
console.log('');
|
||||
}
|
||||
|
||||
if (apply) {
|
||||
console.log('Applying recommendations:');
|
||||
const reverts: string[] = [];
|
||||
for (const r of recs) {
|
||||
await maybeApplyRecommendation(engine, r);
|
||||
console.log(` ✓ ${r.apply_command}`);
|
||||
reverts.push(buildRevertCommand(r));
|
||||
}
|
||||
console.log('');
|
||||
console.log('To revert these changes:');
|
||||
for (const cmd of reverts) console.log(` ${cmd}`);
|
||||
} else {
|
||||
console.log('Run `gbrain search tune --apply` to apply these changes automatically.');
|
||||
}
|
||||
}
|
||||
|
||||
async function maybeApplyRecommendation(engine: BrainEngine, r: TuneRecommendation): Promise<void> {
|
||||
// The apply command is the canonical paste-ready string; here we
|
||||
// re-parse it to call setConfig / unsetConfig directly so the call
|
||||
// happens in-process.
|
||||
const parts = r.apply_command.split(/\s+/);
|
||||
if (parts[0] !== 'gbrain' || parts[1] !== 'config') return;
|
||||
if (parts[2] === 'set' && parts.length === 5) {
|
||||
await engine.setConfig(parts[3], parts[4]);
|
||||
} else if (parts[2] === 'unset' && parts.length === 4) {
|
||||
await engine.unsetConfig(parts[3]);
|
||||
}
|
||||
}
|
||||
|
||||
function buildRevertCommand(r: TuneRecommendation): string {
|
||||
const parts = r.apply_command.split(/\s+/);
|
||||
if (parts[2] === 'set') {
|
||||
return `gbrain config set ${parts[3]} ${String(r.current)}`;
|
||||
} else if (parts[2] === 'unset') {
|
||||
return `gbrain config set ${parts[3]} ${String(r.current)}`;
|
||||
}
|
||||
return r.apply_command;
|
||||
}
|
||||
|
||||
const USAGE = `Usage: gbrain search <modes|stats|tune> [flags]
|
||||
|
||||
Subcommands:
|
||||
modes [--json] Show active mode, bundles, and per-knob source.
|
||||
modes --reset Clear all search.* overrides (mode bundle wins).
|
||||
modes --source <mode> Dry-run: list what --reset would change.
|
||||
stats [--days N] [--json] Cache hit rate, intent mix, budget pressure.
|
||||
tune [--apply] [--json] Print recommendations; --apply mutates config.
|
||||
|
||||
Examples:
|
||||
gbrain search modes
|
||||
gbrain search modes --reset
|
||||
gbrain search stats --days 30 --json
|
||||
gbrain search tune
|
||||
gbrain search tune --apply
|
||||
`;
|
||||
|
||||
export async function runSearch(engine: BrainEngine, args: string[]): Promise<void> {
|
||||
const sub = args[0];
|
||||
const rest = args.slice(1);
|
||||
|
||||
if (!sub || sub === '--help' || sub === '-h') {
|
||||
console.log(USAGE);
|
||||
return;
|
||||
}
|
||||
|
||||
switch (sub) {
|
||||
case 'modes':
|
||||
await runModesSubcommand(engine, rest);
|
||||
return;
|
||||
case 'stats':
|
||||
await runStatsSubcommand(engine, rest);
|
||||
return;
|
||||
case 'tune':
|
||||
await runTuneSubcommand(engine, rest);
|
||||
return;
|
||||
default:
|
||||
console.error(`Unknown subcommand: ${sub}`);
|
||||
console.error(USAGE);
|
||||
process.exit(1);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* `gbrain search modes` is read-only — no DB connection strictly required
|
||||
* for the bundle display IF the engine is given. The dispatch in cli.ts
|
||||
* adds 'search' to its dispatch table so the engine connects normally;
|
||||
* this export is here so future no-engine modes (e.g. `gbrain search --help`
|
||||
* without an engine) could route through it cleanly.
|
||||
*/
|
||||
export const _exports_for_test = {
|
||||
buildModesReport,
|
||||
formatModesText,
|
||||
maybeApplyRecommendation,
|
||||
buildRevertCommand,
|
||||
};
|
||||
|
||||
// Suppress unused-export TS warning — these are intentionally retained for
|
||||
// downstream callers (cli.ts dispatch / future skill linkage).
|
||||
void DEFAULT_SEARCH_MODE;
|
||||
@@ -211,10 +211,21 @@ export function buildGitInvocation(repoPath: string, args: string[], configs: st
|
||||
return [...cfg, '-C', repoPath, ...args];
|
||||
}
|
||||
|
||||
/**
|
||||
* Shell out to git with a generous maxBuffer.
|
||||
*
|
||||
* Node's default maxBuffer is 1 MiB. `git diff --name-status -M` on a
|
||||
* 60–100K file repo easily exceeds that, causing an ENOBUFS crash that
|
||||
* kills the sync process with no error message in the log.
|
||||
*
|
||||
* 100 MiB is generous but still bounded — a 100K-file diff with long
|
||||
* paths tops out around 10–20 MiB in practice.
|
||||
*/
|
||||
function git(repoPath: string, args: string[], configs: string[] = []): string {
|
||||
return execFileSync('git', buildGitInvocation(repoPath, args, configs), {
|
||||
encoding: 'utf-8',
|
||||
timeout: 30000,
|
||||
maxBuffer: 100 * 1024 * 1024,
|
||||
}).trim();
|
||||
}
|
||||
|
||||
|
||||
@@ -257,6 +257,53 @@ export async function runPostUpgrade(args: string[] = []): Promise<void> {
|
||||
await engine.initSchema();
|
||||
console.log(' Schema up to date.');
|
||||
|
||||
// v0.32.3 search-lite mode banner. One-shot: fires at most once per
|
||||
// install (state persisted via `search.mode_upgrade_notice_shown`).
|
||||
// Reframes from "behavior is regressing" to "named modes available"
|
||||
// per [CDX-1+2+3]: the production query op still defaults expand=true
|
||||
// and limit=20 — Garry's behavior is NOT regressing.
|
||||
try {
|
||||
const shown = await engine.getConfig('search.mode_upgrade_notice_shown');
|
||||
const existingMode = await engine.getConfig('search.mode');
|
||||
if (shown !== 'true' && !existingMode) {
|
||||
console.log('');
|
||||
console.log('═══════════════════════════════════════════════════════════════');
|
||||
console.log('[gbrain] v0.32.3 added named search modes (conservative/balanced/tokenmax).');
|
||||
console.log('[gbrain] Your current behavior is unchanged — the `query` op still defaults');
|
||||
console.log('[gbrain] expand=true and limit=20, same as v0.31.x.');
|
||||
console.log('═══════════════════════════════════════════════════════════════');
|
||||
console.log('');
|
||||
console.log('[AGENT] Relay this matrix to your operator and ask which mode they want.');
|
||||
console.log('[AGENT] Recommended default: tokenmax (preserves current retrieval shape).');
|
||||
console.log('[AGENT] Run `gbrain config set search.mode <mode>` after they decide.');
|
||||
console.log('');
|
||||
console.log('Per-query cost @ 10K queries/mo (typical single-user volume):');
|
||||
console.log('');
|
||||
console.log(' Haiku 4.5 Sonnet 4.6 Opus 4.7');
|
||||
console.log(' ($1/M) ($3/M) ($5/M)');
|
||||
console.log(' conservative $40/mo $120/mo $200/mo');
|
||||
console.log(' balanced $100/mo $300/mo $500/mo');
|
||||
console.log(' tokenmax $200/mo $600/mo $1,000/mo');
|
||||
console.log('');
|
||||
console.log(' (scales linearly — multiply by 10 for 100K/mo)');
|
||||
console.log(' 25x corner-to-corner spread. Natural diagonal pairings span ~4x.');
|
||||
console.log('');
|
||||
console.log('To pick:');
|
||||
console.log(' gbrain search modes # see what is running');
|
||||
console.log(' gbrain config set search.mode <conservative|balanced|tokenmax>');
|
||||
console.log(' gbrain search tune # data-driven recommendations');
|
||||
console.log('');
|
||||
console.log('tokenmax bumps limit to 50 (current default is 20). To preserve');
|
||||
console.log('your EXACT current shape:');
|
||||
console.log(' gbrain config set search.mode tokenmax');
|
||||
console.log(' gbrain config set search.searchLimit 20');
|
||||
console.log('');
|
||||
await engine.setConfig('search.mode_upgrade_notice_shown', 'true');
|
||||
}
|
||||
} catch {
|
||||
// Banner is cosmetic; never block the upgrade.
|
||||
}
|
||||
|
||||
// v0.32.7 CJK wave: chunker-version bump → re-embed sweep.
|
||||
// Idempotent — `runReindex` short-circuits when no pages are pending.
|
||||
try {
|
||||
|
||||
+51
-6
@@ -10,7 +10,16 @@
|
||||
*/
|
||||
|
||||
import type { Implementation } from './types.ts';
|
||||
import { AIConfigError } from './errors.ts';
|
||||
|
||||
// Voyage hosted models that accept `output_dimension` (values:
|
||||
// 256 / 512 / 1024 / 2048). Per Voyage's API parameter docs as of 2026-05.
|
||||
// voyage-4-nano is intentionally NOT in this set: it's the open-weight
|
||||
// variant listed separately by Voyage as fixed 1024-dim. Adding it here
|
||||
// would tell the SDK to send `dimensions: N`, which voyageCompatFetch then
|
||||
// rewrites to `output_dimension: N` — and Voyage's hosted nano endpoint
|
||||
// rejects the parameter. The negative regression assertion in
|
||||
// test/ai/gateway.test.ts pins this contract.
|
||||
const VOYAGE_OUTPUT_DIMENSION_MODELS = new Set([
|
||||
'voyage-4-large',
|
||||
'voyage-4',
|
||||
@@ -21,13 +30,32 @@ const VOYAGE_OUTPUT_DIMENSION_MODELS = new Set([
|
||||
'voyage-code-3',
|
||||
]);
|
||||
|
||||
// Voyage's flexible-dim endpoint only accepts these four discrete values.
|
||||
// Per Voyage's API docs (2026-05). Out-of-range requests are rejected with
|
||||
// HTTP 400 by the upstream — catching it locally produces a clearer error
|
||||
// with the valid-values hint. The most common way to hit this in
|
||||
// production: `embedding_model: voyage:voyage-4-large` configured without
|
||||
// `embedding_dimensions`, where the gateway falls back to
|
||||
// DEFAULT_EMBEDDING_DIMENSIONS=1536 (an OpenAI default, not a Voyage one).
|
||||
export const VOYAGE_VALID_OUTPUT_DIMS = [256, 512, 1024, 2048] as const;
|
||||
|
||||
export function supportsVoyageOutputDimension(modelId: string): boolean {
|
||||
return VOYAGE_OUTPUT_DIMENSION_MODELS.has(modelId);
|
||||
}
|
||||
|
||||
export function isValidVoyageOutputDim(dims: number): boolean {
|
||||
return (VOYAGE_VALID_OUTPUT_DIMS as readonly number[]).includes(dims);
|
||||
}
|
||||
|
||||
/**
|
||||
* Build the providerOptions blob for embedMany() that pins output dimensions.
|
||||
*
|
||||
* Matryoshka providers (OpenAI text-embedding-3, Gemini embedding-001) can be
|
||||
* asked to return reduced-dim vectors. Anthropic does not take a dimension
|
||||
* parameter. Most openai-compatible providers do not either, but Voyage's
|
||||
* OpenAI-compatible embeddings endpoint accepts `output_dimension`.
|
||||
* parameter. Most openai-compatible providers do not either. Voyage's
|
||||
* endpoint accepts `output_dimension`, but the AI SDK openai-compatible
|
||||
* adapter only forwards `dimensions`; gateway.ts translates that field to
|
||||
* Voyage's wire name in voyageCompatFetch.
|
||||
*/
|
||||
export function dimsProviderOptions(
|
||||
implementation: Implementation,
|
||||
@@ -53,10 +81,27 @@ export function dimsProviderOptions(
|
||||
return undefined;
|
||||
case 'openai-compatible':
|
||||
// Most openai-compatible providers (Ollama, LM Studio, vLLM, LiteLLM)
|
||||
// do not expose a standard dimensions knob. Voyage's compat endpoint is
|
||||
// the exception: it accepts output_dimension and defaults to 1024 dims.
|
||||
if (VOYAGE_OUTPUT_DIMENSION_MODELS.has(modelId)) {
|
||||
return { openaiCompatible: { output_dimension: dims } };
|
||||
// do not expose a standard dimensions knob. Voyage is the exception,
|
||||
// but it needs the SDK-supported field here so voyageCompatFetch can
|
||||
// translate it to `output_dimension` before the HTTP request is sent.
|
||||
if (supportsVoyageOutputDimension(modelId)) {
|
||||
// Fail-loud at the embed boundary if the user configured a dim
|
||||
// Voyage doesn't accept. The most common path here: a brain with
|
||||
// `embedding_model: voyage:voyage-4-large` but no explicit
|
||||
// `embedding_dimensions`, where the gateway falls back to the
|
||||
// module default (1536). Without this guard, Voyage's HTTP 400 is
|
||||
// the only signal — usually mis-attributed as a transient network
|
||||
// error.
|
||||
if (!isValidVoyageOutputDim(dims)) {
|
||||
throw new AIConfigError(
|
||||
`Voyage model "${modelId}" supports output_dimension only in ` +
|
||||
`{${VOYAGE_VALID_OUTPUT_DIMS.join(', ')}}, got ${dims}.`,
|
||||
`Set \`embedding_dimensions\` to one of ` +
|
||||
`${VOYAGE_VALID_OUTPUT_DIMS.join('/')} in your gbrain config, or ` +
|
||||
`switch to a fixed-dim Voyage model (e.g. voyage-3, voyage-3-lite).`,
|
||||
);
|
||||
}
|
||||
return { openaiCompatible: { dimensions: dims } };
|
||||
}
|
||||
// OpenAI text-embedding-3 family on the openai-compatible adapter
|
||||
// (Azure OpenAI hosts these via its OpenAI-compatible /embeddings
|
||||
|
||||
+28
-3
@@ -139,6 +139,24 @@ const DEFAULT_SAFETY_FACTOR = 0.8;
|
||||
*/
|
||||
const MAX_VOYAGE_RESPONSE_BYTES = 256 * 1024 * 1024;
|
||||
|
||||
/**
|
||||
* Tagged error class for the OOM-defense caps in voyageCompatFetch. The
|
||||
* inbound response-rewriter at the bottom of voyageCompatFetch is wrapped
|
||||
* in a try/catch that silently falls back to the original response on parse
|
||||
* failure — that's correct for "Voyage returned something I can't reshape,
|
||||
* let the SDK handle it" but WRONG for OOM caps where letting the response
|
||||
* through could blow up the worker. The catch checks `instanceof
|
||||
* VoyageResponseTooLargeError` and rethrows in that case.
|
||||
*
|
||||
* Exported for tests; not part of the public surface.
|
||||
*/
|
||||
export class VoyageResponseTooLargeError extends Error {
|
||||
constructor(message: string) {
|
||||
super(message);
|
||||
this.name = 'VoyageResponseTooLargeError';
|
||||
}
|
||||
}
|
||||
|
||||
// ---- Unified auth resolution (D12=A) ----
|
||||
//
|
||||
// Pre-v0.32, openai-compatible auth was duplicated across instantiateEmbedding,
|
||||
@@ -592,7 +610,7 @@ const voyageCompatFetch = (async (input: RequestInfo | URL, init?: RequestInit)
|
||||
if (contentLengthHeader) {
|
||||
const len = parseInt(contentLengthHeader, 10);
|
||||
if (Number.isFinite(len) && len > MAX_VOYAGE_RESPONSE_BYTES) {
|
||||
throw new Error(
|
||||
throw new VoyageResponseTooLargeError(
|
||||
`Voyage response Content-Length=${len} exceeds ${MAX_VOYAGE_RESPONSE_BYTES} bytes — ` +
|
||||
`likely compromised endpoint or misconfiguration`,
|
||||
);
|
||||
@@ -617,7 +635,7 @@ const voyageCompatFetch = (async (input: RequestInfo | URL, init?: RequestInit)
|
||||
// base64 → bytes ratio).
|
||||
const estDecoded = Math.ceil(item.embedding.length * 0.75);
|
||||
if (estDecoded > MAX_VOYAGE_RESPONSE_BYTES) {
|
||||
throw new Error(
|
||||
throw new VoyageResponseTooLargeError(
|
||||
`Voyage embedding base64 exceeds ${MAX_VOYAGE_RESPONSE_BYTES} bytes ` +
|
||||
`(estimated ${estDecoded} bytes from ${item.embedding.length} base64 chars)`,
|
||||
);
|
||||
@@ -646,7 +664,14 @@ const voyageCompatFetch = (async (input: RequestInfo | URL, init?: RequestInit)
|
||||
statusText: resp.statusText,
|
||||
headers: resp.headers,
|
||||
});
|
||||
} catch {
|
||||
} catch (err) {
|
||||
// OOM-cap throws MUST propagate. The catch is here for "Voyage returned
|
||||
// JSON I can't reshape" (parse error, unexpected schema) — falling back
|
||||
// to the original response is correct in that case. Letting the
|
||||
// too-large response through here would defeat the entire purpose of
|
||||
// Layer 2 (the per-embedding cap that fires when Content-Length wasn't
|
||||
// available to Layer 1).
|
||||
if (err instanceof VoyageResponseTooLargeError) throw err;
|
||||
// If parsing/transformation fails, fall back to the original response.
|
||||
return resp;
|
||||
}
|
||||
|
||||
@@ -4,9 +4,15 @@ import type { Recipe } from '../types.ts';
|
||||
* Voyage AI exposes an OpenAI-compatible /embeddings endpoint.
|
||||
* Base URL: https://api.voyageai.com/v1
|
||||
*
|
||||
* Voyage 4 family (Jan 2026): shared embedding space across all v4 variants,
|
||||
* flexible dims (256/512/1024/2048), 32K context, MoE architecture (large).
|
||||
* You can index with voyage-4-large and query with voyage-4-lite — no reindex.
|
||||
* Hosted v4 trio (voyage-4-large / voyage-4 / voyage-4-lite, Jan 2026):
|
||||
* shared embedding space, flexible dims (256/512/1024/2048), 32K context,
|
||||
* MoE architecture (large). You can index with voyage-4-large and query with
|
||||
* voyage-4-lite — no reindex.
|
||||
*
|
||||
* voyage-4-nano is a DIFFERENT thing: an open-weight variant Voyage lists
|
||||
* separately. It does NOT accept the `output_dimension` parameter on
|
||||
* Voyage's hosted API — fixed 1024-dim. See VOYAGE_OUTPUT_DIMENSION_MODELS
|
||||
* in src/core/ai/dims.ts; nano is intentionally excluded.
|
||||
*
|
||||
* voyage-multimodal-3 (v0.27.1): text + image inputs in the same 1024-dim
|
||||
* space. supports_multimodal flips routing to embedMultimodal() in the
|
||||
|
||||
@@ -101,7 +101,7 @@ export async function getCachedTraversal<T>(
|
||||
cluster_generation: row.cluster_generation,
|
||||
};
|
||||
} catch {
|
||||
// Cache table missing on a pre-v56 brain — fall through as miss.
|
||||
// Cache table missing on a pre-v59 brain — fall through as miss.
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
@@ -1147,6 +1147,18 @@ export interface BrainEngine {
|
||||
// Config
|
||||
getConfig(key: string): Promise<string | null>;
|
||||
setConfig(key: string, value: string): Promise<void>;
|
||||
/**
|
||||
* v0.32.3 — delete a config row. Returns the number of rows deleted (0 or 1).
|
||||
* No-op when the key doesn't exist. Used by `gbrain config unset` and by
|
||||
* `gbrain search modes --reset`. Engine-agnostic.
|
||||
*/
|
||||
unsetConfig(key: string): Promise<number>;
|
||||
/**
|
||||
* v0.32.3 — list config keys matching a literal prefix (e.g. "search.").
|
||||
* Used by `gbrain config unset --pattern` and the search-modes --reset path.
|
||||
* Does NOT support glob/regex on purpose — the caller knows the prefix.
|
||||
*/
|
||||
listConfigKeys(prefix: string): Promise<string[]>;
|
||||
|
||||
// Migration support
|
||||
runMigration(version: number, sql: string): Promise<void>;
|
||||
|
||||
@@ -0,0 +1,92 @@
|
||||
/**
|
||||
* v0.32.3 — curated drift-watch list for the eval_drift doctor check.
|
||||
*
|
||||
* Per [CDX-6]: a "search code changed since last eval" warning needs a
|
||||
* precise definition. Too narrow (e.g. only src/core/search/) misses real
|
||||
* regressions like a chunker change. Too wide (every file) trains the
|
||||
* operator to ignore the warning.
|
||||
*
|
||||
* The curated allowlist below names every file whose change MEANINGFULLY
|
||||
* affects retrieval quality. Adding to this list REQUIRES a CHANGELOG line
|
||||
* so coverage grows deliberately.
|
||||
*/
|
||||
|
||||
import { execSync } from 'child_process';
|
||||
import { existsSync } from 'fs';
|
||||
|
||||
/**
|
||||
* Glob-ish patterns watched for retrieval drift. Each pattern is matched
|
||||
* against repo-relative paths via simple `startsWith` semantics (no real
|
||||
* glob expansion) so the matcher is fast + dependency-free.
|
||||
*
|
||||
* If you add a pattern: also add a CHANGELOG line documenting why.
|
||||
*/
|
||||
export const RETRIEVAL_WATCH_PATTERNS: ReadonlyArray<string> = Object.freeze([
|
||||
// Search pipeline core
|
||||
'src/core/search/',
|
||||
// Embedding shape (changing dim or chunker shape moves every result)
|
||||
'src/core/embedding.ts',
|
||||
// Chunkers (recursive + semantic + LLM-guided) — chunk granularity is retrieval
|
||||
'src/core/chunkers/',
|
||||
// AI recipes that drive expansion / embedding choices
|
||||
'src/core/ai/recipes/anthropic.ts',
|
||||
'src/core/ai/recipes/openai.ts',
|
||||
// The query op itself
|
||||
'src/core/operations.ts',
|
||||
]);
|
||||
|
||||
/** Path equality / prefix matcher for the curated list. */
|
||||
export function matchesWatchPattern(path: string, patterns: ReadonlyArray<string> = RETRIEVAL_WATCH_PATTERNS): boolean {
|
||||
for (const p of patterns) {
|
||||
// Trailing-slash pattern = directory prefix
|
||||
if (p.endsWith('/')) {
|
||||
if (path.startsWith(p)) return true;
|
||||
} else {
|
||||
// Bare-file pattern = exact equality
|
||||
if (path === p) return true;
|
||||
}
|
||||
}
|
||||
return false;
|
||||
}
|
||||
|
||||
/**
|
||||
* Return repo-relative paths that have changed in the working tree since
|
||||
* the given commit (or HEAD if no commit). Best-effort: returns [] when
|
||||
* git is unavailable, the repo lacks the commit, or any other failure.
|
||||
*
|
||||
* `commitSha` is a full or short SHA. When omitted, compares HEAD against
|
||||
* working tree (uncommitted changes only).
|
||||
*/
|
||||
export function filesDriftedSince(repoRoot: string, commitSha?: string): string[] {
|
||||
if (!existsSync(repoRoot)) return [];
|
||||
try {
|
||||
const range = commitSha ? `${commitSha}..HEAD` : 'HEAD';
|
||||
const args = commitSha
|
||||
? ['diff', '--name-only', range]
|
||||
: ['diff', '--name-only', 'HEAD'];
|
||||
const out = execSync(`git ${args.join(' ')}`, {
|
||||
cwd: repoRoot,
|
||||
encoding: 'utf-8',
|
||||
stdio: ['pipe', 'pipe', 'pipe'],
|
||||
timeout: 5000,
|
||||
});
|
||||
return out
|
||||
.split('\n')
|
||||
.map(s => s.trim())
|
||||
.filter((s): s is string => s.length > 0);
|
||||
} catch {
|
||||
return [];
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Identify only the changed files that match the retrieval watch list.
|
||||
* Convenience wrapper for the doctor check + future CI gate.
|
||||
*/
|
||||
export function watchedFilesDrifted(
|
||||
repoRoot: string,
|
||||
commitSha?: string,
|
||||
patterns: ReadonlyArray<string> = RETRIEVAL_WATCH_PATTERNS,
|
||||
): string[] {
|
||||
return filesDriftedSince(repoRoot, commitSha).filter((p) => matchesWatchPattern(p, patterns));
|
||||
}
|
||||
@@ -0,0 +1,215 @@
|
||||
/**
|
||||
* v0.32.3 — Single source of truth for evaluation-metric plain-English
|
||||
* glosses. Drives `gbrain search stats` JSON output (`_meta.metric_glossary`
|
||||
* block), `gbrain eval compare` reports, and the auto-generated
|
||||
* `docs/eval/METRIC_GLOSSARY.md` file.
|
||||
*
|
||||
* Per [CDX-25]: glosses live in one `_meta.metric_glossary` block per
|
||||
* response, NOT as sibling `_gloss` fields on every metric. Less invasive
|
||||
* to machine-readable consumers (gbrain-evals repo, CI gates).
|
||||
*
|
||||
* Every entry has THREE fields:
|
||||
* - industry_term: the canonical name used in IR / NLP literature
|
||||
* (preserved verbatim so users searching the literature find what we
|
||||
* report)
|
||||
* - eli10: plain-English explanation a 16-year-old could follow
|
||||
* - range: explicit numeric range + interpretation ("higher is better",
|
||||
* "0..1 where 1 = perfect")
|
||||
*
|
||||
* The doc generator at `scripts/generate-metric-glossary.ts` consumes this
|
||||
* module and writes `docs/eval/METRIC_GLOSSARY.md`. A CI guard
|
||||
* (`scripts/check-eval-glossary-fresh.sh`) regenerates the doc and diffs
|
||||
* against the committed version — out-of-date docs fail the build.
|
||||
*/
|
||||
|
||||
export interface MetricGlossEntry {
|
||||
industry_term: string;
|
||||
eli10: string;
|
||||
range: string;
|
||||
}
|
||||
|
||||
export const METRIC_GLOSSARY: Readonly<Record<string, Readonly<MetricGlossEntry>>> = Object.freeze({
|
||||
// ────────────────────────────────────────────────────────────────────────
|
||||
// Retrieval metrics (IR literature)
|
||||
// ────────────────────────────────────────────────────────────────────────
|
||||
'precision@k': Object.freeze({
|
||||
industry_term: 'Precision at k (P@k)',
|
||||
eli10: 'Of the top k results the engine returned, what fraction were actually relevant? High precision means few junk results in the top of the list.',
|
||||
range: '0..1, higher is better. P@10 = 0.7 means 7 of the top 10 results were on-topic.',
|
||||
}),
|
||||
'recall@k': Object.freeze({
|
||||
industry_term: 'Recall at k (R@k)',
|
||||
eli10: 'Of all the relevant results that exist in the brain, what fraction did the engine find in its top k? High recall means few missed answers.',
|
||||
range: '0..1, higher is better. R@10 = 0.81 means out of every 100 questions, the right answer was in the top 10 for 81 of them.',
|
||||
}),
|
||||
'mrr': Object.freeze({
|
||||
industry_term: 'Mean Reciprocal Rank (MRR)',
|
||||
eli10: 'On average, how far down the list is the FIRST relevant result? An MRR of 1.0 means the first hit is always right; an MRR of 0.5 means it\'s typically at rank 2.',
|
||||
range: '0..1, higher is better. Computed as the average of 1/rank-of-first-relevant-result across all test queries.',
|
||||
}),
|
||||
'ndcg@k': Object.freeze({
|
||||
industry_term: 'Normalized Discounted Cumulative Gain at k (nDCG@k)',
|
||||
eli10: 'Like precision@k, but the engine gets MORE credit for putting good results near the top than near rank k. A perfect ordering scores 1.0; a totally random ordering scores near 0.',
|
||||
range: '0..1, higher is better. nDCG@10 above 0.65 is the common "ship it" threshold for hybrid retrieval on technical corpora.',
|
||||
}),
|
||||
|
||||
// ────────────────────────────────────────────────────────────────────────
|
||||
// Set-similarity / stability metrics (replay + regression checks)
|
||||
// ────────────────────────────────────────────────────────────────────────
|
||||
'jaccard@k': Object.freeze({
|
||||
industry_term: 'Jaccard similarity at k (set Jaccard @k)',
|
||||
eli10: 'How much do two result lists overlap? Compare the top k slugs from the captured baseline against the current run; Jaccard@10 = 1.0 means perfect agreement, 0.0 means zero overlap.',
|
||||
range: '0..1, higher = more stable. Below 0.5 on a stable corpus means retrieval changed significantly.',
|
||||
}),
|
||||
'top1_stability': Object.freeze({
|
||||
industry_term: 'Top-1 stability rate',
|
||||
eli10: 'Fraction of queries where the #1 result is the same between two runs. The most aggressive stability check — small ranking shifts that don\'t change the top answer don\'t hurt it.',
|
||||
range: '0..1, higher = more stable. Above 0.85 typically means safe-to-merge for retrieval changes.',
|
||||
}),
|
||||
|
||||
// ────────────────────────────────────────────────────────────────────────
|
||||
// Statistical-significance metrics (per-mode comparison)
|
||||
// ────────────────────────────────────────────────────────────────────────
|
||||
'p_value': Object.freeze({
|
||||
industry_term: 'p-value (paired bootstrap)',
|
||||
eli10: 'How likely the observed difference between two modes is just noise. Lower = stronger evidence the difference is real. We compute paired bootstrap with 10,000 resamples and Bonferroni correction across the 12 comparisons (3 modes × 4 metrics).',
|
||||
range: '0..1, lower = stronger signal. Below 0.05 is the common "statistically significant" threshold; below 0.01 is strong evidence.',
|
||||
}),
|
||||
'confidence_interval': Object.freeze({
|
||||
industry_term: '95% Confidence Interval (CI)',
|
||||
eli10: 'The range we\'re 95% sure the true value falls inside, given the sample we measured. Narrower CI = more reliable estimate. Computed via bootstrap resampling.',
|
||||
range: 'Two-tuple [low, high]. If 0 is inside the CI for a Δ, the difference isn\'t statistically significant.',
|
||||
}),
|
||||
|
||||
// ────────────────────────────────────────────────────────────────────────
|
||||
// Operational / cost metrics
|
||||
// ────────────────────────────────────────────────────────────────────────
|
||||
'cache_hit_rate': Object.freeze({
|
||||
industry_term: 'Cache hit rate',
|
||||
eli10: 'Fraction of searches that reused a recent cached answer instead of running fresh. Higher hit rate = lower latency + lower LLM spend, but stale results may slip through if the threshold is too loose.',
|
||||
range: '0..1, higher generally better. 0.7-0.9 is the sweet spot for a busy brain; above 0.9 may indicate the similarity threshold is too loose.',
|
||||
}),
|
||||
'avg_results': Object.freeze({
|
||||
industry_term: 'Average results returned',
|
||||
eli10: 'Mean number of search-result rows the engine returned per call. Should be near the active mode\'s searchLimit unless the brain is small or the budget is dropping results.',
|
||||
range: '0..searchLimit. Far below searchLimit suggests budget pressure or sparse retrieval.',
|
||||
}),
|
||||
'avg_tokens': Object.freeze({
|
||||
industry_term: 'Average tokens delivered',
|
||||
eli10: 'Estimated tokens (chars / 4) in the chunk text returned per search call. The direct measure of how much context an agent loop is paying for each search.',
|
||||
range: '0..tokenBudget. Approximates OpenAI tiktoken count for English; off by ~5-10% for Anthropic and worse for non-English.',
|
||||
}),
|
||||
'cost_per_query_usd': Object.freeze({
|
||||
industry_term: 'Cost per query (USD)',
|
||||
eli10: 'Sum of LLM + embedding API charges for one search call. Includes Haiku expansion call (tokenmax mode only) + embedding cost + downstream answer-model cost if measured.',
|
||||
range: '0..unbounded. Conservative mode is typically <\\$0.001 per call; tokenmax with answer-gen can exceed \\$0.01.',
|
||||
}),
|
||||
'p99_latency_ms': Object.freeze({
|
||||
industry_term: 'p99 latency (ms)',
|
||||
eli10: '99th percentile wall-clock time per search call. The latency that 1% of users see — long-tail experience, not the average.',
|
||||
range: '0..unbounded. Warm-cache hits should be <50ms; tokenmax with expansion can exceed 200ms due to the Haiku call.',
|
||||
}),
|
||||
});
|
||||
|
||||
/**
|
||||
* Public accessor — returns the gloss entry for a metric, or null if
|
||||
* unknown. Callers that need the structured shape use this; callers that
|
||||
* just need the plain-English line use eli10For().
|
||||
*/
|
||||
export function getMetricGloss(metric: string): MetricGlossEntry | null {
|
||||
if (METRIC_GLOSSARY[metric]) return METRIC_GLOSSARY[metric];
|
||||
// Fuzzy fallback for @N metrics: `recall@10` → `recall@k`, `ndcg@5` → `ndcg@k`.
|
||||
// The glossary documents the family ("at k"); reports use a specific K value.
|
||||
const atK = metric.match(/^(.+)@\d+$/);
|
||||
if (atK) {
|
||||
const family = `${atK[1]}@k`;
|
||||
if (METRIC_GLOSSARY[family]) return METRIC_GLOSSARY[family];
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
/**
|
||||
* Convenience: return ONLY the plain-English gloss for a metric. Used in
|
||||
* `gbrain search stats` JSON output's _meta.metric_glossary block and in
|
||||
* the eval-compare report's per-metric "Plain English:" lines.
|
||||
*/
|
||||
export function eli10For(metric: string): string | null {
|
||||
const g = getMetricGloss(metric);
|
||||
return g?.eli10 ?? null;
|
||||
}
|
||||
|
||||
/**
|
||||
* Build a `_meta.metric_glossary` block for a set of metrics. Returns an
|
||||
* object suitable for JSON.stringify-ing under the `_meta` key in any
|
||||
* eval / stats response.
|
||||
*
|
||||
* Per [CDX-25]: ONE _meta.metric_glossary per response, NOT sibling
|
||||
* _gloss fields on every numeric metric. Adding a metric to the response
|
||||
* doesn't bloat the JSON; the glossary lives in a single place per
|
||||
* response, indexed by metric name.
|
||||
*/
|
||||
export function buildMetricGlossaryMeta(metrics: ReadonlyArray<string>): Record<string, string> {
|
||||
const out: Record<string, string> = {};
|
||||
for (const m of metrics) {
|
||||
const e = getMetricGloss(m); // routes through fuzzy fallback
|
||||
if (e) out[m] = e.eli10;
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
/**
|
||||
* The list of EVERY metric this module documents. Auto-derived from the
|
||||
* METRIC_GLOSSARY keys so the doc generator can iterate without drift.
|
||||
*/
|
||||
export const ALL_METRICS: ReadonlyArray<string> = Object.freeze(Object.keys(METRIC_GLOSSARY));
|
||||
|
||||
/**
|
||||
* Render the glossary as a Markdown document. Consumed by
|
||||
* `scripts/generate-metric-glossary.ts` to produce
|
||||
* `docs/eval/METRIC_GLOSSARY.md`. The CI guard regenerates this and
|
||||
* diffs against the committed file — out-of-date docs fail the build.
|
||||
*
|
||||
* The output is deterministic: same input → same output bytes.
|
||||
*/
|
||||
export function renderMetricGlossaryMarkdown(): string {
|
||||
const lines: string[] = [];
|
||||
lines.push('# Evaluation Metric Glossary');
|
||||
lines.push('');
|
||||
lines.push('**Auto-generated from `src/core/eval/metric-glossary.ts`. Do not edit by hand.** Run `bun run scripts/generate-metric-glossary.ts` to regenerate.');
|
||||
lines.push('');
|
||||
lines.push('Every metric `gbrain eval *` and `gbrain search stats` reports has a plain-English explanation here. Industry terms are preserved verbatim so users searching the literature find what we report.');
|
||||
lines.push('');
|
||||
|
||||
const groups: Array<[string, string[]]> = [
|
||||
['Retrieval Metrics', ['precision@k', 'recall@k', 'mrr', 'ndcg@k']],
|
||||
['Set-Similarity / Stability Metrics', ['jaccard@k', 'top1_stability']],
|
||||
['Statistical-Significance Metrics', ['p_value', 'confidence_interval']],
|
||||
['Operational / Cost Metrics', ['cache_hit_rate', 'avg_results', 'avg_tokens', 'cost_per_query_usd', 'p99_latency_ms']],
|
||||
];
|
||||
|
||||
for (const [groupTitle, metrics] of groups) {
|
||||
lines.push(`## ${groupTitle}`);
|
||||
lines.push('');
|
||||
for (const m of metrics) {
|
||||
const e = METRIC_GLOSSARY[m];
|
||||
if (!e) continue;
|
||||
lines.push(`### ${e.industry_term}`);
|
||||
lines.push('');
|
||||
lines.push(`**Key:** \`${m}\``);
|
||||
lines.push('');
|
||||
lines.push(`**Plain English:** ${e.eli10}`);
|
||||
lines.push('');
|
||||
lines.push(`**Range:** ${e.range}`);
|
||||
lines.push('');
|
||||
}
|
||||
}
|
||||
|
||||
lines.push('---');
|
||||
lines.push('');
|
||||
lines.push('## Coverage');
|
||||
lines.push('');
|
||||
lines.push(`Every metric printed by any \`gbrain eval *\` or \`gbrain search stats\` command resolves through \`getMetricGloss()\` in \`src/core/eval/metric-glossary.ts\`. Adding a new metric to the glossary REQUIRES updating this doc; the CI guard catches drift.`);
|
||||
lines.push('');
|
||||
|
||||
return lines.join('\n') + '\n';
|
||||
}
|
||||
+182
-2
@@ -2705,9 +2705,11 @@ export const MIGRATIONS: Migration[] = [
|
||||
`,
|
||||
},
|
||||
{
|
||||
version: 56,
|
||||
version: 59,
|
||||
name: 'code_traversal_cache_v0_34',
|
||||
// v0.34 W3b — memoization layer for code_blast / code_flow.
|
||||
// (Originally claimed v56; renumbered to v59 on merge with master which
|
||||
// landed query_cache_search_lite=v55, drift_watch=v56, search_telemetry=v57.)
|
||||
//
|
||||
// Recursive caller/callee walks on a dense (calls + imports + references)
|
||||
// graph can fan out to 200+ nodes per call. During a plan-mode agent
|
||||
@@ -2743,9 +2745,11 @@ export const MIGRATIONS: Migration[] = [
|
||||
`,
|
||||
},
|
||||
{
|
||||
version: 55,
|
||||
version: 58,
|
||||
name: 'edges_backfilled_at_v0_33_2',
|
||||
// v0.33.2 W0c — resumable symbol-resolution backfill watermark.
|
||||
// (Originally claimed v55; renumbered to v58 on merge with master which
|
||||
// landed query_cache_search_lite=v55, drift_watch=v56, search_telemetry=v57.)
|
||||
//
|
||||
// The within-file two-pass resolver (src/core/chunkers/symbol-resolver.ts)
|
||||
// walks every content_chunks row that has unresolved edges
|
||||
@@ -2789,6 +2793,182 @@ export const MIGRATIONS: Migration[] = [
|
||||
WHERE edges_backfilled_at IS NULL;
|
||||
`,
|
||||
},
|
||||
{
|
||||
version: 55,
|
||||
name: 'query_cache_search_lite',
|
||||
// v0.32.x (search-lite, originally claimed v52 in PR #897; renumbered
|
||||
// to v55 on merge with master to sit after eval_contradictions_cache (v52),
|
||||
// eval_contradictions_runs (v53), cjk_wave (v54)).
|
||||
//
|
||||
// Semantic query cache. Cache search results keyed by query embedding
|
||||
// similarity so a near-duplicate query reuses the previous result set
|
||||
// instead of re-running keyword + vector + RRF + dedup. Cache lookup:
|
||||
// `embedding <=> $1 < 0.08` (cosine distance, similarity >= 0.92) using HNSW.
|
||||
//
|
||||
// Schema:
|
||||
// id — SHA-256(query_text + source_id) for diagnostics.
|
||||
// query_text — the raw query for debug + cache-stats output.
|
||||
// source_id — scope by source so multi-source brains don't bleed.
|
||||
// embedding — the query embedding. Same dim as content_chunks.
|
||||
// results — JSONB array of SearchResult rows.
|
||||
// meta — JSONB; what hybridSearch actually did (intent,
|
||||
// vector_enabled, etc.) so cached responses can
|
||||
// surface the same debug info as fresh ones.
|
||||
// ttl_seconds — per-row TTL. Default 3600. Stale rows are skipped
|
||||
// at read time and pruned by `gbrain cache prune`.
|
||||
// created_at — TTL anchor.
|
||||
// hit_count — instrumentation; bumped on each lookup-hit.
|
||||
// last_hit_at — instrumentation.
|
||||
//
|
||||
// Schema is engine-agnostic: HALFVEC when available (matches the facts
|
||||
// table from v45 for consistency), otherwise VECTOR. Embedding dim is
|
||||
// resolved from `config.embedding_dimensions` at migration time so
|
||||
// non-OpenAI brains work — same approach as v45.
|
||||
sql: '',
|
||||
handler: async (engine: BrainEngine) => {
|
||||
// Step 1: resolve embedding dim from config table (same pattern as v45).
|
||||
let embeddingDim = 1536;
|
||||
try {
|
||||
const dimRows = await engine.executeRaw<{ value: string }>(
|
||||
`SELECT value FROM config WHERE key = 'embedding_dimensions'`,
|
||||
);
|
||||
if (dimRows.length > 0) {
|
||||
const parsed = parseInt(dimRows[0].value, 10);
|
||||
if (Number.isFinite(parsed) && parsed > 0 && parsed <= 4096) {
|
||||
embeddingDim = parsed;
|
||||
}
|
||||
}
|
||||
} catch {
|
||||
// No config row yet — fall back to default.
|
||||
}
|
||||
|
||||
// Step 2: pgvector version probe for HALFVEC. Same logic as v45.
|
||||
// We deliberately mirror v45's facts table approach for consistency.
|
||||
let useHalfvec = false;
|
||||
if (engine.kind === 'postgres') {
|
||||
try {
|
||||
const vrows = await engine.executeRaw<{ extversion: string }>(
|
||||
`SELECT extversion FROM pg_extension WHERE extname = 'vector'`,
|
||||
);
|
||||
if (vrows.length > 0) {
|
||||
const v = vrows[0].extversion;
|
||||
const parts = v.split('.');
|
||||
const major = parseInt(parts[0] ?? '0', 10);
|
||||
const minor = parseInt(parts[1] ?? '0', 10);
|
||||
if (major > 0 || (major === 0 && minor >= 7)) {
|
||||
useHalfvec = true;
|
||||
}
|
||||
}
|
||||
} catch {
|
||||
// Probe failed — fall back to VECTOR.
|
||||
}
|
||||
} else {
|
||||
useHalfvec = true;
|
||||
}
|
||||
|
||||
const vecType = useHalfvec ? 'HALFVEC' : 'VECTOR';
|
||||
const opclass = useHalfvec ? 'halfvec_cosine_ops' : 'vector_cosine_ops';
|
||||
|
||||
const ddl = `
|
||||
CREATE TABLE IF NOT EXISTS query_cache (
|
||||
id TEXT PRIMARY KEY,
|
||||
query_text TEXT NOT NULL,
|
||||
source_id TEXT NOT NULL DEFAULT 'default',
|
||||
embedding ${vecType}(${embeddingDim}),
|
||||
results JSONB NOT NULL DEFAULT '[]'::jsonb,
|
||||
meta JSONB NOT NULL DEFAULT '{}'::jsonb,
|
||||
ttl_seconds INTEGER NOT NULL DEFAULT 3600,
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
|
||||
hit_count INTEGER NOT NULL DEFAULT 0,
|
||||
last_hit_at TIMESTAMPTZ
|
||||
);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS idx_query_cache_source_created
|
||||
ON query_cache(source_id, created_at DESC);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS idx_query_cache_embedding_hnsw
|
||||
ON query_cache USING hnsw (embedding ${opclass})
|
||||
WHERE embedding IS NOT NULL;
|
||||
`;
|
||||
|
||||
await engine.runMigration(55, ddl);
|
||||
},
|
||||
},
|
||||
{
|
||||
version: 56,
|
||||
name: 'query_cache_knobs_hash',
|
||||
// v0.32.3 search-lite mode cache contamination hotfix [CDX-4].
|
||||
//
|
||||
// PR #897's query_cache keyed rows on (id, source_id, query_text) only.
|
||||
// The `id` is sha256(source_id::query_text). A tokenmax search
|
||||
// (expansion=on, limit=50) populates a row that a subsequent
|
||||
// conservative call (no-expansion, limit=10) reads back, serving
|
||||
// expanded-and-oversized results to a budget-tight context.
|
||||
//
|
||||
// Fix: extend the row key with a knobs_hash derived from the resolved
|
||||
// search mode bundle. Lookup filters `WHERE knobs_hash = $1 AND
|
||||
// embedding similarity < threshold`. Existing rows have NULL
|
||||
// knobs_hash and are treated as misses (silently re-populated with
|
||||
// the correct hash on first hit — no orphan data, no destructive
|
||||
// migration).
|
||||
//
|
||||
// The PRIMARY KEY stays the existing `id` column (the SHA-256 of
|
||||
// (source_id, query_text, knobs_hash) — the cache code re-derives
|
||||
// it on every write, so a tokenmax write and a conservative write
|
||||
// produce distinct `id` values and live as separate rows).
|
||||
//
|
||||
// Engine-agnostic; idempotent.
|
||||
idempotent: true,
|
||||
sql: `
|
||||
ALTER TABLE query_cache ADD COLUMN IF NOT EXISTS knobs_hash TEXT;
|
||||
|
||||
CREATE INDEX IF NOT EXISTS idx_query_cache_source_knobs_created
|
||||
ON query_cache(source_id, knobs_hash, created_at DESC);
|
||||
`,
|
||||
},
|
||||
{
|
||||
version: 57,
|
||||
name: 'search_telemetry_rollup',
|
||||
// v0.32.3 search-lite: per-day rollup of search-call shape.
|
||||
//
|
||||
// Powers `gbrain search stats [--days N]` and `gbrain search tune` so an
|
||||
// operator (or an agent calling tune) can reason about hit rate, intent
|
||||
// mix, budget pressure, and result-volume averages WITHOUT pulling
|
||||
// per-call rows.
|
||||
//
|
||||
// Schema math per [CDX-17]: sums + counts only, NOT averages. Read-time
|
||||
// derives averages so concurrent ON CONFLICT writes from multiple gbrain
|
||||
// processes accumulate correctly.
|
||||
//
|
||||
// Date-bucketed cache hit/miss per [CDX-18] — query_cache.hit_count is
|
||||
// a LIFETIME counter and can't be sliced by --days. The telemetry table
|
||||
// is the truth for windowed hit rate.
|
||||
//
|
||||
// PK is (date, mode, intent) so the rollup never grows past
|
||||
// 365 days × 3 modes × 4 intents = ~4380 rows/year. Acceptable.
|
||||
//
|
||||
// Engine-agnostic; idempotent.
|
||||
idempotent: true,
|
||||
sql: `
|
||||
CREATE TABLE IF NOT EXISTS search_telemetry (
|
||||
date TEXT NOT NULL,
|
||||
mode TEXT NOT NULL,
|
||||
intent TEXT NOT NULL,
|
||||
count INTEGER NOT NULL DEFAULT 0,
|
||||
sum_results INTEGER NOT NULL DEFAULT 0,
|
||||
sum_tokens INTEGER NOT NULL DEFAULT 0,
|
||||
sum_budget_dropped INTEGER NOT NULL DEFAULT 0,
|
||||
cache_hit INTEGER NOT NULL DEFAULT 0,
|
||||
cache_miss INTEGER NOT NULL DEFAULT 0,
|
||||
first_seen TIMESTAMPTZ NOT NULL DEFAULT now(),
|
||||
last_seen TIMESTAMPTZ NOT NULL DEFAULT now(),
|
||||
PRIMARY KEY (date, mode, intent)
|
||||
);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS idx_search_telemetry_date
|
||||
ON search_telemetry (date DESC);
|
||||
`,
|
||||
},
|
||||
];
|
||||
|
||||
export const LATEST_VERSION = MIGRATIONS.length > 0
|
||||
|
||||
+12
-2
@@ -10,7 +10,7 @@ import { clampSearchLimit } from './engine.ts';
|
||||
import type { GBrainConfig } from './config.ts';
|
||||
import type { PageType } from './types.ts';
|
||||
import { importFromContent } from './import-file.ts';
|
||||
import { hybridSearch } from './search/hybrid.ts';
|
||||
import { hybridSearch, hybridSearchCached } from './search/hybrid.ts';
|
||||
import { expandQuery } from './search/expansion.ts';
|
||||
import { dedupResults } from './search/dedup.ts';
|
||||
import { captureEvalCandidate, isEvalCaptureEnabled, isEvalScrubEnabled } from './eval-capture.ts';
|
||||
@@ -1108,6 +1108,9 @@ const query: Operation = {
|
||||
// search). When the param is the literal '__all__', force-allow
|
||||
// cross-source mode (matches SearchOpts.sourceId contract).
|
||||
let capturedMeta: HybridSearchMeta | null = null;
|
||||
// v0.34 (Codex finding #2): thread ctx.sourceId so multi-source brains
|
||||
// get source-scoped retrieval. Explicit `source_id` param wins over
|
||||
// ctx.sourceId; literal `__all__` opts out (cross-source).
|
||||
const sourceIdParam = typeof p.source_id === 'string' ? p.source_id : undefined;
|
||||
const resolvedSourceId =
|
||||
sourceIdParam !== undefined
|
||||
@@ -1115,7 +1118,10 @@ const query: Operation = {
|
||||
? undefined
|
||||
: sourceIdParam
|
||||
: ctx.sourceId;
|
||||
const results = await hybridSearch(ctx.engine, queryText, {
|
||||
// v0.32.x search-lite: route through hybridSearchCached so semantic cache
|
||||
// + token budget + intent weighting fire automatically. Plain hybridSearch
|
||||
// remains the bare API for callers that opt out.
|
||||
const results = await hybridSearchCached(ctx.engine, queryText, {
|
||||
limit: (p.limit as number) || 20,
|
||||
offset: (p.offset as number) || 0,
|
||||
expansion: expand,
|
||||
@@ -1131,6 +1137,10 @@ const query: Operation = {
|
||||
recency: p.recency as 'off' | 'on' | 'strong' | undefined,
|
||||
since: typeof p.since === 'string' ? p.since : undefined,
|
||||
until: typeof p.until === 'string' ? p.until : undefined,
|
||||
// v0.32.x search-lite: token budget + cache opt-outs.
|
||||
tokenBudget: typeof p.token_budget === 'number' ? (p.token_budget as number) : undefined,
|
||||
useCache: typeof p.use_cache === 'boolean' ? (p.use_cache as boolean) : undefined,
|
||||
intentWeighting: typeof p.intent_weighting === 'boolean' ? (p.intent_weighting as boolean) : undefined,
|
||||
onMeta: (m) => { capturedMeta = m; },
|
||||
});
|
||||
const latency_ms = Date.now() - startedAt;
|
||||
|
||||
@@ -3151,6 +3151,24 @@ export class PGLiteEngine implements BrainEngine {
|
||||
);
|
||||
}
|
||||
|
||||
async unsetConfig(key: string): Promise<number> {
|
||||
const { affectedRows } = await this.db.query(
|
||||
'DELETE FROM config WHERE key = $1',
|
||||
[key],
|
||||
) as { affectedRows?: number };
|
||||
return affectedRows ?? 0;
|
||||
}
|
||||
|
||||
async listConfigKeys(prefix: string): Promise<string[]> {
|
||||
// LIKE-escape the prefix so a user-supplied % or _ doesn't act as a wildcard.
|
||||
const escaped = prefix.replace(/\\/g, '\\\\').replace(/%/g, '\\%').replace(/_/g, '\\_');
|
||||
const { rows } = await this.db.query(
|
||||
`SELECT key FROM config WHERE key LIKE $1 || '%' ESCAPE '\\' ORDER BY key`,
|
||||
[escaped],
|
||||
);
|
||||
return (rows as { key: string }[]).map(r => r.key);
|
||||
}
|
||||
|
||||
// Migration support
|
||||
async runMigration(_version: number, sql: string): Promise<void> {
|
||||
await this.db.exec(sql);
|
||||
|
||||
@@ -3148,6 +3148,23 @@ export class PostgresEngine implements BrainEngine {
|
||||
`;
|
||||
}
|
||||
|
||||
async unsetConfig(key: string): Promise<number> {
|
||||
const sql = this.sql;
|
||||
const result = await sql`DELETE FROM config WHERE key = ${key}` as unknown as { count: number };
|
||||
return result.count ?? 0;
|
||||
}
|
||||
|
||||
async listConfigKeys(prefix: string): Promise<string[]> {
|
||||
const sql = this.sql;
|
||||
// LIKE-escape literal % and _ so a config key with those chars resolves correctly.
|
||||
const escaped = prefix.replace(/\\/g, '\\\\').replace(/%/g, '\\%').replace(/_/g, '\\_');
|
||||
const pattern = `${escaped}%`;
|
||||
const rows = await sql<{ key: string }[]>`
|
||||
SELECT key FROM config WHERE key LIKE ${pattern} ESCAPE '\\' ORDER BY key
|
||||
`;
|
||||
return rows.map(r => r.key);
|
||||
}
|
||||
|
||||
// Migration support
|
||||
async runMigration(_version: number, sqlStr: string): Promise<void> {
|
||||
const conn = this.sql;
|
||||
|
||||
+370
-13
@@ -16,6 +16,17 @@ import { embed } from '../embedding.ts';
|
||||
import { dedupResults } from './dedup.ts';
|
||||
import { autoDetectDetail, classifyQuery } from './query-intent.ts';
|
||||
import { expandAnchors, hydrateChunks } from './two-pass.ts';
|
||||
import { enforceTokenBudget } from './token-budget.ts';
|
||||
import { recordSearchTelemetry } from './telemetry.ts';
|
||||
import {
|
||||
weightsForIntent,
|
||||
effectiveRrfK,
|
||||
applyExactMatchBoost,
|
||||
} from './intent-weights.ts';
|
||||
import {
|
||||
SemanticQueryCache,
|
||||
loadCacheConfig,
|
||||
} from './query-cache.ts';
|
||||
|
||||
const RRF_K = 60;
|
||||
const COMPILED_TRUTH_BOOST = 2.0;
|
||||
@@ -213,10 +224,43 @@ export async function hybridSearch(
|
||||
query: string,
|
||||
opts?: HybridSearchOpts,
|
||||
): Promise<SearchResult[]> {
|
||||
const limit = opts?.limit || 20;
|
||||
// v0.32.3 search-lite mode: resolve the active mode + per-key overrides
|
||||
// once at entry. Mode supplies DEFAULTS for intentWeighting, tokenBudget,
|
||||
// expansion, and searchLimit when the caller leaves those undefined.
|
||||
// Per-call opts and per-key config overrides still win.
|
||||
//
|
||||
// This MUST live in bare hybridSearch (NOT just in hybridSearchCached)
|
||||
// because eval-replay and eval-longmemeval call bare hybridSearch — and
|
||||
// per-mode evals would not test production search if modes lived only in
|
||||
// the wrapper. See `[CDX-5+6]` in the plan.
|
||||
const { loadSearchModeConfig, resolveSearchMode } = await import('./mode.ts');
|
||||
const modeInput = await loadSearchModeConfig(engine);
|
||||
const resolvedMode = resolveSearchMode({
|
||||
mode: modeInput.mode,
|
||||
overrides: modeInput.overrides,
|
||||
perCall: {
|
||||
intentWeighting: opts?.intentWeighting,
|
||||
tokenBudget: opts?.tokenBudget,
|
||||
expansion: opts?.expansion,
|
||||
searchLimit: opts?.limit,
|
||||
},
|
||||
});
|
||||
|
||||
const limit = opts?.limit || resolvedMode.searchLimit;
|
||||
const offset = opts?.offset || 0;
|
||||
const innerLimit = Math.min(limit * 2, MAX_SEARCH_LIMIT);
|
||||
|
||||
// v0.32.x search-lite: classify intent once up front. Drives BOTH the
|
||||
// legacy auto-detail / salience / recency suggestions AND the new
|
||||
// weight-adjustment path. Intent weighting is on by default and can
|
||||
// be disabled via `opts.intentWeighting = false`. The mode bundle
|
||||
// supplies the default when neither per-call nor per-key sets it.
|
||||
const suggestions = classifyQuery(query);
|
||||
const intentWeightingOn = resolvedMode.intentWeighting;
|
||||
const intentWeights = intentWeightingOn
|
||||
? weightsForIntent(suggestions.intent)
|
||||
: weightsForIntent('general');
|
||||
|
||||
// Auto-detect detail level from query intent when caller doesn't specify
|
||||
const detail = opts?.detail ?? autoDetectDetail(query);
|
||||
const detailResolved: 'low' | 'medium' | 'high' | null = detail ?? null;
|
||||
@@ -246,12 +290,23 @@ export async function hybridSearch(
|
||||
// A throwing user callback must never break the search hot path — onMeta
|
||||
// is a public surface (gbrain/search/hybrid) so a third-party closure bug
|
||||
// shouldn't take down query/search responses.
|
||||
//
|
||||
// v0.32.3 search-lite: every emitMeta call ALSO records to the in-process
|
||||
// search_telemetry rollup. Telemetry write is sync (bumps a bucket map),
|
||||
// flush is fire-and-forget on 60s / 100-call thresholds. The hot path
|
||||
// never waits.
|
||||
let lastResultsCount = 0;
|
||||
const emitMeta = (meta: HybridSearchMeta): void => {
|
||||
try {
|
||||
opts?.onMeta?.(meta);
|
||||
} catch {
|
||||
// swallow — capture telemetry is best-effort
|
||||
}
|
||||
try {
|
||||
recordSearchTelemetry(engine, meta, { results_count: lastResultsCount });
|
||||
} catch {
|
||||
// swallow — telemetry must never break the search hot path.
|
||||
}
|
||||
};
|
||||
|
||||
if (DEBUG && detail) {
|
||||
@@ -264,7 +319,11 @@ export async function hybridSearch(
|
||||
// v0.29.1: resolve salience/recency from caller (back-compat aliases for
|
||||
// PR #618's `recencyBoost` numeric scale) or fall back to the heuristic.
|
||||
// The wrapper fires from ALL THREE return paths (codex pass-1 #2 + pass-2 #4).
|
||||
const suggestions = classifyQuery(query);
|
||||
//
|
||||
// v0.32.x search-lite: when caller hasn't set recency and the intent
|
||||
// classifier suggests one, prefer that (suggestedRecency on temporal /
|
||||
// event intents). The legacy heuristic still wins when intent weighting
|
||||
// is off.
|
||||
// Back-compat: recencyBoost: 1|2 → 'on'|'strong'; 0 → 'off'.
|
||||
const legacyRecency: 'off' | 'on' | 'strong' | undefined =
|
||||
opts?.recencyBoost === 2 ? 'strong' :
|
||||
@@ -272,7 +331,20 @@ export async function hybridSearch(
|
||||
opts?.recencyBoost === 0 ? 'off' :
|
||||
undefined;
|
||||
const salienceMode: 'off' | 'on' | 'strong' = opts?.salience ?? suggestions.suggestedSalience;
|
||||
const recencyMode: 'off' | 'on' | 'strong' = opts?.recency ?? legacyRecency ?? suggestions.suggestedRecency;
|
||||
// Intent-weighting recency suggestion is a NUDGE — it only fires when
|
||||
// the caller left recency unspecified AND the legacy heuristic also
|
||||
// didn't fire. The classifier's own suggestedRecency (from v0.29.1)
|
||||
// still wins when it's set; the new intent suggestion is a fallback.
|
||||
const intentRecency =
|
||||
intentWeightingOn && intentWeights.suggestedRecency != null
|
||||
? intentWeights.suggestedRecency
|
||||
: null;
|
||||
const recencyMode: 'off' | 'on' | 'strong' =
|
||||
opts?.recency
|
||||
?? legacyRecency
|
||||
?? (suggestions.suggestedRecency !== 'off'
|
||||
? suggestions.suggestedRecency
|
||||
: (intentRecency ?? suggestions.suggestedRecency));
|
||||
const postFusionOpts = {
|
||||
applyBacklinks: true,
|
||||
salience: salienceMode,
|
||||
@@ -286,15 +358,31 @@ export async function hybridSearch(
|
||||
await runPostFusionStages(engine, keywordResults, postFusionOpts);
|
||||
keywordResults.sort((a, b) => b.score - a.score);
|
||||
}
|
||||
emitMeta({ vector_enabled: false, detail_resolved: detailResolved, expansion_applied: false });
|
||||
return dedupResults(keywordResults).slice(offset, offset + limit);
|
||||
const noEmbedSliced = dedupResults(keywordResults).slice(offset, offset + limit);
|
||||
// v0.32.3 search-lite: budget enforcement on the no-embedding-provider path.
|
||||
const { results: noEmbedBudgeted, meta: noEmbedBudgetMeta } = enforceTokenBudget(noEmbedSliced, resolvedMode.tokenBudget);
|
||||
lastResultsCount = noEmbedBudgeted.length;
|
||||
emitMeta({
|
||||
vector_enabled: false,
|
||||
detail_resolved: detailResolved,
|
||||
expansion_applied: false,
|
||||
intent: suggestions.intent,
|
||||
mode: resolvedMode.resolved_mode,
|
||||
...(resolvedMode.tokenBudget && resolvedMode.tokenBudget > 0
|
||||
? { token_budget: noEmbedBudgetMeta }
|
||||
: {}),
|
||||
});
|
||||
return noEmbedBudgeted;
|
||||
}
|
||||
|
||||
// Determine query variants (optionally with expansion)
|
||||
// expandQuery already includes the original query in its return value,
|
||||
// so we use it directly instead of prepending query again
|
||||
// so we use it directly instead of prepending query again.
|
||||
// v0.32.3 search-lite: expansion fires when (a) resolved mode says yes and
|
||||
// (b) an expandFn is wired in. The mode bundle is the default; per-call
|
||||
// SearchOpts.expansion still wins via resolveSearchMode's chain.
|
||||
let queries = [query];
|
||||
if (opts?.expansion && opts?.expandFn) {
|
||||
if (resolvedMode.expansion && opts?.expandFn) {
|
||||
try {
|
||||
queries = await opts.expandFn(query);
|
||||
if (queries.length === 0) queries = [query];
|
||||
@@ -327,14 +415,38 @@ export async function hybridSearch(
|
||||
await runPostFusionStages(engine, keywordResults, postFusionOpts);
|
||||
keywordResults.sort((a, b) => b.score - a.score);
|
||||
}
|
||||
emitMeta({ vector_enabled: false, detail_resolved: detailResolved, expansion_applied: expansionApplied });
|
||||
return dedupResults(keywordResults).slice(offset, offset + limit);
|
||||
const kwSliced = dedupResults(keywordResults).slice(offset, offset + limit);
|
||||
// v0.32.3 search-lite: budget enforcement on the keyword-fallback path too.
|
||||
const { results: kwBudgeted, meta: kwBudgetMeta } = enforceTokenBudget(kwSliced, resolvedMode.tokenBudget);
|
||||
lastResultsCount = kwBudgeted.length;
|
||||
emitMeta({
|
||||
vector_enabled: false,
|
||||
detail_resolved: detailResolved,
|
||||
expansion_applied: expansionApplied,
|
||||
intent: suggestions.intent,
|
||||
mode: resolvedMode.resolved_mode,
|
||||
...(resolvedMode.tokenBudget && resolvedMode.tokenBudget > 0
|
||||
? { token_budget: kwBudgetMeta }
|
||||
: {}),
|
||||
});
|
||||
return kwBudgeted;
|
||||
}
|
||||
|
||||
// Merge all result lists via RRF (includes normalization + boost)
|
||||
// Skip boost for detail=high (temporal/event queries want natural ranking)
|
||||
const allLists = [...vectorLists, keywordResults];
|
||||
let fused = rrfFusion(allLists, opts?.rrfK ?? RRF_K, detail !== 'high');
|
||||
//
|
||||
// v0.32.x search-lite: when intent weighting is on, run RRF with
|
||||
// per-list effective k values — entity/event intents nudge keyword
|
||||
// contributions up by lowering their k. The base rrfK still controls
|
||||
// the overall RRF shape; intent weights tilt within that shape.
|
||||
const baseRrfK = opts?.rrfK ?? RRF_K;
|
||||
const keywordK = effectiveRrfK(baseRrfK, intentWeights.keywordWeight);
|
||||
const vectorK = effectiveRrfK(baseRrfK, intentWeights.vectorWeight);
|
||||
const allLists: Array<{ list: SearchResult[]; k: number }> = [
|
||||
...vectorLists.map(list => ({ list, k: vectorK })),
|
||||
{ list: keywordResults, k: keywordK },
|
||||
];
|
||||
let fused = rrfFusionWeighted(allLists, detail !== 'high');
|
||||
|
||||
// Cosine re-scoring before dedup so semantically better chunks survive
|
||||
if (queryEmbedding) {
|
||||
@@ -347,6 +459,11 @@ export async function hybridSearch(
|
||||
// both, or neither fires depending on resolved modes.
|
||||
if (fused.length > 0) {
|
||||
await runPostFusionStages(engine, fused, postFusionOpts);
|
||||
// v0.32.x search-lite: intent exact-match boost (entity/event intents).
|
||||
// No-op when boost factor is 1.0 (general intent or weighting disabled).
|
||||
if (intentWeights.exactMatchBoost !== 1.0) {
|
||||
applyExactMatchBoost(fused, query, intentWeights);
|
||||
}
|
||||
fused.sort((a, b) => b.score - a.score);
|
||||
}
|
||||
|
||||
@@ -409,8 +526,248 @@ export async function hybridSearch(
|
||||
return hybridSearch(engine, query, { ...opts, detail: 'high' });
|
||||
}
|
||||
|
||||
emitMeta({ vector_enabled: true, detail_resolved: detailResolved, expansion_applied: expansionApplied });
|
||||
return deduped.slice(offset, offset + limit);
|
||||
const sliced = deduped.slice(offset, offset + limit);
|
||||
// v0.32.3 search-lite: budget enforcement at the main return path.
|
||||
// hybridSearchCached used to be the only place this fired; now bare
|
||||
// hybridSearch enforces it too so eval-replay + eval-longmemeval see
|
||||
// the same budget behavior as the production query op.
|
||||
const { results: budgeted, meta: budgetMeta } = enforceTokenBudget(sliced, resolvedMode.tokenBudget);
|
||||
lastResultsCount = budgeted.length;
|
||||
emitMeta({
|
||||
vector_enabled: true,
|
||||
detail_resolved: detailResolved,
|
||||
expansion_applied: expansionApplied,
|
||||
intent: suggestions.intent,
|
||||
mode: resolvedMode.resolved_mode,
|
||||
...(resolvedMode.tokenBudget && resolvedMode.tokenBudget > 0
|
||||
? { token_budget: budgetMeta }
|
||||
: {}),
|
||||
});
|
||||
return budgeted;
|
||||
}
|
||||
|
||||
// ----------------------------------------------------------------------
|
||||
// v0.32.x (search-lite) — cached + budgeted public wrapper
|
||||
// ----------------------------------------------------------------------
|
||||
|
||||
/**
|
||||
* Public wrapper around hybridSearch that adds the v0.32.x search-lite
|
||||
* features: semantic query cache + token budget enforcement. Both are
|
||||
* additive and backward-compatible; callers that don't opt in see the
|
||||
* same behavior as plain hybridSearch.
|
||||
*
|
||||
* Pipeline:
|
||||
* 1. Cache lookup (if enabled + we can produce a query embedding).
|
||||
* 2. On miss: run hybridSearch normally.
|
||||
* 3. Apply token budget (no-op when budget is undefined).
|
||||
* 4. On miss + successful search: write back to cache (best-effort).
|
||||
*
|
||||
* The cache uses the same embedding the search pipeline would compute,
|
||||
* so an extra embed() call only happens when hybridSearch would have
|
||||
* skipped vector search entirely (no embedding provider configured). In
|
||||
* that case the cache is also skipped — there's no embedding to key on.
|
||||
*/
|
||||
export async function hybridSearchCached(
|
||||
engine: BrainEngine,
|
||||
query: string,
|
||||
opts?: HybridSearchOpts,
|
||||
): Promise<SearchResult[]> {
|
||||
// v0.32.3 search-lite mode: resolve mode + per-key overrides once. The
|
||||
// resolved knob set drives cache enable/threshold/TTL AND the knobs_hash
|
||||
// that scopes the cache row so a tokenmax write can't be served to a
|
||||
// conservative read. See [CDX-4] in the plan.
|
||||
const { loadSearchModeConfig, resolveSearchMode, knobsHash } = await import('./mode.ts');
|
||||
const modeInputForCache = await loadSearchModeConfig(engine);
|
||||
const resolvedForCache = resolveSearchMode({
|
||||
mode: modeInputForCache.mode,
|
||||
overrides: modeInputForCache.overrides,
|
||||
perCall: {
|
||||
cache_enabled: opts?.useCache,
|
||||
tokenBudget: opts?.tokenBudget,
|
||||
expansion: opts?.expansion,
|
||||
intentWeighting: opts?.intentWeighting,
|
||||
searchLimit: opts?.limit,
|
||||
},
|
||||
});
|
||||
const cacheKnobsHash = knobsHash(resolvedForCache);
|
||||
|
||||
// Cache decision: opts.useCache (explicit) wins over global config; global
|
||||
// config wins over mode bundle default. Mode bundle is on for all 3 modes
|
||||
// today; the resolver already folded everything through.
|
||||
const cacheCfg = await loadCacheConfig(engine);
|
||||
const cacheEnabled = resolvedForCache.cache_enabled;
|
||||
const cache = new SemanticQueryCache(engine, {
|
||||
...cacheCfg,
|
||||
enabled: cacheEnabled,
|
||||
similarityThreshold: resolvedForCache.cache_similarity_threshold,
|
||||
ttlSeconds: resolvedForCache.cache_ttl_seconds,
|
||||
});
|
||||
|
||||
// Skip cache entirely when the request asks for two-pass walks or has
|
||||
// a non-default embedding column — those interact with structural state
|
||||
// that the cache can't safely express.
|
||||
const skipCache =
|
||||
!cache.isEnabled() ||
|
||||
(opts?.walkDepth ?? 0) > 0 ||
|
||||
Boolean(opts?.nearSymbol) ||
|
||||
(opts?.embeddingColumn && opts.embeddingColumn !== 'embedding');
|
||||
|
||||
let cacheStatus: 'hit' | 'miss' | 'disabled' = skipCache ? 'disabled' : 'miss';
|
||||
let cacheSimilarity: number | undefined;
|
||||
let cacheAge: number | undefined;
|
||||
|
||||
// We need a query embedding to consult the cache. We try to embed once
|
||||
// here so the same embedding can be threaded back into the search call
|
||||
// if it misses — but the embedding helper isn't cheap, so we only
|
||||
// attempt it when the cache is enabled AND the gateway has an embedding
|
||||
// provider configured.
|
||||
let queryEmbedding: Float32Array | null = null;
|
||||
if (!skipCache) {
|
||||
try {
|
||||
const { isAvailable } = await import('../ai/gateway.ts');
|
||||
if (isAvailable('embedding')) {
|
||||
queryEmbedding = await embed(query);
|
||||
} else {
|
||||
cacheStatus = 'disabled';
|
||||
}
|
||||
} catch {
|
||||
cacheStatus = 'disabled';
|
||||
queryEmbedding = null;
|
||||
}
|
||||
}
|
||||
|
||||
if (!skipCache && queryEmbedding && cacheStatus !== 'disabled') {
|
||||
const hit = await cache.lookup(queryEmbedding, { sourceId: opts?.sourceId, knobsHash: cacheKnobsHash });
|
||||
if (hit.hit && hit.results) {
|
||||
cacheStatus = 'hit';
|
||||
cacheSimilarity = hit.similarity;
|
||||
cacheAge = hit.ageSeconds;
|
||||
|
||||
const limit = opts?.limit || 20;
|
||||
const offset = opts?.offset || 0;
|
||||
const sliced = hit.results.slice(offset, offset + limit);
|
||||
|
||||
// Budget enforcement — same pipeline tail as fresh path.
|
||||
const { results: budgeted, meta: budgetMeta } = enforceTokenBudget(sliced, opts?.tokenBudget);
|
||||
|
||||
// Emit meta describing the cache path.
|
||||
const cachedMeta: HybridSearchMeta = {
|
||||
vector_enabled: hit.meta?.vector_enabled ?? true,
|
||||
detail_resolved: hit.meta?.detail_resolved ?? null,
|
||||
expansion_applied: hit.meta?.expansion_applied ?? false,
|
||||
intent: hit.meta?.intent,
|
||||
cache: {
|
||||
status: 'hit',
|
||||
similarity: cacheSimilarity,
|
||||
age_seconds: cacheAge,
|
||||
},
|
||||
...(opts?.tokenBudget && opts.tokenBudget > 0
|
||||
? { token_budget: budgetMeta }
|
||||
: {}),
|
||||
};
|
||||
try {
|
||||
opts?.onMeta?.(cachedMeta);
|
||||
} catch {
|
||||
// swallow — telemetry is best-effort
|
||||
}
|
||||
return budgeted;
|
||||
}
|
||||
}
|
||||
|
||||
// Cache miss (or disabled): run the normal search. We capture meta so
|
||||
// we can write back to the cache + emit the merged meta to the caller.
|
||||
// The closure-write pattern trips TS's narrowing (it infers `never`), so
|
||||
// we use a single-element box to keep the type stable.
|
||||
const innerMetaBox: { current: HybridSearchMeta | null } = { current: null };
|
||||
const userOnMeta = opts?.onMeta;
|
||||
const results = await hybridSearch(engine, query, {
|
||||
...opts,
|
||||
onMeta: (m) => {
|
||||
innerMetaBox.current = m;
|
||||
// Do NOT call userOnMeta here — we'll emit a merged meta below
|
||||
// that also carries cache + budget info.
|
||||
},
|
||||
});
|
||||
const innerMeta = innerMetaBox.current;
|
||||
|
||||
// Token budget pass (no-op when not set).
|
||||
const { results: budgeted, meta: budgetMeta } = enforceTokenBudget(results, opts?.tokenBudget);
|
||||
|
||||
// Compose the final meta and emit.
|
||||
const finalMeta: HybridSearchMeta = {
|
||||
vector_enabled: innerMeta?.vector_enabled ?? false,
|
||||
detail_resolved: innerMeta?.detail_resolved ?? null,
|
||||
expansion_applied: innerMeta?.expansion_applied ?? false,
|
||||
intent: innerMeta?.intent,
|
||||
cache: { status: cacheStatus },
|
||||
...(opts?.tokenBudget && opts.tokenBudget > 0
|
||||
? { token_budget: budgetMeta }
|
||||
: {}),
|
||||
};
|
||||
try {
|
||||
userOnMeta?.(finalMeta);
|
||||
} catch {
|
||||
// swallow
|
||||
}
|
||||
|
||||
// Best-effort writeback (skip when search returned empty so we don't
|
||||
// cache zero-result queries forever — they often indicate a typo).
|
||||
if (
|
||||
cacheStatus === 'miss' &&
|
||||
queryEmbedding &&
|
||||
results.length > 0 &&
|
||||
(innerMeta?.vector_enabled ?? false)
|
||||
) {
|
||||
void cache
|
||||
.store(query, queryEmbedding, results, finalMeta, { sourceId: opts?.sourceId, knobsHash: cacheKnobsHash })
|
||||
.catch(() => { /* swallow */ });
|
||||
}
|
||||
|
||||
return budgeted;
|
||||
}
|
||||
|
||||
/**
|
||||
* v0.32.x search-lite — weighted RRF. Each list contributes with its own
|
||||
* effective k value, which lets intent weighting bias keyword vs vector
|
||||
* lists without re-weighting individual scores. Wraps rrfFusion internally
|
||||
* by computing weighted contributions in a single pass.
|
||||
*/
|
||||
export function rrfFusionWeighted(
|
||||
lists: Array<{ list: SearchResult[]; k: number }>,
|
||||
applyBoost = true,
|
||||
): SearchResult[] {
|
||||
const scores = new Map<string, { result: SearchResult; score: number }>();
|
||||
|
||||
for (const { list, k } of lists) {
|
||||
for (let rank = 0; rank < list.length; rank++) {
|
||||
const r = list[rank];
|
||||
const key = `${r.slug}:${r.chunk_id ?? r.chunk_text.slice(0, 50)}`;
|
||||
const existing = scores.get(key);
|
||||
const rrfScore = 1 / (k + rank);
|
||||
|
||||
if (existing) {
|
||||
existing.score += rrfScore;
|
||||
} else {
|
||||
scores.set(key, { result: r, score: rrfScore });
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
const entries = Array.from(scores.values());
|
||||
if (entries.length === 0) return [];
|
||||
|
||||
const maxScore = Math.max(...entries.map(e => e.score));
|
||||
if (maxScore > 0) {
|
||||
for (const e of entries) {
|
||||
e.score = e.score / maxScore;
|
||||
const boost = applyBoost && e.result.chunk_source === 'compiled_truth' ? COMPILED_TRUTH_BOOST : 1.0;
|
||||
e.score *= boost;
|
||||
}
|
||||
}
|
||||
|
||||
return entries
|
||||
.sort((a, b) => b.score - a.score)
|
||||
.map(({ result, score }) => ({ ...result, score }));
|
||||
}
|
||||
|
||||
/**
|
||||
|
||||
@@ -0,0 +1,141 @@
|
||||
/**
|
||||
* Zero-LLM Intent → Weight Adjustment (v0.32.x — search-lite)
|
||||
*
|
||||
* Sits on top of the existing query-intent classifier. The classifier
|
||||
* (src/core/search/query-intent.ts) already produces an `intent` field
|
||||
* with 4 values: entity / temporal / event / general. This module maps
|
||||
* that intent onto concrete weight adjustments applied during the hybrid
|
||||
* search pipeline:
|
||||
*
|
||||
* - entity → boost exact slug/title matches (keyword pre-filter favored)
|
||||
* - temporal → increase recency scoring weight (recency = 'on' when caller
|
||||
* left it undefined)
|
||||
* - event → increase keyword weight in hybrid fusion (event queries
|
||||
* tend to have rare named entities that keyword search nails
|
||||
* while vector search smears across paraphrases)
|
||||
* - general → default semantic (no adjustment)
|
||||
*
|
||||
* All adjustments are SUBTLE — they nudge weights, they don't override
|
||||
* caller-explicit options. If the caller passed `recency: 'off'`, intent
|
||||
* weighting will NOT silently re-enable it. The classifier is a default,
|
||||
* not a mandate.
|
||||
*
|
||||
* The original v0.20.0 LLM query expansion path (expandQuery in
|
||||
* expansion.ts) still exists and is opt-in via `opts.expansion = true`.
|
||||
* Intent weighting is the new DEFAULT, and replaces the expansion call
|
||||
* for the common case (simple queries, no API key, fast loop).
|
||||
*
|
||||
* Pure module. No DB, no LLM, no async. Tested in
|
||||
* test/intent-weights.test.ts.
|
||||
*/
|
||||
|
||||
import type { QueryIntent } from './query-intent.ts';
|
||||
import type { SearchResult } from '../types.ts';
|
||||
|
||||
/**
|
||||
* Weight adjustments to apply for a classified intent. All factors are
|
||||
* multiplicative on the existing pipeline weights; the defaults map to
|
||||
* 1.0 (no-op) for the `general` intent. A factor > 1.0 increases the
|
||||
* weight of that signal; < 1.0 decreases it.
|
||||
*
|
||||
* Magnitudes were tuned conservatively (max 1.25x boost) so the existing
|
||||
* search behavior on ambiguous queries stays close to v0.31.x. The point
|
||||
* of the classifier isn't to flip rankings — it's to break ties in
|
||||
* favor of the user's plausible intent.
|
||||
*/
|
||||
export interface IntentWeights {
|
||||
/** Multiplier on the keyword-list rank in RRF fusion. Higher = keyword wins more ties. */
|
||||
keywordWeight: number;
|
||||
/** Multiplier on the vector-list rank in RRF fusion. Higher = semantic wins more ties. */
|
||||
vectorWeight: number;
|
||||
/** Recency tilt to suggest when caller hasn't specified one. */
|
||||
suggestedRecency: 'off' | 'on' | 'strong' | null;
|
||||
/** Score multiplier for results whose slug/title exactly matches the (lowercased) query. */
|
||||
exactMatchBoost: number;
|
||||
}
|
||||
|
||||
const DEFAULT_WEIGHTS: IntentWeights = {
|
||||
keywordWeight: 1.0,
|
||||
vectorWeight: 1.0,
|
||||
suggestedRecency: null,
|
||||
exactMatchBoost: 1.0,
|
||||
};
|
||||
|
||||
const INTENT_WEIGHTS: Record<QueryIntent, IntentWeights> = {
|
||||
entity: {
|
||||
// Entity queries: "who is X", "tell me about Y". The user knows the
|
||||
// name. Reward exact slug/title matches; lean into keyword.
|
||||
keywordWeight: 1.15,
|
||||
vectorWeight: 1.0,
|
||||
suggestedRecency: null,
|
||||
exactMatchBoost: 1.25,
|
||||
},
|
||||
temporal: {
|
||||
// Temporal queries: "what happened last week", "meeting prep". Recency
|
||||
// tilt is the whole game; keyword and vector stay balanced.
|
||||
keywordWeight: 1.0,
|
||||
vectorWeight: 1.0,
|
||||
suggestedRecency: 'on',
|
||||
exactMatchBoost: 1.0,
|
||||
},
|
||||
event: {
|
||||
// Event queries: "announcement", "launched", "raised $". Named events
|
||||
// have rare entity surface forms that keyword search nails (think
|
||||
// company names, dollar amounts). Recency gets a soft tilt too.
|
||||
keywordWeight: 1.20,
|
||||
vectorWeight: 0.95,
|
||||
suggestedRecency: 'on',
|
||||
exactMatchBoost: 1.10,
|
||||
},
|
||||
general: DEFAULT_WEIGHTS,
|
||||
};
|
||||
|
||||
/** Lookup the weights for a classified intent. */
|
||||
export function weightsForIntent(intent: QueryIntent): IntentWeights {
|
||||
return INTENT_WEIGHTS[intent] ?? DEFAULT_WEIGHTS;
|
||||
}
|
||||
|
||||
/**
|
||||
* Apply the per-list rank weighting before RRF. Caller passes the list
|
||||
* source ('keyword' | 'vector') and the weights; we return the effective
|
||||
* RRF k constant to use for THAT list. Lower k = stronger boost on top
|
||||
* ranks; higher k = flatter contribution. So a higher weight maps to a
|
||||
* LOWER k.
|
||||
*
|
||||
* Default RRF_K is 60. With keywordWeight=1.20, the effective k for the
|
||||
* keyword list becomes 60 / 1.20 = 50, which gives top-keyword results
|
||||
* a meaningfully stronger contribution to the fused score.
|
||||
*/
|
||||
export function effectiveRrfK(baseK: number, weight: number): number {
|
||||
if (weight <= 0) return baseK;
|
||||
return baseK / weight;
|
||||
}
|
||||
|
||||
/**
|
||||
* Apply exact-match boost in place. Mutates each result's score by
|
||||
* `weights.exactMatchBoost` when the result's slug or title (lowercased,
|
||||
* trimmed) matches the lowercased query exactly. No-op when the boost
|
||||
* is 1.0. Caller re-sorts after.
|
||||
*
|
||||
* Normalization: slug is matched as-is (slugs are already canonicalized
|
||||
* lowercase-kebab); title is lowercased + trimmed. The query is
|
||||
* lowercased + trimmed once before the loop.
|
||||
*/
|
||||
export function applyExactMatchBoost(
|
||||
results: SearchResult[],
|
||||
query: string,
|
||||
weights: IntentWeights,
|
||||
): void {
|
||||
if (weights.exactMatchBoost === 1.0) return;
|
||||
const q = query.toLowerCase().trim();
|
||||
if (!q) return;
|
||||
// Pre-compute the kebab form for slug-style matches like "garry tan" → "garry-tan".
|
||||
const qKebab = q.replace(/\s+/g, '-');
|
||||
for (const r of results) {
|
||||
const slug = (r.slug ?? '').toLowerCase();
|
||||
const title = (r.title ?? '').toLowerCase().trim();
|
||||
if (slug === q || slug === qKebab || slug.endsWith(`/${qKebab}`) || title === q) {
|
||||
r.score *= weights.exactMatchBoost;
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,371 @@
|
||||
/**
|
||||
* v0.32.3 search-lite mode bundles.
|
||||
*
|
||||
* Three named modes that bundle the search-lite knobs from PR #897 into a
|
||||
* single config key so users pick once at install time and stop thinking
|
||||
* about it. Each mode resolves to a complete knob set; per-call SearchOpts
|
||||
* and per-key config overrides still win — mode just supplies the default.
|
||||
*
|
||||
* The resolution chain matches the v0.31.12 model-tier pattern at
|
||||
* `src/core/model-config.ts:resolveModel`:
|
||||
* per-call opts → per-key config → MODE_BUNDLES[cfg.search.mode] → MODE_BUNDLES.balanced
|
||||
*
|
||||
* `resolveSearchMode` is called at the top of bare `hybridSearch`, NOT just
|
||||
* inside the `hybridSearchCached` wrapper. Eval commands (`eval replay`,
|
||||
* `eval longmemeval`) call bare hybridSearch and must test the same
|
||||
* mode-affected behavior as production. See `[CDX-5+6]` in the plan.
|
||||
*
|
||||
* `knobsHash` produces the SHA-256 the query cache uses to prevent
|
||||
* cross-mode contamination. The PR #897 cache keyed only on
|
||||
* (source_id, query_text) — a tokenmax run with expansion+limit=50 would
|
||||
* populate a row that a subsequent conservative call reads back. Migration
|
||||
* v56 adds `knobs_hash` column; lookup filters by knobs_hash equality AND
|
||||
* embedding similarity. See `[CDX-4]` in the plan.
|
||||
*/
|
||||
|
||||
import { createHash } from 'crypto';
|
||||
|
||||
export type SearchMode = 'conservative' | 'balanced' | 'tokenmax';
|
||||
|
||||
export const SEARCH_MODES: ReadonlyArray<SearchMode> = Object.freeze([
|
||||
'conservative',
|
||||
'balanced',
|
||||
'tokenmax',
|
||||
]);
|
||||
|
||||
/**
|
||||
* A complete knob set for one mode. Every field is required so the bundle
|
||||
* is self-contained and per-key overrides are obvious diffs.
|
||||
*/
|
||||
export interface ModeBundle {
|
||||
/** Semantic query cache (PR #897). Free win; on for everyone. */
|
||||
cache_enabled: boolean;
|
||||
cache_similarity_threshold: number;
|
||||
cache_ttl_seconds: number;
|
||||
/** Zero-LLM intent classifier weight adjustments (PR #897). On for everyone. */
|
||||
intentWeighting: boolean;
|
||||
/**
|
||||
* Per-call token budget cap (PR #897). undefined = no-op (tokenmax).
|
||||
* 4000 = tight (conservative, fits Haiku context loop).
|
||||
* 12000 = balanced (sweet-spot for Sonnet).
|
||||
*/
|
||||
tokenBudget: number | undefined;
|
||||
/**
|
||||
* LLM multi-query expansion (Haiku call per search).
|
||||
* Per CLAUDE.md TODOS the corpus eval shows ~97.6% lift relative to no
|
||||
* expansion — barely measurable. Off for conservative/balanced;
|
||||
* on for tokenmax to preserve power-user retrieval ceiling.
|
||||
*/
|
||||
expansion: boolean;
|
||||
/**
|
||||
* Default `limit` for the operation layer (`src/core/operations.ts:1087`).
|
||||
* Note: production `query` op TODAY defaults to 20. Mode bundle becomes
|
||||
* the default ONLY when the caller omits the field — same chain semantics
|
||||
* as model-tier resolution. See `[CDX-1+2+3]` in the plan: the original
|
||||
* "tokenmax preserves Garry's setup" framing is wrong; tokenmax is an
|
||||
* EXPANSION from the implicit current default (limit 20).
|
||||
*/
|
||||
searchLimit: number;
|
||||
}
|
||||
|
||||
/**
|
||||
* The three mode bundles. Frozen at import time so a typo can't redefine
|
||||
* "conservative" to mean different things on different installs — the
|
||||
* public eval table depends on these being canonical. Power-user
|
||||
* customization happens via per-key config overrides; if there's real
|
||||
* demand for a custom bundle, that's a v0.34 conversation.
|
||||
*/
|
||||
export const MODE_BUNDLES: Readonly<Record<SearchMode, Readonly<ModeBundle>>> = Object.freeze({
|
||||
conservative: Object.freeze({
|
||||
cache_enabled: true,
|
||||
cache_similarity_threshold: 0.92,
|
||||
cache_ttl_seconds: 3600,
|
||||
intentWeighting: true,
|
||||
tokenBudget: 4000,
|
||||
expansion: false,
|
||||
searchLimit: 10,
|
||||
}),
|
||||
balanced: Object.freeze({
|
||||
cache_enabled: true,
|
||||
cache_similarity_threshold: 0.92,
|
||||
cache_ttl_seconds: 3600,
|
||||
intentWeighting: true,
|
||||
tokenBudget: 12000,
|
||||
expansion: false,
|
||||
searchLimit: 25,
|
||||
}),
|
||||
tokenmax: Object.freeze({
|
||||
cache_enabled: true,
|
||||
cache_similarity_threshold: 0.92,
|
||||
cache_ttl_seconds: 3600,
|
||||
intentWeighting: true,
|
||||
tokenBudget: undefined,
|
||||
expansion: true,
|
||||
searchLimit: 50,
|
||||
}),
|
||||
});
|
||||
|
||||
export const DEFAULT_SEARCH_MODE: SearchMode = 'balanced';
|
||||
|
||||
export function isSearchMode(x: unknown): x is SearchMode {
|
||||
return typeof x === 'string' && (SEARCH_MODES as ReadonlyArray<string>).includes(x);
|
||||
}
|
||||
|
||||
/**
|
||||
* Per-key config overrides. Read at search-time from the `config` table.
|
||||
* Every field is optional; an undefined field means "fall through to the
|
||||
* mode bundle default."
|
||||
*/
|
||||
export interface SearchKeyOverrides {
|
||||
cache_enabled?: boolean;
|
||||
cache_similarity_threshold?: number;
|
||||
cache_ttl_seconds?: number;
|
||||
intentWeighting?: boolean;
|
||||
tokenBudget?: number;
|
||||
expansion?: boolean;
|
||||
searchLimit?: number;
|
||||
}
|
||||
|
||||
/**
|
||||
* Per-call opts that can override the bundle for this single search.
|
||||
* Same shape as ModeBundle but every field is optional. These are passed
|
||||
* through from `SearchOpts` / `HybridSearchOpts` so the existing per-call
|
||||
* surface continues to work — mode just provides the default that the
|
||||
* caller's explicit field overrides.
|
||||
*/
|
||||
export interface SearchPerCallOpts {
|
||||
cache_enabled?: boolean;
|
||||
cache_similarity_threshold?: number;
|
||||
cache_ttl_seconds?: number;
|
||||
intentWeighting?: boolean;
|
||||
tokenBudget?: number;
|
||||
expansion?: boolean;
|
||||
searchLimit?: number;
|
||||
}
|
||||
|
||||
/**
|
||||
* Resolve the active search knob set for one search call.
|
||||
*
|
||||
* Resolution chain (matches v0.31.12 model-tier semantics):
|
||||
* 1. perCallOpts.<key> if defined → wins
|
||||
* 2. config.search.<key> if defined → wins
|
||||
* 3. MODE_BUNDLES[config.search.mode].<key> → mode default
|
||||
* 4. MODE_BUNDLES.balanced.<key> → safety fallback when config.search.mode is invalid/unset
|
||||
*
|
||||
* Pure function: no DB calls, no env reads. Caller pre-loads the relevant
|
||||
* config rows (one SELECT for the whole batch of keys, not one per key).
|
||||
*/
|
||||
export interface ResolveSearchModeInput {
|
||||
/** Resolved value of `config.search.mode`. Undefined → fallback to balanced. */
|
||||
mode?: string;
|
||||
/** Resolved per-key overrides from config table. */
|
||||
overrides?: SearchKeyOverrides;
|
||||
/** Per-call opts (SearchOpts / HybridSearchOpts). */
|
||||
perCall?: SearchPerCallOpts;
|
||||
}
|
||||
|
||||
export interface ResolvedSearchKnobs extends ModeBundle {
|
||||
/** Which mode bundle supplied the defaults (after fallback). */
|
||||
resolved_mode: SearchMode;
|
||||
/** True if the caller's `mode` input was a recognized SearchMode. */
|
||||
mode_valid: boolean;
|
||||
}
|
||||
|
||||
export function resolveSearchMode(input: ResolveSearchModeInput): ResolvedSearchKnobs {
|
||||
const requested = typeof input.mode === 'string' ? input.mode.trim().toLowerCase() : '';
|
||||
const valid = isSearchMode(requested);
|
||||
const resolved_mode: SearchMode = valid ? (requested as SearchMode) : DEFAULT_SEARCH_MODE;
|
||||
const bundle = MODE_BUNDLES[resolved_mode];
|
||||
|
||||
const ov = input.overrides ?? {};
|
||||
const pc = input.perCall ?? {};
|
||||
|
||||
const pick = <K extends keyof ModeBundle>(key: K): ModeBundle[K] => {
|
||||
if (pc[key] !== undefined) return pc[key] as ModeBundle[K];
|
||||
if (ov[key] !== undefined) return ov[key] as ModeBundle[K];
|
||||
return bundle[key];
|
||||
};
|
||||
|
||||
return {
|
||||
cache_enabled: pick('cache_enabled'),
|
||||
cache_similarity_threshold: pick('cache_similarity_threshold'),
|
||||
cache_ttl_seconds: pick('cache_ttl_seconds'),
|
||||
intentWeighting: pick('intentWeighting'),
|
||||
tokenBudget: pick('tokenBudget'),
|
||||
expansion: pick('expansion'),
|
||||
searchLimit: pick('searchLimit'),
|
||||
resolved_mode,
|
||||
mode_valid: valid,
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Per-knob source attribution for `gbrain search modes` dashboard.
|
||||
* Tells the user where each resolved value came from so override drift
|
||||
* is legible. Mirrors `gbrain models` (v0.31.12) attribution shape.
|
||||
*/
|
||||
export type KnobSource = 'per-call' | 'override' | 'mode' | 'fallback';
|
||||
|
||||
export interface ResolvedKnobAttribution {
|
||||
knob: keyof ModeBundle;
|
||||
value: ModeBundle[keyof ModeBundle];
|
||||
source: KnobSource;
|
||||
// For 'override' source, the config key path; for 'mode' source, the mode name.
|
||||
source_detail: string;
|
||||
}
|
||||
|
||||
export function attributeKnob<K extends keyof ModeBundle>(
|
||||
knob: K,
|
||||
input: ResolveSearchModeInput,
|
||||
resolved: ResolvedSearchKnobs,
|
||||
): ResolvedKnobAttribution {
|
||||
const pc = input.perCall ?? {};
|
||||
const ov = input.overrides ?? {};
|
||||
if (pc[knob] !== undefined) {
|
||||
return { knob, value: resolved[knob], source: 'per-call', source_detail: 'SearchOpts' };
|
||||
}
|
||||
if (ov[knob] !== undefined) {
|
||||
return { knob, value: resolved[knob], source: 'override', source_detail: `config: search.${knob}` };
|
||||
}
|
||||
if (resolved.mode_valid) {
|
||||
return { knob, value: resolved[knob], source: 'mode', source_detail: `mode: ${resolved.resolved_mode}` };
|
||||
}
|
||||
return { knob, value: resolved[knob], source: 'fallback', source_detail: `mode: ${DEFAULT_SEARCH_MODE} (default — search.mode unset)` };
|
||||
}
|
||||
|
||||
/**
|
||||
* Stable hash of the resolved knob set. Used as part of the query_cache
|
||||
* primary key so a tokenmax cache write can't be served to a conservative
|
||||
* lookup (cross-mode contamination, [CDX-4]).
|
||||
*
|
||||
* Knob order is FIXED so the hash is deterministic across releases. NEVER
|
||||
* reorder or add a knob without bumping a constant — a hash collision would
|
||||
* mean stale cache rows silently reading the wrong shape.
|
||||
*/
|
||||
export const KNOBS_HASH_VERSION = 1;
|
||||
|
||||
export function knobsHash(knobs: ResolvedSearchKnobs): string {
|
||||
// Fixed-order key list. Adding a knob here REQUIRES bumping
|
||||
// KNOBS_HASH_VERSION and is a breaking change for any persisted cache.
|
||||
const parts = [
|
||||
`v=${KNOBS_HASH_VERSION}`,
|
||||
`mode=${knobs.resolved_mode}`,
|
||||
`cache=${knobs.cache_enabled ? 1 : 0}`,
|
||||
`sim=${knobs.cache_similarity_threshold.toFixed(4)}`,
|
||||
`ttl=${knobs.cache_ttl_seconds}`,
|
||||
`iw=${knobs.intentWeighting ? 1 : 0}`,
|
||||
`tb=${knobs.tokenBudget ?? 'none'}`,
|
||||
`exp=${knobs.expansion ? 1 : 0}`,
|
||||
`lim=${knobs.searchLimit}`,
|
||||
];
|
||||
const h = createHash('sha256');
|
||||
h.update(parts.join('|'));
|
||||
return h.digest('hex').slice(0, 16);
|
||||
}
|
||||
|
||||
/**
|
||||
* Convenience: build SearchKeyOverrides from a flat config-table snapshot.
|
||||
* Used by hybridSearch's hot path so the search code pays one round-trip
|
||||
* to load all relevant config keys rather than one per knob.
|
||||
*
|
||||
* Returns sparse overrides — only keys actually present in the config
|
||||
* map appear. Falsy/missing keys fall through to the mode bundle default.
|
||||
*/
|
||||
export function loadOverridesFromConfig(
|
||||
configMap: Record<string, string | undefined>,
|
||||
): SearchKeyOverrides {
|
||||
const out: SearchKeyOverrides = {};
|
||||
const get = (k: string): string | undefined => configMap[k];
|
||||
|
||||
const ce = get('search.cache.enabled');
|
||||
if (ce !== undefined) {
|
||||
out.cache_enabled = ce === '1' || ce.toLowerCase() === 'true';
|
||||
}
|
||||
const st = get('search.cache.similarity_threshold');
|
||||
if (st !== undefined) {
|
||||
const n = parseFloat(st);
|
||||
if (Number.isFinite(n) && n > 0 && n <= 1) out.cache_similarity_threshold = n;
|
||||
}
|
||||
const tt = get('search.cache.ttl_seconds');
|
||||
if (tt !== undefined) {
|
||||
const n = parseInt(tt, 10);
|
||||
if (Number.isFinite(n) && n > 0) out.cache_ttl_seconds = n;
|
||||
}
|
||||
const iw = get('search.intentWeighting');
|
||||
if (iw !== undefined) {
|
||||
out.intentWeighting = iw === '1' || iw.toLowerCase() === 'true';
|
||||
}
|
||||
const tb = get('search.tokenBudget');
|
||||
if (tb !== undefined) {
|
||||
const n = parseInt(tb, 10);
|
||||
if (Number.isFinite(n) && n > 0) out.tokenBudget = n;
|
||||
}
|
||||
const ex = get('search.expansion');
|
||||
if (ex !== undefined) {
|
||||
out.expansion = ex === '1' || ex.toLowerCase() === 'true';
|
||||
}
|
||||
const sl = get('search.searchLimit');
|
||||
if (sl !== undefined) {
|
||||
const n = parseInt(sl, 10);
|
||||
if (Number.isFinite(n) && n > 0) out.searchLimit = n;
|
||||
}
|
||||
|
||||
return out;
|
||||
}
|
||||
|
||||
/** The full list of config keys this module reads. Used by `gbrain search modes --reset`. */
|
||||
export const SEARCH_MODE_CONFIG_KEYS: ReadonlyArray<string> = Object.freeze([
|
||||
'search.cache.enabled',
|
||||
'search.cache.similarity_threshold',
|
||||
'search.cache.ttl_seconds',
|
||||
'search.intentWeighting',
|
||||
'search.tokenBudget',
|
||||
'search.expansion',
|
||||
'search.searchLimit',
|
||||
]);
|
||||
|
||||
/**
|
||||
* The mode-selection config key itself. Separated from SEARCH_MODE_CONFIG_KEYS
|
||||
* because `--reset` clears OVERRIDES (the per-knob keys) but should NOT clear
|
||||
* the operator's mode choice.
|
||||
*/
|
||||
export const SEARCH_MODE_KEY = 'search.mode';
|
||||
|
||||
/**
|
||||
* Load the live mode config (mode + per-key overrides) from the brain engine.
|
||||
* Runs ONE round-trip per knob currently — the BrainEngine.getConfig interface
|
||||
* is single-key. A future v0.34 batch loader can collapse this. Volume is
|
||||
* small (~8 keys); call site is once per search.
|
||||
*
|
||||
* Errors are swallowed and fall through to mode-bundle defaults. The cache
|
||||
* config table predates v0.32.3 and may not exist on very old brains, so
|
||||
* silent fallback is the right shape.
|
||||
*/
|
||||
export async function loadSearchModeConfig(
|
||||
engine: { getConfig(key: string): Promise<string | null> },
|
||||
): Promise<ResolveSearchModeInput> {
|
||||
const safeGet = async (k: string): Promise<string | undefined> => {
|
||||
try {
|
||||
const v = await engine.getConfig(k);
|
||||
return v == null ? undefined : v;
|
||||
} catch {
|
||||
return undefined;
|
||||
}
|
||||
};
|
||||
|
||||
const [mode, ...overrideValues] = await Promise.all([
|
||||
safeGet(SEARCH_MODE_KEY),
|
||||
...SEARCH_MODE_CONFIG_KEYS.map(safeGet),
|
||||
]);
|
||||
|
||||
const configMap: Record<string, string | undefined> = {};
|
||||
SEARCH_MODE_CONFIG_KEYS.forEach((key, i) => {
|
||||
if (overrideValues[i] !== undefined) configMap[key] = overrideValues[i];
|
||||
});
|
||||
|
||||
return {
|
||||
mode,
|
||||
overrides: loadOverridesFromConfig(configMap),
|
||||
};
|
||||
}
|
||||
|
||||
@@ -0,0 +1,372 @@
|
||||
/**
|
||||
* Semantic Query Cache (v0.32.x — search-lite)
|
||||
*
|
||||
* Caches hybridSearch results keyed by query embedding similarity. On
|
||||
* each lookup the cache runs an HNSW cosine-distance probe against
|
||||
* stored query embeddings. If the closest cached query is within
|
||||
* `1 - similarity_threshold` cosine distance (default similarity
|
||||
* >= 0.92, distance < 0.08), we return the stored results instantly
|
||||
* — no keyword search, no vector search, no LLM expansion, no RRF, no
|
||||
* dedup. Otherwise we report a miss and let the caller run the real
|
||||
* search.
|
||||
*
|
||||
* Storage: the `query_cache` table (migration v51) with the same
|
||||
* embedding dim as `content_chunks`. Per-row TTL (default 3600 seconds).
|
||||
* Stale rows are skipped at read time and pruned by `gbrain cache prune`.
|
||||
*
|
||||
* Multi-source isolation: cache lookups scope by `source_id` so brain
|
||||
* A's "who is widget-ceo" cannot return brain B's cached results for
|
||||
* the same query.
|
||||
*
|
||||
* Edge cases:
|
||||
* - No embedding available (no OPENAI key, embed failure): cache is
|
||||
* skipped silently. Caller runs the normal pipeline.
|
||||
* - Table missing (pre-v51 brain): every read swallows the error and
|
||||
* reports a miss. No throw.
|
||||
* - Cache disabled by config or by caller (`useCache: false`): the
|
||||
* cache module is never called.
|
||||
*
|
||||
* Tested in test/query-cache.test.ts.
|
||||
*/
|
||||
|
||||
import { createHash } from 'node:crypto';
|
||||
import type { BrainEngine } from '../engine.ts';
|
||||
import type { SearchResult, HybridSearchMeta } from '../types.ts';
|
||||
|
||||
/** Default cosine similarity threshold for cache hits. */
|
||||
export const DEFAULT_SIMILARITY_THRESHOLD = 0.92;
|
||||
/** Default TTL for cache entries, in seconds. */
|
||||
export const DEFAULT_TTL_SECONDS = 3600;
|
||||
|
||||
export interface CacheLookupResult {
|
||||
hit: boolean;
|
||||
results?: SearchResult[];
|
||||
meta?: HybridSearchMeta;
|
||||
/** Cosine similarity of the matched cached query (0..1). Only set on hit. */
|
||||
similarity?: number;
|
||||
/** Age of the cached entry in seconds. Only set on hit. */
|
||||
ageSeconds?: number;
|
||||
}
|
||||
|
||||
export interface CacheStats {
|
||||
total_rows: number;
|
||||
total_hits: number;
|
||||
fresh_rows: number;
|
||||
stale_rows: number;
|
||||
}
|
||||
|
||||
export interface QueryCacheConfig {
|
||||
enabled?: boolean;
|
||||
similarityThreshold?: number;
|
||||
ttlSeconds?: number;
|
||||
}
|
||||
|
||||
/**
|
||||
* Deterministic ID for a (query, source, knobsHash) tuple. Used as the primary
|
||||
* key so re-caching the exact same (query, mode, knobs) just bumps the row's
|
||||
* hit_count and created_at rather than inserting duplicates.
|
||||
*
|
||||
* v0.32.3 [CDX-4]: knobsHash is now part of the key to prevent cross-mode
|
||||
* cache contamination. A tokenmax write (expansion=on, limit=50) and a
|
||||
* conservative write (no expansion, limit=10) for the same query+source
|
||||
* land in distinct rows. Empty-string knobsHash is accepted (preserves
|
||||
* existing test setups) but production calls always pass the resolved hash.
|
||||
*/
|
||||
export function cacheRowId(queryText: string, sourceId: string, knobsHash = ''): string {
|
||||
const h = createHash('sha256');
|
||||
h.update(`${sourceId}::${queryText}::${knobsHash}`);
|
||||
return h.digest('hex').slice(0, 32);
|
||||
}
|
||||
|
||||
/**
|
||||
* Convert a Float32Array embedding into the pgvector text literal
|
||||
* format: `[v0,v1,v2,...]`. PGLite and Postgres both accept this when
|
||||
* the parameter is cast to `vector` or `halfvec` on the server side.
|
||||
*/
|
||||
function embeddingToPgVector(embedding: Float32Array): string {
|
||||
// Stringify with reasonable precision. pgvector accepts plain decimals.
|
||||
// Use a typed iteration to avoid TS errors on Float32Array forEach.
|
||||
const parts: string[] = new Array(embedding.length);
|
||||
for (let i = 0; i < embedding.length; i++) {
|
||||
parts[i] = embedding[i].toFixed(6);
|
||||
}
|
||||
return `[${parts.join(',')}]`;
|
||||
}
|
||||
|
||||
export class SemanticQueryCache {
|
||||
private similarityThreshold: number;
|
||||
private ttlSeconds: number;
|
||||
private enabled: boolean;
|
||||
|
||||
constructor(
|
||||
private engine: BrainEngine,
|
||||
config?: QueryCacheConfig,
|
||||
) {
|
||||
this.enabled = config?.enabled ?? true;
|
||||
this.similarityThreshold = clampThreshold(config?.similarityThreshold);
|
||||
this.ttlSeconds = clampTtl(config?.ttlSeconds);
|
||||
}
|
||||
|
||||
isEnabled(): boolean {
|
||||
return this.enabled;
|
||||
}
|
||||
|
||||
/**
|
||||
* Look up a cached result set by query embedding similarity. Returns a
|
||||
* miss (hit=false) when:
|
||||
* - cache is disabled,
|
||||
* - embedding is null/empty,
|
||||
* - the table doesn't exist (pre-v51 brain),
|
||||
* - no row is within the similarity threshold,
|
||||
* - the matching row is past its TTL.
|
||||
*
|
||||
* All errors are swallowed and converted to a miss \u2014 the cache must
|
||||
* never break the search hot path.
|
||||
*/
|
||||
async lookup(
|
||||
queryEmbedding: Float32Array | null,
|
||||
opts: { sourceId?: string; knobsHash?: string } = {},
|
||||
): Promise<CacheLookupResult> {
|
||||
if (!this.enabled || !queryEmbedding || queryEmbedding.length === 0) {
|
||||
return { hit: false };
|
||||
}
|
||||
const sourceId = opts.sourceId ?? 'default';
|
||||
const knobsHash = opts.knobsHash ?? '';
|
||||
const distanceThreshold = 1 - this.similarityThreshold;
|
||||
const vec = embeddingToPgVector(queryEmbedding);
|
||||
|
||||
try {
|
||||
// Find the closest cached query within the distance threshold and
|
||||
// freshness window. The TTL check is done in-query (created_at +
|
||||
// ttl_seconds > now) so we never return a stale row.
|
||||
//
|
||||
// v0.32.3 [CDX-4]: knobs_hash filter prevents cross-mode contamination.
|
||||
// A tokenmax write (expansion=on, limit=50) and a conservative read
|
||||
// (no expansion, limit=10) have distinct knobs hashes and miss each
|
||||
// other. Rows with NULL knobs_hash (pre-v0.32.3) are excluded.
|
||||
const rows = await this.engine.executeRaw<{
|
||||
id: string;
|
||||
results: unknown;
|
||||
meta: unknown;
|
||||
distance: number;
|
||||
age_seconds: number;
|
||||
}>(
|
||||
`SELECT id, results, meta,
|
||||
embedding <=> $1::vector AS distance,
|
||||
EXTRACT(EPOCH FROM (now() - created_at))::int AS age_seconds
|
||||
FROM query_cache
|
||||
WHERE source_id = $2
|
||||
AND knobs_hash = $4
|
||||
AND embedding IS NOT NULL
|
||||
AND embedding <=> $1::vector < $3
|
||||
AND created_at + (ttl_seconds || ' seconds')::interval > now()
|
||||
ORDER BY embedding <=> $1::vector
|
||||
LIMIT 1`,
|
||||
[vec, sourceId, distanceThreshold, knobsHash],
|
||||
);
|
||||
|
||||
if (rows.length === 0) return { hit: false };
|
||||
|
||||
const row = rows[0];
|
||||
const results = Array.isArray(row.results)
|
||||
? (row.results as SearchResult[])
|
||||
: safeJsonParse<SearchResult[]>(row.results, []);
|
||||
const meta = safeJsonParse<HybridSearchMeta | undefined>(row.meta, undefined);
|
||||
const similarity = 1 - row.distance;
|
||||
|
||||
// Bump hit_count / last_hit_at \u2014 best-effort.
|
||||
void this.bumpHit(row.id).catch(() => { /* swallow */ });
|
||||
|
||||
return {
|
||||
hit: true,
|
||||
results,
|
||||
meta,
|
||||
similarity,
|
||||
ageSeconds: row.age_seconds,
|
||||
};
|
||||
} catch {
|
||||
// Table missing, vector column missing, or any other failure: miss.
|
||||
return { hit: false };
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Persist a fresh search result set into the cache. Idempotent on
|
||||
* (query_text + source_id) \u2014 re-writes the row with a fresh
|
||||
* created_at. Best-effort: errors are swallowed.
|
||||
*/
|
||||
async store(
|
||||
queryText: string,
|
||||
queryEmbedding: Float32Array | null,
|
||||
results: SearchResult[],
|
||||
meta: HybridSearchMeta,
|
||||
opts: { sourceId?: string; ttlSeconds?: number; knobsHash?: string } = {},
|
||||
): Promise<void> {
|
||||
if (!this.enabled || !queryEmbedding || queryEmbedding.length === 0) return;
|
||||
const sourceId = opts.sourceId ?? 'default';
|
||||
const knobsHash = opts.knobsHash ?? '';
|
||||
const ttl = clampTtl(opts.ttlSeconds ?? this.ttlSeconds);
|
||||
const id = cacheRowId(queryText, sourceId, knobsHash);
|
||||
const vec = embeddingToPgVector(queryEmbedding);
|
||||
|
||||
try {
|
||||
// v0.32.3 [CDX-4]: knobs_hash threaded into the row so concurrent
|
||||
// tokenmax + conservative writes for the same query+source live as
|
||||
// distinct rows. The PK is `id` (which already encodes the hash),
|
||||
// so ON CONFLICT (id) DO UPDATE just refreshes the same-mode row.
|
||||
await this.engine.executeRaw(
|
||||
`INSERT INTO query_cache (id, query_text, source_id, knobs_hash, embedding, results, meta, ttl_seconds, created_at)
|
||||
VALUES ($1, $2, $3, $4, $5::vector, $6::jsonb, $7::jsonb, $8, now())
|
||||
ON CONFLICT (id) DO UPDATE SET
|
||||
query_text = EXCLUDED.query_text,
|
||||
knobs_hash = EXCLUDED.knobs_hash,
|
||||
embedding = EXCLUDED.embedding,
|
||||
results = EXCLUDED.results,
|
||||
meta = EXCLUDED.meta,
|
||||
ttl_seconds = EXCLUDED.ttl_seconds,
|
||||
created_at = now()`,
|
||||
[
|
||||
id,
|
||||
queryText,
|
||||
sourceId,
|
||||
knobsHash,
|
||||
vec,
|
||||
JSON.stringify(results),
|
||||
JSON.stringify(meta),
|
||||
ttl,
|
||||
],
|
||||
);
|
||||
} catch {
|
||||
// swallow \u2014 cache write must never break the search hot path.
|
||||
}
|
||||
}
|
||||
|
||||
/** Clear ALL cache rows (optionally scoped by source). Returns rows deleted. */
|
||||
async clear(opts: { sourceId?: string } = {}): Promise<number> {
|
||||
try {
|
||||
if (opts.sourceId) {
|
||||
const rows = await this.engine.executeRaw<{ n: number }>(
|
||||
`WITH deleted AS (DELETE FROM query_cache WHERE source_id = $1 RETURNING 1)
|
||||
SELECT COUNT(*)::int AS n FROM deleted`,
|
||||
[opts.sourceId],
|
||||
);
|
||||
return rows[0]?.n ?? 0;
|
||||
}
|
||||
const rows = await this.engine.executeRaw<{ n: number }>(
|
||||
`WITH deleted AS (DELETE FROM query_cache RETURNING 1)
|
||||
SELECT COUNT(*)::int AS n FROM deleted`,
|
||||
);
|
||||
return rows[0]?.n ?? 0;
|
||||
} catch {
|
||||
return 0;
|
||||
}
|
||||
}
|
||||
|
||||
/** Delete only stale (past-TTL) rows. Returns rows deleted. */
|
||||
async prune(): Promise<number> {
|
||||
try {
|
||||
const rows = await this.engine.executeRaw<{ n: number }>(
|
||||
`WITH deleted AS (
|
||||
DELETE FROM query_cache
|
||||
WHERE created_at + (ttl_seconds || ' seconds')::interval <= now()
|
||||
RETURNING 1
|
||||
)
|
||||
SELECT COUNT(*)::int AS n FROM deleted`,
|
||||
);
|
||||
return rows[0]?.n ?? 0;
|
||||
} catch {
|
||||
return 0;
|
||||
}
|
||||
}
|
||||
|
||||
/** Summary stats for `gbrain cache stats`. */
|
||||
async stats(): Promise<CacheStats> {
|
||||
try {
|
||||
const rows = await this.engine.executeRaw<{
|
||||
total_rows: number;
|
||||
total_hits: number;
|
||||
fresh_rows: number;
|
||||
stale_rows: number;
|
||||
}>(
|
||||
`SELECT
|
||||
COUNT(*)::int AS total_rows,
|
||||
COALESCE(SUM(hit_count), 0)::int AS total_hits,
|
||||
COUNT(*) FILTER (WHERE created_at + (ttl_seconds || ' seconds')::interval > now())::int AS fresh_rows,
|
||||
COUNT(*) FILTER (WHERE created_at + (ttl_seconds || ' seconds')::interval <= now())::int AS stale_rows
|
||||
FROM query_cache`,
|
||||
);
|
||||
return rows[0] ?? { total_rows: 0, total_hits: 0, fresh_rows: 0, stale_rows: 0 };
|
||||
} catch {
|
||||
return { total_rows: 0, total_hits: 0, fresh_rows: 0, stale_rows: 0 };
|
||||
}
|
||||
}
|
||||
|
||||
private async bumpHit(id: string): Promise<void> {
|
||||
await this.engine.executeRaw(
|
||||
`UPDATE query_cache
|
||||
SET hit_count = hit_count + 1, last_hit_at = now()
|
||||
WHERE id = $1`,
|
||||
[id],
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
function clampThreshold(v: number | undefined): number {
|
||||
if (typeof v !== 'number' || !Number.isFinite(v)) return DEFAULT_SIMILARITY_THRESHOLD;
|
||||
return Math.max(0.5, Math.min(0.999, v));
|
||||
}
|
||||
|
||||
function clampTtl(v: number | undefined): number {
|
||||
if (typeof v !== 'number' || !Number.isFinite(v) || v <= 0) return DEFAULT_TTL_SECONDS;
|
||||
// Cap at 30 days to avoid runaway TTLs.
|
||||
return Math.min(60 * 60 * 24 * 30, Math.floor(v));
|
||||
}
|
||||
|
||||
function safeJsonParse<T>(value: unknown, fallback: T): T {
|
||||
if (value == null) return fallback;
|
||||
if (typeof value === 'object') return value as T;
|
||||
if (typeof value !== 'string') return fallback;
|
||||
try {
|
||||
return JSON.parse(value) as T;
|
||||
} catch {
|
||||
return fallback;
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Resolve cache config from the engine's `config` table. Mirrors how
|
||||
* other modules read runtime config (e.g. embedding_dimensions). All
|
||||
* three keys have sensible defaults; missing rows fall back to those.
|
||||
*/
|
||||
export async function loadCacheConfig(engine: BrainEngine): Promise<QueryCacheConfig> {
|
||||
const keys = [
|
||||
'search.cache.enabled',
|
||||
'search.cache.similarity_threshold',
|
||||
'search.cache.ttl_seconds',
|
||||
];
|
||||
const config: QueryCacheConfig = {
|
||||
enabled: true,
|
||||
similarityThreshold: DEFAULT_SIMILARITY_THRESHOLD,
|
||||
ttlSeconds: DEFAULT_TTL_SECONDS,
|
||||
};
|
||||
try {
|
||||
const rows = await engine.executeRaw<{ key: string; value: string }>(
|
||||
`SELECT key, value FROM config WHERE key = ANY($1)`,
|
||||
[keys],
|
||||
);
|
||||
for (const row of rows) {
|
||||
if (row.key === 'search.cache.enabled') {
|
||||
config.enabled = row.value === '1' || row.value.toLowerCase() === 'true';
|
||||
} else if (row.key === 'search.cache.similarity_threshold') {
|
||||
const v = parseFloat(row.value);
|
||||
if (Number.isFinite(v)) config.similarityThreshold = clampThreshold(v);
|
||||
} else if (row.key === 'search.cache.ttl_seconds') {
|
||||
const v = parseInt(row.value, 10);
|
||||
if (Number.isFinite(v)) config.ttlSeconds = clampTtl(v);
|
||||
}
|
||||
}
|
||||
} catch {
|
||||
// Use defaults.
|
||||
}
|
||||
return config;
|
||||
}
|
||||
@@ -0,0 +1,364 @@
|
||||
/**
|
||||
* v0.32.3 search-lite telemetry rollup writer.
|
||||
*
|
||||
* Architecture decision (D2 in the plan, [CDX-19]): per-process in-memory
|
||||
* bucket, flushed periodically (60s OR 100 calls, whichever first) AND on
|
||||
* process exit via beforeExit/SIGINT/SIGTERM with a 2-second timeout cap.
|
||||
* The search hot path NEVER waits on this write — `record()` is sync and
|
||||
* the flush is fire-and-forget.
|
||||
*
|
||||
* Schema math per [CDX-17]: rows are sums + counts only, NEVER averages.
|
||||
* Read-time derives averages. ON CONFLICT DO UPDATE adds raw values so two
|
||||
* gbrain processes flushing the same (date, mode, intent) tuple accumulate
|
||||
* correctly.
|
||||
*
|
||||
* The bucket map is keyed by `${date}::${mode}::${intent}`. Date is the
|
||||
* UTC ISO date (YYYY-MM-DD). Cross-midnight calls land in distinct buckets
|
||||
* automatically.
|
||||
*
|
||||
* Per-process bucketing means stdio MCP, HTTP MCP, and CLI processes each
|
||||
* maintain their own buffers. Stats are directional, not exact — acceptable
|
||||
* because the consumer is the operator (or an agent running `gbrain search
|
||||
* tune`), not a financial ledger. The "lose last bucket on hard crash"
|
||||
* downside is documented in the methodology doc.
|
||||
*/
|
||||
|
||||
import type { BrainEngine } from '../engine.ts';
|
||||
import type { HybridSearchMeta } from '../types.ts';
|
||||
|
||||
interface Bucket {
|
||||
date: string;
|
||||
mode: string;
|
||||
intent: string;
|
||||
count: number;
|
||||
sum_results: number;
|
||||
sum_tokens: number;
|
||||
sum_budget_dropped: number;
|
||||
cache_hit: number;
|
||||
cache_miss: number;
|
||||
}
|
||||
|
||||
const FLUSH_INTERVAL_MS = 60_000;
|
||||
const FLUSH_THRESHOLD_CALLS = 100;
|
||||
|
||||
/**
|
||||
* Per-process telemetry singleton. Each gbrain process (CLI, stdio MCP,
|
||||
* HTTP MCP) gets one instance. The flush timer and exit hooks are
|
||||
* installed lazily on the first `record()` call so importing this module
|
||||
* has no side effects.
|
||||
*/
|
||||
class TelemetryWriter {
|
||||
private buckets = new Map<string, Bucket>();
|
||||
private pendingCount = 0;
|
||||
private engine: BrainEngine | null = null;
|
||||
private timer: NodeJS.Timeout | null = null;
|
||||
private exitHookInstalled = false;
|
||||
private flushInFlight: Promise<void> | null = null;
|
||||
|
||||
/** Wire the engine. Called once per process at search-time. Subsequent calls are a no-op. */
|
||||
setEngine(engine: BrainEngine): void {
|
||||
if (!this.engine) {
|
||||
this.engine = engine;
|
||||
this.ensureExitHook();
|
||||
this.ensureTimer();
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Record a search call. Sync — never blocks the hot path. Returns
|
||||
* immediately after bumping the in-memory bucket. Flush is async +
|
||||
* fire-and-forget.
|
||||
*/
|
||||
record(meta: HybridSearchMeta, opts: { results_count: number; tokens_estimate?: number } = { results_count: 0 }): void {
|
||||
const date = nowDate();
|
||||
const mode = meta.mode ?? 'unset';
|
||||
const intent = meta.intent ?? 'unset';
|
||||
const key = `${date}::${mode}::${intent}`;
|
||||
|
||||
let b = this.buckets.get(key);
|
||||
if (!b) {
|
||||
b = {
|
||||
date,
|
||||
mode,
|
||||
intent,
|
||||
count: 0,
|
||||
sum_results: 0,
|
||||
sum_tokens: 0,
|
||||
sum_budget_dropped: 0,
|
||||
cache_hit: 0,
|
||||
cache_miss: 0,
|
||||
};
|
||||
this.buckets.set(key, b);
|
||||
}
|
||||
|
||||
b.count += 1;
|
||||
b.sum_results += Math.max(0, Math.floor(opts.results_count));
|
||||
b.sum_tokens += Math.max(0, Math.floor(opts.tokens_estimate ?? meta.token_budget?.used ?? 0));
|
||||
b.sum_budget_dropped += Math.max(0, Math.floor(meta.token_budget?.dropped ?? 0));
|
||||
if (meta.cache?.status === 'hit') b.cache_hit += 1;
|
||||
if (meta.cache?.status === 'miss') b.cache_miss += 1;
|
||||
|
||||
this.pendingCount += 1;
|
||||
if (this.pendingCount >= FLUSH_THRESHOLD_CALLS) {
|
||||
void this.flush().catch(() => { /* swallow */ });
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Drain the bucket map to the database. Idempotent; concurrent flushes
|
||||
* are coalesced via flushInFlight. The bucket map is swapped atomically
|
||||
* before the SQL write so new `record()` calls during flush land in a
|
||||
* fresh map.
|
||||
*/
|
||||
async flush(): Promise<void> {
|
||||
if (this.flushInFlight) return this.flushInFlight;
|
||||
if (!this.engine || this.buckets.size === 0) {
|
||||
this.pendingCount = 0;
|
||||
return;
|
||||
}
|
||||
|
||||
// Swap the map: a new record() call during flush goes into the new map.
|
||||
const snapshot = this.buckets;
|
||||
this.buckets = new Map();
|
||||
this.pendingCount = 0;
|
||||
|
||||
const engine = this.engine;
|
||||
this.flushInFlight = (async () => {
|
||||
try {
|
||||
for (const b of snapshot.values()) {
|
||||
try {
|
||||
await engine.executeRaw(
|
||||
`INSERT INTO search_telemetry
|
||||
(date, mode, intent, count, sum_results, sum_tokens, sum_budget_dropped, cache_hit, cache_miss, first_seen, last_seen)
|
||||
VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, now(), now())
|
||||
ON CONFLICT (date, mode, intent) DO UPDATE SET
|
||||
count = search_telemetry.count + EXCLUDED.count,
|
||||
sum_results = search_telemetry.sum_results + EXCLUDED.sum_results,
|
||||
sum_tokens = search_telemetry.sum_tokens + EXCLUDED.sum_tokens,
|
||||
sum_budget_dropped = search_telemetry.sum_budget_dropped + EXCLUDED.sum_budget_dropped,
|
||||
cache_hit = search_telemetry.cache_hit + EXCLUDED.cache_hit,
|
||||
cache_miss = search_telemetry.cache_miss + EXCLUDED.cache_miss,
|
||||
last_seen = now()`,
|
||||
[b.date, b.mode, b.intent, b.count, b.sum_results, b.sum_tokens, b.sum_budget_dropped, b.cache_hit, b.cache_miss],
|
||||
);
|
||||
} catch {
|
||||
// swallow — telemetry write must never break the hot path.
|
||||
// Per-bucket isolation: one bad row doesn't lose the others.
|
||||
}
|
||||
}
|
||||
} finally {
|
||||
this.flushInFlight = null;
|
||||
}
|
||||
})();
|
||||
return this.flushInFlight;
|
||||
}
|
||||
|
||||
/** Stop the timer and uninstall exit hooks. Called from tests / shutdown. */
|
||||
stop(): void {
|
||||
if (this.timer) {
|
||||
clearInterval(this.timer);
|
||||
this.timer = null;
|
||||
}
|
||||
this.buckets.clear();
|
||||
this.pendingCount = 0;
|
||||
}
|
||||
|
||||
/** Test-only — read the current bucket count without draining. */
|
||||
bucketCountForTest(): number {
|
||||
return this.buckets.size;
|
||||
}
|
||||
|
||||
/** Test-only — read a specific bucket (returns null if absent). */
|
||||
bucketForTest(date: string, mode: string, intent: string): Readonly<Bucket> | null {
|
||||
return this.buckets.get(`${date}::${mode}::${intent}`) ?? null;
|
||||
}
|
||||
|
||||
private ensureTimer(): void {
|
||||
if (this.timer) return;
|
||||
this.timer = setInterval(() => {
|
||||
void this.flush().catch(() => { /* swallow */ });
|
||||
}, FLUSH_INTERVAL_MS);
|
||||
// Unref so the timer doesn't keep the process alive on its own.
|
||||
if (typeof this.timer.unref === 'function') this.timer.unref();
|
||||
}
|
||||
|
||||
private ensureExitHook(): void {
|
||||
if (this.exitHookInstalled) return;
|
||||
this.exitHookInstalled = true;
|
||||
|
||||
// Lossy by design: skip the buffered drain entirely on process exit.
|
||||
//
|
||||
// The earlier implementation installed `process.on('beforeExit', drainOnExit)`
|
||||
// with an inner `Promise.race([flush(), setTimeout(2000)])`. That enqueued
|
||||
// new async work AFTER the event loop had emptied, which kept the process
|
||||
// alive past beforeExit — short-lived CLI invocations (`gbrain query "the"`
|
||||
// exiting after 100ms of work) ended up waiting on the DB write to settle.
|
||||
// On a slow or busy PGLite, the write never settled and the CLI hung
|
||||
// forever. That deadlock surfaced as the `test/e2e/claw-test.test.ts`
|
||||
// hang (the harness spawns short-lived gbrain queries that should exit
|
||||
// in <1s but never did).
|
||||
//
|
||||
// Resolution per [CDX-19]: the periodic flush timer (unref'd) handles
|
||||
// long-running processes (HTTP MCP server, autopilot, jobs work). For
|
||||
// short-lived CLI invocations, telemetry buffering of one search call
|
||||
// is acceptable to lose. Stats are directional, not exact.
|
||||
//
|
||||
// The signal handlers (SIGINT / SIGTERM) also drop — kill -TERM should
|
||||
// exit immediately, not block on a DB write that may never complete.
|
||||
}
|
||||
|
||||
// Test-only: previously inspected by tests. Retained as a no-op so the
|
||||
// test harness's _resetTelemetryWriterForTest doesn't need to know about
|
||||
// the exit-hook decision.
|
||||
flushOnExitForTest(): Promise<void> {
|
||||
return this.flush().catch(() => { /* swallow */ });
|
||||
}
|
||||
}
|
||||
|
||||
/** Module-level singleton, one per process. */
|
||||
let _writer: TelemetryWriter | null = null;
|
||||
|
||||
export function getTelemetryWriter(): TelemetryWriter {
|
||||
if (!_writer) _writer = new TelemetryWriter();
|
||||
return _writer;
|
||||
}
|
||||
|
||||
/**
|
||||
* Convenience entry point for hot-path callers. Wires the engine lazily
|
||||
* and records the call. Never throws.
|
||||
*/
|
||||
export function recordSearchTelemetry(
|
||||
engine: BrainEngine,
|
||||
meta: HybridSearchMeta,
|
||||
opts: { results_count: number; tokens_estimate?: number } = { results_count: 0 },
|
||||
): void {
|
||||
try {
|
||||
const w = getTelemetryWriter();
|
||||
w.setEngine(engine);
|
||||
w.record(meta, opts);
|
||||
} catch {
|
||||
// swallow — telemetry is best-effort.
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Read aggregated stats over a window. Read-time derives averages from
|
||||
* sums + counts so writers can ON CONFLICT-add freely.
|
||||
*/
|
||||
export interface StatsWindow {
|
||||
total_calls: number;
|
||||
cache_hits: number;
|
||||
cache_misses: number;
|
||||
cache_hit_rate: number;
|
||||
avg_results: number;
|
||||
avg_tokens: number;
|
||||
total_budget_dropped: number;
|
||||
intent_distribution: Record<string, number>;
|
||||
mode_distribution: Record<string, number>;
|
||||
window_days: number;
|
||||
oldest_seen?: string;
|
||||
newest_seen?: string;
|
||||
}
|
||||
|
||||
export async function readSearchStats(
|
||||
engine: BrainEngine,
|
||||
opts: { days?: number } = {},
|
||||
): Promise<StatsWindow> {
|
||||
const days = Math.max(1, Math.min(365, opts.days ?? 7));
|
||||
const cutoffDate = new Date(Date.now() - days * 86_400_000).toISOString().slice(0, 10);
|
||||
|
||||
try {
|
||||
const rows = await engine.executeRaw<{
|
||||
mode: string;
|
||||
intent: string;
|
||||
count: number;
|
||||
sum_results: number;
|
||||
sum_tokens: number;
|
||||
sum_budget_dropped: number;
|
||||
cache_hit: number;
|
||||
cache_miss: number;
|
||||
first_seen: string;
|
||||
last_seen: string;
|
||||
}>(
|
||||
`SELECT mode, intent,
|
||||
SUM(count)::int AS count,
|
||||
SUM(sum_results)::int AS sum_results,
|
||||
SUM(sum_tokens)::int AS sum_tokens,
|
||||
SUM(sum_budget_dropped)::int AS sum_budget_dropped,
|
||||
SUM(cache_hit)::int AS cache_hit,
|
||||
SUM(cache_miss)::int AS cache_miss,
|
||||
MIN(first_seen)::text AS first_seen,
|
||||
MAX(last_seen)::text AS last_seen
|
||||
FROM search_telemetry
|
||||
WHERE date >= $1
|
||||
GROUP BY mode, intent`,
|
||||
[cutoffDate],
|
||||
);
|
||||
|
||||
let total_calls = 0;
|
||||
let cache_hits = 0;
|
||||
let cache_misses = 0;
|
||||
let total_results = 0;
|
||||
let total_tokens = 0;
|
||||
let total_budget_dropped = 0;
|
||||
const intent_distribution: Record<string, number> = {};
|
||||
const mode_distribution: Record<string, number> = {};
|
||||
let oldest_seen: string | undefined;
|
||||
let newest_seen: string | undefined;
|
||||
|
||||
for (const r of rows) {
|
||||
total_calls += r.count;
|
||||
cache_hits += r.cache_hit;
|
||||
cache_misses += r.cache_miss;
|
||||
total_results += r.sum_results;
|
||||
total_tokens += r.sum_tokens;
|
||||
total_budget_dropped += r.sum_budget_dropped;
|
||||
intent_distribution[r.intent] = (intent_distribution[r.intent] ?? 0) + r.count;
|
||||
mode_distribution[r.mode] = (mode_distribution[r.mode] ?? 0) + r.count;
|
||||
if (r.first_seen && (!oldest_seen || r.first_seen < oldest_seen)) oldest_seen = r.first_seen;
|
||||
if (r.last_seen && (!newest_seen || r.last_seen > newest_seen)) newest_seen = r.last_seen;
|
||||
}
|
||||
|
||||
const probe_total = cache_hits + cache_misses;
|
||||
return {
|
||||
total_calls,
|
||||
cache_hits,
|
||||
cache_misses,
|
||||
cache_hit_rate: probe_total > 0 ? cache_hits / probe_total : 0,
|
||||
avg_results: total_calls > 0 ? total_results / total_calls : 0,
|
||||
avg_tokens: total_calls > 0 ? total_tokens / total_calls : 0,
|
||||
total_budget_dropped,
|
||||
intent_distribution,
|
||||
mode_distribution,
|
||||
window_days: days,
|
||||
oldest_seen,
|
||||
newest_seen,
|
||||
};
|
||||
} catch {
|
||||
// Table missing or query failed — return empty stats rather than throw.
|
||||
return {
|
||||
total_calls: 0,
|
||||
cache_hits: 0,
|
||||
cache_misses: 0,
|
||||
cache_hit_rate: 0,
|
||||
avg_results: 0,
|
||||
avg_tokens: 0,
|
||||
total_budget_dropped: 0,
|
||||
intent_distribution: {},
|
||||
mode_distribution: {},
|
||||
window_days: days,
|
||||
};
|
||||
}
|
||||
}
|
||||
|
||||
function nowDate(): string {
|
||||
return new Date().toISOString().slice(0, 10);
|
||||
}
|
||||
|
||||
/** Test-only — reset the module-level singleton between test cases. */
|
||||
export function _resetTelemetryWriterForTest(): void {
|
||||
if (_writer) {
|
||||
_writer.stop();
|
||||
_writer = null;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,112 @@
|
||||
/**
|
||||
* Token Budget Enforcement on Search Results (v0.32.x — search-lite)
|
||||
*
|
||||
* Caps the cumulative token cost of a ranked SearchResult[] so callers
|
||||
* (agents, MCP, the query op) can guarantee their search payload fits a
|
||||
* downstream context window. The enforcer is the LAST stage of the search
|
||||
* pipeline — all scoring, ranking, dedup, boosts, two-pass walk are done
|
||||
* before this fires. It does NOT re-rank; it greedily walks top-down and
|
||||
* stops when the next result would push the running total past the
|
||||
* budget.
|
||||
*
|
||||
* Token counting uses a deliberately cheap char/4 heuristic instead of
|
||||
* dropping in a real tokenizer (js-tiktoken is 1.5MB+ and would balloon
|
||||
* the bun build --compile bundle). The heuristic is accurate within
|
||||
* ~10-15% for English text and ~5-25% for mixed code/Unicode — over-
|
||||
* estimating in code (which is what we want for a safety budget). For
|
||||
* a precise count, the caller can subtract real-tokens-vs-heuristic in
|
||||
* post and re-run with a tighter budget.
|
||||
*
|
||||
* Backward-compatibility: when no budget is set (undefined or <=0), the
|
||||
* enforcer is a no-op. The pre-v0.32 contract for search results is
|
||||
* unchanged.
|
||||
*
|
||||
* Pure module. No DB, no LLM, no async. Tested in test/token-budget.test.ts.
|
||||
*/
|
||||
|
||||
import type { SearchResult } from '../types.ts';
|
||||
|
||||
/**
|
||||
* Cheap char/4 token estimate. Returns 0 for empty strings.
|
||||
*
|
||||
* Why char/4: OpenAI's tokenization averages ~4 chars/token for English
|
||||
* prose; closer to 3 for code with lots of punctuation; up to 8 for
|
||||
* CJK. Overshoot is fine for a safety budget. Undershoot would let us
|
||||
* blow past the cap, so we round UP when in doubt.
|
||||
*/
|
||||
export function estimateTokens(text: string | null | undefined): number {
|
||||
if (!text) return 0;
|
||||
// Math.ceil so a 1-char string still costs at least 1 token.
|
||||
return Math.ceil(text.length / 4);
|
||||
}
|
||||
|
||||
/**
|
||||
* Per-result token cost: title + chunk_text. Slug is metadata and
|
||||
* doesn't enter the assistant context, so we don't count it. If a
|
||||
* caller wants a different cost model (e.g. including timeline detail
|
||||
* or compiled_truth length), they can pre-shape the chunk_text before
|
||||
* calling enforceTokenBudget.
|
||||
*/
|
||||
export function resultTokens(r: SearchResult): number {
|
||||
return estimateTokens(r.title) + estimateTokens(r.chunk_text);
|
||||
}
|
||||
|
||||
export interface TokenBudgetMeta {
|
||||
/** Token budget that was applied (verbatim from caller). */
|
||||
budget: number;
|
||||
/** Cumulative token cost of the returned results. */
|
||||
used: number;
|
||||
/** Count of results that were dropped to fit the budget. */
|
||||
dropped: number;
|
||||
/** Count of results actually returned. */
|
||||
kept: number;
|
||||
}
|
||||
|
||||
/**
|
||||
* Greedy top-down budget enforcement. Walks the input in order, accumulates
|
||||
* token costs, and stops as soon as adding the next result would exceed
|
||||
* the budget. Results are NOT re-ranked — caller's order is preserved.
|
||||
*
|
||||
* Edge cases (all preserve the pre-v0.32 contract):
|
||||
* - budget undefined / <= 0: returns input unchanged; dropped=0, kept=N.
|
||||
* - First result alone exceeds budget: returns []; dropped=N, kept=0.
|
||||
* (Intentionally strict: the caller asked for a hard cap.)
|
||||
* - Input empty: returns []; budget unused.
|
||||
*/
|
||||
export function enforceTokenBudget(
|
||||
results: SearchResult[],
|
||||
budget: number | undefined,
|
||||
): { results: SearchResult[]; meta: TokenBudgetMeta } {
|
||||
const safeBudget = typeof budget === 'number' && budget > 0 ? budget : 0;
|
||||
|
||||
if (safeBudget === 0 || results.length === 0) {
|
||||
return {
|
||||
results,
|
||||
meta: {
|
||||
budget: safeBudget,
|
||||
used: results.reduce((acc, r) => acc + resultTokens(r), 0),
|
||||
dropped: 0,
|
||||
kept: results.length,
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
const kept: SearchResult[] = [];
|
||||
let used = 0;
|
||||
for (const r of results) {
|
||||
const cost = resultTokens(r);
|
||||
if (used + cost > safeBudget) break;
|
||||
kept.push(r);
|
||||
used += cost;
|
||||
}
|
||||
|
||||
return {
|
||||
results: kept,
|
||||
meta: {
|
||||
budget: safeBudget,
|
||||
used,
|
||||
dropped: results.length - kept.length,
|
||||
kept: kept.length,
|
||||
},
|
||||
};
|
||||
}
|
||||
@@ -501,6 +501,32 @@ export interface SearchOpts {
|
||||
* Boundary semantics: end-of-day for plain YYYY-MM-DD.
|
||||
*/
|
||||
until?: string;
|
||||
/**
|
||||
* v0.32.x (search-lite): cap the cumulative token cost of returned results.
|
||||
* Applied AFTER all scoring, ranking, dedup, and boosts — the budget is the
|
||||
* LAST stage of the pipeline. Token counting uses a char/4 heuristic (no
|
||||
* tokenizer dep). When undefined or <= 0, this is a no-op (pre-v0.32
|
||||
* behavior).
|
||||
*
|
||||
* Use cases: keep an agent's search payload under its context window;
|
||||
* cap an MCP tool response to fit a router budget; emit a deterministic
|
||||
* upper bound on result size.
|
||||
*/
|
||||
tokenBudget?: number;
|
||||
/**
|
||||
* v0.32.x (search-lite): enable/disable the semantic query cache for this
|
||||
* call. When undefined, the cache decision falls back to global config
|
||||
* (search.cache.enabled, default true). Set to `false` to force a fresh
|
||||
* search; set to `true` to opt in even when global config has it off.
|
||||
*/
|
||||
useCache?: boolean;
|
||||
/**
|
||||
* v0.32.x (search-lite): force enable/disable the zero-LLM intent
|
||||
* classifier weight adjustments. Defaults to enabled. Set to `false` to
|
||||
* pin the legacy (pre-search-lite) weighting — useful when callers want
|
||||
* deterministic behavior independent of query phrasing.
|
||||
*/
|
||||
intentWeighting?: boolean;
|
||||
}
|
||||
|
||||
/**
|
||||
@@ -803,6 +829,45 @@ export interface HybridSearchMeta {
|
||||
detail_resolved: 'low' | 'medium' | 'high' | null;
|
||||
/** True iff multi-query expansion (Haiku) actually fired and produced variants. */
|
||||
expansion_applied: boolean;
|
||||
/**
|
||||
* v0.32.x (search-lite): the intent the zero-LLM classifier inferred for
|
||||
* this query. Surfaced for debugging — agents and the `gbrain query`
|
||||
* command can show "intent: temporal" alongside results to make the
|
||||
* weighting decision auditable.
|
||||
*/
|
||||
intent?: 'entity' | 'temporal' | 'event' | 'general';
|
||||
/**
|
||||
* v0.32.x (search-lite): token budget enforcement metadata. Omitted when
|
||||
* no budget was applied (backward-compatible with pre-search-lite
|
||||
* consumers).
|
||||
*/
|
||||
token_budget?: {
|
||||
budget: number;
|
||||
used: number;
|
||||
kept: number;
|
||||
dropped: number;
|
||||
};
|
||||
/**
|
||||
* v0.32.x (search-lite): cache hit/miss tracking. Omitted when the
|
||||
* semantic query cache wasn't consulted (cache disabled, vector search
|
||||
* unavailable, etc.).
|
||||
*/
|
||||
cache?: {
|
||||
/** 'hit' when results came from the cache; 'miss' when search ran fresh. */
|
||||
status: 'hit' | 'miss' | 'disabled';
|
||||
/** Similarity of the cached query's embedding (0..1). Only set on hit. */
|
||||
similarity?: number;
|
||||
/** Age of the cached entry in seconds. Only set on hit. */
|
||||
age_seconds?: number;
|
||||
};
|
||||
/**
|
||||
* v0.32.3 (search-lite mode): the active search mode for this call.
|
||||
* 'conservative' | 'balanced' | 'tokenmax'. Resolved from
|
||||
* config.search.mode with per-call + per-key overrides applied. Surfaced
|
||||
* so observability sees what mode actually ran (which can differ from
|
||||
* the operator's `config.search.mode` setting if per-call overrides win).
|
||||
*/
|
||||
mode?: 'conservative' | 'balanced' | 'tokenmax';
|
||||
}
|
||||
|
||||
// Config
|
||||
|
||||
+216
-4
@@ -3,12 +3,18 @@ import {
|
||||
configureGateway,
|
||||
resetGateway,
|
||||
isAvailable,
|
||||
embed,
|
||||
getEmbeddingModel,
|
||||
getEmbeddingDimensions,
|
||||
getExpansionModel,
|
||||
VoyageResponseTooLargeError,
|
||||
} from '../../src/core/ai/gateway.ts';
|
||||
import { parseModelId, resolveRecipe } from '../../src/core/ai/model-resolver.ts';
|
||||
import { dimsProviderOptions } from '../../src/core/ai/dims.ts';
|
||||
import {
|
||||
dimsProviderOptions,
|
||||
VOYAGE_VALID_OUTPUT_DIMS,
|
||||
isValidVoyageOutputDim,
|
||||
} from '../../src/core/ai/dims.ts';
|
||||
import { AIConfigError } from '../../src/core/ai/errors.ts';
|
||||
|
||||
describe('gateway configuration', () => {
|
||||
@@ -156,13 +162,219 @@ describe('dims.dimsProviderOptions', () => {
|
||||
expect(opts).toBeUndefined();
|
||||
});
|
||||
|
||||
test('Voyage openai-compatible returns output_dimension', () => {
|
||||
const opts = dimsProviderOptions('openai-compatible', 'voyage-4-large', 2048);
|
||||
expect(opts).toEqual({ openaiCompatible: { output_dimension: 2048 } });
|
||||
test('Voyage flexible-dim models return dimensions for the SDK shim', () => {
|
||||
const opts = dimsProviderOptions('openai-compatible', 'voyage-3-large', 1024);
|
||||
expect(opts).toEqual({ openaiCompatible: { dimensions: 1024 } });
|
||||
const v4Opts = dimsProviderOptions('openai-compatible', 'voyage-4-large', 2048);
|
||||
expect(v4Opts).toEqual({ openaiCompatible: { dimensions: 2048 } });
|
||||
});
|
||||
|
||||
test('Voyage model without flexible dimensions returns undefined', () => {
|
||||
const opts = dimsProviderOptions('openai-compatible', 'voyage-3-lite', 1024);
|
||||
expect(opts).toBeUndefined();
|
||||
});
|
||||
|
||||
// Negative regression pin: voyage-4-nano is an open-weight variant that
|
||||
// Voyage's hosted API rejects `output_dimension` on (fixed 1024-dim).
|
||||
// Don't re-add it to VOYAGE_OUTPUT_DIMENSION_MODELS without cross-checking
|
||||
// Voyage's docs. See src/core/ai/dims.ts for the rationale.
|
||||
test('voyage-4-nano returns undefined (open-weight, fixed-dim)', () => {
|
||||
const opts = dimsProviderOptions('openai-compatible', 'voyage-4-nano', 512);
|
||||
expect(opts).toBeUndefined();
|
||||
});
|
||||
});
|
||||
|
||||
describe('Voyage openai-compatible request shim', () => {
|
||||
beforeEach(() => resetGateway());
|
||||
|
||||
test('sends output_dimension on the actual Voyage embedding request body', async () => {
|
||||
const originalFetch = globalThis.fetch;
|
||||
let requestBody: Record<string, unknown> | undefined;
|
||||
globalThis.fetch = (async (_url: string | URL | Request, init?: RequestInit) => {
|
||||
requestBody = JSON.parse(String(init?.body ?? '{}'));
|
||||
return new Response(JSON.stringify({
|
||||
object: 'list',
|
||||
data: [
|
||||
{
|
||||
object: 'embedding',
|
||||
index: 0,
|
||||
embedding: new Array(2048).fill(0.01),
|
||||
},
|
||||
],
|
||||
model: 'voyage-4-large',
|
||||
usage: { total_tokens: 3 },
|
||||
}), {
|
||||
status: 200,
|
||||
headers: { 'content-type': 'application/json' },
|
||||
});
|
||||
}) as unknown as typeof fetch;
|
||||
|
||||
try {
|
||||
configureGateway({
|
||||
embedding_model: 'voyage:voyage-4-large',
|
||||
embedding_dimensions: 2048,
|
||||
env: { VOYAGE_API_KEY: 'voyage-fake' },
|
||||
});
|
||||
|
||||
const vectors = await embed(['dimension probe']);
|
||||
|
||||
expect(vectors[0].length).toBe(2048);
|
||||
expect(requestBody?.output_dimension).toBe(2048);
|
||||
expect(requestBody?.encoding_format).toBe('base64');
|
||||
} finally {
|
||||
globalThis.fetch = originalFetch;
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
// ─────────────────────────────────────────────────────────────────────
|
||||
// Voyage OOM-cap rethrow regression (Codex P3 follow-up after PR #962).
|
||||
// Pins the contract that VoyageResponseTooLargeError thrown from the
|
||||
// inbound rewriter is NOT swallowed by the surrounding try/catch.
|
||||
// ─────────────────────────────────────────────────────────────────────
|
||||
describe('Voyage OOM-cap: too-large response throws (Codex P3 follow-up)', () => {
|
||||
beforeEach(() => resetGateway());
|
||||
|
||||
test('Layer 1 — Content-Length above cap propagates as VoyageResponseTooLargeError', async () => {
|
||||
const originalFetch = globalThis.fetch;
|
||||
// 257 MB > 256 MB cap.
|
||||
const oversized = String(257 * 1024 * 1024);
|
||||
globalThis.fetch = (async () => {
|
||||
return new Response('{"data": []}', {
|
||||
status: 200,
|
||||
headers: {
|
||||
'content-type': 'application/json',
|
||||
'content-length': oversized,
|
||||
},
|
||||
});
|
||||
}) as unknown as typeof fetch;
|
||||
try {
|
||||
configureGateway({
|
||||
embedding_model: 'voyage:voyage-4-large',
|
||||
embedding_dimensions: 1024,
|
||||
env: { VOYAGE_API_KEY: 'voyage-fake' },
|
||||
});
|
||||
let caught: unknown;
|
||||
try {
|
||||
await embed(['probe']);
|
||||
} catch (e) {
|
||||
caught = e;
|
||||
}
|
||||
// The OOM throw propagates. Provider plumbing may wrap it, but the
|
||||
// VoyageResponseTooLargeError class name + characteristic message
|
||||
// must survive.
|
||||
const msg = caught instanceof Error ? caught.message : String(caught);
|
||||
expect(msg).toContain('Content-Length=');
|
||||
expect(msg).toContain('exceeds');
|
||||
} finally {
|
||||
globalThis.fetch = originalFetch;
|
||||
}
|
||||
});
|
||||
|
||||
test('Layer 2 — oversized base64 embedding string propagates (not swallowed)', async () => {
|
||||
const originalFetch = globalThis.fetch;
|
||||
// Build a JSON response with an `embedding` base64 string that decodes
|
||||
// to > 256 MB. base64 ratio is ~0.75; 360 MB of base64 chars ≈ 270 MB
|
||||
// decoded.
|
||||
const oversizedBase64 = 'A'.repeat(360 * 1024 * 1024);
|
||||
const respBody = `{"object":"list","data":[{"object":"embedding","index":0,"embedding":"${oversizedBase64}"}],"model":"voyage-4-large","usage":{"total_tokens":1}}`;
|
||||
globalThis.fetch = (async () => {
|
||||
// No Content-Length header → Layer 1 skipped, Layer 2 must fire.
|
||||
return new Response(respBody, {
|
||||
status: 200,
|
||||
headers: { 'content-type': 'application/json' },
|
||||
});
|
||||
}) as unknown as typeof fetch;
|
||||
try {
|
||||
configureGateway({
|
||||
embedding_model: 'voyage:voyage-4-large',
|
||||
embedding_dimensions: 1024,
|
||||
env: { VOYAGE_API_KEY: 'voyage-fake' },
|
||||
});
|
||||
let caught: unknown;
|
||||
try {
|
||||
await embed(['probe']);
|
||||
} catch (e) {
|
||||
caught = e;
|
||||
}
|
||||
const msg = caught instanceof Error ? caught.message : String(caught);
|
||||
// The Layer 2 throw fired and was not swallowed by the inbound
|
||||
// try/catch (pre-fix bug: bare `catch {}` returned the original
|
||||
// response and let the AI SDK OOM trying to parse it).
|
||||
expect(msg).toContain('Voyage embedding base64 exceeds');
|
||||
} finally {
|
||||
globalThis.fetch = originalFetch;
|
||||
}
|
||||
}, 15000);
|
||||
|
||||
test('VoyageResponseTooLargeError is exported as a tagged class', () => {
|
||||
expect(VoyageResponseTooLargeError).toBeDefined();
|
||||
const err = new VoyageResponseTooLargeError('test');
|
||||
expect(err).toBeInstanceOf(Error);
|
||||
expect(err).toBeInstanceOf(VoyageResponseTooLargeError);
|
||||
expect(err.name).toBe('VoyageResponseTooLargeError');
|
||||
});
|
||||
});
|
||||
|
||||
// ─────────────────────────────────────────────────────────────────────
|
||||
// Voyage flexible-dim runtime validation (Codex P3 follow-up after PR #962).
|
||||
// The bug class: brain configured for Voyage flexible-dim model without
|
||||
// `embedding_dimensions` → gateway falls back to DEFAULT 1536 → Voyage
|
||||
// HTTP 400. Catch it at the embed-call boundary with a clear AIConfigError.
|
||||
// ─────────────────────────────────────────────────────────────────────
|
||||
describe('Voyage flexible-dim runtime validation', () => {
|
||||
test('rejects 1536 (the default that bites Voyage-first users) with AIConfigError', () => {
|
||||
expect(() => dimsProviderOptions('openai-compatible', 'voyage-4-large', 1536))
|
||||
.toThrow(AIConfigError);
|
||||
expect(() => dimsProviderOptions('openai-compatible', 'voyage-4-large', 1536))
|
||||
.toThrow(/embedding_dimensions|256.*512.*1024.*2048/);
|
||||
});
|
||||
|
||||
test('rejects 3072 with AIConfigError', () => {
|
||||
expect(() => dimsProviderOptions('openai-compatible', 'voyage-3-large', 3072))
|
||||
.toThrow(AIConfigError);
|
||||
});
|
||||
|
||||
test('accepts every Voyage-allowed flexible dim', () => {
|
||||
for (const dim of VOYAGE_VALID_OUTPUT_DIMS) {
|
||||
const opts = dimsProviderOptions('openai-compatible', 'voyage-4-large', dim);
|
||||
expect(opts).toEqual({ openaiCompatible: { dimensions: dim } });
|
||||
}
|
||||
});
|
||||
|
||||
test('VOYAGE_VALID_OUTPUT_DIMS pins exactly the four Voyage values', () => {
|
||||
expect([...VOYAGE_VALID_OUTPUT_DIMS]).toEqual([256, 512, 1024, 2048]);
|
||||
});
|
||||
|
||||
test('isValidVoyageOutputDim returns true only for the four valid sizes', () => {
|
||||
expect(isValidVoyageOutputDim(256)).toBe(true);
|
||||
expect(isValidVoyageOutputDim(512)).toBe(true);
|
||||
expect(isValidVoyageOutputDim(1024)).toBe(true);
|
||||
expect(isValidVoyageOutputDim(2048)).toBe(true);
|
||||
expect(isValidVoyageOutputDim(1536)).toBe(false);
|
||||
expect(isValidVoyageOutputDim(3072)).toBe(false);
|
||||
expect(isValidVoyageOutputDim(0)).toBe(false);
|
||||
expect(isValidVoyageOutputDim(-1)).toBe(false);
|
||||
});
|
||||
|
||||
test('voyage-3-lite (non-flexible-dim) bypasses the validator — still returns undefined', () => {
|
||||
// Sanity: the validator only fires inside the flexible-dim branch, so
|
||||
// a fixed-dim Voyage model with any dim value goes straight through to
|
||||
// the `undefined` return path (no error, no providerOptions).
|
||||
expect(dimsProviderOptions('openai-compatible', 'voyage-3-lite', 1536)).toBeUndefined();
|
||||
expect(dimsProviderOptions('openai-compatible', 'voyage-4-nano', 1536)).toBeUndefined();
|
||||
});
|
||||
|
||||
test('AIConfigError fix hint names the canonical recovery commands', () => {
|
||||
let caught: AIConfigError | undefined;
|
||||
try {
|
||||
dimsProviderOptions('openai-compatible', 'voyage-4-large', 1536);
|
||||
} catch (e) {
|
||||
caught = e as AIConfigError;
|
||||
}
|
||||
expect(caught).toBeInstanceOf(AIConfigError);
|
||||
expect(caught?.fix).toContain('embedding_dimensions');
|
||||
expect(caught?.fix).toContain('256');
|
||||
expect(caught?.fix).toContain('2048');
|
||||
});
|
||||
});
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
* v0.34 W3b — code_traversal_cache module tests.
|
||||
*
|
||||
* Hermetic PGLite test suite covering:
|
||||
* - cache hit returns memoized response (after migration v56)
|
||||
* - cache hit returns memoized response (after migration v59)
|
||||
* - cache miss triggers compute
|
||||
* - D3: cluster_generation bump invalidates cached rows
|
||||
* - clearTraversalCache: source-scoped clear deletes the right rows
|
||||
|
||||
@@ -0,0 +1,257 @@
|
||||
/**
|
||||
* v0.32.3 — `gbrain search modes/stats/tune` CLI tests.
|
||||
*
|
||||
* Covers dispatch + JSON output shape + idempotent --reset + recommendation
|
||||
* generation. Pure unit-level: bypasses the cli.ts entrypoint and calls
|
||||
* runSearch directly against a fresh PGLite engine.
|
||||
*/
|
||||
import { afterAll, beforeAll, beforeEach, describe, expect, test } from 'bun:test';
|
||||
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
|
||||
import { runSearch } from '../src/commands/search.ts';
|
||||
import { recordSearchTelemetry, _resetTelemetryWriterForTest, getTelemetryWriter } from '../src/core/search/telemetry.ts';
|
||||
import type { HybridSearchMeta } from '../src/core/types.ts';
|
||||
|
||||
let engine: PGLiteEngine;
|
||||
|
||||
beforeAll(async () => {
|
||||
engine = new PGLiteEngine();
|
||||
await engine.connect({});
|
||||
await engine.initSchema();
|
||||
});
|
||||
|
||||
afterAll(async () => {
|
||||
await engine.disconnect();
|
||||
});
|
||||
|
||||
beforeEach(async () => {
|
||||
_resetTelemetryWriterForTest();
|
||||
await engine.executeRaw(`DELETE FROM config WHERE key LIKE 'search.%' OR key LIKE 'models.%'`);
|
||||
await engine.executeRaw('DELETE FROM search_telemetry');
|
||||
});
|
||||
|
||||
// Capture-stdout helper so we can assert command output without exec'ing.
|
||||
async function captureRun(fn: () => Promise<void>): Promise<string> {
|
||||
const original = console.log;
|
||||
const captured: string[] = [];
|
||||
console.log = (...args: unknown[]) => { captured.push(args.map(String).join(' ')); };
|
||||
try {
|
||||
await fn();
|
||||
} finally {
|
||||
console.log = original;
|
||||
}
|
||||
return captured.join('\n');
|
||||
}
|
||||
|
||||
const makeMeta = (overrides: Partial<HybridSearchMeta> = {}): HybridSearchMeta => ({
|
||||
vector_enabled: true,
|
||||
detail_resolved: null,
|
||||
expansion_applied: false,
|
||||
intent: 'general',
|
||||
mode: 'balanced',
|
||||
...overrides,
|
||||
});
|
||||
|
||||
describe('gbrain search modes (read-only dashboard)', () => {
|
||||
test('--json emits structured report with all 3 bundles and active mode', async () => {
|
||||
await engine.setConfig('search.mode', 'tokenmax');
|
||||
const out = await captureRun(() => runSearch(engine, ['modes', '--json']));
|
||||
const report = JSON.parse(out);
|
||||
expect(report.schema_version).toBe(2);
|
||||
expect(report.active_mode).toBe('tokenmax');
|
||||
expect(report.active_mode_valid).toBe(true);
|
||||
expect(report.bundles.conservative.searchLimit).toBe(10);
|
||||
expect(report.bundles.balanced.searchLimit).toBe(25);
|
||||
expect(report.bundles.tokenmax.searchLimit).toBe(50);
|
||||
expect(report.resolved.tokenBudget.source).toBe('mode');
|
||||
});
|
||||
|
||||
test('unset mode → balanced fallback with mode_valid=false', async () => {
|
||||
const out = await captureRun(() => runSearch(engine, ['modes', '--json']));
|
||||
const report = JSON.parse(out);
|
||||
expect(report.active_mode).toBe('balanced');
|
||||
expect(report.active_mode_valid).toBe(false);
|
||||
expect(report.resolved.searchLimit.source).toBe('fallback');
|
||||
});
|
||||
|
||||
test('per-key override shows up with source=override', async () => {
|
||||
await engine.setConfig('search.mode', 'conservative');
|
||||
await engine.setConfig('search.cache.enabled', 'false');
|
||||
const out = await captureRun(() => runSearch(engine, ['modes', '--json']));
|
||||
const report = JSON.parse(out);
|
||||
expect(report.resolved.cache_enabled.value).toBe(false);
|
||||
expect(report.resolved.cache_enabled.source).toBe('override');
|
||||
// Other knobs still come from the mode bundle.
|
||||
expect(report.resolved.searchLimit.source).toBe('mode');
|
||||
});
|
||||
|
||||
test('default text output names the active mode', async () => {
|
||||
await engine.setConfig('search.mode', 'tokenmax');
|
||||
const out = await captureRun(() => runSearch(engine, ['modes']));
|
||||
expect(out).toContain('tokenmax');
|
||||
expect(out).toContain('conservative');
|
||||
expect(out).toContain('balanced');
|
||||
});
|
||||
});
|
||||
|
||||
describe('gbrain search modes --reset', () => {
|
||||
test('--source <mode> is a dry-run (no writes)', async () => {
|
||||
await engine.setConfig('search.cache.enabled', 'false');
|
||||
await engine.setConfig('search.tokenBudget', '4000');
|
||||
const out = await captureRun(() => runSearch(engine, ['modes', '--source', 'balanced']));
|
||||
expect(out).toContain('dry run');
|
||||
expect(out).toContain('search.cache.enabled');
|
||||
expect(out).toContain('search.tokenBudget');
|
||||
// Verify nothing was deleted.
|
||||
expect(await engine.getConfig('search.cache.enabled')).toBe('false');
|
||||
expect(await engine.getConfig('search.tokenBudget')).toBe('4000');
|
||||
});
|
||||
|
||||
test('--reset clears every search.* override (but NOT search.mode itself)', async () => {
|
||||
await engine.setConfig('search.mode', 'conservative');
|
||||
await engine.setConfig('search.cache.enabled', 'false');
|
||||
await engine.setConfig('search.tokenBudget', '8000');
|
||||
await engine.setConfig('search.searchLimit', '15');
|
||||
await captureRun(() => runSearch(engine, ['modes', '--reset']));
|
||||
// Mode preserved; overrides gone.
|
||||
expect(await engine.getConfig('search.mode')).toBe('conservative');
|
||||
expect(await engine.getConfig('search.cache.enabled')).toBeNull();
|
||||
expect(await engine.getConfig('search.tokenBudget')).toBeNull();
|
||||
expect(await engine.getConfig('search.searchLimit')).toBeNull();
|
||||
});
|
||||
|
||||
test('--reset on a clean install reports "no overrides"', async () => {
|
||||
const out = await captureRun(() => runSearch(engine, ['modes', '--reset']));
|
||||
expect(out).toContain('No search.* overrides set');
|
||||
});
|
||||
|
||||
test('--reset preserves the upgrade-notice state key', async () => {
|
||||
await engine.setConfig('search.mode_upgrade_notice_shown', 'true');
|
||||
await engine.setConfig('search.tokenBudget', '4000');
|
||||
await captureRun(() => runSearch(engine, ['modes', '--reset']));
|
||||
// Notice key preserved (it's not an "override"); tokenBudget gone.
|
||||
expect(await engine.getConfig('search.mode_upgrade_notice_shown')).toBe('true');
|
||||
expect(await engine.getConfig('search.tokenBudget')).toBeNull();
|
||||
});
|
||||
});
|
||||
|
||||
describe('gbrain search stats', () => {
|
||||
test('empty table → total_calls 0, message about no data', async () => {
|
||||
const out = await captureRun(() => runSearch(engine, ['stats']));
|
||||
expect(out).toContain('Total searches:');
|
||||
expect(out).toContain('0');
|
||||
});
|
||||
|
||||
test('after telemetry writes → hit rate + intent mix surfaced', async () => {
|
||||
const w = getTelemetryWriter();
|
||||
w.setEngine(engine);
|
||||
recordSearchTelemetry(engine, makeMeta({ cache: { status: 'hit' } }), { results_count: 5 });
|
||||
recordSearchTelemetry(engine, makeMeta({ cache: { status: 'hit' } }), { results_count: 7 });
|
||||
recordSearchTelemetry(engine, makeMeta({ cache: { status: 'miss' } }), { results_count: 9 });
|
||||
recordSearchTelemetry(engine, makeMeta({ intent: 'entity' }), { results_count: 3 });
|
||||
await w.flush();
|
||||
|
||||
const out = await captureRun(() => runSearch(engine, ['stats', '--json']));
|
||||
const stats = JSON.parse(out);
|
||||
expect(stats.total_calls).toBe(4);
|
||||
expect(stats.cache_hits).toBe(2);
|
||||
expect(stats.cache_misses).toBe(1);
|
||||
expect(stats.cache_hit_rate).toBeCloseTo(2 / 3, 3);
|
||||
expect(stats._meta.metric_glossary.cache_hit_rate).toBeDefined();
|
||||
});
|
||||
|
||||
test('--days N clamps to [1, 365]', async () => {
|
||||
const out0 = await captureRun(() => runSearch(engine, ['stats', '--days', '0', '--json']));
|
||||
expect(JSON.parse(out0).window_days).toBe(1);
|
||||
const outBig = await captureRun(() => runSearch(engine, ['stats', '--days', '9999', '--json']));
|
||||
expect(JSON.parse(outBig).window_days).toBe(365);
|
||||
});
|
||||
});
|
||||
|
||||
describe('gbrain search tune (recommendations)', () => {
|
||||
test('insufficient data → no_recommendations status', async () => {
|
||||
const out = await captureRun(() => runSearch(engine, ['tune', '--json']));
|
||||
const r = JSON.parse(out);
|
||||
expect(r.status).toBe('insufficient_data');
|
||||
expect(r.recommendations).toEqual([]);
|
||||
});
|
||||
|
||||
test('conservative + high budget drop rate → recommends balanced', async () => {
|
||||
await engine.setConfig('search.mode', 'conservative');
|
||||
const w = getTelemetryWriter();
|
||||
w.setEngine(engine);
|
||||
// 30 calls, each dropping 5 results — strong signal.
|
||||
for (let i = 0; i < 30; i++) {
|
||||
recordSearchTelemetry(engine, makeMeta({
|
||||
mode: 'conservative',
|
||||
token_budget: { budget: 4000, used: 4000, kept: 5, dropped: 5 },
|
||||
}), { results_count: 5 });
|
||||
}
|
||||
await w.flush();
|
||||
|
||||
const out = await captureRun(() => runSearch(engine, ['tune', '--json']));
|
||||
const r = JSON.parse(out);
|
||||
expect(r.status).toBe('has_recommendations');
|
||||
const modeRec = r.recommendations.find((x: { knob: string }) => x.knob === 'search.mode');
|
||||
expect(modeRec).toBeDefined();
|
||||
expect(modeRec.suggested).toBe('balanced');
|
||||
});
|
||||
|
||||
test('tokenmax + Haiku subagent → recommends balanced', async () => {
|
||||
await engine.setConfig('search.mode', 'tokenmax');
|
||||
await engine.setConfig('models.tier.subagent', 'anthropic:claude-haiku-4-5');
|
||||
const w = getTelemetryWriter();
|
||||
w.setEngine(engine);
|
||||
for (let i = 0; i < 25; i++) {
|
||||
recordSearchTelemetry(engine, makeMeta({ mode: 'tokenmax' }), { results_count: 30 });
|
||||
}
|
||||
await w.flush();
|
||||
|
||||
const out = await captureRun(() => runSearch(engine, ['tune', '--json']));
|
||||
const r = JSON.parse(out);
|
||||
const rec = r.recommendations.find((x: { knob: string; suggested: string }) =>
|
||||
x.knob === 'search.mode' && x.suggested === 'balanced'
|
||||
);
|
||||
expect(rec).toBeDefined();
|
||||
expect(rec.reason).toMatch(/Haiku/);
|
||||
});
|
||||
|
||||
test('--apply mutates config', async () => {
|
||||
await engine.setConfig('search.mode', 'conservative');
|
||||
const w = getTelemetryWriter();
|
||||
w.setEngine(engine);
|
||||
for (let i = 0; i < 30; i++) {
|
||||
recordSearchTelemetry(engine, makeMeta({
|
||||
mode: 'conservative',
|
||||
token_budget: { budget: 4000, used: 4000, kept: 5, dropped: 5 },
|
||||
}), { results_count: 5 });
|
||||
}
|
||||
await w.flush();
|
||||
|
||||
await captureRun(() => runSearch(engine, ['tune', '--apply']));
|
||||
expect(await engine.getConfig('search.mode')).toBe('balanced');
|
||||
});
|
||||
});
|
||||
|
||||
describe('gbrain search dispatch', () => {
|
||||
test('--help shows usage', async () => {
|
||||
const out = await captureRun(() => runSearch(engine, ['--help']));
|
||||
expect(out).toContain('Usage:');
|
||||
expect(out).toContain('modes');
|
||||
expect(out).toContain('stats');
|
||||
expect(out).toContain('tune');
|
||||
});
|
||||
|
||||
test('unknown subcommand exits 1', async () => {
|
||||
let exitCode = 0;
|
||||
const originalExit = process.exit;
|
||||
(process.exit as unknown as (code?: number) => void) = ((code?: number) => { exitCode = code ?? 0; throw new Error('exit-' + code); }) as never;
|
||||
const originalErr = console.error;
|
||||
console.error = () => { /* swallow */ };
|
||||
try {
|
||||
await runSearch(engine, ['nonsense']);
|
||||
} catch { /* expected */ }
|
||||
expect(exitCode).toBe(1);
|
||||
process.exit = originalExit;
|
||||
console.error = originalErr;
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,101 @@
|
||||
/**
|
||||
* Pins the v0.32.3 `gbrain config unset` + `listConfigKeys` engine surface.
|
||||
* Required by `gbrain search modes --reset` [CDX-8].
|
||||
*/
|
||||
import { afterAll, beforeAll, beforeEach, describe, expect, test } from 'bun:test';
|
||||
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
|
||||
|
||||
let engine: PGLiteEngine;
|
||||
|
||||
beforeAll(async () => {
|
||||
engine = new PGLiteEngine();
|
||||
await engine.connect({});
|
||||
await engine.initSchema();
|
||||
});
|
||||
|
||||
afterAll(async () => {
|
||||
await engine.disconnect();
|
||||
});
|
||||
|
||||
beforeEach(async () => {
|
||||
await engine.executeRaw(`DELETE FROM config WHERE key LIKE 'test.%' OR key LIKE 'search.%'`);
|
||||
});
|
||||
|
||||
describe('engine.unsetConfig', () => {
|
||||
test('removes an existing key, returns 1', async () => {
|
||||
await engine.setConfig('test.k1', 'v1');
|
||||
const n = await engine.unsetConfig('test.k1');
|
||||
expect(n).toBe(1);
|
||||
const after = await engine.getConfig('test.k1');
|
||||
expect(after).toBeNull();
|
||||
});
|
||||
|
||||
test('returns 0 for a missing key (no error)', async () => {
|
||||
const n = await engine.unsetConfig('test.never-existed');
|
||||
expect(n).toBe(0);
|
||||
});
|
||||
|
||||
test('does not affect other keys', async () => {
|
||||
await engine.setConfig('test.keep', 'keep');
|
||||
await engine.setConfig('test.remove', 'gone');
|
||||
await engine.unsetConfig('test.remove');
|
||||
expect(await engine.getConfig('test.keep')).toBe('keep');
|
||||
expect(await engine.getConfig('test.remove')).toBeNull();
|
||||
});
|
||||
});
|
||||
|
||||
describe('engine.listConfigKeys prefix-matcher', () => {
|
||||
test('empty when no key matches', async () => {
|
||||
expect(await engine.listConfigKeys('test.')).toEqual([]);
|
||||
});
|
||||
|
||||
test('returns matching keys sorted ascending', async () => {
|
||||
await engine.setConfig('test.b', '2');
|
||||
await engine.setConfig('test.a', '1');
|
||||
await engine.setConfig('test.c', '3');
|
||||
await engine.setConfig('other', 'no');
|
||||
const keys = await engine.listConfigKeys('test.');
|
||||
expect(keys).toEqual(['test.a', 'test.b', 'test.c']);
|
||||
});
|
||||
|
||||
test('prefix is an EXACT literal match — no glob wildcards', async () => {
|
||||
await engine.setConfig('test.matchme', 'yes');
|
||||
// % in user input is escaped — does NOT act as wildcard.
|
||||
expect(await engine.listConfigKeys('test%')).toEqual([]);
|
||||
expect(await engine.listConfigKeys('test.match')).toEqual(['test.matchme']);
|
||||
});
|
||||
|
||||
test('underscore in prefix is escaped (literal _)', async () => {
|
||||
await engine.setConfig('test_underscore', 'val');
|
||||
await engine.setConfig('testXunderscore', 'no');
|
||||
const keys = await engine.listConfigKeys('test_');
|
||||
// test_underscore matches because we asked for prefix "test_"; the
|
||||
// testXunderscore does NOT because the _ is escaped to literal.
|
||||
expect(keys).toEqual(['test_underscore']);
|
||||
});
|
||||
|
||||
test('search.* prefix sweep returns every search-mode override key set', async () => {
|
||||
await engine.setConfig('search.cache.enabled', 'true');
|
||||
await engine.setConfig('search.cache.ttl_seconds', '7200');
|
||||
await engine.setConfig('search.tokenBudget', '8000');
|
||||
await engine.setConfig('search.mode', 'tokenmax'); // mode key itself
|
||||
const keys = await engine.listConfigKeys('search.');
|
||||
expect(keys.length).toBe(4);
|
||||
expect(keys).toContain('search.cache.enabled');
|
||||
expect(keys).toContain('search.cache.ttl_seconds');
|
||||
expect(keys).toContain('search.tokenBudget');
|
||||
expect(keys).toContain('search.mode');
|
||||
});
|
||||
});
|
||||
|
||||
describe('round-trip: set → unset → get', () => {
|
||||
test('setting and unsetting in a tight loop is idempotent', async () => {
|
||||
for (let i = 0; i < 5; i++) {
|
||||
await engine.setConfig('test.loopkey', `iteration-${i}`);
|
||||
expect(await engine.getConfig('test.loopkey')).toBe(`iteration-${i}`);
|
||||
const n = await engine.unsetConfig('test.loopkey');
|
||||
expect(n).toBe(1);
|
||||
expect(await engine.getConfig('test.loopkey')).toBeNull();
|
||||
}
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,84 @@
|
||||
/**
|
||||
* v0.32.3 — doctor search_mode + eval_drift check tests.
|
||||
* Pins [CDX-20]: status stays 'ok', no health-score docking; hint lives
|
||||
* in `message`. Tests the two exported helpers directly to avoid the
|
||||
* expensive full runDoctor walk.
|
||||
*/
|
||||
import { afterAll, beforeAll, beforeEach, describe, expect, test } from 'bun:test';
|
||||
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
|
||||
import { checkSearchMode, checkEvalDrift } from '../src/commands/doctor.ts';
|
||||
|
||||
let engine: PGLiteEngine;
|
||||
|
||||
beforeAll(async () => {
|
||||
engine = new PGLiteEngine();
|
||||
await engine.connect({});
|
||||
await engine.initSchema();
|
||||
});
|
||||
|
||||
afterAll(async () => {
|
||||
await engine.disconnect();
|
||||
});
|
||||
|
||||
beforeEach(async () => {
|
||||
await engine.executeRaw(`DELETE FROM config WHERE key LIKE 'search.%'`);
|
||||
});
|
||||
|
||||
describe('checkSearchMode [CDX-20]', () => {
|
||||
test('unset mode → ok with hint to pick a mode', async () => {
|
||||
const c = await checkSearchMode(engine);
|
||||
expect(c.name).toBe('search_mode');
|
||||
expect(c.status).toBe('ok'); // never warn, never dock score
|
||||
expect(c.message).toMatch(/unset/i);
|
||||
expect(c.message).toContain('gbrain search modes');
|
||||
});
|
||||
|
||||
test('mode set, no overrides → ok with "canonical" message', async () => {
|
||||
await engine.setConfig('search.mode', 'balanced');
|
||||
const c = await checkSearchMode(engine);
|
||||
expect(c.status).toBe('ok');
|
||||
expect(c.message).toContain('balanced');
|
||||
expect(c.message).toContain('canonical');
|
||||
});
|
||||
|
||||
test('mode set + overrides → ok with reset hint + override list', async () => {
|
||||
await engine.setConfig('search.mode', 'conservative');
|
||||
await engine.setConfig('search.cache.enabled', 'false');
|
||||
await engine.setConfig('search.tokenBudget', '8000');
|
||||
const c = await checkSearchMode(engine);
|
||||
expect(c.status).toBe('ok'); // [CDX-20]: still ok, never warn
|
||||
expect(c.message).toContain('conservative');
|
||||
expect(c.message).toContain('search.cache.enabled');
|
||||
expect(c.message).toContain('search.tokenBudget');
|
||||
expect(c.message).toContain('gbrain search modes --reset');
|
||||
});
|
||||
|
||||
test('upgrade-notice state key is excluded from override count', async () => {
|
||||
await engine.setConfig('search.mode', 'balanced');
|
||||
await engine.setConfig('search.mode_upgrade_notice_shown', 'true');
|
||||
const c = await checkSearchMode(engine);
|
||||
expect(c.message).toContain('no per-key overrides');
|
||||
});
|
||||
|
||||
test('tokenmax mode is recognized without any override warning', async () => {
|
||||
await engine.setConfig('search.mode', 'tokenmax');
|
||||
const c = await checkSearchMode(engine);
|
||||
expect(c.status).toBe('ok');
|
||||
expect(c.message).toContain('tokenmax');
|
||||
expect(c.message).toContain('canonical');
|
||||
});
|
||||
});
|
||||
|
||||
describe('checkEvalDrift [CDX-6]', () => {
|
||||
test('returns ok status (never warn — per [CDX-20])', async () => {
|
||||
const c = await checkEvalDrift(engine);
|
||||
expect(c.name).toBe('eval_drift');
|
||||
expect(c.status).toBe('ok');
|
||||
});
|
||||
|
||||
test('message is non-empty (either no-drift or drift summary)', async () => {
|
||||
const c = await checkEvalDrift(engine);
|
||||
expect(c.message).toBeTruthy();
|
||||
expect(c.message.length).toBeGreaterThan(0);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,82 @@
|
||||
/**
|
||||
* v0.32.3 — drift-watch module unit tests.
|
||||
* Pins the curated watch-list + matchesWatchPattern semantics.
|
||||
*/
|
||||
import { describe, expect, test } from 'bun:test';
|
||||
import {
|
||||
RETRIEVAL_WATCH_PATTERNS,
|
||||
matchesWatchPattern,
|
||||
watchedFilesDrifted,
|
||||
filesDriftedSince,
|
||||
} from '../src/core/eval/drift-watch.ts';
|
||||
|
||||
describe('RETRIEVAL_WATCH_PATTERNS canonical list', () => {
|
||||
test('includes src/core/search/ prefix', () => {
|
||||
expect(RETRIEVAL_WATCH_PATTERNS).toContain('src/core/search/');
|
||||
});
|
||||
|
||||
test('includes the embedding file', () => {
|
||||
expect(RETRIEVAL_WATCH_PATTERNS).toContain('src/core/embedding.ts');
|
||||
});
|
||||
|
||||
test('includes chunkers/ directory', () => {
|
||||
expect(RETRIEVAL_WATCH_PATTERNS).toContain('src/core/chunkers/');
|
||||
});
|
||||
|
||||
test('includes the query operation definition', () => {
|
||||
expect(RETRIEVAL_WATCH_PATTERNS).toContain('src/core/operations.ts');
|
||||
});
|
||||
|
||||
test('is frozen at module load', () => {
|
||||
expect(Object.isFrozen(RETRIEVAL_WATCH_PATTERNS)).toBe(true);
|
||||
});
|
||||
});
|
||||
|
||||
describe('matchesWatchPattern semantics', () => {
|
||||
test('directory pattern matches any descendant', () => {
|
||||
expect(matchesWatchPattern('src/core/search/hybrid.ts')).toBe(true);
|
||||
expect(matchesWatchPattern('src/core/search/mode.ts')).toBe(true);
|
||||
expect(matchesWatchPattern('src/core/search/deep/nested/file.ts')).toBe(true);
|
||||
});
|
||||
|
||||
test('directory pattern does NOT match a sibling with the same prefix', () => {
|
||||
// src/core/search-related-but-different/foo.ts should NOT match
|
||||
// src/core/search/ because the pattern ends with a slash.
|
||||
expect(matchesWatchPattern('src/core/searchengine.ts')).toBe(false);
|
||||
expect(matchesWatchPattern('src/core/searches/file.ts')).toBe(false);
|
||||
});
|
||||
|
||||
test('bare file pattern requires exact equality', () => {
|
||||
expect(matchesWatchPattern('src/core/embedding.ts')).toBe(true);
|
||||
expect(matchesWatchPattern('src/core/embedding.test.ts')).toBe(false);
|
||||
expect(matchesWatchPattern('src/core/embedding')).toBe(false);
|
||||
});
|
||||
|
||||
test('custom patterns work', () => {
|
||||
const custom = ['foo/', 'bar.ts'];
|
||||
expect(matchesWatchPattern('foo/x.ts', custom)).toBe(true);
|
||||
expect(matchesWatchPattern('bar.ts', custom)).toBe(true);
|
||||
expect(matchesWatchPattern('baz.ts', custom)).toBe(false);
|
||||
});
|
||||
|
||||
test('non-matching path returns false', () => {
|
||||
expect(matchesWatchPattern('docs/eval/METRIC_GLOSSARY.md')).toBe(false);
|
||||
expect(matchesWatchPattern('test/foo.test.ts')).toBe(false);
|
||||
expect(matchesWatchPattern('README.md')).toBe(false);
|
||||
});
|
||||
});
|
||||
|
||||
describe('filesDriftedSince + watchedFilesDrifted graceful failure', () => {
|
||||
test('missing repo root returns empty array', () => {
|
||||
expect(filesDriftedSince('/does/not/exist')).toEqual([]);
|
||||
});
|
||||
|
||||
test('watchedFilesDrifted filters through the same matcher', () => {
|
||||
// Smoke test: should not throw on this repo. Could be empty.
|
||||
const out = watchedFilesDrifted(process.cwd());
|
||||
expect(Array.isArray(out)).toBe(true);
|
||||
for (const p of out) {
|
||||
expect(matchesWatchPattern(p)).toBe(true);
|
||||
}
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,135 @@
|
||||
/**
|
||||
* v0.32.3 — eval-compare report tests.
|
||||
* Pins the markdown + JSON shape and the metric-glossary integration.
|
||||
*/
|
||||
import { afterAll, beforeAll, beforeEach, describe, expect, test } from 'bun:test';
|
||||
import { mkdtempSync, rmSync, writeFileSync } from 'fs';
|
||||
import { tmpdir } from 'os';
|
||||
import { join } from 'path';
|
||||
import { runEvalCompare } from '../src/commands/eval-compare.ts';
|
||||
|
||||
let tmp: string;
|
||||
|
||||
beforeAll(() => {
|
||||
tmp = mkdtempSync(join(tmpdir(), 'gbrain-eval-compare-'));
|
||||
});
|
||||
|
||||
afterAll(() => {
|
||||
rmSync(tmp, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
const SAMPLE_RECORDS = [
|
||||
{
|
||||
schema_version: 2,
|
||||
run_id: 'a-longmemeval-conservative-42',
|
||||
ran_at: '2026-05-12T10:00:00Z',
|
||||
suite: 'longmemeval',
|
||||
mode: 'conservative',
|
||||
commit: 'a',
|
||||
seed: 42,
|
||||
status: 'completed',
|
||||
duration_ms: 1000,
|
||||
metrics: { 'recall@10': 0.71, 'ndcg@10': 0.682 },
|
||||
},
|
||||
{
|
||||
schema_version: 2,
|
||||
run_id: 'a-longmemeval-balanced-42',
|
||||
ran_at: '2026-05-12T10:05:00Z',
|
||||
suite: 'longmemeval',
|
||||
mode: 'balanced',
|
||||
commit: 'a',
|
||||
seed: 42,
|
||||
status: 'completed',
|
||||
duration_ms: 1000,
|
||||
metrics: { 'recall@10': 0.78, 'ndcg@10': 0.741 },
|
||||
},
|
||||
{
|
||||
schema_version: 2,
|
||||
run_id: 'a-longmemeval-tokenmax-42',
|
||||
ran_at: '2026-05-12T10:10:00Z',
|
||||
suite: 'longmemeval',
|
||||
mode: 'tokenmax',
|
||||
commit: 'a',
|
||||
seed: 42,
|
||||
status: 'completed',
|
||||
duration_ms: 1000,
|
||||
metrics: { 'recall@10': 0.81, 'ndcg@10': 0.762 },
|
||||
},
|
||||
];
|
||||
|
||||
function writeJsonl(records: object[]): string {
|
||||
const path = join(tmp, 'eval-results.jsonl');
|
||||
writeFileSync(path, records.map(r => JSON.stringify(r)).join('\n') + '\n', 'utf-8');
|
||||
return path;
|
||||
}
|
||||
|
||||
async function captureRun(fn: () => Promise<void>): Promise<string> {
|
||||
const originalWrite = process.stdout.write.bind(process.stdout);
|
||||
const captured: string[] = [];
|
||||
(process.stdout.write as unknown as (s: string) => boolean) = ((s: string) => { captured.push(s); return true; }) as never;
|
||||
try {
|
||||
await fn();
|
||||
} finally {
|
||||
process.stdout.write = originalWrite;
|
||||
}
|
||||
return captured.join('');
|
||||
}
|
||||
|
||||
beforeEach(() => {
|
||||
rmSync(join(tmp, 'eval-results.jsonl'), { force: true });
|
||||
});
|
||||
|
||||
describe('runEvalCompare', () => {
|
||||
test('--json output has schema_version + grouped + _meta.metric_glossary', async () => {
|
||||
const path = writeJsonl(SAMPLE_RECORDS);
|
||||
const out = await captureRun(() => runEvalCompare(['--json', '--input', path]));
|
||||
const report = JSON.parse(out);
|
||||
expect(report.schema_version).toBe(2);
|
||||
expect(report.grouped.longmemeval).toBeDefined();
|
||||
expect(report.grouped.longmemeval.conservative.metrics['recall@10']).toBe(0.71);
|
||||
expect(report.grouped.longmemeval.tokenmax.metrics['ndcg@10']).toBe(0.762);
|
||||
expect(report._meta.metric_glossary['recall@10']).toBeDefined();
|
||||
expect(report._meta.metric_glossary['ndcg@10']).toBeDefined();
|
||||
expect(report._meta.methodology).toContain('bootstrap');
|
||||
});
|
||||
|
||||
test('--md output names every mode + metric', async () => {
|
||||
const path = writeJsonl(SAMPLE_RECORDS);
|
||||
const out = await captureRun(() => runEvalCompare(['--md', '--input', path]));
|
||||
expect(out).toContain('# Search Mode Comparison');
|
||||
expect(out).toContain('## longmemeval');
|
||||
expect(out).toContain('conservative');
|
||||
expect(out).toContain('balanced');
|
||||
expect(out).toContain('tokenmax');
|
||||
expect(out).toContain('### recall@10');
|
||||
expect(out).toContain('### ndcg@10');
|
||||
expect(out).toContain('Plain English:'); // Glossary line surfaced
|
||||
});
|
||||
|
||||
test('missing file → friendly hint, no crash', async () => {
|
||||
const path = join(tmp, 'does-not-exist.jsonl');
|
||||
const out = await captureRun(() => runEvalCompare(['--md', '--input', path]));
|
||||
expect(out).toContain('No eval-results.jsonl found');
|
||||
expect(out).toContain('gbrain eval run-all');
|
||||
});
|
||||
|
||||
test('--modes filter narrows the table', async () => {
|
||||
const path = writeJsonl(SAMPLE_RECORDS);
|
||||
const out = await captureRun(() => runEvalCompare(['--json', '--modes', 'conservative,tokenmax', '--input', path]));
|
||||
const report = JSON.parse(out);
|
||||
// Records list is filtered.
|
||||
expect(report.records.length).toBe(2);
|
||||
// The grouped table preserves the SEARCH_MODES order (conservative/balanced/tokenmax).
|
||||
// Filtered balanced → null in grouped output.
|
||||
expect(report.grouped.longmemeval.balanced).toBeNull();
|
||||
});
|
||||
|
||||
test('multiple runs for same (suite, mode) → most-recent wins', async () => {
|
||||
const oldRun = { ...SAMPLE_RECORDS[0], ran_at: '2026-05-12T09:00:00Z', metrics: { 'recall@10': 0.5 } };
|
||||
const newRun = { ...SAMPLE_RECORDS[0], ran_at: '2026-05-12T11:00:00Z', metrics: { 'recall@10': 0.9 } };
|
||||
const path = writeJsonl([oldRun, newRun]);
|
||||
const out = await captureRun(() => runEvalCompare(['--json', '--input', path]));
|
||||
const report = JSON.parse(out);
|
||||
expect(report.grouped.longmemeval.conservative.metrics['recall@10']).toBe(0.9);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,197 @@
|
||||
/**
|
||||
* v0.32.3 — eval-run-all orchestrator unit tests.
|
||||
* Pins arg parsing + cost-guard semantics + persist hook + audit trail.
|
||||
*/
|
||||
import { afterAll, beforeAll, beforeEach, describe, expect, test } from 'bun:test';
|
||||
import { mkdtempSync, rmSync, readFileSync, existsSync } from 'fs';
|
||||
import { tmpdir } from 'os';
|
||||
import { join } from 'path';
|
||||
import {
|
||||
parseRunAllArgs,
|
||||
estimateRunCost,
|
||||
evaluateCostGuard,
|
||||
persistRunRecord,
|
||||
type EvalRunRecord,
|
||||
} from '../src/commands/eval-run-all.ts';
|
||||
|
||||
let tmp: string;
|
||||
|
||||
beforeAll(() => {
|
||||
tmp = mkdtempSync(join(tmpdir(), 'gbrain-eval-runall-'));
|
||||
});
|
||||
|
||||
afterAll(() => {
|
||||
rmSync(tmp, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
describe('parseRunAllArgs', () => {
|
||||
test('defaults: all modes, longmemeval+replay suites, seed=42', () => {
|
||||
const opts = parseRunAllArgs([]);
|
||||
expect(opts.modes).toEqual(['conservative', 'balanced', 'tokenmax']);
|
||||
expect(opts.suites).toEqual(['longmemeval', 'replay']);
|
||||
expect(opts.seed).toBe(42);
|
||||
expect(opts.parallel).toBe(1);
|
||||
expect(opts.budgetUsdRetrieval).toBe(5);
|
||||
expect(opts.budgetUsdAnswer).toBe(20);
|
||||
expect(opts.yes).toBe(false);
|
||||
});
|
||||
|
||||
test('--modes filters to a subset', () => {
|
||||
const opts = parseRunAllArgs(['--modes', 'conservative,tokenmax']);
|
||||
expect(opts.modes).toEqual(['conservative', 'tokenmax']);
|
||||
});
|
||||
|
||||
test('--modes rejects invalid mode', () => {
|
||||
expect(() => parseRunAllArgs(['--modes', 'frontier'])).toThrow(/not a valid mode/);
|
||||
});
|
||||
|
||||
test('--suites filters; rejects unknown', () => {
|
||||
const opts = parseRunAllArgs(['--suites', 'longmemeval']);
|
||||
expect(opts.suites).toEqual(['longmemeval']);
|
||||
expect(() => parseRunAllArgs(['--suites', 'foo'])).toThrow(/not a recognized suite/);
|
||||
});
|
||||
|
||||
test('--budget-usd-retrieval + --budget-usd-answer override defaults', () => {
|
||||
const opts = parseRunAllArgs(['--budget-usd-retrieval', '15', '--budget-usd-answer', '50']);
|
||||
expect(opts.budgetUsdRetrieval).toBe(15);
|
||||
expect(opts.budgetUsdAnswer).toBe(50);
|
||||
});
|
||||
|
||||
test('--parallel clamps to mode count', () => {
|
||||
const opts = parseRunAllArgs(['--parallel', '10']);
|
||||
expect(opts.parallel).toBe(3); // clamped to SEARCH_MODES.length
|
||||
});
|
||||
|
||||
test('--parallel rejects 0 / negative', () => {
|
||||
expect(() => parseRunAllArgs(['--parallel', '0'])).toThrow(/must be >= 1/);
|
||||
expect(() => parseRunAllArgs(['--parallel', '-1'])).toThrow(/must be >= 1/);
|
||||
});
|
||||
|
||||
test('--yes flag toggles', () => {
|
||||
expect(parseRunAllArgs(['--yes']).yes).toBe(true);
|
||||
expect(parseRunAllArgs(['-y']).yes).toBe(true);
|
||||
});
|
||||
});
|
||||
|
||||
describe('estimateRunCost', () => {
|
||||
test('returns retrieval/answer/total breakdown', () => {
|
||||
const est = estimateRunCost({ suites: ['longmemeval'], modes: ['balanced'], limit: 100 });
|
||||
expect(est.total_usd).toBe(est.retrieval_usd + est.answer_usd);
|
||||
expect(est.answer_usd).toBeGreaterThan(0); // longmemeval has answer-gen
|
||||
expect(est.retrieval_usd).toBeGreaterThan(0);
|
||||
});
|
||||
|
||||
test('tokenmax incurs expansion cost (Haiku per query)', () => {
|
||||
const bal = estimateRunCost({ suites: ['replay'], modes: ['balanced'], limit: 100 });
|
||||
const tok = estimateRunCost({ suites: ['replay'], modes: ['tokenmax'], limit: 100 });
|
||||
expect(tok.retrieval_usd).toBeGreaterThan(bal.retrieval_usd);
|
||||
});
|
||||
|
||||
test('replay suite has zero answer cost', () => {
|
||||
const est = estimateRunCost({ suites: ['replay'], modes: ['balanced'], limit: 100 });
|
||||
expect(est.answer_usd).toBe(0);
|
||||
});
|
||||
|
||||
test('multi-suite multi-mode estimate sums correctly', () => {
|
||||
const est = estimateRunCost({ suites: ['longmemeval', 'replay'], modes: ['conservative', 'balanced'], limit: 100 });
|
||||
expect(Object.keys(est.per_suite).length).toBe(4); // 2 × 2
|
||||
});
|
||||
});
|
||||
|
||||
describe('evaluateCostGuard', () => {
|
||||
test('within-budget proceeds without --yes', () => {
|
||||
const g = evaluateCostGuard(
|
||||
{ retrieval_usd: 1, answer_usd: 10, total_usd: 11 },
|
||||
{ budgetUsdRetrieval: 5, budgetUsdAnswer: 20, yes: false, isTty: true },
|
||||
);
|
||||
expect(g.proceed).toBe(true);
|
||||
});
|
||||
|
||||
test('over-cap TTY without --yes refuses', () => {
|
||||
const g = evaluateCostGuard(
|
||||
{ retrieval_usd: 30, answer_usd: 30, total_usd: 60 },
|
||||
{ budgetUsdRetrieval: 5, budgetUsdAnswer: 20, yes: false, isTty: true },
|
||||
);
|
||||
expect(g.proceed).toBe(false);
|
||||
expect(g.reason).toContain('--yes');
|
||||
});
|
||||
|
||||
test('over-cap TTY with --yes proceeds', () => {
|
||||
const g = evaluateCostGuard(
|
||||
{ retrieval_usd: 30, answer_usd: 30, total_usd: 60 },
|
||||
{ budgetUsdRetrieval: 5, budgetUsdAnswer: 20, yes: true, isTty: true },
|
||||
);
|
||||
expect(g.proceed).toBe(true);
|
||||
});
|
||||
|
||||
test('over-cap non-TTY without --yes refuses (exit 2 path)', () => {
|
||||
const g = evaluateCostGuard(
|
||||
{ retrieval_usd: 30, answer_usd: 30, total_usd: 60 },
|
||||
{ budgetUsdRetrieval: 5, budgetUsdAnswer: 20, yes: false, isTty: false },
|
||||
);
|
||||
expect(g.proceed).toBe(false);
|
||||
expect(g.reason).toContain('Non-TTY requires --yes');
|
||||
});
|
||||
|
||||
test('only-retrieval-over OR only-answer-over both trigger guard', () => {
|
||||
const a = evaluateCostGuard(
|
||||
{ retrieval_usd: 100, answer_usd: 1, total_usd: 101 },
|
||||
{ budgetUsdRetrieval: 5, budgetUsdAnswer: 20, yes: false, isTty: false },
|
||||
);
|
||||
expect(a.proceed).toBe(false);
|
||||
const b = evaluateCostGuard(
|
||||
{ retrieval_usd: 1, answer_usd: 100, total_usd: 101 },
|
||||
{ budgetUsdRetrieval: 5, budgetUsdAnswer: 20, yes: false, isTty: false },
|
||||
);
|
||||
expect(b.proceed).toBe(false);
|
||||
});
|
||||
});
|
||||
|
||||
describe('persistRunRecord audit trail', () => {
|
||||
beforeEach(() => {
|
||||
rmSync(join(tmp, '.gbrain-evals'), { recursive: true, force: true });
|
||||
rmSync(join(tmp, 'eval-results.jsonl'), { force: true });
|
||||
});
|
||||
|
||||
test('appends to eval-results.jsonl, creates dir if missing', () => {
|
||||
const record: EvalRunRecord = {
|
||||
schema_version: 2,
|
||||
run_id: 'abc123-longmemeval-conservative-42',
|
||||
ran_at: '2026-05-12T12:00:00Z',
|
||||
suite: 'longmemeval',
|
||||
mode: 'conservative',
|
||||
commit: 'abc123',
|
||||
seed: 42,
|
||||
params: {},
|
||||
status: 'completed',
|
||||
duration_ms: 12_345,
|
||||
};
|
||||
persistRunRecord(tmp, record, tmp);
|
||||
const path = join(tmp, 'eval-results.jsonl');
|
||||
expect(existsSync(path)).toBe(true);
|
||||
const content = readFileSync(path, 'utf-8');
|
||||
expect(content).toContain('abc123-longmemeval-conservative-42');
|
||||
const parsed = JSON.parse(content.trim());
|
||||
expect(parsed.mode).toBe('conservative');
|
||||
expect(parsed.schema_version).toBe(2);
|
||||
});
|
||||
|
||||
test('appends multiple records (NDJSON)', () => {
|
||||
const base = {
|
||||
schema_version: 2 as const,
|
||||
ran_at: '2026-05-12T12:00:00Z',
|
||||
suite: 'longmemeval' as const,
|
||||
commit: 'abc',
|
||||
seed: 42,
|
||||
params: {},
|
||||
status: 'completed' as const,
|
||||
duration_ms: 1,
|
||||
};
|
||||
persistRunRecord(tmp, { ...base, run_id: 'a', mode: 'conservative' }, tmp);
|
||||
persistRunRecord(tmp, { ...base, run_id: 'b', mode: 'balanced' }, tmp);
|
||||
persistRunRecord(tmp, { ...base, run_id: 'c', mode: 'tokenmax' }, tmp);
|
||||
const content = readFileSync(join(tmp, 'eval-results.jsonl'), 'utf-8');
|
||||
const lines = content.trim().split('\n');
|
||||
expect(lines.length).toBe(3);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,185 @@
|
||||
/**
|
||||
* v0.32.x search-lite \u2014 hybridSearchCached integration.
|
||||
*
|
||||
* End-to-end PGLite test that confirms the three search-lite features
|
||||
* fire through the actual hybrid pipeline (not the units in isolation):
|
||||
*
|
||||
* 1. Token budget: results are capped after search.
|
||||
* 2. Cache: meta surfaces hit/miss; disabled mode is a clean pass-through.
|
||||
* 3. Intent classifier: meta.intent matches the classifier output.
|
||||
*
|
||||
* Vector search isn't enabled (no embedding provider in test), so we
|
||||
* exercise the keyword-only path \u2014 which still surfaces intent and
|
||||
* budget. The cache path is exercised separately in query-cache.test.ts
|
||||
* because it needs a real embedding to key on.
|
||||
*/
|
||||
|
||||
import { describe, test, expect, beforeAll, afterAll } from 'bun:test';
|
||||
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
|
||||
import { hybridSearchCached } from '../src/core/search/hybrid.ts';
|
||||
import type { PageInput, HybridSearchMeta } from '../src/core/types.ts';
|
||||
|
||||
let engine: PGLiteEngine;
|
||||
const savedKey = process.env.OPENAI_API_KEY;
|
||||
|
||||
beforeAll(async () => {
|
||||
engine = new PGLiteEngine();
|
||||
await engine.connect({});
|
||||
await engine.initSchema();
|
||||
// Insert a small fixture set so keyword search has something to find.
|
||||
// Use long chunk_texts so token budget cuts have observable effect.
|
||||
const longText = 'x'.repeat(800); // ~200 tokens of body text
|
||||
const pages: Array<{ slug: string; page: PageInput }> = [
|
||||
{
|
||||
slug: 'alice-foo',
|
||||
page: {
|
||||
type: 'person',
|
||||
title: 'Alice Foo',
|
||||
compiled_truth: `Alice Foo is a builder. ${longText}`,
|
||||
},
|
||||
},
|
||||
{
|
||||
slug: 'bob-bar',
|
||||
page: {
|
||||
type: 'person',
|
||||
title: 'Bob Bar',
|
||||
compiled_truth: `Bob Bar is a builder. ${longText}`,
|
||||
},
|
||||
},
|
||||
{
|
||||
slug: 'carol-baz',
|
||||
page: {
|
||||
type: 'person',
|
||||
title: 'Carol Baz',
|
||||
compiled_truth: `Carol Baz is a builder. ${longText}`,
|
||||
},
|
||||
},
|
||||
];
|
||||
for (const p of pages) {
|
||||
await engine.putPage(p.slug, p.page);
|
||||
}
|
||||
// Force keyword-only fallback by unsetting the embedding provider key.
|
||||
delete process.env.OPENAI_API_KEY;
|
||||
});
|
||||
|
||||
afterAll(async () => {
|
||||
if (savedKey) process.env.OPENAI_API_KEY = savedKey;
|
||||
try { await engine.disconnect(); } catch { /* ignore */ }
|
||||
});
|
||||
|
||||
describe('hybridSearchCached \u2014 meta surfaces intent', () => {
|
||||
test('entity query classifies as entity', async () => {
|
||||
let meta: HybridSearchMeta | undefined;
|
||||
await hybridSearchCached(engine, 'who is alice-foo', {
|
||||
limit: 5,
|
||||
onMeta: (m) => { meta = m; },
|
||||
});
|
||||
expect(meta?.intent).toBe('entity');
|
||||
});
|
||||
|
||||
test('temporal query classifies as temporal', async () => {
|
||||
let meta: HybridSearchMeta | undefined;
|
||||
await hybridSearchCached(engine, 'what happened last week', {
|
||||
limit: 5,
|
||||
onMeta: (m) => { meta = m; },
|
||||
});
|
||||
expect(meta?.intent).toBe('temporal');
|
||||
});
|
||||
|
||||
test('event query classifies as event', async () => {
|
||||
let meta: HybridSearchMeta | undefined;
|
||||
await hybridSearchCached(engine, 'who raised $10M', {
|
||||
limit: 5,
|
||||
onMeta: (m) => { meta = m; },
|
||||
});
|
||||
expect(meta?.intent).toBe('event');
|
||||
});
|
||||
});
|
||||
|
||||
describe('hybridSearchCached \u2014 token budget', () => {
|
||||
test('budget undefined returns no token_budget meta (no cut)', async () => {
|
||||
let meta: HybridSearchMeta | undefined;
|
||||
const results = await hybridSearchCached(engine, 'alice', {
|
||||
limit: 10,
|
||||
onMeta: (m) => { meta = m; },
|
||||
});
|
||||
// Don't assert non-empty here — keyword tokenization depends on the
|
||||
// pglite analyzer config. What matters: meta is shaped right and
|
||||
// budget metadata is absent when budget isn't set.
|
||||
expect(results).toBeDefined();
|
||||
expect(meta?.token_budget).toBeUndefined();
|
||||
});
|
||||
|
||||
test('budget meta is always emitted when budget is set', async () => {
|
||||
let meta: HybridSearchMeta | undefined;
|
||||
const results = await hybridSearchCached(engine, 'alice', {
|
||||
limit: 10,
|
||||
tokenBudget: 250,
|
||||
onMeta: (m) => { meta = m; },
|
||||
});
|
||||
expect(meta?.token_budget).toBeDefined();
|
||||
expect(meta?.token_budget?.budget).toBe(250);
|
||||
expect(meta?.token_budget?.kept).toBe(results.length);
|
||||
});
|
||||
|
||||
test('tight budget cuts the result set', async () => {
|
||||
// First find out the result count without a budget so the assertion
|
||||
// is robust to the fixture’s actual chunking.
|
||||
const unbounded = await hybridSearchCached(engine, 'builder', { limit: 10 });
|
||||
// Skip the cut test if the fixture happens to return only one row
|
||||
// (keyword search may dedupe by page); the budget enforcement itself
|
||||
// is exhaustively unit-tested in test/token-budget.test.ts.
|
||||
if (unbounded.length < 2) return;
|
||||
|
||||
let meta: HybridSearchMeta | undefined;
|
||||
const results = await hybridSearchCached(engine, 'builder', {
|
||||
limit: 10,
|
||||
tokenBudget: 250, // enough for ~1 row of fixture data
|
||||
onMeta: (m) => { meta = m; },
|
||||
});
|
||||
expect(meta?.token_budget?.budget).toBe(250);
|
||||
expect(meta?.token_budget?.kept).toBe(results.length);
|
||||
expect(meta?.token_budget?.dropped).toBeGreaterThan(0);
|
||||
// The budget must hold: cumulative cost <= budget.
|
||||
expect(meta?.token_budget?.used).toBeLessThanOrEqual(250);
|
||||
});
|
||||
});
|
||||
|
||||
describe('hybridSearchCached \u2014 cache disabled fallback', () => {
|
||||
test('keyword-only path emits cache.status=disabled', async () => {
|
||||
// No embedding available \u2192 cache decision degrades to disabled.
|
||||
let meta: HybridSearchMeta | undefined;
|
||||
await hybridSearchCached(engine, 'who is alice-foo', {
|
||||
limit: 5,
|
||||
onMeta: (m) => { meta = m; },
|
||||
});
|
||||
// cache may be 'disabled' (no embedding provider) or 'miss'.
|
||||
// Either way the field exists.
|
||||
expect(meta?.cache).toBeDefined();
|
||||
expect(['disabled', 'miss']).toContain(meta?.cache?.status ?? '');
|
||||
});
|
||||
|
||||
test('useCache=false explicitly disables', async () => {
|
||||
let meta: HybridSearchMeta | undefined;
|
||||
await hybridSearchCached(engine, 'who is bob-bar', {
|
||||
limit: 5,
|
||||
useCache: false,
|
||||
onMeta: (m) => { meta = m; },
|
||||
});
|
||||
expect(meta?.cache?.status).toBe('disabled');
|
||||
});
|
||||
});
|
||||
|
||||
describe('hybridSearchCached \u2014 intent weighting toggle', () => {
|
||||
test('intentWeighting=false still emits intent in meta (for visibility)', async () => {
|
||||
let meta: HybridSearchMeta | undefined;
|
||||
await hybridSearchCached(engine, 'who is alice-foo', {
|
||||
limit: 5,
|
||||
intentWeighting: false,
|
||||
onMeta: (m) => { meta = m; },
|
||||
});
|
||||
// Intent classification itself still runs (cheap regex); only the
|
||||
// weight adjustment is disabled. So meta.intent stays populated.
|
||||
expect(meta?.intent).toBe('entity');
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,251 @@
|
||||
/**
|
||||
* v0.32.3 search-lite install-time mode picker tests.
|
||||
*
|
||||
* Pure-function coverage (recommendModeFor + parseModeInput) plus the
|
||||
* idempotent runModePicker behavior. The interactive TTY branch is
|
||||
* exercised indirectly via the non-TTY path here; full TTY simulation
|
||||
* lives in the e2e suite (test/e2e/...).
|
||||
*/
|
||||
import { afterAll, beforeAll, beforeEach, describe, expect, test } from 'bun:test';
|
||||
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
|
||||
import {
|
||||
recommendModeFor,
|
||||
parseModeInput,
|
||||
runModePicker,
|
||||
} from '../src/commands/init-mode-picker.ts';
|
||||
import { withEnv } from './helpers/with-env.ts';
|
||||
|
||||
let engine: PGLiteEngine;
|
||||
|
||||
beforeAll(async () => {
|
||||
engine = new PGLiteEngine();
|
||||
await engine.connect({});
|
||||
await engine.initSchema();
|
||||
});
|
||||
|
||||
afterAll(async () => {
|
||||
await engine.disconnect();
|
||||
});
|
||||
|
||||
beforeEach(async () => {
|
||||
await engine.executeRaw(`DELETE FROM config WHERE key LIKE 'search.%' OR key LIKE 'models.%'`);
|
||||
});
|
||||
|
||||
describe('recommendModeFor — auto-suggestion heuristic', () => {
|
||||
test('Opus default → tokenmax', () => {
|
||||
const r = recommendModeFor({ defaultModel: 'anthropic:claude-opus-4-7' });
|
||||
expect(r.mode).toBe('tokenmax');
|
||||
expect(r.reason).toMatch(/Opus/);
|
||||
});
|
||||
|
||||
test('Opus subagent → tokenmax', () => {
|
||||
const r = recommendModeFor({ subagentModel: 'anthropic:claude-opus-4-7' });
|
||||
expect(r.mode).toBe('tokenmax');
|
||||
});
|
||||
|
||||
test('Haiku subagent → conservative', () => {
|
||||
const r = recommendModeFor({ subagentModel: 'anthropic:claude-haiku-4-5' });
|
||||
expect(r.mode).toBe('conservative');
|
||||
expect(r.reason).toMatch(/Haiku/);
|
||||
});
|
||||
|
||||
test('No OpenAI key → conservative (no LLM expansion possible)', () => {
|
||||
const r = recommendModeFor({ hasOpenAIKey: false });
|
||||
expect(r.mode).toBe('conservative');
|
||||
expect(r.reason).toMatch(/No OpenAI/);
|
||||
});
|
||||
|
||||
test('Sonnet / unknown → tokenmax (preserve-v0.31.x default)', () => {
|
||||
const r = recommendModeFor({ subagentModel: 'anthropic:claude-sonnet-4-6', hasOpenAIKey: true });
|
||||
expect(r.mode).toBe('tokenmax');
|
||||
expect(r.reason).toMatch(/v0\.31\.x|preserve/i);
|
||||
});
|
||||
|
||||
test('Empty inputs → tokenmax (preserve-v0.31.x default)', () => {
|
||||
const r = recommendModeFor({});
|
||||
expect(r.mode).toBe('tokenmax');
|
||||
});
|
||||
|
||||
|
||||
test('Haiku subagent wins over Opus default (cost-sensitive takes precedence)', () => {
|
||||
// Reordered in the install-picker DX pass: Haiku check fires BEFORE Opus
|
||||
// because a user running a Haiku subagent loop is signalling cost
|
||||
// sensitivity. Tokenmax over Haiku would silently dump 50-chunk payloads
|
||||
// into a model that struggles past 5-10 chunks. The Haiku floor wins.
|
||||
const r = recommendModeFor({
|
||||
defaultModel: 'anthropic:claude-opus-4-7',
|
||||
subagentModel: 'anthropic:claude-haiku-4-5',
|
||||
hasOpenAIKey: true,
|
||||
});
|
||||
expect(r.mode).toBe('conservative');
|
||||
});
|
||||
});
|
||||
|
||||
describe('MENU_TEXT cost-matrix anchors (must match CLAUDE.md + methodology doc)', () => {
|
||||
test('25x corner-to-corner spread framing is named', async () => {
|
||||
const { MODE_PICKER_MENU } = await import('../src/commands/init-mode-picker.ts');
|
||||
// Updating these REQUIRES bumping CLAUDE.md ## Search Mode + the
|
||||
// methodology doc + README in lockstep.
|
||||
expect(MODE_PICKER_MENU).toContain('25x');
|
||||
expect(MODE_PICKER_MENU).toContain('corner-to-corner');
|
||||
});
|
||||
|
||||
test('cost matrix lists every cell at the natural diagonal and corners', async () => {
|
||||
const { MODE_PICKER_MENU } = await import('../src/commands/init-mode-picker.ts');
|
||||
// Three anchor cells from the natural diagonal at 10K/mo volume. Scales
|
||||
// linearly; multiplying by 10 gives the 100K/mo numbers the methodology
|
||||
// doc + CLAUDE.md cite ("multiply by 10 for 100K/mo" prose anchor).
|
||||
expect(MODE_PICKER_MENU).toContain('$40/mo'); // conservative + Haiku
|
||||
expect(MODE_PICKER_MENU).toContain('$300/mo'); // balanced + Sonnet (default natural)
|
||||
expect(MODE_PICKER_MENU).toContain('$1,000/mo'); // tokenmax + Opus
|
||||
});
|
||||
|
||||
test('volume frame is 10K queries/month with linear-scale callout', async () => {
|
||||
const { MODE_PICKER_MENU } = await import('../src/commands/init-mode-picker.ts');
|
||||
expect(MODE_PICKER_MENU).toContain('10K queries/mo');
|
||||
expect(MODE_PICKER_MENU).toContain('scales linearly');
|
||||
});
|
||||
|
||||
test('all three downstream model rates are explicit', async () => {
|
||||
const { MODE_PICKER_MENU } = await import('../src/commands/init-mode-picker.ts');
|
||||
expect(MODE_PICKER_MENU).toContain('Haiku 4.5');
|
||||
expect(MODE_PICKER_MENU).toContain('Sonnet 4.6');
|
||||
expect(MODE_PICKER_MENU).toContain('Opus 4.7');
|
||||
expect(MODE_PICKER_MENU).toContain('$1/M');
|
||||
expect(MODE_PICKER_MENU).toContain('$3/M');
|
||||
expect(MODE_PICKER_MENU).toContain('$5/M');
|
||||
});
|
||||
|
||||
test('tokenmax Haiku-expansion surcharge is named explicitly', async () => {
|
||||
const { MODE_PICKER_MENU } = await import('../src/commands/init-mode-picker.ts');
|
||||
// Cross-line match — the surcharge phrase can wrap.
|
||||
expect(MODE_PICKER_MENU.replace(/\s+/g, ' ')).toContain('~$1.50 per 1K queries');
|
||||
expect(MODE_PICKER_MENU).toContain('Haiku expansion call');
|
||||
});
|
||||
|
||||
test('cache-hit discount framing is named', async () => {
|
||||
const { MODE_PICKER_MENU } = await import('../src/commands/init-mode-picker.ts');
|
||||
expect(MODE_PICKER_MENU).toContain('cache');
|
||||
// The numbers below are full-payload pre-cache; real loops see 50-80% discount.
|
||||
expect(MODE_PICKER_MENU).toMatch(/50-80%|discount/);
|
||||
});
|
||||
|
||||
test('tune command is surfaced as the next step', async () => {
|
||||
const { MODE_PICKER_MENU } = await import('../src/commands/init-mode-picker.ts');
|
||||
expect(MODE_PICKER_MENU).toContain('gbrain search tune');
|
||||
});
|
||||
});
|
||||
|
||||
describe('parseModeInput — menu choice mapper', () => {
|
||||
test('numeric 1/2/3 → conservative/balanced/tokenmax', () => {
|
||||
expect(parseModeInput('1')).toBe('conservative');
|
||||
expect(parseModeInput('2')).toBe('balanced');
|
||||
expect(parseModeInput('3')).toBe('tokenmax');
|
||||
});
|
||||
|
||||
test('mode names (case-insensitive)', () => {
|
||||
expect(parseModeInput('conservative')).toBe('conservative');
|
||||
expect(parseModeInput('CONSERVATIVE')).toBe('conservative');
|
||||
expect(parseModeInput('TokenMax')).toBe('tokenmax');
|
||||
expect(parseModeInput(' balanced ')).toBe('balanced');
|
||||
});
|
||||
|
||||
test('empty / unrecognized → null', () => {
|
||||
expect(parseModeInput('')).toBeNull();
|
||||
expect(parseModeInput(' ')).toBeNull();
|
||||
expect(parseModeInput('foo')).toBeNull();
|
||||
expect(parseModeInput('4')).toBeNull();
|
||||
expect(parseModeInput('0')).toBeNull();
|
||||
});
|
||||
});
|
||||
|
||||
describe('runModePicker non-TTY surfaces full matrix + [AGENT] directive', () => {
|
||||
test('non-TTY output includes the cost matrix and the [AGENT] directive', async () => {
|
||||
const originalLog = console.log;
|
||||
const captured: string[] = [];
|
||||
console.log = (msg: string) => { captured.push(String(msg)); };
|
||||
try {
|
||||
await runModePicker(engine);
|
||||
} finally {
|
||||
console.log = originalLog;
|
||||
}
|
||||
const out = captured.join('\n');
|
||||
// Matrix corners + diagonal mid
|
||||
expect(out).toContain('$40/mo');
|
||||
expect(out).toContain('$300/mo');
|
||||
expect(out).toContain('$1,000/mo');
|
||||
// 25x spread framing
|
||||
expect(out).toContain('25x corner-to-corner');
|
||||
// Explicit agent directive — load-bearing for agent-platform install paths
|
||||
expect(out).toContain('[AGENT]');
|
||||
expect(out.toLowerCase()).toContain('show this matrix');
|
||||
expect(out).toContain('INSTALL_FOR_AGENTS.md');
|
||||
});
|
||||
});
|
||||
|
||||
describe('runModePicker — non-TTY auto-select + idempotent', () => {
|
||||
test('non-TTY auto-selects + writes config + emits operator hint', async () => {
|
||||
// Bun test runs non-TTY by default.
|
||||
const picked = await runModePicker(engine);
|
||||
// Default model unset → balanced. Should write search.mode.
|
||||
const stored = await engine.getConfig('search.mode');
|
||||
expect(stored).toBe(picked);
|
||||
expect(['conservative', 'balanced', 'tokenmax']).toContain(picked);
|
||||
});
|
||||
|
||||
test('idempotent: second call returns existing mode without overwrite', async () => {
|
||||
await engine.setConfig('search.mode', 'tokenmax');
|
||||
const picked = await runModePicker(engine);
|
||||
expect(picked).toBe('tokenmax');
|
||||
// No accidental overwrite.
|
||||
const stored = await engine.getConfig('search.mode');
|
||||
expect(stored).toBe('tokenmax');
|
||||
});
|
||||
|
||||
test('--force re-prompts even if mode is already set', async () => {
|
||||
await engine.setConfig('search.mode', 'conservative');
|
||||
// Non-TTY + force → re-runs auto-suggest with current inputs.
|
||||
const picked = await runModePicker(engine, { force: true });
|
||||
// With no model hints + no API key state, default is balanced. The picker
|
||||
// will overwrite the existing mode.
|
||||
expect(['conservative', 'balanced', 'tokenmax']).toContain(picked);
|
||||
const stored = await engine.getConfig('search.mode');
|
||||
expect(stored).toBe(picked);
|
||||
});
|
||||
|
||||
test('jsonOutput mode emits a structured event and writes config', async () => {
|
||||
// Capture console.log output.
|
||||
const originalLog = console.log;
|
||||
const captured: string[] = [];
|
||||
console.log = (msg: string) => { captured.push(msg); };
|
||||
try {
|
||||
const picked = await runModePicker(engine, { jsonOutput: true });
|
||||
const stored = await engine.getConfig('search.mode');
|
||||
expect(stored).toBe(picked);
|
||||
const jsonLine = captured.find(l => l.startsWith('{'));
|
||||
expect(jsonLine).toBeDefined();
|
||||
const obj = JSON.parse(jsonLine!);
|
||||
expect(obj.phase).toBe('search_mode_picker');
|
||||
expect(obj.auto).toBe(true);
|
||||
expect(['conservative', 'balanced', 'tokenmax']).toContain(obj.mode);
|
||||
} finally {
|
||||
console.log = originalLog;
|
||||
}
|
||||
});
|
||||
|
||||
test('Opus default model + OpenAI key → picker auto-recommends tokenmax', async () => {
|
||||
// OPENAI_API_KEY must be present — the no-key short-circuit fires before
|
||||
// the Opus check (gbrain can't do vector search without embeddings).
|
||||
await withEnv({ OPENAI_API_KEY: 'sk-test-stub' }, async () => {
|
||||
await engine.setConfig('models.default', 'anthropic:claude-opus-4-7');
|
||||
const picked = await runModePicker(engine);
|
||||
expect(picked).toBe('tokenmax');
|
||||
});
|
||||
});
|
||||
|
||||
test('Haiku subagent → picker auto-recommends conservative', async () => {
|
||||
await engine.setConfig('models.tier.subagent', 'anthropic:claude-haiku-4-5');
|
||||
const picked = await runModePicker(engine);
|
||||
expect(picked).toBe('conservative');
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,120 @@
|
||||
/**
|
||||
* v0.32.x search-lite \u2014 intent \u2192 weight adjustment tests.
|
||||
*
|
||||
* Pure module under test (no DB). Confirms:
|
||||
* - weightsForIntent returns the expected per-intent factors
|
||||
* - effectiveRrfK scales correctly with the weight
|
||||
* - applyExactMatchBoost only fires on exact slug/title matches
|
||||
* - general intent is a no-op (preserves pre-v0.32 behavior)
|
||||
*/
|
||||
|
||||
import { describe, test, expect } from 'bun:test';
|
||||
import {
|
||||
weightsForIntent,
|
||||
effectiveRrfK,
|
||||
applyExactMatchBoost,
|
||||
} from '../src/core/search/intent-weights.ts';
|
||||
import type { SearchResult } from '../src/core/types.ts';
|
||||
|
||||
function makeResult(overrides: Partial<SearchResult> = {}): SearchResult {
|
||||
return {
|
||||
slug: 'test-page',
|
||||
page_id: 1,
|
||||
title: 'Test Title',
|
||||
type: 'concept',
|
||||
chunk_text: 'test chunk text',
|
||||
chunk_source: 'compiled_truth',
|
||||
chunk_id: 1,
|
||||
chunk_index: 0,
|
||||
score: 1.0,
|
||||
stale: false,
|
||||
...overrides,
|
||||
};
|
||||
}
|
||||
|
||||
describe('weightsForIntent', () => {
|
||||
test('general intent is the identity', () => {
|
||||
const w = weightsForIntent('general');
|
||||
expect(w.keywordWeight).toBe(1.0);
|
||||
expect(w.vectorWeight).toBe(1.0);
|
||||
expect(w.suggestedRecency).toBe(null);
|
||||
expect(w.exactMatchBoost).toBe(1.0);
|
||||
});
|
||||
|
||||
test('entity intent boosts keyword + exact match', () => {
|
||||
const w = weightsForIntent('entity');
|
||||
expect(w.keywordWeight).toBeGreaterThan(1.0);
|
||||
expect(w.exactMatchBoost).toBeGreaterThan(1.0);
|
||||
expect(w.suggestedRecency).toBe(null);
|
||||
});
|
||||
|
||||
test('temporal intent suggests recency=on', () => {
|
||||
const w = weightsForIntent('temporal');
|
||||
expect(w.suggestedRecency).toBe('on');
|
||||
});
|
||||
|
||||
test('event intent boosts keyword (rare named entities)', () => {
|
||||
const w = weightsForIntent('event');
|
||||
expect(w.keywordWeight).toBeGreaterThan(1.0);
|
||||
expect(w.suggestedRecency).toBe('on');
|
||||
});
|
||||
});
|
||||
|
||||
describe('effectiveRrfK', () => {
|
||||
test('weight=1.0 returns base k unchanged', () => {
|
||||
expect(effectiveRrfK(60, 1.0)).toBe(60);
|
||||
});
|
||||
|
||||
test('weight > 1 lowers k (stronger top-rank contribution)', () => {
|
||||
expect(effectiveRrfK(60, 1.2)).toBeLessThan(60);
|
||||
expect(effectiveRrfK(60, 1.2)).toBe(50);
|
||||
});
|
||||
|
||||
test('weight <= 0 returns base k (safety)', () => {
|
||||
expect(effectiveRrfK(60, 0)).toBe(60);
|
||||
expect(effectiveRrfK(60, -1)).toBe(60);
|
||||
});
|
||||
});
|
||||
|
||||
describe('applyExactMatchBoost', () => {
|
||||
test('boost=1.0 is a no-op', () => {
|
||||
const results = [makeResult({ slug: 'foo', score: 1.0 })];
|
||||
applyExactMatchBoost(results, 'foo', weightsForIntent('general'));
|
||||
expect(results[0].score).toBe(1.0);
|
||||
});
|
||||
|
||||
test('exact slug match gets boosted (entity intent)', () => {
|
||||
const results = [
|
||||
makeResult({ slug: 'garry-tan', score: 1.0, title: 'Garry Tan' }),
|
||||
makeResult({ slug: 'someone-else', score: 1.0, title: 'Someone Else' }),
|
||||
];
|
||||
applyExactMatchBoost(results, 'garry-tan', weightsForIntent('entity'));
|
||||
// First result has slug=query \u2192 boosted; second is unchanged.
|
||||
expect(results[0].score).toBeGreaterThan(1.0);
|
||||
expect(results[1].score).toBe(1.0);
|
||||
});
|
||||
|
||||
test('query with spaces matches kebab slug ("garry tan" \u2192 "garry-tan")', () => {
|
||||
const results = [makeResult({ slug: 'garry-tan', score: 1.0 })];
|
||||
applyExactMatchBoost(results, 'garry tan', weightsForIntent('entity'));
|
||||
expect(results[0].score).toBeGreaterThan(1.0);
|
||||
});
|
||||
|
||||
test('exact title match (case-insensitive) gets boosted', () => {
|
||||
const results = [makeResult({ slug: 'random-slug', title: 'Garry Tan', score: 1.0 })];
|
||||
applyExactMatchBoost(results, 'GARRY TAN', weightsForIntent('entity'));
|
||||
expect(results[0].score).toBeGreaterThan(1.0);
|
||||
});
|
||||
|
||||
test('partial / substring match does NOT trigger boost', () => {
|
||||
const results = [makeResult({ slug: 'garry-tan', title: 'Garry Tan', score: 1.0 })];
|
||||
applyExactMatchBoost(results, 'garry', weightsForIntent('entity'));
|
||||
expect(results[0].score).toBe(1.0);
|
||||
});
|
||||
|
||||
test('namespaced slug matches via suffix ("people/garry-tan" + query "garry-tan")', () => {
|
||||
const results = [makeResult({ slug: 'people/garry-tan', score: 1.0 })];
|
||||
applyExactMatchBoost(results, 'garry-tan', weightsForIntent('entity'));
|
||||
expect(results[0].score).toBeGreaterThan(1.0);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,128 @@
|
||||
/**
|
||||
* v0.32.3 — metric glossary module tests.
|
||||
* Pins the public surface that drives gbrain search stats / eval compare
|
||||
* output AND the auto-generated METRIC_GLOSSARY.md doc.
|
||||
*/
|
||||
import { describe, expect, test } from 'bun:test';
|
||||
import {
|
||||
METRIC_GLOSSARY,
|
||||
ALL_METRICS,
|
||||
getMetricGloss,
|
||||
eli10For,
|
||||
buildMetricGlossaryMeta,
|
||||
renderMetricGlossaryMarkdown,
|
||||
} from '../src/core/eval/metric-glossary.ts';
|
||||
|
||||
describe('METRIC_GLOSSARY canonical entries', () => {
|
||||
test('every retrieval-IR metric is present', () => {
|
||||
for (const m of ['precision@k', 'recall@k', 'mrr', 'ndcg@k']) {
|
||||
expect(METRIC_GLOSSARY[m]).toBeDefined();
|
||||
}
|
||||
});
|
||||
|
||||
test('every stability metric is present', () => {
|
||||
for (const m of ['jaccard@k', 'top1_stability']) {
|
||||
expect(METRIC_GLOSSARY[m]).toBeDefined();
|
||||
}
|
||||
});
|
||||
|
||||
test('every stat-significance metric is present', () => {
|
||||
for (const m of ['p_value', 'confidence_interval']) {
|
||||
expect(METRIC_GLOSSARY[m]).toBeDefined();
|
||||
}
|
||||
});
|
||||
|
||||
test('every operational metric is present', () => {
|
||||
for (const m of ['cache_hit_rate', 'avg_results', 'avg_tokens', 'cost_per_query_usd', 'p99_latency_ms']) {
|
||||
expect(METRIC_GLOSSARY[m]).toBeDefined();
|
||||
}
|
||||
});
|
||||
|
||||
test('every entry has the three required fields', () => {
|
||||
for (const [key, entry] of Object.entries(METRIC_GLOSSARY)) {
|
||||
expect(entry.industry_term, `${key}.industry_term`).toBeTruthy();
|
||||
expect(entry.eli10, `${key}.eli10`).toBeTruthy();
|
||||
expect(entry.range, `${key}.range`).toBeTruthy();
|
||||
}
|
||||
});
|
||||
|
||||
test('industry_term preserved verbatim (case-sensitive)', () => {
|
||||
expect(METRIC_GLOSSARY['ndcg@k'].industry_term).toBe('Normalized Discounted Cumulative Gain at k (nDCG@k)');
|
||||
expect(METRIC_GLOSSARY['mrr'].industry_term).toBe('Mean Reciprocal Rank (MRR)');
|
||||
});
|
||||
});
|
||||
|
||||
describe('getMetricGloss + eli10For accessors', () => {
|
||||
test('getMetricGloss returns the full entry', () => {
|
||||
const g = getMetricGloss('recall@k');
|
||||
expect(g).not.toBeNull();
|
||||
expect(g!.industry_term).toContain('Recall at k');
|
||||
});
|
||||
|
||||
test('getMetricGloss returns null for unknown metrics', () => {
|
||||
expect(getMetricGloss('made_up_metric')).toBeNull();
|
||||
});
|
||||
|
||||
test('eli10For returns just the plain-English line', () => {
|
||||
const text = eli10For('cache_hit_rate');
|
||||
expect(text).toContain('Fraction of searches');
|
||||
});
|
||||
|
||||
test('eli10For returns null for unknown metric', () => {
|
||||
expect(eli10For('nope')).toBeNull();
|
||||
});
|
||||
});
|
||||
|
||||
describe('buildMetricGlossaryMeta', () => {
|
||||
test('returns a flat record keyed by metric name → eli10 string', () => {
|
||||
const meta = buildMetricGlossaryMeta(['recall@k', 'mrr']);
|
||||
expect(Object.keys(meta).sort()).toEqual(['mrr', 'recall@k']);
|
||||
expect(meta['recall@k']).toContain('relevant');
|
||||
expect(meta['mrr']).toContain('FIRST relevant result');
|
||||
});
|
||||
|
||||
test('unknown metrics silently dropped (no error)', () => {
|
||||
const meta = buildMetricGlossaryMeta(['recall@k', 'made_up']);
|
||||
expect(meta['recall@k']).toBeDefined();
|
||||
expect(meta['made_up']).toBeUndefined();
|
||||
});
|
||||
|
||||
test('empty array → empty object', () => {
|
||||
expect(buildMetricGlossaryMeta([])).toEqual({});
|
||||
});
|
||||
});
|
||||
|
||||
describe('ALL_METRICS roster', () => {
|
||||
test('has every glossary key', () => {
|
||||
expect([...ALL_METRICS].sort()).toEqual(Object.keys(METRIC_GLOSSARY).sort());
|
||||
});
|
||||
|
||||
test('matches the renderer output (no orphans)', () => {
|
||||
const md = renderMetricGlossaryMarkdown();
|
||||
for (const m of ALL_METRICS) {
|
||||
const entry = METRIC_GLOSSARY[m];
|
||||
expect(md, `Markdown should mention ${m}`).toContain(entry.industry_term);
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
describe('renderMetricGlossaryMarkdown determinism', () => {
|
||||
test('repeated calls produce identical output (deterministic)', () => {
|
||||
const a = renderMetricGlossaryMarkdown();
|
||||
const b = renderMetricGlossaryMarkdown();
|
||||
expect(a).toBe(b);
|
||||
});
|
||||
|
||||
test('output includes the auto-generated marker', () => {
|
||||
const md = renderMetricGlossaryMarkdown();
|
||||
expect(md).toContain('Auto-generated from `src/core/eval/metric-glossary.ts`');
|
||||
});
|
||||
|
||||
test('output groups metrics into 4 sections', () => {
|
||||
const md = renderMetricGlossaryMarkdown();
|
||||
expect(md).toContain('## Retrieval Metrics');
|
||||
expect(md).toContain('## Set-Similarity / Stability Metrics');
|
||||
expect(md).toContain('## Statistical-Significance Metrics');
|
||||
expect(md).toContain('## Operational / Cost Metrics');
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,217 @@
|
||||
/**
|
||||
* Regression test for [CDX-4] cross-mode cache contamination.
|
||||
*
|
||||
* Before v0.32.3 (PR #897 as merged), the query_cache primary key was
|
||||
* sha256(source_id::query_text) — a tokenmax search (expansion=on, limit=50)
|
||||
* would populate a row that a subsequent conservative call (no expansion,
|
||||
* limit=10) read back, serving the wrong-shape results.
|
||||
*
|
||||
* After v0.32.3:
|
||||
* - cacheRowId(query, source, knobsHash) — knobsHash is part of the PK
|
||||
* - SemanticQueryCache.lookup({knobsHash}) filters WHERE knobs_hash = $
|
||||
* - SemanticQueryCache.store({knobsHash}) writes the resolved hash
|
||||
*
|
||||
* This test exercises the cache class directly on a fresh PGLite brain
|
||||
* to verify cross-mode writes don't collide.
|
||||
*/
|
||||
import { afterAll, beforeAll, beforeEach, describe, expect, test } from 'bun:test';
|
||||
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
|
||||
import { SemanticQueryCache, cacheRowId } from '../src/core/search/query-cache.ts';
|
||||
import type { SearchResult } from '../src/core/types.ts';
|
||||
import { knobsHash, resolveSearchMode } from '../src/core/search/mode.ts';
|
||||
|
||||
let engine: PGLiteEngine;
|
||||
|
||||
const conservativeHash = knobsHash(resolveSearchMode({ mode: 'conservative' }));
|
||||
const balancedHash = knobsHash(resolveSearchMode({ mode: 'balanced' }));
|
||||
const tokenmaxHash = knobsHash(resolveSearchMode({ mode: 'tokenmax' }));
|
||||
|
||||
beforeAll(async () => {
|
||||
engine = new PGLiteEngine();
|
||||
await engine.connect({});
|
||||
await engine.initSchema();
|
||||
});
|
||||
|
||||
afterAll(async () => {
|
||||
await engine.disconnect();
|
||||
});
|
||||
|
||||
beforeEach(async () => {
|
||||
// Clear cache before each test so writes start from empty state.
|
||||
await engine.executeRaw('DELETE FROM query_cache');
|
||||
});
|
||||
|
||||
const makeEmbedding = (seed: number): Float32Array => {
|
||||
const arr = new Float32Array(1536);
|
||||
for (let i = 0; i < 1536; i++) {
|
||||
arr[i] = Math.sin(seed + i * 0.001);
|
||||
}
|
||||
// Normalize to unit length so cosine similarity is well-defined.
|
||||
let norm = 0;
|
||||
for (let i = 0; i < 1536; i++) norm += arr[i] * arr[i];
|
||||
norm = Math.sqrt(norm);
|
||||
if (norm > 0) for (let i = 0; i < 1536; i++) arr[i] /= norm;
|
||||
return arr;
|
||||
};
|
||||
|
||||
const makeResults = (label: string, n: number): SearchResult[] =>
|
||||
Array.from({ length: n }, (_, i) => ({
|
||||
slug: `${label}/result-${i}`,
|
||||
title: `${label} result ${i}`,
|
||||
chunk_text: `chunk-${label}-${i}`,
|
||||
chunk_id: (i + 1) * 1000,
|
||||
score: 1 / (i + 1),
|
||||
chunk_index: i,
|
||||
type: 'note' as const,
|
||||
chunk_source: 'compiled_truth' as const,
|
||||
page_id: i + 1,
|
||||
stale: false,
|
||||
}));
|
||||
|
||||
describe('cacheRowId is bifurcated by knobsHash', () => {
|
||||
test('same (query, source) but different knobs → different row IDs', () => {
|
||||
const id1 = cacheRowId('what is the meaning of life', 'default', conservativeHash);
|
||||
const id2 = cacheRowId('what is the meaning of life', 'default', tokenmaxHash);
|
||||
expect(id1).not.toBe(id2);
|
||||
});
|
||||
|
||||
test('same (query, source, knobs) → same row ID (idempotent)', () => {
|
||||
const id1 = cacheRowId('what is the meaning of life', 'default', balancedHash);
|
||||
const id2 = cacheRowId('what is the meaning of life', 'default', balancedHash);
|
||||
expect(id1).toBe(id2);
|
||||
});
|
||||
|
||||
test('empty knobsHash still produces a valid ID (test-fixture compatibility)', () => {
|
||||
const id = cacheRowId('q', 'default', '');
|
||||
expect(id).toMatch(/^[0-9a-f]{32}$/);
|
||||
});
|
||||
|
||||
test('all three mode hashes are distinct', () => {
|
||||
expect(conservativeHash).not.toBe(balancedHash);
|
||||
expect(balancedHash).not.toBe(tokenmaxHash);
|
||||
expect(conservativeHash).not.toBe(tokenmaxHash);
|
||||
});
|
||||
});
|
||||
|
||||
describe('SemanticQueryCache cross-mode isolation (CDX-4 hotfix)', () => {
|
||||
test('tokenmax write does NOT contaminate conservative lookup', async () => {
|
||||
const cache = new SemanticQueryCache(engine);
|
||||
const emb = makeEmbedding(1);
|
||||
const tokenmaxResults = makeResults('tokenmax', 50);
|
||||
|
||||
// Write under tokenmax knobs.
|
||||
await cache.store('what is the meaning of life', emb, tokenmaxResults, {
|
||||
vector_enabled: true,
|
||||
detail_resolved: null,
|
||||
expansion_applied: true,
|
||||
}, { knobsHash: tokenmaxHash });
|
||||
|
||||
// Lookup under conservative knobs with the same embedding → MISS.
|
||||
const conservativeHit = await cache.lookup(emb, { knobsHash: conservativeHash });
|
||||
expect(conservativeHit.hit).toBe(false);
|
||||
|
||||
// Lookup under tokenmax knobs with the same embedding → HIT.
|
||||
const tokenmaxHit = await cache.lookup(emb, { knobsHash: tokenmaxHash });
|
||||
expect(tokenmaxHit.hit).toBe(true);
|
||||
expect(tokenmaxHit.results?.length).toBe(50);
|
||||
expect(tokenmaxHit.results?.[0].slug).toBe('tokenmax/result-0');
|
||||
});
|
||||
|
||||
test('three modes coexist as distinct rows for the same query', async () => {
|
||||
const cache = new SemanticQueryCache(engine);
|
||||
const emb = makeEmbedding(2);
|
||||
|
||||
await cache.store('q', emb, makeResults('conservative', 10), {
|
||||
vector_enabled: true, detail_resolved: null, expansion_applied: false,
|
||||
}, { knobsHash: conservativeHash });
|
||||
await cache.store('q', emb, makeResults('balanced', 25), {
|
||||
vector_enabled: true, detail_resolved: null, expansion_applied: false,
|
||||
}, { knobsHash: balancedHash });
|
||||
await cache.store('q', emb, makeResults('tokenmax', 50), {
|
||||
vector_enabled: true, detail_resolved: null, expansion_applied: true,
|
||||
}, { knobsHash: tokenmaxHash });
|
||||
|
||||
const rows = await engine.executeRaw<{ n: number }>(
|
||||
`SELECT COUNT(*)::int AS n FROM query_cache WHERE query_text = 'q'`,
|
||||
);
|
||||
expect(rows[0].n).toBe(3);
|
||||
|
||||
const cHit = await cache.lookup(emb, { knobsHash: conservativeHash });
|
||||
expect(cHit.hit).toBe(true);
|
||||
expect(cHit.results?.length).toBe(10);
|
||||
expect(cHit.results?.[0].slug).toBe('conservative/result-0');
|
||||
|
||||
const bHit = await cache.lookup(emb, { knobsHash: balancedHash });
|
||||
expect(bHit.hit).toBe(true);
|
||||
expect(bHit.results?.length).toBe(25);
|
||||
|
||||
const tHit = await cache.lookup(emb, { knobsHash: tokenmaxHash });
|
||||
expect(tHit.hit).toBe(true);
|
||||
expect(tHit.results?.length).toBe(50);
|
||||
});
|
||||
|
||||
test('legacy rows (NULL knobs_hash) are excluded from lookup', async () => {
|
||||
// Manually insert a row with NULL knobs_hash (simulating pre-v0.32.3 state).
|
||||
const emb = makeEmbedding(3);
|
||||
const vecStr = `[${Array.from(emb).map(v => v.toFixed(6)).join(',')}]`;
|
||||
await engine.executeRaw(
|
||||
`INSERT INTO query_cache (id, query_text, source_id, knobs_hash, embedding, results, meta, ttl_seconds, created_at)
|
||||
VALUES ($1, $2, $3, NULL, $4::vector, $5::jsonb, $6::jsonb, 3600, now())`,
|
||||
[
|
||||
'legacy-row-id',
|
||||
'legacy-query',
|
||||
'default',
|
||||
vecStr,
|
||||
JSON.stringify(makeResults('legacy', 5)),
|
||||
JSON.stringify({ vector_enabled: true, detail_resolved: null, expansion_applied: false }),
|
||||
],
|
||||
);
|
||||
|
||||
const cache = new SemanticQueryCache(engine);
|
||||
// Any mode's lookup → MISS (NULL row is excluded by the knobs_hash filter).
|
||||
expect((await cache.lookup(emb, { knobsHash: conservativeHash })).hit).toBe(false);
|
||||
expect((await cache.lookup(emb, { knobsHash: balancedHash })).hit).toBe(false);
|
||||
expect((await cache.lookup(emb, { knobsHash: tokenmaxHash })).hit).toBe(false);
|
||||
});
|
||||
|
||||
test('same mode written twice updates in place (no duplicate rows)', async () => {
|
||||
const cache = new SemanticQueryCache(engine);
|
||||
const emb = makeEmbedding(4);
|
||||
|
||||
await cache.store('q', emb, makeResults('first', 5), {
|
||||
vector_enabled: true, detail_resolved: null, expansion_applied: false,
|
||||
}, { knobsHash: balancedHash });
|
||||
|
||||
await cache.store('q', emb, makeResults('second', 7), {
|
||||
vector_enabled: true, detail_resolved: null, expansion_applied: false,
|
||||
}, { knobsHash: balancedHash });
|
||||
|
||||
const rows = await engine.executeRaw<{ n: number }>(
|
||||
`SELECT COUNT(*)::int AS n FROM query_cache WHERE query_text = 'q'`,
|
||||
);
|
||||
expect(rows[0].n).toBe(1);
|
||||
|
||||
const hit = await cache.lookup(emb, { knobsHash: balancedHash });
|
||||
expect(hit.hit).toBe(true);
|
||||
expect(hit.results?.length).toBe(7);
|
||||
expect(hit.results?.[0].slug).toBe('second/result-0');
|
||||
});
|
||||
|
||||
test('empty knobsHash arg writes a row but does not collide with mode-hash rows', async () => {
|
||||
const cache = new SemanticQueryCache(engine);
|
||||
const emb = makeEmbedding(5);
|
||||
|
||||
await cache.store('q', emb, makeResults('no-mode', 3), {
|
||||
vector_enabled: true, detail_resolved: null, expansion_applied: false,
|
||||
});
|
||||
|
||||
// No-mode lookup hits its own row.
|
||||
const noModeHit = await cache.lookup(emb);
|
||||
expect(noModeHit.hit).toBe(true);
|
||||
expect(noModeHit.results?.length).toBe(3);
|
||||
|
||||
// Conservative-hash lookup misses (the no-mode row had empty hash).
|
||||
const conservativeHit = await cache.lookup(emb, { knobsHash: conservativeHash });
|
||||
expect(conservativeHit.hit).toBe(false);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,246 @@
|
||||
/**
|
||||
* v0.32.x search-lite \u2014 semantic query cache.
|
||||
*
|
||||
* PGLite-backed test. Confirms:
|
||||
* - migration v51 creates the query_cache table
|
||||
* - store + lookup roundtrip with EXACT same embedding \u2192 hit
|
||||
* - lookup with a similar embedding (cosine > 0.92) \u2192 hit
|
||||
* - lookup with a far embedding \u2192 miss
|
||||
* - TTL expiration: a stale row is skipped at read time
|
||||
* - clear / prune / stats work as advertised
|
||||
* - source_id isolation: brain A's cache doesn't leak to brain B
|
||||
* - disabled cache is a pure no-op
|
||||
*
|
||||
* Uses synthetic Float32Array embeddings so the test doesn't depend on
|
||||
* any external embedding provider.
|
||||
*/
|
||||
|
||||
import { describe, test, expect, beforeAll, afterAll, beforeEach } from 'bun:test';
|
||||
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
|
||||
import { SemanticQueryCache, cacheRowId } from '../src/core/search/query-cache.ts';
|
||||
import type { SearchResult, HybridSearchMeta } from '../src/core/types.ts';
|
||||
|
||||
let engine: PGLiteEngine;
|
||||
|
||||
// Build a stable, normalized embedding. PGLite ships pgvector with 1536-dim
|
||||
// support (the default); a smaller test dim won't match the column. We
|
||||
// truncate / pad to 1536 to match the migration's resolved dim.
|
||||
const DIM = 1536;
|
||||
|
||||
function makeEmbedding(seed: number, dim = DIM): Float32Array {
|
||||
const e = new Float32Array(dim);
|
||||
// Simple deterministic generator with a unique fingerprint per seed
|
||||
// so similar seeds produce similar (cosine > 0.95) vectors and distinct
|
||||
// seeds produce orthogonal-ish ones.
|
||||
for (let i = 0; i < dim; i++) {
|
||||
e[i] = Math.sin(seed * 0.001 + i * 0.01);
|
||||
}
|
||||
// L2-normalize so cosine = dot product.
|
||||
let mag = 0;
|
||||
for (let i = 0; i < dim; i++) mag += e[i] * e[i];
|
||||
mag = Math.sqrt(mag);
|
||||
if (mag > 0) for (let i = 0; i < dim; i++) e[i] /= mag;
|
||||
return e;
|
||||
}
|
||||
|
||||
function makeOrthogonalEmbedding(seed: number, dim = DIM): Float32Array {
|
||||
// Use a totally different basis so cosine is near-zero.
|
||||
const e = new Float32Array(dim);
|
||||
for (let i = 0; i < dim; i++) {
|
||||
e[i] = Math.cos(seed * 13.7 + i * 0.97);
|
||||
}
|
||||
let mag = 0;
|
||||
for (let i = 0; i < dim; i++) mag += e[i] * e[i];
|
||||
mag = Math.sqrt(mag);
|
||||
if (mag > 0) for (let i = 0; i < dim; i++) e[i] /= mag;
|
||||
return e;
|
||||
}
|
||||
|
||||
function makeResult(slug: string): SearchResult {
|
||||
return {
|
||||
slug,
|
||||
page_id: 1,
|
||||
title: `Title for ${slug}`,
|
||||
type: 'concept',
|
||||
chunk_text: `chunk text for ${slug}`,
|
||||
chunk_source: 'compiled_truth',
|
||||
chunk_id: 1,
|
||||
chunk_index: 0,
|
||||
score: 1.0,
|
||||
stale: false,
|
||||
};
|
||||
}
|
||||
|
||||
const META: HybridSearchMeta = {
|
||||
vector_enabled: true,
|
||||
detail_resolved: 'medium',
|
||||
expansion_applied: false,
|
||||
intent: 'general',
|
||||
};
|
||||
|
||||
beforeAll(async () => {
|
||||
engine = new PGLiteEngine();
|
||||
await engine.connect({});
|
||||
await engine.initSchema();
|
||||
});
|
||||
|
||||
afterAll(async () => {
|
||||
try { await engine.disconnect(); } catch { /* ignore */ }
|
||||
});
|
||||
|
||||
beforeEach(async () => {
|
||||
// Wipe the cache between tests so ordering doesn't matter.
|
||||
await engine.executeRaw(`DELETE FROM query_cache`);
|
||||
});
|
||||
|
||||
describe('migration v51 \u2014 query_cache table exists', () => {
|
||||
test('table is present and has expected columns', async () => {
|
||||
const rows = await engine.executeRaw<{ column_name: string }>(
|
||||
`SELECT column_name FROM information_schema.columns
|
||||
WHERE table_name = 'query_cache'`,
|
||||
);
|
||||
const names = rows.map(r => r.column_name);
|
||||
expect(names).toContain('id');
|
||||
expect(names).toContain('query_text');
|
||||
expect(names).toContain('source_id');
|
||||
expect(names).toContain('embedding');
|
||||
expect(names).toContain('results');
|
||||
expect(names).toContain('meta');
|
||||
expect(names).toContain('ttl_seconds');
|
||||
expect(names).toContain('created_at');
|
||||
expect(names).toContain('hit_count');
|
||||
});
|
||||
});
|
||||
|
||||
describe('cacheRowId', () => {
|
||||
test('is deterministic across same input', () => {
|
||||
expect(cacheRowId('hello', 'default')).toBe(cacheRowId('hello', 'default'));
|
||||
});
|
||||
test('differs across source_id', () => {
|
||||
expect(cacheRowId('hello', 'a')).not.toBe(cacheRowId('hello', 'b'));
|
||||
});
|
||||
});
|
||||
|
||||
describe('SemanticQueryCache \u2014 store + lookup', () => {
|
||||
test('roundtrip: exact embedding match returns a hit', async () => {
|
||||
const cache = new SemanticQueryCache(engine);
|
||||
const emb = makeEmbedding(1);
|
||||
const results = [makeResult('a'), makeResult('b')];
|
||||
|
||||
await cache.store('what is foo', emb, results, META);
|
||||
const hit = await cache.lookup(emb);
|
||||
|
||||
expect(hit.hit).toBe(true);
|
||||
expect(hit.results).toHaveLength(2);
|
||||
expect(hit.results?.[0].slug).toBe('a');
|
||||
expect(hit.similarity).toBeGreaterThan(0.99);
|
||||
});
|
||||
|
||||
test('similar embedding (cosine > 0.92) is a hit', async () => {
|
||||
const cache = new SemanticQueryCache(engine);
|
||||
const base = makeEmbedding(100);
|
||||
|
||||
// Construct a near-neighbor: tweak a few dims so cosine stays > 0.92.
|
||||
const near = new Float32Array(base);
|
||||
for (let i = 0; i < 10; i++) near[i] += 0.005;
|
||||
// Re-normalize.
|
||||
let mag = 0;
|
||||
for (let i = 0; i < DIM; i++) mag += near[i] * near[i];
|
||||
mag = Math.sqrt(mag);
|
||||
for (let i = 0; i < DIM; i++) near[i] /= mag;
|
||||
|
||||
await cache.store('what is foo', base, [makeResult('a')], META);
|
||||
const hit = await cache.lookup(near);
|
||||
|
||||
expect(hit.hit).toBe(true);
|
||||
expect(hit.similarity).toBeGreaterThan(0.92);
|
||||
});
|
||||
|
||||
test('orthogonal embedding is a miss', async () => {
|
||||
const cache = new SemanticQueryCache(engine);
|
||||
const a = makeEmbedding(1);
|
||||
const b = makeOrthogonalEmbedding(2);
|
||||
await cache.store('q1', a, [makeResult('a')], META);
|
||||
const hit = await cache.lookup(b);
|
||||
expect(hit.hit).toBe(false);
|
||||
});
|
||||
});
|
||||
|
||||
describe('SemanticQueryCache \u2014 TTL', () => {
|
||||
test('stale row (past TTL) is not returned', async () => {
|
||||
const cache = new SemanticQueryCache(engine, { ttlSeconds: 1 });
|
||||
const emb = makeEmbedding(42);
|
||||
await cache.store('q', emb, [makeResult('a')], META, { ttlSeconds: 1 });
|
||||
|
||||
// Manually rewind created_at to simulate expiration.
|
||||
await engine.executeRaw(
|
||||
`UPDATE query_cache SET created_at = now() - interval '10 seconds'`,
|
||||
);
|
||||
const hit = await cache.lookup(emb);
|
||||
expect(hit.hit).toBe(false);
|
||||
});
|
||||
});
|
||||
|
||||
describe('SemanticQueryCache \u2014 source isolation', () => {
|
||||
test('different source_id cannot read each other\u2019s rows', async () => {
|
||||
const cache = new SemanticQueryCache(engine);
|
||||
const emb = makeEmbedding(7);
|
||||
await cache.store('q', emb, [makeResult('a')], META, { sourceId: 'src-A' });
|
||||
const hitB = await cache.lookup(emb, { sourceId: 'src-B' });
|
||||
expect(hitB.hit).toBe(false);
|
||||
const hitA = await cache.lookup(emb, { sourceId: 'src-A' });
|
||||
expect(hitA.hit).toBe(true);
|
||||
});
|
||||
});
|
||||
|
||||
describe('SemanticQueryCache \u2014 management', () => {
|
||||
test('clear() wipes all rows', async () => {
|
||||
const cache = new SemanticQueryCache(engine);
|
||||
const emb = makeEmbedding(9);
|
||||
await cache.store('q1', emb, [makeResult('a')], META);
|
||||
await cache.store('q2', makeEmbedding(10), [makeResult('b')], META);
|
||||
const removed = await cache.clear();
|
||||
expect(removed).toBeGreaterThanOrEqual(2);
|
||||
const stats = await cache.stats();
|
||||
expect(stats.total_rows).toBe(0);
|
||||
});
|
||||
|
||||
test('prune() deletes only stale rows', async () => {
|
||||
const cache = new SemanticQueryCache(engine);
|
||||
await cache.store('fresh', makeEmbedding(11), [makeResult('a')], META);
|
||||
await cache.store('stale', makeEmbedding(12), [makeResult('b')], META, { ttlSeconds: 1 });
|
||||
await engine.executeRaw(
|
||||
`UPDATE query_cache SET created_at = now() - interval '10 seconds' WHERE query_text = 'stale'`,
|
||||
);
|
||||
const removed = await cache.prune();
|
||||
expect(removed).toBe(1);
|
||||
const stats = await cache.stats();
|
||||
expect(stats.total_rows).toBe(1);
|
||||
expect(stats.fresh_rows).toBe(1);
|
||||
});
|
||||
|
||||
test('stats() reports fresh / stale / total / hit counters', async () => {
|
||||
const cache = new SemanticQueryCache(engine);
|
||||
const emb = makeEmbedding(13);
|
||||
await cache.store('q', emb, [makeResult('a')], META);
|
||||
await cache.lookup(emb); // bump hit
|
||||
// Hit bump is async/fire-and-forget; give it a moment to land.
|
||||
await new Promise(r => setTimeout(r, 50));
|
||||
const stats = await cache.stats();
|
||||
expect(stats.total_rows).toBe(1);
|
||||
expect(stats.fresh_rows).toBe(1);
|
||||
expect(stats.stale_rows).toBe(0);
|
||||
expect(stats.total_hits).toBeGreaterThanOrEqual(1);
|
||||
});
|
||||
});
|
||||
|
||||
describe('SemanticQueryCache \u2014 disabled', () => {
|
||||
test('disabled cache is a pure no-op on lookup', async () => {
|
||||
const cache = new SemanticQueryCache(engine, { enabled: false });
|
||||
const emb = makeEmbedding(99);
|
||||
await cache.store('q', emb, [makeResult('a')], META);
|
||||
// Even after a store call, lookup must miss because enabled=false.
|
||||
const hit = await cache.lookup(emb);
|
||||
expect(hit.hit).toBe(false);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,317 @@
|
||||
/**
|
||||
* Pins the v0.32.3 search-lite mode core: MODE_BUNDLES + resolveSearchMode
|
||||
* + knobsHash. The 3x7 mode table is asserted cell-by-cell because the
|
||||
* public eval methodology doc cites these values verbatim — drift here is
|
||||
* a documentation-honesty bug, not a refactor.
|
||||
*/
|
||||
import { describe, expect, test } from 'bun:test';
|
||||
import {
|
||||
MODE_BUNDLES,
|
||||
SEARCH_MODES,
|
||||
DEFAULT_SEARCH_MODE,
|
||||
isSearchMode,
|
||||
resolveSearchMode,
|
||||
attributeKnob,
|
||||
knobsHash,
|
||||
loadOverridesFromConfig,
|
||||
KNOBS_HASH_VERSION,
|
||||
SEARCH_MODE_CONFIG_KEYS,
|
||||
type SearchMode,
|
||||
} from '../src/core/search/mode.ts';
|
||||
|
||||
describe('SEARCH_MODES + MODE_BUNDLES canonical shape', () => {
|
||||
test('SEARCH_MODES is exactly the 3 expected values', () => {
|
||||
expect([...SEARCH_MODES]).toEqual(['conservative', 'balanced', 'tokenmax']);
|
||||
});
|
||||
|
||||
test('DEFAULT_SEARCH_MODE is balanced (matches v0.31.x current default surface)', () => {
|
||||
expect(DEFAULT_SEARCH_MODE).toBe('balanced');
|
||||
});
|
||||
|
||||
test('MODE_BUNDLES is frozen (cannot be mutated)', () => {
|
||||
expect(Object.isFrozen(MODE_BUNDLES)).toBe(true);
|
||||
expect(Object.isFrozen(MODE_BUNDLES.conservative)).toBe(true);
|
||||
expect(Object.isFrozen(MODE_BUNDLES.balanced)).toBe(true);
|
||||
expect(Object.isFrozen(MODE_BUNDLES.tokenmax)).toBe(true);
|
||||
});
|
||||
|
||||
// The 3x7 cell-by-cell assertion. The methodology doc cites these.
|
||||
test('conservative bundle values are canonical', () => {
|
||||
expect(MODE_BUNDLES.conservative).toEqual({
|
||||
cache_enabled: true,
|
||||
cache_similarity_threshold: 0.92,
|
||||
cache_ttl_seconds: 3600,
|
||||
intentWeighting: true,
|
||||
tokenBudget: 4000,
|
||||
expansion: false,
|
||||
searchLimit: 10,
|
||||
});
|
||||
});
|
||||
|
||||
test('balanced bundle values are canonical', () => {
|
||||
expect(MODE_BUNDLES.balanced).toEqual({
|
||||
cache_enabled: true,
|
||||
cache_similarity_threshold: 0.92,
|
||||
cache_ttl_seconds: 3600,
|
||||
intentWeighting: true,
|
||||
tokenBudget: 12000,
|
||||
expansion: false,
|
||||
searchLimit: 25,
|
||||
});
|
||||
});
|
||||
|
||||
test('tokenmax bundle values are canonical (NOTE: limit=50, NOT current=20)', () => {
|
||||
expect(MODE_BUNDLES.tokenmax).toEqual({
|
||||
cache_enabled: true,
|
||||
cache_similarity_threshold: 0.92,
|
||||
cache_ttl_seconds: 3600,
|
||||
intentWeighting: true,
|
||||
tokenBudget: undefined,
|
||||
expansion: true,
|
||||
searchLimit: 50,
|
||||
});
|
||||
});
|
||||
|
||||
test('cache_enabled is true in every mode (free win)', () => {
|
||||
for (const m of SEARCH_MODES) {
|
||||
expect(MODE_BUNDLES[m].cache_enabled).toBe(true);
|
||||
}
|
||||
});
|
||||
|
||||
test('intentWeighting is true in every mode (zero-LLM cost)', () => {
|
||||
for (const m of SEARCH_MODES) {
|
||||
expect(MODE_BUNDLES[m].intentWeighting).toBe(true);
|
||||
}
|
||||
});
|
||||
|
||||
test('tokenBudget escalates: 4000 → 12000 → undefined', () => {
|
||||
expect(MODE_BUNDLES.conservative.tokenBudget).toBe(4000);
|
||||
expect(MODE_BUNDLES.balanced.tokenBudget).toBe(12000);
|
||||
expect(MODE_BUNDLES.tokenmax.tokenBudget).toBeUndefined();
|
||||
});
|
||||
|
||||
test('searchLimit escalates: 10 → 25 → 50', () => {
|
||||
expect(MODE_BUNDLES.conservative.searchLimit).toBe(10);
|
||||
expect(MODE_BUNDLES.balanced.searchLimit).toBe(25);
|
||||
expect(MODE_BUNDLES.tokenmax.searchLimit).toBe(50);
|
||||
});
|
||||
});
|
||||
|
||||
describe('isSearchMode', () => {
|
||||
test('accepts every documented mode', () => {
|
||||
for (const m of SEARCH_MODES) {
|
||||
expect(isSearchMode(m)).toBe(true);
|
||||
}
|
||||
});
|
||||
test('rejects unknown strings, numbers, null, undefined', () => {
|
||||
expect(isSearchMode('conservativeX')).toBe(false);
|
||||
expect(isSearchMode('')).toBe(false);
|
||||
expect(isSearchMode('CONSERVATIVE')).toBe(false); // case-sensitive at the type guard layer
|
||||
expect(isSearchMode(42)).toBe(false);
|
||||
expect(isSearchMode(null)).toBe(false);
|
||||
expect(isSearchMode(undefined)).toBe(false);
|
||||
});
|
||||
});
|
||||
|
||||
describe('resolveSearchMode resolution chain', () => {
|
||||
test('no inputs → balanced bundle (fallback)', () => {
|
||||
const r = resolveSearchMode({});
|
||||
expect(r.resolved_mode).toBe('balanced');
|
||||
expect(r.mode_valid).toBe(false);
|
||||
expect(r.searchLimit).toBe(25);
|
||||
expect(r.tokenBudget).toBe(12000);
|
||||
expect(r.expansion).toBe(false);
|
||||
});
|
||||
|
||||
test('valid mode picked, no overrides → bundle values pass through', () => {
|
||||
const r = resolveSearchMode({ mode: 'conservative' });
|
||||
expect(r.resolved_mode).toBe('conservative');
|
||||
expect(r.mode_valid).toBe(true);
|
||||
expect(r.searchLimit).toBe(10);
|
||||
expect(r.tokenBudget).toBe(4000);
|
||||
});
|
||||
|
||||
test('invalid mode string → balanced fallback (mode_valid=false)', () => {
|
||||
const r = resolveSearchMode({ mode: 'NUKE_MODE' });
|
||||
expect(r.resolved_mode).toBe('balanced');
|
||||
expect(r.mode_valid).toBe(false);
|
||||
expect(r.searchLimit).toBe(25);
|
||||
});
|
||||
|
||||
test('mode string case-normalized (TokenMax → tokenmax)', () => {
|
||||
const r = resolveSearchMode({ mode: 'TokenMax' });
|
||||
expect(r.resolved_mode).toBe('tokenmax');
|
||||
expect(r.mode_valid).toBe(true);
|
||||
});
|
||||
|
||||
test('per-key override wins over mode bundle (CDX-5 chain)', () => {
|
||||
const r = resolveSearchMode({
|
||||
mode: 'conservative',
|
||||
overrides: { tokenBudget: 99999, cache_enabled: false },
|
||||
});
|
||||
expect(r.resolved_mode).toBe('conservative');
|
||||
expect(r.tokenBudget).toBe(99999);
|
||||
expect(r.cache_enabled).toBe(false);
|
||||
expect(r.searchLimit).toBe(10); // not overridden, still from bundle
|
||||
});
|
||||
|
||||
test('per-call override wins over per-key override', () => {
|
||||
const r = resolveSearchMode({
|
||||
mode: 'conservative',
|
||||
overrides: { tokenBudget: 99999 },
|
||||
perCall: { tokenBudget: 77 },
|
||||
});
|
||||
expect(r.tokenBudget).toBe(77);
|
||||
});
|
||||
|
||||
test('per-call false-y values (false / 0) still beat fallback', () => {
|
||||
const r = resolveSearchMode({
|
||||
mode: 'tokenmax',
|
||||
perCall: { expansion: false, cache_enabled: false },
|
||||
});
|
||||
expect(r.expansion).toBe(false); // beat tokenmax's true
|
||||
expect(r.cache_enabled).toBe(false); // beat tokenmax's true
|
||||
});
|
||||
|
||||
test('undefined fields in perCall fall through (not coerced to false)', () => {
|
||||
const r = resolveSearchMode({
|
||||
mode: 'tokenmax',
|
||||
perCall: { tokenBudget: undefined, expansion: undefined },
|
||||
});
|
||||
expect(r.tokenBudget).toBeUndefined(); // from tokenmax bundle
|
||||
expect(r.expansion).toBe(true); // from tokenmax bundle, NOT overridden
|
||||
});
|
||||
});
|
||||
|
||||
describe('attributeKnob source attribution', () => {
|
||||
test('per-call source labeled correctly', () => {
|
||||
const input = { mode: 'conservative', perCall: { tokenBudget: 999 } };
|
||||
const resolved = resolveSearchMode(input);
|
||||
const a = attributeKnob('tokenBudget', input, resolved);
|
||||
expect(a.source).toBe('per-call');
|
||||
expect(a.value).toBe(999);
|
||||
});
|
||||
|
||||
test('override source labels the config key path', () => {
|
||||
const input = { mode: 'conservative', overrides: { cache_enabled: false } };
|
||||
const resolved = resolveSearchMode(input);
|
||||
const a = attributeKnob('cache_enabled', input, resolved);
|
||||
expect(a.source).toBe('override');
|
||||
expect(a.source_detail).toContain('search.cache_enabled');
|
||||
});
|
||||
|
||||
test('mode source labels the mode name', () => {
|
||||
const input = { mode: 'conservative' };
|
||||
const resolved = resolveSearchMode(input);
|
||||
const a = attributeKnob('searchLimit', input, resolved);
|
||||
expect(a.source).toBe('mode');
|
||||
expect(a.source_detail).toContain('conservative');
|
||||
});
|
||||
|
||||
test('fallback source labels the unset state explicitly', () => {
|
||||
const input = {}; // no mode set
|
||||
const resolved = resolveSearchMode(input);
|
||||
const a = attributeKnob('searchLimit', input, resolved);
|
||||
expect(a.source).toBe('fallback');
|
||||
expect(a.source_detail).toContain('balanced');
|
||||
expect(a.source_detail).toContain('unset');
|
||||
});
|
||||
});
|
||||
|
||||
describe('knobsHash determinism + cross-mode separation (CDX-4)', () => {
|
||||
test('hash is deterministic across calls', () => {
|
||||
const knobs = resolveSearchMode({ mode: 'conservative' });
|
||||
const h1 = knobsHash(knobs);
|
||||
const h2 = knobsHash(knobs);
|
||||
expect(h1).toBe(h2);
|
||||
});
|
||||
|
||||
test('different modes produce different hashes', () => {
|
||||
const c = knobsHash(resolveSearchMode({ mode: 'conservative' }));
|
||||
const b = knobsHash(resolveSearchMode({ mode: 'balanced' }));
|
||||
const t = knobsHash(resolveSearchMode({ mode: 'tokenmax' }));
|
||||
expect(c).not.toBe(b);
|
||||
expect(b).not.toBe(t);
|
||||
expect(c).not.toBe(t);
|
||||
});
|
||||
|
||||
test('per-call override changes the hash (cache key bifurcates)', () => {
|
||||
const a = knobsHash(resolveSearchMode({ mode: 'conservative' }));
|
||||
const b = knobsHash(resolveSearchMode({ mode: 'conservative', perCall: { tokenBudget: 999 } }));
|
||||
expect(a).not.toBe(b);
|
||||
});
|
||||
|
||||
test('hash is short (16 hex chars) and stable shape', () => {
|
||||
const h = knobsHash(resolveSearchMode({ mode: 'balanced' }));
|
||||
expect(h).toMatch(/^[0-9a-f]{16}$/);
|
||||
});
|
||||
|
||||
test('KNOBS_HASH_VERSION constant exposed for migrations to bump on schema change', () => {
|
||||
expect(KNOBS_HASH_VERSION).toBe(1);
|
||||
});
|
||||
});
|
||||
|
||||
describe('loadOverridesFromConfig flat-map parser', () => {
|
||||
test('empty config map → empty overrides', () => {
|
||||
const ov = loadOverridesFromConfig({});
|
||||
expect(ov).toEqual({});
|
||||
});
|
||||
|
||||
test('cache.enabled accepts 1 / 0 / true / false strings', () => {
|
||||
expect(loadOverridesFromConfig({ 'search.cache.enabled': '1' }).cache_enabled).toBe(true);
|
||||
expect(loadOverridesFromConfig({ 'search.cache.enabled': '0' }).cache_enabled).toBe(false);
|
||||
expect(loadOverridesFromConfig({ 'search.cache.enabled': 'true' }).cache_enabled).toBe(true);
|
||||
expect(loadOverridesFromConfig({ 'search.cache.enabled': 'false' }).cache_enabled).toBe(false);
|
||||
expect(loadOverridesFromConfig({ 'search.cache.enabled': 'TRUE' }).cache_enabled).toBe(true);
|
||||
});
|
||||
|
||||
test('numeric keys parse and clamp', () => {
|
||||
expect(loadOverridesFromConfig({ 'search.cache.similarity_threshold': '0.95' }).cache_similarity_threshold).toBe(0.95);
|
||||
expect(loadOverridesFromConfig({ 'search.cache.ttl_seconds': '7200' }).cache_ttl_seconds).toBe(7200);
|
||||
expect(loadOverridesFromConfig({ 'search.tokenBudget': '8000' }).tokenBudget).toBe(8000);
|
||||
expect(loadOverridesFromConfig({ 'search.searchLimit': '30' }).searchLimit).toBe(30);
|
||||
});
|
||||
|
||||
test('invalid numerics are ignored (not coerced to NaN/0)', () => {
|
||||
const ov = loadOverridesFromConfig({
|
||||
'search.cache.similarity_threshold': 'NaN',
|
||||
'search.tokenBudget': 'cheese',
|
||||
'search.searchLimit': '-1',
|
||||
'search.cache.ttl_seconds': '0',
|
||||
});
|
||||
expect(ov.cache_similarity_threshold).toBeUndefined();
|
||||
expect(ov.tokenBudget).toBeUndefined();
|
||||
expect(ov.searchLimit).toBeUndefined();
|
||||
expect(ov.cache_ttl_seconds).toBeUndefined();
|
||||
});
|
||||
|
||||
test('similarity_threshold rejects values outside (0, 1]', () => {
|
||||
expect(loadOverridesFromConfig({ 'search.cache.similarity_threshold': '1.5' }).cache_similarity_threshold).toBeUndefined();
|
||||
expect(loadOverridesFromConfig({ 'search.cache.similarity_threshold': '0' }).cache_similarity_threshold).toBeUndefined();
|
||||
expect(loadOverridesFromConfig({ 'search.cache.similarity_threshold': '-0.1' }).cache_similarity_threshold).toBeUndefined();
|
||||
});
|
||||
});
|
||||
|
||||
describe('SEARCH_MODE_CONFIG_KEYS is the full reset surface', () => {
|
||||
test('every key starts with search. prefix (gbrain config unset --pattern search.* compatibility)', () => {
|
||||
for (const k of SEARCH_MODE_CONFIG_KEYS) {
|
||||
expect(k.startsWith('search.')).toBe(true);
|
||||
}
|
||||
});
|
||||
|
||||
test('every ModeBundle field has a config key (consistency check)', () => {
|
||||
// If a new knob is added to ModeBundle, this test fails until the operator
|
||||
// adds the corresponding config key to SEARCH_MODE_CONFIG_KEYS. That's the
|
||||
// intentional regression guard: `gbrain search modes --reset` must clear
|
||||
// every knob.
|
||||
const knobs = Object.keys(MODE_BUNDLES.balanced);
|
||||
expect(SEARCH_MODE_CONFIG_KEYS.length).toBeGreaterThanOrEqual(knobs.length);
|
||||
});
|
||||
});
|
||||
|
||||
describe('Type-only smoke test (compiler sees SearchMode union)', () => {
|
||||
test('SearchMode union is exactly 3 modes (compile-time)', () => {
|
||||
const valid: SearchMode[] = ['conservative', 'balanced', 'tokenmax'];
|
||||
expect(valid.length).toBe(3);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,243 @@
|
||||
/**
|
||||
* v0.32.3 search-lite telemetry rollup writer tests.
|
||||
*
|
||||
* Pins the architecture decisions from D2 + [CDX-17] + [CDX-18]:
|
||||
* - In-memory bucket flushed periodically (NOT per-call DB write)
|
||||
* - Sums + counts, NEVER pre-averaged columns
|
||||
* - Date-bucketed cache hit/miss derivable over --days N window
|
||||
* - ON CONFLICT DO UPDATE adds raw values (concurrent flushes accumulate)
|
||||
* - Per-bucket isolation: one bad row doesn't lose the others
|
||||
*/
|
||||
import { afterAll, beforeAll, beforeEach, describe, expect, test } from 'bun:test';
|
||||
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
|
||||
import {
|
||||
recordSearchTelemetry,
|
||||
readSearchStats,
|
||||
getTelemetryWriter,
|
||||
_resetTelemetryWriterForTest,
|
||||
} from '../src/core/search/telemetry.ts';
|
||||
import type { HybridSearchMeta } from '../src/core/types.ts';
|
||||
|
||||
let engine: PGLiteEngine;
|
||||
|
||||
beforeAll(async () => {
|
||||
engine = new PGLiteEngine();
|
||||
await engine.connect({});
|
||||
await engine.initSchema();
|
||||
});
|
||||
|
||||
afterAll(async () => {
|
||||
await engine.disconnect();
|
||||
});
|
||||
|
||||
beforeEach(async () => {
|
||||
_resetTelemetryWriterForTest();
|
||||
await engine.executeRaw('DELETE FROM search_telemetry');
|
||||
});
|
||||
|
||||
const makeMeta = (overrides: Partial<HybridSearchMeta> = {}): HybridSearchMeta => ({
|
||||
vector_enabled: true,
|
||||
detail_resolved: null,
|
||||
expansion_applied: false,
|
||||
intent: 'general',
|
||||
mode: 'balanced',
|
||||
...overrides,
|
||||
});
|
||||
|
||||
describe('recordSearchTelemetry — in-memory bucket', () => {
|
||||
test('first record creates a bucket; record() never blocks the caller', () => {
|
||||
const w = getTelemetryWriter();
|
||||
w.setEngine(engine);
|
||||
expect(w.bucketCountForTest()).toBe(0);
|
||||
recordSearchTelemetry(engine, makeMeta(), { results_count: 5 });
|
||||
expect(w.bucketCountForTest()).toBe(1);
|
||||
});
|
||||
|
||||
test('same (date, mode, intent) accumulates into one bucket', () => {
|
||||
const w = getTelemetryWriter();
|
||||
w.setEngine(engine);
|
||||
recordSearchTelemetry(engine, makeMeta(), { results_count: 5 });
|
||||
recordSearchTelemetry(engine, makeMeta(), { results_count: 7 });
|
||||
recordSearchTelemetry(engine, makeMeta(), { results_count: 3 });
|
||||
expect(w.bucketCountForTest()).toBe(1);
|
||||
const today = new Date().toISOString().slice(0, 10);
|
||||
const b = w.bucketForTest(today, 'balanced', 'general');
|
||||
expect(b?.count).toBe(3);
|
||||
expect(b?.sum_results).toBe(15); // 5 + 7 + 3
|
||||
});
|
||||
|
||||
test('different modes / intents create distinct buckets', () => {
|
||||
const w = getTelemetryWriter();
|
||||
w.setEngine(engine);
|
||||
recordSearchTelemetry(engine, makeMeta({ mode: 'conservative', intent: 'entity' }), { results_count: 2 });
|
||||
recordSearchTelemetry(engine, makeMeta({ mode: 'tokenmax', intent: 'temporal' }), { results_count: 9 });
|
||||
recordSearchTelemetry(engine, makeMeta({ mode: 'balanced', intent: 'event' }), { results_count: 4 });
|
||||
expect(w.bucketCountForTest()).toBe(3);
|
||||
});
|
||||
|
||||
test('cache_hit / cache_miss counters fire from meta.cache.status', () => {
|
||||
const w = getTelemetryWriter();
|
||||
w.setEngine(engine);
|
||||
recordSearchTelemetry(engine, makeMeta({ cache: { status: 'hit' } }));
|
||||
recordSearchTelemetry(engine, makeMeta({ cache: { status: 'hit' } }));
|
||||
recordSearchTelemetry(engine, makeMeta({ cache: { status: 'miss' } }));
|
||||
recordSearchTelemetry(engine, makeMeta({ cache: { status: 'disabled' } }));
|
||||
const today = new Date().toISOString().slice(0, 10);
|
||||
const b = w.bucketForTest(today, 'balanced', 'general')!;
|
||||
expect(b.cache_hit).toBe(2);
|
||||
expect(b.cache_miss).toBe(1);
|
||||
expect(b.count).toBe(4);
|
||||
});
|
||||
|
||||
test('sum_budget_dropped accumulates from meta.token_budget.dropped', () => {
|
||||
const w = getTelemetryWriter();
|
||||
w.setEngine(engine);
|
||||
recordSearchTelemetry(engine, makeMeta({ token_budget: { budget: 4000, used: 3800, kept: 8, dropped: 12 } }));
|
||||
recordSearchTelemetry(engine, makeMeta({ token_budget: { budget: 4000, used: 4000, kept: 10, dropped: 7 } }));
|
||||
const today = new Date().toISOString().slice(0, 10);
|
||||
const b = w.bucketForTest(today, 'balanced', 'general')!;
|
||||
expect(b.sum_budget_dropped).toBe(19); // 12 + 7
|
||||
});
|
||||
|
||||
test('missing mode / intent fall back to "unset" — telemetry is non-blocking', () => {
|
||||
const w = getTelemetryWriter();
|
||||
w.setEngine(engine);
|
||||
recordSearchTelemetry(engine, { vector_enabled: true, detail_resolved: null, expansion_applied: false });
|
||||
const today = new Date().toISOString().slice(0, 10);
|
||||
expect(w.bucketForTest(today, 'unset', 'unset')).not.toBeNull();
|
||||
});
|
||||
});
|
||||
|
||||
describe('flush() writes to search_telemetry', () => {
|
||||
test('flush drains the bucket map atomically', async () => {
|
||||
const w = getTelemetryWriter();
|
||||
w.setEngine(engine);
|
||||
recordSearchTelemetry(engine, makeMeta(), { results_count: 5 });
|
||||
recordSearchTelemetry(engine, makeMeta(), { results_count: 7 });
|
||||
expect(w.bucketCountForTest()).toBe(1);
|
||||
await w.flush();
|
||||
expect(w.bucketCountForTest()).toBe(0);
|
||||
|
||||
const rows = await engine.executeRaw<{ count: number; sum_results: number }>(
|
||||
'SELECT count, sum_results FROM search_telemetry',
|
||||
);
|
||||
expect(rows.length).toBe(1);
|
||||
expect(rows[0].count).toBe(2);
|
||||
expect(rows[0].sum_results).toBe(12);
|
||||
});
|
||||
|
||||
test('ON CONFLICT DO UPDATE adds raw values (concurrent-flush semantics)', async () => {
|
||||
const w = getTelemetryWriter();
|
||||
w.setEngine(engine);
|
||||
|
||||
// First flush: 3 calls under balanced/general.
|
||||
recordSearchTelemetry(engine, makeMeta(), { results_count: 5 });
|
||||
recordSearchTelemetry(engine, makeMeta(), { results_count: 5 });
|
||||
recordSearchTelemetry(engine, makeMeta(), { results_count: 5 });
|
||||
await w.flush();
|
||||
|
||||
// Second flush: 2 more calls under same (date, mode, intent).
|
||||
recordSearchTelemetry(engine, makeMeta(), { results_count: 10 });
|
||||
recordSearchTelemetry(engine, makeMeta(), { results_count: 10 });
|
||||
await w.flush();
|
||||
|
||||
const rows = await engine.executeRaw<{ count: number; sum_results: number }>(
|
||||
'SELECT count, sum_results FROM search_telemetry',
|
||||
);
|
||||
expect(rows.length).toBe(1);
|
||||
expect(rows[0].count).toBe(5); // 3 + 2
|
||||
expect(rows[0].sum_results).toBe(35); // (5+5+5) + (10+10)
|
||||
});
|
||||
|
||||
test('flush is no-op when bucket map is empty', async () => {
|
||||
const w = getTelemetryWriter();
|
||||
w.setEngine(engine);
|
||||
await w.flush(); // no records, no rows
|
||||
const rows = await engine.executeRaw<{ n: number }>('SELECT COUNT(*)::int AS n FROM search_telemetry');
|
||||
expect(rows[0].n).toBe(0);
|
||||
});
|
||||
|
||||
test('concurrent flush() calls coalesce (flushInFlight reuse)', async () => {
|
||||
const w = getTelemetryWriter();
|
||||
w.setEngine(engine);
|
||||
recordSearchTelemetry(engine, makeMeta(), { results_count: 1 });
|
||||
// Two simultaneous flush() awaits → both observe the same underlying drain.
|
||||
const [a, b] = await Promise.all([w.flush(), w.flush()]);
|
||||
expect(a).toBeUndefined();
|
||||
expect(b).toBeUndefined();
|
||||
const rows = await engine.executeRaw<{ count: number }>('SELECT count FROM search_telemetry');
|
||||
expect(rows[0].count).toBe(1); // not doubled
|
||||
});
|
||||
});
|
||||
|
||||
describe('readSearchStats — read-time derived averages', () => {
|
||||
test('empty table → all-zero stats', async () => {
|
||||
const s = await readSearchStats(engine, { days: 7 });
|
||||
expect(s.total_calls).toBe(0);
|
||||
expect(s.cache_hit_rate).toBe(0);
|
||||
expect(s.avg_results).toBe(0);
|
||||
});
|
||||
|
||||
test('one bucket flushed → stats derive averages from sums/counts', async () => {
|
||||
const w = getTelemetryWriter();
|
||||
w.setEngine(engine);
|
||||
recordSearchTelemetry(engine, makeMeta(), { results_count: 10 });
|
||||
recordSearchTelemetry(engine, makeMeta(), { results_count: 20 });
|
||||
recordSearchTelemetry(engine, makeMeta(), { results_count: 30 });
|
||||
await w.flush();
|
||||
|
||||
const s = await readSearchStats(engine, { days: 7 });
|
||||
expect(s.total_calls).toBe(3);
|
||||
expect(s.avg_results).toBe(20); // (10 + 20 + 30) / 3 = 20
|
||||
});
|
||||
|
||||
test('cache_hit_rate computed from hits + misses (excludes disabled)', async () => {
|
||||
const w = getTelemetryWriter();
|
||||
w.setEngine(engine);
|
||||
recordSearchTelemetry(engine, makeMeta({ cache: { status: 'hit' } }));
|
||||
recordSearchTelemetry(engine, makeMeta({ cache: { status: 'hit' } }));
|
||||
recordSearchTelemetry(engine, makeMeta({ cache: { status: 'hit' } }));
|
||||
recordSearchTelemetry(engine, makeMeta({ cache: { status: 'miss' } }));
|
||||
recordSearchTelemetry(engine, makeMeta({ cache: { status: 'disabled' } }));
|
||||
await w.flush();
|
||||
|
||||
const s = await readSearchStats(engine, { days: 7 });
|
||||
expect(s.cache_hits).toBe(3);
|
||||
expect(s.cache_misses).toBe(1);
|
||||
expect(s.cache_hit_rate).toBeCloseTo(0.75, 5); // 3 / (3 + 1) = 0.75
|
||||
});
|
||||
|
||||
test('intent_distribution and mode_distribution surface counts', async () => {
|
||||
const w = getTelemetryWriter();
|
||||
w.setEngine(engine);
|
||||
recordSearchTelemetry(engine, makeMeta({ mode: 'conservative', intent: 'entity' }));
|
||||
recordSearchTelemetry(engine, makeMeta({ mode: 'conservative', intent: 'entity' }));
|
||||
recordSearchTelemetry(engine, makeMeta({ mode: 'tokenmax', intent: 'temporal' }));
|
||||
await w.flush();
|
||||
|
||||
const s = await readSearchStats(engine, { days: 7 });
|
||||
expect(s.intent_distribution.entity).toBe(2);
|
||||
expect(s.intent_distribution.temporal).toBe(1);
|
||||
expect(s.mode_distribution.conservative).toBe(2);
|
||||
expect(s.mode_distribution.tokenmax).toBe(1);
|
||||
});
|
||||
|
||||
test('days window clamps to [1, 365]', async () => {
|
||||
const a = await readSearchStats(engine, { days: 0 });
|
||||
expect(a.window_days).toBe(1);
|
||||
const b = await readSearchStats(engine, { days: 9999 });
|
||||
expect(b.window_days).toBe(365);
|
||||
const c = await readSearchStats(engine, {});
|
||||
expect(c.window_days).toBe(7); // default
|
||||
});
|
||||
|
||||
test('missing search_telemetry table → empty stats (graceful)', async () => {
|
||||
// Drop the table to simulate a pre-v0.32.3 brain.
|
||||
await engine.executeRaw('DROP TABLE IF EXISTS search_telemetry');
|
||||
const s = await readSearchStats(engine, { days: 7 });
|
||||
expect(s.total_calls).toBe(0);
|
||||
expect(s.cache_hit_rate).toBe(0);
|
||||
// Restore for subsequent tests in this describe block.
|
||||
await engine.initSchema();
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,123 @@
|
||||
/**
|
||||
* v0.32.x search-lite \u2014 token budget enforcement.
|
||||
*
|
||||
* Pure module under test (no DB, no LLM). Confirms:
|
||||
* - char/4 heuristic estimates correctly
|
||||
* - greedy walk preserves caller ordering
|
||||
* - meta accounting (used / kept / dropped) matches the actual cut
|
||||
* - undefined / <=0 budget is a no-op
|
||||
* - first-result-too-big returns empty list
|
||||
*
|
||||
* Lives in test/token-budget.test.ts to mirror existing search/* test naming.
|
||||
*/
|
||||
|
||||
import { describe, test, expect } from 'bun:test';
|
||||
import { enforceTokenBudget, estimateTokens, resultTokens } from '../src/core/search/token-budget.ts';
|
||||
import type { SearchResult } from '../src/core/types.ts';
|
||||
|
||||
function makeResult(overrides: Partial<SearchResult> = {}): SearchResult {
|
||||
return {
|
||||
slug: 'test-page',
|
||||
page_id: 1,
|
||||
title: 'Test Title',
|
||||
type: 'concept',
|
||||
chunk_text: 'test chunk text',
|
||||
chunk_source: 'compiled_truth',
|
||||
chunk_id: 1,
|
||||
chunk_index: 0,
|
||||
score: 1.0,
|
||||
stale: false,
|
||||
...overrides,
|
||||
};
|
||||
}
|
||||
|
||||
describe('estimateTokens', () => {
|
||||
test('empty / nullish strings cost 0 tokens', () => {
|
||||
expect(estimateTokens('')).toBe(0);
|
||||
expect(estimateTokens(undefined)).toBe(0);
|
||||
expect(estimateTokens(null)).toBe(0);
|
||||
});
|
||||
|
||||
test('rounds up so a single char still costs 1 token', () => {
|
||||
expect(estimateTokens('a')).toBe(1);
|
||||
expect(estimateTokens('abc')).toBe(1);
|
||||
expect(estimateTokens('abcd')).toBe(1);
|
||||
expect(estimateTokens('abcde')).toBe(2);
|
||||
});
|
||||
|
||||
test('scales linearly at ~4 chars / token', () => {
|
||||
// 100 chars \u2192 25 tokens
|
||||
expect(estimateTokens('x'.repeat(100))).toBe(25);
|
||||
// 401 chars \u2192 ceil(401/4) = 101
|
||||
expect(estimateTokens('x'.repeat(401))).toBe(101);
|
||||
});
|
||||
});
|
||||
|
||||
describe('resultTokens', () => {
|
||||
test('sums title + chunk_text', () => {
|
||||
const r = makeResult({ title: 'abcd', chunk_text: 'efgh' }); // 1 + 1 = 2
|
||||
expect(resultTokens(r)).toBe(2);
|
||||
});
|
||||
});
|
||||
|
||||
describe('enforceTokenBudget', () => {
|
||||
test('undefined budget is a no-op', () => {
|
||||
const results = [makeResult({ slug: 'a' }), makeResult({ slug: 'b' })];
|
||||
const { results: kept, meta } = enforceTokenBudget(results, undefined);
|
||||
expect(kept).toHaveLength(2);
|
||||
expect(meta.dropped).toBe(0);
|
||||
expect(meta.kept).toBe(2);
|
||||
expect(meta.budget).toBe(0); // safe-budget normalization
|
||||
});
|
||||
|
||||
test('zero / negative budget is a no-op', () => {
|
||||
const results = [makeResult({ slug: 'a' })];
|
||||
expect(enforceTokenBudget(results, 0).results).toHaveLength(1);
|
||||
expect(enforceTokenBudget(results, -5).results).toHaveLength(1);
|
||||
});
|
||||
|
||||
test('empty input returns empty', () => {
|
||||
const { results, meta } = enforceTokenBudget([], 100);
|
||||
expect(results).toHaveLength(0);
|
||||
expect(meta.kept).toBe(0);
|
||||
expect(meta.dropped).toBe(0);
|
||||
});
|
||||
|
||||
test('greedy top-down: stops as soon as cumulative cost would exceed budget', () => {
|
||||
// Each result: title 'a' (1 tok) + chunk_text 'xxxx' (1 tok) = 2 tokens each
|
||||
const results = [
|
||||
makeResult({ slug: 'a', title: 'a', chunk_text: 'xxxx' }),
|
||||
makeResult({ slug: 'b', title: 'a', chunk_text: 'xxxx' }),
|
||||
makeResult({ slug: 'c', title: 'a', chunk_text: 'xxxx' }),
|
||||
makeResult({ slug: 'd', title: 'a', chunk_text: 'xxxx' }),
|
||||
];
|
||||
// Budget 5 \u2192 fits 2 results (cost 2+2=4); a 3rd would push to 6.
|
||||
const { results: kept, meta } = enforceTokenBudget(results, 5);
|
||||
expect(kept).toHaveLength(2);
|
||||
expect(kept.map(r => r.slug)).toEqual(['a', 'b']);
|
||||
expect(meta.used).toBe(4);
|
||||
expect(meta.dropped).toBe(2);
|
||||
expect(meta.kept).toBe(2);
|
||||
expect(meta.budget).toBe(5);
|
||||
});
|
||||
|
||||
test('preserves caller ordering (never re-ranks)', () => {
|
||||
const results = [
|
||||
makeResult({ slug: 'low-score', score: 0.1, title: 'a', chunk_text: 'xxxx' }),
|
||||
makeResult({ slug: 'high-score', score: 0.9, title: 'a', chunk_text: 'xxxx' }),
|
||||
];
|
||||
const { results: kept } = enforceTokenBudget(results, 2);
|
||||
// 2 tokens fits exactly one result; budget pass should keep the FIRST,
|
||||
// even though it has a worse score \u2014 ordering is caller's contract.
|
||||
expect(kept).toHaveLength(1);
|
||||
expect(kept[0].slug).toBe('low-score');
|
||||
});
|
||||
|
||||
test('first result exceeds budget alone \u2192 returns empty', () => {
|
||||
const big = makeResult({ slug: 'big', title: 'a', chunk_text: 'x'.repeat(1000) });
|
||||
const small = makeResult({ slug: 'small', title: 'a', chunk_text: 'xxxx' });
|
||||
const { results: kept, meta } = enforceTokenBudget([big, small], 5);
|
||||
expect(kept).toHaveLength(0);
|
||||
expect(meta.dropped).toBe(2);
|
||||
});
|
||||
});
|
||||
@@ -59,10 +59,12 @@ describe('v0.31.8 — voyage Content-Length pre-check + per-item cap', () => {
|
||||
|
||||
test('Layer 1 throws on Content-Length over the cap (not silent return)', async () => {
|
||||
const source = await Bun.file(new URL('../src/core/ai/gateway.ts', import.meta.url)).text();
|
||||
// The cap check must use `throw new Error(...)` so the embed flow's
|
||||
// retry/backoff sees a failure, NOT a silent pass-through.
|
||||
// The cap check must throw a VoyageResponseTooLargeError so the inbound
|
||||
// try/catch at the bottom of voyageCompatFetch rethrows it (instead of
|
||||
// the pre-fix bare `catch {}` that swallowed `throw new Error(...)` and
|
||||
// returned the original response — making the cap theatrical).
|
||||
expect(source).toMatch(/exceeds[^`]*MAX_VOYAGE_RESPONSE_BYTES[^`]*bytes/);
|
||||
expect(source).toMatch(/throw new Error\([\s\S]{0,200}Voyage response Content-Length/);
|
||||
expect(source).toMatch(/throw new VoyageResponseTooLargeError\([\s\S]{0,200}Voyage response Content-Length/);
|
||||
});
|
||||
|
||||
test('Layer 2: per-embedding base64 cap fires inside the json.data iteration', async () => {
|
||||
@@ -70,7 +72,18 @@ describe('v0.31.8 — voyage Content-Length pre-check + per-item cap', () => {
|
||||
// Defense-in-depth: even when Content-Length header is missing
|
||||
// (chunked encoding), each embedding string is bounded.
|
||||
expect(source).toMatch(/item\.embedding\.length\s*\*\s*0\.75/);
|
||||
expect(source).toMatch(/throw new Error\([\s\S]{0,200}Voyage embedding base64/);
|
||||
expect(source).toMatch(/throw new VoyageResponseTooLargeError\([\s\S]{0,200}Voyage embedding base64/);
|
||||
});
|
||||
|
||||
test('inbound try/catch rethrows VoyageResponseTooLargeError (Codex P3 follow-up)', async () => {
|
||||
const source = await Bun.file(new URL('../src/core/ai/gateway.ts', import.meta.url)).text();
|
||||
// Critical structural invariant after the Codex P3 fix: the catch
|
||||
// around the inbound JSON-rewrite block MUST rethrow OOM-cap errors.
|
||||
// Pre-fix, a bare `catch {}` swallowed the throw and returned the
|
||||
// original (oversized) response, making Layer 2 ineffective. The
|
||||
// tagged class + instanceof check restores fail-loud behavior.
|
||||
expect(source).toContain('VoyageResponseTooLargeError');
|
||||
expect(source).toMatch(/if\s*\(\s*err\s+instanceof\s+VoyageResponseTooLargeError\s*\)\s*throw\s+err/);
|
||||
});
|
||||
|
||||
test('comment thread documents both layers + the cap-sizing decision', async () => {
|
||||
|
||||
Reference in New Issue
Block a user