2 Commits

Include renames

Author	SHA1	Message	Date
3 people	2504abe47f	v0.35.3.0 fix wave: extract_facts items + git --no-recurse-submodules placement (#1053) ... * refactor(mcp): centralize ParamDef→JSON Schema via shared paramDefToSchema Three duplicate inline mappers existed across the MCP surface: - src/mcp/tool-defs.ts (stdio MCP buildToolDefs) - src/commands/serve-http.ts:837 (live HTTP MCP tools/list) - src/core/minions/tools/brain-allowlist.ts:84 (subagent tool registry) Each had subtly different items propagation. The HTTP MCP variant dropped items entirely, leaving extract_facts.entity_hints broken for OAuth- authenticated remote agents even after a buildToolDefs-only patch. The subagent variant propagated one level of items but used the same shallow shape so nested arrays would silently drop. Extract a single recursive paramDefToSchema helper exported from src/mcp/tool-defs.ts and have all three mappers consume it. Closes the bug class at the architecture level instead of patching one site at a time. The helper copies type, description, enum, default, and recursively rebuilds items so array-of-arrays preserves inner shape. Key ordering (type, description, enum, default, items) matches the pre-v0.34 inline mappers so JSON.stringify output stays byte-stable for every existing operation that does not use nested arrays. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(schema): add items to extract_facts.entity_hints and handle-to-tweet candidates Two array fields shipped without the items property required by JSON Schema. Strict-mode validators (Gemini Pro structured outputs, OpenAI strict tool definitions) reject the entire schema when any type:'array' lacks items. Downstream agents on those providers couldn't use extract_facts or the x_handle_to_tweet resolver. extract_facts.entity_hints — declared items: { type: 'string' } matching the handler at src/core/operations.ts:2733 which already coerces the runtime value to string[]. handle_to_tweet outputSchema.candidates — full XTweetCandidate spec including required + additionalProperties: false. The XTweetCandidate TypeScript interface declares all five fields as required; without required in the JSON Schema, a validator would accept {} as a valid candidate. additionalProperties: false closes the OpenAI strict-mode contract. 19 community PRs (#1028 #999 #980 #979 #910 #904 #847 #832 #863 #862 #812 for entity_hints; #910 caught candidates) converged on these locations. This wave cherry-picks the deepest variant (#910 surfaced both bugs) and centralizes via the paramDefToSchema helper from the preceding commit so the live HTTP MCP tools/list path is also fixed. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Co-Authored-By: DmitryBMsk (PR #910) * fix(git-remote): move --no-recurse-submodules after the subcommand verb Git CLI accepts two flag positions: git [global -c flags] <subcommand> [subcommand flags] [args] Global -c config flags belong before the verb. Subcommand-specific flags (like --no-recurse-submodules) belong after. Pre-v0.34 GIT_SSRF_FLAGS spliced both kinds before the verb, so cloneRepo invoked: git -c http.followRedirects=false ... --no-recurse-submodules clone URL DIR Real git rejects this with exit 129 ("unknown option: --no-recurse-submodules") because --no-recurse-submodules is a clone subcommand flag, not a global config flag. Every remote-source clone broke in production from v0.28 onward. The fake-git harness in test/git-remote.test.ts exits 0 regardless of argv shape, which is why CI never caught it. Split GIT_SSRF_FLAGS (3 -c config flags, spread BEFORE the verb) from GIT_SSRF_SUBCOMMAND_FLAGS (--no-recurse-submodules, spread AFTER the verb). cloneRepo and pullRepo both spread the new constant after their respective verbs. The constant names signal the position rule so future additions land in the right place. 7 community PRs converged on this location (#1023 #1020 #985 #963 #846 #842 — #800 doesn't exist). This wave cherry-picks the semantic- constant approach from #846's GIT_SSRF_SUBCOMMAND_FLAGS name (the clearest signal of the position rule). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * test(mcp+git+resolvers): structural array-items + subcommand-position guards Three new tests / test groups close the bug classes the wave fixes: test/mcp-tool-defs.test.ts — recursive structural guard walks every operation's inputSchema and fails with a property path if any type:'array' lacks items.type. Explicit fixture assertions for extract_facts.entity_hints.items.type and a synthetic nested-array ParamDef pinning items.items.type recursion. Without the explicit fixtures the legacyInlineMap byte-equality test is mirror-theater — mirroring both sides of the equality preserves the blind spot. test/git-remote.test.ts — split snapshot test into GIT_SSRF_FLAGS (3 global -c entries) and GIT_SSRF_SUBCOMMAND_FLAGS (--no-recurse-submodules). cloneRepo + pullRepo argv tests now assert the subcommand flag appears AFTER the verb index. Pre-v0.34 the pinned argv slice prefix included --no-recurse-submodules, which baked the bug into the test suite (codex catch). test/resolvers.test.ts — recursive walk over both inputSchema AND outputSchema for builtin resolvers (xHandleToTweetResolver, urlReachableResolver). Explicit imports rather than getDefaultRegistry(), which starts empty until commands/resolvers.ts runs — codex catch on a hollow-walk failure mode. Dedicated case pins candidates items shape including required + additionalProperties. Reference legacyInlineMap in mcp-tool-defs.test.ts mirrors the new recursive paramDefToSchema helper. No current op uses nested arrays so the byte-equality test stays green for every existing operation. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * test(e2e): raise rerank timeouts for ZE live cold-start The first rerank call of a CI run hits ZeroEntropy's cold-start latency (observed ~5-6s on Tier 2 LLM Skills runners; subsequent calls < 500ms). Two timeouts fired simultaneously at ~5s: 1. bun:test's default 5000ms per-test timeout caused (fail). 2. gateway.rerank's DEFAULT_RERANK_TIMEOUT_MS = 5000 fired right after, reported as "Unhandled error between tests". The next rerank test (top_n=2) ran in 409ms because the API was already warm. Cold-start is the only issue. Pass explicit timeoutMs to each rerank() call and a longer per-test timeout (30s) on both ZE rerank tests. Production DEFAULT_RERANK_TIMEOUT_MS stays at 5s for the search hot path — these E2E tests bypass it locally without changing the default that protects user latency. Unrelated to the fix-wave in this PR (mcp-tool-defs + git-remote + resolver guards). Lands here to keep Tier 2 LLM Skills green. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore: bump version and changelog (v0.35.2.0) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs: sync for v0.35.2.0 Update CLAUDE.md Key files annotations for the v0.35.2.0 fix wave: - src/mcp/tool-defs.ts: document new exported recursive paramDefToSchema helper and the three-consumer centralization (stdio MCP, HTTP MCP tools/list, subagent registry). - src/core/minions/tools/brain-allowlist.ts: paramsToInputSchema now consumes the shared helper. - src/commands/serve-http.ts: tools/list handler now consumes the shared helper (closes the HTTP MCP items-dropped bug class). - src/core/git-remote.ts: new entry. Documents the GIT_SSRF_FLAGS (global config, pre-verb) vs GIT_SSRF_SUBCOMMAND_FLAGS (subcommand-scoped, post-verb) split, the 7-month silent regression, and the position-anchored regression guard in test/git-remote.test.ts. Regenerated llms-full.txt to match. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * chore: rebump version to v0.35.3.0 Queue moved while this PR was open — v0.35.2.0 was claimed by master's v0.35.1.0 sibling work. Advancing one slot. No code changes; only: - VERSION + package.json: 0.35.2.0 → 0.35.3.0 - CHANGELOG.md: rewritten header + inline references - CLAUDE.md: rewritten 4 key-file annotations - llms-full.txt + llms.txt: regenerated to mirror CLAUDE.md Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-17 08:00:29 -07:00
3 people	baf1a47798	v0.35.0.0 feat: ZeroEntropy zembed-1 + zerank-2 reranker (#1008) ... * feat(ai): add ZeroEntropy recipe + reranker touchpoint type Widens `TouchpointKind` with `'reranker'`, adds `RerankerTouchpoint` interface, extends `Recipe.touchpoints` and `AIGatewayConfig` to carry reranker model state. Registers `zeroentropyai` recipe (zembed-1 embeddings + zerank-{2,1,1-small} rerankers) in the recipe registry. Recipe declares the 7 Matryoshka dims (2560/1280/640/320/160/80/40), Voyage-style dense-payload hedge (chars_per_token=1, safety_factor=0.5), and 5MB rerank payload cap. Pinned by test/ai/zeroentropy-recipe.test.ts including F1 regression (implementation literal is 'openai-compatible') and F2 regression (base_url_default ends with /v1, no doubling). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(ai/dims): thread input_type 4th-arg + ZE flexible-dim allowlist `dimsProviderOptions` gains an optional `inputType?: 'query' | 'document'` 4th param so asymmetric providers (ZE zembed-1, Voyage v3+) can route query-side vs document-side encoding. Per-model filtering inside the openai-compatible branch keeps `input_type` from leaking to symmetric providers (OpenAI text-3, DashScope, Zhipu) that would 400 on it. Adds `ZEROENTROPY_VALID_DIMS` allowlist (2560/1280/640/320/160/80/40), `supportsZeroEntropyDimension(modelId)`, and `isValidZeroEntropyDim(dims)`. Throws `AIConfigError` with paste-ready fix hint when zembed-1 is configured with an invalid dim (most common: defaulting to 1536 from DEFAULT_EMBEDDING_DIMENSIONS). The 4th-arg is optional; existing call sites (1 production + N tests across Voyage/OpenAI/DashScope/Zhipu/MiniMax) compile unchanged. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(ai/gateway): zeroEntropyCompatFetch + embedQuery + gateway.rerank() Two seams land together because they share the same recipe + auth path. zeroEntropyCompatFetch handles ZE's non-OpenAI-compatible wire shape: - URL rewrite: SDK's `${base_url}/embeddings` -> `${base_url}/models/embed` - Body inject: `input_type` (default 'document'; 'query' when threaded via providerOptions) + explicit `encoding_format: 'float'` - Response rewrite: `{results: [{embedding}]}` -> `{data: [{embedding, index}]}` so the AI SDK's openai-compat schema validates - `usage.prompt_tokens` injected from `total_tokens` (Voyage hit the same SDK schema requirement at :655) - Layer 1 (Content-Length) + Layer 2 (per-embedding size) OOM caps via tagged `ZeroEntropyResponseTooLargeError` (kept separate from `VoyageResponseTooLargeError` because the Voyage cap tests do structural source-text greps pinning the Voyage name) - Wired in `instantiateEmbedding()` via the existing `recipe.id === 'voyage' ? voyageCompatFetch : ...` ternary pattern embedQuery(text) routes `inputType: 'query'` through dimsProviderOptions for the search hot path. Companion to embed(texts) which now takes an optional 2nd-arg inputType (defaults to undefined -> 'document' for asymmetric providers). gateway.rerank() is the new native HTTP path (no AI-SDK reranking abstraction). Resolves the configured reranker model via `getRerankerModel()` (new accessor), parses + asserts the model is in the recipe's touchpoint.reranker.models allowlist (CDX2-F11: assertTouchpoint does not enforce allowlists for openai-compatible recipes — rerank() does it directly). Posts to `${recipe.base_url}/models/rerank` with bearer auth. Returns `RerankResult[]` sorted by `relevanceScore`. Errors classify into `RerankError.reason: 'auth' | 'rate_limit' | 'network' | 'timeout' | 'payload_too_large' | 'unknown'`. 5s default timeout. Pre-flight payload guard rejects bodies over `recipe.max_payload_bytes` BEFORE any HTTP call so applyReranker can fail-open without burning a round-trip. `_rerankTransport` + `__setRerankTransportForTests` mirror the embed test seam. `AIGatewayConfig.reranker_model` + isAvailable('reranker') branch + configureGateway / reconfigureGatewayWithEngine extensions thread the reranker model through the same state path as embedding/expansion/chat. `applyResolveAuth` + `defaultResolveAuth` widen the touchpoint param to include `'reranker'`. `KnownTouchpointKey` + `getTouchpoint()` in model-resolver widen to cover `'reranker'`. Pinned by: - test/ai/embedQuery.test.ts (8): returns single Float32Array, threads input_type='query' for ZE, drops field for OpenAI text-3, back-compat: legacy embed() callers without 4th arg keep their previous Voyage no-input_type shape - test/ai/rerank.test.ts (21): URL (F2 regression — no /v1/v1/), body shape, bearer header, response parsing, error classification across 6 HTTP shapes, payload pre-flight (no transport call), allowlist enforcement - test/ai/zeroentropy-compat-fetch.test.ts (14): structural source assertions for the shim that mirror test/voyage-response-cap.test.ts — URL rewrite path, body injection, response rewrite, usage.prompt_tokens injection, OOM caps Layer 1 + Layer 2 + instanceof rethrow, instantiateEmbedding wiring branch Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(search): applyReranker + rerank-failure audit + hybrid wire-in src/core/search/rerank.ts — the call-site abstraction. Slices the top `opts.topNIn` deduped candidates, sends to gateway.rerank(), reorders by relevanceScore desc, appends the un-reranked tail in its original RRF order (recall protection). Fail-open on every RerankError.reason: logs via `logRerankFailure` and returns the input array unchanged. Stamps `rerank_score` onto reordered items. `topNOut: null` is the explicit "don't truncate" signal — distinct from `undefined` (fall through to mode bundle); pin in test (CDX2-F16). src/core/rerank-audit.ts — failure-only JSONL audit at `~/.gbrain/audit/rerank-failures-YYYY-Www.jsonl` (ISO-week rotation; mirrors `src/core/audit-slug-fallback.ts`). Exports `logRerankFailure` + `readRecentRerankFailures(days)`. **No `logRerankSuccess`** — CDX2-F22 deliberately drops success-event logging: writing once per tokenmax search is hot-path I/O churn AND success events leak query volume + timing into a local audit. The doctor check reads `search.reranker.enabled` first so "no events in window" gets interpreted correctly (disabled -> healthy by definition; enabled -> healthy because nothing failed). Query text is SHA-256-prefix-hashed (8 hex chars) for privacy. Honors `GBRAIN_AUDIT_DIR`. src/core/search/hybrid.ts — slots `applyReranker` between `dedupResults()` and `enforceTokenBudget()` in the main RRF path. Resolution: per-call `opts.reranker` overrides; otherwise pulled from the resolved mode bundle (tokenmax -> enabled, others -> disabled in commit 5). Cache rows store final reranked results; the bumped knobsHash (commit 5) ensures rows can't leak across reranker configs. src/core/types.ts — adds `SearchOpts.reranker` as a structural type so callers can pass per-call overrides; runtime type lives in src/core/search/rerank.ts (avoids circular import). Tests: - test/search/rerank.test.ts (14): reorder, tail preserve, fail-open on every error class, topNOut null vs number, score stamping, empty + enabled=false pass-through - test/rerank-audit.test.ts (10): JSONL round-trip, error_summary truncated to 200, corrupt rows skipped, missing dir -> [], ISO-week rotation walks current + previous week, no logRerankSuccess export (CDX2-F22 contract) - test/search/hybrid-reranker-integration.test.ts (6): reranker fires when enabled, doesn't when disabled, reorders correctly, preserves tail, stamps rerank_score, fail-opens on rerankerFn throw — uses PGLite + stubbed embed transport, no API keys Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(search/mode): reranker mode-bundle fields + KNOBS_HASH_VERSION v=2 Extends `ModeBundle` with five reranker fields: `reranker_enabled`, `reranker_model`, `reranker_top_n_in`, `reranker_top_n_out`, `reranker_timeout_ms`. Per-mode defaults: - conservative -> enabled=false (cost-sensitive) - balanced -> enabled=false (opt-in via search.reranker.enabled) - tokenmax -> enabled=true (the high-cost-tolerant tier; ~$0.0003/query) Defaults model to `zeroentropyai:zerank-2`, topNIn=30, topNOut=null (no truncate by default; preserves tokenmax's searchLimit=50 end-to-end per CDX2-F16), timeout_ms=5000. `SearchKeyOverrides` + `SearchPerCallOpts` + `resolveSearchMode.pick` all extend to thread the new fields through the resolution chain (per-call -> per-key config -> mode bundle -> default). `loadOverridesFromConfig` adds parsers for the five new `search.reranker.*` config keys. `top_n_out` parsing distinguishes three input shapes (CDX2-F15): key absent -> undefined (fall through to mode bundle) 'null'|'none'|empty -> explicit null (no truncate) positive integer -> that number `SEARCH_MODE_CONFIG_KEYS` extends so `gbrain search modes --reset` clears the reranker overrides too. **KNOBS_HASH_VERSION bumps 1 -> 2** (CDX1-F14). Five new entries appended to `parts[]` (append-only convention CDX2-F13; reordering existing fields would silently rebuild every existing cache row). Includes `reranker_timeout_ms` so a 5s -> 100ms change invalidates stale rows (CDX2-F14: more fail-opens = different search behavior). Mid-rolling-deploy note (CDX2-F12): v=1 and v=2 processes produce distinct cacheRowIds for the same (source_id, query_text). Expect a temporary hit-rate dip + cache-row doubling for hot queries. Clears naturally within `cache.ttl_seconds` (default 3600s). src/commands/search.ts extends `KNOB_DESCRIPTIONS` with five new entries so `gbrain search modes` renders them. test/search-mode.test.ts extends the three bundle fixtures and bumps the KNOBS_HASH_VERSION expectation to 2. Pinned by test/search/knobs-hash-reranker.test.ts (13): each of the 5 reranker fields independently flips the hash, top_n_out=null renders stable, append-only convention enforced via source-position assertion. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(doctor): probeRerankerConfig + reranker_health check `gbrain models doctor` gains two new probes: - `probeRerankerConfig` (zero-network) validates that the configured reranker model resolves through the recipe registry, that the recipe declares a `reranker` touchpoint, and that the model is in `touchpoint.models[]`. Direct allowlist check here — assertTouchpoint does not enforce allowlists for openai-compatible recipes (CDX2-F11). Surfaces paste-ready `gbrain config set search.reranker.model <zerank-2|zerank-1|zerank-1-small>` fix hint. - `probeRerankerReachability` (1-token-equivalent) sends a minimal `{query: "probe", documents: ["probe"]}` rerank to verify auth + URL. Failures classify via `classifyError` into auth/rate_limit/network/ unknown. Skipped silently when reranker is unconfigured. Also extends `probeEmbeddingConfig` with a `providerId === 'zeroentropyai'` branch that catches the silent-1536-default bug class for zembed-1 configurations (same posture as the existing Voyage branch). `ProbeResult.touchpoint` widens to include `'reranker_config'`. `gbrain doctor` adds `checkRerankerHealth` to both the abbreviated (doctorReportRemote) and full (runDoctor) check sets. Logic: 1) Read `search.reranker.enabled` first. Disabled + no failures => 'reranker disabled'. Enabled + no failures => healthy. 2) Walk last 7 days of ~/.gbrain/audit/rerank-failures-*.jsonl. 3) ANY auth failure warns (config-time problem the probe should have caught — surface it). 4) ANY payload_too_large failure warns (workload mismatch). 5) Transient (network/timeout/rate_limit) warns at >=5 in window. Below that they're noise; reranker fails open anyway. CDX2-F21 blind-spot fix: reading enabled state first means "no events" gets interpreted correctly — never confuses "never-used" with "success logging broken" (the latter is impossible because there is no success logging by design, CDX2-F22). Engine-agnostic; file-based + one config-key read. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * test(e2e): ZeroEntropy live API round-trip + wire into Tier 2 CI test/e2e/zeroentropy-live.test.ts exercises the full stack against the real api.zeroentropy.dev: embed (default 2560-dim + flexible 1280), embedQuery (asymmetric query side), batch embed (3 distinct vectors), rerank (3 docs sorted by relevance score, photosynthesis-relevant docs beat the irrelevant cat doc), rerank with topN truncation. Gated on `ZEROENTROPY_API_KEY`: every test prints `[skip]` and returns early without assertions when the env var is unset, so fork PRs and contributor machines without a ZE account stay green. CI wire-up: `.github/workflows/e2e.yml` Tier 2 step adds `test/e2e/zeroentropy-live.test.ts` to its `bun test` invocation and exposes `ZEROENTROPY_API_KEY: ${{ secrets.ZEROENTROPY_API_KEY }}` to the runner. The secret is set on garrytan/gbrain at the repo scope (separately from this commit — set via `gh secret set` so the value never lands in source). Tier 1 stays mechanical (no API keys); Tier 2 is the natural home for provider-live tests because it's already the API-keyed lane. Cost: each full run fires ~6 small HTTP calls totaling well under a cent at the published $0.025/1M-token rate. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * v0.33.3.0 feat: ZeroEntropy zembed-1 + zerank-2 reranker Release notes for the ZeroEntropy support wave: zembed-1 embeddings (flexible-dim 2560/1280/640/320/160/80/40, asymmetric input_type) and zerank-2 cross-encoder reranking land as a new openai-compatible recipe alongside OpenAI/Voyage. Reranker defaults ON for tokenmax mode, OFF for conservative/balanced (~$0.0003/query at tokenmax topNIn=30; rounding error vs the tier's $700/mo Opus pairing per the CLAUDE.md cost matrix). Search now ends with `RRF -> dedup -> reranker -> token-budget` when reranker is enabled; fails open to RRF order on any error class (audit-logged at ~/.gbrain/audit/rerank-failures-*.jsonl). `KNOBS_HASH_VERSION` bumps 1 -> 2 to fold reranker config into the query_cache row key. Rolling-deploy operators should expect a temporary cache hit-rate dip + cache-row doubling for hot queries (clears naturally within `cache.ttl_seconds`, default 3600s). Files in this commit are pure docs / version bump: - VERSION + package.json bump to 0.33.3.0 - CHANGELOG.md release-summary entry with "How to take advantage" block - CLAUDE.md Key Files annotations for the new recipe + rerank.ts + rerank-audit.ts + gateway extensions - docs/ai-providers/zeroentropy.md one-pager (setup, knob reference, failure observability, troubleshooting table) - skills/migrations/v0.33.3.md (purely informational: no required user action; reranker is opt-in everywhere, ZE embedding is opt-in) - llms-full.txt regenerated to match CLAUDE.md Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-15 06:48:36 -07:00