mirror of
https://github.com/garrytan/gbrain.git
synced 2026-07-27 22:15:33 +00:00
aa04988ff1
* fix: adaptive embed batch sizing for Voyage token limits Voyage's tokenizer is 3-4x denser than OpenAI tiktoken, causing batches of 50+ texts to exceed the 120K token-per-batch limit even when DB token counts (from tiktoken) suggest they'd fit. Changes: - Add max_batch_tokens to EmbeddingTouchpoint type (provider-declared limit) - Set Voyage recipe to 120K token limit - Gateway embed() now auto-splits batches using conservative char-to-token estimate (1:1 ratio, 80% budget utilization) - On token-limit errors, embedSubBatch recursively halves and retries (down to single-text batches before giving up) - Reduce embedding.ts BATCH_SIZE from 100 to 50 as a secondary guard - Add tests for batch splitting logic and error pattern matching Fixes infinite retry loops where the same oversized batch would fail repeatedly because WHERE embedding IS NULL re-fetches identical rows. * feat(ai): per-recipe chars_per_token + safety_factor on EmbeddingTouchpoint Voyage's tokenizer runs ~3-4× denser than OpenAI tiktoken on mixed content (code/JSON/CJK), so a global "1 char ≈ 1 token at 80%" estimate either overshoots Voyage's batch cap on dense payloads or kills OpenAI throughput. Move the policy onto the recipe. - types.ts: extend EmbeddingTouchpoint with optional chars_per_token (default 4) and safety_factor (default 0.8). Both only consulted when max_batch_tokens is also set. - voyage.ts: declare chars_per_token=1 + safety_factor=0.5 (60K char budget). * feat(ai/gateway): transport DI + adaptive shrink-on-miss + startup warning Architectural changes to make the embed pipeline testable through the public embed() seam (no private-function DI) and self-healing under tokenizer miscalibration. Per /codex outside-voice review of the original PR #680 plan. - Export splitByTokenBudget + isTokenLimitError as @internal pure helpers; the test file now imports the real functions instead of re-implementing them. - splitByTokenBudget takes chars_per_token as a third parameter (defaults to 4 for OpenAI density when omitted); 0/negative ratios fall back to default. - New __setEmbedTransportForTests(fn) seam — tests inject an embedMany stub and drive recursion / fast-path scenarios through the real embed() call. Production code never reads the override; resetGateway() restores the SDK. - New module-scoped _shrinkState Map<recipeId, {factor, consecutiveSuccesses}>: on token-limit miss, shrink the recipe's effective safety_factor by 0.5 (floor 0.05) so the next embed() pre-splits tighter; after 10 consecutive batch successes, heal back ×1.5 toward the recipe-declared ceiling. - Startup warning (once per process per recipe): configureGateway walks every registered recipe; any embedding touchpoint without max_batch_tokens (except the canonical OpenAI fast-path recipe) emits one stderr line. Future Cohere/Mistral/Jina recipes that forget the field re-create the v0.27 Voyage backfill loop — the warning catches it before traffic hits the cliff. - Embed an ASCII flow diagram in the embed() JSDoc covering the shrinkState + per-recipe budget computation. Test rewrite (23 cases): - Pure helpers: splitByTokenBudget chars_per_token threading, default fallback, isTokenLimitError pattern coverage including non-Error throwables. - Recursion via embed() with stubbed transport: halving + concat-in-order, order preservation across boundaries (slot-0 sentinel asserts mapping), terminal MIN_SUB_BATCH=1 throws normalized error (no infinite loop). - OpenAI fast path: transport called exactly once, no partition, no cross-recipe leakage of voyage shrink state. - Shrink-on-miss: first miss halves factor, floors at 0.05 under repeated misses, heals after wins, healing capped at recipe ceiling. - Startup warning: first call fires once per recipe; subsequent configureGateway calls suppressed within the same process. * chore(embedding): revert BATCH_SIZE 50→100 The PR initially dropped BATCH_SIZE to 50 as a safety guard for Voyage's batch cap, but that halved OpenAI throughput on every embed page even though OpenAI has no such cap. With per-recipe pre-split + recursive halving + adaptive shrink-on-miss now living in the gateway, the outer paginator goes back to its original purpose: progress-callback granularity, not batch protection. * chore: bump version and changelog (v0.28.7) Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * docs: annotate v0.28.7 changes in CLAUDE.md key files --------- Co-authored-by: garrytan-agents <garrytan-agents@users.noreply.github.com> Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>