mirror of
https://github.com/garrytan/gbrain.git
synced 2026-07-30 19:49:14 +00:00
18ec732e1b
Shape requested in #3475's closing review: make capOversizedChunks use a CJK-aware estimate instead of adding a parallel opt-in cap. Measurement (vs the Qwen3-Embedding tokenizer, the strict-backend class from #2826): cl100k matches embedding-family tokenizers on pure-ASCII source (identical counts on English prose and JSON) but undercounts MIXED CJK+ASCII chunks — −31% on URL-dense Korean text. The heuristic fallback (~3.5 chars/token) undercounts CJK ~2.5×. estimateEmbedTokens(): for chunks containing CJK, max(cl100k, per-char- class overestimate — CJK 1.0 / other non-ws 0.75 / ws 0.1). ASCII-only chunks short-circuit to estimateTokens verbatim (bit-identical, pinned); CJK-DOMINANT text is unchanged too (cl100k already exceeds the weighted form, so max() returns today's value — pinned). Only mixed-script chunks, the measured divergence class, estimate higher. Reuses cjk.ts's existing exports — no new module, no config. Also: only the empty-AST branch routed its fallback through capOversizedChunks. The no-language, parse-timeout, no-semantic-nodes (every JSON/YAML fence — their node types aren't in TOP_LEVEL_TYPES) and parse-throw branches shipped word-counted chunks unchecked, letting a 14K-char JSON fence emit ~2,700-token chunks past the 2,000 default cap. Hoist the cap into fallbackChunks so all five emission paths share the net. Hard-split slice budget becomes 1 char/token for CJK-bearing pieces (the weighted estimate can reach 1 token/char). Refs #2826 Co-authored-by: Time Attakc <89218912+time-attack@users.noreply.github.com>