3 people 88287775e5 fix(chunker): estimated-token hard cap — URL-dense/CJK-fallback chunks overflow strict embedding-server token limits ...

Takeover of #2847 (rebased onto current master). Fixes #2826.

- cjk.ts: estimateEmbeddingTokens() — conservative per-char-class token
  estimate (CJK 1.0, other 0.75, whitespace 0.1 per code unit).
- recursive.ts (MARKDOWN_CHUNKER_VERSION 3→4): countWords floored at
  ceil(nonWhitespaceChars/6); capByEstimatedTokens() final pass with
  ChunkOptions.maxTokens (default 1500).
- code.ts (CHUNKER_VERSION 4→5): capCodeChunks() applies the same cap to
  AST-path chunks that splitLargeNode can't subdivide.

Co-authored-by: paul-0320 <paul-0320@users.noreply.github.com>

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

2026-07-21 14:18:40 -07:00

..

code.test.ts

fix(chunker): estimated-token hard cap — URL-dense/CJK-fallback chunks overflow strict embedding-server token limits

2026-07-21 14:18:40 -07:00

recursive.test.ts

fix(chunker): estimated-token hard cap — URL-dense/CJK-fallback chunks overflow strict embedding-server token limits

2026-07-21 14:18:40 -07:00

token-cap.test.ts

fix(chunker): estimated-token hard cap — URL-dense/CJK-fallback chunks overflow strict embedding-server token limits

2026-07-21 14:18:40 -07:00