Files
gbrain/src/core
8837bfe5f2 fix(chunker): cap oversized code chunks so they stay embeddable (#1675)
splitLargeNode can only break up a node that exposes a `body` with >= 2
named children. A node without one -- a giant object/array literal, a single
huge assignment, a massive template literal -- is emitted whole. On real
source that yields a chunk far larger than the embedder's context window; the
embedder then rejects it ("input exceeds context length") and it is never
embedded. Example: a 372 KB service file produced 113 chunks, one a single
281 KB (~70k-token) node -> permanently unembedded.

Add a final safety-net pass (capOversizedChunks) that recursively re-splits
any chunk over a token budget (default 2000, configurable via maxChunkTokens),
with a hard character split as a last resort for no-whitespace content
(minified one-liners). Normal files are untouched.

Verified: that 372 KB file now yields 188 chunks, max ~1.5k tokens, zero
oversized; a small file is unchanged.

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-22 18:46:34 -07:00
..