mirror of
https://github.com/garrytan/gbrain.git
synced 2026-07-27 22:15:33 +00:00
fix(minions): restore rolling conversation prompt-cache on the direct SDK path (#2771)
* fix(minions): restore rolling conversation prompt-cache on the direct SDK path
Regression since v0.42.51 (@ai-sdk bump): the direct/native subagent
path (agent.use_gateway_loop=false) only marks cache_control on the
static system-prompt and last-tool-def blocks (~5.2K tokens). The
`anthroMessages` array — the part of the request that actually grows
every turn — carries no cache marker at all, so Anthropic re-bills the
full conversation as fresh input on every turn instead of reading it
from cache.
Before this regression, cache_read grew with the conversation
(4.6K -> 125K across a session). Since the regression, cache_read is
pinned at the ~5.2K static prefix regardless of conversation length.
Fix: mark the last content block of the last message with
`cache_control: { type: 'ephemeral' }` on every turn, after first
stripping any stale marker left on an earlier message. Anthropic
caches everything up to the last cache_control breakpoint, so this
turns the trailing marker into a rolling window over the growing
conversation while staying within the 4-breakpoint limit (system +
last-tool + 1 rolling = 3 used).
Measured on a real dream synthesize run: cache_read/input ratio
0.000 -> 32-61 across 23 calls, ~78-80% cost reduction for that run
($8.5 no-cache-equivalent -> $1.71), zero dead-letter jobs.
Neither the gateway path (cache_control placed at the top level,
which @ai-sdk 3.x silently ignores — see #2490) nor #2442 (system +
last-tool only) restores this; both leave the conversation body
unrecovered.
* fix(minions): normalize seed message content before caching it
Codex review caught a real gap: a fresh job's seed user message is
initialized as `content: data.prompt` (a plain string), not a content-
block array. The rolling-cache logic added in the previous commit only
attaches cache_control when `Array.isArray(lastMsg.content)`, so it
silently skipped the very first API call — meaning the common
single-tool round-trip (seed prompt -> tool_use -> tool_result -> done)
got no conversation-cache benefit at all, only jobs with 3+ turns did.
Normalize string content to a one-block text array before checking, so
the first call gets the same rolling breakpoint as every later one.
* fix(minions): retain the prior rolling cache breakpoint, not just the newest
Second Codex review pass: with a single rolling marker, deleting every
prior message's cache_control before placing the new one means a turn
that adds more than 20 content blocks since the last marker (e.g. a
large parallel tool_use/tool_result round) can miss Anthropic's cache
lookup entirely -- the API's automatic prefix search only looks back
up to 20 blocks from a breakpoint to find a prior cached prefix.
Keep the immediately-preceding rolling marker in place instead of
stripping down to one; only evict markers older than that. This still
fits the 4-breakpoint budget (system + last-tool + 2 rolling = 4) and
guarantees the previous marker's prefix remains a valid, already-cached
read even when the new marker's own lookback misses.
This commit is contained in:
@@ -475,6 +475,61 @@ export function makeSubagentHandler(deps: SubagentDeps) {
|
||||
// covers the whole request. A mid-call renewal loop would add
|
||||
// complexity; for v0.15 we lean on the 120s TTL + abort-on-signal.
|
||||
try {
|
||||
// --- Patch B (borrow-ahead, hand-authored): rolling conversation prompt-cache ---
|
||||
// Direct path marks cache_control only on static system(485)+last-tool(498) ~5.2K;
|
||||
// the growing anthroMessages conversation is re-billed every turn (6/18 v0.42.51
|
||||
// regression). Anthropic caches up to the last cache_control block, so mark the last
|
||||
// content block of the last message and keep the 4-breakpoint API limit
|
||||
// (system + last-tool + 2 rolling = 4).
|
||||
//
|
||||
// Two rolling markers, not one: Anthropic's automatic cache lookup only walks
|
||||
// back up to 20 content blocks from a breakpoint to find a prior cached prefix
|
||||
// (see prompt-caching docs, "20-block lookback window"). A turn that adds more
|
||||
// than 20 blocks since the last marker (e.g. a large parallel tool_use/tool_result
|
||||
// round) would make a freshly-placed single marker miss the previous cache
|
||||
// entirely. Keeping the immediately-preceding rolling marker in place — and only
|
||||
// evicting anything older than that — guarantees at least that marker's prefix is
|
||||
// still a valid, already-written cache read even when this turn's new marker's
|
||||
// lookback comes up empty.
|
||||
if (anthroMessages.length > 0) {
|
||||
const markerIndices: number[] = [];
|
||||
for (let i = 0; i < anthroMessages.length; i++) {
|
||||
const m = anthroMessages[i] as any;
|
||||
if (Array.isArray(m.content)) {
|
||||
for (const b of m.content) {
|
||||
if (b && typeof b === 'object' && 'cache_control' in b) {
|
||||
markerIndices.push(i);
|
||||
break;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
const keepIdx = markerIndices.length > 0 ? markerIndices[markerIndices.length - 1] : -1;
|
||||
for (const i of markerIndices) {
|
||||
if (i === keepIdx) continue;
|
||||
const m = anthroMessages[i] as any;
|
||||
for (const b of m.content) {
|
||||
if (b && typeof b === 'object' && 'cache_control' in b) delete b.cache_control;
|
||||
}
|
||||
}
|
||||
const lastMsg = anthroMessages[anthroMessages.length - 1] as any;
|
||||
// A fresh job's seed message has string content (see the
|
||||
// `[{ role: 'user', content: data.prompt }]` init above), which
|
||||
// the array-only check below would silently skip — leaving the
|
||||
// very first call, and thus the common single-tool round-trip,
|
||||
// with no rolling breakpoint at all. Normalize to a one-block
|
||||
// array first so it gets the marker like every later turn.
|
||||
if (typeof lastMsg.content === 'string') {
|
||||
lastMsg.content = [{ type: 'text', text: lastMsg.content }];
|
||||
}
|
||||
if (Array.isArray(lastMsg.content) && lastMsg.content.length > 0) {
|
||||
const lastBlock = lastMsg.content[lastMsg.content.length - 1];
|
||||
if (lastBlock && typeof lastBlock === 'object') {
|
||||
lastBlock.cache_control = { type: 'ephemeral' };
|
||||
}
|
||||
}
|
||||
}
|
||||
// --- end Patch B ---
|
||||
const params: Anthropic.MessageCreateParamsNonStreaming = {
|
||||
// v0.41 Bug 3: strip `provider:` prefix at the SDK call site only.
|
||||
// `model` stays qualified everywhere else (persistence, recipe
|
||||
|
||||
Reference in New Issue
Block a user