fix(minions): restore rolling conversation prompt-cache on the direct SDK path (#2771)

* fix(minions): restore rolling conversation prompt-cache on the direct SDK path

Regression since v0.42.51 (@ai-sdk bump): the direct/native subagent
path (agent.use_gateway_loop=false) only marks cache_control on the
static system-prompt and last-tool-def blocks (~5.2K tokens). The
`anthroMessages` array — the part of the request that actually grows
every turn — carries no cache marker at all, so Anthropic re-bills the
full conversation as fresh input on every turn instead of reading it
from cache.

Before this regression, cache_read grew with the conversation
(4.6K -> 125K across a session). Since the regression, cache_read is
pinned at the ~5.2K static prefix regardless of conversation length.

Fix: mark the last content block of the last message with
`cache_control: { type: 'ephemeral' }` on every turn, after first
stripping any stale marker left on an earlier message. Anthropic
caches everything up to the last cache_control breakpoint, so this
turns the trailing marker into a rolling window over the growing
conversation while staying within the 4-breakpoint limit (system +
last-tool + 1 rolling = 3 used).

Measured on a real dream synthesize run: cache_read/input ratio
0.000 -> 32-61 across 23 calls, ~78-80% cost reduction for that run
($8.5 no-cache-equivalent -> $1.71), zero dead-letter jobs.

Neither the gateway path (cache_control placed at the top level,
which @ai-sdk 3.x silently ignores — see #2490) nor #2442 (system +
last-tool only) restores this; both leave the conversation body
unrecovered.

* fix(minions): normalize seed message content before caching it

Codex review caught a real gap: a fresh job's seed user message is
initialized as `content: data.prompt` (a plain string), not a content-
block array. The rolling-cache logic added in the previous commit only
attaches cache_control when `Array.isArray(lastMsg.content)`, so it
silently skipped the very first API call — meaning the common
single-tool round-trip (seed prompt -> tool_use -> tool_result -> done)
got no conversation-cache benefit at all, only jobs with 3+ turns did.

Normalize string content to a one-block text array before checking, so
the first call gets the same rolling breakpoint as every later one.

* fix(minions): retain the prior rolling cache breakpoint, not just the newest

Second Codex review pass: with a single rolling marker, deleting every
prior message's cache_control before placing the new one means a turn
that adds more than 20 content blocks since the last marker (e.g. a
large parallel tool_use/tool_result round) can miss Anthropic's cache
lookup entirely -- the API's automatic prefix search only looks back
up to 20 blocks from a breakpoint to find a prior cached prefix.

Keep the immediately-preceding rolling marker in place instead of
stripping down to one; only evict markers older than that. This still
fits the 4-breakpoint budget (system + last-tool + 2 rolling = 4) and
guarantees the previous marker's prefix remains a valid, already-cached
read even when the new marker's own lookback misses.
This commit is contained in:
Masa
2026-07-17 11:32:31 -07:00
committed by GitHub
parent 9eac872136
commit c4ff8b63a8
+55
View File
@@ -475,6 +475,61 @@ export function makeSubagentHandler(deps: SubagentDeps) {
// covers the whole request. A mid-call renewal loop would add
// complexity; for v0.15 we lean on the 120s TTL + abort-on-signal.
try {
// --- Patch B (borrow-ahead, hand-authored): rolling conversation prompt-cache ---
// Direct path marks cache_control only on static system(485)+last-tool(498) ~5.2K;
// the growing anthroMessages conversation is re-billed every turn (6/18 v0.42.51
// regression). Anthropic caches up to the last cache_control block, so mark the last
// content block of the last message and keep the 4-breakpoint API limit
// (system + last-tool + 2 rolling = 4).
//
// Two rolling markers, not one: Anthropic's automatic cache lookup only walks
// back up to 20 content blocks from a breakpoint to find a prior cached prefix
// (see prompt-caching docs, "20-block lookback window"). A turn that adds more
// than 20 blocks since the last marker (e.g. a large parallel tool_use/tool_result
// round) would make a freshly-placed single marker miss the previous cache
// entirely. Keeping the immediately-preceding rolling marker in place — and only
// evicting anything older than that — guarantees at least that marker's prefix is
// still a valid, already-written cache read even when this turn's new marker's
// lookback comes up empty.
if (anthroMessages.length > 0) {
const markerIndices: number[] = [];
for (let i = 0; i < anthroMessages.length; i++) {
const m = anthroMessages[i] as any;
if (Array.isArray(m.content)) {
for (const b of m.content) {
if (b && typeof b === 'object' && 'cache_control' in b) {
markerIndices.push(i);
break;
}
}
}
}
const keepIdx = markerIndices.length > 0 ? markerIndices[markerIndices.length - 1] : -1;
for (const i of markerIndices) {
if (i === keepIdx) continue;
const m = anthroMessages[i] as any;
for (const b of m.content) {
if (b && typeof b === 'object' && 'cache_control' in b) delete b.cache_control;
}
}
const lastMsg = anthroMessages[anthroMessages.length - 1] as any;
// A fresh job's seed message has string content (see the
// `[{ role: 'user', content: data.prompt }]` init above), which
// the array-only check below would silently skip — leaving the
// very first call, and thus the common single-tool round-trip,
// with no rolling breakpoint at all. Normalize to a one-block
// array first so it gets the marker like every later turn.
if (typeof lastMsg.content === 'string') {
lastMsg.content = [{ type: 'text', text: lastMsg.content }];
}
if (Array.isArray(lastMsg.content) && lastMsg.content.length > 0) {
const lastBlock = lastMsg.content[lastMsg.content.length - 1];
if (lastBlock && typeof lastBlock === 'object') {
lastBlock.cache_control = { type: 'ephemeral' };
}
}
}
// --- end Patch B ---
const params: Anthropic.MessageCreateParamsNonStreaming = {
// v0.41 Bug 3: strip `provider:` prefix at the SDK call site only.
// `model` stays qualified everywhere else (persistence, recipe