feat(ai/gateway): structured-output opt-in + capability-aware expansion fallback (#2372) (#2373)

* fix(gateway): constrain query expansion JSON key to "queries"

The expansion prompt asks the model to "Rewrite the search query below
into 3-4 different, related queries" without naming the JSON key.
On OpenAI-compatible endpoints that don't enforce a strict JSON schema
server-side (e.g. DeepSeek, many self-hosted gateways), the model
picks the prompt-salient noun and emits {"rewrites": [...]}, which
fails ExpansionSchema ({ queries: string[] }) validation. The catch
block only warns for AIConfigError, so the schema-validation failure
silently falls back to [query] and expansion is effectively disabled.

Verified on two providers: oMLX serving Qwen3.6-35B-A3B-6bit at
http://127.0.0.1:8888/v1 and deepseek-v4-flash at
https://api.deepseek.com/v1. With the prompt constraint, both return
{"queries": [...]} and gbrain query latency increases by ~150 ms
(the expansion inference), confirming expansion now runs end-to-end.

Refs #1156

(cherry picked from commit 132973039c)

* fix(gateway): expand() falls back to generateText for openai-compat providers

generateObject() with a Zod schema uses the response_format
json_schema mode, which most openai-compatible providers do not
support. When the provider rejects structured outputs, the expansion
silently returns only the original query — no error, no log, just
degraded retrieval quality.

For openai-compatible recipes, use generateText() with a JSON prompt
and parse the response manually. Native providers (Anthropic, OpenAI,
Google) keep the existing generateObject() path. This fixes silent
expansion failure for all openai-compatible providers: Zhipu/GLM,
DeepSeek, Groq, Together, Ollama, and any future recipe using the
openai-compatible implementation.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
(cherry picked from commit 0e271961c0)

* refactor(ai): lift parseLlmJson into a leaf util

parseLlmJson lived in conversation-parser/llm-base.ts, which imports chat from the gateway. The gateway needs the same tolerant decoder for its expansion fallback, so importing it back would create a dependency cycle and pull the conversation-parser base into the gateway's module graph.

Move the function to src/core/llm-json.ts, a leaf with no provider or gateway imports, and re-export it from llm-base.ts so existing importers (llm-fallback, llm-polish) and its test keep their import path unchanged. Behavior-preserving.

* feat(ai/gateway): structured-output opt-in + capability-aware expansion fallback

Unifies two cherry-picked fixes (preserved in this branch's history) under a single capability flag and one expand() path:

- #1158 (im4saken): names the required "queries" key in the expansion prompt.
- #1618 (punksterlabs): falls back to generateText for openai-compatible providers.

Adds ChatTouchpoint.supports_structured_outputs (default false) and threads it into createOpenAICompatible's supportsStructuredOutputs at the chat and expansion build sites via recipeSupportsStructuredOutputs().

expand() now routes three ways:

- Native providers (Anthropic, OpenAI, Google) use generateObject unchanged.
- openai-compatible recipes that opt into structured outputs request a strict json_schema and fall back to the text path if it is rejected at call time, so a mis-declared capability never drops expansion.
- Every other openai-compatible recipe skips the json_schema attempt and parses the model's text directly, which removes the AI SDK warning and the silent degradation.

parseExpansionResponse() recovers the queries through a tolerant JSON decode plus schema validation, replacing the inline regex parse.

Net: fixes the silent expansion failure for every openai-compatible backend (the #1618 case), keeps the named-key prompt (closes the gap in #1156 that #1158 addresses), and adds strict structured outputs for backends that support them, which the always-generateText approach cannot reach.

Tests: capability gating across recipes plus a synthetic opt-in recipe; schemaless recovery from clean, fenced, and prose-wrapped JSON; null on non-JSON and schema-violating output.

---------

Co-authored-by: im4saken <280051114+im4saken@users.noreply.github.com>
Co-authored-by: Allwin Agnel <allwin.agnel@gmail.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-authored-by: Garry Tan <garrytan@gmail.com>

src/core/ai/gateway.ts

@@ -52,6 +52,7 @@ import {
   openrouterRequiresExplicitPromptCache,
 } from './recipes/openrouter.ts';
 import { resolveModel, TIER_DEFAULTS } from '../model-config.ts';
+import { parseLlmJson } from '../llm-json.ts';
 import type { BrainEngine } from '../engine.ts';
 import { dimsProviderOptions } from './dims.ts';
 import { hasAnthropicKey } from './anthropic-key.ts';
@@ -451,6 +452,20 @@ export function resolveNativeBaseUrl(
   return /\/v1$/.test(trimmed) ? trimmed : `${trimmed}/v1`;
 }
 
+/**
+ * Whether an openai-compatible recipe's backend honors OpenAI structured
+ * outputs. Threaded into `createOpenAICompatible`'s `supportsStructuredOutputs`
+ * at the chat + expansion build sites, and consulted by `expand()` to pick the
+ * strict `generateObject` path over the schemaless text path. Single source of
+ * truth read from the chat touchpoint: the backend serves both chat and
+ * expansion, so the capability is declared once.
+ *
+ * @internal exported for tests.
+ */
+export function recipeSupportsStructuredOutputs(recipe: Recipe): boolean {
+  return recipe.touchpoints.chat?.supports_structured_outputs === true;
+}
+
 /** Configure the gateway. Called by cli.ts#connectEngine. Clears cached models. */
 export function configureGateway(config: AIGatewayConfig): void {
   _config = {
@@ -2418,6 +2433,7 @@ function instantiateExpansion(recipe: Recipe, modelId: string, cfg: AIGatewayCon
         baseURL: compat.baseURL,
         ...(compat.fetch ? { fetch: compat.fetch } : {}),
         ...auth,
+        supportsStructuredOutputs: recipeSupportsStructuredOutputs(recipe),
       }).languageModel(modelId);
     }
   }
@@ -2427,6 +2443,20 @@ const ExpansionSchema = z.object({
   queries: z.array(z.string()).min(1).max(5),
 });
 
+/**
+ * Recover expansion queries from a schemaless model response. Used by the
+ * openai-compatible expansion paths: a tolerant JSON decode plus schema
+ * validation pulls the `queries` array out of the model's text (the prompt
+ * pins it to a bare JSON object). Returns null when the text carries no valid
+ * `{ queries: string[] }` object.
+ *
+ * @internal exported for tests.
+ */
+export function parseExpansionResponse(text: string): string[] | null {
+  const parsed = ExpansionSchema.safeParse(parseLlmJson<unknown>(text));
+  return parsed.success ? parsed.data.queries : null;
+}
+
 /**
  * Expand a search query into up to 4 related queries.
  * Returns the original query PLUS expansions. On failure, returns just the original.
@@ -2443,24 +2473,63 @@ export async function expand(query: string): Promise<string[]> {
     metadata: { query_chars: query.length },
   });
 
+  const expansionPrompt = [
+    'Rewrite the search query below into 3-4 different, related queries that would help find relevant documents. Respond with a JSON object in exactly this shape: {"queries": ["rewrite1", "rewrite2", "rewrite3"]}. The JSON key MUST be exactly "queries" (not "rewrites" or any other variation).',
+    'Return ONLY the JSON object. Do NOT include the original query in the result.',
+    'Each rewrite should emphasize different aspects, synonyms, or framings.',
+    '',
+    `Query: ${query}`,
+  ].join('\n');
+
   try {
     const { model, recipe, modelId } = await resolveExpansionProvider(getExpansionModel());
-    const result = await generateObject({
-      model,
-      schema: ExpansionSchema,
-      // v0.42.20.0 (codex P0) — expansion had NO abortSignal; same stalled-socket
-      // class as chat. Default the chat timeout.
-      abortSignal: withDefaultTimeout(undefined, AI_CHAT_TIMEOUT_MS),
-      prompt: [
-        'Rewrite the search query below into 3-4 different, related queries that would help find relevant documents.',
-        'Return ONLY the JSON object. Do NOT include the original query in the result.',
-        'Each rewrite should emphasize different aspects, synonyms, or framings.',
-        '',
-        `Query: ${query}`,
-      ].join('\n'),
-    });
 
-    const expansions = result.object?.queries ?? [];
+    let expansions: string[];
+
+    // Schemaless text path for openai-compatible backends whose structured-output
+    // support is unknown: the AI SDK can't send a json_schema response_format
+    // there, so generateObject would warn and silently degrade. generateText + a
+    // tolerant parse recovers the queries instead. Fresh abortSignal per call.
+    const viaText = async (): Promise<string[]> => {
+      const { text } = await generateText({
+        model,
+        abortSignal: withDefaultTimeout(undefined, AI_CHAT_TIMEOUT_MS),
+        prompt: expansionPrompt,
+      });
+      return parseExpansionResponse(text) ?? [];
+    };
+
+    if (recipe.implementation !== 'openai-compatible') {
+      // Native providers (Anthropic, OpenAI, Google) support generateObject's
+      // structured output natively — unchanged path.
+      const result = await generateObject({
+        model,
+        schema: ExpansionSchema,
+        abortSignal: withDefaultTimeout(undefined, AI_CHAT_TIMEOUT_MS),
+        prompt: expansionPrompt,
+      });
+      expansions = result.object?.queries ?? [];
+    } else if (recipeSupportsStructuredOutputs(recipe)) {
+      // openai-compatible backend that honors strict json_schema: request the
+      // schema (strict validation), and fall back to the text path if it is
+      // rejected at call time so a mis-declared capability never drops expansion.
+      try {
+        const result = await generateObject({
+          model,
+          schema: ExpansionSchema,
+          abortSignal: withDefaultTimeout(undefined, AI_CHAT_TIMEOUT_MS),
+          prompt: expansionPrompt,
+        });
+        expansions = result.object?.queries ?? [];
+      } catch {
+        expansions = await viaText();
+      }
+    } else {
+      // openai-compatible backend, structured-output support unknown: skip the
+      // json_schema attempt entirely (no SDK warning, no silent degradation).
+      expansions = await viaText();
+    }
+
     // Deduplicate + include the original query
     const seen = new Set<string>();
     const all = [query, ...expansions].filter(q => {
@@ -2926,6 +2995,7 @@ function instantiateChat(recipe: Recipe, modelId: string, cfg: AIGatewayConfig):
         baseURL: compat.baseURL,
         ...(compat.fetch ? { fetch: compat.fetch } : {}),
         ...auth,
+        supportsStructuredOutputs: recipeSupportsStructuredOutputs(recipe),
       }).languageModel(modelId);
     }
     default:

src/core/ai/types.ts

@@ -255,6 +255,17 @@ export interface ChatTouchpoint {
    * model family).
    */
   supports_prompt_cache?: boolean | ((modelId: string) => boolean);
+  /**
+   * Backend honors OpenAI structured outputs (a strict `json_schema`
+   * response_format). Threaded into `createOpenAICompatible`'s
+   * `supportsStructuredOutputs` so query expansion's `generateObject` sends a
+   * real schema (strict validation) instead of degrading to schemaless JSON.
+   * Default false: an openai-compatible recipe may front arbitrary backends,
+   * most of which lack strict json_schema support, so `expand()` routes them
+   * through the schemaless text path. Opt in per recipe when the backend is
+   * known to honor it.
+   */
+  supports_structured_outputs?: boolean;
   max_context_tokens?: number;
   cost_per_1m_input_usd?: number;
   cost_per_1m_output_usd?: number;

src/core/conversation-parser/llm-base.ts

@@ -306,41 +306,7 @@ function splitCacheKey(key: string): [string?, string?, string?] {
   return [shape, model, sha];
 }
 
-/**
- * 4-strategy JSON repair (lifted from `eval/longmemeval/extract.ts:50`
- * for object-shaped output; the original was array-shaped). Caller's
- * `parse` function uses this for tolerant LLM-output decoding.
- *
- * Strategies:
- *   1. Strip ```json...``` fences if present, then JSON.parse.
- *   2. Direct JSON.parse.
- *   3. Find first {...} substring (or [...] if array=true) and parse.
- *   4. Return null.
- *
- * Adversarial input throws caught by caller's try/catch (parse returns
- * null upstream).
- */
-export function parseLlmJson<T>(raw: string, opts: { array?: boolean } = {}): T | null {
-  if (typeof raw !== 'string' || !raw.trim()) return null;
-  const fenceMatch = raw.match(/```(?:json)?\s*\n?([\s\S]*?)```/i);
-  const cleaned = (fenceMatch ? fenceMatch[1] : raw).trim();
-  try {
-    const direct = JSON.parse(cleaned);
-    if (opts.array && Array.isArray(direct)) return direct as T;
-    if (!opts.array && direct !== null && typeof direct === 'object') return direct as T;
-  } catch {
-    // fall through
-  }
-  const pattern = opts.array ? /\[[\s\S]*\]/ : /\{[\s\S]*\}/;
-  const match = cleaned.match(pattern);
-  if (match) {
-    try {
-      const second = JSON.parse(match[0]);
-      if (opts.array && Array.isArray(second)) return second as T;
-      if (!opts.array && second !== null && typeof second === 'object') return second as T;
-    } catch {
-      // fall through
-    }
-  }
-  return null;
-}
+// Tolerant LLM-output JSON decoder. Re-exported from the leaf util so existing
+// importers (llm-fallback, llm-polish) keep their import path while the gateway
+// can reuse it without a dependency cycle.
+export { parseLlmJson } from '../llm-json.ts';

src/core/llm-json.ts

@@ -0,0 +1,37 @@
+/**
+ * Tolerant decode of a JSON object (or array) embedded in LLM output. A leaf
+ * util with no provider/gateway imports so any layer can reuse it without a
+ * dependency cycle.
+ *
+ * Strategies, in order:
+ *   1. Strip ```json...``` fences if present, then JSON.parse.
+ *   2. Direct JSON.parse.
+ *   3. Find the first {...} substring (or [...] when array=true) and parse.
+ *   4. Return null.
+ *
+ * Adversarial input throws are swallowed; callers get null on any failure.
+ */
+export function parseLlmJson<T>(raw: string, opts: { array?: boolean } = {}): T | null {
+  if (typeof raw !== 'string' || !raw.trim()) return null;
+  const fenceMatch = raw.match(/```(?:json)?\s*\n?([\s\S]*?)```/i);
+  const cleaned = (fenceMatch ? fenceMatch[1] : raw).trim();
+  try {
+    const direct = JSON.parse(cleaned);
+    if (opts.array && Array.isArray(direct)) return direct as T;
+    if (!opts.array && direct !== null && typeof direct === 'object') return direct as T;
+  } catch {
+    // fall through
+  }
+  const pattern = opts.array ? /\[[\s\S]*\]/ : /\{[\s\S]*\}/;
+  const match = cleaned.match(pattern);
+  if (match) {
+    try {
+      const second = JSON.parse(match[0]);
+      if (opts.array && Array.isArray(second)) return second as T;
+      if (!opts.array && second !== null && typeof second === 'object') return second as T;
+    } catch {
+      // fall through
+    }
+  }
+  return null;
+}

test/ai/gateway-chat.test.ts

@@ -23,12 +23,15 @@ import {
   isAvailable,
   getChatModel,
   getChatFallbackChain,
+  recipeSupportsStructuredOutputs,
+  parseExpansionResponse,
   chat,
   __setGenerateTextTransportForTests,
 } from '../../src/core/ai/gateway.ts';
 import { parseModelId, resolveRecipe, assertTouchpoint } from '../../src/core/ai/model-resolver.ts';
 import { AIConfigError } from '../../src/core/ai/errors.ts';
 import { listRecipes, getRecipe } from '../../src/core/ai/recipes/index.ts';
+import type { Recipe } from '../../src/core/ai/types.ts';
 
 describe('chat touchpoint — recipe registry', () => {
   test('all six chat-capable providers ship a chat touchpoint with supports_subagent_loop', () => {
@@ -69,6 +72,56 @@ describe('chat touchpoint — recipe registry', () => {
   });
 });
 
+describe('expansion — structured-output capability gating', () => {
+  test('openai-compat chat recipes default to no structured-output support', () => {
+    // The capability is opt-in per recipe: an openai-compatible recipe may front
+    // arbitrary backends, so expand() routes the default through the schemaless
+    // text path rather than requesting a json_schema the backend may reject.
+    for (const id of ['deepseek', 'groq', 'together']) {
+      expect(recipeSupportsStructuredOutputs(getRecipe(id)!)).toBe(false);
+    }
+  });
+
+  test('recipeSupportsStructuredOutputs is false when no chat touchpoint exists', () => {
+    // Embedding-only recipes have no chat touchpoint; the helper must not throw.
+    expect(recipeSupportsStructuredOutputs(getRecipe('voyage')!)).toBe(false);
+  });
+
+  test('recipeSupportsStructuredOutputs is true when a recipe opts in', () => {
+    const optedIn = {
+      id: 'synthetic',
+      touchpoints: { chat: { models: [], supports_tools: true, supports_subagent_loop: true, supports_structured_outputs: true } },
+    } as unknown as Recipe;
+    expect(recipeSupportsStructuredOutputs(optedIn)).toBe(true);
+  });
+});
+
+describe('expansion — schemaless recovery (parseExpansionResponse)', () => {
+  // The openai-compat expansion paths recover queries from raw model text. This
+  // is the testable seam both the default and the strict-fallback paths share.
+  test('recovers queries from clean JSON', () => {
+    expect(parseExpansionResponse('{"queries":["a","b","c"]}')).toEqual(['a', 'b', 'c']);
+  });
+
+  test('recovers queries from fenced JSON', () => {
+    expect(parseExpansionResponse('```json\n{"queries":["a","b"]}\n```')).toEqual(['a', 'b']);
+  });
+
+  test('recovers queries from prose-wrapped JSON', () => {
+    expect(parseExpansionResponse('Here you go: {"queries":["a"]} done')).toEqual(['a']);
+  });
+
+  test('returns null for non-JSON so the caller can drop expansion cleanly', () => {
+    expect(parseExpansionResponse('I cannot help with that.')).toBeNull();
+  });
+
+  test('returns null when the JSON violates the schema', () => {
+    expect(parseExpansionResponse('{"queries":[]}')).toBeNull(); // min(1)
+    expect(parseExpansionResponse('{"rewrites":["a"]}')).toBeNull(); // wrong key
+    expect(parseExpansionResponse('{"queries":[1,2]}')).toBeNull(); // wrong item type
+  });
+});
+
 describe('chat touchpoint — model resolver + aliases (Codex F-OV-5)', () => {
   test('parseModelId handles dated and undated forms identically at parse time', () => {
     expect(parseModelId('anthropic:claude-sonnet-4-6')).toEqual({