fix(cycle): budget the patterns subagent from remaining job time — one phase's fixed 35-min worst case defeats any interval-derived cycle budget (#2781) (#2959)

* fix(cycle): budget the patterns subagent from remaining job time, not a fixed constant (#2781)

The autopilot-cycle job gets an interval-derived timeout stamped at submit,
but the patterns phase submits its subagent with a fixed 30-min job timeout
and waits up to 35 min — one phase's worst case exceeds ANY interval-derived
budget <= 35 min, so the parent job dead-letters mid-patterns and the tail
phases (consolidate -> schema-suggest) starve for days (#2781, the deeper
half left open by the #2852 dispatch-floor fix).

- MinionJobContext.deadlineAtMs: absolute deadline from the claim-time
  timeout_at stamp (the DB ground truth handleTimeouts() sweeps against;
  re-stamped on every claim so retries get a fresh budget). Null when the
  job has no per-job timeout.
- worker: the per-job abort timer now derives its delay from timeout_at
  when present, so the in-process timer, the DB sweeper, and the
  handler-visible deadline agree on ONE absolute instant.
- autopilot-cycle + autopilot-global-maintenance handlers thread
  deadlineAtMs into runCycle; CycleOpts carries it to the patterns phase.
- patterns: clampSubagentBudgets() derives BOTH the child job timeout and
  the wait timeout from the same child deadline (parent deadline minus a
  60s stop-margin reserve — enough for the wait poll + force-evict grace
  + cleanup, deliberately NOT a promise that tail phases complete). Under
  a 2-min minimum the phase skips honestly (insufficient_cycle_budget)
  instead of submitting a guaranteed-kill LLM call; the next cycle
  retries with a fresh budget.
- Direct callers (gbrain dream) pass no deadline and keep the configured
  timeouts unchanged.

Follow-up (separate PR): synthesize has the same shape plus sequential
per-child waits that accumulate N x subagent_wait_timeout_ms past any
parent budget; it needs per-wait remaining-time recomputation.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RaNDjecPPvhC7LRqkxkhGF

* fix(cycle): address review — cancel timed-out patterns child; thread deadline through phase-wrapper handlers

- P1: the child's timeout_ms clock starts at ITS claim, so a queued child
  could outlive the parent deadline the wait was clamped to. On wait
  timeout, cancelJob strips it (waiting -> cancelled; active -> lock
  stripped, worker abort fires next renew tick).
- P2: makePhaseHandler (standalone patterns/synthesize/... minion jobs)
  now threads job.deadlineAtMs into runCycle like the autopilot handlers.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RaNDjecPPvhC7LRqkxkhGF

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Masa
2026-07-20 13:56:00 -07:00
committed by GitHub
co-authored by Claude Fable 5
parent c873ce3014
commit 23e0541d9b
15 changed files with 285 additions and 7 deletions
+3
View File
@@ -1796,6 +1796,7 @@ export async function registerBuiltinHandlers(
brainDir: effectiveBrainDir,
pull,
signal: job.signal, // propagate abort so cycle bails on timeout/cancel
deadlineAtMs: job.deadlineAtMs, // #2781: phases budget sub-work from remaining time
...(sourceId ? { sourceId } : {}),
...(requestedPhases && requestedPhases.length > 0 ? { phases: requestedPhases as any } : {}),
yieldBetweenPhases: async () => {
@@ -1833,6 +1834,7 @@ export async function registerBuiltinHandlers(
brainDir: repoPath,
pull: false, // brain-wide DB/maintenance work never git-pulls
signal: job.signal,
deadlineAtMs: job.deadlineAtMs, // #2781: phases budget sub-work from remaining time
phases,
yieldBetweenPhases: async () => { await new Promise<void>((r) => setImmediate(r)); },
});
@@ -1978,6 +1980,7 @@ export async function registerBuiltinHandlers(
brainDir: repoPath,
phases: [phase as any],
signal: job.signal,
deadlineAtMs: job.deadlineAtMs, // #2781: phases budget sub-work from remaining time
});
return { phase, status: report.status, report };
};
+11
View File
@@ -479,6 +479,16 @@ export interface CycleOpts {
* Validated via `assertValidSourceId` in `cycleLockIdFor` (defense-in-depth).
*/
sourceId?: string;
/**
* Absolute wall-clock deadline (epoch ms) of the enclosing minion job,
* from `MinionJobContext.deadlineAtMs` (the claim-time `timeout_at`
* stamp). Phases that spawn bounded sub-work (patterns' subagent) clamp
* their own timeouts to the REMAINING time so one phase's fixed
* worst-case can't blow past the job budget and dead-letter the whole
* cycle mid-phase (#2781). Unset for direct callers (`gbrain dream`) —
* phases then use their configured timeouts unchanged.
*/
deadlineAtMs?: number | null;
}
// ─── Lock primitives ───────────────────────────────────────────────
@@ -1888,6 +1898,7 @@ export async function runCycle(
brainDir,
dryRun,
yieldDuringPhase: opts.yieldDuringPhase,
deadlineAtMs: opts.deadlineAtMs ?? null,
}));
result.duration_ms = duration_ms;
phaseResults.push(result);
+78 -4
View File
@@ -37,6 +37,57 @@ export interface PatternsPhaseOpts {
brainDir: string;
dryRun: boolean;
yieldDuringPhase?: () => Promise<void>;
/**
* Absolute deadline (epoch ms) of the enclosing minion job, or null for
* direct callers (`gbrain dream`). When set, the subagent's job timeout
* and the wait timeout are clamped so the phase finishes (or times out)
* BEFORE the parent job's budget expires — a fixed 30/35-min default
* inside an interval-derived cycle budget dead-letters the whole cycle
* mid-phase and starves every tail phase (#2781).
*/
deadlineAtMs?: number | null;
}
/**
* Stop-margin reserved under the parent deadline when clamping subagent
* budgets. NOT a promise that tail phases complete — the cycle is allowed
* to go partial and resume next tick. This only guarantees the phase's
* wait returns and the handler unwinds cleanly before the worker's abort
* fires: wait poll interval (5s) + worker force-evict grace (30s) + lock
* and DB cleanup headroom.
*/
export const CYCLE_DEADLINE_RESERVE_MS = 60 * 1000;
/**
* Smallest remaining budget worth submitting a subagent for. Below this,
* the LLM call is near-certain to be killed mid-flight — wasted spend and
* a guaranteed-timeout child — so the phase skips honestly instead
* (`insufficient_cycle_budget`) and the next cycle retries with a fresh
* budget.
*/
export const MIN_PATTERNS_SUBAGENT_BUDGET_MS = 2 * 60 * 1000;
/**
* Clamp the configured subagent budgets to the remaining parent-job time.
* Both timeouts derive from the SAME absolute child deadline
* (`deadlineAtMs - reserve`) so the child job's kill switch and our wait
* agree. Returns null when the remaining budget is below the minimum —
* caller should skip the phase without submitting.
*/
export function clampSubagentBudgets(
config: { subagentTimeoutMs: number; subagentWaitTimeoutMs: number },
deadlineAtMs: number | null | undefined,
nowMs: number,
): { timeoutMs: number; waitTimeoutMs: number } | null {
if (deadlineAtMs == null) {
return { timeoutMs: config.subagentTimeoutMs, waitTimeoutMs: config.subagentWaitTimeoutMs };
}
const childBudgetMs = deadlineAtMs - CYCLE_DEADLINE_RESERVE_MS - nowMs;
if (childBudgetMs < MIN_PATTERNS_SUBAGENT_BUDGET_MS) return null;
return {
timeoutMs: Math.min(config.subagentTimeoutMs, childBudgetMs),
waitTimeoutMs: Math.min(config.subagentWaitTimeoutMs, childBudgetMs),
};
}
export async function runPhasePatterns(
@@ -90,6 +141,19 @@ export async function runPhasePatterns(
'skills/_brain-filing-rules.json missing dream_synthesize_paths.globs'));
}
// #2781: budget the subagent from the REMAINING parent-job time, not
// the fixed config default. Checked after the cheap gates (disabled /
// insufficient_evidence / no_provider) so a skip for budget reasons
// only fires when the phase would otherwise have submitted.
const budgets = clampSubagentBudgets(config, opts.deadlineAtMs, Date.now());
if (budgets === null) {
return skipped(
'insufficient_cycle_budget',
`remaining cycle budget under ${Math.round(MIN_PATTERNS_SUBAGENT_BUDGET_MS / 1000)}s ` +
`(reserve ${Math.round(CYCLE_DEADLINE_RESERVE_MS / 1000)}s); next cycle retries with a fresh budget`,
);
}
const queue = new MinionQueue(engine);
const data: SubagentHandlerData = {
prompt: buildPatternsPrompt(reflections, config.minEvidence, config.outputRoot),
@@ -99,7 +163,7 @@ export async function runPhasePatterns(
};
const submitOpts: Partial<MinionJobInput> = {
max_stalled: 3,
timeout_ms: config.subagentTimeoutMs,
timeout_ms: budgets.timeoutMs,
};
const job = await queue.add('subagent', data as unknown as Record<string, unknown>, submitOpts, {
allowProtectedSubmit: true,
@@ -108,13 +172,23 @@ export async function runPhasePatterns(
let outcome: string;
try {
const final = await waitForCompletion(queue, job.id, {
timeoutMs: config.subagentWaitTimeoutMs,
timeoutMs: budgets.waitTimeoutMs,
pollMs: 5 * 1000,
});
outcome = final.status;
} catch (e) {
if (e instanceof TimeoutError) outcome = 'timeout';
else throw e;
if (e instanceof TimeoutError) {
outcome = 'timeout';
// The child's own timeout_ms clock starts at ITS claim, not at
// submit — a child that sat queued behind other work can outlive
// the parent deadline this wait was clamped to. Cancel it so the
// subagent can't keep spending/writing after the phase gave up
// (waiting child → cancelled immediately; active child → lock
// stripped, worker abort fires on next renew tick).
try { await queue.cancelJob(job.id); } catch { /* best-effort */ }
} else {
throw e;
}
}
if (opts.yieldDuringPhase) {
+6
View File
@@ -200,6 +200,12 @@ export interface MinionJobContext {
attempts_made: number;
/** AbortSignal for cooperative cancellation (fires on timeout, cancel, pause, or lock loss). */
signal: AbortSignal;
/** Absolute wall-clock deadline (epoch ms) from the claim-time `timeout_at` stamp,
* or null when the job has no per-job timeout. This is the DB's ground truth —
* the same instant handleTimeouts() dead-letters against — so handlers that
* spawn bounded sub-work (e.g. autopilot-cycle's subagent phases) can budget
* from the REMAINING time instead of a fixed constant that may exceed it. */
deadlineAtMs: number | null;
/** AbortSignal that fires only on worker process SIGTERM/SIGINT. Handlers sensitive
* to deploy restarts (e.g. the shell handler, which must run a SIGTERM → 5s → SIGKILL
* sequence on its child) listen to this in addition to `signal`. Most handlers can
+10 -2
View File
@@ -900,15 +900,22 @@ export class MinionWorker extends EventEmitter {
// Per-job wall-clock timeout (timer-armed only if `timeout_ms` was
// set on the job; the grace-evict pattern above now lives outside
// this branch).
// this branch). The delay derives from the claim-time `timeout_at`
// stamp when present so this timer, the DB sweeper (handleTimeouts),
// and the handler-visible `deadlineAtMs` all agree on ONE absolute
// deadline instead of three clocks started at slightly different
// instants.
let timeoutTimer: ReturnType<typeof setTimeout> | null = null;
if (job.timeout_ms != null) {
const delayMs = job.timeout_at != null
? Math.max(0, job.timeout_at.getTime() - Date.now())
: job.timeout_ms;
timeoutTimer = setTimeout(() => {
if (!abort.signal.aborted) {
console.warn(`Job ${job.id} (${job.name}) hit per-job timeout (${job.timeout_ms}ms), aborting`);
abort.abort(new Error('timeout'));
}
}, job.timeout_ms);
}, delayMs);
}
const promise = this.executeJob(job, lockToken, abort, lockTimer)
@@ -964,6 +971,7 @@ export class MinionWorker extends EventEmitter {
data: job.data,
attempts_made: job.attempts_made,
signal: abort.signal,
deadlineAtMs: job.timeout_at != null ? job.timeout_at.getTime() : null,
shutdownSignal: this.shutdownAbort.signal,
updateProgress: async (progress: unknown) => {
await this.queue.updateProgress(job.id, lockToken, progress);
+168
View File
@@ -0,0 +1,168 @@
/**
* #2781 — patterns phase budgets its subagent from the REMAINING parent-job
* time instead of a fixed 30/35-min default that can exceed any
* interval-derived cycle budget and dead-letter the whole cycle mid-phase.
*
* Layers:
* 1. Unit tests on the exported pure `clampSubagentBudgets`.
* 2. A real-queue check that `claim` stamps `timeout_at` (the DB ground
* truth `deadlineAtMs` derives from) and leaves it null when the job
* has no per-job timeout.
* 3. Structural assertions pinning the wiring: worker → context →
* handler → runCycle → patterns (matches the house style of
* test/cycle-patterns.test.ts).
*/
import { describe, test, expect, beforeAll, afterAll } from 'bun:test';
import { readFileSync } from 'fs';
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
import { MinionQueue } from '../src/core/minions/queue.ts';
import {
clampSubagentBudgets,
CYCLE_DEADLINE_RESERVE_MS,
MIN_PATTERNS_SUBAGENT_BUDGET_MS,
} from '../src/core/cycle/patterns.ts';
const CONFIG = {
subagentTimeoutMs: 30 * 60 * 1000,
subagentWaitTimeoutMs: 35 * 60 * 1000,
};
describe('clampSubagentBudgets', () => {
const now = 1_000_000_000_000; // fixed epoch ms; the function takes nowMs explicitly
test('null deadline → config passthrough (direct `gbrain dream` back-compat)', () => {
expect(clampSubagentBudgets(CONFIG, null, now)).toEqual({
timeoutMs: CONFIG.subagentTimeoutMs,
waitTimeoutMs: CONFIG.subagentWaitTimeoutMs,
});
expect(clampSubagentBudgets(CONFIG, undefined, now)).toEqual({
timeoutMs: CONFIG.subagentTimeoutMs,
waitTimeoutMs: CONFIG.subagentWaitTimeoutMs,
});
});
test('deadline far away → config values win (no clamping)', () => {
const deadline = now + 2 * 60 * 60 * 1000; // 2h out
expect(clampSubagentBudgets(CONFIG, deadline, now)).toEqual({
timeoutMs: CONFIG.subagentTimeoutMs,
waitTimeoutMs: CONFIG.subagentWaitTimeoutMs,
});
});
test('deadline inside config window → BOTH timeouts clamp to the same child budget', () => {
const deadline = now + 10 * 60 * 1000; // 10 min out
const childBudget = deadline - CYCLE_DEADLINE_RESERVE_MS - now; // 9 min
const budgets = clampSubagentBudgets(CONFIG, deadline, now);
expect(budgets).toEqual({ timeoutMs: childBudget, waitTimeoutMs: childBudget });
// The child's own kill switch never outlives the parent budget.
expect(budgets!.timeoutMs).toBeLessThanOrEqual(deadline - now);
});
test('remaining budget below minimum → null (caller skips, no submit)', () => {
const deadline = now + CYCLE_DEADLINE_RESERVE_MS + MIN_PATTERNS_SUBAGENT_BUDGET_MS - 1;
expect(clampSubagentBudgets(CONFIG, deadline, now)).toBeNull();
});
test('boundary: exactly the minimum budget → submit allowed', () => {
const deadline = now + CYCLE_DEADLINE_RESERVE_MS + MIN_PATTERNS_SUBAGENT_BUDGET_MS;
expect(clampSubagentBudgets(CONFIG, deadline, now)).toEqual({
timeoutMs: MIN_PATTERNS_SUBAGENT_BUDGET_MS,
waitTimeoutMs: MIN_PATTERNS_SUBAGENT_BUDGET_MS,
});
});
test('deadline already past → null, never a negative timeout', () => {
expect(clampSubagentBudgets(CONFIG, now - 1000, now)).toBeNull();
});
});
describe('claim stamps timeout_at (deadlineAtMs ground truth)', () => {
let engine: PGLiteEngine;
let queue: MinionQueue;
beforeAll(async () => {
engine = new PGLiteEngine();
await engine.connect({ database_url: '' }); // in-memory
await engine.initSchema();
queue = new MinionQueue(engine);
});
afterAll(async () => {
await engine.disconnect();
});
test('job with timeout_ms → claim sets timeout_at ≈ now + timeout_ms', async () => {
const before = Date.now();
await queue.add('sync', {}, { timeout_ms: 600_000 });
const claimed = await queue.claim('tok-dl-1', 30000, 'default', ['sync']);
const after = Date.now();
expect(claimed).not.toBeNull();
expect(claimed!.timeout_at).not.toBeNull();
const at = claimed!.timeout_at!.getTime();
expect(at).toBeGreaterThanOrEqual(before + 600_000 - 5_000);
expect(at).toBeLessThanOrEqual(after + 600_000 + 5_000);
});
test('job without timeout_ms and no handler default → timeout_at stays null', async () => {
// 'sync' is not in the long-handler default set, so no stamp either way.
await queue.add('sync', { which: 'no-timeout' });
// Drain the possibly-remaining job from the prior test first.
let claimed = await queue.claim('tok-dl-2', 30000, 'default', ['sync']);
while (claimed && claimed.timeout_ms != null) {
claimed = await queue.claim('tok-dl-2', 30000, 'default', ['sync']);
}
expect(claimed).not.toBeNull();
expect(claimed!.timeout_ms).toBeNull();
expect(claimed!.timeout_at).toBeNull();
});
});
describe('deadline plumbing wiring (structural)', () => {
const workerSrc = readFileSync(new URL('../src/core/minions/worker.ts', import.meta.url), 'utf-8');
const jobsSrc = readFileSync(new URL('../src/commands/jobs.ts', import.meta.url), 'utf-8');
const cycleSrc = readFileSync(new URL('../src/core/cycle.ts', import.meta.url), 'utf-8');
const patternsSrc = readFileSync(new URL('../src/core/cycle/patterns.ts', import.meta.url), 'utf-8');
test('worker exposes deadlineAtMs from the claim-time timeout_at stamp', () => {
expect(workerSrc).toContain('deadlineAtMs: job.timeout_at != null ? job.timeout_at.getTime() : null');
});
test('worker arms its abort timer from timeout_at when present (one absolute deadline)', () => {
expect(workerSrc).toContain('job.timeout_at.getTime() - Date.now()');
});
test('autopilot-cycle, global-maintenance AND phase-wrapper handlers thread deadlineAtMs into runCycle', () => {
const matches = jobsSrc.match(/deadlineAtMs: job\.deadlineAtMs/g) ?? [];
expect(matches.length).toBe(3);
});
test('runCycle forwards deadlineAtMs to the patterns phase', () => {
expect(cycleSrc).toContain('deadlineAtMs: opts.deadlineAtMs ?? null');
});
test('patterns submits + waits with the CLAMPED budgets, not raw config', () => {
expect(patternsSrc).toContain('timeout_ms: budgets.timeoutMs');
expect(patternsSrc).toContain('timeoutMs: budgets.waitTimeoutMs');
expect(patternsSrc).not.toContain('timeout_ms: config.subagentTimeoutMs');
expect(patternsSrc).not.toContain('timeoutMs: config.subagentWaitTimeoutMs');
});
test('patterns cancels the child on wait timeout (child clock starts at ITS claim)', () => {
// A child that sat queued can outlive the parent deadline the wait was
// clamped to; the timeout path must strip it so it can't keep spending.
expect(patternsSrc).toContain('queue.cancelJob(job.id)');
});
test('patterns skips honestly when the remaining budget is too small', () => {
expect(patternsSrc).toContain('insufficient_cycle_budget');
// Budget gate sits AFTER the provider probe so a no-provider brain
// still reports no_provider (cheaper, more actionable reason).
const probeIdx = patternsSrc.indexOf("skipped('no_provider'");
// lastIndexOf: the doc comment on MIN_PATTERNS_SUBAGENT_BUDGET_MS
// mentions the reason string too; the CALL SITE is the later hit.
const budgetIdx = patternsSrc.lastIndexOf('insufficient_cycle_budget');
expect(probeIdx).toBeGreaterThan(0);
expect(budgetIdx).toBeGreaterThan(probeIdx);
});
});
+1
View File
@@ -49,6 +49,7 @@ function makeFakeJobCtx(data: Record<string, unknown>): MinionJobContext {
data,
attempts_made: 1,
signal: new AbortController().signal,
deadlineAtMs: null,
shutdownSignal: new AbortController().signal,
updateProgress: async () => {},
updateTokens: async () => {},
@@ -279,6 +279,7 @@ async function makeCrashedCtx(jobId: number, prompt: string, modelId: string): P
data: { prompt, model: modelId },
attempts_made: 1, // crashed once
signal: abortCtrl.signal,
deadlineAtMs: null,
shutdownSignal: shutdownCtrl.signal,
updateProgress: async () => {},
updateTokens: async () => {},
+1
View File
@@ -92,6 +92,7 @@ async function makeFakeJob(opts: FakeJobOpts): Promise<{ jobId: number; ctx: Min
data: { prompt: opts.prompt, model: opts.model, allowed_tools: opts.allowed_tools },
attempts_made: 0,
signal: abortCtrl.signal,
deadlineAtMs: null,
shutdownSignal: shutdownCtrl.signal,
updateProgress: async () => {},
updateTokens: async (t) => { tokenSink.push(t); },
@@ -68,7 +68,7 @@ async function makeJob(prompt: string, model: string): Promise<{ jobId: number;
const jobId = rows[0].id;
const ctx: MinionJobContext = {
id: jobId, name: 'subagent', data: { prompt, model }, attempts_made: 1,
signal: new AbortController().signal, shutdownSignal: new AbortController().signal,
signal: new AbortController().signal, deadlineAtMs: null, shutdownSignal: new AbortController().signal,
updateProgress: async () => {}, updateTokens: async () => {}, log: async () => {},
isActive: async () => true, readInbox: async () => [],
};
+1
View File
@@ -48,6 +48,7 @@ function fakeJob(data: Record<string, unknown>): MinionJobContext {
data,
attempts_made: 0,
signal: controller.signal,
deadlineAtMs: null,
shutdownSignal: controller.signal,
updateProgress: async () => {},
updateTokens: async () => {},
+1
View File
@@ -58,6 +58,7 @@ function makeJob(data: Record<string, unknown>): MinionJobContext {
data,
attempts_made: 1,
signal: new AbortController().signal,
deadlineAtMs: null,
shutdownSignal: new AbortController().signal,
updateProgress: async () => {},
updateTokens: async () => {},
+1
View File
@@ -52,6 +52,7 @@ function makeCtx(
data,
attempts_made: 0,
signal: opts.signal ?? new AbortController().signal,
deadlineAtMs: null,
shutdownSignal: opts.shutdownSignal ?? new AbortController().signal,
updateProgress: async () => {},
updateTokens: async () => {},
+1
View File
@@ -40,6 +40,7 @@ function ctxWithInbox(
data,
attempts_made: 0,
signal: new AbortController().signal,
deadlineAtMs: null,
shutdownSignal: new AbortController().signal,
async updateProgress(p: unknown) { progress.push(p); },
async updateTokens() {},
+1
View File
@@ -90,6 +90,7 @@ async function makeCtx(input: unknown): Promise<MinionJobContext> {
data: (input as Record<string, unknown>) ?? {},
attempts_made: 0,
signal: ac.signal,
deadlineAtMs: null,
shutdownSignal: shutdown.signal,
async updateProgress() {},
async updateTokens() {},