diff --git a/CHANGELOG.md b/CHANGELOG.md index bc89ae4f7..b564e3bfd 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,42 @@ All notable changes to GBrain will be documented in this file. +## [0.42.15.0] - 2026-06-02 + +**Commands print real data when you run them from a subagent, a pipe, or cron, not just when you have a terminal.** A handful of gbrain commands quietly changed what they printed based on whether a terminal was attached. Run them from a coding agent, a `| cat` pipe, or a cron job and you'd get JSON when you wanted human text, or nothing useful at all. The read commands (`get`, `list`, `search`, `query`) were already fine; this fixes the ones that weren't. + +The clearest case was `gbrain jobs watch`. In a terminal you got the live dashboard; piped or from a subagent you got an endless stream of JSON with no way to a human view. Now the rule is simple and the same for every command: **human output by default, JSON only when you pass `--json`.** A terminal still controls cosmetics (the live cursor-managed dashboard, colors), never the data. + +``` +gbrain jobs watch # terminal: live dashboard. piped: one human snapshot, then exits. +gbrain jobs watch --json # one JSON snapshot (machine-readable) +gbrain jobs watch --follow # stream continuously (human, or JSONL with --json) +``` + +`jobs watch` split into two independent knobs: `--json` picks the format, `--follow` picks the cadence. Non-interactive runs print one snapshot and exit (clean for capture), so a subagent that just wants the current queue state gets it in one shot instead of a loop it has to kill. + +### What else changed + +- **`gbrain reindex --code` refusal is now readable.** When it declines to spend money re-embedding without `--yes` in a non-interactive shell, it now prints a plain-English refusal instead of a JSON error blob (JSON only with `--json`). The safety behavior is unchanged: it still refuses and exits 2 rather than spending unconfirmed. +- **The eval commands stop hiding why they did less work.** `gbrain eval cross-modal` and `gbrain eval takes-quality` run fewer cycles non-interactively (a deliberate cost guard). They already printed the cycle count; now the line says *why* it's low and how to change it: `cycles: 1 (non-interactive default; --cycles N for more)`. Same for the `$1` budget default on `gbrain eval suspected-contradictions` (`--budget-usd N to raise`). + +Nothing here changes a TTY/interactive session: the live `jobs watch` dashboard, prompts, and your normal terminal output are all identical. + +## To take advantage of v0.42.15.0 + +`gbrain upgrade` is all you need. There's no migration and no schema change. + +1. **Update:** + ```bash + gbrain upgrade + ``` +2. **Verify the fix:** run a command non-interactively and confirm you get real output. + ```bash + gbrain jobs watch -.json`. `--batch [--limit N] [--concurrent N] [--max-usd FLOAT] [--yes]` fans out cross-modal scoring across a LongMemEval-shape JSONL; mutually exclusive with `--task` (fail-fast usage error if both set); filters `kind: "by_type_summary"` rows; pre-flight cost estimate refuses if `> --max-usd` without `--yes` (default cap 5.00 USD). Semaphore-bounded fan-out via inline `runWithLimit(items, limit, fn)` (exported for unit tests): max N questions in-flight × 3 model slots = ceiling of 3N parallel API calls (default `--concurrent 3` → 9). Per-question receipts land in a per-batch tempdir and are deleted at end of run; the summary receipt inlines per-question verdicts as JSON, not file paths. Exit precedence (batch-level policy, NOT inherited from aggregate.ts): ERROR > FAIL > INCONCLUSIVE > PASS. DI seam: `runEvalCrossModal(args, opts?: {runEval?: typeof runEval})` mirrors `runEvalLongMemEval(args, {client?})`; tests pass `opts.runEval` to bypass real LLM calls AND the gateway availability check. Pinned by `test/eval-cross-modal-batch.test.ts`. +- `src/commands/eval-cross-modal.ts` — multi-model quality gate. Three different-provider frontier models score the OUTPUT against the TASK on a 5-dim list. Verdict `pass` (exit 0) / `fail` (exit 1) / `inconclusive` (exit 2; <2/3 model successes). Reuses `src/core/ai/gateway.ts:chat()` so config/auth/aliasing comes from the gateway recipe registry — no parallel provider stack. Self-configures the gateway (`configureGateway(loadConfig() + process.env)`) since the cli.ts dispatch bypasses `connectEngine()`. Default cycles 3 in TTY, 1 in non-TTY (partial cost guardrail) via the shared `resolveCycleDefault(explicit, isTty)` in `src/core/eval/cycle-default.ts`; the cost-estimate banner appends `cycleDefaultSuffix(...)` (`for 1 cycle(s) (non-interactive default; --cycles N for more)`) when the value is the silent non-TTY fallback, so the 1-vs-3 difference isn't hidden. Receipts land at `gbrainPath('eval-receipts')/-.json`. `--batch [--limit N] [--concurrent N] [--max-usd FLOAT] [--yes]` fans out cross-modal scoring across a LongMemEval-shape JSONL; mutually exclusive with `--task` (fail-fast usage error if both set); filters `kind: "by_type_summary"` rows; pre-flight cost estimate refuses if `> --max-usd` without `--yes` (default cap 5.00 USD). Semaphore-bounded fan-out via inline `runWithLimit(items, limit, fn)` (exported for unit tests): max N questions in-flight × 3 model slots = ceiling of 3N parallel API calls (default `--concurrent 3` → 9). Per-question receipts land in a per-batch tempdir and are deleted at end of run; the summary receipt inlines per-question verdicts as JSON, not file paths. Exit precedence (batch-level policy, NOT inherited from aggregate.ts): ERROR > FAIL > INCONCLUSIVE > PASS. DI seam: `runEvalCrossModal(args, opts?: {runEval?: typeof runEval})` mirrors `runEvalLongMemEval(args, {client?})`; tests pass `opts.runEval` to bypass real LLM calls AND the gateway availability check. Pinned by `test/eval-cross-modal-batch.test.ts`. +- `src/core/eval/cycle-default.ts` — single source of truth for the eval cycle-count default. Exports `DEFAULT_CYCLES_TTY = 3`, `DEFAULT_CYCLES_NONTTY = 1`, `resolveCycleDefault(explicit, isTty): {cycles, usedNonTtyDefault}`, and `cycleDefaultSuffix(r)` (returns ` (non-interactive default; --cycles N for more)` only when the non-TTY default was applied, else `''`). Consumed by `eval-cross-modal.ts`, `eval-takes-quality.ts` (run + regress), and `takes-quality-eval/runner.ts` (core uses only the constant — library stays TTY-agnostic; the CLI owns the TTY=3 upgrade + banner annotation). `eval-suspected-contradictions.ts` applies the same transparency to its `$5`/`$1` budget default via a `budgetUsdExplicit` flag (the budget is overwritten in-place so explicitness can't be inferred post-hoc). Not shared with `resolveWorkersWithClamp` (different domain, no engine, no dedup). Pinned by `test/eval/cycle-default.test.ts`, `test/eval-suspected-contradictions-budget-default.test.ts`. - `src/core/cross-modal-eval/json-repair.ts` — `parseModelJSON(raw)` named export with a 4-strategy fallback chain (direct parse → fence-strip → trailing-comma + single-quote + embedded-newline repair → regex nuclear option). Adversarial input throws rather than fabricating scores — the aggregator treats a throw as "this model contributed nothing this cycle" so the gate stays correct at >=2/3 successes. - `src/core/cross-modal-eval/aggregate.ts` — pure verdict logic. Pass criterion: `(successes >= 2) AND (every dim mean >= 7) AND (every dim min across models >= 5)`. Inconclusive when <2/3 models returned parseable scores (regression guard for the v1 `Object.values({}).every(...) === true` empty-array PASS bug). - `src/core/cross-modal-eval/runner.ts` — orchestrator. Each cycle runs `Promise.allSettled([gwChat(slotA), gwChat(slotB), gwChat(slotC)])` (bare allSettled, no rate-leases for the CLI path). Stops early on PASS or INCONCLUSIVE; runs up to 3 cycles. Default slots: `openai:gpt-4o` / `anthropic:claude-opus-4-7` / `google:gemini-1.5-pro`. `estimateCost()` exports a small per-model pricing table (drifts; refresh alongside model-family bumps). @@ -230,7 +231,7 @@ per-release `**vX.Y.Z:**` narration — CI enforces this - `src/core/minions/attachments.ts` — Attachment validation (path traversal, null byte, oversize, base64, duplicate detection). - `src/commands/agent.ts` — `gbrain agent run [flags]` CLI. Submits `subagent` (or N children + 1 aggregator) under `{allowProtectedSubmit: true}`. Single-entry `--fanout-manifest` short-circuits. Children get `on_child_fail: 'continue'` + `max_stalled: 3`. `--follow` is the default on TTY; streams logs + polls `waitForCompletion` in parallel. Ctrl-C detaches, does not cancel. - `src/commands/agent-logs.ts` — `gbrain agent logs [--follow] [--since]`. Merges JSONL heartbeat audit + `subagent_messages` into a chronological timeline. `parseSince` accepts ISO-8601 or relative (`5m`, `1h`, `2d`). Transcript tail renders only for terminal jobs. -- `src/commands/jobs.ts` — `gbrain jobs` CLI subcommands + `gbrain jobs work` daemon. `case 'work'` wraps `worker.start()` in try/finally and owns engine lifecycle — calls `engine.disconnect()` on shutdown with loud error logging (the worker must not disconnect an engine it doesn't own; pool slots free immediately on shutdown rather than waiting for TCP keepalive). `jobs submit` surfaces the full `MinionJobInput` retry/backoff/timeout/idempotency surface as flags: `--max-stalled`, `--backoff-type fixed|exponential`, `--backoff-delay`, `--backoff-jitter`, `--timeout-ms`, `--idempotency-key`. `jobs smoke --sigkill-rescue` is the SIGKILL-rescue regression guard. `registerBuiltinHandlers` always registers `subagent` + `subagent_aggregator` (no env flag — `ANTHROPIC_API_KEY` is the cost gate, trust is via `PROTECTED_JOB_NAMES`) and loads `GBRAIN_PLUGIN_PATH` plugins at startup with a loud per-plugin line; `shell` handler still gated by `GBRAIN_ALLOW_SHELL_JOBS=1` (RCE surface). The `autopilot-cycle` handler forwards `job.data.phases` to `runCycle`, validated against `ALL_PHASES` from `src/core/cycle.ts` (invalid names filtered; empty/missing falls back to the default cycle). The `sync` handler resolves `sourceId` at entry from `sources.local_path` (mirrors `cycle.ts:480`) so multi-source brains read the per-source `last_commit` anchor; concurrency routes through `autoConcurrency()` in `src/core/sync-concurrency.ts` (PGLite stays serial); `noEmbed` default is `true`. `gbrain jobs supervisor status` at `jobs.ts:803-826` consumes `summarizeCrashes()` from `src/core/minions/handlers/supervisor-audit.ts` for parity with `gbrain doctor`: JSON adds `crashes_by_cause: {runtime_error, oom_or_external_kill, unknown, legacy}` + `clean_exits_24h`; human output gains per-cause + clean-exits lines. Pinned by 4 source-grep wiring assertions in `test/doctor.test.ts` requiring `crashes_by_cause` + `clean_exits_24h=` in both `doctor.ts` and `jobs.ts`. +- `src/commands/jobs.ts` — `gbrain jobs` CLI subcommands + `gbrain jobs work` daemon. `case 'work'` wraps `worker.start()` in try/finally and owns engine lifecycle — calls `engine.disconnect()` on shutdown with loud error logging (the worker must not disconnect an engine it doesn't own; pool slots free immediately on shutdown rather than waiting for TCP keepalive). `jobs submit` surfaces the full `MinionJobInput` retry/backoff/timeout/idempotency surface as flags: `--max-stalled`, `--backoff-type fixed|exponential`, `--backoff-delay`, `--backoff-jitter`, `--timeout-ms`, `--idempotency-key`. `jobs smoke --sigkill-rescue` is the SIGKILL-rescue regression guard. `registerBuiltinHandlers` always registers `subagent` + `subagent_aggregator` (no env flag — `ANTHROPIC_API_KEY` is the cost gate, trust is via `PROTECTED_JOB_NAMES`) and loads `GBRAIN_PLUGIN_PATH` plugins at startup with a loud per-plugin line; `shell` handler still gated by `GBRAIN_ALLOW_SHELL_JOBS=1` (RCE surface). The `autopilot-cycle` handler forwards `job.data.phases` to `runCycle`, validated against `ALL_PHASES` from `src/core/cycle.ts` (invalid names filtered; empty/missing falls back to the default cycle). The `sync` handler resolves `sourceId` at entry from `sources.local_path` (mirrors `cycle.ts:480`) so multi-source brains read the per-source `last_commit` anchor; concurrency routes through `autoConcurrency()` in `src/core/sync-concurrency.ts` (PGLite stays serial); `noEmbed` default is `true`. `gbrain jobs supervisor status` at `jobs.ts:803-826` consumes `summarizeCrashes()` from `src/core/minions/handlers/supervisor-audit.ts` for parity with `gbrain doctor`: JSON adds `crashes_by_cause: {runtime_error, oom_or_external_kill, unknown, legacy}` + `clean_exits_24h`; human output gains per-cause + clean-exits lines. Pinned by 4 source-grep wiring assertions in `test/doctor.test.ts` requiring `crashes_by_cause` + `clean_exits_24h=` in both `doctor.ts` and `jobs.ts`. `gbrain jobs watch` decouples its two output axes: `--json` picks FORMAT (human default, never gated on isTTY), `--follow` picks LOOP (default `isTTY && !json`). Non-TTY with no flags prints ONE human snapshot then exits (clean for subagent/pipe/cron); `--follow` opts into a continuous stream (human plain per tick, or JSONL with `--json`); a TTY with no flags keeps the live ANSI dashboard. Resolution is the pure `resolveWatchMode(opts, isTTY): {json, follow, useAnsiDashboard}` in `src/commands/jobs-watch.ts`; the dispatch wires `--follow`. Pinned by `test/jobs-watch-mode.test.ts` (format×loop matrix incl. the TTY+`--json`-one-shot case) + `test/e2e/non-tty-output.serial.test.ts` (the `cmd =1.3.10" }, "license": "MIT", - "version": "0.42.14.0" + "version": "0.42.15.0" } diff --git a/src/commands/eval-cross-modal.ts b/src/commands/eval-cross-modal.ts index 5ceacf87d..4d3898fce 100644 --- a/src/commands/eval-cross-modal.ts +++ b/src/commands/eval-cross-modal.ts @@ -24,6 +24,7 @@ import { createHash } from 'crypto'; import { gbrainPath, loadConfig } from '../core/config.ts'; import { configureGateway, isAvailable } from '../core/ai/gateway.ts'; import { runWithLimit } from '../core/worker-pool.ts'; +import { resolveCycleDefault, cycleDefaultSuffix } from '../core/eval/cycle-default.ts'; import { DEFAULT_DIMENSIONS, DEFAULT_SLOTS, @@ -342,7 +343,10 @@ export async function runEvalCrossModal(args: string[], opts: RunCrossModalOpts } const slug = parsed.slug ?? inferSlugFromOutputPath(parsed.output); - const cycles = parsed.cycles ?? (isTTY() ? 3 : 1); + // #1784: resolve the cycle default once; annotate the cost banner below when + // it's the silent non-TTY fallback so the 1-vs-3 difference isn't a surprise. + const cycleDef = resolveCycleDefault(parsed.cycles, isTTY()); + const cycles = cycleDef.cycles; const dimensions = parsed.dimensions ?? DEFAULT_DIMENSIONS; const receiptDir = parsed.receiptDir ?? gbrainPath('eval-receipts'); const maxTokens = parsed.maxTokens ?? 4000; @@ -372,7 +376,7 @@ export async function runEvalCrossModal(args: string[], opts: RunCrossModalOpts const cost = estimateCost(slots, cycles, maxTokens); process.stderr.write( `[eval cross-modal] estimated cost: ~$${cost.perCycleUSD.toFixed(2)}/cycle, ` + - `~$${cost.perRunMaxUSD.toFixed(2)} max for ${cycles} cycle(s).\n`, + `~$${cost.perRunMaxUSD.toFixed(2)} max for ${cycles} cycle(s)${cycleDefaultSuffix(cycleDef)}.\n`, ); for (const note of cost.notes) { process.stderr.write(`[eval cross-modal] note: ${note}\n`); diff --git a/src/commands/eval-suspected-contradictions.ts b/src/commands/eval-suspected-contradictions.ts index 3c4966473..553e08565 100644 --- a/src/commands/eval-suspected-contradictions.ts +++ b/src/commands/eval-suspected-contradictions.ts @@ -62,6 +62,12 @@ interface ParsedFlags { judge?: string; limit?: number; budgetUsd: number; + /** + * #1784: true when --budget-usd was passed explicitly. The TTY-derived + * default ($5 TTY / $1 non-TTY) is overwritten in-place, so explicitness + * can't be inferred post-hoc — track it here to annotate the banner. + */ + budgetUsdExplicit: boolean; output?: string; maxPairChars: number; sampling: 'deterministic' | 'score-first'; @@ -78,7 +84,7 @@ interface ParsedFlags { help: boolean; } -function parseFlags(args: string[]): ParsedFlags { +export function parseFlags(args: string[]): ParsedFlags { // Sub-subcommand: first positional that doesn't start with -- let sub: 'run' | 'trend' | 'review' = 'run'; const rest: string[] = []; @@ -99,6 +105,7 @@ function parseFlags(args: string[]): ParsedFlags { // judge intentionally undefined here — resolved in runRun via resolveModel // so config keys + tier defaults govern. CLI --judge flag wins when set. budgetUsd: isTty ? 5 : 1, + budgetUsdExplicit: false, maxPairChars: 1500, sampling: 'deterministic', noCache: false, @@ -122,7 +129,7 @@ function parseFlags(args: string[]): ParsedFlags { else if (arg === '--top-k') f.topK = Number.parseInt(next(), 10); else if (arg === '--judge') f.judge = next(); else if (arg === '--limit') f.limit = Number.parseInt(next(), 10); - else if (arg === '--budget-usd') f.budgetUsd = Number.parseFloat(next()); + else if (arg === '--budget-usd') { f.budgetUsd = Number.parseFloat(next()); f.budgetUsdExplicit = true; } else if (arg === '--output') f.output = next(); else if (arg === '--max-pair-chars') f.maxPairChars = Number.parseInt(next(), 10); else if (arg === '--sampling') { @@ -264,8 +271,13 @@ async function runRun(engine: BrainEngine, f: ParsedFlags): Promise { fallback: 'anthropic:claude-haiku-4-5', }); + // #1784: annotate the budget when it's the silent non-TTY default ($1) so the + // 5-vs-1 difference isn't a surprise to pipe / cron / subagent callers. + const budgetSuffix = (process.stdout.isTTY !== true && !f.budgetUsdExplicit) + ? ' (non-interactive default; --budget-usd N to raise)' + : ''; console.error( - `Contradiction probe: ${queries.length} queries, top-${f.topK}, judge=${judgeModel}, budget=$${f.budgetUsd.toFixed(2)}.`, + `Contradiction probe: ${queries.length} queries, top-${f.topK}, judge=${judgeModel}, budget=$${f.budgetUsd.toFixed(2)}${budgetSuffix}.`, ); // v0.34 / Lane C: cost-estimate prompt — TTY-only Ctrl-C window before diff --git a/src/commands/eval-takes-quality.ts b/src/commands/eval-takes-quality.ts index 4d16142d8..b63303982 100644 --- a/src/commands/eval-takes-quality.ts +++ b/src/commands/eval-takes-quality.ts @@ -23,6 +23,7 @@ import type { BrainEngine } from '../core/engine.ts'; import { configureGateway } from '../core/ai/gateway.ts'; import { loadConfig } from '../core/config.ts'; import { runEval, DEFAULT_MODEL_PANEL } from '../core/takes-quality-eval/runner.ts'; +import { resolveCycleDefault, cycleDefaultSuffix } from '../core/eval/cycle-default.ts'; import { writeReceipt } from '../core/takes-quality-eval/receipt-write.ts'; import { loadReceiptFromDisk } from '../core/takes-quality-eval/replay.ts'; import { compareReceipts } from '../core/takes-quality-eval/regress.ts'; @@ -138,7 +139,11 @@ export async function runEvalTakesQuality(engine: BrainEngine, args: string[]): if (subcmd === 'run') { const limit = parseIntFlag(argv, '--limit', 100); - const cycles = parseIntFlag(argv, '--cycles', process.stdout.isTTY ? 3 : 1); + // #1784: keep parseIntFlag for value validation; resolveCycleDefault drives + // the banner annotation when the value is the silent non-TTY fallback. + const cycleDef = resolveCycleDefault(undefined, process.stdout.isTTY === true); + const cycles = parseIntFlag(argv, '--cycles', cycleDef.cycles); + const cyclesSuffix = getFlag(argv, '--cycles') === undefined ? cycleDefaultSuffix(cycleDef) : ''; const budgetStr = getFlag(argv, '--budget-usd'); const budgetUsd = budgetStr === undefined ? null : Number(budgetStr); if (budgetStr !== undefined && !Number.isFinite(budgetUsd)) { @@ -153,7 +158,7 @@ export async function runEvalTakesQuality(engine: BrainEngine, args: string[]): if (!json) { process.stderr.write( `[eval takes-quality] sampling ${limit} take(s) from ${source}; ` + - `panel: ${models.join(', ')}; cycles: ${cycles}` + + `panel: ${models.join(', ')}; cycles: ${cycles}${cyclesSuffix}` + (budgetUsd === null ? '' : `; budget: $${budgetUsd.toFixed(2)}`) + '\n', ); @@ -208,11 +213,14 @@ export async function runEvalTakesQuality(engine: BrainEngine, args: string[]): process.exit(2); } const limit = parseIntFlag(argv, '--limit', 100); - const cycles = parseIntFlag(argv, '--cycles', process.stdout.isTTY ? 3 : 1); + // #1784: same annotation treatment as the run subcommand. + const cycleDef = resolveCycleDefault(undefined, process.stdout.isTTY === true); + const cycles = parseIntFlag(argv, '--cycles', cycleDef.cycles); + const cyclesSuffix = getFlag(argv, '--cycles') === undefined ? cycleDefaultSuffix(cycleDef) : ''; const prior = loadReceiptFromDisk(againstPath); if (!json) { - process.stderr.write(`[eval takes-quality regress] running fresh eval to compare against ${againstPath}\n`); + process.stderr.write(`[eval takes-quality regress] running fresh eval (cycles: ${cycles}${cyclesSuffix}) to compare against ${againstPath}\n`); } const result = await runEval(engine, { limit, diff --git a/src/commands/jobs-watch.ts b/src/commands/jobs-watch.ts index ec0dd9381..0c5d5105e 100644 --- a/src/commands/jobs-watch.ts +++ b/src/commands/jobs-watch.ts @@ -9,15 +9,27 @@ * 60fps; 1s keeps the SQL load nominal even when multiple watch sessions * point at the same brain). * - * Rendering: manual ANSI cursor management (no TUI dep). Clears the - * screen on first render, then redraws from the top each tick using - * cursor-home + erase-down. On non-TTY (cron / wrapped redirect), - * falls through to one snapshot line per tick in `--progress-json` - * shape so wrappers can parse. + * Two independent axes (v0.42.11.0, #1784 — decoupled from `isTTY`): + * - FORMAT (what data prints): human by default, JSON only when `--json` is + * passed. NEVER gated on isTTY. + * - LOOP (cadence): `--follow` streams continuously; default is `isTTY` — + * continuous live dashboard in a terminal, ONE snapshot then exit when + * non-TTY (pipe / cron / subagent). Identical data either way, so defaulting + * the loop from isTTY is a cosmetic UX call, not a data gate. * - * Quit: Ctrl-C (SIGINT), 'q', or stdin close — the watcher restores the - * cursor + clears its own region on shutdown so the terminal isn't left - * with a half-rendered dashboard. + * Resulting matrix: + * TTY, no flags → live ANSI dashboard (cursor-managed, loops) + * non-TTY, no flags → ONE human plain-text snapshot, exit + * any + --json → JSON snapshot (one-shot, or JSONL stream w/ --follow) + * any + --follow → continuous (human plain per tick, or JSONL w/ --json) + * + * Rendering: manual ANSI cursor management (no TUI dep) for the live dashboard + * only. Clears the screen on first render, then redraws from the top each tick + * using cursor-home + erase-down. + * + * Quit: in the live dashboard, Ctrl-C (SIGINT) or 'q' restores the cursor + + * clears its region. Non-TTY one-shots (nothing to quit); a non-TTY `--follow` + * stream runs until the process is killed. * * No SSE consumer in v0.41 — local polling against the brain engine is * the foundation. SSE wiring through `serve-http.ts` is filed as a @@ -188,32 +200,63 @@ export async function readSnapshot(engine: BrainEngine): Promise export interface WatchOptions { /** Refresh interval. Default 1000ms. */ refreshMs?: number; - /** Stream JSON snapshots to stdout (non-TTY mode). */ + /** FORMAT axis: emit JSON instead of human text. Default human. Explicit only. */ json?: boolean; + /** + * LOOP axis: stream continuously. Default = `process.stdout.isTTY` — live + * dashboard in a terminal, one snapshot then exit when non-TTY. Pass `true` + * to force a continuous stream even off-TTY (cron tail / log pipe). + */ + follow?: boolean; +} + +export interface WatchMode { + /** FORMAT: emit JSON instead of human text. */ + json: boolean; + /** LOOP: continuous stream vs one-shot. */ + follow: boolean; + /** Live cursor-managed colored dashboard (TTY + human + looping only). */ + useAnsiDashboard: boolean; } /** - * Main entrypoint for `gbrain jobs watch`. Runs until SIGINT or 'q' - * keypress (on TTY). Non-TTY mode loops with --progress-json output. + * Pure resolver for the format × loop matrix (extracted for unit-testing the + * exact TTY-gating contract this command fixes, #1784). The data printed never + * depends on isTTY; only the loop cadence + ANSI cursor management do. + * + * follow default = `isTTY && !json`: a terminal human view is the live + * dashboard (loops), but `--json` (any) and non-TTY both one-shot unless the + * caller passes `--follow` explicitly. Matches the file-header matrix. + */ +export function resolveWatchMode(opts: WatchOptions, isTTY: boolean): WatchMode { + const json = opts.json === true; // FORMAT: explicit only — never from isTTY. + const follow = opts.follow ?? (isTTY && !json); + const useAnsiDashboard = isTTY && !json && follow; + return { json, follow, useAnsiDashboard }; +} + +/** + * Main entrypoint for `gbrain jobs watch`. See the file header for the + * format (`--json`) × loop (`--follow`) matrix. The data printed never depends + * on isTTY; only the loop cadence and the ANSI cursor management do. */ export async function runWatch(engine: BrainEngine, opts: WatchOptions = {}): Promise { const refreshMs = opts.refreshMs ?? 1000; - const isTTY = process.stdout.isTTY === true; - const json = opts.json || !isTTY; + const { json, follow, useAnsiDashboard } = resolveWatchMode(opts, process.stdout.isTTY === true); let stopped = false; const stop = () => { stopped = true; }; - if (isTTY && !json) { + if (useAnsiDashboard) { process.stdout.write(ANSI.cursorHide + ANSI.clear + ANSI.cursorHome); process.on('SIGINT', () => { process.stdout.write(ANSI.cursorShow + ANSI.clear + ANSI.cursorHome); stop(); process.exit(0); }); - // Read stdin for 'q' keypress. + // Read stdin for 'q' keypress (terminal-only affordance). if (process.stdin.isTTY && process.stdin.setRawMode) { process.stdin.setRawMode(true); process.stdin.resume(); @@ -227,15 +270,19 @@ export async function runWatch(engine: BrainEngine, opts: WatchOptions = {}): Pr } } - while (!stopped) { + do { const snap = await readSnapshot(engine); if (json) { process.stdout.write(JSON.stringify({ event: 'jobs.watch.snapshot', ...snap }) + '\n'); - } else { - // TTY: clear + cursor-home + render. + } else if (useAnsiDashboard) { + // Live dashboard: clear + cursor-home + colored render. process.stdout.write(ANSI.cursorHome + ANSI.eraseDown); process.stdout.write(renderSnapshot(snap, { useAnsi: true })); + } else { + // Non-TTY (or --follow without a terminal): plain human snapshot, no ANSI. + process.stdout.write(renderSnapshot(snap, { useAnsi: false }) + '\n'); } + if (!follow) break; // one-shot: render once, exit. await new Promise(r => setTimeout(r, refreshMs)); - } + } while (!stopped); } diff --git a/src/commands/jobs.ts b/src/commands/jobs.ts index c20e72189..8dd5599e1 100644 --- a/src/commands/jobs.ts +++ b/src/commands/jobs.ts @@ -1098,14 +1098,17 @@ HANDLER TYPES (built in) } case 'watch': { - // v0.41 D2 — live TTY dashboard (or JSON snapshots on non-TTY). + // v0.41 D2 — live dashboard; v0.42.11.0 (#1784) decoupled output from TTY. + // Flags: --json (FORMAT, human default), --follow (LOOP, default=isTTY so + // non-TTY one-shots), --refresh-ms=N. Non-TTY no-flag → one human snapshot. try { await queue.ensureSchema(); } catch (e) { console.error(e instanceof Error ? e.message : String(e)); process.exit(1); } const { runWatch } = await import('./jobs-watch.ts'); const refreshArg = args.find(a => a.startsWith('--refresh-ms=')); const refreshMs = refreshArg ? parseInt(refreshArg.split('=')[1] ?? '1000', 10) : 1000; const json = hasFlag(args, '--json'); - await runWatch(engine, { refreshMs, json }); + const follow = hasFlag(args, '--follow') ? true : undefined; // undefined → default to isTTY + await runWatch(engine, { refreshMs, json, follow }); break; } diff --git a/src/commands/reindex-code.ts b/src/commands/reindex-code.ts index 905fd2909..a1dc04db5 100644 --- a/src/commands/reindex-code.ts +++ b/src/commands/reindex-code.ts @@ -376,6 +376,45 @@ export async function runReindexCode( }; } +/** + * v0.42.11.0 (#1784) — what to print when the cost gate refuses to spend + * non-interactively without `--yes`. The REFUSAL (exit 2, no spend) is the + * guardrail and is correct; the FORMAT is a separate axis. Pre-#1784 this path + * always emitted a JSON envelope even without `--json`, violating the repo's + * "human by default" convention. Now: JSON only when `--json` is explicit; + * otherwise a human refusal on stderr. Pure + exported so it's unit-testable + * without a brain or a real cost preview. + */ +export interface CostRefusal { + stdout?: string; + stderr?: string; +} +export function buildCostRefusal(opts: { + json: boolean; + previewMsg: string; + preview: unknown; + costUsd: number; + model: string; +}): CostRefusal { + if (opts.json) { + const envelope = serializeError(errorFor({ + class: 'ConfirmationRequired', + code: 'cost_preview_requires_yes', + message: opts.previewMsg, + hint: 'Pass --yes to proceed, or --dry-run to see the preview and exit 0.', + })); + return { + stdout: JSON.stringify({ error: envelope, preview: opts.preview, costUsd: opts.costUsd, model: opts.model }), + }; + } + return { + stderr: + `${opts.previewMsg}\n` + + 'Refusing to re-embed non-interactively without confirmation. ' + + 'Pass --yes to proceed, or --dry-run for the preview (exit 0).', + }; +} + /** * CLI entrypoint. Parses argv, wires cost-preview gate + JSON/TTY branching, * delegates to runReindexCode. Exit codes: 0 on success/dry-run, 2 on @@ -456,13 +495,11 @@ export async function runReindexCodeCli(engine: BrainEngine, args: string[]): Pr if (!yes) { const isTTY = Boolean(process.stdout.isTTY) && Boolean(process.stdin.isTTY); if (!isTTY || json) { - const envelope = serializeError(errorFor({ - class: 'ConfirmationRequired', - code: 'cost_preview_requires_yes', - message: previewMsg, - hint: 'Pass --yes to proceed, or --dry-run to see the preview and exit 0.', - })); - console.log(JSON.stringify({ error: envelope, preview, costUsd, model: getEmbeddingModelName() })); + // Guardrail unchanged: refuse + exit 2, no spend. Only the FORMAT splits + // on --json now (human refusal on stderr otherwise) — #1784. + const refusal = buildCostRefusal({ json, previewMsg, preview, costUsd, model: getEmbeddingModelName() }); + if (refusal.stdout) console.log(refusal.stdout); + if (refusal.stderr) console.error(refusal.stderr); process.exit(2); } console.log(previewMsg); diff --git a/src/core/eval/cycle-default.ts b/src/core/eval/cycle-default.ts new file mode 100644 index 000000000..d76aafd66 --- /dev/null +++ b/src/core/eval/cycle-default.ts @@ -0,0 +1,65 @@ +/** + * v0.42.11.0 (#1784) — single source of truth for the eval cycle-count default. + * + * Several eval commands (`eval cross-modal`, `eval takes-quality run/regress`) + * and the takes-quality runner core resolved their cycle default as + * `process.stdout.isTTY ? 3 : 1`. The non-TTY value of 1 is a deliberate + * cost-conservative default (each cycle calls frontier models), but the split + * was SILENT — a subagent / pipe / cron run got 1 with nothing explaining why, + * which is the surprise issue #1784 names. + * + * The fix is NOT a new stderr notice line — those commands already print the + * resolved cycle count in their existing banner. Instead, callers ANNOTATE that + * existing banner via `cycleDefaultSuffix` when the value came from the non-TTY + * default, so the operator sees `cycles: 1 (non-interactive default; --cycles N + * for more)` instead of a bare `cycles: 1`. + * + * The runner CORE (`takes-quality-eval/runner.ts`) consumes only + * `DEFAULT_CYCLES_NONTTY` — library code stays TTY-agnostic; the CLI layer owns + * the TTY=3 upgrade + the banner annotation. + * + * Deliberately NOT shared with `resolveWorkersWithClamp` (sync-concurrency.ts): + * different domain, no engine, no per-process dedup. Sharing would be premature. + */ + +/** Interactive (TTY) default: deeper eval, more model calls. */ +export const DEFAULT_CYCLES_TTY = 3; + +/** Non-interactive (pipe / cron / subagent) default: cost-conservative. */ +export const DEFAULT_CYCLES_NONTTY = 1; + +export interface CycleResolution { + /** The effective cycle count. */ + cycles: number; + /** + * True ONLY when no explicit value was given AND we are non-TTY — i.e. the + * caller fell through to `DEFAULT_CYCLES_NONTTY`. This is the case worth + * annotating in the banner so the 1-vs-3 difference isn't silent. + */ + usedNonTtyDefault: boolean; +} + +/** + * Resolve the cycle count from an explicit value + TTY-ness. + * + * explicit set → {explicit, false} (caller asked; no annotation) + * undefined+TTY → {3, false} (interactive default; visible live) + * undefined+!TTY→ {1, true} (cost-safe default; ANNOTATE) + */ +export function resolveCycleDefault( + explicit: number | undefined, + isTty: boolean, +): CycleResolution { + if (explicit !== undefined) return { cycles: explicit, usedNonTtyDefault: false }; + if (isTty) return { cycles: DEFAULT_CYCLES_TTY, usedNonTtyDefault: false }; + return { cycles: DEFAULT_CYCLES_NONTTY, usedNonTtyDefault: true }; +} + +/** + * Banner suffix to append to an EXISTING stderr line that already prints the + * cycle count. Empty string unless the non-TTY default was applied, so the + * common (TTY or explicit) cases get no extra text. + */ +export function cycleDefaultSuffix(r: CycleResolution): string { + return r.usedNonTtyDefault ? ' (non-interactive default; --cycles N for more)' : ''; +} diff --git a/src/core/takes-quality-eval/runner.ts b/src/core/takes-quality-eval/runner.ts index 09fe2b90b..9b18c41d6 100644 --- a/src/core/takes-quality-eval/runner.ts +++ b/src/core/takes-quality-eval/runner.ts @@ -31,6 +31,7 @@ import { } from './receipt-name.ts'; import type { TakesQualityReceipt } from './receipt.ts'; import { estimateCost, getPricing, PricingNotFoundError } from './pricing.ts'; +import { DEFAULT_CYCLES_NONTTY } from '../eval/cycle-default.ts'; export const DEFAULT_MODEL_PANEL = [ 'openai:gpt-4o', @@ -145,7 +146,10 @@ async function callOneModel( export async function runEval(engine: BrainEngine, opts: RunOpts = {}): Promise { const limit = opts.limit ?? 100; - const cycles = opts.cycles ?? (process.stdout.isTTY ? 3 : 1); + // Library core stays TTY-agnostic (#1784): default to the cost-conservative + // value. The CLI layer (eval-takes-quality.ts) owns the TTY=3 upgrade + the + // banner annotation; it always passes an explicit `cycles` down. + const cycles = opts.cycles ?? DEFAULT_CYCLES_NONTTY; const models = opts.models ?? DEFAULT_MODEL_PANEL; const budgetUsd = opts.budgetUsd ?? null; const source = opts.source ?? 'db'; diff --git a/test/e2e/non-tty-output.serial.test.ts b/test/e2e/non-tty-output.serial.test.ts new file mode 100644 index 000000000..aee2bf945 --- /dev/null +++ b/test/e2e/non-tty-output.serial.test.ts @@ -0,0 +1,61 @@ +/** + * v0.42.11.0 (#1784) — non-TTY output contract E2E (the `cmd { + let home: string; + let env: NodeJS.ProcessEnv; + + beforeAll(() => { + home = mkdtempSync(join(tmpdir(), 'gbrain-nontty-e2e-')); + env = { ...process.env, GBRAIN_HOME: home }; + // Fresh local PGLite brain so `jobs watch` has an engine to read. + execFileSync('bun', ['run', 'src/cli.ts', 'init', '--pglite', '--no-embedding', '--non-interactive'], { + cwd: process.cwd(), env, stdio: 'ignore', + }); + }, 60_000); + + afterAll(() => { + try { rmSync(home, { recursive: true, force: true }); } catch { /* best-effort */ } + }); + + test('non-TTY, no flags → ONE human snapshot, non-empty, not JSON, exit 0', () => { + // stdin 'ignore' + stdout 'pipe' makes the child non-TTY on both ends. + const out = execFileSync('bun', ['run', 'src/cli.ts', 'jobs', 'watch'], { + cwd: process.cwd(), env, stdio: ['ignore', 'pipe', 'ignore'], encoding: 'utf-8', timeout: 30_000, + }); + expect(out.length).toBeGreaterThan(0); // the bug was (no output) + expect(out).toContain('gbrain jobs watch'); // human renderer header + expect(out).toContain('Queue'); // human panel + expect(out.trimStart().startsWith('{')).toBe(false); // NOT JSON by default + }, 35_000); + + test('non-TTY, --json → ONE JSON snapshot, parses, exit 0', () => { + const out = execFileSync('bun', ['run', 'src/cli.ts', 'jobs', 'watch', '--json'], { + cwd: process.cwd(), env, stdio: ['ignore', 'pipe', 'ignore'], encoding: 'utf-8', timeout: 30_000, + }); + const firstLine = out.split('\n').find(l => l.trim().length > 0) ?? ''; + const parsed = JSON.parse(firstLine); + expect(parsed.event).toBe('jobs.watch.snapshot'); + expect(parsed.queue_health).toBeDefined(); + }, 35_000); +}); diff --git a/test/eval-suspected-contradictions-budget-default.test.ts b/test/eval-suspected-contradictions-budget-default.test.ts new file mode 100644 index 000000000..1477fd52f --- /dev/null +++ b/test/eval-suspected-contradictions-budget-default.test.ts @@ -0,0 +1,37 @@ +/** + * v0.42.11.0 (#1784) — suspected-contradictions budget-default detection. + * + * The TTY-derived budget default ($5 TTY / $1 non-TTY) is overwritten in-place, + * so the banner needs an explicit `budgetUsdExplicit` flag to know whether to + * annotate the value as the silent non-interactive default. These run non-TTY + * (bun test pipes stdout), so the default resolves to $1. + */ +import { describe, test, expect } from 'bun:test'; +import { parseFlags } from '../src/commands/eval-suspected-contradictions.ts'; + +describe('parseFlags budgetUsdExplicit', () => { + test('no --budget-usd → non-TTY default $1, not flagged explicit', () => { + const f = parseFlags([]); + expect(f.budgetUsd).toBe(1); // bun test is non-TTY + expect(f.budgetUsdExplicit).toBe(false); + }); + + test('--budget-usd 3 → value 3, flagged explicit', () => { + const f = parseFlags(['--budget-usd', '3']); + expect(f.budgetUsd).toBe(3); + expect(f.budgetUsdExplicit).toBe(true); + }); + + test('explicit value equal to the default still counts as explicit', () => { + const f = parseFlags(['--budget-usd', '1']); + expect(f.budgetUsd).toBe(1); + expect(f.budgetUsdExplicit).toBe(true); + }); + + test('subcommand positional does not break flag parsing', () => { + const f = parseFlags(['run', '--budget-usd', '2']); + expect(f.sub).toBe('run'); + expect(f.budgetUsd).toBe(2); + expect(f.budgetUsdExplicit).toBe(true); + }); +}); diff --git a/test/eval/cycle-default.test.ts b/test/eval/cycle-default.test.ts new file mode 100644 index 000000000..49cc4760b --- /dev/null +++ b/test/eval/cycle-default.test.ts @@ -0,0 +1,60 @@ +/** + * v0.42.11.0 (#1784) — cycle-default resolver unit tests. + * + * Pins the 4 branches of resolveCycleDefault + the banner-suffix contract. + * Pure functions, no DB, no env mutation. + */ +import { describe, test, expect } from 'bun:test'; +import { + resolveCycleDefault, + cycleDefaultSuffix, + DEFAULT_CYCLES_TTY, + DEFAULT_CYCLES_NONTTY, +} from '../../src/core/eval/cycle-default.ts'; + +describe('resolveCycleDefault', () => { + test('explicit value wins (TTY) — no annotation', () => { + const r = resolveCycleDefault(5, true); + expect(r).toEqual({ cycles: 5, usedNonTtyDefault: false }); + }); + + test('explicit value wins (non-TTY) — no annotation', () => { + const r = resolveCycleDefault(5, false); + expect(r).toEqual({ cycles: 5, usedNonTtyDefault: false }); + }); + + test('TTY default → 3, no annotation', () => { + const r = resolveCycleDefault(undefined, true); + expect(r).toEqual({ cycles: DEFAULT_CYCLES_TTY, usedNonTtyDefault: false }); + expect(r.cycles).toBe(3); + }); + + test('non-TTY default → 1, ANNOTATED (the silent-degradation case)', () => { + const r = resolveCycleDefault(undefined, false); + expect(r).toEqual({ cycles: DEFAULT_CYCLES_NONTTY, usedNonTtyDefault: true }); + expect(r.cycles).toBe(1); + }); + + test('explicit 1 in non-TTY is NOT flagged as the default (user asked for it)', () => { + const r = resolveCycleDefault(1, false); + expect(r.usedNonTtyDefault).toBe(false); + }); +}); + +describe('cycleDefaultSuffix', () => { + test('empty unless the non-TTY default was applied', () => { + expect(cycleDefaultSuffix({ cycles: 3, usedNonTtyDefault: false })).toBe(''); + expect(cycleDefaultSuffix({ cycles: 5, usedNonTtyDefault: false })).toBe(''); + }); + + test('names the --cycles override when annotated', () => { + const s = cycleDefaultSuffix({ cycles: 1, usedNonTtyDefault: true }); + expect(s).toContain('non-interactive default'); + expect(s).toContain('--cycles'); + }); + + test('round-trips with resolveCycleDefault non-TTY path', () => { + expect(cycleDefaultSuffix(resolveCycleDefault(undefined, false))).not.toBe(''); + expect(cycleDefaultSuffix(resolveCycleDefault(undefined, true))).toBe(''); + }); +}); diff --git a/test/jobs-watch-mode.test.ts b/test/jobs-watch-mode.test.ts new file mode 100644 index 000000000..0e506fca1 --- /dev/null +++ b/test/jobs-watch-mode.test.ts @@ -0,0 +1,44 @@ +/** + * v0.42.15.0 (#1784) — jobs-watch format × loop matrix. + * + * Pins resolveWatchMode so the TTY-gating contract can't regress. The bug this + * guards: TTY + --json used to loop forever (follow defaulted to isTTY); the + * documented matrix says --json one-shots unless --follow. Caught by codex in + * the pre-landing review; the e2e missed it (non-TTY only). + */ +import { describe, test, expect } from 'bun:test'; +import { resolveWatchMode } from '../src/commands/jobs-watch.ts'; + +describe('resolveWatchMode — format × loop matrix', () => { + test('TTY, no flags → live dashboard (json off, follow on, ansi on)', () => { + expect(resolveWatchMode({}, true)).toEqual({ json: false, follow: true, useAnsiDashboard: true }); + }); + + test('non-TTY, no flags → one human snapshot (the bug fix: not JSON, one-shot)', () => { + expect(resolveWatchMode({}, false)).toEqual({ json: false, follow: false, useAnsiDashboard: false }); + }); + + test('TTY + --json → ONE JSON snapshot, NOT a forever loop (codex finding)', () => { + expect(resolveWatchMode({ json: true }, true)).toEqual({ json: true, follow: false, useAnsiDashboard: false }); + }); + + test('non-TTY + --json → one JSON snapshot', () => { + expect(resolveWatchMode({ json: true }, false)).toEqual({ json: true, follow: false, useAnsiDashboard: false }); + }); + + test('TTY + --follow → live dashboard loops', () => { + expect(resolveWatchMode({ follow: true }, true)).toEqual({ json: false, follow: true, useAnsiDashboard: true }); + }); + + test('non-TTY + --follow → plain human stream (loops, no ansi)', () => { + expect(resolveWatchMode({ follow: true }, false)).toEqual({ json: false, follow: true, useAnsiDashboard: false }); + }); + + test('--json --follow → JSONL stream, never the ansi dashboard', () => { + expect(resolveWatchMode({ json: true, follow: true }, true)).toEqual({ json: true, follow: true, useAnsiDashboard: false }); + }); + + test('explicit --follow false on a TTY → one-shot (override wins)', () => { + expect(resolveWatchMode({ follow: false }, true)).toEqual({ json: false, follow: false, useAnsiDashboard: false }); + }); +}); diff --git a/test/reindex-cost-refusal.test.ts b/test/reindex-cost-refusal.test.ts new file mode 100644 index 000000000..3e778abf6 --- /dev/null +++ b/test/reindex-cost-refusal.test.ts @@ -0,0 +1,52 @@ +/** + * v0.42.11.0 (#1784) — reindex-code cost-refusal format unit tests. + * + * The cost gate still REFUSES to spend non-interactively without --yes (exit 2, + * tested at the CLI layer). This pins the separate FORMAT axis: JSON only when + * --json is explicit; otherwise a human refusal on stderr. Pure function — no + * brain, no real cost preview. + */ +import { describe, test, expect } from 'bun:test'; +import { buildCostRefusal } from '../src/commands/reindex-code.ts'; + +const PREVIEW_MSG = 'reindex-code: 42 code page(s), ~12,345 tokens, est. $0.12 on voyage:voyage-code-3.'; + +describe('buildCostRefusal', () => { + test('--json → machine-readable envelope on stdout, nothing on stderr', () => { + const r = buildCostRefusal({ + json: true, + previewMsg: PREVIEW_MSG, + preview: { totalPages: 42, totalTokens: 12345 }, + costUsd: 0.12, + model: 'voyage:voyage-code-3', + }); + expect(r.stderr).toBeUndefined(); + expect(typeof r.stdout).toBe('string'); + const parsed = JSON.parse(r.stdout!); + expect(parsed.error).toBeDefined(); + // The structured refusal must still carry the confirmation code + context. + expect(JSON.stringify(parsed.error)).toContain('cost_preview_requires_yes'); + expect(parsed.costUsd).toBe(0.12); + expect(parsed.model).toBe('voyage:voyage-code-3'); + expect(parsed.preview).toEqual({ totalPages: 42, totalTokens: 12345 }); + }); + + test('no --json → human refusal on stderr, nothing on stdout (not JSON)', () => { + const r = buildCostRefusal({ + json: false, + previewMsg: PREVIEW_MSG, + preview: {}, + costUsd: 0.12, + model: 'voyage:voyage-code-3', + }); + expect(r.stdout).toBeUndefined(); + expect(typeof r.stderr).toBe('string'); + // Human text: includes the preview, the refusal reason, and both escape hatches. + expect(r.stderr).toContain(PREVIEW_MSG); + expect(r.stderr).toContain('Refusing to re-embed'); + expect(r.stderr).toContain('--yes'); + expect(r.stderr).toContain('--dry-run'); + // Must NOT be a JSON envelope. + expect(r.stderr!.trimStart().startsWith('{')).toBe(false); + }); +});