mirror of
https://github.com/garrytan/gbrain.git
synced 2026-07-27 22:15:33 +00:00
* feat(progress): step 1 - shared ProgressReporter + CliOptions Adds the foundation for v0.14.2's bulk-action progress streaming work: - src/core/progress.ts: dependency-free reporter with auto/human/json/quiet modes, TTY-aware rendering, time+item rate gating, heartbeat helper for slow single queries, dot-composed child phases, EPIPE defense (both sync throw and async 'error' event), and a singleton module-level signal coordinator so SIGINT/SIGTERM emits abort events for all live phases without leaking per-instance listeners. - src/core/cli-options.ts: parseGlobalFlags() for --quiet / --progress-json / --progress-interval=<ms> (both space and = forms), plus cliOptsToProgressOptions() that resolves to the right mode. Non-TTY default is human-plain one-line-per-event; JSON is explicit opt-in so shell pipelines don't suddenly see structured noise. - test/progress.test.ts (17 cases): mode resolution, rate gating, no-fake- totals on heartbeat paths, EPIPE paths, SIGINT singleton, child phase composition. - test/cli-options.test.ts (14 cases): flag parsing, invalid values, interleaved flags, mode resolution. Follow-ups wire doctor/embed/files/export/extract/import/sync/migrate/ repair-jsonb/backlinks/orphans/lint/integrity/eval/autopilot/jobs plus the apply-migrations orchestrators through this reporter, and route Minion handlers to job.updateProgress instead of stderr. See the plan in ~/.claude/plans/. 1682 unit tests pass. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(progress): step 2 - wire global flags into cli.ts Parse --quiet / --progress-json / --progress-interval from argv BEFORE command dispatch, strip them, stash resolved CliOptions on a module-level singleton (same pattern as Commander's program.opts()) and on every OperationContext created for shared-op dispatch. - src/cli.ts: parseGlobalFlags(rawArgs) at the top of main(); setCliOptions once; dispatch sees only the stripped argv. Fixes the "gbrain --progress-json doctor" unknown-command case that Codex flagged. - src/core/cli-options.ts: expose setCliOptions/getCliOptions/ _resetCliOptionsForTest singleton. Commands that want progress call getCliOptions() to construct their reporter. - src/core/operations.ts: OperationContext gains optional cliOpts field so shared-op handlers (and MCP-invoked ops that need a reporter) can read the same settings. MCP callers leave it undefined and consumers default to quiet. - test/cli-options.test.ts: +4 cases covering singleton round-trip and an integration smoke spawning `bun src/cli.ts --progress-json --version` to prove the global flag survives dispatch. 45 relevant unit tests pass (progress + cli-options + cli.test.ts). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(progress): step 3a - doctor + orphans heartbeat streaming Doctor on a 52K-page brain used to sit silent for 10+ minutes while the DB checks ran, then get killed by an agent timeout. Wired through the new reporter so agents see which check is running and the slow ones heartbeat every second. doctor.ts: - Start a single `doctor.db_checks` phase around the DB section, with a per-check heartbeat before each step (connection, pgvector, rls, schema_version, embeddings, graph_coverage, integrity, jsonb_integrity, markdown_body_completeness). - jsonb_integrity now scans 5 targets, not 4: added page_versions. frontmatter so the check surface matches `repair-jsonb` (per Codex review of the plan — the old 4-target scan missed a known repair site). Per-target heartbeat so 50K-row scans show incremental progress. - markdown_body_completeness: wrap the existing query in a 1s heartbeat timer. The regex scan over rd.data ->> 'content' can't be paginated usefully; this just lets agents see life during the sequential scan. No fake totals — the LIMIT 100 query has no meaningful total count. - integrity sample: same heartbeat pattern around the 500-page scan. orphans.ts: - findOrphans() wraps the NOT EXISTS anti-join in a 1s heartbeat. Keyset pagination was considered and rejected: without an index on links.to_page_id it's no faster than the full scan, and may re-plan the anti-join per batch. A schema migration adding that index is the right fix and is queued for v0.14.3. Follow-ups: - Step 3b: wire embed/files/export (the \r-only stdout offenders). - Step 5: end-to-end progress test spawning `gbrain doctor --progress-json` against a fixture brain, asserting stderr events and clean stdout. All existing unit tests continue to pass (76/76 in doctor + orphans + progress + cli-options). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(progress): step 3b - embed + files + export stderr progress Replaces the \r-on-stdout progress pattern in the three worst offenders (embed, files sync, export) with the shared reporter on stderr. Stdout now carries only final summaries, so scripts and tests that grep for counts ("Embedded N chunks", "Files sync complete", "Exported N pages") still work when output is piped. - embed.ts: runEmbedCore accepts an optional onProgress callback. The CLI wrapper builds a reporter and passes reporter.tick(); Minion handlers will pass job.updateProgress in Step 4. Worker-pool is single-threaded JS so no rate-gate race (per Codex review #18). - files.ts syncFiles(): tick per file; summary preserved on stdout. - export.ts: tick per page; summary preserved on stdout. Also fixes a --quiet flag collision. `skillpack-check` has its own --quiet mode (suppress all stdout). parseGlobalFlags strips --quiet globally now, and skillpack-check reads the resolved CliOptions singleton via getCliOptions() instead of re-parsing argv. Test updated to match the stripping behavior. 1686 unit tests pass. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(progress): step 3c - extract + import + sync reporter streaming Extract, import, and sync now stream per-file progress to stderr through the shared reporter. All three kept their stdout summaries + JSON action-events intact so existing tests + agent scripts are unaffected. - extract.ts (4 paths: links/timeline × fs/db): replaced the ad-hoc `process.stderr.write({event:"progress"...})` lines with reporter ticks. Same channel (stderr), canonical schema now, visible in both text and --json modes. Stdout action-events (`add_link` / `add_timeline`) untouched — tests grep them. - import.ts: the logProgress() function that printed every 100 files to stdout is now a progress.tick() call per file. Rate-gated by the reporter. Stdout still gets the final "Import complete (Xs)" summary and the --json payload. - sync.ts: three new phases (`sync.deletes`, `sync.renames`, `sync.imports`) tick per file, so big syncs show each step rather than a single end-of-run summary. Phase hierarchy ready to be child()-chained into runImport / runEmbed later, per Codex review #26. Updated the #132 nested-transaction regression test in test/sync.test.ts to also accept the new hoisted-loop shape — the guarantee (this loop is not wrapped in engine.transaction) still holds. 1686 unit tests pass. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(progress): step 3d - migrate/repair/backlinks/lint/integrity/eval Wires the remaining bulk commands through the reporter: - migrate-engine: phase starts (migrate.copy_pages, migrate.copy_links), per-page tick. Old \"Progress: N/total\" stdout logs replaced by stderr ticks; final stdout summary preserved. - repair-jsonb: per-column start + a heartbeat timer while each UPDATE runs (minutes on 50K-row tables). CRITICAL: stdout stays clean so migrations/v0_12_2.ts's JSON.parse(child.stdout) still works. Per Codex review #12. - backlinks: 1s heartbeat around findBacklinkGaps() (sync double-walk of the brain dir). - lint: tick per page; per-issue stdout output preserved. - integrity auto: tick per page in the main resolver loop. The separate ~/.gbrain/integrity-progress.jsonl resume marker is untouched (its role shifts from live progress reporting to resume-only). - eval: add an onProgress option to core's runEval(), CLI wraps with a reporter. Phases: eval.single / eval.ab. Tick per query. core/search/eval.ts gains a RunEvalOptions type so future callers (MCP eval op, Minion handlers) can also hook in without the reporter. 1686 unit tests pass. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(progress): step 3e - onProgress callbacks on core libs - src/core/embedding.ts: embedBatch() gains an optional EmbedBatchOptions.onBatchComplete callback, fired after each 100-item sub-batch. CLI wrappers pass reporter.tick; Minion handlers can pass job.updateProgress. - src/core/enrichment-service.ts: enrichEntities() config gains onProgress(done, total, name) fired after each entity. Same split: CLI -> reporter, Minion -> DB-backed progress. No CLI behavior change on its own. Wiring these callbacks into the Minion handlers is Step 4. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(progress): step 4 - orchestrators + upgrade + minion handlers - cli-options.ts: childGlobalFlags() returns the flag suffix to append to child gbrain subprocesses. Empty string by default, " --quiet --progress-json" when the parent has them set, so child behavior inherits the parent's progress-mode without scattering string-concat logic across every execSync site. - migrations/v0_12_2.ts: each execSync inherits the parent's global flags. Phase C (repair-jsonb --dry-run --json) pins explicit stdio to ['ignore','pipe','inherit'] so child stderr streams straight through while stdout stays captured for JSON.parse. Per Codex review #12. - migrations/v0_12_0.ts + v0_11_0.ts: same childGlobalFlags wiring at each gbrain-subcommand execSync. - upgrade.ts: post-upgrade timeout bumped 300s → 30min (1_800_000 ms) with GBRAIN_POST_UPGRADE_TIMEOUT_MS override. The old 300s cap killed v0.12.0 graph-backfill migrations on 50K+ brains; the heartbeat wiring added in v0.14.2 makes long waits observable, so a generous ceiling no longer means users stare at a silent terminal. - jobs.ts: the embed Minion handler passes job.updateProgress as the onProgress callback, so per-job progress is durable in minion_jobs and readable via `gbrain jobs get <id>`. Primary Minion progress channel is DB-backed — stderr from `jobs work` stays coarse for daemon liveness only. Per Codex review #20. 1686 unit tests pass. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(progress): step 5 - E2E doctor-progress test + CI guard scripts/check-progress-to-stdout.sh greps src/ for the banned `process.stdout.write('\r…')` pattern that v0.14.2 removed from the bulk-action codepaths. Wired into the `bun run test` script so any future regression that puts progress back on stdout fails fast. An empty allowlist documents the position: every known call site was migrated; new exceptions need a rationale in the allowlist. test/e2e/doctor-progress.test.ts (Tier 1, needs Postgres + pgvector): - `gbrain --progress-json doctor --json`: stderr carries JSONL progress events with the canonical {event, phase, ts} shape, starts + finishes for `doctor.db_checks`. Stdout stays parseable JSON — no progress pollution. - `gbrain doctor` (no flag): human-plain progress goes to stderr only, stdout stays free of `[doctor.db_checks]`. - `gbrain --quiet doctor`: reporter emits nothing; doctor still runs to completion. test/cli-options.test.ts: +2 spawning integration tests. One verifies `gbrain --progress-json --version` keeps stdout clean of progress events (single-shot commands that don't use a reporter aren't affected). One guards the skillpack-check --quiet regression — --quiet suppresses stdout by reading the resolved CliOptions singleton, not re-parsing argv. Full test matrix: bun run test -> 1726 pass / 184 skipped (no DB) / 0 fail bun run test:e2e -> 136 pass / 13 skipped / 0 fail Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(progress): step 6 - docs + v0.14.2 release bump - VERSION + package.json bumped to 0.14.2. - docs/progress-events.md (new): canonical JSON event schema reference. Stable from v0.14.2, additive only. Lists every phase name shipped in this release, the five event types (start/tick/heartbeat/finish/ abort), the TTY/non-TTY rendering rules, subprocess inheritance semantics, and the Minion DB-backed progress model. - CLAUDE.md: "Bulk-action progress reporting" section under the build instructions; Key files entries for src/core/progress.ts, src/core/cli-options.ts, scripts/check-progress-to-stdout.sh, and docs/progress-events.md; doctor.ts entry updated to note the v0.14.2 5-target jsonb_integrity scan + heartbeat wiring. - CHANGELOG.md v0.14.2: full release summary per project voice rules. The "numbers that matter" table, per-command before/after grid, backward-compat warnings for stdout→stderr moves, and an itemized changes section covering reporter/CLI plumbing/schema/Minion handlers/doctor fixes/upgrade timeout/CI guard/tests. No em dashes. Real file paths, real commands, real numbers. - skills/migrations/v0.14.2.md (new): agent migration note. Mechanical step is "nothing" since v0.14.2 is purely additive. Walks agents through the three new global flags, the 14 wired commands, the event schema cheat sheet, Minion progress via job.updateProgress, and scripts/verification commands. Full test matrix: bun run test (unit + guards) -> 1726 pass / 184 skipped / 0 fail bun run test:e2e (Postgres) -> 141 pass / 8 skipped / 0 fail Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore: bump version to 0.15.2, restore master's [0.14.2] CHANGELOG entry Master sits at 0.14.2 (reliability wave). This PR lands on top as 0.15.2 (progress streaming wave). Splits the merge-time combined CHANGELOG entry back into two discrete release sections so history stays honest: - [0.15.2] = progress reporter, CliOptions, 14 wired commands, Minion embed handler, doctor jsonb_integrity 5-target fix, upgrade timeout bump, CI guard, progress unit+E2E tests. - [0.14.2] = master's eight root-cause bug fixes, restored verbatim from origin/master. Touched files: - VERSION + package.json: 0.14.2 -> 0.15.2 (next patch off master). - skills/migrations/v0.14.2.md -> skills/migrations/v0.15.2.md (rename + rewrite frontmatter + body to v0.15.2). - CHANGELOG.md: split into two entries; progress-wave refs renamed v0.14.2 -> v0.15.2; reliability-wave entry restored from master. - src/core/progress.ts, src/commands/doctor.ts, src/commands/sync.ts, src/commands/upgrade.ts, docs/progress-events.md, test/sync.test.ts: progress-wave v0.14.2 references -> v0.15.2. The remaining v0.14.2 references in test/e2e/migration-flow.test.ts (Bug 3 context) and CLAUDE.md (reliability-wave key commands, Bug 3 ledger move) correctly point at master's 0.14.2 release. Test matrix after version bump: bun run test -> 1780 pass / 179 skipped / 0 fail Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
638 lines
28 KiB
TypeScript
638 lines
28 KiB
TypeScript
/**
|
|
* CLI handler for `gbrain jobs` subcommands.
|
|
* Thin wrapper around MinionQueue and MinionWorker.
|
|
*/
|
|
|
|
import type { BrainEngine } from '../core/engine.ts';
|
|
import { MinionQueue } from '../core/minions/queue.ts';
|
|
import { MinionWorker } from '../core/minions/worker.ts';
|
|
import type { MinionJob, MinionJobStatus } from '../core/minions/types.ts';
|
|
|
|
function parseFlag(args: string[], flag: string): string | undefined {
|
|
const idx = args.indexOf(flag);
|
|
return idx >= 0 && idx + 1 < args.length ? args[idx + 1] : undefined;
|
|
}
|
|
|
|
function hasFlag(args: string[], flag: string): boolean {
|
|
return args.includes(flag);
|
|
}
|
|
|
|
function formatJob(job: MinionJob): string {
|
|
const dur = job.finished_at && job.started_at
|
|
? `${((job.finished_at.getTime() - job.started_at.getTime()) / 1000).toFixed(1)}s`
|
|
: '—';
|
|
const stalled = job.status === 'active' && job.lock_until && job.lock_until < new Date()
|
|
? ' (stalled?)' : '';
|
|
return ` ${String(job.id).padEnd(6)} ${job.name.padEnd(14)} ${(job.status + stalled).padEnd(20)} ${job.queue.padEnd(10)} ${dur.padEnd(8)} ${job.created_at.toISOString().slice(0, 19)}`;
|
|
}
|
|
|
|
function formatJobDetail(job: MinionJob): string {
|
|
const lines = [
|
|
`Job #${job.id}: ${job.name} (${job.status.toUpperCase()}${job.status === 'dead' ? ` after ${job.attempts_made} attempts` : ''})`,
|
|
` Queue: ${job.queue} | Priority: ${job.priority}`,
|
|
` Attempts: ${job.attempts_made}/${job.max_attempts} (started: ${job.attempts_started})`,
|
|
` Backoff: ${job.backoff_type} ${job.backoff_delay}ms (jitter: ${job.backoff_jitter})`,
|
|
];
|
|
if (job.started_at) lines.push(` Started: ${job.started_at.toISOString()}`);
|
|
if (job.finished_at) lines.push(` Finished: ${job.finished_at.toISOString()}`);
|
|
if (job.lock_token) lines.push(` Lock: ${job.lock_token} (until ${job.lock_until?.toISOString()})`);
|
|
if (job.delay_until) lines.push(` Delayed until: ${job.delay_until.toISOString()}`);
|
|
if (job.parent_job_id) lines.push(` Parent: job #${job.parent_job_id} (on_child_fail: ${job.on_child_fail})`);
|
|
if (job.error_text) lines.push(` Error: ${job.error_text}`);
|
|
if (job.stacktrace.length > 0) {
|
|
lines.push(` History:`);
|
|
for (const entry of job.stacktrace) lines.push(` - ${entry}`);
|
|
}
|
|
if (job.progress != null) lines.push(` Progress: ${JSON.stringify(job.progress)}`);
|
|
if (job.result != null) lines.push(` Result: ${JSON.stringify(job.result)}`);
|
|
lines.push(` Data: ${JSON.stringify(job.data)}`);
|
|
return lines.join('\n');
|
|
}
|
|
|
|
export async function runJobs(engine: BrainEngine, args: string[]): Promise<void> {
|
|
const sub = args[0];
|
|
|
|
if (!sub || sub === '--help' || sub === '-h') {
|
|
console.log(`gbrain jobs — Minions job queue
|
|
|
|
USAGE
|
|
gbrain jobs submit <name> [--params JSON] [--follow] [--priority N]
|
|
[--delay Nms] [--max-attempts N] [--max-stalled N]
|
|
[--backoff-type fixed|exponential] [--backoff-delay Nms]
|
|
[--backoff-jitter 0..1] [--timeout-ms Nms]
|
|
[--idempotency-key K] [--queue Q] [--dry-run]
|
|
gbrain jobs list [--status S] [--queue Q] [--limit N]
|
|
gbrain jobs get <id>
|
|
gbrain jobs cancel <id>
|
|
gbrain jobs retry <id>
|
|
gbrain jobs prune [--older-than 30d]
|
|
gbrain jobs delete <id>
|
|
gbrain jobs stats
|
|
gbrain jobs smoke
|
|
gbrain jobs work [--queue Q] [--concurrency N]
|
|
|
|
HANDLER TYPES (built in)
|
|
sync Pull and embed new pages from the repo
|
|
embed (Re-)embed pages; --params '{"slug":...}' or '{"all":true}'
|
|
lint Run page linter; --params '{"dir":"...","fix":true}'
|
|
import Bulk import markdown; --params '{"dir":"..."}'
|
|
extract Extract links + timeline entries; '{"mode":"all"}'
|
|
backlinks Check or fix back-links; '{"action":"fix"}'
|
|
autopilot-cycle One autopilot pass (sync+extract+embed+backlinks)
|
|
shell Run a command or argv. Requires GBRAIN_ALLOW_SHELL_JOBS=1
|
|
on the worker. Params: {cmd?, argv?, cwd, env?}.
|
|
See: docs/guides/minions-shell-jobs.md
|
|
`);
|
|
return;
|
|
}
|
|
|
|
const queue = new MinionQueue(engine);
|
|
|
|
switch (sub) {
|
|
case 'submit': {
|
|
const name = args[1];
|
|
if (!name) {
|
|
console.error('Error: job name required. Usage: gbrain jobs submit <name>');
|
|
process.exit(1);
|
|
}
|
|
|
|
const paramsStr = parseFlag(args, '--params');
|
|
let data: Record<string, unknown> = {};
|
|
if (paramsStr) {
|
|
try { data = JSON.parse(paramsStr); }
|
|
catch { console.error('Error: --params must be valid JSON'); process.exit(1); }
|
|
}
|
|
|
|
const priority = parseInt(parseFlag(args, '--priority') ?? '0', 10);
|
|
const delay = parseInt(parseFlag(args, '--delay') ?? '0', 10);
|
|
const maxAttempts = parseInt(parseFlag(args, '--max-attempts') ?? '3', 10);
|
|
const maxStalledRaw = parseFlag(args, '--max-stalled');
|
|
const maxStalled = maxStalledRaw !== undefined ? parseInt(maxStalledRaw, 10) : undefined;
|
|
// v0.13.1 field audit: expose retry/backoff/timeout/idempotency knobs so
|
|
// users can tune Minions behavior without dropping into TypeScript.
|
|
const backoffTypeRaw = parseFlag(args, '--backoff-type');
|
|
const backoffType = backoffTypeRaw === 'fixed' || backoffTypeRaw === 'exponential'
|
|
? backoffTypeRaw
|
|
: undefined;
|
|
const backoffDelayRaw = parseFlag(args, '--backoff-delay');
|
|
const backoffDelay = backoffDelayRaw !== undefined ? parseInt(backoffDelayRaw, 10) : undefined;
|
|
const backoffJitterRaw = parseFlag(args, '--backoff-jitter');
|
|
const backoffJitter = backoffJitterRaw !== undefined ? parseFloat(backoffJitterRaw) : undefined;
|
|
const timeoutMsRaw = parseFlag(args, '--timeout-ms');
|
|
const timeoutMs = timeoutMsRaw !== undefined ? parseInt(timeoutMsRaw, 10) : undefined;
|
|
if (timeoutMsRaw !== undefined && (isNaN(timeoutMs!) || timeoutMs! <= 0)) {
|
|
console.error('Error: --timeout-ms must be a positive integer (milliseconds)');
|
|
process.exit(1);
|
|
}
|
|
const idempotencyKey = parseFlag(args, '--idempotency-key');
|
|
const queueName = parseFlag(args, '--queue') ?? 'default';
|
|
const dryRun = hasFlag(args, '--dry-run');
|
|
const follow = hasFlag(args, '--follow');
|
|
|
|
if (dryRun) {
|
|
console.log(`[DRY RUN] Would submit job:`);
|
|
console.log(` Name: ${name}`);
|
|
console.log(` Queue: ${queueName}`);
|
|
console.log(` Priority: ${priority}`);
|
|
console.log(` Max attempts: ${maxAttempts}`);
|
|
if (maxStalled !== undefined) console.log(` Max stalled: ${maxStalled}`);
|
|
if (backoffType) console.log(` Backoff type: ${backoffType}`);
|
|
if (backoffDelay !== undefined) console.log(` Backoff delay: ${backoffDelay}ms`);
|
|
if (backoffJitter !== undefined) console.log(` Backoff jitter: ${backoffJitter}`);
|
|
if (timeoutMs !== undefined) console.log(` Timeout: ${timeoutMs}ms`);
|
|
if (idempotencyKey) console.log(` Idempotency key: ${idempotencyKey}`);
|
|
if (delay > 0) console.log(` Delay: ${delay}ms`);
|
|
console.log(` Data: ${JSON.stringify(data)}`);
|
|
return;
|
|
}
|
|
|
|
try {
|
|
await queue.ensureSchema();
|
|
} catch (e) {
|
|
console.error(e instanceof Error ? e.message : String(e));
|
|
process.exit(1);
|
|
}
|
|
|
|
// The CLI path is a trusted submitter. Pass {allowProtectedSubmit: true}
|
|
// ONLY for protected names, not blanket-set for every submission, so any
|
|
// future protected name forces explicit opt-in at the call site.
|
|
const { isProtectedJobName } = await import('../core/minions/protected-names.ts');
|
|
const trusted = isProtectedJobName(name) ? { allowProtectedSubmit: true } : undefined;
|
|
const job = await queue.add(name, data, {
|
|
priority,
|
|
delay: delay > 0 ? delay : undefined,
|
|
max_attempts: maxAttempts,
|
|
max_stalled: maxStalled,
|
|
backoff_type: backoffType,
|
|
backoff_delay: backoffDelay,
|
|
backoff_jitter: backoffJitter,
|
|
timeout_ms: timeoutMs,
|
|
idempotency_key: idempotencyKey,
|
|
queue: queueName,
|
|
}, trusted);
|
|
|
|
// Submission audit log (operational trace, not forensic insurance).
|
|
try {
|
|
const { logShellSubmission } = await import('../core/minions/handlers/shell-audit.ts');
|
|
if (name.trim() === 'shell') {
|
|
logShellSubmission({
|
|
caller: 'cli',
|
|
remote: false,
|
|
job_id: job.id,
|
|
cwd: typeof data.cwd === 'string' ? data.cwd : '',
|
|
cmd_display: typeof data.cmd === 'string' ? data.cmd.slice(0, 80) : undefined,
|
|
argv_display: Array.isArray(data.argv)
|
|
? (data.argv as unknown[]).filter((a): a is string => typeof a === 'string').map((a) => a.slice(0, 80))
|
|
: undefined,
|
|
});
|
|
}
|
|
} catch { /* audit failures never block submission */ }
|
|
|
|
// Starvation warning (DX polish). Fire for every non-`--follow` shell submit
|
|
// regardless of the submitter's own `GBRAIN_ALLOW_SHELL_JOBS` — the submitter
|
|
// env is a weak proxy for the worker env (they may run on different machines),
|
|
// so the warning remains useful any time the job might sit in 'waiting'.
|
|
if (!follow && name.trim() === 'shell') {
|
|
process.stderr.write(
|
|
`\n⚠ Shell jobs require GBRAIN_ALLOW_SHELL_JOBS=1 on the worker process.\n` +
|
|
` Your job was queued (id=${job.id}) but will sit in 'waiting' until a\n` +
|
|
` worker with the env flag starts. To run now:\n\n` +
|
|
` GBRAIN_ALLOW_SHELL_JOBS=1 gbrain jobs submit shell \\\n` +
|
|
` --params '...' --follow\n\n` +
|
|
` Or start a persistent worker (Postgres only — PGLite uses --follow):\n\n` +
|
|
` GBRAIN_ALLOW_SHELL_JOBS=1 gbrain jobs work\n\n`,
|
|
);
|
|
}
|
|
|
|
if (follow) {
|
|
console.log(`Job #${job.id} submitted (${name}). Executing inline...`);
|
|
// Inline execution: run the job in this process
|
|
const worker = new MinionWorker(engine, { queue: queueName, pollInterval: 100 });
|
|
|
|
// Register built-in handlers
|
|
await registerBuiltinHandlers(worker, engine);
|
|
|
|
if (!worker.registeredNames.includes(name)) {
|
|
console.error(`Error: Unknown job type '${name}'.`);
|
|
console.error(`Available types: ${worker.registeredNames.join(', ')}`);
|
|
console.error(`Register custom types with worker.register('${name}', handler).`);
|
|
process.exit(1);
|
|
}
|
|
|
|
// Run worker for one job then stop
|
|
const startTime = Date.now();
|
|
const workerPromise = worker.start();
|
|
// Poll until this job completes
|
|
const pollInterval = setInterval(async () => {
|
|
const updated = await queue.getJob(job.id);
|
|
if (updated && ['completed', 'failed', 'dead', 'cancelled'].includes(updated.status)) {
|
|
worker.stop();
|
|
clearInterval(pollInterval);
|
|
}
|
|
}, 200);
|
|
await workerPromise;
|
|
clearInterval(pollInterval);
|
|
|
|
const final = await queue.getJob(job.id);
|
|
const elapsed = ((Date.now() - startTime) / 1000).toFixed(1);
|
|
if (final?.status === 'completed') {
|
|
console.log(`Job #${job.id} completed in ${elapsed}s`);
|
|
if (final.result) console.log(`Result: ${JSON.stringify(final.result)}`);
|
|
} else {
|
|
console.error(`Job #${job.id} ${final?.status}: ${final?.error_text}`);
|
|
process.exit(1);
|
|
}
|
|
} else {
|
|
console.log(JSON.stringify(job, null, 2));
|
|
}
|
|
break;
|
|
}
|
|
|
|
case 'list': {
|
|
const status = parseFlag(args, '--status') as MinionJobStatus | undefined;
|
|
const queueName = parseFlag(args, '--queue');
|
|
const limit = parseInt(parseFlag(args, '--limit') ?? '20', 10);
|
|
|
|
try { await queue.ensureSchema(); }
|
|
catch (e) { console.error(e instanceof Error ? e.message : String(e)); process.exit(1); }
|
|
|
|
const jobs = await queue.getJobs({ status, queue: queueName, limit });
|
|
|
|
if (jobs.length === 0) {
|
|
console.log('No jobs found.');
|
|
return;
|
|
}
|
|
|
|
console.log(` ${'ID'.padEnd(6)} ${'Name'.padEnd(14)} ${'Status'.padEnd(20)} ${'Queue'.padEnd(10)} ${'Time'.padEnd(8)} Created`);
|
|
console.log(' ' + '─'.repeat(80));
|
|
for (const job of jobs) console.log(formatJob(job));
|
|
console.log(`\n ${jobs.length} jobs shown`);
|
|
break;
|
|
}
|
|
|
|
case 'get': {
|
|
const id = parseInt(args[1], 10);
|
|
if (isNaN(id)) { console.error('Error: job ID required. Usage: gbrain jobs get <id>'); process.exit(1); }
|
|
|
|
try { await queue.ensureSchema(); }
|
|
catch (e) { console.error(e instanceof Error ? e.message : String(e)); process.exit(1); }
|
|
|
|
const job = await queue.getJob(id);
|
|
if (!job) { console.error(`Job #${id} not found.`); process.exit(1); }
|
|
console.log(formatJobDetail(job));
|
|
break;
|
|
}
|
|
|
|
case 'cancel': {
|
|
const id = parseInt(args[1], 10);
|
|
if (isNaN(id)) { console.error('Error: job ID required.'); process.exit(1); }
|
|
|
|
try { await queue.ensureSchema(); }
|
|
catch (e) { console.error(e instanceof Error ? e.message : String(e)); process.exit(1); }
|
|
|
|
const cancelled = await queue.cancelJob(id);
|
|
if (cancelled) {
|
|
console.log(`Job #${id} cancelled.`);
|
|
} else {
|
|
console.error(`Could not cancel job #${id} (may already be completed/dead).`);
|
|
process.exit(1);
|
|
}
|
|
break;
|
|
}
|
|
|
|
case 'retry': {
|
|
const id = parseInt(args[1], 10);
|
|
if (isNaN(id)) { console.error('Error: job ID required.'); process.exit(1); }
|
|
|
|
try { await queue.ensureSchema(); }
|
|
catch (e) { console.error(e instanceof Error ? e.message : String(e)); process.exit(1); }
|
|
|
|
const retried = await queue.retryJob(id);
|
|
if (retried) {
|
|
console.log(`Job #${id} re-queued for retry.`);
|
|
} else {
|
|
console.error(`Could not retry job #${id} (must be failed or dead).`);
|
|
process.exit(1);
|
|
}
|
|
break;
|
|
}
|
|
|
|
case 'delete': {
|
|
const id = parseInt(args[1], 10);
|
|
if (isNaN(id)) { console.error('Error: job ID required.'); process.exit(1); }
|
|
|
|
try { await queue.ensureSchema(); }
|
|
catch (e) { console.error(e instanceof Error ? e.message : String(e)); process.exit(1); }
|
|
|
|
const removed = await queue.removeJob(id);
|
|
if (removed) {
|
|
console.log(`Job #${id} deleted.`);
|
|
} else {
|
|
console.error(`Could not delete job #${id} (must be in a terminal status).`);
|
|
process.exit(1);
|
|
}
|
|
break;
|
|
}
|
|
|
|
case 'prune': {
|
|
const olderThanStr = parseFlag(args, '--older-than') ?? '30d';
|
|
const days = parseInt(olderThanStr, 10);
|
|
if (isNaN(days) || days <= 0) {
|
|
console.error('Error: --older-than must be a positive number (days). Example: --older-than 30d');
|
|
process.exit(1);
|
|
}
|
|
|
|
try { await queue.ensureSchema(); }
|
|
catch (e) { console.error(e instanceof Error ? e.message : String(e)); process.exit(1); }
|
|
|
|
const count = await queue.prune({ olderThan: new Date(Date.now() - days * 86400000) });
|
|
console.log(`Pruned ${count} jobs older than ${days} days.`);
|
|
break;
|
|
}
|
|
|
|
case 'stats': {
|
|
try { await queue.ensureSchema(); }
|
|
catch (e) { console.error(e instanceof Error ? e.message : String(e)); process.exit(1); }
|
|
|
|
const stats = await queue.getStats();
|
|
|
|
console.log('Job Stats (last 24h):');
|
|
if (stats.by_type.length > 0) {
|
|
console.log(` ${'Type'.padEnd(14)} ${'Total'.padEnd(7)} ${'Done'.padEnd(7)} ${'Failed'.padEnd(8)} ${'Dead'.padEnd(6)} Avg Time`);
|
|
for (const t of stats.by_type) {
|
|
const avgTime = t.avg_duration_ms != null ? `${(t.avg_duration_ms / 1000).toFixed(1)}s` : '—';
|
|
console.log(` ${t.name.padEnd(14)} ${String(t.total).padEnd(7)} ${String(t.completed).padEnd(7)} ${String(t.failed).padEnd(8)} ${String(t.dead).padEnd(6)} ${avgTime}`);
|
|
}
|
|
} else {
|
|
console.log(' No jobs in the last 24 hours.');
|
|
}
|
|
console.log(`\n Queue health: ${stats.queue_health.waiting} waiting, ${stats.queue_health.active} active, ${stats.queue_health.stalled} stalled`);
|
|
break;
|
|
}
|
|
|
|
case 'smoke': {
|
|
const startTime = Date.now();
|
|
try { await queue.ensureSchema(); }
|
|
catch (e) {
|
|
console.error(`SMOKE FAIL — schema init: ${e instanceof Error ? e.message : String(e)}`);
|
|
process.exit(1);
|
|
}
|
|
|
|
const sigkillRescue = hasFlag(args, '--sigkill-rescue');
|
|
|
|
const worker = new MinionWorker(engine, { queue: 'smoke', pollInterval: 100 });
|
|
worker.register('noop', async () => ({ ok: true, at: new Date().toISOString() }));
|
|
|
|
const job = await queue.add('noop', {}, { queue: 'smoke', max_attempts: 1 });
|
|
const workerPromise = worker.start();
|
|
|
|
const timeoutMs = 15000;
|
|
let final: MinionJob | null = null;
|
|
for (let elapsed = 0; elapsed < timeoutMs; elapsed += 100) {
|
|
await new Promise(r => setTimeout(r, 100));
|
|
final = await queue.getJob(job.id);
|
|
if (final && ['completed', 'failed', 'dead', 'cancelled'].includes(final.status)) break;
|
|
}
|
|
worker.stop();
|
|
await workerPromise;
|
|
|
|
const elapsedSec = ((Date.now() - startTime) / 1000).toFixed(2);
|
|
if (final?.status !== 'completed') {
|
|
console.error(`SMOKE FAIL — job #${job.id} status: ${final?.status ?? 'timeout'} (${elapsedSec}s elapsed)`);
|
|
if (final?.error_text) console.error(` Error: ${final.error_text}`);
|
|
process.exit(1);
|
|
}
|
|
|
|
// --sigkill-rescue: regression case for #219. Simulates a SIGKILL
|
|
// mid-flight by directly manipulating lock_until via handleStalled.
|
|
// Verifies that with the v0.13.1 schema default (max_stalled=5), a
|
|
// stalled job is REQUEUED rather than dead-lettered on first stall.
|
|
// Full subprocess-level SIGKILL lives in test/e2e/minions.test.ts.
|
|
if (sigkillRescue) {
|
|
const rescueJob = await queue.add('noop', {}, { queue: 'smoke' });
|
|
|
|
// Transition to active with a past lock_until, mimicking a worker
|
|
// that claimed and then got SIGKILL'd mid-run.
|
|
await engine.executeRaw(
|
|
`UPDATE minion_jobs
|
|
SET status='active',
|
|
lock_token='smoke-sigkill-rescue',
|
|
lock_until=now() - interval '1 minute',
|
|
started_at=now() - interval '2 minute',
|
|
attempts_started = attempts_started + 1
|
|
WHERE id=$1`,
|
|
[rescueJob.id]
|
|
);
|
|
|
|
const result = await queue.handleStalled();
|
|
const afterStall = await queue.getJob(rescueJob.id);
|
|
|
|
if (afterStall?.status === 'dead') {
|
|
console.error(
|
|
`SMOKE FAIL (--sigkill-rescue) — job #${rescueJob.id} was dead-lettered on first stall. ` +
|
|
`This is the #219 regression: schema default max_stalled should rescue, not dead-letter. ` +
|
|
`handleStalled: ${JSON.stringify(result)}`
|
|
);
|
|
process.exit(1);
|
|
}
|
|
if (afterStall?.status !== 'waiting') {
|
|
console.error(
|
|
`SMOKE FAIL (--sigkill-rescue) — unexpected status after stall: ${afterStall?.status}. ` +
|
|
`Expected 'waiting' (rescued). handleStalled: ${JSON.stringify(result)}`
|
|
);
|
|
process.exit(1);
|
|
}
|
|
try { await queue.removeJob(rescueJob.id); } catch { /* non-fatal cleanup */ }
|
|
}
|
|
|
|
const cfg = (await import('../core/config.ts')).loadConfig();
|
|
const engineLabel = cfg?.engine ?? 'unknown';
|
|
const tag = sigkillRescue ? ' + SIGKILL rescue' : '';
|
|
console.log(`SMOKE PASS — Minions healthy${tag} in ${elapsedSec}s (engine: ${engineLabel})`);
|
|
if (engineLabel === 'pglite') {
|
|
console.log('Note: the `gbrain jobs work` daemon requires Postgres. PGLite');
|
|
console.log('supports inline execution only (`submit --follow`).');
|
|
}
|
|
try { await queue.removeJob(job.id); } catch { /* non-fatal cleanup */ }
|
|
process.exit(0);
|
|
}
|
|
|
|
case 'work': {
|
|
// Check if PGLite
|
|
const config = (await import('../core/config.ts')).loadConfig();
|
|
if (config?.engine === 'pglite') {
|
|
console.error('Error: Worker daemon requires Postgres. PGLite uses an exclusive file lock that blocks other processes.');
|
|
console.error('Use --follow for inline execution: gbrain jobs submit <name> --follow');
|
|
process.exit(1);
|
|
}
|
|
|
|
const queueName = parseFlag(args, '--queue') ?? 'default';
|
|
const concurrency = parseInt(parseFlag(args, '--concurrency') ?? '1', 10);
|
|
|
|
try { await queue.ensureSchema(); }
|
|
catch (e) { console.error(e instanceof Error ? e.message : String(e)); process.exit(1); }
|
|
|
|
const worker = new MinionWorker(engine, { queue: queueName, concurrency });
|
|
await registerBuiltinHandlers(worker, engine);
|
|
|
|
console.log(`Minion worker started (queue: ${queueName}, concurrency: ${concurrency})`);
|
|
console.log(`Registered handlers: ${worker.registeredNames.join(', ')}`);
|
|
await worker.start();
|
|
break;
|
|
}
|
|
|
|
default:
|
|
console.error(`Unknown subcommand: ${sub}. Run 'gbrain jobs --help' for usage.`);
|
|
process.exit(1);
|
|
}
|
|
}
|
|
|
|
/**
|
|
* Register built-in job handlers.
|
|
*
|
|
* Handlers call library-level Core functions (runSyncCore via performSync,
|
|
* runExtractCore, runEmbedCore, runBacklinksCore) directly — NOT the CLI
|
|
* wrappers. CLI wrappers call process.exit(1) on validation errors; if a
|
|
* worker claimed a badly-formed job and ran one, the WORKER PROCESS would
|
|
* die and every in-flight job would go stalled. Library Cores throw
|
|
* instead, so one bad job fails one job — not the worker.
|
|
*
|
|
* Per the v0.11.1 plan (Codex architecture #5 — tension 3).
|
|
*/
|
|
export async function registerBuiltinHandlers(worker: MinionWorker, engine: BrainEngine): Promise<void> {
|
|
worker.register('sync', async (job) => {
|
|
const { performSync } = await import('./sync.ts');
|
|
const repoPath = typeof job.data.repoPath === 'string' ? job.data.repoPath : undefined;
|
|
const noPull = !!job.data.noPull;
|
|
const noEmbed = job.data.noEmbed !== false;
|
|
const result = await performSync(engine, { repoPath, noPull, noEmbed });
|
|
return result;
|
|
});
|
|
|
|
worker.register('embed', async (job) => {
|
|
const { runEmbedCore } = await import('./embed.ts');
|
|
// Primary Minion progress channel is job.updateProgress (DB-backed,
|
|
// readable via `gbrain jobs get <id>`). Stderr from the worker daemon
|
|
// only emits coarse job-start / job-done lines; per-page detail lives
|
|
// in the DB. Per Codex review #20.
|
|
await runEmbedCore(engine, {
|
|
slug: typeof job.data.slug === 'string' ? job.data.slug : undefined,
|
|
slugs: Array.isArray(job.data.slugs) ? (job.data.slugs as string[]) : undefined,
|
|
all: !!job.data.all,
|
|
stale: job.data.all ? false : (job.data.stale !== false),
|
|
onProgress: (done, total, embedded) => {
|
|
// Fire-and-forget: progress updates are best-effort and must not
|
|
// block the worker loop.
|
|
job.updateProgress({ done, total, embedded, phase: 'embed.pages' }).catch(() => {});
|
|
},
|
|
});
|
|
return { embedded: true };
|
|
});
|
|
|
|
worker.register('lint', async (job) => {
|
|
const { runLintCore } = await import('./lint.ts');
|
|
const target = typeof job.data.dir === 'string' ? job.data.dir : '.';
|
|
const result = await runLintCore({ target, fix: !!job.data.fix, dryRun: !!job.data.dryRun });
|
|
return result;
|
|
});
|
|
|
|
worker.register('import', async (job) => {
|
|
// import.ts Core extraction deferred to v0.12.0 (import has parallel
|
|
// workers + checkpointing). Keep the CLI wrapper call but note the
|
|
// worker-kill risk is bounded: import's only process.exit fires on
|
|
// a missing dir arg, which this handler always passes.
|
|
const { runImport } = await import('./import.ts');
|
|
const importArgs: string[] = [];
|
|
if (job.data.dir) importArgs.push(String(job.data.dir));
|
|
if (job.data.noEmbed) importArgs.push('--no-embed');
|
|
await runImport(engine, importArgs);
|
|
return { imported: true };
|
|
});
|
|
|
|
worker.register('extract', async (job) => {
|
|
const { runExtractCore } = await import('./extract.ts');
|
|
const mode = (typeof job.data.mode === 'string' && ['links', 'timeline', 'all'].includes(job.data.mode))
|
|
? (job.data.mode as 'links' | 'timeline' | 'all')
|
|
: 'all';
|
|
const dir = typeof job.data.dir === 'string'
|
|
? job.data.dir
|
|
: (await engine.getConfig('sync.repo_path')) ?? '.';
|
|
return await runExtractCore(engine, { mode, dir, dryRun: !!job.data.dryRun });
|
|
});
|
|
|
|
worker.register('backlinks', async (job) => {
|
|
const { runBacklinksCore } = await import('./backlinks.ts');
|
|
const action: 'check' | 'fix' = job.data.action === 'check' ? 'check' : 'fix';
|
|
const dir = typeof job.data.dir === 'string'
|
|
? job.data.dir
|
|
: (await engine.getConfig('sync.repo_path')) ?? '.';
|
|
return await runBacklinksCore({ action, dir, dryRun: !!job.data.dryRun });
|
|
});
|
|
|
|
// The killer handler. Autopilot submits ONE `autopilot-cycle` per cycle
|
|
// (idempotency_key on cycle slot) instead of a 4-job parent-child DAG,
|
|
// because Minions' parent/child is NOT a depends_on primitive (Codex
|
|
// H3/H4). Each step is wrapped in its own try/catch; the handler returns
|
|
// `{ partial: true, failed_steps: [...] }` when any step fails. It does
|
|
// NOT throw on partial failure — that would cause the Minion to retry,
|
|
// and an intermittent extract bug would block every future cycle.
|
|
worker.register('autopilot-cycle', async (job) => {
|
|
const { performSync } = await import('./sync.ts');
|
|
const { runExtractCore } = await import('./extract.ts');
|
|
const { runEmbedCore } = await import('./embed.ts');
|
|
const { runBacklinksCore } = await import('./backlinks.ts');
|
|
|
|
const repoPath = typeof job.data.repoPath === 'string'
|
|
? job.data.repoPath
|
|
: (await engine.getConfig('sync.repo_path')) ?? '.';
|
|
|
|
const steps: Record<string, unknown> = {};
|
|
const failed: string[] = [];
|
|
|
|
// Bug 8 — Between phases, yield to the event loop. The worker's lock
|
|
// renewal runs on a timer (src/core/minions/worker.ts); without a
|
|
// periodic yield, long CPU-bound phases starve the renewal callback
|
|
// and the job gets killed by the stalled-sweeper. A single
|
|
// `await new Promise(r => setImmediate(r))` gives the timer a chance
|
|
// to fire. The per-phase body is async+await already, so each phase
|
|
// internally yields on its own I/O boundaries — this is a belt for
|
|
// the gap between phases.
|
|
//
|
|
// Follow-up (deferred to v0.15): thread ctx.signal / ctx.shutdownSignal
|
|
// through each core fn so mid-phase cancellation works on huge brains.
|
|
const yieldToLoop = () => new Promise<void>(r => setImmediate(r));
|
|
|
|
try { steps.sync = await performSync(engine, { repoPath, noEmbed: true }); }
|
|
catch (e) { steps.sync = { error: e instanceof Error ? e.message : String(e) }; failed.push('sync'); }
|
|
await yieldToLoop();
|
|
|
|
try { steps.extract = await runExtractCore(engine, { mode: 'all', dir: repoPath }); }
|
|
catch (e) { steps.extract = { error: e instanceof Error ? e.message : String(e) }; failed.push('extract'); }
|
|
await yieldToLoop();
|
|
|
|
try { await runEmbedCore(engine, { stale: true }); steps.embed = { embedded: true }; }
|
|
catch (e) { steps.embed = { error: e instanceof Error ? e.message : String(e) }; failed.push('embed'); }
|
|
await yieldToLoop();
|
|
|
|
try { steps.backlinks = await runBacklinksCore({ action: 'fix', dir: repoPath }); }
|
|
catch (e) { steps.backlinks = { error: e instanceof Error ? e.message : String(e) }; failed.push('backlinks'); }
|
|
|
|
if (failed.length > 0) {
|
|
return { partial: true, failed_steps: failed, steps };
|
|
}
|
|
return { partial: false, steps };
|
|
});
|
|
|
|
// Shell handler: registered ONLY when GBRAIN_ALLOW_SHELL_JOBS=1 is set on the
|
|
// worker process. Default-closed; opt-in per-host. Without the flag, shell
|
|
// jobs submitted via CLI insert rows but no worker claims them (they sit in
|
|
// 'waiting' — the CLI prints a starvation warning for that case).
|
|
if (process.env.GBRAIN_ALLOW_SHELL_JOBS === '1') {
|
|
const { shellHandler } = await import('../core/minions/handlers/shell.ts');
|
|
worker.register('shell', shellHandler);
|
|
process.stderr.write('[minion worker] shell handler enabled (GBRAIN_ALLOW_SHELL_JOBS=1)\n');
|
|
} else {
|
|
process.stderr.write('[minion worker] shell handler disabled (set GBRAIN_ALLOW_SHELL_JOBS=1 to enable)\n');
|
|
}
|
|
}
|