mirror of
https://github.com/garrytan/gbrain.git
synced 2026-07-30 03:12:32 +00:00
docs: update project documentation for Minions job queue
CLAUDE.md: added Minions files to key files, updated operation count (36), BrainEngine method count (38), test file count (45), added jobs CLI commands. CHANGELOG.md: added Minions entry to v0.10.0 (background jobs, retry, stall detection, worker daemon). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 4.6
parent
b7726bf1f1
commit
34529ac60f
@@ -6,6 +6,8 @@ All notable changes to GBrain will be documented in this file.
|
||||
|
||||
### Added
|
||||
|
||||
- **Background jobs that don't die.** Minions is a BullMQ-inspired job queue built directly into GBrain. No Redis. No external dependencies. Submit `gbrain jobs submit embed --follow` and it runs with automatic retry, exponential backoff, and stall detection. Kill the process mid-job? Stall detection catches it and requeues. Run `gbrain jobs work` to start a persistent worker daemon that processes jobs from the queue. Jobs are first-class: submit, list, cancel, retry, prune, stats, all from the CLI or MCP. Your agent can now run long operations (14K+ page embeds, bulk enrichment) as durable background jobs instead of fragile inline commands.
|
||||
|
||||
- **Your agent now has 24 skills, not 8.** 16 new brain skills generalized from a production deployment with 14,700+ pages. Signal detection, brain-first lookup, content ingestion (articles, video, meetings), entity enrichment, task management, cron scheduling, reports, and cross-modal review. All shipped as fat markdown files your agent reads on demand.
|
||||
|
||||
- **Signal detector fires on every message.** A cheap sub-agent spawns in parallel to capture original thinking and entity mentions. Ideas get preserved with exact phrasing. Entities get brain pages. The brain compounds on autopilot.
|
||||
|
||||
@@ -9,7 +9,7 @@ cron scheduling, reports, identity, and access control.
|
||||
|
||||
## Architecture
|
||||
|
||||
Contract-first: `src/core/operations.ts` defines ~30 shared operations. CLI and MCP
|
||||
Contract-first: `src/core/operations.ts` defines ~36 shared operations. CLI and MCP
|
||||
server are both generated from this single source. Engine factory (`src/core/engine-factory.ts`)
|
||||
dynamically imports the configured engine (`'pglite'` or `'postgres'`). Skills are fat
|
||||
markdown files (tool-agnostic, work with both CLI and plugin contexts).
|
||||
@@ -19,7 +19,7 @@ markdown files (tool-agnostic, work with both CLI and plugin contexts).
|
||||
- `src/core/operations.ts` — Contract-first operation definitions (the foundation)
|
||||
- `src/core/engine.ts` — Pluggable engine interface (BrainEngine)
|
||||
- `src/core/engine-factory.ts` — Engine factory with dynamic imports (`'pglite'` | `'postgres'`)
|
||||
- `src/core/pglite-engine.ts` — PGLite (embedded Postgres 17.5 via WASM) implementation, all 37 BrainEngine methods
|
||||
- `src/core/pglite-engine.ts` — PGLite (embedded Postgres 17.5 via WASM) implementation, all 38 BrainEngine methods
|
||||
- `src/core/pglite-schema.ts` — PGLite-specific DDL (pgvector, pg_trgm, triggers)
|
||||
- `src/core/postgres-engine.ts` — Postgres + pgvector implementation (Supabase / self-hosted)
|
||||
- `src/core/utils.ts` — Shared SQL utilities extracted from postgres-engine.ts
|
||||
@@ -42,6 +42,10 @@ markdown files (tool-agnostic, work with both CLI and plugin contexts).
|
||||
- `src/core/transcription.ts` — Audio transcription: Groq Whisper (default), OpenAI fallback, ffmpeg segmentation for >25MB
|
||||
- `src/core/enrichment-service.ts` — Global enrichment service: entity slug generation, tier auto-escalation, batch throttling
|
||||
- `src/core/data-research.ts` — Recipe validation, field extraction (MRR/ARR regex), dedup, tracker parsing, HTML stripping
|
||||
- `src/core/minions/` — Minions job queue: BullMQ-inspired, Postgres-native (queue, worker, backoff, types)
|
||||
- `src/core/minions/queue.ts` — MinionQueue class (submit, claim, complete, fail, stall detection, parent-child)
|
||||
- `src/core/minions/worker.ts` — MinionWorker class (handler registry, lock renewal, graceful shutdown)
|
||||
- `src/commands/jobs.ts` — `gbrain jobs` CLI subcommands + `gbrain jobs work` daemon
|
||||
- `src/mcp/server.ts` — MCP stdio server (generated from operations)
|
||||
- `src/commands/auth.ts` — Standalone token management (create/list/revoke/test)
|
||||
- `src/commands/upgrade.ts` — Self-update CLI with post-upgrade feature discovery
|
||||
@@ -100,9 +104,18 @@ Key commands added in v0.7:
|
||||
- `gbrain init` — defaults to PGLite (no Supabase needed), scans repo size, suggests Supabase for 1000+ files
|
||||
- `gbrain migrate --to supabase` / `gbrain migrate --to pglite` — bidirectional engine migration
|
||||
|
||||
Key commands added for Minions (job queue):
|
||||
- `gbrain jobs submit <name> [--params JSON] [--follow] [--dry-run]` — submit a background job
|
||||
- `gbrain jobs list [--status S] [--queue Q]` — list jobs with filters
|
||||
- `gbrain jobs get <id>` — job details with attempt history
|
||||
- `gbrain jobs cancel/retry/delete <id>` — manage job lifecycle
|
||||
- `gbrain jobs prune [--older-than 30d]` — clean old completed/dead jobs
|
||||
- `gbrain jobs stats` — job health dashboard
|
||||
- `gbrain jobs work [--queue Q] [--concurrency N]` — start worker daemon (Postgres only)
|
||||
|
||||
## Testing
|
||||
|
||||
`bun test` runs all tests (34 unit test files + 5 E2E test files). Unit tests run
|
||||
`bun test` runs all tests (45 unit test files + 5 E2E test files). Unit tests run
|
||||
without a database. E2E tests skip gracefully when `DATABASE_URL` is not set.
|
||||
|
||||
Unit tests: `test/markdown.test.ts` (frontmatter parsing), `test/chunkers/recursive.test.ts`
|
||||
@@ -133,7 +146,8 @@ parity), `test/cli.test.ts` (CLI structure), `test/config.test.ts` (config redac
|
||||
`test/fail-improve.test.ts` (deterministic/LLM cascade, JSONL logging, test generation, rotation),
|
||||
`test/transcription.test.ts` (provider detection, format validation, API key errors),
|
||||
`test/enrichment-service.test.ts` (entity slugification, extraction, tier escalation),
|
||||
`test/data-research.test.ts` (recipe validation, MRR/ARR extraction, dedup, tracker parsing, HTML stripping).
|
||||
`test/data-research.test.ts` (recipe validation, MRR/ARR extraction, dedup, tracker parsing, HTML stripping),
|
||||
`test/minions.test.ts` (Minions job queue: CRUD, state machine, backoff, stall detection, dependencies, worker lifecycle, lock management, claim mechanics).
|
||||
|
||||
E2E tests (`test/e2e/`): Run against real Postgres+pgvector. Require `DATABASE_URL`.
|
||||
- `bun run test:e2e` runs Tier 1 (mechanical, all operations, no API keys)
|
||||
|
||||
Reference in New Issue
Block a user