mirror of
https://github.com/garrytan/gbrain.git
synced 2026-07-28 06:23:01 +00:00
bench: production Wintermute benchmark — Minions 753ms vs sub-agent timeout
Real deployment: 45K-page brain on Render+Supabase. Task: pull 99 tweets, write brain page, commit, sync. Minions: 753ms, $0. Sub-agent: gateway timeout (>10s, couldn't even spawn under production load). Also: 19,240 tweets backfilled across 36 months in 15 min at $0. Sub-agents would cost $1.08 and fail 40% of spawns.
This commit is contained in:
@@ -0,0 +1,125 @@
|
||||
# Production Benchmark: Minions vs OpenClaw Sub-agents (Real Deployment)
|
||||
|
||||
**Date:** 2026-04-18
|
||||
**Environment:** Wintermute on Render (ephemeral container, Supabase Postgres)
|
||||
**GBrain:** v0.11.0 (minions-jobs branch)
|
||||
**OpenClaw:** 2026.4.10
|
||||
**Brain:** 45,798 pages, 98K chunks, 25K links, 79K timeline entries
|
||||
**Task:** Pull and ingest one month of @garrytan tweets from the X Enterprise API
|
||||
|
||||
## Context
|
||||
|
||||
This is a **production benchmark**, not a lab test. The existing lab benchmark
|
||||
(docs/benchmarks/2026-04-18-minions-vs-openclaw-subagents.md) uses trivial
|
||||
prompts on localhost Postgres. This benchmark uses a real 45K-page brain on
|
||||
Supabase, pulling real tweets from the X Enterprise API ($50K/mo firehose),
|
||||
and writing real brain pages.
|
||||
|
||||
## The Task
|
||||
|
||||
Pull @garrytan tweets for one month (May 2020), parse them into a structured
|
||||
brain page with frontmatter, engagement metrics, and links, commit to the
|
||||
brain repo, and submit a sync job to gbrain.
|
||||
|
||||
## Method 1: Minions (deterministic pipeline)
|
||||
|
||||
```bash
|
||||
# 1. Pull tweets from X API
|
||||
curl -s -H "Authorization: Bearer $X_BEARER_TOKEN" \
|
||||
"https://api.x.com/2/tweets/search/all?query=from:garrytan&max_results=100&start_time=2020-05-01T00:00:00Z&end_time=2020-06-01T00:00:00Z&tweet.fields=created_at,public_metrics" \
|
||||
> /tmp/bench-tweets.json
|
||||
|
||||
# 2. Parse + write brain page (python)
|
||||
python3 parse_and_write.py
|
||||
|
||||
# 3. Git commit
|
||||
cd /data/brain && git add media/x/garrytan/2020-05.md && git commit -m "x-archive: 2020-05"
|
||||
|
||||
# 4. Submit sync to Minions
|
||||
gbrain jobs submit sync --params '{"repo":"/data/brain","noPull":true}'
|
||||
```
|
||||
|
||||
**Result: 753ms total.** 99 tweets pulled, page written, committed, sync job queued.
|
||||
|
||||
Breakdown:
|
||||
- X API call: ~300ms
|
||||
- Python parse + write: ~50ms
|
||||
- Git commit: ~100ms
|
||||
- gbrain jobs submit: ~300ms
|
||||
|
||||
Cost: $0.00 (no LLM tokens)
|
||||
|
||||
## Method 2: OpenClaw Sub-agent (sessions_spawn)
|
||||
|
||||
```javascript
|
||||
sessions_spawn({
|
||||
task: "Pull @garrytan tweets for June 2020 and save as a brain page...",
|
||||
model: "anthropic/claude-sonnet-4-20250514",
|
||||
mode: "run",
|
||||
runTimeoutSeconds: 120
|
||||
})
|
||||
```
|
||||
|
||||
**Result: GATEWAY TIMEOUT (>10,000ms).** The sub-agent could not even spawn
|
||||
within the 10-second gateway timeout. On a production Render container running
|
||||
a 45K-page brain with 19 active cron jobs, the gateway is under enough load
|
||||
that sub-agent spawning is unreliable.
|
||||
|
||||
When sub-agents DO successfully spawn (off-peak), the expected path is:
|
||||
1. Gateway receives spawn request (~500ms)
|
||||
2. Create session, load context (~2-3s) — AGENTS.md, SOUL.md, skills, memory
|
||||
3. Model reads task, plans approach (~2-3s)
|
||||
4. Model calls `exec` tool for curl (~1s)
|
||||
5. Model calls `exec` tool for python (~1s)
|
||||
6. Model calls `exec` tool for git (~1s)
|
||||
7. Model reports result (~1s)
|
||||
**Estimated: 10-15s + ~$0.03 in tokens per invocation**
|
||||
|
||||
## Comparison
|
||||
|
||||
| Metric | Minions | Sub-agent |
|
||||
|--------|---------|-----------|
|
||||
| **Wall time** | **753ms** | **>10,000ms** (gateway timeout) |
|
||||
| **Token cost** | $0.00 | ~$0.03 per run |
|
||||
| **Success rate** | 100% | 0% (timeout on first attempt) |
|
||||
| **Survives restart** | ✅ (Postgres) | ❌ (dies with process) |
|
||||
| **Progress tracking** | ✅ `gbrain jobs get <id>` | ❌ poll sessions_list |
|
||||
| **Auto-retry** | ✅ 3 attempts, exponential backoff | ❌ manual re-spawn |
|
||||
| **Concurrency** | ✅ FOR UPDATE SKIP LOCKED | ❌ hope-based maxConcurrent |
|
||||
| **Steerable** | ✅ inbox messages | ❌ fire and forget |
|
||||
| **Results persisted** | ✅ job record | ❌ lost on compaction |
|
||||
| **Memory** | ~2MB per in-flight job | ~80MB per spawned session |
|
||||
|
||||
## The Scaling Story
|
||||
|
||||
We pulled 19,240 tweets across 36 months (2021-2023) using the Minions
|
||||
approach in a single bash loop. Total time: ~15 minutes. Cost: $0.00 in
|
||||
LLM tokens.
|
||||
|
||||
The same task via sub-agents would require 36 spawns × ~$0.03 = ~$1.08
|
||||
in tokens, take 36 × 15s = 9 minutes best-case, and fail on ~40% of
|
||||
spawns under load (per the fan-out benchmark).
|
||||
|
||||
At scale (100+ months of backfill, or 1000+ batch enrichment jobs),
|
||||
Minions is the only viable path. Sub-agents hit the gateway timeout wall,
|
||||
burn tokens on deterministic work, and provide no durability.
|
||||
|
||||
## When Sub-agents Still Win
|
||||
|
||||
Sub-agents are correct for **judgment work**:
|
||||
- Email triage (LLM decides priority, drafts reply)
|
||||
- Social radar (LLM assesses severity, decides to alert)
|
||||
- Meeting prep (LLM synthesizes brain pages into briefing)
|
||||
- Cold email research (LLM decides notability)
|
||||
|
||||
These tasks require an LLM to make decisions. Minions can't do that —
|
||||
its handlers are code, not models. The routing rule:
|
||||
|
||||
> **Deterministic** (same input → same steps → same output) → **Minions**
|
||||
> **Judgment** (input requires assessment/decision) → **Sub-agents**
|
||||
|
||||
## One-Line Summary
|
||||
|
||||
Minions completed a production tweet ingestion in 753ms for $0.
|
||||
Sub-agents couldn't even spawn. For deterministic brain-write work,
|
||||
Minions is not incrementally better — it's categorically different.
|
||||
Reference in New Issue
Block a user