- _loop_local: two-stage compaction at 22k tokens (tiktoken cl100k).
Stage 1 elides old tool messages to [tool output elided: N chars, exit=X],
preserving tool_call_id. Stage 2 folds turn pairs into a synthesized
system note. Last 3 turns + system + initial user always intact.
Emergency-compact retry on a 32k BadRequest from vLLM.
- _loop_cloud_openai: assistant content is "" not None on tool-only turns
(OpenAI 400's on null). Bash output decoded with errors="replace" and
stubbed to [binary output: N bytes, exit=X] on >5% replacement or NUL.
- _openai_retry: local-vLLM branch now retries APIConnectionError /
APITimeoutError / InternalServerError 3x with 1/2/4s backoff.
- Tests: 18 cases covering compaction shape, token budget, tool_call_id
preservation, null-content guard, binary decode, local retry.
Addresses ctx-overflow in Gemma-31B SWE n=100 cells (61 + 19 errored
rows) and the OpenAI/Gemini adapter errors across ~19 cells in the
n=100 hybrid sweep.
Adds an opt-in paper-faithful worker pool to the toolorchestra RL path,
gated on `method_cfg.pool = "paper"`. Default cells are unaffected.
- New worker types: `tavily-search` (wraps WebSearchTool), `openrouter`
(code/math/generalist specialists), `modal-python` (one-shot Modal
Sandbox Python exec).
- `_paper_expert_for` maps orchestrator slots to Tavily / GPT-5 /
GPT-5-mini / Llama-3.3-70B / Qwen-2.5-Coder-32B; `enhance_reasoning`
output is exec'd in Modal.
- Fixes a pre-existing RL-mode bug where vLLM's tool parser swallowed the
orchestrator's tool call into the SDK field, leaving content empty and
forcing the answer-1 fallback every turn — paper mode now surfaces SDK
tool_calls via `_call_orchestrator_with_tool_calls`.
- New smoke cell `toolorchestra-papermatch-orch8b-opus47-gaia-n5`.
Skipped vs. paper: FAISS RAG, Qwen2.5-Math-{72B,7B} (not on OpenRouter).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The toolorchestra cell that produced 0.560/$31.58 on GAIA (orch8b-opus47)
only ran the RL orchestrator-8B path on GAIA-shaped questions; SWE-bench
fell back to a one-shot cloud call. This unblocks the toolorchestra x SWE
column in the hybrid ablation: now the same n=100 orchestrator-8b +
opus-4-7 cell can run on swebench-verified, gated by
method_cfg.swe_use_agent_loop = true.
_run_rl now mirrors the prompted path's SWE handling — clones a shared
workdir per task, routes enhance_reasoning / answer worker dispatches
through run_swe_agent_loop on that workdir, falls back to the frontier
worker through the SWE loop too if the orchestrator never picks
`answer`, and appends the working-tree diff to the final answer so the
SWE-bench Modal scorer can extract the patch. bash turns count as
tool_calls per row (matches the 7f432453 contract). All cleanup runs
through a try/finally guard so /tmp doesn't leak cloned repos.
Smoke cell `toolorchestra-orch8b-opus47-swe-smoke2` (n=2) runs 2/2
without errors, score 1.0/0.0 (one diff fix lands, one doesn't —
expected for n=2). tool_calls populated, swe_mode=True in traces,
workdirs cleaned up. The n=100 cell now carries the
swe_use_agent_loop=true + matching swe_* knobs.
Unblocks the minions × Google axis of the n=100 hybrid ablation. The
vendored Minions library already ships a `GeminiClient` and the
`Minion.execute_task` path already type-dispatches on it (passing a
Pydantic `response_schema` to coerce the supervisor JSON into the same
{decision, message, answer} shape Opus / GPT-5 emit). All we were
missing was the `cloud_endpoint == "gemini"` branch in MinionsAgent.
Also adds an idempotent patch to the vendored `GeminiClient.schat`:
upstream subtracts `candidates_token_count` from `total_token_count`
without `None`-checking, which `TypeError`s on Gemini 2.5 Pro responses
that burn the whole budget on thinking. Patch falls back to
`prompt_token_count` directly and defaults missing fields to 0, so
`tokens_cloud` / `cost_total_usd` / `n_cloud_calls` still populate in
`summary.json` for the hybrid runner.
Registry: adds `minions-qwen27b-gemini25pro-{gaia,swe}-n100` cells.
Concurrent hybrid SWE cells were colliding on the swebench-harness
shared cache because run_id was keyed only on instance_id
(`oj-<instance_id>`). The second cell to score the same task hit
"1 instances already run, skipping..." and silently scored 0 with
`reason: no_report` (or read the first cell's verdict on a write race).
Previously papered over by manual cache wipes + rescores; this is the
root-cause fix.
run_id is now `oj-<cell>-<instance>` when a cell name is supplied,
falling back to the legacy `oj-<instance>` form for single-cell
callers. Same-cell resumes still hit the harness cache (run_id is
deterministic for a given cell + instance). Cell name is sanitized
to filesystem-safe `[A-Za-z0-9._-]+` since it lands in both the
`logs/run_evaluation/<run_id>/...` subtree and the
`<model>.<run_id>.json` summary filename.
Plumbed cell_name through `runner._process -> score -> _score_swebench
-> SWEBenchHarnessScorer -> _run_harness -> _build_run_id`. Regression
test under `tests/agents/hybrid/test_swebench_run_id_isolation.py`
pins: (a) different cells produce different run_ids, (b) same cell is
stable, (c) legacy `None` cell preserves old format, (d) hostile cell
names sanitize cleanly, (e) end-to-end wiring from scorer constructor
through to the run_id.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Follow-up to 7f432453 (tool_calls). Every row now also carries
n_cloud_calls and n_local_calls so the per-task LLM round-trip
budget is queryable alongside cost / latency / tool_calls.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
results-table.md caveat (3) flagged that tool_calls was missing from
results.jsonl for every paradigm and the legacy skillorchestra values
came from a defunct telemetry path. Wire the count back through the
canonical row-write path so future ablation cells populate it.
Definition per paradigm:
- SWE-bench: sum of bash turns from each run_swe_agent_loop subloop
(one tool call = one bash command the agent ran).
- GAIA: native web_search invocations from the cloud backbone. Zero
on one-shot GAIA paths and on paradigms whose GAIA path is pure
text passing (minions w/o prefetch).
runner._run_one now reads meta["tool_calls"] into the top-level row;
summary.json picks up tool_calls_total. Existing n=100 rows stay "—"
(no rerun).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Tests, ablation registries, rescore script, and m2 distillation configs that
were left untracked alongside the recent feature commits. Junk excluded
(minion_logs/, oj-debug.oj-debug.json — runtime artifacts).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
GPT-5 family and Gemini 2.5 Pro consume the output-token budget on hidden
chain-of-thought before emitting visible answer text. At max_tokens=4096
they silently truncate with empty answers (finish_reason=length / MAX_TOKENS)
on GAIA — surveyed n=100 cell logs show:
- cloud-only-gpt5-gaia: 26/100 empty answers
- cloud-only-gpt5mini-gaia: 6 empty (+41 short — partial truncations)
- cloud-only-gemini25pro-gaia: 18/100 empty answers
- cloud-only-opus47-gaia: 2/100 empty (Opus isn't a reasoning model in
this sense, kept at 4096)
Adds _prices.is_reasoning_model + default_max_output_tokens helpers and
threads them through baseline_cloud's three call sites. Cells that opt in
via method_cfg.cloud_max_tokens override the new default unchanged.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
mini_swe_agent.py adds recovery nudges so the agent doesn't exit silently
when the cloud model produces an empty turn:
- _loop_cloud_openai: finish_reason='length' with empty content/tool_calls
→ re-prompt instead of breaking. Hit gpt-5-mini on SWE n=100 (caused 0-score
exits with empty final_summary).
- _loop_cloud_gemini: MALFORMED_FUNCTION_CALL / MAX_TOKENS with empty parts
→ same recovery. Hit gemini-2.5-flash on SWE n=100 (24/100 tasks).
Also: proper Gemini Schema-shaped bash tool params, gpt-5-family handling,
missing is_gpt5_family import. 5 regression tests cover both recovery paths
and natural-stop sanity guards.
scripts/ablation/run_sweep.py parse_cell anchors on the n100 suffix and
walks left, so advisors-*-gaia-n100 cells report GAIA as bench instead of
'qwen27b'. Old logic only handled cloud-only/skillorchestra-qwen prefixes
explicitly.
Companion working-tree changes shipped here: Gemini price entries,
baseline_cloud module registration, swebench dataset variant note,
swebench_harness Modal cgroup-v2 patch (file write swallows
FileNotFoundError on cgroup paths so Modal v2 sandboxes don't die).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds opt-in `method_cfg.web_search = { enabled, max_uses }` schema and
plumbs the native Anthropic server-side `web_search_20250305` tool
through every hybrid paradigm's GAIA codepath. Default OFF preserves
back-compat with all currently running n=100 cells; only the next
rendered sweep opts in.
Engine layer: `_base.py` gets `build_web_search_tool`, `web_search_cfg`,
and `_call_anthropic_agent` (multi-turn loop with `gaia_max_turns`
default 8). Anthropic cost (`$0.01 / search`) added to `tokens_cloud`.
Per-paradigm wiring (cloud-side Anthropic calls only):
* baseline_cloud: GAIA one-shot upgraded to agent loop with tools
* advisors: executor1 + executor2 passes declare tools
* minions: existing prefetch retained as legacy default; new schema
overrides
* skillorchestra: cloud-route specialist gets tools
* conductor: anthropic worker steps get tools when enabled
* toolorchestra: already used native web_search worker - unchanged,
now surfaces web_search_uses in meta
* archon: thread-local web_search tool injection on Anthropic
ranker/fuser generators
Trace + aggregation:
* Per-row `web_search_uses: int` in results.jsonl
* Summary `web_search_uses_total` in summary.json
* make_report.py adds `web_searches_mean` column to the HTML table
Tests: tests/agents/hybrid/test_web_search_wiring.py - confirms the
tool is declared when enabled, NOT declared when omitted (default),
and gracefully skipped on non-Anthropic endpoints (no fake local
web_search).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Both paradigms previously hardcoded a heterogeneous worker pool (Opus +
gpt-5-mini + Anthropic web-search) and only the orchestrator/planner role
swapped when `cell.cloud.model` changed. Adds an optional
`method_cfg.worker_pool` that, when set, strictly replaces the default
pool, so cells can ablate pool composition without code changes.
Behavior:
- Absent override = legacy default pool (unchanged).
- Each entry: `id` (int), `name` (str), `endpoint`/`type` (whitelisted),
`model` (validated against `_prices.PRICES` for cloud workers;
`anthropic-web-search` may omit it).
- `model = "$local"` / `"<local>"` substitutes `config.local.model`;
`"$cloud"` / `"<cloud>"` substitutes the cell's cloud model.
- Validation runs at agent `__init__` — bad pools fail fast, not mid-task.
- Strict replace (not merge) so the cell.toml is the single source of
truth for the worker pool.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The ternary `"anthropic" if backbone == "cloud" else "anthropic"` returned
the same value in both branches. Behaviorally a no-op today (when
backbone="local" the value is unused by run_swe_agent_loop's local path),
but the code was misleading. Replace with per-branch assignment so it's
obvious what each path sets.
The hard fallback path (no parsed final_answer after max_turns) was
calling workers[-1], which in the default pool is gpt-5-mini, not the
strongest worker as the comment claimed. Now picks the worker with the
highest output-token price excluding search workers, so Opus wins by
default and the fallback actually exercises the frontier tier.
Bug A: DEFAULT_AGENT_COMPETENCE / cost / router prompt hardcoded
`cloud-opus-4-7` with Opus's $0.30/task, so every non-Opus cell
(haiku45, gpt-5, gemini-*) routed under Opus's price + capability — the
cheaper cloud's lower cost was invisible to the router.
Bug B: priors had no overlap (cloud floor 0.70 > local ceiling 0.65) and
lambda_cost=0.5 with $0.30 per-task cost gave a 0.15 penalty that was
swamped by the competence gap. Net effect: 95-98% of tasks routed to
cloud across all n=100 cells.
This patch:
- Plumbs self._cloud_model into the agent key: cloud-<configured-model>.
chosen_agent trace field reflects the real cloud (preserves the schema
field name; only the *value* changes).
- Adds MODEL_COST_USD_PER_TASK calibrated against cloud-only-*-gaia-n100
empirical per-task costs (opus47 $0.014, haiku45 $0.002, gpt-5 $0.025,
gpt-5-mini $0.003, gemini-2.5-pro $0.002, gemini-2.5-flash $0.0006).
- Adds per-cloud competence priors so the router sees haiku/flash with
realistically lower competence (floors ~0.50-0.55) than Opus/Pro
(~0.70+). Local ceiling raised to 0.75 (format_compliance) so local
can win pure-arithmetic / format-heavy mixes.
- Exposes lambda_cost in method_cfg (default 2.0, up from 0.5).
- Allows method_cfg.agent_competence / agent_cost_usd partial overrides
so future cells can plug in calibrated priors without code changes.
Test updated: _ROUTER_JSON emits the new cell-specific key, and a new
test_skillorchestra_chosen_agent_uses_cloud_model_name asserts the
contract end-to-end for a haiku45 cell.
- `ruff check --fix` handled 31 auto-fixable issues across the hybrid
module: unused imports, import sorting, removed stale unused
shutil/tempfile imports in advisors.py, etc.
- Add `src/openjarvis/agents/hybrid/*.py` to the project's E501
per-file-ignores list. Same relaxation already in place for
`evals/datasets/` and `evals/scorers/` — hybrid is research code
with long prompt strings and per-paradigm config dicts that don't
benefit from mid-sentence line breaks.
- archon.py: add `List` to the typing imports (was used in an
annotation; PEP 563 protected it from runtime NameError, but ruff
was right to flag it as undefined).
- runner.py: split `f.seek(0); f.truncate(); f.write(...); f.flush()`
into four statements, and rename ambiguous `l` -> `line` in the
results-loading comprehension.
Total: 79 ruff errors -> 0 on `src/ tests/` (the scope CI checks).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
cloud.py:
- Skip server_tool_use blocks from tool_calls (they're already in
content_blocks; agents would otherwise try to execute Anthropic
server-side tools like web_search as if they were local).
- Drop the synthetic tool_use_id fallback (f"{btype}_{len(...)}").
Anthropic always returns ids for tool_use blocks; the fallback
only masked missing ids with fakes that fail Anthropic's
tool_use_id matching on the next assistant turn. Now we log a
warning and skip the block.
- Streaming/non-streaming parity: emit content_blocks and
tool_results at end-of-stream in _stream_full_anthropic so
streaming callers see the same rich blocks as generate().
StreamChunk gets two new optional fields.
swebench_harness.py:
- Log a warning when set_cpu_quota is missing instead of silently
no-op'ing — previously a swebench API change would have produced
a silent 0-score sweep with no diagnostic.
- Delete the prior report JSON before invoking the subprocess so a
stale file from a crashed run can't be re-read as fresh.
archon.py:
- Token tally is now thread-local (threading.local). The runner
reuses one ArchonAgent across a ThreadPoolExecutor; the previous
module-global dict let concurrent tasks reset and read each
other's counters, corrupting per-task cost_usd / tokens_*.
toolorchestra.py:
- Wrap the orchestration loop in try/finally so shared_workdir is
always cleaned up. Previously rmtree only ran on the success
path; any exception in the turn loop, worker call, or diff
extraction leaked the cloned repo (hundreds of MB at n=500).
Matches the pattern conductor.py and mini_swe_agent.py use.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Replace the n=30 reproduction snippet with the upstream harness's headline
numbers (GAIA n=165, SWE-Verified n=500) and a verdict column so it's
obvious at a glance which paradigms are worth keeping (minions,
skillorchestra), which split by bench (conductor, advisors), and which
are dominated (archon).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Minions (swe mode): skip the upstream Minions library — it doesn't fit
SWE-bench (its premise is 'small reads long docs, big summarizes').
Replace with: cloud supervisor writes a high-level fix plan (no tools),
local Qwen runs mini-SWE-agent with that plan as additional context.
Mirrors Minions's 'cloud supervises, local does the work' shape on a
benchmark where the upstream library is wrong tool.
Archon (swe mode): K independent local mini-SWE-agent runs producing K
diverse candidate patches; cloud ranker picks the best by summary +
diff. Skips the 'fuser' layer (can't fuse two diffs cleanly).
Add registry/swe_agent_loop.toml with one cell per paradigm exercising
the new mode (n=3 smokes; same model pairing as the headline cells).
All 6 paradigms + standalone mini_swe_agent = 9 SWE-agent cells total.
Smoke imports clean across all paradigms. Cost warning: each cell now
runs ~30 LLM turns/task with bash; budget -3/task with Opus.
Same 'swe_use_agent_loop' opt-in flag as Conductor/SkillOrchestra.
Advisors (swe mode):
- Initial executor pass = full mini-SWE-agent run on a fresh workdir.
- Advisor (local) critiques the produced patch (sees summary + diff).
- Final executor pass = SECOND mini-SWE-agent run on ANOTHER fresh
workdir, with the initial summary + advisor feedback folded into the
prompt. Don't reuse the initial workdir — that would smuggle the
bad-fix into the new attempt; the final pass should incorporate only
the parts of the advisor feedback that hold up.
ToolOrchestra (swe mode):
- One SHARED workdir created at start of run, cloned from the SWE-bench
repo at base_commit.
- Each call_worker action whose worker is a solver (vllm/anthropic)
runs as a mini-SWE-agent subloop on the shared workdir. Web-search
workers stay one-shot (search isn't an agent loop). OpenAI workers
fall back to one-shot too (loop tool format isn't wired for OpenAI).
- After loop ends, the framed final answer wraps the working-tree
git diff so the swebench scorer picks it up.
Add 'swe_use_agent_loop' cfg flag to both paradigms. When on AND the
task is SWE-bench-shaped (has repo + base_commit + problem_statement),
the worker call is replaced by a run_swe_agent_loop subloop.
Conductor: one shared workdir across all plan steps so step N+1 builds
on step N's edits. Each step's subtask becomes the initial_prompt; the
worker's model + endpoint determines the agent-loop backbone
(vllm → local, anthropic → cloud, openai workers fall back to one-shot
since the agent-loop tool format isn't wired for OpenAI today). After
the last step, the framed final answer is the working-tree git diff.
SkillOrchestra: same flag, no workdir sharing (just one worker invoked).
Whichever of local-qwen / cloud-opus the router picks runs as
run_swe_agent_loop instead of a single completion.
Cells stay one-shot by default — set method_cfg.swe_use_agent_loop=true
to opt in. Smoke imports clean; behavior on non-SWE benches unchanged.
Refactor MiniSWEAgent so the workdir setup + multi-turn bash loop +
diff extraction live in a top-level function. Paradigms can now call
run_swe_agent_loop(task, backbone, model, ...) to replace their
one-shot worker call with a real agent loop on SWE-bench tasks.
MiniSWEAgent itself becomes a thin LocalCloudAgent wrapper around the
function (~30 lines). Behavior unchanged for the standalone agent;
trace event kinds still use 'mini_swe_*' prefix (paradigms pass
trace_prefix= to namespace their subloops in the log).
Adds `MiniSWEAgent` — the environment-interaction loop the paradigms
were missing for SWE-bench. Every existing paradigm (Minions, Conductor,
Archon, Advisors, etc.) is an *orchestration* loop but the worker
ultimately runs one-shot "predict the patch from the issue text." No
shell, no repo browsing, no test execution. That's why every paradigm
caps near 0.30 on SWE-bench-Verified — the model literally cannot look
at the code.
mini-SWE-agent fills the gap: one bash tool, a per-task git clone, and
a multi-turn loop that lets the model grep / cat / sed / pytest its way
to a real fix. Upstream (https://github.com/swe-agent/mini-swe-agent)
gets ~0.77 on SWE-bench-Verified with a frontier backbone.
This is the vendored option (a) from the plan — no Docker, no upstream
dependency, ~330 lines total:
- `MiniSWEAgent._run_paradigm`: clone the SWE-bench repo into a tempdir,
run the loop, extract the final diff via `git diff`, wrap it in a
```diff fence so the existing swebench scorer reads it.
- Two backbone modes (`cfg["backbone"]`): `cloud` (Anthropic, multi-turn
with `tools=[bash]`), `local` (vLLM OpenAI-compatible with the same).
Both routes integrate with the LocalCloudAgent trace buffer — every
bash invocation lands as a `mini_swe_bash` event with full command,
stdout, stderr, exit code, latency.
- Each `bash` command runs as a fresh subshell in the workdir with a
configurable timeout and 10K-char output cap.
- No `submit` tool — the loop ends when the model produces a turn with
no tool_calls, or `max_turns` hits.
Safety: model can `rm -rf` its own workdir. Workdir is disposable.
Network is available (pip etc.). Don't run on a host with secrets in
$CWD. Docker sandboxing is a follow-up.
Registry: `registry/mini_swe_agent.toml` ships three cells (n=3 cloud,
n=30 cloud, n=3 local-backbone). Smoke (n=1, max_turns=10): the agent
explored the django repo, edited `django/contrib/auth/forms.py`, ran
the relevant tests, produced an 829-char patch. ~$0.25 / 75s per task
with Opus 4.7 at max_turns=10. Cost+time will scale roughly linearly
with max_turns; production cells use max_turns=30-50.
Follow-up: wrap Minions/Conductor workers in this loop so the
orchestration paradigms get the same 0.30 → ~0.77 jump (task #13).
Same gap as the hybrid agents had: `_generate_anthropic` walked `resp.content`
and only matched `type == "tool_use"` for tool calls. Anthropic server-side
tools (web_search) emit `server_tool_use` blocks for the model's actual
query and `web_search_tool_result` blocks for the result payload — both
were silently dropped. So every OpenJarvis agent using Anthropic web_search
had its search queries and results invisible in traces.db.
Fix:
- Add `_serialize_anthropic_block(block)` — recursive JSON-safe converter
that handles text / tool_use / server_tool_use / web_search_tool_result
/ tool_result / thinking blocks, preserving nested citation content.
- `_generate_anthropic` now returns a full `content_blocks` list (every
block in order) plus extended `tool_calls` (tool_use + server_tool_use,
with a `server_side` flag) and a new `tool_results` field
(web_search_tool_result + tool_result).
- `instrumented_engine` propagates these into the INFERENCE_END event.
- `TraceCollector._on_inference_end` records them on the generate step.
Backward compat: existing callers that only read `result["tool_calls"]`
still see the same shape; new fields are additive.
Confirmed against a real Opus web_search call: serializer captures the
`server_tool_use` block with the actual query (`{"query": "when did
SpaceX first launch Starship"}`), the `web_search_tool_result` block
with 10 nested citation entries, and the synthesizing text block.
Previous trace events only stored the joined `text` blocks from Anthropic
responses and `message.content` from OpenAI/vLLM — every tool_use block,
server_tool_use block, web_search_tool_result, and OpenAI-style tool_calls
list was being thrown away.
Now each call event includes:
Anthropic events:
- `content_blocks`: full list of every block the model emitted, with all
fields preserved (text / tool_use / server_tool_use /
web_search_tool_result / thinking). Nested content (e.g. search-result
citations) recurses.
- `tool_calls`: filtered subset (tool_use + server_tool_use).
- `tool_results`: filtered subset (web_search_tool_result + tool_result).
- `tools_declared`, `tool_choice`, `output_config`, `stop_reason`.
OpenAI / vLLM events:
- `tool_calls`: serialized `message.tool_calls` (id, function name, args).
- `reasoning_content`: thinking-model intermediate reasoning if present.
- `tools_declared`, `tool_choice`, `finish_reason`.
Confirmed against a real Anthropic call with `tools=[web_search]`,
`tool_choice={"type":"any"}`: the resulting event contains the
`server_tool_use` block with the actual search query, the
`web_search_tool_result` with 10 nested citation blocks, and all
subsequent text blocks the model emitted to synthesize the answer.
Previously the runner created the `logs/` directory but never wrote
anything. Wire a thread-local trace buffer through the SDK helpers so
every cloud/local call is captured automatically, plus paradigm hooks
for the events that bypass the helpers (Minions's library protocol,
Archon's layer pipeline, conductor's plan + step dispatch, ToolOrchestra
action turns, SkillOrchestra routing decision).
Mechanics:
- `_TRACE_STATE` is a `threading.local()` — `run()` opens a fresh trace
per task and writes a digested JSON to `<log_dir>/<task_id>.json` in
the `finally` block (so we get a record even on hard failures).
- `_call_anthropic` / `_call_openai` / `_call_vllm` append full events
(system, user, response, token counts, latency, model, schema config)
to the active trace.
- `LocalCloudAgent.record_trace_event(...)` is the public hook for
paradigms with library-side execution.
- Nothing is truncated — full prompts, responses, and tool-call payloads
land in the JSON.
- Runner threads `log_dir` through `context.metadata["log_dir"]`.
Verified: re-running `advisors-gaia-qwen9b-opus-3` produces three
`logs/gaia-*.json` files, each ~13KB with all 3 turns (cloud executor →
local advisor → cloud final) and the full text of each prompt/response.
GAIADataset and SWEBenchDataset both build a formatted prompt (`rec.problem`)
and discard the raw input text. The hybrid paradigm runner needs the raw
text so it can apply the GAIA / SWE-bench answer-format instructions from
`agents.hybrid._prompts.format_prompt(task)` — otherwise paradigms see a
double-wrapped prompt and the GAIA "FINAL ANSWER: <x>" rule is missing.
Additive change — adds `metadata["question"]` for GAIA and
`metadata["problem_statement"]` for SWE-bench. No existing consumer
breaks; the runner now picks up the raw text correctly.
Verified end-to-end with `python -m openjarvis.agents.hybrid.runner
--cell advisors-gaia-qwen9b-opus-3`: 3/3 tasks, acc=0.667, $0.04 — matches
the hybrid harness's cell exactly.
Final piece — make the ported paradigms runnable end-to-end through the
same flow as the hybrid harness:
- `runner.py`: `python -m openjarvis.agents.hybrid.runner --cell <name>`.
Loads cells from packaged registry TOMLs, constructs the registered
agent via AgentRegistry, iterates tasks from OpenJarvis's existing
GAIADataset / SWEBenchDataset, scores via gaia exact-match or the new
Modal-backed SWE-bench harness scorer, writes
`results.jsonl`+`summary.json` in the same shape as the hybrid harness
so existing rescore / dashboard scripts work unmodified. Supports
resume (drops errored rows), per-cell flock, ThreadPoolExecutor
concurrency, and the same per-row heartbeat format.
- `registry/{advisors,archon,conductor,minions,skillorchestra,toolorchestra}.toml`:
35 cells ported verbatim from the hybrid harness — same models, same N,
same `method_cfg` — so OpenJarvis runs should reproduce the numbers in
`hybrid-local-cloud-compute/docs/results.md`. ToolOrchestra cells
rewritten from the original NotImplementedError stubs to point at the
prompted port (cloud-as-orchestrator, no Nemotron-Orchestrator-8B).
- `scripts/new_experiment.sh`: mirror of hybrid's scaffolder. Maps
shortnames (`qwen3.5-27b`, `claude-opus-4-7`, …) to full HF ids +
endpoints, picks method-appropriate `method_cfg` defaults, appends a
`[cells.<name>]` block to the right registry TOML.
- `README.md`: quickstart, paradigm taxonomy, reproducibility table
pointing at the hybrid results for headline numbers.
End-to-end smoke: `python -m openjarvis.agents.hybrid.runner --help` works,
all 6 agents register, all 35 cells load.
Adds `SWEBenchHarnessScorer` alongside the existing structural-only
scorer. Hands one prediction at a time to `python -m swebench.harness.
run_evaluation`, parses the report JSON it writes, returns the resolved
bool. Backend selectable via `SWEBENCH_BACKEND` env var
(modal default, docker fallback).
Carries forward the two upstream-swebench patches from the hybrid
harness, both idempotent and lazy:
1. **Modal cgroup-v2 fix** — `swebench/harness/modal_eval/run_evaluation_modal.py`
writes to `/sys/fs/cgroup/cpu/cpu.shares` (cgroup v1). Modal v2 sandboxes
are cgroup v2 — the path doesn't exist and every sandbox died on the
write. This was the root cause of "Modal harness failing 100%"; wrapping
`set_cpu_quota` in try/except fixes it.
2. **Rescore `*_ids` fix** — older swebench wrote `resolved_instances` as
a list; current swebench puts the count there and the actual ids in
`resolved_ids`. Reading `*_ids` first; falling back to the list-typed
legacy field if the user is on an older install.
Patch extraction supports the same three formats as the hybrid harness:
fenced `\`\`\`diff`, fenced raw (`\`\`\`\ndiff --git`), and unfenced
`diff --git`.
Inference-only port of NVlabs ToolOrchestra (arXiv:2511.21689). The
paper's RL-trained Nemotron-Orchestrator-8B is NOT in the loop — the
hybrid harness adapter is a documented stub for exactly that reason
(needs a separate vLLM srun for the 8B checkpoint, a FAISS wiki
retriever, a Tavily key, and an upstream refactor).
This port keeps the same scope discipline: a cloud model plays the
orchestrator role and dispatches to a mixed pool of workers (local
Qwen for cheap extraction, Anthropic web_search for unknowns, Opus
4.7 / gpt-5-mini for hard reasoning or final synthesis). Reactive
multi-turn loop emitting JSON action objects per turn; forces a
final answer at the budget cap; hard fallback to the strongest worker
on persistent parse failure.
Treat results as preliminary until a real Orchestrator-8B deployment
exists — included here so OpenJarvis has registry entries for all six
paradigms and so the distillation pipeline can slot ToolOrchestra in
alongside the rest.
Registered as `toolorchestra`. The hybrid harness `toolorchestra_adapter.py`
remains a NotImplementedError stub; this is the first runnable version.
Inference-time skill-aware router from arXiv:2602.19672. The paper's
4-phase explore/learn/select pipeline is out of scope (needs multi-model
serving + FRAMES wiki retriever + a multi-hour learning loop) — the
hybrid harness reproduced only the deployment-time `select_agent()`
path with a hand-curated skill taxonomy + per-agent competence priors
calibrated to Qwen3.5-27B-FP8 vs Opus 4.7. This port keeps that scope.
Per task: Opus reads the question, picks skill weights from a 7-skill
catalog (factual_recall, multi_step_reasoning, arithmetic, web_grounding,
long_text_extraction, format_compliance, code_or_logic), scores each
agent under `score = weighted_competence - 0.5 * avg_cost`, routes to
the winner. JSON schema enforced via Anthropic `output_config`; balanced-
brace fallback if the SDK ignores the schema. Soft-fail row on
unparseable JSON to match hybrid's `err=1` on n=30.
Hybrid harness result: `skillorchestra-gaia-qwen27b-opus-30` = 0.500 acc,
$0.02/task — 30× cheaper than baseline-cloud (0.567 / $0.66) at ~7pp
lower accuracy. Best cost-efficient GAIA paradigm.
Registered as `skillorchestra`. Ported from
`hybrid-local-cloud-compute/adapters/skillorchestra_adapter.py`.
Layered (generator → ranker → fuser) sampling: K local generators propose
candidates, cloud ranker scores them, cloud fuser synthesizes the final
answer. Paper: arXiv:2409.15254. Two presets: `ensemble_rank_fuse` (the
full layered pipeline) and `single_local` (debug shim).
Carries the same patch story as the hybrid adapter:
- Eager-import stubs for groq/google/litellm so Archon's `utils.py` loads
in the OpenJarvis venv without those deps.
- Custom `vllm_local` model_type wired into Archon's GENERATE_MAP via a
per-run closure (endpoint can vary per cell).
- OpenAI / Anthropic generators replaced with token-tallying wrappers so
we can charge cost — Archon ignores `usage` by default.
- Opus 4.7 temperature stripping at the SDK level.
Archon library is lazy-imported. The hybrid clone at
`hybrid-local-cloud-compute/external/Archon/src` is the default
location; override with `ARCHON_SRC` env var.
Hybrid harness results: `archon-gaia-qwen27b-opus-ensemble-K5-30` =
0.333 acc, $0.20/task; `archon-gaia-qwen27b-singlelocal-30` = 0.033 acc.
Underperforms baseline-cloud on GAIA — included for completeness, not as
a recommended paradigm.
Registered as `archon`. Ported from
`hybrid-local-cloud-compute/adapters/archon_adapter.py`.
Cloud supervisor decomposes + reads back; local worker(s) do bulk
reading. Two modes: `minion` (single worker, default) / `minions`
(parallel workers + aggregator).
Carries forward every compatibility patch from the hybrid adapter so the
n=500 numbers transfer:
- Strip `temperature` for Opus 4.7+ (rejected with 400).
- Inject server-side `output_config` JSON schema per supervisor turn
(first-turn `{reasoning, message}` vs conversation-turn anyOf
`{decision, message|answer}`), picked by sniffing the prompt for
`"decision": "provide_final_answer"`.
- Replace `minions._extract_json` with a JSON-first wrapper.
- Inject `timeout=600`/`max_retries=5` into `anthropic.Anthropic()` so
Minions's bare-client construction doesn't 60s-timeout under SWE-bench
concurrency=8.
- GAIA-only `web_search` prefetch so the worker has a real doc to read.
Soft-fail row on JSONDecodeError/BadRequestError/prompt-too-long — same
deterministic-failure handling that produced the n=165 GAIA `err=6` rows
in the hybrid harness without crashing the cell.
`minions` library import is lazy; install via
`uv pip install -e /matx/u/aspark/hybrid-local-cloud-compute/external/minions`.
Hybrid harness results being reproduced:
- `minions-swebenchverified-qwen27b-opus-500` = 0.274 acc, $0.09/task
(+3.8pp vs baseline-cloud at 10× cheaper — cleanest paradigm-validation
result in the harness).
- `minions-gaia-qwen27b-opus-165` = 0.576 acc, $0.67/task
(~tied with baseline-cloud at 1.6× cheaper).
Registered as `minions`. Ported from
`hybrid-local-cloud-compute/adapters/minions_adapter.py`.
Inference-only Stage-1 repro: zero-shot cloud planner substitutes for the
paper's RL-trained Qwen2.5-7B conductor. Emits 3 JSON lists
(model_id/subtasks/access_list), executor runs ≤5 steps over a worker pool.
Two-tier parse fallback: JSON → Python-literal regex → single call to the
strongest worker. Worker pool defaults to {local Qwen if vLLM up, Opus 4.7,
gpt-5-mini}; override via `cfg["workers"]`.
Hybrid harness result: `conductor-swebenchverified-opusplan-30` = 0.367 acc
($0.22/task) vs baseline-cloud 0.267 ($3.36/task) — +10pp at ~15× cheaper.
Registered as `conductor`. Ported from
`hybrid-local-cloud-compute/adapters/conductor_adapter.py`.
Three-step executor (cloud) → advisor (local) → executor (cloud) loop from
arXiv:2510.02453. Inference-only: the paper's RL-trained advisor would
land higher; this matches the hybrid harness's `advisors-gaia-qwen9b-opus-30`
cell at 0.533 acc / $0.02 per task.
Registered as `advisors`. Local model id auto-resolves against vLLM's
`/v1/models` so cell configs naming Qwen3.5-9B don't 404 when the server
is actually serving 27B-FP8.
Ported from `hybrid-local-cloud-compute/adapters/advisors_adapter.py`.
Add `openjarvis.agents.hybrid` — the home for the local+cloud paradigms
being ported from `/matx/u/aspark/hybrid-local-cloud-compute`. This commit
ships only the shared base (no paradigm yet):
- `_base.LocalCloudAgent`: ABC over `BaseAgent` with `_call_anthropic`,
`_call_openai`, `_call_vllm` SDK helpers; standardized AgentResult
metadata shape (tokens_local/tokens_cloud/cost_usd/latency_s/traces);
Opus 4.7 temperature stripping; GPT-5 family `max_completion_tokens`;
soft-fail row helper for deterministic adapter failures.
- `_prices.py`: ported verbatim from hybrid's `prices.py` so the n=500
cost numbers stay aligned with `hybrid-local-cloud-compute/docs/results.md`.
- `_prompts.py`: GAIA + SWE-bench-Verified answer-format instructions,
same `format_prompt(task)` dispatch as `benches/__init__.py`.
- `agents/__init__.py`: wire the new subpackage so its sub-modules
register on package import.
Hybrid harness stays untouched — this is the OpenJarvis-native port.
- calculator: add `ln` alias for `math.log`, replace `^` with `**` before
AST parsing so caret-as-power works, convert SyntaxError → ValueError for
consistent error handling, and return `math.inf` on division by zero
(matching the documented meval behaviour) instead of raising an exception
- file_read / file_write: replace the hardcoded `str(path).startswith(str(d) + "/")
guard with `Path.is_relative_to()` so allowed-directory checks work on
Windows (and any OS whose separator is not `/`)
- git_tool: catch `NotADirectoryError` raised by `subprocess.run` on Windows
when `cwd` points to a non-existent path, returning a proper error result
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: Robby Manihani <manihani@stanford.edu>
* fix: close KnowledgeStore connections in connectors_router endpoints
KnowledgeStore opens a persistent SQLite connection in __init__ but was
never closed at the three instantiation sites in connectors_router.py,
leaking one file descriptor (plus its WAL handle) per call. The worst
offender is _connector_summary(), which is invoked per-connector on
every GET /connectors poll from the frontend — after ~100 polls the
backend hits the default macOS per-process FD limit (256) and asyncio's
accept() starts failing with OSError: [Errno 24] Too many open files.
Add __enter__/__exit__ to KnowledgeStore (close() already existed) and
wrap all three instantiation sites in `with` blocks so the connection
is released deterministically when the scope exits.
- connectors/store.py: add context manager protocol
- server/connectors_router.py: use `with KnowledgeStore() as store:`
in _connector_summary, _ingest, and _run_sync
* test: cover KnowledgeStore context manager + connectors_router close leak
- test_store.py: add two tests for the new __enter__/__exit__ protocol
verifying the connection closes on normal exit and on exceptions.
- test_connectors_router.py: add a regression test that monkeypatches
KnowledgeStore.close to count invocations and asserts every store
opened by GET /v1/connectors is paired with a close.
Also fix a pre-existing bug in the test_connectors_router fixture: the
router is created with prefix="/v1/connectors" internally (line 92 of
connectors_router.py), and the fixture was wrapping it again with
prefix="/v1", producing "/v1/v1/connectors". All 6 existing router
tests were failing with 404 before this fix. 5 of them now pass; the
remaining failure (test_trigger_sync) is an unrelated KeyError on a
missing response field and is out of scope for this PR.
* test: drop connectors_router regression test — skipped in CI anyway
The regression test added in the previous commit relies on FastAPI
being importable, but CI's dev-extra install does not include fastapi
(it lives in the `server` optional group). All tests in
test_connectors_router.py silently skip on CI with "fastapi not
installed", so the regression test would never actually execute there.
Revert test_connectors_router.py to the upstream main version. The
fixture bug I fixed (double prefix) and the connection-leak regression
test both deserve a separate PR scoped to test infrastructure — that
PR should either add fastapi to dev deps, split server tests into
their own CI job, or both.
The KnowledgeStore context manager tests in test_store.py are kept
because they have no fastapi dependency and will run in CI.
Andrew Park