Same 'swe_use_agent_loop' opt-in flag as Conductor/SkillOrchestra.
Advisors (swe mode):
- Initial executor pass = full mini-SWE-agent run on a fresh workdir.
- Advisor (local) critiques the produced patch (sees summary + diff).
- Final executor pass = SECOND mini-SWE-agent run on ANOTHER fresh
workdir, with the initial summary + advisor feedback folded into the
prompt. Don't reuse the initial workdir — that would smuggle the
bad-fix into the new attempt; the final pass should incorporate only
the parts of the advisor feedback that hold up.
ToolOrchestra (swe mode):
- One SHARED workdir created at start of run, cloned from the SWE-bench
repo at base_commit.
- Each call_worker action whose worker is a solver (vllm/anthropic)
runs as a mini-SWE-agent subloop on the shared workdir. Web-search
workers stay one-shot (search isn't an agent loop). OpenAI workers
fall back to one-shot too (loop tool format isn't wired for OpenAI).
- After loop ends, the framed final answer wraps the working-tree
git diff so the swebench scorer picks it up.
Add 'swe_use_agent_loop' cfg flag to both paradigms. When on AND the
task is SWE-bench-shaped (has repo + base_commit + problem_statement),
the worker call is replaced by a run_swe_agent_loop subloop.
Conductor: one shared workdir across all plan steps so step N+1 builds
on step N's edits. Each step's subtask becomes the initial_prompt; the
worker's model + endpoint determines the agent-loop backbone
(vllm → local, anthropic → cloud, openai workers fall back to one-shot
since the agent-loop tool format isn't wired for OpenAI today). After
the last step, the framed final answer is the working-tree git diff.
SkillOrchestra: same flag, no workdir sharing (just one worker invoked).
Whichever of local-qwen / cloud-opus the router picks runs as
run_swe_agent_loop instead of a single completion.
Cells stay one-shot by default — set method_cfg.swe_use_agent_loop=true
to opt in. Smoke imports clean; behavior on non-SWE benches unchanged.
Refactor MiniSWEAgent so the workdir setup + multi-turn bash loop +
diff extraction live in a top-level function. Paradigms can now call
run_swe_agent_loop(task, backbone, model, ...) to replace their
one-shot worker call with a real agent loop on SWE-bench tasks.
MiniSWEAgent itself becomes a thin LocalCloudAgent wrapper around the
function (~30 lines). Behavior unchanged for the standalone agent;
trace event kinds still use 'mini_swe_*' prefix (paradigms pass
trace_prefix= to namespace their subloops in the log).
Adds `MiniSWEAgent` — the environment-interaction loop the paradigms
were missing for SWE-bench. Every existing paradigm (Minions, Conductor,
Archon, Advisors, etc.) is an *orchestration* loop but the worker
ultimately runs one-shot "predict the patch from the issue text." No
shell, no repo browsing, no test execution. That's why every paradigm
caps near 0.30 on SWE-bench-Verified — the model literally cannot look
at the code.
mini-SWE-agent fills the gap: one bash tool, a per-task git clone, and
a multi-turn loop that lets the model grep / cat / sed / pytest its way
to a real fix. Upstream (https://github.com/swe-agent/mini-swe-agent)
gets ~0.77 on SWE-bench-Verified with a frontier backbone.
This is the vendored option (a) from the plan — no Docker, no upstream
dependency, ~330 lines total:
- `MiniSWEAgent._run_paradigm`: clone the SWE-bench repo into a tempdir,
run the loop, extract the final diff via `git diff`, wrap it in a
```diff fence so the existing swebench scorer reads it.
- Two backbone modes (`cfg["backbone"]`): `cloud` (Anthropic, multi-turn
with `tools=[bash]`), `local` (vLLM OpenAI-compatible with the same).
Both routes integrate with the LocalCloudAgent trace buffer — every
bash invocation lands as a `mini_swe_bash` event with full command,
stdout, stderr, exit code, latency.
- Each `bash` command runs as a fresh subshell in the workdir with a
configurable timeout and 10K-char output cap.
- No `submit` tool — the loop ends when the model produces a turn with
no tool_calls, or `max_turns` hits.
Safety: model can `rm -rf` its own workdir. Workdir is disposable.
Network is available (pip etc.). Don't run on a host with secrets in
$CWD. Docker sandboxing is a follow-up.
Registry: `registry/mini_swe_agent.toml` ships three cells (n=3 cloud,
n=30 cloud, n=3 local-backbone). Smoke (n=1, max_turns=10): the agent
explored the django repo, edited `django/contrib/auth/forms.py`, ran
the relevant tests, produced an 829-char patch. ~$0.25 / 75s per task
with Opus 4.7 at max_turns=10. Cost+time will scale roughly linearly
with max_turns; production cells use max_turns=30-50.
Follow-up: wrap Minions/Conductor workers in this loop so the
orchestration paradigms get the same 0.30 → ~0.77 jump (task #13).
Same gap as the hybrid agents had: `_generate_anthropic` walked `resp.content`
and only matched `type == "tool_use"` for tool calls. Anthropic server-side
tools (web_search) emit `server_tool_use` blocks for the model's actual
query and `web_search_tool_result` blocks for the result payload — both
were silently dropped. So every OpenJarvis agent using Anthropic web_search
had its search queries and results invisible in traces.db.
Fix:
- Add `_serialize_anthropic_block(block)` — recursive JSON-safe converter
that handles text / tool_use / server_tool_use / web_search_tool_result
/ tool_result / thinking blocks, preserving nested citation content.
- `_generate_anthropic` now returns a full `content_blocks` list (every
block in order) plus extended `tool_calls` (tool_use + server_tool_use,
with a `server_side` flag) and a new `tool_results` field
(web_search_tool_result + tool_result).
- `instrumented_engine` propagates these into the INFERENCE_END event.
- `TraceCollector._on_inference_end` records them on the generate step.
Backward compat: existing callers that only read `result["tool_calls"]`
still see the same shape; new fields are additive.
Confirmed against a real Opus web_search call: serializer captures the
`server_tool_use` block with the actual query (`{"query": "when did
SpaceX first launch Starship"}`), the `web_search_tool_result` block
with 10 nested citation entries, and the synthesizing text block.
Previous trace events only stored the joined `text` blocks from Anthropic
responses and `message.content` from OpenAI/vLLM — every tool_use block,
server_tool_use block, web_search_tool_result, and OpenAI-style tool_calls
list was being thrown away.
Now each call event includes:
Anthropic events:
- `content_blocks`: full list of every block the model emitted, with all
fields preserved (text / tool_use / server_tool_use /
web_search_tool_result / thinking). Nested content (e.g. search-result
citations) recurses.
- `tool_calls`: filtered subset (tool_use + server_tool_use).
- `tool_results`: filtered subset (web_search_tool_result + tool_result).
- `tools_declared`, `tool_choice`, `output_config`, `stop_reason`.
OpenAI / vLLM events:
- `tool_calls`: serialized `message.tool_calls` (id, function name, args).
- `reasoning_content`: thinking-model intermediate reasoning if present.
- `tools_declared`, `tool_choice`, `finish_reason`.
Confirmed against a real Anthropic call with `tools=[web_search]`,
`tool_choice={"type":"any"}`: the resulting event contains the
`server_tool_use` block with the actual search query, the
`web_search_tool_result` with 10 nested citation blocks, and all
subsequent text blocks the model emitted to synthesize the answer.
Previously the runner created the `logs/` directory but never wrote
anything. Wire a thread-local trace buffer through the SDK helpers so
every cloud/local call is captured automatically, plus paradigm hooks
for the events that bypass the helpers (Minions's library protocol,
Archon's layer pipeline, conductor's plan + step dispatch, ToolOrchestra
action turns, SkillOrchestra routing decision).
Mechanics:
- `_TRACE_STATE` is a `threading.local()` — `run()` opens a fresh trace
per task and writes a digested JSON to `<log_dir>/<task_id>.json` in
the `finally` block (so we get a record even on hard failures).
- `_call_anthropic` / `_call_openai` / `_call_vllm` append full events
(system, user, response, token counts, latency, model, schema config)
to the active trace.
- `LocalCloudAgent.record_trace_event(...)` is the public hook for
paradigms with library-side execution.
- Nothing is truncated — full prompts, responses, and tool-call payloads
land in the JSON.
- Runner threads `log_dir` through `context.metadata["log_dir"]`.
Verified: re-running `advisors-gaia-qwen9b-opus-3` produces three
`logs/gaia-*.json` files, each ~13KB with all 3 turns (cloud executor →
local advisor → cloud final) and the full text of each prompt/response.
GAIADataset and SWEBenchDataset both build a formatted prompt (`rec.problem`)
and discard the raw input text. The hybrid paradigm runner needs the raw
text so it can apply the GAIA / SWE-bench answer-format instructions from
`agents.hybrid._prompts.format_prompt(task)` — otherwise paradigms see a
double-wrapped prompt and the GAIA "FINAL ANSWER: <x>" rule is missing.
Additive change — adds `metadata["question"]` for GAIA and
`metadata["problem_statement"]` for SWE-bench. No existing consumer
breaks; the runner now picks up the raw text correctly.
Verified end-to-end with `python -m openjarvis.agents.hybrid.runner
--cell advisors-gaia-qwen9b-opus-3`: 3/3 tasks, acc=0.667, $0.04 — matches
the hybrid harness's cell exactly.
Final piece — make the ported paradigms runnable end-to-end through the
same flow as the hybrid harness:
- `runner.py`: `python -m openjarvis.agents.hybrid.runner --cell <name>`.
Loads cells from packaged registry TOMLs, constructs the registered
agent via AgentRegistry, iterates tasks from OpenJarvis's existing
GAIADataset / SWEBenchDataset, scores via gaia exact-match or the new
Modal-backed SWE-bench harness scorer, writes
`results.jsonl`+`summary.json` in the same shape as the hybrid harness
so existing rescore / dashboard scripts work unmodified. Supports
resume (drops errored rows), per-cell flock, ThreadPoolExecutor
concurrency, and the same per-row heartbeat format.
- `registry/{advisors,archon,conductor,minions,skillorchestra,toolorchestra}.toml`:
35 cells ported verbatim from the hybrid harness — same models, same N,
same `method_cfg` — so OpenJarvis runs should reproduce the numbers in
`hybrid-local-cloud-compute/docs/results.md`. ToolOrchestra cells
rewritten from the original NotImplementedError stubs to point at the
prompted port (cloud-as-orchestrator, no Nemotron-Orchestrator-8B).
- `scripts/new_experiment.sh`: mirror of hybrid's scaffolder. Maps
shortnames (`qwen3.5-27b`, `claude-opus-4-7`, …) to full HF ids +
endpoints, picks method-appropriate `method_cfg` defaults, appends a
`[cells.<name>]` block to the right registry TOML.
- `README.md`: quickstart, paradigm taxonomy, reproducibility table
pointing at the hybrid results for headline numbers.
End-to-end smoke: `python -m openjarvis.agents.hybrid.runner --help` works,
all 6 agents register, all 35 cells load.
Adds `SWEBenchHarnessScorer` alongside the existing structural-only
scorer. Hands one prediction at a time to `python -m swebench.harness.
run_evaluation`, parses the report JSON it writes, returns the resolved
bool. Backend selectable via `SWEBENCH_BACKEND` env var
(modal default, docker fallback).
Carries forward the two upstream-swebench patches from the hybrid
harness, both idempotent and lazy:
1. **Modal cgroup-v2 fix** — `swebench/harness/modal_eval/run_evaluation_modal.py`
writes to `/sys/fs/cgroup/cpu/cpu.shares` (cgroup v1). Modal v2 sandboxes
are cgroup v2 — the path doesn't exist and every sandbox died on the
write. This was the root cause of "Modal harness failing 100%"; wrapping
`set_cpu_quota` in try/except fixes it.
2. **Rescore `*_ids` fix** — older swebench wrote `resolved_instances` as
a list; current swebench puts the count there and the actual ids in
`resolved_ids`. Reading `*_ids` first; falling back to the list-typed
legacy field if the user is on an older install.
Patch extraction supports the same three formats as the hybrid harness:
fenced `\`\`\`diff`, fenced raw (`\`\`\`\ndiff --git`), and unfenced
`diff --git`.
Inference-only port of NVlabs ToolOrchestra (arXiv:2511.21689). The
paper's RL-trained Nemotron-Orchestrator-8B is NOT in the loop — the
hybrid harness adapter is a documented stub for exactly that reason
(needs a separate vLLM srun for the 8B checkpoint, a FAISS wiki
retriever, a Tavily key, and an upstream refactor).
This port keeps the same scope discipline: a cloud model plays the
orchestrator role and dispatches to a mixed pool of workers (local
Qwen for cheap extraction, Anthropic web_search for unknowns, Opus
4.7 / gpt-5-mini for hard reasoning or final synthesis). Reactive
multi-turn loop emitting JSON action objects per turn; forces a
final answer at the budget cap; hard fallback to the strongest worker
on persistent parse failure.
Treat results as preliminary until a real Orchestrator-8B deployment
exists — included here so OpenJarvis has registry entries for all six
paradigms and so the distillation pipeline can slot ToolOrchestra in
alongside the rest.
Registered as `toolorchestra`. The hybrid harness `toolorchestra_adapter.py`
remains a NotImplementedError stub; this is the first runnable version.
Inference-time skill-aware router from arXiv:2602.19672. The paper's
4-phase explore/learn/select pipeline is out of scope (needs multi-model
serving + FRAMES wiki retriever + a multi-hour learning loop) — the
hybrid harness reproduced only the deployment-time `select_agent()`
path with a hand-curated skill taxonomy + per-agent competence priors
calibrated to Qwen3.5-27B-FP8 vs Opus 4.7. This port keeps that scope.
Per task: Opus reads the question, picks skill weights from a 7-skill
catalog (factual_recall, multi_step_reasoning, arithmetic, web_grounding,
long_text_extraction, format_compliance, code_or_logic), scores each
agent under `score = weighted_competence - 0.5 * avg_cost`, routes to
the winner. JSON schema enforced via Anthropic `output_config`; balanced-
brace fallback if the SDK ignores the schema. Soft-fail row on
unparseable JSON to match hybrid's `err=1` on n=30.
Hybrid harness result: `skillorchestra-gaia-qwen27b-opus-30` = 0.500 acc,
$0.02/task — 30× cheaper than baseline-cloud (0.567 / $0.66) at ~7pp
lower accuracy. Best cost-efficient GAIA paradigm.
Registered as `skillorchestra`. Ported from
`hybrid-local-cloud-compute/adapters/skillorchestra_adapter.py`.
Layered (generator → ranker → fuser) sampling: K local generators propose
candidates, cloud ranker scores them, cloud fuser synthesizes the final
answer. Paper: arXiv:2409.15254. Two presets: `ensemble_rank_fuse` (the
full layered pipeline) and `single_local` (debug shim).
Carries the same patch story as the hybrid adapter:
- Eager-import stubs for groq/google/litellm so Archon's `utils.py` loads
in the OpenJarvis venv without those deps.
- Custom `vllm_local` model_type wired into Archon's GENERATE_MAP via a
per-run closure (endpoint can vary per cell).
- OpenAI / Anthropic generators replaced with token-tallying wrappers so
we can charge cost — Archon ignores `usage` by default.
- Opus 4.7 temperature stripping at the SDK level.
Archon library is lazy-imported. The hybrid clone at
`hybrid-local-cloud-compute/external/Archon/src` is the default
location; override with `ARCHON_SRC` env var.
Hybrid harness results: `archon-gaia-qwen27b-opus-ensemble-K5-30` =
0.333 acc, $0.20/task; `archon-gaia-qwen27b-singlelocal-30` = 0.033 acc.
Underperforms baseline-cloud on GAIA — included for completeness, not as
a recommended paradigm.
Registered as `archon`. Ported from
`hybrid-local-cloud-compute/adapters/archon_adapter.py`.
Cloud supervisor decomposes + reads back; local worker(s) do bulk
reading. Two modes: `minion` (single worker, default) / `minions`
(parallel workers + aggregator).
Carries forward every compatibility patch from the hybrid adapter so the
n=500 numbers transfer:
- Strip `temperature` for Opus 4.7+ (rejected with 400).
- Inject server-side `output_config` JSON schema per supervisor turn
(first-turn `{reasoning, message}` vs conversation-turn anyOf
`{decision, message|answer}`), picked by sniffing the prompt for
`"decision": "provide_final_answer"`.
- Replace `minions._extract_json` with a JSON-first wrapper.
- Inject `timeout=600`/`max_retries=5` into `anthropic.Anthropic()` so
Minions's bare-client construction doesn't 60s-timeout under SWE-bench
concurrency=8.
- GAIA-only `web_search` prefetch so the worker has a real doc to read.
Soft-fail row on JSONDecodeError/BadRequestError/prompt-too-long — same
deterministic-failure handling that produced the n=165 GAIA `err=6` rows
in the hybrid harness without crashing the cell.
`minions` library import is lazy; install via
`uv pip install -e /matx/u/aspark/hybrid-local-cloud-compute/external/minions`.
Hybrid harness results being reproduced:
- `minions-swebenchverified-qwen27b-opus-500` = 0.274 acc, $0.09/task
(+3.8pp vs baseline-cloud at 10× cheaper — cleanest paradigm-validation
result in the harness).
- `minions-gaia-qwen27b-opus-165` = 0.576 acc, $0.67/task
(~tied with baseline-cloud at 1.6× cheaper).
Registered as `minions`. Ported from
`hybrid-local-cloud-compute/adapters/minions_adapter.py`.
Inference-only Stage-1 repro: zero-shot cloud planner substitutes for the
paper's RL-trained Qwen2.5-7B conductor. Emits 3 JSON lists
(model_id/subtasks/access_list), executor runs ≤5 steps over a worker pool.
Two-tier parse fallback: JSON → Python-literal regex → single call to the
strongest worker. Worker pool defaults to {local Qwen if vLLM up, Opus 4.7,
gpt-5-mini}; override via `cfg["workers"]`.
Hybrid harness result: `conductor-swebenchverified-opusplan-30` = 0.367 acc
($0.22/task) vs baseline-cloud 0.267 ($3.36/task) — +10pp at ~15× cheaper.
Registered as `conductor`. Ported from
`hybrid-local-cloud-compute/adapters/conductor_adapter.py`.
Three-step executor (cloud) → advisor (local) → executor (cloud) loop from
arXiv:2510.02453. Inference-only: the paper's RL-trained advisor would
land higher; this matches the hybrid harness's `advisors-gaia-qwen9b-opus-30`
cell at 0.533 acc / $0.02 per task.
Registered as `advisors`. Local model id auto-resolves against vLLM's
`/v1/models` so cell configs naming Qwen3.5-9B don't 404 when the server
is actually serving 27B-FP8.
Ported from `hybrid-local-cloud-compute/adapters/advisors_adapter.py`.
Add `openjarvis.agents.hybrid` — the home for the local+cloud paradigms
being ported from `/matx/u/aspark/hybrid-local-cloud-compute`. This commit
ships only the shared base (no paradigm yet):
- `_base.LocalCloudAgent`: ABC over `BaseAgent` with `_call_anthropic`,
`_call_openai`, `_call_vllm` SDK helpers; standardized AgentResult
metadata shape (tokens_local/tokens_cloud/cost_usd/latency_s/traces);
Opus 4.7 temperature stripping; GPT-5 family `max_completion_tokens`;
soft-fail row helper for deterministic adapter failures.
- `_prices.py`: ported verbatim from hybrid's `prices.py` so the n=500
cost numbers stay aligned with `hybrid-local-cloud-compute/docs/results.md`.
- `_prompts.py`: GAIA + SWE-bench-Verified answer-format instructions,
same `format_prompt(task)` dispatch as `benches/__init__.py`.
- `agents/__init__.py`: wire the new subpackage so its sub-modules
register on package import.
Hybrid harness stays untouched — this is the OpenJarvis-native port.
- calculator: add `ln` alias for `math.log`, replace `^` with `**` before
AST parsing so caret-as-power works, convert SyntaxError → ValueError for
consistent error handling, and return `math.inf` on division by zero
(matching the documented meval behaviour) instead of raising an exception
- file_read / file_write: replace the hardcoded `str(path).startswith(str(d) + "/")
guard with `Path.is_relative_to()` so allowed-directory checks work on
Windows (and any OS whose separator is not `/`)
- git_tool: catch `NotADirectoryError` raised by `subprocess.run` on Windows
when `cwd` points to a non-existent path, returning a proper error result
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: Robby Manihani <manihani@stanford.edu>
* fix: close KnowledgeStore connections in connectors_router endpoints
KnowledgeStore opens a persistent SQLite connection in __init__ but was
never closed at the three instantiation sites in connectors_router.py,
leaking one file descriptor (plus its WAL handle) per call. The worst
offender is _connector_summary(), which is invoked per-connector on
every GET /connectors poll from the frontend — after ~100 polls the
backend hits the default macOS per-process FD limit (256) and asyncio's
accept() starts failing with OSError: [Errno 24] Too many open files.
Add __enter__/__exit__ to KnowledgeStore (close() already existed) and
wrap all three instantiation sites in `with` blocks so the connection
is released deterministically when the scope exits.
- connectors/store.py: add context manager protocol
- server/connectors_router.py: use `with KnowledgeStore() as store:`
in _connector_summary, _ingest, and _run_sync
* test: cover KnowledgeStore context manager + connectors_router close leak
- test_store.py: add two tests for the new __enter__/__exit__ protocol
verifying the connection closes on normal exit and on exceptions.
- test_connectors_router.py: add a regression test that monkeypatches
KnowledgeStore.close to count invocations and asserts every store
opened by GET /v1/connectors is paired with a close.
Also fix a pre-existing bug in the test_connectors_router fixture: the
router is created with prefix="/v1/connectors" internally (line 92 of
connectors_router.py), and the fixture was wrapping it again with
prefix="/v1", producing "/v1/v1/connectors". All 6 existing router
tests were failing with 404 before this fix. 5 of them now pass; the
remaining failure (test_trigger_sync) is an unrelated KeyError on a
missing response field and is out of scope for this PR.
* test: drop connectors_router regression test — skipped in CI anyway
The regression test added in the previous commit relies on FastAPI
being importable, but CI's dev-extra install does not include fastapi
(it lives in the `server` optional group). All tests in
test_connectors_router.py silently skip on CI with "fastapi not
installed", so the regression test would never actually execute there.
Revert test_connectors_router.py to the upstream main version. The
fixture bug I fixed (double prefix) and the connection-leak regression
test both deserve a separate PR scoped to test infrastructure — that
PR should either add fastapi to dev deps, split server tests into
their own CI job, or both.
The KnowledgeStore context manager tests in test_store.py are kept
because they have no fastapi dependency and will run in CI.
* feat(mining): add Pearl model conversion workflow
* test(mining): tolerate missing Docker device request type
* docs(mining): record Gemma and Qwen conversion evidence
* docs(mining): record Qwen local validation evidence
---------
Co-authored-by: krypticmouse <herumbshandilya123@gmail.com>
routing.types appears to have been an in-progress module that was
consolidated into routing.router before it was created. The lazy
import inside the router policy block was silently caught by the
surrounding try/except, causing the policy to never activate and
the executor to always fall back to the configured model regardless
of any router_policy setting.
Corrects the import to openjarvis.learning.routing.router where
build_routing_context is defined and exported.
The desktop app's chat-completions stream relies on a raw browser
fetch from the Tauri webview to the FastAPI backend, so it is
subject to CORS. The default allowlist only contained
`tauri://localhost`, which is the production webview origin on
macOS, Linux, and iOS. On Windows and Android, Tauri 2 serves the
app from `http://tauri.localhost` (or `https://tauri.localhost`
when `windows.useHttpsScheme` is enabled), so the preflight for
`POST /v1/chat/completions` was rejected and the streaming fetch
threw `TypeError: Failed to fetch` before any byte was read.
Symptom users reported: in the Logs tab the request appears to
succeed up to "Request sent", followed immediately by
"Stream error: Failed to fetch" and "Response: 22 chars" -- which
is exactly the length of the synthesized fallback string
`Error: Failed to fetch` written by InputArea.tsx when streamChat
throws.
Add `http://tauri.localhost` and `https://tauri.localhost` to both
default origin lists (`ServerConfig.cors_origins` and the
`create_app` fallback), and add a regression test that drives a
real preflight against `/v1/chat/completions` for each of the
three Tauri origin schemes.
Browser model loading kept working because it is routed through
the Rust `tauriInvoke('fetch_models')` command, which is a
server-to-server HTTP call not subject to browser CORS -- only
streaming chat goes through the webview's fetch.
Removes the pull_request trigger so the workflow only runs on explicit
@claude mentions or manual workflow_dispatch.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Andrew Park