Previous trace events only stored the joined `text` blocks from Anthropic
responses and `message.content` from OpenAI/vLLM — every tool_use block,
server_tool_use block, web_search_tool_result, and OpenAI-style tool_calls
list was being thrown away.
Now each call event includes:
Anthropic events:
- `content_blocks`: full list of every block the model emitted, with all
fields preserved (text / tool_use / server_tool_use /
web_search_tool_result / thinking). Nested content (e.g. search-result
citations) recurses.
- `tool_calls`: filtered subset (tool_use + server_tool_use).
- `tool_results`: filtered subset (web_search_tool_result + tool_result).
- `tools_declared`, `tool_choice`, `output_config`, `stop_reason`.
OpenAI / vLLM events:
- `tool_calls`: serialized `message.tool_calls` (id, function name, args).
- `reasoning_content`: thinking-model intermediate reasoning if present.
- `tools_declared`, `tool_choice`, `finish_reason`.
Confirmed against a real Anthropic call with `tools=[web_search]`,
`tool_choice={"type":"any"}`: the resulting event contains the
`server_tool_use` block with the actual search query, the
`web_search_tool_result` with 10 nested citation blocks, and all
subsequent text blocks the model emitted to synthesize the answer.
Previously the runner created the `logs/` directory but never wrote
anything. Wire a thread-local trace buffer through the SDK helpers so
every cloud/local call is captured automatically, plus paradigm hooks
for the events that bypass the helpers (Minions's library protocol,
Archon's layer pipeline, conductor's plan + step dispatch, ToolOrchestra
action turns, SkillOrchestra routing decision).
Mechanics:
- `_TRACE_STATE` is a `threading.local()` — `run()` opens a fresh trace
per task and writes a digested JSON to `<log_dir>/<task_id>.json` in
the `finally` block (so we get a record even on hard failures).
- `_call_anthropic` / `_call_openai` / `_call_vllm` append full events
(system, user, response, token counts, latency, model, schema config)
to the active trace.
- `LocalCloudAgent.record_trace_event(...)` is the public hook for
paradigms with library-side execution.
- Nothing is truncated — full prompts, responses, and tool-call payloads
land in the JSON.
- Runner threads `log_dir` through `context.metadata["log_dir"]`.
Verified: re-running `advisors-gaia-qwen9b-opus-3` produces three
`logs/gaia-*.json` files, each ~13KB with all 3 turns (cloud executor →
local advisor → cloud final) and the full text of each prompt/response.
GAIADataset and SWEBenchDataset both build a formatted prompt (`rec.problem`)
and discard the raw input text. The hybrid paradigm runner needs the raw
text so it can apply the GAIA / SWE-bench answer-format instructions from
`agents.hybrid._prompts.format_prompt(task)` — otherwise paradigms see a
double-wrapped prompt and the GAIA "FINAL ANSWER: <x>" rule is missing.
Additive change — adds `metadata["question"]` for GAIA and
`metadata["problem_statement"]` for SWE-bench. No existing consumer
breaks; the runner now picks up the raw text correctly.
Verified end-to-end with `python -m openjarvis.agents.hybrid.runner
--cell advisors-gaia-qwen9b-opus-3`: 3/3 tasks, acc=0.667, $0.04 — matches
the hybrid harness's cell exactly.
Final piece — make the ported paradigms runnable end-to-end through the
same flow as the hybrid harness:
- `runner.py`: `python -m openjarvis.agents.hybrid.runner --cell <name>`.
Loads cells from packaged registry TOMLs, constructs the registered
agent via AgentRegistry, iterates tasks from OpenJarvis's existing
GAIADataset / SWEBenchDataset, scores via gaia exact-match or the new
Modal-backed SWE-bench harness scorer, writes
`results.jsonl`+`summary.json` in the same shape as the hybrid harness
so existing rescore / dashboard scripts work unmodified. Supports
resume (drops errored rows), per-cell flock, ThreadPoolExecutor
concurrency, and the same per-row heartbeat format.
- `registry/{advisors,archon,conductor,minions,skillorchestra,toolorchestra}.toml`:
35 cells ported verbatim from the hybrid harness — same models, same N,
same `method_cfg` — so OpenJarvis runs should reproduce the numbers in
`hybrid-local-cloud-compute/docs/results.md`. ToolOrchestra cells
rewritten from the original NotImplementedError stubs to point at the
prompted port (cloud-as-orchestrator, no Nemotron-Orchestrator-8B).
- `scripts/new_experiment.sh`: mirror of hybrid's scaffolder. Maps
shortnames (`qwen3.5-27b`, `claude-opus-4-7`, …) to full HF ids +
endpoints, picks method-appropriate `method_cfg` defaults, appends a
`[cells.<name>]` block to the right registry TOML.
- `README.md`: quickstart, paradigm taxonomy, reproducibility table
pointing at the hybrid results for headline numbers.
End-to-end smoke: `python -m openjarvis.agents.hybrid.runner --help` works,
all 6 agents register, all 35 cells load.
Adds `SWEBenchHarnessScorer` alongside the existing structural-only
scorer. Hands one prediction at a time to `python -m swebench.harness.
run_evaluation`, parses the report JSON it writes, returns the resolved
bool. Backend selectable via `SWEBENCH_BACKEND` env var
(modal default, docker fallback).
Carries forward the two upstream-swebench patches from the hybrid
harness, both idempotent and lazy:
1. **Modal cgroup-v2 fix** — `swebench/harness/modal_eval/run_evaluation_modal.py`
writes to `/sys/fs/cgroup/cpu/cpu.shares` (cgroup v1). Modal v2 sandboxes
are cgroup v2 — the path doesn't exist and every sandbox died on the
write. This was the root cause of "Modal harness failing 100%"; wrapping
`set_cpu_quota` in try/except fixes it.
2. **Rescore `*_ids` fix** — older swebench wrote `resolved_instances` as
a list; current swebench puts the count there and the actual ids in
`resolved_ids`. Reading `*_ids` first; falling back to the list-typed
legacy field if the user is on an older install.
Patch extraction supports the same three formats as the hybrid harness:
fenced `\`\`\`diff`, fenced raw (`\`\`\`\ndiff --git`), and unfenced
`diff --git`.
Inference-only port of NVlabs ToolOrchestra (arXiv:2511.21689). The
paper's RL-trained Nemotron-Orchestrator-8B is NOT in the loop — the
hybrid harness adapter is a documented stub for exactly that reason
(needs a separate vLLM srun for the 8B checkpoint, a FAISS wiki
retriever, a Tavily key, and an upstream refactor).
This port keeps the same scope discipline: a cloud model plays the
orchestrator role and dispatches to a mixed pool of workers (local
Qwen for cheap extraction, Anthropic web_search for unknowns, Opus
4.7 / gpt-5-mini for hard reasoning or final synthesis). Reactive
multi-turn loop emitting JSON action objects per turn; forces a
final answer at the budget cap; hard fallback to the strongest worker
on persistent parse failure.
Treat results as preliminary until a real Orchestrator-8B deployment
exists — included here so OpenJarvis has registry entries for all six
paradigms and so the distillation pipeline can slot ToolOrchestra in
alongside the rest.
Registered as `toolorchestra`. The hybrid harness `toolorchestra_adapter.py`
remains a NotImplementedError stub; this is the first runnable version.
Inference-time skill-aware router from arXiv:2602.19672. The paper's
4-phase explore/learn/select pipeline is out of scope (needs multi-model
serving + FRAMES wiki retriever + a multi-hour learning loop) — the
hybrid harness reproduced only the deployment-time `select_agent()`
path with a hand-curated skill taxonomy + per-agent competence priors
calibrated to Qwen3.5-27B-FP8 vs Opus 4.7. This port keeps that scope.
Per task: Opus reads the question, picks skill weights from a 7-skill
catalog (factual_recall, multi_step_reasoning, arithmetic, web_grounding,
long_text_extraction, format_compliance, code_or_logic), scores each
agent under `score = weighted_competence - 0.5 * avg_cost`, routes to
the winner. JSON schema enforced via Anthropic `output_config`; balanced-
brace fallback if the SDK ignores the schema. Soft-fail row on
unparseable JSON to match hybrid's `err=1` on n=30.
Hybrid harness result: `skillorchestra-gaia-qwen27b-opus-30` = 0.500 acc,
$0.02/task — 30× cheaper than baseline-cloud (0.567 / $0.66) at ~7pp
lower accuracy. Best cost-efficient GAIA paradigm.
Registered as `skillorchestra`. Ported from
`hybrid-local-cloud-compute/adapters/skillorchestra_adapter.py`.
Layered (generator → ranker → fuser) sampling: K local generators propose
candidates, cloud ranker scores them, cloud fuser synthesizes the final
answer. Paper: arXiv:2409.15254. Two presets: `ensemble_rank_fuse` (the
full layered pipeline) and `single_local` (debug shim).
Carries the same patch story as the hybrid adapter:
- Eager-import stubs for groq/google/litellm so Archon's `utils.py` loads
in the OpenJarvis venv without those deps.
- Custom `vllm_local` model_type wired into Archon's GENERATE_MAP via a
per-run closure (endpoint can vary per cell).
- OpenAI / Anthropic generators replaced with token-tallying wrappers so
we can charge cost — Archon ignores `usage` by default.
- Opus 4.7 temperature stripping at the SDK level.
Archon library is lazy-imported. The hybrid clone at
`hybrid-local-cloud-compute/external/Archon/src` is the default
location; override with `ARCHON_SRC` env var.
Hybrid harness results: `archon-gaia-qwen27b-opus-ensemble-K5-30` =
0.333 acc, $0.20/task; `archon-gaia-qwen27b-singlelocal-30` = 0.033 acc.
Underperforms baseline-cloud on GAIA — included for completeness, not as
a recommended paradigm.
Registered as `archon`. Ported from
`hybrid-local-cloud-compute/adapters/archon_adapter.py`.
Cloud supervisor decomposes + reads back; local worker(s) do bulk
reading. Two modes: `minion` (single worker, default) / `minions`
(parallel workers + aggregator).
Carries forward every compatibility patch from the hybrid adapter so the
n=500 numbers transfer:
- Strip `temperature` for Opus 4.7+ (rejected with 400).
- Inject server-side `output_config` JSON schema per supervisor turn
(first-turn `{reasoning, message}` vs conversation-turn anyOf
`{decision, message|answer}`), picked by sniffing the prompt for
`"decision": "provide_final_answer"`.
- Replace `minions._extract_json` with a JSON-first wrapper.
- Inject `timeout=600`/`max_retries=5` into `anthropic.Anthropic()` so
Minions's bare-client construction doesn't 60s-timeout under SWE-bench
concurrency=8.
- GAIA-only `web_search` prefetch so the worker has a real doc to read.
Soft-fail row on JSONDecodeError/BadRequestError/prompt-too-long — same
deterministic-failure handling that produced the n=165 GAIA `err=6` rows
in the hybrid harness without crashing the cell.
`minions` library import is lazy; install via
`uv pip install -e /matx/u/aspark/hybrid-local-cloud-compute/external/minions`.
Hybrid harness results being reproduced:
- `minions-swebenchverified-qwen27b-opus-500` = 0.274 acc, $0.09/task
(+3.8pp vs baseline-cloud at 10× cheaper — cleanest paradigm-validation
result in the harness).
- `minions-gaia-qwen27b-opus-165` = 0.576 acc, $0.67/task
(~tied with baseline-cloud at 1.6× cheaper).
Registered as `minions`. Ported from
`hybrid-local-cloud-compute/adapters/minions_adapter.py`.
Inference-only Stage-1 repro: zero-shot cloud planner substitutes for the
paper's RL-trained Qwen2.5-7B conductor. Emits 3 JSON lists
(model_id/subtasks/access_list), executor runs ≤5 steps over a worker pool.
Two-tier parse fallback: JSON → Python-literal regex → single call to the
strongest worker. Worker pool defaults to {local Qwen if vLLM up, Opus 4.7,
gpt-5-mini}; override via `cfg["workers"]`.
Hybrid harness result: `conductor-swebenchverified-opusplan-30` = 0.367 acc
($0.22/task) vs baseline-cloud 0.267 ($3.36/task) — +10pp at ~15× cheaper.
Registered as `conductor`. Ported from
`hybrid-local-cloud-compute/adapters/conductor_adapter.py`.
Three-step executor (cloud) → advisor (local) → executor (cloud) loop from
arXiv:2510.02453. Inference-only: the paper's RL-trained advisor would
land higher; this matches the hybrid harness's `advisors-gaia-qwen9b-opus-30`
cell at 0.533 acc / $0.02 per task.
Registered as `advisors`. Local model id auto-resolves against vLLM's
`/v1/models` so cell configs naming Qwen3.5-9B don't 404 when the server
is actually serving 27B-FP8.
Ported from `hybrid-local-cloud-compute/adapters/advisors_adapter.py`.
Add `openjarvis.agents.hybrid` — the home for the local+cloud paradigms
being ported from `/matx/u/aspark/hybrid-local-cloud-compute`. This commit
ships only the shared base (no paradigm yet):
- `_base.LocalCloudAgent`: ABC over `BaseAgent` with `_call_anthropic`,
`_call_openai`, `_call_vllm` SDK helpers; standardized AgentResult
metadata shape (tokens_local/tokens_cloud/cost_usd/latency_s/traces);
Opus 4.7 temperature stripping; GPT-5 family `max_completion_tokens`;
soft-fail row helper for deterministic adapter failures.
- `_prices.py`: ported verbatim from hybrid's `prices.py` so the n=500
cost numbers stay aligned with `hybrid-local-cloud-compute/docs/results.md`.
- `_prompts.py`: GAIA + SWE-bench-Verified answer-format instructions,
same `format_prompt(task)` dispatch as `benches/__init__.py`.
- `agents/__init__.py`: wire the new subpackage so its sub-modules
register on package import.
Hybrid harness stays untouched — this is the OpenJarvis-native port.
- calculator: add `ln` alias for `math.log`, replace `^` with `**` before
AST parsing so caret-as-power works, convert SyntaxError → ValueError for
consistent error handling, and return `math.inf` on division by zero
(matching the documented meval behaviour) instead of raising an exception
- file_read / file_write: replace the hardcoded `str(path).startswith(str(d) + "/")
guard with `Path.is_relative_to()` so allowed-directory checks work on
Windows (and any OS whose separator is not `/`)
- git_tool: catch `NotADirectoryError` raised by `subprocess.run` on Windows
when `cwd` points to a non-existent path, returning a proper error result
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: Robby Manihani <manihani@stanford.edu>
* fix: close KnowledgeStore connections in connectors_router endpoints
KnowledgeStore opens a persistent SQLite connection in __init__ but was
never closed at the three instantiation sites in connectors_router.py,
leaking one file descriptor (plus its WAL handle) per call. The worst
offender is _connector_summary(), which is invoked per-connector on
every GET /connectors poll from the frontend — after ~100 polls the
backend hits the default macOS per-process FD limit (256) and asyncio's
accept() starts failing with OSError: [Errno 24] Too many open files.
Add __enter__/__exit__ to KnowledgeStore (close() already existed) and
wrap all three instantiation sites in `with` blocks so the connection
is released deterministically when the scope exits.
- connectors/store.py: add context manager protocol
- server/connectors_router.py: use `with KnowledgeStore() as store:`
in _connector_summary, _ingest, and _run_sync
* test: cover KnowledgeStore context manager + connectors_router close leak
- test_store.py: add two tests for the new __enter__/__exit__ protocol
verifying the connection closes on normal exit and on exceptions.
- test_connectors_router.py: add a regression test that monkeypatches
KnowledgeStore.close to count invocations and asserts every store
opened by GET /v1/connectors is paired with a close.
Also fix a pre-existing bug in the test_connectors_router fixture: the
router is created with prefix="/v1/connectors" internally (line 92 of
connectors_router.py), and the fixture was wrapping it again with
prefix="/v1", producing "/v1/v1/connectors". All 6 existing router
tests were failing with 404 before this fix. 5 of them now pass; the
remaining failure (test_trigger_sync) is an unrelated KeyError on a
missing response field and is out of scope for this PR.
* test: drop connectors_router regression test — skipped in CI anyway
The regression test added in the previous commit relies on FastAPI
being importable, but CI's dev-extra install does not include fastapi
(it lives in the `server` optional group). All tests in
test_connectors_router.py silently skip on CI with "fastapi not
installed", so the regression test would never actually execute there.
Revert test_connectors_router.py to the upstream main version. The
fixture bug I fixed (double prefix) and the connection-leak regression
test both deserve a separate PR scoped to test infrastructure — that
PR should either add fastapi to dev deps, split server tests into
their own CI job, or both.
The KnowledgeStore context manager tests in test_store.py are kept
because they have no fastapi dependency and will run in CI.
* feat(mining): add Pearl model conversion workflow
* test(mining): tolerate missing Docker device request type
* docs(mining): record Gemma and Qwen conversion evidence
* docs(mining): record Qwen local validation evidence
---------
Co-authored-by: krypticmouse <herumbshandilya123@gmail.com>
routing.types appears to have been an in-progress module that was
consolidated into routing.router before it was created. The lazy
import inside the router policy block was silently caught by the
surrounding try/except, causing the policy to never activate and
the executor to always fall back to the configured model regardless
of any router_policy setting.
Corrects the import to openjarvis.learning.routing.router where
build_routing_context is defined and exported.
The desktop app's chat-completions stream relies on a raw browser
fetch from the Tauri webview to the FastAPI backend, so it is
subject to CORS. The default allowlist only contained
`tauri://localhost`, which is the production webview origin on
macOS, Linux, and iOS. On Windows and Android, Tauri 2 serves the
app from `http://tauri.localhost` (or `https://tauri.localhost`
when `windows.useHttpsScheme` is enabled), so the preflight for
`POST /v1/chat/completions` was rejected and the streaming fetch
threw `TypeError: Failed to fetch` before any byte was read.
Symptom users reported: in the Logs tab the request appears to
succeed up to "Request sent", followed immediately by
"Stream error: Failed to fetch" and "Response: 22 chars" -- which
is exactly the length of the synthesized fallback string
`Error: Failed to fetch` written by InputArea.tsx when streamChat
throws.
Add `http://tauri.localhost` and `https://tauri.localhost` to both
default origin lists (`ServerConfig.cors_origins` and the
`create_app` fallback), and add a regression test that drives a
real preflight against `/v1/chat/completions` for each of the
three Tauri origin schemes.
Browser model loading kept working because it is routed through
the Rust `tauriInvoke('fetch_models')` command, which is a
server-to-server HTTP call not subject to browser CORS -- only
streaming chat goes through the webview's fetch.
Removes the pull_request trigger so the workflow only runs on explicit
@claude mentions or manual workflow_dispatch.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
server/app.py defaulted to allow_origins=["*"] combined with
allow_credentials=True when cors_origins was not passed. That
combination is invalid per the CORS spec — browsers reject it — and
when used loosely it signals that any cross-origin request with
credentials is allowed, exposing session cookies and auth headers.
The reference config at configs/openjarvis/config.toml binds to
0.0.0.0 without setting cors_origins, so this fallback fired in every
default deployment.
Default to a closed list (Vite dev origin + Tauri webview) instead of
the wildcard. Users who actually need a custom origin already pass it
via cors_origins.
Closes#222
The dashboard at /dashboard renders blank once the Rust security module
is compiled (uv run maturin develop ...). The middleware sets
"Content-Security-Policy: default-src 'self'", which blocks the inline
scripts and styles the dashboard HTML relies on.
Loosen the policy to allow 'unsafe-inline' and 'unsafe-eval' so the
dashboard works once the security middleware is active. The same string
is exported via SECURITY_HEADERS for tests, so update both call sites.
Closes#261
Andrew Park