- _loop_local: two-stage compaction at 22k tokens (tiktoken cl100k).
Stage 1 elides old tool messages to [tool output elided: N chars, exit=X],
preserving tool_call_id. Stage 2 folds turn pairs into a synthesized
system note. Last 3 turns + system + initial user always intact.
Emergency-compact retry on a 32k BadRequest from vLLM.
- _loop_cloud_openai: assistant content is "" not None on tool-only turns
(OpenAI 400's on null). Bash output decoded with errors="replace" and
stubbed to [binary output: N bytes, exit=X] on >5% replacement or NUL.
- _openai_retry: local-vLLM branch now retries APIConnectionError /
APITimeoutError / InternalServerError 3x with 1/2/4s backoff.
- Tests: 18 cases covering compaction shape, token budget, tool_call_id
preservation, null-content guard, binary decode, local retry.
Addresses ctx-overflow in Gemma-31B SWE n=100 cells (61 + 19 errored
rows) and the OpenAI/Gemini adapter errors across ~19 cells in the
n=100 hybrid sweep.
Concurrent hybrid SWE cells were colliding on the swebench-harness
shared cache because run_id was keyed only on instance_id
(`oj-<instance_id>`). The second cell to score the same task hit
"1 instances already run, skipping..." and silently scored 0 with
`reason: no_report` (or read the first cell's verdict on a write race).
Previously papered over by manual cache wipes + rescores; this is the
root-cause fix.
run_id is now `oj-<cell>-<instance>` when a cell name is supplied,
falling back to the legacy `oj-<instance>` form for single-cell
callers. Same-cell resumes still hit the harness cache (run_id is
deterministic for a given cell + instance). Cell name is sanitized
to filesystem-safe `[A-Za-z0-9._-]+` since it lands in both the
`logs/run_evaluation/<run_id>/...` subtree and the
`<model>.<run_id>.json` summary filename.
Plumbed cell_name through `runner._process -> score -> _score_swebench
-> SWEBenchHarnessScorer -> _run_harness -> _build_run_id`. Regression
test under `tests/agents/hybrid/test_swebench_run_id_isolation.py`
pins: (a) different cells produce different run_ids, (b) same cell is
stable, (c) legacy `None` cell preserves old format, (d) hostile cell
names sanitize cleanly, (e) end-to-end wiring from scorer constructor
through to the run_id.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Follow-up to 7f432453 (tool_calls). Every row now also carries
n_cloud_calls and n_local_calls so the per-task LLM round-trip
budget is queryable alongside cost / latency / tool_calls.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
results-table.md caveat (3) flagged that tool_calls was missing from
results.jsonl for every paradigm and the legacy skillorchestra values
came from a defunct telemetry path. Wire the count back through the
canonical row-write path so future ablation cells populate it.
Definition per paradigm:
- SWE-bench: sum of bash turns from each run_swe_agent_loop subloop
(one tool call = one bash command the agent ran).
- GAIA: native web_search invocations from the cloud backbone. Zero
on one-shot GAIA paths and on paradigms whose GAIA path is pure
text passing (minions w/o prefetch).
runner._run_one now reads meta["tool_calls"] into the top-level row;
summary.json picks up tool_calls_total. Existing n=100 rows stay "—"
(no rerun).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Tests, ablation registries, rescore script, and m2 distillation configs that
were left untracked alongside the recent feature commits. Junk excluded
(minion_logs/, oj-debug.oj-debug.json — runtime artifacts).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
mini_swe_agent.py adds recovery nudges so the agent doesn't exit silently
when the cloud model produces an empty turn:
- _loop_cloud_openai: finish_reason='length' with empty content/tool_calls
→ re-prompt instead of breaking. Hit gpt-5-mini on SWE n=100 (caused 0-score
exits with empty final_summary).
- _loop_cloud_gemini: MALFORMED_FUNCTION_CALL / MAX_TOKENS with empty parts
→ same recovery. Hit gemini-2.5-flash on SWE n=100 (24/100 tasks).
Also: proper Gemini Schema-shaped bash tool params, gpt-5-family handling,
missing is_gpt5_family import. 5 regression tests cover both recovery paths
and natural-stop sanity guards.
scripts/ablation/run_sweep.py parse_cell anchors on the n100 suffix and
walks left, so advisors-*-gaia-n100 cells report GAIA as bench instead of
'qwen27b'. Old logic only handled cloud-only/skillorchestra-qwen prefixes
explicitly.
Companion working-tree changes shipped here: Gemini price entries,
baseline_cloud module registration, swebench dataset variant note,
swebench_harness Modal cgroup-v2 patch (file write swallows
FileNotFoundError on cgroup paths so Modal v2 sandboxes don't die).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds opt-in `method_cfg.web_search = { enabled, max_uses }` schema and
plumbs the native Anthropic server-side `web_search_20250305` tool
through every hybrid paradigm's GAIA codepath. Default OFF preserves
back-compat with all currently running n=100 cells; only the next
rendered sweep opts in.
Engine layer: `_base.py` gets `build_web_search_tool`, `web_search_cfg`,
and `_call_anthropic_agent` (multi-turn loop with `gaia_max_turns`
default 8). Anthropic cost (`$0.01 / search`) added to `tokens_cloud`.
Per-paradigm wiring (cloud-side Anthropic calls only):
* baseline_cloud: GAIA one-shot upgraded to agent loop with tools
* advisors: executor1 + executor2 passes declare tools
* minions: existing prefetch retained as legacy default; new schema
overrides
* skillorchestra: cloud-route specialist gets tools
* conductor: anthropic worker steps get tools when enabled
* toolorchestra: already used native web_search worker - unchanged,
now surfaces web_search_uses in meta
* archon: thread-local web_search tool injection on Anthropic
ranker/fuser generators
Trace + aggregation:
* Per-row `web_search_uses: int` in results.jsonl
* Summary `web_search_uses_total` in summary.json
* make_report.py adds `web_searches_mean` column to the HTML table
Tests: tests/agents/hybrid/test_web_search_wiring.py - confirms the
tool is declared when enabled, NOT declared when omitted (default),
and gracefully skipped on non-Anthropic endpoints (no fake local
web_search).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Both paradigms previously hardcoded a heterogeneous worker pool (Opus +
gpt-5-mini + Anthropic web-search) and only the orchestrator/planner role
swapped when `cell.cloud.model` changed. Adds an optional
`method_cfg.worker_pool` that, when set, strictly replaces the default
pool, so cells can ablate pool composition without code changes.
Behavior:
- Absent override = legacy default pool (unchanged).
- Each entry: `id` (int), `name` (str), `endpoint`/`type` (whitelisted),
`model` (validated against `_prices.PRICES` for cloud workers;
`anthropic-web-search` may omit it).
- `model = "$local"` / `"<local>"` substitutes `config.local.model`;
`"$cloud"` / `"<cloud>"` substitutes the cell's cloud model.
- Validation runs at agent `__init__` — bad pools fail fast, not mid-task.
- Strict replace (not merge) so the cell.toml is the single source of
truth for the worker pool.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Bug A: DEFAULT_AGENT_COMPETENCE / cost / router prompt hardcoded
`cloud-opus-4-7` with Opus's $0.30/task, so every non-Opus cell
(haiku45, gpt-5, gemini-*) routed under Opus's price + capability — the
cheaper cloud's lower cost was invisible to the router.
Bug B: priors had no overlap (cloud floor 0.70 > local ceiling 0.65) and
lambda_cost=0.5 with $0.30 per-task cost gave a 0.15 penalty that was
swamped by the competence gap. Net effect: 95-98% of tasks routed to
cloud across all n=100 cells.
This patch:
- Plumbs self._cloud_model into the agent key: cloud-<configured-model>.
chosen_agent trace field reflects the real cloud (preserves the schema
field name; only the *value* changes).
- Adds MODEL_COST_USD_PER_TASK calibrated against cloud-only-*-gaia-n100
empirical per-task costs (opus47 $0.014, haiku45 $0.002, gpt-5 $0.025,
gpt-5-mini $0.003, gemini-2.5-pro $0.002, gemini-2.5-flash $0.0006).
- Adds per-cloud competence priors so the router sees haiku/flash with
realistically lower competence (floors ~0.50-0.55) than Opus/Pro
(~0.70+). Local ceiling raised to 0.75 (format_compliance) so local
can win pure-arithmetic / format-heavy mixes.
- Exposes lambda_cost in method_cfg (default 2.0, up from 0.5).
- Allows method_cfg.agent_competence / agent_cost_usd partial overrides
so future cells can plug in calibrated priors without code changes.
Test updated: _ROUTER_JSON emits the new cell-specific key, and a new
test_skillorchestra_chosen_agent_uses_cloud_model_name asserts the
contract end-to-end for a haiku45 cell.
* fix: close KnowledgeStore connections in connectors_router endpoints
KnowledgeStore opens a persistent SQLite connection in __init__ but was
never closed at the three instantiation sites in connectors_router.py,
leaking one file descriptor (plus its WAL handle) per call. The worst
offender is _connector_summary(), which is invoked per-connector on
every GET /connectors poll from the frontend — after ~100 polls the
backend hits the default macOS per-process FD limit (256) and asyncio's
accept() starts failing with OSError: [Errno 24] Too many open files.
Add __enter__/__exit__ to KnowledgeStore (close() already existed) and
wrap all three instantiation sites in `with` blocks so the connection
is released deterministically when the scope exits.
- connectors/store.py: add context manager protocol
- server/connectors_router.py: use `with KnowledgeStore() as store:`
in _connector_summary, _ingest, and _run_sync
* test: cover KnowledgeStore context manager + connectors_router close leak
- test_store.py: add two tests for the new __enter__/__exit__ protocol
verifying the connection closes on normal exit and on exceptions.
- test_connectors_router.py: add a regression test that monkeypatches
KnowledgeStore.close to count invocations and asserts every store
opened by GET /v1/connectors is paired with a close.
Also fix a pre-existing bug in the test_connectors_router fixture: the
router is created with prefix="/v1/connectors" internally (line 92 of
connectors_router.py), and the fixture was wrapping it again with
prefix="/v1", producing "/v1/v1/connectors". All 6 existing router
tests were failing with 404 before this fix. 5 of them now pass; the
remaining failure (test_trigger_sync) is an unrelated KeyError on a
missing response field and is out of scope for this PR.
* test: drop connectors_router regression test — skipped in CI anyway
The regression test added in the previous commit relies on FastAPI
being importable, but CI's dev-extra install does not include fastapi
(it lives in the `server` optional group). All tests in
test_connectors_router.py silently skip on CI with "fastapi not
installed", so the regression test would never actually execute there.
Revert test_connectors_router.py to the upstream main version. The
fixture bug I fixed (double prefix) and the connection-leak regression
test both deserve a separate PR scoped to test infrastructure — that
PR should either add fastapi to dev deps, split server tests into
their own CI job, or both.
The KnowledgeStore context manager tests in test_store.py are kept
because they have no fastapi dependency and will run in CI.
* feat(mining): add Pearl model conversion workflow
* test(mining): tolerate missing Docker device request type
* docs(mining): record Gemma and Qwen conversion evidence
* docs(mining): record Qwen local validation evidence
---------
Co-authored-by: krypticmouse <herumbshandilya123@gmail.com>
The desktop app's chat-completions stream relies on a raw browser
fetch from the Tauri webview to the FastAPI backend, so it is
subject to CORS. The default allowlist only contained
`tauri://localhost`, which is the production webview origin on
macOS, Linux, and iOS. On Windows and Android, Tauri 2 serves the
app from `http://tauri.localhost` (or `https://tauri.localhost`
when `windows.useHttpsScheme` is enabled), so the preflight for
`POST /v1/chat/completions` was rejected and the streaming fetch
threw `TypeError: Failed to fetch` before any byte was read.
Symptom users reported: in the Logs tab the request appears to
succeed up to "Request sent", followed immediately by
"Stream error: Failed to fetch" and "Response: 22 chars" -- which
is exactly the length of the synthesized fallback string
`Error: Failed to fetch` written by InputArea.tsx when streamChat
throws.
Add `http://tauri.localhost` and `https://tauri.localhost` to both
default origin lists (`ServerConfig.cors_origins` and the
`create_app` fallback), and add a regression test that drives a
real preflight against `/v1/chat/completions` for each of the
three Tauri origin schemes.
Browser model loading kept working because it is routed through
the Rust `tauriInvoke('fetch_models')` command, which is a
server-to-server HTTP call not subject to browser CORS -- only
streaming chat goes through the webview's fetch.
* feat: add Apple Contacts connector (macOS AddressBook)
Reads directly from ~/Library/Application Support/AddressBook/AddressBook-v22.abcddb
in read-only mode. Extracts names, phone numbers, emails, postal addresses, URLs,
social profiles, and notes for each contact. Requires Full Disk Access like
iMessage and Apple Notes connectors.
- New connector: src/openjarvis/connectors/apple_contacts.py
- Registered in connectors/__init__.py
- Added frontend catalog entry (PIM category) and icon mapping
- MCP tools: contacts_search, contacts_get_contact
* test: add comprehensive tests for Apple Contacts connector
15 tests covering: is_connected (exists/missing), sync yields all real
contacts (skips system rows), extracts all field types (phone, email,
address, URL, social, notes), cleans Apple label markup, org-only
contacts, minimal contacts, since filter, sync_status tracking,
structured metadata, disconnect, mcp_tools, registry, empty DB,
missing DB. Uses fake SQLite database — no real Contacts DB needed.
* fix: scan iCloud/Exchange source databases for all contacts
The main AddressBook database only contains locally-created contacts.
Synced contacts (iCloud, Exchange, etc.) live under
Sources/<UUID>/AddressBook-v22.abcddb. Now scans all source databases
and deduplicates by ZUNIQUEID across sources.
Adds tests for multi-source scanning and cross-source deduplication.
Update all human-readable strings (docstrings, comments, descriptions,
log messages, TOML config descriptions) to correctly say
"DeepResearchBench" instead of "LiveResearchBench" for the
deep_research_bench benchmark. Class names and file paths are
unchanged for backward compatibility.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- openjarvis.__version__ now comes from importlib.metadata.version("openjarvis")
instead of a hardcoded string, so it stays in sync with pyproject.toml on every
release. Tests assert against openjarvis.__version__ rather than a literal.
- Bump actions/checkout@v4 → @v6 and astral-sh/setup-uv@v4 → @v8 across every
workflow ahead of the 2026-06-02 Node.js 20 deprecation on GitHub Actions runners.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Qwen3.5 MoE models provide better quality per GB of memory since only
a fraction of parameters are active per token. Updates the tier table:
≤8 GB → qwen3.5:2b
8-16GB → qwen3.5:4b
16-32GB → qwen3.5:9b
32GB+ → qwen3.5:27b
Closes#132 — jarvis init now recommends a Qwen3 dense model based on
available memory using an explicit tier table:
≤8 GB → qwen3:1.7b (~1.1 GB)
8-16GB → qwen3:4b (~2.5 GB)
16-32GB → qwen3:8b (~5.2 GB)
32GB+ → qwen3:14b (~9.3 GB)
Adds qwen3:0.6b, 1.7b, 4b, 14b, 30b to the model catalog. Falls back
to scanning all compatible models if the tiered pick doesn't support
the selected engine. Shows the user why the model was chosen and how
to download later if declined.
Closes#164 — Qwen and other models with baked-in corporate identities
would claim to be cloud services when running locally. Adds a
configurable default_system_prompt to AgentConfig that is injected as a
fallback when no other system prompt is provided. The default tells the
model it is running locally through OpenJarvis. Users can override or
clear it via config.toml.
Tests were storing digests with local time but get_today() filters
by UTC date, causing failures when local date != UTC date.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(evals): add tool_choice=auto + fix traces thread safety
Two fixes for eval accuracy and stability:
1. TauBench agent: add tool_choice="auto" to match tau2's native
LLMAgent behavior. Without this, Qwen and GPT-5.4 score 10-14pp
below leaderboard because the models don't receive explicit
tool-calling guidance.
2. SystemBuilder: apply self._traces flag to config.traces.enabled.
Previously builder.traces(False) was a no-op — traces stayed
enabled, creating SQLite connections in the main thread that
crashed when accessed from ThreadPoolExecutor worker threads
in GAIA evals ("SQLite objects created in a thread can only be
used in that same thread").
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat: enrich inference events with model response content and add Trace.messages
Add content, tool_calls, and finish_reason fields to INFERENCE_END events
published by InstrumentedEngine. For non-instrumented engines, BaseAgent._generate()
now publishes INFERENCE_START/END events with the same rich data. Add _message_to_dict
helper and messages field to Trace dataclass for full conversation capture.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat: enhance TraceCollector with rich content, tool details, and messages
Capture model response content, tool_calls, and finish_reason in GENERATE
steps; store tool arguments and result text in TOOL_CALL steps; extract
conversation messages from AgentResult.metadata into Trace.messages; and
implement the last_trace property. Also adds a messages column to the
TraceStore schema so messages survive the SQLite round-trip.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat: agents store conversation messages in AgentResult.metadata
Both NativeReActAgent and MonitorOperativeAgent now serialize their
internal messages list via _message_to_dict and include it in the
returned AgentResult.metadata under the "messages" key. This enables
TraceCollector to capture full conversation traces.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat: wire TraceCollector into system._run_agent and JarvisAgentBackend
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat: persist rich trace data in eval trace JSONL files
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat: add TerminalBench config for Qwen 3.5-122B
* fix: add SQLite migration for traces.messages column on existing databases
* fix: check trace_store instead of shared config for trace enablement
* feat(evals): add ToolCall-15, LiveCodeBench, LiveResearchBench + telemetry
Three new benchmark integrations and full telemetry wiring for the
NeurIPS 2026 IPW/IPJ experiments.
## New Benchmarks
### ToolCall-15
15-scenario tool calling accuracy benchmark across 5 categories.
All scenarios defined inline with deterministic scoring (0/1/2 per
scenario). Fast to run (~5min/model) — ideal for optimization loops.
### LiveCodeBench
Competitive programming from LeetCode/AtCoder/CodeForces via
HuggingFace dataset. Sandboxed code execution with per-test
timeouts. Single-turn generation via jarvis-direct backend.
### LiveResearchBench
100 expert-curated deep research tasks. LLM-as-judge scoring
across 4 dimensions (comprehensiveness, insight, instruction
following, readability). Uses web_search tool for live research.
## Telemetry Wiring
- FLOPs estimation: 2 * active_params * total_tokens (MoE-aware)
- Energy/power capture flows from InstrumentedEngine through
backends to EvalResult and RunSummary
- New telemetry_summary section in output JSON with IPW/IPJ
- JarvisDirectBackend now propagates gpu_metrics flag
- TauBench forwards telemetry flags to SystemBuilder
## Experiment Plan
Added docs/experiments/neurips-2026-plan.md tracking the full
experiment matrix: 9 models x 7 benchmarks across NVIDIA, AMD,
and Apple hardware stacks.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Jon Saad-Falcon <jonsaadfalcon@gmail.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Andrew Park