Qwen3.5 MoE models provide better quality per GB of memory since only
a fraction of parameters are active per token. Updates the tier table:
≤8 GB → qwen3.5:2b
8-16GB → qwen3.5:4b
16-32GB → qwen3.5:9b
32GB+ → qwen3.5:27b
Closes#132 — jarvis init now recommends a Qwen3 dense model based on
available memory using an explicit tier table:
≤8 GB → qwen3:1.7b (~1.1 GB)
8-16GB → qwen3:4b (~2.5 GB)
16-32GB → qwen3:8b (~5.2 GB)
32GB+ → qwen3:14b (~9.3 GB)
Adds qwen3:0.6b, 1.7b, 4b, 14b, 30b to the model catalog. Falls back
to scanning all compatible models if the tiered pick doesn't support
the selected engine. Shows the user why the model was chosen and how
to download later if declined.
Closes#164 — Qwen and other models with baked-in corporate identities
would claim to be cloud services when running locally. Adds a
configurable default_system_prompt to AgentConfig that is injected as a
fallback when no other system prompt is provided. The default tells the
model it is running locally through OpenJarvis. Users can override or
clear it via config.toml.
Tests were storing digests with local time but get_today() filters
by UTC date, causing failures when local date != UTC date.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(evals): add tool_choice=auto + fix traces thread safety
Two fixes for eval accuracy and stability:
1. TauBench agent: add tool_choice="auto" to match tau2's native
LLMAgent behavior. Without this, Qwen and GPT-5.4 score 10-14pp
below leaderboard because the models don't receive explicit
tool-calling guidance.
2. SystemBuilder: apply self._traces flag to config.traces.enabled.
Previously builder.traces(False) was a no-op — traces stayed
enabled, creating SQLite connections in the main thread that
crashed when accessed from ThreadPoolExecutor worker threads
in GAIA evals ("SQLite objects created in a thread can only be
used in that same thread").
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat: enrich inference events with model response content and add Trace.messages
Add content, tool_calls, and finish_reason fields to INFERENCE_END events
published by InstrumentedEngine. For non-instrumented engines, BaseAgent._generate()
now publishes INFERENCE_START/END events with the same rich data. Add _message_to_dict
helper and messages field to Trace dataclass for full conversation capture.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat: enhance TraceCollector with rich content, tool details, and messages
Capture model response content, tool_calls, and finish_reason in GENERATE
steps; store tool arguments and result text in TOOL_CALL steps; extract
conversation messages from AgentResult.metadata into Trace.messages; and
implement the last_trace property. Also adds a messages column to the
TraceStore schema so messages survive the SQLite round-trip.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat: agents store conversation messages in AgentResult.metadata
Both NativeReActAgent and MonitorOperativeAgent now serialize their
internal messages list via _message_to_dict and include it in the
returned AgentResult.metadata under the "messages" key. This enables
TraceCollector to capture full conversation traces.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat: wire TraceCollector into system._run_agent and JarvisAgentBackend
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat: persist rich trace data in eval trace JSONL files
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat: add TerminalBench config for Qwen 3.5-122B
* fix: add SQLite migration for traces.messages column on existing databases
* fix: check trace_store instead of shared config for trace enablement
* feat(evals): add ToolCall-15, LiveCodeBench, LiveResearchBench + telemetry
Three new benchmark integrations and full telemetry wiring for the
NeurIPS 2026 IPW/IPJ experiments.
## New Benchmarks
### ToolCall-15
15-scenario tool calling accuracy benchmark across 5 categories.
All scenarios defined inline with deterministic scoring (0/1/2 per
scenario). Fast to run (~5min/model) — ideal for optimization loops.
### LiveCodeBench
Competitive programming from LeetCode/AtCoder/CodeForces via
HuggingFace dataset. Sandboxed code execution with per-test
timeouts. Single-turn generation via jarvis-direct backend.
### LiveResearchBench
100 expert-curated deep research tasks. LLM-as-judge scoring
across 4 dimensions (comprehensiveness, insight, instruction
following, readability). Uses web_search tool for live research.
## Telemetry Wiring
- FLOPs estimation: 2 * active_params * total_tokens (MoE-aware)
- Energy/power capture flows from InstrumentedEngine through
backends to EvalResult and RunSummary
- New telemetry_summary section in output JSON with IPW/IPJ
- JarvisDirectBackend now propagates gpu_metrics flag
- TauBench forwards telemetry flags to SystemBuilder
## Experiment Plan
Added docs/experiments/neurips-2026-plan.md tracking the full
experiment matrix: 9 models x 7 benchmarks across NVIDIA, AMD,
and Apple hardware stacks.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Jon Saad-Falcon <jonsaadfalcon@gmail.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
test_config.py: SecurityConfig.mode default changed from "warn" to "redact"
test_config_phase3.py: ServerConfig.host default changed from "0.0.0.0" to "127.0.0.1"
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- BoundaryGuard degrades gracefully when Rust scanners unavailable
- BoundaryGuard tests use lightweight mock scanners (no Rust needed)
- Server-dependent tests use pytest.importorskip for starlette/fastapi
- Fix import ordering in test files
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
CI installs `--extra dev` but not `--extra server`, so fastapi is
unavailable. All other server tests use pytest.importorskip("fastapi")
to skip gracefully — this file was missing the guard.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- test_deep_research_prompt.py (8 tests): date injection, time, day of week,
dynamic generation, response types, tool descriptions, /no_think, Jarvis name
- test_slack_messaging.py (7 tests): connect with/without tokens, error state,
disconnect, handler registration, send without token
- test_connector_health.py (44 tests): all 11 connectors instantiate, have
required methods, report not-connected without creds, have metadata.
Plus KnowledgeStore data presence checks.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add exchange_google_token() and run_oauth_flow() to oauth.py for the
full authorization code exchange. Update gdrive, gcalendar, and
gcontacts connectors to trigger the browser-based OAuth flow when a
client_id:client_secret pair is provided, prefer access_token over raw
token, and require an actual access_token for is_connected(). auth_url()
now returns the Cloud Console credentials page when no client_id is
stored.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- personal_deep_research.toml: add schedule_value="" for template test compat
- test_deep_research_tools_wiring: skip when fastapi not installed in CI
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add deep_research_agent parameter to ChannelBridge.__init__ and update
_handle_chat to try DeepResearchAgent first, falling back to system.ask()
when no research agent is configured.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Resolve tool names from config["tools"] into actual tool instances
using ToolRegistry, inject runtime deps, and pass them to the agent
constructor instead of hardcoded tools=[].
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
