* fix(evals): add tool_choice=auto + fix traces thread safety
Two fixes for eval accuracy and stability:
1. TauBench agent: add tool_choice="auto" to match tau2's native
LLMAgent behavior. Without this, Qwen and GPT-5.4 score 10-14pp
below leaderboard because the models don't receive explicit
tool-calling guidance.
2. SystemBuilder: apply self._traces flag to config.traces.enabled.
Previously builder.traces(False) was a no-op — traces stayed
enabled, creating SQLite connections in the main thread that
crashed when accessed from ThreadPoolExecutor worker threads
in GAIA evals ("SQLite objects created in a thread can only be
used in that same thread").
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat: enrich inference events with model response content and add Trace.messages
Add content, tool_calls, and finish_reason fields to INFERENCE_END events
published by InstrumentedEngine. For non-instrumented engines, BaseAgent._generate()
now publishes INFERENCE_START/END events with the same rich data. Add _message_to_dict
helper and messages field to Trace dataclass for full conversation capture.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat: enhance TraceCollector with rich content, tool details, and messages
Capture model response content, tool_calls, and finish_reason in GENERATE
steps; store tool arguments and result text in TOOL_CALL steps; extract
conversation messages from AgentResult.metadata into Trace.messages; and
implement the last_trace property. Also adds a messages column to the
TraceStore schema so messages survive the SQLite round-trip.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat: agents store conversation messages in AgentResult.metadata
Both NativeReActAgent and MonitorOperativeAgent now serialize their
internal messages list via _message_to_dict and include it in the
returned AgentResult.metadata under the "messages" key. This enables
TraceCollector to capture full conversation traces.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat: wire TraceCollector into system._run_agent and JarvisAgentBackend
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat: persist rich trace data in eval trace JSONL files
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat: add TerminalBench config for Qwen 3.5-122B
* fix: add SQLite migration for traces.messages column on existing databases
* fix: check trace_store instead of shared config for trace enablement
* feat(evals): add ToolCall-15, LiveCodeBench, LiveResearchBench + telemetry
Three new benchmark integrations and full telemetry wiring for the
NeurIPS 2026 IPW/IPJ experiments.
## New Benchmarks
### ToolCall-15
15-scenario tool calling accuracy benchmark across 5 categories.
All scenarios defined inline with deterministic scoring (0/1/2 per
scenario). Fast to run (~5min/model) — ideal for optimization loops.
### LiveCodeBench
Competitive programming from LeetCode/AtCoder/CodeForces via
HuggingFace dataset. Sandboxed code execution with per-test
timeouts. Single-turn generation via jarvis-direct backend.
### LiveResearchBench
100 expert-curated deep research tasks. LLM-as-judge scoring
across 4 dimensions (comprehensiveness, insight, instruction
following, readability). Uses web_search tool for live research.
## Telemetry Wiring
- FLOPs estimation: 2 * active_params * total_tokens (MoE-aware)
- Energy/power capture flows from InstrumentedEngine through
backends to EvalResult and RunSummary
- New telemetry_summary section in output JSON with IPW/IPJ
- JarvisDirectBackend now propagates gpu_metrics flag
- TauBench forwards telemetry flags to SystemBuilder
## Experiment Plan
Added docs/experiments/neurips-2026-plan.md tracking the full
experiment matrix: 9 models x 7 benchmarks across NVIDIA, AMD,
and Apple hardware stacks.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Jon Saad-Falcon <jonsaadfalcon@gmail.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
test_config.py: SecurityConfig.mode default changed from "warn" to "redact"
test_config_phase3.py: ServerConfig.host default changed from "0.0.0.0" to "127.0.0.1"
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- BoundaryGuard degrades gracefully when Rust scanners unavailable
- BoundaryGuard tests use lightweight mock scanners (no Rust needed)
- Server-dependent tests use pytest.importorskip for starlette/fastapi
- Fix import ordering in test files
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
CI installs `--extra dev` but not `--extra server`, so fastapi is
unavailable. All other server tests use pytest.importorskip("fastapi")
to skip gracefully — this file was missing the guard.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- test_deep_research_prompt.py (8 tests): date injection, time, day of week,
dynamic generation, response types, tool descriptions, /no_think, Jarvis name
- test_slack_messaging.py (7 tests): connect with/without tokens, error state,
disconnect, handler registration, send without token
- test_connector_health.py (44 tests): all 11 connectors instantiate, have
required methods, report not-connected without creds, have metadata.
Plus KnowledgeStore data presence checks.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add exchange_google_token() and run_oauth_flow() to oauth.py for the
full authorization code exchange. Update gdrive, gcalendar, and
gcontacts connectors to trigger the browser-based OAuth flow when a
client_id:client_secret pair is provided, prefer access_token over raw
token, and require an actual access_token for is_connected(). auth_url()
now returns the Cloud Console credentials page when no client_id is
stored.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- personal_deep_research.toml: add schedule_value="" for template test compat
- test_deep_research_tools_wiring: skip when fastapi not installed in CI
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add deep_research_agent parameter to ChannelBridge.__init__ and update
_handle_chat to try DeepResearchAgent first, falling back to system.ask()
when no research agent is configured.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Resolve tool names from config["tools"] into actual tool instances
using ToolRegistry, inject runtime deps, and pass them to the agent
constructor instead of hardcoded tools=[].
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Adds `codex/` prefixed model support using the OpenAI Responses API —
the same protocol used by zeroclaw and other Codex-compatible tools.
Live-tested against gpt-5-mini-2025-08-07 with:
- Generate (non-streaming): confirmed working
- System prompt → instructions mapping: confirmed working
- SSE streaming: confirmed working (9 chunks)
- End-to-end via `jarvis ask`: confirmed working
Implementation:
- Default endpoint: api.openai.com/v1/responses (standard API key)
- Override via OPENAI_CODEX_BASE_URL for ChatGPT OAuth tokens
(e.g. chatgpt.com/backend-api/codex)
- Auth via OPENAI_CODEX_API_KEY env var
- Responses API format: input array, instructions field, output_text extraction
- Handles reasoning+message output blocks correctly
- SSE streaming parses response.output_text.delta events
Models: codex/gpt-4o, codex/gpt-4o-mini, codex/o3-mini,
codex/gpt-5-mini, codex/gpt-5-mini-2025-08-07
Usage:
export OPENAI_CODEX_API_KEY="your-api-key-or-oauth-token"
jarvis ask "Hello" --model codex/gpt-5-mini-2025-08-07
Closes#134
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Incorporates the useful new features from PR #135 (by @gridworks) on top
of the existing PrivacyScanner implementation:
- Add DNS configuration check (macOS, via scutil --dns)
- Add --json flag to `jarvis scan` for machine-readable output
- Add --no-scan flag to `jarvis init` to skip the post-init audit
- Expand remote-access process list (ngrok, tailscaled, cloudflared, ZeroTier)
- Upgrade `jarvis scan` output from plain text to Rich table
- Add GET /v1/security/scan API endpoint
- Add tests for all new features (30 tests, all passing)
Closes#133
Co-Authored-By: gridworks <5502067+gridworks@users.noreply.github.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
