* fix(evals): add tool_choice=auto + fix traces thread safety
Two fixes for eval accuracy and stability:
1. TauBench agent: add tool_choice="auto" to match tau2's native
LLMAgent behavior. Without this, Qwen and GPT-5.4 score 10-14pp
below leaderboard because the models don't receive explicit
tool-calling guidance.
2. SystemBuilder: apply self._traces flag to config.traces.enabled.
Previously builder.traces(False) was a no-op — traces stayed
enabled, creating SQLite connections in the main thread that
crashed when accessed from ThreadPoolExecutor worker threads
in GAIA evals ("SQLite objects created in a thread can only be
used in that same thread").
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat: enrich inference events with model response content and add Trace.messages
Add content, tool_calls, and finish_reason fields to INFERENCE_END events
published by InstrumentedEngine. For non-instrumented engines, BaseAgent._generate()
now publishes INFERENCE_START/END events with the same rich data. Add _message_to_dict
helper and messages field to Trace dataclass for full conversation capture.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat: enhance TraceCollector with rich content, tool details, and messages
Capture model response content, tool_calls, and finish_reason in GENERATE
steps; store tool arguments and result text in TOOL_CALL steps; extract
conversation messages from AgentResult.metadata into Trace.messages; and
implement the last_trace property. Also adds a messages column to the
TraceStore schema so messages survive the SQLite round-trip.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat: agents store conversation messages in AgentResult.metadata
Both NativeReActAgent and MonitorOperativeAgent now serialize their
internal messages list via _message_to_dict and include it in the
returned AgentResult.metadata under the "messages" key. This enables
TraceCollector to capture full conversation traces.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat: wire TraceCollector into system._run_agent and JarvisAgentBackend
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat: persist rich trace data in eval trace JSONL files
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat: add TerminalBench config for Qwen 3.5-122B
* fix: add SQLite migration for traces.messages column on existing databases
* fix: check trace_store instead of shared config for trace enablement
* feat(evals): add ToolCall-15, LiveCodeBench, LiveResearchBench + telemetry
Three new benchmark integrations and full telemetry wiring for the
NeurIPS 2026 IPW/IPJ experiments.
## New Benchmarks
### ToolCall-15
15-scenario tool calling accuracy benchmark across 5 categories.
All scenarios defined inline with deterministic scoring (0/1/2 per
scenario). Fast to run (~5min/model) — ideal for optimization loops.
### LiveCodeBench
Competitive programming from LeetCode/AtCoder/CodeForces via
HuggingFace dataset. Sandboxed code execution with per-test
timeouts. Single-turn generation via jarvis-direct backend.
### LiveResearchBench
100 expert-curated deep research tasks. LLM-as-judge scoring
across 4 dimensions (comprehensiveness, insight, instruction
following, readability). Uses web_search tool for live research.
## Telemetry Wiring
- FLOPs estimation: 2 * active_params * total_tokens (MoE-aware)
- Energy/power capture flows from InstrumentedEngine through
backends to EvalResult and RunSummary
- New telemetry_summary section in output JSON with IPW/IPJ
- JarvisDirectBackend now propagates gpu_metrics flag
- TauBench forwards telemetry flags to SystemBuilder
## Experiment Plan
Added docs/experiments/neurips-2026-plan.md tracking the full
experiment matrix: 9 models x 7 benchmarks across NVIDIA, AMD,
and Apple hardware stacks.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Jon Saad-Falcon <jonsaadfalcon@gmail.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Closes#158 — the README and installation docs assumed uv was already
installed without explaining how to get it, creating a barrier for new
users (especially on macOS). Adds a prerequisites table to both the
README and docs/getting-started/installation.md with platform-specific
install commands, and links to the existing macOS step-by-step guide.
- Skip SetupWizard on launch, go straight to Chat page
- Add Data Sources page with sidebar nav (separate tabs for data sources + messaging channels)
- Add "Connect your data" banner + quick-action buttons on Chat empty state
- Add hint on deep research agent pointing to Data Sources + Messaging tabs
- Consistent naming: "Data Sources" and "Messaging Channels" everywhere
- Fix Apple Notes / iMessage setup (remove broken system prefs link)
- Fix Slack data source: auto-join public channels, rate limit retry, is_member filter
- Add channels:join scope to all Slack manifests + docs
- Fix Gmail: use gmail_imap connector (IMAP + app password), increase limit to 5000
- Fix sync endpoint: run in background thread, return immediately
- Show sync progress with progress bar, error messages, Sync Now / Re-sync / Retry buttons
- Add triggerSync() API + SyncStatusDisplay component
- Rewrite all connector setup instructions with precise click-by-click steps
- Notion: share all pages at once via top-level page sharing
- Obsidian: show how to find vault path via Obsidian UI or Finder
- SendBlue: add link to API Credentials page, add ngrok webhook step
- Slack messaging: add Copy button for JSON manifest
- Add OpenJarvis Slack icon asset
- Shorten deep research template description
- Fix desktop app port (8222 → 8000 to match server default)
- Disable auto-updater for local dev builds
- Register Slack, Outlook, GCalendar connectors in __init__.py
- Auto-create default agent when setting up messaging channels
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Roadmap: updated messaging status (iMessage/SMS + Slack working,
WhatsApp Baileys blocked, Meta Cloud API planned).
Testing plan: 12 query types across 3 platforms (Interact, Slack,
iMessage/SMS) with execution phases and success criteria.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Added SendBlue (iMessage/SMS) and Slack messaging setup with detailed
step-by-step instructions, App Manifest JSON, webhook registration,
and troubleshooting tables. Reorganized doc into two sections:
Messaging Channels (talk to agent) and Data Connectors (search data).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Register sendblue in channels/__init__.py so ChannelRegistry works
- Auto-restore SendBlue bindings on server startup from database,
re-creating ChannelBridge + DeepResearchAgent so webhooks survive
server restarts
- Add sendblue to CLI channel_cmd.py (_get_channel, help text) and
SystemBuilder._resolve_channel() for config.toml support
- Health check endpoint GET /v1/channels/sendblue/health returns
channel_connected, bridge_wired, ready status
- Frontend: SendBlueWizard checks health on mount, shows
"Disconnected" badge with "Reconnect" button when bridge is dead
- Docs: CLI usage, server restart behavior, ngrok re-registration,
troubleshooting table, health check endpoint
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Covers both data connector (read messages) and messaging channel (DM the agent).
Includes the full App Manifest JSON, required scopes table, and all the gotchas
we discovered (Request URL, reinstall requirement, App Token vs Bot Token, etc.).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Covers setup instructions and troubleshooting for all 12 connectors:
Gmail, Google Drive, Calendar, Contacts, Slack, Notion, Granola,
Apple Notes, iMessage, Outlook, Obsidian, Dropbox.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Replace the recompute approach from #146 with per-token outlier
detection. Entries whose energy, FLOPs, or dollar savings exceed
generous per-token thresholds (~1000x legitimate values) are hidden
from the leaderboard entirely. All displayed values come directly
from the database — no rewriting of submitted data.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Energy and FLOPs are now derived from total_tokens on the leaderboard
display using Claude Opus 4.6 constants, rather than trusting submitted
DB values. This prevents gaming (e.g. TotallyNoire submitting 14B Wh
from 15M tokens). Dollar savings are clamped at the theoretical max
($25/1M tokens) on both the leaderboard and frontend submission side.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Comprehensive step-by-step guide covering Homebrew, uv, Rust, llama.cpp,
model download, Python 3.12 pin (PyO3 compat), and common pitfalls.
Cherry-picked from PR #131 by @gridworks — cleaned up to include only
the docs content (removed duplicate files, binary artifacts, and
unrelated lockfile changes from the original PR).
Co-Authored-By: gridworks <5502067+gridworks@users.noreply.github.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Dollar savings were previously summed across all three cloud providers
(GPT-5.3 + Claude Opus 4.6 + Gemini 3.1 Pro), inflating the reported
number by ~3x. Now uses only Claude Opus 4.6 pricing as the baseline,
with an asterisk footnote on the leaderboard explaining this.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Merge main into fix/ssrf-check, keeping both the auto-recover
logic for error-state agents and the async streaming support
for the send_message endpoint.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
3-task plan: wire since param for incremental sync, content-addressed
AttachmentStore with SHA-256 dedup, POST /sync trigger endpoint.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Exposes /v1/connectors/* REST API for desktop wizard and CLI:
list, detail, connect, disconnect, sync status.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
3-task plan: query classifier, ChannelAgent with thread pool and
escalation links, integration test with real KnowledgeStore.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add `stream: bool` parameter to `POST /v1/managed-agents/{id}/messages`.
When `stream=true`, the agent processes the message synchronously and
returns an SSE stream (OpenAI-compatible format) with token-by-token
response, tool result events, and usage metadata.
This enables real-time voice assistants and chat UIs to receive agent
responses as they are generated, rather than polling for completion.
- Extend SendMessageRequest with `stream` field (default: false)
- Add _stream_managed_agent() helper using asyncio.to_thread()
- Build AgentContext from conversation history for multi-turn support
- Emit tool_results as named SSE events
- Persist agent response in DB after streaming completes
- Add 6 new tests covering streaming behavior
- Update agents.md documentation with streaming examples
The energy_wh_saved and flops_saved values were orders of magnitude too
high because a scaling factor that grows linearly with N was applied to
the energy calculation, making it scale as O(N³) instead of O(N²).
Replace the buggy scale-factor approach with a direct FLOP-to-energy
conversion using each provider's per-token constants. Add regression
tests to prevent recurrence.
Closes#95
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat: add MiniMax as cloud inference provider
Add MiniMax M2.5 and M2.5-highspeed as a 5th cloud provider alongside
OpenAI, Anthropic, Google, and OpenRouter. Uses the OpenAI-compatible
API at api.minimax.io/v1 via the existing openai SDK dependency.
Changes:
- Add MiniMax client init, generate, and streaming in CloudEngine
- Add MiniMax models to model catalog with correct pricing
- Add MINIMAX_API_KEY environment variable support
- Add temperature clamping (0.01-1.0) per MiniMax API constraints
- Add 19 unit tests and 3 integration tests
- Update docs and README with MiniMax provider info
* feat: upgrade MiniMax default model to M2.7
- Add MiniMax-M2.7 and MiniMax-M2.7-highspeed to model list
- Set MiniMax-M2.7 as default model (first in list)
- Keep all previous models (M2.5, M2.5-highspeed) as alternatives
- Update pricing table with M2.7 entries
- Update model catalog with M2.7 specs
- Update docs to list all available MiniMax models
- Update unit tests (23 passing) and integration tests (3 passing)
---------
Co-authored-by: Octopus <octo-patch@users.noreply.github.com>
- Move "Current Focus Areas" and "How to Get Involved" to top of roadmap
- Remove Version History section entirely
- Fix "take" workflow: clarify that claiming happens on GitHub issues,
with link to create new issues if none exists
- Remove Contributing Guide from MkDocs nav (linked from roadmap instead)
- Link to CONTRIBUTING.md on GitHub from "How to Get Involved" section
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Reorganize roadmap around the 5 workstreams: Continuous Operators,
Mobile & Messaging, Secure Cloud Collaboration, Tutorials, Hardware
- Add concrete "Where you can help" tables with maturity tags and
good-first-issue markers under each workstream
- Change "GRPO training from trace data" to "Post-training data"
- Fix 404 link: CONTRIBUTING.md link now points to GitHub, not docs site
- Remove !!! tip admonition from docs/development/contributing.md
- Preserve version history in collapsible section
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
