* fix(evals): add tool_choice=auto + fix traces thread safety
Two fixes for eval accuracy and stability:
1. TauBench agent: add tool_choice="auto" to match tau2's native
LLMAgent behavior. Without this, Qwen and GPT-5.4 score 10-14pp
below leaderboard because the models don't receive explicit
tool-calling guidance.
2. SystemBuilder: apply self._traces flag to config.traces.enabled.
Previously builder.traces(False) was a no-op — traces stayed
enabled, creating SQLite connections in the main thread that
crashed when accessed from ThreadPoolExecutor worker threads
in GAIA evals ("SQLite objects created in a thread can only be
used in that same thread").
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat: enrich inference events with model response content and add Trace.messages
Add content, tool_calls, and finish_reason fields to INFERENCE_END events
published by InstrumentedEngine. For non-instrumented engines, BaseAgent._generate()
now publishes INFERENCE_START/END events with the same rich data. Add _message_to_dict
helper and messages field to Trace dataclass for full conversation capture.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat: enhance TraceCollector with rich content, tool details, and messages
Capture model response content, tool_calls, and finish_reason in GENERATE
steps; store tool arguments and result text in TOOL_CALL steps; extract
conversation messages from AgentResult.metadata into Trace.messages; and
implement the last_trace property. Also adds a messages column to the
TraceStore schema so messages survive the SQLite round-trip.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat: agents store conversation messages in AgentResult.metadata
Both NativeReActAgent and MonitorOperativeAgent now serialize their
internal messages list via _message_to_dict and include it in the
returned AgentResult.metadata under the "messages" key. This enables
TraceCollector to capture full conversation traces.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat: wire TraceCollector into system._run_agent and JarvisAgentBackend
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat: persist rich trace data in eval trace JSONL files
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat: add TerminalBench config for Qwen 3.5-122B
* fix: add SQLite migration for traces.messages column on existing databases
* fix: check trace_store instead of shared config for trace enablement
* feat(evals): add ToolCall-15, LiveCodeBench, LiveResearchBench + telemetry
Three new benchmark integrations and full telemetry wiring for the
NeurIPS 2026 IPW/IPJ experiments.
## New Benchmarks
### ToolCall-15
15-scenario tool calling accuracy benchmark across 5 categories.
All scenarios defined inline with deterministic scoring (0/1/2 per
scenario). Fast to run (~5min/model) — ideal for optimization loops.
### LiveCodeBench
Competitive programming from LeetCode/AtCoder/CodeForces via
HuggingFace dataset. Sandboxed code execution with per-test
timeouts. Single-turn generation via jarvis-direct backend.
### LiveResearchBench
100 expert-curated deep research tasks. LLM-as-judge scoring
across 4 dimensions (comprehensiveness, insight, instruction
following, readability). Uses web_search tool for live research.
## Telemetry Wiring
- FLOPs estimation: 2 * active_params * total_tokens (MoE-aware)
- Energy/power capture flows from InstrumentedEngine through
backends to EvalResult and RunSummary
- New telemetry_summary section in output JSON with IPW/IPJ
- JarvisDirectBackend now propagates gpu_metrics flag
- TauBench forwards telemetry flags to SystemBuilder
## Experiment Plan
Added docs/experiments/neurips-2026-plan.md tracking the full
experiment matrix: 9 models x 7 benchmarks across NVIDIA, AMD,
and Apple hardware stacks.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Jon Saad-Falcon <jonsaadfalcon@gmail.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Closes#158 — the README and installation docs assumed uv was already
installed without explaining how to get it, creating a barrier for new
users (especially on macOS). Adds a prerequisites table to both the
README and docs/getting-started/installation.md with platform-specific
install commands, and links to the existing macOS step-by-step guide.
Server returns template IDs with underscores (code_reviewer, research_monitor)
but TEMPLATE_INSTRUCTIONS used hyphens. Added both formats + prompts for
personal_deep_research and inbox_triager templates.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Adds a new Upload / Paste data source that lets users paste text or upload
documents (.txt, .md, .pdf, .docx, .csv) directly into the knowledge base.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Clarify that Gmail uses App Passwords (not OAuth login popup), emphasize
2-Step Verification prerequisite, and add a collapsible troubleshooting
section for common issues. Also improve the error message when Gmail
auth fails to specifically mention App Password requirements.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Move the Interact tab to the first position in agent detail view and
default to it when opening an agent. Add "(optional)" hint to Advanced
Settings and add (?) tooltips to Memory Extraction, Observation
Compression, Retrieval Strategy, and Task Decomposition labels.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add TEMPLATE_INSTRUCTIONS map with sensible defaults for daily-briefing,
research-monitor, code-reviewer, and meeting-prep templates so users get
a starting prompt when selecting a template. Show a warning hint when the
instruction contains [bracketed placeholders] that need to be replaced.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Replace the manual/interval/cron schedule picker with friendlier presets
(Daily, Weekly, Every N hours, Custom cron) that generate the correct
cron expressions or interval values behind the scenes. Update
formatSchedule() to render human-readable labels like "Daily at 9:00 AM"
and "Weekly on Mon, Wed, Fri at 9:00 AM".
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Frontend: add progress stages (Connecting → Authenticating → Connected →
Syncing) with spinner and progress bar to the data source connect flow.
Previously sources would silently stay "Not connected" after setup.
Show error message on failure instead of swallowing exceptions.
- Obsidian connector: use timezone-aware datetime (tz=timezone.utc) in
fromtimestamp() to fix "can't compare offset-naive and offset-aware
datetimes" crash during incremental sync.
- Google Calendar connector: catch HTTPStatusError when listing events
for individual calendars (e.g. US Holidays returning 404) so one
inaccessible calendar doesn't crash the entire sync.
- Agent SSE timeout: increase progress queue timeout from 120s to 600s
so complex multi-hop deep research queries aren't killed mid-execution
on slower local models.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Two fixes for eval accuracy and stability:
1. TauBench agent: add tool_choice="auto" to match tau2's native
LLMAgent behavior. Without this, Qwen and GPT-5.4 score 10-14pp
below leaderboard because the models don't receive explicit
tool-calling guidance.
2. SystemBuilder: apply self._traces flag to config.traces.enabled.
Previously builder.traces(False) was a no-op — traces stayed
enabled, creating SQLite connections in the main thread that
crashed when accessed from ThreadPoolExecutor worker threads
in GAIA evals ("SQLite objects created in a thread can only be
used in that same thread").
Co-authored-by: Jon Saad-Falcon <jonsaadfalcon@gmail.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Granola API rejects +00:00 timezone offset and bare ISO timestamps.
Use strftime with explicit Z suffix which the API accepts.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Remove gmail (REST API) from registered connectors, keep only gmail_imap
- Track background sync state at router level so status endpoint reports
"syncing" while background thread is alive, "error" with message on failure
- Fixes sync UI bouncing back to "Sync Now" immediately after clicking
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Skip SetupWizard on launch, go straight to Chat page
- Add Data Sources page with sidebar nav (separate tabs for data sources + messaging channels)
- Add "Connect your data" banner + quick-action buttons on Chat empty state
- Add hint on deep research agent pointing to Data Sources + Messaging tabs
- Consistent naming: "Data Sources" and "Messaging Channels" everywhere
- Fix Apple Notes / iMessage setup (remove broken system prefs link)
- Fix Slack data source: auto-join public channels, rate limit retry, is_member filter
- Add channels:join scope to all Slack manifests + docs
- Fix Gmail: use gmail_imap connector (IMAP + app password), increase limit to 5000
- Fix sync endpoint: run in background thread, return immediately
- Show sync progress with progress bar, error messages, Sync Now / Re-sync / Retry buttons
- Add triggerSync() API + SyncStatusDisplay component
- Rewrite all connector setup instructions with precise click-by-click steps
- Notion: share all pages at once via top-level page sharing
- Obsidian: show how to find vault path via Obsidian UI or Finder
- SendBlue: add link to API Credentials page, add ngrok webhook step
- Slack messaging: add Copy button for JSON manifest
- Add OpenJarvis Slack icon asset
- Shorten deep research template description
- Fix desktop app port (8222 → 8000 to match server default)
- Disable auto-updater for local dev builds
- Register Slack, Outlook, GCalendar connectors in __init__.py
- Auto-create default agent when setting up messaging channels
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
test_config.py: SecurityConfig.mode default changed from "warn" to "redact"
test_config_phase3.py: ServerConfig.host default changed from "0.0.0.0" to "127.0.0.1"
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- BoundaryGuard degrades gracefully when Rust scanners unavailable
- BoundaryGuard tests use lightweight mock scanners (no Rust needed)
- Server-dependent tests use pytest.importorskip for starlette/fastapi
- Fix import ordering in test files
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add check_bind_safety() to auth_middleware — refuses to bind non-loopback
without API key. Add cors_origins parameter to create_app() so CORS uses
config values instead of hardcoded wildcard "*".
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Detailed plan covering network exposure, boundary guard, webhook
validation, file permissions, log sanitization, credential scoping,
CORS hardening, and security profiles. Each task has failing tests
first, then implementation, then verification.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Tailwind CSS v4.2 (@tailwindcss/oxide) requires Node >= 20 for native
bindings. Without this, `npm install` succeeds but `vite` fails at
runtime with a missing native binding error.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Roadmap: updated messaging status (iMessage/SMS + Slack working,
WhatsApp Baileys blocked, Meta Cloud API planned).
Testing plan: 12 query types across 3 platforms (Interact, Slack,
iMessage/SMS) with execution phases and success criteria.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Added SendBlue (iMessage/SMS) and Slack messaging setup with detailed
step-by-step instructions, App Manifest JSON, webhook registration,
and troubleshooting tables. Reorganized doc into two sections:
Messaging Channels (talk to agent) and Data Connectors (search data).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
