Introduce ResultTracker ABC with pluggable experiment tracking for the
eval framework, enabling per-sample streaming to W&B and summary-row
appending to Google Sheets — critical for the NeurIPS paper sweep.
- ResultTracker ABC: on_run_start/on_result/on_summary/on_run_end
- WandbTracker: per-sample wandb.log with sample/ prefix, flattened
MetricStats in run summary, reinit=True for suite mode
- SheetsTracker: summary-only (no per-sample API calls), 29-column
canonical row, idempotent header, service account + ADC auth
- EvalRunner: trackers wired into lifecycle, all calls try/except
wrapped so tracker failures never abort an eval run
- CLI: 7 new flags (--wandb-project/entity/tags/group,
--sheets-id/worksheet/creds) on both jarvis eval run and
python -m openjarvis.evals run
- Config: tracker fields parsed from TOML [run] section, propagated
through expand_suite() for suite mode
- pyproject.toml: eval-wandb and eval-sheets optional dependency groups
- 9 new tests (lifecycle, crash resilience, mock W&B/Sheets)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Adds openjarvis[speech] (faster-whisper) and openjarvis[speech-deepgram]
(deepgram-sdk) extras to pyproject.toml.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The codebase simplification PR consolidated per-engine files into
openai_compat_engines.py but docs still referenced the old module paths,
breaking the mkdocs build. Also adds pymessenger to channel-messenger
optional dependency.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Move the standalone evals framework from the project root into the
openjarvis package. Rewrite all ~50+ import statements from 'from evals.'
to 'from openjarvis.evals.' across the package, CLI, and tests. Remove
the evals-specific pyproject.toml (no longer a standalone package).
Update ruff per-file-ignores paths and fix line-length violations
introduced by the longer import paths.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Rewrite .github/workflows/desktop.yml: 2-job pipeline (validate + build-and-release)
with rolling desktop-latest pre-release on push to main and stable desktop-v* releases
- Add UpdateChecker component: checks for updates on startup + every 30 min,
background download with progress bar, one-click relaunch
- Configure Tauri updater: endpoints pointing to desktop-latest release, pubkey placeholder
- Add tauri-plugin-process for relaunch support (Cargo.toml, lib.rs, package.json)
- Add macOS Entitlements.plist for notarization (network + file access, no sandbox)
- Add scripts/bump-desktop-version.sh for atomic version bumps across 3 config files
- Add desktop/README.md with dev setup, auto-update architecture, signing docs
- Update .gitignore for desktop/node_modules, dist, target
- Configure macOS minimumSystemVersion, Windows timestampUrl
- Include all Phase 14-21 work: agent hardening, RBAC, taint tracking, workflows,
skills, knowledge graph, sessions, A2A, MCP templates, WASM sandbox, TUI dashboard,
production tools, CLI expansion, API expansion, learning productionization,
Tauri desktop app, and 10 new channels
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Pull shared boilerplate (event emission, message building, generation,
think-tag stripping) into BaseAgent concrete helpers and ToolUsingAgent
intermediate base class with tool-call loop
- Add NativeReActAgent and NativeOpenHandsAgent as clean implementations
built on the new base classes
- Simplify SimpleAgent, OrchestratorAgent, ReActAgent, OpenHandsAgent,
and RLMAgent to use inherited helpers instead of duplicated logic
- Remove CustomAgent (superseded by BaseAgent subclassing)
- Add backward-compat tests, base agent tests, native agent tests
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Evaluation framework (evals/): benchmarking system for measuring accuracy
across four categories — Chat (WildChat), Reasoning (SuperGPQA), RAG (FRAMES),
and Agentic (GAIA). Two backends: jarvis-direct (engine-level) and jarvis-agent
(agent-level with tool calling), both supporting local and cloud models.
Datasets adapted from IPW, scorers include exact match, LLM letter extraction,
and LLM-as-judge. Parallel execution via ThreadPoolExecutor with incremental
JSONL output. CLI: python -m evals {run,run-all,summarize,list}. 57 tests pass.
SVG fix: center logo content within viewBox by wrapping icon+text in a
translate(90,0) group, eliminating the left-shift visible in the README.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add URL detection, normalization, and direct fetching to web_search tool
(handles standalone URLs, embedded URLs, arxiv PDF→abs conversion)
- Add URL pre-expansion in OpenHands agent to bypass tool loop for URL queries
- Add input truncation (_truncate_if_needed) to prevent context window overflow
- Fix stream_bridge ToolResult field access (content/latency_seconds vs output/latency_ms)
- Add XML tool call format parser alongside Action:/Action Input: format
- Reduce default max_turns from 10→3 to cap worst-case response time
- Add empty response fallback text in frontend useChat hook
- Add channels system, security module, and model catalog updates
- Add 30+ new tests for URL fetching, truncation, and agent behavior
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Sets up a complete documentation website with 7 navigable sections (Home, Getting
Started, User Guide, Architecture, API Reference, Deployment, Development), light/dark
mode, search, code copy, and Mermaid diagram support. API reference pages use
mkdocstrings to auto-generate docs from source docstrings. GitHub Actions workflow
deploys to GitHub Pages on push to main.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add ReAct and OpenHands agents, WebSearch and CodeInterpreter tools,
full MCP protocol layer (server/client/transport), Gemini cloud engine
support, 12 new model specs (4 local MoE + 8 cloud), trace system,
and comprehensive test coverage across all dimensions (hardware, engine,
memory, agents, tools, MCP, integration).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
