Introduce ResultTracker ABC with pluggable experiment tracking for the
eval framework, enabling per-sample streaming to W&B and summary-row
appending to Google Sheets — critical for the NeurIPS paper sweep.
- ResultTracker ABC: on_run_start/on_result/on_summary/on_run_end
- WandbTracker: per-sample wandb.log with sample/ prefix, flattened
MetricStats in run summary, reinit=True for suite mode
- SheetsTracker: summary-only (no per-sample API calls), 29-column
canonical row, idempotent header, service account + ADC auth
- EvalRunner: trackers wired into lifecycle, all calls try/except
wrapped so tracker failures never abort an eval run
- CLI: 7 new flags (--wandb-project/entity/tags/group,
--sheets-id/worksheet/creds) on both jarvis eval run and
python -m openjarvis.evals run
- Config: tracker fields parsed from TOML [run] section, propagated
through expand_suite() for suite mode
- pyproject.toml: eval-wandb and eval-sheets optional dependency groups
- 9 new tests (lifecycle, crash resilience, mock W&B/Sheets)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The transcribe_audio command uses reqwest::multipart which requires
the "multipart" feature flag on the reqwest crate.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Updates test count to ~3295, adds Speech subsystem section, API
endpoints, Phase 24 row, and speech extras to the docs.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Adds openjarvis[speech] (faster-whisper) and openjarvis[speech-deepgram]
(deepgram-sdk) extras to pyproject.toml.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Proxies multipart audio upload and health check to the Python backend
for the desktop app's speech-to-text integration.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Adds transcribeAudio() and fetchSpeechHealth() to the API client,
and the useSpeech React hook for managing mic recording lifecycle.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Define the abstract interface for the speech subsystem: SpeechBackend ABC
with transcribe/health/supported_formats methods, TranscriptionResult and
Segment dataclasses for structured transcription output.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add SpeechConfig dataclass with backend, model, language, device, and
compute_type fields. Wire it into JarvisConfig and load_config() so
[speech] TOML sections are loaded automatically.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
New Speech subsystem with SpeechBackend ABC, registry-based backends
(Faster-Whisper, OpenAI, Deepgram), API endpoints, and frontend mic
button for desktop app and browser voice input.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The codebase simplification PR consolidated per-engine files into
openai_compat_engines.py but docs still referenced the old module paths,
breaking the mkdocs build. Also adds pymessenger to channel-messenger
optional dependency.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace vllm.py, sglang.py, llamacpp.py, mlx.py, lmstudio.py with
openai_compat_engines.py using dynamic class creation and registration.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Delete the entire src/openjarvis/memory/ directory (11 files) which
was a pure re-export shim pointing to openjarvis.tools.storage.*.
Update all imports across src/ and tests/ to use the canonical
openjarvis.tools.storage imports directly.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Fix mock.patch() target in test_config.py (evals.cli -> openjarvis.evals.cli)
- Remove stale sys.path manipulation in evals/tests/conftest.py
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Move the standalone evals framework from the project root into the
openjarvis package. Rewrite all ~50+ import statements from 'from evals.'
to 'from openjarvis.evals.' across the package, CLI, and tests. Remove
the evals-specific pyproject.toml (no longer a standalone package).
Update ruff per-file-ignores paths and fix line-length violations
introduced by the longer import paths.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
