Update test expectations across all test modules to align with the
Rust (PyO3) backend that replaced Python implementations:
- calculator: meval uses ^/ln/floor(), returns floats, inf for 1/0
- shell_exec: output format uses "Exit code: N\n--- stdout ---\n..."
- git_tool: always --oneline, mock Rust bridge for git-not-found
- http_request: force httpx fallback so respx mocks work
- memory: UUID is 36 chars (hyphenated), BM25Memory() takes no args
- loop_guard: Rust blocks on 2nd identical call, capitalized messages
- ssrf: Rust returns "Invalid URL" for unparseable URLs
- mcp: division by zero returns inf (success), not error
- integration: calculator results are float strings ("4.0" not "4")
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Move 12 optimize-related test files from tests/ root into tests/learning/
to match the module reorganization. All 305 tests pass from new locations.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Fix all ruff E501 line-length violations
- Add noqa: E402 for intentional late imports in runner scripts
- Remove unused imports in tb_agent.py
- Update test_llm_optimizer assertions to expect fixed params in
unparseable/missing-params cases (intended behavior change)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Keep both system_prompt (optimization) and episode_mode (Phase 25) in
RunConfig. Merge terminalbench_native metadata using v2 task_data dict
API while preserving new create_task_env/verify_requirements methods.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Remove duplicate config structs in Rust config.rs and Python config.py
- Remove duplicate match arms in Rust discovery.rs
- Fix E501 line-too-long in evals/cli.py
- Fix E402 imports-not-at-top in test_ama_bench.py and test_loghub.py
- Fix E501 long lines in test_loghub.py
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add four new OpenAI-compatible inference engines across the full stack:
Rust backend:
- Config structs with serde defaults (ExoEngineConfig, NexaEngineConfig,
UzuEngineConfig, AppleFmEngineConfig) in openjarvis-core
- Factory constructors on OpenAICompatEngine (exo, nexa, uzu, apple_fm)
- Engine enum variants with delegate_engine! macro dispatch
- Discovery and resolution in get_engine_static()
- PyO3 bridge support in PyEngine
Python frontend:
- Data-driven engine class registration in openai_compat_engines.py
- Config dataclasses with backward-compat host properties
- Discovery host map entries
- DEFAULT_SEARCH_SPACE updated with new engine backends
- Apple FM shim (FastAPI wrapper for python-apple-fm-sdk, macOS only)
Config & tests:
- TOML config sections for all four engines
- Unit tests for factory methods, enum variants, and search space
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Remove unused imports (os, pytest) in test_rust_bridge.py
- Add pytest.importorskip for fastapi/starlette in test_learning_api.py
- Fix AppleEnergyMonitor test fixtures: set _zeus_ok, _chip_name,
_tdp_watts attributes when using __new__ to skip __init__
- Fix uninitialized monitor test to match CPU-time fallback behavior
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
str(Role.SYSTEM) returns 'Role.SYSTEM' on Python 3.10, not 'system'.
Use getattr(m, 'role', None) == 'system' which works because Role
inherits from str. Also wires compress_context into operative agent.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Surface per-instance scores (SampleScore), structured feedback
(TrialFeedback), Pareto frontier tracking, pillar-targeted mutation,
and config merge strategies. Store migration adds new columns with
backward-compatible deserialization.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add the orchestration layer that ties together the LLM optimizer, trial
runner, and persistence into a propose-evaluate-analyze loop with early
stopping and recipe export. 36 new tests pass.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Implement LLMOptimizer that uses a cloud LLM to propose and analyze
OpenJarvis configurations, inspired by DSPy's GEPA approach with
textual trace feedback rather than just scalar rewards. 46 tests pass.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add TrialRunner to bridge TrialConfig to the eval framework (EvalRunner),
TraceJudge for LLM-as-judge scoring of agent traces, and FeedbackCollector
for aggregating explicit, thumbs, and judge-driven feedback signals.
68 tests pass covering all three modules.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Synthesize reusable benchmarks from interaction traces with feedback
scores. Mines high-quality traces, groups by query class, picks the
best reference per class, and exposes results through DatasetProvider
and LLM-judge Scorer so EvalRunner can evaluate against personal
workflow patterns.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add the optimize module with core data types (SearchDimension, SearchSpace,
TrialConfig, TrialResult, OptimizationRun) and search space builder
(build_search_space, DEFAULT_SEARCH_SPACE) covering all 5 pillars. Includes
TrialConfig.to_recipe() mapping and SearchSpace.to_prompt_description()
for LLM-readable rendering. 66 tests pass.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Introduce ResultTracker ABC with pluggable experiment tracking for the
eval framework, enabling per-sample streaming to W&B and summary-row
appending to Google Sheets — critical for the NeurIPS paper sweep.
- ResultTracker ABC: on_run_start/on_result/on_summary/on_run_end
- WandbTracker: per-sample wandb.log with sample/ prefix, flattened
MetricStats in run summary, reinit=True for suite mode
- SheetsTracker: summary-only (no per-sample API calls), 29-column
canonical row, idempotent header, service account + ADC auth
- EvalRunner: trackers wired into lifecycle, all calls try/except
wrapped so tracker failures never abort an eval run
- CLI: 7 new flags (--wandb-project/entity/tags/group,
--sheets-id/worksheet/creds) on both jarvis eval run and
python -m openjarvis.evals run
- Config: tracker fields parsed from TOML [run] section, propagated
through expand_suite() for suite mode
- pyproject.toml: eval-wandb and eval-sheets optional dependency groups
- 9 new tests (lifecycle, crash resilience, mock W&B/Sheets)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Define the abstract interface for the speech subsystem: SpeechBackend ABC
with transcribe/health/supported_formats methods, TranscriptionResult and
Segment dataclasses for structured transcription output.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add SpeechConfig dataclass with backend, model, language, device, and
compute_type fields. Wire it into JarvisConfig and load_config() so
[speech] TOML sections are loaded automatically.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace vllm.py, sglang.py, llamacpp.py, mlx.py, lmstudio.py with
openai_compat_engines.py using dynamic class creation and registration.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Delete the entire src/openjarvis/memory/ directory (11 files) which
was a pure re-export shim pointing to openjarvis.tools.storage.*.
Update all imports across src/ and tests/ to use the canonical
openjarvis.tools.storage imports directly.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Move the standalone evals framework from the project root into the
openjarvis package. Rewrite all ~50+ import statements from 'from evals.'
to 'from openjarvis.evals.' across the package, CLI, and tests. Remove
the evals-specific pyproject.toml (no longer a standalone package).
Update ruff per-file-ignores paths and fix line-length violations
introduced by the longer import paths.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add std_latency to LatencyBenchmark metrics. New _render_stats_table
in bench_cmd.py detects stats-pattern keys (mean_X, p50_X, min_X,
max_X, std_X, p95_X) and renders Avg/Median/Min/Max/Std/P95 columns.
Falls back to simple key-value for non-stats metrics.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
New log_config.py with setup_logging() configuring the openjarvis
logger (WARNING default, DEBUG on --verbose, ERROR on --quiet).
RotatingFileHandler (5MB, 3 backups) enabled in verbose mode.
Flags added to root CLI group and forwarded via click context.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
New hints.py with hint_no_config(), hint_no_engine(), hint_no_model().
Wire hint_no_engine into ask.py EngineConnectionError handlers to
show actionable suggestions when engine is unreachable.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
5-step guided setup: detect hardware, write config, check engine,
verify model, run test query. Skips config step if already present
unless --force is used. Exits with helpful message on engine failure.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
