Keep both system_prompt (optimization) and episode_mode (Phase 25) in
RunConfig. Merge terminalbench_native metadata using v2 task_data dict
API while preserving new create_task_env/verify_requirements methods.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Remove duplicate config structs in Rust config.rs and Python config.py
- Remove duplicate match arms in Rust discovery.rs
- Fix E501 line-too-long in evals/cli.py
- Fix E402 imports-not-at-top in test_ama_bench.py and test_loghub.py
- Fix E501 long lines in test_loghub.py
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add four new OpenAI-compatible inference engines across the full stack:
Rust backend:
- Config structs with serde defaults (ExoEngineConfig, NexaEngineConfig,
UzuEngineConfig, AppleFmEngineConfig) in openjarvis-core
- Factory constructors on OpenAICompatEngine (exo, nexa, uzu, apple_fm)
- Engine enum variants with delegate_engine! macro dispatch
- Discovery and resolution in get_engine_static()
- PyO3 bridge support in PyEngine
Python frontend:
- Data-driven engine class registration in openai_compat_engines.py
- Config dataclasses with backward-compat host properties
- Discovery host map entries
- DEFAULT_SEARCH_SPACE updated with new engine backends
- Apple FM shim (FastAPI wrapper for python-apple-fm-sdk, macOS only)
Config & tests:
- TOML config sections for all four engines
- Unit tests for factory methods, enum variants, and search space
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
str(Role.SYSTEM) returns 'Role.SYSTEM' on Python 3.10, not 'system'.
Use getattr(m, 'role', None) == 'system' which works because Role
inherits from str. Also wires compress_context into operative agent.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Surface per-instance scores (SampleScore), structured feedback
(TrialFeedback), Pareto frontier tracking, pillar-targeted mutation,
and config merge strategies. Store migration adds new columns with
backward-compatible deserialization.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add the orchestration layer that ties together the LLM optimizer, trial
runner, and persistence into a propose-evaluate-analyze loop with early
stopping and recipe export. 36 new tests pass.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Implement LLMOptimizer that uses a cloud LLM to propose and analyze
OpenJarvis configurations, inspired by DSPy's GEPA approach with
textual trace feedback rather than just scalar rewards. 46 tests pass.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add TrialRunner to bridge TrialConfig to the eval framework (EvalRunner),
TraceJudge for LLM-as-judge scoring of agent traces, and FeedbackCollector
for aggregating explicit, thumbs, and judge-driven feedback signals.
68 tests pass covering all three modules.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Synthesize reusable benchmarks from interaction traces with feedback
scores. Mines high-quality traces, groups by query class, picks the
best reference per class, and exposes results through DatasetProvider
and LLM-judge Scorer so EvalRunner can evaluate against personal
workflow patterns.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add the optimize module with core data types (SearchDimension, SearchSpace,
TrialConfig, TrialResult, OptimizationRun) and search space builder
(build_search_space, DEFAULT_SEARCH_SPACE) covering all 5 pillars. Includes
TrialConfig.to_recipe() mapping and SearchSpace.to_prompt_description()
for LLM-readable rendering. 66 tests pass.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Introduce ResultTracker ABC with pluggable experiment tracking for the
eval framework, enabling per-sample streaming to W&B and summary-row
appending to Google Sheets — critical for the NeurIPS paper sweep.
- ResultTracker ABC: on_run_start/on_result/on_summary/on_run_end
- WandbTracker: per-sample wandb.log with sample/ prefix, flattened
MetricStats in run summary, reinit=True for suite mode
- SheetsTracker: summary-only (no per-sample API calls), 29-column
canonical row, idempotent header, service account + ADC auth
- EvalRunner: trackers wired into lifecycle, all calls try/except
wrapped so tracker failures never abort an eval run
- CLI: 7 new flags (--wandb-project/entity/tags/group,
--sheets-id/worksheet/creds) on both jarvis eval run and
python -m openjarvis.evals run
- Config: tracker fields parsed from TOML [run] section, propagated
through expand_suite() for suite mode
- pyproject.toml: eval-wandb and eval-sheets optional dependency groups
- 9 new tests (lifecycle, crash resilience, mock W&B/Sheets)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Define the abstract interface for the speech subsystem: SpeechBackend ABC
with transcribe/health/supported_formats methods, TranscriptionResult and
Segment dataclasses for structured transcription output.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add SpeechConfig dataclass with backend, model, language, device, and
compute_type fields. Wire it into JarvisConfig and load_config() so
[speech] TOML sections are loaded automatically.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace vllm.py, sglang.py, llamacpp.py, mlx.py, lmstudio.py with
openai_compat_engines.py using dynamic class creation and registration.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Delete the entire src/openjarvis/memory/ directory (11 files) which
was a pure re-export shim pointing to openjarvis.tools.storage.*.
Update all imports across src/ and tests/ to use the canonical
openjarvis.tools.storage imports directly.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Fix mock.patch() target in test_config.py (evals.cli -> openjarvis.evals.cli)
- Remove stale sys.path manipulation in evals/tests/conftest.py
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Move the standalone evals framework from the project root into the
openjarvis package. Rewrite all ~50+ import statements from 'from evals.'
to 'from openjarvis.evals.' across the package, CLI, and tests. Remove
the evals-specific pyproject.toml (no longer a standalone package).
Update ruff per-file-ignores paths and fix line-length violations
introduced by the longer import paths.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-locate recipe TOML files with the loader code that reads them,
eliminating the fragile parents[3] path traversal. Updates
_PROJECT_RECIPES_DIR to use a relative package data path instead.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
