Surface per-instance scores (SampleScore), structured feedback
(TrialFeedback), Pareto frontier tracking, pillar-targeted mutation,
and config merge strategies. Store migration adds new columns with
backward-compatible deserialization.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add the orchestration layer that ties together the LLM optimizer, trial
runner, and persistence into a propose-evaluate-analyze loop with early
stopping and recipe export. 36 new tests pass.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Implement LLMOptimizer that uses a cloud LLM to propose and analyze
OpenJarvis configurations, inspired by DSPy's GEPA approach with
textual trace feedback rather than just scalar rewards. 46 tests pass.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add TrialRunner to bridge TrialConfig to the eval framework (EvalRunner),
TraceJudge for LLM-as-judge scoring of agent traces, and FeedbackCollector
for aggregating explicit, thumbs, and judge-driven feedback signals.
68 tests pass covering all three modules.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Synthesize reusable benchmarks from interaction traces with feedback
scores. Mines high-quality traces, groups by query class, picks the
best reference per class, and exposes results through DatasetProvider
and LLM-judge Scorer so EvalRunner can evaluate against personal
workflow patterns.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add the optimize module with core data types (SearchDimension, SearchSpace,
TrialConfig, TrialResult, OptimizationRun) and search space builder
(build_search_space, DEFAULT_SEARCH_SPACE) covering all 5 pillars. Includes
TrialConfig.to_recipe() mapping and SearchSpace.to_prompt_description()
for LLM-readable rendering. 66 tests pass.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Introduce ResultTracker ABC with pluggable experiment tracking for the
eval framework, enabling per-sample streaming to W&B and summary-row
appending to Google Sheets — critical for the NeurIPS paper sweep.
- ResultTracker ABC: on_run_start/on_result/on_summary/on_run_end
- WandbTracker: per-sample wandb.log with sample/ prefix, flattened
MetricStats in run summary, reinit=True for suite mode
- SheetsTracker: summary-only (no per-sample API calls), 29-column
canonical row, idempotent header, service account + ADC auth
- EvalRunner: trackers wired into lifecycle, all calls try/except
wrapped so tracker failures never abort an eval run
- CLI: 7 new flags (--wandb-project/entity/tags/group,
--sheets-id/worksheet/creds) on both jarvis eval run and
python -m openjarvis.evals run
- Config: tracker fields parsed from TOML [run] section, propagated
through expand_suite() for suite mode
- pyproject.toml: eval-wandb and eval-sheets optional dependency groups
- 9 new tests (lifecycle, crash resilience, mock W&B/Sheets)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Define the abstract interface for the speech subsystem: SpeechBackend ABC
with transcribe/health/supported_formats methods, TranscriptionResult and
Segment dataclasses for structured transcription output.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add SpeechConfig dataclass with backend, model, language, device, and
compute_type fields. Wire it into JarvisConfig and load_config() so
[speech] TOML sections are loaded automatically.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace vllm.py, sglang.py, llamacpp.py, mlx.py, lmstudio.py with
openai_compat_engines.py using dynamic class creation and registration.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Delete the entire src/openjarvis/memory/ directory (11 files) which
was a pure re-export shim pointing to openjarvis.tools.storage.*.
Update all imports across src/ and tests/ to use the canonical
openjarvis.tools.storage imports directly.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Fix mock.patch() target in test_config.py (evals.cli -> openjarvis.evals.cli)
- Remove stale sys.path manipulation in evals/tests/conftest.py
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Move the standalone evals framework from the project root into the
openjarvis package. Rewrite all ~50+ import statements from 'from evals.'
to 'from openjarvis.evals.' across the package, CLI, and tests. Remove
the evals-specific pyproject.toml (no longer a standalone package).
Update ruff per-file-ignores paths and fix line-length violations
introduced by the longer import paths.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-locate recipe TOML files with the loader code that reads them,
eliminating the fragile parents[3] path traversal. Updates
_PROJECT_RECIPES_DIR to use a relative package data path instead.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add std_latency to LatencyBenchmark metrics. New _render_stats_table
in bench_cmd.py detects stats-pattern keys (mean_X, p50_X, min_X,
max_X, std_X, p95_X) and renders Avg/Median/Min/Max/Std/P95 columns.
Falls back to simple key-value for non-stats metrics.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Wrap engine.generate() in ask.py with console.status() spinner.
Wrap memory indexing loop in memory_cmd.py with rich.progress.track().
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
New log_config.py with setup_logging() configuring the openjarvis
logger (WARNING default, DEBUG on --verbose, ERROR on --quiet).
RotatingFileHandler (5MB, 3 backups) enabled in verbose mode.
Flags added to root CLI group and forwarded via click context.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
New hints.py with hint_no_config(), hint_no_engine(), hint_no_model().
Wire hint_no_engine into ask.py EngineConnectionError handlers to
show actionable suggestions when engine is unreachable.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
5-step guided setup: detect hardware, write config, check engine,
verify model, run test query. Skips config step if already present
unless --force is used. Exits with helpful message on engine failure.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add StepTypeStats dataclass with full descriptive statistics per step
type (count, avg/median/min/max/std duration, total energy, and
avg/median/min/max/std input/output tokens)
- Add total_energy_joules, total_generate_energy_joules, and
step_type_stats fields to TraceSummary
- Compute per-step-type aggregations in summary() method
- Add 3 tests covering energy totals, step-type stats, and dataclass fields
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add tokens_per_joule column to TelemetryStore schema, INSERT, and migration
- Add avg_tokens_per_joule field to ModelStats and EngineStats
- Add AVG(tokens_per_joule) to per_model_stats() and per_engine_stats() SQL
queries using _safe_col() pattern for backward compatibility
- Add 3 tests for store/retrieve, multi-record aggregation, and engine stats
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Computes completion_tokens / energy_joules in both generate() and
stream() methods. Zero when energy or tokens are zero.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
