Commit Graph
167 Commits
Author SHA1 Message Date
Jon Saad-FalconandClaude Opus 4.6 c457a332f1 feat(rust): add monitor_operative and native_openhands agents
MonitorOperativeAgent with 4 configurable strategies (memory extraction,
observation compression, retrieval, task decomposition). NativeOpenHandsAgent
with CodeAct pattern (code blocks + structured tool calls).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-07 00:25:41 +00:00
Jon Saad-FalconandClaude Opus 4.6 9fb2d3a981 feat(rust): add optimization module to openjarvis-learning crate
Port optimization types, store, search space, engine, and LLM optimizer
from Python to Rust. Includes SearchDimension, SearchSpace, TrialConfig,
TrialResult, OptimizationStore (SQLite), OptimizationEngine with Pareto
frontier, LLMOptimizer with OptimizerBackend trait. 47 tests pass.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-07 00:17:25 +00:00
Jon Saad-FalconandClaude Opus 4.6 e280cc41ff test: consolidate test files into subdirectories
Move remaining top-level test files to appropriate subdirectories:
- test_system.py -> tests/sdk/
- test_system_learning.py -> tests/learning/
- test_integration*.py -> tests/integration/
- test_docker.py -> tests/deployment/
- test_rust_bridge.py -> tests/core/

Fix ROOT path in test_docker.py for new directory depth.
All 3772 tests pass, 57 skipped (optional deps).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-06 23:32:54 +00:00
Jon Saad-FalconandClaude Opus 4.6 3677dbab6c style: fix all Ruff lint errors in src/
Shorten docstrings in backward-compat shims to fix E501 (line too long).
Sort imports in learning/__init__.py. Zero Ruff violations remaining.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-06 23:28:57 +00:00
Jon Saad-FalconandClaude Opus 4.6 1aae88ccde refactor: move optimize test files into tests/learning/
Move 12 optimize-related test files from tests/ root into tests/learning/
to match the module reorganization. All 305 tests pass from new locations.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-06 23:27:19 +00:00
Jon Saad-FalconandClaude Opus 4.6 9d4e935af8 refactor: move optimize/ into learning/optimize/ with backward-compat shims
Move the optimization module under the Learning pillar where it
conceptually belongs. All internal imports updated to
openjarvis.learning.optimize. Backward-compatibility shims at the old
openjarvis.optimize path re-export everything. External consumers
(CLI optimize_cmd, server api_routes) updated to import from new location.
All 305 optimize-related tests pass.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-06 23:26:36 +00:00
Jon Saad-FalconandClaude Opus 4.6 418892b38a chore: clean extraneous files from repo, update .gitignore
Remove NFS lock artifacts, empty research output files, results/ and
site/ build artifacts. Add .ruff_cache/, .nfs*, research_mining_* to
.gitignore.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-06 23:16:44 +00:00
Jon Saad-FalconandClaude Opus 4.6 f19970dece chore: remove extraneous files from repo root
Remove internal docs (CLAUDE.md, README.md, ROADMAP.md, VISION.md, NOTES.md),
testing scripts (run_dry_run.py, run_full_optimization.py), and update .gitignore
to prevent re-accumulation of logs/, traces/, coding_task_*, *.jsonl, *.npz, and
NFS lock files.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-06 23:11:31 +00:00
Jon Saad-FalconandGitHub 2ed56fb0b3 Merge pull request #17 from HazyResearch/feat/multi-benchmark-optimization
feat: joint multi-benchmark optimization with LLM-guided search
2026-03-06 11:09:14 -08:00
Tarun SureshandClaude Opus 4.6 2ac1993ea8 fix: resolve lint errors and update tests for fixed params injection
- Fix all ruff E501 line-length violations
- Add noqa: E402 for intentional late imports in runner scripts
- Remove unused imports in tb_agent.py
- Update test_llm_optimizer assertions to expect fixed params in
  unparseable/missing-params cases (intended behavior change)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-06 18:58:53 +00:00
Tarun SureshandClaude Opus 4.6 df96348e81 merge: resolve conflicts with origin/main (Phase 25)
Keep both system_prompt (optimization) and episode_mode (Phase 25) in
RunConfig. Merge terminalbench_native metadata using v2 task_data dict
API while preserving new create_task_env/verify_requirements methods.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-06 18:53:27 +00:00
Tarun SureshandClaude Opus 4.6 bf24ffc527 feat: joint multi-benchmark optimization with LLM-guided search
Add Claude Opus-guided optimization loop that jointly optimizes agent
configs across multiple benchmarks (TerminalBench-native, GAIA, HLE)
with weighted accuracy aggregation and Pareto frontier computation.

Key additions:
- MultiBenchTrialRunner with native terminal-bench v2 Docker execution
- LLM optimizer: fixed params injection, structured trial feedback
- OptimizationStore (SQLite) with per-benchmark score persistence
- System prompt passthrough from optimizer → eval → agent
- Custom terminal-bench agent (OpenJarvisTerminalBenchAgent) avoiding
  LiteLLM serialization issues with agent_import_path
- TOML configs for Qwen3-235B joint agentic optimization
- CLI: `jarvis optimize` command with dry-run support

Early results on Qwen3-235B-A22B-Instruct-2507-FP8 (4xA100):
  Best config: native_openhands, temp=0.0, 20 turns → 5.6% weighted acc
  (TB2=5%, GAIA=6%, HLE=6%)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-06 18:44:24 +00:00
Jon Saad-FalconandGitHub 90c6066b10 Merge pull request #16 from HazyResearch/feat/savings-meter-benchmarks-comparison
feat: eval pipeline upgrade + savings meter benchmarks
2026-03-05 20:48:57 -08:00
Jon Saad-FalconandClaude Opus 4.6 50ca80dfa4 merge: resolve conflicts with main (keep both benchmark sets)
Merge origin/main into feat/savings-meter-benchmarks-comparison.
Both sides added new benchmarks to cli.py — keep all:
- Our branch: use-case benchmarks (email_triage, morning_brief, etc.)
- Main: agentic benchmarks (loghub, ama-bench, workarena, etc.)

dataset.py: keep both create_task_env/verify_requirements and iter_episodes.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-06 04:38:26 +00:00
Jon Saad-FalconandClaude Opus 4.6 d89bbcce52 feat: upgrade eval pipeline with agentic runner, telemetry session, and savings meter
Add 9 capabilities to match IPW pipeline:

Eval Pipeline:
- AgenticRunner for multi-turn agent execution with per-turn trace decomposition
- QueryTrace/TurnTrace data model for agentic workload telemetry
- EventRecorder for thread-safe agent event collection
- TerminalBenchTaskEnv for Docker-based task execution
- Cost computation via engine/cloud.py PRICING table
- Rich export: JSONL, HF Arrow, summary JSON, artifacts manifest
- CLI: --agentic, --concurrency, --query-timeout flags

Telemetry:
- TelemetrySession with background-sampling ring buffer (Python fallback)
- Phase metrics: prefill/decode energy split at TTFT boundary
- ITL percentile tracking (p50/p90/p95/p99)
- FLOPs estimation and MFU computation
- EnergyMonitor.snapshot() method

Rust Performance Layer:
- Ring buffer with binary search O(log n) window queries
- Trapezoidal energy integration
- Phase metrics, ITL stats, FLOPs estimation in Rust
- PyO3 bindings for all new telemetry modules (50 Rust tests)

Savings Meter & Benchmarks:
- Use-case benchmark datasets (coding, email, research, knowledge, morning brief)
- Savings dashboard component with cost comparison visualization
- Cloud cost calculator and comparison server routes
- Use-case eval configs for multiple agent/engine combinations

Tests: 80 new tests (3779 total pass, 37 skipped, 0 failures)
Lint: ruff check src/ tests/ — all checks passed

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-06 04:22:55 +00:00
Jon Saad-FalconandGitHub 2a78b6e537 Merge pull request #15 from HazyResearch/feat/integrate-exo-nexa-uzu-applefm-engines
feat: integrate Exo, Nexa, Uzu, and Apple FM engines
2026-03-05 18:16:24 -08:00
Jon Saad-FalconandClaude Opus 4.6 5b22d991f3 fix: remove duplicate definitions and resolve lint errors
- Remove duplicate config structs in Rust config.rs and Python config.py
- Remove duplicate match arms in Rust discovery.rs
- Fix E501 line-too-long in evals/cli.py
- Fix E402 imports-not-at-top in test_ama_bench.py and test_loghub.py
- Fix E501 long lines in test_loghub.py

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-06 02:13:17 +00:00
Jon Saad-FalconandClaude Opus 4.6 3040f468b6 feat: integrate Exo, Nexa, Uzu, and Apple FM inference engines
Add four new OpenAI-compatible inference engines across the full stack:

Rust backend:
- Config structs with serde defaults (ExoEngineConfig, NexaEngineConfig,
  UzuEngineConfig, AppleFmEngineConfig) in openjarvis-core
- Factory constructors on OpenAICompatEngine (exo, nexa, uzu, apple_fm)
- Engine enum variants with delegate_engine! macro dispatch
- Discovery and resolution in get_engine_static()
- PyO3 bridge support in PyEngine

Python frontend:
- Data-driven engine class registration in openai_compat_engines.py
- Config dataclasses with backward-compat host properties
- Discovery host map entries
- DEFAULT_SEARCH_SPACE updated with new engine backends
- Apple FM shim (FastAPI wrapper for python-apple-fm-sdk, macOS only)

Config & tests:
- TOML config sections for all four engines
- Unit tests for factory methods, enum variants, and search space

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-06 02:00:10 +00:00
Gabriel Bo e278d3ac5e desktop app warning for damaged mac app 2026-03-05 14:21:05 -08:00
Gabriel Bo 152b3bf92d merge conflict in index.md 2026-03-05 13:47:26 -08:00
Gabriel Bo 0ad9fdba8a feat: removing downloads and consolidating all to getting started 2026-03-05 13:45:26 -08:00
Jon Saad-FalconandGitHub 6bf872c079 Merge pull request #14 from HazyResearch/desktop-app
Desktop app
2026-03-05 13:30:52 -08:00
Jon Saad-FalconandGitHub c8f3c01407 Merge pull request #13 from HazyResearch/feat/auto-gen-api-reference
feat: add auto-generated API reference via gen-files + literate-nav
2026-03-05 13:25:19 -08:00
Gabriel Bo fdde659347 uploaded working dmg for macos, ubuntu, and windows 2026-03-05 13:19:27 -08:00
Gabriel Bo 25f0d0ca1e fixing desktop set up backend 2026-03-05 13:14:42 -08:00
Jon Saad-FalconandClaude Opus 4.6 0fb7bf8dd9 feat: add auto-generated API reference via gen-files + literate-nav
- Add gen-files and literate-nav mkdocs plugins
- Create docs/gen_ref_pages.py (adapted from IPW) to auto-generate API reference pages
- Replace 15 manual API nav entries with single auto-generated api-reference/ section
- Add mkdocs-gen-files and mkdocs-literate-nav to docs extra dependencies

Cross-pollinated from intelligence-per-watt docs patterns.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-05 20:52:15 +00:00
Jon Saad-FalconandGitHub 8a0aee7e38 Merge pull request #12 from HazyResearch/fix/codebase-cleanup-bugs-and-lint
fix: codebase cleanup — bug fixes, test hardening, and lint
2026-03-04 22:02:36 -08:00
Jon Saad-FalconandClaude Opus 4.6 6bb323a9fc fix: resolve CI failures — lint, import guards, and test fixtures
- Remove unused imports (os, pytest) in test_rust_bridge.py
- Add pytest.importorskip for fastapi/starlette in test_learning_api.py
- Fix AppleEnergyMonitor test fixtures: set _zeus_ok, _chip_name,
  _tdp_watts attributes when using __new__ to skip __init__
- Fix uninitialized monitor test to match CPU-time fallback behavior

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-05 05:59:32 +00:00
Jon Saad-FalconandClaude Opus 4.6 c488e2a263 style: fix remaining import guards and unused imports
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-05 05:47:29 +00:00
Jon Saad-Falcon 449bdac2a1 Merge remote-tracking branch 'origin/main' into fix/codebase-cleanup-bugs-and-lint 2026-03-05 05:44:23 +00:00
Jon Saad-FalconandClaude Opus 4.6 2c5a9a76d7 ci: add lint and test workflow for pushes and PRs
Runs ruff check and pytest on every push to main and every PR.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-05 05:38:45 +00:00
Jon Saad-FalconandClaude Opus 4.6 62f2ea2252 fix: fix loop_guard role comparison for Python 3.10 compatibility
str(Role.SYSTEM) returns 'Role.SYSTEM' on Python 3.10, not 'system'.
Use getattr(m, 'role', None) == 'system' which works because Role
inherits from str. Also wires compress_context into operative agent.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-05 05:38:27 +00:00
Jon Saad-FalconandGitHub ac1bc4ded1 Merge pull request #11 from HazyResearch/python-rust
Use Rust utils in Python SDK
2026-03-04 21:31:24 -08:00
Jon Saad-FalconandClaude Opus 4.6 82ef2664a6 fix: codebase cleanup — bug fixes, test hardening, and lint
- Remove pynvml from core deps (optional, behind pynvml import guard)
- Fix SystemBuilder wiring bugs (agent/tools/telemetry initialization)
- Fix LoopGuard hash tracking and ping-pong detection edge cases
- Fix OrchestratorAgent/NativeReActAgent tool_calls handling
- Fix OpenAI-compat engine streaming and error handling
- Fix Ollama engine model listing and health checks
- Fix eval display, config parsing, and runner edge cases
- Fix guardrails engine scan mode handling
- Fix dashboard panel rendering and data formatting
- Fix web_search tool error handling
- Fix quickstart CLI detection and setup flow
- Harden channel tests (WhatsApp, Matrix, IRC, Signal, Teams, etc.)
- Harden optimizer, trace judge, and personal synthesizer tests
- Add missing test coverage for types, engine, system, and security

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-05 05:29:42 +00:00
Jon Saad-FalconandGitHub 07d283f705 Merge pull request #10 from HazyResearch/composition
Add unified composition system
2026-03-04 21:29:27 -08:00
krypticmouse 2cd131847c Add unified composition system 2026-03-04 19:35:33 -08:00
krypticmouse b03a0662bb Use Rust utils in Python SDK 2026-03-04 19:15:50 -08:00
Herumb ShandilyaandGitHub 83330183fb Merge pull request #9 from HazyResearch/rust-migration
Add Generics, reduce dynamic dispatch, python integration
2026-03-04 18:14:31 -08:00
krypticmouse 77026f6803 Add Generics, reduce dynamic dispatch, python integration 2026-03-04 18:13:45 -08:00
Jon Saad-FalconandGitHub c58bab27ec Merge pull request #8 from HazyResearch/feat/optimize
feat(optimize): trace-aware multi-objective optimization engine
2026-03-04 17:14:14 -08:00
Jon Saad-FalconandClaude Opus 4.6 3c39995476 feat(optimize): add trace-aware multi-objective optimization
Surface per-instance scores (SampleScore), structured feedback
(TrialFeedback), Pareto frontier tracking, pillar-targeted mutation,
and config merge strategies. Store migration adds new columns with
backward-compatible deserialization.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-05 01:04:30 +00:00
Jon Saad-FalconandClaude Opus 4.6 552ce4cba0 feat(optimize): add CLI commands, API routes, and config integration
- Add `jarvis optimize` CLI group (run, status, results, best, personal)
- Add `jarvis feedback` CLI group (score, thumbs, evaluate, stats)
- Add OptimizeConfig to JarvisConfig with TOML loading support
- Add optimization + feedback event types to EventType enum
- Add TraceStore.update_feedback() for recording trace scores
- Add /v1/feedback and /v1/optimize API routes
- Register optimize/feedback commands in cli/__init__.py
- Update optimize/__init__.py with all Phase 25 exports

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-05 00:01:52 +00:00
Jon Saad-FalconandClaude Opus 4.6 d234471c03 feat(optimize): add optimization engine, SQLite store, and config loader
Add the orchestration layer that ties together the LLM optimizer, trial
runner, and persistence into a propose-evaluate-analyze loop with early
stopping and recipe export. 36 new tests pass.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-04 23:56:30 +00:00
Jon Saad-FalconandClaude Opus 4.6 44dc78ff7a feat(optimize): add LLM optimizer for config tuning
Implement LLMOptimizer that uses a cloud LLM to propose and analyze
OpenJarvis configurations, inspired by DSPy's GEPA approach with
textual trace feedback rather than just scalar rewards. 46 tests pass.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-04 23:50:19 +00:00
Jon Saad-FalconandClaude Opus 4.6 9ef339f531 feat(optimize): add trial runner and feedback system
Add TrialRunner to bridge TrialConfig to the eval framework (EvalRunner),
TraceJudge for LLM-as-judge scoring of agent traces, and FeedbackCollector
for aggregating explicit, thumbs, and judge-driven feedback signals.
68 tests pass covering all three modules.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-04 23:49:23 +00:00
Jon Saad-FalconandClaude Opus 4.6 8437fccbb3 feat(optimize): add personal benchmark system for trace-driven eval
Synthesize reusable benchmarks from interaction traces with feedback
scores. Mines high-quality traces, groups by query class, picks the
best reference per class, and exposes results through DatasetProvider
and LLM-judge Scorer so EvalRunner can evaluate against personal
workflow patterns.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-04 23:46:53 +00:00
Jon Saad-FalconandClaude Opus 4.6 529cd77ccd feat(optimize): add foundation types and search space builder
Add the optimize module with core data types (SearchDimension, SearchSpace,
TrialConfig, TrialResult, OptimizationRun) and search space builder
(build_search_space, DEFAULT_SEARCH_SPACE) covering all 5 pillars. Includes
TrialConfig.to_recipe() mapping and SearchSpace.to_prompt_description()
for LLM-readable rendering. 66 tests pass.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-04 23:43:54 +00:00
Jon Saad-FalconandGitHub d1b68df3fc Merge pull request #7 from HazyResearch/rust-migration
Python to Rust Migration [WIP]
2026-03-04 11:27:56 -08:00
Orhun AkenginandClaude Opus 4.6 896404b4e0 Merge main into rust-migration, resolve conflicts
- .gitignore: keep both Rust target/ and Claude plan artifacts/Tauri schemas sections
- uv.lock: keep main's provides-extras with speech, eval-wandb, eval-sheets extras

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-04 18:56:04 +00:00
Jon Saad-FalconandGitHub 21870cfb37 Merge pull request #4 from HazyResearch/gabebo-ui
webapp + desktop app UI setup w/ working backend + pods
2026-03-03 22:08:43 -08:00