mirror of
https://github.com/open-jarvis/OpenJarvis.git
synced 2026-07-30 19:02:16 +00:00
removing unnecessary files
This commit is contained in:
Gabriel Bo
@@ -1,177 +0,0 @@
|
||||
# Differentiated Functionalities Design
|
||||
|
||||
**Date:** 2026-02-28
|
||||
**Approach:** Learning Flywheel (Approach A)
|
||||
**Status:** Approved
|
||||
|
||||
## Vision
|
||||
|
||||
Make OpenJarvis's functionalities genuinely differentiated through four pillars of uniqueness:
|
||||
|
||||
1. **Fully open-source on-device stack** — every component runs locally, user controls data
|
||||
2. **Native on-device intelligence** — make local LMs punch above their weight via trace-driven learning
|
||||
3. **Latency, privacy, energy as first-class** — alongside accuracy, not afterthoughts
|
||||
4. **Composable, programmable abstractions** — Intelligence, Agent, Tools, Engine, Learning as optimized components
|
||||
|
||||
**Priority:** On-device depth first. Trace-driven learning is the core differentiator.
|
||||
|
||||
## Design Decisions
|
||||
|
||||
- **Approach A selected:** Build the trace-to-learn-to-eval loop as the backbone, hang user-facing features off it
|
||||
- **Learning targets:** Full stack — routing, agent logic, AND model weights (LoRA/QLoRA)
|
||||
- **Eval purpose:** Research platform — rigorous before/after measurement with reproducible configs
|
||||
- **Composition layers:** Python SDK + TOML configs + pre-built recipes
|
||||
- **Operators:** 3 recipes (researcher, correspondent, sentinel) — not a new abstraction, just recipe + schedule + channel
|
||||
- **Interleave:** Wire up real learning AND build new user-facing features simultaneously
|
||||
|
||||
## Section 1: Trace-Driven Learning Pipeline
|
||||
|
||||
The core differentiator. Makes the learning pipeline real, not just infrastructure.
|
||||
|
||||
### Current State
|
||||
|
||||
`TraceStore` captures traces, `TraceAnalyzer` computes stats, GRPO/Bandit policies update internal routing state, `SFTRouterPolicy` and `AgentAdvisorPolicy` exist but don't produce real training jobs, `ICLUpdaterPolicy` updates in-context examples with versioning/rollback.
|
||||
|
||||
### New Components
|
||||
|
||||
**Trace-to-Training Data pipeline** (`learning/training/data.py`): Mines the `TraceStore` for supervised pairs. For successful traces (user rated positively or task completed), extracts `(input, preferred_output)` pairs. For routing, extracts `(query_class, best_model)` pairs. For agent logic, extracts `(query_type, best_agent_config)` tuples. Filters: minimum trace quality score, deduplication, privacy scanning (strip PII before training).
|
||||
|
||||
**LoRA/QLoRA fine-tuning** (`learning/training/lora.py`): Wraps `transformers` + `peft` + `trl` for actual weight updates on local models. Takes training data from the pipeline, runs LoRA fine-tuning with configurable rank/alpha/dropout. Produces adapter weights saved alongside the base model. Energy-aware via existing `EnergyMonitor`. Guarded by hardware detection — only runs if sufficient VRAM/RAM.
|
||||
|
||||
**Agent config evolution** (`learning/agent_learning.py`): Makes `AgentAdvisorPolicy` actually rewrite agent TOML configs. Analyzes traces to identify: which tools were useful vs. never called, which system prompt patterns led to success, optimal `max_turns` per query class. Writes updated configs with version control (git-tracked, rollback via `ICLUpdaterPolicy` pattern).
|
||||
|
||||
**Learning orchestrator** (`learning/orchestrator.py`): Coordinates all learning. Runs on a schedule (e.g., nightly) or on-demand. Steps: collect new traces, mine training data, run evals (baseline), fine-tune model / update agent configs, run evals (after), accept/reject based on improvement threshold. Atomic: if evals don't improve, rollback automatically.
|
||||
|
||||
### Data Flow
|
||||
|
||||
```
|
||||
User queries -> Traces recorded -> TraceStore (SQLite)
|
||||
|
|
||||
LearningOrchestrator (scheduled/on-demand)
|
||||
| | |
|
||||
TrainingDataMiner AgentConfigEvolver RoutingPolicyUpdater
|
||||
| | |
|
||||
LoRA fine-tune TOML config update GRPO/Bandit update
|
||||
| | |
|
||||
EvalHarness (before/after comparison)
|
||||
|
|
||||
Accept (deploy) or Reject (rollback)
|
||||
```
|
||||
|
||||
## Section 2: Eval Framework
|
||||
|
||||
Proves the learning flywheel works and serves as the research platform.
|
||||
|
||||
### Five Workload-Type Eval Suites
|
||||
|
||||
- **Chat:** Multi-turn conversation quality (MT-Bench style, coherence, helpfulness)
|
||||
- **Reasoning:** Logic/math/science (GSM8K, ARC, MMLU subsets that run locally)
|
||||
- **RAG:** Retrieval-augmented accuracy (measures whether memory retrieval improves answers vs. without, uses OpenJarvis's own memory backends)
|
||||
- **Agentic:** Task completion rate for multi-step tool-use workflows (custom scenarios: research a topic, triage inbox, etc.)
|
||||
- **Coding:** Code generation correctness (HumanEval, MBPP subsets)
|
||||
|
||||
### On-Device Metrics as First-Class
|
||||
|
||||
Every eval records: accuracy, latency (TTFT + total), energy consumption (via `EnergyMonitor`), peak memory, tokens/second. Results are multi-dimensional, not a single number. Configurable weighting (e.g., 40% accuracy, 30% latency, 20% energy, 10% memory).
|
||||
|
||||
### Before/After Measurement
|
||||
|
||||
`LearningOrchestrator` calls the eval harness before learning (baseline) and after (candidate). Improvement measured per-dimension. Configurable acceptance threshold (e.g., accept if accuracy improves >2% without latency regressing >10%).
|
||||
|
||||
### Reproducible Configs
|
||||
|
||||
Every eval run fully specified by TOML (model, agent, tools, dataset, metrics, hardware). Results are JSONL with full provenance. CLI: `jarvis eval run`, `jarvis eval compare`, `jarvis eval report`.
|
||||
|
||||
### Ablation Support
|
||||
|
||||
Run the same eval with one variable changed (with vs. without LoRA, with vs. without a tool, with vs. without memory). Built-in diffing and statistical significance testing.
|
||||
|
||||
## Section 3: Composable Abstractions and Recipes
|
||||
|
||||
The user-facing layer for interacting with the pillars.
|
||||
|
||||
### Recipe System (`recipes/`)
|
||||
|
||||
A recipe is a curated composition of all 5 pillars in a single TOML file: model selection, engine preference, agent type + system prompt, tool set, learning policy, and eval config. Recipes are opinionated defaults that work well together.
|
||||
|
||||
Users load recipes via `jarvis ask --recipe coding_assistant "Fix this bug"` or `Jarvis(recipe="coding_assistant")` in Python.
|
||||
|
||||
### Agent Templates (15-20)
|
||||
|
||||
Pre-configured agent TOML manifests with system prompts, tool sets, and behavioral parameters.
|
||||
|
||||
Categories:
|
||||
- **Coding:** code-reviewer, debugger, architect
|
||||
- **Research:** deep-researcher, fact-checker, summarizer
|
||||
- **Productivity:** inbox-triager, meeting-prep, note-taker
|
||||
- **General:** assistant, tutor, translator, writer
|
||||
|
||||
### Bundled Skills (20-30)
|
||||
|
||||
Focused on on-device use cases where local execution matters.
|
||||
|
||||
Categories:
|
||||
- **File management:** organize, deduplicate, backup
|
||||
- **Personal knowledge:** extract, summarize, index
|
||||
- **Development:** lint, test, review
|
||||
- **Productivity:** calendar prep, email draft, todo management
|
||||
|
||||
### Three Composition Layers
|
||||
|
||||
- **CLI flags:** `--recipe`, `--agent`, `--tools`, `--model` for quick use
|
||||
- **TOML configs:** Full pillar configuration for persistent setups
|
||||
- **Python SDK:** `SystemBuilder` for programmatic composition, research scripts, custom pipelines
|
||||
|
||||
## Section 4: Operator Recipes
|
||||
|
||||
OpenJarvis's answer to autonomous task agents. Operators are NOT a new abstraction — they are recipe + schedule + channel output, composed via TOML.
|
||||
|
||||
### Deep Researcher (`recipes/operators/researcher.toml`)
|
||||
|
||||
Autonomous research agent. Given a topic, searches the web, cross-references sources, evaluates credibility, builds a knowledge graph entry, produces a cited report. Runs on-demand or scheduled.
|
||||
|
||||
Tools: `web_search`, `http_request`, `memory_store`, `memory_search`, `kg_add_entity`, `kg_add_relation`, `file_write`.
|
||||
|
||||
Learning: traces which search strategies yield useful results, evolves search query patterns over time.
|
||||
|
||||
### Correspondent (`recipes/operators/correspondent.toml`)
|
||||
|
||||
Messaging triage across Slack/Gmail/Discord. Monitors incoming messages, classifies urgency (urgent/normal/low/ignore), drafts responses for high-priority items, summarizes the rest into a daily digest.
|
||||
|
||||
Tools: `memory_store`, `memory_search`, `think`, `llm_call`.
|
||||
|
||||
Channels: Slack, email, Discord.
|
||||
|
||||
Learning: adapts to user's triage preferences over time.
|
||||
|
||||
### Sentinel (`recipes/operators/sentinel.toml`)
|
||||
|
||||
Monitors Twitter, Reddit, Mastodon, Google Trends, RSS feeds, and specified URLs for changes relevant to user-defined topics. Produces alerts when something significant surfaces — trending discussions, sentiment shifts, breaking news, competitor activity.
|
||||
|
||||
Tools: `web_search`, `http_request`, `memory_store`, `memory_search`, `kg_add_entity`.
|
||||
|
||||
Channels: Twitter, Reddit, Mastodon (read via APIs), Google Trends.
|
||||
|
||||
Learning: refines what "significant" means based on which alerts the user acts on vs. dismisses.
|
||||
|
||||
## Section 5: Testing and Validation
|
||||
|
||||
### Learning Pipeline Tests
|
||||
|
||||
Unit tests for `TrainingDataMiner` (mock traces, verify correct pairs), `LoRATrainer` (mock training, verify adapter saved), `AgentConfigEvolver` (mock traces, verify TOML rewritten), `LearningOrchestrator` (mock all steps, verify accept/reject logic). Integration test: synthetic traces, actual LoRA training on tiny model, verify loss decreases.
|
||||
|
||||
### Eval Framework Tests
|
||||
|
||||
Each of the 5 eval suites has a smoke mode (5-10 examples, <30 seconds). Verify multi-dimensional scoring. Verify before/after comparison and acceptance thresholds. Verify ablation diffing.
|
||||
|
||||
### Recipe and Template Tests
|
||||
|
||||
Each recipe loads without error. Each agent template produces a valid agent. Each skill executes its steps. Composition tests: `--recipe` flag wires all pillars correctly.
|
||||
|
||||
### Operator Tests
|
||||
|
||||
Each operator recipe loads and schedules correctly. Mock tool execution to verify agent loop completes. Verify learning feedback loop (trace, config update, re-eval).
|
||||
|
||||
### Regression
|
||||
|
||||
All ~2940 existing tests continue to pass. Target ~200-300 additional tests for new functionality.
|
||||
File diff suppressed because it is too large
Load Diff
@@ -1,319 +0,0 @@
|
||||
# Experience Polish Design: Eval, CLI, Install, Dashboard
|
||||
|
||||
**Date**: 2026-02-28
|
||||
**Status**: Approved
|
||||
**Approach**: Vertical slices (3 mini-phases, independently shippable)
|
||||
|
||||
## Context
|
||||
|
||||
OpenJarvis has strong infrastructure (multi-vendor energy monitoring, eval framework, Tauri desktop, PWA) but the end-to-end user experience has gaps. This design addresses: eval output quality, CLI polish, installation flow, and dashboard aesthetics.
|
||||
|
||||
### Current State (from audits)
|
||||
|
||||
| Area | Score | Key Gaps |
|
||||
|------|-------|----------|
|
||||
| Energy Monitors | 8.5/10 | IPJ/IPW only in evals, not core telemetry |
|
||||
| Eval Framework | 9/10 | Token stats null, no grouped tables, no trace aggregation |
|
||||
| CLI Formatting | 9/10 | No logging, no verbose/quiet, bench lacks full stats |
|
||||
| Installation | 8.5/10 | No quickstart, no auto-suggestions in errors |
|
||||
| Desktop App | 8/10 | No settings panel, placeholder icon, stub tray |
|
||||
| Browser/PWA | 6/10 | No CI/CD, no versioning, no error boundary |
|
||||
|
||||
### Key Decisions
|
||||
|
||||
- **Benchmark focus**: GAIA with OpenHands (TerminalBench deferred)
|
||||
- **IPW** = task_accuracy / avg_power_watts (task-level, NOT per-inference)
|
||||
- **IPJ** = task_accuracy / avg_energy_joules (task-level, NOT per-inference)
|
||||
- **tokens_per_joule** = per-inference efficiency metric in core telemetry
|
||||
- **Table layout**: Grouped panels by default, `--compact` for single dense table
|
||||
- **Trace detail**: Summary + step-type breakdown by default, `--trace-detail` for full listing
|
||||
- **Installation**: Non-interactive `jarvis quickstart` (zero prompts)
|
||||
- **Dashboard style**: Clean, minimalistic (ChatGPT/Claude aesthetic) with unique side panels
|
||||
|
||||
---
|
||||
|
||||
## Phase 23a: Perfect Eval Run
|
||||
|
||||
Everything needed so `python -m evals run -c config.toml` produces beautiful, complete results.
|
||||
|
||||
### 1. Core Telemetry: tokens_per_joule
|
||||
|
||||
**Files**:
|
||||
- `src/openjarvis/core/types.py` — Add `tokens_per_joule` to `TelemetryRecord`
|
||||
- `src/openjarvis/telemetry/instrumented_engine.py` — Compute `tokens_per_joule = completion_tokens / energy_joules`
|
||||
- `src/openjarvis/telemetry/store.py` — Schema migration for new column
|
||||
- `src/openjarvis/telemetry/aggregator.py` — Add `avg_tokens_per_joule` to `ModelStats`/`EngineStats`
|
||||
|
||||
**Not** adding IPJ/IPW to core telemetry — they require task accuracy and are inherently eval-level.
|
||||
|
||||
### 2. Eval Metrics: Fix Token Stats + Strengthen IPJ/IPW
|
||||
|
||||
**Files**:
|
||||
- `evals/core/runner.py`:
|
||||
- Wire `prompt_tokens` and `completion_tokens` from engine response into `EvalResult`
|
||||
- Verify IPW = `accuracy / avg(power_watts)` across all scored samples
|
||||
- Verify IPJ = `accuracy / avg(energy_joules)` across all scored samples
|
||||
- Ensure `energy_joules` and `power_watts` populated from telemetry for every sample
|
||||
- `evals/core/types.py`:
|
||||
- Add `total_energy_joules` (sum over all samples) to `RunSummary`
|
||||
- Add `avg_power_watts` (mean over all samples) to `RunSummary`
|
||||
- Add `trace_steps: int`, `trace_energy_joules: float` to `EvalResult`
|
||||
- Add `trace_step_type_stats: Dict[str, StepTypeStats]` to `RunSummary`
|
||||
|
||||
### 3. Eval Display: Grouped Tables
|
||||
|
||||
Rewrite `evals/core/display.py` with these functions:
|
||||
|
||||
#### `print_accuracy_panel(summary)`
|
||||
```
|
||||
╭─ Accuracy ──────────────────────────────────────────────────╮
|
||||
│ Overall Accuracy 42.0% (84/200) │
|
||||
│ Level 1 58.8% (40/68) │
|
||||
│ Level 2 35.0% (35/100) │
|
||||
│ Level 3 28.1% (9/32) │
|
||||
╰─────────────────────────────────────────────────────────────╯
|
||||
```
|
||||
|
||||
#### `print_latency_table(summary)`
|
||||
```
|
||||
╭─ Latency & Throughput ──────────────────────────────────────╮
|
||||
│ Metric Avg Median Min Max Std │
|
||||
│ Latency (s) 15.41 12.30 2.10 89.50 11.20 │
|
||||
│ TTFT (ms) 145.2 120.0 45.0 890.0 85.3 │
|
||||
│ Throughput (tok/s) 41.9 38.5 12.0 95.0 15.2 │
|
||||
│ Avg Input Tokens 1024 890 128 4096 520 │
|
||||
│ Avg Output Tokens 256 210 32 2048 180 │
|
||||
╰─────────────────────────────────────────────────────────────╯
|
||||
```
|
||||
|
||||
#### `print_energy_table(summary)`
|
||||
```
|
||||
╭─ Energy & Efficiency ───────────────────────────────────────╮
|
||||
│ Metric Avg Median Min Max Std │
|
||||
│ Energy (J) 46502 38500 4145 410522 89390 │
|
||||
│ Power (W) 883.8 870.2 650.0 1050.0 85.0 │
|
||||
│ GPU Util (%) 46.4 48.0 12.0 92.0 18.5 │
|
||||
│ Energy/Token (mJ) 12.5 10.8 3.2 45.0 8.1 │
|
||||
│ Tokens/Joule 80.0 92.6 22.2 312.5 65.0 │
|
||||
│ ────────────────────────────────────────────────────────── │
|
||||
│ IPW (acc/W) 0.00048 │
|
||||
│ IPJ (acc/J) 9.03e-06 │
|
||||
│ Total Energy (kJ) 9300.4 │
|
||||
╰─────────────────────────────────────────────────────────────╯
|
||||
```
|
||||
|
||||
#### `print_trace_summary(summary)`
|
||||
```
|
||||
╭─ Agentic Trace Summary ────────────────────────────────────╮
|
||||
│ Total Steps: 1240 │ Avg Steps/Sample: 6.2 │
|
||||
│ │
|
||||
│ Step Type Count Avg Duration Avg Energy Avg In Tok Avg Out Tok │
|
||||
│ Median/Min/Max/Std for each column │
|
||||
│ generate 580 8.2s 38200 J 890 256 │
|
||||
│ tool_call 420 3.1s — — — │
|
||||
│ retrieve 120 0.8s — — — │
|
||||
│ route 120 0.01s — — — │
|
||||
╰─────────────────────────────────────────────────────────────╯
|
||||
```
|
||||
|
||||
All metrics in trace summary show avg/median/min/max/std where applicable.
|
||||
|
||||
#### `print_compact_table(summary)` — `--compact` flag
|
||||
Single dense table with all 17 metrics as rows, columns: avg/median/min/max/std.
|
||||
|
||||
#### CLI flags
|
||||
- `--compact`: Dense single table
|
||||
- `--trace-detail`: Full per-step listing for each sample
|
||||
|
||||
### 4. Per-Step Trace Aggregation
|
||||
|
||||
**Files**:
|
||||
- `src/openjarvis/traces/analyzer.py`:
|
||||
- Add `TraceSummary.total_energy_joules` — sum of `step.metadata['energy_joules']`
|
||||
- Add `TraceSummary.total_generate_energy_joules` — sum for GENERATE steps
|
||||
- Add `TraceSummary.step_type_stats` — dict mapping step type to `{count, avg_duration, median_duration, min_duration, max_duration, std_duration, total_energy, avg_input_tokens, median_input_tokens, min_input_tokens, max_input_tokens, std_input_tokens, avg_output_tokens, median_output_tokens, min_output_tokens, max_output_tokens, std_output_tokens}`
|
||||
- `evals/core/runner.py` — After each agent eval sample, extract `TraceSummary` and store in `EvalResult.metadata`
|
||||
|
||||
---
|
||||
|
||||
## Phase 23b: Perfect First Experience
|
||||
|
||||
Everything needed so a new user goes from zero to first eval in one sitting.
|
||||
|
||||
### 5. `jarvis quickstart` Command
|
||||
|
||||
**New file**: `src/openjarvis/cli/quickstart_cmd.py`
|
||||
|
||||
Non-interactive flow (5 numbered steps):
|
||||
1. Detect hardware (platform, CPU, RAM, GPU vendor/model/VRAM)
|
||||
2. Write config to `~/.openjarvis/config.toml` (skip if exists, unless `--force`)
|
||||
3. Check engine health (try auto-detected engine)
|
||||
4. Verify model availability (list models from engine)
|
||||
5. Test query ("What is 2+2?") with latency + energy measurement
|
||||
|
||||
Each step prints a status line. If a step fails, print a helpful suggestion and exit gracefully.
|
||||
|
||||
Flags: `--force` (redo everything)
|
||||
|
||||
**Register** in `src/openjarvis/cli/__init__.py`.
|
||||
|
||||
### 6. Error Message Auto-Suggestions
|
||||
|
||||
**New file**: `src/openjarvis/cli/hints.py`
|
||||
|
||||
Centralized hint functions:
|
||||
- `hint_no_config()` → "Config not found. Run: `jarvis quickstart`"
|
||||
- `hint_no_engine()` → "No engine responding. Run: `jarvis doctor`"
|
||||
- `hint_no_model()` → "No model available. Try: `ollama pull qwen3:8b`"
|
||||
|
||||
**Wire into**: `ask.py`, `serve.py`, `bench_cmd.py`, `chat_cmd.py` at failure points.
|
||||
|
||||
### 7. Global Logging & Verbose/Quiet Flags
|
||||
|
||||
**Changes**:
|
||||
- `src/openjarvis/cli/__init__.py` — Add `--verbose` / `--quiet` to root `cli` group
|
||||
- **New file**: `src/openjarvis/cli/log_config.py` — Centralized logging setup
|
||||
- `RichHandler` for console (respects quiet flag)
|
||||
- `RotatingFileHandler` for `~/.openjarvis/cli.log` (5 MB max, 3 backups)
|
||||
- Default level: WARNING; `--verbose`: DEBUG; `--quiet`: ERROR
|
||||
|
||||
**Add progress indicators**:
|
||||
- `ask.py` — Spinner during generation
|
||||
- `memory_cmd.py` — Progress bar for indexing
|
||||
|
||||
### 8. Bench CLI Full Stats Tables
|
||||
|
||||
**Changes**:
|
||||
- `src/openjarvis/cli/bench_cmd.py` — Show full stats table (avg/median/min/max/std) matching eval style
|
||||
- `src/openjarvis/bench/latency.py` — Export all percentile stats (already computes internally)
|
||||
- `src/openjarvis/bench/throughput.py` — Add stats computation
|
||||
- `src/openjarvis/bench/energy.py` — Add stats computation
|
||||
|
||||
---
|
||||
|
||||
## Phase 23c: Perfect Dashboard
|
||||
|
||||
Desktop and browser apps polished for demos and daily use.
|
||||
|
||||
### Design Aesthetic
|
||||
|
||||
**Clean, minimalistic** — inspired by ChatGPT and Claude web interfaces:
|
||||
- Generous whitespace, muted color palette, subtle borders
|
||||
- Sans-serif typography, comfortable line heights
|
||||
- Smooth transitions, no visual clutter
|
||||
|
||||
**Differentiated** via unique side panels:
|
||||
- Energy dashboard with real-time power charts
|
||||
- Trace debugger with color-coded timeline
|
||||
- Learning curve visualization
|
||||
- Memory browser with relevance scoring
|
||||
|
||||
The side panels are what make OpenJarvis unique vs generic chat UIs. Keep them accessible but not overwhelming — collapsible, clean data density.
|
||||
|
||||
### 9. Desktop: Settings Panel
|
||||
|
||||
**New file**: `desktop/src/components/SettingsPanel.tsx`
|
||||
- API URL configuration (default: `localhost:8000`)
|
||||
- Auto-update interval setting
|
||||
- Theme toggle (dark/light) — currently hardcoded dark
|
||||
- Persist to `localStorage`
|
||||
|
||||
### 10. Desktop: Windows Icon
|
||||
|
||||
Replace 108-byte placeholder `desktop/src-tauri/icons/icon.ico` with proper multi-resolution icon generated from existing `icon.png`.
|
||||
|
||||
### 11. Desktop: System Tray
|
||||
|
||||
Implement stubbed tray menu in `desktop/src-tauri/src/lib.rs`:
|
||||
- Show/Hide window toggle
|
||||
- Health status indicator (green/red dot)
|
||||
- Quit action
|
||||
|
||||
### 12. Browser/PWA: CI/CD
|
||||
|
||||
**New file**: `.github/workflows/frontend.yml`
|
||||
- Trigger on changes to `frontend/`
|
||||
- `npm ci && npm run build`
|
||||
- Commit built output to `src/openjarvis/server/static/`
|
||||
- Add `version` field to `manifest.webmanifest` (auto-bumped)
|
||||
|
||||
### 13. Browser/PWA: Error Boundary + Config
|
||||
|
||||
- Wrap `<App />` in React error boundary (graceful crash recovery)
|
||||
- Support `VITE_API_URL` environment variable (default `localhost:8000`)
|
||||
|
||||
### 14. Browser/PWA: Style Refresh
|
||||
|
||||
Refactor the 12,971-line `App.css`:
|
||||
- Modularize into component-scoped CSS files
|
||||
- Clean minimalist aesthetic (ChatGPT/Claude-inspired)
|
||||
- Catppuccin-based dark theme (keep existing) + light theme option
|
||||
- Collapsible side panels with smooth transitions
|
||||
|
||||
---
|
||||
|
||||
## Implementation Order
|
||||
|
||||
### Phase 23a (highest priority — enables GAIA research)
|
||||
1. Core telemetry: `tokens_per_joule`
|
||||
2. Eval metrics: fix token stats, strengthen IPJ/IPW
|
||||
3. Per-step trace aggregation in `TraceAnalyzer`
|
||||
4. Eval display: grouped tables, `--compact`, `--trace-detail`
|
||||
|
||||
### Phase 23b (high priority — onboarding & CLI)
|
||||
5. `jarvis quickstart` command
|
||||
6. Error message auto-suggestions
|
||||
7. Global logging + verbose/quiet flags
|
||||
8. Bench CLI full stats tables
|
||||
|
||||
### Phase 23c (lower priority — dashboard polish)
|
||||
9. Desktop settings panel
|
||||
10. Windows icon fix
|
||||
11. System tray implementation
|
||||
12. PWA CI/CD
|
||||
13. PWA error boundary + config
|
||||
14. PWA style refresh
|
||||
|
||||
---
|
||||
|
||||
## Files Modified (Summary)
|
||||
|
||||
### Phase 23a
|
||||
| File | Change |
|
||||
|------|--------|
|
||||
| `src/openjarvis/core/types.py` | Add `tokens_per_joule` to `TelemetryRecord` |
|
||||
| `src/openjarvis/telemetry/instrumented_engine.py` | Compute `tokens_per_joule` |
|
||||
| `src/openjarvis/telemetry/store.py` | Schema migration |
|
||||
| `src/openjarvis/telemetry/aggregator.py` | Add `avg_tokens_per_joule` |
|
||||
| `src/openjarvis/traces/analyzer.py` | Add energy aggregation, step-type stats |
|
||||
| `evals/core/types.py` | Add trace fields, total_energy, avg_power |
|
||||
| `evals/core/runner.py` | Fix token stats, wire trace data, verify IPJ/IPW |
|
||||
| `evals/core/display.py` | Rewrite: grouped panels, compact mode |
|
||||
| `evals/cli.py` | Add `--compact`, `--trace-detail` flags |
|
||||
|
||||
### Phase 23b
|
||||
| File | Change |
|
||||
|------|--------|
|
||||
| `src/openjarvis/cli/quickstart_cmd.py` | **New**: quickstart command |
|
||||
| `src/openjarvis/cli/hints.py` | **New**: centralized hint system |
|
||||
| `src/openjarvis/cli/log_config.py` | **New**: logging setup |
|
||||
| `src/openjarvis/cli/__init__.py` | Register quickstart, add verbose/quiet |
|
||||
| `src/openjarvis/cli/ask.py` | Add hints, progress spinner |
|
||||
| `src/openjarvis/cli/serve.py` | Add hints |
|
||||
| `src/openjarvis/cli/bench_cmd.py` | Full stats tables |
|
||||
| `src/openjarvis/cli/chat_cmd.py` | Add hints |
|
||||
| `src/openjarvis/cli/memory_cmd.py` | Progress bar for indexing |
|
||||
| `src/openjarvis/bench/latency.py` | Export full stats |
|
||||
| `src/openjarvis/bench/throughput.py` | Add stats computation |
|
||||
| `src/openjarvis/bench/energy.py` | Add stats computation |
|
||||
|
||||
### Phase 23c
|
||||
| File | Change |
|
||||
|------|--------|
|
||||
| `desktop/src/components/SettingsPanel.tsx` | **New**: settings UI |
|
||||
| `desktop/src-tauri/icons/icon.ico` | Replace placeholder |
|
||||
| `desktop/src-tauri/src/lib.rs` | Implement system tray |
|
||||
| `.github/workflows/frontend.yml` | **New**: PWA CI/CD |
|
||||
| `frontend/src/App.tsx` | Error boundary wrapper |
|
||||
| `frontend/vite.config.ts` | VITE_API_URL support |
|
||||
| `frontend/src/App.css` | Modularize, style refresh |
|
||||
File diff suppressed because it is too large
Load Diff
@@ -1,71 +0,0 @@
|
||||
# Codebase Simplification Design
|
||||
|
||||
**Date:** 2026-03-02
|
||||
**Goal:** Simplify repository structure and clean up code while preserving all functionality.
|
||||
**Approach:** Incremental commits, tests verified after each step.
|
||||
|
||||
## 1. Target Root Structure
|
||||
|
||||
Reduce root from 32 items to 21 items.
|
||||
|
||||
### Items removed from git tracking
|
||||
- `get-pip.py` (2.2MB, regenerable)
|
||||
- `site/` (9.6MB, mkdocs build output)
|
||||
- `results/` (5.7MB, eval output)
|
||||
|
||||
### Items moved
|
||||
| Source | Destination |
|
||||
|--------|------------|
|
||||
| `Dockerfile`, `Dockerfile.gpu`, `Dockerfile.gpu.rocm`, `Dockerfile.sandbox` | `deploy/docker/` |
|
||||
| `docker-compose.yml`, `docker-compose.gpu.rocm.yml` | `deploy/docker/` |
|
||||
| `recipes/*.toml` | `src/openjarvis/recipes/data/` |
|
||||
| `templates/agents/*.toml` | `src/openjarvis/templates/data/` |
|
||||
| `skills/builtin/*.toml` | `src/openjarvis/skills/data/` |
|
||||
| `operators/*.toml` | `src/openjarvis/operators/data/` |
|
||||
| `evals/` (entire directory) | `src/openjarvis/evals/` |
|
||||
|
||||
### Items kept at root
|
||||
- `CLAUDE.md`, `README.md`, `VISION.md`, `ROADMAP.md`, `NOTES.md`
|
||||
- `mkdocs.yml`, `pyproject.toml`, `uv.lock`, `LICENSE`
|
||||
- `assets/`, `configs/`, `deploy/`, `desktop/`, `docs/`, `frontend/`, `scripts/`, `src/`, `tests/`
|
||||
|
||||
## 2. Data Directory Migration
|
||||
|
||||
Each module's `discover_*()` function defaults to `Path(__file__).parent / "data"` for built-in TOML files. `pyproject.toml` declares package data via `[tool.hatch.build.targets.wheel]`.
|
||||
|
||||
### evals/ migration
|
||||
The entire evals directory (Python modules + configs + dataset loaders) moves into `src/openjarvis/evals/` as a subpackage. CLI `jarvis eval` and `python -m openjarvis.evals` entry points updated.
|
||||
|
||||
## 3. Code Cleanups
|
||||
|
||||
### 3a. Remove memory/ shim package
|
||||
Delete `src/openjarvis/memory/` (11 files, 108 lines of pure re-exports). Update all imports to use `openjarvis.tools.storage.*` directly.
|
||||
|
||||
### 3b. Consolidate engine wrappers
|
||||
Replace 5 near-identical 17-line files (vllm.py, sglang.py, llamacpp.py, mlx.py, lmstudio.py) with a single data-driven registration in `openai_compat_engines.py`.
|
||||
|
||||
### 3c. Refactor repetitive __init__.py imports
|
||||
Replace 26 identical try-except blocks in channels/__init__.py (and similar in learning/, engine/) with loop-based auto-imports using `importlib.import_module()`.
|
||||
|
||||
## 4. Test & CI Impact
|
||||
|
||||
- Update tests importing from `openjarvis.memory.*` to `openjarvis.tools.storage.*`
|
||||
- Delete `tests/memory/` (duplicates `tests/tools/` storage tests)
|
||||
- Update engine tests for consolidated module
|
||||
- Update CI workflows referencing moved paths
|
||||
- Update `pyproject.toml` entry points
|
||||
- All ~3240 tests must pass after each commit
|
||||
|
||||
## 5. Commit Sequence
|
||||
|
||||
1. Clean git artifacts (get-pip.py, site/, results/) + update .gitignore
|
||||
2. Move Docker files to deploy/docker/
|
||||
3. Move recipes/ TOML data into src/openjarvis/recipes/data/
|
||||
4. Move templates/ TOML data into src/openjarvis/templates/data/
|
||||
5. Move skills/ TOML data into src/openjarvis/skills/data/
|
||||
6. Move operators/ TOML data into src/openjarvis/operators/data/
|
||||
7. Move evals/ into src/openjarvis/evals/
|
||||
8. Remove memory/ shim package + update imports
|
||||
9. Consolidate engine wrappers into single file
|
||||
10. Refactor repetitive __init__.py imports
|
||||
11. Update CLAUDE.md to reflect new structure
|
||||
@@ -1,782 +0,0 @@
|
||||
# Codebase Simplification Implementation Plan
|
||||
|
||||
> **For Claude:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task.
|
||||
|
||||
**Goal:** Simplify the OpenJarvis repository structure by reducing root sprawl, moving data files into the package, removing backward-compat shims, and consolidating repetitive code patterns.
|
||||
|
||||
**Architecture:** Incremental commits, each independently testable. Data TOML files move into `src/openjarvis/*/data/` as package data. The `evals/` framework becomes `src/openjarvis/evals/` with all internal imports rewritten from `evals.` to `openjarvis.evals.`. The `memory/` shim package is removed. Engine wrappers are consolidated.
|
||||
|
||||
**Tech Stack:** Python, TOML, hatch/hatchling build system, pytest, ruff
|
||||
|
||||
---
|
||||
|
||||
### Task 1: Remove get-pip.py from git tracking
|
||||
|
||||
**Files:**
|
||||
- Delete: `get-pip.py`
|
||||
- Modify: `.gitignore`
|
||||
|
||||
**Step 1: Add get-pip.py to .gitignore**
|
||||
|
||||
In `.gitignore`, add after the `# Project` section (after line 44):
|
||||
|
||||
```
|
||||
get-pip.py
|
||||
```
|
||||
|
||||
**Step 2: Remove get-pip.py from git tracking and delete**
|
||||
|
||||
Run:
|
||||
```bash
|
||||
git rm get-pip.py
|
||||
```
|
||||
|
||||
**Step 3: Commit**
|
||||
|
||||
```bash
|
||||
git add .gitignore
|
||||
git commit -m "chore: remove get-pip.py from repository (2.2MB regenerable artifact)"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 2: Move Docker files to deploy/docker/
|
||||
|
||||
**Files:**
|
||||
- Move: `Dockerfile` -> `deploy/docker/Dockerfile`
|
||||
- Move: `Dockerfile.gpu` -> `deploy/docker/Dockerfile.gpu`
|
||||
- Move: `Dockerfile.gpu.rocm` -> `deploy/docker/Dockerfile.gpu.rocm`
|
||||
- Move: `Dockerfile.sandbox` -> `deploy/docker/Dockerfile.sandbox`
|
||||
- Move: `docker-compose.yml` -> `deploy/docker/docker-compose.yml`
|
||||
- Move: `docker-compose.gpu.rocm.yml` -> `deploy/docker/docker-compose.gpu.rocm.yml`
|
||||
- Modify: `deploy/docker/docker-compose.yml` (update dockerfile paths)
|
||||
- Modify: `deploy/docker/docker-compose.gpu.rocm.yml` (update dockerfile path)
|
||||
|
||||
**Step 1: Move all Docker files**
|
||||
|
||||
```bash
|
||||
mv Dockerfile deploy/docker/Dockerfile
|
||||
mv Dockerfile.gpu deploy/docker/Dockerfile.gpu
|
||||
mv Dockerfile.gpu.rocm deploy/docker/Dockerfile.gpu.rocm
|
||||
mv Dockerfile.sandbox deploy/docker/Dockerfile.sandbox
|
||||
mv docker-compose.yml deploy/docker/docker-compose.yml
|
||||
mv docker-compose.gpu.rocm.yml deploy/docker/docker-compose.gpu.rocm.yml
|
||||
```
|
||||
|
||||
**Step 2: Update docker-compose.yml build contexts**
|
||||
|
||||
In `deploy/docker/docker-compose.yml`, change the `build` section from:
|
||||
```yaml
|
||||
build:
|
||||
context: .
|
||||
dockerfile: Dockerfile
|
||||
```
|
||||
to:
|
||||
```yaml
|
||||
build:
|
||||
context: ../..
|
||||
dockerfile: deploy/docker/Dockerfile
|
||||
```
|
||||
|
||||
**Step 3: Update docker-compose.gpu.rocm.yml**
|
||||
|
||||
In `deploy/docker/docker-compose.gpu.rocm.yml`, change:
|
||||
```yaml
|
||||
dockerfile: Dockerfile.gpu.rocm
|
||||
```
|
||||
to:
|
||||
```yaml
|
||||
context: ../..
|
||||
dockerfile: deploy/docker/Dockerfile.gpu.rocm
|
||||
```
|
||||
|
||||
**Step 4: Commit**
|
||||
|
||||
```bash
|
||||
git add -A deploy/docker/
|
||||
git add -A Dockerfile* docker-compose*
|
||||
git commit -m "refactor: move Docker files to deploy/docker/"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 3: Move recipes/ TOML data into package
|
||||
|
||||
**Files:**
|
||||
- Move: `recipes/*.toml` -> `src/openjarvis/recipes/data/*.toml`
|
||||
- Move: `recipes/operators/*.toml` -> `src/openjarvis/recipes/data/operators/*.toml`
|
||||
- Move: `recipes/operators/*.md` -> `src/openjarvis/recipes/data/operators/*.md`
|
||||
- Modify: `src/openjarvis/recipes/loader.py` (lines 15-16, update `_PROJECT_RECIPES_DIR`)
|
||||
- Modify: `pyproject.toml` (add package data include)
|
||||
- Test: `tests/recipes/` (run existing tests)
|
||||
|
||||
**Step 1: Create data directory and move files**
|
||||
|
||||
```bash
|
||||
mkdir -p src/openjarvis/recipes/data/operators
|
||||
mv recipes/*.toml src/openjarvis/recipes/data/
|
||||
mv recipes/operators/*.toml src/openjarvis/recipes/data/operators/
|
||||
mv recipes/operators/*.md src/openjarvis/recipes/data/operators/
|
||||
rmdir recipes/operators
|
||||
rmdir recipes
|
||||
```
|
||||
|
||||
**Step 2: Update loader.py**
|
||||
|
||||
In `src/openjarvis/recipes/loader.py`, change line 16 from:
|
||||
```python
|
||||
_PROJECT_RECIPES_DIR = Path(__file__).resolve().parents[3] / "recipes"
|
||||
```
|
||||
to:
|
||||
```python
|
||||
_PROJECT_RECIPES_DIR = Path(__file__).resolve().parent / "data"
|
||||
```
|
||||
|
||||
**Step 3: Run tests**
|
||||
|
||||
```bash
|
||||
uv run pytest tests/recipes/ -v
|
||||
```
|
||||
Expected: All recipe tests pass.
|
||||
|
||||
**Step 4: Commit**
|
||||
|
||||
```bash
|
||||
git add src/openjarvis/recipes/data/ src/openjarvis/recipes/loader.py
|
||||
git add recipes/
|
||||
git commit -m "refactor: move recipes/ TOML data into src/openjarvis/recipes/data/"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 4: Move templates/ agent TOML data into package
|
||||
|
||||
**Files:**
|
||||
- Move: `templates/agents/*.toml` -> `src/openjarvis/templates/data/*.toml`
|
||||
- Modify: `src/openjarvis/templates/agent_templates.py` (lines 68-71, update `_builtin_templates_dir`)
|
||||
- Test: `tests/templates/` (run existing tests)
|
||||
|
||||
**Step 1: Create data directory and move files**
|
||||
|
||||
```bash
|
||||
mkdir -p src/openjarvis/templates/data
|
||||
mv templates/agents/*.toml src/openjarvis/templates/data/
|
||||
rmdir templates/agents
|
||||
rmdir templates
|
||||
```
|
||||
|
||||
**Step 2: Update agent_templates.py**
|
||||
|
||||
In `src/openjarvis/templates/agent_templates.py`, change lines 68-71 from:
|
||||
```python
|
||||
def _builtin_templates_dir() -> Path:
|
||||
"""Return the path to the built-in templates shipped with the package."""
|
||||
# templates/agents/ at project root, 3 levels above this file
|
||||
return Path(__file__).resolve().parents[3] / "templates" / "agents"
|
||||
```
|
||||
to:
|
||||
```python
|
||||
def _builtin_templates_dir() -> Path:
|
||||
"""Return the path to the built-in templates shipped with the package."""
|
||||
return Path(__file__).resolve().parent / "data"
|
||||
```
|
||||
|
||||
**Step 3: Run tests**
|
||||
|
||||
```bash
|
||||
uv run pytest tests/templates/ -v
|
||||
```
|
||||
Expected: All template tests pass.
|
||||
|
||||
**Step 4: Commit**
|
||||
|
||||
```bash
|
||||
git add src/openjarvis/templates/data/ src/openjarvis/templates/agent_templates.py
|
||||
git add templates/
|
||||
git commit -m "refactor: move templates/agents/ TOML data into src/openjarvis/templates/data/"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 5: Move skills/builtin/ TOML data into package
|
||||
|
||||
**Files:**
|
||||
- Move: `skills/builtin/*.toml` -> `src/openjarvis/skills/data/*.toml`
|
||||
- Modify: `tests/skills/test_bundled_skills.py` (line 12, update `BUILTIN_DIR` path)
|
||||
- Test: `tests/skills/` (run existing tests)
|
||||
|
||||
**Step 1: Create data directory and move files**
|
||||
|
||||
```bash
|
||||
mkdir -p src/openjarvis/skills/data
|
||||
mv skills/builtin/*.toml src/openjarvis/skills/data/
|
||||
rmdir skills/builtin
|
||||
rmdir skills
|
||||
```
|
||||
|
||||
**Step 2: Update test_bundled_skills.py**
|
||||
|
||||
In `tests/skills/test_bundled_skills.py`, change line 12 from:
|
||||
```python
|
||||
BUILTIN_DIR = Path(__file__).resolve().parents[2] / "skills" / "builtin"
|
||||
```
|
||||
to:
|
||||
```python
|
||||
BUILTIN_DIR = Path(__file__).resolve().parents[2] / "src" / "openjarvis" / "skills" / "data"
|
||||
```
|
||||
|
||||
**Step 3: Run tests**
|
||||
|
||||
```bash
|
||||
uv run pytest tests/skills/ -v
|
||||
```
|
||||
Expected: All skill tests pass (20 bundled skills found).
|
||||
|
||||
**Step 4: Commit**
|
||||
|
||||
```bash
|
||||
git add src/openjarvis/skills/data/ tests/skills/test_bundled_skills.py
|
||||
git add skills/
|
||||
git commit -m "refactor: move skills/builtin/ TOML data into src/openjarvis/skills/data/"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 6: Move operators/ TOML data into package
|
||||
|
||||
**Files:**
|
||||
- Move: `operators/*.toml` -> `src/openjarvis/operators/data/*.toml`
|
||||
- Modify: `src/openjarvis/cli/operators_cmd.py` (lines 35, 247, 274 — replace `Path("operators")` with package data path)
|
||||
- Test: `tests/operators/` (run existing tests)
|
||||
|
||||
**Step 1: Create data directory and move files**
|
||||
|
||||
```bash
|
||||
mkdir -p src/openjarvis/operators/data
|
||||
mv operators/*.toml src/openjarvis/operators/data/
|
||||
rmdir operators
|
||||
```
|
||||
|
||||
**Step 2: Update operators_cmd.py**
|
||||
|
||||
The CLI currently checks `Path("operators")` (cwd-relative). We need to replace this with the package data directory. In `src/openjarvis/cli/operators_cmd.py`:
|
||||
|
||||
Add a helper at the top of the file (after imports):
|
||||
```python
|
||||
def _builtin_operators_dir() -> Path:
|
||||
"""Return the path to built-in operator manifests shipped with the package."""
|
||||
return Path(__file__).resolve().parents[1] / "operators" / "data"
|
||||
```
|
||||
|
||||
Then update the three places that reference `Path("operators")`:
|
||||
|
||||
Line 35: Change `local_ops = Path("operators")` to `local_ops = _builtin_operators_dir()`
|
||||
|
||||
Line 247: Change `dirs = [DEFAULT_CONFIG_DIR / "operators", Path("operators")]` to `dirs = [DEFAULT_CONFIG_DIR / "operators", _builtin_operators_dir()]`
|
||||
|
||||
Line 274: Change `for d in [DEFAULT_CONFIG_DIR / "operators", Path("operators")]:` to `for d in [DEFAULT_CONFIG_DIR / "operators", _builtin_operators_dir()]:`
|
||||
|
||||
**Step 3: Run tests**
|
||||
|
||||
```bash
|
||||
uv run pytest tests/operators/ -v
|
||||
```
|
||||
Expected: All operator tests pass.
|
||||
|
||||
**Step 4: Commit**
|
||||
|
||||
```bash
|
||||
git add src/openjarvis/operators/data/ src/openjarvis/cli/operators_cmd.py
|
||||
git add operators/
|
||||
git commit -m "refactor: move operators/ TOML data into src/openjarvis/operators/data/"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 7: Move evals/ into src/openjarvis/evals/
|
||||
|
||||
This is the most complex migration. The `evals/` directory is a standalone Python package with internal imports like `from evals.core.config import ...`. All these must be rewritten to `from openjarvis.evals.core.config import ...`.
|
||||
|
||||
**Files:**
|
||||
- Move: `evals/` -> `src/openjarvis/evals/` (entire directory tree)
|
||||
- Modify: All `*.py` files in `src/openjarvis/evals/` (rewrite `from evals.` to `from openjarvis.evals.`)
|
||||
- Modify: `src/openjarvis/cli/eval_cmd.py` (rewrite `from evals.` imports)
|
||||
- Modify: `tests/evals/*.py` (rewrite `from evals.` imports)
|
||||
- Modify: `pyproject.toml` (update ruff per-file-ignores paths)
|
||||
|
||||
**Step 1: Move the directory**
|
||||
|
||||
```bash
|
||||
# Move evals/ into src/openjarvis/evals/
|
||||
# First remove any existing __pycache__
|
||||
find evals/ -type d -name __pycache__ -exec rm -rf {} + 2>/dev/null
|
||||
mv evals/ src/openjarvis/evals/
|
||||
```
|
||||
|
||||
**Step 2: Rewrite all internal imports in evals/**
|
||||
|
||||
Run a find-and-replace across all Python files in `src/openjarvis/evals/`:
|
||||
```bash
|
||||
find src/openjarvis/evals/ -name "*.py" -exec sed -i 's/from evals\./from openjarvis.evals./g; s/import evals\./import openjarvis.evals./g' {} +
|
||||
```
|
||||
|
||||
**Step 3: Rewrite imports in eval_cmd.py**
|
||||
|
||||
In `src/openjarvis/cli/eval_cmd.py`, replace all occurrences of:
|
||||
- `from evals.core.config import` -> `from openjarvis.evals.core.config import`
|
||||
- `from evals.cli import` -> `from openjarvis.evals.cli import`
|
||||
- `from evals.core.types import` -> `from openjarvis.evals.core.types import`
|
||||
|
||||
**Step 4: Rewrite imports in tests/evals/**
|
||||
|
||||
Run find-and-replace across test files:
|
||||
```bash
|
||||
find tests/evals/ -name "*.py" -exec sed -i 's/from evals\./from openjarvis.evals./g; s/import evals\./import openjarvis.evals./g' {} +
|
||||
```
|
||||
|
||||
**Step 5: Update pyproject.toml**
|
||||
|
||||
Change lines 134-136 from:
|
||||
```toml
|
||||
[tool.ruff.lint.per-file-ignores]
|
||||
"evals/datasets/*.py" = ["E501"]
|
||||
"evals/scorers/*.py" = ["E501"]
|
||||
```
|
||||
to:
|
||||
```toml
|
||||
[tool.ruff.lint.per-file-ignores]
|
||||
"src/openjarvis/evals/datasets/*.py" = ["E501"]
|
||||
"src/openjarvis/evals/scorers/*.py" = ["E501"]
|
||||
```
|
||||
|
||||
**Step 6: Update __main__.py**
|
||||
|
||||
In `src/openjarvis/evals/__main__.py`, the import was rewritten in step 2 but verify it now reads:
|
||||
```python
|
||||
"""Allow running as ``python -m openjarvis.evals``."""
|
||||
|
||||
from openjarvis.evals.cli import main
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
```
|
||||
|
||||
**Step 7: Run tests**
|
||||
|
||||
```bash
|
||||
uv run pytest tests/evals/ -v
|
||||
```
|
||||
Expected: All eval tests pass.
|
||||
|
||||
**Step 8: Lint check**
|
||||
|
||||
```bash
|
||||
uv run ruff check src/openjarvis/evals/ tests/evals/ src/openjarvis/cli/eval_cmd.py
|
||||
```
|
||||
Expected: No import errors.
|
||||
|
||||
**Step 9: Commit**
|
||||
|
||||
```bash
|
||||
git add src/openjarvis/evals/ tests/evals/ src/openjarvis/cli/eval_cmd.py pyproject.toml
|
||||
git commit -m "refactor: move evals/ into src/openjarvis/evals/ as proper subpackage"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 8: Remove memory/ backward-compat shim package
|
||||
|
||||
**Files:**
|
||||
- Delete: `src/openjarvis/memory/` (entire directory, 11 files)
|
||||
- Modify: `src/openjarvis/cli/memory_cmd.py` (lines 15-16)
|
||||
- Modify: `src/openjarvis/cli/ask.py` (lines 130, 304)
|
||||
- Modify: `src/openjarvis/sdk.py` (lines 36, 59-60, 391, 431)
|
||||
- Modify: `tests/tools/test_storage_stubs.py` (lines 68-69, 80)
|
||||
- Modify: `tests/tools/test_retrieval.py` (line 7)
|
||||
- Modify: `tests/cli/test_memory_cmd.py` (line 12)
|
||||
- Modify: `tests/cli/test_ask_context.py` (lines 34, 55)
|
||||
- Modify: `tests/test_integration_extended.py` (lines 402, 419)
|
||||
- Delete: `tests/memory/` (entire directory — these test the shim, not the real backends)
|
||||
|
||||
**Step 1: Update src/ imports**
|
||||
|
||||
In `src/openjarvis/cli/memory_cmd.py`, change:
|
||||
```python
|
||||
from openjarvis.memory.chunking import ChunkConfig
|
||||
from openjarvis.memory.ingest import ingest_path
|
||||
```
|
||||
to:
|
||||
```python
|
||||
from openjarvis.tools.storage.chunking import ChunkConfig
|
||||
from openjarvis.tools.storage.ingest import ingest_path
|
||||
```
|
||||
|
||||
In `src/openjarvis/cli/ask.py`, change all occurrences of:
|
||||
- `from openjarvis.memory.context import` -> `from openjarvis.tools.storage.context import`
|
||||
- `from openjarvis.memory.sqlite import` -> `from openjarvis.tools.storage.sqlite import`
|
||||
|
||||
In `src/openjarvis/sdk.py`, change:
|
||||
- `from openjarvis.memory.sqlite import SqliteMemory` -> `from openjarvis.tools.storage.sqlite import SQLiteMemory as SqliteMemory`
|
||||
- `from openjarvis.memory.chunking import ChunkConfig` -> `from openjarvis.tools.storage.chunking import ChunkConfig`
|
||||
- `from openjarvis.memory.ingest import ingest_path` -> `from openjarvis.tools.storage.ingest import ingest_path`
|
||||
- `from openjarvis.memory.context import ContextConfig, inject_context` -> `from openjarvis.tools.storage.context import ContextConfig, inject_context`
|
||||
|
||||
**Step 2: Update test imports**
|
||||
|
||||
In `tests/tools/test_storage_stubs.py`, change:
|
||||
- `from openjarvis.memory._stubs import MemoryBackend as MB` -> `from openjarvis.tools.storage._stubs import MemoryBackend as MB`
|
||||
- `from openjarvis.memory._stubs import RetrievalResult as RR` -> `from openjarvis.tools.storage._stubs import RetrievalResult as RR`
|
||||
- `from openjarvis.memory.sqlite import SQLiteMemory as S2` -> `from openjarvis.tools.storage.sqlite import SQLiteMemory as S2`
|
||||
|
||||
In `tests/tools/test_retrieval.py`, change:
|
||||
- `from openjarvis.memory._stubs import MemoryBackend, RetrievalResult` -> `from openjarvis.tools.storage._stubs import MemoryBackend, RetrievalResult`
|
||||
|
||||
In `tests/cli/test_memory_cmd.py`, change:
|
||||
- `from openjarvis.memory.sqlite import SQLiteMemory` -> `from openjarvis.tools.storage.sqlite import SQLiteMemory`
|
||||
|
||||
In `tests/cli/test_ask_context.py`, change:
|
||||
- `from openjarvis.memory.sqlite import SQLiteMemory` -> `from openjarvis.tools.storage.sqlite import SQLiteMemory`
|
||||
|
||||
In `tests/test_integration_extended.py`, change:
|
||||
- `from openjarvis.memory.sqlite import SQLiteMemory` -> `from openjarvis.tools.storage.sqlite import SQLiteMemory`
|
||||
- `from openjarvis.memory.bm25 import BM25Memory` -> `from openjarvis.tools.storage.bm25 import BM25Memory`
|
||||
|
||||
**Step 3: Update tests/memory/ imports and move to tests/tools/**
|
||||
|
||||
The `tests/memory/` files test the same backends as `tests/tools/test_storage_*.py`. Since they import from the shim, we need to rewrite their imports and merge them. However, since `tests/tools/` already has storage tests, the simplest approach is:
|
||||
|
||||
Rewrite all `tests/memory/*.py` imports from `openjarvis.memory.*` to `openjarvis.tools.storage.*`:
|
||||
```bash
|
||||
find tests/memory/ -name "*.py" -exec sed -i 's/from openjarvis\.memory\./from openjarvis.tools.storage./g' {} +
|
||||
```
|
||||
|
||||
Then move the directory to be a subdirectory of tests/tools:
|
||||
```bash
|
||||
mv tests/memory tests/tools/memory_tests
|
||||
```
|
||||
|
||||
Actually, the cleanest approach: just rewrite imports in-place and keep `tests/memory/` as-is. The tests are valuable and test different aspects than `tests/tools/`.
|
||||
|
||||
**Step 4: Delete the shim package**
|
||||
|
||||
```bash
|
||||
rm -rf src/openjarvis/memory/
|
||||
```
|
||||
|
||||
**Step 5: Run full test suite**
|
||||
|
||||
```bash
|
||||
uv run pytest tests/memory/ tests/tools/ tests/cli/ tests/sdk/ -v
|
||||
```
|
||||
Expected: All tests pass.
|
||||
|
||||
**Step 6: Commit**
|
||||
|
||||
```bash
|
||||
git add -A src/openjarvis/memory/ src/openjarvis/cli/memory_cmd.py src/openjarvis/cli/ask.py src/openjarvis/sdk.py
|
||||
git add tests/memory/ tests/tools/ tests/cli/ tests/test_integration_extended.py
|
||||
git commit -m "refactor: remove memory/ backward-compat shims, use tools.storage directly"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 9: Consolidate engine wrappers into single file
|
||||
|
||||
**Files:**
|
||||
- Delete: `src/openjarvis/engine/vllm.py`
|
||||
- Delete: `src/openjarvis/engine/sglang.py`
|
||||
- Delete: `src/openjarvis/engine/llamacpp.py`
|
||||
- Delete: `src/openjarvis/engine/mlx.py`
|
||||
- Delete: `src/openjarvis/engine/lmstudio.py`
|
||||
- Create: `src/openjarvis/engine/openai_compat_engines.py`
|
||||
- Modify: `src/openjarvis/engine/__init__.py` (update imports)
|
||||
- Test: Run `tests/engine/` tests
|
||||
|
||||
**Step 1: Write the consolidated engine file**
|
||||
|
||||
Create `src/openjarvis/engine/openai_compat_engines.py`:
|
||||
|
||||
```python
|
||||
"""Data-driven registration of OpenAI-compatible inference engines."""
|
||||
|
||||
from openjarvis.core.registry import EngineRegistry
|
||||
from openjarvis.engine._openai_compat import _OpenAICompatibleEngine
|
||||
|
||||
_ENGINES = {
|
||||
"vllm": ("VLLMEngine", "http://localhost:8000"),
|
||||
"sglang": ("SGLangEngine", "http://localhost:30000"),
|
||||
"llamacpp": ("LlamaCppEngine", "http://localhost:8080"),
|
||||
"mlx": ("MLXEngine", "http://localhost:8080"),
|
||||
"lmstudio": ("LMStudioEngine", "http://localhost:1234"),
|
||||
}
|
||||
|
||||
for _key, (_cls_name, _default_host) in _ENGINES.items():
|
||||
_cls = type(
|
||||
_cls_name,
|
||||
(_OpenAICompatibleEngine,),
|
||||
{"engine_id": _key, "_default_host": _default_host},
|
||||
)
|
||||
EngineRegistry.register(_key)(_cls)
|
||||
globals()[_cls_name] = _cls
|
||||
|
||||
__all__ = [name for name, _ in _ENGINES.values()]
|
||||
```
|
||||
|
||||
**Step 2: Update engine/__init__.py**
|
||||
|
||||
Replace lines 5-11 (the 5 individual imports) with a single import:
|
||||
|
||||
Change from:
|
||||
```python
|
||||
import openjarvis.engine.llamacpp # noqa: F401
|
||||
import openjarvis.engine.mlx # noqa: F401
|
||||
|
||||
# Import engine modules to trigger @EngineRegistry.register() decorators
|
||||
import openjarvis.engine.ollama # noqa: F401
|
||||
import openjarvis.engine.sglang # noqa: F401
|
||||
import openjarvis.engine.vllm # noqa: F401
|
||||
```
|
||||
|
||||
To:
|
||||
```python
|
||||
# Import engine modules to trigger @EngineRegistry.register() decorators
|
||||
import openjarvis.engine.ollama # noqa: F401
|
||||
import openjarvis.engine.openai_compat_engines # noqa: F401
|
||||
```
|
||||
|
||||
**Step 3: Delete the 5 individual wrapper files**
|
||||
|
||||
```bash
|
||||
rm src/openjarvis/engine/vllm.py
|
||||
rm src/openjarvis/engine/sglang.py
|
||||
rm src/openjarvis/engine/llamacpp.py
|
||||
rm src/openjarvis/engine/mlx.py
|
||||
rm src/openjarvis/engine/lmstudio.py
|
||||
```
|
||||
|
||||
**Step 4: Run tests**
|
||||
|
||||
```bash
|
||||
uv run pytest tests/engine/ -v
|
||||
```
|
||||
Expected: All engine tests pass.
|
||||
|
||||
**Step 5: Commit**
|
||||
|
||||
```bash
|
||||
git add src/openjarvis/engine/
|
||||
git commit -m "refactor: consolidate 5 OpenAI-compat engine wrappers into single data-driven file"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 10: Refactor repetitive __init__.py imports
|
||||
|
||||
**Files:**
|
||||
- Modify: `src/openjarvis/channels/__init__.py`
|
||||
- Modify: `src/openjarvis/engine/__init__.py` (optional engines section)
|
||||
- Test: Run `tests/channels/` and `tests/engine/` tests
|
||||
|
||||
**Step 1: Refactor channels/__init__.py**
|
||||
|
||||
Replace the entire file content with:
|
||||
|
||||
```python
|
||||
"""Channel abstraction for multi-platform messaging."""
|
||||
|
||||
import importlib
|
||||
|
||||
from openjarvis.channels._stubs import (
|
||||
BaseChannel,
|
||||
ChannelHandler,
|
||||
ChannelMessage,
|
||||
ChannelStatus,
|
||||
)
|
||||
|
||||
# Trigger registration of built-in channels.
|
||||
# Each module uses @ChannelRegistry.register() — importing is sufficient.
|
||||
_CHANNEL_MODULES = [
|
||||
"telegram",
|
||||
"discord_channel",
|
||||
"slack",
|
||||
"webhook",
|
||||
"email_channel",
|
||||
"whatsapp",
|
||||
"signal_channel",
|
||||
"google_chat",
|
||||
"irc_channel",
|
||||
"webchat",
|
||||
"teams",
|
||||
"matrix_channel",
|
||||
"mattermost",
|
||||
"feishu",
|
||||
"bluebubbles",
|
||||
"whatsapp_baileys",
|
||||
"line_channel",
|
||||
"viber_channel",
|
||||
"messenger_channel",
|
||||
"reddit_channel",
|
||||
"mastodon_channel",
|
||||
"xmpp_channel",
|
||||
"rocketchat_channel",
|
||||
"zulip_channel",
|
||||
"twitch_channel",
|
||||
"nostr_channel",
|
||||
]
|
||||
|
||||
for _mod in _CHANNEL_MODULES:
|
||||
try:
|
||||
importlib.import_module(f".{_mod}", __name__)
|
||||
except ImportError:
|
||||
pass
|
||||
|
||||
__all__ = [
|
||||
"BaseChannel",
|
||||
"ChannelHandler",
|
||||
"ChannelMessage",
|
||||
"ChannelStatus",
|
||||
]
|
||||
```
|
||||
|
||||
**Step 2: Refactor optional engine imports in engine/__init__.py**
|
||||
|
||||
Replace lines 19-29 with:
|
||||
|
||||
```python
|
||||
# Optional engines — only register if their SDK deps are present
|
||||
for _optional in ("cloud", "litellm"):
|
||||
try:
|
||||
importlib.import_module(f".{_optional}", __name__)
|
||||
except ImportError:
|
||||
pass
|
||||
```
|
||||
|
||||
And add `import importlib` at the top.
|
||||
|
||||
**Step 3: Run tests**
|
||||
|
||||
```bash
|
||||
uv run pytest tests/channels/ tests/engine/ -v
|
||||
```
|
||||
Expected: All tests pass.
|
||||
|
||||
**Step 4: Commit**
|
||||
|
||||
```bash
|
||||
git add src/openjarvis/channels/__init__.py src/openjarvis/engine/__init__.py
|
||||
git commit -m "refactor: replace repetitive try/except imports with loop-based auto-registration"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 11: Update pyproject.toml package data includes
|
||||
|
||||
**Files:**
|
||||
- Modify: `pyproject.toml`
|
||||
|
||||
**Step 1: Add package data includes**
|
||||
|
||||
The TOML data files that moved into `src/openjarvis/` need to be included in wheel builds. In `pyproject.toml`, after the existing `[tool.hatch.build.targets.wheel.force-include]` section, add the data directories.
|
||||
|
||||
Actually, since all the data directories are under `src/openjarvis/` and `packages = ["src/openjarvis"]` is already set, hatch will include them automatically as long as they are inside the package tree. But TOML files are non-Python files, so we need to tell hatch to include them.
|
||||
|
||||
Add to pyproject.toml after line 114:
|
||||
|
||||
```toml
|
||||
[tool.hatch.build.targets.wheel.shared-data]
|
||||
|
||||
[tool.hatch.build]
|
||||
include = [
|
||||
"src/openjarvis/**/*.py",
|
||||
"src/openjarvis/**/*.toml",
|
||||
"src/openjarvis/**/*.md",
|
||||
"src/openjarvis/agents/claude_code_runner/**",
|
||||
"src/openjarvis/channels/whatsapp_baileys_bridge/**",
|
||||
]
|
||||
```
|
||||
|
||||
Wait — hatchling by default includes all files under `packages`. Non-Python files inside the package are included by default. No change needed here. Verify with:
|
||||
|
||||
```bash
|
||||
uv run python -c "from pathlib import Path; d = Path('src/openjarvis/recipes/data'); print(list(d.glob('*.toml')))"
|
||||
```
|
||||
|
||||
**Step 2: Commit if any changes were needed**
|
||||
|
||||
If no pyproject.toml changes needed, skip this task.
|
||||
|
||||
---
|
||||
|
||||
### Task 12: Update CLAUDE.md to reflect new structure
|
||||
|
||||
**Files:**
|
||||
- Modify: `CLAUDE.md`
|
||||
|
||||
**Step 1: Update all references to moved paths**
|
||||
|
||||
In CLAUDE.md, make the following updates:
|
||||
|
||||
1. Update the `python -m evals` command references to `python -m openjarvis.evals`
|
||||
2. Remove references to top-level `recipes/`, `templates/`, `skills/`, `operators/` directories
|
||||
3. Update Docker command references to use `deploy/docker/` paths
|
||||
4. Update the architecture section to reflect that data files are now package data
|
||||
5. Remove mentions of `memory/` backward-compat shims
|
||||
6. Remove the 5 individual engine wrapper files from architecture description
|
||||
|
||||
**Step 2: Run linting to verify no broken references**
|
||||
|
||||
```bash
|
||||
uv run ruff check src/ tests/
|
||||
```
|
||||
|
||||
**Step 3: Run full test suite**
|
||||
|
||||
```bash
|
||||
uv run pytest tests/ -v --tb=short 2>&1 | tail -20
|
||||
```
|
||||
Expected: ~3240 tests pass.
|
||||
|
||||
**Step 4: Commit**
|
||||
|
||||
```bash
|
||||
git add CLAUDE.md
|
||||
git commit -m "docs: update CLAUDE.md to reflect simplified repository structure"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 13: Final verification
|
||||
|
||||
**Step 1: Run full test suite**
|
||||
|
||||
```bash
|
||||
uv run pytest tests/ -v --tb=short 2>&1 | tail -30
|
||||
```
|
||||
Expected: ~3240 tests pass, ~44 skipped.
|
||||
|
||||
**Step 2: Run linter**
|
||||
|
||||
```bash
|
||||
uv run ruff check src/ tests/
|
||||
```
|
||||
Expected: Clean.
|
||||
|
||||
**Step 3: Verify root structure**
|
||||
|
||||
```bash
|
||||
ls -la | grep -v __pycache__ | grep -v '^\.'
|
||||
```
|
||||
Expected: 21 or fewer items at root (down from 32).
|
||||
|
||||
**Step 4: Verify package data is accessible**
|
||||
|
||||
```bash
|
||||
uv run python -c "
|
||||
from openjarvis.recipes.loader import discover_recipes
|
||||
from openjarvis.templates.agent_templates import discover_templates
|
||||
print(f'Recipes: {len(discover_recipes())}')
|
||||
print(f'Templates: {len(discover_templates())}')
|
||||
print('All data accessible.')
|
||||
"
|
||||
```
|
||||
Expected: Recipes: 3, Templates: 15, All data accessible.
|
||||
@@ -1,280 +0,0 @@
|
||||
# Speech-to-Text Design
|
||||
|
||||
**Date:** 2026-03-03
|
||||
**Status:** Approved
|
||||
**Scope:** Add voice input (speech-to-text) to OpenJarvis — desktop app and browser
|
||||
|
||||
## Overview
|
||||
|
||||
Add a new Speech subsystem to OpenJarvis that lets users speak commands instead of typing them. The system transcribes audio to text using local open-source models (Faster-Whisper) by default, with cloud backends (OpenAI, Deepgram) as alternatives. Transcribed text is inserted into the input box for user review before sending.
|
||||
|
||||
## Design Decisions
|
||||
|
||||
| Decision | Choice | Rationale |
|
||||
|----------|--------|-----------|
|
||||
| Scope | STT only (no TTS) | Ship voice input first; TTS follows later |
|
||||
| Runtime | Separate process | Avoid VRAM conflicts with the LLM engine |
|
||||
| Surfaces | Desktop app + browser | Both use the same React frontend |
|
||||
| UX mode | Record-then-transcribe | Simpler to implement, works with all backends |
|
||||
| Architecture | New Speech subsystem | Fits OpenJarvis patterns (ABC + registry + decorator) |
|
||||
|
||||
## Architecture
|
||||
|
||||
### New Module: `src/openjarvis/speech/`
|
||||
|
||||
```
|
||||
src/openjarvis/speech/
|
||||
├── __init__.py # Imports, ensure_registered()
|
||||
├── _stubs.py # SpeechBackend ABC, TranscriptionResult dataclass
|
||||
├── faster_whisper.py # FasterWhisperBackend (local, default)
|
||||
├── whisper_cpp.py # WhisperCppBackend (local, llama.cpp ecosystem)
|
||||
├── openai_whisper.py # OpenAIWhisperBackend (cloud)
|
||||
├── deepgram.py # DeepgramBackend (cloud)
|
||||
└── _discovery.py # Auto-discover available backend (local preferred)
|
||||
```
|
||||
|
||||
### Core Types (`_stubs.py`)
|
||||
|
||||
```python
|
||||
@dataclass
|
||||
class Segment:
|
||||
text: str
|
||||
start: float # Start time in seconds
|
||||
end: float # End time in seconds
|
||||
confidence: float | None
|
||||
|
||||
@dataclass
|
||||
class TranscriptionResult:
|
||||
text: str # The transcribed text
|
||||
language: str | None # Detected language code (e.g., "en")
|
||||
confidence: float | None # Overall confidence [0, 1]
|
||||
duration_seconds: float # Audio duration
|
||||
segments: list[Segment] # Word/phrase-level timing (optional)
|
||||
|
||||
class SpeechBackend(ABC):
|
||||
backend_id: str
|
||||
|
||||
@abstractmethod
|
||||
def transcribe(self, audio: bytes, *, format: str = "wav",
|
||||
language: str | None = None) -> TranscriptionResult: ...
|
||||
|
||||
@abstractmethod
|
||||
def health(self) -> bool: ...
|
||||
|
||||
@abstractmethod
|
||||
def supported_formats(self) -> list[str]: ...
|
||||
```
|
||||
|
||||
### Registry
|
||||
|
||||
New `SpeechRegistry` added to `core/registry.py` using `RegistryBase[T]`. Backends register via `@SpeechRegistry.register("faster-whisper")`.
|
||||
|
||||
### Discovery (`_discovery.py`)
|
||||
|
||||
Priority order (local-first):
|
||||
1. Faster-Whisper (if `faster-whisper` package installed)
|
||||
2. WhisperCpp (if `whisper-cpp-python` package installed)
|
||||
3. OpenAI Whisper API (if `OPENAI_API_KEY` set)
|
||||
4. Deepgram (if `DEEPGRAM_API_KEY` set)
|
||||
|
||||
Function: `get_speech_backend(config) -> SpeechBackend | None`
|
||||
|
||||
### Config
|
||||
|
||||
New `[speech]` section in `JarvisConfig`:
|
||||
|
||||
```toml
|
||||
[speech]
|
||||
backend = "auto" # "auto", "faster-whisper", "whisper-cpp", "openai", "deepgram"
|
||||
model = "base" # Whisper model size: tiny, base, small, medium, large-v3
|
||||
language = "" # Empty = auto-detect
|
||||
device = "auto" # "auto", "cpu", "cuda"
|
||||
compute_type = "float16" # "float16", "int8", "float32"
|
||||
```
|
||||
|
||||
New `SpeechConfig` dataclass in `core/config.py`.
|
||||
|
||||
## API Layer
|
||||
|
||||
### New Endpoints
|
||||
|
||||
```
|
||||
POST /v1/speech/transcribe
|
||||
Content-Type: multipart/form-data
|
||||
Body: audio file (field name: "file")
|
||||
Optional form fields: language, model
|
||||
|
||||
Response 200: {
|
||||
"text": "Hello, what's the weather like?",
|
||||
"language": "en",
|
||||
"confidence": 0.94,
|
||||
"duration_seconds": 2.3
|
||||
}
|
||||
|
||||
GET /v1/speech/backends
|
||||
Response 200: {
|
||||
"backends": ["faster-whisper"],
|
||||
"active": "faster-whisper",
|
||||
"model": "base"
|
||||
}
|
||||
|
||||
GET /v1/speech/health
|
||||
Response 200: {"available": true, "backend": "faster-whisper", "model": "base"}
|
||||
Response 200: {"available": false, "reason": "No speech backend installed"}
|
||||
```
|
||||
|
||||
### Server Wiring
|
||||
|
||||
- `SystemBuilder.with_speech(backend=..., model=...)` — configure speech
|
||||
- `JarvisSystem.speech` — active `SpeechBackend` instance or `None`
|
||||
- Speech backend initializes lazily on first `transcribe()` call
|
||||
- Routes added to `server/api_routes.py`
|
||||
|
||||
## Frontend Integration
|
||||
|
||||
### Web Frontend (`frontend/src/`)
|
||||
|
||||
**New component: `MicButton.tsx`**
|
||||
- Sits next to the Send button in `InputArea.tsx`
|
||||
- States: idle (mic icon), recording (pulsing red), transcribing (spinner)
|
||||
- Uses `MediaRecorder` API to capture audio
|
||||
- Sends audio blob as multipart to `/v1/speech/transcribe`
|
||||
- Appends transcribed text to input textarea
|
||||
- Hidden if `/v1/speech/health` returns `available: false`
|
||||
|
||||
**New hook: `useSpeech.ts`**
|
||||
- Manages microphone permissions, `MediaRecorder` lifecycle
|
||||
- `startRecording()`, `stopRecording()`, `isRecording`, `isTranscribing`, `error`
|
||||
- Checks `navigator.mediaDevices` support
|
||||
|
||||
**Changes to existing components:**
|
||||
- `InputArea.tsx` — add `MicButton` next to send button
|
||||
- `App.tsx` — check speech availability on load
|
||||
|
||||
### Desktop App (`desktop/`)
|
||||
|
||||
Same React UI works in the Tauri WebView.
|
||||
|
||||
**New Tauri command: `transcribe_audio`** in `lib.rs` — proxies multipart POST to backend for cases where WebView fetch has CORS issues.
|
||||
|
||||
### Audio Format
|
||||
|
||||
Record as WebM/Opus (native browser format). Faster-Whisper handles WebM directly. Server-side conversion to WAV as fallback if a backend requires it.
|
||||
|
||||
### User Flow
|
||||
|
||||
```
|
||||
User clicks mic button
|
||||
→ Browser requests microphone permission (first time)
|
||||
→ Recording starts (button pulses red)
|
||||
User clicks mic button again (or releases)
|
||||
→ Recording stops
|
||||
→ Button shows spinner
|
||||
→ Audio blob sent to POST /v1/speech/transcribe
|
||||
→ Response text inserted into input textarea
|
||||
→ User reviews/edits text
|
||||
→ User clicks Send (normal flow)
|
||||
```
|
||||
|
||||
## Backend Implementations
|
||||
|
||||
### Faster-Whisper (Default Local)
|
||||
|
||||
```python
|
||||
@SpeechRegistry.register("faster-whisper")
|
||||
class FasterWhisperBackend(SpeechBackend):
|
||||
backend_id = "faster-whisper"
|
||||
# Uses CTranslate2-based Faster-Whisper (4x faster than original Whisper)
|
||||
# Lazy model loading on first transcribe() call
|
||||
# Model sizes: tiny (39M), base (74M), small (244M), medium (769M), large-v3 (1.5B)
|
||||
# GPU: device="cuda", compute_type="float16" or "int8"
|
||||
# CPU: device="cpu", compute_type="int8"
|
||||
```
|
||||
|
||||
Optional dep: `openjarvis[speech]` → `faster-whisper>=1.0`
|
||||
|
||||
### OpenAI Whisper API (Cloud)
|
||||
|
||||
```python
|
||||
@SpeechRegistry.register("openai")
|
||||
class OpenAIWhisperBackend(SpeechBackend):
|
||||
backend_id = "openai"
|
||||
# Uses openai.audio.transcriptions.create()
|
||||
# Requires OPENAI_API_KEY
|
||||
# Uses existing openai dependency
|
||||
```
|
||||
|
||||
### Deepgram (Cloud)
|
||||
|
||||
```python
|
||||
@SpeechRegistry.register("deepgram")
|
||||
class DeepgramBackend(SpeechBackend):
|
||||
backend_id = "deepgram"
|
||||
# Uses deepgram-sdk
|
||||
# Requires DEEPGRAM_API_KEY
|
||||
```
|
||||
|
||||
Optional dep: `openjarvis[speech-deepgram]` → `deepgram-sdk>=3.0`
|
||||
|
||||
## Error Handling
|
||||
|
||||
| Scenario | Behavior |
|
||||
|----------|----------|
|
||||
| Mic permission denied | Toast error in frontend, no crash |
|
||||
| No speech backend available | Mic button hidden, health endpoint returns `available: false` |
|
||||
| Audio too short / silence | Return empty text with low confidence |
|
||||
| Model loading failure | Health returns false, error logged |
|
||||
| Network failure (cloud) | Error response, frontend shows retry option |
|
||||
| Unsupported audio format | Server converts to WAV, or returns 400 with message |
|
||||
|
||||
## Testing
|
||||
|
||||
### Backend Tests
|
||||
- `tests/speech/test_faster_whisper.py` — mock `faster_whisper` import, test transcribe with synthetic WAV
|
||||
- `tests/speech/test_openai_whisper.py` — mock `openai` client, test transcribe
|
||||
- `tests/speech/test_deepgram.py` — mock `deepgram` client, test transcribe
|
||||
- `tests/speech/test_discovery.py` — test auto-discovery priority order
|
||||
|
||||
### API Tests
|
||||
- `tests/server/test_speech_routes.py` — test endpoints with mocked backend
|
||||
|
||||
### Config Tests
|
||||
- Test `SpeechConfig` defaults, TOML parsing, `[speech]` section
|
||||
|
||||
All optional-dep backends behind `pytest.importorskip()`.
|
||||
|
||||
## Optional Dependencies
|
||||
|
||||
```toml
|
||||
[project.optional-dependencies]
|
||||
speech = ["faster-whisper>=1.0"]
|
||||
speech-deepgram = ["deepgram-sdk>=3.0"]
|
||||
```
|
||||
|
||||
## Files Changed (Estimated)
|
||||
|
||||
### New Files (~12)
|
||||
- `src/openjarvis/speech/__init__.py`
|
||||
- `src/openjarvis/speech/_stubs.py`
|
||||
- `src/openjarvis/speech/faster_whisper.py`
|
||||
- `src/openjarvis/speech/openai_whisper.py`
|
||||
- `src/openjarvis/speech/deepgram.py`
|
||||
- `src/openjarvis/speech/_discovery.py`
|
||||
- `frontend/src/components/MicButton.tsx`
|
||||
- `frontend/src/hooks/useSpeech.ts`
|
||||
- `tests/speech/test_faster_whisper.py`
|
||||
- `tests/speech/test_openai_whisper.py`
|
||||
- `tests/speech/test_deepgram.py`
|
||||
- `tests/speech/test_discovery.py`
|
||||
- `tests/server/test_speech_routes.py`
|
||||
|
||||
### Modified Files (~8)
|
||||
- `src/openjarvis/core/registry.py` — add `SpeechRegistry`
|
||||
- `src/openjarvis/core/config.py` — add `SpeechConfig`, `[speech]` section
|
||||
- `src/openjarvis/system.py` — add `with_speech()`, wire speech backend
|
||||
- `src/openjarvis/server/api_routes.py` — add speech endpoints
|
||||
- `frontend/src/components/InputArea.tsx` — add MicButton
|
||||
- `frontend/src/App.tsx` — check speech availability
|
||||
- `desktop/src-tauri/src/lib.rs` — add `transcribe_audio` command
|
||||
- `pyproject.toml` — add `[speech]` and `[speech-deepgram]` extras
|
||||
- `tests/conftest.py` — add SpeechRegistry to `_clean_registries`
|
||||
File diff suppressed because it is too large
Load Diff
Reference in New Issue
Block a user