OpenJarvis/NOTES.md

OpenJarvis Development Notes

Living document tracking implementation progress, testing state, lessons learned, dead ends, and practices for ongoing development. Updated across sessions.

---

Current State (2026-02-21)

• Version: 1.0.0 (trace system added, targeting v1.1)
• All 6 roadmap phases complete (Phase 0 through Phase 5) + Phase 6 trace system in progress
• Tests: 576 passed, 8 skipped, 0 failures
• Lint: ruff clean (select = ["E", "F", "I", "W"])
• Source files: 76 Python files in src/openjarvis/
• Test files: 78 Python files in tests/
• Python: 3.13 (compatible with 3.10+)
• Package manager: uv with hatchling build backend

8 Skipped Tests (Optional Dependencies)

Test	Missing Dep	Install Extra
tests/memory/test_bm25.py	rank_bm25	openjarvis[memory-bm25]
tests/memory/test_colbert.py	colbert	openjarvis[memory-colbert]
tests/memory/test_embeddings.py	sentence_transformers	openjarvis[memory-faiss]
tests/memory/test_faiss.py	faiss	openjarvis[memory-faiss]
tests/server/test_models_pydantic.py	pydantic	openjarvis[server]
tests/server/test_routes.py	fastapi	openjarvis[server]
tests/test_integration.py:165	fastapi	openjarvis[server]
tests/test_integration.py:190	fastapi	openjarvis[server]

---

Phase Completion Log

Phase	Version	Deliverables	Test Count (cumulative)
Phase 0	v0.1	Scaffolding, registries, core types, config, CLI skeleton, event bus	~60
Phase 1	v0.2	Intelligence + Inference — jarvis ask end-to-end, heuristic router, engine discovery, basic telemetry	~160
Phase 2	v0.3	Memory — SQLite/FAISS/ColBERT/BM25/Hybrid backends, document ingest pipeline, context injection, jarvis memory CLI	~270
Phase 3	v0.4	Agents (Simple/Orchestrator/Custom/OpenClaw stub), tool system (Calculator/Think/Retrieval/LLM/FileRead), OpenAI-compatible API server, jarvis serve	~360
Phase 4	v0.5	Learning — HeuristicRouter, HeuristicRewardFunction, GRPORouterPolicy stub, TelemetryAggregator, jarvis telemetry CLI, --router CLI option	~432
Phase 5	v1.0	SDK (Jarvis class), OpenClaw infrastructure (protocol/transport/plugin), benchmarks (jarvis bench), Docker, docs	~520
Phase 6	v1.1	Trace system (TraceStore, TraceCollector, TraceAnalyzer), trace-driven learning (TraceDrivenPolicy)	~576

---

Architecture Quick Reference

Directory Layout

src/openjarvis/
├── __init__.py          # __version__ = "1.0.0", exports Jarvis, MemoryHandle
├── sdk.py               # Python SDK: Jarvis class + MemoryHandle
├── core/
│   ├── registry.py      # RegistryBase[T] + 7 typed registries
│   ├── types.py         # Message, Conversation, ModelSpec, ToolResult, TelemetryRecord
│   ├── config.py        # JarvisConfig dataclass hierarchy, TOML loader
│   └── events.py        # EventBus pub/sub (synchronous)
├── intelligence/        # ModelRegistry, HeuristicRouter, model catalog
├── traces/              # TraceStore, TraceCollector, TraceAnalyzer
├── learning/            # RouterPolicyRegistry, HeuristicRouter, TraceDrivenPolicy, GRPO stub
├── memory/              # SQLite/FAISS/ColBERT/BM25/Hybrid backends, chunking, ingest
├── agents/              # Simple/Orchestrator/Custom/OpenClaw agents + protocol/transport
├── engine/              # Ollama/vLLM/llama.cpp/Cloud engine wrappers
├── tools/               # Calculator/Think/Retrieval/LLM/FileRead tools
├── bench/               # Latency/Throughput benchmarks, BenchmarkSuite
├── telemetry/           # TelemetryStore, TelemetryAggregator, instrumented_generate
├── server/              # FastAPI OpenAI-compatible API server
└── cli/                 # Click CLI: init, ask, serve, model, memory, telemetry, bench

7 Registries

All use RegistryBase[T] with @XRegistry.register("name") or register_value():

1. ModelRegistry — ModelSpec objects
2. EngineRegistry — InferenceEngine implementations
3. MemoryRegistry — MemoryBackend implementations
4. AgentRegistry — BaseAgent implementations
5. ToolRegistry — BaseTool implementations
6. RouterPolicyRegistry — RouterPolicy implementations
7. BenchmarkRegistry — BaseBenchmark implementations

---

Patterns and Practices

The ensure_registered() Pattern

Problem: The _clean_registries autouse fixture in tests/conftest.py calls .clear() on every registry before each test. Module-level @XRegistry.register("name") decorators only fire once at import time (Python caches modules in sys.modules). After registry clearing, the decorations never re-fire, leaving registries empty for subsequent tests.

Solution: Use lazy registration via ensure_registered():

# src/openjarvis/bench/latency.py
_registered = False

def ensure_registered() -> None:
    global _registered
    if _registered:
        return
    from openjarvis.core.registry import BenchmarkRegistry
    if not BenchmarkRegistry.contains("latency"):
        BenchmarkRegistry.register_value("latency", LatencyBenchmark)
    _registered = True

Then in __init__.py:

def ensure_registered() -> None:
    from openjarvis.bench.latency import ensure_registered as _reg_latency
    _reg_latency()

And in test files, use an autouse fixture:

@pytest.fixture(autouse=True)
def _register_latency():
    from openjarvis.bench import ensure_registered
    ensure_registered()

Where this pattern is used: bench/latency.py, bench/throughput.py, learning/heuristic_policy.py, learning/grpo_policy.py, learning/heuristic_reward.py

Where this pattern is NOT needed: Agents, engines, memory backends, and tools use @register decorators that work fine because their test files explicitly import and re-register as needed, or the test module import triggers registration.

Test Infrastructure

• tests/conftest.py — _clean_registries autouse fixture clears all 7 registries + clears EventBus default listeners before each test. Critical for test isolation.
• Mock engine pattern — Almost every test that touches the engine layer uses a MagicMock() with .engine_id, .health(), .list_models(), .generate() stubbed:

  def _make_engine(content="Hello"):
      engine = MagicMock()
      engine.engine_id = "mock"
      engine.health.return_value = True
      engine.list_models.return_value = ["test-model"]
      engine.generate.return_value = {
          "content": content,
          "usage": {"prompt_tokens": 5, "completion_tokens": 3, "total_tokens": 8},
          "model": "test-model",
          "finish_reason": "stop",
      }
      return engine
  

• CLI tests use Click's CliRunner with patch("openjarvis.cli.X.get_engine", ...) to mock the engine layer.
• Memory tests use tmp_path fixture for SQLite DB paths and test files.
• Optional dep tests use pytest.importorskip("module_name") at module level.

Config Defaults

JarvisConfig() with no arguments produces sane defaults:

• Engine: auto-discover (Ollama, vLLM, llama.cpp, cloud in priority order)
• Memory: sqlite backend, ~/.openjarvis/memory.db
• Agent: simple (no Node.js dependency)
• Intelligence: qwen3:8b default, qwen3:0.6b fallback
• Telemetry: enabled, ~/.openjarvis/telemetry.db
• Learning: heuristic default policy

File Naming Conventions

• ABCs and shared dataclasses: _stubs.py (e.g., agents/_stubs.py, bench/_stubs.py, tools/_stubs.py)
• Internal helpers: _discovery.py, _base.py (underscore prefix)
• CLI commands: *_cmd.py (e.g., bench_cmd.py, telemetry_cmd.py, memory_cmd.py)
• Test files mirror source: tests/agents/test_openclaw.py tests src/openjarvis/agents/openclaw.py

Import Structure

• Package __init__.py files import submodules to trigger registration
• Try/except around optional dependency imports:

  try:
      from openjarvis.engine.ollama import OllamaEngine  # noqa: F401
  except ImportError:
      pass
  

• Top-level openjarvis/__init__.py exports: Jarvis, MemoryHandle, __version__

---

Dead Ends and Gotchas

1. @register Decorator vs. ensure_registered()

Dead end: Initially used @BenchmarkRegistry.register("latency") class decorator in bench/latency.py. This caused ~10 test failures because:

• Registry cleared between tests by conftest.py
• Module already in sys.modules, so import openjarvis.bench is a no-op on second import
• Registry stays empty after clearing

Fix: Switched to ensure_registered() pattern (see above). This is the pattern already used by learning/ modules.

Rule of thumb: If a module is imported at package init time AND its registry gets cleared in tests, use ensure_registered(). If registration only happens in test fixtures or explicit calls, @register is fine.

2. Chunk Attribute Names

memory/chunking.py Chunk dataclass uses content (not text). ChunkConfig uses chunk_overlap (not overlap). Easy to get wrong because these aren't obvious from the field names alone. Always read _stubs.py or the actual dataclass before using.

3. Test Content Size for Chunking

ChunkConfig.min_chunk_size=50 tokens by default. A test string like "This is test content." produces 0 chunks. Use at least ~100 words:

words = " ".join(f"word{i}" for i in range(100))

4. Version String Locations

Version is defined in three places that must stay in sync:

1. src/openjarvis/__init__.py — __version__ = "1.0.0"
2. pyproject.toml — version = "1.0.0"
3. src/openjarvis/server/app.py — FastAPI version="1.0.0" constructor arg

Tests that check version: tests/cli/test_cli.py::test_version_flag

5. Server Import Guards

The server/ module requires fastapi, uvicorn, pydantic. These are behind the [server] optional extra. All test files that touch server code use pytest.importorskip("fastapi"). The server __init__.py wraps imports in try/except.

6. patch() Targets for Engine Mocking

When mocking get_engine in CLI tests, the patch target must be the importing module, not the source module:

# CORRECT — patches where it's imported
patch("openjarvis.cli.bench_cmd.get_engine", return_value=("mock", engine))

# WRONG — patches the source, doesn't affect the already-imported reference
patch("openjarvis.engine._discovery.get_engine", return_value=("mock", engine))

Same for SDK tests: patch("openjarvis.sdk.get_engine", ...).

7. EventBus Clearing

EventBus() creates a new instance each time, but EventBus._default_listeners is a class variable. The conftest.py fixture resets it. If tests subscribe to events, subscriptions won't persist across tests.

8. Module Shadowing in CLI Package

In cli/__init__.py, from openjarvis.cli.ask import ask imports the Click command. This shadows the module name. When you try mock.patch("openjarvis.cli.ask.get_engine"), Python resolves openjarvis.cli.ask as the Click command (via attribute lookup on the package), not the module.

Fix: Use importlib.import_module("openjarvis.cli.ask") to get the actual module object, then mock.patch.object(module, "get_engine").

---

Post-v1.0: Unimplemented Ideas from VISION.md

These are mentioned in VISION.md but not in the roadmap phases. They represent future work:

Learning / Router

• Learned router via GRPO (Group Relative Policy Optimization) — GRPORouterPolicy is a stub
• Preference learning from user feedback
• Continual fine-tuning on accumulated trajectories
• Multi-objective optimization: quality vs. latency vs. energy vs. cost

Memory

• ConversationMemory — sliding window with automatic summarization of older turns
• Personal Notes — user-created persistent notes and preferences
• Episodic Memory — records of past interactions, tool uses, and outcomes
• Vector DB adapters (Qdrant, ChromaDB) for users with existing infrastructure

Tools

• WebSearch tool (Tavily, SearXNG, DuckDuckGo)
• CodeInterpreter tool (sandboxed Python execution)
• FileWrite tool (safe file writing with path validation)
• MCP (Model Context Protocol) compatibility

Engines

• SGLang engine backend (structured generation, constrained decoding)
• MLX engine backend (Apple Silicon native, Metal acceleration)
• Complete vLLM integration (tensor parallelism config, multi-GPU)

OpenClaw

• Full OpenClaw gateway integration (WebSocket, :18789)
• OpenClaw skill composition
• Context compaction in OpenClaw agent
• openjarvis-openclaw as separate plugin package (currently inline)

Infrastructure

• Documentation site (MkDocs or similar)
• Getting started guide
• Plugin development guide
• API reference docs
• CI/CD pipeline
• PyPI publishing

---

Testing Recipes

Run all tests

uv sync --extra dev
uv run pytest tests/ -v --tb=short

Run a specific module's tests

uv run pytest tests/bench/ -v
uv run pytest tests/sdk/ -v
uv run pytest tests/agents/test_openclaw.py -v

Run with optional deps (server)

uv sync --extra dev --extra server
uv run pytest tests/server/ -v  # No longer skipped

Lint

uv run ruff check src/ tests/
uv run ruff check src/ tests/ --fix  # Auto-fix

Quick smoke test

uv run jarvis --version          # 1.0.0
uv run jarvis --help             # All subcommands
python -c "from openjarvis import Jarvis; print(Jarvis)"

---

Adding New Components

New Benchmark

1. Create src/openjarvis/bench/my_benchmark.py:

   from openjarvis.bench._stubs import BaseBenchmark, BenchmarkResult
   
   class MyBenchmark(BaseBenchmark):
       @property
       def name(self) -> str: return "my-bench"
       @property
       def description(self) -> str: return "Description"
       def run(self, engine, model, *, num_samples=10) -> BenchmarkResult: ...
   
   _registered = False
   def ensure_registered():
       global _registered
       if _registered: return
       from openjarvis.core.registry import BenchmarkRegistry
       if not BenchmarkRegistry.contains("my-bench"):
           BenchmarkRegistry.register_value("my-bench", MyBenchmark)
       _registered = True
   

2. Import in bench/__init__.py ensure_registered()
3. Add test file tests/bench/test_my_benchmark.py with autouse fixture calling ensure_registered()

New Tool

1. Create src/openjarvis/tools/my_tool.py:

   from openjarvis.core.registry import ToolRegistry
   from openjarvis.tools._stubs import BaseTool, ToolSpec
   
   @ToolRegistry.register("my-tool")
   class MyTool(BaseTool):
       @property
       def spec(self) -> ToolSpec: ...
       def execute(self, input: str, **params) -> str: ...
   

2. Import in tools/__init__.py
3. Add test file tests/tools/test_my_tool.py

New Memory Backend

1. Create src/openjarvis/memory/my_backend.py:

   from openjarvis.core.registry import MemoryRegistry
   from openjarvis.memory._stubs import MemoryBackend, RetrievalResult
   
   @MemoryRegistry.register("my-backend")
   class MyBackend(MemoryBackend):
       def store(self, content, *, source="", metadata=None) -> str: ...
       def retrieve(self, query, top_k=5) -> list[RetrievalResult]: ...
       def delete(self, doc_id) -> bool: ...
       def clear(self) -> None: ...
   

2. Import in memory/__init__.py with try/except for optional deps
3. Add test file with pytest.importorskip() if using optional deps
4. Add optional dep group in pyproject.toml if needed

New Agent

1. Create src/openjarvis/agents/my_agent.py:

   from openjarvis.agents._stubs import AgentResult, BaseAgent
   from openjarvis.core.registry import AgentRegistry
   
   @AgentRegistry.register("my-agent")
   class MyAgent(BaseAgent):
       agent_id = "my-agent"
       def __init__(self, engine, model, *, bus=None, **kwargs): ...
       def run(self, input, context=None, **kwargs) -> AgentResult: ...
   

2. Import in agents/__init__.py
3. Add test file tests/agents/test_my_agent.py

New Engine

1. Create src/openjarvis/engine/my_engine.py:

   from openjarvis.core.registry import EngineRegistry
   from openjarvis.engine._stubs import InferenceEngine
   
   @EngineRegistry.register("my-engine")
   class MyEngine(InferenceEngine):
       engine_id = "my-engine"
       def generate(self, messages, *, model, **kwargs) -> dict: ...
       def stream(self, messages, *, model, **kwargs): ...
       def list_models(self) -> list[str]: ...
       def health(self) -> bool: ...
   

2. Import in engine/__init__.py with try/except
3. Add to _discovery.py engine priority list if auto-discoverable

---

Session Log

Session 1 (2026-02-16) — Phase 5 Implementation

Scope: Full Phase 5 (v1.0) — SDK, OpenClaw, Benchmarks, Docker, Docs

Work completed:

• Step 1: Added BenchmarkRegistry to core/registry.py, updated conftest.py
• Step 2: Created bench/ package — _stubs.py, latency.py, throughput.py, __init__.py; CLI bench_cmd.py
• Step 3: Created sdk.py — Jarvis class + MemoryHandle; updated __init__.py exports
• Step 4: Created OpenClaw infra — openclaw_protocol.py, openclaw_transport.py, openclaw_plugin.py; rewrote openclaw.py from stub
• Step 5: Created Dockerfile, Dockerfile.gpu, docker-compose.yml, deploy/systemd/openjarvis.service, deploy/launchd/com.openjarvis.plist
• Step 6: Version bump to 1.0.0, updated README.md, CLAUDE.md

Bugs fixed during implementation:

1. Ruff lint: 17 issues (E501, I001, F401, F841) — all fixed
2. Registry clearing broke @register decorators — switched to ensure_registered() for bench modules
3. ChunkConfig(overlap=...) should be ChunkConfig(chunk_overlap=...) — fixed
4. chunk.text should be chunk.content — fixed
5. Test content too short for chunking (0 chunks produced) — used 100 words

Final: 520 passed, 8 skipped, 0 failures, ruff clean

Session 2 (2026-02-17) — Test Fixes + Live vLLM Testing

Scope: Fix broken tests, set up live vLLM inference testing

Work completed:

• Fixed 6 failed + 13 errored tests in tests/cli/test_ask_router.py and tests/cli/test_ask_agent.py
  • Root cause: from openjarvis.cli.ask import ask in cli/__init__.py shadows the ask module with the Click command object. When mock.patch("openjarvis.cli.ask.get_engine") resolves, it tries to patch an attribute on the Click command, not the module.
  • Fix: Use importlib.import_module("openjarvis.cli.ask") + mock.patch.object(_ask_mod, "get_engine") instead of string-based patching.
• Added tool fallback in _openai_compat.py: if server returns 400 when tools are sent (e.g., vLLM without --enable-auto-tool-choice), retry without tools.
• Verified live vLLM testing: existing vLLM server on port 8003 with Qwen/Qwen3-8B
• Tested: jarvis ask, jarvis bench run, jarvis model list, jarvis memory index/search, jarvis telemetry stats, SDK Jarvis.ask() and ask_full()

Gotcha discovered: 8. Module shadowing with from X import Y — If a package's __init__.py does from openjarvis.cli.ask import ask, then openjarvis.cli.ask in sys.modules is the module, but accessing it via attribute lookup on openjarvis.cli gives the imported object (the Click command). Use importlib.import_module() for reliable module access when patching.

Live vLLM setup notes:

• vLLM 0.15.1 running on Lambda cluster (8x A100-SXM4-80GB)
• Config: ~/.openjarvis/config.toml with vllm_host = "http://localhost:8003" and default_model = "Qwen/Qwen3-8B"
• Tool calling requires --enable-auto-tool-choice --tool-call-parser hermes flags on vLLM server
• Without tool support, orchestrator falls back to reasoning-only mode

Final: 520 passed, 8 skipped, 0 failures, ruff clean

Session 3 (2026-02-21) — Trace System & Research Direction

Scope: Design new research direction (abstractions for local AI), implement trace system

Design decisions made:

• OpenJarvis repositioned as a research framework for studying on-device AI
• Four core abstractions: Intelligence, Engine, Agentic Logic, Memory
• Learning is a cross-cutting concern driven by interaction traces
• Agentic Logic should be pluggable — users bring their own architecture (ReAct, OpenHands-style, etc.)
• Trace collection is the bridge between static and learned agents
• Evolve existing codebase rather than full redesign
• Name stays as OpenJarvis
• Learning focus: telemetry-driven routing/tool policies (lightweight, always-on)
• Agent-model coupling: loose (any agent, any model)

Work completed:

• Added StepType enum, TraceStep, Trace dataclasses to core/types.py
• Added TRACE_STEP, TRACE_COMPLETE event types to core/events.py
• Created traces/ package:
  • store.py — TraceStore: SQLite-backed, save/get/list with filters, event bus subscription
  • collector.py — TraceCollector: wraps any BaseAgent, subscribes to EventBus, records steps automatically
  • analyzer.py — TraceAnalyzer: per-route stats, per-tool stats, summaries, export, query-type filtering
• Created learning/trace_policy.py — TraceDrivenPolicy: learns routing from trace outcomes, batch/online updates, registered as "learned" policy
• Registered TraceDrivenPolicy in learning/__init__.py
• Added 56 new tests across 4 test files in tests/traces/ and tests/learning/test_trace_policy.py
• Updated all markdown documentation (README, VISION, ROADMAP, NOTES, CLAUDE)

Final: 576 passed, 8 skipped, 0 failures, ruff clean