OpenJarvis/docs/plans/2026-03-02-codebase-simplification.md

Codebase Simplification Implementation Plan

For Claude: REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task.

Goal: Simplify the OpenJarvis repository structure by reducing root sprawl, moving data files into the package, removing backward-compat shims, and consolidating repetitive code patterns.

Architecture: Incremental commits, each independently testable. Data TOML files move into src/openjarvis/*/data/ as package data. The evals/ framework becomes src/openjarvis/evals/ with all internal imports rewritten from evals. to openjarvis.evals.. The memory/ shim package is removed. Engine wrappers are consolidated.

Tech Stack: Python, TOML, hatch/hatchling build system, pytest, ruff

---

Task 1: Remove get-pip.py from git tracking

Files:

• Delete: get-pip.py
• Modify: .gitignore

Step 1: Add get-pip.py to .gitignore

In .gitignore, add after the # Project section (after line 44):

get-pip.py

Step 2: Remove get-pip.py from git tracking and delete

Run:

git rm get-pip.py

Step 3: Commit

git add .gitignore
git commit -m "chore: remove get-pip.py from repository (2.2MB regenerable artifact)"

---

Task 2: Move Docker files to deploy/docker/

Files:

• Move: Dockerfile -> deploy/docker/Dockerfile
• Move: Dockerfile.gpu -> deploy/docker/Dockerfile.gpu
• Move: Dockerfile.gpu.rocm -> deploy/docker/Dockerfile.gpu.rocm
• Move: Dockerfile.sandbox -> deploy/docker/Dockerfile.sandbox
• Move: docker-compose.yml -> deploy/docker/docker-compose.yml
• Move: docker-compose.gpu.rocm.yml -> deploy/docker/docker-compose.gpu.rocm.yml
• Modify: deploy/docker/docker-compose.yml (update dockerfile paths)
• Modify: deploy/docker/docker-compose.gpu.rocm.yml (update dockerfile path)

Step 1: Move all Docker files

mv Dockerfile deploy/docker/Dockerfile
mv Dockerfile.gpu deploy/docker/Dockerfile.gpu
mv Dockerfile.gpu.rocm deploy/docker/Dockerfile.gpu.rocm
mv Dockerfile.sandbox deploy/docker/Dockerfile.sandbox
mv docker-compose.yml deploy/docker/docker-compose.yml
mv docker-compose.gpu.rocm.yml deploy/docker/docker-compose.gpu.rocm.yml

Step 2: Update docker-compose.yml build contexts

In deploy/docker/docker-compose.yml, change the build section from:

    build:
      context: .
      dockerfile: Dockerfile

to:

    build:
      context: ../..
      dockerfile: deploy/docker/Dockerfile

Step 3: Update docker-compose.gpu.rocm.yml

In deploy/docker/docker-compose.gpu.rocm.yml, change:

      dockerfile: Dockerfile.gpu.rocm

to:

      context: ../..
      dockerfile: deploy/docker/Dockerfile.gpu.rocm

Step 4: Commit

git add -A deploy/docker/
git add -A Dockerfile* docker-compose*
git commit -m "refactor: move Docker files to deploy/docker/"

---

Task 3: Move recipes/ TOML data into package

Files:

• Move: recipes/*.toml -> src/openjarvis/recipes/data/*.toml
• Move: recipes/operators/*.toml -> src/openjarvis/recipes/data/operators/*.toml
• Move: recipes/operators/*.md -> src/openjarvis/recipes/data/operators/*.md
• Modify: src/openjarvis/recipes/loader.py (lines 15-16, update _PROJECT_RECIPES_DIR)
• Modify: pyproject.toml (add package data include)
• Test: tests/recipes/ (run existing tests)

Step 1: Create data directory and move files

mkdir -p src/openjarvis/recipes/data/operators
mv recipes/*.toml src/openjarvis/recipes/data/
mv recipes/operators/*.toml src/openjarvis/recipes/data/operators/
mv recipes/operators/*.md src/openjarvis/recipes/data/operators/
rmdir recipes/operators
rmdir recipes

Step 2: Update loader.py

In src/openjarvis/recipes/loader.py, change line 16 from:

_PROJECT_RECIPES_DIR = Path(__file__).resolve().parents[3] / "recipes"

to:

_PROJECT_RECIPES_DIR = Path(__file__).resolve().parent / "data"

Step 3: Run tests

uv run pytest tests/recipes/ -v

Expected: All recipe tests pass.

Step 4: Commit

git add src/openjarvis/recipes/data/ src/openjarvis/recipes/loader.py
git add recipes/
git commit -m "refactor: move recipes/ TOML data into src/openjarvis/recipes/data/"

---

Task 4: Move templates/ agent TOML data into package

Files:

• Move: templates/agents/*.toml -> src/openjarvis/templates/data/*.toml
• Modify: src/openjarvis/templates/agent_templates.py (lines 68-71, update _builtin_templates_dir)
• Test: tests/templates/ (run existing tests)

Step 1: Create data directory and move files

mkdir -p src/openjarvis/templates/data
mv templates/agents/*.toml src/openjarvis/templates/data/
rmdir templates/agents
rmdir templates

Step 2: Update agent_templates.py

In src/openjarvis/templates/agent_templates.py, change lines 68-71 from:

def _builtin_templates_dir() -> Path:
    """Return the path to the built-in templates shipped with the package."""
    # templates/agents/ at project root, 3 levels above this file
    return Path(__file__).resolve().parents[3] / "templates" / "agents"

to:

def _builtin_templates_dir() -> Path:
    """Return the path to the built-in templates shipped with the package."""
    return Path(__file__).resolve().parent / "data"

Step 3: Run tests

uv run pytest tests/templates/ -v

Expected: All template tests pass.

Step 4: Commit

git add src/openjarvis/templates/data/ src/openjarvis/templates/agent_templates.py
git add templates/
git commit -m "refactor: move templates/agents/ TOML data into src/openjarvis/templates/data/"

---

Task 5: Move skills/builtin/ TOML data into package

Files:

• Move: skills/builtin/*.toml -> src/openjarvis/skills/data/*.toml
• Modify: tests/skills/test_bundled_skills.py (line 12, update BUILTIN_DIR path)
• Test: tests/skills/ (run existing tests)

Step 1: Create data directory and move files

mkdir -p src/openjarvis/skills/data
mv skills/builtin/*.toml src/openjarvis/skills/data/
rmdir skills/builtin
rmdir skills

Step 2: Update test_bundled_skills.py

In tests/skills/test_bundled_skills.py, change line 12 from:

BUILTIN_DIR = Path(__file__).resolve().parents[2] / "skills" / "builtin"

to:

BUILTIN_DIR = Path(__file__).resolve().parents[2] / "src" / "openjarvis" / "skills" / "data"

Step 3: Run tests

uv run pytest tests/skills/ -v

Expected: All skill tests pass (20 bundled skills found).

Step 4: Commit

git add src/openjarvis/skills/data/ tests/skills/test_bundled_skills.py
git add skills/
git commit -m "refactor: move skills/builtin/ TOML data into src/openjarvis/skills/data/"

---

Task 6: Move operators/ TOML data into package

Files:

• Move: operators/*.toml -> src/openjarvis/operators/data/*.toml
• Modify: src/openjarvis/cli/operators_cmd.py (lines 35, 247, 274 — replace Path("operators") with package data path)
• Test: tests/operators/ (run existing tests)

Step 1: Create data directory and move files

mkdir -p src/openjarvis/operators/data
mv operators/*.toml src/openjarvis/operators/data/
rmdir operators

Step 2: Update operators_cmd.py

The CLI currently checks Path("operators") (cwd-relative). We need to replace this with the package data directory. In src/openjarvis/cli/operators_cmd.py:

Add a helper at the top of the file (after imports):

def _builtin_operators_dir() -> Path:
    """Return the path to built-in operator manifests shipped with the package."""
    return Path(__file__).resolve().parents[1] / "operators" / "data"

Then update the three places that reference Path("operators"):

Line 35: Change local_ops = Path("operators") to local_ops = _builtin_operators_dir()

Line 247: Change dirs = [DEFAULT_CONFIG_DIR / "operators", Path("operators")] to dirs = [DEFAULT_CONFIG_DIR / "operators", _builtin_operators_dir()]

Line 274: Change for d in [DEFAULT_CONFIG_DIR / "operators", Path("operators")]: to for d in [DEFAULT_CONFIG_DIR / "operators", _builtin_operators_dir()]:

Step 3: Run tests

uv run pytest tests/operators/ -v

Expected: All operator tests pass.

Step 4: Commit

git add src/openjarvis/operators/data/ src/openjarvis/cli/operators_cmd.py
git add operators/
git commit -m "refactor: move operators/ TOML data into src/openjarvis/operators/data/"

---

Task 7: Move evals/ into src/openjarvis/evals/

This is the most complex migration. The evals/ directory is a standalone Python package with internal imports like from evals.core.config import .... All these must be rewritten to from openjarvis.evals.core.config import ....

Files:

• Move: evals/ -> src/openjarvis/evals/ (entire directory tree)
• Modify: All *.py files in src/openjarvis/evals/ (rewrite from evals. to from openjarvis.evals.)
• Modify: src/openjarvis/cli/eval_cmd.py (rewrite from evals. imports)
• Modify: tests/evals/*.py (rewrite from evals. imports)
• Modify: pyproject.toml (update ruff per-file-ignores paths)

Step 1: Move the directory

# Move evals/ into src/openjarvis/evals/
# First remove any existing __pycache__
find evals/ -type d -name __pycache__ -exec rm -rf {} + 2>/dev/null
mv evals/ src/openjarvis/evals/

Step 2: Rewrite all internal imports in evals/

Run a find-and-replace across all Python files in src/openjarvis/evals/:

find src/openjarvis/evals/ -name "*.py" -exec sed -i 's/from evals\./from openjarvis.evals./g; s/import evals\./import openjarvis.evals./g' {} +

Step 3: Rewrite imports in eval_cmd.py

In src/openjarvis/cli/eval_cmd.py, replace all occurrences of:

• from evals.core.config import -> from openjarvis.evals.core.config import
• from evals.cli import -> from openjarvis.evals.cli import
• from evals.core.types import -> from openjarvis.evals.core.types import

Step 4: Rewrite imports in tests/evals/

Run find-and-replace across test files:

find tests/evals/ -name "*.py" -exec sed -i 's/from evals\./from openjarvis.evals./g; s/import evals\./import openjarvis.evals./g' {} +

Step 5: Update pyproject.toml

Change lines 134-136 from:

[tool.ruff.lint.per-file-ignores]
"evals/datasets/*.py" = ["E501"]
"evals/scorers/*.py" = ["E501"]

to:

[tool.ruff.lint.per-file-ignores]
"src/openjarvis/evals/datasets/*.py" = ["E501"]
"src/openjarvis/evals/scorers/*.py" = ["E501"]

Step 6: Update __main__.py

In src/openjarvis/evals/__main__.py, the import was rewritten in step 2 but verify it now reads:

"""Allow running as ``python -m openjarvis.evals``."""

from openjarvis.evals.cli import main

if __name__ == "__main__":
    main()

Step 7: Run tests

uv run pytest tests/evals/ -v

Expected: All eval tests pass.

Step 8: Lint check

uv run ruff check src/openjarvis/evals/ tests/evals/ src/openjarvis/cli/eval_cmd.py

Expected: No import errors.

Step 9: Commit

git add src/openjarvis/evals/ tests/evals/ src/openjarvis/cli/eval_cmd.py pyproject.toml
git commit -m "refactor: move evals/ into src/openjarvis/evals/ as proper subpackage"

---

Task 8: Remove memory/ backward-compat shim package

Files:

• Delete: src/openjarvis/memory/ (entire directory, 11 files)
• Modify: src/openjarvis/cli/memory_cmd.py (lines 15-16)
• Modify: src/openjarvis/cli/ask.py (lines 130, 304)
• Modify: src/openjarvis/sdk.py (lines 36, 59-60, 391, 431)
• Modify: tests/tools/test_storage_stubs.py (lines 68-69, 80)
• Modify: tests/tools/test_retrieval.py (line 7)
• Modify: tests/cli/test_memory_cmd.py (line 12)
• Modify: tests/cli/test_ask_context.py (lines 34, 55)
• Modify: tests/test_integration_extended.py (lines 402, 419)
• Delete: tests/memory/ (entire directory — these test the shim, not the real backends)

Step 1: Update src/ imports

In src/openjarvis/cli/memory_cmd.py, change:

from openjarvis.memory.chunking import ChunkConfig
from openjarvis.memory.ingest import ingest_path

to:

from openjarvis.tools.storage.chunking import ChunkConfig
from openjarvis.tools.storage.ingest import ingest_path

In src/openjarvis/cli/ask.py, change all occurrences of:

• from openjarvis.memory.context import -> from openjarvis.tools.storage.context import
• from openjarvis.memory.sqlite import -> from openjarvis.tools.storage.sqlite import

In src/openjarvis/sdk.py, change:

• from openjarvis.memory.sqlite import SqliteMemory -> from openjarvis.tools.storage.sqlite import SQLiteMemory as SqliteMemory
• from openjarvis.memory.chunking import ChunkConfig -> from openjarvis.tools.storage.chunking import ChunkConfig
• from openjarvis.memory.ingest import ingest_path -> from openjarvis.tools.storage.ingest import ingest_path
• from openjarvis.memory.context import ContextConfig, inject_context -> from openjarvis.tools.storage.context import ContextConfig, inject_context

Step 2: Update test imports

In tests/tools/test_storage_stubs.py, change:

• from openjarvis.memory._stubs import MemoryBackend as MB -> from openjarvis.tools.storage._stubs import MemoryBackend as MB
• from openjarvis.memory._stubs import RetrievalResult as RR -> from openjarvis.tools.storage._stubs import RetrievalResult as RR
• from openjarvis.memory.sqlite import SQLiteMemory as S2 -> from openjarvis.tools.storage.sqlite import SQLiteMemory as S2

In tests/tools/test_retrieval.py, change:

• from openjarvis.memory._stubs import MemoryBackend, RetrievalResult -> from openjarvis.tools.storage._stubs import MemoryBackend, RetrievalResult

In tests/cli/test_memory_cmd.py, change:

• from openjarvis.memory.sqlite import SQLiteMemory -> from openjarvis.tools.storage.sqlite import SQLiteMemory

In tests/cli/test_ask_context.py, change:

• from openjarvis.memory.sqlite import SQLiteMemory -> from openjarvis.tools.storage.sqlite import SQLiteMemory

In tests/test_integration_extended.py, change:

• from openjarvis.memory.sqlite import SQLiteMemory -> from openjarvis.tools.storage.sqlite import SQLiteMemory
• from openjarvis.memory.bm25 import BM25Memory -> from openjarvis.tools.storage.bm25 import BM25Memory

Step 3: Update tests/memory/ imports and move to tests/tools/

The tests/memory/ files test the same backends as tests/tools/test_storage_*.py. Since they import from the shim, we need to rewrite their imports and merge them. However, since tests/tools/ already has storage tests, the simplest approach is:

Rewrite all tests/memory/*.py imports from openjarvis.memory.* to openjarvis.tools.storage.*:

find tests/memory/ -name "*.py" -exec sed -i 's/from openjarvis\.memory\./from openjarvis.tools.storage./g' {} +

Then move the directory to be a subdirectory of tests/tools:

mv tests/memory tests/tools/memory_tests

Actually, the cleanest approach: just rewrite imports in-place and keep tests/memory/ as-is. The tests are valuable and test different aspects than tests/tools/.

Step 4: Delete the shim package

rm -rf src/openjarvis/memory/

Step 5: Run full test suite

uv run pytest tests/memory/ tests/tools/ tests/cli/ tests/sdk/ -v

Expected: All tests pass.

Step 6: Commit

git add -A src/openjarvis/memory/ src/openjarvis/cli/memory_cmd.py src/openjarvis/cli/ask.py src/openjarvis/sdk.py
git add tests/memory/ tests/tools/ tests/cli/ tests/test_integration_extended.py
git commit -m "refactor: remove memory/ backward-compat shims, use tools.storage directly"

---

Task 9: Consolidate engine wrappers into single file

Files:

• Delete: src/openjarvis/engine/vllm.py
• Delete: src/openjarvis/engine/sglang.py
• Delete: src/openjarvis/engine/llamacpp.py
• Delete: src/openjarvis/engine/mlx.py
• Delete: src/openjarvis/engine/lmstudio.py
• Create: src/openjarvis/engine/openai_compat_engines.py
• Modify: src/openjarvis/engine/__init__.py (update imports)
• Test: Run tests/engine/ tests

Step 1: Write the consolidated engine file

Create src/openjarvis/engine/openai_compat_engines.py:

"""Data-driven registration of OpenAI-compatible inference engines."""

from openjarvis.core.registry import EngineRegistry
from openjarvis.engine._openai_compat import _OpenAICompatibleEngine

_ENGINES = {
    "vllm": ("VLLMEngine", "http://localhost:8000"),
    "sglang": ("SGLangEngine", "http://localhost:30000"),
    "llamacpp": ("LlamaCppEngine", "http://localhost:8080"),
    "mlx": ("MLXEngine", "http://localhost:8080"),
    "lmstudio": ("LMStudioEngine", "http://localhost:1234"),
}

for _key, (_cls_name, _default_host) in _ENGINES.items():
    _cls = type(
        _cls_name,
        (_OpenAICompatibleEngine,),
        {"engine_id": _key, "_default_host": _default_host},
    )
    EngineRegistry.register(_key)(_cls)
    globals()[_cls_name] = _cls

__all__ = [name for name, _ in _ENGINES.values()]

Step 2: Update engine/__init__.py

Replace lines 5-11 (the 5 individual imports) with a single import:

Change from:

import openjarvis.engine.llamacpp  # noqa: F401
import openjarvis.engine.mlx  # noqa: F401

# Import engine modules to trigger @EngineRegistry.register() decorators
import openjarvis.engine.ollama  # noqa: F401
import openjarvis.engine.sglang  # noqa: F401
import openjarvis.engine.vllm  # noqa: F401

To:

# Import engine modules to trigger @EngineRegistry.register() decorators
import openjarvis.engine.ollama  # noqa: F401
import openjarvis.engine.openai_compat_engines  # noqa: F401

Step 3: Delete the 5 individual wrapper files

rm src/openjarvis/engine/vllm.py
rm src/openjarvis/engine/sglang.py
rm src/openjarvis/engine/llamacpp.py
rm src/openjarvis/engine/mlx.py
rm src/openjarvis/engine/lmstudio.py

Step 4: Run tests

uv run pytest tests/engine/ -v

Expected: All engine tests pass.

Step 5: Commit

git add src/openjarvis/engine/
git commit -m "refactor: consolidate 5 OpenAI-compat engine wrappers into single data-driven file"

---

Task 10: Refactor repetitive __init__.py imports

Files:

• Modify: src/openjarvis/channels/__init__.py
• Modify: src/openjarvis/engine/__init__.py (optional engines section)
• Test: Run tests/channels/ and tests/engine/ tests

Step 1: Refactor channels/__init__.py

Replace the entire file content with:

"""Channel abstraction for multi-platform messaging."""

import importlib

from openjarvis.channels._stubs import (
    BaseChannel,
    ChannelHandler,
    ChannelMessage,
    ChannelStatus,
)

# Trigger registration of built-in channels.
# Each module uses @ChannelRegistry.register() — importing is sufficient.
_CHANNEL_MODULES = [
    "telegram",
    "discord_channel",
    "slack",
    "webhook",
    "email_channel",
    "whatsapp",
    "signal_channel",
    "google_chat",
    "irc_channel",
    "webchat",
    "teams",
    "matrix_channel",
    "mattermost",
    "feishu",
    "bluebubbles",
    "whatsapp_baileys",
    "line_channel",
    "viber_channel",
    "messenger_channel",
    "reddit_channel",
    "mastodon_channel",
    "xmpp_channel",
    "rocketchat_channel",
    "zulip_channel",
    "twitch_channel",
    "nostr_channel",
]

for _mod in _CHANNEL_MODULES:
    try:
        importlib.import_module(f".{_mod}", __name__)
    except ImportError:
        pass

__all__ = [
    "BaseChannel",
    "ChannelHandler",
    "ChannelMessage",
    "ChannelStatus",
]

Step 2: Refactor optional engine imports in engine/__init__.py

Replace lines 19-29 with:

# Optional engines — only register if their SDK deps are present
for _optional in ("cloud", "litellm"):
    try:
        importlib.import_module(f".{_optional}", __name__)
    except ImportError:
        pass

And add import importlib at the top.

Step 3: Run tests

uv run pytest tests/channels/ tests/engine/ -v

Expected: All tests pass.

Step 4: Commit

git add src/openjarvis/channels/__init__.py src/openjarvis/engine/__init__.py
git commit -m "refactor: replace repetitive try/except imports with loop-based auto-registration"

---

Task 11: Update pyproject.toml package data includes

Files:

• Modify: pyproject.toml

Step 1: Add package data includes

The TOML data files that moved into src/openjarvis/ need to be included in wheel builds. In pyproject.toml, after the existing [tool.hatch.build.targets.wheel.force-include] section, add the data directories.

Actually, since all the data directories are under src/openjarvis/ and packages = ["src/openjarvis"] is already set, hatch will include them automatically as long as they are inside the package tree. But TOML files are non-Python files, so we need to tell hatch to include them.

Add to pyproject.toml after line 114:

[tool.hatch.build.targets.wheel.shared-data]

[tool.hatch.build]
include = [
    "src/openjarvis/**/*.py",
    "src/openjarvis/**/*.toml",
    "src/openjarvis/**/*.md",
    "src/openjarvis/agents/claude_code_runner/**",
    "src/openjarvis/channels/whatsapp_baileys_bridge/**",
]

Wait — hatchling by default includes all files under packages. Non-Python files inside the package are included by default. No change needed here. Verify with:

uv run python -c "from pathlib import Path; d = Path('src/openjarvis/recipes/data'); print(list(d.glob('*.toml')))"

Step 2: Commit if any changes were needed

If no pyproject.toml changes needed, skip this task.

---

Task 12: Update CLAUDE.md to reflect new structure

Files:

• Modify: CLAUDE.md

Step 1: Update all references to moved paths

In CLAUDE.md, make the following updates:

1. Update the python -m evals command references to python -m openjarvis.evals
2. Remove references to top-level recipes/, templates/, skills/, operators/ directories
3. Update Docker command references to use deploy/docker/ paths
4. Update the architecture section to reflect that data files are now package data
5. Remove mentions of memory/ backward-compat shims
6. Remove the 5 individual engine wrapper files from architecture description

Step 2: Run linting to verify no broken references

uv run ruff check src/ tests/

Step 3: Run full test suite

uv run pytest tests/ -v --tb=short 2>&1 | tail -20

Expected: ~3240 tests pass.

Step 4: Commit

git add CLAUDE.md
git commit -m "docs: update CLAUDE.md to reflect simplified repository structure"

---

Task 13: Final verification

Step 1: Run full test suite

uv run pytest tests/ -v --tb=short 2>&1 | tail -30

Expected: ~3240 tests pass, ~44 skipped.

Step 2: Run linter

uv run ruff check src/ tests/

Expected: Clean.

Step 3: Verify root structure

ls -la | grep -v __pycache__ | grep -v '^\.'

Expected: 21 or fewer items at root (down from 32).

Step 4: Verify package data is accessible

uv run python -c "
from openjarvis.recipes.loader import discover_recipes
from openjarvis.templates.agent_templates import discover_templates
print(f'Recipes: {len(discover_recipes())}')
print(f'Templates: {len(discover_templates())}')
print('All data accessible.')
"

Expected: Recipes: 3, Templates: 15, All data accessible.