The monolith->package split landed while #558 was adding provider-aware web
search to toolorchestra.py upstream. Rebasing resolved that as delete-vs-modify,
which would have silently dropped the feature: the package always routed search
to Anthropic. Port it -- search now follows cloud_endpoint (openai-web-search /
gemini-web-search / anthropic-web-search), with the matching worker types,
per-search cost surcharges, and pool validation.
_TOOLORCH_SEARCH_TYPES replaces the scattered == "anthropic-web-search" checks,
so the one-shot guard and the "needs a solver" rule now cover every search type
rather than just the Anthropic one -- previously a search worker could have been
picked as the synthesis fallback.
Also, to get back under the ruff gate CI now enforces (it landed in the 95
commits we were behind): formatting, import order, and one real bug -- calculator
had "abs" twice in the same dict literal, and eval_backend had four imports
stranded below a function body.
The build_orchestrator_sft driver test read OJ_DATA_ROOT from the ambient
environment, so it wrote to the real experiments tree instead of tmp_path for
anyone who had sourced .env. Drop the var in the test.
--out was being used as a path, so runs overwrote each other and a curated
file couldn't be traced back to the rollout that produced it. It is now a
tag: every run lands in <OJ_DATA_ROOT>/raw/{label}_{MMDD}[_{tag}]/data.jsonl
and keeps the file name fixed.
raw/ sits above the sft/rl fork deliberately. SFT keeps only correct and
well-formed rows, but GRPO needs the failures, so the unfiltered rollouts
have to survive reject-sampling. Naming matches across stages -- only the
stage word differs -- so raw/qwen_0707/ pairs with sft/qwen_clean_0707.jsonl
by eye.
OJ_DATA_ROOT defaults to a repo-relative data/orchestrator so a fresh clone
works, and points outside the checkout in a real workspace.
The orchestrator SFT pipeline was living entirely in the working tree and
never got committed. One script in that lineage (clean_sft_data.py) has
already been lost with no way to recover it, so track the rest before the
same thing happens again:
run_sft_fsdp.py full-parameter FSDP trainer (launched via accelerate)
sft_tokenize.py conversation -> tokens + assistant-only masking
render_sft_data.py record -> chat-template rendering
make_splits.py train / holdout / overfit splits
chat_orchestrator.py REPL for eyeballing a trained checkpoint's routing
format_eval_sample.py, upload_to_braintrust.py
scripts/train/ fsdp4.yaml, fsdp8.yaml, eval sbatch
toolorchestra/tracing.py was also untracked despite being imported by
unified.py and rollout.py, which left the branch broken for a fresh
checkout. Add it along with the rest of the ToolOrchestra package split.
Also here: mmlu_pro / supergpqa scorers, expert pricing for the OpenRouter
Qwen builds (estimates, flagged in-file), an expanded calculator/web_search
tool surface, and the reject-sampling clean gate that run_sft_fsdp reads
via --require-clean.
GRPO is left at its upstream state -- the RL stage hasn't been run yet, so
it stays out of this change. Paths in the sbatch are env-driven rather than
hardcoded to one cluster home.
Tools:
- Bridge 4 more real OpenJarvis tools into the orchestrator catalog
(think, apply_patch, pdf_extract, db_query); catalog is now 6 models
+ 11 basic tools.
- Fix the dispatch path so confirmation-gated tools actually run: the
ToolExecutor was built with no confirm callback, so shell_exec (and
any requires_confirmation tool) returned a "requires confirmation"
error instead of executing -- which silently broke TerminalBench.
Headless eval/rollouts now auto-approve.
- Drop dead-end candidates: git_* (need the unbuilt openjarvis_rust
extension and just duplicate shell_exec), browser_* (needs Playwright),
and knowledge_search/sql/retrieval (personal-data RAG, empty on the
academic benchmarks).
SFT data pipeline:
- Reasoning-task loaders (GeneralThought-430K + OpenThoughts3) with an
8K cold-start set and a 30K GRPO prompt pool.
- Domain-dispatched verifier (math/code checkers + Gemini judge fallback,
since the OpenAI key is dead).
- Rejection-sampling generator + unified <tool_call> serializer matching
the GRPO rollout format; base self-sampling driver.
- Drop the old paradigm/tier scaffolding (adp_loader, build, paradigms,
select, serialize, tiers) replaced by the unified path.
Training/eval:
- SFT + GRPO configs for Qwen3.5-9B and gemma-4-12B-it.
- OrchestratorBackend + eval harness over GAIA/TerminalBench/TauBench/
MMLU-Pro/SuperGPQA.
- Cost-aware GRPO reward.
Ignore generated data/ artifacts.
Train a single ~9B orchestrator (Qwen3-8B) for local<->cloud routing via SFT
on synthetic traces derived from NeuLab ADP (agent-data-collection).
- sft_data/: full ADP trajectory loader, tier model + pricing, 4 paradigm
renderers, reward-ranked selection, conversations-JSONL serializer, CLI
- wire sft_trainer._generate_traces() to the pipeline (no GPU / no API keys)
- Qwen3-8B SFT config + scripts/orchestrator CLI wrapper
- offline fixture tests (12)
Heuristic competence labels for the cold-start; real paradigm execution +
GRPO are the documented v2.
Previously `core/config.py` defined `DEFAULT_CONFIG_DIR = Path.home() /
".openjarvis"` as a by-value module constant imported into ~45 modules, and 34
modules hardcoded `Path.home() / ".openjarvis"` directly. The installer honored
`OPENJARVIS_HOME` but the Python runtime ignored it, producing a split-brain
layout (some modules honored the override, the core config dir did not). Eval
dataset caches also scattered into `~/.cache/<benchmark>`.
This introduces a single env-aware resolver in `openjarvis/core/paths.py` and
routes every state/config/cache path through it. OpenJarvis now keeps ALL of
its state under ONE root, resolved in priority order:
1. $OPENJARVIS_HOME
2. $XDG_DATA_HOME/openjarvis (single nested dir, when XDG_DATA_HOME is set)
3. ~/.openjarvis (default — unchanged, so existing installs are
untouched and no data migration is required)
Implementation:
- New `core/paths.py`: get_config_dir / get_config_path / get_data_dir /
get_cache_dir, with a source-tree rejection guard (fails loudly per
REVIEW.md if the root resolves inside the repo).
- `core/config.py`: DEFAULT_CONFIG_DIR / DEFAULT_CONFIG_PATH are now resolved
via the env-aware resolver at import (real attributes, so existing
monkeypatch.setattr-based tests keep working). All dataclass field defaults
that pointed at ~/.openjarvis converted to default_factory so they honor the
override at instantiation.
- Routed all 34 hardcoders plus several string-literal escapees the original
audit missed: prompt_loader / description_loader (were OPENJARVIS_HOME-only,
no XDG), swebench_harness cache, tools/{memory,skill,user_profile}_manage
defaults, server trace.db fallbacks, doctor_cmd hints.
- spec_search storage/paths now delegates to the unified resolver (gains XDG);
its ConfigurationError is aliased to the core one.
- Eval dataset caches moved from ~/.cache/<name> to <root>/cache/<name>
(~/.cache/huggingface left alone — it is HF's own cache).
- Docs + installer comment + `jarvis config path` to show resolved dirs.
Read-only macOS connectors and OS service files (LaunchAgents/systemd) are
intentionally left untouched.
Fixes#462
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Memory tools degraded silently and misleadingly when the mandatory
`openjarvis_rust` extension was absent from the *serving* venv:
- `POST /v1/memory/store` returned HTTP 200 `{"status":"stored","note":
"no backend available"}` and stored nothing (silent data loss).
- `POST /v1/memory/index` returned a generic "No memory backend available",
and the desktop frontend discarded the server `detail` and threw a blanket
"Failed to index path", blaming the path instead of the real cause.
- `GET /v1/memory/config` reported `backend_type: sqlite` even though no
backend could be constructed.
Root cause: `SQLiteMemory.__init__` calls `get_rust_module()` (which raises
ImportError by design — the Rust ext is mandatory, no Python fallback), and
`_get_memory_backend` swallowed that ImportError and returned `None`,
conflating "native extension missing" (a hard install error) with "memory
intentionally disabled" (benign). A chunking floor also silently dropped whole
short documents, and the installer never verified the extension imported from
the serving venv before writing its success marker.
Fix (no fake Python fallback — the Rust ext stays mandatory by design):
- Add `MemoryBackendUnavailable` + `RUST_MISSING_HINT` in tools/storage/_stubs.
`SQLiteMemory.__init__` translates the bridge ImportError into this clear,
actionable error ("run `uv run maturin develop ...`").
- `_get_memory_backend` distinguishes the two cases: a missing native ext
raises HTTP 503 with the actionable hint; a benign unconfigured backend
still returns `None` (graceful path preserved for search/stats).
- `/store` now returns 503 instead of a 200 silent no-op.
- `/config` reports `available: false` + `detail` instead of falsely claiming
a healthy `backend_type`.
- `/index` adds a `note` when `chunks_indexed == 0` so "indexed" never
silently means "stored nothing".
- chunk_text no longer drops an entire short document below `min_chunk_size`
(the floor only discards tiny *trailing* fragments now).
- Frontend `storeMemory`/`indexMemoryPath` surface the server `detail` instead
of blanket strings; `MemoryConfig` gains optional `available`/`detail`.
(Left the pre-existing `backend` vs `backend_type` mismatch untouched.)
- build-extension.sh verifies `import openjarvis_rust` succeeds in the serving
venv before writing the `extension-built` marker.
Regression tests: tests/server/test_api_routes.py::TestMemoryRustMissing mocks
`get_rust_module` to raise ImportError and asserts /store (503, not 200 no-op),
/index (actionable detail, not "Failed to index path"), and /config
(available:false) all surface the clear error; tests/memory/test_chunking.py
asserts short-only docs are kept while tiny trailing fragments are still
filtered.
Fixes#502
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Consolidates two reports that overlap in scope:
- #476 (@senki): install.sh hardcoded `--python 3.11` but
pyproject.toml declares `requires-python = ">=3.10,<3.14"`. The
installer should track the project's allowed range, not pin a
conservative-three-years-ago version.
- #484 (@sanjayravit): install.sh crashed on hosts with no
python3/python on PATH because `mark_done`, `beacon`, and
`get_anon_id` all called $PY_CMD via inline Python heredocs.
This PR closes both gaps with five surgical edits to install.sh
(all behavior-preserving for the existing happy path; the bats
tests prove it):
1. get_anon_id — replace `python3 -c uuid` with POSIX
`/dev/urandom + od` + bash substring expansion. Same UUID v4
shape; works on Python-less hosts.
2. beacon — replace the 40-line Python heredoc with curl + a
shell-built JSON payload. All inputs are from controlled
sources (event ∈ fixed vocabulary, stage from stage_label(),
numeric ids/codes from validated arithmetic, anon_id from a
fresh UUID) so no general-purpose JSON escaping is needed.
`|| true` is load-bearing — PostHog 5xx must never abort an
install via the ERR trap.
3. mark_done — replace `python3 -c json.load+update+dump` with
awk that regenerates the file from scratch. Idempotent: if
the key is already marked, return early. Robust against
prior format drift. wsl key is always rewritten last so a
later FORCE_WSL=1 re-run correctly updates it.
4. parse_requires_python — new helper. Greps the project's
requires-python field, handles inclusive (`<=3.13`) and
exclusive (`<3.14`) upper bounds correctly, falls back to
3.11 if pyproject can't be parsed (the previous hardcoded
value — safe under the existing 3.10-3.13 range).
5. create_venv — call parse_requires_python instead of
hardcoding 3.11. The existing uv-managed-Python fallback
(from #444) still kicks in if the host doesn't have the
target version installed.
Adversarial review caught two real bugs before commit:
- HIGH: parse_requires_python's exclusive-bound regex would
also match the digits after `<=` (inclusive bound) and then
incorrectly subtract 1 — producing 3.12 from `<=3.13`. Fixed
by checking the inclusive form first.
- MEDIUM: the new "no Python on PATH" bats test silently skips
symlinking `pgrep` on hosts where it isn't at /usr/bin or
/bin (some minimal BusyBox configurations). Added an explicit
"no matching process" fallback so start_ollama's check works.
New bats coverage:
- `install succeeds with no system Python on PATH (#484)` —
builds a PATH that excludes python3/python and exercises the
full install. Asserts state file is written and contains
greppable step keys.
- `mark_done is idempotent — second mark of same key doesn't
duplicate` — re-runs the install and verifies install_uv
appears exactly once in install-state.json.
- `create_venv picks newest in requires-python range, not
hardcoded 3.11 (#476)` — asserts uv was called with
`--python 3.13` (the upper minor of `>=3.10,<3.14`).
The git stub now includes `requires-python = ">=3.10,<3.14"`
in its fake pyproject.toml so create_venv has something
realistic to parse.
@sanjayravit — your PR #484 motivated this consolidation;
closing that one as superseded with credit.
Co-authored-by: krypticmouse <herumbshandilya123@gmail.com>
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
PR A of the post-cluster install-hardening pair. A multi-agent audit (4 install paths × 2 agents each) showed the README's "copy this one-liner and chat" claim broke on every fresh laptop because of missing git, missing Python 3.11, a racing Ollama daemon, and silently-swallowed model-pull failures. This PR closes the macOS / Linux / WSL2 gaps.
Changes in scripts/install/install.sh:
1. Auto-install missing tools (was: hard refuse):
- macOS: xcode-select --install + 10-min poll. Refuses fast under SSH (no display for the dialog).
- Linux: detects apt-get / dnf / yum / pacman / zypper / apk. Pre-checks `sudo -n true` and refuses fast with actionable guidance if sudo would prompt (stdin is the curl pipe — any prompt silently hangs under `set -euo pipefail`). Uses `;` not `&&` between apt-get update and install. Skips sudo entirely if already root.
2. Auto-install Python 3.11 via uv when missing. Captures real errors to $STATE_DIR/venv-create.err so disk-full / permission-denied isn't hidden behind "downloading managed Python".
3. Replace `sleep 1` after `ollama serve` with a 60-second poll on `ollama list`. Caller guards with `|| true` so timeout surfaces as a warning in the final banner instead of aborting under `set -e`.
4. Track MODEL_PULL_OK + PATH_MODIFIED. The completion banner now tells the truth: if the model pull failed, point at `jarvis doctor` (not bare `jarvis` which would crash). If PATH was just written to ~/.bashrc / ~/.zshrc, print the exact `source <rc> && jarvis` so it works in the same shell.
Adversarial review caught 5 real bugs before commit:
- SSH/headless macOS xcode-select hang → pre-detect SSH session
- sudo no-TTY silent abort → `sudo -n true` precheck
- wait_for_ollama timeout tripping ERR trap → `|| true` at call sites
- swallowed venv stderr hiding diagnostics → capture to log + surface on fallback failure
- contradictory "PATH new + model missing" banner → conditional NEXT_CMD
All 26 install bats tests pass.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Reimplements the useful parts of #385 cleanly.
Adds two small cross-platform helpers under `openjarvis.core.utils`:
- `get_python_executable()` — prefers `python3`, falls back to `python` for Windows / minimal distros that only ship the unversioned name.
- `open_browser(url)` — `webbrowser.open` by default; on Windows uses `cmd /c start "" <url>` to avoid console-host edge cases.
Swapped at every hardcoded `python3` / `webbrowser.open` site: `connectors/oauth.py`, `evals/scorers/livecodebench.py`, `scripts/oauth_all.py`, `scripts/install/install.sh` (adds `PY_CMD` detection block), `scripts/quickstart.sh` (adds `MINGW*|MSYS*|CYGWIN*) cmd /c start` case), and the two affected test files. Test files wrap `get_python_executable()` in `shlex.quote()` before interpolating into `shell=True` strings — Windows interpreter paths often contain spaces.
Deliberately different from #385: `openjarvis.core.__init__` does NOT re-export `DEFAULT_CONFIG_DIR` (would have raised ImportError because it's in `openjarvis.core.config`, not the package `__init__`; re-exporting would also force eager import of the heavy config module at every `import openjarvis.core`). `oauth.py` keeps `from openjarvis.core.config import DEFAULT_CONFIG_DIR` alongside the new `from openjarvis.core import open_browser`.
Original API surface and call-site sweep by @sanjayravit in #385 — huge thanks for the careful Windows-compatibility audit. This PR preserves your design while fixing the ImportError edge cases caught during review.
Co-Authored-By: sanjayravit <sanjayravit@users.noreply.github.com>
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The documented install command pointed at `https://openjarvis.ai/install.sh`,
but that domain is community-operated (not controlled by this project) and
its TLS config broke — every new user hit `sslv3 alert handshake failure`
(#337, #352). Since we can't fix a domain we don't control, this moves the
installer onto infrastructure we DO control: the project's GitHub Pages
docs site.
Changes:
- **`docs/gen_install_script.py`** (new) + **`mkdocs.yml`**: a `gen-files`
hook copies `scripts/install/install.sh` verbatim into the built site at
`install.sh` on every `mkdocs build`. Single source of truth — the script
stays at `scripts/install/install.sh` (still bundled into the wheel as
`_install_scripts/`); the published copy can't drift. Verified locally:
`mkdocs build` emits `site/install.sh` byte-identical to the source.
New canonical URL: `https://open-jarvis.github.io/OpenJarvis/install.sh`
— HTTPS always valid (GitHub's cert), fully under project control.
- **`README.md`**: canonical command switched to the github.io URL for
both the Installation and Quick Start blocks. Also addresses the uv
discoverability gap — explicitly states the curl installer downloads uv
for you (no prerequisite), and that the Windows **desktop .exe** expects
uv to be installed first, with the exact PowerShell command.
- **`docs/getting-started/{install,wsl2,macos,linux}.md`**: canonical URL
switched to github.io. `install.md` gains an "Install URL" info note
explaining the github.io URL is canonical and that the older
`openjarvis.ai` URL is community-operated with intermittent TLS issues.
- **`scripts/install/install.sh`** + **`jarvis-wrapper.sh`**: usage comment,
the WSL re-run hint (added in #399), and the wrapper's re-install message
all updated to the github.io URL.
Migration note for maintainers: ask whoever operates `openjarvis.ai` to
CNAME it to `open-jarvis.github.io`. Once they do, `openjarvis.ai/install.sh`
will serve this same GitHub Pages content with a valid GitHub-managed cert,
and the nicer brand URL can become canonical again with zero further code
changes. Until then, the github.io URL works and is under our control.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Addresses the second half of the Discord support thread on the
"Jarvis server did not become healthy in time" issue (PR #398 fixed
the diagnostic gap; this fixes discoverability of the right install
path so users don't end up there in the first place).
Three changes, all surface improvements:
1. **`README.md`** — explicit Windows section in the Installation
block. Previously, the only Windows mention was a footnote
("Platforms: ... WSL2 on Windows") that came AFTER the `curl … |
bash` install command. Users on PowerShell would copy/paste the
command, get a syntax error, then try to debug bash on Windows.
Now the README clearly says: bash installer is macOS/Linux only;
Windows users have two paths (WSL2 with one-time `wsl --install`
setup, or the desktop .exe from Releases). Both link to the
relevant docs.
2. **`scripts/install/install.sh`** — early bail when running under
Git Bash / MSYS2 / Cygwin (MINGW*, MSYS*, CYGWIN* per `uname -s`).
These environments aren't WSL — `uv` and `git` will install to
Windows-side paths that OpenJarvis can't reach, Ollama integration
silently breaks, and the user gets to debug it 3 minutes into a
doomed install. The bail message points at both the WSL2 setup
command (`wsl --install -d Ubuntu-24.04`) and the desktop .exe
download as alternatives.
Verified the case-match doesn't fire on Linux (`uname -s` →
`Linux`, matches the wildcard fall-through, not the MINGW patterns).
3. **`frontend/src-tauri/src/lib.rs`** — when `resolve_bin("uv")`
can't find uv, the per-OS error message now contains the exact
install command for the user's OS, ready to copy/paste. On Windows
that's the `irm https://astral.sh/uv/install.ps1 | iex` command
Marc kept reposting on the Discord support thread (5/12-5/14).
On macOS/Linux it's the standard `curl | sh` installer.
The previous generic "Install it from https://astral.sh/uv" left
users guessing whether to use winget, scoop, pip, or the official
installer — which is exactly the confusion the Discord thread
captured.
None of these are root-cause code fixes (Discord users' uv installs
fail for environment-specific reasons we can't diagnose remotely),
but together they remove the three biggest friction sources we saw:
copy-paste install command that can't possibly work, doomed git-bash
installs that fail mysteriously, and missing exact install commands
when uv isn't found.
The Tauri change ships in the desktop binary; the README change is
visible immediately on the repo page; the install.sh change reaches
users via openjarvis.ai (when it's restored) and via the GitHub-raw
fallback from PR #398.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(mining): add Pearl model conversion workflow
* test(mining): tolerate missing Docker device request type
* docs(mining): record Gemma and Qwen conversion evidence
* docs(mining): record Qwen local validation evidence
---------
Co-authored-by: krypticmouse <herumbshandilya123@gmail.com>
The project had two overlapping directories: desktop/ (Tauri Rust backend +
stale React components) and frontend/ (real React app + dead Tauri stub).
This consolidates everything under frontend/:
- Move desktop/src-tauri/ → frontend/src-tauri/ (the real 1,720-line Rust
backend with Ollama sidecar, backend lifecycle, cloud keys, overlay, etc.)
- Preserve 9 old desktop React components in frontend/src/components/Desktop/
(excluded from TS build — APIs have drifted, kept for future integration)
- Delete the old frontend/src-tauri/ stub (246 lines, never compiled)
- Delete desktop/ entirely
- Fix tauri.conf.json frontendDist path (../../frontend/dist → ../dist)
- Update CI workflow, bump script, .gitignore, and docs paths
- Rename setup/ → Setup/ for consistent PascalCase component directories
Companion to #143 which fixed the frontend to use Claude Opus 4.6 as
the sole baseline. This migration recomputes existing Supabase rows
using the exact closed-form: new = T/3.8M + 10*old/19, derived from
the original triple-provider formula.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Andrew Park