The orchestrator SFT pipeline was living entirely in the working tree and
never got committed. One script in that lineage (clean_sft_data.py) has
already been lost with no way to recover it, so track the rest before the
same thing happens again:
run_sft_fsdp.py full-parameter FSDP trainer (launched via accelerate)
sft_tokenize.py conversation -> tokens + assistant-only masking
render_sft_data.py record -> chat-template rendering
make_splits.py train / holdout / overfit splits
format_eval_sample.py, upload_to_braintrust.py
scripts/train/ fsdp4.yaml, fsdp8.yaml, eval sbatch
toolorchestra/tracing.py was also untracked despite being imported by
unified.py and rollout.py, which left the branch broken for a fresh
checkout. Add it along with the rest of the ToolOrchestra package split.
Also here: mmlu_pro / supergpqa scorers, expert pricing for the OpenRouter
Qwen builds (estimates, flagged in-file), an expanded calculator/web_search
tool surface, and the reject-sampling clean gate that run_sft_fsdp reads
via --require-clean.
Paths in the sbatch and litellm helpers are now env-driven rather than
hardcoded to one cluster home.
Tools:
- Bridge 4 more real OpenJarvis tools into the orchestrator catalog
(think, apply_patch, pdf_extract, db_query); catalog is now 6 models
+ 11 basic tools.
- Fix the dispatch path so confirmation-gated tools actually run: the
ToolExecutor was built with no confirm callback, so shell_exec (and
any requires_confirmation tool) returned a "requires confirmation"
error instead of executing -- which silently broke TerminalBench.
Headless eval/rollouts now auto-approve.
- Drop dead-end candidates: git_* (need the unbuilt openjarvis_rust
extension and just duplicate shell_exec), browser_* (needs Playwright),
and knowledge_search/sql/retrieval (personal-data RAG, empty on the
academic benchmarks).
SFT data pipeline:
- Reasoning-task loaders (GeneralThought-430K + OpenThoughts3) with an
8K cold-start set and a 30K GRPO prompt pool.
- Domain-dispatched verifier (math/code checkers + Gemini judge fallback,
since the OpenAI key is dead).
- Rejection-sampling generator + unified <tool_call> serializer matching
the GRPO rollout format; base self-sampling driver.
- Drop the old paradigm/tier scaffolding (adp_loader, build, paradigms,
select, serialize, tiers) replaced by the unified path.
Training/eval:
- SFT + GRPO configs for Qwen3.5-9B and gemma-4-12B-it.
- OrchestratorBackend + eval harness over GAIA/TerminalBench/TauBench/
MMLU-Pro/SuperGPQA.
- Cost-aware GRPO reward.
Ignore generated data/ artifacts.
Train a single ~9B orchestrator (Qwen3-8B) for local<->cloud routing via SFT
on synthetic traces derived from NeuLab ADP (agent-data-collection).
- sft_data/: full ADP trajectory loader, tier model + pricing, 4 paradigm
renderers, reward-ranked selection, conversations-JSONL serializer, CLI
- wire sft_trainer._generate_traces() to the pipeline (no GPU / no API keys)
- Qwen3-8B SFT config + scripts/orchestrator CLI wrapper
- offline fixture tests (12)
Heuristic competence labels for the cold-start; real paradigm execution +
GRPO are the documented v2.
Consolidates two reports that overlap in scope:
- #476 (@senki): install.sh hardcoded `--python 3.11` but
pyproject.toml declares `requires-python = ">=3.10,<3.14"`. The
installer should track the project's allowed range, not pin a
conservative-three-years-ago version.
- #484 (@sanjayravit): install.sh crashed on hosts with no
python3/python on PATH because `mark_done`, `beacon`, and
`get_anon_id` all called $PY_CMD via inline Python heredocs.
This PR closes both gaps with five surgical edits to install.sh
(all behavior-preserving for the existing happy path; the bats
tests prove it):
1. get_anon_id — replace `python3 -c uuid` with POSIX
`/dev/urandom + od` + bash substring expansion. Same UUID v4
shape; works on Python-less hosts.
2. beacon — replace the 40-line Python heredoc with curl + a
shell-built JSON payload. All inputs are from controlled
sources (event ∈ fixed vocabulary, stage from stage_label(),
numeric ids/codes from validated arithmetic, anon_id from a
fresh UUID) so no general-purpose JSON escaping is needed.
`|| true` is load-bearing — PostHog 5xx must never abort an
install via the ERR trap.
3. mark_done — replace `python3 -c json.load+update+dump` with
awk that regenerates the file from scratch. Idempotent: if
the key is already marked, return early. Robust against
prior format drift. wsl key is always rewritten last so a
later FORCE_WSL=1 re-run correctly updates it.
4. parse_requires_python — new helper. Greps the project's
requires-python field, handles inclusive (`<=3.13`) and
exclusive (`<3.14`) upper bounds correctly, falls back to
3.11 if pyproject can't be parsed (the previous hardcoded
value — safe under the existing 3.10-3.13 range).
5. create_venv — call parse_requires_python instead of
hardcoding 3.11. The existing uv-managed-Python fallback
(from #444) still kicks in if the host doesn't have the
target version installed.
Adversarial review caught two real bugs before commit:
- HIGH: parse_requires_python's exclusive-bound regex would
also match the digits after `<=` (inclusive bound) and then
incorrectly subtract 1 — producing 3.12 from `<=3.13`. Fixed
by checking the inclusive form first.
- MEDIUM: the new "no Python on PATH" bats test silently skips
symlinking `pgrep` on hosts where it isn't at /usr/bin or
/bin (some minimal BusyBox configurations). Added an explicit
"no matching process" fallback so start_ollama's check works.
New bats coverage:
- `install succeeds with no system Python on PATH (#484)` —
builds a PATH that excludes python3/python and exercises the
full install. Asserts state file is written and contains
greppable step keys.
- `mark_done is idempotent — second mark of same key doesn't
duplicate` — re-runs the install and verifies install_uv
appears exactly once in install-state.json.
- `create_venv picks newest in requires-python range, not
hardcoded 3.11 (#476)` — asserts uv was called with
`--python 3.13` (the upper minor of `>=3.10,<3.14`).
The git stub now includes `requires-python = ">=3.10,<3.14"`
in its fake pyproject.toml so create_venv has something
realistic to parse.
@sanjayravit — your PR #484 motivated this consolidation;
closing that one as superseded with credit.
Co-authored-by: krypticmouse <herumbshandilya123@gmail.com>
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
PR A of the post-cluster install-hardening pair. A multi-agent audit (4 install paths × 2 agents each) showed the README's "copy this one-liner and chat" claim broke on every fresh laptop because of missing git, missing Python 3.11, a racing Ollama daemon, and silently-swallowed model-pull failures. This PR closes the macOS / Linux / WSL2 gaps.
Changes in scripts/install/install.sh:
1. Auto-install missing tools (was: hard refuse):
- macOS: xcode-select --install + 10-min poll. Refuses fast under SSH (no display for the dialog).
- Linux: detects apt-get / dnf / yum / pacman / zypper / apk. Pre-checks `sudo -n true` and refuses fast with actionable guidance if sudo would prompt (stdin is the curl pipe — any prompt silently hangs under `set -euo pipefail`). Uses `;` not `&&` between apt-get update and install. Skips sudo entirely if already root.
2. Auto-install Python 3.11 via uv when missing. Captures real errors to $STATE_DIR/venv-create.err so disk-full / permission-denied isn't hidden behind "downloading managed Python".
3. Replace `sleep 1` after `ollama serve` with a 60-second poll on `ollama list`. Caller guards with `|| true` so timeout surfaces as a warning in the final banner instead of aborting under `set -e`.
4. Track MODEL_PULL_OK + PATH_MODIFIED. The completion banner now tells the truth: if the model pull failed, point at `jarvis doctor` (not bare `jarvis` which would crash). If PATH was just written to ~/.bashrc / ~/.zshrc, print the exact `source <rc> && jarvis` so it works in the same shell.
Adversarial review caught 5 real bugs before commit:
- SSH/headless macOS xcode-select hang → pre-detect SSH session
- sudo no-TTY silent abort → `sudo -n true` precheck
- wait_for_ollama timeout tripping ERR trap → `|| true` at call sites
- swallowed venv stderr hiding diagnostics → capture to log + surface on fallback failure
- contradictory "PATH new + model missing" banner → conditional NEXT_CMD
All 26 install bats tests pass.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Reimplements the useful parts of #385 cleanly.
Adds two small cross-platform helpers under `openjarvis.core.utils`:
- `get_python_executable()` — prefers `python3`, falls back to `python` for Windows / minimal distros that only ship the unversioned name.
- `open_browser(url)` — `webbrowser.open` by default; on Windows uses `cmd /c start "" <url>` to avoid console-host edge cases.
Swapped at every hardcoded `python3` / `webbrowser.open` site: `connectors/oauth.py`, `evals/scorers/livecodebench.py`, `scripts/oauth_all.py`, `scripts/install/install.sh` (adds `PY_CMD` detection block), `scripts/quickstart.sh` (adds `MINGW*|MSYS*|CYGWIN*) cmd /c start` case), and the two affected test files. Test files wrap `get_python_executable()` in `shlex.quote()` before interpolating into `shell=True` strings — Windows interpreter paths often contain spaces.
Deliberately different from #385: `openjarvis.core.__init__` does NOT re-export `DEFAULT_CONFIG_DIR` (would have raised ImportError because it's in `openjarvis.core.config`, not the package `__init__`; re-exporting would also force eager import of the heavy config module at every `import openjarvis.core`). `oauth.py` keeps `from openjarvis.core.config import DEFAULT_CONFIG_DIR` alongside the new `from openjarvis.core import open_browser`.
Original API surface and call-site sweep by @sanjayravit in #385 — huge thanks for the careful Windows-compatibility audit. This PR preserves your design while fixing the ImportError edge cases caught during review.
Co-Authored-By: sanjayravit <sanjayravit@users.noreply.github.com>
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The documented install command pointed at `https://openjarvis.ai/install.sh`,
but that domain is community-operated (not controlled by this project) and
its TLS config broke — every new user hit `sslv3 alert handshake failure`
(#337, #352). Since we can't fix a domain we don't control, this moves the
installer onto infrastructure we DO control: the project's GitHub Pages
docs site.
Changes:
- **`docs/gen_install_script.py`** (new) + **`mkdocs.yml`**: a `gen-files`
hook copies `scripts/install/install.sh` verbatim into the built site at
`install.sh` on every `mkdocs build`. Single source of truth — the script
stays at `scripts/install/install.sh` (still bundled into the wheel as
`_install_scripts/`); the published copy can't drift. Verified locally:
`mkdocs build` emits `site/install.sh` byte-identical to the source.
New canonical URL: `https://open-jarvis.github.io/OpenJarvis/install.sh`
— HTTPS always valid (GitHub's cert), fully under project control.
- **`README.md`**: canonical command switched to the github.io URL for
both the Installation and Quick Start blocks. Also addresses the uv
discoverability gap — explicitly states the curl installer downloads uv
for you (no prerequisite), and that the Windows **desktop .exe** expects
uv to be installed first, with the exact PowerShell command.
- **`docs/getting-started/{install,wsl2,macos,linux}.md`**: canonical URL
switched to github.io. `install.md` gains an "Install URL" info note
explaining the github.io URL is canonical and that the older
`openjarvis.ai` URL is community-operated with intermittent TLS issues.
- **`scripts/install/install.sh`** + **`jarvis-wrapper.sh`**: usage comment,
the WSL re-run hint (added in #399), and the wrapper's re-install message
all updated to the github.io URL.
Migration note for maintainers: ask whoever operates `openjarvis.ai` to
CNAME it to `open-jarvis.github.io`. Once they do, `openjarvis.ai/install.sh`
will serve this same GitHub Pages content with a valid GitHub-managed cert,
and the nicer brand URL can become canonical again with zero further code
changes. Until then, the github.io URL works and is under our control.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Addresses the second half of the Discord support thread on the
"Jarvis server did not become healthy in time" issue (PR #398 fixed
the diagnostic gap; this fixes discoverability of the right install
path so users don't end up there in the first place).
Three changes, all surface improvements:
1. **`README.md`** — explicit Windows section in the Installation
block. Previously, the only Windows mention was a footnote
("Platforms: ... WSL2 on Windows") that came AFTER the `curl … |
bash` install command. Users on PowerShell would copy/paste the
command, get a syntax error, then try to debug bash on Windows.
Now the README clearly says: bash installer is macOS/Linux only;
Windows users have two paths (WSL2 with one-time `wsl --install`
setup, or the desktop .exe from Releases). Both link to the
relevant docs.
2. **`scripts/install/install.sh`** — early bail when running under
Git Bash / MSYS2 / Cygwin (MINGW*, MSYS*, CYGWIN* per `uname -s`).
These environments aren't WSL — `uv` and `git` will install to
Windows-side paths that OpenJarvis can't reach, Ollama integration
silently breaks, and the user gets to debug it 3 minutes into a
doomed install. The bail message points at both the WSL2 setup
command (`wsl --install -d Ubuntu-24.04`) and the desktop .exe
download as alternatives.
Verified the case-match doesn't fire on Linux (`uname -s` →
`Linux`, matches the wildcard fall-through, not the MINGW patterns).
3. **`frontend/src-tauri/src/lib.rs`** — when `resolve_bin("uv")`
can't find uv, the per-OS error message now contains the exact
install command for the user's OS, ready to copy/paste. On Windows
that's the `irm https://astral.sh/uv/install.ps1 | iex` command
Marc kept reposting on the Discord support thread (5/12-5/14).
On macOS/Linux it's the standard `curl | sh` installer.
The previous generic "Install it from https://astral.sh/uv" left
users guessing whether to use winget, scoop, pip, or the official
installer — which is exactly the confusion the Discord thread
captured.
None of these are root-cause code fixes (Discord users' uv installs
fail for environment-specific reasons we can't diagnose remotely),
but together they remove the three biggest friction sources we saw:
copy-paste install command that can't possibly work, doomed git-bash
installs that fail mysteriously, and missing exact install commands
when uv isn't found.
The Tauri change ships in the desktop binary; the README change is
visible immediately on the repo page; the install.sh change reaches
users via openjarvis.ai (when it's restored) and via the GitHub-raw
fallback from PR #398.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(mining): add Pearl model conversion workflow
* test(mining): tolerate missing Docker device request type
* docs(mining): record Gemma and Qwen conversion evidence
* docs(mining): record Qwen local validation evidence
---------
Co-authored-by: krypticmouse <herumbshandilya123@gmail.com>
The project had two overlapping directories: desktop/ (Tauri Rust backend +
stale React components) and frontend/ (real React app + dead Tauri stub).
This consolidates everything under frontend/:
- Move desktop/src-tauri/ → frontend/src-tauri/ (the real 1,720-line Rust
backend with Ollama sidecar, backend lifecycle, cloud keys, overlay, etc.)
- Preserve 9 old desktop React components in frontend/src/components/Desktop/
(excluded from TS build — APIs have drifted, kept for future integration)
- Delete the old frontend/src-tauri/ stub (246 lines, never compiled)
- Delete desktop/ entirely
- Fix tauri.conf.json frontendDist path (../../frontend/dist → ../dist)
- Update CI workflow, bump script, .gitignore, and docs paths
- Rename setup/ → Setup/ for consistent PascalCase component directories
Companion to #143 which fixed the frontend to use Claude Opus 4.6 as
the sole baseline. This migration recomputes existing Supabase rows
using the exact closed-form: new = T/3.8M + 10*old/19, derived from
the original triple-provider formula.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Andrew Park