The monolith->package split landed while #558 was adding provider-aware web
search to toolorchestra.py upstream. Rebasing resolved that as delete-vs-modify,
which would have silently dropped the feature: the package always routed search
to Anthropic. Port it -- search now follows cloud_endpoint (openai-web-search /
gemini-web-search / anthropic-web-search), with the matching worker types,
per-search cost surcharges, and pool validation.
_TOOLORCH_SEARCH_TYPES replaces the scattered == "anthropic-web-search" checks,
so the one-shot guard and the "needs a solver" rule now cover every search type
rather than just the Anthropic one -- previously a search worker could have been
picked as the synthesis fallback.
Also, to get back under the ruff gate CI now enforces (it landed in the 95
commits we were behind): formatting, import order, and one real bug -- calculator
had "abs" twice in the same dict literal, and eval_backend had four imports
stranded below a function body.
The build_orchestrator_sft driver test read OJ_DATA_ROOT from the ambient
environment, so it wrote to the real experiments tree instead of tmp_path for
anyone who had sourced .env. Drop the var in the test.
--out was being used as a path, so runs overwrote each other and a curated
file couldn't be traced back to the rollout that produced it. It is now a
tag: every run lands in <OJ_DATA_ROOT>/raw/{label}_{MMDD}[_{tag}]/data.jsonl
and keeps the file name fixed.
raw/ sits above the sft/rl fork deliberately. SFT keeps only correct and
well-formed rows, but GRPO needs the failures, so the unfiltered rollouts
have to survive reject-sampling. Naming matches across stages -- only the
stage word differs -- so raw/qwen_0707/ pairs with sft/qwen_clean_0707.jsonl
by eye.
OJ_DATA_ROOT defaults to a repo-relative data/orchestrator so a fresh clone
works, and points outside the checkout in a real workspace.
The orchestrator SFT pipeline was living entirely in the working tree and
never got committed. One script in that lineage (clean_sft_data.py) has
already been lost with no way to recover it, so track the rest before the
same thing happens again:
run_sft_fsdp.py full-parameter FSDP trainer (launched via accelerate)
sft_tokenize.py conversation -> tokens + assistant-only masking
render_sft_data.py record -> chat-template rendering
make_splits.py train / holdout / overfit splits
chat_orchestrator.py REPL for eyeballing a trained checkpoint's routing
format_eval_sample.py, upload_to_braintrust.py
scripts/train/ fsdp4.yaml, fsdp8.yaml, eval sbatch
toolorchestra/tracing.py was also untracked despite being imported by
unified.py and rollout.py, which left the branch broken for a fresh
checkout. Add it along with the rest of the ToolOrchestra package split.
Also here: mmlu_pro / supergpqa scorers, expert pricing for the OpenRouter
Qwen builds (estimates, flagged in-file), an expanded calculator/web_search
tool surface, and the reject-sampling clean gate that run_sft_fsdp reads
via --require-clean.
GRPO is left at its upstream state -- the RL stage hasn't been run yet, so
it stays out of this change. Paths in the sbatch are env-driven rather than
hardcoded to one cluster home.
Tools:
- Bridge 4 more real OpenJarvis tools into the orchestrator catalog
(think, apply_patch, pdf_extract, db_query); catalog is now 6 models
+ 11 basic tools.
- Fix the dispatch path so confirmation-gated tools actually run: the
ToolExecutor was built with no confirm callback, so shell_exec (and
any requires_confirmation tool) returned a "requires confirmation"
error instead of executing -- which silently broke TerminalBench.
Headless eval/rollouts now auto-approve.
- Drop dead-end candidates: git_* (need the unbuilt openjarvis_rust
extension and just duplicate shell_exec), browser_* (needs Playwright),
and knowledge_search/sql/retrieval (personal-data RAG, empty on the
academic benchmarks).
SFT data pipeline:
- Reasoning-task loaders (GeneralThought-430K + OpenThoughts3) with an
8K cold-start set and a 30K GRPO prompt pool.
- Domain-dispatched verifier (math/code checkers + Gemini judge fallback,
since the OpenAI key is dead).
- Rejection-sampling generator + unified <tool_call> serializer matching
the GRPO rollout format; base self-sampling driver.
- Drop the old paradigm/tier scaffolding (adp_loader, build, paradigms,
select, serialize, tiers) replaced by the unified path.
Training/eval:
- SFT + GRPO configs for Qwen3.5-9B and gemma-4-12B-it.
- OrchestratorBackend + eval harness over GAIA/TerminalBench/TauBench/
MMLU-Pro/SuperGPQA.
- Cost-aware GRPO reward.
Ignore generated data/ artifacts.
Train a single ~9B orchestrator (Qwen3-8B) for local<->cloud routing via SFT
on synthetic traces derived from NeuLab ADP (agent-data-collection).
- sft_data/: full ADP trajectory loader, tier model + pricing, 4 paradigm
renderers, reward-ranked selection, conversations-JSONL serializer, CLI
- wire sft_trainer._generate_traces() to the pipeline (no GPU / no API keys)
- Qwen3-8B SFT config + scripts/orchestrator CLI wrapper
- offline fixture tests (12)
Heuristic competence labels for the cold-start; real paradigm execution +
GRPO are the documented v2.
Andrew Park