mirror of
https://github.com/open-jarvis/OpenJarvis.git
synced 2026-07-27 21:05:34 +00:00
Andrew Park
e6c47e9532
The orchestrator SFT pipeline was living entirely in the working tree and never got committed. One script in that lineage (clean_sft_data.py) has already been lost with no way to recover it, so track the rest before the same thing happens again: run_sft_fsdp.py full-parameter FSDP trainer (launched via accelerate) sft_tokenize.py conversation -> tokens + assistant-only masking render_sft_data.py record -> chat-template rendering make_splits.py train / holdout / overfit splits chat_orchestrator.py REPL for eyeballing a trained checkpoint's routing format_eval_sample.py, upload_to_braintrust.py scripts/train/ fsdp4.yaml, fsdp8.yaml, eval sbatch toolorchestra/tracing.py was also untracked despite being imported by unified.py and rollout.py, which left the branch broken for a fresh checkout. Add it along with the rest of the ToolOrchestra package split. Also here: mmlu_pro / supergpqa scorers, expert pricing for the OpenRouter Qwen builds (estimates, flagged in-file), an expanded calculator/web_search tool surface, and the reject-sampling clean gate that run_sft_fsdp reads via --require-clean. GRPO is left at its upstream state -- the RL stage hasn't been run yet, so it stays out of this change. Paths in the sbatch are env-driven rather than hardcoded to one cluster home.
26 lines
660 B
YAML
26 lines
660 B
YAML
compute_environment: LOCAL_MACHINE
|
|
distributed_type: FSDP
|
|
downcast_bf16: 'no'
|
|
machine_rank: 0
|
|
main_process_ip: null
|
|
main_process_port: null
|
|
main_training_function: main
|
|
mixed_precision: bf16
|
|
num_machines: 1
|
|
num_processes: 8
|
|
rdzv_backend: static
|
|
same_network: true
|
|
use_cpu: false
|
|
fsdp_config:
|
|
fsdp_version: 1
|
|
fsdp_auto_wrap_policy: SIZE_BASED_WRAP
|
|
fsdp_min_num_params: 100000000
|
|
fsdp_sharding_strategy: FULL_SHARD
|
|
fsdp_state_dict_type: FULL_STATE_DICT
|
|
fsdp_offload_params: false
|
|
fsdp_backward_prefetch: BACKWARD_PRE
|
|
fsdp_forward_prefetch: false
|
|
fsdp_use_orig_params: true
|
|
fsdp_cpu_ram_efficient_loading: true
|
|
fsdp_sync_module_states: true
|