admin/OpenJarvis
1
0
Fork 0
mirror of https://github.com/open-jarvis/OpenJarvis.git synced 2026-07-31 03:12:16 +00:00
Code Issues Packages Projects Releases Wiki Activity
37 Commits 11 Branches 77 Tags
2aebcd7d770d3daa4da61e1ddcf9fb6c9fe83e69
Commit Graph
4 Commits
Author SHA1 Message Date
Jon Saad-FalconandClaude Opus 4.6 9f5e97eed9 Add eval framework efficiency metrics: MFU, MBU, IPW, IPJ 
Extend the eval framework to compute per-sample hardware efficiency
metrics when model hardware parameters are provided in TOML config:

- ModelConfig gains param_count_b, active_params_b, gpu_peak_tflops,
  gpu_peak_bandwidth_gb_s, num_gpus fields
- RunConfig gains metadata dict, populated by expand_suite() from
  model hardware params
- EvalRunner._process_one() computes IPW (Intelligence Per Watt),
  IPJ (Intelligence Per Joule), MFU, and MBU per sample
- All telemetry fields (energy, power, GPU util, throughput, MFU, MBU,
  IPW, IPJ) written to JSONL output and summary JSON
- RunSummary includes MetricStats (mean/median/min/max/std) for all
  telemetry metrics plus total_energy_joules
- GLM-4.7-Flash eval config enriched with A100 SXM hardware params
- 146 eval tests pass (26 new tests for telemetry, efficiency metrics,
  metadata flow, MetricStats helpers)
- User guide and API docs updated with new fields and output format

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-02-25 21:17:53 +00:00
Jon Saad-FalconandClaude Opus 4.6 9ec402ab4a Enrich agent tool awareness, normalize engine tool_calls, add telemetry and eval config 
- Add build_tool_descriptions() shared builder for enriched agent system
  prompts (NativeReAct, NativeOpenHands, RLM, Orchestrator structured mode)
- Normalize tool_calls to flat {id, name, arguments} across CloudEngine
  (OpenAI/Anthropic/Google), LiteLLM, and Ollama
- Add Anthropic tool_use extraction and input_schema conversion
- Add Google function_call extraction
- Make ReAct/OpenHands parsing case-insensitive
- Add telemetry efficiency, GPU monitor, and vLLM metrics modules
- Add TOML-based eval suite config system
- Update documentation and changelog

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-02-25 01:58:18 +00:00
Jon Saad-FalconandClaude Opus 4.6 323d7ff032 Add TOML config system for eval suites, pillar-aligned config, and documentation 
- Eval config: TOML-based suite configs defining models x benchmarks matrix,
  loaded via --config flag. Includes load_eval_config(), expand_suite(),
  7 config dataclasses, 3 example configs, and 61 new tests.
- Pillar-aligned config: generation params in IntelligenceConfig, nested
  engine/learning configs, agent objective/system_prompt/context_from_memory,
  structured learning sub-policies, TOML migration layer.
- Documentation: evaluations user guide, evals API reference, updated
  mkdocs.yml navigation, updated architecture docs.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-02-24 03:34:05 +00:00
Jon Saad-FalconandClaude Opus 4.6 bd49383201 Add evaluation framework and center README logo SVGs 
Evaluation framework (evals/): benchmarking system for measuring accuracy
across four categories — Chat (WildChat), Reasoning (SuperGPQA), RAG (FRAMES),
and Agentic (GAIA). Two backends: jarvis-direct (engine-level) and jarvis-agent
(agent-level with tool calling), both supporting local and cloud models.
Datasets adapted from IPW, scorers include exact match, LLM letter extraction,
and LLM-as-judge. Parallel execution via ThreadPoolExecutor with incremental
JSONL output. CLI: python -m evals {run,run-all,summarize,list}. 57 tests pass.

SVG fix: center logo content within viewBox by wrapping icon+text in a
translate(90,0) group, eliminating the left-shift visible in the README.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-02-22 23:48:43 +00:00