- Add total_input_tokens, total_output_tokens, avg_power_watts to
_compute_summary() and include in RunSummary constructor
- Add _output_path and _traces_dir as proper slots on RunSummary (fixes
pre-existing bug: slots=True prevented dynamic attribute setting)
- Update _summary_to_dict to serialize new fields
- Fix pre-existing test_metric_stats_to_dict to check p90/p95/p99 keys
- Add 2 new tests: total token counts and avg power in summary
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Extend the eval framework to compute per-sample hardware efficiency
metrics when model hardware parameters are provided in TOML config:
- ModelConfig gains param_count_b, active_params_b, gpu_peak_tflops,
gpu_peak_bandwidth_gb_s, num_gpus fields
- RunConfig gains metadata dict, populated by expand_suite() from
model hardware params
- EvalRunner._process_one() computes IPW (Intelligence Per Watt),
IPJ (Intelligence Per Joule), MFU, and MBU per sample
- All telemetry fields (energy, power, GPU util, throughput, MFU, MBU,
IPW, IPJ) written to JSONL output and summary JSON
- RunSummary includes MetricStats (mean/median/min/max/std) for all
telemetry metrics plus total_energy_joules
- GLM-4.7-Flash eval config enriched with A100 SXM hardware params
- 146 eval tests pass (26 new tests for telemetry, efficiency metrics,
metadata flow, MetricStats helpers)
- User guide and API docs updated with new fields and output format
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Evaluation framework (evals/): benchmarking system for measuring accuracy
across four categories — Chat (WildChat), Reasoning (SuperGPQA), RAG (FRAMES),
and Agentic (GAIA). Two backends: jarvis-direct (engine-level) and jarvis-agent
(agent-level with tool calling), both supporting local and cloud models.
Datasets adapted from IPW, scorers include exact match, LLM letter extraction,
and LLM-as-judge. Parallel execution via ThreadPoolExecutor with incremental
JSONL output. CLI: python -m evals {run,run-all,summarize,list}. 57 tests pass.
SVG fix: center logo content within viewBox by wrapping icon+text in a
translate(90,0) group, eliminating the left-shift visible in the README.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
