28ef745384 fix(leaderboard): correct telemetry pipeline and outlier handling (#498)
* fix(leaderboard): correct telemetry pipeline and outlier handling

The public leaderboard at /leaderboard showed a clear bimodal Wh/token
distribution: most users at ~3-5 J/token, ~30% inflated by 1000-4000×.
A 5-agent investigation workflow + 4-agent verification pass traced the
inflation to a cluster of related bugs across telemetry, server, and
display layers. This PR fixes the in-repo half. Backfilling existing
Supabase data is a separate follow-up.

Bug 1 — Dual telemetry recording (`server/routes.py`)
=====================================================

`_handle_direct` wrapped the engine with `instrumented_generate`
unconditionally. When `app.state.engine` was already an
`InstrumentedEngine` (the common case when telemetry is wired in),
BOTH layers published `TELEMETRY_RECORD` — once from the inner
`InstrumentedEngine.generate`, once from the outer wrapper. Every
chat-completion request was counted twice in the leaderboard pipeline.

Fix: detect the InstrumentedEngine and unwrap to `._inner` before
passing to the wrapper so only one layer fires.

Bug 2 — KV-cache fallback over-counts multi-turn (`server/savings.py`)
=====================================================================

`compute_savings` falls back from `prompt_tokens_evaluated` to
`prompt_tokens` when the KV-cache-aware count is missing. But routes.py
aggregates by summing each turn's full prompt — which counts the system
prompt N times for an N-turn conversation. The fallback inflated FLOPs
and energy by N×.

Fix: use 0 (conservative under-count) when the evaluated count is
missing rather than falling back to the inflated `prompt_tokens` sum.

Bug 3 — TelemetryRecord lacked methodology versioning
=====================================================

There was no per-record version tag, so legacy (pre-fix) and current
records were silently aggregated together in the public leaderboard
even though they used different methodologies.

Fix: add `token_counting_version: Optional[int]` field to the
`TelemetryRecord` dataclass + a nullable column to the SQLite schema
(with idempotent migration); the constant moves from `server.savings`
to `core.types` to avoid the server→telemetry layering. New records
write the current version; pre-fix rows remain NULL. The aggregator
gains a `current_methodology_only=True` flag that filters NULL rows out
of leaderboard sums — local dashboards leave it off so historical
aggregates still render.

Bug 4 — Leaderboard JS displayed missing telemetry as legit zeros
=================================================================

Rows with significant token counts but `energy_wh_saved = 0` and
`flops_saved = 0` rendered as `0.00 Wh / 0 FLOPs` — visually identical
to a user who genuinely did almost nothing. The headline totals also
included these rows.

Fix: new `isMissingTelemetry()` detector renders missing-telemetry
energy/FLOPs cells as `—` (with a tooltip explaining why); a new
`.lb-missing` CSS class differentiates the placeholder visually.

Bug 5 — Outlier filter was too generous
=======================================

`MAX_ENERGY_WH_PER_TOKEN = 10` left a 10,000× margin that admitted
every Group B row even though those values are physically impossible
(would imply a space-heater per token). Same for the FLOPs cap at
`1e17`.

Fix: tighten to `0.5` Wh/token (still 500× over a typical consumer GPU)
and `1e15` FLOPs/token (still 10,000× over typical). This removes
existing pre-fix corrupt rows from the public view without touching
Supabase.

Tests
=====

New regression tests (all pass, ruff clean):

- `tests/server/test_routes.py::test_instrumented_engine_unwrapped_to_avoid_dual_telemetry`
  — pins the bug 1 fix; asserts exactly ONE TELEMETRY_RECORD event per
  request when the engine is already an InstrumentedEngine.
- `tests/server/test_savings.py` (NEW file, 3 tests) — pins the bug 2
  fix (FLOPs not inflated via fallback) and the cost-side invariant
  (dollar savings still use full prompt_tokens because cloud providers
  bill per input token even when local KV cache hit).
- `tests/telemetry/test_aggregator.py::TestMethodologyFilter` (3 tests)
  — default behaviour includes legacy rows (local dashboard parity);
  `current_methodology_only=True` excludes them; the summary surface
  honors the same filter.

Unrelated test note
===================

`tests/telemetry/test_energy_wiring.py::TestTelemetryStatsEnergy::test_export_includes_energy_fields`
and `::TestEndToEndPipeline::test_ask_to_export_with_energy` are flaky
on this branch but ALSO flaky on `main` (confirmed via stash + the
banner-only run). Root cause: `_version_check.py:132-135` prints the
"new version of OpenJarvis is available" banner to stdout outside the
throttle window, contaminating `json.loads(result.output)` in those two
tests. Reproducible with `OPENJARVIS_NO_UPDATE_CHECK=1` → all pass.
That's a separate bug (the banner shouldn't write to the same stream
as machine-readable output); not addressing it here to keep this PR
focused on the leaderboard pipeline.

What this PR explicitly does NOT do
====================================

- Backfill ~30% of Supabase rows that already shipped with inflated
  values from pre-fix clients. That requires Supabase write access and
  is the natural follow-up — see PR comments for proposed dry-run audit
  query and the additive `methodology_version` column migration.
- Distinguish which past submissions came from buggy vs correct clients
  retroactively (no `app_version` tag on existing submissions).

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* test(evals): fix test_energy_scales_linearly under leaderboard fix

CI on #498 failed in the slow `test` job:

    tests/evals/test_use_case_benchmarks.py::TestSavings::
    test_energy_scales_linearly — ZeroDivisionError: float division by zero

Root cause: the test (and `test_energy_wh_matches_direct_formula`)
called `compute_savings(N, 0)` without `prompt_tokens_evaluated`. Under
the OLD buggy fallback, an unset evaluated count silently became N,
energy scaled linearly with N, and the test passed by accident. Under
this branch's conservative fix the fallback is 0, FLOPs collapse to 0,
and the `p10.energy_wh / p1.energy_wh` ratio divides 0 by 0.

The test's own docstring says "evaluated tokens (KV-cache model)" — it
was always meant to exercise the explicit-evaluated path, the API
call just didn't match. Pass `prompt_tokens_evaluated` explicitly so
the test now expresses the invariant it claims to.

Same one-line fix for `test_energy_wh_matches_direct_formula` plus an
explicit `flops > 0` sanity assertion — that test was passing trivially
with `0 == 0` after the conservative fallback fix, masking whether the
formula was actually being exercised.

No production code change in this commit. Verified locally: all 6
TestSavings tests pass; the 5 new regression tests added on this branch
still pass.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

---------

Co-authored-by: krypticmouse <herumbshandilya123@gmail.com>
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
2026-06-04 16:59:24 -07:00
2026-05-25 17:49:40 +00:00
2026-03-12 17:29:39 +00:00

OpenJarvis

Personal AI, On Personal Devices.

Project Docs Python License Discord X / Twitter


OpenJarvis demo reel

Documentation

Project Site

Leaderboard

Roadmap

Why OpenJarvis?

Personal AI agents are exploding in popularity, but nearly all of them still route intelligence through cloud APIs. Your "personal" AI continues to depend on someone else's server. At the same time, our Intelligence Per Watt research showed that local language models already handle 88.7% of single-turn chat and reasoning queries, with intelligence efficiency improving 5.3× from 2023 to 2025. The models and hardware are increasingly ready. What has been missing is the software stack to make local-first personal AI practical.

OpenJarvis is that stack. It is a framework for local-first personal AI, built around three core ideas: shared primitives for building on-device agents; evaluations that treat energy, FLOPs, latency, and dollar cost as first-class constraints alongside accuracy; and a learning loop that improves models using local trace data. The goal is simple: make it possible to build personal AI agents that run locally by default, calling the cloud only when truly necessary. OpenJarvis aims to be both a research platform and a production foundation for local AI, in the spirit of PyTorch.

Installation

Pick your platform and run one command. Each installer handles uv, the Python venv, Ollama, and a starter model — about 3 minutes on broadband.

Platform One-liner
macOS · Linux · WSL2 curl -fsSL https://open-jarvis.github.io/OpenJarvis/install.sh | bash
Native Windows irm https://open-jarvis.github.io/OpenJarvis/install.ps1 | iex
Desktop GUI Download .exe / .dmg / .deb / .rpm / .AppImage from the latest release

Then jarvis to start. The Rust extension and larger models continue downloading in the background; jarvis doctor shows status.

Platform-specific notes (WSL2 setup, native-Windows scheduled-task service, desktop prerequisites, manual / contributor install): see the installation docs.

Quick Start

jarvis                          # start chatting (default: chat-simple)
jarvis init --preset <name>     # switch to a starter config

Prefix jarvis ... with uv run, or source .venv/bin/activate first.

Preset What it does
morning-digest-mac / morning-digest-linux / morning-digest-minimal Spoken daily briefing from email, calendar, health, news
deep-research Multi-hop research across indexed docs with citations
code-assistant Agent with code execution, file I/O, and shell access
scheduled-monitor Stateful agent on a schedule with memory
chat-simple Lightweight conversation, no tools

Example:

jarvis init --preset morning-digest-mac
jarvis connect gdrive          # one OAuth covers Gmail / Calendar / Tasks
jarvis digest --fresh          # generate and play your first briefing

Per-preset deep dives: morning digest · deep research · code assistant · scheduled monitor · chat simple · or the full quickstart guide.

Skills

Skills teach agents how to better use tools and improve their reasoning. Every skill is a tool — agents discover them from a catalog and invoke them on demand.

# Install skills from public sources
jarvis skill install hermes:arxiv
jarvis skill sync hermes --category research

# Use skills with any agent
jarvis ask "Use the code-explainer skill to explain this Python code: for i in range(5): print(i*2)"

# Optimize skills from your trace history
jarvis optimize skills --policy dspy

# Benchmark the impact
jarvis bench skills --max-samples 5 --seeds 42

Import from Hermes Agent (~150 skills), OpenClaw (~13,700 community skills), or any GitHub repo. Skills follow the agentskills.io open standard.

See the Skills User Guide and Skills Tutorial for details.

Built-in Agents

OpenJarvis ships with eight built-in agents across three execution modes (on-demand, scheduled, continuous):

Agent Type What it does
morning_digest Scheduled Daily briefing from email, calendar, health, news — with TTS audio
deep_research On-demand Multi-hop research with citations across web and local docs
monitor_operative Continuous Long-horizon monitoring with memory, compression, and retrieval
orchestrator On-demand Multi-turn reasoning with automatic tool selection
native_react On-demand ReAct (Thought-Action-Observation) loop agent
operative Continuous Persistent autonomous agent with state management
native_openhands On-demand CodeAct — generates and executes Python code
simple On-demand Single-turn chat, no tools

See the User Guide and Tutorials for detailed setup instructions.

Full documentation — including Docker deployment, cloud engines, development setup, and tutorials — at open-jarvis.github.io/OpenJarvis.

Community

Contributing

We welcome contributions! See the Contributing Guide for incentives, contribution types, and the PR process.

Quick start for contributors:

git clone https://github.com/open-jarvis/OpenJarvis.git
cd OpenJarvis
uv sync --extra dev
uv run pre-commit install
uv run pytest tests/ -v

Browse the Roadmap for areas where help is needed. Comment "take" on any issue to get auto-assigned.

About

OpenJarvis is part of Intelligence Per Watt, a research initiative studying the intelligence efficiency of AI systems. The project is developed at Hazy Research and the Scaling Intelligence Lab at Stanford SAIL.

Sponsors

Laude InstituteStanford MarloweGoogle Cloud PlatformLambda LabsOllamaIBM ResearchStanford HAI

Citation

@misc{saadfalcon2026openjarvispersonalaipersonal,
      title={OpenJarvis: Personal AI, On Personal Devices}, 
      author={Jon Saad-Falcon and Avanika Narayan and Robby Manihani and Tanvir Bhathal and Herumb Shandilya and Hakki Orhun Akengin and Gabriel Bo and Andrew Park and Matthew Hart and Caia Costello and Chuan Li and Christopher Ré and Azalia Mirhoseini},
      year={2026},
      eprint={2605.17172},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/2605.17172}, 
}

License

Apache 2.0

S
Description
No description provided
Readme
104 MiB
Languages
Python 82.8%
Rust 8.6%
TypeScript 7.1%
Shell 0.7%
HTML 0.3%
Other 0.5%