# Benchmarks The benchmarking framework measures inference engine performance with reproducible, standardized tests. It includes built-in benchmarks for latency and throughput, a suite runner for batch execution, and support for custom benchmarks. ## Overview OpenJarvis ships with two benchmarks: | Benchmark | Registry Key | Measures | |---------------|----------------|-----------------------------------------------| | **Latency** | `latency` | Per-call inference latency (mean, p50, p95, min, max) | | **Throughput**| `throughput` | Tokens per second throughput | --- ## BaseBenchmark ABC All benchmarks implement the `BaseBenchmark` abstract base class. ```python from abc import ABC, abstractmethod from openjarvis.bench._stubs import BenchmarkResult from openjarvis.engine._stubs import InferenceEngine class BaseBenchmark(ABC): @property @abstractmethod def name(self) -> str: """Short identifier for this benchmark.""" @property @abstractmethod def description(self) -> str: """Human-readable description of what this benchmark measures.""" @abstractmethod def run( self, engine: InferenceEngine, model: str, *, num_samples: int = 10, ) -> BenchmarkResult: """Execute the benchmark and return results.""" ``` ### BenchmarkResult Each benchmark run produces a `BenchmarkResult`: | Field | Type | Description | |------------------|------------------|------------------------------------------| | `benchmark_name` | `str` | Name of the benchmark | | `model` | `str` | Model used | | `engine` | `str` | Engine backend used | | `metrics` | `dict[str, float]` | Key-value pairs of measured metrics | | `metadata` | `dict[str, Any]` | Additional metadata | | `samples` | `int` | Number of samples run | | `errors` | `int` | Number of errors encountered | --- ## Built-in Benchmarks ### Latency Benchmark Measures per-call inference latency using short, fixed prompts. Each sample sends a simple prompt to the engine and measures wall-clock time. **Prompts used:** The benchmark rotates through a set of short canned prompts ("Hello", "What is 2+2?", "Explain gravity in one sentence") to keep input variation consistent across runs. **Metrics produced:** | Metric | Description | |-----------------|-----------------------------------------------------| | `mean_latency` | Average latency across all successful samples | | `p50_latency` | Median latency (50th percentile) | | `p95_latency` | 95th percentile latency (tail performance) | | `min_latency` | Fastest single call | | `max_latency` | Slowest single call | **Example output:** ``` latency (10 samples, 0 errors) mean_latency: 0.2345 p50_latency: 0.2100 p95_latency: 0.3800 min_latency: 0.1500 max_latency: 0.4200 ``` ### Throughput Benchmark Measures inference throughput in tokens per second. Each sample sends a longer prompt ("Write a short paragraph about artificial intelligence") and measures both the time taken and the number of completion tokens generated. **Metrics produced:** | Metric | Description | |-----------------------|------------------------------------------------| | `tokens_per_second` | Total completion tokens / total time | | `total_tokens` | Total completion tokens across all samples | | `total_time_seconds` | Total wall-clock time across all samples | **Example output:** ``` throughput (10 samples, 0 errors) tokens_per_second: 45.6789 total_tokens: 1250.0000 total_time_seconds: 27.3600 ``` --- ## Interpreting Results ### Latency Metrics - **mean_latency:** The average response time. Use this for general performance comparison. - **p50_latency (median):** The typical response time. Less affected by outliers than the mean. - **p95_latency:** The worst-case response time for 95% of requests. Critical for user experience -- if this is too high, some users will experience noticeable delays. - **min/max_latency:** The best and worst individual calls. A large gap between min and max indicates inconsistent performance. !!! tip "What to look for" A healthy setup has `p95 / p50 < 2`. If the p95 is much higher than the median, investigate whether the engine is experiencing contention, thermal throttling, or memory pressure. ### Throughput Metrics - **tokens_per_second:** The main throughput indicator. Higher is better. Typical ranges: - CPU-only: 5-20 tokens/second - Consumer GPU (RTX 3060-4090): 30-100 tokens/second - Data-center GPU (A100, H100): 100-500+ tokens/second - **total_tokens / total_time:** The raw data behind the throughput calculation. Useful for verifying that the engine is generating meaningful output (not returning empty responses). --- ## BenchmarkSuite The `BenchmarkSuite` class runs a collection of benchmarks and provides aggregation and serialization utilities. ```python from openjarvis.bench._stubs import BenchmarkSuite from openjarvis.bench.latency import LatencyBenchmark from openjarvis.bench.throughput import ThroughputBenchmark suite = BenchmarkSuite([LatencyBenchmark(), ThroughputBenchmark()]) # Run all benchmarks results = suite.run_all(engine, model, num_samples=20) # Serialize to JSONL (one JSON object per line) jsonl = suite.to_jsonl(results) # Get a summary dict summary = suite.summary(results) ``` ### Methods | Method | Returns | Description | |-------------------------|--------------------|--------------------------------------------| | `run_all(engine, model, num_samples=10)` | `list[BenchmarkResult]` | Run all benchmarks sequentially | | `to_jsonl(results)` | `str` | Serialize results to JSONL format | | `summary(results)` | `dict[str, Any]` | Create a summary dictionary | ### JSONL Format Each line in the JSONL output is a JSON object: ```json {"benchmark_name": "latency", "model": "qwen3:8b", "engine": "ollama", "metrics": {"mean_latency": 0.234, "p50_latency": 0.21, "p95_latency": 0.38, "min_latency": 0.15, "max_latency": 0.42}, "metadata": {}, "samples": 10, "errors": 0} {"benchmark_name": "throughput", "model": "qwen3:8b", "engine": "ollama", "metrics": {"tokens_per_second": 45.67, "total_tokens": 1250.0, "total_time_seconds": 27.36}, "metadata": {}, "samples": 10, "errors": 0} ``` ### Summary Format ```json { "benchmark_count": 2, "benchmarks": [ { "name": "latency", "model": "qwen3:8b", "engine": "ollama", "metrics": {"mean_latency": 0.234, ...}, "samples": 10, "errors": 0 }, { "name": "throughput", "model": "qwen3:8b", "engine": "ollama", "metrics": {"tokens_per_second": 45.67, ...}, "samples": 10, "errors": 0 } ] } ``` --- ## CLI Usage ```bash # Run all benchmarks with default settings (10 samples) jarvis bench run # Run with more samples for better statistical accuracy jarvis bench run -n 50 # Run only the latency benchmark jarvis bench run -b latency # Run only the throughput benchmark with 20 samples jarvis bench run -b throughput -n 20 # Specify model and engine jarvis bench run -m qwen3:8b -e ollama # Output JSON summary to stdout jarvis bench run --json # Write JSONL results to a file jarvis bench run -o results.jsonl # Combine options jarvis bench run -b latency -n 100 -m qwen3:8b --json -o latency.jsonl ``` | Option | Type | Default | Description | |----------------------------|--------|---------|------------------------------------------| | `-m`, `--model MODEL` | string | auto | Model to benchmark | | `-e`, `--engine ENGINE` | string | auto | Engine backend | | `-n`, `--samples N` | int | `10` | Number of samples per benchmark | | `-b`, `--benchmark NAME` | string | all | Specific benchmark to run (`latency` or `throughput`) | | `-o`, `--output PATH` | path | none | Write JSONL results to file | | `--json` | flag | off | Output JSON summary to stdout | --- ## Adding Custom Benchmarks Create a custom benchmark by subclassing `BaseBenchmark` and registering it with the `BenchmarkRegistry`. ### Step 1: Implement the Benchmark ```python import time from openjarvis.bench._stubs import BaseBenchmark, BenchmarkResult from openjarvis.core.registry import BenchmarkRegistry from openjarvis.core.types import Message, Role from openjarvis.engine._stubs import InferenceEngine class ContextLengthBenchmark(BaseBenchmark): """Measures how latency scales with input length.""" @property def name(self) -> str: return "context_length" @property def description(self) -> str: return "Measures latency scaling with increasing input length" def run( self, engine: InferenceEngine, model: str, *, num_samples: int = 10, ) -> BenchmarkResult: latencies = {} errors = 0 for length in [100, 500, 1000, 2000]: prompt = "x " * length messages = [Message(role=Role.USER, content=prompt)] t0 = time.time() try: engine.generate(messages, model=model) latencies[f"latency_{length}_tokens"] = time.time() - t0 except Exception: errors += 1 return BenchmarkResult( benchmark_name=self.name, model=model, engine=engine.engine_id, metrics=latencies, samples=len(latencies), errors=errors, ) ``` ### Step 2: Register the Benchmark Use the `ensure_registered()` pattern to survive registry clearing in tests: ```python def ensure_registered() -> None: """Register the benchmark if not already present.""" if not BenchmarkRegistry.contains("context_length"): BenchmarkRegistry.register_value("context_length", ContextLengthBenchmark) ``` Alternatively, use the decorator at class definition time: ```python @BenchmarkRegistry.register("context_length") class ContextLengthBenchmark(BaseBenchmark): ... ``` !!! info "The `ensure_registered()` Pattern" The `ensure_registered()` function is preferred over the decorator for benchmark modules because it survives registry clearing during testing. The built-in `latency` and `throughput` benchmarks both use this pattern. The benchmark CLI command calls `ensure_registered()` before looking up benchmarks. ### Step 3: Use Your Benchmark Once registered, your benchmark is available through the CLI: ```bash jarvis bench run -b context_length ``` And through the `BenchmarkSuite`: ```python from openjarvis.core.registry import BenchmarkRegistry bench_cls = BenchmarkRegistry.get("context_length") bench = bench_cls() result = bench.run(engine, model, num_samples=5) ```