Commit Graph
3 Commits
Author SHA1 Message Date
Jon Saad-FalconandClaude Opus 4.6 d89bbcce52 feat: upgrade eval pipeline with agentic runner, telemetry session, and savings meter
Add 9 capabilities to match IPW pipeline:

Eval Pipeline:
- AgenticRunner for multi-turn agent execution with per-turn trace decomposition
- QueryTrace/TurnTrace data model for agentic workload telemetry
- EventRecorder for thread-safe agent event collection
- TerminalBenchTaskEnv for Docker-based task execution
- Cost computation via engine/cloud.py PRICING table
- Rich export: JSONL, HF Arrow, summary JSON, artifacts manifest
- CLI: --agentic, --concurrency, --query-timeout flags

Telemetry:
- TelemetrySession with background-sampling ring buffer (Python fallback)
- Phase metrics: prefill/decode energy split at TTFT boundary
- ITL percentile tracking (p50/p90/p95/p99)
- FLOPs estimation and MFU computation
- EnergyMonitor.snapshot() method

Rust Performance Layer:
- Ring buffer with binary search O(log n) window queries
- Trapezoidal energy integration
- Phase metrics, ITL stats, FLOPs estimation in Rust
- PyO3 bindings for all new telemetry modules (50 Rust tests)

Savings Meter & Benchmarks:
- Use-case benchmark datasets (coding, email, research, knowledge, morning brief)
- Savings dashboard component with cost comparison visualization
- Cloud cost calculator and comparison server routes
- Use-case eval configs for multiple agent/engine combinations

Tests: 80 new tests (3779 total pass, 37 skipped, 0 failures)
Lint: ruff check src/ tests/ — all checks passed

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-06 04:22:55 +00:00
Gabriel Bo 62dd0ce6c9 quick start local on doc page + gh pg redesign setup 2026-03-03 14:03:26 -08:00
Jon Saad-FalconandClaude Opus 4.6 a4c4081ff4 Add desktop distribution pipeline: rolling releases, auto-updates, code signing
- Rewrite .github/workflows/desktop.yml: 2-job pipeline (validate + build-and-release)
  with rolling desktop-latest pre-release on push to main and stable desktop-v* releases
- Add UpdateChecker component: checks for updates on startup + every 30 min,
  background download with progress bar, one-click relaunch
- Configure Tauri updater: endpoints pointing to desktop-latest release, pubkey placeholder
- Add tauri-plugin-process for relaunch support (Cargo.toml, lib.rs, package.json)
- Add macOS Entitlements.plist for notarization (network + file access, no sandbox)
- Add scripts/bump-desktop-version.sh for atomic version bumps across 3 config files
- Add desktop/README.md with dev setup, auto-update architecture, signing docs
- Update .gitignore for desktop/node_modules, dist, target
- Configure macOS minimumSystemVersion, Windows timestampUrl
- Include all Phase 14-21 work: agent hardening, RBAC, taint tracking, workflows,
  skills, knowledge graph, sessions, A2A, MCP templates, WASM sandbox, TUI dashboard,
  production tools, CLI expansion, API expansion, learning productionization,
  Tauri desktop app, and 10 new channels

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-27 19:14:05 +00:00