Commit Graph
8 Commits
Author SHA1 Message Date
Jon Saad-FalconandClaude Opus 4.6 784b880130 test(evals): add PinchBench integration tests — full grading pipeline
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-25 09:31:02 -07:00
Jon Saad-FalconandClaude Opus 4.6 73cb1f1bbd feat(evals): add PinchBench grading helpers — transcript translation, automated/LLM judge/hybrid scoring
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-25 09:26:12 -07:00
Jon Saad-FalconandClaude Opus 4.6 8e8d4a68e4 feat(evals): add PinchBench dataset provider — repo clone and task markdown parsing
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-25 09:25:39 -07:00
Jon Saad-FalconandClaude Sonnet 4.6 c5cb6a9e49 feat(evals): add tool_calls field to TurnTrace for rich tool data
Adds a `tool_calls` list field to `TurnTrace` capturing name, arguments,
and result for each tool invocation, enabling PinchBench transcript integration.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-25 09:20:25 -07:00
Jon Saad-Falcon 00f71ae504 fix: update energy scaling test for linear KV-cache FLOPs model
The FLOPs formula changed from quadratic (P*N*(N+1)) to linear
(2*P*T_evaluated) with KV-cache awareness. Update the test
expectation from ~100x to ~10x for 10x token increase.
2026-03-24 18:16:11 -07:00
8468f08c41 fix: correct O(N²) energy/FLOPs savings calculation (#95) (#97)
The energy_wh_saved and flops_saved values were orders of magnitude too
high because a scaling factor that grows linearly with N was applied to
the energy calculation, making it scale as O(N³) instead of O(N²).

Replace the buggy scale-factor approach with a direct FLOP-to-energy
conversion using each provider's per-token constants. Add regression
tests to prevent recurrence.

Closes #95

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-20 18:52:18 -07:00
Prathap PandGitHub c2756964a7 fix(channels): wire channel→agent handler and fix Telegram send pipeline (#94)
* fix(channels): wire channel→agent handler and fix Telegram send pipeline

* format code

* add supported tests
2026-03-20 18:35:18 -07:00
Jon Saad-Falconandkrypticmouse 8798e2ee4f init commit 2026-03-12 17:29:39 +00:00