mirror of
https://github.com/garrytan/gbrain.git
synced 2026-07-27 22:15:33 +00:00
3933eb6a79
* docs(designs): 2026-05 embedder shootout eval plan Adds docs/designs/2026_05_EVAL_PLAN.md — the approved plan + 6 Conductor session briefs for the OpenAI vs Voyage vs ZeroEntropy embedder comparison. Why: produce a publishable comparison report for v0.35.x release notes pinning "which embedder wins, and does zerank-2 carry the win for ZeroEntropy" against public LongMemEval + in-house BrainBench. Each session brief is self-contained — repo, branch, commits, verify, ship, deliverable, hand-off. Stewardable one section per Conductor session. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(pricing): add voyage-4-large + zembed-1 to EMBEDDING_PRICING v0.35.0.0 shipped ZeroEntropy zembed-1 + zerank-2 reranker support and expanded the Voyage allow-list to include voyage-4-large. The pricing table missed both, so `gbrain upgrade`'s post-upgrade reembed prompt silently fell back to "estimate unavailable" for users on these models. - voyage:voyage-4-large @ $0.18/MTok (same as voyage-3-large) - zeroentropyai:zembed-1 @ $0.05/MTok New test file pins both entries plus the openai/voyage-3-large baselines, case-insensitive provider matching, bare-model openai-default fallback, table integrity (lowercase providers, finite non-negative prices), and the estimateCostFromChars approximation. 11 cases, 46 expect() calls. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(exports): expose gbrain/ai/gateway with canary test Adds ./ai/gateway to the package.json exports map so external eval consumers (notably gbrain-evals, the sibling repo running the embedder shootout in docs/designs/2026_05_EVAL_PLAN.md) can call configureGateway directly to swap embedding providers per cell. Why: pre-v0.35.1.0, gbrain-evals adapters hardcoded gbrain/embedding, which means every retrieval adapter was OpenAI-only. The newly-exposed gateway lets adapters route through Voyage and ZeroEntropy without forking gbrain or duplicating the recipe wiring. - package.json: add "./ai/gateway" -> "./src/core/ai/gateway.ts" - scripts/check-exports-count.sh: bump expected count 17 -> 18 - test/public-exports.test.ts: add canary pinning configureGateway + embed, bump expected count assertion Pre-existing import-resolution failures in this test file (16 on master) are unrelated to this change — they're a longstanding Bun package self-import behavior. The count + EXPECTED_EXPORTS list-match assertions both pass cleanly. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(eval): add --resume-from <jsonl> to gbrain eval longmemeval Multi-cell embedder shootouts spend $50+/cell on the gpt-4o judge after gbrain emits hypotheses. A mid-run abort (rate-limit, cost-cap, OS interrupt, SIGKILL) previously meant re-paying the full cell. This flag makes those aborts cheap: re-invoke with --resume-from pointed at the partial JSONL and only the unanswered question_ids re-run. Behavior: - Read question_ids from the file; skip them on this run. - Rows with non-empty hypothesis count as done. - Rows with hypothesis="" AND an error field are NOT skipped (retry case for per-question failures recorded by the existing try/catch). - Corrupt trailing lines (SIGKILL'd writer mid-line) are silently skipped with a stderr warn. - When --resume-from path == --output path, the output emitter opens the file in append mode instead of truncating, so the existing rows survive. - Empty resume case (all questions already done) returns immediately without spinning up the brain or calling the client. New exported helper loadResumeSet() makes the parser unit-testable. 6 new test cases pinning: - File-not-found returns empty set - Well-formed JSONL load - Error-row retry semantics (empty hypothesis + error -> not in set) - Truncated final line recovery - End-to-end resume against the 5-question mini fixture - All-done early-return (stub client must NOT be invoked) All 18 cases in test/eval-longmemeval.test.ts green; bun run typecheck clean. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore: v0.35.1.0 Bumps VERSION + package.json + CHANGELOG entry for the embedder-shootout prereq release. Three additive changes from the prior 4 commits: - pricing: voyage-4-large + zembed-1 entries - exports: gbrain/ai/gateway is now public - eval: gbrain eval longmemeval --resume-from <jsonl> Each commit on this branch is independently bisect-friendly and CI-green; the CHANGELOG entry is the user-facing rollup. No migrations, no breaking changes — the gateway export expands the surface, the resume-from flag is additive, the pricing patch only changes "estimate unavailable" -> a real dollar figure for two specific models. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
49 lines
2.0 KiB
Bash
Executable File
49 lines
2.0 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
# CI guard: the public exports surface never shrinks silently (v0.21.0).
|
|
#
|
|
# Precedent: scripts/check-jsonb-pattern.sh + check-progress-to-stdout.sh
|
|
# are grep-based structural guards wired into `bun run test`. This one
|
|
# counts the entries in package.json "exports" and fails when the count
|
|
# drops below the v0.21.0 baseline (17 entries).
|
|
#
|
|
# Policy (from CLAUDE.md):
|
|
# "Removing any of these is a breaking change going forward."
|
|
#
|
|
# If you're legitimately removing a public export: bump gbrain's minor
|
|
# version, note the removal in CHANGELOG.md under a "Breaking changes"
|
|
# bullet, then bump EXPECTED_COUNT below. Anything else is a regression.
|
|
#
|
|
# Adding a new export: update EXPECTED_COUNT to match AND extend the
|
|
# EXPECTED_EXPORTS list in test/public-exports.test.ts so the runtime
|
|
# contract test pins the canary symbol.
|
|
|
|
set -euo pipefail
|
|
|
|
EXPECTED_COUNT=18
|
|
|
|
# Count top-level keys in the exports object. `node -e` parses JSON
|
|
# reliably without needing jq (which isn't in every CI environment).
|
|
ACTUAL=$(node -e "
|
|
const pkg = require('./package.json');
|
|
console.log(Object.keys(pkg.exports || {}).length);
|
|
")
|
|
|
|
if [ "$ACTUAL" -lt "$EXPECTED_COUNT" ]; then
|
|
echo "❌ public-exports guard: package.json exports shrank from $EXPECTED_COUNT to $ACTUAL"
|
|
echo " Removing a public export is a breaking change (see CLAUDE.md)."
|
|
echo " If intentional: bump gbrain minor version + update EXPECTED_COUNT in"
|
|
echo " scripts/check-exports-count.sh and EXPECTED_EXPORTS in"
|
|
echo " test/public-exports.test.ts, AND add a CHANGELOG 'Breaking changes' bullet."
|
|
exit 1
|
|
fi
|
|
|
|
if [ "$ACTUAL" -gt "$EXPECTED_COUNT" ]; then
|
|
echo "⚠️ public-exports guard: package.json exports grew from $EXPECTED_COUNT to $ACTUAL"
|
|
echo " Additive public API change. Update EXPECTED_COUNT in this script + the"
|
|
echo " EXPECTED_EXPORTS list in test/public-exports.test.ts to lock the new"
|
|
echo " canary symbols."
|
|
exit 1
|
|
fi
|
|
|
|
echo "✓ public-exports guard: $ACTUAL entries (matches baseline $EXPECTED_COUNT)"
|