mirror of
https://github.com/garrytan/gbrain.git
synced 2026-07-30 11:22:34 +00:00
b91350d778
* fix(autopilot,eval): nightly quality probe enable path works end-to-end + wire conversation-parser probe Takeover of #2629 and #2630 (rebased onto master; dropped the test/engine-find-trajectory.test.ts hunk both PRs carried — master already ships the equivalent gateway-dims fix). #2629 — nightly quality probe enable path: - autopilot + doctor read the probe flag dual-plane (DB config row from 'gbrain config set' wins, ~/.gbrain/config.json fallback) via new resolveProbeEnabled/resolveProbeMaxUsd helpers - resolveRepoRoot prefers the gbrain package root where the committed fixture lives, not the brain repoPath - rate_limited skips no longer write an audit row every autopilot cycle - eval-longmemeval strips 'provider:' recipe ids before raw Anthropic SDK calls and emits the gold answer for downstream judges - cross-modal batch folds the gold answer into the judge task; probe passes QA-shaped dimensions instead of the agent-response rubric - DEFAULT_SLOTS slot A moves to openai:gpt-5.2 (gpt-4o left the recipe); new consistency test pins every default slot to its recipe #2630 — conversation-parser nightly probe wire-up: - autopilot step 4.6 invokes runConversationParserNightlyProbe (dual-plane flag + D10 tokenmax mode-gate, package-root fixtures, 24h gate, audit trail via new src/core/audit-parser-probe.ts) - doctor's conversation_parser_probe_health replaces the hardcoded 'Skipped' stub with a real pure-function check over the audit trail Co-authored-by: p3ob7o <p3ob7o@users.noreply.github.com> Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(pricing): add openai:gpt-5.2 canonical entry for the new default slot A DEFAULT_SLOTS slot A moved to openai:gpt-5.2, which had no CANONICAL_PRICING entry — estimateCost silently dropped slot A from the --max-usd pre-flight and est_cost_usd audit rows (~1/3 under-count on the default panel). Rates from the OpenAI recipe chat touchpoint (verified 2026-04-20). Also refresh the --slot-a-model help text default and pin a pricing-presence assertion in the DEFAULT_SLOTS consistency test. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Sinabina <sinabina@Sinabinas-MacBook-Pro-4.local> Co-authored-by: p3ob7o <p3ob7o@users.noreply.github.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: Garry Tan <garrytan@gmail.com>
64 lines
2.2 KiB
TypeScript
64 lines
2.2 KiB
TypeScript
/**
|
|
* Nightly conversation-parser probe audit trail.
|
|
*
|
|
* One event per REAL probe run lands in
|
|
* `~/.gbrain/audit/parser-probe-YYYY-Www.jsonl` (ISO-week rotation via the
|
|
* shared audit-writer primitive; honors `GBRAIN_AUDIT_DIR`).
|
|
* Scheduler-cadence skips (`rate_limited`) are NOT logged — the autopilot
|
|
* loop ticks every few minutes, so logging every skip would flood the
|
|
* audit file with rows that carry no signal.
|
|
*
|
|
* Read by `gbrain doctor`'s `conversation_parser_probe_health` check and
|
|
* by the autopilot wiring's 24h rate-limit gate (`parserProbeRanWithin`).
|
|
*/
|
|
|
|
import { createAuditWriter } from './audit/audit-writer.ts';
|
|
import type { NightlyProbeResult } from './conversation-parser/nightly-probe.ts';
|
|
|
|
export type ParserProbeAuditEvent = NightlyProbeResult;
|
|
|
|
const writer = createAuditWriter<ParserProbeAuditEvent>({
|
|
featureName: 'parser-probe',
|
|
errorLabel: 'gbrain',
|
|
errorMessagePrefix: 'parser-probe audit ',
|
|
errorTrailer: '; probe continues',
|
|
});
|
|
|
|
/** Append one parser-probe event. Best-effort; never throws. */
|
|
export function logParserProbeEvent(event: ParserProbeAuditEvent): void {
|
|
writer.log(event);
|
|
}
|
|
|
|
/**
|
|
* Read recent parser-probe events (current + previous ISO week, filtered
|
|
* to the window). Missing files and corrupt rows are skipped silently.
|
|
*/
|
|
export function readRecentParserProbeEvents(
|
|
days = 7,
|
|
now: Date = new Date(),
|
|
): ParserProbeAuditEvent[] {
|
|
return writer.readRecent(days, now);
|
|
}
|
|
|
|
/** Exposed for tests pinning the rotation edge cases. */
|
|
export function computeParserProbeAuditFilename(now: Date = new Date()): string {
|
|
return writer.computeFilename(now);
|
|
}
|
|
|
|
/**
|
|
* 24h rate-limit gate for the autopilot wiring: true when any audited run
|
|
* happened within `windowMs` of `now`. Only REAL outcomes are audited (see
|
|
* module header), so a pass/fail today blocks re-runs until tomorrow while
|
|
* scheduler-cadence skips never extend the window.
|
|
*/
|
|
export function parserProbeRanWithin(
|
|
windowMs: number,
|
|
now: Date = new Date(),
|
|
): boolean {
|
|
const cutoff = now.getTime() - windowMs;
|
|
return readRecentParserProbeEvents(2, now).some((ev) => {
|
|
const ts = Date.parse(ev.ts);
|
|
return Number.isFinite(ts) && ts >= cutoff;
|
|
});
|
|
}
|