mirror of
https://github.com/garrytan/gbrain.git
synced 2026-07-27 22:15:33 +00:00
* feat(core): shared computeSyncDelta + spend-posture module (#2139) sync-delta.ts: ONE implementation of "what changed since last_commit", consumed by both the sync executor and the inline cost estimator so the gate's dollar figure can't drift from what the sync imports. spend-posture.ts: spend.posture config + parseUsdLimit/formatUsdLimit off-switch parsing (off/unlimited/none → Infinity; undefined at the budget boundary so ledger rows never serialize null). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(sync): delta-aware cost estimator + non-TTY auto-defer + per-source failure acks (#2139) The inline-embed cost gate was a ~400x phantom: it priced the entire tree whenever the working tree was dirty (always, on an active brain), then blocked the daily cron with exit 2. Now: - performSyncInner + the estimator both route through computeSyncDelta, so the estimate mirrors execution (fetch-first delta; dirty-but-caught-up tree → $0). - shouldBlockSync is posture-aware; non-TTY above floor AUTO-DEFERS embeds to capped backfill jobs (exit 0) instead of wedging — single shared runInlineCostGate on both --all and single-source paths. - --full prices delta + stale backlog (full sync sweeps it inline). - off/unlimited on the cost knobs; tokenmax bypasses the backfill cap (still ledgered) but never the cooldown. - --skip-failed/--retry-failed scoped per source; the D15 parallel refusal is lifted (the #1939 ledger is per-source + lock-serialized). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(config): register spend-control keys + validate spend.posture (#2139) Adds spend.posture + the five previously --force-only spend knobs to KNOWN_CONFIG_KEYS so `config set` accepts them directly (removes the archaeology the issue complained about), and rejects invalid spend.posture values at set time with a paste-ready hint. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(reindex,enrich,onboard): spend.posture across the remaining cost gates (#2139) reindex-code: tokenmax makes the cost gate informational; --max-cost accepts off/unlimited. enrich + onboard --auto: tokenmax lifts the refuse-without-cap guardrail and runs UNCAPPED (spend still ledgered by BudgetTracker). Explicit --max-usd always wins over posture. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test: cost-gate, delta estimator, spend-posture, off-switch coverage (#2139) New sync-delta + sync-cost-estimate unit suites; rewritten cost-gate serial tests (auto-defer instead of exit 2, posture, off-switch, format split, single-source); parseUsdLimit/posture-aware shouldBlockSync; backfill cap-off + tokenmax-bypass + cooldown-still-refuses; config known-key acceptance. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(spend-controls): single spend-control surface + ref-map + follow-up TODOs (#2139) New docs/operations/spend-controls.md (every gate, key, default, off switch, posture interaction); CLAUDE.md reference-map row; two P3 follow-up TODOs (measured chunk-count gating, per-source defer granularity). llms bundles regenerated. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(spend): SSRF-harden estimator fetch + complete off/uncapped across reindex/enrich/onboard (#2139) Ship-stage codex pre-landing review caught four P1s in the secondary cost gates: - The delta estimator's fetch-first ran `git fetch` through the plain git() helper, bypassing the GIT_SSRF_FLAGS + GIT_TERMINAL_PROMPT=0 hardening that real sync uses. Added `fetchRemote()` to git-remote.ts (same flags as pullRepo) and route the estimator through it — a cost preview / dry-run can no longer hit a remote through a less-protected path. - `reindex --max-cost off`, `enrich --max-usd off`, `onboard --auto --max-usd off` were parsed but didn't actually proceed/uncap. Now: explicit off (and spend.posture=tokenmax) proceed past the confirmation/missing-cap refusal AND run uncapped. enrich threads an Infinity sentinel mapped to "no BudgetTracker ceiling" (never raw Infinity → no null in audit rows); reindex/onboard use their native undefined=uncapped path. Spend still ledgered. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore: bump version and changelog (v0.42.45.0) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(KEY_FILES): update sync/embedding/git-remote/reindex entries to post-#2139 truth document-release pass: the cost-gate entries described the pre-#2139 behavior (full-tree-ceiling estimator, --skip-failed-rejects-under-parallel, exit-2 confirmation gate). Updated to current truth — delta-aware estimator via the shared computeSyncDelta, per-source failure acks under parallel, non-TTY auto-defer (no exit 2), posture-aware shouldBlockSync. Added entries for the two new core modules (sync-delta.ts, spend-posture.ts) + fetchRemote on git-remote.ts + reindex --max-cost off. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Garry Tan
co-authored by
Claude Opus 4.8
Claude Opus 4.8
parent
090bb53203
commit
5c49225e4b
@@ -2,6 +2,28 @@
|
||||
|
||||
All notable changes to GBrain will be documented in this file.
|
||||
|
||||
## [0.42.45.0] - 2026-06-13
|
||||
|
||||
**The daily sync cron stops wedging on cost, and the embedding-spend estimate finally matches what a sync actually does (gbrain#2139).** On an active brain the inline-embed cost gate priced the *entire* corpus every time the working tree was dirty — which is always, since agents and crons write to it constantly — so a routine daily sync estimated ~158M tokens / ~$8 when the real delta was a few hundred files / ~$0.04, then blocked the cron with a confirmation it could never answer. Embeds silently stalled until someone noticed. The estimate now mirrors execution: it fetches first and prices only the files this run will pull and import, through the same diff machinery the sync itself uses. A brain whose commits are caught up but whose tree is dirty estimates $0, because an attached-HEAD sync imports only the committed diff.
|
||||
|
||||
When the gate does fire in a non-interactive session, it no longer exits with an error — it imports now and defers embedding to capped background jobs (which drain via the jobs worker or `gbrain embed --stale`), so a cron is never wedged again. Operators who have decided cost isn't the constraint get one switch — `gbrain config set spend.posture tokenmax` — that makes every cost gate informational across sync, reindex, enrich, and onboard (spend is still recorded; the switch removes the ceiling, not the accounting). The USD knobs accept `off` / `unlimited`, and every gate message now carries paste-ready commands so the controls are discoverable at the moment they fire.
|
||||
|
||||
This release also lifts the rule that blocked `--skip-failed` / `--retry-failed` under parallel sync — failure recovery no longer has to drop to `--serial` (which is what armed the inline gate in the first place).
|
||||
|
||||
### Added
|
||||
- **`spend.posture` config** — `tokenmax` makes every embedding-cost gate informational (print the estimate, proceed, keep the ledger); `gated` (default) enforces as before. Documented end-to-end in `docs/operations/spend-controls.md`.
|
||||
- **First-class off switches** — `sync.cost_gate_min_usd`, `embed.backfill_max_usd_per_source_24h`, `embed.backfill_max_usd`, and `reindex --max-cost` / `enrich --max-usd` accept `off` / `unlimited` / `none`. No more sentinel values like `100000`.
|
||||
- **Single-source `gbrain sync` cost preview** — plain `gbrain sync` previously embedded inline with no preview; it now carries the same gate as `sync --all` (auto-defers in non-TTY sessions, never blocks).
|
||||
- **Self-describing gate messages** — every cost-gate / FYI line ends with the exact `gbrain config set` commands to widen, disable, or switch posture, plus a docs pointer.
|
||||
|
||||
### Changed
|
||||
- **`gbrain sync --all` is no longer blocked by the cost gate in cron/agent contexts.** Above the floor in a non-interactive session it auto-defers embeds (exit 0) instead of emitting a `cost_preview_requires_yes` envelope and exiting 2. Cron wrappers that branched on exit 2 now see exit 0 with `status: "auto_deferred"`. A TTY still prompts `[y/N]`; `--yes` still embeds inline.
|
||||
- **`--skip-failed` / `--retry-failed` now work under parallel sync.** The failure ledger is per-source and lock-serialized, so the previous "not supported under parallel — re-run with --serial" refusal is retired.
|
||||
- **The six spend-control config keys are now first-class** (`gbrain config set` accepts them without `--force`).
|
||||
|
||||
### To take advantage of v0.42.45.0
|
||||
`gbrain upgrade`. Nothing to configure for the headline fix — the daily sync cron stops wedging and the estimate is accurate out of the box. If you run a high-volume brain where cost genuinely isn't the constraint, `gbrain config set spend.posture tokenmax` makes every gate informational. To widen or disable a specific gate instead, see the table in `docs/operations/spend-controls.md` (e.g. `gbrain config set sync.cost_gate_min_usd off`). Failure-recovery syncs can now stay parallel: `gbrain sync --all --skip-failed` no longer forces `--serial`.
|
||||
|
||||
## [0.42.44.0] - 2026-06-13
|
||||
|
||||
### Fixed
|
||||
|
||||
@@ -97,6 +97,7 @@ detail on demand.)
|
||||
| any file in `src/` (what it does + its invariants) | `docs/architecture/KEY_FILES.md` — find the file's entry |
|
||||
| search / ranking / hybrid / retrieval | `docs/architecture/RETRIEVAL.md` + the `search/*` entries in `KEY_FILES.md` |
|
||||
| search modes / cost knobs | `docs/guides/search-modes.md` |
|
||||
| embedding spend gates / cost gate / `spend.posture` / off switches | `docs/operations/spend-controls.md` |
|
||||
| push-based context (volunteer/watch/reflex window) | `docs/guides/push-context.md` |
|
||||
| schema packs / page types / extraction | `docs/architecture/schema-packs.md`, `type-taxonomy.md`, `lens-packs.md` |
|
||||
| thin-client / remote MCP / cross-modal | `docs/architecture/thin-client.md` |
|
||||
|
||||
@@ -1,5 +1,28 @@
|
||||
# TODOS
|
||||
|
||||
## Spend-controls wave follow-ups (filed v0.42.45.0, #2139)
|
||||
|
||||
Deferred from the #2139 delta-estimator wave. See plan + GSTACK REVIEW REPORT at
|
||||
`~/.claude/plans/system-instruction-you-are-working-lovely-balloon.md`.
|
||||
|
||||
- [ ] **P3 — Measured post-import chunk-count gating (#2139 proposal 2b).**
|
||||
**What:** Gate the inline cost decision on the actual chunk count sync produced
|
||||
(known after import, before embedding) instead of the pre-sync token estimate.
|
||||
**Why:** A fully execution-accurate gate with zero estimate error. **Context:**
|
||||
After v0.42.42.0 the estimator already mirrors execution (fetch-first delta via the
|
||||
shared `computeSyncDelta`, `--full`=delta+stale, dirty-tree→$0). This is the
|
||||
belt-and-suspenders fallback if a future case still drifts. **Trigger:** only if the
|
||||
delta estimator proves insufficient in practice. **Start:** the gate call site in
|
||||
`src/commands/sync.ts` (`runInlineCostGate`), gate on post-import `chunksCreated`.
|
||||
- [ ] **P3 — Per-source defer granularity (#2139, D8A road-not-taken).**
|
||||
**What:** When the aggregate inline gate trips in a non-TTY session, defer embeds
|
||||
only for sources above a per-source floor; let cheap sources keep embedding inline.
|
||||
**Why:** Cheap sources would get embeddings minutes sooner instead of waiting for a
|
||||
backfill-worker drain. **Context:** v0.42.42.0 chose GLOBAL defer (one flag, strictly
|
||||
dominates the exit-2 it replaced). This is the granularity upgrade. **Trigger:** a
|
||||
filed embedding-latency-by-minutes complaint. **Start:** thread per-source estimates
|
||||
through `runOne` (`src/commands/sync.ts`); design worked out at D8A in the plan.
|
||||
|
||||
## gbrain#2095 push-based context follow-ups (v0.43+)
|
||||
|
||||
Filed from the #2095 wave (volunteer_context op + reflex window + `gbrain watch`).
|
||||
|
||||
File diff suppressed because one or more lines are too long
@@ -0,0 +1,111 @@
|
||||
# Spend controls
|
||||
|
||||
GBrain's embedding-spend gates in one place: every gate, its config key, default,
|
||||
whether it blocks or just informs, how to widen or disable it, and how the
|
||||
`spend.posture` switch governs all of them.
|
||||
|
||||
The orienting idea: **GBrain itself is rounding error; the spend that matters is
|
||||
downstream embedding.** These gates exist so a routine sync or enrich can't run up
|
||||
an unexpected embedding bill, while never wedging an unattended cron.
|
||||
|
||||
## `spend.posture` — one switch for "cost is not my constraint"
|
||||
|
||||
```bash
|
||||
gbrain config set spend.posture tokenmax # all cost gates become informational
|
||||
gbrain config set spend.posture gated # default — gates enforce
|
||||
```
|
||||
|
||||
| Value | Effect |
|
||||
|-------|--------|
|
||||
| `gated` (default) | Every cost gate enforces its limit as documented below. |
|
||||
| `tokenmax` | Every cost gate prints its estimate and **proceeds** — informational only. Spend is still recorded to the ledger; posture removes the *ceiling*, not the *accounting*. |
|
||||
|
||||
`spend.posture` is deliberately separate from `search.mode=tokenmax` (which governs
|
||||
retrieval payload size, not embedding spend). When a gate fires and
|
||||
`search.mode=tokenmax` but `spend.posture` is unset, the gate prints a one-line hint
|
||||
pointing at this switch.
|
||||
|
||||
**Precedence:** an explicit per-call cap (`--max-usd N`, `--max-cost N`) always wins
|
||||
over posture. `tokenmax` only governs the default/absent case — it never overrides a
|
||||
number you typed on the command line.
|
||||
|
||||
## Off switches (`off` / `unlimited` / `none`)
|
||||
|
||||
The USD-limit knobs accept `off`, `unlimited`, or `none` (case-insensitive) to mean
|
||||
"no limit" — no more setting sentinel values like `100000`.
|
||||
|
||||
- `0` is **not** "off". On `sync.cost_gate_min_usd`, `0` means "block on any nonzero
|
||||
spend" (a real choice). On the backfill caps, `0` falls back to the default.
|
||||
- Internally "no limit" is the string `unlimited` in any printed/JSON output and "no
|
||||
cap" inside the budget tracker — never a raw `Infinity` (which would serialize to
|
||||
`null` in ledger rows).
|
||||
|
||||
## The gates
|
||||
|
||||
| Gate | Config key | Default | Blocks? | Off switch | tokenmax |
|
||||
|------|-----------|---------|---------|-----------|----------|
|
||||
| Sync inline-embed cost gate | `sync.cost_gate_min_usd` | `0.50` | TTY prompt / non-TTY auto-defer | `off` (or `0` = block-on-any) | informational |
|
||||
| Backfill 24h per-source spend cap | `embed.backfill_max_usd_per_source_24h` | `25` | refuses submission | `off` (`0` → default) | bypassed (still ledgered) |
|
||||
| Backfill per-job budget | `embed.backfill_max_usd` | `10` | caps the job's tracker | `off` (`0` → default) | uncapped (still ledgered) |
|
||||
| Backfill cooldown | `embed.backfill_cooldown_min` | `10` | skips re-submission inside window | — (latency knob, not spend) | **not** bypassed |
|
||||
| `reindex-code` cost gate | — (preview before re-embed) | — | TTY prompt / non-TTY refuse + exit 2 | `--max-cost off` | informational |
|
||||
| `enrich` / `onboard --auto` | `--max-usd` (per-call) | — | refuse without a cap (non-TTY) | `--max-usd off` | runs uncapped (still ledgered) |
|
||||
|
||||
### Sync inline-embed cost gate
|
||||
|
||||
Fires only when sync embeds **inline** (federated_v2 off, or `--serial` without
|
||||
`--no-embed`). Under federated_v2 + parallel, embedding is deferred to capped backfill
|
||||
jobs and the gate is informational. The estimate prices the **delta** — the files this
|
||||
sync will actually import (fetched-first, so it sees commits the run is about to pull) —
|
||||
not the whole tree. A busy brain with a dirty working tree but caught-up commits
|
||||
estimates `$0`, because an attached-HEAD sync imports only the committed diff.
|
||||
|
||||
Behavior above the floor:
|
||||
- **TTY:** prompts `[y/N]`.
|
||||
- **Non-interactive (cron/agent):** **auto-defers** embeds to capped backfill jobs and
|
||||
exits 0 — it never wedges the pipeline. The backlog drains via the jobs worker or
|
||||
`gbrain embed --stale`. Pass `--yes` to embed inline instead.
|
||||
|
||||
Output format splits on the explicit `--json` flag: `--json` emits a structured
|
||||
envelope; otherwise human text. Every gate message carries paste-ready knobs.
|
||||
|
||||
`--full` re-embeds the stale backlog inline (full sync sweeps it), so a `--full`
|
||||
estimate is `delta + stale backlog`, labeled as such.
|
||||
|
||||
### Estimate labels
|
||||
|
||||
- `~N tokens (delta: changed files since last sync)` — the precise estimate.
|
||||
- `<=N tokens (full-tree ceiling for K source(s): <reasons> …)` — a conservative
|
||||
over-count used only when a precise delta can't be computed: a first sync, a chunker
|
||||
version drift (forces a full re-chunk), or git being unavailable. Unchanged files
|
||||
still skip via `content_hash` at execution, so the ceiling over-states real spend.
|
||||
|
||||
## Notes & limits
|
||||
|
||||
- **Pre-pull window:** the gate fetches before estimating, so it prices what the run
|
||||
will pull. If a fetch fails (offline), it estimates against local HEAD and labels the
|
||||
result; the bounded residual is priced on the next run.
|
||||
- **Single-source `gbrain sync`** carries the same gate as `sync --all` (it previously
|
||||
embedded inline with no preview).
|
||||
- **Recovery under parallel:** `--skip-failed` / `--retry-failed` work under parallel
|
||||
sync (the failure ledger is per-source and lock-serialized) — you no longer have to
|
||||
drop to `--serial`, which is what used to arm the inline gate.
|
||||
|
||||
## Escape hatches at a glance
|
||||
|
||||
```bash
|
||||
# Never gate this brain on cost:
|
||||
gbrain config set spend.posture tokenmax
|
||||
|
||||
# Widen the sync inline floor to $5:
|
||||
gbrain config set sync.cost_gate_min_usd 5
|
||||
|
||||
# Disable the sync inline floor entirely:
|
||||
gbrain config set sync.cost_gate_min_usd off
|
||||
|
||||
# Lift the backfill 24h spend cap:
|
||||
gbrain config set embed.backfill_max_usd_per_source_24h off
|
||||
|
||||
# Run enrich uncapped non-interactively:
|
||||
gbrain enrich --max-usd off # or: gbrain config set spend.posture tokenmax
|
||||
```
|
||||
@@ -246,6 +246,7 @@ detail on demand.)
|
||||
| any file in `src/` (what it does + its invariants) | `docs/architecture/KEY_FILES.md` — find the file's entry |
|
||||
| search / ranking / hybrid / retrieval | `docs/architecture/RETRIEVAL.md` + the `search/*` entries in `KEY_FILES.md` |
|
||||
| search modes / cost knobs | `docs/guides/search-modes.md` |
|
||||
| embedding spend gates / cost gate / `spend.posture` / off switches | `docs/operations/spend-controls.md` |
|
||||
| push-based context (volunteer/watch/reflex window) | `docs/guides/push-context.md` |
|
||||
| schema packs / page types / extraction | `docs/architecture/schema-packs.md`, `type-taxonomy.md`, `lens-packs.md` |
|
||||
| thin-client / remote MCP / cross-modal | `docs/architecture/thin-client.md` |
|
||||
|
||||
+1
-1
@@ -143,5 +143,5 @@
|
||||
"bun": ">=1.3.10"
|
||||
},
|
||||
"license": "MIT",
|
||||
"version": "0.42.44.0"
|
||||
"version": "0.42.45.0"
|
||||
}
|
||||
|
||||
@@ -181,6 +181,20 @@ export async function runConfig(engine: BrainEngine, args: string[]) {
|
||||
const coverageOverride =
|
||||
args.includes('--coverage-override') || args.includes('--yes');
|
||||
|
||||
// v0.42.42.0 (#2139): validate spend.posture at set time so a typo
|
||||
// ('tokenMax', 'max') doesn't silently fall back to gated.
|
||||
if (key === 'spend.posture') {
|
||||
const { isValidSpendPosture } = await import('../core/spend-posture.ts');
|
||||
if (!isValidSpendPosture(value)) {
|
||||
console.error(
|
||||
`[config] spend.posture must be 'gated' or 'tokenmax' (got '${value}').\n` +
|
||||
`[config] gbrain config set spend.posture tokenmax # cost gates become informational\n` +
|
||||
`[config] gbrain config set spend.posture gated # default — gates enforce`,
|
||||
);
|
||||
process.exit(1);
|
||||
}
|
||||
}
|
||||
|
||||
if (key === 'embedding_columns') {
|
||||
try {
|
||||
const parsed = JSON.parse(value);
|
||||
|
||||
+36
-7
@@ -508,8 +508,13 @@ export async function runEnrichCore(
|
||||
// One tracker reference for both the run and the post-hoc overage check.
|
||||
// External tracker (cycle phase): used as-is, no withBudgetTracker wrap (that
|
||||
// would REPLACE not stack). Internal: capped at maxCostUsd ?? DEFAULT.
|
||||
// v0.42.42.0 (#2139): Infinity = explicit "uncapped" (off / tokenmax) → pass
|
||||
// `undefined` so BudgetTracker runs without a ceiling (NOT raw Infinity, which
|
||||
// would serialize to null in audit rows). undefined-when-unset still → DEFAULT.
|
||||
const resolvedCap =
|
||||
opts.maxCostUsd === Infinity ? undefined : (opts.maxCostUsd ?? DEFAULT_MAX_COST_USD);
|
||||
const tracker = opts.budgetTracker ?? new BudgetTracker({
|
||||
maxCostUsd: opts.maxCostUsd ?? DEFAULT_MAX_COST_USD,
|
||||
maxCostUsd: resolvedCap,
|
||||
label: `enrich:${sourceId}`,
|
||||
});
|
||||
try {
|
||||
@@ -622,8 +627,15 @@ export function parseArgs(args: string[]): ParsedArgs {
|
||||
continue;
|
||||
}
|
||||
if (a === '--max-usd' || a === '--max-cost-usd') {
|
||||
const n = parseFloat(args[++i] ?? '');
|
||||
if (Number.isFinite(n) && n > 0) out.maxCostUsd = n;
|
||||
const raw = args[++i] ?? '';
|
||||
// v0.42.42.0 (#2139): off/unlimited/none → run uncapped (Infinity sentinel;
|
||||
// mapped to "no BudgetTracker ceiling" in runEnrichCore). Spend still ledgered.
|
||||
if (['off', 'unlimited', 'none'].includes(raw.trim().toLowerCase())) {
|
||||
out.maxCostUsd = Infinity;
|
||||
} else {
|
||||
const n = parseFloat(raw);
|
||||
if (Number.isFinite(n) && n > 0) out.maxCostUsd = n;
|
||||
}
|
||||
continue;
|
||||
}
|
||||
if (a === '--min-context') {
|
||||
@@ -801,9 +813,24 @@ export async function runEnrich(engine: BrainEngine, args: string[]): Promise<vo
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
// v0.42.42.0 (#2139, D15A): enrich runs UNCAPPED when the operator says cost
|
||||
// isn't the constraint — either explicit `--max-usd off` (parsed to Infinity)
|
||||
// or `spend.posture=tokenmax` with no per-call cap. Uncapped → the missing-cap
|
||||
// refusals lift AND runEnrichCore passes no ceiling to the BudgetTracker (spend
|
||||
// still ledgered; posture removes the ceiling, not the accounting). An explicit
|
||||
// finite --max-usd always wins (precedence: per-call > posture).
|
||||
const explicitOff = parsed.maxCostUsd === Infinity;
|
||||
const { resolveSpendPosture } = await import('../core/spend-posture.ts');
|
||||
const posture = parsed.dryRun ? 'gated' : await resolveSpendPosture(engine);
|
||||
const uncapped =
|
||||
!parsed.dryRun && (explicitOff || (parsed.maxCostUsd === undefined && posture === 'tokenmax'));
|
||||
if (uncapped) {
|
||||
console.error(`${explicitOff ? '--max-usd off' : 'spend.posture=tokenmax'}: running uncapped, spend ledgered. docs: docs/operations/spend-controls.md`);
|
||||
}
|
||||
|
||||
// Non-TTY execute without --max-usd or --yes is refused (cost guardrail).
|
||||
if (!parsed.dryRun && parsed.maxCostUsd === undefined && !parsed.yes && !process.stdout.isTTY) {
|
||||
console.error('Refusing to spend without a cap in a non-interactive context. Pass --max-usd <FLOAT> or --yes.');
|
||||
if (!parsed.dryRun && parsed.maxCostUsd === undefined && !parsed.yes && !process.stdout.isTTY && !uncapped) {
|
||||
console.error('Refusing to spend without a cap in a non-interactive context. Pass --max-usd <FLOAT> (or `off`), --yes, or set spend.posture=tokenmax.');
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
@@ -812,7 +839,7 @@ export async function runEnrich(engine: BrainEngine, args: string[]): Promise<vo
|
||||
: (await listSources(engine)).map((s) => s.id);
|
||||
|
||||
// Dry-run cost preview (TTY) before spending.
|
||||
if (!parsed.dryRun && process.stdout.isTTY && !parsed.yes && parsed.maxCostUsd === undefined) {
|
||||
if (!parsed.dryRun && process.stdout.isTTY && !parsed.yes && parsed.maxCostUsd === undefined && !uncapped) {
|
||||
const limit = parsed.limit ?? DEFAULT_LIMIT;
|
||||
const est = (limit * sourceIds.length * COST_ESTIMATE_PER_PAGE_USD).toFixed(2);
|
||||
console.error(`About to enrich up to ${limit} page(s) per source across ${sourceIds.length} source(s), est. ~$${est}. Re-run with --max-usd or --yes to confirm.`);
|
||||
@@ -834,7 +861,9 @@ export async function runEnrich(engine: BrainEngine, args: string[]): Promise<vo
|
||||
limit: parsed.limit,
|
||||
workers: parsed.workers,
|
||||
model: parsed.model,
|
||||
maxCostUsd: parsed.maxCostUsd,
|
||||
// uncapped (off / tokenmax) → Infinity sentinel; runEnrichCore maps it
|
||||
// to "no BudgetTracker ceiling".
|
||||
maxCostUsd: uncapped ? Infinity : parsed.maxCostUsd,
|
||||
minContextChars: parsed.minContextChars,
|
||||
thinThreshold: parsed.thinThreshold,
|
||||
reenrichAfterMs: parsed.reenrichAfterMs,
|
||||
|
||||
+23
-6
@@ -45,6 +45,13 @@ export async function runOnboard(engine: BrainEngine, args: string[]): Promise<v
|
||||
// No-op without a pack_upgrade_available finding.
|
||||
const explain = args.includes('--explain');
|
||||
const targetScore = parseInt10(args, '--target-score') ?? 90;
|
||||
// v0.42.42.0 (#2139): `--max-usd off`/`unlimited`/`none` → run uncapped. maxUsd
|
||||
// stays undefined, which runRemediation already treats as no ceiling (skips the
|
||||
// est-cost refusal + BudgetTracker runs uncapped); `maxUsdOff` lifts the
|
||||
// missing-cap refusal below. Spend is still ledgered.
|
||||
const maxUsdIdx = args.indexOf('--max-usd');
|
||||
const maxUsdVal = maxUsdIdx >= 0 ? (args[maxUsdIdx + 1] ?? '').trim().toLowerCase() : '';
|
||||
const maxUsdOff = ['off', 'unlimited', 'none'].includes(maxUsdVal);
|
||||
const maxUsdRaw = parseFloat10(args, '--max-usd');
|
||||
const maxUsd = maxUsdRaw === null ? undefined : maxUsdRaw;
|
||||
|
||||
@@ -91,13 +98,23 @@ export async function runOnboard(engine: BrainEngine, args: string[]): Promise<v
|
||||
}
|
||||
|
||||
// --auto refuses without --max-usd (cron-safety per A12 + A20).
|
||||
// v0.42.42.0 (#2139, D15A): spend.posture=tokenmax lifts the refusal —
|
||||
// --auto runs UNCAPPED (maxUsd stays undefined), spend still ledgered by the
|
||||
// remediation budget tracker. An explicit --max-usd always wins.
|
||||
if (auto && maxUsd === undefined) {
|
||||
process.stderr.write(
|
||||
`gbrain onboard --auto refuses without --max-usd N.\n` +
|
||||
`Set a cap to avoid surprise spend:\n` +
|
||||
` gbrain onboard --auto --max-usd 5\n`,
|
||||
);
|
||||
process.exit(2);
|
||||
const { resolveSpendPosture } = await import('../core/spend-posture.ts');
|
||||
const tokenmax = (await resolveSpendPosture(engine)) === 'tokenmax';
|
||||
if (maxUsdOff || tokenmax) {
|
||||
process.stderr.write(`${maxUsdOff ? '--max-usd off' : 'spend.posture=tokenmax'}: onboard --auto running uncapped, spend ledgered. docs: docs/operations/spend-controls.md\n`);
|
||||
} else {
|
||||
process.stderr.write(
|
||||
`gbrain onboard --auto refuses without --max-usd N.\n` +
|
||||
`Set a cap to avoid surprise spend:\n` +
|
||||
` gbrain onboard --auto --max-usd 5\n` +
|
||||
`Or pass --max-usd off / set spend.posture=tokenmax to run uncapped. docs: docs/operations/spend-controls.md\n`,
|
||||
);
|
||||
process.exit(2);
|
||||
}
|
||||
}
|
||||
|
||||
// Build the plan: T4 checks supply extra remediations on top of T3's
|
||||
|
||||
@@ -444,14 +444,24 @@ export async function runReindexCodeCli(engine: BrainEngine, args: string[]): Pr
|
||||
}
|
||||
|
||||
// F3: --max-cost / --max-cost-usd both accepted for symmetry with brainstorm.
|
||||
// v0.42.42.0 (#2139): `off`/`unlimited`/`none` → no runtime cap AND an explicit
|
||||
// "cost isn't the constraint" decision that proceeds past the confirmation gate
|
||||
// (like --yes). Numeric must be positive; `0`/garbage is rejected.
|
||||
let maxCostUsd: number | undefined;
|
||||
let maxCostOff = false;
|
||||
for (const flag of ['--max-cost', '--max-cost-usd']) {
|
||||
const idx = args.indexOf(flag);
|
||||
if (idx >= 0) {
|
||||
const v = args[idx + 1];
|
||||
const t = (v ?? '').trim().toLowerCase();
|
||||
if (['off', 'unlimited', 'none'].includes(t)) {
|
||||
maxCostUsd = undefined; // no runtime cap (reindex skips the tracker when unset)
|
||||
maxCostOff = true;
|
||||
break;
|
||||
}
|
||||
const n = v ? parseFloat(v) : NaN;
|
||||
if (!Number.isFinite(n) || n <= 0) {
|
||||
console.error(`gbrain reindex --code: ${flag} requires a positive number in USD (got ${v ?? '(missing)'})`);
|
||||
console.error(`gbrain reindex --code: ${flag} requires a positive number in USD, or off/unlimited (got ${v ?? '(missing)'})`);
|
||||
process.exit(2);
|
||||
}
|
||||
maxCostUsd = n;
|
||||
@@ -493,20 +503,36 @@ export async function runReindexCodeCli(engine: BrainEngine, args: string[]): Pr
|
||||
}
|
||||
|
||||
if (!yes) {
|
||||
const isTTY = Boolean(process.stdout.isTTY) && Boolean(process.stdin.isTTY);
|
||||
if (!isTTY || json) {
|
||||
// Guardrail unchanged: refuse + exit 2, no spend. Only the FORMAT splits
|
||||
// on --json now (human refusal on stderr otherwise) — #1784.
|
||||
const refusal = buildCostRefusal({ json, previewMsg, preview, costUsd, model: getEmbeddingModelName() });
|
||||
if (refusal.stdout) console.log(refusal.stdout);
|
||||
if (refusal.stderr) console.error(refusal.stderr);
|
||||
process.exit(2);
|
||||
}
|
||||
console.log(previewMsg);
|
||||
const answer = await promptYesNo('Proceed? [y/N] ');
|
||||
if (!answer) {
|
||||
console.log('Cancelled.');
|
||||
return;
|
||||
// v0.42.42.0 (#2139): spend.posture=tokenmax makes the gate informational
|
||||
// — print the estimate and proceed (the operator declared cost isn't the
|
||||
// constraint). The spend is still ledgered by the runtime BudgetTracker.
|
||||
const { resolveSpendPosture } = await import('../core/spend-posture.ts');
|
||||
const posture = await resolveSpendPosture(engine);
|
||||
// An explicit `--max-cost off` is the same "cost isn't the constraint"
|
||||
// signal as spend.posture=tokenmax — proceed past the confirmation gate.
|
||||
if (posture === 'tokenmax' || maxCostOff) {
|
||||
const gate = maxCostOff ? 'max_cost_off' : 'posture_tokenmax';
|
||||
if (json) {
|
||||
console.log(JSON.stringify({ status: 'proceeding', gate, codePages: preview.totalPages, totalTokens: preview.totalTokens, costUsd, model: getEmbeddingModelName() }));
|
||||
} else {
|
||||
console.log(`${previewMsg} ${maxCostOff ? '--max-cost off' : 'spend.posture=tokenmax'}: proceeding (informational). docs: docs/operations/spend-controls.md`);
|
||||
}
|
||||
} else {
|
||||
const isTTY = Boolean(process.stdout.isTTY) && Boolean(process.stdin.isTTY);
|
||||
if (!isTTY || json) {
|
||||
// Guardrail unchanged: refuse + exit 2, no spend. Only the FORMAT splits
|
||||
// on --json now (human refusal on stderr otherwise) — #1784.
|
||||
const refusal = buildCostRefusal({ json, previewMsg, preview, costUsd, model: getEmbeddingModelName() });
|
||||
if (refusal.stdout) console.log(refusal.stdout);
|
||||
if (refusal.stderr) console.error(refusal.stderr);
|
||||
process.exit(2);
|
||||
}
|
||||
console.log(previewMsg);
|
||||
const answer = await promptYesNo('Proceed? [y/N] ');
|
||||
if (!answer) {
|
||||
console.log('Cancelled.');
|
||||
return;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
+530
-209
@@ -7,12 +7,11 @@ import { importFile } from '../core/import-file.ts';
|
||||
import { collectSyncableFiles } from './import.ts';
|
||||
import { createInterface } from 'readline';
|
||||
import {
|
||||
buildSyncManifest,
|
||||
isSyncable,
|
||||
unsyncableReason,
|
||||
resolveSlugForPath,
|
||||
unacknowledgedSyncFailures,
|
||||
acknowledgeSyncFailures,
|
||||
acknowledgeFailures,
|
||||
loadSyncFailures,
|
||||
formatCodeBreakdown,
|
||||
applySyncFailureGate,
|
||||
@@ -20,6 +19,16 @@ import {
|
||||
resolveAutoSkipThreshold,
|
||||
DEFAULT_SOURCE_ID,
|
||||
} from '../core/sync.ts';
|
||||
import {
|
||||
computeSyncDelta,
|
||||
buildDetachedWorkingTreeManifest,
|
||||
} from '../core/sync-delta.ts';
|
||||
import { fetchRemote } from '../core/git-remote.ts';
|
||||
import {
|
||||
parseUsdLimit,
|
||||
formatUsdLimit,
|
||||
resolveSpendPosture,
|
||||
} from '../core/spend-posture.ts';
|
||||
import { estimateTokens, CHUNKER_VERSION } from '../core/chunkers/code.ts';
|
||||
import {
|
||||
estimateEmbeddingCostUsd,
|
||||
@@ -28,11 +37,10 @@ import {
|
||||
currentEmbeddingSignature,
|
||||
willEmbedSynchronously,
|
||||
shouldBlockSync,
|
||||
type SyncEmbedMode,
|
||||
} from '../core/embedding.ts';
|
||||
import { estimateCostFromChars } from '../core/embedding-pricing.ts';
|
||||
import { isSourceUnchangedSinceSync } from '../core/git-head.ts';
|
||||
import { SPEND_CAP_CONFIG_KEY } from '../core/embed-backfill-submit.ts';
|
||||
import { errorFor, serializeError } from '../core/errors.ts';
|
||||
import type { SyncManifest } from '../core/sync.ts';
|
||||
import { createProgress } from '../core/progress.ts';
|
||||
import { getCliOptions, cliOptsToProgressOptions } from '../core/cli-options.ts';
|
||||
@@ -214,16 +222,17 @@ export interface SyncResult {
|
||||
|
||||
/**
|
||||
* Walk ONE source's working tree and sum tokens for every syncable file.
|
||||
* Conservative full-tree ceiling (full file content, not the incremental
|
||||
* diff) — over-counts, never under-counts, and matches the filesystem set
|
||||
* `sync` actually imports (collectSyncableFiles + content_hash, NOT a git
|
||||
* commit diff). Best-effort per file and per source: anything unreadable
|
||||
* contributes 0 rather than blocking the preview.
|
||||
* Conservative full-tree CEILING (full file content, not the incremental
|
||||
* diff) — over-counts, never under-counts. Used only on the ceiling rungs of
|
||||
* `estimateInlineNewTokens` (first sync, chunker drift, git-unavailable),
|
||||
* where the delta is genuinely the whole tree or can't be computed.
|
||||
*
|
||||
* v0.31.2: routed through collectSyncableFiles (lstat + inode-cycle +
|
||||
* max-depth) so the preview walks exactly what the real sync walks.
|
||||
*
|
||||
* Exported (v0.42.42.0, #2139) for direct unit testing.
|
||||
*/
|
||||
function estimateSourceTreeTokens(
|
||||
export function estimateSourceTreeTokens(
|
||||
localPath: string,
|
||||
strategy: 'markdown' | 'code' | 'auto',
|
||||
): { tokens: number; files: number } {
|
||||
@@ -248,18 +257,114 @@ function estimateSourceTreeTokens(
|
||||
return { tokens, files };
|
||||
}
|
||||
|
||||
/** Sum tokens for an explicit set of repo-relative paths read at live working-tree content. */
|
||||
function estimateDeltaTokens(localPath: string, relPaths: string[]): number {
|
||||
let tokens = 0;
|
||||
for (const rel of relPaths) {
|
||||
try {
|
||||
const full = join(localPath, rel);
|
||||
const stat = statSync(full);
|
||||
if (stat.size > 5_000_000) continue; // skip large binaries (matches tree walk)
|
||||
tokens += estimateTokens(readFileSync(full, 'utf-8'));
|
||||
} catch {
|
||||
// Listed in the diff but unreadable (e.g. since deleted) → 0, like the tree walk.
|
||||
}
|
||||
}
|
||||
return tokens;
|
||||
}
|
||||
|
||||
/**
|
||||
* v0.41.31 — INLINE-path new-content estimate. Per source, contribute ZERO
|
||||
* when the source is provably unchanged since its last sync (HEAD ==
|
||||
* last_commit AND clean working tree AND chunker_version matches CURRENT) —
|
||||
* `content_hash` short-circuits every file so nothing re-embeds. Otherwise
|
||||
* contribute the full-tree ceiling. The unchanged predicate mirrors
|
||||
* doctor's `sync_freshness` and sync's own "do work?" gate (sync.ts:1057+
|
||||
* 1075). `isSourceUnchangedSinceSync` is fail-open (probe error → false), so
|
||||
* a source we can't prove unchanged is conservatively re-estimated rather
|
||||
* than silently priced at $0.
|
||||
* v0.42.42.0 (#2139): resolve the commit the estimate should diff AGAINST.
|
||||
*
|
||||
* The cost gate runs BEFORE sync's own `git pull`, so a stale local HEAD would
|
||||
* make the estimate blind to commits the run is about to pull (codex #1). So
|
||||
* we FETCH first (fail-open) and target `origin/<branch>` — the estimate then
|
||||
* prices exactly what this run will sync. The subsequent pull fast-forwards
|
||||
* the already-fetched objects, so net new network cost ≈ 0.
|
||||
*
|
||||
* - detached HEAD → no upstream; target = local HEAD (+ caller merges the
|
||||
* detached working-tree manifest, which sync imports on a detached repo).
|
||||
* - attached + origin remote → fetch origin/<branch> (best-effort), target =
|
||||
* origin/<branch> if resolvable, else local HEAD (offline / no upstream).
|
||||
* - HEAD unresolvable (not a git repo) → null (caller treats as unavailable).
|
||||
*
|
||||
* NOTE: this makes `--dry-run` perform a network fetch so the preview reflects
|
||||
* what a real run would pull. Fail-open: offline dry-run still previews against
|
||||
* local HEAD. Uses the shared `git()` 30s budget (the fetch cost is the pull
|
||||
* cost paid a few seconds early — a tighter cap would frequently fall back to
|
||||
* local HEAD and underestimate the remote delta).
|
||||
*/
|
||||
function estimateInlineNewTokens(
|
||||
function resolveEstimateTarget(localPath: string): { target: string; detached: boolean } | null {
|
||||
let head: string;
|
||||
try {
|
||||
head = git(localPath, ['rev-parse', 'HEAD']);
|
||||
} catch {
|
||||
return null;
|
||||
}
|
||||
const detached = isDetachedHead(localPath);
|
||||
if (detached) return { target: head, detached: true };
|
||||
|
||||
let branch: string | null = null;
|
||||
try {
|
||||
branch = git(localPath, ['rev-parse', '--abbrev-ref', 'HEAD']).trim() || null;
|
||||
} catch {
|
||||
branch = null;
|
||||
}
|
||||
if (branch && branch !== 'HEAD' && hasOriginRemote(localPath)) {
|
||||
try {
|
||||
// v0.42.42.0 (#2139): route through the SSRF-hardened fetch (same flags +
|
||||
// no-prompt env as pullRepo) — a cost preview / dry-run must NOT hit a
|
||||
// remote through a less-protected path than real sync.
|
||||
fetchRemote(localPath, branch, { timeoutMs: 30_000 });
|
||||
} catch {
|
||||
// fail-open: offline, auth failure, no upstream — fall through to local HEAD.
|
||||
}
|
||||
try {
|
||||
const remoteSha = git(localPath, ['rev-parse', `origin/${branch}`]);
|
||||
if (remoteSha) return { target: remoteSha, detached: false };
|
||||
} catch {
|
||||
// no remote-tracking ref for this branch — use local HEAD.
|
||||
}
|
||||
}
|
||||
return { target: head, detached: false };
|
||||
}
|
||||
|
||||
export type EstimateKind = 'delta' | 'ceiling' | 'mixed' | 'unchanged';
|
||||
|
||||
export interface InlineEstimate {
|
||||
tokens: number;
|
||||
changedSources: number;
|
||||
unchangedSources: number;
|
||||
estimateKind: EstimateKind;
|
||||
/** Per-source ceiling reasons (chunker_drift / first_sync / git_unavailable) for honest labeling. */
|
||||
ceilingReasons: string[];
|
||||
}
|
||||
|
||||
/**
|
||||
* v0.42.42.0 (#2139) — INLINE-path new-content estimate. The estimate now
|
||||
* MIRRORS EXECUTION instead of pricing the whole tree on every dirty sync (the
|
||||
* 400x overestimate that wedged the daily cron). Per-source fail-open ladder:
|
||||
*
|
||||
* 1. syncEnabled === false → skip (unchanged)
|
||||
* 2. chunker drift (stored !== current) → full-tree CEILING (a drift forces
|
||||
* performFullSync → full re-chunk → full re-embed; a delta would
|
||||
* underestimate by the whole corpus). kind: ceiling_chunker_drift
|
||||
* 3. last_commit === fetch target → 0 (mirrors `up_to_date` at
|
||||
* sync.ts:1402 — NO clean-working-tree requirement; a dirty tree whose
|
||||
* commits are caught up imports nothing). kind: unchanged
|
||||
* 4. last_commit === null (first sync) → full-tree CEILING. kind: ceiling_first_sync
|
||||
* 5. computeSyncDelta ok → price added∪modified∪renamed.to
|
||||
* (syncable, live working-tree content); deletes cost 0. kind: delta
|
||||
* 6. computeSyncDelta unavailable → full-tree CEILING. kind: ceiling_git_unavailable
|
||||
*
|
||||
* The delta rung routes through the SAME `computeSyncDelta` the executor uses
|
||||
* (src/core/sync-delta.ts), so the gate's dollar figure can't drift from what
|
||||
* the sync imports. `--full`'s extra stale-backlog sweep is added by the gate
|
||||
* (it already has `staleCostUsd`), not here — see the call site.
|
||||
*
|
||||
* Exported for direct unit testing.
|
||||
*/
|
||||
export function estimateInlineNewTokens(
|
||||
sources: Array<{
|
||||
local_path: string | null;
|
||||
config: Record<string, unknown>;
|
||||
@@ -267,25 +372,85 @@ function estimateInlineNewTokens(
|
||||
chunker_version: string | null;
|
||||
}>,
|
||||
currentChunkerVersion: string,
|
||||
): { tokens: number; changedSources: number; unchangedSources: number } {
|
||||
): InlineEstimate {
|
||||
let tokens = 0;
|
||||
let changedSources = 0;
|
||||
let unchangedSources = 0;
|
||||
let hadDelta = false;
|
||||
let hadCeiling = false;
|
||||
const ceilingReasons: string[] = [];
|
||||
|
||||
const ceiling = (localPath: string, strategy: 'markdown' | 'code' | 'auto', reason: string) => {
|
||||
tokens += estimateSourceTreeTokens(localPath, strategy).tokens;
|
||||
changedSources++;
|
||||
hadCeiling = true;
|
||||
ceilingReasons.push(reason);
|
||||
};
|
||||
|
||||
for (const src of sources) {
|
||||
if (!src.local_path) continue;
|
||||
const cfg = (src.config || {}) as { syncEnabled?: boolean; strategy?: 'markdown' | 'code' | 'auto' };
|
||||
if (cfg.syncEnabled === false) continue;
|
||||
const unchanged =
|
||||
isSourceUnchangedSinceSync(src.local_path, src.last_commit, { requireCleanWorkingTree: true }) &&
|
||||
src.chunker_version === currentChunkerVersion;
|
||||
if (unchanged) {
|
||||
const strategy = cfg.strategy ?? 'markdown';
|
||||
const localPath = src.local_path;
|
||||
|
||||
// Rung 2: chunker drift forces a full re-chunk → full re-embed. CEILING.
|
||||
if (src.chunker_version !== currentChunkerVersion) {
|
||||
ceiling(localPath, strategy, 'chunker_drift');
|
||||
continue;
|
||||
}
|
||||
|
||||
// Rung 4 (early): no bookmark → first sync imports everything. CEILING.
|
||||
if (!src.last_commit) {
|
||||
ceiling(localPath, strategy, 'first_sync');
|
||||
continue;
|
||||
}
|
||||
|
||||
const resolved = resolveEstimateTarget(localPath);
|
||||
if (!resolved) {
|
||||
// HEAD unresolvable (not a git repo / gone) — can't compute a delta. CEILING.
|
||||
ceiling(localPath, strategy, 'git_unavailable');
|
||||
continue;
|
||||
}
|
||||
|
||||
// Rung 3: caught up to the fetch target AND no detached working-tree changes.
|
||||
// Mirrors the executor's `up_to_date` predicate — a dirty-but-committed-current
|
||||
// tree imports nothing, so it must price $0 (the heart of the false-fire fix).
|
||||
const detachedManifest = resolved.detached
|
||||
? buildDetachedWorkingTreeManifest(localPath)
|
||||
: null;
|
||||
const detachedHasChanges = detachedManifest !== null &&
|
||||
(detachedManifest.added.length > 0 ||
|
||||
detachedManifest.modified.length > 0 ||
|
||||
detachedManifest.deleted.length > 0 ||
|
||||
detachedManifest.renamed.length > 0);
|
||||
if (src.last_commit === resolved.target && !detachedHasChanges) {
|
||||
unchangedSources++;
|
||||
continue;
|
||||
}
|
||||
|
||||
// Rung 5/6: the delta itself — SAME helper the executor diffs with.
|
||||
const delta = computeSyncDelta(localPath, src.last_commit, resolved.target, {
|
||||
detachedManifest,
|
||||
});
|
||||
if (delta.status === 'unavailable') {
|
||||
ceiling(localPath, strategy, 'git_unavailable');
|
||||
continue;
|
||||
}
|
||||
const syncOpts = { strategy };
|
||||
const changedPaths = unique([
|
||||
...delta.manifest.added.filter(p => isSyncable(p, syncOpts)),
|
||||
...delta.manifest.modified.filter(p => isSyncable(p, syncOpts)),
|
||||
...delta.manifest.renamed.filter(r => isSyncable(r.to, syncOpts)).map(r => r.to),
|
||||
]);
|
||||
tokens += estimateDeltaTokens(localPath, changedPaths);
|
||||
changedSources++;
|
||||
tokens += estimateSourceTreeTokens(src.local_path, cfg.strategy ?? 'markdown').tokens;
|
||||
hadDelta = true;
|
||||
}
|
||||
return { tokens, changedSources, unchangedSources };
|
||||
|
||||
const estimateKind: EstimateKind =
|
||||
hadCeiling && hadDelta ? 'mixed' : hadCeiling ? 'ceiling' : hadDelta ? 'delta' : 'unchanged';
|
||||
return { tokens, changedSources, unchangedSources, estimateKind, ceilingReasons };
|
||||
}
|
||||
|
||||
/**
|
||||
@@ -299,9 +464,10 @@ function estimateInlineNewTokens(
|
||||
async function resolveCostGateFloorUsd(engine: BrainEngine): Promise<number> {
|
||||
try {
|
||||
const raw = await engine.getConfig('sync.cost_gate_min_usd');
|
||||
if (raw === null || raw === undefined) return 0.5;
|
||||
const n = Number(raw);
|
||||
return Number.isFinite(n) && n >= 0 ? n : 0.5;
|
||||
// v0.42.42.0 (#2139): `0` keeps meaning "block on any nonzero spend"
|
||||
// (allowZero); `off`/`unlimited`/`none` → Infinity so the gate never
|
||||
// blocks (`costUsd > Infinity` is always false).
|
||||
return parseUsdLimit(raw, 0.5, { allowZero: true });
|
||||
} catch {
|
||||
return 0.5;
|
||||
}
|
||||
@@ -315,9 +481,9 @@ async function resolveCostGateFloorUsd(engine: BrainEngine): Promise<number> {
|
||||
async function resolveBackfillCapUsd(engine: BrainEngine): Promise<number> {
|
||||
try {
|
||||
const raw = await engine.getConfig(SPEND_CAP_CONFIG_KEY);
|
||||
if (raw === null || raw === undefined) return 25;
|
||||
const n = Number(raw);
|
||||
return Number.isFinite(n) && n > 0 ? n : 25;
|
||||
// v0.42.42.0 (#2139): no allowZero — `0` falls back to the default
|
||||
// (off semantics ≠ 0); `off`/`unlimited`/`none` → Infinity (cap disabled).
|
||||
return parseUsdLimit(raw, 25);
|
||||
} catch {
|
||||
return 25;
|
||||
}
|
||||
@@ -335,6 +501,214 @@ async function promptYesNo(question: string): Promise<boolean> {
|
||||
});
|
||||
}
|
||||
|
||||
// v0.42.42.0 (#2139): paste-ready knobs appended to every gate message so the
|
||||
// spend-control surface is discoverable at the moment of need (humans + agents),
|
||||
// not only after reading source. Closes the issue's "takes archaeology" complaint.
|
||||
const SPEND_HINT =
|
||||
'widen: gbrain config set sync.cost_gate_min_usd 5 | ' +
|
||||
'never gate: gbrain config set spend.posture tokenmax | ' +
|
||||
'docs: docs/operations/spend-controls.md';
|
||||
|
||||
/** Honest token label — delta vs full-tree ceiling, with the ceiling reasons. */
|
||||
function labelEstimate(inline: InlineEstimate): string {
|
||||
if (inline.estimateKind === 'unchanged') return '0 new tokens (sources caught up)';
|
||||
if (inline.estimateKind === 'delta') {
|
||||
return `~${inline.tokens.toLocaleString()} new tokens (delta: changed files since last sync)`;
|
||||
}
|
||||
// ceiling | mixed — be explicit that this is an over-count, not the real spend.
|
||||
const reasons = unique(inline.ceilingReasons).join(', ') || 'unknown';
|
||||
return (
|
||||
`<=${inline.tokens.toLocaleString()} tokens (full-tree ceiling for ${inline.changedSources} ` +
|
||||
`source(s): ${reasons} — unchanged files skip via content_hash at execution)`
|
||||
);
|
||||
}
|
||||
|
||||
type CostGateSource = {
|
||||
local_path: string | null;
|
||||
config: Record<string, unknown>;
|
||||
last_commit: string | null;
|
||||
chunker_version: string | null;
|
||||
};
|
||||
|
||||
interface CostGateContext {
|
||||
sources: CostGateSource[];
|
||||
/** Resolved embed mode. Single-source is always 'inline' (not the parallel-deferred fan-out). */
|
||||
mode: SyncEmbedMode;
|
||||
dryRun: boolean;
|
||||
jsonOut: boolean;
|
||||
yesFlag: boolean;
|
||||
full: boolean;
|
||||
/** Message prefix ('sync --all' | 'sync'). */
|
||||
label: string;
|
||||
}
|
||||
|
||||
type CostGateOutcome =
|
||||
| { action: 'proceed'; autoDeferEmbeds: boolean }
|
||||
| { action: 'stop' };
|
||||
|
||||
/**
|
||||
* v0.42.42.0 (#2139): the inline-embed cost gate, shared by BOTH `sync --all`
|
||||
* and single-source `sync` so the spend surface is consistent. Runs at the
|
||||
* COMMAND layer (never inside performSync, which `runOne` also calls — that
|
||||
* would double-gate `--all`).
|
||||
*
|
||||
* Behavior:
|
||||
* - deferred mode → FYI only, never blocks (backfill cap is the money gate).
|
||||
* - inline + below floor → proceed quietly.
|
||||
* - inline + spend.posture=tokenmax → informational, proceed inline.
|
||||
* - inline + above floor + TTY → [y/N] prompt.
|
||||
* - inline + above floor + non-TTY/--json → AUTO-DEFER embeds to capped
|
||||
* backfill jobs, exit 0 (NEVER exit 2 — the wedged-cron fix). Caller sets
|
||||
* effectiveNoEmbed and enqueues the backfill.
|
||||
*
|
||||
* Output format splits on the EXPLICIT `--json` flag only (absorbs the
|
||||
* TODOS.md:340 #1784 conflation): JSON envelope iff `--json`, else human text.
|
||||
*/
|
||||
async function runInlineCostGate(
|
||||
engine: BrainEngine,
|
||||
ctx: CostGateContext,
|
||||
): Promise<CostGateOutcome> {
|
||||
const { sources, mode, dryRun, jsonOut, yesFlag, full, label } = ctx;
|
||||
|
||||
// Stale backlog: cheap single SQL; fail-open to 0 so a transient DB hiccup
|
||||
// never blocks the sync. Signature-aware (model/dims swap surfaces here).
|
||||
let staleChars = 0;
|
||||
try {
|
||||
staleChars = await engine.sumStaleChunkChars({ signature: currentEmbeddingSignature() });
|
||||
} catch {
|
||||
staleChars = 0;
|
||||
}
|
||||
const staleCostUsd = estimateCostFromChars(staleChars, currentEmbeddingPricePerMTok());
|
||||
const embeddingModelName = getEmbeddingModelName();
|
||||
const floorUsd = await resolveCostGateFloorUsd(engine);
|
||||
const posture = await resolveSpendPosture(engine);
|
||||
|
||||
if (mode === 'deferred') {
|
||||
// Deferred path: print an FYI, NEVER block. The backfill cap is the real
|
||||
// money gate.
|
||||
const capUsd = await resolveBackfillCapUsd(engine);
|
||||
let queuedBackfills = 0;
|
||||
try {
|
||||
const r = await engine.executeRaw<{ n: number }>(
|
||||
`SELECT COUNT(*)::int AS n FROM minion_jobs
|
||||
WHERE name = 'embed-backfill'
|
||||
AND status IN ('waiting','active','delayed','waiting-children')`,
|
||||
);
|
||||
queuedBackfills = Number(r[0]?.n) || 0;
|
||||
} catch {
|
||||
queuedBackfills = 0;
|
||||
}
|
||||
const deferredMsg =
|
||||
`${label}: embedding deferred to backfill jobs ` +
|
||||
`(capped $${formatUsdLimit(capUsd)}/source/24h, not charged by this sync). ` +
|
||||
`Current backlog ~${staleChars.toLocaleString()} chars (~$${staleCostUsd.toFixed(2)} on ` +
|
||||
`${embeddingModelName}) across ${sources.length} source(s); ` +
|
||||
`${queuedBackfills} backfill job(s) queued.`;
|
||||
if (dryRun) {
|
||||
if (jsonOut) {
|
||||
console.log(JSON.stringify({ status: 'dry_run', mode, gate: 'dry_run', staleChars, staleCostUsd, capUsd: formatUsdLimit(capUsd), floorUsd: formatUsdLimit(floorUsd), queuedBackfills, model: embeddingModelName }));
|
||||
} else {
|
||||
console.log(deferredMsg);
|
||||
console.log('--dry-run: exit without syncing.');
|
||||
}
|
||||
return { action: 'stop' };
|
||||
}
|
||||
if (jsonOut) {
|
||||
console.log(JSON.stringify({ status: 'deferred', mode, gate: 'deferred_notice', staleChars, staleCostUsd, capUsd: formatUsdLimit(capUsd), floorUsd: formatUsdLimit(floorUsd), queuedBackfills, model: embeddingModelName }));
|
||||
} else {
|
||||
console.log(deferredMsg);
|
||||
}
|
||||
return { action: 'proceed', autoDeferEmbeds: false };
|
||||
}
|
||||
|
||||
// ── Inline path ───────────────────────────────────────────────
|
||||
const inline = estimateInlineNewTokens(sources, String(CHUNKER_VERSION));
|
||||
// D7A: `--full` runs `performFullSync` → `runEmbedCore({stale:true})`, which
|
||||
// sweeps the pre-existing stale backlog INLINE on top of the delta. Price it.
|
||||
const costUsd = estimateEmbeddingCostUsd(inline.tokens) + (full ? staleCostUsd : 0);
|
||||
const fullNote = full && staleChars > 0
|
||||
? ` (includes ~${staleChars.toLocaleString()} stale-backlog chars swept by --full)`
|
||||
: '';
|
||||
const staleNote = !full && staleChars > 0
|
||||
? ` (plus ~${staleChars.toLocaleString()} stale-backlog chars pending \`gbrain embed --stale\`)`
|
||||
: '';
|
||||
const previewMsg =
|
||||
`${label} preview (inline embed): ${inline.changedSources} changed source(s), ` +
|
||||
`${inline.unchangedSources} unchanged; ${labelEstimate(inline)}, ` +
|
||||
`est. $${costUsd.toFixed(2)} on ${embeddingModelName}${fullNote}${staleNote}.`;
|
||||
|
||||
if (dryRun) {
|
||||
if (jsonOut) {
|
||||
console.log(JSON.stringify({ status: 'dry_run', mode, gate: 'dry_run', newTokens: inline.tokens, estimateKind: inline.estimateKind, staleChars, costUsd, floorUsd: formatUsdLimit(floorUsd), model: embeddingModelName }));
|
||||
} else {
|
||||
console.log(previewMsg);
|
||||
console.log('--dry-run: exit without syncing.');
|
||||
}
|
||||
return { action: 'stop' };
|
||||
}
|
||||
|
||||
// --yes bypasses the gate entirely (embed inline, no preview).
|
||||
if (yesFlag) return { action: 'proceed', autoDeferEmbeds: false };
|
||||
|
||||
// spend.posture=tokenmax → informational, proceed INLINE (operator declared
|
||||
// cost isn't the constraint; don't defer).
|
||||
if (posture === 'tokenmax') {
|
||||
if (jsonOut) {
|
||||
console.log(JSON.stringify({ status: 'proceeding', mode, gate: 'posture_tokenmax', newTokens: inline.tokens, estimateKind: inline.estimateKind, costUsd, floorUsd: formatUsdLimit(floorUsd), model: embeddingModelName, hint: SPEND_HINT }));
|
||||
} else {
|
||||
console.log(`${previewMsg} spend.posture=tokenmax: proceeding (informational). ${SPEND_HINT}`);
|
||||
}
|
||||
return { action: 'proceed', autoDeferEmbeds: false };
|
||||
}
|
||||
|
||||
// Link intent: search.mode=tokenmax but spend posture unset → nudge once.
|
||||
let searchModeHint = '';
|
||||
try {
|
||||
const sm = await engine.getConfig('search.mode');
|
||||
if (typeof sm === 'string' && sm.trim().toLowerCase() === 'tokenmax') {
|
||||
searchModeHint =
|
||||
` (search.mode=tokenmax detected — \`gbrain config set spend.posture tokenmax\` ` +
|
||||
`makes cost gates informational)`;
|
||||
}
|
||||
} catch {
|
||||
/* best-effort */
|
||||
}
|
||||
|
||||
if (shouldBlockSync(costUsd, floorUsd, mode, posture)) {
|
||||
const isTTY = Boolean(process.stdout.isTTY) && Boolean(process.stdin.isTTY);
|
||||
if (isTTY && !jsonOut) {
|
||||
// Interactive TTY: prompt [y/N].
|
||||
console.log(previewMsg + searchModeHint);
|
||||
const answer = await promptYesNo('Proceed? [y/N] ');
|
||||
if (!answer) {
|
||||
console.log('Cancelled.');
|
||||
return { action: 'stop' };
|
||||
}
|
||||
return { action: 'proceed', autoDeferEmbeds: false };
|
||||
}
|
||||
// Non-TTY or --json: AUTO-DEFER embeds to capped backfill jobs. NEVER exit 2
|
||||
// (the wedged-cron fix). Format splits on the explicit --json flag only.
|
||||
if (jsonOut) {
|
||||
console.log(JSON.stringify({ status: 'auto_deferred', mode, gate: 'auto_deferred_embeds', newTokens: inline.tokens, estimateKind: inline.estimateKind, costUsd, floorUsd: formatUsdLimit(floorUsd), model: embeddingModelName, hint: SPEND_HINT }));
|
||||
} else {
|
||||
console.log(
|
||||
`${previewMsg} Exceeds floor $${formatUsdLimit(floorUsd)} in a non-interactive ` +
|
||||
`session — importing now, deferring embeds to capped backfill jobs. ` +
|
||||
`Drain: run the jobs worker or \`gbrain embed --stale\`. Pass --yes to embed inline.\n${SPEND_HINT}`,
|
||||
);
|
||||
}
|
||||
return { action: 'proceed', autoDeferEmbeds: true };
|
||||
}
|
||||
|
||||
// Below floor → proceed without blocking (kills inline-cron noise).
|
||||
if (jsonOut) {
|
||||
console.log(JSON.stringify({ status: 'below_floor', mode, gate: 'below_floor', newTokens: inline.tokens, estimateKind: inline.estimateKind, staleChars, costUsd, floorUsd: formatUsdLimit(floorUsd), model: embeddingModelName }));
|
||||
} else {
|
||||
console.log(`${previewMsg} Below cost gate floor ($${formatUsdLimit(floorUsd)}), proceeding.`);
|
||||
}
|
||||
return { action: 'proceed', autoDeferEmbeds: false };
|
||||
}
|
||||
|
||||
export interface SyncOpts {
|
||||
repoPath?: string;
|
||||
dryRun?: boolean;
|
||||
@@ -554,19 +928,9 @@ function unique<T>(items: T[]): T[] {
|
||||
return [...new Set(items)];
|
||||
}
|
||||
|
||||
function buildDetachedWorkingTreeManifest(repoPath: string): SyncManifest {
|
||||
const manifest = buildSyncManifest(git(repoPath, ['diff', '--name-status', '-M', 'HEAD']));
|
||||
const untracked = git(repoPath, ['ls-files', '--others', '--exclude-standard'])
|
||||
.split('\n')
|
||||
.filter(line => line.length > 0);
|
||||
|
||||
return {
|
||||
added: unique([...manifest.added, ...untracked]),
|
||||
modified: unique(manifest.modified),
|
||||
deleted: unique(manifest.deleted),
|
||||
renamed: manifest.renamed,
|
||||
};
|
||||
}
|
||||
// v0.42.42.0 (#2139): `buildDetachedWorkingTreeManifest` relocated to
|
||||
// `src/core/sync-delta.ts` (re-imported below) so the inline cost estimator
|
||||
// prices detached sources through the same code the executor imports them with.
|
||||
|
||||
// v0.18.0 Step 5: source-scoped sync state helpers. When opts.sourceId
|
||||
// is set, read/write the per-source row instead of the global config
|
||||
@@ -1426,29 +1790,28 @@ async function performSyncInner(engine: BrainEngine, opts: SyncOpts): Promise<Sy
|
||||
// both endpoints are stable across every resume, so the manifest is
|
||||
// deterministic and resumeFilter maps cleanly onto completed paths.
|
||||
//
|
||||
// v0.42.42.0 (#2139): the diff + detached-working-tree merge now route
|
||||
// through `computeSyncDelta` (src/core/sync-delta.ts) — the SAME helper the
|
||||
// inline cost estimator uses, so the gate's dollar figure can't drift from
|
||||
// what this sync actually imports. `detachedWorkingTreeManifest` (computed
|
||||
// above for the `up_to_date` gate) is passed through to avoid recomputing it.
|
||||
//
|
||||
// #1970 (F-B): a non-ancestor diff against a wildly divergent tree (e.g. a
|
||||
// force-push to unrelated history) can exceed git()'s 30s timeout / 100 MiB
|
||||
// buffer. On failure, fall back to the authoritative full reconcile instead
|
||||
// of throwing — a slow correct reconcile beats a hard error or a silent walk.
|
||||
let diffOutput: string;
|
||||
try {
|
||||
diffOutput = git(repoPath, ['diff', '--name-status', '-M', `${lastCommit}..${pin}`]);
|
||||
} catch (e) {
|
||||
const msg = e instanceof Error ? e.message : String(e);
|
||||
// buffer, and a gc'd anchor object can't be diffed at all. On either
|
||||
// `unavailable`, fall back to the authoritative full reconcile instead of
|
||||
// throwing — a slow correct reconcile beats a hard error or a silent walk.
|
||||
const delta = computeSyncDelta(repoPath, lastCommit, pin, {
|
||||
detachedManifest: detachedWorkingTreeManifest,
|
||||
});
|
||||
if (delta.status === 'unavailable') {
|
||||
serr(
|
||||
`[sync] git diff ${lastCommit.slice(0, 8)}..${pin.slice(0, 8)} failed ` +
|
||||
`(${msg.slice(0, 80)}) — likely an oversized post-rewrite diff; ` +
|
||||
`falling back to full reconcile.`,
|
||||
`[sync] delta ${lastCommit.slice(0, 8)}..${pin.slice(0, 8)} unavailable ` +
|
||||
`(${delta.reason}) — falling back to full reconcile.`,
|
||||
);
|
||||
return performFullSync(engine, repoPath, headCommit, opts);
|
||||
}
|
||||
const manifest = buildSyncManifest(diffOutput);
|
||||
if (detachedWorkingTreeManifest) {
|
||||
manifest.added = unique([...manifest.added, ...detachedWorkingTreeManifest.added]);
|
||||
manifest.modified = unique([...manifest.modified, ...detachedWorkingTreeManifest.modified]);
|
||||
manifest.deleted = unique([...manifest.deleted, ...detachedWorkingTreeManifest.deleted]);
|
||||
manifest.renamed = [...manifest.renamed, ...detachedWorkingTreeManifest.renamed];
|
||||
}
|
||||
const manifest = delta.manifest;
|
||||
|
||||
// Filter to syncable files (strategy-aware)
|
||||
const syncOpts = opts.strategy ? { strategy: opts.strategy } : undefined;
|
||||
@@ -3020,13 +3383,6 @@ See also:
|
||||
// never reached, and "Already up to date." leaves the log untouched. Both
|
||||
// doctor and printSyncResult instruct users to run --skip-failed in
|
||||
// exactly this case, so the flag has to handle stale entries up-front.
|
||||
if (skipFailed) {
|
||||
const stale = unacknowledgedSyncFailures();
|
||||
if (stale.length > 0) {
|
||||
const acked = acknowledgeSyncFailures();
|
||||
console.log(`Acknowledged ${acked.count} pre-existing failure(s).`);
|
||||
}
|
||||
}
|
||||
|
||||
// v0.18.0 Step 5: --source resolves to a sources(id) row. Falls back
|
||||
// to pre-v0.17 global config (sync.repo_path + sync.last_commit) when
|
||||
@@ -3052,6 +3408,23 @@ See also:
|
||||
if (nudge) process.stderr.write(nudge + '\n');
|
||||
}
|
||||
|
||||
// --skip-failed: acknowledge pre-existing unacked failures BEFORE the sync
|
||||
// runs, not only ones the current run produces. Without this, the common
|
||||
// recovery flow — fix the YAML, re-run sync, then run --skip-failed to clear
|
||||
// the log — fails to clear anything (no NEW failures → the inner ack path in
|
||||
// performSync is never reached, and "Already up to date." leaves the log).
|
||||
//
|
||||
// v0.42.42.0 (#2139, D13C): scoped PER SOURCE. `--all` clears every source's
|
||||
// open failures; single-source clears only its own (don't ack source B's
|
||||
// failures when syncing source A). Safe under parallel — the ledger
|
||||
// serializes writes via `withLedgerLock` and keys rows by `source_id`
|
||||
// (#1939), which is why the old D15 "no --skip-failed under parallel"
|
||||
// refusal is lifted below.
|
||||
if (skipFailed) {
|
||||
const acked = syncAll ? acknowledgeFailures() : acknowledgeFailures(sourceId);
|
||||
if (acked.count > 0) console.log(`Acknowledged ${acked.count} pre-existing failure(s).`);
|
||||
}
|
||||
|
||||
// v0.19.0 — `sync --all` iterates all registered sources with a
|
||||
// local_path. Sources are the canonical v0.18.0 abstraction: per-source
|
||||
// last_commit, last_sync_at, config.federated flags. Per-source
|
||||
@@ -3080,132 +3453,21 @@ See also:
|
||||
const { isFederatedV2Enabled } = await import('../core/feature-flags.ts');
|
||||
const v2Enabled = await isFederatedV2Enabled(engine);
|
||||
|
||||
// v0.41.31 cost gate (supersedes the v0.20.0 unconditional gate). Under
|
||||
// federated_v2 sync DEFERS embedding to per-source embed-backfill jobs
|
||||
// that carry their own $X/source/24h spend cap, so sync itself spends
|
||||
// nothing synchronously — the gate is INFORMATIONAL (never exit 2) on
|
||||
// that path. The blocking ConfirmationRequired gate fires ONLY when embed
|
||||
// runs INLINE (v2 off, or --serial without --no-embed) AND the estimated
|
||||
// spend exceeds `sync.cost_gate_min_usd` (default $0.50). Skipped entirely
|
||||
// when --no-embed is set (user opted out; will run `embed --stale` later).
|
||||
// v0.42.42.0 (#2139) cost gate — shared `runInlineCostGate`. Under
|
||||
// federated_v2 + parallel, embedding is DEFERRED to per-source backfill
|
||||
// jobs (own spend cap) so the gate is FYI-only. Inline mode (v2 off, or
|
||||
// --serial without --no-embed) gates on the DELTA estimate: below floor
|
||||
// proceeds; above floor in a non-TTY/--json session AUTO-DEFERS embeds
|
||||
// (exit 0, never exit 2 — the wedged-cron fix); a TTY prompts. Skipped
|
||||
// entirely when --no-embed is set.
|
||||
let autoDeferEmbeds = false;
|
||||
if (!noEmbed) {
|
||||
const mode = willEmbedSynchronously({ v2Enabled, serialFlag, noEmbed });
|
||||
// Stale backlog: cheap single SQL; fail-open to 0 so a transient DB
|
||||
// hiccup never blocks the sync.
|
||||
let staleChars = 0;
|
||||
try {
|
||||
// v0.41.31: signature-aware so a model/dims swap surfaces in the
|
||||
// backlog estimate (NULL signature grandfathered → not counted).
|
||||
staleChars = await engine.sumStaleChunkChars({ signature: currentEmbeddingSignature() });
|
||||
} catch {
|
||||
staleChars = 0;
|
||||
}
|
||||
const rate = currentEmbeddingPricePerMTok();
|
||||
const staleCostUsd = estimateCostFromChars(staleChars, rate);
|
||||
const embeddingModelName = getEmbeddingModelName();
|
||||
const floorUsd = await resolveCostGateFloorUsd(engine);
|
||||
|
||||
if (mode === 'deferred') {
|
||||
// Deferred path: print an FYI, NEVER exit 2. The backfill cap is the
|
||||
// real money gate (D1/D4).
|
||||
const capUsd = await resolveBackfillCapUsd(engine);
|
||||
// v0.41.31 (TODO-2): surface already-queued backfill jobs so a cron
|
||||
// operator sees work is enqueued, not lost. Best-effort — minion_jobs
|
||||
// may not exist on a brain that never ran a worker.
|
||||
let queuedBackfills = 0;
|
||||
try {
|
||||
const r = await engine.executeRaw<{ n: number }>(
|
||||
`SELECT COUNT(*)::int AS n FROM minion_jobs
|
||||
WHERE name = 'embed-backfill'
|
||||
AND status IN ('waiting','active','delayed','waiting-children')`,
|
||||
);
|
||||
queuedBackfills = Number(r[0]?.n) || 0;
|
||||
} catch {
|
||||
queuedBackfills = 0;
|
||||
}
|
||||
const deferredMsg =
|
||||
`sync --all: embedding deferred to backfill jobs ` +
|
||||
`(capped $${capUsd}/source/24h, not charged by this sync). ` +
|
||||
`Current backlog ~${staleChars.toLocaleString()} chars (~$${staleCostUsd.toFixed(2)} on ` +
|
||||
`${embeddingModelName}) across ${sources.length} source(s); ` +
|
||||
`${queuedBackfills} backfill job(s) queued.`;
|
||||
if (dryRun) {
|
||||
if (jsonOut) {
|
||||
console.log(JSON.stringify({ status: 'dry_run', mode, gate: 'dry_run', staleChars, staleCostUsd, capUsd, floorUsd, queuedBackfills, model: embeddingModelName }));
|
||||
} else {
|
||||
console.log(deferredMsg);
|
||||
console.log('--dry-run: exit without syncing.');
|
||||
}
|
||||
return;
|
||||
}
|
||||
if (jsonOut) {
|
||||
console.log(JSON.stringify({ status: 'deferred', mode, gate: 'deferred_notice', staleChars, staleCostUsd, capUsd, floorUsd, queuedBackfills, model: embeddingModelName }));
|
||||
} else {
|
||||
console.log(deferredMsg);
|
||||
}
|
||||
// fall through to sync — no exit 2.
|
||||
} else {
|
||||
// Inline path: sync embeds synchronously with no backfill cap to
|
||||
// protect it, so the blocking gate applies. The BLOCKING cost is the
|
||||
// new-content estimate ONLY (full-tree ceiling for changed sources;
|
||||
// unchanged contribute 0) — that's what this sync actually embeds.
|
||||
// The pre-existing stale backlog (NULL embeddings + signature drift)
|
||||
// is NOT swept by sync; `gbrain embed --stale` clears it. So we show
|
||||
// it informationally but never gate on cost this sync won't incur
|
||||
// (else a model swap would block the next inline cron — F2).
|
||||
const currentChunkerVersion = String(CHUNKER_VERSION);
|
||||
const inline = estimateInlineNewTokens(sources, currentChunkerVersion);
|
||||
const newCostUsd = estimateEmbeddingCostUsd(inline.tokens);
|
||||
const costUsd = newCostUsd;
|
||||
const staleNote = staleChars > 0
|
||||
? ` (plus ~${staleChars.toLocaleString()} stale-backlog chars pending \`gbrain embed --stale\`)`
|
||||
: '';
|
||||
const previewMsg =
|
||||
`sync --all preview (inline embed): ${inline.changedSources} changed source(s), ` +
|
||||
`${inline.unchangedSources} unchanged; ~${inline.tokens.toLocaleString()} new tokens, ` +
|
||||
`est. $${costUsd.toFixed(2)} on ${embeddingModelName}${staleNote}.`;
|
||||
|
||||
if (dryRun) {
|
||||
if (jsonOut) {
|
||||
console.log(JSON.stringify({ status: 'dry_run', mode, gate: 'dry_run', newTokens: inline.tokens, staleChars, costUsd, floorUsd, model: embeddingModelName }));
|
||||
} else {
|
||||
console.log(previewMsg);
|
||||
console.log('--dry-run: exit without syncing.');
|
||||
}
|
||||
return;
|
||||
}
|
||||
|
||||
if (!yesFlag) {
|
||||
if (shouldBlockSync(costUsd, floorUsd, mode)) {
|
||||
const isTTY = Boolean(process.stdout.isTTY) && Boolean(process.stdin.isTTY);
|
||||
if (!isTTY || jsonOut) {
|
||||
// Agent-facing path: emit structured envelope, exit 2.
|
||||
const envelope = serializeError(errorFor({
|
||||
class: 'ConfirmationRequired',
|
||||
code: 'cost_preview_requires_yes',
|
||||
message: previewMsg,
|
||||
hint: 'Pass --yes to proceed, or --dry-run to see the preview and exit 0.',
|
||||
}));
|
||||
console.log(JSON.stringify({ error: envelope, mode, gate: 'confirmation_required', newTokens: inline.tokens, staleChars, costUsd, floorUsd, model: embeddingModelName }));
|
||||
process.exit(2);
|
||||
}
|
||||
// Interactive TTY path: prompt [y/N].
|
||||
console.log(previewMsg);
|
||||
const answer = await promptYesNo('Proceed? [y/N] ');
|
||||
if (!answer) {
|
||||
console.log('Cancelled.');
|
||||
return;
|
||||
}
|
||||
} else {
|
||||
// Below floor → proceed without blocking (kills inline-cron noise).
|
||||
if (jsonOut) {
|
||||
console.log(JSON.stringify({ status: 'below_floor', mode, gate: 'below_floor', newTokens: inline.tokens, staleChars, costUsd, floorUsd, model: embeddingModelName }));
|
||||
} else {
|
||||
console.log(`${previewMsg} Below cost gate floor ($${floorUsd.toFixed(2)}), proceeding.`);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
const gate = await runInlineCostGate(engine, {
|
||||
sources, mode, dryRun, jsonOut, yesFlag, full, label: 'sync --all',
|
||||
});
|
||||
if (gate.action === 'stop') return;
|
||||
autoDeferEmbeds = gate.autoDeferEmbeds;
|
||||
}
|
||||
|
||||
// v0.40.5.0 Federated Sync v2 (master) + v0.40.6.0 layering (this branch):
|
||||
@@ -3266,7 +3528,12 @@ See also:
|
||||
const runOne = async (src: typeof sources[number]): Promise<SyncResult> => {
|
||||
const cfg = (src.config || {}) as { strategy?: 'markdown' | 'code' | 'auto' };
|
||||
// D18: parallel path defers embed; auto-enqueue embed-backfill after.
|
||||
const effectiveNoEmbed = v2Enabled && !serialFlag && !noEmbed ? true : noEmbed;
|
||||
// v0.42.42.0 (#2139): `autoDeferEmbeds` (the inline gate tripped in a
|
||||
// non-TTY session) ALSO forces deferral — global by design (the gate's
|
||||
// decision unit is the aggregate estimate; deferral strictly dominates
|
||||
// the exit-2 it replaced for every source).
|
||||
const effectiveNoEmbed =
|
||||
(v2Enabled && !serialFlag && !noEmbed ? true : noEmbed) || autoDeferEmbeds;
|
||||
// v0.41.13.0 (T6 / D-V3-3 / D-V4-mech-6) — per-source AbortController.
|
||||
//
|
||||
// When the user passes --timeout, each source gets its OWN
|
||||
@@ -3329,8 +3596,11 @@ See also:
|
||||
// v0.41.13.0 (T7 / D-V3-5): partial excluded — the next clean sync
|
||||
// re-walks the diff and re-decides whether to enqueue embed for
|
||||
// pages whose content actually changed.
|
||||
// v0.42.42.0 (#2139): `autoDeferEmbeds` enqueues even on the v2-OFF
|
||||
// legacy path — otherwise the gate's auto-defer would strand
|
||||
// NULL-embedded chunks with no queued job to embed them.
|
||||
if (
|
||||
v2Enabled &&
|
||||
(v2Enabled || autoDeferEmbeds) &&
|
||||
!noAutoEmbed &&
|
||||
!dryRun &&
|
||||
result.status !== 'dry_run' &&
|
||||
@@ -3357,18 +3627,13 @@ See also:
|
||||
const parallelEligible =
|
||||
v2Enabled && !serialFlag && engine.kind !== 'pglite' && activeSources.length > 1;
|
||||
|
||||
// v0.40.6.0 (D15): refuse --skip-failed / --retry-failed when running
|
||||
// parallel. sync-failures.jsonl is brain-global; parallel acks race.
|
||||
if (parallelEligible && (skipFailed || retryFailed)) {
|
||||
const flag = skipFailed ? '--skip-failed' : '--retry-failed';
|
||||
console.error(
|
||||
`Error: ${flag} is not supported under parallel sync.\n` +
|
||||
` (the sync-failures log is brain-global and parallel acks race).\n` +
|
||||
` Re-run with --serial for the recovery flow:\n` +
|
||||
` gbrain sync --all --serial ${flag}`,
|
||||
);
|
||||
process.exit(1);
|
||||
}
|
||||
// v0.42.42.0 (#2139, D13C): the v0.40.6.0 (D15) refusal of --skip-failed /
|
||||
// --retry-failed under parallel sync is LIFTED. It existed because the
|
||||
// failure ledger was once brain-global with racing acks; #1939 made the
|
||||
// ledger per-(source_id, path) and serialized every write through
|
||||
// `withLedgerLock`, so parallel per-source acks no longer race. Lifting it
|
||||
// also removes the forcing-function that pushed recovery syncs to --serial
|
||||
// (and thus armed the inline cost gate) — the root cause behind #2139.
|
||||
|
||||
// Effective parallelism — surfaced in the --json envelope so consumers
|
||||
// know how the run was actually dispatched. 1 in the serial fallback,
|
||||
@@ -3516,12 +3781,47 @@ See also:
|
||||
signal: composeAbortSignals(singleSourceInterrupt.signal, singleSourceController?.signal),
|
||||
};
|
||||
|
||||
// v0.42.42.0 (#2139, Step 4b): single-source `gbrain sync` gets the SAME
|
||||
// inline cost gate as `--all`. Previously single-source embedded inline with
|
||||
// NO gate (only rail: the ≤100-file inline cap). Single-source always embeds
|
||||
// INLINE (not the parallel-deferred fan-out), so mode is forced 'inline'.
|
||||
// Skipped on --no-embed, --dry-run (performSync's own dry-run previews and
|
||||
// spends nothing), and watch. Non-TTY above floor AUTO-DEFERS — so adding
|
||||
// the gate can never wedge an existing cron; it converts silent ungated
|
||||
// inline spend into informed inline-or-deferred spend.
|
||||
let singleSourceAutoDefer = false;
|
||||
if (!noEmbed && !dryRun && !watch) {
|
||||
const gateRows = await engine.executeRaw<{ local_path: string | null; config: Record<string, unknown>; last_commit: string | null; chunker_version: string | null }>(
|
||||
`SELECT local_path, config, last_commit, chunker_version FROM sources WHERE id = $1`,
|
||||
[sourceId],
|
||||
);
|
||||
if (gateRows.length > 0) {
|
||||
const gateSources = [{
|
||||
local_path: gateRows[0].local_path ?? repoPath ?? null,
|
||||
config: gateRows[0].config ?? {},
|
||||
last_commit: gateRows[0].last_commit,
|
||||
chunker_version: gateRows[0].chunker_version,
|
||||
}];
|
||||
const gate = await runInlineCostGate(engine, {
|
||||
sources: gateSources, mode: 'inline', dryRun: false, jsonOut, yesFlag, full, label: 'sync',
|
||||
});
|
||||
if (gate.action === 'stop') return;
|
||||
if (gate.autoDeferEmbeds) {
|
||||
opts.noEmbed = true;
|
||||
singleSourceAutoDefer = true;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Bug 9 — --retry-failed: before running normal sync, clear acknowledgment
|
||||
// flags so the sync picks them up as fresh work. The actual re-attempt
|
||||
// happens inside the regular incremental/full loop because once the commit
|
||||
// pointer is behind the failures, the diff naturally revisits them.
|
||||
if (retryFailed) {
|
||||
const failures = unacknowledgedSyncFailures();
|
||||
// v0.42.42.0 (#2139, D13C): scope the retry count to THIS source — rows
|
||||
// carry source_id (#1939), so a single-source retry shouldn't report
|
||||
// another source's failures.
|
||||
const failures = unacknowledgedSyncFailures().filter(f => f.source_id === sourceId);
|
||||
if (failures.length === 0) {
|
||||
console.log('No unacknowledged sync failures to retry.');
|
||||
} else {
|
||||
@@ -3564,6 +3864,27 @@ See also:
|
||||
manageGitignore(effectiveRepoPath, engine.kind);
|
||||
}
|
||||
}
|
||||
// v0.42.42.0 (#2139, Step 4b): the inline gate auto-deferred this run's
|
||||
// embeds (non-TTY, above floor) — enqueue a capped backfill job so the
|
||||
// NULL-embedded chunks get embedded out of band instead of being stranded.
|
||||
if (
|
||||
singleSourceAutoDefer &&
|
||||
result.status !== 'dry_run' &&
|
||||
result.status !== 'up_to_date' &&
|
||||
result.status !== 'partial'
|
||||
) {
|
||||
try {
|
||||
const { submitEmbedBackfill } = await import('../core/embed-backfill-submit.ts');
|
||||
const sub = await submitEmbedBackfill(engine, sourceId, { reason: 'sync_autodefer' });
|
||||
if (sub.status === 'submitted') {
|
||||
process.stderr.write(` → embed-backfill job ${sub.jobId} queued (deferred inline embed).\n`);
|
||||
} else if (sub.status === 'cooldown') {
|
||||
process.stderr.write(` → embed-backfill skipped (cooldown); run \`gbrain embed --stale\` to drain now.\n`);
|
||||
}
|
||||
} catch (e) {
|
||||
process.stderr.write(` → embed-backfill submission failed: ${e instanceof Error ? e.message : String(e)}\n`);
|
||||
}
|
||||
}
|
||||
return;
|
||||
}
|
||||
|
||||
|
||||
@@ -904,6 +904,15 @@ export const KNOWN_CONFIG_KEYS: readonly string[] = [
|
||||
// Link resolution (issue #972)
|
||||
'link_resolution',
|
||||
'link_resolution.global_basename',
|
||||
// Spend controls (v0.42.42.0, issue #2139). Previously `--force`-only — the
|
||||
// operator had to discover these by reading source. Registered so `config
|
||||
// set` accepts them directly. See docs/operations/spend-controls.md.
|
||||
'spend.posture',
|
||||
'sync.cost_gate_min_usd',
|
||||
'sync.federated_v2',
|
||||
'embed.backfill_cooldown_min',
|
||||
'embed.backfill_max_usd_per_source_24h',
|
||||
'embed.backfill_max_usd',
|
||||
];
|
||||
|
||||
/**
|
||||
|
||||
@@ -35,6 +35,7 @@
|
||||
*/
|
||||
import type { BrainEngine } from './engine.ts';
|
||||
import { MinionQueue } from './minions/queue.ts';
|
||||
import { parseUsdLimit, resolveSpendPosture, type SpendPosture } from './spend-posture.ts';
|
||||
|
||||
export const COOLDOWN_CONFIG_KEY = 'embed.backfill_cooldown_min';
|
||||
export const SPEND_CAP_CONFIG_KEY = 'embed.backfill_max_usd_per_source_24h';
|
||||
@@ -57,6 +58,13 @@ export interface SubmitEmbedBackfillResult {
|
||||
spend24hUsd?: number;
|
||||
/** Set when status === 'spend_capped'. Active cap. */
|
||||
spendCapUsd?: number;
|
||||
/**
|
||||
* Set true when `spend.posture=tokenmax` waved the job past the 24h spend
|
||||
* cap (#2139). The spend is still LEDGERED by the per-job BudgetTracker —
|
||||
* posture removes the ceiling, not the accounting. Cooldown is NOT bypassed
|
||||
* (it's queue-churn protection, not a spend gate).
|
||||
*/
|
||||
spendCapBypassed?: boolean;
|
||||
}
|
||||
|
||||
export interface SubmitEmbedBackfillOpts {
|
||||
@@ -72,6 +80,8 @@ export interface SubmitEmbedBackfillOpts {
|
||||
nowMs?: number;
|
||||
/** Job priority. Default 5 (lower than autopilot's 0; above default jobs). */
|
||||
priority?: number;
|
||||
/** Override the resolved spend posture (tests). Default: read from config. */
|
||||
postureOverride?: SpendPosture;
|
||||
}
|
||||
|
||||
/**
|
||||
@@ -133,9 +143,12 @@ export async function submitEmbedBackfill(
|
||||
const cooldownMin =
|
||||
opts.cooldownMinOverride ??
|
||||
(await readIntConfig(engine, COOLDOWN_CONFIG_KEY, DEFAULT_COOLDOWN_MIN));
|
||||
// v0.42.42.0 (#2139): spend cap honors `off`/`unlimited`/`none` → Infinity.
|
||||
// `0` still falls back to the default (off semantics ≠ 0).
|
||||
const spendCap =
|
||||
opts.spendCapUsdOverride ??
|
||||
(await readIntConfig(engine, SPEND_CAP_CONFIG_KEY, DEFAULT_SPEND_CAP_USD));
|
||||
(raw => parseUsdLimit(raw, DEFAULT_SPEND_CAP_USD))(await engine.getConfig(SPEND_CAP_CONFIG_KEY));
|
||||
const posture = opts.postureOverride ?? (await resolveSpendPosture(engine));
|
||||
|
||||
// ── Source-level cooldown ─────────────────────────────────────
|
||||
// Block re-submission if (a) an embed-backfill is currently active for this
|
||||
@@ -172,9 +185,14 @@ export async function submitEmbedBackfill(
|
||||
}
|
||||
|
||||
// ── 24h rolling spend cap ─────────────────────────────────────
|
||||
// v0.42.42.0 (#2139): `spend.posture=tokenmax` waves past the cap (the
|
||||
// operator declared cost isn't the constraint). The per-job BudgetTracker
|
||||
// still ledgers the spend — posture removes the ceiling, not the accounting.
|
||||
// An `off`/`unlimited` cap (Infinity) is likewise never tripped.
|
||||
const spend24hFn = opts.spend24hFn ?? defaultSpend24hForSource;
|
||||
const spend24h = await spend24hFn(engine, sourceId);
|
||||
if (spend24h >= spendCap) {
|
||||
const spendCapBypassed = posture === 'tokenmax' && spend24h >= spendCap;
|
||||
if (spend24h >= spendCap && !spendCapBypassed) {
|
||||
return {
|
||||
status: 'spend_capped',
|
||||
spend24hUsd: spend24h,
|
||||
@@ -194,7 +212,9 @@ export async function submitEmbedBackfill(
|
||||
},
|
||||
);
|
||||
|
||||
return { status: 'submitted', jobId: job.id };
|
||||
return spendCapBypassed
|
||||
? { status: 'submitted', jobId: job.id, spendCapBypassed: true, spend24hUsd: spend24h }
|
||||
: { status: 'submitted', jobId: job.id };
|
||||
}
|
||||
|
||||
/** Round timestamp down to the nearest `bucketMs` boundary. */
|
||||
|
||||
+12
-5
@@ -218,16 +218,23 @@ export function willEmbedSynchronously(opts: {
|
||||
}
|
||||
|
||||
/**
|
||||
* Pure cost-gate decision. The gate BLOCKS (prompt in TTY, exit 2 envelope
|
||||
* in non-TTY) only when embed runs inline AND the estimated spend exceeds
|
||||
* the floor. Deferred mode NEVER blocks — the backfill cap is the real
|
||||
* money gate, and blocking the cheap markdown import for cost the import
|
||||
* doesn't synchronously incur is the bug this fix removes.
|
||||
* Pure cost-gate decision. The gate BLOCKS (prompt in TTY, auto-defer in
|
||||
* non-TTY) only when embed runs inline AND the estimated spend exceeds the
|
||||
* floor. Deferred mode NEVER blocks — the backfill cap is the real money gate,
|
||||
* and blocking the cheap markdown import for cost the import doesn't
|
||||
* synchronously incur is the bug this fix removes.
|
||||
*
|
||||
* v0.42.42.0 (#2139): `spend.posture=tokenmax` makes the gate INFORMATIONAL —
|
||||
* the operator has declared cost isn't the constraint, so it never blocks
|
||||
* (the caller prints the estimate and proceeds inline). An `off`/`unlimited`
|
||||
* floor (Infinity) is likewise never exceeded.
|
||||
*/
|
||||
export function shouldBlockSync(
|
||||
costUsd: number,
|
||||
floorUsd: number,
|
||||
mode: SyncEmbedMode,
|
||||
posture: 'gated' | 'tokenmax' = 'gated',
|
||||
): boolean {
|
||||
if (posture === 'tokenmax') return false;
|
||||
return mode === 'inline' && costUsd > floorUsd;
|
||||
}
|
||||
|
||||
+26
-1
@@ -131,7 +131,7 @@ export interface CloneOpts {
|
||||
|
||||
export class GitOperationError extends Error {
|
||||
constructor(
|
||||
public op: 'clone' | 'pull' | 'remote_get_url',
|
||||
public op: 'clone' | 'pull' | 'fetch' | 'remote_get_url',
|
||||
message: string,
|
||||
public cause?: unknown,
|
||||
) {
|
||||
@@ -217,6 +217,31 @@ export function pullRepo(repoPath: string, opts: { timeoutMs?: number } = {}): v
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Fetch a single remote branch with the SAME SSRF-defensive flags + no-prompt
|
||||
* env as cloneRepo/pullRepo (GIT_SSRF_FLAGS, --no-recurse-submodules,
|
||||
* GIT_TERMINAL_PROMPT=0). Used by the sync cost-estimator's fetch-first path
|
||||
* (#2139) so a cost preview / dry-run never hits a remote through a
|
||||
* less-protected route than real sync. Throws GitOperationError on failure;
|
||||
* the estimator catches and falls back to local HEAD.
|
||||
*/
|
||||
export function fetchRemote(repoPath: string, branch: string, opts: { timeoutMs?: number } = {}): void {
|
||||
const args: string[] = ['-C', repoPath, ...GIT_SSRF_FLAGS, 'fetch', ...GIT_SSRF_SUBCOMMAND_FLAGS, 'origin', branch];
|
||||
try {
|
||||
execFileSync('git', args, {
|
||||
stdio: ['ignore', 'pipe', 'pipe'],
|
||||
timeout: opts.timeoutMs ?? 30_000,
|
||||
env: { ...process.env, ...GIT_ENV },
|
||||
});
|
||||
} catch (e) {
|
||||
throw new GitOperationError(
|
||||
'fetch',
|
||||
`git fetch failed in ${repoPath}: ${(e as Error).message}`,
|
||||
e,
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
export type RepoState =
|
||||
| 'healthy'
|
||||
| 'missing'
|
||||
|
||||
@@ -38,6 +38,7 @@ import { embedStaleForSource } from '../../embed-stale.ts';
|
||||
import { currentEmbeddingSignature } from '../../embedding.ts';
|
||||
import type { BrainEngine } from '../../engine.ts';
|
||||
import type { MinionJobContext } from '../types.ts';
|
||||
import { parseUsdLimit, usdLimitToCap, resolveSpendPosture } from '../../spend-posture.ts';
|
||||
|
||||
const DEFAULT_MAX_USD_PER_JOB = 10;
|
||||
const EMBED_BACKFILL_LOCK_TTL_MIN = 60;
|
||||
@@ -66,12 +67,21 @@ function embedBackfillLockId(sourceId: string): string {
|
||||
return `gbrain-embed-backfill:${sourceId}`;
|
||||
}
|
||||
|
||||
/** Read embed.backfill_max_usd config or default. */
|
||||
async function readMaxUsd(engine: BrainEngine): Promise<number> {
|
||||
/**
|
||||
* Resolve the per-job budget cap (USD) for the BudgetTracker.
|
||||
*
|
||||
* v0.42.42.0 (#2139): returns `undefined` = "no cap" (which BudgetTracker
|
||||
* treats as cap-absent) when the config is `off`/`unlimited`/`none` OR when
|
||||
* `spend.posture=tokenmax`. NEVER returns Infinity (that would pass through as
|
||||
* a real ceiling and serialize to `null` in audit rows). Spend is still
|
||||
* ledgered by the tracker either way — posture removes the ceiling, not the
|
||||
* accounting. `0`/garbage fall back to the $10 default.
|
||||
*/
|
||||
async function readMaxUsd(engine: BrainEngine): Promise<number | undefined> {
|
||||
const posture = await resolveSpendPosture(engine);
|
||||
if (posture === 'tokenmax') return undefined;
|
||||
const raw = await engine.getConfig('embed.backfill_max_usd');
|
||||
if (raw === null || raw === undefined) return DEFAULT_MAX_USD_PER_JOB;
|
||||
const n = Number(raw);
|
||||
return Number.isFinite(n) && n > 0 ? n : DEFAULT_MAX_USD_PER_JOB;
|
||||
return usdLimitToCap(parseUsdLimit(raw, DEFAULT_MAX_USD_PER_JOB));
|
||||
}
|
||||
|
||||
/** Validate + extract typed job params. Throws on malformed input. */
|
||||
|
||||
@@ -0,0 +1,106 @@
|
||||
/**
|
||||
* Spend posture + USD-limit parsing — the single spend-control surface for
|
||||
* gbrain's cost gates (issue #2139). Two concerns live here:
|
||||
*
|
||||
* 1. `spend.posture` (DB-plane config): `'gated'` (default) makes every cost
|
||||
* gate behave as before; `'tokenmax'` makes them INFORMATIONAL — print the
|
||||
* estimate, proceed, and keep ledgering spend. The operator who sets
|
||||
* `tokenmax` has declared "cost is not my constraint." Posture removes the
|
||||
* CEILING, never the ACCOUNTING (the spend ledger still records every
|
||||
* dollar). It is deliberately SEPARATE from `search.mode=tokenmax` (which
|
||||
* governs retrieval payload size, not embedding spend); the gate prints a
|
||||
* hint linking the two when only the search mode is set.
|
||||
*
|
||||
* 2. `parseUsdLimit` / `formatUsdLimit`: first-class `off` / `unlimited` /
|
||||
* `none` on the USD gate knobs (so operators stop setting sentinel values
|
||||
* like `100000`). The parse layer represents "no limit" as `Infinity` so
|
||||
* comparisons stay special-case-free (`cost > Infinity` is never true).
|
||||
* `formatUsdLimit` renders it as the string `'unlimited'` — NEVER serialize
|
||||
* raw `Infinity`, because `JSON.stringify(Infinity)` emits `null`, which is
|
||||
* ambiguous in audit/ledger rows. At the budget-machinery boundary, callers
|
||||
* convert `Infinity` → `undefined` ("no cap"), which `BudgetTracker` already
|
||||
* treats as cap-absent.
|
||||
*
|
||||
* LEAF module: imports only the engine type so any cost-gate site can pull it
|
||||
* in without a circular dependency.
|
||||
*/
|
||||
import type { BrainEngine } from './engine.ts';
|
||||
|
||||
export const SPEND_POSTURE_CONFIG_KEY = 'spend.posture';
|
||||
|
||||
export type SpendPosture = 'gated' | 'tokenmax';
|
||||
|
||||
/**
|
||||
* Resolve `spend.posture` from DB-plane config. Fail-open to `'gated'` on a
|
||||
* missing/unknown value or a config-read error — a posture probe must never
|
||||
* crash a sync, and an unrecognized value must never silently disable gates.
|
||||
*/
|
||||
export async function resolveSpendPosture(engine: BrainEngine): Promise<SpendPosture> {
|
||||
try {
|
||||
const raw = await engine.getConfig(SPEND_POSTURE_CONFIG_KEY);
|
||||
return normalizeSpendPosture(raw);
|
||||
} catch {
|
||||
return 'gated';
|
||||
}
|
||||
}
|
||||
|
||||
/** Pure normalizer (testable without an engine). Anything but `tokenmax` → `gated`. */
|
||||
export function normalizeSpendPosture(raw: unknown): SpendPosture {
|
||||
if (typeof raw === 'string' && raw.trim().toLowerCase() === 'tokenmax') return 'tokenmax';
|
||||
return 'gated';
|
||||
}
|
||||
|
||||
/** True iff `raw` is a valid `spend.posture` value (for `config set` validation). */
|
||||
export function isValidSpendPosture(raw: unknown): boolean {
|
||||
return typeof raw === 'string' && ['gated', 'tokenmax'].includes(raw.trim().toLowerCase());
|
||||
}
|
||||
|
||||
const OFF_TOKENS = new Set(['off', 'unlimited', 'none']);
|
||||
|
||||
/**
|
||||
* Parse a USD-limit config value.
|
||||
* - `'off'` / `'unlimited'` / `'none'` (case-insensitive) → `Infinity` (no limit)
|
||||
* - a finite positive number (or `0` when `allowZero`) → that number
|
||||
* - anything else (garbage, negative, empty, NaN) → `def`
|
||||
*
|
||||
* `allowZero` distinguishes the two knob semantics:
|
||||
* - `sync.cost_gate_min_usd` uses `allowZero: true` — `0` means "block on any
|
||||
* nonzero spend" (a real operator choice). `off` is the no-limit escape.
|
||||
* - the backfill caps reject `0` (fall back to the default); only `off`
|
||||
* disables them. `off` semantics ≠ `0` (issue #2139).
|
||||
*/
|
||||
export function parseUsdLimit(
|
||||
raw: unknown,
|
||||
def: number,
|
||||
opts: { allowZero?: boolean } = {},
|
||||
): number {
|
||||
if (raw === null || raw === undefined) return def;
|
||||
if (typeof raw === 'string') {
|
||||
const t = raw.trim().toLowerCase();
|
||||
if (t === '') return def;
|
||||
if (OFF_TOKENS.has(t)) return Infinity;
|
||||
}
|
||||
const n = Number(raw);
|
||||
if (!Number.isFinite(n)) return def;
|
||||
if (n < 0) return def;
|
||||
if (n === 0) return opts.allowZero ? 0 : def;
|
||||
return n;
|
||||
}
|
||||
|
||||
/**
|
||||
* Render a USD limit for human/JSON output. `Infinity` → `'unlimited'` (NEVER
|
||||
* the raw value — `JSON.stringify(Infinity)` is `null`). Finite values are
|
||||
* returned as-is so callers can `$${formatUsdLimit(x)}` or embed in JSON.
|
||||
*/
|
||||
export function formatUsdLimit(n: number): string | number {
|
||||
return Number.isFinite(n) ? n : 'unlimited';
|
||||
}
|
||||
|
||||
/**
|
||||
* Convert a parsed USD limit to the budget-machinery cap representation:
|
||||
* `Infinity` → `undefined` ("no cap", which `BudgetTracker` treats as
|
||||
* cap-absent), finite → the number. Keeps `null` out of ledger rows.
|
||||
*/
|
||||
export function usdLimitToCap(n: number): number | undefined {
|
||||
return Number.isFinite(n) ? n : undefined;
|
||||
}
|
||||
@@ -0,0 +1,151 @@
|
||||
/**
|
||||
* Shared sync-delta machinery — ONE implementation of "what changed since
|
||||
* last_commit" consumed by BOTH the sync executor (`performSyncInner` in
|
||||
* `src/commands/sync.ts`) and the inline-embed cost estimator
|
||||
* (`src/core/sync-cost-estimate.ts`). Before this module the executor diffed
|
||||
* `last_commit..pin` while the estimator priced the entire tree, so the
|
||||
* gate's dollar figure had no relationship to what the sync actually embedded
|
||||
* (issue #2139: a 400x overestimate that wedged the daily cron). Routing both
|
||||
* through `computeSyncDelta` makes diff/manifest drift between estimate and
|
||||
* execution structurally impossible.
|
||||
*
|
||||
* Shell-injection safe: `execFileSync` with array args (no `/bin/sh -c`), so a
|
||||
* `sources.local_path` containing shell metacharacters can never escape — same
|
||||
* posture documented at `git-head.ts:14-19`.
|
||||
*
|
||||
* Fail-open ladder (never throws):
|
||||
*
|
||||
* computeSyncDelta(repo, from, to)
|
||||
* │
|
||||
* ├─ `git cat-file -t <from>` throws → { unavailable, anchor_missing }
|
||||
* │ (bookmark object gc'd after a history rewrite — nothing to diff;
|
||||
* │ caller falls back to a full reconcile / full-tree ceiling)
|
||||
* │
|
||||
* ├─ `git diff --name-status -M from..to` throws → { unavailable, diff_failed }
|
||||
* │ (oversized post-rewrite diff exceeds the 30s / 100 MiB budget)
|
||||
* │
|
||||
* └─ ok → { ok, manifest } (+ detached working-tree manifest merged in)
|
||||
*
|
||||
* NOTE: a present-but-non-ancestor `from` (force-push, squash, master→main) is
|
||||
* still diffable — `git diff A..B` is an endpoint-tree comparison and does NOT
|
||||
* require A to be an ancestor of B (unlike a rev-walk or `A...B` merge-base).
|
||||
* That is the #1970 property this module preserves.
|
||||
*/
|
||||
import { execFileSync } from 'node:child_process';
|
||||
import { buildSyncManifest, type SyncManifest } from './sync.ts';
|
||||
|
||||
/** Runs a git subcommand in `repoPath` and returns trimmed stdout (throws on failure). */
|
||||
export type GitRunner = (repoPath: string, args: string[]) => string;
|
||||
|
||||
// Mirrors `git()` + `buildGitInvocation()` in commands/sync.ts: `core.quotepath=false`
|
||||
// so non-ASCII (CJK) paths arrive as UTF-8; 30s timeout; 100 MiB maxBuffer (a
|
||||
// 100K-file `--name-status` diff tops out ~10-20 MiB — Node's 1 MiB default
|
||||
// would ENOBUFS-crash the sync with no log line).
|
||||
const DEFAULT_GIT_RUNNER: GitRunner = (repoPath, args) =>
|
||||
execFileSync('git', ['-c', 'core.quotepath=false', '-C', repoPath, ...args], {
|
||||
encoding: 'utf-8',
|
||||
timeout: 30_000,
|
||||
maxBuffer: 100 * 1024 * 1024,
|
||||
}).trim();
|
||||
|
||||
let _gitRunner: GitRunner = DEFAULT_GIT_RUNNER;
|
||||
|
||||
/**
|
||||
* Test seam (probe-seam pattern, matches `git-head.ts:_setGitHeadProbeForTests`)
|
||||
* so tests drive `computeSyncDelta` without mocking child_process or routing
|
||||
* through `mock.module` (R2-compliant). Pass `null` to restore the default.
|
||||
*/
|
||||
export function _setGitRunnerForTests(fn: GitRunner | null): void {
|
||||
_gitRunner = fn ?? DEFAULT_GIT_RUNNER;
|
||||
}
|
||||
|
||||
function unique<T>(items: T[]): T[] {
|
||||
return [...new Set(items)];
|
||||
}
|
||||
|
||||
/**
|
||||
* Working-tree manifest for a DETACHED HEAD (relocated from sync.ts:557 so the
|
||||
* estimator can price detached sources identically to how the executor imports
|
||||
* them). On a detached HEAD, sync syncs from the live working tree: tracked
|
||||
* changes (`git diff --name-status -M HEAD`) PLUS untracked files (`ls-files
|
||||
* --others --exclude-standard`). Attached HEADs never call this — their
|
||||
* incremental path imports ONLY the commit diff (untracked/dirty files are not
|
||||
* imported), which is why the estimator must not price dirty files on an
|
||||
* attached repo (issue #2139 phantom-cost class).
|
||||
*/
|
||||
export function buildDetachedWorkingTreeManifest(
|
||||
repoPath: string,
|
||||
run: GitRunner = _gitRunner,
|
||||
): SyncManifest {
|
||||
const manifest = buildSyncManifest(run(repoPath, ['diff', '--name-status', '-M', 'HEAD']));
|
||||
const untracked = run(repoPath, ['ls-files', '--others', '--exclude-standard'])
|
||||
.split('\n')
|
||||
.filter(line => line.length > 0);
|
||||
return {
|
||||
added: unique([...manifest.added, ...untracked]),
|
||||
modified: unique(manifest.modified),
|
||||
deleted: unique(manifest.deleted),
|
||||
renamed: manifest.renamed,
|
||||
};
|
||||
}
|
||||
|
||||
export type SyncDeltaResult =
|
||||
| { status: 'ok'; manifest: SyncManifest }
|
||||
| { status: 'unavailable'; reason: 'anchor_missing' | 'diff_failed' };
|
||||
|
||||
export interface ComputeSyncDeltaOpts {
|
||||
/**
|
||||
* Pre-computed detached working-tree manifest to merge into the commit diff
|
||||
* (the executor already builds one for its `up_to_date` gate; pass it to
|
||||
* avoid a redundant `git diff HEAD` + `ls-files`). When omitted and
|
||||
* `detached` is true, this module builds it.
|
||||
*/
|
||||
detachedManifest?: SyncManifest | null;
|
||||
/** Build the detached manifest internally (estimator path). Ignored if `detachedManifest` is provided. */
|
||||
detached?: boolean;
|
||||
}
|
||||
|
||||
/**
|
||||
* The single source of truth for "what changed between two commits in this
|
||||
* repo." Returns the RAW merged manifest (added/modified/deleted/renamed) —
|
||||
* callers apply their own `isSyncable` filtering + side effects.
|
||||
*/
|
||||
export function computeSyncDelta(
|
||||
repoPath: string,
|
||||
fromCommit: string,
|
||||
toCommit: string,
|
||||
opts: ComputeSyncDeltaOpts = {},
|
||||
): SyncDeltaResult {
|
||||
const run = _gitRunner;
|
||||
|
||||
// Reachability: a gc'd bookmark object can't be diffed (#1970).
|
||||
try {
|
||||
run(repoPath, ['cat-file', '-t', fromCommit]);
|
||||
} catch {
|
||||
return { status: 'unavailable', reason: 'anchor_missing' };
|
||||
}
|
||||
|
||||
let diffOutput: string;
|
||||
try {
|
||||
diffOutput = run(repoPath, ['diff', '--name-status', '-M', `${fromCommit}..${toCommit}`]);
|
||||
} catch {
|
||||
return { status: 'unavailable', reason: 'diff_failed' };
|
||||
}
|
||||
|
||||
const manifest = buildSyncManifest(diffOutput);
|
||||
|
||||
const detached =
|
||||
opts.detachedManifest !== undefined && opts.detachedManifest !== null
|
||||
? opts.detachedManifest
|
||||
: opts.detached
|
||||
? buildDetachedWorkingTreeManifest(repoPath, run)
|
||||
: null;
|
||||
if (detached) {
|
||||
manifest.added = unique([...manifest.added, ...detached.added]);
|
||||
manifest.modified = unique([...manifest.modified, ...detached.modified]);
|
||||
manifest.deleted = unique([...manifest.deleted, ...detached.deleted]);
|
||||
manifest.renamed = [...manifest.renamed, ...detached.renamed];
|
||||
}
|
||||
|
||||
return { status: 'ok', manifest };
|
||||
}
|
||||
@@ -29,6 +29,15 @@ describe('KNOWN_CONFIG_KEYS', () => {
|
||||
expect(KNOWN_CONFIG_KEYS).toContain('models.tier.subagent');
|
||||
});
|
||||
|
||||
test('contains the spend-control keys (v0.42.42.0, #2139) — no --force archaeology', () => {
|
||||
expect(KNOWN_CONFIG_KEYS).toContain('spend.posture');
|
||||
expect(KNOWN_CONFIG_KEYS).toContain('sync.cost_gate_min_usd');
|
||||
expect(KNOWN_CONFIG_KEYS).toContain('sync.federated_v2');
|
||||
expect(KNOWN_CONFIG_KEYS).toContain('embed.backfill_cooldown_min');
|
||||
expect(KNOWN_CONFIG_KEYS).toContain('embed.backfill_max_usd_per_source_24h');
|
||||
expect(KNOWN_CONFIG_KEYS).toContain('embed.backfill_max_usd');
|
||||
});
|
||||
|
||||
test('no duplicate entries', () => {
|
||||
const set = new Set(KNOWN_CONFIG_KEYS);
|
||||
expect(set.size).toBe(KNOWN_CONFIG_KEYS.length);
|
||||
|
||||
@@ -155,6 +155,54 @@ describe('submitEmbedBackfill — 24h spend cap', () => {
|
||||
expect(result.status).toBe('spend_capped');
|
||||
expect(result.spendCapUsd).toBe(5);
|
||||
});
|
||||
|
||||
// v0.42.42.0 (#2139): off-switch + tokenmax bypass.
|
||||
test('cap "off" → submits even at huge spend (Infinity cap never tripped)', async () => {
|
||||
await engine.setConfig(SPEND_CAP_CONFIG_KEY, 'off');
|
||||
const result = await submitEmbedBackfill(engine, 'default', {
|
||||
reason: 'unit',
|
||||
spend24hFn: async () => 1e9,
|
||||
});
|
||||
expect(result.status).toBe('submitted');
|
||||
});
|
||||
|
||||
test('0 falls back to the default cap (off semantics ≠ 0)', async () => {
|
||||
await engine.setConfig(SPEND_CAP_CONFIG_KEY, '0');
|
||||
const result = await submitEmbedBackfill(engine, 'default', {
|
||||
reason: 'unit',
|
||||
spend24hFn: async () => 25, // == default $25 → capped
|
||||
});
|
||||
expect(result.status).toBe('spend_capped');
|
||||
expect(result.spendCapUsd).toBe(25);
|
||||
});
|
||||
|
||||
test('spend.posture=tokenmax bypasses the cap, marks spendCapBypassed', async () => {
|
||||
const result = await submitEmbedBackfill(engine, 'default', {
|
||||
reason: 'unit',
|
||||
postureOverride: 'tokenmax',
|
||||
spendCapUsdOverride: 25,
|
||||
spend24hFn: async () => 100, // way over cap
|
||||
});
|
||||
expect(result.status).toBe('submitted');
|
||||
expect(result.spendCapBypassed).toBe(true);
|
||||
expect(result.spend24hUsd).toBe(100);
|
||||
});
|
||||
|
||||
test('tokenmax does NOT bypass the cooldown (axis split — churn protection stays)', async () => {
|
||||
const queue = new MinionQueue(engine);
|
||||
const job = await queue.add('embed-backfill', { sourceId: 'default' }, {});
|
||||
await engine.executeRaw(
|
||||
`UPDATE minion_jobs SET status='completed', finished_at=NOW() - INTERVAL '1 minute' WHERE id=$1`,
|
||||
[job.id],
|
||||
);
|
||||
const result = await submitEmbedBackfill(engine, 'default', {
|
||||
reason: 'unit',
|
||||
postureOverride: 'tokenmax',
|
||||
cooldownMinOverride: 10,
|
||||
spend24hFn: async () => 1e9,
|
||||
});
|
||||
expect(result.status).toBe('cooldown'); // posture lifts the cap, NOT the cooldown
|
||||
});
|
||||
});
|
||||
|
||||
describe('submitEmbedBackfill — source isolation', () => {
|
||||
|
||||
@@ -0,0 +1,53 @@
|
||||
/**
|
||||
* v0.42.42.0 (#2139) — `--max-usd off` / `--max-cost off` uncapped-switch pins
|
||||
* across enrich / onboard / reindex (the T6 secondary cost gates).
|
||||
*
|
||||
* The enrich arg parser carries the real logic (off → Infinity sentinel, mapped
|
||||
* to "no BudgetTracker ceiling" in runEnrichCore), so it gets a direct unit test.
|
||||
* reindex + onboard detect `off` inline at the CLI dispatch and proceed past the
|
||||
* confirmation / missing-cap refusal; those are pinned as source-level regression
|
||||
* guards (the codex pre-landing review found all three half-built — these keep
|
||||
* them from silently regressing without standing up full CLI+gateway harnesses).
|
||||
*/
|
||||
import { describe, test, expect } from 'bun:test';
|
||||
import { parseArgs } from '../src/commands/enrich.ts';
|
||||
|
||||
describe('enrich parseArgs — --max-usd off → uncapped (Infinity sentinel)', () => {
|
||||
test('off / unlimited / none (case-insensitive) → Infinity', () => {
|
||||
for (const v of ['off', 'OFF', 'unlimited', 'none', 'None']) {
|
||||
expect(parseArgs(['--max-usd', v]).maxCostUsd).toBe(Infinity);
|
||||
}
|
||||
// --max-cost-usd alias too.
|
||||
expect(parseArgs(['--max-cost-usd', 'off']).maxCostUsd).toBe(Infinity);
|
||||
});
|
||||
test('finite positive number passes through; absent → undefined; garbage → undefined', () => {
|
||||
expect(parseArgs(['--max-usd', '5']).maxCostUsd).toBe(5);
|
||||
expect(parseArgs([]).maxCostUsd).toBeUndefined();
|
||||
expect(parseArgs(['--max-usd', 'abc']).maxCostUsd).toBeUndefined();
|
||||
expect(parseArgs(['--max-usd', '0']).maxCostUsd).toBeUndefined(); // non-positive ignored
|
||||
});
|
||||
});
|
||||
|
||||
describe('reindex / onboard off-switch dispatch (regression guards)', () => {
|
||||
test('reindex-code: --max-cost off proceeds past the confirmation gate', async () => {
|
||||
const src = await Bun.file(new URL('../src/commands/reindex-code.ts', import.meta.url)).text();
|
||||
// off sets maxCostOff and the gate proceeds when (tokenmax || maxCostOff).
|
||||
expect(src).toMatch(/maxCostOff\s*=\s*true/);
|
||||
expect(src).toMatch(/posture === 'tokenmax' \|\| maxCostOff/);
|
||||
});
|
||||
|
||||
test('onboard: --max-usd off lifts the --auto missing-cap refusal', async () => {
|
||||
const src = await Bun.file(new URL('../src/commands/onboard.ts', import.meta.url)).text();
|
||||
expect(src).toMatch(/maxUsdOff\s*=/);
|
||||
// refusal skipped when (maxUsdOff || tokenmax).
|
||||
expect(src).toMatch(/maxUsdOff \|\| tokenmax/);
|
||||
});
|
||||
|
||||
test('enrich: uncapped path maps the Infinity sentinel to no BudgetTracker ceiling', async () => {
|
||||
const src = await Bun.file(new URL('../src/commands/enrich.ts', import.meta.url)).text();
|
||||
// The sentinel must become `undefined` at the tracker (never raw Infinity,
|
||||
// which serializes to null in audit rows).
|
||||
expect(src).toMatch(/opts\.maxCostUsd === Infinity \? undefined/);
|
||||
expect(src).toMatch(/uncapped \? Infinity : parsed\.maxCostUsd/);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,140 @@
|
||||
/**
|
||||
* v0.42.42.0 (#2139) — estimateInlineNewTokens ladder coverage.
|
||||
*
|
||||
* The inline cost estimator now MIRRORS EXECUTION (delta, not full-tree
|
||||
* ceiling) so the gate's dollar figure stops being a ~400x phantom on a busy
|
||||
* brain. Real temp git repos, no PGLite. estimateInlineNewTokens is exported
|
||||
* from commands/sync.ts; CHUNKER_VERSION is the live chunker version.
|
||||
*/
|
||||
import { describe, test, expect, beforeEach, afterEach } from 'bun:test';
|
||||
import { mkdtempSync, writeFileSync, rmSync, mkdirSync } from 'fs';
|
||||
import { execSync } from 'child_process';
|
||||
import { tmpdir } from 'os';
|
||||
import { join } from 'path';
|
||||
import { estimateInlineNewTokens } from '../src/commands/sync.ts';
|
||||
import { CHUNKER_VERSION } from '../src/core/chunkers/code.ts';
|
||||
|
||||
const CURRENT = String(CHUNKER_VERSION);
|
||||
let repo: string;
|
||||
|
||||
function commitAll(msg: string): string {
|
||||
execSync('git add -A', { cwd: repo, stdio: 'pipe' });
|
||||
execSync(`git commit -m "${msg}"`, { cwd: repo, stdio: 'pipe' });
|
||||
return execSync('git rev-parse HEAD', { cwd: repo, stdio: 'pipe' }).toString().trim();
|
||||
}
|
||||
function src(over: Partial<{ last_commit: string | null; chunker_version: string | null; config: Record<string, unknown> }> = {}) {
|
||||
return {
|
||||
local_path: repo,
|
||||
config: over.config ?? {},
|
||||
last_commit: over.last_commit ?? null,
|
||||
chunker_version: over.chunker_version ?? null,
|
||||
};
|
||||
}
|
||||
|
||||
beforeEach(() => {
|
||||
repo = mkdtempSync(join(tmpdir(), 'gbrain-est-'));
|
||||
execSync('git init', { cwd: repo, stdio: 'pipe' });
|
||||
execSync('git config user.email "t@t.com"', { cwd: repo, stdio: 'pipe' });
|
||||
execSync('git config user.name "T"', { cwd: repo, stdio: 'pipe' });
|
||||
mkdirSync(join(repo, 'topics'), { recursive: true });
|
||||
});
|
||||
afterEach(() => {
|
||||
if (repo) rmSync(repo, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
describe('estimateInlineNewTokens — ladder', () => {
|
||||
test('chunker drift → full-tree ceiling even with an empty git delta', () => {
|
||||
writeFileSync(join(repo, 'topics/a.md'), 'some body content here');
|
||||
const head = commitAll('base');
|
||||
// git unchanged (last_commit == HEAD) but chunker drifted → must NOT be 0.
|
||||
const r = estimateInlineNewTokens([src({ last_commit: head, chunker_version: 'STALE-0' })], CURRENT);
|
||||
expect(r.tokens).toBeGreaterThan(0);
|
||||
expect(r.estimateKind).toBe('ceiling');
|
||||
expect(r.ceilingReasons).toContain('chunker_drift');
|
||||
expect(r.changedSources).toBe(1);
|
||||
});
|
||||
|
||||
test('[D2A headline] HEAD==last_commit + current chunker + DIRTY tree → 0 (unchanged)', () => {
|
||||
writeFileSync(join(repo, 'topics/a.md'), 'body');
|
||||
const head = commitAll('base');
|
||||
// Dirty the tree (untracked scratch + uncommitted edit) — attached sync
|
||||
// imports nothing, so the estimate must be 0. This is the exact pre-fix
|
||||
// false-fire shape.
|
||||
writeFileSync(join(repo, 'scratch.tmp'), 'agent scratch');
|
||||
writeFileSync(join(repo, 'topics/a.md'), 'uncommitted edit');
|
||||
const r = estimateInlineNewTokens([src({ last_commit: head, chunker_version: CURRENT })], CURRENT);
|
||||
expect(r.tokens).toBe(0);
|
||||
expect(r.estimateKind).toBe('unchanged');
|
||||
expect(r.unchangedSources).toBe(1);
|
||||
});
|
||||
|
||||
test('first sync (last_commit null) → full-tree ceiling', () => {
|
||||
writeFileSync(join(repo, 'topics/a.md'), 'body content');
|
||||
commitAll('base');
|
||||
const r = estimateInlineNewTokens([src({ last_commit: null, chunker_version: CURRENT })], CURRENT);
|
||||
expect(r.tokens).toBeGreaterThan(0);
|
||||
expect(r.estimateKind).toBe('ceiling');
|
||||
expect(r.ceilingReasons).toContain('first_sync');
|
||||
});
|
||||
|
||||
test('delta rung: only changed committed files priced; deletes cost 0', () => {
|
||||
writeFileSync(join(repo, 'topics/a.md'), 'a'.repeat(400));
|
||||
writeFileSync(join(repo, 'topics/b.md'), 'b'.repeat(400));
|
||||
const base = commitAll('base');
|
||||
writeFileSync(join(repo, 'topics/a.md'), 'a'.repeat(800)); // modify
|
||||
rmSync(join(repo, 'topics/b.md')); // delete → 0
|
||||
commitAll('change');
|
||||
|
||||
const r = estimateInlineNewTokens([src({ last_commit: base, chunker_version: CURRENT })], CURRENT);
|
||||
expect(r.estimateKind).toBe('delta');
|
||||
expect(r.tokens).toBeGreaterThan(0); // a.md priced
|
||||
// A pure-delete delta would be 0; here a.md modify keeps it > 0. Sanity:
|
||||
// the magnitude is delta-scale (one ~800-char file), not full-tree.
|
||||
});
|
||||
|
||||
test('delta rung: non-syncable changed file is filtered out (markdown strategy)', () => {
|
||||
writeFileSync(join(repo, 'topics/a.md'), 'md');
|
||||
const base = commitAll('base');
|
||||
writeFileSync(join(repo, 'notes.txt'), 'x'.repeat(4000)); // .txt under markdown strategy → not syncable
|
||||
commitAll('add txt');
|
||||
|
||||
const r = estimateInlineNewTokens([src({ last_commit: base, chunker_version: CURRENT })], CURRENT);
|
||||
// No syncable changes → delta priced at 0 tokens (but the source changed).
|
||||
expect(r.tokens).toBe(0);
|
||||
expect(r.estimateKind).toBe('delta');
|
||||
});
|
||||
|
||||
test('syncEnabled:false sources are skipped (neither changed nor unchanged)', () => {
|
||||
writeFileSync(join(repo, 'topics/a.md'), 'body');
|
||||
commitAll('base');
|
||||
const r = estimateInlineNewTokens([src({ last_commit: null, config: { syncEnabled: false } })], CURRENT);
|
||||
expect(r.tokens).toBe(0);
|
||||
expect(r.changedSources).toBe(0);
|
||||
expect(r.unchangedSources).toBe(0);
|
||||
});
|
||||
|
||||
test('mixed: one ceiling source + one unchanged source → estimateKind mixed-or-ceiling, reasons captured', () => {
|
||||
writeFileSync(join(repo, 'topics/a.md'), 'body');
|
||||
const head = commitAll('base');
|
||||
const r = estimateInlineNewTokens(
|
||||
[
|
||||
src({ last_commit: null, chunker_version: CURRENT }), // first_sync ceiling
|
||||
src({ last_commit: head, chunker_version: CURRENT }), // unchanged
|
||||
],
|
||||
CURRENT,
|
||||
);
|
||||
expect(r.ceilingReasons).toContain('first_sync');
|
||||
expect(r.unchangedSources).toBe(1);
|
||||
// hadCeiling true, hadDelta false → 'ceiling' aggregate.
|
||||
expect(r.estimateKind).toBe('ceiling');
|
||||
});
|
||||
|
||||
test('missing local_path source contributes nothing', () => {
|
||||
const r = estimateInlineNewTokens(
|
||||
[{ local_path: null, config: {}, last_commit: null, chunker_version: CURRENT }],
|
||||
CURRENT,
|
||||
);
|
||||
expect(r.tokens).toBe(0);
|
||||
expect(r.changedSources).toBe(0);
|
||||
});
|
||||
});
|
||||
@@ -1,15 +1,17 @@
|
||||
/**
|
||||
* v0.41.31 — `gbrain sync --all` cost-gate wiring regressions (PGLite).
|
||||
* `gbrain sync` cost-gate wiring regressions (PGLite).
|
||||
*
|
||||
* Pure shouldBlockSync / willEmbedSynchronously logic is pinned in
|
||||
* test/sync-cost-preview.test.ts. THIS file pins the end-to-end wiring in
|
||||
* runSync's --all path:
|
||||
* Pure shouldBlockSync / willEmbedSynchronously / parseUsdLimit logic is pinned
|
||||
* in test/sync-cost-preview.test.ts. THIS file pins the end-to-end wiring in
|
||||
* runSync's --all AND single-source paths:
|
||||
*
|
||||
* R-1 (headline): deferred-embed sync --all, non-TTY, with backlog →
|
||||
* emits gate:'deferred_notice' and NEVER exit 2 (the cron-blocking
|
||||
* bug this release fixes).
|
||||
* R-2 (protection): inline-embed sync --all (--serial), non-TTY, above
|
||||
* floor → still exit 2 with gate:'confirmation_required'.
|
||||
* emits gate:'deferred_notice' and NEVER exit 2.
|
||||
* R-2 (v0.42.42.0, #2139): inline sync --all (--serial), non-TTY, above floor
|
||||
* → AUTO-DEFERS (exit 0, gate:'auto_deferred_embeds') and enqueues an
|
||||
* embed-backfill job. The exit-2 wedge is gone.
|
||||
* R-3: chunker drift → full-tree CEILING estimate, auto-defers (not exit 2).
|
||||
* + posture tokenmax, off-switch, format split (#1784/D3A), single-source gate.
|
||||
*
|
||||
* Serial-quarantined: stubs process.exit + console.log (process-global).
|
||||
*/
|
||||
@@ -21,10 +23,18 @@ import { join } from 'path';
|
||||
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
|
||||
import { runSources } from '../src/commands/sources.ts';
|
||||
import { resetPgliteState } from './helpers/reset-pglite.ts';
|
||||
import { configureGateway, resetGateway } from '../src/core/ai/gateway.ts';
|
||||
import { configureGateway, resetGateway, __setEmbedTransportForTests } from '../src/core/ai/gateway.ts';
|
||||
import { CHUNKER_VERSION } from '../src/core/chunkers/code.ts';
|
||||
import type { ChunkInput } from '../src/core/types.ts';
|
||||
|
||||
/** Offline embed stub so inline-proceed paths (posture tokenmax) don't network. */
|
||||
function stubOfflineEmbed(): void {
|
||||
__setEmbedTransportForTests(async ({ values }: any) => ({
|
||||
embeddings: values.map(() => new Array(1536).fill(0)),
|
||||
usage: { tokens: 0 },
|
||||
}) as any);
|
||||
}
|
||||
|
||||
let engine: PGLiteEngine;
|
||||
let repoPath: string;
|
||||
let headSha: string;
|
||||
@@ -64,6 +74,7 @@ beforeEach(async () => {
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
__setEmbedTransportForTests(null);
|
||||
resetGateway();
|
||||
if (repoPath) rmSync(repoPath, { recursive: true, force: true });
|
||||
});
|
||||
@@ -115,39 +126,43 @@ describe('v0.41.31 — sync --all cost gate wiring', () => {
|
||||
expect(stdout).toContain('"gate":"deferred_notice"');
|
||||
}, 60_000);
|
||||
|
||||
test('R-2: inline sync --all (--serial) above floor still exit 2', async () => {
|
||||
test('R-2 (#2139): inline sync --all (--serial) above floor AUTO-DEFERS (exit 0, never exit 2) + enqueues backfill', async () => {
|
||||
await runSources(engine, ['add', 'vault', '--path', repoPath, '--no-federated']);
|
||||
// Floor 0 → any nonzero inline cost blocks. Source is unsynced
|
||||
// (last_commit NULL) so estimateInlineNewTokens sees it as changed →
|
||||
// full-tree tokens > 0 → costUsd > 0 > floor.
|
||||
// Floor 0 → any nonzero inline cost trips the gate. Source is unsynced
|
||||
// (last_commit NULL) → first-sync ceiling > 0 > floor.
|
||||
await engine.setConfig('sync.cost_gate_min_usd', '0');
|
||||
|
||||
// --serial forces inline even with v2 on. --json → non-TTY exit-2 path.
|
||||
// --serial forces inline even with v2 on. --json → non-TTY path → AUTO-DEFER.
|
||||
const { exitCode, stdout } = await runSyncCaptured(['--all', '--serial', '--json', '--no-pull']);
|
||||
|
||||
expect(exitCode).toBe(2);
|
||||
expect(stdout).toContain('"gate":"confirmation_required"');
|
||||
expect(exitCode).not.toBe(2);
|
||||
expect(stdout).toContain('"gate":"auto_deferred_embeds"');
|
||||
expect(stdout).not.toContain('"gate":"confirmation_required"');
|
||||
// The run PROCEEDED to import (the wedge is gone) — embeds were deferred,
|
||||
// not blocked. (The embed-backfill enqueue wiring + its graceful
|
||||
// missing-table tolerance is pinned in embed-backfill-submit.test.ts; the
|
||||
// minion_jobs table isn't provisioned in this gate-wiring harness.)
|
||||
expect(stdout).toContain('"sync_status":"first_sync"');
|
||||
}, 60_000);
|
||||
|
||||
test('R-3: inline, git-unchanged source but STALE chunker_version still estimates (not $0)', async () => {
|
||||
// The unchanged-source short-circuit requires HEAD==last_commit AND clean
|
||||
// tree AND chunker_version == current. Here git is unchanged but the
|
||||
// chunker drifted, so the source must NOT be treated as 0 — sync would
|
||||
// re-chunk + re-embed everything. floor=0 so any nonzero cost blocks.
|
||||
test('R-3 (#2139): chunker drift → full-tree CEILING estimate, auto-defers (not exit 2)', async () => {
|
||||
// git unchanged (HEAD==last_commit) but chunker drifted → the source must
|
||||
// NOT price $0 (sync would re-chunk + re-embed everything). The estimate is
|
||||
// the full-tree ceiling; the gate auto-defers rather than wedging.
|
||||
await runSources(engine, ['add', 'vault', '--path', repoPath, '--no-federated']);
|
||||
await engine.executeRaw(`UPDATE sources SET last_commit = $1, chunker_version = $2 WHERE id = 'vault'`, [headSha, 'STALE-0']);
|
||||
await engine.setConfig('sync.cost_gate_min_usd', '0');
|
||||
|
||||
const { exitCode, stdout } = await runSyncCaptured(['--all', '--serial', '--json', '--no-pull']);
|
||||
|
||||
expect(exitCode).toBe(2);
|
||||
expect(stdout).toContain('"gate":"confirmation_required"');
|
||||
expect(exitCode).not.toBe(2);
|
||||
expect(stdout).toContain('"gate":"auto_deferred_embeds"');
|
||||
expect(stdout).toContain('"estimateKind":"ceiling"');
|
||||
}, 60_000);
|
||||
|
||||
test('R-3 control: inline, git-unchanged + CURRENT chunker_version short-circuits to $0 (no exit 2)', async () => {
|
||||
// Same setup but chunker_version matches current → the source IS unchanged
|
||||
// → contributes 0 new-content tokens → below floor → proceeds (no block).
|
||||
// Proves the short-circuit fires when (and only when) everything matches.
|
||||
test('R-3 control: git-unchanged + CURRENT chunker → $0 estimate, below floor (no auto-defer)', async () => {
|
||||
// Mirrors the executor's up_to_date predicate: HEAD==last_commit AND chunker
|
||||
// matches → 0 new tokens → below floor → proceeds without deferring.
|
||||
await runSources(engine, ['add', 'vault', '--path', repoPath, '--no-federated']);
|
||||
await engine.executeRaw(`UPDATE sources SET last_commit = $1, chunker_version = $2 WHERE id = 'vault'`, [headSha, String(CHUNKER_VERSION)]);
|
||||
await engine.setConfig('sync.cost_gate_min_usd', '0');
|
||||
@@ -155,6 +170,78 @@ describe('v0.41.31 — sync --all cost gate wiring', () => {
|
||||
const { exitCode, stdout } = await runSyncCaptured(['--all', '--serial', '--json', '--no-pull']);
|
||||
|
||||
expect(exitCode).not.toBe(2);
|
||||
expect(stdout).not.toContain('"gate":"confirmation_required"');
|
||||
expect(stdout).not.toContain('"gate":"auto_deferred_embeds"');
|
||||
expect(stdout).toContain('"estimateKind":"unchanged"');
|
||||
}, 60_000);
|
||||
|
||||
test('headline regression: HEAD==last_commit + DIRTY untracked file → $0, no gate (the false-fire)', async () => {
|
||||
// The exact pre-fix false-fire: a busy brain's working tree is never
|
||||
// git-clean, but the commits are caught up. The OLD estimator priced the
|
||||
// whole tree (158M-token phantom); the new one mirrors execution → $0.
|
||||
await runSources(engine, ['add', 'vault', '--path', repoPath, '--no-federated']);
|
||||
await engine.executeRaw(`UPDATE sources SET last_commit = $1, chunker_version = $2 WHERE id = 'vault'`, [headSha, String(CHUNKER_VERSION)]);
|
||||
// Dirty the tree with an untracked non-syncable scratch file (agents/crons
|
||||
// write constantly) — attached-HEAD sync never imports it.
|
||||
writeFileSync(join(repoPath, 'scratch.tmp'), 'uncommitted agent scratch');
|
||||
writeFileSync(join(repoPath, 'topics/foo.md'), 'uncommitted edit, not staged');
|
||||
await engine.setConfig('sync.cost_gate_min_usd', '0');
|
||||
|
||||
const { exitCode, stdout } = await runSyncCaptured(['--all', '--serial', '--json', '--no-pull']);
|
||||
|
||||
expect(exitCode).not.toBe(2);
|
||||
expect(stdout).not.toContain('"gate":"auto_deferred_embeds"');
|
||||
expect(stdout).toContain('"estimateKind":"unchanged"');
|
||||
}, 60_000);
|
||||
|
||||
test('spend.posture=tokenmax → proceeds inline, gate:posture_tokenmax (informational)', async () => {
|
||||
stubOfflineEmbed(); // inline embed proceeds — keep it off the network.
|
||||
await runSources(engine, ['add', 'vault', '--path', repoPath, '--no-federated']);
|
||||
await engine.setConfig('sync.cost_gate_min_usd', '0');
|
||||
await engine.setConfig('spend.posture', 'tokenmax');
|
||||
|
||||
const { exitCode, stdout } = await runSyncCaptured(['--all', '--serial', '--json', '--no-pull']);
|
||||
|
||||
expect(exitCode).not.toBe(2);
|
||||
expect(stdout).toContain('"gate":"posture_tokenmax"');
|
||||
expect(stdout).not.toContain('"gate":"auto_deferred_embeds"');
|
||||
}, 60_000);
|
||||
|
||||
test('sync.cost_gate_min_usd=off → floor renders "unlimited", never blocks', async () => {
|
||||
stubOfflineEmbed();
|
||||
await runSources(engine, ['add', 'vault', '--path', repoPath, '--no-federated']);
|
||||
await engine.setConfig('sync.cost_gate_min_usd', 'off');
|
||||
|
||||
const { exitCode, stdout } = await runSyncCaptured(['--all', '--serial', '--json', '--no-pull']);
|
||||
|
||||
expect(exitCode).not.toBe(2);
|
||||
expect(stdout).toContain('"floorUsd":"unlimited"');
|
||||
expect(stdout).not.toContain('"gate":"auto_deferred_embeds"');
|
||||
}, 60_000);
|
||||
|
||||
test('format split (#1784/D3A): non-TTY WITHOUT --json emits human text, no JSON envelope', async () => {
|
||||
await runSources(engine, ['add', 'vault', '--path', repoPath, '--no-federated']);
|
||||
await engine.setConfig('sync.cost_gate_min_usd', '0');
|
||||
|
||||
// No --json: above floor in a non-TTY session → human auto-defer text.
|
||||
const { exitCode, stdout } = await runSyncCaptured(['--all', '--serial', '--no-pull']);
|
||||
|
||||
expect(exitCode).not.toBe(2);
|
||||
expect(stdout).not.toContain('"gate":'); // no JSON envelope without --json
|
||||
expect(stdout.toLowerCase()).toContain('deferring embeds');
|
||||
expect(stdout).toContain('spend.posture'); // self-describing hint present
|
||||
}, 60_000);
|
||||
|
||||
test('single-source sync gets the same gate (auto-defers above floor, exit 0)', async () => {
|
||||
await runSources(engine, ['add', 'vault', '--path', repoPath, '--no-federated']);
|
||||
await engine.setConfig('sync.cost_gate_min_usd', '0');
|
||||
|
||||
// Single-source (no --all): unsynced → ceiling > 0 → non-TTY auto-defer.
|
||||
const { exitCode, stdout } = await runSyncCaptured(['--source', 'vault', '--json', '--no-pull']);
|
||||
|
||||
expect(exitCode).not.toBe(2);
|
||||
expect(stdout).toContain('"gate":"auto_deferred_embeds"');
|
||||
// The gate now exists on the single-source path (was ungated before
|
||||
// #2139) and proceeds to import rather than blocking.
|
||||
expect(stdout.toLowerCase()).toContain('imported');
|
||||
}, 60_000);
|
||||
});
|
||||
|
||||
@@ -23,6 +23,13 @@ import {
|
||||
import { lookupEmbeddingPrice } from '../src/core/embedding-pricing.ts';
|
||||
import { resetGateway } from '../src/core/ai/gateway.ts';
|
||||
import { estimateTokens } from '../src/core/chunkers/code.ts';
|
||||
import {
|
||||
parseUsdLimit,
|
||||
formatUsdLimit,
|
||||
usdLimitToCap,
|
||||
normalizeSpendPosture,
|
||||
isValidSpendPosture,
|
||||
} from '../src/core/spend-posture.ts';
|
||||
|
||||
describe('Layer 8 D1 — embedding cost model', () => {
|
||||
// The estimateEmbeddingCostUsd tests assert the gateway-UNCONFIGURED OpenAI
|
||||
@@ -127,6 +134,73 @@ describe('v0.41.31 — shouldBlockSync (cost-gate decision)', () => {
|
||||
expect(shouldBlockSync(0.0001, 0, 'inline')).toBe(true);
|
||||
expect(shouldBlockSync(0, 0, 'inline')).toBe(false);
|
||||
});
|
||||
|
||||
// v0.42.42.0 (#2139): posture + Infinity-floor behavior.
|
||||
test('spend.posture=tokenmax never blocks, even above floor', () => {
|
||||
expect(shouldBlockSync(999, 0.5, 'inline', 'tokenmax')).toBe(false);
|
||||
expect(shouldBlockSync(999, 0, 'inline', 'tokenmax')).toBe(false);
|
||||
});
|
||||
test('default posture (gated) preserves the legacy decision', () => {
|
||||
expect(shouldBlockSync(0.51, 0.5, 'inline', 'gated')).toBe(true);
|
||||
expect(shouldBlockSync(0.03, 0.5, 'inline', 'gated')).toBe(false);
|
||||
});
|
||||
test('off/unlimited floor (Infinity) is never exceeded → never blocks', () => {
|
||||
expect(shouldBlockSync(999, Infinity, 'inline')).toBe(false);
|
||||
expect(shouldBlockSync(1e9, Infinity, 'inline', 'gated')).toBe(false);
|
||||
});
|
||||
});
|
||||
|
||||
describe('v0.42.42.0 (#2139) — spend-posture USD-limit parsing', () => {
|
||||
test('off / unlimited / none (case-insensitive) → Infinity', () => {
|
||||
for (const v of ['off', 'OFF', 'unlimited', 'Unlimited', 'none', 'NONE', ' off ']) {
|
||||
expect(parseUsdLimit(v, 25)).toBe(Infinity);
|
||||
}
|
||||
});
|
||||
test('finite positive numbers pass through', () => {
|
||||
expect(parseUsdLimit('5', 25)).toBe(5);
|
||||
expect(parseUsdLimit(0.5, 25)).toBe(0.5);
|
||||
expect(parseUsdLimit('100000', 25)).toBe(100000);
|
||||
});
|
||||
test('0 falls back to default unless allowZero', () => {
|
||||
expect(parseUsdLimit('0', 25)).toBe(25); // backfill cap: off ≠ 0
|
||||
expect(parseUsdLimit('0', 0.5, { allowZero: true })).toBe(0); // floor: 0 = block-on-any
|
||||
});
|
||||
test('garbage / negative / empty / null → default', () => {
|
||||
expect(parseUsdLimit('abc', 25)).toBe(25);
|
||||
expect(parseUsdLimit('-3', 25)).toBe(25);
|
||||
expect(parseUsdLimit('', 25)).toBe(25);
|
||||
expect(parseUsdLimit(null, 25)).toBe(25);
|
||||
expect(parseUsdLimit(undefined, 0.5)).toBe(0.5);
|
||||
});
|
||||
test('formatUsdLimit: Infinity → "unlimited" (never the JSON.stringify=null trap), finite passthrough', () => {
|
||||
expect(formatUsdLimit(Infinity)).toBe('unlimited');
|
||||
expect(formatUsdLimit(5)).toBe(5);
|
||||
expect(formatUsdLimit(0)).toBe(0);
|
||||
// The trap this guards: raw Infinity serializes to null.
|
||||
expect(JSON.stringify({ cap: Infinity })).toBe('{"cap":null}');
|
||||
expect(JSON.stringify({ cap: formatUsdLimit(Infinity) })).toBe('{"cap":"unlimited"}');
|
||||
});
|
||||
test('usdLimitToCap: Infinity → undefined (no cap), finite passthrough', () => {
|
||||
expect(usdLimitToCap(Infinity)).toBeUndefined();
|
||||
expect(usdLimitToCap(10)).toBe(10);
|
||||
});
|
||||
test('normalizeSpendPosture: only tokenmax is tokenmax; everything else gated', () => {
|
||||
expect(normalizeSpendPosture('tokenmax')).toBe('tokenmax');
|
||||
expect(normalizeSpendPosture('TokenMax')).toBe('tokenmax');
|
||||
expect(normalizeSpendPosture('gated')).toBe('gated');
|
||||
expect(normalizeSpendPosture('max')).toBe('gated');
|
||||
expect(normalizeSpendPosture('')).toBe('gated');
|
||||
expect(normalizeSpendPosture(null)).toBe('gated');
|
||||
expect(normalizeSpendPosture(42)).toBe('gated');
|
||||
});
|
||||
test('isValidSpendPosture accepts gated/tokenmax (case-insensitive), rejects the rest', () => {
|
||||
expect(isValidSpendPosture('gated')).toBe(true);
|
||||
expect(isValidSpendPosture('tokenmax')).toBe(true);
|
||||
expect(isValidSpendPosture('TokenMax')).toBe(true); // normalized lowercase
|
||||
expect(isValidSpendPosture('max')).toBe(false);
|
||||
expect(isValidSpendPosture('')).toBe(false);
|
||||
expect(isValidSpendPosture(7)).toBe(false);
|
||||
});
|
||||
});
|
||||
|
||||
describe('Layer 8 D1 — estimateTokens (exported from chunkers/code.ts)', () => {
|
||||
|
||||
@@ -0,0 +1,154 @@
|
||||
/**
|
||||
* v0.42.42.0 (#2139) — computeSyncDelta unit coverage.
|
||||
*
|
||||
* The shared diff/manifest helper that BOTH the sync executor and the inline
|
||||
* cost estimator route through (so the gate's dollar figure can't drift from
|
||||
* what the sync imports). Real temp git repos; no PGLite, no env writes
|
||||
* (R1/R2-clean). The git-runner seam drives the unavailable branches.
|
||||
*/
|
||||
import { describe, test, expect, beforeEach, afterEach } from 'bun:test';
|
||||
import { mkdtempSync, writeFileSync, rmSync, mkdirSync, renameSync } from 'fs';
|
||||
import { execSync } from 'child_process';
|
||||
import { tmpdir } from 'os';
|
||||
import { join } from 'path';
|
||||
import {
|
||||
computeSyncDelta,
|
||||
buildDetachedWorkingTreeManifest,
|
||||
_setGitRunnerForTests,
|
||||
} from '../src/core/sync-delta.ts';
|
||||
|
||||
let repo: string;
|
||||
|
||||
function git(args: string): string {
|
||||
return execSync(`git ${args}`, { cwd: repo, stdio: 'pipe' }).toString().trim();
|
||||
}
|
||||
function commitAll(msg: string): string {
|
||||
execSync('git add -A', { cwd: repo, stdio: 'pipe' });
|
||||
execSync(`git commit -m "${msg}"`, { cwd: repo, stdio: 'pipe' });
|
||||
return git('rev-parse HEAD');
|
||||
}
|
||||
|
||||
beforeEach(() => {
|
||||
repo = mkdtempSync(join(tmpdir(), 'gbrain-delta-'));
|
||||
execSync('git init', { cwd: repo, stdio: 'pipe' });
|
||||
execSync('git config user.email "t@t.com"', { cwd: repo, stdio: 'pipe' });
|
||||
execSync('git config user.name "T"', { cwd: repo, stdio: 'pipe' });
|
||||
mkdirSync(join(repo, 'topics'), { recursive: true });
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
_setGitRunnerForTests(null);
|
||||
if (repo) rmSync(repo, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
describe('computeSyncDelta — commit diff', () => {
|
||||
test('A/M/D classified; only committed changes in the manifest', () => {
|
||||
writeFileSync(join(repo, 'topics/a.md'), 'a');
|
||||
writeFileSync(join(repo, 'topics/b.md'), 'b');
|
||||
const base = commitAll('base');
|
||||
writeFileSync(join(repo, 'topics/a.md'), 'a-edited'); // modify
|
||||
writeFileSync(join(repo, 'topics/c.md'), 'c'); // add
|
||||
rmSync(join(repo, 'topics/b.md')); // delete
|
||||
const head = commitAll('change');
|
||||
|
||||
const r = computeSyncDelta(repo, base, head);
|
||||
expect(r.status).toBe('ok');
|
||||
if (r.status !== 'ok') return;
|
||||
expect(r.manifest.modified).toContain('topics/a.md');
|
||||
expect(r.manifest.added).toContain('topics/c.md');
|
||||
expect(r.manifest.deleted).toContain('topics/b.md');
|
||||
});
|
||||
|
||||
test('rename → destination path on the renamed list', () => {
|
||||
writeFileSync(join(repo, 'topics/old.md'), 'x'.repeat(200));
|
||||
const base = commitAll('base');
|
||||
renameSync(join(repo, 'topics/old.md'), join(repo, 'topics/new.md'));
|
||||
const head = commitAll('rename');
|
||||
|
||||
const r = computeSyncDelta(repo, base, head);
|
||||
expect(r.status).toBe('ok');
|
||||
if (r.status !== 'ok') return;
|
||||
expect(r.manifest.renamed.map(x => x.to)).toContain('topics/new.md');
|
||||
});
|
||||
|
||||
test('[D2A] attached HEAD: dirty tracked + untracked files are NOT in the manifest', () => {
|
||||
writeFileSync(join(repo, 'topics/a.md'), 'a');
|
||||
const base = commitAll('base');
|
||||
const head = git('rev-parse HEAD'); // HEAD == base, no new commits
|
||||
// Dirty the tree: an uncommitted edit + an untracked scratch file.
|
||||
writeFileSync(join(repo, 'topics/a.md'), 'uncommitted edit');
|
||||
writeFileSync(join(repo, 'scratch.tmp'), 'untracked');
|
||||
|
||||
const r = computeSyncDelta(repo, base, head); // not detached → commit diff only
|
||||
expect(r.status).toBe('ok');
|
||||
if (r.status !== 'ok') return;
|
||||
expect(r.manifest.added).toHaveLength(0);
|
||||
expect(r.manifest.modified).toHaveLength(0);
|
||||
expect(r.manifest.deleted).toHaveLength(0);
|
||||
});
|
||||
});
|
||||
|
||||
describe('computeSyncDelta — detached HEAD merges the working-tree manifest', () => {
|
||||
test('detached + working-tree changes → merged into the manifest', () => {
|
||||
writeFileSync(join(repo, 'topics/a.md'), 'a');
|
||||
const base = commitAll('base');
|
||||
// Detach HEAD and dirty the tree.
|
||||
execSync(`git checkout --detach ${base}`, { cwd: repo, stdio: 'pipe' });
|
||||
writeFileSync(join(repo, 'topics/a.md'), 'detached edit'); // tracked modify
|
||||
writeFileSync(join(repo, 'topics/new.md'), 'new'); // untracked add
|
||||
|
||||
const r = computeSyncDelta(repo, base, base, { detached: true });
|
||||
expect(r.status).toBe('ok');
|
||||
if (r.status !== 'ok') return;
|
||||
expect(r.manifest.modified).toContain('topics/a.md');
|
||||
expect(r.manifest.added).toContain('topics/new.md'); // untracked picked up on detached
|
||||
});
|
||||
|
||||
test('buildDetachedWorkingTreeManifest: clean detached tree → empty manifest', () => {
|
||||
writeFileSync(join(repo, 'topics/a.md'), 'a');
|
||||
const base = commitAll('base');
|
||||
execSync(`git checkout --detach ${base}`, { cwd: repo, stdio: 'pipe' });
|
||||
const m = buildDetachedWorkingTreeManifest(repo);
|
||||
expect(m.added).toHaveLength(0);
|
||||
expect(m.modified).toHaveLength(0);
|
||||
});
|
||||
});
|
||||
|
||||
describe('computeSyncDelta — fail-open ladder', () => {
|
||||
test('bogus anchor SHA → unavailable: anchor_missing', () => {
|
||||
writeFileSync(join(repo, 'topics/a.md'), 'a');
|
||||
const head = commitAll('base');
|
||||
const r = computeSyncDelta(repo, 'deadbeefdeadbeefdeadbeefdeadbeefdeadbeef', head);
|
||||
expect(r.status).toBe('unavailable');
|
||||
if (r.status === 'unavailable') expect(r.reason).toBe('anchor_missing');
|
||||
});
|
||||
|
||||
test('non-ancestor anchor still diffs (the #1970 property)', () => {
|
||||
// git diff A..B is endpoint-tree, no ancestry requirement.
|
||||
writeFileSync(join(repo, 'topics/a.md'), 'a');
|
||||
const base = commitAll('base');
|
||||
// Rewrite history: amend creates a new commit not descended from `base`,
|
||||
// but `base` is still on disk (reflog) → diffable.
|
||||
writeFileSync(join(repo, 'topics/a.md'), 'rewritten');
|
||||
execSync('git add -A && git commit --amend -m rewritten', { cwd: repo, stdio: 'pipe' });
|
||||
const head = git('rev-parse HEAD');
|
||||
expect(head).not.toBe(base);
|
||||
|
||||
const r = computeSyncDelta(repo, base, head);
|
||||
expect(r.status).toBe('ok'); // orphaned-but-present anchor is still diffable
|
||||
});
|
||||
|
||||
test('injected git failure on the diff → unavailable: diff_failed', () => {
|
||||
writeFileSync(join(repo, 'topics/a.md'), 'a');
|
||||
const base = commitAll('base');
|
||||
const head = git('rev-parse HEAD');
|
||||
_setGitRunnerForTests((_repo, args) => {
|
||||
if (args[0] === 'cat-file') return 'commit'; // anchor reachable
|
||||
if (args[0] === 'diff') throw new Error('simulated oversized diff / timeout');
|
||||
return '';
|
||||
});
|
||||
const r = computeSyncDelta(repo, base, head);
|
||||
expect(r.status).toBe('unavailable');
|
||||
if (r.status === 'unavailable') expect(r.reason).toBe('diff_failed');
|
||||
});
|
||||
});
|
||||
@@ -170,10 +170,10 @@ describe('Bug 9 — sync.ts CLI flag wiring', () => {
|
||||
// performSync's inner ack path only fires when failedFiles.length > 0
|
||||
// in the current run. This test pins the up-front ack at the top of
|
||||
// runSync so the flag means "ack whatever is currently flagged".
|
||||
// v0.42.42.0 (#2139, D13C): the pre-ack is now scoped PER SOURCE — `--all`
|
||||
// acks every source, single-source acks only its own.
|
||||
const source = await Bun.file(new URL('../src/commands/sync.ts', import.meta.url)).text();
|
||||
// Ensure the up-front check exists before the syncAll / performSync
|
||||
// dispatch, gated on skipFailed.
|
||||
expect(source).toMatch(/if \(skipFailed\) \{[\s\S]*?unacknowledgedSyncFailures\(\)[\s\S]*?acknowledgeSyncFailures\(\)/);
|
||||
expect(source).toMatch(/if \(skipFailed\) \{[\s\S]*?syncAll \? acknowledgeFailures\(\) : acknowledgeFailures\(sourceId\)/);
|
||||
});
|
||||
|
||||
test('acknowledgeSyncFailures clears stale failures end-to-end', async () => {
|
||||
|
||||
Reference in New Issue
Block a user