mirror of
https://github.com/garrytan/gbrain.git
synced 2026-07-27 21:19:18 +00:00
v0.41.31.0 feat(embed): delta-aware sync --all cost gate + real stale-embedding semantics (#1632)
* fix(cost): embedding cost preview uses configured model rate, not hardcoded OpenAI
The sync --all cost gate computed spend from a hardcoded
EMBEDDING_COST_PER_1K_TOKENS = 0.00013 (OpenAI text-embedding-3-large)
and labeled the preview with the back-compat EMBEDDING_MODEL constant,
regardless of the actually-configured embedding model. A brain running a
cheaper model (e.g. zeroentropyai:zembed-1 @ $0.05/Mtok) saw a preview
that named the wrong provider and over-stated spend ~2.6x ($337 vs $130
on a 2.6B-token corpus).
estimateEmbeddingCostUsd now resolves the live model via the gateway and
prices it through embedding-pricing.ts (the existing per-provider:model
table), falling back to the OpenAI rate only when the gateway is
unconfigured (unit-test context) or the model is unknown. sync.ts surfaces
the real model name in the preview message and JSON.
Regression test pins model-aware pricing: openai 3-large vs zembed-1 must
produce materially different previews; collapsing both to the OpenAI number
fails the assertion.
* fix(cost): sync --all gate is informational when embed is deferred; delta-aware inline gate
Under federated_v2 (default), sync --all DEFERS embedding to per-source
embed-backfill jobs that already cap spend at $25/source/24h. The v0.20
cost gate predated that cap and fired ConfirmationRequired + exit 2 on
EVERY non-TTY sync --all without --yes, regardless of cost — blocking
nightly crons over already-synced corpora and forcing permanent --yes.
The gate is now mode-aware:
- Deferred embed (v2 default): print an FYI deferred notice (cap-aware,
"not charged by this sync") + the stale-chunk backlog estimate, and
NEVER exit 2. The backfill cap is the real money gate.
- Inline embed (v2 off, or --serial without --no-embed): keep the
blocking gate, but estimate the actual delta — full-tree ceiling for
changed sources (unchanged sources contribute 0 via the same git +
chunker_version "do work?" gate doctor/sync use) + stale backlog — and
block only when it exceeds the new configurable floor
sync.cost_gate_min_usd (default $0.50).
New pure helpers in embedding.ts (willEmbedSynchronously, shouldBlockSync)
keep the decision logic hermetically testable. New engine method
sumStaleChunkChars (both engines) prices the embedding backlog via
estimateCostFromChars. estimateSyncAllCost's per-source walk extracted to
estimateSourceTreeTokens (reused by the inline estimator).
Regressions pinned: R-1 deferred non-TTY never exit 2 (headline), R-2
inline above-floor still exit 2 (protection), plus the willEmbedSynchronously
/ shouldBlockSync matrix and sumStaleChunkChars engine + scope + embed_skip
coverage.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(embed): real stale semantics — re-embed on model/dims swap (migration v108)
Pre-v0.41.30 "stale" meant only `embedding IS NULL`, so swapping the
embedding model or dimensions left the whole corpus silently embedded under
the OLD model — `embed --stale` ignored it and search quality quietly
degraded.
New `pages.embedding_signature` (TEXT, migration v108) stamps the embedding
provenance (`<provider:model>:<dims>`) whenever a page's chunks are embedded.
A later model/dims swap makes the stored signature differ from the current
one, which the embed paths now detect and re-embed.
GRANDFATHER (critical): the stale predicate is
`embedding IS NULL OR (embedding_signature IS NOT NULL AND <> $current)`
so a NULL signature is NEVER stale. After the migration every existing page
has NULL → none flagged → the next `embed --stale` does NOT re-embed the
whole corpus. Signatures are stamped going forward only.
Surface:
- countStaleChunks / sumStaleChunkChars gain an optional `signature` opt
that widens staleness (read-only; used by the dry-run preview + the
sync cost preview, which is now signature-aware).
- invalidateStaleSignatureEmbeddings(signature, sourceId?) NULLs the
embeddings of signature-mismatched pages so the EXISTING NULL-embedding
cursor (listStaleChunks, untouched) re-embeds them — keeps the keyset
pagination logic intact.
- setPageEmbeddingSignature stamps after a page's chunks land.
- Both embed loops wired: `gbrain embed --stale`/`--all` (embed.ts) and the
embed-backfill minion (embed-stale.ts) invalidate-then-stamp.
Migration v108 + bootstrap probe (both engines) + REQUIRED_BOOTSTRAP_COVERAGE
entry. Pinned by test/embedding-signature-stale.test.ts (R-4 grandfather,
mismatch detection, matching no-op, scoped invalidate, stamp) + the
bootstrap-coverage gate.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(sync): surface embed-backfill job state in sources status + deferred notice
Under federated_v2, `sync --all` exits 0 and embedding lags behind in
embed-backfill jobs (subject to cooldown + the per-source 24h cap). Pre-fix
an operator had no signal those jobs were queued or lagging — the sync looked
"done" while embeddings trickled in later.
`gbrain sources status` now shows a BACKFILL column per source
(active(N)/queued(N)/idle) plus the last completion timestamp, read from
minion_jobs. The deferred-sync notice appends "N backfill job(s) queued" so a
cron operator sees work is enqueued, not lost. Both reads are best-effort —
a brain that never ran a worker (no minion_jobs table) reports idle/0 instead
of crashing the dashboard.
SyncStatusReportSource gains backfill_queued / backfill_active /
backfill_last_completed_at (additive; JSON envelope schema_version unchanged).
Pinned by a new case in test/e2e/sync-status-pglite.test.ts.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test: add currentEmbeddingSignature to embedding.ts mocks + sync stale EXPECTED_PHASES
Commit 3 made embed.ts import currentEmbeddingSignature from embedding.ts.
Four tests mock.module the whole embedding.ts and omitted the new export, so
embed.ts (imported transitively) failed at load with "Export named
'currentEmbeddingSignature' not found". Add the export to each mock:
embed.serial.test.ts, e2e/cycle.test.ts, e2e/dream.test.ts,
e2e/dream-cycle-phase-order-pglite.test.ts.
Also sync the stale EXPECTED_PHASES in dream-cycle-phase-order-pglite.test.ts
to match cycle.ts ALL_PHASES — extract_atoms, synthesize_concepts, and
conversation_facts_backfill drifted in after the test was last touched
(v0.41.0.0) and were never added, so both phase-order assertions were failing
on the branch before this wave (confirmed against 0906ab0a). The dry-run cycle
emits all 20 phases, so mirroring the constant makes both assertions pass.
Pre-existing, unrelated: cycle.test.ts / dream.test.ts have 5 runCycle
failures via direct `bun test` (the conversation_facts_backfill phase uses the
module-singleton getConnection) — present identically at 0906ab0a, not touched
by this wave.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test: pin R-3 chunker-drift regression + embed-signature stamp-call wiring
Ship-workflow coverage audit flagged two gaps:
- R-3 (mandatory regression) had no dedicated test: the inline unchanged-source
short-circuit requires git-unchanged AND chunker_version match, but nothing
pinned the chunker half. Add a case where git is unchanged (HEAD==last_commit,
clean) but chunker_version is stale → estimate still fires (exit 2), plus a
control where chunker matches → short-circuits to $0 (no block).
- The embed loops' setPageEmbeddingSignature call-site was only kept green by the
mock, never asserted. Add a test that runs `embed --all` and asserts the stamp
fires once per page with the current signature.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(embed): stamp embedding_signature on the inline import + per-slug paths (F1); inline cost gate counts new-content only (F2)
Adversarial review caught that the stale-detection feature was inert for
non-federated/inline brains: the embed-write paths that DON'T go through
embed.ts/embed-stale.ts never stamped pages.embedding_signature.
F1 — stamp at the remaining write sites:
- embedPage (gbrain embed <slug> + sync's post-import runEmbedCore({slugs}))
- importFromContent markdown branch (inline import/sync embed + gbrain import)
- importCodeFile (only when EVERY chunk was freshly embedded this call —
reuse-by-hash carries old-model vectors, so a mixed page stays unstamped
rather than falsely marked current)
Without this, inline-synced pages kept NULL signatures → grandfathered → never
re-embedded on a model/dims swap. Now all embed-write paths stamp.
F2 — coupled regression the F1 fix would otherwise introduce: the inline cost
gate added the stale backlog (NULL + signature drift) into the BLOCKING cost,
but `gbrain sync` inline only embeds new/changed content — the backlog is
`gbrain embed --stale`'s job. Once F1 gives inline brains real signatures, a
model swap would inflate the inline gate and block the next cron for cost the
sync never incurs. Inline blocking cost is now new-content only; the stale
backlog is shown informationally ("pending gbrain embed --stale"). Deferred
path keeps the signature-aware backlog FYI (the backfill does clear it).
Pinned by test/import-signature-stamp.serial.test.ts (inline stamp + --no-embed
NULL) and the existing R-2/R-3 inline-gate tests (still exit 2 on new-content).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* chore: bump version and changelog (v0.41.30.0)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(sources): wire BACKFILL into the real `sources status` path (P0a); guard partial-page signature stamping (P0b)
Codex adversarial review caught two issues in the v0.41.30 wave:
P0a — `gbrain sources status` routes through computeAllSourceMetrics
(source-health.ts), not the buildSyncStatusReport helper where the BACKFILL
column was added, so the CLI never showed it. Add per-source embed-backfill
active/queued counts to computeAllSourceMetrics (one extra FILTER on the
existing minion_jobs query) and render a BACKFILL column in `sources status`.
The deferred-sync notice's queued-job count (live sync path) already worked.
P0b — embedPage / embedAllStale / embed-stale stamped embedding_signature
unconditionally after embedding only the STALE subset of a page's chunks. A
partially-embedded page (some chunks preserved from a prior embed under
unknown/old provenance) would be falsely marked current, hiding the old
vectors from future stale detection. Now stamp only when EVERY chunk of the
page was (re)embedded this pass (toEmbed === chunks / stale === existing).
importFromContent embeds the full chunk set so it stays unconditional;
importCodeFile already had the equivalent guard. `gbrain embed --all` fully
re-embeds and stamps mixed pages.
Accepted as documented limitations (not fixed): the inline cost gate can
over-estimate a >100-file `--serial` sync that performSync will defer
(non-default mode, conservative-high bias), and model-swap invalidation NULLs
drifted vectors before re-embed (a deliberate, rare operation).
Pinned by a new backfill-counts case in test/source-health.test.ts.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs: update project documentation for v0.41.30.0
Refresh CLAUDE.md Key Files + Commands for the embedding cost-model + stale-semantics wave: model-aware cost helpers in embedding.ts (currentEmbeddingPricePerMTok / currentEmbeddingSignature / willEmbedSynchronously / shouldBlockSync), the embedding-signature stale-detection engine quartet (sumStaleChunkChars / setPageEmbeddingSignature / invalidateStaleSignatureEmbeddings + widened countStaleChunks), migration v108, signature stamping across embed.ts / import-file.ts, the mode-aware sync --all cost gate + sync.cost_gate_min_usd config key, and the sources status BACKFILL column. Add a same-dimension-swap auto-reembed note to docs/embedding-migrations.md. Regenerate llms-full.txt.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* chore: re-version 0.41.30.0 → 0.41.31.0 (queue slot)
Mechanical version-string sweep across VERSION, package.json, CHANGELOG,
CLAUDE.md, docs, source/test comments, and regenerated llms bundles. No logic
change.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(test): pin embedding dim in cosine-rescore-column test (CI shard-5 contamination)
CI shard 5 failed deterministically with `expected 1280 dimensions, not 1536`.
Root cause: cosine-rescore-column.test.ts hardcodes 1536-dim `embedding`
vectors and asserts length 1536, but its beforeAll ran `initSchema()` with no
gateway config. initSchema sizes the `embedding` column from
getEmbeddingDimensions(), whose default is 1280 (zeroentropyai:zembed-1). The
test only passed by inheriting a leaked 1536 gateway config from an earlier
test (or, locally, from ~/.gbrain). When the v0.41.31 merge shifted the
weight-aware shard bin-packing, the file order changed so the 1280 default won
in CI → vector(1280) column → 1536 insert rejected. (Passed locally because
the dev machine's ~/.gbrain resolves 1536.)
Fix: configureGateway({ openai:text-embedding-3-large, 1536 }) in beforeAll
BEFORE connect/initSchema so the column is deterministically vector(1536)
regardless of ambient/leaked state, and resetGateway() in afterAll for
hygiene. Proven: under a forced-1280 gateway preload the old test reproduces
the exact CI error and the fixed test passes (4/4).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: t <t@t>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Garry Tan
co-authored by
t
Claude Opus 4.8
t
Claude Opus 4.8
parent
63977054af
commit
146a8f1eed
+104
@@ -2,6 +2,110 @@
|
||||
|
||||
All notable changes to GBrain will be documented in this file.
|
||||
|
||||
## [0.41.31.0] - 2026-05-30
|
||||
|
||||
**Your nightly `gbrain sync --all` cron stops getting blocked. It used to
|
||||
demand `--yes` on every non-interactive run even when nothing needed
|
||||
embedding. Now it just runs. And when you change your embedding model,
|
||||
`gbrain embed --stale` actually re-embeds the pages still on the old model
|
||||
instead of silently leaving them stale.**
|
||||
|
||||
Before this release, `gbrain sync --all` always stopped and asked for
|
||||
confirmation in any non-interactive run (cron, script, piped output) based on
|
||||
a guess of your whole corpus's embedding cost. On an already-synced brain that
|
||||
guess was large and the real work was zero, so the cron exited with an error
|
||||
every night and you had to wire in `--yes` permanently, which defeats the
|
||||
safety check.
|
||||
|
||||
Here's what changed. On the default fast sync path, embedding doesn't run
|
||||
during the sync itself; it's handed to background jobs that already cap their
|
||||
own spend at $25 per source per day. So the sync command now prints what's
|
||||
queued and proceeds. It never blocks. You'll only see a confirmation prompt
|
||||
when sync embeds inline (the older non-parallel mode), and even then only when
|
||||
the new content this sync embeds crosses a dollar threshold you control.
|
||||
|
||||
**The threshold knob:**
|
||||
```
|
||||
gbrain config set sync.cost_gate_min_usd 0.50 # default; raise or lower to taste
|
||||
```
|
||||
At or below this estimate an inline sync proceeds silently. Set it to 0 to
|
||||
confirm on any nonzero cost.
|
||||
|
||||
**Real stale detection.** gbrain now records which model and dimensions
|
||||
produced each page's vectors. Change your embedding model (OpenAI to Voyage,
|
||||
or a different dimension) and `gbrain embed --stale` finds and re-embeds every
|
||||
page still on the old model. Before, "stale" only meant "never embedded," so a
|
||||
model swap was invisible and search quietly mixed vector spaces.
|
||||
|
||||
This does NOT trigger a surprise re-embed of your whole brain on upgrade. Pages
|
||||
that predate this release have no recorded signature, and gbrain treats "no
|
||||
signature" as "leave it alone." Only pages embedded after you upgrade carry a
|
||||
signature, and only an actual model change marks them stale.
|
||||
|
||||
**See your embedding backlog.** `gbrain sources status` gains a BACKFILL column
|
||||
showing whether each source has embed jobs queued, running, or idle, plus when
|
||||
the last one finished. After a `sync --all` you can tell at a glance whether
|
||||
embeddings are caught up or still trickling in.
|
||||
|
||||
## To take advantage of v0.41.31.0
|
||||
|
||||
`gbrain upgrade` does this automatically (it applies migration v108, which adds
|
||||
the per-page embedding-signature column). If `gbrain doctor` warns about a
|
||||
partial migration:
|
||||
|
||||
1. **Apply migrations manually:**
|
||||
```bash
|
||||
gbrain apply-migrations --yes
|
||||
```
|
||||
2. **Verify the cron no longer blocks:**
|
||||
```bash
|
||||
gbrain sync --all --dry-run # previews, exits 0, no confirmation demanded
|
||||
gbrain sources status # BACKFILL column present
|
||||
```
|
||||
3. **After a future model swap:** `gbrain embed --stale` re-embeds the drifted
|
||||
pages. (Pages embedded before this upgrade keep their old vectors until they
|
||||
are next re-embedded for some other reason — the no-surprise-cost
|
||||
grandfather behavior.)
|
||||
4. **If anything looks wrong,** file an issue at
|
||||
https://github.com/garrytan/gbrain/issues with the output of `gbrain doctor`.
|
||||
|
||||
### Itemized changes
|
||||
|
||||
#### Sync cost gate
|
||||
- `gbrain sync --all` is mode-aware. When embedding is deferred to backfill
|
||||
jobs (the federated_v2 default), the gate is informational only: it prints a
|
||||
deferred notice (cap-aware, "not charged by this sync") plus the current
|
||||
backlog and never exits 2. The blocking confirmation gate fires only when
|
||||
sync embeds inline AND the new-content estimate exceeds
|
||||
`sync.cost_gate_min_usd` (default $0.50).
|
||||
- The inline estimate is delta-aware: sources unchanged since their last sync
|
||||
(HEAD == last_commit, clean tree, chunker version current) contribute 0;
|
||||
changed sources contribute the full-tree ceiling. The pre-existing stale
|
||||
backlog is shown informationally but does not gate inline sync (that backlog
|
||||
is `gbrain embed --stale`'s job, not sync's).
|
||||
- The cost preview prices against the configured model's real rate rather than
|
||||
a hardcoded OpenAI rate.
|
||||
|
||||
#### Real stale semantics (migration v108)
|
||||
- New `pages.embedding_signature` column records `<provider:model>:<dims>` when
|
||||
a page's chunks are embedded. `gbrain embed --stale`, the embed-backfill
|
||||
jobs, and the sync cost preview treat a page as stale when its signature
|
||||
differs from the current model's. NULL signature is grandfathered (never
|
||||
stale) so upgrading does not mass-re-embed.
|
||||
- All embed-write paths stamp the signature: `gbrain embed`/`--all`/`--stale`,
|
||||
the embed-backfill minion, `gbrain sync`'s inline import, and `gbrain import`.
|
||||
- New engine methods on both Postgres and PGLite: `sumStaleChunkChars`,
|
||||
`setPageEmbeddingSignature`, `invalidateStaleSignatureEmbeddings`, plus an
|
||||
optional `signature` filter on `countStaleChunks` / `sumStaleChunkChars`.
|
||||
|
||||
#### Backfill visibility
|
||||
- `gbrain sources status` shows per-source embed-backfill state
|
||||
(queued/active/idle + last completion). The deferred sync notice appends the
|
||||
queued-job count so a cron operator sees work is enqueued, not lost.
|
||||
|
||||
#### Configuration
|
||||
- `sync.cost_gate_min_usd` (default 0.50) sets the inline-sync confirmation
|
||||
floor.
|
||||
## [0.41.30.0] - 2026-05-30
|
||||
|
||||
**`gbrain lsd --save` (and `brainstorm --save`) now actually writes the
|
||||
|
||||
@@ -10,6 +10,34 @@ change automatically.
|
||||
this mismatch and refuse to silently proceed. This doc is the recipe
|
||||
they point at.
|
||||
|
||||
## Same-dimension model swaps (v0.41.31.0 — automatic)
|
||||
|
||||
If you switch to a different model at the **same** dimension count
|
||||
(e.g. one 1536-dim provider to another, or a re-tuned model that keeps
|
||||
its width), the column type doesn't change, so no `ALTER`/wipe recipe
|
||||
is needed. As of v0.41.31.0, gbrain stamps an embedding-provenance
|
||||
signature (`<provider:model>:<dims>`) onto each page when its chunks are
|
||||
embedded. After you point the config at the new model, the stored
|
||||
signatures differ from the current one, and `gbrain embed --stale`
|
||||
re-embeds exactly those pages:
|
||||
|
||||
```bash
|
||||
# After switching to the new same-dim model in your config:
|
||||
gbrain embed --stale # re-embeds signature-drifted pages
|
||||
gbrain embed --stale --dry-run # preview the count without re-embedding
|
||||
```
|
||||
|
||||
Under federated_v2, the same drift is picked up by the per-source
|
||||
`embed-backfill` jobs that `gbrain sync --all` enqueues (capped
|
||||
`$X/source/24h`). **Grandfather:** pages embedded before v0.41.31.0
|
||||
carry a NULL signature and are NEVER flagged stale, so upgrading to
|
||||
v0.41.31.0 does NOT trigger a whole-corpus re-embed. Signatures only
|
||||
get stamped going forward.
|
||||
|
||||
A **dimension** change still requires the wipe-and-reinit (PGLite) or
|
||||
column-alter (Postgres) recipe below — the on-disk `vector(N)` width
|
||||
genuinely has to change.
|
||||
|
||||
## Why we don't do this automatically
|
||||
|
||||
Switching dimensions requires:
|
||||
|
||||
+15
-9
File diff suppressed because one or more lines are too long
+1
-1
@@ -141,5 +141,5 @@
|
||||
"bun": ">=1.3.10"
|
||||
},
|
||||
"license": "MIT",
|
||||
"version": "0.41.30.0"
|
||||
"version": "0.41.31.0"
|
||||
}
|
||||
|
||||
+46
-3
@@ -1,5 +1,5 @@
|
||||
import type { BrainEngine } from '../core/engine.ts';
|
||||
import { embedBatch } from '../core/embedding.ts';
|
||||
import { embedBatch, currentEmbeddingSignature } from '../core/embedding.ts';
|
||||
import type { ChunkInput } from '../core/types.ts';
|
||||
import { chunkText } from '../core/chunkers/recursive.ts';
|
||||
import { createProgress, type ProgressReporter } from '../core/progress.ts';
|
||||
@@ -378,6 +378,16 @@ async function embedPage(
|
||||
}));
|
||||
|
||||
await engine.upsertChunks(slug, updated, opts);
|
||||
// v0.41.31: stamp provenance so a later model/dims swap is detectable as
|
||||
// stale. embedPage is the per-slug path used by `gbrain embed <slug>` AND
|
||||
// by `gbrain sync`'s post-import embed step (runEmbedCore({slugs})).
|
||||
// Guard: only stamp when EVERY chunk was (re)embedded this pass. If some
|
||||
// chunks were preserved from a prior embed (unknown/old provenance), the
|
||||
// page is mixed — don't claim it's current. `embed --all` fully re-embeds
|
||||
// such a page and then stamps it.
|
||||
if (toEmbed.length === chunks.length) {
|
||||
await engine.setPageEmbeddingSignature(slug, { sourceId, signature: currentEmbeddingSignature() });
|
||||
}
|
||||
result.embedded += toEmbed.length;
|
||||
result.pages_processed++;
|
||||
slog(`${slug}: embedded ${toEmbed.length} chunks`);
|
||||
@@ -396,6 +406,10 @@ async function embedAll(
|
||||
catchUp?: boolean;
|
||||
},
|
||||
) {
|
||||
// v0.41.31: current embedding provenance signature. Stamped onto pages
|
||||
// when their chunks are (re)embedded so a later model/dimension swap is
|
||||
// detectable as stale.
|
||||
const signature = currentEmbeddingSignature();
|
||||
// ─────────────────────────────────────────────────────────────
|
||||
// Stale-only fast path: avoid the listPages + per-page getChunks
|
||||
// bomb that pulled every page row + every chunk's embedding column
|
||||
@@ -412,7 +426,7 @@ async function embedAll(
|
||||
if (staleOnly) {
|
||||
// D7: thread sourceId so `gbrain embed --stale --source X` actually scopes.
|
||||
// v0.41.18.0 (A13): thread batchSize/priority/catchUp into the stale path.
|
||||
return await embedAllStale(engine, sourceId, dryRun, result, onProgress, staleOpts);
|
||||
return await embedAllStale(engine, sourceId, dryRun, result, onProgress, staleOpts, signature);
|
||||
}
|
||||
|
||||
// v0.31.12: when sourceId is set, scope listPages to that source.
|
||||
@@ -482,6 +496,9 @@ async function embedAll(
|
||||
token_count: c.token_count || Math.ceil(c.chunk_text.length / 4),
|
||||
}));
|
||||
await engine.upsertChunks(page.slug, updated, pageOpts);
|
||||
// v0.41.31: stamp embedding provenance so a later model swap is
|
||||
// detectable as stale.
|
||||
await engine.setPageEmbeddingSignature(page.slug, { sourceId: pageSourceId, signature });
|
||||
result.embedded += toEmbed.length;
|
||||
} catch (e: unknown) {
|
||||
serr(`\n Error embedding ${page.slug}: ${e instanceof Error ? e.message : e}`);
|
||||
@@ -543,13 +560,31 @@ async function embedAllStale(
|
||||
priority?: 'recent';
|
||||
catchUp?: boolean;
|
||||
},
|
||||
signature?: string,
|
||||
) {
|
||||
// D7: thread sourceId so source-scoped runs only count + visit
|
||||
// that source's NULL embeddings.
|
||||
const sourceOpt = sourceId ? { sourceId } : undefined;
|
||||
|
||||
// v0.41.31: re-embed pages whose embedding_signature drifted (model/dims
|
||||
// swap). dry-run must NOT mutate, so it counts signature-stale via the
|
||||
// widened predicate; a live run NULLs them first so the existing
|
||||
// NULL-embedding cursor (listStaleChunks) picks them up unchanged.
|
||||
if (!dryRun && signature) {
|
||||
const invalidated = await engine.invalidateStaleSignatureEmbeddings({
|
||||
signature,
|
||||
...(sourceId && { sourceId }),
|
||||
});
|
||||
if (invalidated > 0) {
|
||||
slog(`[embed] invalidated ${invalidated} chunk(s) embedded under a prior model signature`);
|
||||
}
|
||||
}
|
||||
|
||||
// Pre-flight: 0 stale chunks → nothing to do, no further DB reads.
|
||||
const staleCount = await engine.countStaleChunks(sourceOpt);
|
||||
// dry-run includes signature-drift in the count without mutating.
|
||||
const staleCount = await engine.countStaleChunks(
|
||||
dryRun && signature ? { ...sourceOpt, signature } : sourceOpt,
|
||||
);
|
||||
if (staleCount === 0) {
|
||||
if (dryRun) {
|
||||
slog('[dry-run] Would embed 0 chunks (0 stale found)');
|
||||
@@ -671,6 +706,14 @@ async function embedAllStale(
|
||||
token_count: c.token_count || Math.ceil(c.chunk_text.length / 4),
|
||||
}));
|
||||
await engine.upsertChunks(slug, merged, { sourceId: keySourceId });
|
||||
// v0.41.31: stamp provenance after the page's chunks are embedded —
|
||||
// but only when EVERY chunk was stale (fully re-embedded this pass).
|
||||
// A partially-stale page keeps preserved chunks of unknown/old
|
||||
// provenance, so don't claim it's current. (After invalidate, a
|
||||
// signature-drifted page IS fully stale → this stamps it.)
|
||||
if (signature && stale.length === existing.length) {
|
||||
await engine.setPageEmbeddingSignature(slug, { sourceId: keySourceId, signature });
|
||||
}
|
||||
result.embedded += stale.length;
|
||||
} catch (e: unknown) {
|
||||
// Budget-fired aborts are expected on the way out; don't spam
|
||||
|
||||
+10
-3
@@ -599,22 +599,29 @@ async function runStatus(engine: BrainEngine, args: string[]): Promise<void> {
|
||||
return;
|
||||
}
|
||||
|
||||
// Human-readable table: SOURCE | LAG | EMBED | FAILS | QUEUE | PAGES | LAST SYNC
|
||||
// Human-readable table: SOURCE | LAG | EMBED | BACKFILL | FAILS | QUEUE | PAGES | LAST SYNC
|
||||
console.log('SOURCES — health');
|
||||
console.log('────────────────');
|
||||
console.log(
|
||||
` ${'SOURCE'.padEnd(20)} ${'LAG'.padEnd(8)} ${'EMBED'.padEnd(7)} ${'FAILS'.padEnd(6)} ${'QUEUE'.padEnd(6)} ${'PAGES'.padStart(8)} LAST SYNC`,
|
||||
` ${'SOURCE'.padEnd(20)} ${'LAG'.padEnd(8)} ${'EMBED'.padEnd(7)} ${'BACKFILL'.padEnd(9)} ${'FAILS'.padEnd(6)} ${'QUEUE'.padEnd(6)} ${'PAGES'.padStart(8)} LAST SYNC`,
|
||||
);
|
||||
for (const m of metrics) {
|
||||
const lag = m.lag_seconds === null
|
||||
? 'never'
|
||||
: formatLag(m.lag_seconds);
|
||||
const embed = `${m.embed_coverage_pct.toFixed(0)}%`;
|
||||
// v0.41.31: embed-backfill state (active beats queued beats idle) so a
|
||||
// cron operator sees deferred embedding work after `sync --all`.
|
||||
const backfill = m.backfill_active > 0
|
||||
? `active(${m.backfill_active})`
|
||||
: m.backfill_queued > 0
|
||||
? `queued(${m.backfill_queued})`
|
||||
: 'idle';
|
||||
const fails = String(m.failed_jobs_24h);
|
||||
const queue = String(m.queue_depth);
|
||||
const pages = m.total_pages.toLocaleString();
|
||||
const sync = m.last_sync_at ? new Date(m.last_sync_at).toISOString().slice(0, 19).replace('T', ' ') : 'never';
|
||||
console.log(` ${m.source_id.padEnd(20)} ${lag.padEnd(8)} ${embed.padEnd(7)} ${fails.padEnd(6)} ${queue.padEnd(6)} ${pages.padStart(8)} ${sync}`);
|
||||
console.log(` ${m.source_id.padEnd(20)} ${lag.padEnd(8)} ${embed.padEnd(7)} ${backfill.padEnd(9)} ${fails.padEnd(6)} ${queue.padEnd(6)} ${pages.padStart(8)} ${sync}`);
|
||||
}
|
||||
console.log('');
|
||||
for (const m of metrics) {
|
||||
|
||||
+300
-100
@@ -17,7 +17,17 @@ import {
|
||||
formatCodeBreakdown,
|
||||
} from '../core/sync.ts';
|
||||
import { estimateTokens, CHUNKER_VERSION } from '../core/chunkers/code.ts';
|
||||
import { EMBEDDING_MODEL, estimateEmbeddingCostUsd } from '../core/embedding.ts';
|
||||
import {
|
||||
estimateEmbeddingCostUsd,
|
||||
getEmbeddingModelName,
|
||||
currentEmbeddingPricePerMTok,
|
||||
currentEmbeddingSignature,
|
||||
willEmbedSynchronously,
|
||||
shouldBlockSync,
|
||||
} from '../core/embedding.ts';
|
||||
import { estimateCostFromChars } from '../core/embedding-pricing.ts';
|
||||
import { isSourceUnchangedSinceSync } from '../core/git-head.ts';
|
||||
import { SPEND_CAP_CONFIG_KEY } from '../core/embed-backfill-submit.ts';
|
||||
import { errorFor, serializeError } from '../core/errors.ts';
|
||||
import type { SyncManifest } from '../core/sync.ts';
|
||||
import { createProgress } from '../core/progress.ts';
|
||||
@@ -78,64 +88,114 @@ export interface SyncResult {
|
||||
}
|
||||
|
||||
/**
|
||||
* v0.20.0 Cathedral II Layer 8 (D1) — walk each source's working tree and
|
||||
* sum tokens for every syncable file. This is a conservative overestimate
|
||||
* (full file content, not just the incremental diff) because `sync --all`
|
||||
* on a source that hasn't been synced yet WILL embed every file in the
|
||||
* working tree. For already-synced sources with only incremental changes,
|
||||
* the overestimate is the ceiling, not the floor — users never get
|
||||
* surprised by MORE cost than the preview claims. The false-high bias is
|
||||
* intentional: a lower estimate that undersells the real bill would be
|
||||
* worse than one that oversells.
|
||||
* Walk ONE source's working tree and sum tokens for every syncable file.
|
||||
* Conservative full-tree ceiling (full file content, not the incremental
|
||||
* diff) — over-counts, never under-counts, and matches the filesystem set
|
||||
* `sync` actually imports (collectSyncableFiles + content_hash, NOT a git
|
||||
* commit diff). Best-effort per file and per source: anything unreadable
|
||||
* contributes 0 rather than blocking the preview.
|
||||
*
|
||||
* v0.31.2: routed through collectSyncableFiles (lstat + inode-cycle +
|
||||
* max-depth) so the preview walks exactly what the real sync walks.
|
||||
*/
|
||||
function estimateSyncAllCost(sources: Array<{ local_path: string | null; config: Record<string, unknown> }>): {
|
||||
totalTokens: number;
|
||||
totalFiles: number;
|
||||
activeSources: number;
|
||||
perSource: Array<{ path: string; tokens: number; files: number }>;
|
||||
} {
|
||||
let totalTokens = 0;
|
||||
let totalFiles = 0;
|
||||
let activeSources = 0;
|
||||
const perSource: Array<{ path: string; tokens: number; files: number }> = [];
|
||||
function estimateSourceTreeTokens(
|
||||
localPath: string,
|
||||
strategy: 'markdown' | 'code' | 'auto',
|
||||
): { tokens: number; files: number } {
|
||||
let tokens = 0;
|
||||
let files = 0;
|
||||
try {
|
||||
const fileList = collectSyncableFiles(localPath, { strategy });
|
||||
for (const fullPath of fileList) {
|
||||
try {
|
||||
const stat = statSync(fullPath);
|
||||
if (stat.size > 5_000_000) continue; // skip large binaries
|
||||
const content = readFileSync(fullPath, 'utf-8');
|
||||
tokens += estimateTokens(content);
|
||||
files++;
|
||||
} catch {
|
||||
// Best-effort per file; sync itself tolerates the same.
|
||||
}
|
||||
}
|
||||
} catch {
|
||||
// Best-effort: a source whose local_path is gone/unreadable contributes 0.
|
||||
}
|
||||
return { tokens, files };
|
||||
}
|
||||
|
||||
/**
|
||||
* v0.41.31 — INLINE-path new-content estimate. Per source, contribute ZERO
|
||||
* when the source is provably unchanged since its last sync (HEAD ==
|
||||
* last_commit AND clean working tree AND chunker_version matches CURRENT) —
|
||||
* `content_hash` short-circuits every file so nothing re-embeds. Otherwise
|
||||
* contribute the full-tree ceiling. The unchanged predicate mirrors
|
||||
* doctor's `sync_freshness` and sync's own "do work?" gate (sync.ts:1057+
|
||||
* 1075). `isSourceUnchangedSinceSync` is fail-open (probe error → false), so
|
||||
* a source we can't prove unchanged is conservatively re-estimated rather
|
||||
* than silently priced at $0.
|
||||
*/
|
||||
function estimateInlineNewTokens(
|
||||
sources: Array<{
|
||||
local_path: string | null;
|
||||
config: Record<string, unknown>;
|
||||
last_commit: string | null;
|
||||
chunker_version: string | null;
|
||||
}>,
|
||||
currentChunkerVersion: string,
|
||||
): { tokens: number; changedSources: number; unchangedSources: number } {
|
||||
let tokens = 0;
|
||||
let changedSources = 0;
|
||||
let unchangedSources = 0;
|
||||
for (const src of sources) {
|
||||
if (!src.local_path) continue;
|
||||
const cfg = (src.config || {}) as { syncEnabled?: boolean; strategy?: 'markdown' | 'code' | 'auto' };
|
||||
if (cfg.syncEnabled === false) continue;
|
||||
activeSources++;
|
||||
let sourceTokens = 0;
|
||||
let sourceFiles = 0;
|
||||
try {
|
||||
// v0.31.2: cost preview routed through collectSyncableFiles
|
||||
// (single hardened walker; see import.ts). Previously
|
||||
// walkSyncableFiles used statSync (followed symlinks). New walker
|
||||
// uses lstat + inode-cycle + max-depth so the preview matches
|
||||
// what the real sync will actually walk.
|
||||
const files = collectSyncableFiles(src.local_path, { strategy: cfg.strategy ?? 'markdown' });
|
||||
for (const fullPath of files) {
|
||||
try {
|
||||
const stat = statSync(fullPath);
|
||||
if (stat.size > 5_000_000) continue; // skip large binaries
|
||||
const content = readFileSync(fullPath, 'utf-8');
|
||||
sourceTokens += estimateTokens(content);
|
||||
sourceFiles++;
|
||||
} catch {
|
||||
// Best-effort per file. Skip unreadable files silently;
|
||||
// sync itself tolerates the same.
|
||||
}
|
||||
}
|
||||
} catch {
|
||||
// Best-effort: a source whose local_path is gone or unreadable just
|
||||
// contributes 0. The sync itself would have failed anyway; no point
|
||||
// blocking the preview on a pre-existing fault.
|
||||
const unchanged =
|
||||
isSourceUnchangedSinceSync(src.local_path, src.last_commit, { requireCleanWorkingTree: true }) &&
|
||||
src.chunker_version === currentChunkerVersion;
|
||||
if (unchanged) {
|
||||
unchangedSources++;
|
||||
continue;
|
||||
}
|
||||
totalTokens += sourceTokens;
|
||||
totalFiles += sourceFiles;
|
||||
perSource.push({ path: src.local_path, tokens: sourceTokens, files: sourceFiles });
|
||||
changedSources++;
|
||||
tokens += estimateSourceTreeTokens(src.local_path, cfg.strategy ?? 'markdown').tokens;
|
||||
}
|
||||
return { tokens, changedSources, unchangedSources };
|
||||
}
|
||||
|
||||
return { totalTokens, totalFiles, activeSources, perSource };
|
||||
/**
|
||||
* Resolve the inline-path cost-gate floor in USD. Config key
|
||||
* `sync.cost_gate_min_usd` (DB plane), default $0.50. Below this estimate
|
||||
* the inline gate proceeds without blocking. Fail-open to the default on a
|
||||
* missing/invalid value or a config-read error (the gate must never crash
|
||||
* the sync). Accepts 0 (an operator can set the floor to $0 to make the
|
||||
* gate block on any nonzero inline cost).
|
||||
*/
|
||||
async function resolveCostGateFloorUsd(engine: BrainEngine): Promise<number> {
|
||||
try {
|
||||
const raw = await engine.getConfig('sync.cost_gate_min_usd');
|
||||
if (raw === null || raw === undefined) return 0.5;
|
||||
const n = Number(raw);
|
||||
return Number.isFinite(n) && n >= 0 ? n : 0.5;
|
||||
} catch {
|
||||
return 0.5;
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Resolve the per-source embed-backfill 24h spend cap (USD) for the deferred
|
||||
* notice. Mirrors embed-backfill-submit.ts's own resolution of
|
||||
* SPEND_CAP_CONFIG_KEY (default 25). Fail-open to the default.
|
||||
*/
|
||||
async function resolveBackfillCapUsd(engine: BrainEngine): Promise<number> {
|
||||
try {
|
||||
const raw = await engine.getConfig(SPEND_CAP_CONFIG_KEY);
|
||||
if (raw === null || raw === undefined) return 25;
|
||||
const n = Number(raw);
|
||||
return Number.isFinite(n) && n > 0 ? n : 25;
|
||||
} catch {
|
||||
return 25;
|
||||
}
|
||||
}
|
||||
|
||||
/** Interactive [y/N] prompt. Resolves false on non-y answers or EOF. */
|
||||
@@ -2231,62 +2291,149 @@ See also:
|
||||
// source (no checkout) has nothing for `sync` to pull. Sources with
|
||||
// syncEnabled=false in config.jsonb are skipped too.
|
||||
if (syncAll) {
|
||||
const sources = await engine.executeRaw<{ id: string; name: string; local_path: string | null; config: Record<string, unknown> }>(
|
||||
`SELECT id, name, local_path, config FROM sources WHERE local_path IS NOT NULL`,
|
||||
// v0.41.31: SELECT carries last_commit + chunker_version so the inline
|
||||
// cost preview's "unchanged source → 0" short-circuit can mirror sync's
|
||||
// own "do work?" gate (sync.ts:1057+1075) + doctor's sync_freshness.
|
||||
// Both columns predate v0.41 (writeSyncAnchor / writeChunkerVersion); no
|
||||
// schema migration needed.
|
||||
const sources = await engine.executeRaw<{ id: string; name: string; local_path: string | null; config: Record<string, unknown>; last_commit: string | null; chunker_version: string | null }>(
|
||||
`SELECT id, name, local_path, config, last_commit, chunker_version FROM sources WHERE local_path IS NOT NULL`,
|
||||
);
|
||||
if (!sources || sources.length === 0) {
|
||||
console.log('No sources with local_path configured. Use `gbrain sources add <id> --path <path>` first.');
|
||||
return;
|
||||
}
|
||||
|
||||
// v0.20.0 Cathedral II Layer 8 D1 — cost preview + ConfirmationRequired
|
||||
// gate. Before kicking off a multi-source sync that may embed tens of
|
||||
// thousands of chunks (real money), walk the sync-diff set(s), sum
|
||||
// tokens, compute USD estimate, and gate:
|
||||
// - TTY + !json + !yes → interactive [y/N] prompt
|
||||
// - non-TTY OR --json OR piped → emit ConfirmationRequired envelope,
|
||||
// exit 2 (reserve 1 for runtime errors)
|
||||
// - --yes → skip prompt entirely
|
||||
// - --dry-run → preview + exit 0
|
||||
// Skipped entirely when --no-embed is set (user already opted out of
|
||||
// the cost and will run `embed --stale` later).
|
||||
// v0.41.31 — mode-aware cost gate. Resolve federated_v2 ONCE here so both
|
||||
// the gate (below) and the fan-out (further down) share it.
|
||||
const { isFederatedV2Enabled } = await import('../core/feature-flags.ts');
|
||||
const v2Enabled = await isFederatedV2Enabled(engine);
|
||||
|
||||
// v0.41.31 cost gate (supersedes the v0.20.0 unconditional gate). Under
|
||||
// federated_v2 sync DEFERS embedding to per-source embed-backfill jobs
|
||||
// that carry their own $X/source/24h spend cap, so sync itself spends
|
||||
// nothing synchronously — the gate is INFORMATIONAL (never exit 2) on
|
||||
// that path. The blocking ConfirmationRequired gate fires ONLY when embed
|
||||
// runs INLINE (v2 off, or --serial without --no-embed) AND the estimated
|
||||
// spend exceeds `sync.cost_gate_min_usd` (default $0.50). Skipped entirely
|
||||
// when --no-embed is set (user opted out; will run `embed --stale` later).
|
||||
if (!noEmbed) {
|
||||
const preview = estimateSyncAllCost(sources);
|
||||
const costUsd = estimateEmbeddingCostUsd(preview.totalTokens);
|
||||
const previewMsg =
|
||||
`sync --all preview: ${preview.totalFiles} files across ${preview.activeSources} source(s), ` +
|
||||
`~${preview.totalTokens.toLocaleString()} tokens, est. $${costUsd.toFixed(2)} on ${EMBEDDING_MODEL}.`;
|
||||
|
||||
if (dryRun) {
|
||||
if (jsonOut) {
|
||||
console.log(JSON.stringify({ status: 'dry_run', preview, costUsd, model: EMBEDDING_MODEL }));
|
||||
} else {
|
||||
console.log(previewMsg);
|
||||
console.log('--dry-run: exit without syncing.');
|
||||
}
|
||||
return;
|
||||
const mode = willEmbedSynchronously({ v2Enabled, serialFlag, noEmbed });
|
||||
// Stale backlog: cheap single SQL; fail-open to 0 so a transient DB
|
||||
// hiccup never blocks the sync.
|
||||
let staleChars = 0;
|
||||
try {
|
||||
// v0.41.31: signature-aware so a model/dims swap surfaces in the
|
||||
// backlog estimate (NULL signature grandfathered → not counted).
|
||||
staleChars = await engine.sumStaleChunkChars({ signature: currentEmbeddingSignature() });
|
||||
} catch {
|
||||
staleChars = 0;
|
||||
}
|
||||
const rate = currentEmbeddingPricePerMTok();
|
||||
const staleCostUsd = estimateCostFromChars(staleChars, rate);
|
||||
const embeddingModelName = getEmbeddingModelName();
|
||||
const floorUsd = await resolveCostGateFloorUsd(engine);
|
||||
|
||||
if (!yesFlag) {
|
||||
const isTTY = Boolean(process.stdout.isTTY) && Boolean(process.stdin.isTTY);
|
||||
if (!isTTY || jsonOut) {
|
||||
// Agent-facing path: emit structured envelope, exit 2.
|
||||
const envelope = serializeError(errorFor({
|
||||
class: 'ConfirmationRequired',
|
||||
code: 'cost_preview_requires_yes',
|
||||
message: previewMsg,
|
||||
hint: 'Pass --yes to proceed, or --dry-run to see the preview and exit 0.',
|
||||
}));
|
||||
console.log(JSON.stringify({ error: envelope, preview, costUsd, model: EMBEDDING_MODEL }));
|
||||
process.exit(2);
|
||||
if (mode === 'deferred') {
|
||||
// Deferred path: print an FYI, NEVER exit 2. The backfill cap is the
|
||||
// real money gate (D1/D4).
|
||||
const capUsd = await resolveBackfillCapUsd(engine);
|
||||
// v0.41.31 (TODO-2): surface already-queued backfill jobs so a cron
|
||||
// operator sees work is enqueued, not lost. Best-effort — minion_jobs
|
||||
// may not exist on a brain that never ran a worker.
|
||||
let queuedBackfills = 0;
|
||||
try {
|
||||
const r = await engine.executeRaw<{ n: number }>(
|
||||
`SELECT COUNT(*)::int AS n FROM minion_jobs
|
||||
WHERE name = 'embed-backfill'
|
||||
AND status IN ('waiting','active','delayed','waiting-children')`,
|
||||
);
|
||||
queuedBackfills = Number(r[0]?.n) || 0;
|
||||
} catch {
|
||||
queuedBackfills = 0;
|
||||
}
|
||||
// Interactive TTY path: prompt [y/N].
|
||||
console.log(previewMsg);
|
||||
const answer = await promptYesNo('Proceed? [y/N] ');
|
||||
if (!answer) {
|
||||
console.log('Cancelled.');
|
||||
const deferredMsg =
|
||||
`sync --all: embedding deferred to backfill jobs ` +
|
||||
`(capped $${capUsd}/source/24h, not charged by this sync). ` +
|
||||
`Current backlog ~${staleChars.toLocaleString()} chars (~$${staleCostUsd.toFixed(2)} on ` +
|
||||
`${embeddingModelName}) across ${sources.length} source(s); ` +
|
||||
`${queuedBackfills} backfill job(s) queued.`;
|
||||
if (dryRun) {
|
||||
if (jsonOut) {
|
||||
console.log(JSON.stringify({ status: 'dry_run', mode, gate: 'dry_run', staleChars, staleCostUsd, capUsd, floorUsd, queuedBackfills, model: embeddingModelName }));
|
||||
} else {
|
||||
console.log(deferredMsg);
|
||||
console.log('--dry-run: exit without syncing.');
|
||||
}
|
||||
return;
|
||||
}
|
||||
if (jsonOut) {
|
||||
console.log(JSON.stringify({ status: 'deferred', mode, gate: 'deferred_notice', staleChars, staleCostUsd, capUsd, floorUsd, queuedBackfills, model: embeddingModelName }));
|
||||
} else {
|
||||
console.log(deferredMsg);
|
||||
}
|
||||
// fall through to sync — no exit 2.
|
||||
} else {
|
||||
// Inline path: sync embeds synchronously with no backfill cap to
|
||||
// protect it, so the blocking gate applies. The BLOCKING cost is the
|
||||
// new-content estimate ONLY (full-tree ceiling for changed sources;
|
||||
// unchanged contribute 0) — that's what this sync actually embeds.
|
||||
// The pre-existing stale backlog (NULL embeddings + signature drift)
|
||||
// is NOT swept by sync; `gbrain embed --stale` clears it. So we show
|
||||
// it informationally but never gate on cost this sync won't incur
|
||||
// (else a model swap would block the next inline cron — F2).
|
||||
const currentChunkerVersion = String(CHUNKER_VERSION);
|
||||
const inline = estimateInlineNewTokens(sources, currentChunkerVersion);
|
||||
const newCostUsd = estimateEmbeddingCostUsd(inline.tokens);
|
||||
const costUsd = newCostUsd;
|
||||
const staleNote = staleChars > 0
|
||||
? ` (plus ~${staleChars.toLocaleString()} stale-backlog chars pending \`gbrain embed --stale\`)`
|
||||
: '';
|
||||
const previewMsg =
|
||||
`sync --all preview (inline embed): ${inline.changedSources} changed source(s), ` +
|
||||
`${inline.unchangedSources} unchanged; ~${inline.tokens.toLocaleString()} new tokens, ` +
|
||||
`est. $${costUsd.toFixed(2)} on ${embeddingModelName}${staleNote}.`;
|
||||
|
||||
if (dryRun) {
|
||||
if (jsonOut) {
|
||||
console.log(JSON.stringify({ status: 'dry_run', mode, gate: 'dry_run', newTokens: inline.tokens, staleChars, costUsd, floorUsd, model: embeddingModelName }));
|
||||
} else {
|
||||
console.log(previewMsg);
|
||||
console.log('--dry-run: exit without syncing.');
|
||||
}
|
||||
return;
|
||||
}
|
||||
|
||||
if (!yesFlag) {
|
||||
if (shouldBlockSync(costUsd, floorUsd, mode)) {
|
||||
const isTTY = Boolean(process.stdout.isTTY) && Boolean(process.stdin.isTTY);
|
||||
if (!isTTY || jsonOut) {
|
||||
// Agent-facing path: emit structured envelope, exit 2.
|
||||
const envelope = serializeError(errorFor({
|
||||
class: 'ConfirmationRequired',
|
||||
code: 'cost_preview_requires_yes',
|
||||
message: previewMsg,
|
||||
hint: 'Pass --yes to proceed, or --dry-run to see the preview and exit 0.',
|
||||
}));
|
||||
console.log(JSON.stringify({ error: envelope, mode, gate: 'confirmation_required', newTokens: inline.tokens, staleChars, costUsd, floorUsd, model: embeddingModelName }));
|
||||
process.exit(2);
|
||||
}
|
||||
// Interactive TTY path: prompt [y/N].
|
||||
console.log(previewMsg);
|
||||
const answer = await promptYesNo('Proceed? [y/N] ');
|
||||
if (!answer) {
|
||||
console.log('Cancelled.');
|
||||
return;
|
||||
}
|
||||
} else {
|
||||
// Below floor → proceed without blocking (kills inline-cron noise).
|
||||
if (jsonOut) {
|
||||
console.log(JSON.stringify({ status: 'below_floor', mode, gate: 'below_floor', newTokens: inline.tokens, staleChars, costUsd, floorUsd, model: embeddingModelName }));
|
||||
} else {
|
||||
console.log(`${previewMsg} Below cost gate floor ($${floorUsd.toFixed(2)}), proceeding.`);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -2302,8 +2449,7 @@ See also:
|
||||
// - withSourcePrefix wrap inside runOne so slog/serr lines from
|
||||
// performSync get the [<source-id>] prefix under parallel mode (D6)
|
||||
// - stable JSON envelope {schema_version:1, sources, ...} when --json
|
||||
const { isFederatedV2Enabled } = await import('../core/feature-flags.ts');
|
||||
const v2Enabled = await isFederatedV2Enabled(engine);
|
||||
// v0.41.31: v2Enabled resolved once above (cost gate). Reused here.
|
||||
const activeSources = sources.filter((s) => {
|
||||
const cfg = (s.config || {}) as { syncEnabled?: boolean };
|
||||
return cfg.syncEnabled !== false;
|
||||
@@ -2791,6 +2937,13 @@ export interface SyncStatusReportSource {
|
||||
chunks_total: number;
|
||||
chunks_unembedded: number;
|
||||
embedding_coverage_pct: number;
|
||||
// v0.41.31: embed-backfill job visibility (federated_v2 defers embedding
|
||||
// to these jobs; without this an operator can't see queued/lagging work
|
||||
// after `sync --all` exits 0). Best-effort — all 0 / null on brains
|
||||
// without the minion_jobs table.
|
||||
backfill_queued: number;
|
||||
backfill_active: number;
|
||||
backfill_last_completed_at: string | null;
|
||||
}
|
||||
|
||||
export interface SyncStatusReport {
|
||||
@@ -2892,6 +3045,43 @@ export async function buildSyncStatusReport(
|
||||
});
|
||||
}
|
||||
|
||||
// v0.41.31: per-source embed-backfill job state. Best-effort — the
|
||||
// minion_jobs table doesn't exist on every brain (a brain that never ran
|
||||
// a worker has the pre-minions schema), and the dashboard must not crash
|
||||
// for that. A failure → empty map → all sources report 0/null.
|
||||
type BackfillRow = {
|
||||
source_id: string | null;
|
||||
queued: string | number;
|
||||
active: string | number;
|
||||
last_completed_at: string | Date | null;
|
||||
};
|
||||
const backfillMap = new Map<string, { queued: number; active: number; last_completed_at: string | null }>();
|
||||
if (sourceIds.length > 0) {
|
||||
try {
|
||||
const backfillRows = await engine.executeRaw<BackfillRow>(
|
||||
`SELECT data->>'sourceId' AS source_id,
|
||||
COUNT(*) FILTER (WHERE status IN ('waiting','delayed','waiting-children'))::int AS queued,
|
||||
COUNT(*) FILTER (WHERE status = 'active')::int AS active,
|
||||
MAX(finished_at) FILTER (WHERE status = 'completed') AS last_completed_at
|
||||
FROM minion_jobs
|
||||
WHERE name = 'embed-backfill' AND data->>'sourceId' = ANY($1::text[])
|
||||
GROUP BY data->>'sourceId'`,
|
||||
[sourceIds],
|
||||
);
|
||||
for (const r of backfillRows) {
|
||||
if (!r.source_id) continue;
|
||||
const last = r.last_completed_at;
|
||||
backfillMap.set(r.source_id, {
|
||||
queued: Number(r.queued) || 0,
|
||||
active: Number(r.active) || 0,
|
||||
last_completed_at: last == null ? null : (last instanceof Date ? last.toISOString() : last),
|
||||
});
|
||||
}
|
||||
} catch {
|
||||
// minion_jobs absent / unreadable → leave backfillMap empty.
|
||||
}
|
||||
}
|
||||
|
||||
const now = Date.now();
|
||||
const out: SyncStatusReportSource[] = sources.map((src) => {
|
||||
const cfgEntry = (src.config || {}) as { syncEnabled?: boolean };
|
||||
@@ -2928,6 +3118,9 @@ export async function buildSyncStatusReport(
|
||||
chunks_total: counts.chunks_total,
|
||||
chunks_unembedded: counts.chunks_unembedded,
|
||||
embedding_coverage_pct: embeddingCoveragePct,
|
||||
backfill_queued: backfillMap.get(src.id)?.queued ?? 0,
|
||||
backfill_active: backfillMap.get(src.id)?.active ?? 0,
|
||||
backfill_last_completed_at: backfillMap.get(src.id)?.last_completed_at ?? null,
|
||||
};
|
||||
});
|
||||
|
||||
@@ -2968,7 +3161,7 @@ export function printSyncStatusReport(
|
||||
write(' (no sources registered)');
|
||||
return;
|
||||
}
|
||||
const headers = ['SOURCE', 'STATE', 'STALENESS', 'PAGES', 'EMBEDDED', 'LAST SYNC'];
|
||||
const headers = ['SOURCE', 'STATE', 'STALENESS', 'PAGES', 'EMBEDDED', 'BACKFILL', 'LAST SYNC'];
|
||||
const rows = report.sources.map((s) => {
|
||||
const stale = s.staleness_hours === null
|
||||
? 'never'
|
||||
@@ -2976,21 +3169,28 @@ export function printSyncStatusReport(
|
||||
const stateBits: string[] = [];
|
||||
if (!s.sync_enabled) stateBits.push('disabled');
|
||||
stateBits.push(s.staleness_class);
|
||||
// BACKFILL: active beats queued beats idle for the at-a-glance cell.
|
||||
const backfill = s.backfill_active > 0
|
||||
? `active(${s.backfill_active})`
|
||||
: s.backfill_queued > 0
|
||||
? `queued(${s.backfill_queued})`
|
||||
: 'idle';
|
||||
return [
|
||||
s.name,
|
||||
stateBits.join(','),
|
||||
stale,
|
||||
String(s.pages),
|
||||
`${s.embedding_coverage_pct}%`,
|
||||
backfill,
|
||||
s.last_sync_at ?? '(never)',
|
||||
];
|
||||
});
|
||||
const widths = headers.map((h, i) =>
|
||||
Math.max(h.length, ...rows.map((r) => r[i].length)),
|
||||
);
|
||||
// Numeric columns (PAGES at index 3, EMBEDDED at index 4, STALENESS
|
||||
// at index 2) right-pad-left so digits align cleanly. Text columns
|
||||
// left-pad-right per the existing `sources list` convention.
|
||||
// Numeric columns (STALENESS=2, PAGES=3, EMBEDDED=4) right-pad-left so
|
||||
// digits align cleanly. Text columns (incl. BACKFILL=5) left-pad-right
|
||||
// per the existing `sources list` convention.
|
||||
const NUMERIC_COLS = new Set([2, 3, 4]);
|
||||
const fmt = (cells: string[]) =>
|
||||
cells.map((c, i) => (NUMERIC_COLS.has(i) ? c.padStart(widths[i]) : c.padEnd(widths[i]))).join(' ');
|
||||
|
||||
@@ -51,6 +51,14 @@ export interface EmbedStaleOpts {
|
||||
* the gateway. Production callers leave it unset.
|
||||
*/
|
||||
embedFn?: (texts: string[], opts: { abortSignal?: AbortSignal }) => Promise<Float32Array[]>;
|
||||
/**
|
||||
* v0.41.31: current embedding provenance signature (`<provider:model>:<dims>`).
|
||||
* When set, embeddings stamped under a DIFFERENT signature are invalidated
|
||||
* (NULLed) at the start so they flow through the NULL cursor and get
|
||||
* re-embedded; each page's signature is stamped after its chunks land.
|
||||
* Omit to keep the legacy `embedding IS NULL`-only behavior.
|
||||
*/
|
||||
embeddingSignature?: string;
|
||||
}
|
||||
|
||||
export interface EmbedStaleResult {
|
||||
@@ -109,6 +117,18 @@ export async function embedStaleForSource(
|
||||
done: false,
|
||||
aborted: false,
|
||||
};
|
||||
const signature = opts.embeddingSignature;
|
||||
|
||||
// v0.41.31: invalidate embeddings stamped under a prior model signature so
|
||||
// the NULL cursor below re-embeds them. GRANDFATHER: NULL signature
|
||||
// untouched. Best-effort — a failure here must not abort the backfill.
|
||||
if (signature) {
|
||||
try {
|
||||
await engine.invalidateStaleSignatureEmbeddings({ signature, sourceId });
|
||||
} catch {
|
||||
// Non-fatal: fall through to the NULL-only stale loop.
|
||||
}
|
||||
}
|
||||
|
||||
for (;;) {
|
||||
if (signal?.aborted) {
|
||||
@@ -170,6 +190,13 @@ export async function embedStaleForSource(
|
||||
token_count: c.token_count || Math.ceil(c.chunk_text.length / 4),
|
||||
}));
|
||||
await engine.upsertChunks(slug, merged, { sourceId: keySourceId });
|
||||
// v0.41.31: stamp provenance only when EVERY chunk was stale (fully
|
||||
// re-embedded this pass) — a partially-stale page keeps preserved
|
||||
// chunks of unknown provenance, so don't claim current. After the
|
||||
// invalidate pass above, signature-drifted pages ARE fully stale.
|
||||
if (signature && stale.length === existing.length) {
|
||||
await engine.setPageEmbeddingSignature(slug, { sourceId: keySourceId, signature });
|
||||
}
|
||||
result.embedded += stale.length;
|
||||
result.pagesProcessed += 1;
|
||||
} catch (e: unknown) {
|
||||
|
||||
+99
-5
@@ -12,6 +12,7 @@ import {
|
||||
getEmbeddingModel as gatewayGetModel,
|
||||
getEmbeddingDimensions as gatewayGetDims,
|
||||
} from './ai/gateway.ts';
|
||||
import { lookupEmbeddingPrice } from './embedding-pricing.ts';
|
||||
|
||||
// v0.27.1: re-export multimodal embedding so callers can pull both text and
|
||||
// image embedding APIs from `src/core/embedding`. import-image-file consumes
|
||||
@@ -127,13 +128,106 @@ export const EMBEDDING_MODEL = 'text-embedding-3-large';
|
||||
export const EMBEDDING_DIMENSIONS = 1536;
|
||||
|
||||
/**
|
||||
* USD cost per 1k tokens for text-embedding-3-large. Used by
|
||||
* `gbrain sync --all` cost preview and `reindex-code` to surface
|
||||
* expected spend before accepting expensive operations.
|
||||
* USD cost per 1k tokens for text-embedding-3-large. Retained for back-compat
|
||||
* with callers/tests that import it directly; new cost math resolves the
|
||||
* ACTUAL configured model's rate via embedding-pricing.ts instead of assuming
|
||||
* OpenAI. (Hardcoding this rate produced cost previews that named the wrong
|
||||
* provider and over-stated spend ~2.6x when the brain ran on a cheaper model.)
|
||||
*/
|
||||
export const EMBEDDING_COST_PER_1K_TOKENS = 0.00013;
|
||||
|
||||
/** Compute USD cost estimate for embedding `tokens` at current model rate. */
|
||||
/**
|
||||
* Resolve the price-per-1M-tokens for the currently-configured embedding
|
||||
* model. Falls back to the OpenAI text-embedding-3-large rate only when the
|
||||
* model is unknown to the pricing table.
|
||||
*/
|
||||
export function currentEmbeddingPricePerMTok(): number {
|
||||
let modelString: string;
|
||||
try {
|
||||
modelString = gatewayGetModel(); // e.g. 'zeroentropyai:zembed-1'
|
||||
} catch {
|
||||
// Gateway not configured (e.g. unit tests, cost preview before connect).
|
||||
// Fall back to the OpenAI text-embedding-3-large default rate.
|
||||
return 0.13;
|
||||
}
|
||||
const hit = lookupEmbeddingPrice(modelString);
|
||||
return hit.kind === 'known' ? hit.pricePerMTok : 0.13;
|
||||
}
|
||||
|
||||
/**
|
||||
* Compute USD cost estimate for embedding `tokens` at the CURRENT configured
|
||||
* model's rate (not a hardcoded OpenAI rate).
|
||||
*/
|
||||
export function estimateEmbeddingCostUsd(tokens: number): number {
|
||||
return (tokens / 1000) * EMBEDDING_COST_PER_1K_TOKENS;
|
||||
return (tokens / 1_000_000) * currentEmbeddingPricePerMTok();
|
||||
}
|
||||
|
||||
/**
|
||||
* Embedding provenance signature for the currently-configured model:
|
||||
* `<provider:model>:<dims>` (e.g. `openai:text-embedding-3-large:1536`).
|
||||
* Stamped onto `pages.embedding_signature` when a page's chunks are
|
||||
* embedded so a later model/dimension swap can be detected as stale.
|
||||
*
|
||||
* Deliberately does NOT include the chunker version — chunker drift is
|
||||
* already tracked per-page via `pages.chunker_version` (used by sync +
|
||||
* doctor). This signature is strictly about the EMBEDDING space.
|
||||
*
|
||||
* Falls back to the OpenAI default signature when the gateway is
|
||||
* unconfigured (unit-test context), matching the other estimator fallbacks.
|
||||
*/
|
||||
export function currentEmbeddingSignature(): string {
|
||||
try {
|
||||
return `${gatewayGetModel()}:${gatewayGetDims()}`;
|
||||
} catch {
|
||||
return `${EMBEDDING_MODEL}:${EMBEDDING_DIMENSIONS}`;
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Whether a `gbrain sync --all` invocation will embed at sync time
|
||||
* ('inline') or defer embedding to per-source `embed-backfill` minion jobs
|
||||
* ('deferred'). Under federated_v2 the default path defers; the backfill
|
||||
* jobs carry their own 10-min cooldown + $25/source/24h spend cap, so the
|
||||
* sync-time cost gate only BLOCKS on the inline path. See sync.ts:2346
|
||||
* (`effectiveNoEmbed`) — this mirrors that resolution exactly.
|
||||
*/
|
||||
export type SyncEmbedMode = 'deferred' | 'inline';
|
||||
|
||||
/**
|
||||
* Resolve the embed mode from the same three signals sync.ts uses to
|
||||
* compute `effectiveNoEmbed`. Single source of truth so the cost gate and
|
||||
* the actual embed decision can never drift.
|
||||
*
|
||||
* effectiveNoEmbed = v2Enabled && !serialFlag && !noEmbed ? true : noEmbed
|
||||
*
|
||||
* Embed runs INLINE iff that resolves to false:
|
||||
* - v2 off → inline (legacy synchronous embed)
|
||||
* - v2 on + --serial + !--no-embed → inline
|
||||
* - v2 on (parallel) → deferred (backfill jobs)
|
||||
* - --no-embed (any path) → the caller skips the gate entirely;
|
||||
* we report 'deferred' for completeness.
|
||||
*/
|
||||
export function willEmbedSynchronously(opts: {
|
||||
v2Enabled: boolean;
|
||||
serialFlag: boolean;
|
||||
noEmbed: boolean;
|
||||
}): SyncEmbedMode {
|
||||
const effectiveNoEmbed =
|
||||
opts.v2Enabled && !opts.serialFlag && !opts.noEmbed ? true : opts.noEmbed;
|
||||
return effectiveNoEmbed ? 'deferred' : 'inline';
|
||||
}
|
||||
|
||||
/**
|
||||
* Pure cost-gate decision. The gate BLOCKS (prompt in TTY, exit 2 envelope
|
||||
* in non-TTY) only when embed runs inline AND the estimated spend exceeds
|
||||
* the floor. Deferred mode NEVER blocks — the backfill cap is the real
|
||||
* money gate, and blocking the cheap markdown import for cost the import
|
||||
* doesn't synchronously incur is the bug this fix removes.
|
||||
*/
|
||||
export function shouldBlockSync(
|
||||
costUsd: number,
|
||||
floorUsd: number,
|
||||
mode: SyncEmbedMode,
|
||||
): boolean {
|
||||
return mode === 'inline' && costUsd > floorUsd;
|
||||
}
|
||||
|
||||
+32
-1
@@ -956,7 +956,38 @@ export interface BrainEngine {
|
||||
* `gbrain embed --stale --source media-corpus` expect only that
|
||||
* source's NULLs touched; the caller threads `sourceId` here.
|
||||
*/
|
||||
countStaleChunks(opts?: { sourceId?: string }): Promise<number>;
|
||||
countStaleChunks(opts?: { sourceId?: string; signature?: string }): Promise<number>;
|
||||
/**
|
||||
* Sum of LENGTH(chunk_text) over stale chunks — the character-count
|
||||
* backlog the embed phase / embed-backfill will process. Sibling of
|
||||
* countStaleChunks (same stale predicate + embed_skip filter + optional
|
||||
* sourceId scope); used by the `gbrain sync --all` cost preview to price
|
||||
* the embedding backlog via estimateCostFromChars. Returns 0 on an
|
||||
* empty/fully-embedded brain.
|
||||
*
|
||||
* v0.41.31: `signature` (optional) widens "stale" to ALSO include chunks
|
||||
* whose page `embedding_signature` is set AND differs from the current
|
||||
* model signature (a model/dims swap). NULL signature is GRANDFATHERED
|
||||
* (never counted) so the post-migration corpus isn't flagged en masse.
|
||||
* Omit `signature` for the legacy `embedding IS NULL`-only count.
|
||||
*/
|
||||
sumStaleChunkChars(opts?: { sourceId?: string; signature?: string }): Promise<number>;
|
||||
/**
|
||||
* Stamp `pages.embedding_signature = signature` for one page. Called after
|
||||
* a page's chunks are (re)embedded so a later model swap can detect it as
|
||||
* stale. Idempotent. No-op if the page doesn't exist.
|
||||
*/
|
||||
setPageEmbeddingSignature(slug: string, opts: { sourceId?: string; signature: string }): Promise<void>;
|
||||
/**
|
||||
* NULL out the embeddings (and embedded_at) of every chunk whose page
|
||||
* `embedding_signature` is set AND differs from `signature` — i.e. pages
|
||||
* embedded under a now-stale model. Returns the chunk count invalidated.
|
||||
* The embed-stale loop calls this BEFORE listStaleChunks so signature-
|
||||
* drift pages flow through the existing NULL-embedding cursor (keeps
|
||||
* listStaleChunks's keyset pagination untouched). GRANDFATHER: NULL
|
||||
* signature is never invalidated. `sourceId` scopes the sweep.
|
||||
*/
|
||||
invalidateStaleSignatureEmbeddings(opts: { signature: string; sourceId?: string }): Promise<number>;
|
||||
/**
|
||||
* Return every chunk where embedding IS NULL, with the metadata needed
|
||||
* to call embedBatch + upsertChunks. The `embedding` column is omitted
|
||||
|
||||
+16
-1
@@ -8,7 +8,7 @@ import { chunkText } from './chunkers/recursive.ts';
|
||||
import { chunkCodeText, chunkCodeTextFull, detectCodeLanguage, CHUNKER_VERSION } from './chunkers/code.ts';
|
||||
import { findChunkForOffset } from './chunkers/edge-extractor.ts';
|
||||
import { extractCodeRefs, imageOfCandidates } from './link-extraction.ts';
|
||||
import { embedBatch, embedMultimodal } from './embedding.ts';
|
||||
import { embedBatch, embedMultimodal, currentEmbeddingSignature } from './embedding.ts';
|
||||
import { slugifyPath, slugifyCodePath, isCodeFilePath } from './sync.ts';
|
||||
import type { ChunkInput, PageInput, PageType } from './types.ts';
|
||||
import { computeEffectiveDate } from './effective-date.ts';
|
||||
@@ -658,6 +658,13 @@ export async function importFromContent(
|
||||
|
||||
if (chunks.length > 0) {
|
||||
await tx.upsertChunks(slug, chunks, txOpts);
|
||||
// v0.41.31: stamp embedding provenance when this import actually
|
||||
// embedded (not --no-embed), so a later model/dims swap is detectable
|
||||
// as stale via embed --stale. The deferred/backfill + per-slug embed
|
||||
// paths stamp too; this covers the inline import/sync path.
|
||||
if (!opts.noEmbed) {
|
||||
await tx.setPageEmbeddingSignature(slug, { sourceId, signature: currentEmbeddingSignature() });
|
||||
}
|
||||
} else {
|
||||
// Content is empty — delete stale chunks so they don't ghost in search results
|
||||
await tx.deleteChunks(slug, txOpts);
|
||||
@@ -968,6 +975,14 @@ export async function importCodeFile(
|
||||
|
||||
if (chunks.length > 0) {
|
||||
await tx.upsertChunks(slug, chunks, txOpts);
|
||||
// v0.41.31: stamp embedding provenance ONLY when every chunk was
|
||||
// freshly embedded with the current model this call (no reuse-by-hash
|
||||
// carrying old-model vectors). Mixed pages stay unstamped rather than
|
||||
// falsely marked current; `reindex --code --force` / `embed --stale`
|
||||
// handle the swap for those.
|
||||
if (!opts.noEmbed && needsEmbedIndexes.length === chunks.length) {
|
||||
await tx.setPageEmbeddingSignature(slug, { sourceId, signature: currentEmbeddingSignature() });
|
||||
}
|
||||
} else {
|
||||
await tx.deleteChunks(slug, txOpts);
|
||||
}
|
||||
|
||||
@@ -4920,6 +4920,32 @@ export const MIGRATIONS: Migration[] = [
|
||||
EXECUTE FUNCTION bump_page_generation_clock_fn();
|
||||
`,
|
||||
},
|
||||
{
|
||||
version: 108,
|
||||
name: 'pages_embedding_signature',
|
||||
// v0.41.31 — embedding provenance for real stale semantics.
|
||||
//
|
||||
// Adds `pages.embedding_signature TEXT NULL` = `<provider:model>:<dims>`
|
||||
// stamped when a page's chunks are embedded (setPageEmbeddingSignature).
|
||||
// A later model/dimension swap makes the stored signature differ from
|
||||
// the current one, so countStaleChunks/sumStaleChunkChars (with the
|
||||
// `signature` opt) and invalidateStaleSignatureEmbeddings can detect and
|
||||
// re-embed those pages.
|
||||
//
|
||||
// GRANDFATHER (critical): the stale predicate is
|
||||
// `embedding_signature IS NOT NULL AND embedding_signature <> $current`
|
||||
// so a NULL signature is NEVER stale. After this migration every existing
|
||||
// page has NULL — none are flagged — so the next `embed --stale` does NOT
|
||||
// re-embed the whole corpus. Signatures only get stamped going forward.
|
||||
//
|
||||
// No index: the column is read only via a JOINed pages row in the
|
||||
// chunk-grain stale queries; no standalone lookup hot path. ADD COLUMN
|
||||
// with no DEFAULT (NULL) is metadata-only on Postgres 11+ / PGLite 17.5.
|
||||
idempotent: true,
|
||||
sql: `
|
||||
ALTER TABLE pages ADD COLUMN IF NOT EXISTS embedding_signature TEXT NULL;
|
||||
`,
|
||||
},
|
||||
];
|
||||
|
||||
export const LATEST_VERSION = MIGRATIONS.length > 0
|
||||
|
||||
@@ -35,6 +35,7 @@ import { tryAcquireDbLock } from '../../db-lock.ts';
|
||||
import { BudgetTracker, BudgetExhausted } from '../../budget/budget-tracker.ts';
|
||||
import { withBudgetTracker } from '../../ai/gateway.ts';
|
||||
import { embedStaleForSource } from '../../embed-stale.ts';
|
||||
import { currentEmbeddingSignature } from '../../embedding.ts';
|
||||
import type { BrainEngine } from '../../engine.ts';
|
||||
import type { MinionJobContext } from '../types.ts';
|
||||
|
||||
@@ -123,6 +124,9 @@ export function makeEmbedBackfillHandler(engine: BrainEngine) {
|
||||
embedStaleForSource(engine, sourceId, {
|
||||
batchSize,
|
||||
signal: job.signal,
|
||||
// v0.41.31: re-embed pages whose model signature drifted + stamp
|
||||
// provenance as chunks land.
|
||||
embeddingSignature: currentEmbeddingSignature(),
|
||||
onProgress: ({ embedded, chunksProcessed, cursor }) => {
|
||||
// Fire-and-forget; updateProgress returns a Promise but the
|
||||
// handler is sync inside the loop.
|
||||
|
||||
+94
-21
@@ -422,7 +422,9 @@ export class PGLiteEngine implements BrainEngine {
|
||||
EXISTS (SELECT 1 FROM information_schema.columns
|
||||
WHERE table_schema='public' AND table_name='sources' AND column_name='trust_frontmatter_overrides') AS sources_trust_fm_exists,
|
||||
EXISTS (SELECT 1 FROM information_schema.columns
|
||||
WHERE table_schema='public' AND table_name='pages' AND column_name='generation') AS pages_generation_exists
|
||||
WHERE table_schema='public' AND table_name='pages' AND column_name='generation') AS pages_generation_exists,
|
||||
EXISTS (SELECT 1 FROM information_schema.columns
|
||||
WHERE table_schema='public' AND table_name='pages' AND column_name='embedding_signature') AS pages_embedding_signature_exists
|
||||
`);
|
||||
const probe = rows[0] as {
|
||||
pages_exists: boolean;
|
||||
@@ -463,6 +465,7 @@ export class PGLiteEngine implements BrainEngine {
|
||||
sources_cr_mode_exists: boolean;
|
||||
sources_trust_fm_exists: boolean;
|
||||
pages_generation_exists: boolean;
|
||||
pages_embedding_signature_exists: boolean;
|
||||
};
|
||||
|
||||
const needsPagesBootstrap = probe.pages_exists && !probe.source_id_exists;
|
||||
@@ -530,6 +533,10 @@ export class PGLiteEngine implements BrainEngine {
|
||||
// body; bootstrap only needs to add the column on pre-v91 brains so
|
||||
// the CREATE INDEX doesn't crash.
|
||||
const needsPagesGeneration = probe.pages_exists && !probe.pages_generation_exists;
|
||||
// v0.41.31 (v108): pages.embedding_signature for real stale semantics.
|
||||
// No SCHEMA_SQL index references it today; bootstrap is defense-in-depth
|
||||
// so future schema work doesn't wedge pre-v108 brains.
|
||||
const needsPagesEmbeddingSignature = probe.pages_exists && !probe.pages_embedding_signature_exists;
|
||||
|
||||
// Fresh installs (no tables yet) and modern brains both no-op.
|
||||
if (!needsPagesBootstrap && !needsLinksBootstrap && !needsChunksBootstrap
|
||||
@@ -539,7 +546,8 @@ export class PGLiteEngine implements BrainEngine {
|
||||
&& !needsFilesBootstrap && !needsOauthClientsBootstrap
|
||||
&& !needsSourcesArchive && !needsPagesLastRetrievedAt
|
||||
&& !needsPagesProvenance
|
||||
&& !needsContextualRetrievalColumns && !needsPagesGeneration) return;
|
||||
&& !needsContextualRetrievalColumns && !needsPagesGeneration
|
||||
&& !needsPagesEmbeddingSignature) return;
|
||||
|
||||
process.stderr.write(' Pre-v0.21 brain detected, applying forward-reference bootstrap\n');
|
||||
|
||||
@@ -767,6 +775,15 @@ export class PGLiteEngine implements BrainEngine {
|
||||
ALTER TABLE pages ADD COLUMN IF NOT EXISTS generation BIGINT NOT NULL DEFAULT 1;
|
||||
`);
|
||||
}
|
||||
|
||||
if (needsPagesEmbeddingSignature) {
|
||||
// v108 (pages_embedding_signature): embedding provenance for real
|
||||
// stale semantics. NULL grandfathered (never stale). v108 runs later
|
||||
// via runMigrations and is idempotent.
|
||||
await this.db.exec(`
|
||||
ALTER TABLE pages ADD COLUMN IF NOT EXISTS embedding_signature TEXT;
|
||||
`);
|
||||
}
|
||||
}
|
||||
|
||||
async withReservedConnection<T>(fn: (conn: ReservedConnection) => Promise<T>): Promise<T> {
|
||||
@@ -2061,35 +2078,91 @@ export class PGLiteEngine implements BrainEngine {
|
||||
return (rows as Record<string, unknown>[]).map(r => rowToChunk(r));
|
||||
}
|
||||
|
||||
async countStaleChunks(opts?: { sourceId?: string }): Promise<number> {
|
||||
// D7: source-scoped count for `gbrain embed --stale --source X`.
|
||||
// v0.41 (D4+D8+Codex r2 #11): always JOIN pages so embed-skip filter
|
||||
// applies via `NOT (frontmatter ? 'embed_skip')`. PGLite is
|
||||
// PostgreSQL 17.5 in WASM and supports the full JSONB operator set.
|
||||
if (opts?.sourceId === undefined) {
|
||||
const { rows } = await this.db.query(
|
||||
`SELECT count(*)::int AS count
|
||||
FROM content_chunks cc
|
||||
JOIN pages p ON p.id = cc.page_id
|
||||
WHERE cc.embedding IS NULL
|
||||
AND NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'embed_skip')`,
|
||||
);
|
||||
const count = (rows[0] as { count: number } | undefined)?.count ?? 0;
|
||||
return Number(count);
|
||||
/**
|
||||
* Build the stale-chunk WHERE clause + positional params. embed_skip is
|
||||
* always excluded. `signature` widens "stale" to include embedding_signature
|
||||
* drift (NULL grandfathered → never stale). Shared by countStaleChunks +
|
||||
* sumStaleChunkChars so they can't drift.
|
||||
*/
|
||||
private buildStaleChunkWhere(opts?: { sourceId?: string; signature?: string }): { where: string; params: unknown[] } {
|
||||
const params: unknown[] = [];
|
||||
const conds: string[] = [];
|
||||
if (opts?.signature !== undefined) {
|
||||
params.push(opts.signature);
|
||||
conds.push(`(cc.embedding IS NULL OR (p.embedding_signature IS NOT NULL AND p.embedding_signature <> $${params.length}))`);
|
||||
} else {
|
||||
conds.push(`cc.embedding IS NULL`);
|
||||
}
|
||||
conds.push(`NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'embed_skip')`);
|
||||
if (opts?.sourceId !== undefined) {
|
||||
params.push(opts.sourceId);
|
||||
conds.push(`p.source_id = $${params.length}`);
|
||||
}
|
||||
return { where: conds.join(' AND '), params };
|
||||
}
|
||||
|
||||
async countStaleChunks(opts?: { sourceId?: string; signature?: string }): Promise<number> {
|
||||
// D7: source-scoped count for `gbrain embed --stale --source X`. Always
|
||||
// JOIN pages so embed-skip + signature predicates apply. PGLite is
|
||||
// PostgreSQL 17.5 in WASM and supports the full JSONB operator set.
|
||||
const { where, params } = this.buildStaleChunkWhere(opts);
|
||||
const { rows } = await this.db.query(
|
||||
`SELECT count(*)::int AS count
|
||||
FROM content_chunks cc
|
||||
JOIN pages p ON p.id = cc.page_id
|
||||
WHERE cc.embedding IS NULL
|
||||
AND p.source_id = $1
|
||||
AND NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'embed_skip')`,
|
||||
[opts.sourceId],
|
||||
WHERE ${where}`,
|
||||
params,
|
||||
);
|
||||
const count = (rows[0] as { count: number } | undefined)?.count ?? 0;
|
||||
return Number(count);
|
||||
}
|
||||
|
||||
async sumStaleChunkChars(opts?: { sourceId?: string; signature?: string }): Promise<number> {
|
||||
// Sibling of countStaleChunks: same stale predicate, summing chunk_text
|
||||
// length for the sync cost preview. ::bigint guards int4 overflow.
|
||||
const { where, params } = this.buildStaleChunkWhere(opts);
|
||||
const { rows } = await this.db.query(
|
||||
`SELECT COALESCE(SUM(LENGTH(cc.chunk_text)), 0)::bigint AS chars
|
||||
FROM content_chunks cc
|
||||
JOIN pages p ON p.id = cc.page_id
|
||||
WHERE ${where}`,
|
||||
params,
|
||||
);
|
||||
const chars = (rows[0] as { chars: number | string } | undefined)?.chars ?? 0;
|
||||
return Number(chars);
|
||||
}
|
||||
|
||||
async setPageEmbeddingSignature(slug: string, opts: { sourceId?: string; signature: string }): Promise<void> {
|
||||
await this.db.query(
|
||||
`UPDATE pages SET embedding_signature = $1 WHERE slug = $2 AND source_id = $3`,
|
||||
[opts.signature, slug, opts.sourceId ?? 'default'],
|
||||
);
|
||||
}
|
||||
|
||||
async invalidateStaleSignatureEmbeddings(opts: { signature: string; sourceId?: string }): Promise<number> {
|
||||
// NULL out embeddings whose page signature is set AND differs from the
|
||||
// current model signature. GRANDFATHER: NULL signature untouched. Feeds
|
||||
// the existing NULL-embedding cursor so listStaleChunks stays unchanged.
|
||||
const params: unknown[] = [opts.signature];
|
||||
let srcClause = '';
|
||||
if (opts.sourceId !== undefined) {
|
||||
params.push(opts.sourceId);
|
||||
srcClause = ` AND p.source_id = $${params.length}`;
|
||||
}
|
||||
const { rows } = await this.db.query(
|
||||
`UPDATE content_chunks cc
|
||||
SET embedding = NULL, embedded_at = NULL
|
||||
FROM pages p
|
||||
WHERE cc.page_id = p.id
|
||||
AND cc.embedding IS NOT NULL
|
||||
AND p.embedding_signature IS NOT NULL
|
||||
AND p.embedding_signature <> $1${srcClause}
|
||||
RETURNING cc.page_id`,
|
||||
params,
|
||||
);
|
||||
return (rows as unknown[]).length;
|
||||
}
|
||||
|
||||
async listStaleChunks(opts?: {
|
||||
batchSize?: number;
|
||||
afterPageId?: number;
|
||||
|
||||
+98
-30
@@ -455,7 +455,9 @@ export class PostgresEngine implements BrainEngine {
|
||||
EXISTS (SELECT 1 FROM information_schema.columns
|
||||
WHERE table_schema = current_schema() AND table_name = 'sources' AND column_name = 'trust_frontmatter_overrides') AS sources_trust_fm_exists,
|
||||
EXISTS (SELECT 1 FROM information_schema.columns
|
||||
WHERE table_schema = current_schema() AND table_name = 'pages' AND column_name = 'generation') AS pages_generation_exists
|
||||
WHERE table_schema = current_schema() AND table_name = 'pages' AND column_name = 'generation') AS pages_generation_exists,
|
||||
EXISTS (SELECT 1 FROM information_schema.columns
|
||||
WHERE table_schema = current_schema() AND table_name = 'pages' AND column_name = 'embedding_signature') AS pages_embedding_signature_exists
|
||||
`;
|
||||
const probe = probeRows[0]!;
|
||||
|
||||
@@ -530,6 +532,7 @@ export class PostgresEngine implements BrainEngine {
|
||||
sources_cr_mode_exists?: boolean;
|
||||
sources_trust_fm_exists?: boolean;
|
||||
pages_generation_exists?: boolean;
|
||||
pages_embedding_signature_exists?: boolean;
|
||||
};
|
||||
const needsContextualRetrievalColumns = (probe.pages_exists
|
||||
&& (!probeCr.pages_cr_mode_exists || !probeCr.pages_corpus_generation_exists))
|
||||
@@ -540,6 +543,9 @@ export class PostgresEngine implements BrainEngine {
|
||||
// it. Pre-v91 brains crash without the column; bootstrap adds it before
|
||||
// SCHEMA_SQL replay creates the index.
|
||||
const needsPagesGeneration = probe.pages_exists && !probeCr.pages_generation_exists;
|
||||
// v0.41.31 (v108): pages.embedding_signature for real stale semantics.
|
||||
// No SCHEMA_SQL index references it; bootstrap is defense-in-depth.
|
||||
const needsPagesEmbeddingSignature = probe.pages_exists && !probeCr.pages_embedding_signature_exists;
|
||||
|
||||
if (!needsPagesBootstrap && !needsLinksBootstrap && !needsChunksBootstrap
|
||||
&& !needsPagesDeletedAt && !needsMcpLogBootstrap && !needsSubagentProviderId
|
||||
@@ -548,7 +554,8 @@ export class PostgresEngine implements BrainEngine {
|
||||
&& !needsOauthClientsBootstrap && !needsSourcesArchive
|
||||
&& !needsPagesLastRetrievedAt
|
||||
&& !needsPagesProvenance
|
||||
&& !needsContextualRetrievalColumns && !needsPagesGeneration) return;
|
||||
&& !needsContextualRetrievalColumns && !needsPagesGeneration
|
||||
&& !needsPagesEmbeddingSignature) return;
|
||||
|
||||
process.stderr.write(' Pre-v0.21 brain detected, applying forward-reference bootstrap\n');
|
||||
|
||||
@@ -774,6 +781,15 @@ export class PostgresEngine implements BrainEngine {
|
||||
ALTER TABLE pages ADD COLUMN IF NOT EXISTS generation BIGINT NOT NULL DEFAULT 1;
|
||||
`);
|
||||
}
|
||||
|
||||
if (needsPagesEmbeddingSignature) {
|
||||
// v108 (pages_embedding_signature): embedding provenance for real stale
|
||||
// semantics. NULL grandfathered. v108 runs later via runMigrations and
|
||||
// is idempotent.
|
||||
await conn.unsafe(`
|
||||
ALTER TABLE pages ADD COLUMN IF NOT EXISTS embedding_signature TEXT;
|
||||
`);
|
||||
}
|
||||
}
|
||||
|
||||
async transaction<T>(fn: (engine: BrainEngine) => Promise<T>): Promise<T> {
|
||||
@@ -2093,36 +2109,88 @@ export class PostgresEngine implements BrainEngine {
|
||||
return rows.map((r) => rowToChunk(r as Record<string, unknown>));
|
||||
}
|
||||
|
||||
async countStaleChunks(opts?: { sourceId?: string }): Promise<number> {
|
||||
const sql = this.sql;
|
||||
// v0.41 (D4+D8+Codex r2 #11): the embed-skip filter requires JOIN
|
||||
// pages so we always join — the pre-v0.41 "fast path" without join
|
||||
// is gone. JSONB `?` existence check is cheap on the small set of
|
||||
// skipped pages; full-scan benefits from the partial index on
|
||||
// embedding IS NULL regardless.
|
||||
//
|
||||
// D7: source_id scoping. NULL/undefined = scan all sources;
|
||||
// a value scopes to that source so `gbrain embed --stale --source X`
|
||||
// does what it says.
|
||||
if (opts?.sourceId === undefined) {
|
||||
const [row] = await sql`
|
||||
SELECT count(*)::int AS count
|
||||
FROM content_chunks cc
|
||||
JOIN pages p ON p.id = cc.page_id
|
||||
WHERE cc.embedding IS NULL
|
||||
AND NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'embed_skip')
|
||||
`;
|
||||
return Number((row as { count?: number } | undefined)?.count ?? 0);
|
||||
/**
|
||||
* Build the stale-chunk WHERE clause + positional params for sql.unsafe.
|
||||
* embed_skip always excluded. `signature` widens "stale" to include
|
||||
* embedding_signature drift (NULL grandfathered). Shared by
|
||||
* countStaleChunks + sumStaleChunkChars (parity with the PGLite sibling).
|
||||
*/
|
||||
private buildStaleChunkWhere(opts?: { sourceId?: string; signature?: string }): { where: string; params: unknown[] } {
|
||||
const params: unknown[] = [];
|
||||
const conds: string[] = [];
|
||||
if (opts?.signature !== undefined) {
|
||||
params.push(opts.signature);
|
||||
conds.push(`(cc.embedding IS NULL OR (p.embedding_signature IS NOT NULL AND p.embedding_signature <> $${params.length}))`);
|
||||
} else {
|
||||
conds.push(`cc.embedding IS NULL`);
|
||||
}
|
||||
const [row] = await sql`
|
||||
SELECT count(*)::int AS count
|
||||
FROM content_chunks cc
|
||||
JOIN pages p ON p.id = cc.page_id
|
||||
WHERE cc.embedding IS NULL
|
||||
AND p.source_id = ${opts.sourceId}
|
||||
AND NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'embed_skip')
|
||||
conds.push(`NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'embed_skip')`);
|
||||
if (opts?.sourceId !== undefined) {
|
||||
params.push(opts.sourceId);
|
||||
conds.push(`p.source_id = $${params.length}`);
|
||||
}
|
||||
return { where: conds.join(' AND '), params };
|
||||
}
|
||||
|
||||
async countStaleChunks(opts?: { sourceId?: string; signature?: string }): Promise<number> {
|
||||
// Always JOIN pages so the embed_skip + signature predicates apply.
|
||||
// D7: source_id scoping. v0.41.31: optional signature widens staleness
|
||||
// to embedding_signature drift (NULL grandfathered).
|
||||
const { where, params } = this.buildStaleChunkWhere(opts);
|
||||
const rows = await this.sql.unsafe(
|
||||
`SELECT count(*)::int AS count
|
||||
FROM content_chunks cc
|
||||
JOIN pages p ON p.id = cc.page_id
|
||||
WHERE ${where}`,
|
||||
params as Parameters<typeof this.sql.unsafe>[1],
|
||||
);
|
||||
return Number((rows[0] as { count?: number } | undefined)?.count ?? 0);
|
||||
}
|
||||
|
||||
async sumStaleChunkChars(opts?: { sourceId?: string; signature?: string }): Promise<number> {
|
||||
// Sibling of countStaleChunks: same stale predicate, summing chunk_text
|
||||
// length for the sync cost preview. ::bigint guards int4 overflow.
|
||||
const { where, params } = this.buildStaleChunkWhere(opts);
|
||||
const rows = await this.sql.unsafe(
|
||||
`SELECT COALESCE(SUM(LENGTH(cc.chunk_text)), 0)::bigint AS chars
|
||||
FROM content_chunks cc
|
||||
JOIN pages p ON p.id = cc.page_id
|
||||
WHERE ${where}`,
|
||||
params as Parameters<typeof this.sql.unsafe>[1],
|
||||
);
|
||||
return Number((rows[0] as { chars?: number | string } | undefined)?.chars ?? 0);
|
||||
}
|
||||
|
||||
async setPageEmbeddingSignature(slug: string, opts: { sourceId?: string; signature: string }): Promise<void> {
|
||||
const sql = this.sql;
|
||||
await sql`
|
||||
UPDATE pages SET embedding_signature = ${opts.signature}
|
||||
WHERE slug = ${slug} AND source_id = ${opts.sourceId ?? 'default'}
|
||||
`;
|
||||
return Number((row as { count?: number } | undefined)?.count ?? 0);
|
||||
}
|
||||
|
||||
async invalidateStaleSignatureEmbeddings(opts: { signature: string; sourceId?: string }): Promise<number> {
|
||||
// NULL embeddings whose page signature is set AND differs from current.
|
||||
// GRANDFATHER: NULL signature untouched. Feeds the NULL-embedding cursor
|
||||
// so listStaleChunks stays unchanged. RETURNING → row count.
|
||||
const params: unknown[] = [opts.signature];
|
||||
let srcClause = '';
|
||||
if (opts.sourceId !== undefined) {
|
||||
params.push(opts.sourceId);
|
||||
srcClause = ` AND p.source_id = $${params.length}`;
|
||||
}
|
||||
const rows = await this.sql.unsafe(
|
||||
`UPDATE content_chunks cc
|
||||
SET embedding = NULL, embedded_at = NULL
|
||||
FROM pages p
|
||||
WHERE cc.page_id = p.id
|
||||
AND cc.embedding IS NOT NULL
|
||||
AND p.embedding_signature IS NOT NULL
|
||||
AND p.embedding_signature <> $1${srcClause}
|
||||
RETURNING cc.page_id`,
|
||||
params as Parameters<typeof this.sql.unsafe>[1],
|
||||
);
|
||||
return (rows as unknown[]).length;
|
||||
}
|
||||
|
||||
async listStaleChunks(opts?: {
|
||||
|
||||
@@ -33,6 +33,10 @@ export interface SourceMetrics {
|
||||
failed_jobs_24h: number;
|
||||
/** Waiting + active + delayed jobs (sync OR embed-backfill) for this source. */
|
||||
queue_depth: number;
|
||||
/** v0.41.31: embed-backfill jobs specifically active right now. */
|
||||
backfill_active: number;
|
||||
/** v0.41.31: embed-backfill jobs queued (waiting/delayed/waiting-children). */
|
||||
backfill_queued: number;
|
||||
tracked_branch: string | null;
|
||||
priority_label: PriorityLabel;
|
||||
/** Webhook configured? (true iff config.webhook_secret is set.) */
|
||||
@@ -131,7 +135,7 @@ export async function computeAllSourceMetrics(
|
||||
const cfg = parseSourceConfig(src.config);
|
||||
const pages = pageCounts.get(src.id) ?? 0;
|
||||
const chunkStats = chunkCounts.get(src.id) ?? { total: 0, embedded: 0 };
|
||||
const jobStats = jobCounts.get(src.id) ?? { failed_24h: 0, queue_depth: 0 };
|
||||
const jobStats = jobCounts.get(src.id) ?? { failed_24h: 0, queue_depth: 0, backfill_active: 0, backfill_queued: 0 };
|
||||
|
||||
const embedCoverage = chunkStats.total === 0
|
||||
? 100
|
||||
@@ -155,6 +159,8 @@ export async function computeAllSourceMetrics(
|
||||
lag_seconds: lagSeconds,
|
||||
failed_jobs_24h: jobStats.failed_24h,
|
||||
queue_depth: jobStats.queue_depth,
|
||||
backfill_active: jobStats.backfill_active,
|
||||
backfill_queued: jobStats.backfill_queued,
|
||||
tracked_branch: typeof cfg.tracked_branch === 'string' ? cfg.tracked_branch : null,
|
||||
priority_label: resolvePriorityLabel(src.id, src.config),
|
||||
webhook_configured: typeof cfg.webhook_secret === 'string' && cfg.webhook_secret.length > 0,
|
||||
@@ -189,21 +195,30 @@ async function chunkCountsBySource(engine: BrainEngine): Promise<Map<string, { t
|
||||
return m;
|
||||
}
|
||||
|
||||
async function jobCountsBySource(engine: BrainEngine): Promise<Map<string, { failed_24h: number; queue_depth: number }>> {
|
||||
type JobStats = { failed_24h: number; queue_depth: number; backfill_active: number; backfill_queued: number };
|
||||
|
||||
async function jobCountsBySource(engine: BrainEngine): Promise<Map<string, JobStats>> {
|
||||
// Pre-v0.11 brains don't have minion_jobs; return empty map.
|
||||
try {
|
||||
const rows = await engine.executeRaw<{ source_id: string; failed_24h: number; queue_depth: number }>(
|
||||
const rows = await engine.executeRaw<{ source_id: string; failed_24h: number; queue_depth: number; backfill_active: number; backfill_queued: number }>(
|
||||
`SELECT data->>'sourceId' AS source_id,
|
||||
COUNT(*) FILTER (WHERE status IN ('failed','dead') AND created_at > NOW() - INTERVAL '24 hours')::int AS failed_24h,
|
||||
COUNT(*) FILTER (WHERE status IN ('waiting','active','delayed'))::int AS queue_depth
|
||||
COUNT(*) FILTER (WHERE status IN ('waiting','active','delayed'))::int AS queue_depth,
|
||||
COUNT(*) FILTER (WHERE name = 'embed-backfill' AND status = 'active')::int AS backfill_active,
|
||||
COUNT(*) FILTER (WHERE name = 'embed-backfill' AND status IN ('waiting','delayed','waiting-children'))::int AS backfill_queued
|
||||
FROM minion_jobs
|
||||
WHERE name IN ('sync','embed-backfill')
|
||||
AND data->>'sourceId' IS NOT NULL
|
||||
GROUP BY data->>'sourceId'`,
|
||||
);
|
||||
const m = new Map<string, { failed_24h: number; queue_depth: number }>();
|
||||
const m = new Map<string, JobStats>();
|
||||
for (const r of rows) {
|
||||
m.set(r.source_id, { failed_24h: Number(r.failed_24h), queue_depth: Number(r.queue_depth) });
|
||||
m.set(r.source_id, {
|
||||
failed_24h: Number(r.failed_24h),
|
||||
queue_depth: Number(r.queue_depth),
|
||||
backfill_active: Number(r.backfill_active),
|
||||
backfill_queued: Number(r.backfill_queued),
|
||||
});
|
||||
}
|
||||
return m;
|
||||
} catch {
|
||||
|
||||
@@ -19,6 +19,7 @@ import { PGLiteEngine } from '../src/core/pglite-engine.ts';
|
||||
import {
|
||||
EmbeddingColumnNotRegisteredError,
|
||||
} from '../src/core/search/embedding-column.ts';
|
||||
import { configureGateway, resetGateway } from '../src/core/ai/gateway.ts';
|
||||
import type { PageInput, ChunkInput } from '../src/core/types.ts';
|
||||
|
||||
let engine: PGLiteEngine;
|
||||
@@ -26,6 +27,17 @@ let chunkId: number;
|
||||
let chunkId2: number;
|
||||
|
||||
beforeAll(async () => {
|
||||
// Pin the embedding dim to 1536 BEFORE initSchema. initSchema sizes the
|
||||
// `embedding` column from getEmbeddingDimensions() (default 1280 =
|
||||
// zeroentropyai). This test hardcodes 1536-dim vectors + asserts 1536, so
|
||||
// it must NOT inherit ambient/leaked gateway state (which is 1536 from a
|
||||
// local ~/.gbrain config but 1280 in CI → vector(1280) → insert fails).
|
||||
// Pinning here makes the column deterministically 1536 regardless of order.
|
||||
configureGateway({
|
||||
embedding_model: 'openai:text-embedding-3-large',
|
||||
embedding_dimensions: 1536,
|
||||
env: { OPENAI_API_KEY: 'sk-test-cosine-rescore' },
|
||||
});
|
||||
engine = new PGLiteEngine();
|
||||
await engine.connect({});
|
||||
await engine.initSchema();
|
||||
@@ -87,6 +99,7 @@ beforeAll(async () => {
|
||||
|
||||
afterAll(async () => {
|
||||
await engine.disconnect();
|
||||
resetGateway();
|
||||
});
|
||||
|
||||
describe('getEmbeddingsByChunkIds — column parameter (D9)', () => {
|
||||
|
||||
@@ -25,6 +25,8 @@ mock.module('../../src/core/embedding.ts', () => ({
|
||||
// Deterministic fake vector for each chunk.
|
||||
return texts.map(() => new Float32Array(1536));
|
||||
},
|
||||
// v0.41.31: embed phase reads the current signature to stamp provenance.
|
||||
currentEmbeddingSignature: () => 'test:model:1536',
|
||||
}));
|
||||
|
||||
const { runCycle } = await import('../../src/core/cycle.ts');
|
||||
|
||||
@@ -46,6 +46,8 @@ mock.module('../../src/core/embedding.ts', () => ({
|
||||
EMBEDDING_DIMENSIONS: 1536,
|
||||
EMBEDDING_COST_PER_1K_TOKENS: 0.00013,
|
||||
estimateEmbeddingCostUsd: (tokens: number) => (tokens / 1000) * 0.00013,
|
||||
// v0.41.31: embed phase reads the current signature to stamp provenance.
|
||||
currentEmbeddingSignature: () => 'text-embedding-3-large:1536',
|
||||
}));
|
||||
|
||||
const { runCycle, ALL_PHASES } = await import('../../src/core/cycle.ts');
|
||||
@@ -106,6 +108,9 @@ async function withoutAnthropicKey<T>(body: () => Promise<T>): Promise<T> {
|
||||
// v0.31 — added `consolidate` between recompute_emotional_weight and embed
|
||||
// v0.33 — added `resolve_symbol_edges` between extract and patterns
|
||||
type CyclePhase = (typeof ALL_PHASES)[number];
|
||||
// Mirrors src/core/cycle.ts ALL_PHASES order exactly. v0.41.31: synced the
|
||||
// three phases that drifted in after this test was last touched (v0.41.0.0):
|
||||
// extract_atoms (v0.41 T9), synthesize_concepts, conversation_facts_backfill.
|
||||
const EXPECTED_PHASES: CyclePhase[] = [
|
||||
'lint',
|
||||
'backlinks',
|
||||
@@ -113,13 +118,16 @@ const EXPECTED_PHASES: CyclePhase[] = [
|
||||
'synthesize',
|
||||
'extract',
|
||||
'extract_facts', // v0.32.2 — reconcile fence → DB facts index
|
||||
'extract_atoms', // v0.41 T9 — pack-gated atom extraction
|
||||
'resolve_symbol_edges', // v0.33.3 — within-file symbol resolution
|
||||
'patterns',
|
||||
'synthesize_concepts', // v0.41 — concept synthesis after patterns
|
||||
'recompute_emotional_weight', // v0.29
|
||||
'consolidate', // v0.31
|
||||
'propose_takes', // v0.36.1.0 — hindsight calibration wave
|
||||
'grade_takes', // v0.36.1.0
|
||||
'calibration_profile', // v0.36.1.0
|
||||
'conversation_facts_backfill', // v0.41.11.0 — config-gated (default off)
|
||||
'embed',
|
||||
'orphans',
|
||||
'schema-suggest', // v0.39.0.0 — passive schema-suggest after orphans
|
||||
|
||||
@@ -19,6 +19,8 @@ import { hasDatabase, setupDB, teardownDB, getEngine, getConn } from './helpers.
|
||||
// Mock embedBatch so embed phase doesn't call OpenAI.
|
||||
mock.module('../../src/core/embedding.ts', () => ({
|
||||
embedBatch: async (texts: string[]) => texts.map(() => new Float32Array(1536)),
|
||||
// v0.41.31: embed phase reads the current signature to stamp provenance.
|
||||
currentEmbeddingSignature: () => 'test:model:1536',
|
||||
}));
|
||||
|
||||
const { runDream } = await import('../../src/commands/dream.ts');
|
||||
|
||||
@@ -197,6 +197,43 @@ describe('buildSyncStatusReport against real PGLite (IRON RULE regression for Bl
|
||||
expect(b.embedding_coverage_pct).toBe(100);
|
||||
});
|
||||
|
||||
test('v0.41.31: embed-backfill job state surfaced per source (TODO-2)', async () => {
|
||||
// Seed embed-backfill minion jobs for source-a: 2 queued + 1 active +
|
||||
// 1 completed. source-b has none → idle.
|
||||
await engine.executeRaw(
|
||||
`INSERT INTO minion_jobs (name, status, data) VALUES
|
||||
('embed-backfill', 'waiting', '{"sourceId":"source-a"}'::jsonb),
|
||||
('embed-backfill', 'waiting', '{"sourceId":"source-a"}'::jsonb),
|
||||
('embed-backfill', 'active', '{"sourceId":"source-a"}'::jsonb)`,
|
||||
);
|
||||
await engine.executeRaw(
|
||||
`INSERT INTO minion_jobs (name, status, data, finished_at)
|
||||
VALUES ('embed-backfill', 'completed', '{"sourceId":"source-a"}'::jsonb, now())`,
|
||||
);
|
||||
|
||||
const sources = await engine.executeRaw<{
|
||||
id: string; name: string; local_path: string | null; config: Record<string, unknown>;
|
||||
}>(
|
||||
`SELECT id, name, local_path, config FROM sources
|
||||
WHERE local_path IS NOT NULL AND archived IS NOT TRUE ORDER BY id`,
|
||||
);
|
||||
const report = await buildSyncStatusReport(engine, sources);
|
||||
const byId = new Map(report.sources.map((s) => [s.source_id, s]));
|
||||
|
||||
const a = byId.get('source-a')!;
|
||||
expect(a.backfill_queued).toBe(2);
|
||||
expect(a.backfill_active).toBe(1);
|
||||
expect(a.backfill_last_completed_at).not.toBeNull();
|
||||
|
||||
const b = byId.get('source-b')!;
|
||||
expect(b.backfill_queued).toBe(0);
|
||||
expect(b.backfill_active).toBe(0);
|
||||
expect(b.backfill_last_completed_at).toBeNull();
|
||||
|
||||
// Clean up so sibling tests in this describe see a clean minion_jobs.
|
||||
await engine.executeRaw(`DELETE FROM minion_jobs WHERE name = 'embed-backfill'`);
|
||||
});
|
||||
|
||||
test('soft-deleted pages excluded from pages count (v0.26.5 regression)', async () => {
|
||||
// Verifies the `WHERE pg.deleted_at IS NULL` clause in BOTH subqueries
|
||||
// of the dashboard SQL. Pre-fix the original PR query would have
|
||||
|
||||
@@ -32,6 +32,11 @@ mock.module('../src/core/embedding.ts', () => ({
|
||||
activeEmbedCalls--;
|
||||
}
|
||||
},
|
||||
// v0.41.31: embedAll/embedAllStale read the current embedding signature to
|
||||
// stamp provenance. The mock returns a stable value; the mock engine's
|
||||
// setPageEmbeddingSignature / invalidateStaleSignatureEmbeddings resolve to
|
||||
// null via the Proxy default, so the signature value is inert here.
|
||||
currentEmbeddingSignature: () => 'test:model:1536',
|
||||
}));
|
||||
|
||||
// Import AFTER mocking.
|
||||
@@ -105,6 +110,29 @@ describe('runEmbed --all (parallel)', () => {
|
||||
expect(maxConcurrentEmbedCalls).toBeLessThanOrEqual(10);
|
||||
});
|
||||
|
||||
test('v0.41.31: stamps embedding_signature after embedding each page (--all)', async () => {
|
||||
const pages = [{ slug: 'a', source_id: 'default' }, { slug: 'b', source_id: 'default' }];
|
||||
const chunksBySlug = new Map(
|
||||
pages.map(p => [
|
||||
p.slug,
|
||||
[{ chunk_index: 0, chunk_text: `text ${p.slug}`, chunk_source: 'compiled_truth', embedded_at: null, token_count: 4 }],
|
||||
]),
|
||||
);
|
||||
const engine = mockEngine({
|
||||
listPages: async () => pages,
|
||||
getChunks: async (slug: string) => chunksBySlug.get(slug) || [],
|
||||
upsertChunks: async () => {},
|
||||
});
|
||||
|
||||
await runEmbed(engine, ['--all']);
|
||||
|
||||
// The wiring gap this pins: embedAll must CALL setPageEmbeddingSignature
|
||||
// after upsertChunks, with the current signature (mocked to test:model:1536).
|
||||
const stampCalls = (engine as any)._calls.filter((c: any) => c.method === 'setPageEmbeddingSignature');
|
||||
expect(stampCalls.length).toBe(2); // one per page
|
||||
expect(stampCalls[0].args[1]).toEqual({ sourceId: 'default', signature: 'test:model:1536' });
|
||||
});
|
||||
|
||||
test('respects GBRAIN_EMBED_CONCURRENCY=1 (serial)', async () => {
|
||||
const pages = Array.from({ length: 5 }, (_, i) => ({ slug: `page-${i}` }));
|
||||
const chunksBySlug = new Map(
|
||||
|
||||
@@ -0,0 +1,121 @@
|
||||
/**
|
||||
* v0.41.31 — real stale semantics via pages.embedding_signature (PGLite).
|
||||
*
|
||||
* Pins the commit-3 contract:
|
||||
* - R-4 (grandfather, CRITICAL): a page embedded under a NULL signature is
|
||||
* NEVER stale. After the v108 migration every existing page has NULL, so
|
||||
* the next embed --stale must NOT re-embed the whole corpus.
|
||||
* - signature mismatch (model/dims swap) → counted as stale.
|
||||
* - matching signature → not stale.
|
||||
* - invalidateStaleSignatureEmbeddings NULLs only mismatched (grandfathered
|
||||
* NULL + matching untouched) and returns the count.
|
||||
* - setPageEmbeddingSignature stamps.
|
||||
*
|
||||
* Canonical PGLite block (CLAUDE.md R3+R4).
|
||||
*/
|
||||
import { describe, test, expect, beforeAll, afterAll, beforeEach } from 'bun:test';
|
||||
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
|
||||
import { resetPgliteState } from './helpers/reset-pglite.ts';
|
||||
import type { ChunkInput } from '../src/core/types.ts';
|
||||
|
||||
let engine: PGLiteEngine;
|
||||
let colDim: number;
|
||||
|
||||
beforeAll(async () => {
|
||||
engine = new PGLiteEngine();
|
||||
await engine.connect({});
|
||||
await engine.initSchema();
|
||||
}, 30000);
|
||||
|
||||
afterAll(async () => {
|
||||
await engine.disconnect();
|
||||
});
|
||||
|
||||
beforeEach(async () => {
|
||||
await resetPgliteState(engine);
|
||||
const rows = await engine.executeRaw<{ dim: number }>(
|
||||
`SELECT atttypmod AS dim FROM pg_attribute
|
||||
WHERE attrelid = 'content_chunks'::regclass AND attname = 'embedding' AND attnum > 0`,
|
||||
);
|
||||
colDim = Number(rows[0]?.dim);
|
||||
});
|
||||
|
||||
/**
|
||||
* Seed a page with one EMBEDDED chunk (non-null vector) and a given
|
||||
* embedding_signature (null → grandfathered legacy state).
|
||||
*/
|
||||
async function seedEmbedded(slug: string, text: string, signature: string | null, sourceId?: string): Promise<void> {
|
||||
await engine.putPage(slug, { type: 'note', title: slug, compiled_truth: `# ${slug}` }, sourceId ? { sourceId } : undefined);
|
||||
const chunks: ChunkInput[] = [
|
||||
{ chunk_index: 0, chunk_text: text, chunk_source: 'compiled_truth', token_count: 4, embedding: undefined },
|
||||
];
|
||||
await engine.upsertChunks(slug, chunks, sourceId ? { sourceId } : undefined);
|
||||
// Flip the chunk to a non-null vector sized to the actual column dim.
|
||||
await engine.executeRaw(
|
||||
`UPDATE content_chunks
|
||||
SET embedding = ('[' || array_to_string(array_fill(0.0::real, ARRAY[$1::int]), ',') || ']')::vector
|
||||
WHERE page_id = (SELECT id FROM pages WHERE slug = $2 AND source_id = $3)`,
|
||||
[colDim, slug, sourceId ?? 'default'],
|
||||
);
|
||||
if (signature !== null) {
|
||||
await engine.setPageEmbeddingSignature(slug, { sourceId, signature });
|
||||
}
|
||||
}
|
||||
|
||||
describe('embedding_signature stale semantics', () => {
|
||||
test('R-4 GRANDFATHER: NULL signature is never stale', async () => {
|
||||
await seedEmbedded('legacy', 'abcde', null); // embedded, NULL signature
|
||||
// No NULL embeddings, NULL signature → not stale under any signature.
|
||||
expect(await engine.countStaleChunks({ signature: 'openai:m:1536' })).toBe(0);
|
||||
expect(await engine.sumStaleChunkChars({ signature: 'openai:m:1536' })).toBe(0);
|
||||
});
|
||||
|
||||
test('signature MISMATCH (model swap) is counted as stale', async () => {
|
||||
await seedEmbedded('drifted', 'abcde', 'openai:old:1536'); // 5 chars
|
||||
expect(await engine.countStaleChunks({ signature: 'voyage:new:1024' })).toBe(1);
|
||||
expect(await engine.sumStaleChunkChars({ signature: 'voyage:new:1024' })).toBe(5);
|
||||
// Without the signature opt, the legacy NULL-only predicate ignores it.
|
||||
expect(await engine.countStaleChunks()).toBe(0);
|
||||
});
|
||||
|
||||
test('MATCHING signature is not stale', async () => {
|
||||
await seedEmbedded('fresh', 'abcde', 'voyage:new:1024');
|
||||
expect(await engine.countStaleChunks({ signature: 'voyage:new:1024' })).toBe(0);
|
||||
expect(await engine.sumStaleChunkChars({ signature: 'voyage:new:1024' })).toBe(0);
|
||||
});
|
||||
|
||||
test('invalidateStaleSignatureEmbeddings NULLs only mismatched; grandfathered + matching untouched', async () => {
|
||||
await seedEmbedded('old', 'abcde', 'openai:old:1536'); // mismatched → invalidate
|
||||
await seedEmbedded('legacy', 'fghij', null); // grandfathered → keep
|
||||
await seedEmbedded('new', 'klmno', 'voyage:new:1024'); // matching → keep
|
||||
|
||||
const invalidated = await engine.invalidateStaleSignatureEmbeddings({ signature: 'voyage:new:1024' });
|
||||
expect(invalidated).toBe(1); // only 'old'
|
||||
|
||||
// Now exactly the 'old' page's chunk is NULL → legacy stale count = 1.
|
||||
expect(await engine.countStaleChunks()).toBe(1);
|
||||
// Re-running is idempotent (nothing left to invalidate).
|
||||
expect(await engine.invalidateStaleSignatureEmbeddings({ signature: 'voyage:new:1024' })).toBe(0);
|
||||
});
|
||||
|
||||
test('invalidate is sourceId-scoped', async () => {
|
||||
await engine.executeRaw(
|
||||
`INSERT INTO sources (id, name, config) VALUES ('other', 'other', '{}'::jsonb) ON CONFLICT (id) DO NOTHING`,
|
||||
);
|
||||
await seedEmbedded('a', 'abcde', 'openai:old:1536'); // default
|
||||
await seedEmbedded('b', 'fghij', 'openai:old:1536', 'other'); // other
|
||||
const n = await engine.invalidateStaleSignatureEmbeddings({ signature: 'voyage:new:1024', sourceId: 'default' });
|
||||
expect(n).toBe(1);
|
||||
expect(await engine.countStaleChunks({ sourceId: 'default' })).toBe(1);
|
||||
expect(await engine.countStaleChunks({ sourceId: 'other' })).toBe(0); // untouched
|
||||
});
|
||||
|
||||
test('setPageEmbeddingSignature stamps the page', async () => {
|
||||
await engine.putPage('p', { type: 'note', title: 'p', compiled_truth: '# p' });
|
||||
await engine.setPageEmbeddingSignature('p', { signature: 'openai:m:1536' });
|
||||
const rows = await engine.executeRaw<{ embedding_signature: string | null }>(
|
||||
`SELECT embedding_signature FROM pages WHERE slug = 'p' AND source_id = 'default'`,
|
||||
);
|
||||
expect(rows[0]?.embedding_signature).toBe('openai:m:1536');
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,76 @@
|
||||
/**
|
||||
* v0.41.31 (F1) — importFromContent stamps pages.embedding_signature when it
|
||||
* embeds inline.
|
||||
*
|
||||
* The inline import/sync path (importFromContent) writes embeddings without
|
||||
* going through embed.ts/embed-stale.ts. Before the F1 fix it never stamped
|
||||
* the signature, so non-federated/inline brains kept NULL signatures forever
|
||||
* → grandfathered → `embed --stale` never re-embedded them after a model swap
|
||||
* (the headline feature silently inert). This pins the stamp.
|
||||
*
|
||||
* Serial: stubs the gateway embed transport (process-global module state).
|
||||
*/
|
||||
import { describe, test, expect, beforeAll, afterAll, beforeEach, afterEach } from 'bun:test';
|
||||
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
|
||||
import { resetPgliteState } from './helpers/reset-pglite.ts';
|
||||
import { importFromContent } from '../src/core/import-file.ts';
|
||||
import { configureGateway, resetGateway, __setEmbedTransportForTests } from '../src/core/ai/gateway.ts';
|
||||
import { currentEmbeddingSignature } from '../src/core/embedding.ts';
|
||||
|
||||
let engine: PGLiteEngine;
|
||||
let colDim: number;
|
||||
|
||||
beforeAll(async () => {
|
||||
engine = new PGLiteEngine();
|
||||
await engine.connect({});
|
||||
await engine.initSchema();
|
||||
}, 30000);
|
||||
|
||||
afterAll(async () => {
|
||||
await engine.disconnect();
|
||||
});
|
||||
|
||||
beforeEach(async () => {
|
||||
await resetPgliteState(engine);
|
||||
const rows = await engine.executeRaw<{ dim: number }>(
|
||||
`SELECT atttypmod AS dim FROM pg_attribute
|
||||
WHERE attrelid = 'content_chunks'::regclass AND attname = 'embedding' AND attnum > 0`,
|
||||
);
|
||||
colDim = Number(rows[0]?.dim);
|
||||
configureGateway({
|
||||
embedding_model: 'openai:text-embedding-3-large',
|
||||
embedding_dimensions: colDim,
|
||||
env: { OPENAI_API_KEY: 'sk-test-import-stamp' },
|
||||
});
|
||||
// Fake transport (AI SDK embedMany shape): receives { values }, returns
|
||||
// one zero-vector (sized to the column) per input value.
|
||||
__setEmbedTransportForTests(async ({ values }: { values: string[] }) => ({
|
||||
embeddings: values.map(() => Array(colDim).fill(0)),
|
||||
usage: { tokens: 0 },
|
||||
}) as any);
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
__setEmbedTransportForTests(null);
|
||||
resetGateway();
|
||||
});
|
||||
|
||||
async function signatureOf(slug: string): Promise<string | null> {
|
||||
const rows = await engine.executeRaw<{ embedding_signature: string | null }>(
|
||||
`SELECT embedding_signature FROM pages WHERE slug = $1 AND source_id = 'default'`,
|
||||
[slug],
|
||||
);
|
||||
return rows[0]?.embedding_signature ?? null;
|
||||
}
|
||||
|
||||
describe('importFromContent embedding_signature stamping (F1)', () => {
|
||||
test('embeds inline → stamps the current signature', async () => {
|
||||
await importFromContent(engine, 'concepts/stamped', '# Stamped\n\nsome body content to chunk and embed.', {});
|
||||
expect(await signatureOf('concepts/stamped')).toBe(currentEmbeddingSignature());
|
||||
});
|
||||
|
||||
test('--no-embed → leaves signature NULL (grandfathered, not stale)', async () => {
|
||||
await importFromContent(engine, 'concepts/unstamped', '# Unstamped\n\nbody content.', { noEmbed: true });
|
||||
expect(await signatureOf('concepts/unstamped')).toBeNull();
|
||||
});
|
||||
});
|
||||
@@ -158,6 +158,10 @@ const REQUIRED_BOOTSTRAP_COVERAGE: ForwardReference[] = [
|
||||
// pages_generation_idx (CREATE INDEX ON pages (generation)) so bootstrap
|
||||
// probes guard pre-v91 brains.
|
||||
{ kind: 'column', table: 'pages', column: 'generation' },
|
||||
// v0.41.31 (v108) — pages.embedding_signature TEXT for real stale
|
||||
// semantics. No SCHEMA_SQL index references it; bootstrap probe is
|
||||
// defense-in-depth (and satisfies the MIGRATIONS ADD COLUMN coverage gate).
|
||||
{ kind: 'column', table: 'pages', column: 'embedding_signature' },
|
||||
];
|
||||
|
||||
test('applyForwardReferenceBootstrap covers every forward reference declared in REQUIRED_BOOTSTRAP_COVERAGE', async () => {
|
||||
|
||||
@@ -167,6 +167,24 @@ describe('computeAllSourceMetrics', () => {
|
||||
expect(result.find((m) => m.source_id === 'other')!.total_pages).toBe(1);
|
||||
});
|
||||
|
||||
test('v0.41.31: embed-backfill active/queued counts per source (CLI BACKFILL column)', async () => {
|
||||
// 2 queued + 1 active embed-backfill for default; a non-backfill 'sync'
|
||||
// job must NOT inflate the backfill counts (only the generic queue_depth).
|
||||
await engine.executeRaw(
|
||||
`INSERT INTO minion_jobs (name, status, data) VALUES
|
||||
('embed-backfill', 'waiting', '{"sourceId":"default"}'::jsonb),
|
||||
('embed-backfill', 'waiting', '{"sourceId":"default"}'::jsonb),
|
||||
('embed-backfill', 'active', '{"sourceId":"default"}'::jsonb),
|
||||
('sync', 'waiting', '{"sourceId":"default"}'::jsonb)`,
|
||||
);
|
||||
const sources = await loadAllSources(engine);
|
||||
const dflt = (await computeAllSourceMetrics(engine, sources)).find((m) => m.source_id === 'default')!;
|
||||
expect(dflt.backfill_active).toBe(1);
|
||||
expect(dflt.backfill_queued).toBe(2);
|
||||
// generic queue_depth includes the sync job too (4 total waiting/active).
|
||||
expect(dflt.queue_depth).toBe(4);
|
||||
});
|
||||
|
||||
test('webhook_configured reflects config.webhook_secret presence', async () => {
|
||||
await engine.executeRaw(
|
||||
`INSERT INTO sources (id, name, config) VALUES ('webhooky', 'webhooky', '{"federated":true,"webhook_secret":"x","github_repo":"a/b"}'::jsonb)`,
|
||||
|
||||
@@ -0,0 +1,113 @@
|
||||
/**
|
||||
* v0.41.31 — BrainEngine.sumStaleChunkChars tests (PGLite).
|
||||
*
|
||||
* Sibling of countStaleChunks: sums LENGTH(chunk_text) over stale chunks so
|
||||
* the `gbrain sync --all` cost preview can price the embedding backlog.
|
||||
* Validates: empty brain → 0, exact char sum, sourceId scope, embed_skip
|
||||
* exclusion, and that non-null (already-embedded) chunks are NOT counted.
|
||||
*
|
||||
* Uses the canonical PGLite block (one engine per file, resetPgliteState in
|
||||
* beforeEach) per CLAUDE.md test-isolation rules R3+R4.
|
||||
*/
|
||||
import { describe, test, expect, beforeAll, afterAll, beforeEach } from 'bun:test';
|
||||
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
|
||||
import { resetPgliteState } from './helpers/reset-pglite.ts';
|
||||
import type { ChunkInput } from '../src/core/types.ts';
|
||||
|
||||
let engine: PGLiteEngine;
|
||||
|
||||
beforeAll(async () => {
|
||||
engine = new PGLiteEngine();
|
||||
await engine.connect({});
|
||||
await engine.initSchema();
|
||||
}, 30000);
|
||||
|
||||
afterAll(async () => {
|
||||
await engine.disconnect();
|
||||
});
|
||||
|
||||
beforeEach(async () => {
|
||||
await resetPgliteState(engine);
|
||||
});
|
||||
|
||||
/** Seed a page + stale (NULL-embedding) chunks with explicit chunk_text. */
|
||||
async function seedStale(
|
||||
slug: string,
|
||||
texts: string[],
|
||||
opts?: { sourceId?: string; frontmatter?: Record<string, unknown> },
|
||||
): Promise<void> {
|
||||
await engine.putPage(
|
||||
slug,
|
||||
{
|
||||
type: 'note',
|
||||
title: slug,
|
||||
compiled_truth: `# ${slug}`,
|
||||
...(opts?.frontmatter ? { frontmatter: opts.frontmatter } : {}),
|
||||
},
|
||||
opts?.sourceId ? { sourceId: opts.sourceId } : undefined,
|
||||
);
|
||||
const chunks: ChunkInput[] = texts.map((t, i) => ({
|
||||
chunk_index: i,
|
||||
chunk_text: t,
|
||||
chunk_source: 'compiled_truth',
|
||||
token_count: 4,
|
||||
embedding: undefined, // NULL = stale
|
||||
}));
|
||||
await engine.upsertChunks(slug, chunks, opts?.sourceId ? { sourceId: opts.sourceId } : undefined);
|
||||
}
|
||||
|
||||
describe('sumStaleChunkChars', () => {
|
||||
test('empty brain returns 0', async () => {
|
||||
expect(await engine.sumStaleChunkChars()).toBe(0);
|
||||
});
|
||||
|
||||
test('sums LENGTH(chunk_text) across stale chunks', async () => {
|
||||
await seedStale('a', ['abcde', 'fghij']); // 5 + 5
|
||||
await seedStale('b', ['xyz']); // 3
|
||||
expect(await engine.sumStaleChunkChars()).toBe(13);
|
||||
});
|
||||
|
||||
test('scopes to sourceId when provided', async () => {
|
||||
await engine.executeRaw(
|
||||
`INSERT INTO sources (id, name, config) VALUES ('other', 'other', '{"federated":true}'::jsonb) ON CONFLICT (id) DO NOTHING`,
|
||||
);
|
||||
await seedStale('a', ['abcde']); // default, 5
|
||||
await seedStale('b', ['xyzwv'], { sourceId: 'other' }); // other, 5
|
||||
expect(await engine.sumStaleChunkChars({ sourceId: 'default' })).toBe(5);
|
||||
expect(await engine.sumStaleChunkChars({ sourceId: 'other' })).toBe(5);
|
||||
expect(await engine.sumStaleChunkChars()).toBe(10); // all sources
|
||||
});
|
||||
|
||||
test('excludes pages with embed_skip frontmatter', async () => {
|
||||
await seedStale('keep', ['abcde']); // 5, counted
|
||||
await seedStale('skip', ['1234567890'], {
|
||||
frontmatter: { embed_skip: { at: '2026-01-01T00:00:00Z', reason: 'oversize' } },
|
||||
});
|
||||
expect(await engine.sumStaleChunkChars()).toBe(5);
|
||||
});
|
||||
|
||||
test('does NOT count already-embedded (non-null) chunks', async () => {
|
||||
// Stale page → counted.
|
||||
await seedStale('stale', ['abcde']); // 5
|
||||
// "Embedded" page: seed as stale, then flip its chunk's embedding to a
|
||||
// non-null vector via raw SQL. We bypass upsertChunks so the test doesn't
|
||||
// depend on gateway-configured dimensions (keeps it robust regardless of
|
||||
// any leaked gateway state from sibling files).
|
||||
await seedStale('done', ['this is embedded and should not count']);
|
||||
// Size the embedding to the ACTUAL column dimension (pgvector stores it
|
||||
// in atttypmod) so the test is robust to whatever dim initSchema chose.
|
||||
const dimRows = await engine.executeRaw<{ dim: number }>(
|
||||
`SELECT atttypmod AS dim FROM pg_attribute
|
||||
WHERE attrelid = 'content_chunks'::regclass AND attname = 'embedding' AND attnum > 0`,
|
||||
);
|
||||
const dim = Number(dimRows[0]?.dim);
|
||||
expect(dim).toBeGreaterThan(0);
|
||||
await engine.executeRaw(
|
||||
`UPDATE content_chunks
|
||||
SET embedding = ('[' || array_to_string(array_fill(0.0::real, ARRAY[$1::int]), ',') || ']')::vector
|
||||
WHERE page_id = (SELECT id FROM pages WHERE slug = 'done' AND source_id = 'default')`,
|
||||
[dim],
|
||||
);
|
||||
expect(await engine.sumStaleChunkChars()).toBe(5);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,160 @@
|
||||
/**
|
||||
* v0.41.31 — `gbrain sync --all` cost-gate wiring regressions (PGLite).
|
||||
*
|
||||
* Pure shouldBlockSync / willEmbedSynchronously logic is pinned in
|
||||
* test/sync-cost-preview.test.ts. THIS file pins the end-to-end wiring in
|
||||
* runSync's --all path:
|
||||
*
|
||||
* R-1 (headline): deferred-embed sync --all, non-TTY, with backlog →
|
||||
* emits gate:'deferred_notice' and NEVER exit 2 (the cron-blocking
|
||||
* bug this release fixes).
|
||||
* R-2 (protection): inline-embed sync --all (--serial), non-TTY, above
|
||||
* floor → still exit 2 with gate:'confirmation_required'.
|
||||
*
|
||||
* Serial-quarantined: stubs process.exit + console.log (process-global).
|
||||
*/
|
||||
import { describe, test, expect, beforeAll, afterAll, beforeEach, afterEach } from 'bun:test';
|
||||
import { mkdtempSync, writeFileSync, rmSync, mkdirSync } from 'fs';
|
||||
import { execSync } from 'child_process';
|
||||
import { tmpdir } from 'os';
|
||||
import { join } from 'path';
|
||||
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
|
||||
import { runSources } from '../src/commands/sources.ts';
|
||||
import { resetPgliteState } from './helpers/reset-pglite.ts';
|
||||
import { configureGateway, resetGateway } from '../src/core/ai/gateway.ts';
|
||||
import { CHUNKER_VERSION } from '../src/core/chunkers/code.ts';
|
||||
import type { ChunkInput } from '../src/core/types.ts';
|
||||
|
||||
let engine: PGLiteEngine;
|
||||
let repoPath: string;
|
||||
let headSha: string;
|
||||
|
||||
beforeAll(async () => {
|
||||
engine = new PGLiteEngine();
|
||||
await engine.connect({});
|
||||
await engine.initSchema();
|
||||
}, 60_000);
|
||||
|
||||
afterAll(async () => {
|
||||
if (engine) await engine.disconnect();
|
||||
}, 60_000);
|
||||
|
||||
beforeEach(async () => {
|
||||
await resetPgliteState(engine);
|
||||
// Configure the gateway with a dummy key so the pre-gate embedding-creds
|
||||
// preflight passes (diagnoseEmbedding reads gateway configure-time state,
|
||||
// not live env). The gate runs before any real embed call, so no network
|
||||
// request is made.
|
||||
configureGateway({
|
||||
embedding_model: 'openai:text-embedding-3-large',
|
||||
embedding_dimensions: 1536,
|
||||
env: { OPENAI_API_KEY: 'sk-test-costgate' },
|
||||
});
|
||||
repoPath = mkdtempSync(join(tmpdir(), 'gbrain-costgate-'));
|
||||
execSync('git init', { cwd: repoPath, stdio: 'pipe' });
|
||||
execSync('git config user.email "t@t.com"', { cwd: repoPath, stdio: 'pipe' });
|
||||
execSync('git config user.name "T"', { cwd: repoPath, stdio: 'pipe' });
|
||||
mkdirSync(join(repoPath, 'topics'), { recursive: true });
|
||||
writeFileSync(
|
||||
join(repoPath, 'topics/foo.md'),
|
||||
['---', 'type: concept', 'title: Foo', '---', '', 'some body content to estimate.'].join('\n'),
|
||||
);
|
||||
execSync('git add -A && git commit -m initial', { cwd: repoPath, stdio: 'pipe' });
|
||||
headSha = execSync('git rev-parse HEAD', { cwd: repoPath, stdio: 'pipe' }).toString().trim();
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
resetGateway();
|
||||
if (repoPath) rmSync(repoPath, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
/** Run runSync(args) with process.exit + console.log captured. */
|
||||
async function runSyncCaptured(args: string[]): Promise<{ exitCode: number | undefined; stdout: string }> {
|
||||
const { runSync } = await import('../src/commands/sync.ts');
|
||||
const origExit = process.exit;
|
||||
const origLog = console.log.bind(console);
|
||||
const out: string[] = [];
|
||||
let exitCode: number | undefined;
|
||||
console.log = (...a: unknown[]) => {
|
||||
out.push(a.map((x) => (typeof x === 'string' ? x : JSON.stringify(x))).join(' '));
|
||||
};
|
||||
process.exit = ((code?: number) => {
|
||||
exitCode = code;
|
||||
throw new Error('__exit__');
|
||||
}) as typeof process.exit;
|
||||
try {
|
||||
await runSync(engine, args);
|
||||
} catch (e) {
|
||||
if ((e as Error).message !== '__exit__') throw e;
|
||||
} finally {
|
||||
process.exit = origExit;
|
||||
console.log = origLog;
|
||||
}
|
||||
return { exitCode, stdout: out.join('\n') };
|
||||
}
|
||||
|
||||
describe('v0.41.31 — sync --all cost gate wiring', () => {
|
||||
test('R-1: deferred sync --all (non-TTY) emits deferred_notice and never exit 2', async () => {
|
||||
await runSources(engine, ['add', 'vault', '--path', repoPath, '--no-federated']);
|
||||
// Make the fan-out a clean no-op: last_commit == HEAD so performSync
|
||||
// reports up_to_date (no git pull, no backfill submit).
|
||||
await engine.executeRaw(`UPDATE sources SET last_commit = $1 WHERE id = 'vault'`, [headSha]);
|
||||
// Seed a stale backlog so the deferred notice has a non-zero figure.
|
||||
await engine.putPage('vault/note', { type: 'note', title: 'note', compiled_truth: '# note' }, { sourceId: 'vault' });
|
||||
const chunks: ChunkInput[] = [
|
||||
{ chunk_index: 0, chunk_text: 'x'.repeat(500), chunk_source: 'compiled_truth', token_count: 4, embedding: undefined },
|
||||
];
|
||||
await engine.upsertChunks('vault/note', chunks, { sourceId: 'vault' });
|
||||
|
||||
// v2 default ON → deferred. --json → agent path. NO --yes. --no-pull
|
||||
// because the synthetic repo has no remote.
|
||||
const { exitCode, stdout } = await runSyncCaptured(['--all', '--json', '--no-pull']);
|
||||
|
||||
// The headline assertion: NOT blocked.
|
||||
expect(exitCode).not.toBe(2);
|
||||
expect(stdout).toContain('"gate":"deferred_notice"');
|
||||
}, 60_000);
|
||||
|
||||
test('R-2: inline sync --all (--serial) above floor still exit 2', async () => {
|
||||
await runSources(engine, ['add', 'vault', '--path', repoPath, '--no-federated']);
|
||||
// Floor 0 → any nonzero inline cost blocks. Source is unsynced
|
||||
// (last_commit NULL) so estimateInlineNewTokens sees it as changed →
|
||||
// full-tree tokens > 0 → costUsd > 0 > floor.
|
||||
await engine.setConfig('sync.cost_gate_min_usd', '0');
|
||||
|
||||
// --serial forces inline even with v2 on. --json → non-TTY exit-2 path.
|
||||
const { exitCode, stdout } = await runSyncCaptured(['--all', '--serial', '--json', '--no-pull']);
|
||||
|
||||
expect(exitCode).toBe(2);
|
||||
expect(stdout).toContain('"gate":"confirmation_required"');
|
||||
}, 60_000);
|
||||
|
||||
test('R-3: inline, git-unchanged source but STALE chunker_version still estimates (not $0)', async () => {
|
||||
// The unchanged-source short-circuit requires HEAD==last_commit AND clean
|
||||
// tree AND chunker_version == current. Here git is unchanged but the
|
||||
// chunker drifted, so the source must NOT be treated as 0 — sync would
|
||||
// re-chunk + re-embed everything. floor=0 so any nonzero cost blocks.
|
||||
await runSources(engine, ['add', 'vault', '--path', repoPath, '--no-federated']);
|
||||
await engine.executeRaw(`UPDATE sources SET last_commit = $1, chunker_version = $2 WHERE id = 'vault'`, [headSha, 'STALE-0']);
|
||||
await engine.setConfig('sync.cost_gate_min_usd', '0');
|
||||
|
||||
const { exitCode, stdout } = await runSyncCaptured(['--all', '--serial', '--json', '--no-pull']);
|
||||
|
||||
expect(exitCode).toBe(2);
|
||||
expect(stdout).toContain('"gate":"confirmation_required"');
|
||||
}, 60_000);
|
||||
|
||||
test('R-3 control: inline, git-unchanged + CURRENT chunker_version short-circuits to $0 (no exit 2)', async () => {
|
||||
// Same setup but chunker_version matches current → the source IS unchanged
|
||||
// → contributes 0 new-content tokens → below floor → proceeds (no block).
|
||||
// Proves the short-circuit fires when (and only when) everything matches.
|
||||
await runSources(engine, ['add', 'vault', '--path', repoPath, '--no-federated']);
|
||||
await engine.executeRaw(`UPDATE sources SET last_commit = $1, chunker_version = $2 WHERE id = 'vault'`, [headSha, String(CHUNKER_VERSION)]);
|
||||
await engine.setConfig('sync.cost_gate_min_usd', '0');
|
||||
|
||||
const { exitCode, stdout } = await runSyncCaptured(['--all', '--serial', '--json', '--no-pull']);
|
||||
|
||||
expect(exitCode).not.toBe(2);
|
||||
expect(stdout).not.toContain('"gate":"confirmation_required"');
|
||||
}, 60_000);
|
||||
});
|
||||
@@ -14,23 +14,54 @@
|
||||
*/
|
||||
|
||||
import { describe, test, expect } from 'bun:test';
|
||||
import { EMBEDDING_COST_PER_1K_TOKENS, estimateEmbeddingCostUsd } from '../src/core/embedding.ts';
|
||||
import {
|
||||
EMBEDDING_COST_PER_1K_TOKENS,
|
||||
estimateEmbeddingCostUsd,
|
||||
willEmbedSynchronously,
|
||||
shouldBlockSync,
|
||||
} from '../src/core/embedding.ts';
|
||||
import { lookupEmbeddingPrice } from '../src/core/embedding-pricing.ts';
|
||||
import { estimateTokens } from '../src/core/chunkers/code.ts';
|
||||
|
||||
describe('Layer 8 D1 — embedding cost model', () => {
|
||||
test('EMBEDDING_COST_PER_1K_TOKENS is text-embedding-3-large pricing', () => {
|
||||
// Update this when OpenAI changes text-embedding-3-large pricing.
|
||||
// As of 2026-04-24: $0.00013 / 1k tokens.
|
||||
test('EMBEDDING_COST_PER_1K_TOKENS back-compat constant is the OpenAI 3-large rate', () => {
|
||||
// Retained only for back-compat imports. Live cost math now resolves the
|
||||
// CONFIGURED model's rate via embedding-pricing.ts (see model-aware test
|
||||
// below). As of 2026-04-24: $0.00013 / 1k tokens.
|
||||
expect(EMBEDDING_COST_PER_1K_TOKENS).toBe(0.00013);
|
||||
});
|
||||
|
||||
test('estimateEmbeddingCostUsd scales linearly with tokens', () => {
|
||||
test('estimateEmbeddingCostUsd scales linearly (gateway-unconfigured fallback = OpenAI rate)', () => {
|
||||
// With no gateway configured (unit-test context) the estimator falls back
|
||||
// to the OpenAI text-embedding-3-large rate ($0.13/Mtok = $0.00013/1k).
|
||||
expect(estimateEmbeddingCostUsd(0)).toBe(0);
|
||||
expect(estimateEmbeddingCostUsd(1000)).toBeCloseTo(0.00013, 5);
|
||||
expect(estimateEmbeddingCostUsd(10_000)).toBeCloseTo(0.0013, 4);
|
||||
expect(estimateEmbeddingCostUsd(1_000_000)).toBeCloseTo(0.13, 4);
|
||||
});
|
||||
|
||||
test('cost preview uses the CONFIGURED model rate, not a hardcoded OpenAI rate', () => {
|
||||
// Regression: the cost gate previously hardcoded $0.00013/1k (OpenAI
|
||||
// text-embedding-3-large) regardless of the configured embedding model,
|
||||
// so a brain on a cheaper model (e.g. zeroentropyai:zembed-1 @ $0.05/Mtok)
|
||||
// saw a preview that named the wrong provider and over-stated spend ~2.6x.
|
||||
// The pricing table is the single source of truth per provider:model.
|
||||
const TOKENS = 2_590_710_262; // a real large-brain sync preview
|
||||
const openai = lookupEmbeddingPrice('openai:text-embedding-3-large');
|
||||
const zeroentropy = lookupEmbeddingPrice('zeroentropyai:zembed-1');
|
||||
expect(openai.kind).toBe('known');
|
||||
expect(zeroentropy.kind).toBe('known');
|
||||
if (openai.kind === 'known' && zeroentropy.kind === 'known') {
|
||||
const openaiCost = (TOKENS / 1_000_000) * openai.pricePerMTok;
|
||||
const zeCost = (TOKENS / 1_000_000) * zeroentropy.pricePerMTok;
|
||||
// The two models must produce materially different previews; a fix that
|
||||
// collapses both to the OpenAI number would regress this assertion.
|
||||
expect(openaiCost).toBeCloseTo(336.79, 1);
|
||||
expect(zeCost).toBeCloseTo(129.54, 1);
|
||||
expect(zeCost).toBeLessThan(openaiCost);
|
||||
}
|
||||
});
|
||||
|
||||
test('5K-file TS repo sanity check: ~$5 at ~400k tokens', () => {
|
||||
// A 5K-file TS repo at ~80 tokens/file averages ~400k tokens. Cost:
|
||||
// 400_000 / 1000 * 0.00013 = $0.052 ≈ $0.05. Not $5. The CHANGELOG
|
||||
@@ -43,6 +74,50 @@ describe('Layer 8 D1 — embedding cost model', () => {
|
||||
});
|
||||
});
|
||||
|
||||
describe('v0.41.31 — willEmbedSynchronously (embed-mode resolver)', () => {
|
||||
// Mirrors sync.ts:2346 effectiveNoEmbed = v2 && !serial && !noEmbed ? true : noEmbed.
|
||||
// Embed runs INLINE iff that resolves to false.
|
||||
test('v2 off → inline (legacy synchronous embed)', () => {
|
||||
expect(willEmbedSynchronously({ v2Enabled: false, serialFlag: false, noEmbed: false })).toBe('inline');
|
||||
});
|
||||
test('v2 on + parallel → deferred (backfill jobs)', () => {
|
||||
expect(willEmbedSynchronously({ v2Enabled: true, serialFlag: false, noEmbed: false })).toBe('deferred');
|
||||
});
|
||||
test('v2 on + --serial → inline', () => {
|
||||
expect(willEmbedSynchronously({ v2Enabled: true, serialFlag: true, noEmbed: false })).toBe('inline');
|
||||
});
|
||||
test('--no-embed forces deferred regardless of v2/serial', () => {
|
||||
expect(willEmbedSynchronously({ v2Enabled: false, serialFlag: false, noEmbed: true })).toBe('deferred');
|
||||
expect(willEmbedSynchronously({ v2Enabled: true, serialFlag: true, noEmbed: true })).toBe('deferred');
|
||||
});
|
||||
});
|
||||
|
||||
describe('v0.41.31 — shouldBlockSync (cost-gate decision)', () => {
|
||||
// R-1: deferred NEVER blocks, even at absurd cost (the headline fix — a
|
||||
// nightly cron over a synced corpus must not exit 2).
|
||||
test('R-1: deferred never blocks, even at $999', () => {
|
||||
expect(shouldBlockSync(999, 0.5, 'deferred')).toBe(false);
|
||||
expect(shouldBlockSync(0, 0.5, 'deferred')).toBe(false);
|
||||
});
|
||||
// R-2: inline still blocks above the floor (protection preserved where
|
||||
// sync actually spends synchronously).
|
||||
test('R-2: inline blocks above floor', () => {
|
||||
expect(shouldBlockSync(0.51, 0.5, 'inline')).toBe(true);
|
||||
expect(shouldBlockSync(130, 0.5, 'inline')).toBe(true);
|
||||
});
|
||||
test('inline at exactly the floor does NOT block (boundary)', () => {
|
||||
expect(shouldBlockSync(0.5, 0.5, 'inline')).toBe(false);
|
||||
});
|
||||
test('inline below floor does not block (kills cents-level cron noise)', () => {
|
||||
expect(shouldBlockSync(0.03, 0.5, 'inline')).toBe(false);
|
||||
expect(shouldBlockSync(0, 0.5, 'inline')).toBe(false);
|
||||
});
|
||||
test('floor of 0 makes inline block on any nonzero cost', () => {
|
||||
expect(shouldBlockSync(0.0001, 0, 'inline')).toBe(true);
|
||||
expect(shouldBlockSync(0, 0, 'inline')).toBe(false);
|
||||
});
|
||||
});
|
||||
|
||||
describe('Layer 8 D1 — estimateTokens (exported from chunkers/code.ts)', () => {
|
||||
test('empty string is 0 tokens', () => {
|
||||
expect(estimateTokens('')).toBe(0);
|
||||
|
||||
Reference in New Issue
Block a user