mirror of
https://github.com/garrytan/gbrain.git
synced 2026-07-27 22:15:33 +00:00
Merge remote-tracking branch 'origin/master' into garrytan/newest-agents-filed-issues
# Conflicts: # TODOS.md # src/cli.ts # src/core/migrate.ts
This commit is contained in:
Garry Tan
@@ -2,6 +2,39 @@
|
||||
|
||||
All notable changes to GBrain will be documented in this file.
|
||||
|
||||
## [0.42.41.0] - 2026-06-11
|
||||
|
||||
**A correctness-and-reliability wave: your conversation facts survive a cycle, write-through stops polluting other repos, autopilot rides out a DB blip instead of crash-looping, and concurrent PGLite processes stop corrupting each other.** A triage of open reports surfaced six bugs with no fix yet plus a batch of community PRs; this ships them together, each with a regression test.
|
||||
|
||||
The headline is data durability. The `extract_facts` cycle phase reconciles a page's facts from its `## Facts` fence by deleting then reinserting — but conversation facts (written by `extract-conversation-facts`) live on pages that have no fence, so a cycle could delete them and reinsert nothing. A failed sync made it worse by escalating the phase to a full-brain walk. Both are fixed: the reconcile now protects non-fence (`cli:`-origin) facts, the destructive phase no longer inherits the failed-sync full-walk, and a reconcile that removes far more than it adds now reports `warn` instead of a silent `ok`.
|
||||
|
||||
Three more silent failures, each found in production and now self-healing or loud:
|
||||
- **Timeline writes that silently stopped.** A migration renumbered during a merge could be recorded as applied without its index change ever running, so every timeline insert failed its `ON CONFLICT`. `gbrain` now repairs the index shape on every migrate pass (dedupe-then-rebuild, even when nothing is pending), `gbrain doctor` reports the drift, and the meetings extractor no longer swallows batch errors.
|
||||
- **Autopilot crash-loop on transient DB errors.** The health-probe recovery called `connect()` without its config, so every reconnect threw and the process exited on any blip. It now uses `reconnect()`, which restores the captured config; `reconnect()` is a first-class method on both engines.
|
||||
- **WAL corruption from concurrent PGLite processes.** A live, working process holding the data-dir lock could have it stolen after five minutes. The lock now heartbeats while held and is only reclaimed from a dead or genuinely stalled holder.
|
||||
|
||||
### Added
|
||||
- **`timeline_dedup_index` doctor check + always-run repair (#2038).** Detects and heals a stale `idx_timeline_dedup` shape; `gbrain apply-migrations --force-schema` triggers the repair on demand. Reported by @jbarol.
|
||||
- **`BrainEngine.reconnect()` on both engines (#2034).** Config-restoring reconnect, replacing the `disconnect()`+bare-`connect()` pattern.
|
||||
|
||||
### Fixed
|
||||
- **Conversation facts survive a fence reconcile (#1928).** The cycle's per-page wipe now excludes non-fence (`cli:`) facts, the destructive phase no longer full-walks on a failed sync, and net-negative reconciles surface as `warn`.
|
||||
- **`put_page` write-through no longer leaks into an unrelated source's repo (#2018).** It mirrors to the assigned source's own `local_path`; a source without one is skipped rather than written into the global repo path.
|
||||
- **Autopilot survives transient DB errors instead of crash-looping (#2034).**
|
||||
- **Concurrent PGLite processes no longer corrupt the WAL (#2058).** Heartbeat + steal-grace replaces the age-only stale-lock check.
|
||||
- **Timeline migration drift self-heals; the extractor stops swallowing errors (#2038, #2057).** A Date-typed batch date round-trips correctly (verified by test).
|
||||
- **A cwd `.env` `DATABASE_URL` no longer silently retargets the brain (#2064, closes #427).** Reported by @bomliu.
|
||||
- **`gbrain sync --strategy code` honors `.gitignore` and skips `vendor`/`dist`/`build`/`venv` (#2052, #2020).** Reported by @aphaiboon and @dMac716.
|
||||
- **Asymmetric embedding `input_type` reaches the wire across every openai-compatible recipe (#2033, supersedes #1400).** Query vectors are no longer document-typed. By @pabloglzg; original diagnosis by @billy-armstrong.
|
||||
- **`updateSourceConfig` JSONB merge is atomic (#2074).** Eliminates a concurrent-writer lost-update race. Reported by @pai-scaffolde.
|
||||
- **`gbrain doctor` correctness pass (#2075):** stale-lock hints, content sanity, graph coverage, exit code, gateway guard. Reported by @pai-scaffolde.
|
||||
- **`gbrain search` returns results instead of exiting 0 empty on slow poolers; scoped `code-callers`/`code-callees` find their edges (#2073).** Migration v116 backfills NULL edge `source_id` and indexes `from_symbol_qualified`. Reported by @jbarol.
|
||||
- **OAuth scope handling (#2009, #2072):** an omitted authorize scope now defaults to the client's registered grant (clamped to it, so no widening) instead of an empty grant that never self-heals; legacy token source grants are honored through a single shared scope parser. By @austinrarnett and @maxpetrusenkoagent.
|
||||
|
||||
### To take advantage of v0.42.41.0
|
||||
|
||||
`gbrain upgrade`. The timeline-index repair and migration v116 run automatically on the next migrate pass — if `gbrain doctor` flagged a timeline or call-graph problem before, re-run it after upgrade. No config changes required; the facts, write-through, autopilot, and lock fixes apply on restart.
|
||||
|
||||
## [0.42.40.0] - 2026-06-09
|
||||
|
||||
**`gbrain extract --stale` no longer aborts partway through a brain that contains emoji or other non-BMP characters.** On a large brain, link/timeline extraction could die with `invalid input syntax for type json` and commit nothing — and because the staleness bookmark only advances on a clean finish, every retry re-hit the same point and extraction stayed wedged. The cause: the link-context excerpt was sliced by raw UTF-16 index, so a window boundary landing inside an emoji's surrogate pair left an unpaired surrogate half in the text, which Postgres rejects when the batch is serialized to JSONB — taking down the whole batch, not just the one row. (PGLite is more permissive here, so this primarily bit the managed-Postgres engine.)
|
||||
|
||||
@@ -23,6 +23,31 @@ are the bar). Plan + GSTACK REVIEW REPORT at
|
||||
string window (`user:`/`assistant:` prefixes) to avoid a dual-shape contract.
|
||||
If MCP callers accumulate parsing bugs, add a structured array param beside it.
|
||||
**Where:** `src/core/operations.ts:volunteer_context` + `src/core/context/volunteer.ts:parseWindow`.
|
||||
## gbrain triage wave follow-ups (filed v0.42.41.0)
|
||||
|
||||
Deferred from the v0.42.41.0 fix wave (eng-reviewed as separate scope, not hotfixes).
|
||||
See plan + GSTACK REVIEW REPORT at
|
||||
`~/.claude/plans/system-instruction-you-are-working-zany-thacker.md`.
|
||||
|
||||
- [ ] **P1 — supervisor: retry-with-backoff instead of hard stop on transient DB outages (#1994).**
|
||||
`max_crashes_exceeded` gives up permanently; a transient pooler blip that trips the
|
||||
counter wedges the supervisor until manual restart. **Why:** the #2034 reconnect fix
|
||||
makes the engine recover, but the supervisor still hard-stops. **Where:**
|
||||
`src/core/minions/supervisor.ts` crash-count loop — add exponential backoff with a
|
||||
much higher (or no) permanent-give-up threshold for recoverable errors.
|
||||
- [ ] **P2 — PGLite `reindex-frontmatter` / backfill statement_timeout boost (#1963).**
|
||||
Community RCA: `SET LOCAL statement_timeout` is gated on `engine.kind === 'postgres'`,
|
||||
so PGLite inherits the 30s session default and trips on non-trivial batches; the CLI
|
||||
then swallows the error and exits 0. **Where:** `src/core/backfill-effective-date.ts`
|
||||
(boost on PGLite too, or per-row updates) + the cli.ts catch that hides it.
|
||||
- [ ] **P2 — autopilot drain-worker concurrency self-deadlock (#2050).** Drain-worker
|
||||
runs at concurrency=1, so any cycle phase that spawns a subagent (patterns, synthesize)
|
||||
deadlocks waiting on a worker slot it can't get. **Where:** autopilot drain-worker
|
||||
dispatch — raise concurrency or exempt subagent-spawning phases.
|
||||
- [ ] **P3 — name-keyed migration ledger (#2038 structural follow-up).** The always-run
|
||||
index drift probe heals the one known case; the general fix is keying applied-migration
|
||||
tracking by stable name rather than version integer so a renumber can't strand a
|
||||
migration as recorded-but-not-executed. **Where:** `src/core/migrate.ts` ledger.
|
||||
|
||||
## gbrain#1981 Retrieval Reflex follow-ups (v0.43+)
|
||||
|
||||
@@ -1821,11 +1846,6 @@ Three items deferred:
|
||||
self-heals via stale-reclaim). The common sync SUCCESS path already drains via
|
||||
handleCliOnly's finally. Convert for graceful drain on sync error exits.
|
||||
|
||||
- [ ] **(v0.42.20.0 follow-up) Decouple the op-dispatch force-exit timer** so it
|
||||
wraps `engine.disconnect()` only (it's armed before the handler today, doubling
|
||||
as a blanket handler watchdog) and fix its misleading "engine.disconnect() did
|
||||
not return…" message that fires even when the handler (not disconnect) was slow.
|
||||
|
||||
- [ ] **(v0.42.20.0 follow-up) Gateway idle-timeout (vs absolute) for streaming
|
||||
chat.** `withDefaultTimeout` uses an absolute `AbortSignal.timeout`; a streaming
|
||||
generation actively producing tokens past the chat default (300s) would abort.
|
||||
@@ -3595,6 +3615,18 @@ keeping both skills' triggers intact for chaining.
|
||||
|
||||
## Completed
|
||||
|
||||
### ~~(v0.42.20.0 follow-up) Decouple the op-dispatch force-exit timer~~
|
||||
**Completed:** v0.42.39.0 (2026-06-10)
|
||||
|
||||
The timer now arms at teardown entry (inside the op-dispatch finally, before
|
||||
drain + disconnect) so it bounds ONLY disconnect — no longer doubling as a
|
||||
blanket handler watchdog that killed slow-but-healthy ops at 10s with exit 0
|
||||
and empty stdout. Its "engine.disconnect() did not return…" message is now
|
||||
accurate by construction (it can only fire during teardown). Read-scope
|
||||
handlers + context build got their own explicit wallclock bound (180s default,
|
||||
`--timeout=Ns`, exit 124, hard-exit after teardown) in the same wave. Pinned by
|
||||
`test/cli-force-exit-teardown-arming.test.ts`.
|
||||
|
||||
### ~~Checks 5 + 6 for check-resolvable~~
|
||||
**Completed:** v0.19.0 (2026-04-22)
|
||||
|
||||
|
||||
File diff suppressed because one or more lines are too long
@@ -43,8 +43,8 @@ genuinely has to change.
|
||||
Switching dimensions requires:
|
||||
|
||||
1. Dropping the HNSW vector index (pgvector won't survive an `ALTER COLUMN TYPE`).
|
||||
2. Altering the column type (Postgres only — PGLite cannot do this).
|
||||
3. Wiping every existing embedding (the old vectors are unusable in the new space).
|
||||
2. Wiping every existing embedding (the old vectors are unusable in the new space — and pgvector refuses to cast them across dimensions, so this must happen before the alter).
|
||||
3. Altering the column type (Postgres only — PGLite cannot do this).
|
||||
4. Re-embedding the entire corpus (can take hours on a 50K-page brain and costs $1-100 in API calls depending on model).
|
||||
5. Conditionally recreating the index (HNSW supports up to 2000 dimensions per pgvector; above that you must use exact scans).
|
||||
|
||||
@@ -115,12 +115,17 @@ BEGIN;
|
||||
-- 1. Drop the HNSW index. It can't survive the column type change.
|
||||
DROP INDEX IF EXISTS idx_chunks_embedding;
|
||||
|
||||
-- 2. Alter the column type.
|
||||
ALTER TABLE content_chunks ALTER COLUMN embedding TYPE vector(<NEW_DIMS>);
|
||||
|
||||
-- 3. Clear stale embeddings so they don't survive into the new space.
|
||||
-- 2. Clear stale embeddings FIRST. This must happen BEFORE the column
|
||||
-- alter: pgvector refuses to cast existing vectors across dimensions
|
||||
-- ("expected <NEW_DIMS> dimensions, not <OLD_DIMS>"), so altering a
|
||||
-- column that still holds old-width vectors aborts the transaction.
|
||||
-- NULLs cast fine. (The old vectors are unusable in the new space
|
||||
-- anyway — this is the wipe step from the rationale above.)
|
||||
UPDATE content_chunks SET embedding = NULL, embedded_at = NULL;
|
||||
|
||||
-- 3. Alter the column type (all rows are NULL now, so the cast succeeds).
|
||||
ALTER TABLE content_chunks ALTER COLUMN embedding TYPE vector(<NEW_DIMS>);
|
||||
|
||||
-- 4. Recreate the HNSW index ONLY IF dims <= 2000. Above that, leave it
|
||||
-- indexless and rely on exact scans (gbrain searchVector handles this
|
||||
-- automatically — search just gets slower, not broken).
|
||||
|
||||
@@ -58,7 +58,7 @@ Per-call knobs on the op/watch: `max_pages`, `min_confidence`, `session_id`,
|
||||
|
||||
## Storage + privacy
|
||||
|
||||
Volunteered pages log to `context_volunteer_events` (migration v116): slug,
|
||||
Volunteered pages log to `context_volunteer_events` (migration v117): slug,
|
||||
arm, confidence, channel, optional session/turn — the rationale is a
|
||||
deterministic template string, never raw conversation text. Rows are pruned
|
||||
after 90 days by the dream cycle's purge phase. Synopses pass through the same
|
||||
|
||||
+1
-1
@@ -143,5 +143,5 @@
|
||||
"bun": ">=1.3.10"
|
||||
},
|
||||
"license": "MIT",
|
||||
"version": "0.42.40.0"
|
||||
"version": "0.42.41.0"
|
||||
}
|
||||
|
||||
+101
-8
@@ -354,9 +354,91 @@ async function main() {
|
||||
|
||||
// Local engine path (unchanged behavior for local installs).
|
||||
const engine = await connectEngine();
|
||||
// v0.41.8.0 (#1247, #1269, #1290): the search / query / get_page
|
||||
// op handlers fire-and-forget `bumpLastRetrievedAt` after returning
|
||||
// results. On PGLite that IIFE keeps Bun's event loop alive past
|
||||
// engine.disconnect(), hanging the CLI at ~95-98% CPU until SIGKILL.
|
||||
// Drain the fire-and-forget set BEFORE disconnect; force-exit only
|
||||
// if the drain itself times out (preserves stderr diagnostic signal
|
||||
// AND guarantees the CLI doesn't re-hang at the disconnect layer).
|
||||
//
|
||||
// Defense-in-depth (adversarial-review C13): `engine.disconnect()` itself
|
||||
// can hang on PGLite (db.close() or releaseLock racing OS-level FS state).
|
||||
// The unref'd hard-exit fallback is armed inside drainThenDisconnect (called
|
||||
// from the `finally` below), so it bounds ONLY the teardown phase (drain +
|
||||
// disconnect) — the same helper every owner-disconnect site uses. It used to
|
||||
// be armed HERE, before the try, which silently killed any op whose BODY ran
|
||||
// past the deadline: on a slow Postgres pooler (6-10s per fresh connection)
|
||||
// a healthy `gbrain search` was force-exited mid-handler with code 0 and
|
||||
// ZERO stdout — an empty "success" indistinguishable from no results. The
|
||||
// exitCode honor (v0.42.20.0) can't help there: a mid-op kill fires before
|
||||
// any error path sets exitCode. Op-body wallclock bounds are the read-scope
|
||||
// withTimeout wrap inside the try below, not this teardown backstop.
|
||||
// Daemons (`serve`) are excluded so they stay alive.
|
||||
// Wallclock bound for READ-scope op handlers. With the hard-deadline timer
|
||||
// correctly scoped to teardown, a genuinely WEDGED read handler (hung pooler
|
||||
// connection mid-query) would otherwise hang the CLI forever — the #1633
|
||||
// zombie class the old (buggy) pre-try timer accidentally bounded at 10s.
|
||||
// 180s sits far above any healthy slow-pooler run (6-10s/connection);
|
||||
// --timeout=Ns overrides. Writes/admin stay unbounded: a long import/embed
|
||||
// must never be killed by a default deadline.
|
||||
const READ_OP_TIMEOUT_MS = 180_000;
|
||||
// Set when a wallclock bound fired. The abandoned (timed-out but still
|
||||
// running) handler can hold ref'd sockets/timers that keep Bun's event loop
|
||||
// alive after main() returns — so the finally must hard-exit after teardown
|
||||
// on this path, or the timeout print is followed by an immortal process:
|
||||
// the same zombie class, resurrected through the timeout door (adversarial
|
||||
// review finding).
|
||||
let wallclockTimedOut = false;
|
||||
|
||||
try {
|
||||
const ctx = await makeContext(engine, params);
|
||||
const rawResult = await op.handler(ctx, params);
|
||||
const { withTimeout, OperationTimeoutError } = await import('./core/timeout.ts');
|
||||
const wallclockMs = getCliOptions().timeoutMs ?? READ_OP_TIMEOUT_MS;
|
||||
const onWallclockTimeout = (e: InstanceType<typeof OperationTimeoutError>) => {
|
||||
const hint = getCliOptions().timeoutMs
|
||||
? ''
|
||||
: ` (default ${e.ms}ms; pass --timeout=Ns to override)`;
|
||||
console.error(`${e.label} timed out${hint}.`);
|
||||
setCliExitCode(124);
|
||||
wallclockTimedOut = true;
|
||||
};
|
||||
|
||||
// Context build does DB I/O (resolveSourceId) and runs for EVERY op —
|
||||
// a wedged pooler connection here would otherwise hang reads, writes,
|
||||
// and admin alike with no bound at all (adversarial review finding).
|
||||
let ctx: Awaited<ReturnType<typeof makeContext>>;
|
||||
try {
|
||||
ctx = await withTimeout(
|
||||
makeContext(engine, params),
|
||||
wallclockMs,
|
||||
`gbrain ${command}: context`,
|
||||
);
|
||||
} catch (e: unknown) {
|
||||
if (e instanceof OperationTimeoutError) {
|
||||
onWallclockTimeout(e);
|
||||
return; // the finally below still drains + disconnects, then exits
|
||||
}
|
||||
throw e;
|
||||
}
|
||||
|
||||
let rawResult: unknown;
|
||||
if (op.scope === 'read') {
|
||||
try {
|
||||
rawResult = await withTimeout(
|
||||
op.handler(ctx, params),
|
||||
wallclockMs,
|
||||
`gbrain ${command}`,
|
||||
);
|
||||
} catch (e: unknown) {
|
||||
if (e instanceof OperationTimeoutError) {
|
||||
onWallclockTimeout(e);
|
||||
return; // the finally below still drains + disconnects, then exits
|
||||
}
|
||||
throw e;
|
||||
}
|
||||
} else {
|
||||
rawResult = await op.handler(ctx, params);
|
||||
}
|
||||
// ENG-2 (renderer parity by data shape): JSON-round-trip the local-engine
|
||||
// path's return value so renderers see the same shape they'd see on the
|
||||
// routed path. Date → ISO string; bigint → string (postgres.js shape);
|
||||
@@ -369,7 +451,8 @@ async function main() {
|
||||
// STILL runs (drains every background-work sink + disconnects). A bare
|
||||
// process.exit(1) here would skip the finally → skip the drain + disconnect
|
||||
// (leaves facts/cache/eval-capture writes racing teardown). The finally's
|
||||
// drain bounds teardown; the outer hard-deadline timer bounds a hung one.
|
||||
// drain bounds teardown; the hard-deadline timer armed at teardown entry
|
||||
// bounds a hung one.
|
||||
if (e instanceof OperationError) {
|
||||
console.error(`Error [${e.code}]: ${e.message}`);
|
||||
if (e.suggestion) console.error(` Fix: ${e.suggestion}`);
|
||||
@@ -382,6 +465,15 @@ async function main() {
|
||||
// ~0ms fast path; capture/import that DO enqueue pay up to 1s (+ facts
|
||||
// shutdown grace) while in-flight Haiku finishes.
|
||||
await drainThenDisconnect(engine, { drainTimeoutMs: 1000 });
|
||||
// Wallclock-timeout path (master v0.42.41.0): the ABANDONED handler
|
||||
// (withTimeout races, it does not cancel) can hold ref'd sockets / SDK
|
||||
// retry timers that keep Bun's event loop alive. The entrypoint
|
||||
// flush-exit fires when main() resolves and covers this; the explicit
|
||||
// call here is belt-and-braces in case a future caller invokes this op
|
||||
// path outside the guarded entrypoint.
|
||||
if (wallclockTimedOut) {
|
||||
void flushStdoutThenExit(getCliExitCode());
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -400,11 +492,12 @@ async function main() {
|
||||
* │ busy-loop (#1762)
|
||||
* └── engine.disconnect() (best-effort), then clear the deadline
|
||||
*
|
||||
* The 10s timer is armed HERE — around the teardown window only — not before
|
||||
* the op handler (the pre-v0.43 placement would have force-killed any op
|
||||
* slower than 10s). On the happy path the timer never fires: main() resolves
|
||||
* and the entrypoint's flushStdoutThenExit ends the process deliberately
|
||||
* (#2084's fix for lingering embedding/PgBouncer sockets riding the backstop).
|
||||
* The 10s timer is armed HERE — around the teardown window only — never
|
||||
* before the op handler (master's v0.42.41.0 triage wave fixed the same
|
||||
* pre-armed-timer bug independently: it force-killed any op slower than
|
||||
* 10s). On the happy path the timer never fires: main() resolves and the
|
||||
* entrypoint's flushStdoutThenExit ends the process deliberately (#2084's
|
||||
* fix for lingering embedding/PgBouncer sockets riding the backstop).
|
||||
*/
|
||||
const DISCONNECT_HARD_DEADLINE_MS = 10_000;
|
||||
export async function drainThenDisconnect(
|
||||
|
||||
@@ -530,8 +530,13 @@ export async function runAutopilot(engine: BrainEngine, args: string[]) {
|
||||
autopilotReconnectFails = 0; // reset on success
|
||||
} catch (probeErr) {
|
||||
try {
|
||||
await engine.disconnect();
|
||||
await (engine as any).connect?.();
|
||||
// #2034: use reconnect() — it restores the config captured at connect()
|
||||
// and avoids the null-connection window. The previous
|
||||
// `disconnect()` + bare `connect()` lost the config (throwing
|
||||
// `database_url undefined` on every retry → FATAL restart-loop on any
|
||||
// transient DB blip) AND tore down the pool postgres.js can otherwise
|
||||
// self-heal.
|
||||
await engine.reconnect({ error: probeErr });
|
||||
autopilotReconnectFails = 0;
|
||||
} catch (e) {
|
||||
logError('reconnect', e);
|
||||
|
||||
+94
-11
@@ -510,6 +510,33 @@ export async function doctorReportRemote(engine: BrainEngine): Promise<DoctorRep
|
||||
checks.push({ name: 'schema_version', status: 'warn', message: 'Could not check schema version' });
|
||||
}
|
||||
|
||||
// 2b. #2038: idx_timeline_dedup shape. A renumbered-during-merge migration
|
||||
// (v102) can be recorded-as-applied without its DDL running, leaving the
|
||||
// 3-column index in place — every timeline write then fails the 4-column
|
||||
// ON CONFLICT. The version counter can't see this, so check the index SHAPE.
|
||||
try {
|
||||
const { checkTimelineDedupIndex } = await import('../core/timeline-dedup-repair.ts');
|
||||
const idx = await checkTimelineDedupIndex(engine);
|
||||
if (!idx.tablePresent || !idx.needsRepair) {
|
||||
checks.push({
|
||||
name: 'timeline_dedup_index',
|
||||
status: 'ok',
|
||||
message: idx.tablePresent ? 'idx_timeline_dedup has the 4-column shape' : 'no timeline_entries table yet',
|
||||
});
|
||||
} else {
|
||||
checks.push({
|
||||
name: 'timeline_dedup_index',
|
||||
status: 'fail',
|
||||
message:
|
||||
`idx_timeline_dedup is ${idx.indexPresent ? `(${idx.columns.join(', ')})` : 'absent'}, ` +
|
||||
`expected (page_id, date, summary, source) — timeline writes are failing (#2038). ` +
|
||||
`Run \`gbrain apply-migrations --force-schema\` to heal it.`,
|
||||
});
|
||||
}
|
||||
} catch {
|
||||
checks.push({ name: 'timeline_dedup_index', status: 'warn', message: 'Could not check idx_timeline_dedup shape' });
|
||||
}
|
||||
|
||||
// 3. Brain score
|
||||
try {
|
||||
const health = await engine.getHealth();
|
||||
@@ -2385,8 +2412,14 @@ export async function checkStaleLocks(
|
||||
}
|
||||
const lines = stale.slice(0, 10).map(s => {
|
||||
const ageH = Math.floor(s.age_ms / 3600_000);
|
||||
const source = s.id.startsWith('gbrain-sync:') ? s.id.slice('gbrain-sync:'.length) : null;
|
||||
const breakHint = source ? `gbrain sync --break-lock --source ${source}` : `gbrain sync --break-lock`;
|
||||
let breakHint = 'gbrain doctor';
|
||||
if (s.id.startsWith('gbrain-sync:')) {
|
||||
breakHint = `gbrain sync --break-lock --source ${s.id.slice('gbrain-sync:'.length)}`;
|
||||
} else if (s.id.startsWith('gbrain-cycle:')) {
|
||||
breakHint = `gbrain dream --break-lock --source ${s.id.slice('gbrain-cycle:'.length)}`;
|
||||
} else if (s.id === 'gbrain-cycle') {
|
||||
breakHint = 'gbrain dream --break-lock';
|
||||
}
|
||||
return ` ${s.id} (pid ${s.holder_pid} on ${s.holder_host}, age ${ageH}h) → ${breakHint}`;
|
||||
});
|
||||
const tail = stale.length > 10 ? ` ... and ${stale.length - 10} more.` : null;
|
||||
@@ -2614,7 +2647,7 @@ async function checkEmbeddingEnvOverride(engine: BrainEngine): Promise<Check> {
|
||||
};
|
||||
}
|
||||
|
||||
async function checkSubagentCapability(engine: BrainEngine): Promise<Check> {
|
||||
export async function checkSubagentCapability(engine: BrainEngine): Promise<Check> {
|
||||
try {
|
||||
const { classifyCapabilities } = await import('../core/ai/capabilities.ts');
|
||||
const tierSubagent = await engine.getConfig('models.tier.subagent');
|
||||
@@ -2675,8 +2708,11 @@ async function checkSubagentCapability(engine: BrainEngine): Promise<Check> {
|
||||
const { loadConfig } = await import('../core/config.ts');
|
||||
const cfg = loadConfig();
|
||||
const chatModel = cfg?.chat_model;
|
||||
const gatewayLoopRaw = await engine.getConfig('agent.use_gateway_loop').catch(() => null);
|
||||
const gatewayLoopEnabled = typeof gatewayLoopRaw === 'string'
|
||||
&& ['true', '1', 'yes', 'on'].includes(gatewayLoopRaw.trim().toLowerCase());
|
||||
const { isAnthropicProvider } = await import('../core/model-config.ts');
|
||||
if (chatModel && !isAnthropicProvider(chatModel) && !process.env.ANTHROPIC_API_KEY) {
|
||||
if (chatModel && !isAnthropicProvider(chatModel) && !process.env.ANTHROPIC_API_KEY && !gatewayLoopEnabled) {
|
||||
return {
|
||||
name: 'subagent_capability',
|
||||
status: 'warn',
|
||||
@@ -5610,8 +5646,29 @@ export async function buildChecks(
|
||||
"SELECT COUNT(*)::int AS count FROM pages WHERE type IN ('entity', 'person', 'company', 'organization')",
|
||||
))[0]?.count ?? 0;
|
||||
|
||||
const linkPct = ((health.link_coverage ?? 0) * 100).toFixed(0);
|
||||
const timelinePct = ((health.timeline_coverage ?? 0) * 100).toFixed(0);
|
||||
// Compute coverage against eligible entities only — exclude test fixtures
|
||||
// (`tools/gbrain/test/*`) and template stubs (`templates/new-person`) so
|
||||
// that brains seeded only with code sources don't get spurious warnings
|
||||
// about missing link/timeline coverage on pages that are test fixtures, not
|
||||
// real knowledge entities.
|
||||
const eligibleStats = (await engine.executeRaw<{ entities: number; linked_from: number; timeline: number }>(
|
||||
`WITH eligible AS (
|
||||
SELECT id FROM pages
|
||||
WHERE type IN ('entity','person','company','organization')
|
||||
AND slug NOT LIKE 'tools/gbrain/test/%'
|
||||
AND slug <> 'templates/new-person'
|
||||
)
|
||||
SELECT
|
||||
(SELECT count(*)::int FROM eligible) AS entities,
|
||||
(SELECT count(DISTINCT from_page_id)::int FROM links WHERE from_page_id IN (SELECT id FROM eligible)) AS linked_from,
|
||||
(SELECT count(DISTINCT page_id)::int FROM timeline_entries WHERE page_id IN (SELECT id FROM eligible)) AS timeline`,
|
||||
))[0] ?? { entities: entityCount, linked_from: 0, timeline: 0 };
|
||||
|
||||
const eligibleEntityCount = Number(eligibleStats.entities ?? entityCount);
|
||||
const linkCoverage = eligibleEntityCount > 0 ? Number(eligibleStats.linked_from ?? 0) / eligibleEntityCount : 0;
|
||||
const timelineCoverage = eligibleEntityCount > 0 ? Number(eligibleStats.timeline ?? 0) / eligibleEntityCount : 0;
|
||||
const linkPct = (linkCoverage * 100).toFixed(0);
|
||||
const timelinePct = (timelineCoverage * 100).toFixed(0);
|
||||
if (entityCount === 0) {
|
||||
// Markdown-only / journal / wiki brain — no entity pages to compute
|
||||
// coverage against. Coverage formula is structurally inapplicable.
|
||||
@@ -5620,13 +5677,19 @@ export async function buildChecks(
|
||||
status: 'ok',
|
||||
message: 'No entity pages — graph_coverage not applicable (markdown-only brain)',
|
||||
});
|
||||
} else if ((health.link_coverage ?? 0) >= 0.5 && (health.timeline_coverage ?? 0) >= 0.5) {
|
||||
} else if (eligibleEntityCount === 0) {
|
||||
checks.push({
|
||||
name: 'graph_coverage',
|
||||
status: 'ok',
|
||||
message: `Only code/test fixture entity pages found (${entityCount}); graph_coverage not applicable`,
|
||||
});
|
||||
} else if (linkCoverage >= 0.5 && timelineCoverage >= 0.5) {
|
||||
checks.push({ name: 'graph_coverage', status: 'ok', message: `Entity link coverage ${linkPct}%, timeline ${timelinePct}%` });
|
||||
} else {
|
||||
checks.push({
|
||||
name: 'graph_coverage',
|
||||
status: 'warn',
|
||||
message: `Entity link coverage ${linkPct}%, timeline ${timelinePct}% (${entityCount} entity pages). Run: gbrain extract all`,
|
||||
message: `Entity link coverage ${linkPct}%, timeline ${timelinePct}% (${eligibleEntityCount} entity pages). Run: gbrain extract all`,
|
||||
});
|
||||
}
|
||||
|
||||
@@ -6162,12 +6225,29 @@ export async function buildChecks(
|
||||
.slice(0, 3)
|
||||
.map(([s, n]) => `${s}=${n}`)
|
||||
.join(', ');
|
||||
// Audit events are evidence, not automatically breakage. A large code
|
||||
// source can legitimately emit many WARN events (oversize/markup-heavy)
|
||||
// while remaining searchable and intentionally flagged. Fail on hard
|
||||
// dispositions (content actually blocked or hidden); warn on soft
|
||||
// dispositions or volume. This keeps doctor from treating expected
|
||||
// code-corpus telemetry as an unhealthy brain.
|
||||
//
|
||||
// v0.42 renamed the hard path: a rejected page emits `reject` and a
|
||||
// quarantined (hidden) junk page emits `quarantine`; `hard_block` is now
|
||||
// only the pre-v0.42 legacy alias. Counting `hard_block` alone let fresh
|
||||
// junk-ingest evidence (`reject`/`quarantine`) clear as `ok` whenever
|
||||
// fewer than 10 events landed. `flag` is a warn disposition (still
|
||||
// searchable, agent warned on retrieval), so it joins `soft_block`.
|
||||
const hardBlocked =
|
||||
summary.by_type.hard_block + summary.by_type.reject + summary.by_type.quarantine;
|
||||
const softBlocked = summary.by_type.soft_block + summary.by_type.flag;
|
||||
const status: 'ok' | 'warn' | 'fail' =
|
||||
events.length >= 100 ? 'fail' : events.length >= 10 ? 'warn' : 'ok';
|
||||
hardBlocked > 0 ? 'fail' :
|
||||
(softBlocked > 0 || events.length >= 10) ? 'warn' : 'ok';
|
||||
checks.push({
|
||||
name: 'content_sanity_audit_recent',
|
||||
status,
|
||||
message: `${events.length} events (hard=${summary.by_type.hard_block} soft=${summary.by_type.soft_block} warn=${summary.by_type.warn})${topPatterns ? ', patterns: ' + topPatterns : ''}${topSources ? ', sources: ' + topSources : ''}. (Local audit only — multi-host operators set GBRAIN_AUDIT_DIR.)`,
|
||||
message: `${events.length} events (hard=${hardBlocked} [hard_block=${summary.by_type.hard_block} reject=${summary.by_type.reject} quarantine=${summary.by_type.quarantine}] soft=${softBlocked} [soft_block=${summary.by_type.soft_block} flag=${summary.by_type.flag}] warn=${summary.by_type.warn})${topPatterns ? ', patterns: ' + topPatterns : ''}${topSources ? ', sources: ' + topSources : ''}. (Local audit only — multi-host operators set GBRAIN_AUDIT_DIR.)`,
|
||||
});
|
||||
}
|
||||
} catch (err) {
|
||||
@@ -7145,7 +7225,10 @@ export async function runDoctor(
|
||||
} catch { /* best-effort */ }
|
||||
}
|
||||
|
||||
process.exit(hasFail ? 1 : 0);
|
||||
// Use process.exitCode instead of process.exit() so cleanup handlers
|
||||
// (e.g. Bun unload events, open database connections) still run before
|
||||
// the process terminates. process.exit() is a hard kill that bypasses them.
|
||||
process.exitCode = hasFail ? 1 : 0;
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
@@ -855,6 +855,17 @@ Status (v0.42):
|
||||
if (!jsonMode) {
|
||||
console.log(`Timeline from meetings: ${r.entries_created} entries on ${r.entities_touched} entity pages from ${r.meetings_scanned} meetings`);
|
||||
}
|
||||
// #2057 (codex): batch failures are no longer swallowed silently — make
|
||||
// them visible at the command surface (and non-zero exit) instead of
|
||||
// printing a clean "N entries" success over failed inserts.
|
||||
if (r.batch_errors > 0) {
|
||||
console.error(
|
||||
`[extract timeline] ${r.batch_errors} batch(es) failed to insert` +
|
||||
(r.first_batch_error ? ` (first error: ${r.first_batch_error})` : '') +
|
||||
` — timeline is incomplete.`,
|
||||
);
|
||||
process.exitCode = 1;
|
||||
}
|
||||
} else if (byMention || ner) {
|
||||
// v0.41.18.0 (T7): combined --by-mention + --ner walk shares one
|
||||
// gazetteer; saves an entire pass on big brains. When only one
|
||||
|
||||
+64
-5
@@ -11,6 +11,7 @@ import {
|
||||
isCodeFilePath,
|
||||
isMarkdownFilePath,
|
||||
isImageFilePath as isImageFilePathFromSync,
|
||||
pruneDir,
|
||||
type SyncStrategy,
|
||||
} from '../core/sync.ts';
|
||||
import { sortNewestFirst } from '../core/sort-newest-first.ts';
|
||||
@@ -512,6 +513,51 @@ function isCollectibleForWalker(
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Git-aware fast path for `collectSyncableFiles`. Returns the strategy-filtered
|
||||
* list of syncable files when `dir` is inside a git work tree (paths absolute,
|
||||
* sorted), or `null` when `dir` is not a git repo / git is unavailable — in
|
||||
* which case the caller falls back to the recursive FS walk.
|
||||
*
|
||||
* Honors `.gitignore` (the whole point): `git ls-files --cached --others
|
||||
* --exclude-standard` lists tracked + untracked-not-ignored files, so vendored
|
||||
* / build / generated trees never reach the importer. `-z` (NUL-delimited)
|
||||
* survives paths with spaces/newlines. Each path is lstat-checked to preserve
|
||||
* the walker's no-symlink policy and to drop submodule gitlinks (which surface
|
||||
* as a single non-regular entry).
|
||||
*/
|
||||
function gitListSyncableFiles(
|
||||
dir: string,
|
||||
strategy: SyncStrategy,
|
||||
multimodalOn: boolean,
|
||||
): string[] | null {
|
||||
let stdout: string;
|
||||
try {
|
||||
stdout = execFileSync(
|
||||
'git',
|
||||
['-C', dir, 'ls-files', '--cached', '--others', '--exclude-standard', '-z'],
|
||||
{ encoding: 'utf8', maxBuffer: 512 * 1024 * 1024, stdio: ['ignore', 'pipe', 'ignore'] },
|
||||
);
|
||||
} catch {
|
||||
return null; // not a git work tree, or git not on PATH → FS-walk fallback
|
||||
}
|
||||
const files: string[] = [];
|
||||
for (const rel of stdout.split('\0')) {
|
||||
if (!rel) continue;
|
||||
if (!isCollectibleForWalker(rel, strategy, multimodalOn)) continue;
|
||||
const full = join(dir, rel);
|
||||
let st;
|
||||
try {
|
||||
st = lstatSync(full);
|
||||
} catch {
|
||||
continue; // ls-files raced a deletion, or unreadable
|
||||
}
|
||||
if (st.isSymbolicLink() || !st.isFile()) continue;
|
||||
files.push(full);
|
||||
}
|
||||
return files.sort();
|
||||
}
|
||||
|
||||
/**
|
||||
* v0.31.2 (codex C4 + C5 + C8): unified walker with five hardenings:
|
||||
*
|
||||
@@ -532,6 +578,19 @@ function isCollectibleForWalker(
|
||||
export function collectSyncableFiles(dir: string, opts: CollectOpts = {}): string[] {
|
||||
const strategy: SyncStrategy = opts.strategy ?? 'markdown';
|
||||
const multimodalOn = process.env.GBRAIN_EMBEDDING_MULTIMODAL === 'true';
|
||||
|
||||
// v0.42.x (#1159 --respect-gitignore / #1483 .gbrainignore): when `dir` is a
|
||||
// git work tree, enumerate via `git ls-files` so the walk honors
|
||||
// `.gitignore`. Pre-fix the recursive FS walk below descended into every
|
||||
// git-ignored tree — `vendor/` (PHP Composer), `storage/`, `public/build/`,
|
||||
// etc. — so a Laravel/PHP repo's `--strategy code` sync tried to import ~50k
|
||||
// dependency/build files (and bloated DB + embedding cost on any repo with
|
||||
// vendored data/fixtures). `--cached --others --exclude-standard` = tracked
|
||||
// PLUS untracked-not-ignored, so uncommitted source is still indexed. Non-git
|
||||
// dirs (or git unavailable) fall through to the FS walk below.
|
||||
const gitFiles = gitListSyncableFiles(dir, strategy, multimodalOn);
|
||||
if (gitFiles) return gitFiles;
|
||||
|
||||
const maxDepth = resolveMaxWalkDepth();
|
||||
const visitedInodes = new Map<string, true>();
|
||||
const files: string[] = [];
|
||||
@@ -548,11 +607,11 @@ export function collectSyncableFiles(dir: string, opts: CollectOpts = {}): strin
|
||||
return;
|
||||
}
|
||||
for (const entry of entries) {
|
||||
// Skip hidden dirs (.git, .claude, .raw, etc.) and `node_modules`/`ops`.
|
||||
// Same set the legacy walkers honored, surfaced once at the top of
|
||||
// every iteration.
|
||||
if (entry.startsWith('.')) continue;
|
||||
if (entry === 'node_modules' || entry === 'ops') continue;
|
||||
// Descent-time prune through the canonical gate (single source of truth
|
||||
// in core/sync.ts) instead of a hand-maintained inline list that drifted
|
||||
// from it. Skips hidden dirs (`.git`, `.raw`, etc.), `node_modules`,
|
||||
// `vendor`, `dist`, `build`, `venv` (#2020), `ops`, and git submodules.
|
||||
if (!pruneDir(entry, d)) continue;
|
||||
|
||||
const full = join(d, entry);
|
||||
let stat;
|
||||
|
||||
@@ -6,7 +6,7 @@ import { homedir } from 'os';
|
||||
|
||||
const __filename = fileURLToPath(import.meta.url);
|
||||
const __dirname = dirname(__filename);
|
||||
import { saveConfig, loadConfig, loadConfigFileOnly, toEngineConfig, gbrainPath, configPath, isThinClient, type GBrainConfig } from '../core/config.ts';
|
||||
import { saveConfig, loadConfig, loadConfigFileOnly, toEngineConfig, gbrainPath, configPath, isThinClient, effectiveEnvDatabaseUrl, type GBrainConfig } from '../core/config.ts';
|
||||
import { createEngine } from '../core/engine-factory.ts';
|
||||
import { discoverOAuth, mintClientCredentialsToken, smokeTestMcp } from '../core/remote-mcp-probe.ts';
|
||||
import { runInitEmbedCheck } from '../core/init-embed-check.ts';
|
||||
@@ -133,7 +133,11 @@ export async function runInit(args: string[]) {
|
||||
if (manualUrl) {
|
||||
databaseUrl = manualUrl;
|
||||
} else if (isNonInteractive) {
|
||||
const envUrl = process.env.GBRAIN_DATABASE_URL || process.env.DATABASE_URL;
|
||||
// effectiveEnvDatabaseUrl applies the #427 guard: a DATABASE_URL that Bun
|
||||
// auto-loaded from a .env in cwd must not seed a brain config — that is
|
||||
// the "init inside a web-app checkout writes the app's DB into
|
||||
// ~/.gbrain/config.json" failure mode.
|
||||
const envUrl = effectiveEnvDatabaseUrl();
|
||||
if (envUrl) {
|
||||
databaseUrl = envUrl;
|
||||
} else {
|
||||
|
||||
@@ -8,7 +8,7 @@
|
||||
*/
|
||||
|
||||
import { createEngine } from '../core/engine-factory.ts';
|
||||
import { loadConfig, saveConfig, toEngineConfig, gbrainPath, type GBrainConfig } from '../core/config.ts';
|
||||
import { loadConfig, saveConfig, toEngineConfig, gbrainPath, effectiveEnvDatabaseUrl, type GBrainConfig } from '../core/config.ts';
|
||||
import type { BrainEngine } from '../core/engine.ts';
|
||||
import type { EngineConfig } from '../core/types.ts';
|
||||
import { writeFileSync, readFileSync, existsSync, unlinkSync } from 'fs';
|
||||
@@ -91,7 +91,8 @@ export async function runMigrateEngine(sourceEngine: BrainEngine, args: string[]
|
||||
// Build target config
|
||||
const targetConfig: EngineConfig = { engine: opts.targetEngine };
|
||||
if (opts.targetEngine === 'postgres') {
|
||||
targetConfig.database_url = opts.targetUrl || process.env.GBRAIN_DATABASE_URL || process.env.DATABASE_URL;
|
||||
// #427 guard: don't let a cwd-.env DATABASE_URL become a migration target.
|
||||
targetConfig.database_url = opts.targetUrl || effectiveEnvDatabaseUrl();
|
||||
if (!targetConfig.database_url) {
|
||||
console.error('Target is Supabase but no connection string provided. Use: --url <connection_string>');
|
||||
process.exit(1);
|
||||
|
||||
+102
-7
@@ -797,6 +797,26 @@ export function isAvailable(touchpoint: TouchpointKind, modelOverride?: string):
|
||||
|
||||
// ---- Embedding ----
|
||||
|
||||
/**
|
||||
* Carries the asymmetric-embedding `input_type` ('query' | 'document')
|
||||
* across the AI SDK boundary, from embedSubBatch() to the per-recipe fetch
|
||||
* shims below (#1400).
|
||||
*
|
||||
* Why this exists: `dimsProviderOptions()` correctly emits `input_type`
|
||||
* into `providerOptions.openaiCompatible` for asymmetric models (ZE
|
||||
* zembed-1, Voyage v3+), but the AI SDK's openai-compatible adapter
|
||||
* validates providerOptions against a fixed schema (`dimensions`, `user`)
|
||||
* and silently DROPS every other field before building the wire body. By
|
||||
* the time a compat shim sees the request, the threaded 'query' is gone —
|
||||
* so every embedQuery() was encoded document-side and asymmetric retrieval
|
||||
* silently collapsed to surface-token overlap.
|
||||
*
|
||||
* embedSubBatch() populates the store only when providerOptions actually
|
||||
* threads an input_type; shims that need a default (ZE) apply their own.
|
||||
* Same module-level ALS pattern as `__budgetStore` below.
|
||||
*/
|
||||
const __embedInputTypeStore = new AsyncLocalStorage<'query' | 'document'>();
|
||||
|
||||
/**
|
||||
* Voyage AI compatibility shim. Voyage's `/v1/embeddings` endpoint is OpenAI-shaped
|
||||
* but diverges on two parameters:
|
||||
@@ -835,6 +855,15 @@ const voyageCompatFetch = (async (input: RequestInfo | URL, init?: RequestInit)
|
||||
if (typeof dims === 'number') parsed.output_dimension = dims;
|
||||
mutated = true;
|
||||
}
|
||||
// Recover the SDK-stripped input_type (#1400) — opt-in, mirroring
|
||||
// the dims.ts Voyage branch: inject only when a caller actually
|
||||
// threaded one; when absent, the field stays off the wire
|
||||
// (pre-v0.35.0.0 behavior preserved).
|
||||
const threadedInputType = __embedInputTypeStore.getStore();
|
||||
if (threadedInputType !== undefined && parsed.input_type === undefined) {
|
||||
parsed.input_type = threadedInputType;
|
||||
mutated = true;
|
||||
}
|
||||
if (mutated) {
|
||||
const newBody = JSON.stringify(parsed);
|
||||
// Drop Content-Length so fetch recomputes from the new body.
|
||||
@@ -1005,10 +1034,13 @@ const zeroEntropyCompatFetch = (async (input: RequestInfo | URL, init?: RequestI
|
||||
parsed.encoding_format = 'float';
|
||||
mutated = true;
|
||||
}
|
||||
// Default input_type when caller didn't thread one (document-side
|
||||
// embedding is the correct default for ingest paths).
|
||||
// Recover the SDK-stripped input_type (#1400): the threaded value
|
||||
// arrives via __embedInputTypeStore because the openai-compatible
|
||||
// adapter drops it from providerOptions before building the body.
|
||||
// Document-side default preserved for paths that didn't thread one
|
||||
// (ingest).
|
||||
if (parsed.input_type === undefined) {
|
||||
parsed.input_type = 'document';
|
||||
parsed.input_type = __embedInputTypeStore.getStore() ?? 'document';
|
||||
mutated = true;
|
||||
}
|
||||
if (mutated) {
|
||||
@@ -1096,6 +1128,56 @@ const zeroEntropyCompatFetch = (async (input: RequestInfo | URL, init?: RequestI
|
||||
}
|
||||
}) as unknown as typeof fetch;
|
||||
|
||||
/**
|
||||
* Generic asymmetric-embedding shim for openai-compatible recipes that
|
||||
* ship no compat fetch of their own (llama-server, litellm, ollama, ...).
|
||||
* Those deployments are the canonical local/proxy paths for serving
|
||||
* asymmetric models (e.g. a zembed-1 behind llama.cpp, vLLM, or an
|
||||
* OpenAI-compatible proxy), and they need the same `input_type` signal the
|
||||
* hosted ZE endpoint gets — the serving layer can't apply query-side vs
|
||||
* document-side encoding without it.
|
||||
*
|
||||
* Strictly opt-in at the wire level: when no input_type was threaded (the
|
||||
* model isn't asymmetric — dims.ts threads it by model id, never for
|
||||
* Azure/DashScope/Zhipu-style fixed models — or the caller didn't ask),
|
||||
* the request passes through byte-identical. When one WAS threaded
|
||||
* (recovered from __embedInputTypeStore — see #1400), inject it into the
|
||||
* JSON body; OpenAI-compatible servers that don't understand the field
|
||||
* ignore unknown body keys (llama-server does), and asymmetric-aware
|
||||
* servers/proxies read it. URL + response shape are untouched — these
|
||||
* endpoints are already OpenAI-shaped. Recipes with their own compat
|
||||
* fetch (voyage, zeroentropyai, azure) never reach this shim: the
|
||||
* `compat.fetch ??` precedence in instantiateEmbedding wins.
|
||||
*/
|
||||
const openAICompatAsymmetricFetch = (async (input: RequestInfo | URL, init?: RequestInit) => {
|
||||
const threadedInputType = __embedInputTypeStore.getStore();
|
||||
if (threadedInputType === undefined) return fetch(input as any, init);
|
||||
// Inject only on the string-URL + string-body shape the AI SDK actually
|
||||
// sends. A Request-shaped input may carry its body on the Request object
|
||||
// itself, where a rebuild would silently drop it — pass it through
|
||||
// untouched instead (unreachable via the SDK today; preserves the
|
||||
// byte-identical contract if it ever becomes reachable).
|
||||
if (typeof input !== 'string' && !(input instanceof URL)) {
|
||||
return fetch(input as any, init);
|
||||
}
|
||||
let baseInit: RequestInit = init ?? {};
|
||||
if (baseInit.body && typeof baseInit.body === 'string') {
|
||||
try {
|
||||
const parsed = JSON.parse(baseInit.body);
|
||||
if (parsed && typeof parsed === 'object' && parsed.input_type === undefined) {
|
||||
parsed.input_type = threadedInputType;
|
||||
// Drop Content-Length so fetch recomputes from the new body.
|
||||
const headers = new Headers(baseInit.headers ?? {});
|
||||
headers.delete('content-length');
|
||||
baseInit = { ...baseInit, body: JSON.stringify(parsed), headers };
|
||||
}
|
||||
} catch {
|
||||
// Body wasn't JSON — pass through untouched.
|
||||
}
|
||||
}
|
||||
return fetch(typeof input === 'string' ? input : input.toString(), baseInit);
|
||||
}) as unknown as typeof fetch;
|
||||
|
||||
async function resolveEmbeddingProvider(modelStr: string): Promise<{ model: any; recipe: Recipe; modelId: string }> {
|
||||
const { parsed, recipe } = resolveRecipe(modelStr);
|
||||
assertTouchpoint(recipe, 'embedding', parsed.modelId, getExtendedModelsForProvider(parsed.providerId));
|
||||
@@ -1149,15 +1231,19 @@ function instantiateEmbedding(recipe: Recipe, modelId: string, cfg: AIGatewayCon
|
||||
// wrapper via resolveOpenAICompatConfig. Azure recipes ship their own
|
||||
// fetch (api-version splice); voyage doesn't — use voyageCompatFetch.
|
||||
// ZeroEntropy needs zeroEntropyCompatFetch (URL path + body input_type
|
||||
// + response shape rewrite + OOM caps). Same per-recipe-id branch
|
||||
// pattern as voyage so adding a third compat shim is one more case.
|
||||
// + response shape rewrite + OOM caps). Every other openai-compat
|
||||
// recipe (llama-server, litellm, ollama, ...) falls through to
|
||||
// openAICompatAsymmetricFetch so the threaded input_type reaches
|
||||
// asymmetric models there too (#1400) — a strict pass-through when no
|
||||
// input_type was threaded, so symmetric deployments see zero wire
|
||||
// change.
|
||||
const fetchWrapper =
|
||||
compat.fetch ??
|
||||
(recipe.id === 'voyage'
|
||||
? voyageCompatFetch
|
||||
: recipe.id === 'zeroentropyai'
|
||||
? zeroEntropyCompatFetch
|
||||
: undefined);
|
||||
: openAICompatAsymmetricFetch);
|
||||
const client = createOpenAICompatible({
|
||||
name: recipe.id,
|
||||
baseURL: compat.baseURL,
|
||||
@@ -1472,7 +1558,7 @@ async function embedSubBatch(
|
||||
opts?: EmbedOpts,
|
||||
): Promise<Float32Array[]> {
|
||||
try {
|
||||
const result = await _embedTransport({
|
||||
const callTransport = () => _embedTransport({
|
||||
model,
|
||||
values: texts,
|
||||
providerOptions: providerOpts,
|
||||
@@ -1483,6 +1569,15 @@ async function embedSubBatch(
|
||||
abortSignal: withDefaultTimeout(opts?.abortSignal, AI_EMBED_TIMEOUT_MS),
|
||||
...(opts?.maxRetries !== undefined && { maxRetries: opts.maxRetries }),
|
||||
});
|
||||
// Carry the threaded input_type across the SDK boundary via
|
||||
// __embedInputTypeStore (the adapter strips it from providerOptions —
|
||||
// see the store's doc comment). Populated only when dimsProviderOptions
|
||||
// actually emitted one, so non-asymmetric paths run store-empty and the
|
||||
// fetch shims leave their wire bodies untouched.
|
||||
const threadedInputType = providerOpts?.openaiCompatible?.input_type;
|
||||
const result = await (threadedInputType === 'query' || threadedInputType === 'document'
|
||||
? __embedInputTypeStore.run(threadedInputType, callTransport)
|
||||
: callTransport());
|
||||
|
||||
if (!Array.isArray(result.embeddings) || result.embeddings.length !== texts.length) {
|
||||
throw new AIConfigError(
|
||||
|
||||
+98
-3
@@ -397,6 +397,96 @@ export function loadConfigFileOnly(): GBrainConfig | null {
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* #427 guard — DATABASE_URL hijack via Bun's cwd .env auto-load.
|
||||
*
|
||||
* Bun merges `.env` files from the process cwd into process.env before any
|
||||
* user code runs. For a globally-installed tool that is a footgun: running
|
||||
* gbrain inside any checkout whose `.env` defines DATABASE_URL (Next.js,
|
||||
* Hono, Supabase, most web apps) silently retargets the brain at that app's
|
||||
* database. Reads hit the wrong DB; `apply-migrations` can write gbrain's
|
||||
* schema — including its DDL event trigger — into a production app database
|
||||
* (see the v0.42.8 report on #427).
|
||||
*
|
||||
* Bun gives no way to ask which vars came from a .env file (the merge
|
||||
* happens before module load), so we re-parse the .env files Bun auto-loads
|
||||
* from cwd and treat DATABASE_URL as "not operator-provided" when its value
|
||||
* matches one of them. Deliberate overrides still work two ways:
|
||||
* - GBRAIN_DATABASE_URL: namespaced to this tool, never auto-ignored;
|
||||
* - exporting DATABASE_URL in the shell: exported vars win over .env in
|
||||
* Bun, and a deliberate export that happens to EQUAL the cwd .env value
|
||||
* would have selected the same database anyway — ignoring it changes
|
||||
* the outcome only by honoring the brain config, which is the safe
|
||||
* reading of ambiguous intent.
|
||||
*
|
||||
* The file list is a superset of Bun's auto-load set across NODE_ENV values
|
||||
* so the guard doesn't depend on replicating Bun's exact selection logic.
|
||||
*/
|
||||
const CWD_DOTENV_FILES = ['.env', '.env.local', '.env.development', '.env.production', '.env.test'];
|
||||
|
||||
/**
|
||||
* All values assigned to `key` across the .env files in `dir`. Collecting
|
||||
* every assignment (rather than emulating override order) keeps the guard
|
||||
* independent of dotenv precedence rules — a match against ANY assignment
|
||||
* means the value is file-origin. Exported for tests.
|
||||
*/
|
||||
export function dotenvValuesForKey(key: string, dir: string = process.cwd()): Set<string> {
|
||||
const values = new Set<string>();
|
||||
const assignment = /^(?:export\s+)?([A-Za-z_][A-Za-z0-9_]*)\s*=\s*(.*)$/;
|
||||
for (const name of CWD_DOTENV_FILES) {
|
||||
let content: string;
|
||||
try {
|
||||
content = readFileSync(join(dir, name), 'utf-8');
|
||||
} catch {
|
||||
continue; // missing/unreadable file — nothing to guard against
|
||||
}
|
||||
for (const rawLine of content.split('\n')) {
|
||||
const line = rawLine.trim();
|
||||
if (!line || line.startsWith('#')) continue;
|
||||
const m = line.match(assignment);
|
||||
if (!m || m[1] !== key) continue;
|
||||
let v = m[2].trim();
|
||||
if ((v.startsWith('"') && v.endsWith('"') && v.length >= 2) ||
|
||||
(v.startsWith("'") && v.endsWith("'") && v.length >= 2)) {
|
||||
v = v.slice(1, -1);
|
||||
} else {
|
||||
const hash = v.indexOf(' #');
|
||||
if (hash !== -1) v = v.slice(0, hash).trim();
|
||||
}
|
||||
if (v) values.add(v);
|
||||
}
|
||||
}
|
||||
return values;
|
||||
}
|
||||
|
||||
let warnedCwdEnvDbUrlIgnored = false;
|
||||
|
||||
/**
|
||||
* The env-provided DB URL gbrain should honor, with the #427 guard applied:
|
||||
* a DATABASE_URL whose value matches an assignment in a cwd .env file is
|
||||
* treated as belonging to the project in cwd, not to gbrain, and ignored
|
||||
* with a one-time stderr notice. GBRAIN_DATABASE_URL is always honored.
|
||||
* `dir` is injectable for tests; callers use the default.
|
||||
*/
|
||||
export function effectiveEnvDatabaseUrl(dir: string = process.cwd()): string | undefined {
|
||||
if (process.env.GBRAIN_DATABASE_URL) return process.env.GBRAIN_DATABASE_URL;
|
||||
const url = process.env.DATABASE_URL;
|
||||
if (!url) return undefined;
|
||||
if (dotenvValuesForKey('DATABASE_URL', dir).has(url)) {
|
||||
if (!warnedCwdEnvDbUrlIgnored) {
|
||||
warnedCwdEnvDbUrlIgnored = true;
|
||||
console.warn(
|
||||
'[config] Ignoring DATABASE_URL auto-loaded by Bun from a .env file in the current ' +
|
||||
'directory — it belongs to the project here, not to gbrain. Using the engine from ' +
|
||||
'~/.gbrain/config.json instead. To point gbrain at that database deliberately, set ' +
|
||||
'GBRAIN_DATABASE_URL.',
|
||||
);
|
||||
}
|
||||
return undefined;
|
||||
}
|
||||
return url;
|
||||
}
|
||||
|
||||
export function loadConfig(): GBrainConfig | null {
|
||||
let fileConfig: GBrainConfig | null = null;
|
||||
try {
|
||||
@@ -405,8 +495,8 @@ export function loadConfig(): GBrainConfig | null {
|
||||
fileConfig = migrateLegacyEmbeddingConfig(parsed) as unknown as GBrainConfig;
|
||||
} catch { /* no config file */ }
|
||||
|
||||
// Try env vars
|
||||
const dbUrl = process.env.GBRAIN_DATABASE_URL || process.env.DATABASE_URL;
|
||||
// Try env vars (cwd-.env-origin DATABASE_URL excluded — see #427 guard above)
|
||||
const dbUrl = effectiveEnvDatabaseUrl();
|
||||
|
||||
if (!fileConfig && !dbUrl) return null;
|
||||
|
||||
@@ -945,7 +1035,12 @@ export function gbrainPath(...segments: string[]): string {
|
||||
*/
|
||||
export function getDbUrlSource(): DbUrlSource {
|
||||
if (process.env.GBRAIN_DATABASE_URL) return 'env:GBRAIN_DATABASE_URL';
|
||||
if (process.env.DATABASE_URL) return 'env:DATABASE_URL';
|
||||
// Same #427 guard as loadConfig: a DATABASE_URL that Bun auto-loaded from
|
||||
// a cwd .env file is not an operator-provided source. Keeping this in
|
||||
// lockstep with loadConfig matters because doctor uses this to tell the
|
||||
// user where the URL came from — reporting env:DATABASE_URL while
|
||||
// loadConfig ignores it would send them debugging the wrong layer.
|
||||
if (effectiveEnvDatabaseUrl()) return 'env:DATABASE_URL';
|
||||
if (!existsSync(configPath())) return null;
|
||||
try {
|
||||
const raw = readFileSync(configPath(), 'utf-8');
|
||||
|
||||
+36
-1
@@ -1033,6 +1033,21 @@ async function runPhaseExtractFacts(
|
||||
|| result.phantomsSkippedDrift)
|
||||
? `, ${result.phantomsRedirected} phantom(s) redirected (${result.phantomsAmbiguous} ambiguous, ${result.phantomsSkippedDrift} drift-skipped)`
|
||||
: '';
|
||||
// #1928: a reconcile that deletes far more facts than it reinserts is the
|
||||
// signature of the conversation-facts wipe (factsDeleted 1829, inserted 0
|
||||
// read as a no-op "ok" before this guard). Surface net deletion above a
|
||||
// floor as `warn` so the daily report and doctor make it visible instead
|
||||
// of it reading like a clean run.
|
||||
const NET_DELETION_WARN_FLOOR = 50;
|
||||
const netDeleted = result.factsDeleted - result.factsInserted;
|
||||
const netDeletionWarn = netDeleted >= NET_DELETION_WARN_FLOOR;
|
||||
if (netDeletionWarn) {
|
||||
result.warnings.push(
|
||||
`net_fact_deletion: reconcile removed ${result.factsDeleted} fact(s) and ` +
|
||||
`reinserted ${result.factsInserted} (net -${netDeleted}). If unexpected, a ` +
|
||||
`destructive full walk may have wiped non-fence facts (#1928).`,
|
||||
);
|
||||
}
|
||||
return {
|
||||
phase: 'extract_facts',
|
||||
status: result.warnings.length > 0 ? 'warn' : 'ok',
|
||||
@@ -1586,6 +1601,11 @@ export async function runCycle(
|
||||
// and which slugs synthesize wrote so recompute_emotional_weight can
|
||||
// pick up the union of (sync ∪ synthesize) for v0.29 incremental mode.
|
||||
let syncPagesAffected: string[] | undefined;
|
||||
// #1928 (codex): true ONLY when the sync phase actually RAN its work (not
|
||||
// when it was skipped for no-engine / no-brainDir). The destructive
|
||||
// extract_facts guard keys off this so a SKIPPED sync still allows a
|
||||
// legitimate full reconcile — only a sync that ran and failed suppresses it.
|
||||
let syncAttempted = false;
|
||||
let synthesizeWrittenSlugs: string[] | undefined;
|
||||
if (phases.includes('sync')) {
|
||||
checkAborted(opts.signal);
|
||||
@@ -1601,6 +1621,7 @@ export async function runCycle(
|
||||
phaseResults.push(skipNoBrainDir('sync'));
|
||||
} else {
|
||||
progress.start('cycle.sync');
|
||||
syncAttempted = true; // sync ran its work; undefined pagesAffected now means failure
|
||||
const { result, duration_ms } = await timePhase(() => runPhaseSync(engine, brainDir, dryRun, pull, phases.includes('extract')));
|
||||
result.duration_ms = duration_ms;
|
||||
// Capture changed slugs for incremental extract.
|
||||
@@ -1700,8 +1721,22 @@ export async function runCycle(
|
||||
// the sources table doesn't recognize this brainDir (pre-multi-
|
||||
// source installs).
|
||||
const xfSourceId = cycleSourceId ?? 'default';
|
||||
// #1928: extract_facts is DESTRUCTIVE (wipe-and-reinsert per page). It
|
||||
// must NOT inherit the "sync failed ⇒ undefined ⇒ full walk" fallback
|
||||
// that's safe for link/timeline extract. When the sync phase RAN but
|
||||
// failed, syncPagesAffected is undefined (a successful no-op sync
|
||||
// returns []). In that case pass [] (no-op) so a lock-contention or
|
||||
// transient sync failure can't escalate into a brain-wide fact wipe.
|
||||
// undefined still reaches here (intended full reconcile) when the sync
|
||||
// phase was absent OR skipped (no engine / no brainDir — extract_facts
|
||||
// supports no-brainDir DB reconciliation). Only a sync that actually
|
||||
// RAN and came back with undefined pagesAffected is a real failure
|
||||
// (#1928, codex: keying off phases.includes('sync') wrongly suppressed
|
||||
// the skipped-sync full reconcile).
|
||||
const syncRanButFailed = syncAttempted && syncPagesAffected === undefined;
|
||||
const xfSlugs = syncRanButFailed ? [] : syncPagesAffected;
|
||||
const { result, duration_ms } = await timePhase(() =>
|
||||
runPhaseExtractFacts(engine, brainDir, xfSourceId, dryRun, syncPagesAffected, opts.signal));
|
||||
runPhaseExtractFacts(engine, brainDir, xfSourceId, dryRun, xfSlugs, opts.signal));
|
||||
result.duration_ms = duration_ms;
|
||||
phaseResults.push(result);
|
||||
progress.finish();
|
||||
|
||||
@@ -222,10 +222,15 @@ export async function runExtractFacts(
|
||||
|
||||
if (opts.dryRun) continue;
|
||||
|
||||
// Wipe-and-reinsert per page. The deleteFactsForPage call targets
|
||||
// source_markdown_slug = slug only, so NULL-source_markdown_slug
|
||||
// legacy rows survive (the partial-UNIQUE-index keyspace).
|
||||
const deleted = await engine.deleteFactsForPage(slug, sourceId);
|
||||
// Wipe-and-reinsert per page. The delete targets source_markdown_slug =
|
||||
// slug only, so NULL-source_markdown_slug legacy rows survive (the
|
||||
// partial-UNIQUE-index keyspace). #1928: `cli:`-origin facts (conversation
|
||||
// facts from extract-conversation-facts) are NOT fence-owned — the page
|
||||
// carries no `## Facts` fence to recreate them — so they MUST survive this
|
||||
// reconcile. Exclude them from the wipe.
|
||||
const deleted = await engine.deleteFactsForPage(slug, sourceId, {
|
||||
excludeSourcePrefixes: ['cli:'],
|
||||
});
|
||||
result.factsDeleted += deleted.deleted;
|
||||
|
||||
if (parsed.facts.length === 0) continue;
|
||||
|
||||
@@ -173,6 +173,7 @@ export const META_CHECK_NAMES: ReadonlySet<string> = new Set([
|
||||
'schema_pack_source_drift',
|
||||
'schema_version',
|
||||
'slug_fallback_audit',
|
||||
'timeline_dedup_index',
|
||||
'upgrade_errors',
|
||||
]);
|
||||
|
||||
|
||||
@@ -217,8 +217,10 @@ export function embeddingMismatchMessage(opts: EmbeddingMismatchOpts): string {
|
||||
``,
|
||||
` BEGIN;`,
|
||||
` DROP INDEX IF EXISTS idx_chunks_embedding;`,
|
||||
` ALTER TABLE content_chunks ALTER COLUMN embedding TYPE vector(${requestedDims});`,
|
||||
` -- NULL embeddings BEFORE the alter: pgvector refuses to cast existing`,
|
||||
` -- vectors across dimensions and aborts the transaction. NULLs cast fine.`,
|
||||
` UPDATE content_chunks SET embedding = NULL, embedded_at = NULL;`,
|
||||
` ALTER TABLE content_chunks ALTER COLUMN embedding TYPE vector(${requestedDims});`,
|
||||
` ${reindexLine.split('\n').join('\n ')}`,
|
||||
` COMMIT;`,
|
||||
``,
|
||||
@@ -573,11 +575,25 @@ export function buildFactsAlterRecipe(
|
||||
): string {
|
||||
const opclass = columnType === 'halfvec' ? 'halfvec_cosine_ops' : 'vector_cosine_ops';
|
||||
const targetType = columnType === 'halfvec' ? `halfvec(${configuredDims})` : `vector(${configuredDims})`;
|
||||
const dimsChanged = columnDims !== configuredDims;
|
||||
return [
|
||||
`-- ALTER ${columnType}(${columnDims}) → ${columnType}(${configuredDims}) on indexed column.`,
|
||||
`-- HOLD a maintenance window: this rewrites every row's embedding.`,
|
||||
`-- Coordinate with any active extract-conversation-facts backfill.`,
|
||||
`DROP INDEX IF EXISTS idx_facts_embedding_hnsw;`,
|
||||
// Same-dim type swaps (halfvec <-> vector) cast row data losslessly and
|
||||
// MUST keep it. Cross-dimension changes are different: pgvector refuses
|
||||
// to cast existing vectors across dimensions ("expected N dimensions,
|
||||
// not M") and aborts the transaction — and the old-space vectors are
|
||||
// unusable at the new width anyway. NULL them first; the facts pipeline
|
||||
// re-embeds on the next write.
|
||||
...(dimsChanged
|
||||
? [
|
||||
`-- Dimension change: NULL embeddings BEFORE the alter — pgvector`,
|
||||
`-- refuses cross-dimension casts and aborts the transaction.`,
|
||||
`UPDATE facts SET embedding = NULL;`,
|
||||
]
|
||||
: []),
|
||||
`ALTER TABLE facts ALTER COLUMN embedding TYPE ${targetType}`,
|
||||
` USING embedding::${targetType};`,
|
||||
`CREATE INDEX idx_facts_embedding_hnsw`,
|
||||
|
||||
+23
-1
@@ -650,6 +650,15 @@ export interface BrainEngine {
|
||||
// Lifecycle
|
||||
connect(config: EngineConfig): Promise<void>;
|
||||
disconnect(): Promise<void>;
|
||||
/**
|
||||
* Recover a dropped connection using the config captured at the last
|
||||
* `connect()`. Callers (autopilot health probe, batchRetry) MUST use this
|
||||
* instead of `disconnect()` + bare `connect()`: the latter loses the config
|
||||
* (#2034 — a bare `connect()` with no args throws `database_url undefined`
|
||||
* forever) AND opens a null-connection window. Implemented on BOTH engines
|
||||
* for parity so the call is never a silent no-op.
|
||||
*/
|
||||
reconnect(ctx?: { error?: unknown }): Promise<void>;
|
||||
initSchema(): Promise<void>;
|
||||
transaction<T>(fn: (engine: BrainEngine) => Promise<T>): Promise<T>;
|
||||
/**
|
||||
@@ -1678,7 +1687,20 @@ export interface BrainEngine {
|
||||
* until the v0_32_2 migration backfills them. Cycle-phase callers in
|
||||
* commit 7 add the empty-fence-guard as a belt-and-suspenders check.
|
||||
*/
|
||||
deleteFactsForPage(slug: string, source_id: string): Promise<{ deleted: number }>;
|
||||
/**
|
||||
* #1928: `excludeSourcePrefixes` protects facts whose `source` matches any
|
||||
* given prefix (e.g. `['cli:']` for conversation facts written by
|
||||
* `extract-conversation-facts`) from a fence reconcile. Those rows are NOT
|
||||
* fence-owned — a destructive wipe-and-reinsert pass would delete them and
|
||||
* never recreate them (the page has no `## Facts` fence). Omitted ⇒ legacy
|
||||
* behavior (delete every fact on the page coordinate). NULL/empty `source`
|
||||
* rows are always deletable (fence default).
|
||||
*/
|
||||
deleteFactsForPage(
|
||||
slug: string,
|
||||
source_id: string,
|
||||
opts?: { excludeSourcePrefixes?: string[] },
|
||||
): Promise<{ deleted: number }>;
|
||||
|
||||
/**
|
||||
* Mark a fact expired. Never DELETE. Returns true iff a row was updated.
|
||||
|
||||
@@ -28,6 +28,15 @@ export interface ExtractTimelineFromMeetingsResult {
|
||||
entries_created: number;
|
||||
/** Distinct entity pages that received at least one new timeline entry. */
|
||||
entities_touched: number;
|
||||
/**
|
||||
* #2057: batches that failed to insert. Previously swallowed by a bare
|
||||
* `catch {}`, which let a brain-wide timeline-write failure read as a clean
|
||||
* "0 entries" run. Non-zero here means inserts are failing — surfaced on
|
||||
* stderr too.
|
||||
*/
|
||||
batch_errors: number;
|
||||
/** First batch-insert error message, when batch_errors > 0. */
|
||||
first_batch_error?: string;
|
||||
}
|
||||
|
||||
interface MeetingRow {
|
||||
@@ -71,7 +80,7 @@ export async function extractTimelineFromMeetings(
|
||||
);
|
||||
|
||||
if (meetings.length === 0) {
|
||||
return { meetings_scanned: 0, entries_created: 0, entities_touched: 0 };
|
||||
return { meetings_scanned: 0, entries_created: 0, entities_touched: 0, batch_errors: 0 };
|
||||
}
|
||||
|
||||
// 2. Fetch all 'attended' edges (one round-trip, scoped to the loaded
|
||||
@@ -106,14 +115,23 @@ export async function extractTimelineFromMeetings(
|
||||
let entriesCreated = 0;
|
||||
const entitiesTouched = new Set<string>();
|
||||
let meetingsScanned = 0;
|
||||
let batchErrors = 0;
|
||||
let firstBatchError: string | undefined;
|
||||
|
||||
async function flush() {
|
||||
if (batch.length === 0) return;
|
||||
if (!dryRun) {
|
||||
try {
|
||||
entriesCreated += await engine.addTimelineEntriesBatch(batch);
|
||||
} catch {
|
||||
// batch error — drop; per-meeting progress continues
|
||||
} catch (e) {
|
||||
// #2057: do NOT swallow. A bare `catch {}` here hid a brain-wide
|
||||
// timeline-write failure (the run reported 0 entries with no error).
|
||||
// Count + surface it on stderr; the per-meeting loop still continues so
|
||||
// one bad batch isn't fatal to the rest.
|
||||
batchErrors += 1;
|
||||
const msg = e instanceof Error ? e.message : String(e);
|
||||
if (!firstBatchError) firstBatchError = msg;
|
||||
console.error(`[extract timeline] batch insert failed (${batch.length} row(s)): ${msg}`);
|
||||
}
|
||||
} else {
|
||||
entriesCreated += batch.length;
|
||||
@@ -182,5 +200,7 @@ export async function extractTimelineFromMeetings(
|
||||
meetings_scanned: meetingsScanned,
|
||||
entries_created: entriesCreated,
|
||||
entities_touched: entitiesTouched.size,
|
||||
batch_errors: batchErrors,
|
||||
first_batch_error: firstBatchError,
|
||||
};
|
||||
}
|
||||
|
||||
+20
-1
@@ -1193,7 +1193,15 @@ export async function importCodeFile(
|
||||
// chunk IDs are stable.
|
||||
if (extractedEdges.length > 0 && chunks.length > 0) {
|
||||
try {
|
||||
const persistedChunks = await engine.getChunks(slug, sourceId ? { sourceId } : undefined);
|
||||
// Normalize ONCE: '' and undefined both mean the schema-default source
|
||||
// (pages.source_id DEFAULT 'default'). Using the normalized value for
|
||||
// BOTH the chunk lookup and the edge stamp keeps them in lockstep —
|
||||
// an unscoped getChunks here could fan out to same-slug chunks from
|
||||
// another source, and a '' stamp would FK-violate against sources(id)
|
||||
// and silently drop the file's whole call graph in the best-effort
|
||||
// catch below (adversarial review findings).
|
||||
const edgeSourceId = sourceId || 'default';
|
||||
const persistedChunks = await engine.getChunks(slug, { sourceId: edgeSourceId });
|
||||
const byIndex = new Map<number, { id?: number; symbol_name_qualified?: string | null; start_line?: number | null; end_line?: number | null }>();
|
||||
for (const pc of persistedChunks) {
|
||||
byIndex.set(pc.chunk_index, pc);
|
||||
@@ -1231,6 +1239,17 @@ export async function importCodeFile(
|
||||
from_symbol_qualified: from.symbol_name_qualified,
|
||||
to_symbol_qualified: e.toSymbol,
|
||||
edge_type: e.edgeType,
|
||||
// Stamp the source: getCallersOf/getCalleesOf add
|
||||
// `AND source_id = <scoped>` whenever a worktree pin / --source is
|
||||
// in play, and a NULL here never matches that filter — so every
|
||||
// scoped call-graph query silently returned 0 rows on
|
||||
// multi-source brains even though the edges existed. The fallback
|
||||
// is 'default', NOT null: an unscoped import lands its pages under
|
||||
// the schema default (pages.source_id DEFAULT 'default'), so a
|
||||
// NULL-stamped edge would be invisible to the matching scoped
|
||||
// query getCallersOf(sym, { sourceId: 'default' }) — the same bug
|
||||
// through the other door.
|
||||
source_id: edgeSourceId,
|
||||
});
|
||||
}
|
||||
|
||||
|
||||
@@ -0,0 +1,22 @@
|
||||
/**
|
||||
* Derive a legacy bearer token's source scope from its stored
|
||||
* `access_tokens.permissions.source_id` grant.
|
||||
*
|
||||
* ARRAY = federated read grant, exposed through `allowedSources` with the
|
||||
* first granted source as the scalar write floor. STRING = scalar source.
|
||||
* Missing, empty, or garbage values fail closed to the historical `default`
|
||||
* floor and NEVER widen to all sources.
|
||||
*/
|
||||
export function parseLegacyTokenScope(rawSource: unknown): { sourceId: string; allowedSources?: string[] } {
|
||||
if (Array.isArray(rawSource)) {
|
||||
const allowedSources = (rawSource as unknown[]).filter(s => typeof s === 'string' && s.length > 0) as string[];
|
||||
if (allowedSources.length > 0) {
|
||||
return { sourceId: allowedSources[0], allowedSources };
|
||||
}
|
||||
return { sourceId: 'default' };
|
||||
}
|
||||
if (typeof rawSource === 'string' && rawSource.length > 0) {
|
||||
return { sourceId: rawSource };
|
||||
}
|
||||
return { sourceId: 'default' };
|
||||
}
|
||||
@@ -5188,6 +5188,54 @@ export const MIGRATIONS: Migration[] = [
|
||||
},
|
||||
{
|
||||
version: 116,
|
||||
name: 'code_edges_source_backfill_and_callee_index',
|
||||
// Repair + index pass for the call graph:
|
||||
//
|
||||
// 1. BACKFILL: importCodeFile built CodeEdgeInput rows without source_id,
|
||||
// so every extracted edge landed NULL. getCallersOf/getCalleesOf add
|
||||
// `AND source_id = <scoped>` whenever a worktree pin / --source is in
|
||||
// play — NULL never matches, so scoped call-graph queries silently
|
||||
// returned 0 rows on multi-source brains even though the edges
|
||||
// existed. The write path now stamps `sourceId ?? 'default'`; this
|
||||
// backfill repairs rows written before the fix by deriving each
|
||||
// edge's source from its own from_chunk's page (pages.source_id is
|
||||
// NOT NULL DEFAULT 'default', so COALESCE is belt-and-braces only).
|
||||
//
|
||||
// 2. INDEXES: getCalleesOf filters BOTH edge tables on
|
||||
// from_symbol_qualified, which had no index anywhere — every callee
|
||||
// lookup was a sequential scan, amplified per-BFS-node by the
|
||||
// recursive code walk (one getCalleesOf per frontier node, up to
|
||||
// maxNodes). With NULL edges repaired, scoped walks actually expand,
|
||||
// so the latent seq-scan cost becomes real. Plain CREATE INDEX (not
|
||||
// CONCURRENTLY): edge tables are modest (mirrors the v58 resolver
|
||||
// index). Keep in sync with src/schema.sql.
|
||||
idempotent: true,
|
||||
sql: `
|
||||
UPDATE code_edges_symbol e
|
||||
SET source_id = COALESCE(p.source_id, 'default')
|
||||
FROM content_chunks c
|
||||
JOIN pages p ON p.id = c.page_id
|
||||
WHERE c.id = e.from_chunk_id
|
||||
AND e.source_id IS NULL;
|
||||
|
||||
UPDATE code_edges_chunk e
|
||||
SET source_id = COALESCE(p.source_id, 'default')
|
||||
FROM content_chunks c
|
||||
JOIN pages p ON p.id = c.page_id
|
||||
WHERE c.id = e.from_chunk_id
|
||||
AND e.source_id IS NULL;
|
||||
|
||||
CREATE INDEX IF NOT EXISTS idx_code_edges_symbol_from_symbol
|
||||
ON code_edges_symbol (from_symbol_qualified);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS idx_code_edges_chunk_from_symbol
|
||||
ON code_edges_chunk (from_symbol_qualified);
|
||||
`,
|
||||
},
|
||||
{
|
||||
// Renumbered 116 -> 117 at merge: master's v0.42.41.0 triage wave
|
||||
// claimed v116 (code_edges_source_backfill_and_callee_index) first.
|
||||
version: 117,
|
||||
name: 'context_volunteer_events_table',
|
||||
// #2095 push-based context: feedback-loop log of every page the brain
|
||||
// VOLUNTEERED (via the volunteer_context op, the retrieval-reflex pointer
|
||||
@@ -5560,6 +5608,26 @@ export async function runMigrations(engine: BrainEngine): Promise<{ applied: num
|
||||
const sorted = [...MIGRATIONS].sort((a, b) => a.version - b.version);
|
||||
|
||||
const pending = sorted.filter(m => m.version > current);
|
||||
|
||||
// #2038: schema-drift self-heal. A migration renumbered during a master
|
||||
// merge (v102 timeline dedup, originally v99) can be recorded-as-applied
|
||||
// without its DDL ever running — the version counter can't see it. Repair
|
||||
// the known drift on EVERY pass, including when nothing is pending (the
|
||||
// affected brains are stamped AHEAD of the missing migration, so they never
|
||||
// reach the loop below). Best-effort + idempotent: a no-op on a healthy
|
||||
// index; `doctor` surfaces it independently if this ever fails.
|
||||
try {
|
||||
const { repairTimelineDedupIndex } = await import('./timeline-dedup-repair.ts');
|
||||
const r = await repairTimelineDedupIndex(engine);
|
||||
if (r.repaired) {
|
||||
console.error(
|
||||
`[migrate] healed idx_timeline_dedup drift (#2038): ${r.before.join(',') || '(absent)'} ` +
|
||||
`→ page_id,date,summary,source` +
|
||||
(r.collapsedDuplicates > 0 ? ` (collapsed ${r.collapsedDuplicates} duplicate row(s))` : ''),
|
||||
);
|
||||
}
|
||||
} catch { /* best-effort; doctor reports the drift if this couldn't run */ }
|
||||
|
||||
if (pending.length === 0) {
|
||||
return { applied: 0, current };
|
||||
}
|
||||
|
||||
@@ -1023,16 +1023,17 @@ export class MinionSupervisor {
|
||||
error: errMsg,
|
||||
queue: this.opts.queue,
|
||||
});
|
||||
// Attempt to reconnect the engine if it supports it
|
||||
// Attempt to reconnect the engine. #2034: reconnect() is now a
|
||||
// first-class BrainEngine method on both engines (it restores the
|
||||
// config captured at connect()), so call it directly — the old
|
||||
// feature-detection cast is no longer needed.
|
||||
try {
|
||||
if ('reconnect' in this.engine && typeof (this.engine as Record<string, unknown>).reconnect === 'function') {
|
||||
await (this.engine as unknown as { reconnect(): Promise<void> }).reconnect();
|
||||
this.consecutiveHealthFailures = 0;
|
||||
this.emit('health_warn', {
|
||||
reason: 'db_reconnected',
|
||||
queue: this.opts.queue,
|
||||
});
|
||||
}
|
||||
await this.engine.reconnect({ error: errMsg });
|
||||
this.consecutiveHealthFailures = 0;
|
||||
this.emit('health_warn', {
|
||||
reason: 'db_reconnected',
|
||||
queue: this.opts.queue,
|
||||
});
|
||||
} catch (reconnErr) {
|
||||
this.emit('health_error', {
|
||||
error: `reconnect failed: ${reconnErr instanceof Error ? reconnErr.message : String(reconnErr)}`,
|
||||
|
||||
+55
-18
@@ -21,10 +21,12 @@ import type {
|
||||
} from '@modelcontextprotocol/sdk/shared/auth.js';
|
||||
import type { OAuthServerProvider, AuthorizationParams } from '@modelcontextprotocol/sdk/server/auth/provider.js';
|
||||
import type { OAuthRegisteredClientsStore } from '@modelcontextprotocol/sdk/server/auth/clients.js';
|
||||
import type { AuthInfo } from '@modelcontextprotocol/sdk/server/auth/types.js';
|
||||
import type { AuthInfo as SdkAuthInfo } from '@modelcontextprotocol/sdk/server/auth/types.js';
|
||||
import { InvalidTokenError } from '@modelcontextprotocol/sdk/server/auth/errors.js';
|
||||
import { hashToken, generateToken, isUndefinedColumnError } from './utils.ts';
|
||||
import { hasScope, assertAllowedScopes, parseScopeString, InvalidScopeError } from './scope.ts';
|
||||
import type { AuthInfo as CoreAuthInfo } from './operations.ts';
|
||||
import { parseLegacyTokenScope } from './legacy-token-scope.ts';
|
||||
import type { SqlQuery, SqlValue } from './sql-query.ts';
|
||||
export type { SqlQuery, SqlValue };
|
||||
|
||||
@@ -389,10 +391,18 @@ export class GBrainOAuthProvider implements OAuthServerProvider {
|
||||
// as a fully-admin access token. Mirrors the filter pattern already used
|
||||
// by exchangeClientCredentials (this file) and exchangeRefreshToken's F3
|
||||
// subset enforcement (RFC 6749 §6) so all three grant entry points clamp
|
||||
// consistently. Empty/omitted requested scope inherits the empty-stored
|
||||
// shape (existing behavior; not a security boundary).
|
||||
// consistently. When the client requests NO scope, RFC 6749 §3.3 lets the
|
||||
// server fall back to a default — we default to the client's full
|
||||
// registered scope (matching exchangeClientCredentials, which already does
|
||||
// `requestedScope ? ... : allowedScopes`). Previously an omitted request
|
||||
// granted the empty set, which then propagated into the access+refresh
|
||||
// tokens and never self-healed: every op failed `insufficient_scope` even
|
||||
// though the client was registered with `read write`. Clients that omit
|
||||
// `scope` on /authorize (e.g. some MCP connectors) hit this. Still clamped
|
||||
// to the allowed set, so an explicit over-broad request can't escalate.
|
||||
const allowedScopes = parseScopeString(client.scope);
|
||||
const grantedScopes = (params.scopes || []).filter(s => hasScope(allowedScopes, s));
|
||||
const requestedScopes = (params.scopes && params.scopes.length) ? params.scopes : allowedScopes;
|
||||
const grantedScopes = requestedScopes.filter(s => hasScope(allowedScopes, s));
|
||||
|
||||
await this.sql`
|
||||
INSERT INTO oauth_codes (code_hash, client_id, scopes, code_challenge,
|
||||
@@ -539,7 +549,7 @@ export class GBrainOAuthProvider implements OAuthServerProvider {
|
||||
// Token Verification
|
||||
// -------------------------------------------------------------------------
|
||||
|
||||
async verifyAccessToken(token: string): Promise<AuthInfo> {
|
||||
async verifyAccessToken(token: string): Promise<SdkAuthInfo> {
|
||||
const tokenHash = hashToken(token);
|
||||
const now = Math.floor(Date.now() / 1000);
|
||||
|
||||
@@ -629,14 +639,29 @@ export class GBrainOAuthProvider implements OAuthServerProvider {
|
||||
// operations.ts prefers this array over scalar sourceId when set
|
||||
// and non-empty.
|
||||
allowedSources,
|
||||
} as AuthInfo;
|
||||
} as CoreAuthInfo as SdkAuthInfo;
|
||||
}
|
||||
|
||||
// Fallback: legacy access_tokens table (backward compat)
|
||||
const legacyRows = await this.sql`
|
||||
SELECT name FROM access_tokens
|
||||
WHERE token_hash = ${tokenHash} AND revoked_at IS NULL
|
||||
`;
|
||||
// Fallback: legacy access_tokens table (backward compat). Modern legacy
|
||||
// rows may carry permissions.source_id from the pre-OAuth bearer-token
|
||||
// path; OAuth transport must preserve that same source grant instead of
|
||||
// pinning every legacy token to `default`.
|
||||
let legacyRows: Record<string, unknown>[];
|
||||
try {
|
||||
legacyRows = await this.sql`
|
||||
SELECT name, permissions FROM access_tokens
|
||||
WHERE token_hash = ${tokenHash} AND revoked_at IS NULL
|
||||
`;
|
||||
} catch (err) {
|
||||
if (isUndefinedColumnError(err, 'permissions')) {
|
||||
legacyRows = await this.sql`
|
||||
SELECT name FROM access_tokens
|
||||
WHERE token_hash = ${tokenHash} AND revoked_at IS NULL
|
||||
`;
|
||||
} else {
|
||||
throw err;
|
||||
}
|
||||
}
|
||||
|
||||
if (legacyRows.length > 0) {
|
||||
// Legacy tokens get full admin access (grandfather in).
|
||||
@@ -646,19 +671,31 @@ export class GBrainOAuthProvider implements OAuthServerProvider {
|
||||
UPDATE access_tokens SET last_used_at = now() WHERE token_hash = ${tokenHash}
|
||||
`;
|
||||
const name = legacyRows[0].name as string;
|
||||
const permissionsRaw = legacyRows[0].permissions;
|
||||
let permissions: unknown = permissionsRaw;
|
||||
if (typeof permissionsRaw === 'string') {
|
||||
try {
|
||||
permissions = JSON.parse(permissionsRaw);
|
||||
} catch {
|
||||
permissions = undefined;
|
||||
}
|
||||
}
|
||||
const sourceGrant = permissions && typeof permissions === 'object'
|
||||
? (permissions as Record<string, unknown>).source_id
|
||||
: undefined;
|
||||
const { sourceId, allowedSources } = parseLegacyTokenScope(sourceGrant);
|
||||
return {
|
||||
token,
|
||||
clientId: name,
|
||||
clientName: name,
|
||||
scopes: ['read', 'write', 'admin'],
|
||||
expiresAt: Math.floor(Date.now() / 1000) + 365 * 24 * 3600, // Legacy tokens never expire — set 1yr future
|
||||
// v0.34.1 (#861, D13): legacy bearer tokens default to 'default'
|
||||
// source — matches the pre-v0.34 effective behavior where the
|
||||
// serve-http transport fell back to GBRAIN_SOURCE/'default' for
|
||||
// any caller without explicit scope. Operators who want a
|
||||
// narrower scope for legacy tokens migrate to OAuth.
|
||||
sourceId: 'default',
|
||||
} as AuthInfo;
|
||||
// Legacy tokens without an explicit permissions.source_id grant keep
|
||||
// the historical 'default' source floor. Array grants become
|
||||
// allowedSources for federated reads, matching legacy HTTP transport.
|
||||
sourceId,
|
||||
allowedSources,
|
||||
} as CoreAuthInfo as SdkAuthInfo;
|
||||
}
|
||||
|
||||
throw new InvalidTokenError('Invalid token');
|
||||
|
||||
@@ -199,6 +199,9 @@ export class PGLiteEngine implements BrainEngine {
|
||||
readonly kind = 'pglite' as const;
|
||||
private _db: PGLiteDB | null = null;
|
||||
private _lock: LockHandle | null = null;
|
||||
// #2034: captured at connect() so reconnect() can restore the same data dir
|
||||
// after a drop, matching PostgresEngine's _savedConfig contract.
|
||||
private _savedConfig: EngineConfig | null = null;
|
||||
// Tier 3: when GBRAIN_PGLITE_SNAPSHOT loaded a post-initSchema state into
|
||||
// PGlite.create(loadDataDir), initSchema is a no-op (schema is already
|
||||
// present + migrations already applied). Saves ~1-3s per fresh test PGLite.
|
||||
@@ -211,6 +214,7 @@ export class PGLiteEngine implements BrainEngine {
|
||||
|
||||
// Lifecycle
|
||||
async connect(config: EngineConfig): Promise<void> {
|
||||
this._savedConfig = config; // #2034: remember for reconnect()
|
||||
const dataDir = config.database_path || undefined; // undefined = in-memory
|
||||
|
||||
// Acquire file lock to prevent concurrent PGLite access (crashes with Aborted())
|
||||
@@ -294,6 +298,27 @@ export class PGLiteEngine implements BrainEngine {
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* #2034: engine-parity reconnect. PGLite is single-writer in-process so it
|
||||
* doesn't suffer the pool-drop class PostgresEngine.reconnect() handles, but
|
||||
* the method MUST exist so callers (autopilot health probe, worker/queue
|
||||
* claim-error recovery) can call `engine.reconnect()` uniformly.
|
||||
*
|
||||
* IN-MEMORY (no `database_path`) is a NO-OP: there is no persistent backing,
|
||||
* the connection can't recoverably "drop" in-process, and a disconnect+reopen
|
||||
* would DISCARD all state. This matches the long-standing assumption the
|
||||
* worker/queue recovery paths are written against ("PGLite has no pooler
|
||||
* reaping so reconnect is absent" — src/core/minions/queue.ts). A FILE-backed
|
||||
* engine genuinely re-opens the same data dir (state persists on disk).
|
||||
*/
|
||||
async reconnect(_ctx?: { error?: unknown }): Promise<void> {
|
||||
if (!this._savedConfig) return; // never connected — nothing to restore
|
||||
if (!this._savedConfig.database_path) return; // in-memory — no-op, preserve state
|
||||
const config = this._savedConfig;
|
||||
await this.disconnect();
|
||||
await this.connect(config);
|
||||
}
|
||||
|
||||
async initSchema(): Promise<void> {
|
||||
// Tier 3: snapshot was loaded into PGlite — schema + migrations already
|
||||
// applied. Nothing to do. Returns immediately.
|
||||
@@ -3604,7 +3629,25 @@ export class PGLiteEngine implements BrainEngine {
|
||||
return { inserted: ids.length, ids };
|
||||
}
|
||||
|
||||
async deleteFactsForPage(slug: string, source_id: string): Promise<{ deleted: number }> {
|
||||
async deleteFactsForPage(
|
||||
slug: string,
|
||||
source_id: string,
|
||||
opts?: { excludeSourcePrefixes?: string[] },
|
||||
): Promise<{ deleted: number }> {
|
||||
const prefixes = opts?.excludeSourcePrefixes;
|
||||
if (prefixes && prefixes.length > 0) {
|
||||
// #1928: keep rows whose `source` matches an excluded prefix (e.g.
|
||||
// `cli:` conversation facts). COALESCE so NULL/empty-source fence rows
|
||||
// stay deletable — only the explicitly-protected prefixes survive.
|
||||
const patterns = prefixes.map(p => `${p}%`);
|
||||
const result = await this.db.query(
|
||||
`DELETE FROM facts
|
||||
WHERE source_id = $1 AND source_markdown_slug = $2
|
||||
AND NOT (COALESCE(source, '') LIKE ANY($3::text[]))`,
|
||||
[source_id, slug, patterns],
|
||||
);
|
||||
return { deleted: result.affectedRows ?? 0 };
|
||||
}
|
||||
const result = await this.db.query(
|
||||
`DELETE FROM facts WHERE source_id = $1 AND source_markdown_slug = $2`,
|
||||
[source_id, slug],
|
||||
@@ -4914,7 +4957,7 @@ export class PGLiteEngine implements BrainEngine {
|
||||
e.from_chunk_id, e.to_chunk_id, e.from_symbol_qualified,
|
||||
e.to_symbol_qualified, e.edge_type,
|
||||
JSON.stringify(e.edge_metadata ?? {}),
|
||||
e.source_id ?? null,
|
||||
e.source_id ?? 'default',
|
||||
);
|
||||
}
|
||||
const res = await this.db.query(
|
||||
@@ -4935,7 +4978,7 @@ export class PGLiteEngine implements BrainEngine {
|
||||
params.push(
|
||||
e.from_chunk_id, e.from_symbol_qualified, e.to_symbol_qualified, e.edge_type,
|
||||
JSON.stringify(e.edge_metadata ?? {}),
|
||||
e.source_id ?? null,
|
||||
e.source_id ?? 'default',
|
||||
);
|
||||
}
|
||||
const res = await this.db.query(
|
||||
|
||||
+100
-11
@@ -19,11 +19,72 @@ import { join } from 'path';
|
||||
|
||||
const LOCK_DIR_NAME = '.gbrain-lock';
|
||||
const LOCK_FILE = 'lock';
|
||||
const STALE_THRESHOLD_MS = 5 * 60 * 1000; // 5 minutes — embed jobs can be long
|
||||
|
||||
// #2058: refresh the lock's `refreshed_at` while held so a long-running but
|
||||
// LIVE holder (embed jobs run for many minutes) is never mistaken for stale.
|
||||
const HEARTBEAT_INTERVAL_MS = 30_000;
|
||||
|
||||
// #2058: a holder whose heartbeat refreshed within this window is ALIVE and is
|
||||
// NEVER stolen, regardless of how old the lock is. Only a holder that STOPPED
|
||||
// refreshing past this grace (hung, crashed without cleanup, or a PID since
|
||||
// reused by an unrelated process) is reaped. Pairing heartbeat-age with PID
|
||||
// liveness is what defeats both the WAL-corruption bug (stealing a live
|
||||
// writer) AND the PID-reuse false-positive (a recycled PID reading as "alive").
|
||||
// Env-overridable as an incident escape hatch, matching the sync-lock knobs.
|
||||
function stealGraceMs(): number {
|
||||
const env = parseInt(process.env.GBRAIN_PGLITE_LOCK_STEAL_GRACE_SECONDS ?? '', 10);
|
||||
return Number.isFinite(env) && env > 0 ? env * 1000 : 10 * 60 * 1000; // default 600s
|
||||
}
|
||||
|
||||
export interface LockHandle {
|
||||
lockDir: string;
|
||||
acquired: boolean;
|
||||
/**
|
||||
* #2058: heartbeat timer + lock-file path, set when a real (on-disk) lock is
|
||||
* held so `releaseLock` can stop refreshing. Absent for the in-memory engine
|
||||
* (no lock file, no concurrent access possible).
|
||||
*/
|
||||
heartbeat?: ReturnType<typeof setInterval>;
|
||||
lockPath?: string;
|
||||
/**
|
||||
* #2058 (codex): our ownership token (`<pid>:<acquired_at>`). If we stall
|
||||
* past the steal grace, another process can reap + re-acquire. When we
|
||||
* resume, the heartbeat and release MUST verify the on-disk lock is STILL
|
||||
* ours before touching it — otherwise a resumed stale holder would refresh
|
||||
* or delete the NEW owner's live lock, re-opening the concurrent-writer hole.
|
||||
*/
|
||||
ownerToken?: string;
|
||||
}
|
||||
|
||||
/** The on-disk lock identity, used to detect "we were reaped and replaced". */
|
||||
function tokenOf(lockData: { pid?: unknown; acquired_at?: unknown }): string {
|
||||
return `${lockData.pid}:${lockData.acquired_at}`;
|
||||
}
|
||||
|
||||
/**
|
||||
* #2058: keep the held lock's `refreshed_at` current so a concurrent acquirer
|
||||
* can tell a live, working holder from a hung/dead one. Best-effort: if the
|
||||
* file is gone (we're being reaped) the write simply fails. `.unref()` so the
|
||||
* timer never keeps the process alive on its own. Ownership-checked: if the
|
||||
* on-disk lock is no longer ours (we were reaped past grace and replaced), stop
|
||||
* the heartbeat instead of clobbering the new owner's lock.
|
||||
*/
|
||||
function startHeartbeat(lockPath: string, ownerToken: string): ReturnType<typeof setInterval> {
|
||||
const timer = setInterval(() => {
|
||||
try {
|
||||
const raw = JSON.parse(readFileSync(lockPath, 'utf-8'));
|
||||
if (tokenOf(raw) !== ownerToken) {
|
||||
// We were reaped and someone else owns it now — do NOT refresh their
|
||||
// lock. Stand down.
|
||||
clearInterval(timer);
|
||||
return;
|
||||
}
|
||||
raw.refreshed_at = Date.now();
|
||||
writeFileSync(lockPath, JSON.stringify(raw), { mode: 0o644 });
|
||||
} catch { /* best-effort — file removed or transient FS error */ }
|
||||
}, HEARTBEAT_INTERVAL_MS);
|
||||
(timer as { unref?: () => void }).unref?.();
|
||||
return timer;
|
||||
}
|
||||
|
||||
function getLockDir(dataDir: string | undefined): string {
|
||||
@@ -75,16 +136,23 @@ export async function acquireLock(dataDir: string | undefined, opts?: { timeoutM
|
||||
const lockPid = lockData.pid as number;
|
||||
const lockTime = lockData.acquired_at as number;
|
||||
|
||||
// Is the locking process still alive?
|
||||
if (!isProcessAlive(lockPid)) {
|
||||
// Stale lock — clean it up
|
||||
// #2058: classify by PID liveness AND heartbeat freshness. A holder
|
||||
// that is alive AND refreshed its heartbeat within the steal grace is
|
||||
// genuinely working (e.g. a multi-minute embed) and is NEVER reaped —
|
||||
// force-removing it here is what corrupted the single-writer WAL.
|
||||
const alive = isProcessAlive(lockPid);
|
||||
const lastRefresh = (lockData.refreshed_at as number | undefined) ?? lockTime;
|
||||
const sinceRefresh = Date.now() - lastRefresh;
|
||||
if (!alive) {
|
||||
// Holder process is gone — reap.
|
||||
try { rmSync(lockDir, { recursive: true, force: true }); } catch { /* race condition, try again */ }
|
||||
} else if (Date.now() - lockTime > STALE_THRESHOLD_MS) {
|
||||
// Lock held for too long — assume stale (e.g., process hung)
|
||||
// Still alive but probably stuck — force remove
|
||||
} else if (sinceRefresh > stealGraceMs()) {
|
||||
// PID is alive but the heartbeat stopped past the grace window:
|
||||
// either the holder hung, or this PID was reused by an unrelated
|
||||
// process (the real holder died and stopped refreshing). Reap.
|
||||
try { rmSync(lockDir, { recursive: true, force: true }); } catch { /* race condition */ }
|
||||
} else {
|
||||
// Lock is held by a live process — wait and retry
|
||||
// Live holder refreshing within grace — wait and retry.
|
||||
await new Promise(r => setTimeout(r, 1000));
|
||||
continue;
|
||||
}
|
||||
@@ -97,15 +165,19 @@ export async function acquireLock(dataDir: string | undefined, opts?: { timeoutM
|
||||
// Try to acquire lock (atomic mkdir)
|
||||
try {
|
||||
mkdirSync(lockDir, { recursive: false });
|
||||
// We got the lock — write our PID
|
||||
// We got the lock — write our PID. #2058: seed `refreshed_at` and start
|
||||
// the heartbeat so this holder reads as alive-and-working to others.
|
||||
const lockPath = join(lockDir, LOCK_FILE);
|
||||
const now = Date.now();
|
||||
writeFileSync(lockPath, JSON.stringify({
|
||||
pid: process.pid,
|
||||
acquired_at: Date.now(),
|
||||
acquired_at: now,
|
||||
refreshed_at: now,
|
||||
command: process.argv.slice(1).join(' '),
|
||||
}), { mode: 0o644 });
|
||||
|
||||
return { lockDir, acquired: true };
|
||||
const ownerToken = tokenOf({ pid: process.pid, acquired_at: now });
|
||||
return { lockDir, acquired: true, lockPath, ownerToken, heartbeat: startHeartbeat(lockPath, ownerToken) };
|
||||
} catch (e: unknown) {
|
||||
// mkdir failed — someone else grabbed it between our check and mkdir
|
||||
// This is fine, we'll retry
|
||||
@@ -138,8 +210,25 @@ export async function acquireLock(dataDir: string | undefined, opts?: { timeoutM
|
||||
* Release a previously acquired lock.
|
||||
*/
|
||||
export async function releaseLock(lock: LockHandle): Promise<void> {
|
||||
// #2058: stop the heartbeat first so it can't recreate/rewrite the lock file
|
||||
// after we remove it.
|
||||
if (lock.heartbeat) {
|
||||
clearInterval(lock.heartbeat);
|
||||
lock.heartbeat = undefined;
|
||||
}
|
||||
if (!lock.lockDir || !lock.acquired) return;
|
||||
|
||||
// #2058 (codex): only remove the lock if it is STILL ours. If we were reaped
|
||||
// past the grace and another process re-acquired, removing its live lock
|
||||
// would let a third process in alongside it — the corruption this fix exists
|
||||
// to prevent. Unreadable/absent lock falls through to a best-effort remove.
|
||||
if (lock.ownerToken) {
|
||||
try {
|
||||
const raw = JSON.parse(readFileSync(join(lock.lockDir, LOCK_FILE), 'utf-8'));
|
||||
if (tokenOf(raw) !== lock.ownerToken) return; // someone else owns it now
|
||||
} catch { /* unreadable/gone — fall through to best-effort cleanup */ }
|
||||
}
|
||||
|
||||
try {
|
||||
rmSync(lock.lockDir, { recursive: true, force: true });
|
||||
} catch {
|
||||
|
||||
@@ -946,7 +946,7 @@ CREATE TABLE IF NOT EXISTS op_checkpoint_paths (
|
||||
);
|
||||
|
||||
-- #2095 push-based context: feedback-loop log of volunteered pages.
|
||||
-- Mirrors migration v116 + src/schema.sql. "Used" derives from
|
||||
-- Mirrors migration v117 + src/schema.sql. "Used" derives from
|
||||
-- pages.last_retrieved_at > volunteered_at; 90-day prune in the dream
|
||||
-- cycle's purge phase.
|
||||
CREATE TABLE IF NOT EXISTS context_volunteer_events (
|
||||
|
||||
@@ -1276,11 +1276,28 @@ export class PostgresEngine implements BrainEngine {
|
||||
}
|
||||
|
||||
async updateSourceConfig(sourceId: string, patch: Record<string, unknown>): Promise<boolean> {
|
||||
// v0.38: atomic JSONB merge. `||` is the Postgres concat operator —
|
||||
// for jsonb, right-side keys overwrite left-side; nested object keys
|
||||
// are NOT deep-merged (use jsonb_set for nested paths). The patch
|
||||
// shape this autopilot wave uses is flat (`last_full_cycle_at`,
|
||||
// `archive_*`, etc.) so concat is sufficient. Idempotent on re-run.
|
||||
// Atomic single-statement merge. The previous read-then-write form dropped
|
||||
// concurrent updates: two callers patching different keys could both read
|
||||
// the same old config and the later `SET config = ...` clobbered the
|
||||
// earlier patch. These keys are written by background cycle/autopilot
|
||||
// paths, so the merge must happen inside the UPDATE (parity with
|
||||
// pglite-engine.updateSourceConfig, which already uses JSONB `||`).
|
||||
//
|
||||
// The CASE normalizes historical bad shapes inline (so `config` is re-read
|
||||
// against the row-locked latest version — a CTE/subquery snapshot would
|
||||
// reintroduce the lost-update race under READ COMMITTED): older code paths
|
||||
// could store config as a JSONB string (double-encoded) or as a JSONB array
|
||||
// of patch objects. We coerce those to a flat object before the `||` merge
|
||||
// so doctor and source routing keep getting flat keys.
|
||||
//
|
||||
// String branch guard: a JSONB string whose inner text is NOT itself valid
|
||||
// JSON (one of the historical bad shapes this path repairs) would make the
|
||||
// bare `::jsonb` cast raise `invalid input syntax for type json`, failing
|
||||
// the whole UPDATE. Postgres has no `try_cast`, so we gate the cast with
|
||||
// the SQL `IS JSON` predicate (Postgres 16+): parseable inner text is
|
||||
// double-encoded config and gets parsed; unparseable text falls back to `{}`.
|
||||
// The guard keeps the merge a single atomic statement (no extra round-trip,
|
||||
// no lost-update race).
|
||||
//
|
||||
// MUST use sql.json(patch) inside the template tag — postgres-js's
|
||||
// positional executeRaw + `$1::jsonb` cast DOUBLE-ENCODES the
|
||||
@@ -1294,7 +1311,25 @@ export class PostgresEngine implements BrainEngine {
|
||||
const sql = this.sql;
|
||||
const result = await sql`
|
||||
UPDATE sources
|
||||
SET config = COALESCE(config, '{}'::jsonb) || ${sql.json(patch as Parameters<typeof sql.json>[0])}
|
||||
SET config =
|
||||
CASE
|
||||
WHEN jsonb_typeof(config) = 'object' THEN config
|
||||
WHEN jsonb_typeof(config) = 'string'
|
||||
THEN CASE
|
||||
WHEN (config #>> '{}') IS JSON
|
||||
THEN COALESCE(NULLIF((config #>> '{}'), '')::jsonb, '{}'::jsonb)
|
||||
ELSE '{}'::jsonb
|
||||
END
|
||||
WHEN jsonb_typeof(config) = 'array'
|
||||
THEN COALESCE(
|
||||
(SELECT jsonb_object_agg(kv.key, kv.value)
|
||||
FROM jsonb_array_elements(config) elem,
|
||||
jsonb_each(elem) kv),
|
||||
'{}'::jsonb
|
||||
)
|
||||
ELSE '{}'::jsonb
|
||||
END
|
||||
|| ${sql.json(patch as Parameters<typeof sql.json>[0])}
|
||||
WHERE id = ${sourceId}
|
||||
`;
|
||||
return (result.count ?? 0) > 0;
|
||||
@@ -3745,8 +3780,26 @@ export class PostgresEngine implements BrainEngine {
|
||||
return { inserted: ids.length, ids };
|
||||
}
|
||||
|
||||
async deleteFactsForPage(slug: string, source_id: string): Promise<{ deleted: number }> {
|
||||
async deleteFactsForPage(
|
||||
slug: string,
|
||||
source_id: string,
|
||||
opts?: { excludeSourcePrefixes?: string[] },
|
||||
): Promise<{ deleted: number }> {
|
||||
const sql = this.sql;
|
||||
const prefixes = opts?.excludeSourcePrefixes;
|
||||
if (prefixes && prefixes.length > 0) {
|
||||
// #1928: keep rows whose `source` matches an excluded prefix (e.g.
|
||||
// `cli:` conversation facts). COALESCE so NULL/empty-source fence rows
|
||||
// stay deletable — only the explicitly-protected prefixes survive.
|
||||
const patterns = prefixes.map(p => `${p}%`);
|
||||
const result = await sql`
|
||||
DELETE FROM facts
|
||||
WHERE source_id = ${source_id}
|
||||
AND source_markdown_slug = ${slug}
|
||||
AND NOT (COALESCE(source, '') LIKE ANY(${patterns}))
|
||||
`;
|
||||
return { deleted: result.count ?? 0 };
|
||||
}
|
||||
const result = await sql`
|
||||
DELETE FROM facts WHERE source_id = ${source_id} AND source_markdown_slug = ${slug}
|
||||
`;
|
||||
@@ -5119,7 +5172,7 @@ export class PostgresEngine implements BrainEngine {
|
||||
const toQual = resolved.map(e => e.to_symbol_qualified);
|
||||
const edgeTypes = resolved.map(e => e.edge_type);
|
||||
const metas = resolved.map(e => JSON.stringify(e.edge_metadata ?? {}));
|
||||
const sources = resolved.map(e => e.source_id ?? null);
|
||||
const sources = resolved.map(e => e.source_id ?? 'default');
|
||||
const res = await sql`
|
||||
INSERT INTO code_edges_chunk (from_chunk_id, to_chunk_id, from_symbol_qualified, to_symbol_qualified, edge_type, edge_metadata, source_id)
|
||||
SELECT * FROM unnest(
|
||||
@@ -5139,7 +5192,7 @@ export class PostgresEngine implements BrainEngine {
|
||||
const toQual = unresolved.map(e => e.to_symbol_qualified);
|
||||
const edgeTypes = unresolved.map(e => e.edge_type);
|
||||
const metas = unresolved.map(e => JSON.stringify(e.edge_metadata ?? {}));
|
||||
const sources = unresolved.map(e => e.source_id ?? null);
|
||||
const sources = unresolved.map(e => e.source_id ?? 'default');
|
||||
const res = await sql`
|
||||
INSERT INTO code_edges_symbol (from_chunk_id, from_symbol_qualified, to_symbol_qualified, edge_type, edge_metadata, source_id)
|
||||
SELECT * FROM unnest(
|
||||
|
||||
@@ -390,6 +390,11 @@ CREATE INDEX IF NOT EXISTS idx_code_edges_chunk_to
|
||||
ON code_edges_chunk(to_chunk_id, edge_type);
|
||||
CREATE INDEX IF NOT EXISTS idx_code_edges_chunk_to_symbol
|
||||
ON code_edges_chunk(to_symbol_qualified, edge_type);
|
||||
-- getCalleesOf filters on from_symbol_qualified; without this index every
|
||||
-- callee lookup is a sequential scan, amplified per-BFS-node by the
|
||||
-- recursive code walk. Mirrors migration v116.
|
||||
CREATE INDEX IF NOT EXISTS idx_code_edges_chunk_from_symbol
|
||||
ON code_edges_chunk(from_symbol_qualified);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS code_edges_symbol (
|
||||
id SERIAL PRIMARY KEY,
|
||||
@@ -407,6 +412,10 @@ CREATE INDEX IF NOT EXISTS idx_code_edges_symbol_from
|
||||
ON code_edges_symbol(from_chunk_id, edge_type);
|
||||
CREATE INDEX IF NOT EXISTS idx_code_edges_symbol_to
|
||||
ON code_edges_symbol(to_symbol_qualified, edge_type);
|
||||
-- getCalleesOf companion to idx_code_edges_chunk_from_symbol above.
|
||||
-- Mirrors migration v116.
|
||||
CREATE INDEX IF NOT EXISTS idx_code_edges_symbol_from_symbol
|
||||
ON code_edges_symbol(from_symbol_qualified);
|
||||
|
||||
-- ============================================================
|
||||
-- links: cross-references between pages
|
||||
@@ -706,7 +715,7 @@ CREATE TABLE IF NOT EXISTS op_checkpoint_paths (
|
||||
-- (volunteer_context op / retrieval-reflex pointers / gbrain watch).
|
||||
-- "Used" derives from pages.last_retrieved_at > volunteered_at; rationale is
|
||||
-- a deterministic template string, never raw conversation text. 90-day prune
|
||||
-- in the dream cycle's purge phase. Mirrors migration v116.
|
||||
-- in the dream cycle's purge phase. Mirrors migration v117.
|
||||
CREATE TABLE IF NOT EXISTS context_volunteer_events (
|
||||
id BIGSERIAL PRIMARY KEY,
|
||||
source_id TEXT NOT NULL,
|
||||
|
||||
+10
-1
@@ -738,7 +738,16 @@ export function attributeKnob<K extends keyof ModeBundle>(
|
||||
// to take effect immediately (one-time global cache cold-miss on upgrade; refills
|
||||
// within cache.ttl_seconds). Same cache-key-contamination convention as the
|
||||
// autocut / title_boost / graph_signals bumps above.
|
||||
export const KNOBS_HASH_VERSION = 10;
|
||||
//
|
||||
// bump 10→11 (#1400 input_type fix): asymmetric embedding models (zembed-1
|
||||
// hosted or local, Voyage v3+) had their query-side input_type stripped by
|
||||
// the AI SDK before the wire, so every cached row's key embedding AND result
|
||||
// set were computed with document-side query vectors. The fix changes what
|
||||
// embedQuery() produces for those providers; pre-fix rows must not be served
|
||||
// to post-fix lookups. Same one-time global cold-miss pattern as the bumps
|
||||
// above (the hash is global, not per-provider); refills within
|
||||
// cache.ttl_seconds (3600s default).
|
||||
export const KNOBS_HASH_VERSION = 11;
|
||||
|
||||
/**
|
||||
* v0.36 (D8 / CDX-2) — second-arg context for the cache key. The
|
||||
|
||||
@@ -241,6 +241,19 @@ function matchesAnyGlob(path: string, patterns?: string[]): boolean {
|
||||
*/
|
||||
const PRUNE_DIR_NAMES = new Set<string>([
|
||||
'node_modules',
|
||||
// Dependency / build-output trees that are git-ignored on virtually every
|
||||
// repo and never contain hand-authored source worth indexing. `vendor`
|
||||
// (PHP Composer / Go / Ruby bundle), `dist` + `build` (compiled output).
|
||||
// Closes the silent-pollution bug where a Laravel/PHP repo's full code sync
|
||||
// walked ~50k `vendor/` files (#1483 / #1159 / maintainer #1942).
|
||||
'vendor',
|
||||
'dist',
|
||||
'build',
|
||||
// Python venv: vendored dependency tree, the `node_modules` analogue (#2020).
|
||||
// Like `node_modules` it lacks a leading dot so isSyncable's dot-prefix
|
||||
// exclusion misses it; explicit entry keeps incremental sync consistent
|
||||
// with the first-sync walker in commands/import.ts.
|
||||
'venv',
|
||||
'.raw',
|
||||
'ops',
|
||||
]);
|
||||
|
||||
@@ -0,0 +1,120 @@
|
||||
/**
|
||||
* #2038 — idx_timeline_dedup schema-drift self-heal.
|
||||
*
|
||||
* Migration v102 (`timeline_entries_source_in_dedup`) widens the dedup index
|
||||
* from (page_id, date, summary) to (page_id, date, summary, source). It was
|
||||
* renumbered from v99 during a master merge, so a brain that ran the OLD v99
|
||||
* variant has its version counter stamped PAST v102 while the index stayed
|
||||
* 3-column. `runMigrations` then can't see the drift (it early-returns when no
|
||||
* version is pending), and every `addTimelineEntry(esBatch)` fails with
|
||||
* "no unique or exclusion constraint matching the ON CONFLICT specification"
|
||||
* because both insert sites infer on the 4-column tuple — timeline writes
|
||||
* silently break brain-wide.
|
||||
*
|
||||
* The version counter can't detect this, so the repair is keyed off the actual
|
||||
* index SHAPE and runs on every migrate pass (including the no-pending path).
|
||||
* Idempotent: a no-op when the index is already 4-column.
|
||||
*/
|
||||
|
||||
import type { BrainEngine } from './engine.ts';
|
||||
|
||||
const INDEX_NAME = 'idx_timeline_dedup';
|
||||
const EXPECTED_COLUMNS = ['page_id', 'date', 'summary', 'source'];
|
||||
|
||||
export interface TimelineDedupStatus {
|
||||
/** The timeline_entries table exists (nothing to repair if not). */
|
||||
tablePresent: boolean;
|
||||
/** The index exists. */
|
||||
indexPresent: boolean;
|
||||
/** Indexed columns in order (empty when the index is absent). */
|
||||
columns: string[];
|
||||
/** Index exists in the wrong (pre-v102) shape — needs a rebuild. */
|
||||
needsRepair: boolean;
|
||||
}
|
||||
|
||||
/** Parse the column list out of a pg_indexes `indexdef` string. */
|
||||
function parseIndexColumns(indexdef: string): string[] {
|
||||
const open = indexdef.lastIndexOf('(');
|
||||
const close = indexdef.lastIndexOf(')');
|
||||
if (open < 0 || close < 0 || close < open) return [];
|
||||
return indexdef
|
||||
.slice(open + 1, close)
|
||||
.split(',')
|
||||
.map(c => c.trim().split(/\s+/)[0]) // drop any "col DESC"/opclass suffix
|
||||
.filter(Boolean);
|
||||
}
|
||||
|
||||
export async function checkTimelineDedupIndex(engine: BrainEngine): Promise<TimelineDedupStatus> {
|
||||
const tbl = await engine.executeRaw<{ reg: string | null }>(
|
||||
`SELECT to_regclass('timeline_entries')::text AS reg`,
|
||||
);
|
||||
const tablePresent = !!tbl[0]?.reg;
|
||||
if (!tablePresent) {
|
||||
return { tablePresent: false, indexPresent: false, columns: [], needsRepair: false };
|
||||
}
|
||||
const rows = await engine.executeRaw<{ indexdef: string }>(
|
||||
`SELECT indexdef FROM pg_indexes WHERE indexname = $1`,
|
||||
[INDEX_NAME],
|
||||
);
|
||||
const indexPresent = rows.length > 0;
|
||||
const columns = indexPresent ? parseIndexColumns(rows[0].indexdef) : [];
|
||||
const correct =
|
||||
columns.length === EXPECTED_COLUMNS.length &&
|
||||
EXPECTED_COLUMNS.every((c, i) => columns[i] === c);
|
||||
// An ABSENT index is also "needs repair" — the migration that creates it was
|
||||
// skipped. (A fresh brain always has it, created by the migration chain.)
|
||||
return { tablePresent, indexPresent, columns, needsRepair: !correct };
|
||||
}
|
||||
|
||||
export interface TimelineDedupRepairResult {
|
||||
repaired: boolean;
|
||||
before: string[];
|
||||
collapsedDuplicates: number;
|
||||
reason: 'already_correct' | 'no_table' | 'rebuilt';
|
||||
}
|
||||
|
||||
/**
|
||||
* Heal the index if it's missing the v102 4-column shape. Dedupes FIRST —
|
||||
* the loose 3-column index let rows differing only by `source` coexist, and
|
||||
* `CREATE UNIQUE INDEX` would throw on those collisions otherwise. Keeps the
|
||||
* earliest row (min ctid) of each 4-tuple group.
|
||||
*/
|
||||
export async function repairTimelineDedupIndex(engine: BrainEngine): Promise<TimelineDedupRepairResult> {
|
||||
const status = await checkTimelineDedupIndex(engine);
|
||||
if (!status.tablePresent) {
|
||||
return { repaired: false, before: [], collapsedDuplicates: 0, reason: 'no_table' };
|
||||
}
|
||||
if (!status.needsRepair) {
|
||||
return { repaired: false, before: status.columns, collapsedDuplicates: 0, reason: 'already_correct' };
|
||||
}
|
||||
|
||||
// Keep the lowest `id` per 4-tuple group — deterministic and consistent with
|
||||
// the existing v-migration dedup rule (`a.id > b.id`), unlike `ctid` which is
|
||||
// a physical tuple location that can preserve an arbitrary duplicate.
|
||||
const del = await engine.executeRaw<{ n: string }>(
|
||||
`WITH d AS (
|
||||
DELETE FROM timeline_entries t
|
||||
USING (
|
||||
SELECT page_id, date, summary, source, MIN(id) AS keep
|
||||
FROM timeline_entries
|
||||
GROUP BY page_id, date, summary, source
|
||||
HAVING COUNT(*) > 1
|
||||
) dup
|
||||
WHERE t.page_id = dup.page_id
|
||||
AND t.date = dup.date
|
||||
AND t.summary = dup.summary
|
||||
AND t.source IS NOT DISTINCT FROM dup.source
|
||||
AND t.id <> dup.keep
|
||||
RETURNING 1
|
||||
)
|
||||
SELECT COUNT(*)::text AS n FROM d`,
|
||||
);
|
||||
const collapsedDuplicates = parseInt(del[0]?.n ?? '0', 10);
|
||||
|
||||
await engine.executeRaw(`DROP INDEX IF EXISTS ${INDEX_NAME}`);
|
||||
await engine.executeRaw(
|
||||
`CREATE UNIQUE INDEX IF NOT EXISTS ${INDEX_NAME}
|
||||
ON timeline_entries(page_id, date, summary, source)`,
|
||||
);
|
||||
return { repaired: true, before: status.columns, collapsedDuplicates, reason: 'rebuilt' };
|
||||
}
|
||||
+48
-11
@@ -22,7 +22,7 @@
|
||||
*/
|
||||
|
||||
import { existsSync, statSync, mkdirSync, writeFileSync, renameSync, unlinkSync } from 'fs';
|
||||
import { dirname } from 'path';
|
||||
import { dirname, join } from 'path';
|
||||
import { randomBytes } from 'crypto';
|
||||
import type { BrainEngine } from './engine.ts';
|
||||
import { serializePageToMarkdown, resolvePageFilePath } from './markdown.ts';
|
||||
@@ -37,12 +37,16 @@ export interface WriteThroughResult {
|
||||
path?: string;
|
||||
/**
|
||||
* Non-error reasons the file was not written:
|
||||
* - no_repo_configured: `sync.repo_path` is unset (DB-only by design).
|
||||
* - repo_not_found: `sync.repo_path` set but missing / not a directory.
|
||||
* - no_repo_configured: the resolved target (source `local_path` or, for a
|
||||
* sole-source brain, `sync.repo_path`) is unset (DB-only by design).
|
||||
* - repo_not_found: target set but missing / not a directory.
|
||||
* - source_repo_belongs_to_other_source: the assigned source has no
|
||||
* `local_path`, and `sync.repo_path` is another source's own working tree
|
||||
* — #2018: writing here would pollute that sibling's repo, so we skip.
|
||||
* - page_not_found_after_write: the DB row isn't readable back (the caller's
|
||||
* DB write failed or targeted a different source).
|
||||
*/
|
||||
skipped?: 'no_repo_configured' | 'repo_not_found' | 'page_not_found_after_write';
|
||||
skipped?: 'no_repo_configured' | 'repo_not_found' | 'source_repo_belongs_to_other_source' | 'page_not_found_after_write';
|
||||
/** Set when the render/write/rename itself threw (EACCES, ENOTDIR, disk full). */
|
||||
error?: string;
|
||||
}
|
||||
@@ -67,12 +71,46 @@ export async function writePageThrough(
|
||||
): Promise<WriteThroughResult> {
|
||||
const sourceId = opts.sourceId ?? 'default';
|
||||
try {
|
||||
const repoPath = await engine.getConfig('sync.repo_path');
|
||||
if (!repoPath) {
|
||||
return { written: false, skipped: 'no_repo_configured' };
|
||||
}
|
||||
if (!existsSync(repoPath) || !statSync(repoPath).isDirectory()) {
|
||||
return { written: false, skipped: 'repo_not_found' };
|
||||
// #2018: pick the disk target so a page is NEVER written into a different
|
||||
// source's working tree. Two legitimate topologies, plus the leak guard:
|
||||
// 1. The assigned source has its OWN `local_path` (a separate working
|
||||
// tree) → write at that tree's root (matches how `scanOneSource` reads
|
||||
// it back; never nested under `.sources/`).
|
||||
// 2. No per-source `local_path` → nest under the host repo
|
||||
// (`sync.repo_path`): default at the root, non-default under
|
||||
// `.sources/<id>/` (the established multi-source layout).
|
||||
// 3. LEAK GUARD: if `sync.repo_path` is literally ANOTHER source's own
|
||||
// `local_path`, nesting this page there would pollute that sibling's
|
||||
// git repo (the reported bug). Skip instead.
|
||||
let filePath: string;
|
||||
const srcRows = await engine.executeRaw<{ local_path: string | null }>(
|
||||
`SELECT local_path FROM sources WHERE id = $1`,
|
||||
[sourceId],
|
||||
);
|
||||
const sourceLocalPath = srcRows[0]?.local_path ?? null;
|
||||
if (sourceLocalPath) {
|
||||
if (!existsSync(sourceLocalPath) || !statSync(sourceLocalPath).isDirectory()) {
|
||||
return { written: false, skipped: 'repo_not_found' };
|
||||
}
|
||||
filePath = join(sourceLocalPath, `${slug}.md`);
|
||||
} else {
|
||||
const repoPath = await engine.getConfig('sync.repo_path');
|
||||
if (!repoPath) {
|
||||
return { written: false, skipped: 'no_repo_configured' };
|
||||
}
|
||||
if (!existsSync(repoPath) || !statSync(repoPath).isDirectory()) {
|
||||
return { written: false, skipped: 'repo_not_found' };
|
||||
}
|
||||
// Leak guard: refuse to write into a path that is some OTHER source's
|
||||
// own working tree (#2018).
|
||||
const collide = await engine.executeRaw<{ one: number }>(
|
||||
`SELECT 1 AS one FROM sources WHERE id <> $1 AND local_path = $2 LIMIT 1`,
|
||||
[sourceId, repoPath],
|
||||
);
|
||||
if (collide.length > 0) {
|
||||
return { written: false, skipped: 'source_repo_belongs_to_other_source' };
|
||||
}
|
||||
filePath = resolvePageFilePath(repoPath, slug, sourceId);
|
||||
}
|
||||
|
||||
const writtenPage = await engine.getPage(slug, { sourceId });
|
||||
@@ -85,7 +123,6 @@ export async function writePageThrough(
|
||||
frontmatterOverrides: opts.frontmatterOverrides,
|
||||
});
|
||||
|
||||
const filePath = resolvePageFilePath(repoPath, slug, sourceId);
|
||||
mkdirSync(dirname(filePath), { recursive: true });
|
||||
|
||||
// Atomic write: unique temp sibling + rename. Unique name (pid + random)
|
||||
|
||||
@@ -34,6 +34,8 @@ import { VERSION } from '../version.ts';
|
||||
import { dispatchToolCall } from './dispatch.ts';
|
||||
import { buildDefaultLimiters, type RateLimiter } from './rate-limit.ts';
|
||||
import { sqlQueryForEngine } from '../core/sql-query.ts';
|
||||
import { parseLegacyTokenScope } from '../core/legacy-token-scope.ts';
|
||||
export { parseLegacyTokenScope };
|
||||
|
||||
const DEFAULT_BODY_CAP = 1024 * 1024; // 1 MiB
|
||||
|
||||
@@ -84,28 +86,8 @@ interface AuthResult {
|
||||
auth?: AuthInfo;
|
||||
}
|
||||
|
||||
/**
|
||||
* #1336: derive a legacy bearer token's source scope from its stored
|
||||
* `permissions.source_id`. An ARRAY value is a federated_read grant →
|
||||
* `allowedSources` (scoped reads across exactly those sources). A STRING value
|
||||
* scopes the scalar floor. Anything else → 'default' (preserves pre-v0.34
|
||||
* behavior). NEVER widened to "all": an empty/garbage value keeps the 'default'
|
||||
* floor and no federated grant.
|
||||
*/
|
||||
export function parseLegacyTokenScope(rawSource: unknown): { sourceId: string; allowedSources?: string[] } {
|
||||
if (Array.isArray(rawSource)) {
|
||||
const allowedSources = (rawSource as unknown[]).filter(s => typeof s === 'string' && s.length > 0) as string[];
|
||||
if (allowedSources.length > 0) {
|
||||
// Scalar floor: the first granted source (write authority); reads span the array.
|
||||
return { sourceId: allowedSources[0], allowedSources };
|
||||
}
|
||||
return { sourceId: 'default' };
|
||||
}
|
||||
if (typeof rawSource === 'string' && rawSource.length > 0) {
|
||||
return { sourceId: rawSource };
|
||||
}
|
||||
return { sourceId: 'default' };
|
||||
}
|
||||
/* Legacy token source-scope parsing lives in core/legacy-token-scope.ts and is
|
||||
* re-exported above so the legacy HTTP transport and OAuth provider cannot drift. */
|
||||
|
||||
/** Read up to `cap` bytes off req.body. Returns null if cap exceeded. */
|
||||
async function readBodyWithCap(req: Request, cap: number): Promise<string | null> {
|
||||
|
||||
+10
-1
@@ -386,6 +386,11 @@ CREATE INDEX IF NOT EXISTS idx_code_edges_chunk_to
|
||||
ON code_edges_chunk(to_chunk_id, edge_type);
|
||||
CREATE INDEX IF NOT EXISTS idx_code_edges_chunk_to_symbol
|
||||
ON code_edges_chunk(to_symbol_qualified, edge_type);
|
||||
-- getCalleesOf filters on from_symbol_qualified; without this index every
|
||||
-- callee lookup is a sequential scan, amplified per-BFS-node by the
|
||||
-- recursive code walk. Mirrors migration v116.
|
||||
CREATE INDEX IF NOT EXISTS idx_code_edges_chunk_from_symbol
|
||||
ON code_edges_chunk(from_symbol_qualified);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS code_edges_symbol (
|
||||
id SERIAL PRIMARY KEY,
|
||||
@@ -403,6 +408,10 @@ CREATE INDEX IF NOT EXISTS idx_code_edges_symbol_from
|
||||
ON code_edges_symbol(from_chunk_id, edge_type);
|
||||
CREATE INDEX IF NOT EXISTS idx_code_edges_symbol_to
|
||||
ON code_edges_symbol(to_symbol_qualified, edge_type);
|
||||
-- getCalleesOf companion to idx_code_edges_chunk_from_symbol above.
|
||||
-- Mirrors migration v116.
|
||||
CREATE INDEX IF NOT EXISTS idx_code_edges_symbol_from_symbol
|
||||
ON code_edges_symbol(from_symbol_qualified);
|
||||
|
||||
-- ============================================================
|
||||
-- links: cross-references between pages
|
||||
@@ -702,7 +711,7 @@ CREATE TABLE IF NOT EXISTS op_checkpoint_paths (
|
||||
-- (volunteer_context op / retrieval-reflex pointers / gbrain watch).
|
||||
-- "Used" derives from pages.last_retrieved_at > volunteered_at; rationale is
|
||||
-- a deterministic template string, never raw conversation text. 90-day prune
|
||||
-- in the dream cycle's purge phase. Mirrors migration v116.
|
||||
-- in the dream cycle's purge phase. Mirrors migration v117.
|
||||
CREATE TABLE IF NOT EXISTS context_volunteer_events (
|
||||
id BIGSERIAL PRIMARY KEY,
|
||||
source_id TEXT NOT NULL,
|
||||
|
||||
@@ -53,13 +53,15 @@ describe('zeroEntropyCompatFetch — shim structural shape', () => {
|
||||
expect(src).not.toContain('/v1/v1/');
|
||||
});
|
||||
|
||||
test('body injects input_type default "document"', async () => {
|
||||
test('body recovers threaded input_type, defaulting to "document"', async () => {
|
||||
const src = await Bun.file(GATEWAY_PATH).text();
|
||||
// The wrapper defaults input_type to 'document' when caller didn't
|
||||
// thread one (matches the document-side correctness for sync /
|
||||
// import / embed CLI paths).
|
||||
// The wrapper recovers the SDK-stripped input_type from
|
||||
// __embedInputTypeStore (#1400) and still defaults to 'document' when
|
||||
// no caller threaded one (document-side correctness for sync / import /
|
||||
// embed CLI paths). Wire-level behavioral coverage lives in
|
||||
// test/embed-input-type-wire.test.ts.
|
||||
expect(src).toMatch(/parsed\.input_type\s*===\s*undefined/);
|
||||
expect(src).toMatch(/parsed\.input_type\s*=\s*['"]document['"]/);
|
||||
expect(src).toMatch(/parsed\.input_type\s*=\s*__embedInputTypeStore\.getStore\(\)\s*\?\?\s*['"]document['"]/);
|
||||
});
|
||||
|
||||
test('body forces encoding_format=float (CDX2-F2)', async () => {
|
||||
|
||||
@@ -0,0 +1,56 @@
|
||||
/**
|
||||
* Structural regression — the DISCONNECT_HARD_DEADLINE_MS force-exit timer in
|
||||
* cli.ts main() must be armed at TEARDOWN ENTRY (inside the finally, before
|
||||
* the drain + disconnect), never before the op-dispatch try block.
|
||||
*
|
||||
* Pre-fix bug: the 10s unref'd setTimeout was armed BEFORE the try, so any op
|
||||
* whose handler ran past 10s wall-clock was killed mid-flight with
|
||||
* process.exit(0) and ZERO stdout — an empty "success" indistinguishable from
|
||||
* no results (a healthy `gbrain search` on a slow Postgres pooler hit this on
|
||||
* every run). Armed in the finally, the timer still bounds a hung
|
||||
* drain/disconnect (the C13 contract) but can no longer kill a
|
||||
* slow-but-progressing op body.
|
||||
*
|
||||
* Source-grep is the right tool here (same rationale as
|
||||
* fix-wave-structural.test.ts): the rule is "this arming must stay at this
|
||||
* location". A behavioral test would need >10s of real wall-clock plus a
|
||||
* deliberately slow op handler in a spawned CLI — slow and flaky by
|
||||
* construction.
|
||||
*/
|
||||
|
||||
import { describe, test, expect } from 'bun:test';
|
||||
import { readFileSync } from 'fs';
|
||||
|
||||
describe('cli.ts — disconnect hard-deadline armed at teardown entry, not before the op body', () => {
|
||||
test('forceExitTimer setTimeout lives inside the finally, gated on the daemon guard, before the drain', () => {
|
||||
const src = readFileSync('src/cli.ts', 'utf8');
|
||||
|
||||
const decl = src.indexOf('const DISCONNECT_HARD_DEADLINE_MS');
|
||||
expect(decl).toBeGreaterThan(-1);
|
||||
const tryIdx = src.indexOf('try {', decl);
|
||||
expect(tryIdx).toBeGreaterThan(-1);
|
||||
const finallyIdx = src.indexOf('} finally {', tryIdx);
|
||||
expect(finallyIdx).toBeGreaterThan(-1);
|
||||
const armIdx = src.indexOf('forceExitTimer = setTimeout', decl);
|
||||
expect(armIdx).toBeGreaterThan(-1);
|
||||
const drainIdx = src.indexOf('drainAllBackgroundWorkForCliExit', finallyIdx);
|
||||
expect(drainIdx).toBeGreaterThan(-1);
|
||||
|
||||
// NO arming between the deadline declaration and the op-body try — a
|
||||
// pre-try timer kills slow-but-progressing op handlers mid-flight with
|
||||
// exit 0 and empty stdout. (`setTimeout(` matches only a call site; the
|
||||
// `ReturnType<typeof setTimeout>` type annotation stays allowed.)
|
||||
expect(src.slice(decl, tryIdx)).not.toContain('setTimeout(');
|
||||
|
||||
// The arming sits AFTER the finally opens (teardown entry) and BEFORE the
|
||||
// drain + disconnect it exists to bound.
|
||||
expect(armIdx).toBeGreaterThan(finallyIdx);
|
||||
expect(armIdx).toBeLessThan(drainIdx);
|
||||
|
||||
// Still gated on the daemon-survival guard so `serve` stays alive, and
|
||||
// still unref'd + cleared on clean teardown.
|
||||
expect(src.slice(finallyIdx, drainIdx)).toMatch(/if \(shouldForceExitAfterMain\(\)\)/);
|
||||
expect(src.slice(finallyIdx, drainIdx)).toContain('forceExitTimer.unref?.()');
|
||||
expect(src.slice(drainIdx)).toContain('if (forceExitTimer) clearTimeout(forceExitTimer)');
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,144 @@
|
||||
import { mkdtempSync, rmSync, writeFileSync } from 'fs';
|
||||
import { tmpdir } from 'os';
|
||||
import { join } from 'path';
|
||||
import { describe, expect, test } from 'bun:test';
|
||||
import { dotenvValuesForKey, effectiveEnvDatabaseUrl } from '../src/core/config.ts';
|
||||
import { withEnv } from './helpers/with-env.ts';
|
||||
|
||||
// #427 — Bun auto-loads `.env` from the process cwd, so running gbrain inside
|
||||
// any web-app checkout whose `.env` defines DATABASE_URL silently retargets
|
||||
// the brain at that app's database. These tests pin the guard:
|
||||
// effectiveEnvDatabaseUrl() must treat a DATABASE_URL whose value matches a
|
||||
// cwd .env assignment as file-origin (ignored), while still honoring
|
||||
// GBRAIN_DATABASE_URL and genuinely exported DATABASE_URLs.
|
||||
//
|
||||
// The dir is injected instead of process.chdir'd so these tests stay safe in
|
||||
// the parallel shard runner (cwd is process-global, same reason with-env.ts
|
||||
// exists for process.env).
|
||||
|
||||
function tmpProject(envFiles: Record<string, string>): string {
|
||||
const dir = mkdtempSync(join(tmpdir(), 'gbrain-env-hijack-'));
|
||||
for (const [name, content] of Object.entries(envFiles)) {
|
||||
writeFileSync(join(dir, name), content);
|
||||
}
|
||||
return dir;
|
||||
}
|
||||
|
||||
describe('dotenvValuesForKey', () => {
|
||||
test('collects assignments across all auto-loaded .env variants', () => {
|
||||
const dir = tmpProject({
|
||||
'.env': 'DATABASE_URL=postgres://app:pw@prod.example.test:5432/app\n',
|
||||
'.env.local': "DATABASE_URL='postgres://app:pw@local.example.test:5432/app'\n",
|
||||
});
|
||||
try {
|
||||
const values = dotenvValuesForKey('DATABASE_URL', dir);
|
||||
expect(values.has('postgres://app:pw@prod.example.test:5432/app')).toBe(true);
|
||||
expect(values.has('postgres://app:pw@local.example.test:5432/app')).toBe(true);
|
||||
} finally {
|
||||
rmSync(dir, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
test('handles export prefix, double quotes, comments, and unrelated keys', () => {
|
||||
const dir = tmpProject({
|
||||
'.env': [
|
||||
'# comment line',
|
||||
'export DATABASE_URL="postgres://quoted.example.test/db"',
|
||||
'OTHER_KEY=not-a-db-url',
|
||||
'EMPTY=',
|
||||
'',
|
||||
].join('\n'),
|
||||
});
|
||||
try {
|
||||
const values = dotenvValuesForKey('DATABASE_URL', dir);
|
||||
expect(values.has('postgres://quoted.example.test/db')).toBe(true);
|
||||
expect(values.size).toBe(1);
|
||||
} finally {
|
||||
rmSync(dir, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
test('returns empty set when no .env files exist', () => {
|
||||
const dir = mkdtempSync(join(tmpdir(), 'gbrain-env-hijack-none-'));
|
||||
try {
|
||||
expect(dotenvValuesForKey('DATABASE_URL', dir).size).toBe(0);
|
||||
} finally {
|
||||
rmSync(dir, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
describe('effectiveEnvDatabaseUrl (#427 guard)', () => {
|
||||
const HIJACK_URL = 'postgres://app:pw@victim-prod.example.test:5432/app';
|
||||
const OPERATOR_URL = 'postgres://operator@deliberate.example.test:5432/brain';
|
||||
|
||||
test('ignores DATABASE_URL whose value matches a cwd .env assignment', async () => {
|
||||
const dir = tmpProject({ '.env': `DATABASE_URL=${HIJACK_URL}\n` });
|
||||
try {
|
||||
await withEnv(
|
||||
{ GBRAIN_DATABASE_URL: undefined, DATABASE_URL: HIJACK_URL },
|
||||
() => {
|
||||
expect(effectiveEnvDatabaseUrl(dir)).toBeUndefined();
|
||||
},
|
||||
);
|
||||
} finally {
|
||||
rmSync(dir, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
test('honors an exported DATABASE_URL that differs from the cwd .env', async () => {
|
||||
const dir = tmpProject({ '.env': `DATABASE_URL=${HIJACK_URL}\n` });
|
||||
try {
|
||||
await withEnv(
|
||||
{ GBRAIN_DATABASE_URL: undefined, DATABASE_URL: OPERATOR_URL },
|
||||
() => {
|
||||
expect(effectiveEnvDatabaseUrl(dir)).toBe(OPERATOR_URL);
|
||||
},
|
||||
);
|
||||
} finally {
|
||||
rmSync(dir, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
test('GBRAIN_DATABASE_URL is never ignored, even when it matches the cwd .env', async () => {
|
||||
const dir = tmpProject({ '.env': `GBRAIN_DATABASE_URL=${OPERATOR_URL}\nDATABASE_URL=${HIJACK_URL}\n` });
|
||||
try {
|
||||
await withEnv(
|
||||
{ GBRAIN_DATABASE_URL: OPERATOR_URL, DATABASE_URL: HIJACK_URL },
|
||||
() => {
|
||||
expect(effectiveEnvDatabaseUrl(dir)).toBe(OPERATOR_URL);
|
||||
},
|
||||
);
|
||||
} finally {
|
||||
rmSync(dir, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
test('honors DATABASE_URL when no .env files exist in cwd', async () => {
|
||||
const dir = mkdtempSync(join(tmpdir(), 'gbrain-env-hijack-clean-'));
|
||||
try {
|
||||
await withEnv(
|
||||
{ GBRAIN_DATABASE_URL: undefined, DATABASE_URL: OPERATOR_URL },
|
||||
() => {
|
||||
expect(effectiveEnvDatabaseUrl(dir)).toBe(OPERATOR_URL);
|
||||
},
|
||||
);
|
||||
} finally {
|
||||
rmSync(dir, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
test('returns undefined when neither env var is set', async () => {
|
||||
const dir = mkdtempSync(join(tmpdir(), 'gbrain-env-hijack-unset-'));
|
||||
try {
|
||||
await withEnv(
|
||||
{ GBRAIN_DATABASE_URL: undefined, DATABASE_URL: undefined },
|
||||
() => {
|
||||
expect(effectiveEnvDatabaseUrl(dir)).toBeUndefined();
|
||||
},
|
||||
);
|
||||
} finally {
|
||||
rmSync(dir, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
});
|
||||
@@ -317,8 +317,10 @@ describe('Eng-review D3 — executeRaw has no per-call retry wrapper', () => {
|
||||
it('Supervisor still has the 3-strikes-then-reconnect path', () => {
|
||||
const src = readFileSync(resolve('src/core/minions/supervisor.ts'), 'utf-8');
|
||||
expect(src).toContain('consecutiveHealthFailures');
|
||||
// Supervisor invokes reconnect via a typed cast after 3 consecutive failures.
|
||||
expect(src).toMatch(/reconnect\(\): Promise<void>/);
|
||||
// #2034: reconnect() is now a first-class BrainEngine method, so the
|
||||
// supervisor calls it directly after 3 consecutive failures (the prior
|
||||
// `(engine as unknown as { reconnect(): Promise<void> })` cast was removed).
|
||||
expect(src).toContain('this.engine.reconnect(');
|
||||
expect(src).toContain('this.consecutiveHealthFailures >= 3');
|
||||
});
|
||||
});
|
||||
|
||||
@@ -136,15 +136,17 @@ describe('D2 — knobsHash differs across cross-modal knob values', () => {
|
||||
return resolveSearchMode({ mode: 'balanced' });
|
||||
}
|
||||
|
||||
test('KNOBS_HASH_VERSION is 10 (cross-modal still appended; 9→10 relational recall)', () => {
|
||||
test('KNOBS_HASH_VERSION is 11 (cross-modal still appended; 10→11 asymmetric input_type fix)', () => {
|
||||
// v0.35 ladder: 1→2 reranker, 2→3 floor_ratio. v0.36 piggybacks on v=3
|
||||
// with 7 cross-modal knobs + column/provider context. v0.40.4 (salem) +
|
||||
// v0.39 T21 (master) bump to v=4 for graph_signals + schema-pack fields.
|
||||
// v0.40.3.0 D8 bumps to v=5 (sequenced behind salem's v=4 graph-signals).
|
||||
// v0.41.22.0 (type-unification): 5→6 for alias_resolved post-fusion boost.
|
||||
// T2: 6→7 title_boost. v0.42.3.0: 7→8 autocut. issue #1777: 8→9 archive/ demote.
|
||||
// v0.43: 9→10 relational recall arm.
|
||||
expect(KNOBS_HASH_VERSION).toBe(10);
|
||||
// v0.43: 9→10 relational recall arm. #1400: 10→11 query-side input_type
|
||||
// finally reaches asymmetric providers — pre-fix rows were keyed on
|
||||
// document-side query vectors.
|
||||
expect(KNOBS_HASH_VERSION).toBe(11);
|
||||
});
|
||||
|
||||
test('flipping unified_multimodal changes the hash', () => {
|
||||
|
||||
@@ -0,0 +1,247 @@
|
||||
/**
|
||||
* #1400 — asymmetric `input_type` must survive the AI SDK boundary and
|
||||
* reach the WIRE BODY.
|
||||
*
|
||||
* test/asymmetric-encoding-contract.test.ts pins that embedQuery() threads
|
||||
* `input_type: 'query'` into the transport's providerOptions. This file
|
||||
* pins the layer BELOW that contract: the AI SDK's openai-compatible
|
||||
* adapter validates providerOptions against a fixed schema and silently
|
||||
* drops `input_type` before building the HTTP body. Without the
|
||||
* `__embedInputTypeStore` recovery in the per-recipe fetch shims, every
|
||||
* query was encoded document-side (ZE shim's hard default) or with no
|
||||
* input_type at all (Voyage, llama-server) — asymmetric retrieval silently
|
||||
* collapsed while the providerOptions-level test stayed green.
|
||||
*
|
||||
* These tests run the REAL SDK transport with a mocked global fetch and
|
||||
* assert on the outbound request body — the only place the regression is
|
||||
* observable.
|
||||
*/
|
||||
|
||||
import { afterEach, beforeEach, describe, expect, test } from 'bun:test';
|
||||
import { configureGateway, embed, embedQuery, resetGateway } from '../src/core/ai/gateway.ts';
|
||||
|
||||
type FetchHandler = (url: string, init: RequestInit) => Promise<Response>;
|
||||
let fetchHandler: FetchHandler | null = null;
|
||||
const origFetch = globalThis.fetch;
|
||||
|
||||
beforeEach(() => {
|
||||
fetchHandler = null;
|
||||
globalThis.fetch = (async (url: string | URL | Request, init?: RequestInit) => {
|
||||
if (!fetchHandler) {
|
||||
throw new Error('fetch called but no handler installed');
|
||||
}
|
||||
return fetchHandler(typeof url === 'string' ? url : url.toString(), init ?? {});
|
||||
}) as typeof fetch;
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
globalThis.fetch = origFetch;
|
||||
resetGateway();
|
||||
});
|
||||
|
||||
/** OpenAI-shaped /v1/embeddings response (llama-server is already OpenAI-shaped). */
|
||||
function openAIShapedResponse(dims: number, count: number): Response {
|
||||
const vec = Array.from({ length: dims }, () => 0.1);
|
||||
return new Response(
|
||||
JSON.stringify({
|
||||
data: Array.from({ length: count }, (_, i) => ({ object: 'embedding', index: i, embedding: vec })),
|
||||
usage: { prompt_tokens: 3, total_tokens: 3 },
|
||||
}),
|
||||
{ status: 200, headers: { 'Content-Type': 'application/json' } },
|
||||
);
|
||||
}
|
||||
|
||||
/** ZE-shaped /v1/models/embed response (zeroEntropyCompatFetch rewrites results→data). */
|
||||
function zeShapedResponse(dims: number, count: number): Response {
|
||||
const vec = Array.from({ length: dims }, () => 0.1);
|
||||
return new Response(
|
||||
JSON.stringify({
|
||||
results: Array.from({ length: count }, () => ({ embedding: vec })),
|
||||
usage: { total_bytes: 12, total_tokens: 3 },
|
||||
}),
|
||||
{ status: 200, headers: { 'Content-Type': 'application/json' } },
|
||||
);
|
||||
}
|
||||
|
||||
/** Voyage-shaped response: base64 Float32 LE embeddings (rewriter decodes). */
|
||||
function voyageShapedResponse(dims: number, count: number): Response {
|
||||
const b64 = Buffer.from(new Float32Array(dims).fill(0.1).buffer).toString('base64');
|
||||
return new Response(
|
||||
JSON.stringify({
|
||||
data: Array.from({ length: count }, (_, i) => ({ object: 'embedding', index: i, embedding: b64 })),
|
||||
usage: { total_tokens: 3 },
|
||||
}),
|
||||
{ status: 200, headers: { 'Content-Type': 'application/json' } },
|
||||
);
|
||||
}
|
||||
|
||||
describe('ZeroEntropy hosted — input_type reaches the wire body', () => {
|
||||
function configureZE() {
|
||||
configureGateway({
|
||||
embedding_model: 'zeroentropyai:zembed-1',
|
||||
embedding_dimensions: 1280,
|
||||
env: { ZEROENTROPY_API_KEY: 'sk-fake' },
|
||||
});
|
||||
}
|
||||
|
||||
test('embedQuery sends input_type=query (not the document default)', async () => {
|
||||
configureZE();
|
||||
let capturedUrl = '';
|
||||
let capturedBody: any = null;
|
||||
fetchHandler = async (url, init) => {
|
||||
capturedUrl = url;
|
||||
capturedBody = JSON.parse(init.body as string);
|
||||
return zeShapedResponse(1280, 1);
|
||||
};
|
||||
|
||||
await embedQuery('what does foo bar do?');
|
||||
// Sanity: the ZE shim ran (URL path rewritten).
|
||||
expect(capturedUrl).toContain('/models/embed');
|
||||
expect(capturedBody.input_type).toBe('query');
|
||||
});
|
||||
|
||||
test('embed (index path) sends input_type=document', async () => {
|
||||
configureZE();
|
||||
let capturedBody: any = null;
|
||||
fetchHandler = async (_url, init) => {
|
||||
capturedBody = JSON.parse(init.body as string);
|
||||
return zeShapedResponse(1280, 1);
|
||||
};
|
||||
|
||||
await embed(['this is a document being indexed']);
|
||||
expect(capturedBody.input_type).toBe('document');
|
||||
});
|
||||
});
|
||||
|
||||
describe('openai-compatible recipes (local/proxy asymmetric models) — input_type reaches the wire body', () => {
|
||||
function configureLlamaServer(modelId: string, dims: number) {
|
||||
configureGateway({
|
||||
embedding_model: `llama-server:${modelId}`,
|
||||
embedding_dimensions: dims,
|
||||
env: {},
|
||||
});
|
||||
}
|
||||
|
||||
test('embedQuery against a local zembed-1 sends input_type=query', async () => {
|
||||
configureLlamaServer('zembed-1', 1280);
|
||||
let capturedUrl = '';
|
||||
let capturedBody: any = null;
|
||||
fetchHandler = async (url, init) => {
|
||||
capturedUrl = url;
|
||||
capturedBody = JSON.parse(init.body as string);
|
||||
return openAIShapedResponse(1280, 1);
|
||||
};
|
||||
|
||||
await embedQuery('what does foo bar do?');
|
||||
// URL untouched — llama-server's /v1/embeddings is already OpenAI-shaped.
|
||||
expect(capturedUrl).toContain('/embeddings');
|
||||
expect(capturedUrl).not.toContain('/models/embed');
|
||||
expect(capturedBody.input_type).toBe('query');
|
||||
});
|
||||
|
||||
test('embed (index path) against a local zembed-1 sends input_type=document', async () => {
|
||||
configureLlamaServer('zembed-1', 1280);
|
||||
let capturedBody: any = null;
|
||||
fetchHandler = async (_url, init) => {
|
||||
capturedBody = JSON.parse(init.body as string);
|
||||
return openAIShapedResponse(1280, 1);
|
||||
};
|
||||
|
||||
await embed(['this is a document being indexed']);
|
||||
expect(capturedBody.input_type).toBe('document');
|
||||
});
|
||||
|
||||
test('non-asymmetric model: wire body carries NO input_type (strict pass-through)', async () => {
|
||||
// dims.ts only threads input_type for recognized asymmetric models;
|
||||
// for anything else the shim must leave the body untouched so vanilla
|
||||
// llama-server deployments see zero wire change.
|
||||
configureLlamaServer('my-gguf', 768);
|
||||
let capturedBody: any = null;
|
||||
fetchHandler = async (_url, init) => {
|
||||
capturedBody = JSON.parse(init.body as string);
|
||||
return openAIShapedResponse(768, 1);
|
||||
};
|
||||
|
||||
await embedQuery('hello');
|
||||
expect(capturedBody).not.toBeNull();
|
||||
expect('input_type' in capturedBody).toBe(false);
|
||||
});
|
||||
|
||||
test('litellm proxying an asymmetric model: embedQuery sends input_type=query', async () => {
|
||||
// The shim is the fallthrough default for every openai-compatible
|
||||
// recipe without its own compat fetch — dims.ts threads input_type by
|
||||
// model id, so a zembed-1 behind a LiteLLM proxy (e.g. fronting vLLM)
|
||||
// gets the same signal as llama-server.
|
||||
configureGateway({
|
||||
embedding_model: 'litellm:zembed-1',
|
||||
embedding_dimensions: 1280,
|
||||
env: { LITELLM_API_KEY: 'sk-fake' },
|
||||
base_urls: { litellm: 'http://localhost:4000' },
|
||||
});
|
||||
let capturedBody: any = null;
|
||||
fetchHandler = async (_url, init) => {
|
||||
capturedBody = JSON.parse(init.body as string);
|
||||
return openAIShapedResponse(1280, 1);
|
||||
};
|
||||
|
||||
await embedQuery('what does foo bar do?');
|
||||
expect(capturedBody.input_type).toBe('query');
|
||||
});
|
||||
|
||||
test('ollama serving an asymmetric model: embedQuery sends input_type=query', async () => {
|
||||
configureGateway({
|
||||
embedding_model: 'ollama:zembed-1',
|
||||
embedding_dimensions: 1280,
|
||||
env: {},
|
||||
});
|
||||
let capturedBody: any = null;
|
||||
fetchHandler = async (_url, init) => {
|
||||
capturedBody = JSON.parse(init.body as string);
|
||||
return openAIShapedResponse(1280, 1);
|
||||
};
|
||||
|
||||
await embedQuery('what does foo bar do?');
|
||||
expect(capturedBody.input_type).toBe('query');
|
||||
});
|
||||
});
|
||||
|
||||
describe('Voyage hosted — input_type reaches the wire body (opt-in preserved)', () => {
|
||||
function configureVoyage() {
|
||||
configureGateway({
|
||||
embedding_model: 'voyage:voyage-3-large',
|
||||
embedding_dimensions: 1024,
|
||||
env: { VOYAGE_API_KEY: 'sk-fake' },
|
||||
});
|
||||
}
|
||||
|
||||
test('embedQuery sends input_type=query', async () => {
|
||||
configureVoyage();
|
||||
let capturedBody: any = null;
|
||||
fetchHandler = async (_url, init) => {
|
||||
capturedBody = JSON.parse(init.body as string);
|
||||
return voyageShapedResponse(1024, 1);
|
||||
};
|
||||
|
||||
await embedQuery('what does foo bar do?');
|
||||
expect(capturedBody.input_type).toBe('query');
|
||||
// Existing voyage translation still applies on the same body.
|
||||
expect(capturedBody.output_dimension).toBe(1024);
|
||||
expect(capturedBody.encoding_format).toBe('base64');
|
||||
});
|
||||
|
||||
test('embed (index path) keeps input_type OFF the wire (pre-v0.35.0.0 opt-in shape)', async () => {
|
||||
// dims.ts deliberately emits no input_type for Voyage unless threaded
|
||||
// (`...(inputType ? { input_type: inputType } : {})`); the shim must
|
||||
// not invent a default for it.
|
||||
configureVoyage();
|
||||
let capturedBody: any = null;
|
||||
fetchHandler = async (_url, init) => {
|
||||
capturedBody = JSON.parse(init.body as string);
|
||||
return voyageShapedResponse(1024, 1);
|
||||
};
|
||||
|
||||
await embed(['this is a document being indexed']);
|
||||
expect(capturedBody).not.toBeNull();
|
||||
expect('input_type' in capturedBody).toBe(false);
|
||||
});
|
||||
});
|
||||
@@ -142,6 +142,27 @@ describe('buildFactsAlterRecipe', () => {
|
||||
const recipe = buildFactsAlterRecipe(1536, 1280, 'halfvec');
|
||||
expect(recipe).toMatch(/DROP INDEX[\s\S]*ALTER TABLE[\s\S]*CREATE INDEX/);
|
||||
});
|
||||
|
||||
test('dimension change NULLs embeddings BEFORE the alter (pgvector refuses cross-dim casts)', () => {
|
||||
// Same defect class fixed for content_chunks in embeddingMismatchMessage:
|
||||
// pgvector aborts a cross-dimension ALTER while rows still hold old-width
|
||||
// vectors. The dims-change recipe must wipe first; order pinned.
|
||||
const recipe = buildFactsAlterRecipe(1536, 1280, 'halfvec');
|
||||
const nullIdx = recipe.indexOf('UPDATE facts SET embedding = NULL');
|
||||
const alterIdx = recipe.indexOf('ALTER TABLE facts ALTER COLUMN embedding TYPE');
|
||||
expect(nullIdx).toBeGreaterThan(-1);
|
||||
expect(alterIdx).toBeGreaterThan(-1);
|
||||
expect(nullIdx).toBeLessThan(alterIdx);
|
||||
});
|
||||
|
||||
test('same-dim type swap PRESERVES embeddings (no NULL wipe)', () => {
|
||||
// halfvec(1536) <-> vector(1536): the USING cast is lossless and the
|
||||
// whole point is keeping the data. A wipe here would destroy valid
|
||||
// embeddings for no reason.
|
||||
const recipe = buildFactsAlterRecipe(1536, 1536, 'vector');
|
||||
expect(recipe).not.toContain('UPDATE facts SET embedding = NULL');
|
||||
expect(recipe).toContain('USING embedding::vector(1536)');
|
||||
});
|
||||
});
|
||||
|
||||
describe('FactsEmbeddingDimMismatchError', () => {
|
||||
|
||||
@@ -103,6 +103,26 @@ describe('embeddingMismatchMessage', () => {
|
||||
expect(msg).toContain('docs/embedding-migrations.md');
|
||||
});
|
||||
|
||||
test('Postgres recipe NULLs embeddings BEFORE the column alter (pgvector refuses cross-dim casts)', () => {
|
||||
// pgvector aborts `ALTER COLUMN TYPE vector(N)` with "expected N
|
||||
// dimensions, not M" while rows still hold old-width vectors — which is
|
||||
// every brain running this recipe. The UPDATE must precede the ALTER
|
||||
// (NULLs cast fine). Order pinned so the printed recipe can't drift from
|
||||
// the corrected docs/embedding-migrations.md again.
|
||||
const msg = embeddingMismatchMessage({
|
||||
currentDims: 1536,
|
||||
requestedDims: 768,
|
||||
requestedModel: 'nomic-embed-text',
|
||||
source: 'init',
|
||||
engineKind: 'postgres',
|
||||
});
|
||||
const nullIdx = msg.indexOf('UPDATE content_chunks SET embedding = NULL');
|
||||
const alterIdx = msg.indexOf('ALTER TABLE content_chunks ALTER COLUMN embedding TYPE vector(768)');
|
||||
expect(nullIdx).toBeGreaterThan(-1);
|
||||
expect(alterIdx).toBeGreaterThan(-1);
|
||||
expect(nullIdx).toBeLessThan(alterIdx);
|
||||
});
|
||||
|
||||
test('Postgres branch skips HNSW recreate when requested dims exceed pgvector cap', () => {
|
||||
// Codex finding #8: 2048d (Voyage 4 Large) cannot be HNSW-indexed in pgvector.
|
||||
// The recipe must NOT instruct a CREATE INDEX HNSW for that dim.
|
||||
|
||||
@@ -0,0 +1,78 @@
|
||||
/**
|
||||
* #1928 regression — extract_facts must NOT wipe conversation facts.
|
||||
*
|
||||
* `extract-conversation-facts` writes facts with source='cli:...' and
|
||||
* source_markdown_slug=<transcript slug>. Those pages carry no `## Facts`
|
||||
* fence, so the cycle's wipe-and-reinsert reconcile (deleteFactsForPage +
|
||||
* insertFactsBatch) used to delete them and reinsert nothing — a brain-wide
|
||||
* conversation-facts wipe on a failed-sync full walk (status `ok`, 0
|
||||
* inserted, 1829 rows gone in the original report).
|
||||
*
|
||||
* The fix scopes the cycle delete with excludeSourcePrefixes: ['cli:'].
|
||||
* These tests pin: (a) the exclusion protects cli:-origin rows while still
|
||||
* deleting fence-owned rows on the same page coordinate, and (b) the default
|
||||
* (no opts) behavior is unchanged — every fact on the coordinate is deleted.
|
||||
*/
|
||||
|
||||
import { describe, test, expect, beforeAll, afterAll } from 'bun:test';
|
||||
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
|
||||
|
||||
let engine: PGLiteEngine;
|
||||
|
||||
beforeAll(async () => {
|
||||
engine = new PGLiteEngine();
|
||||
await engine.connect({});
|
||||
await engine.initSchema();
|
||||
});
|
||||
|
||||
afterAll(async () => {
|
||||
await engine.disconnect();
|
||||
});
|
||||
|
||||
async function seedPageWithMixedFacts(slug: string, sourceId: string) {
|
||||
await engine.insertFacts(
|
||||
[
|
||||
// Fence-owned row (what extract_facts recreates from the page fence).
|
||||
{ fact: 'fence fact', kind: 'fact', source: 'fence', row_num: 1, source_markdown_slug: slug },
|
||||
// Empty-source row — also fence-default; must stay deletable.
|
||||
{ fact: 'blank-source fact', kind: 'fact', source: '', row_num: 2, source_markdown_slug: slug },
|
||||
// Conversation fact — NOT fence-owned; must survive the reconcile.
|
||||
{ fact: 'conversation fact', kind: 'fact', source: 'cli:extract-conversation-facts', row_num: 3, source_markdown_slug: slug },
|
||||
],
|
||||
{ source_id: sourceId },
|
||||
);
|
||||
}
|
||||
|
||||
async function factSourcesOnPage(slug: string, sourceId: string): Promise<string[]> {
|
||||
const rows = await engine.executeRaw<{ source: string }>(
|
||||
`SELECT COALESCE(source, '') AS source FROM facts
|
||||
WHERE source_id = $1 AND source_markdown_slug = $2 ORDER BY row_num`,
|
||||
[sourceId, slug],
|
||||
);
|
||||
return rows.map(r => r.source);
|
||||
}
|
||||
|
||||
describe('#1928 deleteFactsForPage excludeSourcePrefixes', () => {
|
||||
test('protects cli:-origin facts, still deletes fence + empty-source rows', async () => {
|
||||
await seedPageWithMixedFacts('transcripts/2026-06-01', 'default');
|
||||
expect((await factSourcesOnPage('transcripts/2026-06-01', 'default')).length).toBe(3);
|
||||
|
||||
const { deleted } = await engine.deleteFactsForPage('transcripts/2026-06-01', 'default', {
|
||||
excludeSourcePrefixes: ['cli:'],
|
||||
});
|
||||
|
||||
expect(deleted).toBe(2); // fence + blank-source removed
|
||||
const survivors = await factSourcesOnPage('transcripts/2026-06-01', 'default');
|
||||
expect(survivors).toEqual(['cli:extract-conversation-facts']);
|
||||
});
|
||||
|
||||
test('default behavior (no opts) deletes every fact on the coordinate', async () => {
|
||||
await seedPageWithMixedFacts('transcripts/2026-06-02', 'default');
|
||||
expect((await factSourcesOnPage('transcripts/2026-06-02', 'default')).length).toBe(3);
|
||||
|
||||
const { deleted } = await engine.deleteFactsForPage('transcripts/2026-06-02', 'default');
|
||||
|
||||
expect(deleted).toBe(3);
|
||||
expect((await factSourcesOnPage('transcripts/2026-06-02', 'default')).length).toBe(0);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,112 @@
|
||||
/**
|
||||
* Regression — importCodeFile stamps source_id on extracted call-graph edges.
|
||||
*
|
||||
* Pre-fix bug: importCodeFile built CodeEdgeInput rows WITHOUT source_id, so
|
||||
* every extracted edge landed NULL in code_edges_symbol. getCallersOf /
|
||||
* getCalleesOf add `AND source_id = <scoped>` whenever a worktree pin or
|
||||
* --source is in play — NULL never matches that filter, so scoped call-graph
|
||||
* queries silently returned 0 rows on multi-source brains even though the
|
||||
* edges existed. Pre-existing coverage (cathedral-ii-brainbench.test.ts,
|
||||
* code-edges.test.ts) only ever queried with { allSources: true }, which
|
||||
* bypasses the filter — exactly why the NULL never surfaced.
|
||||
*
|
||||
* This test imports a caller/callee pair under a non-default source and
|
||||
* asserts (a) the persisted code_edges_symbol rows carry the source_id, and
|
||||
* (b) the SCOPED getCallersOf/getCalleesOf — the user-visible path that
|
||||
* returned 0 — now find the edge, while a different source scope does not.
|
||||
*/
|
||||
|
||||
import { describe, test, expect, beforeAll, afterAll } from 'bun:test';
|
||||
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
|
||||
import { importCodeFile } from '../src/core/import-file.ts';
|
||||
import { runSources } from '../src/commands/sources.ts';
|
||||
|
||||
let engine: PGLiteEngine;
|
||||
|
||||
beforeAll(async () => {
|
||||
engine = new PGLiteEngine();
|
||||
await engine.connect({});
|
||||
await engine.initSchema();
|
||||
await runSources(engine, ['add', 'testsrc', '--no-federated']);
|
||||
|
||||
// Same fixture shape as cathedral-ii-brainbench: runner() calls helper(),
|
||||
// Layer 5 edge extraction captures the unresolved 'calls' edge — but here
|
||||
// the import is pinned to a non-default source.
|
||||
await importCodeFile(
|
||||
engine,
|
||||
'src/a.ts',
|
||||
'export function runner() { return helper(); }\n',
|
||||
{ noEmbed: true, sourceId: 'testsrc' },
|
||||
);
|
||||
await importCodeFile(
|
||||
engine,
|
||||
'src/b.ts',
|
||||
'export function helper() { return 42; }\n',
|
||||
{ noEmbed: true, sourceId: 'testsrc' },
|
||||
);
|
||||
}, 60_000);
|
||||
|
||||
afterAll(async () => {
|
||||
if (engine) await engine.disconnect();
|
||||
}, 30_000);
|
||||
|
||||
describe('importCodeFile — source_id stamped on extracted call-graph edges', () => {
|
||||
test('edges land with the import source_id and scoped caller/callee queries match', async () => {
|
||||
// (a) The persisted unresolved edge rows carry the source, not NULL.
|
||||
const rows = await engine.executeRaw<{ source_id: string | null }>(
|
||||
`SELECT source_id FROM code_edges_symbol WHERE from_symbol_qualified = $1`,
|
||||
['runner'],
|
||||
);
|
||||
expect(rows.length).toBeGreaterThanOrEqual(1);
|
||||
for (const r of rows) {
|
||||
expect(r.source_id).toBe('testsrc');
|
||||
}
|
||||
|
||||
// (b) The scoped queries — the path that silently returned 0 pre-fix.
|
||||
const callers = await engine.getCallersOf('helper', { sourceId: 'testsrc' });
|
||||
const fromRunner = callers.find(r => r.from_symbol_qualified === 'runner');
|
||||
expect(fromRunner).toBeDefined();
|
||||
expect(fromRunner!.edge_type).toBe('calls');
|
||||
expect(fromRunner!.source_id).toBe('testsrc');
|
||||
|
||||
const callees = await engine.getCalleesOf('runner', { sourceId: 'testsrc' });
|
||||
expect(callees.some(r => r.to_symbol_qualified === 'helper')).toBe(true);
|
||||
|
||||
// Source isolation still holds: a different scope must NOT see the edge.
|
||||
const otherScope = await engine.getCallersOf('helper', { sourceId: 'default' });
|
||||
expect(otherScope.find(r => r.from_symbol_qualified === 'runner')).toBeUndefined();
|
||||
});
|
||||
|
||||
test('UNSCOPED import stamps edges with the schema-default source, not NULL', async () => {
|
||||
// The other door of the same bug: an import WITHOUT opts.sourceId (legacy
|
||||
// unscoped callers — `gbrain reindex --code` with no --source) lands its
|
||||
// pages under the schema default (pages.source_id DEFAULT 'default').
|
||||
// If its edges were stamped NULL, the matching scoped query
|
||||
// getCallersOf(sym, { sourceId: 'default' }) — a worktree pinned to
|
||||
// default, --source default, GBRAIN_SOURCE=default — would miss them.
|
||||
await importCodeFile(
|
||||
engine,
|
||||
'src/c.ts',
|
||||
'export function unscopedRunner() { return unscopedHelper(); }\n',
|
||||
{ noEmbed: true },
|
||||
);
|
||||
await importCodeFile(
|
||||
engine,
|
||||
'src/d.ts',
|
||||
'export function unscopedHelper() { return 7; }\n',
|
||||
{ noEmbed: true },
|
||||
);
|
||||
|
||||
const rows = await engine.executeRaw<{ source_id: string | null }>(
|
||||
`SELECT source_id FROM code_edges_symbol WHERE from_symbol_qualified = $1`,
|
||||
['unscopedRunner'],
|
||||
);
|
||||
expect(rows.length).toBeGreaterThanOrEqual(1);
|
||||
for (const r of rows) {
|
||||
expect(r.source_id).toBe('default');
|
||||
}
|
||||
|
||||
const callers = await engine.getCallersOf('unscopedHelper', { sourceId: 'default' });
|
||||
expect(callers.some(r => r.from_symbol_qualified === 'unscopedRunner')).toBe(true);
|
||||
});
|
||||
});
|
||||
@@ -135,4 +135,51 @@ describe('engine.updateSourceConfig', () => {
|
||||
const all = await engine.listAllSources();
|
||||
expect(all.find(s => s.id === 'delta')!.config.last_full_cycle_at).toBe('2026-05-22T11:00:00.000Z');
|
||||
});
|
||||
|
||||
// IS JSON guard: the postgres-engine atomic merge gates its `::jsonb` cast
|
||||
// behind the SQL `IS JSON` predicate so a historical bad row whose config is
|
||||
// a JSONB string of NON-JSON text normalizes to `{}` instead of raising
|
||||
// `invalid input syntax for type json` and aborting the UPDATE.
|
||||
//
|
||||
// We exercise the SQL CASE expression directly via executeRaw against PGLite
|
||||
// (which ships Postgres 17 + IS JSON parity) rather than calling
|
||||
// PostgresEngine.updateSourceConfig (which requires a live Postgres pool).
|
||||
test('IS-JSON guard: non-JSON string config normalizes to {} on merge', async () => {
|
||||
const patch = { merged_key: 'v' };
|
||||
|
||||
const guarded = (configExpr: string) => `
|
||||
SELECT (
|
||||
CASE
|
||||
WHEN jsonb_typeof(${configExpr}) = 'object' THEN ${configExpr}
|
||||
WHEN jsonb_typeof(${configExpr}) = 'string'
|
||||
THEN CASE
|
||||
WHEN (${configExpr} #>> '{}') IS JSON
|
||||
THEN COALESCE(NULLIF((${configExpr} #>> '{}'), '')::jsonb, '{}'::jsonb)
|
||||
ELSE '{}'::jsonb
|
||||
END
|
||||
WHEN jsonb_typeof(${configExpr}) = 'array'
|
||||
THEN COALESCE(
|
||||
(SELECT jsonb_object_agg(kv.key, kv.value)
|
||||
FROM jsonb_array_elements(${configExpr}) elem,
|
||||
jsonb_each(elem) kv),
|
||||
'{}'::jsonb
|
||||
)
|
||||
ELSE '{}'::jsonb
|
||||
END || $1::jsonb
|
||||
) AS result`;
|
||||
|
||||
// 1. JSONB string holding NON-JSON text → normalizes to {} then merges patch.
|
||||
const bad = await engine.executeRaw<{ result: Record<string, unknown> }>(
|
||||
guarded(`to_jsonb('garbage text'::text)`),
|
||||
[JSON.stringify(patch)],
|
||||
);
|
||||
expect(bad[0].result).toEqual({ merged_key: 'v' });
|
||||
|
||||
// 2. JSONB string holding double-encoded valid JSON object → parsed + merged.
|
||||
const good = await engine.executeRaw<{ result: Record<string, unknown> }>(
|
||||
guarded(`to_jsonb('{"x":1}'::text)`),
|
||||
[JSON.stringify(patch)],
|
||||
);
|
||||
expect(good[0].result).toEqual({ x: 1, merged_key: 'v' });
|
||||
});
|
||||
});
|
||||
|
||||
@@ -2181,7 +2181,7 @@ describe('v112 — pages_links_extracted_at', () => {
|
||||
|
||||
|
||||
// v0.43 (#2095): context_volunteer_events — push-based-context feedback log.
|
||||
describe('v116 — context_volunteer_events_table', () => {
|
||||
describe('v117 — context_volunteer_events_table', () => {
|
||||
let engine: PGLiteEngine;
|
||||
beforeAll(async () => {
|
||||
engine = new PGLiteEngine();
|
||||
@@ -2190,15 +2190,15 @@ describe('v116 — context_volunteer_events_table', () => {
|
||||
}, 60_000);
|
||||
afterAll(async () => { if (engine) await engine.disconnect(); }, 60_000);
|
||||
|
||||
test('v116 entry exists, named + idempotent', () => {
|
||||
const m = MIGRATIONS.find(x => x.version === 116);
|
||||
test('v117 entry exists, named + idempotent', () => {
|
||||
const m = MIGRATIONS.find(x => x.version === 117);
|
||||
expect(m).toBeDefined();
|
||||
expect(m!.name).toBe('context_volunteer_events_table');
|
||||
expect(m!.idempotent).toBe(true);
|
||||
});
|
||||
|
||||
test('LATEST_VERSION is at or above 116', () => {
|
||||
expect(LATEST_VERSION).toBeGreaterThanOrEqual(116);
|
||||
test('LATEST_VERSION is at or above 117', () => {
|
||||
expect(LATEST_VERSION).toBeGreaterThanOrEqual(117);
|
||||
});
|
||||
|
||||
test('table exists after initSchema with the documented columns', async () => {
|
||||
|
||||
@@ -0,0 +1,89 @@
|
||||
/**
|
||||
* Authorize-grant scope default (RFC 6749 §3.3).
|
||||
*
|
||||
* When a client omits `scope` on /authorize, the granted scope must default to
|
||||
* the client's full registered scope — NOT the empty set. Regression guard for
|
||||
* the bug where an omitted request granted [], which then propagated into the
|
||||
* access + refresh tokens and never self-healed: every op failed
|
||||
* `insufficient_scope` even though the client was registered `read write`
|
||||
* (some MCP connectors omit `scope` on /authorize). The clamp must still hold —
|
||||
* an explicit over-broad request cannot escalate past the client's allowed set.
|
||||
*/
|
||||
import { describe, test, expect, beforeAll, afterAll, beforeEach } from 'bun:test';
|
||||
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
|
||||
import { GBrainOAuthProvider } from '../src/core/oauth-provider.ts';
|
||||
import { sqlQueryForEngine } from '../src/core/sql-query.ts';
|
||||
|
||||
let engine: PGLiteEngine;
|
||||
let provider: GBrainOAuthProvider;
|
||||
let sql: ReturnType<typeof sqlQueryForEngine>;
|
||||
|
||||
beforeAll(async () => {
|
||||
engine = new PGLiteEngine();
|
||||
await engine.connect({});
|
||||
await engine.initSchema();
|
||||
sql = sqlQueryForEngine(engine);
|
||||
provider = new GBrainOAuthProvider({ sql });
|
||||
});
|
||||
|
||||
afterAll(async () => {
|
||||
await engine.disconnect();
|
||||
});
|
||||
|
||||
beforeEach(async () => {
|
||||
await (engine as any).db.exec('DELETE FROM oauth_tokens');
|
||||
await (engine as any).db.exec('DELETE FROM oauth_codes');
|
||||
await (engine as any).db.exec('DELETE FROM oauth_clients');
|
||||
});
|
||||
|
||||
// authorize() writes the granted scope into oauth_codes then redirects; we
|
||||
// assert on the stored grant directly, so the redirect is a no-op.
|
||||
const noopRes = { redirect() {} } as any;
|
||||
|
||||
async function authorizeAndReadScopes(
|
||||
scope: string,
|
||||
requested: string[] | undefined,
|
||||
): Promise<string[]> {
|
||||
const reg = await provider.registerClientManual(
|
||||
'authz-test', ['authorization_code'], scope, ['https://example.test/cb'],
|
||||
);
|
||||
const client = await provider.clientsStore.getClient(reg.clientId);
|
||||
expect(client).toBeTruthy();
|
||||
await provider.authorize(
|
||||
client!,
|
||||
{
|
||||
scopes: requested,
|
||||
codeChallenge: 'test-challenge',
|
||||
redirectUri: 'https://example.test/cb',
|
||||
state: 'xyz',
|
||||
} as any,
|
||||
noopRes,
|
||||
);
|
||||
const rows = (await sql`
|
||||
SELECT scopes FROM oauth_codes WHERE client_id = ${reg.clientId}
|
||||
`) as Array<{ scopes: string[] }>;
|
||||
expect(rows.length).toBe(1);
|
||||
return rows[0].scopes ?? [];
|
||||
}
|
||||
|
||||
describe('authorize() scope default — omitted scope inherits client grant', () => {
|
||||
test('omitted scope → inherits full registered scope', async () => {
|
||||
expect((await authorizeAndReadScopes('read write', undefined)).sort()).toEqual(['read', 'write']);
|
||||
});
|
||||
|
||||
test('empty scope array → inherits full registered scope', async () => {
|
||||
expect((await authorizeAndReadScopes('read write', [])).sort()).toEqual(['read', 'write']);
|
||||
});
|
||||
|
||||
test('explicit subset is honored (not overridden to full)', async () => {
|
||||
expect(await authorizeAndReadScopes('read write admin', ['read'])).toEqual(['read']);
|
||||
});
|
||||
|
||||
test('clamp preserved: over-broad request cannot escalate', async () => {
|
||||
expect(await authorizeAndReadScopes('read', ['read', 'admin'])).toEqual(['read']);
|
||||
});
|
||||
|
||||
test('clamp preserved: requesting only a disallowed scope grants nothing (no inheritance)', async () => {
|
||||
expect(await authorizeAndReadScopes('read write', ['admin'])).toEqual([]);
|
||||
});
|
||||
});
|
||||
@@ -12,6 +12,7 @@ import {
|
||||
import { hashToken, generateToken } from '../src/core/utils.ts';
|
||||
import { PGLITE_SCHEMA_SQL } from '../src/core/pglite-schema.ts';
|
||||
import { InvalidTokenError } from '@modelcontextprotocol/sdk/server/auth/errors.js';
|
||||
import type { AuthInfo as CoreAuthInfo } from '../src/core/operations.ts';
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Test setup: in-memory PGLite with OAuth tables
|
||||
@@ -310,6 +311,33 @@ describe('verifyAccessToken', () => {
|
||||
expect(authInfo.clientId).toBe('legacy-agent');
|
||||
expect(authInfo.scopes).toEqual(['read', 'write', 'admin']); // grandfathered full access
|
||||
});
|
||||
|
||||
test('legacy access_tokens fallback honors permissions.source_id array grants', async () => {
|
||||
// oauth.test.ts initializes the static PGLite schema blob, not the full
|
||||
// migration stack. Add the v38 permissions column here so the row matches
|
||||
// a modern brain carrying a legacy-token source grant.
|
||||
await sql`
|
||||
ALTER TABLE access_tokens
|
||||
ADD COLUMN IF NOT EXISTS permissions JSONB NOT NULL DEFAULT '{"takes_holders":["world"]}'::jsonb
|
||||
`;
|
||||
|
||||
const legacyToken = generateToken('gbrain_');
|
||||
const hash = hashToken(legacyToken);
|
||||
await sql`
|
||||
INSERT INTO access_tokens (id, name, token_hash, permissions)
|
||||
VALUES (
|
||||
${crypto.randomUUID()},
|
||||
${'legacy-federated-agent'},
|
||||
${hash},
|
||||
${JSON.stringify({ source_id: ['default', 'src-a', 'src-b'] })}::jsonb
|
||||
)
|
||||
`;
|
||||
|
||||
const authInfo = await provider.verifyAccessToken(legacyToken) as CoreAuthInfo;
|
||||
expect(authInfo.clientId).toBe('legacy-federated-agent');
|
||||
expect(authInfo.sourceId).toBe('default');
|
||||
expect(authInfo.allowedSources).toEqual(['default', 'src-a', 'src-b']);
|
||||
});
|
||||
});
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
@@ -3,6 +3,7 @@ import { mkdirSync, rmSync, existsSync, readFileSync, writeFileSync } from 'fs';
|
||||
import { join } from 'path';
|
||||
import { tmpdir } from 'os';
|
||||
import { acquireLock, releaseLock, type LockHandle } from '../src/core/pglite-lock';
|
||||
import { withEnv } from './helpers/with-env.ts';
|
||||
|
||||
const TEST_DIR = join(tmpdir(), 'gbrain-lock-test-' + process.pid);
|
||||
|
||||
@@ -99,3 +100,96 @@ describe('pglite-lock', () => {
|
||||
await releaseLock(lock2);
|
||||
});
|
||||
});
|
||||
|
||||
describe('pglite-lock #2058 heartbeat + steal-grace', () => {
|
||||
beforeEach(() => {
|
||||
if (existsSync(TEST_DIR)) rmSync(TEST_DIR, { recursive: true, force: true });
|
||||
mkdirSync(TEST_DIR, { recursive: true });
|
||||
});
|
||||
afterEach(() => {
|
||||
if (existsSync(TEST_DIR)) rmSync(TEST_DIR, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
function writeHolder(fields: { pid: number; acquiredAgoMs: number; refreshedAgoMs: number }) {
|
||||
const lockDir = join(TEST_DIR, '.gbrain-lock');
|
||||
mkdirSync(lockDir, { recursive: true });
|
||||
const now = Date.now();
|
||||
writeFileSync(join(lockDir, 'lock'), JSON.stringify({
|
||||
pid: fields.pid,
|
||||
acquired_at: now - fields.acquiredAgoMs,
|
||||
refreshed_at: now - fields.refreshedAgoMs,
|
||||
command: 'test holder',
|
||||
}));
|
||||
}
|
||||
|
||||
test('[REGRESSION] a LIVE holder with a fresh heartbeat is NOT stolen even when the lock is old', async () => {
|
||||
// The WAL-corruption bug: a >5min embed used to get its lock force-removed.
|
||||
// Now an alive holder that heartbeated recently is left alone regardless of
|
||||
// age. acquired 20min ago, but refreshed just now → must wait, not steal.
|
||||
writeHolder({ pid: process.pid, acquiredAgoMs: 20 * 60_000, refreshedAgoMs: 0 });
|
||||
|
||||
await expect(acquireLock(TEST_DIR, { timeoutMs: 1200 })).rejects.toThrow(/Timed out/);
|
||||
// Holder's lock still present (was never stolen).
|
||||
expect(existsSync(join(TEST_DIR, '.gbrain-lock'))).toBe(true);
|
||||
});
|
||||
|
||||
test('a LIVE PID whose heartbeat went stale past the grace window IS reaped', async () => {
|
||||
// PID is alive (our own) but hasn't refreshed in 20min (> 600s grace):
|
||||
// hung holder, or a reused PID whose real holder is gone. Reap + acquire.
|
||||
writeHolder({ pid: process.pid, acquiredAgoMs: 25 * 60_000, refreshedAgoMs: 20 * 60_000 });
|
||||
|
||||
const lock = await acquireLock(TEST_DIR, { timeoutMs: 2000 });
|
||||
expect(lock.acquired).toBe(true);
|
||||
await releaseLock(lock);
|
||||
});
|
||||
|
||||
test('GBRAIN_PGLITE_LOCK_STEAL_GRACE_SECONDS tunes the grace window', async () => {
|
||||
// withEnv keeps the process-global mutation isolated across shard files.
|
||||
await withEnv({ GBRAIN_PGLITE_LOCK_STEAL_GRACE_SECONDS: '5' }, async () => {
|
||||
// Refreshed 30s ago — fresh under the 600s default, STALE under 5s.
|
||||
writeHolder({ pid: process.pid, acquiredAgoMs: 60_000, refreshedAgoMs: 30_000 });
|
||||
const lock = await acquireLock(TEST_DIR, { timeoutMs: 2000 });
|
||||
expect(lock.acquired).toBe(true);
|
||||
await releaseLock(lock);
|
||||
});
|
||||
});
|
||||
|
||||
test('[REGRESSION] releaseLock does NOT remove a lock that was stolen + re-acquired by another process', async () => {
|
||||
// We acquire, then simulate a steal: another process reaped us past grace
|
||||
// and now owns the lock (different pid + acquired_at). Our releaseLock must
|
||||
// NOT delete their live lock — doing so would let a third process in
|
||||
// alongside the new owner (the #2058 corruption class).
|
||||
const lock: LockHandle = await acquireLock(TEST_DIR);
|
||||
expect(lock.acquired).toBe(true);
|
||||
expect(lock.ownerToken).toBeDefined();
|
||||
if (lock.heartbeat) clearInterval(lock.heartbeat); // stop our heartbeat for a deterministic test
|
||||
|
||||
// Overwrite the lock file as if process B re-acquired it.
|
||||
const lockFile = join(TEST_DIR, '.gbrain-lock', 'lock');
|
||||
const bNow = Date.now() + 1;
|
||||
writeFileSync(lockFile, JSON.stringify({ pid: 999999, acquired_at: bNow, refreshed_at: bNow, command: 'process B' }));
|
||||
|
||||
await releaseLock(lock); // our (stale) handle
|
||||
|
||||
// B's lock survives — we did not clobber it.
|
||||
expect(existsSync(join(TEST_DIR, '.gbrain-lock'))).toBe(true);
|
||||
const after = JSON.parse(readFileSync(lockFile, 'utf-8'));
|
||||
expect(after.pid).toBe(999999);
|
||||
|
||||
// Cleanup for afterEach.
|
||||
rmSync(join(TEST_DIR, '.gbrain-lock'), { recursive: true, force: true });
|
||||
});
|
||||
|
||||
test('acquire starts a heartbeat and seeds refreshed_at; release clears it', async () => {
|
||||
const lock: LockHandle = await acquireLock(TEST_DIR);
|
||||
expect(lock.acquired).toBe(true);
|
||||
expect(lock.heartbeat).toBeDefined();
|
||||
const data = JSON.parse(readFileSync(join(TEST_DIR, '.gbrain-lock', 'lock'), 'utf-8'));
|
||||
expect(data.refreshed_at).toBeDefined();
|
||||
expect(typeof data.refreshed_at).toBe('number');
|
||||
|
||||
await releaseLock(lock);
|
||||
expect(lock.heartbeat).toBeUndefined();
|
||||
expect(existsSync(join(TEST_DIR, '.gbrain-lock'))).toBe(false);
|
||||
});
|
||||
});
|
||||
|
||||
@@ -0,0 +1,45 @@
|
||||
/**
|
||||
* #2034 — engine-parity reconnect().
|
||||
*
|
||||
* The autopilot health-probe recovery path used `disconnect()` + bare
|
||||
* `connect()`, which (a) threw `database_url undefined` forever on Postgres
|
||||
* because the config was lost, and (b) was a silent no-op on PGLite which had
|
||||
* no reconnect() at all. Both engines now expose `reconnect()`; this pins the
|
||||
* PGLite side: it restores connectivity against the config captured at the
|
||||
* last connect(), and is a safe no-op when never connected.
|
||||
*/
|
||||
|
||||
import { describe, test, expect } from 'bun:test';
|
||||
import { mkdtempSync, rmSync } from 'fs';
|
||||
import { join } from 'path';
|
||||
import { tmpdir } from 'os';
|
||||
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
|
||||
|
||||
describe('#2034 PGLiteEngine.reconnect', () => {
|
||||
test('restores connectivity and persisted state against the saved config', async () => {
|
||||
const dir = mkdtempSync(join(tmpdir(), 'gbrain-reconnect-'));
|
||||
const engine = new PGLiteEngine();
|
||||
try {
|
||||
await engine.connect({ database_path: dir });
|
||||
await engine.initSchema();
|
||||
await engine.setConfig('reconnect.probe', 'pre');
|
||||
|
||||
// Reconnect with no args — the #2034 contract: it must reuse the config
|
||||
// captured at connect(), not require it to be re-passed.
|
||||
await engine.reconnect();
|
||||
|
||||
// Still queryable, and persisted state survived (same data dir).
|
||||
const rows = await engine.executeRaw<{ x: number }>('SELECT 1 AS x');
|
||||
expect(rows[0]?.x).toBe(1);
|
||||
expect(await engine.getConfig('reconnect.probe')).toBe('pre');
|
||||
} finally {
|
||||
await engine.disconnect();
|
||||
rmSync(dir, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
test('reconnect() before any connect() is a safe no-op', async () => {
|
||||
const engine = new PGLiteEngine();
|
||||
await expect(engine.reconnect()).resolves.toBeUndefined();
|
||||
});
|
||||
});
|
||||
@@ -89,7 +89,7 @@ describe('alias_resolved boost stage', () => {
|
||||
});
|
||||
|
||||
describe('KNOBS_HASH_VERSION', () => {
|
||||
it('is 10 (9→10 relational recall arm invalidates rel-off cache rows, v0.43)', () => {
|
||||
expect(KNOBS_HASH_VERSION).toBe(10);
|
||||
it('is 11 (10→11 asymmetric input_type fix invalidates document-side query-vector rows, #1400)', () => {
|
||||
expect(KNOBS_HASH_VERSION).toBe(11);
|
||||
});
|
||||
});
|
||||
|
||||
@@ -403,7 +403,11 @@ describe('knobsHash determinism + cross-mode separation (CDX-4)', () => {
|
||||
// must not be served from a cache row written before the policy change.
|
||||
// v0.43: bumped 9→10 for the relational recall arm (rel=/reld=) — a
|
||||
// relational-on write must not be served to a relational-off lookup.
|
||||
expect(KNOBS_HASH_VERSION).toBe(10);
|
||||
// #1400: bumped 10→11 for the asymmetric input_type fix — embedQuery()
|
||||
// now produces query-side vectors for asymmetric providers (zembed-1,
|
||||
// Voyage v3+), so rows keyed on pre-fix document-side query vectors
|
||||
// must not be served to post-fix lookups.
|
||||
expect(KNOBS_HASH_VERSION).toBe(11);
|
||||
});
|
||||
|
||||
test('T1 (codex): floor_ratio set vs unset produces DIFFERENT hashes (cache contamination prevention)', () => {
|
||||
@@ -568,8 +572,8 @@ describe('v0.40.4 — graph_signals knob', () => {
|
||||
});
|
||||
|
||||
describe('v0.42.3.0 — autocut knobs', () => {
|
||||
test('KNOBS_HASH_VERSION is 10 (9→10 relational recall arm, v0.43)', () => {
|
||||
expect(KNOBS_HASH_VERSION).toBe(10);
|
||||
test('KNOBS_HASH_VERSION is 11 (10→11 asymmetric input_type fix, #1400)', () => {
|
||||
expect(KNOBS_HASH_VERSION).toBe(11);
|
||||
});
|
||||
|
||||
test('bundle defaults: conservative off, balanced/tokenmax on @0.20', () => {
|
||||
|
||||
@@ -43,7 +43,7 @@ function baseKnobs(): ResolvedSearchKnobs {
|
||||
}
|
||||
|
||||
describe('KNOBS_HASH_VERSION + version invariants', () => {
|
||||
test('version is 10 (…; 7→8 autocut; 8→9 archive-demote #1777; 9→10 relational recall)', () => {
|
||||
test('version is 11 (…; 8→9 archive-demote #1777; 9→10 relational recall; 10→11 asymmetric input_type #1400)', () => {
|
||||
// v0.35.0.0: 1→2 to fold reranker fields. v0.35.6.0: 2→3 to fold
|
||||
// floor_ratio. v0.36 wave: piggybacks on v=3 with 7 cross-modal knobs
|
||||
// (D2) PLUS column + provider context (D8/CDX-2 cross-column isolation).
|
||||
@@ -58,7 +58,10 @@ describe('KNOBS_HASH_VERSION + version invariants', () => {
|
||||
// autocut. issue #1777: 8→9 archive/ demote (search-exclude policy change
|
||||
// isn't in the hash, so the bump invalidates archive-excluded cache rows).
|
||||
// v0.43: 9→10 relational recall arm (rel=/reld=).
|
||||
expect(KNOBS_HASH_VERSION).toBe(10);
|
||||
// #1400: 10→11 asymmetric input_type fix — embedQuery() now produces
|
||||
// query-side vectors for asymmetric providers, so rows keyed on
|
||||
// pre-fix document-side query vectors must not be served.
|
||||
expect(KNOBS_HASH_VERSION).toBe(11);
|
||||
});
|
||||
|
||||
test('hash is 16 hex chars regardless of reranker config', () => {
|
||||
|
||||
@@ -0,0 +1,76 @@
|
||||
/**
|
||||
* #2057 regression — addTimelineEntriesBatch must accept JS Date `date` values.
|
||||
*
|
||||
* The original bug: callers source `date` from SQL rows (e.g.
|
||||
* `meeting.effective_date`), which arrive as JS Date objects. The OLD insert
|
||||
* bound them into `${dates}::text[]`, which threw `cannot cast type timestamp
|
||||
* with time zone to text[]`, and a bare `catch {}` in the meetings extractor
|
||||
* swallowed it — timeline stayed empty forever.
|
||||
*
|
||||
* The #1861 refactor moved the insert to `jsonb_to_recordset` + `v.date::date`,
|
||||
* where a Date serializes to an ISO string that casts cleanly. This test pins
|
||||
* that a Date-bearing batch round-trips, so a future refactor can't silently
|
||||
* reintroduce the cast failure. (The companion fix un-silences the extractor's
|
||||
* catch so any future failure is visible rather than a phantom "0 entries".)
|
||||
*/
|
||||
|
||||
import { describe, test, expect, beforeAll, afterAll } from 'bun:test';
|
||||
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
|
||||
import { importFromContent } from '../src/core/import-file.ts';
|
||||
import type { TimelineBatchInput } from '../src/core/engine.ts';
|
||||
|
||||
let engine: PGLiteEngine;
|
||||
|
||||
beforeAll(async () => {
|
||||
engine = new PGLiteEngine();
|
||||
await engine.connect({});
|
||||
await engine.initSchema();
|
||||
await importFromContent(engine, 'people/alice-example', `---\ntitle: Alice\ntype: note\n---\n\n# Alice\n`, {
|
||||
noEmbed: true,
|
||||
sourceId: 'default',
|
||||
sourcePath: 'people/alice-example.md',
|
||||
});
|
||||
});
|
||||
|
||||
afterAll(async () => {
|
||||
await engine.disconnect();
|
||||
});
|
||||
|
||||
describe('#2057 addTimelineEntriesBatch with Date date values', () => {
|
||||
test('a Date `date` is inserted and round-trips as the right calendar day', async () => {
|
||||
// Mimic the real caller: `date` typed string on the interface, but a JS
|
||||
// Date at runtime (straight off a TIMESTAMPTZ column). The cast must hold.
|
||||
const effectiveDate = new Date('2026-04-03T00:00:00.000Z');
|
||||
const batch: TimelineBatchInput[] = [
|
||||
{
|
||||
slug: 'people/alice-example',
|
||||
date: effectiveDate as unknown as string,
|
||||
source: 'cli:test',
|
||||
summary: 'met alice',
|
||||
source_id: 'default',
|
||||
},
|
||||
];
|
||||
|
||||
const inserted = await engine.addTimelineEntriesBatch(batch);
|
||||
expect(inserted).toBe(1);
|
||||
|
||||
const rows = await engine.executeRaw<{ date: string }>(
|
||||
`SELECT date::text AS date FROM timeline_entries WHERE summary = 'met alice'`,
|
||||
);
|
||||
expect(rows.length).toBe(1);
|
||||
expect(rows[0].date).toBe('2026-04-03');
|
||||
});
|
||||
|
||||
test('a plain ISO string `date` still works (no regression)', async () => {
|
||||
const inserted = await engine.addTimelineEntriesBatch([
|
||||
{
|
||||
slug: 'people/alice-example',
|
||||
date: '2026-05-01',
|
||||
source: 'cli:test',
|
||||
summary: 'string-dated entry',
|
||||
source_id: 'default',
|
||||
},
|
||||
]);
|
||||
expect(inserted).toBe(1);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,119 @@
|
||||
/**
|
||||
* #2038 — idx_timeline_dedup schema-drift self-heal.
|
||||
*
|
||||
* A brain that ran the pre-renumber v99 variant of the dedup migration is
|
||||
* stamped past v102 with the OLD 3-column index. `runMigrations` early-returns
|
||||
* (nothing pending) so a migration verify-hook can't fix it. The repair is
|
||||
* keyed off the index SHAPE and runs regardless. These tests simulate the
|
||||
* drifted states directly and pin: detection, rebuild, dedupe-before-rebuild
|
||||
* (only possible when the index was absent), and idempotency.
|
||||
*/
|
||||
|
||||
import { describe, test, expect, beforeAll, afterAll } from 'bun:test';
|
||||
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
|
||||
import {
|
||||
checkTimelineDedupIndex,
|
||||
repairTimelineDedupIndex,
|
||||
} from '../src/core/timeline-dedup-repair.ts';
|
||||
import { importFromContent } from '../src/core/import-file.ts';
|
||||
|
||||
let engine: PGLiteEngine;
|
||||
let pageId: string;
|
||||
|
||||
beforeAll(async () => {
|
||||
engine = new PGLiteEngine();
|
||||
await engine.connect({});
|
||||
await engine.initSchema();
|
||||
await importFromContent(engine, 'people/alice-example', `---\ntitle: Alice\ntype: note\n---\n\n# Alice\n`, {
|
||||
noEmbed: true,
|
||||
sourceId: 'default',
|
||||
sourcePath: 'people/alice-example.md',
|
||||
});
|
||||
const pid = await engine.executeRaw<{ id: string }>(
|
||||
`SELECT id::text AS id FROM pages WHERE slug = 'people/alice-example' AND source_id = 'default'`,
|
||||
);
|
||||
pageId = pid[0].id;
|
||||
});
|
||||
|
||||
afterAll(async () => {
|
||||
await engine.disconnect();
|
||||
});
|
||||
|
||||
/** Force the index back to the broken pre-v102 3-column shape. */
|
||||
async function regressTo3Col() {
|
||||
await engine.executeRaw(`DELETE FROM timeline_entries`);
|
||||
await engine.executeRaw(`DROP INDEX IF EXISTS idx_timeline_dedup`);
|
||||
await engine.executeRaw(
|
||||
`CREATE UNIQUE INDEX idx_timeline_dedup ON timeline_entries(page_id, date, summary)`,
|
||||
);
|
||||
}
|
||||
|
||||
/** The other drift shape: the index was dropped entirely, letting true
|
||||
* 4-tuple duplicates accumulate that would block a naive CREATE UNIQUE INDEX. */
|
||||
async function regressToAbsentWithDupes() {
|
||||
await engine.executeRaw(`DELETE FROM timeline_entries`);
|
||||
await engine.executeRaw(`DROP INDEX IF EXISTS idx_timeline_dedup`);
|
||||
await engine.executeRaw(
|
||||
`INSERT INTO timeline_entries (page_id, date, summary, source, detail)
|
||||
VALUES ($1, '2026-04-03', 'met alice', 'meeting', ''),
|
||||
($1, '2026-04-03', 'met alice', 'meeting', ''),
|
||||
($1, '2026-04-03', 'met alice', 'cli:extract', '')`,
|
||||
[pageId],
|
||||
);
|
||||
}
|
||||
|
||||
describe('#2038 idx_timeline_dedup drift repair', () => {
|
||||
test('detects the 3-column drift', async () => {
|
||||
await regressTo3Col();
|
||||
const status = await checkTimelineDedupIndex(engine);
|
||||
expect(status.tablePresent).toBe(true);
|
||||
expect(status.indexPresent).toBe(true);
|
||||
expect(status.columns).toEqual(['page_id', 'date', 'summary']);
|
||||
expect(status.needsRepair).toBe(true);
|
||||
});
|
||||
|
||||
test('rebuilds the 3-column index to 4 columns (no dupes to collapse)', async () => {
|
||||
await regressTo3Col();
|
||||
await engine.executeRaw(
|
||||
`INSERT INTO timeline_entries (page_id, date, summary, source, detail)
|
||||
VALUES ($1, '2026-04-03', 'met alice', 'meeting', '')`,
|
||||
[pageId],
|
||||
);
|
||||
|
||||
const res = await repairTimelineDedupIndex(engine);
|
||||
expect(res.repaired).toBe(true);
|
||||
expect(res.reason).toBe('rebuilt');
|
||||
expect(res.collapsedDuplicates).toBe(0);
|
||||
|
||||
const after = await checkTimelineDedupIndex(engine);
|
||||
expect(after.columns).toEqual(['page_id', 'date', 'summary', 'source']);
|
||||
expect(after.needsRepair).toBe(false);
|
||||
});
|
||||
|
||||
test('dedupes true 4-tuple duplicates before building the unique index', async () => {
|
||||
await regressToAbsentWithDupes(); // index absent + a real (meeting) dup
|
||||
|
||||
const before = await checkTimelineDedupIndex(engine);
|
||||
expect(before.indexPresent).toBe(false);
|
||||
expect(before.needsRepair).toBe(true);
|
||||
|
||||
const res = await repairTimelineDedupIndex(engine);
|
||||
expect(res.repaired).toBe(true);
|
||||
expect(res.collapsedDuplicates).toBe(1); // one of the two 'meeting' rows removed
|
||||
|
||||
const after = await checkTimelineDedupIndex(engine);
|
||||
expect(after.columns).toEqual(['page_id', 'date', 'summary', 'source']);
|
||||
const rows = await engine.executeRaw<{ n: string }>(
|
||||
`SELECT COUNT(*)::text AS n FROM timeline_entries`,
|
||||
);
|
||||
expect(parseInt(rows[0].n, 10)).toBe(2); // meeting (deduped) + cli:extract
|
||||
});
|
||||
|
||||
test('idempotent — a second repair is a no-op', async () => {
|
||||
await regressTo3Col();
|
||||
await repairTimelineDedupIndex(engine);
|
||||
const second = await repairTimelineDedupIndex(engine);
|
||||
expect(second.repaired).toBe(false);
|
||||
expect(second.reason).toBe('already_correct');
|
||||
});
|
||||
});
|
||||
@@ -120,6 +120,58 @@ describe('writePageThrough', () => {
|
||||
expect(res).toEqual({ written: false, skipped: 'page_not_found_after_write' });
|
||||
});
|
||||
|
||||
test('[REGRESSION #2018] default page (null local_path) in a multi-source brain → skipped, no leak into a sibling source repo', async () => {
|
||||
// A sibling federated source with its OWN working tree.
|
||||
const siblingDir = path.join(tmpRoot, 'housefax');
|
||||
fs.mkdirSync(siblingDir, { recursive: true });
|
||||
await engine.executeRaw(
|
||||
`INSERT INTO sources (id, name, local_path, config) VALUES ('housefax', 'Housefax', $1, '{}'::jsonb)`,
|
||||
[siblingDir],
|
||||
);
|
||||
// The leak trigger: global sync.repo_path points at the sibling's tree
|
||||
// while the default source (re-seeded by resetPgliteState) has no
|
||||
// local_path of its own.
|
||||
await engine.setConfig('sync.repo_path', siblingDir);
|
||||
|
||||
const slug = 'internal/cross-cutting-note';
|
||||
await seedPage(slug); // sourceId 'default'
|
||||
|
||||
const res = await writePageThrough(engine, slug, { sourceId: 'default' });
|
||||
|
||||
expect(res).toEqual({ written: false, skipped: 'source_repo_belongs_to_other_source' });
|
||||
// The sibling source's repo stays clean — the whole point of #2018.
|
||||
expect(walkFiles(siblingDir).some((f) => f.endsWith('.md'))).toBe(false);
|
||||
});
|
||||
|
||||
test('[#2018] page assigned to a source with its own local_path writes to that tree root, not the global path', async () => {
|
||||
const alphaDir = path.join(tmpRoot, 'alpha-repo');
|
||||
fs.mkdirSync(alphaDir, { recursive: true });
|
||||
const globalDir = path.join(tmpRoot, 'global-repo');
|
||||
fs.mkdirSync(globalDir, { recursive: true });
|
||||
await engine.executeRaw(
|
||||
`INSERT INTO sources (id, name, local_path, config) VALUES ('alpha', 'Alpha', $1, '{}'::jsonb)`,
|
||||
[alphaDir],
|
||||
);
|
||||
// Must NOT be used — the assigned source has its own tree.
|
||||
await engine.setConfig('sync.repo_path', globalDir);
|
||||
|
||||
const slug = 'notes/alpha-thing';
|
||||
await importFromContent(engine, slug, `---\ntitle: T\ntype: note\n---\n\n# Body\n`, {
|
||||
noEmbed: true,
|
||||
sourceId: 'alpha',
|
||||
sourcePath: `${slug}.md`,
|
||||
});
|
||||
|
||||
const res = await writePageThrough(engine, slug, { sourceId: 'alpha' });
|
||||
|
||||
expect(res.written).toBe(true);
|
||||
// File at the source's tree ROOT, never nested under `.sources/<id>/`.
|
||||
expect(res.path).toBe(path.join(alphaDir, `${slug}.md`));
|
||||
expect(fs.existsSync(path.join(alphaDir, `${slug}.md`))).toBe(true);
|
||||
// The global repo path is untouched.
|
||||
expect(walkFiles(globalDir).some((f) => f.endsWith('.md'))).toBe(false);
|
||||
});
|
||||
|
||||
test('[REGRESSION] mkdir ENOTDIR (parent is a file) → error, no partial .md, no .tmp', async () => {
|
||||
await engine.setConfig('sync.repo_path', brainDir);
|
||||
// Block the `wiki/` directory by putting a FILE named "wiki" under the repo,
|
||||
|
||||
Reference in New Issue
Block a user