mirror of
https://github.com/garrytan/gbrain.git
synced 2026-07-27 22:15:33 +00:00
Merge remote-tracking branch 'origin/master' into tmp-2798-rebase
# Conflicts: # TODOS.md
This commit is contained in:
Sinabina
@@ -2,6 +2,42 @@
|
||||
|
||||
All notable changes to GBrain will be documented in this file.
|
||||
|
||||
## [0.42.60.0] - 2026-07-16
|
||||
|
||||
**Eleven verified community fixes: Windows brains no longer risk losing subdirectory pages on a full sync, agent tool loops on non-Anthropic providers survive interruption instead of dead-lettering, multi-source brains get two source-isolation gaps closed, and the search cache stops leaking results across exclude policies. Every fix was reproduced and reviewed against master before landing.**
|
||||
|
||||
### Fixed
|
||||
- **Windows: `sync --full` no longer deletes subdirectory pages.** A path-separator mismatch made every subdirectory page look stale during full-sync reconcile, so a routine full sync could delete them. Paths are now normalized before comparison, and a mass-delete safety valve blocks any reconcile that would remove most of a source's pages. (#2828, #2836, contributed by @1alessio)
|
||||
- **Gateway tool loops on non-Anthropic providers are reliable across resume.** Tool-result turns are persisted as they happen, interrupted jobs reconcile dangling tool calls on resume instead of dead-lettering with unbalanced-transcript errors, `Date` values in tool outputs no longer crash serialization, DeepSeek reasoning-only replies are read correctly instead of as empty, and `openrouter_api_key` in config reaches the gateway. (#2820, consolidating community fixes #2062, #2065, #2257, #2274, #2336, #2487, #2491, #2572, #2614, #2617, #2806; contributed by @time-attack and the original PR authors)
|
||||
- **Claude 5 models get output-token headroom.** Thinking-default models no longer have long answers silently truncated by the old 4096-token default output cap; Claude 5 chat calls now default to 32000 output tokens (16000 for `think`). Other providers keep their existing caps, so smaller-limit providers are unaffected. (#2820)
|
||||
- **Bulk import survives huge fence-less files.** The markdown lexer is skipped when a page contains no code fences, removing an out-of-memory crash on large tables and notes during bulk import. (#2437, #2440, contributed by @irresi)
|
||||
- **`file_list` no longer crashes on Postgres brains over MCP.** BIGINT file sizes are normalized before JSON serialization; the CLI files listing gets the same fix. (#472, contributed by @vinsew)
|
||||
- **`gbrain config set auto_chronicle true` works as documented.** The Life Chronicle config keys (and `takes.bootstrap_enabled`) are registered, so the documented enable commands stop being rejected as unknown keys. (#2632, contributed by @p3ob7o)
|
||||
- **Orphan reports skip generated corpus roots.** `raw/`, `atoms/`, and `skills/` no longer inflate the orphan ratio by default; `--include-pseudo` still shows everything. (#2068, contributed by @mgunnin)
|
||||
|
||||
### Security
|
||||
- **The search cache honors your hard-exclude policy.** Cached search results are now keyed on the effective hard-exclude/include slug-prefix policy, so a process with `GBRAIN_SEARCH_EXCLUDE` set can never be served cached rows written under a different policy — and vice versa. (#2825, #2885)
|
||||
- **Take-writes are source-scoped.** When a source resolves (via `--source`, `GBRAIN_SOURCE`, or the dotfile chain), CLI take commands look pages up within that source instead of first-match-by-slug, closing a cross-source write path on brains where the same slug exists in multiple sources. Brains without a resolvable source keep the previous lookup. (#2684, #2698, contributed by @RerankerGuo)
|
||||
- **Image pages land in the right source.** Imported images are stamped with the syncing source (and their auto-links stay within it) instead of always landing in `default`. (#2706, #2718, contributed by @RerankerGuo)
|
||||
- **The admin bootstrap token no longer prints to a non-terminal stream.** The one-time token is withheld when its output stream is a pipe, log, or CI capture instead of an interactive terminal. (#2625, contributed by @irresi)
|
||||
|
||||
### Internal
|
||||
- Pinned embedding dimensions in a doctor test to eliminate a shard-order flake in CI. (#2801, contributed by @p3ob7o)
|
||||
|
||||
### To take advantage of v0.42.60.0
|
||||
|
||||
`gbrain upgrade`. No new schema migrations.
|
||||
|
||||
1. **Windows users with git-synced sources:** re-run `gbrain sync --full` once after upgrading — if a pre-upgrade sync deleted subdirectory pages, they re-import from the repo.
|
||||
2. **Your first search after upgrading may be a cache miss** (the cache key now includes the exclude policy). Speeds return to normal as the cache refills within its TTL.
|
||||
3. **If agent jobs previously dead-lettered** with unbalanced tool-call transcript errors on OpenAI-compatible providers, retry them with `gbrain jobs retry <id>` — resume now reconciles the transcript.
|
||||
4. **Verify:**
|
||||
```bash
|
||||
gbrain doctor
|
||||
gbrain stats
|
||||
```
|
||||
5. **If any step fails,** please file an issue at https://github.com/garrytan/gbrain/issues with the output of `gbrain doctor`.
|
||||
|
||||
## [0.42.59.0] - 2026-07-13
|
||||
|
||||
**Five community-reported fixes, each reproduced and verified before/after on both engines (PGLite + real Postgres): an upgrade wedge that locked pre-v121 brains out of migrations, two data-integrity holes in engine migration, silent deletion of facts containing pipe characters, confidently-wrong entity attribution on ambiguous names, and tightened source-scope enforcement in `think`.**
|
||||
|
||||
@@ -1,5 +1,19 @@
|
||||
# TODOS
|
||||
|
||||
## community fix-wave follow-ups (filed v0.42.60.0)
|
||||
|
||||
- [ ] **P1 — take-writes source scoping fails open when source resolution errors (#2684 residual).**
|
||||
`resolveTakesSourceId` (src/commands/takes.ts) swallows resolution errors and returns
|
||||
`undefined`, which falls back to the unscoped slug-only page lookup — so an invalid
|
||||
`GBRAIN_SOURCE` (or a broken dotfile chain) silently restores the pre-#2698 cross-source
|
||||
write behavior on multi-source brains. Decide fail-closed semantics: error out when a
|
||||
source was explicitly requested but doesn't resolve; keep the unscoped fallback only for
|
||||
brains with no source configuration at all. Add a regression test for the invalid-source
|
||||
path. Found by cross-model adversarial review during the v0.42.60.0 release ship.
|
||||
- [ ] **P2 — cherry-pick #2112's uncovered doctor.ts hunk.** Fix-wave A (#2820) superseded
|
||||
most of #2112 but not its `checkSubagentCapability` fix (check explicit `models.subagent`
|
||||
before `models.tier.subagent`). Refile or cherry-pick; the rest of that PR is covered.
|
||||
|
||||
## v0.42.59.0 follow-ups (five-fix rollup #2735–#2739)
|
||||
|
||||
Filed as follow-ups from v0.42.59.0 (bootstrap probe for
|
||||
@@ -219,10 +233,7 @@ job) and sync. See CLAUDE.md "Pace Mode".
|
||||
`sourceScopeOpts`) and `code_def` (`operations.ts:4155` — brain-wide raw SQL over
|
||||
`content_chunks`; confirm whether brain-wide is intentional before scoping). A remote
|
||||
federated client (grant set, dispatch-default `ctx.sourceId='default'`) reads these against
|
||||
`default` or unscoped, not its grant. *(Partially done by v0.42.59.0: the engine methods
|
||||
`searchTakes`/`searchTakesVector` now accept the source-scope predicates and the `think`
|
||||
gather path threads them; the standalone `takes_search` op handler still doesn't route
|
||||
through `sourceScopeOpts(ctx)`.)*
|
||||
`default` or unscoped, not its grant.
|
||||
- **Why:** same cross-source correctness/isolation class #2200 targets; a federated client
|
||||
can't read chunks/raw-data/versions for an authorized non-default source, `resolve_slugs`
|
||||
can fuzzy-resolve across all sources, and `takes_search`/`code_def` query without the grant.
|
||||
@@ -964,7 +975,7 @@ default-off; these are the gates and extensions before any default flip.
|
||||
|
||||
- [ ] **v0.42+: cross-surface ablation before flipping `search.adaptive_return` default.** The gate ships default-off. Before turning it on in any `MODE_BUNDLES` tier, run the recall ablation (adaptive off vs on, recall-preserving caps) across `gbrain eval longmemeval`, `gbrain eval whoknows`, `gbrain eval suspected-contradictions`, and the BrainBench-Real replay (sibling gbrain-evals repo). Confirm recall@k / answer quality does not regress; pick the safe caps; probably flip `tokenmax` first (broadest searchLimit, most noise). On-surface evidence (the PrecisionMemBench precision/recall frontier: off 0.076/0.99, e1/o2 0.40/0.91, e1/o1 0.58/0.82) is recorded in `gbrain-evals/docs/benchmarks/2026-05-29-precisionmembench.md`. Priority: P2.
|
||||
- [ ] **v0.42+: fold adaptive-return params into KNOBS_HASH so adaptive-on calls can cache.** v0.41.33.0 skips `hybridSearchCached` entirely when the gate is on (cache-safe but cache-cold). Fold `adaptive_return` enabled + caps + `minKeep` into `knobsHash()` (append-only, bump `KNOBS_HASH_VERSION`) so a gate-on write segregates from a gate-off row and adaptive calls cache correctly. Required before any default flip (else default-on means cache-cold everywhere). See `src/core/search/mode.ts` KNOBS_HASH parts + `return-policy.ts`. Priority: P2 (paired with the default-flip ablation above).
|
||||
- [ ] **v0.42+: gentle adaptive gate on `think`'s gather stage (A3).** The plan's A3 decision was a gentler return-gate on `runThink`'s gather candidates (cleaner context, fewer tokens per reasoning call). Deferred because the benefit is unvalidated without a longmemeval answer-quality run, and trimming the answer path (even default-off) carries regression risk. gather fuses 4 streams (page / takes-keyword / takes-vector / graph); the gate must operate on the fused output with a higher min-keep than search, validated on `gbrain eval longmemeval` answer quality (not retrieval precision). *(The scope plumbing this was gated on landed in v0.42.59.0: `RunThinkOpts` carries `sourceId`/`allowedSources` and `runGather` threads the scope through every stream, so the gate work no longer blocks on it.)* Priority: P2.
|
||||
- [ ] **v0.42+: gentle adaptive gate on `think`'s gather stage (A3).** The plan's A3 decision was a gentler return-gate on `runThink`'s gather candidates (cleaner context, fewer tokens per reasoning call). Deferred because the benefit is unvalidated without a longmemeval answer-quality run, and trimming the answer path (even default-off) carries regression risk. gather fuses 4 streams (page / takes-keyword / takes-vector / graph); the gate must operate on the fused output with a higher min-keep than search, validated on `gbrain eval longmemeval` answer quality (not retrieval precision). Also: `RunThinkOpts` has no `sourceId` today, so think's gather runs unscoped (codex finding) — scope-isolated think needs that plumbing first. Priority: P2.
|
||||
- [ ] **v0.42+: `--explain` human header for adaptive_return.** The decision is in `HybridSearchMeta.adaptive_return` and surfaces in `--json` today. The per-result `explain-formatter.ts` is result-scoped and can't render a per-query meta line; the human `gbrain search --explain` header needs the meta threaded through `cli.ts:formatResult` (it currently only receives `results`). Add a one-line gate-decision header (intent / cap / kept of total). Priority: P3.
|
||||
- [ ] **v0.42+: structured-alias / facts-mode fidelity for the PrecisionMemBench eval.** The gbrain-evals benchmark seeds beliefs as pages with aliases in the body (real FTS). A second fidelity that exercises gbrain's structured alias/entity-resolution layer (facts with `valid_until` + entity resolution) would measure gbrain's structured-belief path on the 23 alias cases. Lives in gbrain-evals (`eval/precisionmembench/seed.ts` throws on `fidelity:'structured'` today). Priority: P3.
|
||||
|
||||
|
||||
File diff suppressed because one or more lines are too long
+8
-1
@@ -74,13 +74,20 @@ to the HTTP server, so no migration is required.
|
||||
gbrain serve --http --port 3131
|
||||
```
|
||||
|
||||
On first start, the server prints an **admin bootstrap token** to stderr:
|
||||
On first start in an interactive terminal, the server prints an **admin
|
||||
bootstrap token** to stderr:
|
||||
|
||||
```
|
||||
Admin bootstrap token: 3a1f9c...
|
||||
Open http://localhost:3131/admin and paste it to log in.
|
||||
```
|
||||
|
||||
On a non-TTY start (systemd, Docker, any piped or captured logs) the generated
|
||||
token is hidden so it never lands in log storage. For headless deploys either
|
||||
set `GBRAIN_ADMIN_BOOTSTRAP_TOKEN` to a value you control before starting, or
|
||||
run `gbrain serve --http --print-admin-token` once on a trusted terminal to
|
||||
force printing.
|
||||
|
||||
Save this token. Open `http://localhost:3131/admin` and paste it to access the
|
||||
dashboard. The dashboard shows live activity, registered clients, request logs,
|
||||
and per-client config export.
|
||||
|
||||
@@ -158,7 +158,7 @@ gbrain serve --http --port 3131 --bind 0.0.0.0
|
||||
|
||||
The `--bind 0.0.0.0` is important. By default the server binds to localhost only, which is correct for a personal install but blocks remote teammates. Setting `0.0.0.0` accepts connections from any interface.
|
||||
|
||||
The server prints an admin bootstrap token to stderr on first start. Save it. You'll use it once for the admin dashboard.
|
||||
The server prints an admin bootstrap token to stderr on first start when run in an interactive terminal. Save it. You'll use it once for the admin dashboard. On a non-TTY start (systemd, Docker, piped logs) the token is hidden from logs — set `GBRAIN_ADMIN_BOOTSTRAP_TOKEN` yourself or pass `--print-admin-token` on a trusted terminal instead.
|
||||
|
||||
For development, tunnel the local server out via ngrok:
|
||||
|
||||
|
||||
+8
-1
@@ -3643,13 +3643,20 @@ to the HTTP server, so no migration is required.
|
||||
gbrain serve --http --port 3131
|
||||
```
|
||||
|
||||
On first start, the server prints an **admin bootstrap token** to stderr:
|
||||
On first start in an interactive terminal, the server prints an **admin
|
||||
bootstrap token** to stderr:
|
||||
|
||||
```
|
||||
Admin bootstrap token: 3a1f9c...
|
||||
Open http://localhost:3131/admin and paste it to log in.
|
||||
```
|
||||
|
||||
On a non-TTY start (systemd, Docker, any piped or captured logs) the generated
|
||||
token is hidden so it never lands in log storage. For headless deploys either
|
||||
set `GBRAIN_ADMIN_BOOTSTRAP_TOKEN` to a value you control before starting, or
|
||||
run `gbrain serve --http --print-admin-token` once on a trusted terminal to
|
||||
force printing.
|
||||
|
||||
Save this token. Open `http://localhost:3131/admin` and paste it to access the
|
||||
dashboard. The dashboard shows live activity, registered clients, request logs,
|
||||
and per-client config export.
|
||||
|
||||
+1
-1
@@ -144,5 +144,5 @@
|
||||
"bun": ">=1.3.10"
|
||||
},
|
||||
"license": "MIT",
|
||||
"version": "0.42.59.0"
|
||||
"version": "0.42.60.0"
|
||||
}
|
||||
|
||||
@@ -130,6 +130,37 @@ export function shouldSpawnAutopilotWorker(args: string[]): boolean {
|
||||
return !args.includes('--no-worker');
|
||||
}
|
||||
|
||||
export function isPidAlive(pid: number): boolean {
|
||||
if (!Number.isFinite(pid) || pid <= 0) return false;
|
||||
try {
|
||||
process.kill(pid, 0);
|
||||
return true;
|
||||
} catch (error: unknown) {
|
||||
return (error as NodeJS.ErrnoException).code === 'EPERM';
|
||||
}
|
||||
}
|
||||
|
||||
export function decideLockAcquisition(
|
||||
lockPath: string,
|
||||
currentPid: number,
|
||||
): { action: 'acquire' } | { action: 'exit'; holderPid: number } | { action: 'takeover'; reason: string } {
|
||||
if (!existsSync(lockPath)) return { action: 'acquire' };
|
||||
|
||||
let raw = '';
|
||||
try {
|
||||
raw = readFileSync(lockPath, 'utf-8').trim();
|
||||
} catch {
|
||||
// An unreadable lock cannot prove another process is alive.
|
||||
}
|
||||
|
||||
const holderPid = Number.parseInt(raw, 10);
|
||||
const sameProcess = Number.isFinite(holderPid) && holderPid === currentPid;
|
||||
const alive = !sameProcess && isPidAlive(holderPid);
|
||||
|
||||
if (alive) return { action: 'exit', holderPid };
|
||||
return { action: 'takeover', reason: `dead pid ${raw || '<empty>'}` };
|
||||
}
|
||||
|
||||
// ── Self-upgrade silent channel (v0.42; opt-in, supervisor-relaunch) ─────────
|
||||
|
||||
/**
|
||||
@@ -351,14 +382,13 @@ export async function runAutopilot(engine: BrainEngine, args: string[]) {
|
||||
const lockPath = gbrainHomePath('autopilot.lock');
|
||||
try {
|
||||
mkdirSync(gbrainHomePath(), { recursive: true });
|
||||
if (existsSync(lockPath)) {
|
||||
const stat = require('fs').statSync(lockPath);
|
||||
const ageMinutes = (Date.now() - stat.mtimeMs) / 60000;
|
||||
if (ageMinutes < 10) {
|
||||
console.error('Another autopilot instance is running (lock file is fresh). Exiting.');
|
||||
process.exit(0);
|
||||
}
|
||||
console.log('Stale lock file found (>10 min). Taking over.');
|
||||
const decision = decideLockAcquisition(lockPath, process.pid);
|
||||
if (decision.action === 'exit') {
|
||||
console.error(`Another autopilot instance is running (pid ${decision.holderPid}). Exiting.`);
|
||||
process.exit(0);
|
||||
}
|
||||
if (decision.action === 'takeover') {
|
||||
console.log(`Stale autopilot lock found (${decision.reason}). Taking over.`);
|
||||
}
|
||||
writeFileSync(lockPath, String(process.pid));
|
||||
} catch { /* best-effort */ }
|
||||
|
||||
@@ -257,7 +257,9 @@ function buildChapterPrompt(
|
||||
|
||||
return `You are analyzing one chapter of "${bookTitle}"${authorLine} for the user.
|
||||
|
||||
Your output is a markdown two-column table where the LEFT column preserves the chapter's actual content (stories, frameworks, statistics, named examples) and the RIGHT column maps each idea to the user's actual life using their words, situations, and patterns from the brain.
|
||||
Your output is a two-column HTML table where the LEFT column preserves the chapter's actual content (stories, frameworks, statistics, named examples) and the RIGHT column maps each idea to the user's actual life using their words, situations, and patterns from the brain.
|
||||
|
||||
CRITICAL: Use an HTML <table> with valign="top" on EVERY cell — NOT a markdown pipe table. Markdown pipe tables have no way to set vertical alignment, so every renderer except GitHub middle-aligns the rows, which is unreadable when the two columns have different lengths. The HTML <table valign="top"> form top-aligns everywhere (GitHub, PDF, Obsidian).
|
||||
|
||||
This is chapter ${chapter.index} of ${totalChapters}.
|
||||
|
||||
@@ -276,11 +278,12 @@ Return ONLY a single markdown section in this exact shape:
|
||||
### Key Ideas
|
||||
[2-4 sentence thesis of the chapter — what the author is actually arguing.]
|
||||
|
||||
| What the Author Says | How This Applies to You |
|
||||
|---|---|
|
||||
| [Detailed paragraph: a section/argument from the chapter, preserving stories, stats, frameworks, named examples. Use \`<br><br>\` for paragraph breaks within the cell.] | [Specific personal connection: name dates, people, exact quotes from the user, real situations. Same \`<br><br>\` for breaks.] |
|
||||
| [Next section] | [Next mirror] |
|
||||
| [4-10 rows depending on chapter density] | |
|
||||
<table>
|
||||
<tr><th align="left">What the Author Says</th><th align="left">How This Applies to You</th></tr>
|
||||
<tr><td valign="top">[Detailed paragraph: a section/argument from the chapter, preserving stories, stats, frameworks, named examples. Use \`<br><br>\` for paragraph breaks within the cell.]</td><td valign="top">[Specific personal connection: name dates, people, exact quotes from the user, real situations. Same \`<br><br>\` for breaks.]</td></tr>
|
||||
<tr><td valign="top">[Next section]</td><td valign="top">[Next mirror]</td></tr>
|
||||
[4-10 rows depending on chapter density]
|
||||
</table>
|
||||
\`\`\`
|
||||
|
||||
## RULES
|
||||
@@ -290,6 +293,7 @@ Return ONLY a single markdown section in this exact shape:
|
||||
- 4-10 rows per chapter. If a section honestly doesn't apply, write \`*This section is less directly relevant because [specific reason].*\` Don't force connections.
|
||||
- Never generic ("This might apply if you've ever felt..."). Never sycophantic. Never preach.
|
||||
- Use \`<br><br>\` for paragraph breaks inside table cells, not literal newlines.
|
||||
- EVERY <td> MUST carry valign="top". Never emit a markdown pipe table (| ... | ... |) — always the HTML <table> form above.
|
||||
|
||||
You have ${DEFAULT_MAX_TURNS} turns and read-only tools (get_page, search). You CANNOT call put_page — your output is the markdown text in your final message. The CLI assembles all chapters and writes the brain page.
|
||||
|
||||
@@ -314,7 +318,7 @@ title: "${opts.title} — Personalized"
|
||||
type: book-analysis${authorLine}
|
||||
date: ${today}
|
||||
context: "${contextSummary.replace(/"/g, '\\"')}"
|
||||
tags: [book, personalized, two-column]
|
||||
tags: [book, personalized, two-column-htmltable-valign-top]
|
||||
---`;
|
||||
|
||||
const intro = `# ${opts.title} — Personalized
|
||||
|
||||
@@ -494,6 +494,38 @@ export function extractTimelineFromContent(content: string, slug: string): Extra
|
||||
entries.push({ slug, date: match[1], source: 'markdown', summary: match[2].trim(), detail: detail || undefined });
|
||||
}
|
||||
|
||||
// Format 3: Inline citation — [Source: <source>, YYYY-MM-DD]
|
||||
//
|
||||
// This is the citation convention gbrain's own quality rules require on
|
||||
// every brain write (skills/conventions/quality.md), so dated evidence is
|
||||
// pervasive in curated pages — but until now the extractor could not see
|
||||
// it, and a page whose dates all live in citations scored zero timeline
|
||||
// coverage. The entry's summary is the sentence the citation annotates
|
||||
// (the surrounding line with citation markers stripped).
|
||||
//
|
||||
// Lines already captured by Format 1 are skipped: a timeline bullet often
|
||||
// carries its own [Source: ...] citation, and re-extracting it would file
|
||||
// a duplicate entry under a different (source, summary) shape that the
|
||||
// DB-level uniqueness cannot collapse.
|
||||
const citationPattern = /\[Source:\s*([^\]]+?),\s*(\d{4}-\d{2}-\d{2})\s*\]/g;
|
||||
const bulletLinePattern = /^-\s+\*\*\d{4}-\d{2}-\d{2}\*\*\s*\|/;
|
||||
for (const line of content.split(/\r?\n/)) {
|
||||
if (bulletLinePattern.test(line)) continue;
|
||||
const lineMatches = [...line.matchAll(citationPattern)];
|
||||
if (lineMatches.length === 0) continue;
|
||||
// Strip every citation marker from the line to leave the annotated text.
|
||||
const summary = line
|
||||
.replace(/\[Source:[^\]]*\]/g, '')
|
||||
.replace(/^[-*>#\s]+/, '')
|
||||
.replace(/\s+/g, ' ')
|
||||
.trim()
|
||||
.slice(0, 300);
|
||||
if (!summary) continue; // a bare citation with no surrounding text is not an event
|
||||
for (const m of lineMatches) {
|
||||
entries.push({ slug, date: m[2], source: m[1].trim().slice(0, 200), summary });
|
||||
}
|
||||
}
|
||||
|
||||
return entries;
|
||||
}
|
||||
|
||||
|
||||
@@ -116,8 +116,7 @@ async function listFiles(engine: BrainEngine, slug?: string) {
|
||||
|
||||
console.log(`${rows.length} file(s):`);
|
||||
for (const row of rows) {
|
||||
const sizeBytes = row.size_bytes as number | null;
|
||||
const size = sizeBytes ? `${Math.round(sizeBytes / 1024)}KB` : '?';
|
||||
const size = row.size_bytes ? `${Math.round(Number(row.size_bytes) / 1024)}KB` : '?';
|
||||
console.log(` ${row.page_slug || '(unlinked)'} / ${row.filename} [${size}, ${row.mime_type || '?'}]`);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -53,7 +53,15 @@ const DENY_PREFIXES = [
|
||||
];
|
||||
|
||||
/** First slug segments where no inbound links is expected */
|
||||
const FIRST_SEGMENT_EXCLUSIONS = new Set(['scratch', 'thoughts', 'catalog', 'entities']);
|
||||
const FIRST_SEGMENT_EXCLUSIONS = new Set([
|
||||
'scratch',
|
||||
'thoughts',
|
||||
'catalog',
|
||||
'entities',
|
||||
'raw',
|
||||
'atoms',
|
||||
'skills',
|
||||
]);
|
||||
|
||||
// --- Filter logic ---
|
||||
|
||||
|
||||
@@ -23,6 +23,7 @@ import {
|
||||
addAliasToType,
|
||||
addLinkTypeToPack,
|
||||
addPrefixToType,
|
||||
BUNDLED_PACK_NAMES,
|
||||
addTypeToPack,
|
||||
invalidatePackCache,
|
||||
loadActivePack,
|
||||
@@ -179,7 +180,7 @@ async function runActive(_args: string[]): Promise<void> {
|
||||
}
|
||||
|
||||
function runList(_args: string[]): void {
|
||||
const bundled = ['gbrain-base', 'gbrain-recommended'];
|
||||
const bundled = [...BUNDLED_PACK_NAMES];
|
||||
const installedDir = gbrainPath('schema-packs');
|
||||
const installed: string[] = [];
|
||||
if (existsSync(installedDir)) {
|
||||
@@ -366,12 +367,12 @@ function runUse(args: string[]): void {
|
||||
}
|
||||
|
||||
function packPathByName(name: string): string | null {
|
||||
if (name === 'gbrain-base') {
|
||||
if (BUNDLED_PACK_NAMES.has(name)) {
|
||||
// Resolve bundled YAML — try a few locations.
|
||||
const here = dirname(new URL(import.meta.url).pathname);
|
||||
const candidates = [
|
||||
join(here, '..', 'core', 'schema-pack', 'base', 'gbrain-base.yaml'),
|
||||
join(here, '..', '..', 'src', 'core', 'schema-pack', 'base', 'gbrain-base.yaml'),
|
||||
join(here, '..', 'core', 'schema-pack', 'base', `${name}.yaml`),
|
||||
join(here, '..', '..', 'src', 'core', 'schema-pack', 'base', `${name}.yaml`),
|
||||
];
|
||||
for (const c of candidates) {
|
||||
if (existsSync(c)) return c;
|
||||
|
||||
@@ -90,6 +90,26 @@ export function resolveBootstrapToken(
|
||||
return { kind: 'ok', token: trimmed, fromEnv: true };
|
||||
}
|
||||
|
||||
/**
|
||||
* #2624: decide whether the generated admin bootstrap token is hidden from
|
||||
* the startup banner. Fail-safe default: a generated token is NOT printed
|
||||
* unless stderr is an interactive TTY, so containerized (non-TTY) deploys
|
||||
* never ship the secret to centralized log storage. Env-sourced tokens are
|
||||
* always hidden (operator already holds them). Explicit --suppress hides
|
||||
* everything; --print-admin-token forces the raw value even on a non-TTY.
|
||||
*/
|
||||
export function shouldSuppressBootstrapPrint(opts: {
|
||||
suppress: boolean;
|
||||
fromEnv: boolean;
|
||||
forcePrint: boolean;
|
||||
isTty: boolean;
|
||||
}): boolean {
|
||||
if (opts.suppress) return true;
|
||||
if (opts.fromEnv) return true;
|
||||
if (opts.forcePrint) return false;
|
||||
return !opts.isTty;
|
||||
}
|
||||
|
||||
export type ProbeHealthResult =
|
||||
| { ok: true; status: 200; body: { status: 'ok'; version: string; engine: string; [k: string]: unknown } }
|
||||
| { ok: false; status: 503; body: { error: 'service_unavailable'; error_description: string } };
|
||||
@@ -304,6 +324,14 @@ interface ServeHttpOptions {
|
||||
* tracking the regenerated value through other means.
|
||||
*/
|
||||
suppressBootstrapToken?: boolean;
|
||||
/**
|
||||
* #2624: force-print the generated admin bootstrap token even on a
|
||||
* non-TTY (containerized) start. By default the raw token is only printed
|
||||
* when stderr is an interactive TTY, so it never lands in centralized log
|
||||
* storage for headless deploys. Set this when you genuinely need the value
|
||||
* captured to a non-interactive log and accept the leak.
|
||||
*/
|
||||
printAdminToken?: boolean;
|
||||
}
|
||||
|
||||
/**
|
||||
@@ -530,7 +558,12 @@ export async function runServeHttp(engine: BrainEngine, options: ServeHttpOption
|
||||
let bootstrapToken: string = resolved.token;
|
||||
let bootstrapFromEnv: boolean = resolved.fromEnv;
|
||||
const bootstrapHash = createHash('sha256').update(bootstrapToken).digest('hex');
|
||||
const suppressBootstrapPrint = options.suppressBootstrapToken === true;
|
||||
const suppressBootstrapPrint = shouldSuppressBootstrapPrint({
|
||||
suppress: options.suppressBootstrapToken === true,
|
||||
fromEnv: bootstrapFromEnv,
|
||||
forcePrint: options.printAdminToken === true,
|
||||
isTty: process.stderr.isTTY === true,
|
||||
});
|
||||
const adminSessions = new Map<string, number>(); // sessionId → expiresAt
|
||||
|
||||
// SSE clients for live activity feed
|
||||
@@ -2166,10 +2199,10 @@ export async function runServeHttp(engine: BrainEngine, options: ServeHttpOption
|
||||
║ MCP: http://localhost:${port}/mcp${' '.repeat(Math.max(0, 21 - String(port).length))}║
|
||||
║ Health: http://localhost:${port}/health${' '.repeat(Math.max(0, 18 - String(port).length))}║
|
||||
╠══════════════════════════════════════════════════════╣
|
||||
${suppressBootstrapPrint
|
||||
? '║ Admin Token: suppressed (--suppress-bootstrap-token) ║\n╚══════════════════════════════════════════════════════╝'
|
||||
: bootstrapFromEnv
|
||||
? '║ Admin Token: from $GBRAIN_ADMIN_BOOTSTRAP_TOKEN ║\n╚══════════════════════════════════════════════════════╝'
|
||||
${bootstrapFromEnv
|
||||
? '║ Admin Token: from $GBRAIN_ADMIN_BOOTSTRAP_TOKEN ║\n╚══════════════════════════════════════════════════════╝'
|
||||
: suppressBootstrapPrint
|
||||
? '║ Admin Token: hidden (non-TTY log-leak guard) ║\n║ set $GBRAIN_ADMIN_BOOTSTRAP_TOKEN, or pass ║\n║ --print-admin-token on a trusted terminal. ║\n╚══════════════════════════════════════════════════════╝'
|
||||
: `║ Admin Token (paste into /admin login): ║\n║ ${bootstrapToken.substring(0, 50)} ║\n║ ${bootstrapToken.substring(50).padEnd(50)} ║\n╚══════════════════════════════════════════════════════╝`}
|
||||
`);
|
||||
});
|
||||
|
||||
@@ -118,8 +118,13 @@ export async function runServe(
|
||||
// restart.
|
||||
const suppressBootstrapToken = args.includes('--suppress-bootstrap-token');
|
||||
|
||||
// #2624: by default the generated token only prints on an interactive
|
||||
// TTY (never into container log storage). --print-admin-token forces the
|
||||
// raw value even on a non-TTY start.
|
||||
const printAdminToken = args.includes('--print-admin-token');
|
||||
|
||||
const { runServeHttp } = await import('./serve-http.ts');
|
||||
await runServeHttp(engine, { port, tokenTtl, enableDcr, enableDcrInsecure, publicUrl, logFullParams, bind, suppressBootstrapToken });
|
||||
await runServeHttp(engine, { port, tokenTtl, enableDcr, enableDcrInsecure, publicUrl, logFullParams, bind, suppressBootstrapToken, printAdminToken });
|
||||
return;
|
||||
}
|
||||
|
||||
|
||||
+109
-12
@@ -3109,23 +3109,44 @@ async function performFullSync(
|
||||
// repo-relative (importFile uses `relative(dir, filePath)`), so relativize
|
||||
// to the same form before membership-testing — otherwise every page looks
|
||||
// stale and the reconcile would wrongly delete live pages.
|
||||
const current = new Set(
|
||||
collectSyncableFiles(repoPath, { strategy: opts.strategy ?? 'markdown' })
|
||||
.map(abs => relative(repoPath, abs)),
|
||||
);
|
||||
//
|
||||
// #2828: planReconcileDeletes ALSO normalizes path separators on both sides
|
||||
// of the membership test. On a Windows checkout `path.relative` yields
|
||||
// backslash paths while a stored source_path can hold git-derived forward
|
||||
// slashes; without normalization every file-backed page mismatches, looks
|
||||
// stale, and the reconcile wipes the whole source.
|
||||
const currentFiles = collectSyncableFiles(repoPath, { strategy: opts.strategy ?? 'markdown' })
|
||||
.map(abs => relative(repoPath, abs));
|
||||
const rows = await engine.executeRaw<{ slug: string; source_path: string | null }>(
|
||||
`SELECT slug, source_path FROM pages WHERE source_id = $1 AND source_path IS NOT NULL AND deleted_at IS NULL`,
|
||||
[sid],
|
||||
);
|
||||
const staleSlugs = rows
|
||||
.filter(r => r.source_path != null
|
||||
&& isSyncable(r.source_path, reconcileSyncOpts)
|
||||
&& !current.has(r.source_path))
|
||||
.map(r => r.slug);
|
||||
if (staleSlugs.length > 0) {
|
||||
const plan = planReconcileDeletes(
|
||||
rows,
|
||||
currentFiles,
|
||||
p => isSyncable(p, reconcileSyncOpts),
|
||||
);
|
||||
if (plan.staleSlugs.length > 0 && plan.massDelete && !massReconcileAllowed()) {
|
||||
// #2828 mass-delete safety valve: a reconcile that would sweep more than
|
||||
// half of the pages this strategy manages, on a source with a non-trivial
|
||||
// number of them, is almost always a path-comparison bug or the wrong repo
|
||||
// path — NOT a genuine bulk deletion. Skip the delete and warn loudly
|
||||
// instead of silently wiping the brain.
|
||||
serr(
|
||||
`\n WARNING: refusing to reconcile-delete ${plan.staleSlugs.length} of ` +
|
||||
`${plan.reconcilableCount} file-backed page(s) for source '${sid}' ` +
|
||||
`(> ${Math.round(MASS_RECONCILE_RATIO * 100)}% of them).\n` +
|
||||
` A full sync removes pages only when their backing file is gone. Deleting\n` +
|
||||
` this many at once almost always means the paths were compared wrong (e.g.\n` +
|
||||
` a path-separator mismatch) or the WRONG repo path was synced — not that\n` +
|
||||
` you actually deleted that many files. No pages were deleted.\n` +
|
||||
` If this bulk removal is genuinely intended, re-run with ` +
|
||||
`GBRAIN_ALLOW_MASS_RECONCILE=1 to restore the old behavior.`,
|
||||
);
|
||||
} else if (plan.staleSlugs.length > 0) {
|
||||
const deleteScopedOpts = { sourceId: sid };
|
||||
for (let i = 0; i < staleSlugs.length; i += DELETE_BATCH_SIZE) {
|
||||
const batch = staleSlugs.slice(i, i + DELETE_BATCH_SIZE);
|
||||
for (let i = 0; i < plan.staleSlugs.length; i += DELETE_BATCH_SIZE) {
|
||||
const batch = plan.staleSlugs.slice(i, i + DELETE_BATCH_SIZE);
|
||||
try {
|
||||
const deleted = await engine.deletePages(batch, deleteScopedOpts);
|
||||
reconciledDeletes += deleted.length;
|
||||
@@ -3179,6 +3200,82 @@ async function performFullSync(
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* #2828 full-sync reconcile safety-valve thresholds. A reconcile that would
|
||||
* delete more than MASS_RECONCILE_RATIO of the file-backed pages a strategy
|
||||
* manages, on a source that holds more than MASS_RECONCILE_MIN_PAGES of them, is
|
||||
* treated as a suspected path-comparison bug rather than a real bulk deletion.
|
||||
*/
|
||||
export const MASS_RECONCILE_RATIO = 0.5;
|
||||
export const MASS_RECONCILE_MIN_PAGES = 20;
|
||||
|
||||
/**
|
||||
* Normalize path separators so a page whose stored `source_path` was written
|
||||
* with a different separator than the local OS's `path.relative` produces (e.g.
|
||||
* git-derived forward-slash paths on a Windows checkout) still compares equal.
|
||||
* Without this, on Windows every file-backed page looks stale and the reconcile
|
||||
* wrongly deletes the whole source (#2828).
|
||||
*/
|
||||
function normalizeReconcilePath(p: string): string {
|
||||
return p.replace(/\\/g, '/');
|
||||
}
|
||||
|
||||
export interface ReconcilePlan {
|
||||
/** Slugs whose backing file is genuinely gone; safe to reconcile-delete. */
|
||||
staleSlugs: string[];
|
||||
/**
|
||||
* File-backed, in-strategy pages the reconcile can act on. This is the
|
||||
* denominator for the mass-delete valve (the exact population at risk).
|
||||
*/
|
||||
reconcilableCount: number;
|
||||
/**
|
||||
* True when `staleSlugs` would sweep more than MASS_RECONCILE_RATIO of
|
||||
* `reconcilableCount`, on a source with more than MASS_RECONCILE_MIN_PAGES of
|
||||
* them — the mass-delete signal that trips the safety valve.
|
||||
*/
|
||||
massDelete: boolean;
|
||||
}
|
||||
|
||||
/**
|
||||
* #2828: decide which file-backed pages a full-sync reconcile should delete, and
|
||||
* whether that deletion is suspiciously large. Pure and exported so both the
|
||||
* separator normalization and the mass-delete valve are unit-testable without a
|
||||
* live engine or a Windows host.
|
||||
*
|
||||
* @param rows pages with a non-null `source_path` (deleted_at IS NULL).
|
||||
* @param currentFiles repo-relative paths present in the working tree.
|
||||
* @param isSyncablePath predicate excluding metafiles and the wrong strategy.
|
||||
*/
|
||||
export function planReconcileDeletes(
|
||||
rows: ReadonlyArray<{ slug: string; source_path: string | null }>,
|
||||
currentFiles: Iterable<string>,
|
||||
isSyncablePath: (p: string) => boolean,
|
||||
): ReconcilePlan {
|
||||
const current = new Set<string>();
|
||||
for (const f of currentFiles) current.add(normalizeReconcilePath(f));
|
||||
const reconcilable = rows.filter(
|
||||
r => r.source_path != null && isSyncablePath(r.source_path),
|
||||
);
|
||||
const staleSlugs = reconcilable
|
||||
.filter(r => !current.has(normalizeReconcilePath(r.source_path as string)))
|
||||
.map(r => r.slug);
|
||||
const massDelete =
|
||||
reconcilable.length > MASS_RECONCILE_MIN_PAGES &&
|
||||
staleSlugs.length > reconcilable.length * MASS_RECONCILE_RATIO;
|
||||
return { staleSlugs, reconcilableCount: reconcilable.length, massDelete };
|
||||
}
|
||||
|
||||
/**
|
||||
* #2828 escape hatch: `GBRAIN_ALLOW_MASS_RECONCILE=1` restores the pre-valve
|
||||
* behavior for the rare intentional bulk removal. Env-only (an incident-time
|
||||
* override), mirroring `resolveStallAbortSeconds`' pure, env-parameterized shape.
|
||||
*/
|
||||
export function massReconcileAllowed(
|
||||
env: Record<string, string | undefined> = process.env,
|
||||
): boolean {
|
||||
return env.GBRAIN_ALLOW_MASS_RECONCILE === '1';
|
||||
}
|
||||
|
||||
/**
|
||||
* Grace window (seconds) between the watchdog's SIGTERM and SIGKILL. SIGTERM
|
||||
* gives a responsive loop a clean shutdown; SIGKILL is the starvation backstop.
|
||||
|
||||
+36
-19
@@ -28,6 +28,7 @@ import {
|
||||
type ParsedTake,
|
||||
} from '../core/takes-fence.ts';
|
||||
import { withPageLock } from '../core/page-lock.ts';
|
||||
import { resolveSourceId } from '../core/source-resolver.ts';
|
||||
|
||||
// --- Helpers ---
|
||||
|
||||
@@ -83,18 +84,31 @@ function ensureFloat(raw: string | undefined, fallback: number): number {
|
||||
return n;
|
||||
}
|
||||
|
||||
async function getPageId(engine: BrainEngine, slug: string): Promise<number> {
|
||||
const rows = await engine.executeRaw<{ id: number }>(
|
||||
`SELECT id FROM pages WHERE slug = $1 LIMIT 1`,
|
||||
[slug],
|
||||
);
|
||||
async function getPageId(engine: BrainEngine, slug: string, sourceId?: string): Promise<number> {
|
||||
const rows = sourceId
|
||||
? await engine.executeRaw<{ id: number }>(
|
||||
`SELECT id FROM pages WHERE slug = $1 AND source_id = $2 LIMIT 1`,
|
||||
[slug, sourceId],
|
||||
)
|
||||
: await engine.executeRaw<{ id: number }>(
|
||||
`SELECT id FROM pages WHERE slug = $1 LIMIT 1`,
|
||||
[slug],
|
||||
);
|
||||
if (!rows[0]) {
|
||||
console.error(`Page not found in brain: ${slug}. Run \`gbrain sync\` first.`);
|
||||
console.error(`Page not found in brain: ${slug}${sourceId ? ` (source=${sourceId})` : ''}. Run \`gbrain sync\` first.`);
|
||||
process.exit(1);
|
||||
}
|
||||
return rows[0].id;
|
||||
}
|
||||
|
||||
async function resolveTakesSourceId(engine: BrainEngine): Promise<string | undefined> {
|
||||
try {
|
||||
return await resolveSourceId(engine, null);
|
||||
} catch {
|
||||
return undefined;
|
||||
}
|
||||
}
|
||||
|
||||
function readBodyOrEmpty(path: string): string {
|
||||
if (!existsSync(path)) return '';
|
||||
return readFileSync(path, 'utf-8');
|
||||
@@ -169,7 +183,7 @@ async function cmdSearch(engine: BrainEngine, args: string[]): Promise<void> {
|
||||
}
|
||||
}
|
||||
|
||||
async function cmdAdd(engine: BrainEngine, args: string[]): Promise<void> {
|
||||
async function cmdAdd(engine: BrainEngine, args: string[], sourceId?: string): Promise<void> {
|
||||
const slug = args[0];
|
||||
if (!slug) {
|
||||
console.error('Usage: gbrain takes add <slug> --claim "..." --kind <k> --who <h> [--weight 0.5] [--source "..."] [--since YYYY-MM]');
|
||||
@@ -195,7 +209,7 @@ async function cmdAdd(engine: BrainEngine, args: string[]): Promise<void> {
|
||||
writeBody(path, nextBody);
|
||||
|
||||
// Mirror to DB. Page may not be in DB yet if not synced — caller must run sync first.
|
||||
const pageId = await getPageId(engine, slug);
|
||||
const pageId = await getPageId(engine, slug, sourceId);
|
||||
await engine.addTakesBatch([{
|
||||
page_id: pageId, row_num: rowNum, claim, kind, holder, weight,
|
||||
since_date: since, source, active: true, superseded_by: null,
|
||||
@@ -204,7 +218,7 @@ async function cmdAdd(engine: BrainEngine, args: string[]): Promise<void> {
|
||||
});
|
||||
}
|
||||
|
||||
async function cmdUpdate(engine: BrainEngine, args: string[]): Promise<void> {
|
||||
async function cmdUpdate(engine: BrainEngine, args: string[], sourceId?: string): Promise<void> {
|
||||
const slug = args[0];
|
||||
const rowNumStr = flagValue(args, '--row');
|
||||
if (!slug || !rowNumStr) {
|
||||
@@ -223,7 +237,7 @@ async function cmdUpdate(engine: BrainEngine, args: string[]): Promise<void> {
|
||||
const brainDir = await resolveBrainDir(engine, dirArg ?? null);
|
||||
|
||||
await withPageLock(slug, async () => {
|
||||
const pageId = await getPageId(engine, slug);
|
||||
const pageId = await getPageId(engine, slug, sourceId);
|
||||
await engine.updateTake(pageId, rowNum, fields);
|
||||
|
||||
// Sync the markdown table: read fence, find row, apply field updates, re-render.
|
||||
@@ -254,7 +268,7 @@ async function cmdUpdate(engine: BrainEngine, args: string[]): Promise<void> {
|
||||
});
|
||||
}
|
||||
|
||||
async function cmdSupersede(engine: BrainEngine, args: string[]): Promise<void> {
|
||||
async function cmdSupersede(engine: BrainEngine, args: string[], sourceId?: string): Promise<void> {
|
||||
const slug = args[0];
|
||||
const rowNumStr = flagValue(args, '--row');
|
||||
if (!slug || !rowNumStr) {
|
||||
@@ -268,7 +282,7 @@ async function cmdSupersede(engine: BrainEngine, args: string[]): Promise<void>
|
||||
const brainDir = await resolveBrainDir(engine, dirArg ?? null);
|
||||
|
||||
await withPageLock(slug, async () => {
|
||||
const pageId = await getPageId(engine, slug);
|
||||
const pageId = await getPageId(engine, slug, sourceId);
|
||||
|
||||
// Read existing row to inherit kind/holder unless overridden
|
||||
const existing = await engine.listTakes({ page_id: pageId, active: false, limit: 500 });
|
||||
@@ -302,7 +316,7 @@ async function cmdSupersede(engine: BrainEngine, args: string[]): Promise<void>
|
||||
});
|
||||
}
|
||||
|
||||
async function cmdResolve(engine: BrainEngine, args: string[]): Promise<void> {
|
||||
async function cmdResolve(engine: BrainEngine, args: string[], sourceId?: string): Promise<void> {
|
||||
const slug = args[0];
|
||||
const rowNumStr = flagValue(args, '--row');
|
||||
const qualityStr = flagValue(args, '--quality');
|
||||
@@ -347,7 +361,7 @@ async function cmdResolve(engine: BrainEngine, args: string[]): Promise<void> {
|
||||
const resolvedBy = flagValue(args, '--by') ?? 'garry';
|
||||
const dirArg = flagValue(args, '--dir');
|
||||
|
||||
const pageId = await getPageId(engine, slug);
|
||||
const pageId = await getPageId(engine, slug, sourceId);
|
||||
await engine.resolveTake(pageId, rowNum, {
|
||||
quality,
|
||||
outcome,
|
||||
@@ -564,10 +578,10 @@ Common flags:
|
||||
|
||||
switch (sub) {
|
||||
case 'search': return cmdSearch(engine, rest);
|
||||
case 'add': return cmdAdd(engine, rest);
|
||||
case 'update': return cmdUpdate(engine, rest);
|
||||
case 'supersede': return cmdSupersede(engine, rest);
|
||||
case 'resolve': return cmdResolve(engine, rest);
|
||||
case 'add': return cmdAdd(engine, rest, await resolveTakesSourceId(engine));
|
||||
case 'update': return cmdUpdate(engine, rest, await resolveTakesSourceId(engine));
|
||||
case 'supersede': return cmdSupersede(engine, rest, await resolveTakesSourceId(engine));
|
||||
case 'resolve': return cmdResolve(engine, rest, await resolveTakesSourceId(engine));
|
||||
case 'scorecard': return cmdScorecard(engine, rest);
|
||||
case 'calibration': return cmdCalibration(engine, rest);
|
||||
case 'revisit': return cmdRevisit(engine, rest);
|
||||
@@ -591,7 +605,8 @@ async function cmdExtract(engine: BrainEngine, rest: string[]): Promise<void> {
|
||||
const sub = rest[0];
|
||||
if (sub !== '--from-pages') {
|
||||
process.stderr.write(
|
||||
'Usage: gbrain takes extract --from-pages [--yes] [--dry-run] [--source-id <id>] [--max-pages N] [--holder <name>]\n',
|
||||
'Usage: gbrain takes extract --from-pages [--yes] [--dry-run] [--source-id <id>] [--max-pages N (clamped to 1000)] [--include-covered] [--holder <name>]\n' +
|
||||
'Runs progress: pages that already hold takes are skipped, so repeat runs sweep a large corpus in slices. --include-covered rescans everything (refresh).\n',
|
||||
);
|
||||
process.exit(1);
|
||||
}
|
||||
@@ -604,6 +619,7 @@ async function cmdExtract(engine: BrainEngine, rest: string[]): Promise<void> {
|
||||
const maxPages = maxPagesRaw ? Math.max(1, Math.min(1000, parseInt(maxPagesRaw, 10) || 50)) : 50;
|
||||
const holderIdx = rest.indexOf('--holder');
|
||||
const holder = holderIdx >= 0 ? rest[holderIdx + 1] : 'system';
|
||||
const includeCovered = rest.includes('--include-covered');
|
||||
|
||||
// A12 consent gate.
|
||||
const bootstrapEnabledCfg = await engine.getConfig('takes.bootstrap_enabled');
|
||||
@@ -628,6 +644,7 @@ async function cmdExtract(engine: BrainEngine, rest: string[]): Promise<void> {
|
||||
dryRun,
|
||||
sourceIdFilter,
|
||||
maxPages,
|
||||
includeCovered,
|
||||
holder,
|
||||
});
|
||||
if (result.llm_unavailable) {
|
||||
|
||||
@@ -34,6 +34,10 @@ export function buildGatewayConfig(c: GBrainConfig): AIGatewayConfig {
|
||||
// plane field now exists (GBrainConfig type) and gets mapped here, so
|
||||
// setting it via `~/.gbrain/config.json` propagates into the gateway.
|
||||
if (c.zeroentropy_api_key) envFromConfig.ZEROENTROPY_API_KEY = c.zeroentropy_api_key;
|
||||
// Same seam for OpenRouter: `gbrain config set openrouter_api_key X` (or
|
||||
// config.json) must reach the openrouter recipe's OPENROUTER_API_KEY.
|
||||
// process.env still wins via the later spread.
|
||||
if (c.openrouter_api_key) envFromConfig.OPENROUTER_API_KEY = c.openrouter_api_key;
|
||||
|
||||
// v0.32 codex finding #4+#5 fix: thread local-server _BASE_URL env vars
|
||||
// into base_urls so the gateway hits the user's configured port. Without
|
||||
|
||||
+157
-30
@@ -377,7 +377,8 @@ export function applyOpenAICompatConfig(
|
||||
cfg: AIGatewayConfig,
|
||||
): { baseURL: string; fetch?: typeof fetch } {
|
||||
if (recipe.resolveOpenAICompatConfig) {
|
||||
return recipe.resolveOpenAICompatConfig(cfg.env);
|
||||
const resolved = recipe.resolveOpenAICompatConfig(cfg.env);
|
||||
return { ...resolved, fetch: resolved.fetch ?? recipe.compat?.fetch };
|
||||
}
|
||||
const baseURL = cfg.base_urls?.[recipe.id] ?? recipe.base_url_default;
|
||||
if (!baseURL) {
|
||||
@@ -386,7 +387,7 @@ export function applyOpenAICompatConfig(
|
||||
recipe.setup_hint,
|
||||
);
|
||||
}
|
||||
return { baseURL };
|
||||
return { baseURL, fetch: recipe.compat?.fetch };
|
||||
}
|
||||
|
||||
/**
|
||||
@@ -2383,6 +2384,58 @@ export interface ChatToolDef {
|
||||
* production subagent jobs) throws "messages do not match the ModelMessage[]
|
||||
* schema" the moment the model calls a tool. Surfaced by the SkillOpt eval.
|
||||
*/
|
||||
/**
|
||||
* Default per-call max output tokens. Thinking-by-default Claude 5 models
|
||||
* (`anthropic:claude-*-5`) burn a large chunk of the budget on internal
|
||||
* reasoning before emitting any text, so a 4096 default leaves them with empty
|
||||
* final text on the subagent tool loop. Give those models headroom; providers
|
||||
* bill actual tokens, not the cap, so it is free for the models that don't use
|
||||
* it. Everything else keeps 4096 on purpose: raising the default blanket-wide
|
||||
* would exceed some openai-compat providers' hard max-output caps (DeepSeek
|
||||
* 8192, gpt-4o 16384) and 400 on them — a regression for exactly the
|
||||
* non-Anthropic subagent users the gateway loop exists to serve.
|
||||
*/
|
||||
const DEFAULT_MAX_OUTPUT_TOKENS = 4096;
|
||||
const THINKING_MODEL_MAX_OUTPUT_TOKENS = 32000;
|
||||
const THINKING_BY_DEFAULT_MODEL_RE = /^anthropic[:/]claude-[a-z0-9]+-5(?:[.-]|$)/i;
|
||||
function defaultMaxOutputTokens(modelStr: string | undefined): number {
|
||||
return modelStr && THINKING_BY_DEFAULT_MODEL_RE.test(modelStr)
|
||||
? THINKING_MODEL_MAX_OUTPUT_TOKENS
|
||||
: DEFAULT_MAX_OUTPUT_TOKENS;
|
||||
}
|
||||
|
||||
/**
|
||||
* Deep-serialize a tool output into a plain JSON value for the AI SDK v6
|
||||
* ModelMessage schema. node-postgres returns `timestamptz` columns as JS
|
||||
* `Date` instances, and AI SDK v6's `JSONValue` schema rejects a raw Date,
|
||||
* throwing "Invalid prompt ... ModelMessage[] schema" the moment a
|
||||
* timestamp-bearing tool result (e.g. `brain_get_page`, `brain_list_pages`)
|
||||
* is fed back — dead-lettering the whole multi-tool loop. The JSON round-trip
|
||||
* runs `Date.prototype.toJSON` (ISO string) recursively and drops `undefined`.
|
||||
* This is a serialization fix at the SDK boundary, NOT a `::jsonb` DB cast —
|
||||
* it never touches Postgres. (BigInt / circular outputs still throw in
|
||||
* JSON.stringify; those aren't LLM-serializable and are out of scope.)
|
||||
*/
|
||||
function toJsonSafe(value: unknown): unknown {
|
||||
try {
|
||||
return JSON.parse(JSON.stringify(value ?? null));
|
||||
} catch {
|
||||
// BigInt / circular output isn't LLM-serializable; degrade to a string
|
||||
// rather than throwing and dead-lettering the whole tool loop.
|
||||
return safeStringify(value);
|
||||
}
|
||||
}
|
||||
|
||||
/** Stringify that never throws (bigint/circular fall back to String()). */
|
||||
function safeStringify(value: unknown): string {
|
||||
if (typeof value === 'string') return value;
|
||||
try {
|
||||
return JSON.stringify(value ?? null);
|
||||
} catch {
|
||||
return String(value);
|
||||
}
|
||||
}
|
||||
|
||||
export function toModelMessages(messages: ChatMessage[]): unknown[] {
|
||||
return messages.map((m) => {
|
||||
if (typeof m.content === 'string') return { role: m.role, content: m.content };
|
||||
@@ -2398,24 +2451,86 @@ export function toModelMessages(messages: ChatMessage[]): unknown[] {
|
||||
toolCallId: b.toolCallId,
|
||||
toolName: b.toolName,
|
||||
output: b.isError
|
||||
? { type: 'error-text' as const, value: typeof b.output === 'string' ? b.output : JSON.stringify(b.output) }
|
||||
? { type: 'error-text' as const, value: safeStringify(b.output) }
|
||||
: (typeof b.output === 'string'
|
||||
? { type: 'text' as const, value: b.output }
|
||||
: { type: 'json' as const, value: (b.output ?? null) as never }),
|
||||
: { type: 'json' as const, value: toJsonSafe(b.output) as never }),
|
||||
})),
|
||||
};
|
||||
}
|
||||
return {
|
||||
role: m.role,
|
||||
content: blocks.map((b) => {
|
||||
if (b.type === 'text') return { type: 'text' as const, text: b.text };
|
||||
if (b.type === 'tool-call') return { type: 'tool-call' as const, toolCallId: b.toolCallId, toolName: b.toolName, input: b.input };
|
||||
return b;
|
||||
}),
|
||||
// Drop text blocks whose `text` isn't a string: reasoning models
|
||||
// (DeepSeek v4, etc.) surface `text: null/undefined` thinking parts that
|
||||
// AI SDK v6's Zod schema rejects, poisoning the whole call. `''` is valid
|
||||
// and kept.
|
||||
content: blocks
|
||||
.filter((b) => b.type !== 'text' || typeof b.text === 'string')
|
||||
.map((b) => {
|
||||
if (b.type === 'text') return { type: 'text' as const, text: b.text };
|
||||
if (b.type === 'tool-call') return { type: 'tool-call' as const, toolCallId: b.toolCallId, toolName: b.toolName, input: b.input };
|
||||
return b;
|
||||
}),
|
||||
};
|
||||
});
|
||||
}
|
||||
|
||||
/**
|
||||
* Last-resort normalization at the `chat()` boundary: back-fill error stubs for
|
||||
* any assistant tool-call that isn't answered by the immediately-following
|
||||
* tool-result turn. The subagent handler already balances its own transcript
|
||||
* (see reconcileGatewayReplay), so this is a no-op there — it exists for the
|
||||
* paths reconcile can't reach: a partially-answered turn, a provider that
|
||||
* duplicates or drops tool-call IDs (local vLLM), or a `finishReason:'length'`
|
||||
* truncation mid-batch. Without it those histories throw
|
||||
* AI_MissingToolResultsError inside `generateText`. No-op on balanced input.
|
||||
*
|
||||
* @internal exported for tests.
|
||||
*/
|
||||
export function repairToolPairing(messages: ChatMessage[]): ChatMessage[] {
|
||||
const out: ChatMessage[] = [];
|
||||
for (let i = 0; i < messages.length; i++) {
|
||||
const m = messages[i];
|
||||
out.push(m);
|
||||
if (typeof m.content === 'string' || m.role !== 'assistant') continue;
|
||||
|
||||
const calls = m.content.filter(
|
||||
(b): b is Extract<ChatBlock, { type: 'tool-call' }> => b.type === 'tool-call',
|
||||
);
|
||||
if (calls.length === 0) continue;
|
||||
|
||||
// v6 only accepts results in the immediately-following message.
|
||||
const next = messages[i + 1];
|
||||
const nextBlocks = next && typeof next.content !== 'string' ? next.content : [];
|
||||
const resolved = new Set(
|
||||
nextBlocks
|
||||
.filter((b): b is Extract<ChatBlock, { type: 'tool-result' }> => b.type === 'tool-result')
|
||||
.map((b) => b.toolCallId),
|
||||
);
|
||||
|
||||
const missing = calls.filter((c) => !resolved.has(c.toolCallId));
|
||||
if (missing.length === 0) continue;
|
||||
|
||||
const stubs: ChatBlock[] = missing.map((c) => ({
|
||||
type: 'tool-result',
|
||||
toolCallId: c.toolCallId,
|
||||
toolName: c.toolName,
|
||||
output: 'tool result unavailable (recovered after interrupted run)',
|
||||
isError: true,
|
||||
}));
|
||||
|
||||
if (resolved.size > 0) {
|
||||
// A tool-result message follows but is incomplete — merge the stubs in.
|
||||
out.push({ role: next!.role, content: [...(nextBlocks as ChatBlock[]), ...stubs] });
|
||||
i++; // the merged message replaces the original; don't emit it twice.
|
||||
} else {
|
||||
// No following tool-result message at all — synthesize one.
|
||||
out.push({ role: 'user', content: stubs });
|
||||
}
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
export interface ChatResult {
|
||||
/** Final text content concatenated from text blocks. */
|
||||
text: string;
|
||||
@@ -2676,6 +2791,22 @@ async function classifyGatewayGuardrail(input: {
|
||||
}
|
||||
}
|
||||
|
||||
export function toAISDKTools(tools: ChatToolDef[] | undefined): Record<string, any> | undefined {
|
||||
if (!tools || tools.length === 0) return undefined;
|
||||
return tools.reduce((acc, t) => {
|
||||
acc[t.name] = {
|
||||
description: t.description,
|
||||
// AI SDK v6 requires a Schema (carrying the schema symbol), not a plain
|
||||
// `{jsonSchema}` object — the bare object makes asSchema() treat it as a
|
||||
// thunk and call schema(), throwing "schema is not a function". Wrap the
|
||||
// raw JSON Schema with the SDK's jsonSchema() helper so tool calls work
|
||||
// through the real toolLoop (skillopt rollouts + subagent jobs).
|
||||
inputSchema: jsonSchema(t.inputSchema as any),
|
||||
};
|
||||
return acc;
|
||||
}, {} as Record<string, any>);
|
||||
}
|
||||
|
||||
export async function chat(opts: ChatOpts): Promise<ChatResult> {
|
||||
const tracker = __budgetStore.getStore() ?? null;
|
||||
const modelStrEarly = opts.model ?? getChatModel();
|
||||
@@ -2697,7 +2828,7 @@ export async function chat(opts: ChatOpts): Promise<ChatResult> {
|
||||
}
|
||||
}
|
||||
const estimatedInputTokens = estimateChatInputTokens(opts);
|
||||
const maxOutputTokens = opts.maxTokens ?? 4096;
|
||||
const maxOutputTokens = opts.maxTokens ?? defaultMaxOutputTokens(modelStrEarly);
|
||||
|
||||
// TX5: reserve BEFORE the provider call. Throws BudgetExhausted on cost,
|
||||
// runtime, or no_pricing (when cap is set). Pre-resolution model id is
|
||||
@@ -2763,21 +2894,7 @@ export async function chat(opts: ChatOpts): Promise<ChatResult> {
|
||||
const supportsCache = recipe.touchpoints.chat?.supports_prompt_cache === true;
|
||||
const useCache = !!opts.cacheSystem && supportsCache;
|
||||
|
||||
// Build messages. Anthropic prompt-cache markers ride on system + last tool
|
||||
// via providerOptions; the AI SDK accepts the system as a string for
|
||||
// generateText, so cache markers go through providerOptions.anthropic.
|
||||
const tools = (opts.tools ?? []).reduce((acc, t) => {
|
||||
acc[t.name] = {
|
||||
description: t.description,
|
||||
// AI SDK v6 requires a Schema (carrying the schema symbol), not a plain
|
||||
// `{jsonSchema}` object — the bare object makes asSchema() treat it as a
|
||||
// thunk and call schema(), throwing "schema is not a function". Wrap the
|
||||
// raw JSON Schema with the SDK's jsonSchema() helper so tool calls work
|
||||
// through the real toolLoop (skillopt rollouts + subagent jobs).
|
||||
inputSchema: jsonSchema(t.inputSchema as any),
|
||||
};
|
||||
return acc;
|
||||
}, {} as Record<string, any>);
|
||||
const tools = toAISDKTools(opts.tools);
|
||||
|
||||
const providerOptions: Record<string, any> = {};
|
||||
if (useCache) {
|
||||
@@ -2804,9 +2921,9 @@ export async function chat(opts: ChatOpts): Promise<ChatResult> {
|
||||
const result = await generateText({
|
||||
model,
|
||||
system: opts.system,
|
||||
messages: toModelMessages(opts.messages) as any,
|
||||
messages: toModelMessages(repairToolPairing(opts.messages)) as any,
|
||||
tools: opts.tools && opts.tools.length > 0 ? tools : undefined,
|
||||
maxOutputTokens: opts.maxTokens ?? 4096,
|
||||
maxOutputTokens: opts.maxTokens ?? defaultMaxOutputTokens(modelStr),
|
||||
// v0.42.20.0 — default a chat timeout (composes with the caller's signal,
|
||||
// shorter wins). Covers native-anthropic (the default provider + facts Haiku).
|
||||
abortSignal: withDefaultTimeout(opts.abortSignal, AI_CHAT_TIMEOUT_MS),
|
||||
@@ -2957,6 +3074,14 @@ export interface ToolLoopOpts {
|
||||
) => Promise<{ gbrainToolUseId: string }>;
|
||||
onToolCallComplete?: (gbrainToolUseId: string, output: unknown) => Promise<void>;
|
||||
onToolCallFailed?: (gbrainToolUseId: string, error: string) => Promise<void>;
|
||||
/**
|
||||
* Persist the tool-result user turn that closes each tool round, BEFORE it is
|
||||
* appended to the in-memory history. Without this the loop only kept the
|
||||
* tool-result turn in memory, so a resumed job reloaded assistant tool-calls
|
||||
* with no matching results and non-Anthropic providers rejected the
|
||||
* unbalanced history (AI_MissingToolResultsError). Fires per completed round.
|
||||
*/
|
||||
onToolResultTurn?: (turnIdx: number, messageIdx: number, blocks: ChatBlock[]) => Promise<void>;
|
||||
|
||||
/** Optional per-call heartbeat for observability. */
|
||||
onHeartbeat?: (event: string, data: Record<string, unknown>) => void;
|
||||
@@ -2991,7 +3116,7 @@ export interface ToolLoopResult {
|
||||
*/
|
||||
export async function toolLoop(opts: ToolLoopOpts): Promise<ToolLoopResult> {
|
||||
const maxTurns = opts.maxTurns ?? 20;
|
||||
const maxTokens = opts.maxTokens ?? 4096;
|
||||
const maxTokens = opts.maxTokens ?? defaultMaxOutputTokens(opts.model ?? getChatModel());
|
||||
const handlers = opts.toolHandlers;
|
||||
const totalUsage: ChatResult['usage'] = {
|
||||
input_tokens: 0,
|
||||
@@ -3180,9 +3305,11 @@ export async function toolLoop(opts: ToolLoopOpts): Promise<ToolLoopResult> {
|
||||
|
||||
if (stopReason === 'aborted') break;
|
||||
|
||||
// Feed all tool results back as a single user message.
|
||||
// Persist + feed all tool results back as a single user message. The
|
||||
// persist-before-push mirrors onAssistantTurn's write-ordering: a crash
|
||||
// after this leaves a balanced transcript for the next resume.
|
||||
const userMessageIdx = messageIdx++;
|
||||
void userMessageIdx;
|
||||
await opts.onToolResultTurn?.(turnIdx, userMessageIdx, toolResultBlocks);
|
||||
messages.push({ role: 'user', content: toolResultBlocks });
|
||||
|
||||
turnIdx++;
|
||||
|
||||
@@ -1,5 +1,64 @@
|
||||
import type { Recipe } from '../types.ts';
|
||||
|
||||
/**
|
||||
* `deepseek-reasoner` returns its answer in a separate `reasoning_content`
|
||||
* field and leaves `content` empty/whitespace when the whole response was
|
||||
* reasoning. The AI SDK's openai-compatible adapter reads only `content`, so
|
||||
* the model appears to answer with nothing. This transport shim promotes
|
||||
* `reasoning_content` into `content` when `content` is empty, before the
|
||||
* adapter parses the body. Fail-open: any error returns the original response.
|
||||
* Non-streaming JSON chat completions only.
|
||||
*
|
||||
* @internal exported for tests.
|
||||
*/
|
||||
// Cast through `unknown` because TS's `typeof fetch` includes a `preconnect`
|
||||
// member the arrow function does not implement (matches azure-openai.ts).
|
||||
export const deepseekReasoningContentCompatFetch = (async (
|
||||
input: RequestInfo | URL,
|
||||
init?: RequestInit,
|
||||
): Promise<Response> => {
|
||||
const res = await fetch(input as any, init as any);
|
||||
try {
|
||||
if (!res.ok) return res;
|
||||
const ctype = res.headers.get('content-type') ?? '';
|
||||
if (!ctype.includes('application/json')) return res;
|
||||
const json = await res.clone().json();
|
||||
const choices = Array.isArray(json?.choices) ? json.choices : [];
|
||||
let modified = false;
|
||||
for (const choice of choices) {
|
||||
const msg = choice?.message;
|
||||
if (!msg) continue;
|
||||
// A tool-call turn legitimately carries content:null — the answer is the
|
||||
// tool call, not text. NEVER promote reasoning_content there: DeepSeek's
|
||||
// chain-of-thought must not be fed back to the model (it would be
|
||||
// persisted as assistant text and replayed every subsequent turn,
|
||||
// contaminating context and inflating tokens). Only promote on a terminal
|
||||
// text turn whose content is empty.
|
||||
const hasToolCalls = Array.isArray(msg.tool_calls) && msg.tool_calls.length > 0;
|
||||
const content = msg.content;
|
||||
const reasoning = msg.reasoning_content;
|
||||
const contentEmpty = content == null || (typeof content === 'string' && content.trim() === '');
|
||||
if (!hasToolCalls && contentEmpty && typeof reasoning === 'string' && reasoning.trim() !== '') {
|
||||
msg.content = reasoning;
|
||||
modified = true;
|
||||
}
|
||||
}
|
||||
if (!modified) return res;
|
||||
// Rebuild with a fresh header set: the body length changed, so the
|
||||
// upstream content-length / content-encoding would now be wrong.
|
||||
const headers = new Headers(res.headers);
|
||||
headers.delete('content-length');
|
||||
headers.delete('content-encoding');
|
||||
return new Response(JSON.stringify(json), {
|
||||
status: res.status,
|
||||
statusText: res.statusText,
|
||||
headers,
|
||||
});
|
||||
} catch {
|
||||
return res;
|
||||
}
|
||||
}) as unknown as typeof fetch;
|
||||
|
||||
/**
|
||||
* DeepSeek exposes an OpenAI-compatible /v1/chat/completions endpoint.
|
||||
* Useful as the second hop in a refusal-fallback chain and for cheap-
|
||||
@@ -29,4 +88,5 @@ export const deepseek: Recipe = {
|
||||
},
|
||||
},
|
||||
setup_hint: 'Get an API key at https://platform.deepseek.com/api_keys, then `export DEEPSEEK_API_KEY=...`',
|
||||
compat: { fetch: deepseekReasoningContentCompatFetch },
|
||||
};
|
||||
|
||||
@@ -326,6 +326,18 @@ export interface Recipe {
|
||||
baseURL: string;
|
||||
fetch?: typeof fetch;
|
||||
};
|
||||
/**
|
||||
* Optional inbound-response rewriter for openai-compatible recipes whose wire
|
||||
* shape needs normalizing before the AI SDK adapter parses it. `fetch` wraps
|
||||
* the transport and MUST be fail-open (return the original response on any
|
||||
* error). Used by DeepSeek to promote `reasoning_content` into `content` when
|
||||
* the reasoner returns an empty `content` (the adapter reads only `content`).
|
||||
* Applied by `applyOpenAICompatConfig`; a `resolveOpenAICompatConfig`-provided
|
||||
* fetch takes precedence when both are present.
|
||||
*/
|
||||
compat?: {
|
||||
fetch?: typeof fetch;
|
||||
};
|
||||
/**
|
||||
* v0.32 (D13=A): optional runtime readiness check for local-server
|
||||
* recipes (ollama, llama-server, future lmstudio-recipe). Returns
|
||||
|
||||
@@ -41,6 +41,13 @@ export interface GBrainConfig {
|
||||
* merge → buildGatewayConfig env dict → recipe reads ZEROENTROPY_API_KEY.
|
||||
*/
|
||||
zeroentropy_api_key?: string;
|
||||
/**
|
||||
* OpenRouter API key. File-plane slot so `gbrain config set
|
||||
* openrouter_api_key X` (or config.json) reaches the openrouter recipe:
|
||||
* file plane → loadConfig env merge → buildGatewayConfig env dict → recipe
|
||||
* reads OPENROUTER_API_KEY.
|
||||
*/
|
||||
openrouter_api_key?: string;
|
||||
/** AI gateway config (v0.14+). v0.36+ default: "zeroentropyai:zembed-1" / 1280 / "anthropic:claude-haiku-4-5-20251001". */
|
||||
embedding_model?: string;
|
||||
embedding_dimensions?: number;
|
||||
@@ -526,6 +533,7 @@ export function loadConfig(): GBrainConfig | null {
|
||||
...(process.env.OPENAI_API_KEY ? { openai_api_key: process.env.OPENAI_API_KEY } : {}),
|
||||
...(process.env.ANTHROPIC_API_KEY ? { anthropic_api_key: process.env.ANTHROPIC_API_KEY } : {}),
|
||||
...(process.env.ZEROENTROPY_API_KEY ? { zeroentropy_api_key: process.env.ZEROENTROPY_API_KEY } : {}),
|
||||
...(process.env.OPENROUTER_API_KEY ? { openrouter_api_key: process.env.OPENROUTER_API_KEY } : {}),
|
||||
...(process.env.GBRAIN_EMBEDDING_MODEL ? { embedding_model: process.env.GBRAIN_EMBEDDING_MODEL } : {}),
|
||||
...(process.env.GBRAIN_EMBEDDING_DIMENSIONS ? { embedding_dimensions: parseInt(process.env.GBRAIN_EMBEDDING_DIMENSIONS, 10) } : {}),
|
||||
...(process.env.GBRAIN_EXPANSION_MODEL ? { expansion_model: process.env.GBRAIN_EXPANSION_MODEL } : {}),
|
||||
@@ -815,6 +823,8 @@ export const KNOWN_CONFIG_KEYS: readonly string[] = [
|
||||
'database_path',
|
||||
'openai_api_key',
|
||||
'anthropic_api_key',
|
||||
'zeroentropy_api_key',
|
||||
'openrouter_api_key',
|
||||
'embedding_model',
|
||||
'embedding_dimensions',
|
||||
'embedding_disabled',
|
||||
@@ -836,6 +846,11 @@ export const KNOWN_CONFIG_KEYS: readonly string[] = [
|
||||
'sync',
|
||||
'sync.repo_path',
|
||||
'sync.last_commit',
|
||||
// Gateway-native subagent loop toggle (routes subagent jobs through the
|
||||
// provider-agnostic gateway.toolLoop for non-Anthropic providers). The
|
||||
// subagent handler's error message tells users to `config set` this, so it
|
||||
// must be a known key or `config set` rejects it without --force.
|
||||
'agent.use_gateway_loop',
|
||||
// DB-plane (v0.32.3 search modes + related)
|
||||
'search.mode',
|
||||
'search.cache.enabled',
|
||||
@@ -921,6 +936,16 @@ export const KNOWN_CONFIG_KEYS: readonly string[] = [
|
||||
// operator had to discover these by reading source. Registered so `config
|
||||
// set` accepts them directly. See docs/operations/spend-controls.md.
|
||||
'spend.posture',
|
||||
// Life Chronicle (v0.42.56.0, #2390). The release notes' enable command is
|
||||
// `gbrain config set auto_chronicle true`, but the key was never registered
|
||||
// — so the documented command failed with "Unknown config key" and the
|
||||
// operator had to discover --force by reading source. Same class as the
|
||||
// spend-controls registration above.
|
||||
'auto_chronicle',
|
||||
// Takes bootstrap (v0.41.18.0, A12). The onboard remediation's two-gate
|
||||
// consent reads this key, and enabling it is the documented path to
|
||||
// `gbrain takes extract --from-pages` — same unregistered-key class.
|
||||
'takes.bootstrap_enabled',
|
||||
'sync.cost_gate_min_usd',
|
||||
'sync.federated_v2',
|
||||
'embed.backfill_cooldown_min',
|
||||
@@ -943,6 +968,7 @@ export const KNOWN_CONFIG_KEY_PREFIXES: readonly string[] = [
|
||||
'content_sanity.', // v0.41 content-sanity tunables
|
||||
'mcp.', // mcp.publish_skills, mcp.skills_dir (PR1 skill catalog)
|
||||
'autopilot.', // autopilot.nightly_quality_probe.*, autopilot.auto_drain.* (#1685)
|
||||
'chronicle.', // chronicle.tz + future Life Chronicle knobs (#2390)
|
||||
'self_upgrade.', // v0.42 self-upgrade (mode, quiet_hours, state)
|
||||
];
|
||||
|
||||
|
||||
+109
-34
@@ -9,24 +9,27 @@
|
||||
// 4. Write each atom via engine.putPage(slug, page, {sourceId})
|
||||
// with sourceId threaded so federated brains route correctly.
|
||||
//
|
||||
// Idempotency (D1 from /plan-eng-review):
|
||||
// Each atom carries frontmatter.source_hash (16-char sha256 prefix).
|
||||
// Before processing a transcript/page, query "any atom with this
|
||||
// source_hash exists in this source?". If yes, skip. Closes both:
|
||||
// - PR #1414's primary concern (page-side re-extraction)
|
||||
// - Pre-existing v0.41.2.0 transcript-side date-stamp duplicate bug
|
||||
// (atom slugs are `atoms/YYYY-MM-DD/<title>`, so re-discovered
|
||||
// transcripts on day N+1 used to write second atoms; now skipped).
|
||||
// Idempotency (per-atom, via deterministic slug):
|
||||
// Each atom's slug is `atoms/<source-date>/<stem>-<title-hash>` — built from
|
||||
// the SOURCE date (the transcript's own date / the page slug), NOT the run
|
||||
// date, plus a 6-char hash of the title. Re-extracting the same atom resolves
|
||||
// to the SAME slug, so engine.putPage upserts in place instead of minting a
|
||||
// duplicate. This closes three bugs in one scheme:
|
||||
// - PR #1414's page-side re-extraction.
|
||||
// - The cross-day transcript duplicate: append-only transcripts grow daily,
|
||||
// so a run-date prefix (`atoms/<today>/…`) used to re-mint the same atom
|
||||
// under a new date every day. A source-date prefix is stable, so it now
|
||||
// upserts.
|
||||
// - The "trailing-dash twin": the stem routes through slugifySegment (the
|
||||
// FS-import normalizer) and re-strips a trailing dash after the 60-char
|
||||
// truncation, so the two write paths can no longer disagree on `…would`
|
||||
// vs `…would-` and persist the same atom twice.
|
||||
//
|
||||
// Known limitation (D9 #2 — documented, not blocking):
|
||||
// If extraction writes atom 1 of 3 then atom 2 throws, source_hash
|
||||
// filter sees atom 1 exists and skips on next discovery. Atoms 2+3
|
||||
// stay missing until content_hash changes. Acceptable for v0.41.2.1:
|
||||
// - Haiku call failure is rare; network/budget failures rarer.
|
||||
// - Content edits trigger natural re-extract via new content_hash.
|
||||
// - The original incident (duplicate atoms) is fully closed.
|
||||
// Per-atom idempotency via deterministic slug is v0.42+ TODO
|
||||
// (see TODOS.md).
|
||||
// The source_hash batch check (atomsExistingForHashes) is retained ONLY as a
|
||||
// cost fast-path — it skips re-running Haiku on a transcript whose whole-file
|
||||
// hash is unchanged. On append-only sources that hash changes daily so the
|
||||
// fast-path won't skip, but the deterministic slug makes the re-run upsert
|
||||
// rather than duplicate, so correctness no longer depends on it.
|
||||
//
|
||||
// Config:
|
||||
// Reads dream.synthesize.session_corpus_dir + meeting_transcripts_dir
|
||||
@@ -51,6 +54,8 @@ import type { ProgressReporter } from '../progress.ts';
|
||||
import { chat as gatewayChat } from '../ai/gateway.ts';
|
||||
import { writeReceipt } from '../extract/receipt-writer.ts';
|
||||
import { upsertExtractRollup } from '../extract/rollup-writer.ts';
|
||||
import { createHash } from 'crypto';
|
||||
import { slugifySegment } from '../sync.ts';
|
||||
|
||||
const DEFAULT_BUDGET_USD = 0.3;
|
||||
|
||||
@@ -61,16 +66,56 @@ const ATOM_TYPES = [
|
||||
'critique', 'collection',
|
||||
] as const;
|
||||
|
||||
// v0.41.2.1 (D2): brain-page discovery constants. Hardcoded for now;
|
||||
// future pack-aware refactor is a one-line change to pull from the
|
||||
// active pack manifest (symmetric with the existing
|
||||
// src/core/facts/eligibility.ts:49 TODO).
|
||||
const EXTRACTABLE_PAGE_TYPES = [
|
||||
// v0.41.2.1 (D2): brain-page discovery constants.
|
||||
//
|
||||
// Legacy floor: the pre-pack hardcoded atom-extraction types. Retained as a
|
||||
// back-compat union member so a gbrain-base brain never loses an extraction
|
||||
// target when we begin honoring the pack manifest's `extractable` flags.
|
||||
const LEGACY_EXTRACTABLE_TYPES = [
|
||||
'meeting', 'source', 'article', 'video', 'book', 'original',
|
||||
] as const;
|
||||
|
||||
// Synthesis outputs are never extraction inputs: extracting atoms from atoms or
|
||||
// concepts would loop (concepts are synthesized FROM atoms). Mirrors
|
||||
// facts/eligibility.ts, which likewise excludes `concept` despite its
|
||||
// extractable:true flag being a documented forward-compat marker.
|
||||
const SYNTHESIS_OUTPUT_TYPES = new Set<string>(['atom', 'concept']);
|
||||
|
||||
const PAGE_DISCOVERY_BUDGET = 50;
|
||||
const MIN_PAGE_CHARS_FOR_EXTRACTION = 500;
|
||||
|
||||
/**
|
||||
* Pure allowlist policy: the legacy floor UNION the pack's `extractable: true`
|
||||
* types, MINUS synthesis outputs. Exported for unit tests; keep I/O-free.
|
||||
*/
|
||||
export function unionExtractableTypes(packExtractable: Iterable<string>): string[] {
|
||||
const types = new Set<string>(LEGACY_EXTRACTABLE_TYPES);
|
||||
for (const t of packExtractable) types.add(t);
|
||||
for (const t of SYNTHESIS_OUTPUT_TYPES) types.delete(t);
|
||||
return [...types];
|
||||
}
|
||||
|
||||
/**
|
||||
* Resolve the atom-extraction type allowlist from the active schema pack.
|
||||
* Closes the D2 TODO of honoring the pack manifest (so a type declared
|
||||
* extractable — e.g. `note` — actually extracts) while preserving behavior for
|
||||
* gbrain-base via the legacy-floor union. Fail-soft: any pack-load error falls
|
||||
* back to the legacy floor.
|
||||
*/
|
||||
async function resolveExtractableTypes(): Promise<string[]> {
|
||||
let packExtractable: Iterable<string> = [];
|
||||
try {
|
||||
const { loadConfig } = await import('../config.ts');
|
||||
const { loadActivePack } = await import('../schema-pack/load-active.ts');
|
||||
const { extractableTypesFromPack } = await import('../schema-pack/extractable.ts');
|
||||
const resolved = await loadActivePack({ cfg: loadConfig(), remote: false });
|
||||
packExtractable = extractableTypesFromPack(resolved.manifest);
|
||||
} catch {
|
||||
// Pack unavailable (test seams, bootstrap) — legacy floor only.
|
||||
}
|
||||
return unionExtractableTypes(packExtractable);
|
||||
}
|
||||
|
||||
export interface ExtractAtomsOpts {
|
||||
brainDir?: string;
|
||||
sourceId?: string;
|
||||
@@ -195,7 +240,7 @@ export async function discoverExtractablePages(
|
||||
`;
|
||||
const params: unknown[] = [
|
||||
sourceId,
|
||||
EXTRACTABLE_PAGE_TYPES as unknown as string[],
|
||||
await resolveExtractableTypes(),
|
||||
MIN_PAGE_CHARS_FOR_EXTRACTION,
|
||||
PAGE_DISCOVERY_BUDGET,
|
||||
];
|
||||
@@ -272,9 +317,10 @@ export async function countExtractAtomsBacklog(
|
||||
AND atom.frontmatter->>'source_hash' = substring(p.content_hash from 1 for 16)
|
||||
AND atom.deleted_at IS NULL
|
||||
)`;
|
||||
const extractableTypes = await resolveExtractableTypes();
|
||||
const params = scoped
|
||||
? [sourceId, EXTRACTABLE_PAGE_TYPES as unknown as string[], MIN_PAGE_CHARS_FOR_EXTRACTION]
|
||||
: [EXTRACTABLE_PAGE_TYPES as unknown as string[], MIN_PAGE_CHARS_FOR_EXTRACTION];
|
||||
? [sourceId, extractableTypes, MIN_PAGE_CHARS_FOR_EXTRACTION]
|
||||
: [extractableTypes, MIN_PAGE_CHARS_FOR_EXTRACTION];
|
||||
const rows = await engine.executeRaw<{ cnt: string | number }>(sql, params);
|
||||
return Number(rows[0]?.cnt ?? 0);
|
||||
} catch (err) {
|
||||
@@ -517,7 +563,8 @@ export async function runPhaseExtractAtoms(
|
||||
|
||||
if (!opts.dryRun) {
|
||||
for (const atom of atoms) {
|
||||
const slug = `atoms/${todayDate()}/${slugify(atom.title)}`;
|
||||
const srcRef = item.kind === 'transcript' ? item.filePath : item.slug;
|
||||
const slug = atomSlug(atom.title, srcRef);
|
||||
const originFrontmatter =
|
||||
item.kind === 'transcript'
|
||||
? { source_path: item.filePath }
|
||||
@@ -691,12 +738,40 @@ function todayDate(): string {
|
||||
return new Date().toISOString().slice(0, 10);
|
||||
}
|
||||
|
||||
function slugify(s: string): string {
|
||||
return s
|
||||
.toLowerCase()
|
||||
.replace(/[^a-z0-9\s-]/g, '')
|
||||
.trim()
|
||||
.replace(/\s+/g, '-')
|
||||
.replace(/-+/g, '-')
|
||||
.slice(0, 60);
|
||||
/**
|
||||
* Canonical slug stem for an atom title. Routes through slugifySegment (the
|
||||
* same normalizer the FS-import path uses) and RE-STRIPS a trailing dash after
|
||||
* the 60-char truncation — the cut can land on a hyphen and re-introduce one.
|
||||
* Two writers disagreeing on that trailing dash (`…would` vs `…would-`) was the
|
||||
* "trailing-dash twin" duplicate bug.
|
||||
*/
|
||||
function atomSlugStem(title: string): string {
|
||||
return slugifySegment(title).slice(0, 60).replace(/-+$/g, '') || 'untitled';
|
||||
}
|
||||
|
||||
/**
|
||||
* Pull a YYYY-MM-DD date from a source reference — a transcript file path like
|
||||
* `…/2026-06-11-telegram.md`, or a dated page slug. Checks the basename first
|
||||
* to avoid matching a date in a parent directory. Falls back to the run date
|
||||
* only when the source carries no date, so dated sources are fully deterministic.
|
||||
*/
|
||||
function sourceDate(ref: string): string {
|
||||
const base = ref.split('/').pop() ?? ref;
|
||||
const m = base.match(/(\d{4}-\d{2}-\d{2})/) ?? ref.match(/(\d{4}-\d{2}-\d{2})/);
|
||||
return m ? m[1] : todayDate();
|
||||
}
|
||||
|
||||
/**
|
||||
* Deterministic per-atom slug: `atoms/<source-date>/<stem>-<title-hash>`.
|
||||
* - Date comes from the SOURCE, not the run date, so re-extracting an
|
||||
* append-only transcript on a later day yields the SAME slug → putPage
|
||||
* upserts instead of minting a cross-day duplicate.
|
||||
* - The 6-char title hash keeps two distinct atoms whose titles share the
|
||||
* first 60 chars on separate slugs, so a deterministic slug never silently
|
||||
* clobbers a *different* atom. Hash is over the title only (not body) so an
|
||||
* LLM rewording the body on re-extraction still upserts rather than dupes.
|
||||
*/
|
||||
function atomSlug(title: string, srcRef: string): string {
|
||||
const hash = createHash('sha256').update(title).digest('hex').slice(0, 6);
|
||||
return `atoms/${sourceDate(srcRef)}/${atomSlugStem(title)}-${hash}`;
|
||||
}
|
||||
|
||||
@@ -46,6 +46,13 @@ export interface ExtractTakesFromPagesOpts {
|
||||
sourceIdFilter?: string;
|
||||
/** Max pages to classify per run (caps cost). Default 50. */
|
||||
maxPages?: number;
|
||||
/**
|
||||
* Also rescan pages that already hold takes (refresh semantics).
|
||||
* Default false: bootstrap runs skip covered pages, so repeated runs
|
||||
* PROGRESS through a corpus larger than one run's cap instead of
|
||||
* rescanning the same most-recently-updated slice forever.
|
||||
*/
|
||||
includeCovered?: boolean;
|
||||
/** Owner identifier for the inserted takes. Default 'system'. */
|
||||
holder?: string;
|
||||
/** Model override; defaults to facts.extraction_model. */
|
||||
@@ -132,12 +139,21 @@ export async function extractTakesFromPages(
|
||||
// Fetch eligible pages. Order by updated_at DESC so recently-edited
|
||||
// pages get bootstrapped first.
|
||||
const typesList = ALLOWED_PAGE_TYPES.map((t) => `'${t}'`).join(', ');
|
||||
// Bootstrap progression: skip pages that already hold takes (opt out via
|
||||
// includeCovered). Without this, the updated_at-DESC + LIMIT selection made
|
||||
// every re-run rescan the same most-recent slice — a corpus larger than one
|
||||
// run's cap could never be fully bootstrapped (and each rescan re-spent LLM
|
||||
// budget on covered pages for upsert-identical rows).
|
||||
const coveredFilter = opts.includeCovered
|
||||
? ''
|
||||
: `AND NOT EXISTS (SELECT 1 FROM takes t WHERE t.page_id = pages.id)`;
|
||||
const pages = await engine.executeRaw<PageRow>(
|
||||
`SELECT id, slug, source_id, type, compiled_truth, updated_at
|
||||
FROM pages
|
||||
WHERE type IN (${typesList})
|
||||
AND deleted_at IS NULL
|
||||
AND length(COALESCE(compiled_truth, '')) > 200
|
||||
${coveredFilter}
|
||||
${sourceFilter}
|
||||
ORDER BY updated_at DESC
|
||||
LIMIT ${maxPages}`,
|
||||
|
||||
+29
-7
@@ -115,6 +115,17 @@ async function extractFencedChunks(
|
||||
startChunkIndex: number,
|
||||
): Promise<ChunkInput[]> {
|
||||
const out: ChunkInput[] = [];
|
||||
// Fast path: most pages (prose, tables, converted docs) contain no code
|
||||
// fence at all, so there is nothing for this function to extract. marked's
|
||||
// lexer still allocates transient memory proportional to page size on every
|
||||
// call — a ~2MB table-heavy page spikes ~110MB of heap just to produce zero
|
||||
// fenced chunks. During bulk import those per-page spikes stack on top of
|
||||
// accumulated chunk/embedding memory and can OOM the worker, and the
|
||||
// try/catch below cannot rescue an OOM (it is process death, not a throw).
|
||||
// Skip the lexer entirely when no fence marker (``` or ~~~) is present.
|
||||
// The `\r` in the line-start class mirrors marked's own `\r\n|\r → \n`
|
||||
// normalization, so CR/CRLF-only documents don't lose a real fence.
|
||||
if (!/(^|[\r\n])[ \t]{0,3}(```|~~~)/.test(markdown)) return out;
|
||||
let tokens: ReturnType<typeof marked.lexer>;
|
||||
try {
|
||||
tokens = marked.lexer(markdown);
|
||||
@@ -1304,6 +1315,8 @@ const NEEDS_DECODE = new Set(['.heic', '.heif', '.avif']);
|
||||
export interface ImportTransactionSpec {
|
||||
slug: string;
|
||||
hadExisting: boolean;
|
||||
/** Source containing the page, chunks, file row, and type-specific writes. */
|
||||
sourceId?: string;
|
||||
page: PageInput;
|
||||
/** When undefined, no chunk write happens. When [], deletes any prior chunks. */
|
||||
chunks?: ChunkInput[];
|
||||
@@ -1317,23 +1330,26 @@ export async function withImportTransaction(
|
||||
engine: BrainEngine,
|
||||
spec: ImportTransactionSpec,
|
||||
): Promise<void> {
|
||||
const sourceId = spec.sourceId ?? 'default';
|
||||
const txOpts = spec.sourceId ? { sourceId: spec.sourceId } : undefined;
|
||||
await engine.transaction(async (tx) => {
|
||||
if (spec.hadExisting) await tx.createVersion(spec.slug);
|
||||
await tx.putPage(spec.slug, spec.page);
|
||||
if (spec.hadExisting) await tx.createVersion(spec.slug, txOpts);
|
||||
await tx.putPage(spec.slug, spec.page, txOpts);
|
||||
if (spec.file) {
|
||||
// page_id resolution after putPage so the new row's id is available.
|
||||
const stored = await tx.getPage(spec.slug);
|
||||
const stored = await tx.getPage(spec.slug, txOpts);
|
||||
await tx.upsertFile({
|
||||
...spec.file,
|
||||
source_id: sourceId,
|
||||
page_slug: spec.slug,
|
||||
page_id: stored?.id ?? null,
|
||||
});
|
||||
}
|
||||
if (spec.chunks !== undefined) {
|
||||
if (spec.chunks.length > 0) {
|
||||
await tx.upsertChunks(spec.slug, spec.chunks);
|
||||
await tx.upsertChunks(spec.slug, spec.chunks, txOpts);
|
||||
} else {
|
||||
await tx.deleteChunks(spec.slug);
|
||||
await tx.deleteChunks(spec.slug, txOpts);
|
||||
}
|
||||
}
|
||||
if (spec.after) await spec.after(tx);
|
||||
@@ -1563,10 +1579,14 @@ export async function importImageFile(
|
||||
// and slugifyPath would already preserve it). Recompute with the file
|
||||
// extension preserved so the page slug is stable + collision-free.
|
||||
const imageSlug = relativePath.replace(/[\\\/]/g, '/').toLowerCase();
|
||||
const sourceOpts = opts.sourceId ? { sourceId: opts.sourceId } : undefined;
|
||||
const linkOpts = opts.sourceId
|
||||
? { fromSourceId: opts.sourceId, toSourceId: opts.sourceId, originSourceId: opts.sourceId }
|
||||
: undefined;
|
||||
const buf = readFileSync(filePath);
|
||||
const hash = createHash('sha256').update(buf).digest('hex');
|
||||
|
||||
const existing = await engine.getPage(imageSlug);
|
||||
const existing = await engine.getPage(imageSlug, sourceOpts);
|
||||
if (existing?.content_hash === hash) {
|
||||
return { slug: imageSlug, status: 'skipped', chunks: 0 };
|
||||
}
|
||||
@@ -1642,6 +1662,7 @@ export async function importImageFile(
|
||||
await withImportTransaction(engine, {
|
||||
slug: imageSlug,
|
||||
hadExisting: !!existing,
|
||||
sourceId: opts.sourceId,
|
||||
page: {
|
||||
type: 'image',
|
||||
page_kind: 'image',
|
||||
@@ -1659,13 +1680,14 @@ export async function importImageFile(
|
||||
// throws when the target doesn't exist; we silently skip for now and
|
||||
// let `gbrain reconcile-links` pick up later additions.
|
||||
for (const candidate of imageOfCandidates(imageSlug)) {
|
||||
const sibling = await tx.getPage(candidate);
|
||||
const sibling = await tx.getPage(candidate, sourceOpts);
|
||||
if (sibling) {
|
||||
try {
|
||||
await tx.addLink(
|
||||
imageSlug, candidate,
|
||||
filename,
|
||||
'image_of', 'manual', imageSlug, 'frontmatter',
|
||||
linkOpts,
|
||||
);
|
||||
} catch { /* sibling vanished mid-tx; skip */ }
|
||||
break; // one canonical link per image
|
||||
|
||||
@@ -1156,6 +1156,31 @@ export function parseTimelineEntries(content: string): TimelineCandidate[] {
|
||||
result.push({ date, summary, detail: detailLines.join(' ').trim() });
|
||||
i = j;
|
||||
}
|
||||
|
||||
// Format 3: inline citation — [Source: <source>, YYYY-MM-DD]. The citation
|
||||
// convention gbrain's own quality rules require on every brain write;
|
||||
// until now this parser (the db-source extract + ingest path) could not
|
||||
// see it, so a page whose dates all live in citations scored zero
|
||||
// timeline coverage. Kept in sync with extractTimelineFromContent's
|
||||
// Format 3 (the fs-source path). Lines already captured by the timeline
|
||||
// bullet pass are skipped (a bullet often carries its own citation).
|
||||
const citationRe = /\[Source:\s*([^\]]+?),\s*(\d{4}-\d{2}-\d{2})\s*\]/g;
|
||||
for (const line of lines) {
|
||||
if (TIMELINE_LINE_RE.test(line)) continue;
|
||||
const matches = [...line.matchAll(citationRe)];
|
||||
if (matches.length === 0) continue;
|
||||
const summary = line
|
||||
.replace(/\[Source:[^\]]*\]/g, '')
|
||||
.replace(/^[-*>#\s]+/, '')
|
||||
.replace(/\s+/g, ' ')
|
||||
.trim()
|
||||
.slice(0, 300);
|
||||
if (!summary) continue;
|
||||
for (const m of matches) {
|
||||
if (!isValidDate(m[2])) continue;
|
||||
result.push({ date: m[2], summary, detail: `Source: ${m[1].trim().slice(0, 200)}` });
|
||||
}
|
||||
}
|
||||
return result;
|
||||
}
|
||||
|
||||
|
||||
@@ -777,22 +777,57 @@ async function runSubagentViaGateway(args: GatewayRunArgs): Promise<SubagentResu
|
||||
});
|
||||
}
|
||||
|
||||
// Convert prior Anthropic-shape messages → ChatMessage with ChatBlock content.
|
||||
// v1 rows store Anthropic content blocks ({type:'tool_use'|'tool_result'|...});
|
||||
// we adapt them to ChatBlock shape (type: 'tool-call' | 'tool-result' | 'text').
|
||||
const priorChatMessages: ChatMessage[] = priorMessages.map(m => ({
|
||||
role: m.role as 'user' | 'assistant',
|
||||
content: adaptContentBlocksToChatBlocks(m.content_blocks),
|
||||
}));
|
||||
// Token rollup across the prior transcript (returned as-is on the terminal
|
||||
// early-return path; the loop adds only NEW-turn usage otherwise).
|
||||
const priorTokens = { in: 0, out: 0, cache_read: 0, cache_create: 0 };
|
||||
for (const m of priorMessages) {
|
||||
if (m.tokens_in) priorTokens.in += m.tokens_in;
|
||||
if (m.tokens_out) priorTokens.out += m.tokens_out;
|
||||
if (m.tokens_cache_read) priorTokens.cache_read += m.tokens_cache_read;
|
||||
if (m.tokens_cache_create) priorTokens.cache_create += m.tokens_cache_create;
|
||||
}
|
||||
|
||||
// Reconcile an unbalanced transcript from a prior crashed/resumed run. The
|
||||
// gateway loop persists each assistant turn but (pre-fix) never persisted the
|
||||
// following tool-result user turn, so a resumed job reloads assistant
|
||||
// tool-calls with no matching results — which non-Anthropic (openai-compat)
|
||||
// providers reject with AI_MissingToolResultsError, dead-lettering the job.
|
||||
// reconcileGatewayReplay heals every such dangling turn from settled tool
|
||||
// executions (mirroring the legacy Anthropic path) and reports the terminal
|
||||
// case where the prior run already reached end_turn.
|
||||
const priorToolsV1 = await loadPriorTools(engine, ctx.id);
|
||||
const { chatMessages: priorChatMessages, nextMessageIdx: reconciledNextIdx, terminalText } =
|
||||
await reconcileGatewayReplay({
|
||||
engine,
|
||||
jobId: ctx.id,
|
||||
priorMessages,
|
||||
priorTools: priorToolsV1,
|
||||
toolDefs,
|
||||
signal: ctx.signal,
|
||||
});
|
||||
|
||||
// Terminal early-return (#1151 parity): the prior run already reached
|
||||
// end_turn. Non-Anthropic providers reject a trailing assistant "prefill",
|
||||
// so surface the persisted text and skip the loop entirely.
|
||||
if (terminalText !== null) {
|
||||
return {
|
||||
result: terminalText,
|
||||
turns_count: priorChatMessages.filter(m => m.role === 'assistant').length,
|
||||
stop_reason: 'end_turn',
|
||||
tokens: priorTokens,
|
||||
};
|
||||
}
|
||||
|
||||
// Initial seed message if no prior state.
|
||||
const initialMessages: ChatMessage[] = priorChatMessages.length === 0
|
||||
? [{ role: 'user', content: data.prompt }]
|
||||
: [];
|
||||
|
||||
// Persist seed user message at idx 0 if fresh start.
|
||||
let nextMessageIdx = priorChatMessages.length;
|
||||
if (nextMessageIdx === 0) {
|
||||
// Persist seed user message at idx 0 if fresh start. reconciledNextIdx is
|
||||
// max(known message_idx) + 1 (0 when no prior rows), which keeps the loop's
|
||||
// subsequent writes clear of any healed tool-result turn we just inserted.
|
||||
let nextMessageIdx = reconciledNextIdx;
|
||||
if (priorChatMessages.length === 0) {
|
||||
await persistMessage(engine, ctx.id, {
|
||||
message_idx: 0,
|
||||
role: 'user',
|
||||
@@ -904,6 +939,22 @@ async function runSubagentViaGateway(args: GatewayRunArgs): Promise<SubagentResu
|
||||
[errorMsg, gbrainToolUseId],
|
||||
);
|
||||
},
|
||||
// Persist the tool-result user turn so a resume reloads a balanced
|
||||
// transcript. JSON.stringify inside persistMessage ISO-izes any Date in
|
||||
// tool output, and the value binds through $N::text::jsonb (never
|
||||
// JSON.stringify into a bare ::jsonb).
|
||||
onToolResultTurn: async (_turnIdx, messageIdx, blocks) => {
|
||||
await persistMessage(engine, ctx.id, {
|
||||
message_idx: messageIdx,
|
||||
role: 'user',
|
||||
content_blocks: blocks as unknown as ContentBlock[],
|
||||
tokens_in: null,
|
||||
tokens_out: null,
|
||||
tokens_cache_read: null,
|
||||
tokens_cache_create: null,
|
||||
model: null,
|
||||
});
|
||||
},
|
||||
onHeartbeat: heartbeat,
|
||||
});
|
||||
|
||||
@@ -934,6 +985,158 @@ async function runSubagentViaGateway(args: GatewayRunArgs): Promise<SubagentResu
|
||||
};
|
||||
}
|
||||
|
||||
interface ReconcileArgs {
|
||||
engine: BrainEngine;
|
||||
jobId: number;
|
||||
priorMessages: PersistedMessage[];
|
||||
priorTools: PersistedToolExec[];
|
||||
toolDefs: ToolDef[];
|
||||
signal: AbortSignal;
|
||||
}
|
||||
|
||||
interface ReconcileResult {
|
||||
chatMessages: ChatMessage[];
|
||||
/** max(known/healed message_idx) + 1 — 0 when there is no prior transcript. */
|
||||
nextMessageIdx: number;
|
||||
/** Non-null when the transcript already ended on an assistant text turn. */
|
||||
terminalText: string | null;
|
||||
}
|
||||
|
||||
/**
|
||||
* Heal an unbalanced gateway replay transcript before it reaches the provider.
|
||||
*
|
||||
* The gateway loop persists each assistant turn but historically never
|
||||
* persisted the following tool-result user turn, so a resumed job reloaded
|
||||
* assistant tool-calls with no matching results. Non-Anthropic (openai-compat)
|
||||
* providers reject that with AI_MissingToolResultsError and the job
|
||||
* dead-letters. This walks EVERY prior assistant turn that carries tool-call
|
||||
* blocks (not just the tail — a pre-fix multi-turn job persisted several
|
||||
* consecutive dangling assistant turns) and, if the turn isn't already answered
|
||||
* by the following tool-result user turn, synthesizes that turn from the
|
||||
* settled `subagent_tool_executions` rows with the same semantics as the legacy
|
||||
* Anthropic replay path:
|
||||
* - complete → stored output
|
||||
* - failed → stored error (isError)
|
||||
* - idempotent-pending / missing row → re-dispatch, persist, use output
|
||||
* - non-idempotent-pending → throw (cannot safely re-run)
|
||||
* - tool no longer registered → persist failed + error stub
|
||||
* Each synthesized turn is persisted at `assistant.message_idx + 1` (the free
|
||||
* slot the pre-fix loop skipped) via `ON CONFLICT DO NOTHING`, so the next
|
||||
* resume stays balanced.
|
||||
*/
|
||||
async function reconcileGatewayReplay(args: ReconcileArgs): Promise<ReconcileResult> {
|
||||
const { engine, jobId, priorMessages, priorTools, toolDefs, signal } = args;
|
||||
|
||||
const work = priorMessages.map(m => {
|
||||
const adapted = adaptContentBlocksToChatBlocks(m.content_blocks);
|
||||
return {
|
||||
message_idx: m.message_idx,
|
||||
role: m.role,
|
||||
blocks: typeof adapted === 'string' ? [{ type: 'text', text: adapted } as ChatBlock] : adapted,
|
||||
};
|
||||
});
|
||||
|
||||
// Settled executions, looked up by (assistant message_idx, provider
|
||||
// tool_use_id) with an ordinal-position fallback for legacy rows.
|
||||
const execByKey = new Map<string, PersistedToolExec>();
|
||||
const execByMsg = new Map<number, PersistedToolExec[]>();
|
||||
for (const t of priorTools) {
|
||||
execByKey.set(`${t.message_idx}:${t.tool_use_id}`, t);
|
||||
const arr = execByMsg.get(t.message_idx) ?? [];
|
||||
arr.push(t);
|
||||
execByMsg.set(t.message_idx, arr);
|
||||
}
|
||||
|
||||
let maxIdx = work.reduce((mx, w) => Math.max(mx, w.message_idx), -1);
|
||||
|
||||
for (let i = 0; i < work.length; i++) {
|
||||
const msg = work[i];
|
||||
if (msg.role !== 'assistant') continue;
|
||||
const toolCalls = msg.blocks.filter(
|
||||
(b): b is Extract<ChatBlock, { type: 'tool-call' }> => b.type === 'tool-call',
|
||||
);
|
||||
if (toolCalls.length === 0) continue;
|
||||
|
||||
// Skip if a following tool-result user turn exists AT ALL. A fully-answered
|
||||
// turn is already balanced; a PARTIALLY-answered turn (only reachable from
|
||||
// externally-corrupted data — gbrain persists all of a turn's results in one
|
||||
// message) is left for repairToolPairing() at the chat() boundary, which
|
||||
// back-fills only the missing ids. Synthesizing a full turn here would
|
||||
// duplicate the answered results and collide with the persisted row.
|
||||
const next = work[i + 1];
|
||||
if (next && next.role === 'user' && next.blocks.some(b => b.type === 'tool-result')) continue;
|
||||
|
||||
const results: ChatBlock[] = [];
|
||||
for (let callIdx = 0; callIdx < toolCalls.length; callIdx++) {
|
||||
const call = toolCalls[callIdx];
|
||||
// Prefer an exact (message_idx, provider tool_use_id) match. The
|
||||
// positional fallback is used only when the row at that ordinal is for
|
||||
// the SAME tool, so a missing row can't mis-attribute a sibling's output.
|
||||
const fallback = execByMsg.get(msg.message_idx)?.[callIdx];
|
||||
const exec = execByKey.get(`${msg.message_idx}:${call.toolCallId}`)
|
||||
?? (fallback && fallback.tool_name === call.toolName ? fallback : undefined);
|
||||
if (exec?.status === 'complete') {
|
||||
results.push({ type: 'tool-result', toolCallId: call.toolCallId, toolName: call.toolName, output: exec.output ?? null });
|
||||
continue;
|
||||
}
|
||||
if (exec?.status === 'failed') {
|
||||
results.push({ type: 'tool-result', toolCallId: call.toolCallId, toolName: call.toolName, output: exec.error ?? 'tool failed', isError: true });
|
||||
continue;
|
||||
}
|
||||
const toolDef = toolDefs.find(t => t.name === call.toolName);
|
||||
if (!toolDef) {
|
||||
await persistToolExecFailed(engine, jobId, msg.message_idx, call.toolCallId, call.toolName, call.input, `tool "${call.toolName}" is not in the registry for this subagent`);
|
||||
results.push({ type: 'tool-result', toolCallId: call.toolCallId, toolName: call.toolName, output: `tool "${call.toolName}" is not available`, isError: true });
|
||||
continue;
|
||||
}
|
||||
if (exec?.status === 'pending' && !toolDef.idempotent) {
|
||||
throw new Error(`non-idempotent tool "${call.toolName}" pending on resume; cannot safely re-run`);
|
||||
}
|
||||
await persistToolExecPending(engine, jobId, msg.message_idx, call.toolCallId, call.toolName, call.input);
|
||||
try {
|
||||
const output = await toolDef.execute(call.input, { engine, jobId, remote: true, signal });
|
||||
await persistToolExecComplete(engine, jobId, call.toolCallId, output);
|
||||
results.push({ type: 'tool-result', toolCallId: call.toolCallId, toolName: call.toolName, output });
|
||||
} catch (e) {
|
||||
const errText = e instanceof Error ? (e.stack ?? e.message) : String(e);
|
||||
await persistToolExecFailed(engine, jobId, msg.message_idx, call.toolCallId, call.toolName, call.input, errText);
|
||||
results.push({ type: 'tool-result', toolCallId: call.toolCallId, toolName: call.toolName, output: errText, isError: true });
|
||||
}
|
||||
}
|
||||
|
||||
const resultIdx = msg.message_idx + 1;
|
||||
await persistMessage(engine, jobId, {
|
||||
message_idx: resultIdx,
|
||||
role: 'user',
|
||||
content_blocks: results as unknown as ContentBlock[],
|
||||
tokens_in: null, tokens_out: null, tokens_cache_read: null, tokens_cache_create: null, model: null,
|
||||
});
|
||||
maxIdx = Math.max(maxIdx, resultIdx);
|
||||
work.splice(i + 1, 0, { message_idx: resultIdx, role: 'user' as const, blocks: results });
|
||||
i++; // skip the turn we just inserted
|
||||
}
|
||||
|
||||
const chatMessages: ChatMessage[] = work.map(w => ({ role: w.role, content: w.blocks }));
|
||||
|
||||
// Terminal case: the transcript already ends on an assistant turn that
|
||||
// carried real text and made no tool calls (prior run reached end_turn).
|
||||
// Surface its text; skip the loop. An assistant turn whose blocks are all
|
||||
// empty (e.g. a reasoning-only/null-text turn that adaptation dropped) is NOT
|
||||
// terminal — falling through lets the loop re-issue the call rather than
|
||||
// returning an empty result.
|
||||
const lastMsg = work[work.length - 1];
|
||||
let terminalText: string | null = null;
|
||||
if (lastMsg && lastMsg.role === 'assistant' && !lastMsg.blocks.some(b => b.type === 'tool-call')) {
|
||||
const text = lastMsg.blocks
|
||||
.filter((b): b is Extract<ChatBlock, { type: 'text' }> => b.type === 'text')
|
||||
.map(b => b.text)
|
||||
.join('\n');
|
||||
if (text.trim() !== '') terminalText = text;
|
||||
}
|
||||
|
||||
return { chatMessages, nextMessageIdx: maxIdx + 1, terminalText };
|
||||
}
|
||||
|
||||
function recipeIdFromModel(modelString: string): string {
|
||||
const idx = modelString.indexOf(':');
|
||||
return idx > 0 ? modelString.slice(0, idx) : 'anthropic';
|
||||
@@ -1087,7 +1290,8 @@ async function loadPriorTools(engine: BrainEngine, jobId: number): Promise<Persi
|
||||
const rows = await engine.executeRaw<Record<string, unknown>>(
|
||||
`SELECT message_idx, tool_use_id, tool_name, input, status, output, error
|
||||
FROM subagent_tool_executions
|
||||
WHERE job_id = $1`,
|
||||
WHERE job_id = $1
|
||||
ORDER BY message_idx, COALESCE(ordinal, 0), id`,
|
||||
[jobId],
|
||||
);
|
||||
return rows.map(r => ({
|
||||
|
||||
@@ -52,11 +52,18 @@ export interface ModelPricing {
|
||||
*/
|
||||
export const CANONICAL_PRICING: Record<string, ModelPricing> = {
|
||||
// ── Anthropic ──────────────────────────────────────────────────────────
|
||||
// Fable 5: Anthropic's top tier, above Opus. $10 in / $50 out.
|
||||
'anthropic:claude-fable-5': { input: 10.00, output: 50.00 },
|
||||
// Opus 4.x: $5 in / $25 out. 4.8 (released 2026-05-28) shares 4.7's
|
||||
// per-token rate — closes gbrain#1819.
|
||||
'anthropic:claude-opus-4-8': { input: 5.00, output: 25.00 },
|
||||
'anthropic:claude-opus-4-7': { input: 5.00, output: 25.00 },
|
||||
'anthropic:claude-opus-4-6': { input: 5.00, output: 25.00 },
|
||||
// Sonnet 5 (released 2026-06-29): same $3/$15 sticker as 4.6. The launch
|
||||
// intro discount ($2/$10 through 2026-08-31) is deliberately NOT modeled —
|
||||
// the table carries standard rates so estimates stay conservative and
|
||||
// don't need a time-bombed edit when the promo lapses.
|
||||
'anthropic:claude-sonnet-5': { input: 3.00, output: 15.00 },
|
||||
'anthropic:claude-sonnet-4-6': { input: 3.00, output: 15.00 },
|
||||
// Haiku 4.5 — both the dateless canonical id and the dated snapshot.
|
||||
'anthropic:claude-haiku-4-5': { input: 1.00, output: 5.00 },
|
||||
|
||||
+10
-4
@@ -2686,10 +2686,16 @@ const file_list: Operation = {
|
||||
handler: async (_ctx, p) => {
|
||||
const sql = db.getConnection();
|
||||
const slug = p.slug as string | undefined;
|
||||
if (slug) {
|
||||
return sql`SELECT id, page_slug, filename, storage_path, mime_type, size_bytes, content_hash, created_at FROM files WHERE page_slug = ${slug} ORDER BY filename LIMIT ${FILE_LIST_LIMIT}`;
|
||||
}
|
||||
return sql`SELECT id, page_slug, filename, storage_path, mime_type, size_bytes, content_hash, created_at FROM files ORDER BY page_slug, filename LIMIT ${FILE_LIST_LIMIT}`;
|
||||
const rows = slug
|
||||
? await sql`SELECT id, page_slug, filename, storage_path, mime_type, size_bytes, content_hash, created_at FROM files WHERE page_slug = ${slug} ORDER BY filename LIMIT ${FILE_LIST_LIMIT}`
|
||||
: await sql`SELECT id, page_slug, filename, storage_path, mime_type, size_bytes, content_hash, created_at FROM files ORDER BY page_slug, filename LIMIT ${FILE_LIST_LIMIT}`;
|
||||
// Postgres returns size_bytes (BIGINT) as native BigInt — JSON.stringify
|
||||
// throws on those, breaking MCP callers. PGLite returns Number already.
|
||||
// 9 PB ceiling (2^53 bytes) is far above any plausible file size.
|
||||
return rows.map((r: Record<string, unknown>) => ({
|
||||
...r,
|
||||
size_bytes: r.size_bytes == null ? null : Number(r.size_bytes),
|
||||
}));
|
||||
},
|
||||
};
|
||||
|
||||
|
||||
@@ -15,6 +15,7 @@ import type { SearchResult, SearchOpts, HybridSearchMeta } from '../types.ts';
|
||||
import { embed, embedQuery } from '../embedding.ts';
|
||||
import { registerBackgroundWorkDrainer } from '../background-work.ts';
|
||||
import { resolveEmbeddingColumn, isCacheSafe } from './embedding-column.ts';
|
||||
import { resolveHardExcludes } from './source-boost.ts';
|
||||
import {
|
||||
resolveAdaptiveReturn,
|
||||
applyAdaptiveReturn,
|
||||
@@ -1631,6 +1632,12 @@ export async function hybridSearchCached(
|
||||
const cacheKnobsHash = knobsHash(resolvedForCache, {
|
||||
embeddingColumn: resolvedColCached.name,
|
||||
embeddingModel: resolvedColCached.embeddingModel,
|
||||
// #2825 — fold the resolved hard-exclude prefix list (defaults ∪
|
||||
// GBRAIN_SEARCH_EXCLUDE ∪ per-call exclude_slug_prefixes, minus
|
||||
// include_slug_prefixes — exactly what the engines' query-build path
|
||||
// resolves) into the cache key so a row written under one exclude
|
||||
// policy can't be served to a lookup under another.
|
||||
hardExcludes: resolveHardExcludes(opts?.exclude_slug_prefixes, opts?.include_slug_prefixes),
|
||||
});
|
||||
|
||||
// Cache decision: opts.useCache (explicit) wins over global config; global
|
||||
|
||||
+26
-1
@@ -747,7 +747,16 @@ export function attributeKnob<K extends keyof ModeBundle>(
|
||||
// to post-fix lookups. Same one-time global cold-miss pattern as the bumps
|
||||
// above (the hash is global, not per-provider); refills within
|
||||
// cache.ttl_seconds (3600s default).
|
||||
export const KNOBS_HASH_VERSION = 11;
|
||||
//
|
||||
// bump 11→12 (2026-07-16, #2825): the resolved hard-exclude slug-prefix list
|
||||
// (defaults ∪ GBRAIN_SEARCH_EXCLUDE ∪ exclude_slug_prefixes, minus
|
||||
// include_slug_prefixes) folds into the key via ctx.hardExcludes. It only
|
||||
// applied at DB-query build time (cache miss), so a process with
|
||||
// GBRAIN_SEARCH_EXCLUDE set could be served cached rows containing excluded
|
||||
// slugs written by a process without it, and vice versa. Same one-time
|
||||
// global cold-miss pattern as the bumps above; refills within
|
||||
// cache.ttl_seconds (3600s default).
|
||||
export const KNOBS_HASH_VERSION = 12;
|
||||
|
||||
/**
|
||||
* v0.36 (D8 / CDX-2) — second-arg context for the cache key. The
|
||||
@@ -776,6 +785,16 @@ export interface KnobsHashContext {
|
||||
*/
|
||||
schemaPack?: string;
|
||||
schemaPackVersion?: string;
|
||||
/**
|
||||
* v=12 (#2825): the RESOLVED effective hard-exclude prefix list — the same
|
||||
* value resolveHardExcludes() produces at query-build time (defaults ∪
|
||||
* GBRAIN_SEARCH_EXCLUDE ∪ per-call exclude_slug_prefixes, minus
|
||||
* include_slug_prefixes). Folded (sorted, so input order is irrelevant)
|
||||
* into the hash so a cache row written under one exclude policy can never
|
||||
* be served to a lookup under another. Undefined falls back to the literal
|
||||
* 'none' for legacy callers that don't thread excludes.
|
||||
*/
|
||||
hardExcludes?: string[];
|
||||
}
|
||||
|
||||
export function knobsHash(
|
||||
@@ -863,6 +882,12 @@ export function knobsHash(
|
||||
// test/model-pricing.test.ts-style drift guards and the mode tests.
|
||||
`rel=${knobs.relationalRetrieval ? 1 : 0}`,
|
||||
`reld=${knobs.relational_retrieval_depth ?? 2}`,
|
||||
// v=12 addition (#2825, append-only): resolved hard-exclude prefixes.
|
||||
// Before this, resolveHardExcludes() only ran at DB-query build time
|
||||
// (cache miss), so cached rows leaked GBRAIN_SEARCH_EXCLUDE'd slugs
|
||||
// across processes. Sorted copy so ['a/','b/'] and ['b/','a/'] hash
|
||||
// identically; undefined falls back to 'none' for legacy callers.
|
||||
`hx=${ctx?.hardExcludes ? [...ctx.hardExcludes].sort().join(',') : 'none'}`,
|
||||
];
|
||||
const h = createHash('sha256');
|
||||
h.update(parts.join('|'));
|
||||
|
||||
@@ -37,6 +37,7 @@ const SUPPORTED_MODELS = [
|
||||
'openai:gpt-5.5',
|
||||
'anthropic:claude-opus-4-8',
|
||||
'anthropic:claude-opus-4-7',
|
||||
'anthropic:claude-sonnet-5',
|
||||
'anthropic:claude-sonnet-4-6',
|
||||
'anthropic:claude-haiku-4-5',
|
||||
'google:gemini-1.5-pro',
|
||||
|
||||
+14
-1
@@ -152,6 +152,19 @@ export interface ThinkResult {
|
||||
|
||||
const DEFAULT_MAX_OUTPUT_TOKENS = 4000;
|
||||
|
||||
// Thinking-by-default Claude 5 models (`anthropic:claude-*-5`) spend a large
|
||||
// share of the output budget on internal reasoning before emitting any answer,
|
||||
// so the 4000 default leaves `think` with empty or truncated text. Give those
|
||||
// models headroom; providers bill actual tokens, not the cap. Everything else
|
||||
// keeps 4000.
|
||||
const THINKING_DEFAULT_MAX_OUTPUT_TOKENS = 16000;
|
||||
const THINKING_BY_DEFAULT_MODEL_RE = /^anthropic[:/]claude-[a-z0-9]+-5(?:[.-]|$)/i;
|
||||
export function maxOutputTokensFor(modelStr: string): number {
|
||||
return THINKING_BY_DEFAULT_MODEL_RE.test(modelStr)
|
||||
? THINKING_DEFAULT_MAX_OUTPUT_TOKENS
|
||||
: DEFAULT_MAX_OUTPUT_TOKENS;
|
||||
}
|
||||
|
||||
function inferIntent(question: string, anchor?: string): string {
|
||||
if (anchor) return 'entity';
|
||||
const q = question.toLowerCase();
|
||||
@@ -465,7 +478,7 @@ export async function runThink(
|
||||
}
|
||||
const result = await client.create({
|
||||
model: modelUsed,
|
||||
max_tokens: DEFAULT_MAX_OUTPUT_TOKENS,
|
||||
max_tokens: maxOutputTokensFor(normalizeModelId(modelUsed)),
|
||||
system: systemPrompt,
|
||||
messages: [{ role: 'user', content: userMessage }],
|
||||
});
|
||||
|
||||
@@ -86,6 +86,26 @@ describe('buildGatewayConfig env-baseURL passthrough', () => {
|
||||
});
|
||||
});
|
||||
|
||||
describe('buildGatewayConfig config-plane API-key folding', () => {
|
||||
test('openrouter_api_key folds into gateway env as OPENROUTER_API_KEY', async () => {
|
||||
await withEnv({ OPENROUTER_API_KEY: undefined }, async () => {
|
||||
const cfg = buildGatewayConfig({
|
||||
openrouter_api_key: 'sk-or-config-plane',
|
||||
} as unknown as GBrainConfig);
|
||||
expect(cfg.env.OPENROUTER_API_KEY).toBe('sk-or-config-plane');
|
||||
});
|
||||
});
|
||||
|
||||
test('a real OPENROUTER_API_KEY process.env value wins over the config-plane fallback', async () => {
|
||||
await withEnv({ OPENROUTER_API_KEY: 'sk-or-env-plane' }, async () => {
|
||||
const cfg = buildGatewayConfig({
|
||||
openrouter_api_key: 'sk-or-config-plane',
|
||||
} as unknown as GBrainConfig);
|
||||
expect(cfg.env.OPENROUTER_API_KEY).toBe('sk-or-env-plane');
|
||||
});
|
||||
});
|
||||
});
|
||||
|
||||
describe('buildGatewayConfig env empty-string clobber guard (#1249)', () => {
|
||||
test('an empty-string process.env value does NOT clobber a valid config-plane key', async () => {
|
||||
// Claude Code injects ANTHROPIC_API_KEY='' to neuter subprocess LLM calls.
|
||||
|
||||
@@ -0,0 +1,124 @@
|
||||
/**
|
||||
* Pins the DeepSeek reasoning_content transport shim. `deepseek-reasoner`
|
||||
* returns its answer in a separate `reasoning_content` field and leaves
|
||||
* `content` empty when the whole turn was reasoning; the AI SDK's
|
||||
* openai-compatible adapter reads only `content`, so the model appears to
|
||||
* answer with nothing. The shim promotes `reasoning_content` into `content`
|
||||
* when `content` is empty, fail-open on anything unexpected.
|
||||
*/
|
||||
import { describe, test, expect, afterEach } from 'bun:test';
|
||||
import { deepseekReasoningContentCompatFetch, deepseek } from '../../src/core/ai/recipes/deepseek.ts';
|
||||
import { applyOpenAICompatConfig } from '../../src/core/ai/gateway.ts';
|
||||
import type { Recipe, AIGatewayConfig } from '../../src/core/ai/types.ts';
|
||||
|
||||
const realFetch = globalThis.fetch;
|
||||
afterEach(() => { globalThis.fetch = realFetch; });
|
||||
|
||||
function stubFetch(body: unknown, init?: { status?: number; contentType?: string }) {
|
||||
globalThis.fetch = (async () =>
|
||||
new Response(JSON.stringify(body), {
|
||||
status: init?.status ?? 200,
|
||||
headers: { 'content-type': init?.contentType ?? 'application/json' },
|
||||
})) as unknown as typeof fetch;
|
||||
}
|
||||
|
||||
describe('deepseekReasoningContentCompatFetch', () => {
|
||||
test('promotes reasoning_content when content is empty', async () => {
|
||||
stubFetch({ choices: [{ message: { role: 'assistant', content: '', reasoning_content: 'the answer' } }] });
|
||||
const res = await deepseekReasoningContentCompatFetch('https://api.deepseek.com/v1/chat/completions');
|
||||
const json = await res.json();
|
||||
expect(json.choices[0].message.content).toBe('the answer');
|
||||
});
|
||||
|
||||
test('promotes when content is null or whitespace-only', async () => {
|
||||
stubFetch({ choices: [{ message: { content: null, reasoning_content: 'from null' } }, { message: { content: ' ', reasoning_content: 'from ws' } }] });
|
||||
const res = await deepseekReasoningContentCompatFetch('u');
|
||||
const json = await res.json();
|
||||
expect(json.choices[0].message.content).toBe('from null');
|
||||
expect(json.choices[1].message.content).toBe('from ws');
|
||||
});
|
||||
|
||||
test('leaves non-empty content untouched (no duplication)', async () => {
|
||||
stubFetch({ choices: [{ message: { content: 'real content', reasoning_content: 'ignored' } }] });
|
||||
const res = await deepseekReasoningContentCompatFetch('u');
|
||||
const json = await res.json();
|
||||
expect(json.choices[0].message.content).toBe('real content');
|
||||
});
|
||||
|
||||
test('both empty: stays empty, no crash', async () => {
|
||||
stubFetch({ choices: [{ message: { content: '', reasoning_content: '' } }] });
|
||||
const res = await deepseekReasoningContentCompatFetch('u');
|
||||
const json = await res.json();
|
||||
expect(json.choices[0].message.content).toBe('');
|
||||
});
|
||||
|
||||
test('tool-call turn (content:null + tool_calls) is NOT promoted — never feed CoT back', async () => {
|
||||
// content:null is the standard OpenAI shape on a tool-call turn. Promoting
|
||||
// reasoning_content here would inject the whole chain-of-thought as assistant
|
||||
// text, which the loop persists + replays every turn. Must be left alone.
|
||||
stubFetch({ choices: [{ finish_reason: 'tool_calls', message: {
|
||||
content: null,
|
||||
reasoning_content: 'INTERNAL CHAIN OF THOUGHT — must not leak',
|
||||
tool_calls: [{ id: 'call_1', type: 'function', function: { name: 'brain_search', arguments: '{}' } }],
|
||||
} }] });
|
||||
const res = await deepseekReasoningContentCompatFetch('u');
|
||||
const json = await res.json();
|
||||
expect(json.choices[0].message.content).toBeNull();
|
||||
expect(json.choices[0].message.tool_calls).toHaveLength(1);
|
||||
});
|
||||
|
||||
test('rebuilt response drops stale content-length header', async () => {
|
||||
globalThis.fetch = (async () => new Response(JSON.stringify({ choices: [{ message: { content: '', reasoning_content: 'x' } }] }), {
|
||||
status: 200, headers: { 'content-type': 'application/json', 'content-length': '999999' },
|
||||
})) as unknown as typeof fetch;
|
||||
const res = await deepseekReasoningContentCompatFetch('u');
|
||||
expect(res.headers.get('content-length')).toBeNull();
|
||||
expect((await res.json()).choices[0].message.content).toBe('x');
|
||||
});
|
||||
});
|
||||
|
||||
describe('applyOpenAICompatConfig — compat.fetch wiring (gateway seam)', () => {
|
||||
const cfg = { env: {}, base_urls: {} } as unknown as AIGatewayConfig;
|
||||
|
||||
test('threads recipe.compat.fetch onto the resolved config (deepseek)', () => {
|
||||
// Guards the src/core/ai/gateway.ts wiring: without `?? recipe.compat?.fetch`
|
||||
// the DeepSeek shim would never install in production.
|
||||
const resolved = applyOpenAICompatConfig(deepseek, cfg);
|
||||
expect(resolved.fetch).toBe(deepseekReasoningContentCompatFetch);
|
||||
});
|
||||
|
||||
test('a resolveOpenAICompatConfig-provided fetch takes precedence over compat.fetch', () => {
|
||||
const ownFetch = (async () => new Response('{}')) as unknown as typeof fetch;
|
||||
const recipe = {
|
||||
id: 'x', name: 'X', tier: 'openai-compat', implementation: 'openai-compatible',
|
||||
touchpoints: {},
|
||||
compat: { fetch: deepseekReasoningContentCompatFetch },
|
||||
resolveOpenAICompatConfig: () => ({ baseURL: 'http://x', fetch: ownFetch }),
|
||||
} as unknown as Recipe;
|
||||
expect(applyOpenAICompatConfig(recipe, cfg).fetch).toBe(ownFetch);
|
||||
});
|
||||
|
||||
test('falls back to compat.fetch when resolveOpenAICompatConfig omits a fetch', () => {
|
||||
const recipe = {
|
||||
id: 'y', name: 'Y', tier: 'openai-compat', implementation: 'openai-compatible',
|
||||
touchpoints: {},
|
||||
compat: { fetch: deepseekReasoningContentCompatFetch },
|
||||
resolveOpenAICompatConfig: () => ({ baseURL: 'http://y' }),
|
||||
} as unknown as Recipe;
|
||||
expect(applyOpenAICompatConfig(recipe, cfg).fetch).toBe(deepseekReasoningContentCompatFetch);
|
||||
});
|
||||
|
||||
test('fail-open on non-ok / non-json responses', async () => {
|
||||
stubFetch({ error: 'nope' }, { status: 500 });
|
||||
const res = await deepseekReasoningContentCompatFetch('u');
|
||||
expect(res.status).toBe(500);
|
||||
globalThis.fetch = (async () =>
|
||||
new Response('plain text', { status: 200, headers: { 'content-type': 'text/plain' } })) as unknown as typeof fetch;
|
||||
const res2 = await deepseekReasoningContentCompatFetch('u');
|
||||
expect(await res2.text()).toBe('plain text');
|
||||
});
|
||||
|
||||
test('recipe wires the shim via compat.fetch', () => {
|
||||
expect(deepseek.compat?.fetch).toBe(deepseekReasoningContentCompatFetch);
|
||||
});
|
||||
});
|
||||
@@ -150,6 +150,47 @@ describe('gateway.toolLoop (v0.38 D11 — provider-agnostic loop control)', () =
|
||||
expect(events[4]).toBe('onAssistantTurn(1)'); // final assistant turn
|
||||
});
|
||||
|
||||
it('persists the tool-result user turn via onToolResultTurn before the next chat', async () => {
|
||||
let turn = 0;
|
||||
__setChatTransportForTests(async () => {
|
||||
turn++;
|
||||
if (turn === 1) {
|
||||
return {
|
||||
text: '',
|
||||
blocks: [{ type: 'tool-call', toolCallId: 'tc1', toolName: 'search', input: { q: 'x' } }] as ChatBlock[],
|
||||
stopReason: 'tool_calls',
|
||||
usage: { input_tokens: 1, output_tokens: 1, cache_read_tokens: 0, cache_creation_tokens: 0 },
|
||||
model: 'anthropic:claude-sonnet-4-6',
|
||||
providerId: 'anthropic',
|
||||
};
|
||||
}
|
||||
return {
|
||||
text: 'done',
|
||||
blocks: [{ type: 'text', text: 'done' }] as ChatBlock[],
|
||||
stopReason: 'end',
|
||||
usage: { input_tokens: 1, output_tokens: 1, cache_read_tokens: 0, cache_creation_tokens: 0 },
|
||||
model: 'anthropic:claude-sonnet-4-6',
|
||||
providerId: 'anthropic',
|
||||
};
|
||||
});
|
||||
|
||||
const resultTurns: Array<{ turnIdx: number; messageIdx: number; blocks: ChatBlock[] }> = [];
|
||||
await toolLoop({
|
||||
initialMessages: [{ role: 'user', content: 'go' }],
|
||||
tools: [{ name: 'search', description: 's', inputSchema: { type: 'object' } }],
|
||||
toolHandlers: new Map([['search', { idempotent: true, async execute() { return { hits: 1 }; } }]]),
|
||||
onToolResultTurn: async (turnIdx, messageIdx, blocks) => {
|
||||
resultTurns.push({ turnIdx, messageIdx, blocks });
|
||||
},
|
||||
});
|
||||
|
||||
// Fired exactly once, for the single tool round, carrying the tool-result.
|
||||
expect(resultTurns).toHaveLength(1);
|
||||
expect(resultTurns[0].turnIdx).toBe(0);
|
||||
expect(resultTurns[0].blocks[0].type).toBe('tool-result');
|
||||
expect((resultTurns[0].blocks[0] as Extract<ChatBlock, { type: 'tool-result' }>).toolCallId).toBe('tc1');
|
||||
});
|
||||
|
||||
it('replay short-circuits a complete prior tool execution', async () => {
|
||||
let chatCalls = 0;
|
||||
__setChatTransportForTests(async () => {
|
||||
@@ -230,6 +271,30 @@ describe('gateway.toolLoop (v0.38 D11 — provider-agnostic loop control)', () =
|
||||
).rejects.toThrow(/non-idempotent.*pending/i);
|
||||
});
|
||||
|
||||
it('defaults max output tokens per model: 4096 for non-thinking, 32000 for Claude 5', async () => {
|
||||
const seen: Array<number | undefined> = [];
|
||||
__setChatTransportForTests(async (opts) => {
|
||||
seen.push(opts.maxTokens);
|
||||
return {
|
||||
text: 'ok',
|
||||
blocks: [{ type: 'text', text: 'ok' }] as ChatBlock[],
|
||||
stopReason: 'end',
|
||||
usage: { input_tokens: 1, output_tokens: 1, cache_read_tokens: 0, cache_creation_tokens: 0 },
|
||||
model: opts.model ?? 'anthropic:claude-sonnet-4-6',
|
||||
providerId: 'anthropic',
|
||||
};
|
||||
});
|
||||
|
||||
await toolLoop({ model: 'openai:gpt-4o', initialMessages: [{ role: 'user', content: 'hi' }], tools: [], toolHandlers: new Map() });
|
||||
await toolLoop({ model: 'anthropic:claude-sonnet-4-6', initialMessages: [{ role: 'user', content: 'hi' }], tools: [], toolHandlers: new Map() });
|
||||
await toolLoop({ model: 'anthropic:claude-sonnet-5', initialMessages: [{ role: 'user', content: 'hi' }], tools: [], toolHandlers: new Map() });
|
||||
await toolLoop({ model: 'anthropic:claude-fable-5', initialMessages: [{ role: 'user', content: 'hi' }], tools: [], toolHandlers: new Map() });
|
||||
|
||||
// Non-thinking / non-Claude-5 stay 4096 (safe under openai-compat caps);
|
||||
// thinking-by-default Claude 5 models get 32000 headroom.
|
||||
expect(seen).toEqual([4096, 4096, 32000, 32000]);
|
||||
});
|
||||
|
||||
it('hits max_turns when the model keeps calling tools', async () => {
|
||||
__setChatTransportForTests(async () => ({
|
||||
text: '',
|
||||
|
||||
@@ -0,0 +1,100 @@
|
||||
/**
|
||||
* Pins `repairToolPairing` (the chat()-boundary safety net) and proves, against
|
||||
* the REAL AI SDK v6 `generateText`, that the two failure modes this wave fixes
|
||||
* are gone:
|
||||
* - an unbalanced tool history (assistant tool-call with no tool-result) is
|
||||
* back-filled so v6 no longer throws AI_MissingToolResultsError, and
|
||||
* - a Date-bearing tool-result (Postgres timestamptz) passes v6's ModelMessage
|
||||
* JSONValue schema after `toModelMessages` ISO-izes it, whereas a raw Date
|
||||
* is still rejected (control).
|
||||
*
|
||||
* MockLanguageModelV3 = no network / no keys.
|
||||
*/
|
||||
import { describe, expect, it } from 'bun:test';
|
||||
import { generateText } from 'ai';
|
||||
import { MockLanguageModelV3 } from 'ai/test';
|
||||
import { repairToolPairing, toModelMessages, type ChatMessage } from '../../src/core/ai/gateway.ts';
|
||||
|
||||
function mockModel(): MockLanguageModelV3 {
|
||||
return new MockLanguageModelV3({
|
||||
doGenerate: async () => ({
|
||||
content: [{ type: 'text', text: 'ok' }],
|
||||
finishReason: 'stop',
|
||||
usage: { inputTokens: 1, outputTokens: 1, totalTokens: 2 },
|
||||
warnings: [],
|
||||
}),
|
||||
} as any);
|
||||
}
|
||||
|
||||
describe('repairToolPairing', () => {
|
||||
it('is a no-op on a balanced history', () => {
|
||||
const msgs: ChatMessage[] = [
|
||||
{ role: 'user', content: 'hi' },
|
||||
{ role: 'assistant', content: [{ type: 'tool-call', toolCallId: 'c1', toolName: 'search', input: {} }] },
|
||||
{ role: 'user', content: [{ type: 'tool-result', toolCallId: 'c1', toolName: 'search', output: { ok: 1 } }] },
|
||||
];
|
||||
expect(repairToolPairing(msgs)).toEqual(msgs);
|
||||
});
|
||||
|
||||
it('synthesizes a tool-result turn when the assistant tool-call is fully unanswered', () => {
|
||||
const msgs: ChatMessage[] = [
|
||||
{ role: 'assistant', content: [{ type: 'tool-call', toolCallId: 'c1', toolName: 'search', input: {} }] },
|
||||
];
|
||||
const out = repairToolPairing(msgs);
|
||||
expect(out).toHaveLength(2);
|
||||
expect(out[1].role).toBe('user');
|
||||
const block = (out[1].content as any[])[0];
|
||||
expect(block).toMatchObject({ type: 'tool-result', toolCallId: 'c1', isError: true });
|
||||
});
|
||||
|
||||
it('merges stubs into a PARTIALLY-answered turn without duplicating the answered id', () => {
|
||||
const msgs: ChatMessage[] = [
|
||||
{ role: 'assistant', content: [
|
||||
{ type: 'tool-call', toolCallId: 'a', toolName: 'search', input: {} },
|
||||
{ type: 'tool-call', toolCallId: 'b', toolName: 'search', input: {} },
|
||||
] },
|
||||
{ role: 'user', content: [{ type: 'tool-result', toolCallId: 'a', toolName: 'search', output: { ok: 1 } }] },
|
||||
];
|
||||
const out = repairToolPairing(msgs);
|
||||
expect(out).toHaveLength(2); // merged in place, not appended
|
||||
const ids = (out[1].content as any[]).map((x) => x.toolCallId);
|
||||
expect(ids).toEqual(['a', 'b']); // 'a' kept once, 'b' back-filled
|
||||
expect((out[1].content as any[]).filter((x) => x.toolCallId === 'a')).toHaveLength(1);
|
||||
});
|
||||
});
|
||||
|
||||
describe('real AI SDK v6 validation', () => {
|
||||
it('an unbalanced history passes generateText after repairToolPairing', async () => {
|
||||
const model = mockModel();
|
||||
const unbalanced: ChatMessage[] = [
|
||||
{ role: 'user', content: 'go' },
|
||||
{ role: 'assistant', content: [{ type: 'tool-call', toolCallId: 'c1', toolName: 'search', input: {} }] },
|
||||
// no tool-result turn
|
||||
];
|
||||
const result = await generateText({
|
||||
model: model as any,
|
||||
messages: toModelMessages(repairToolPairing(unbalanced)) as any,
|
||||
});
|
||||
expect(result.text).toBe('ok');
|
||||
const prompt = model.doGenerateCalls[0]!.prompt as any[];
|
||||
expect(prompt.some((m) => m.role === 'tool')).toBe(true); // stub promoted to tool role
|
||||
});
|
||||
|
||||
it('a Date-bearing tool-result passes after toModelMessages ISO-izes it; a raw Date is rejected', async () => {
|
||||
const withDate: ChatMessage[] = [
|
||||
{ role: 'user', content: 'go' },
|
||||
{ role: 'assistant', content: [{ type: 'tool-call', toolCallId: 'c1', toolName: 'brain_get_page', input: {} }] },
|
||||
{ role: 'user', content: [{ type: 'tool-result', toolCallId: 'c1', toolName: 'brain_get_page', output: { updated_at: new Date('2026-06-26T06:56:59.000Z') } }] },
|
||||
];
|
||||
// Fixed path: converted history validates.
|
||||
await expect(generateText({ model: mockModel() as any, messages: toModelMessages(withDate) as any })).resolves.toBeDefined();
|
||||
|
||||
// Control: a raw Date placed straight into a v6 ModelMessage json value is rejected.
|
||||
const rawDateMessages = [
|
||||
{ role: 'user', content: 'go' },
|
||||
{ role: 'assistant', content: [{ type: 'tool-call', toolCallId: 'c1', toolName: 'brain_get_page', input: {} }] },
|
||||
{ role: 'tool', content: [{ type: 'tool-result', toolCallId: 'c1', toolName: 'brain_get_page', output: { type: 'json', value: { updated_at: new Date('2026-06-26T06:56:59.000Z') } } }] },
|
||||
];
|
||||
await expect(generateText({ model: mockModel() as any, messages: rawDateMessages as any })).rejects.toThrow();
|
||||
});
|
||||
});
|
||||
@@ -1,7 +1,7 @@
|
||||
import { describe, expect, it } from 'bun:test';
|
||||
import { generateText, jsonSchema } from 'ai';
|
||||
import { MockLanguageModelV3 } from 'ai/test';
|
||||
import { toModelMessages, type ChatMessage } from '../../src/core/ai/gateway.ts';
|
||||
import { toAISDKTools, toModelMessages, type ChatMessage } from '../../src/core/ai/gateway.ts';
|
||||
|
||||
// v0.42 AI SDK v6 fix — the regression guard that the original bug evaded.
|
||||
// Every gateway/toolLoop test stubs the chat transport, which short-circuits
|
||||
@@ -36,13 +36,13 @@ const internalMessages: ChatMessage[] = [
|
||||
describe('gateway tool schema + message shape (real AI SDK v6)', () => {
|
||||
it('jsonSchema()-wrapped tools + adapted messages pass generateText without throwing', async () => {
|
||||
const model = mockModel();
|
||||
// Built exactly as gateway.chat() builds it (the primary fix).
|
||||
const tools = {
|
||||
search: {
|
||||
const tools = toAISDKTools([
|
||||
{
|
||||
name: 'search',
|
||||
description: 'search the brain',
|
||||
inputSchema: jsonSchema({ type: 'object', properties: { q: { type: 'string' } } } as any),
|
||||
inputSchema: { type: 'object', properties: { q: { type: 'string' } } },
|
||||
},
|
||||
};
|
||||
]);
|
||||
|
||||
const result = await generateText({
|
||||
model: model as any,
|
||||
|
||||
@@ -0,0 +1,58 @@
|
||||
import { describe, test, expect, beforeEach, afterEach } from 'bun:test';
|
||||
import { mkdtempSync, rmSync, writeFileSync } from 'fs';
|
||||
import { join } from 'path';
|
||||
import { tmpdir } from 'os';
|
||||
import { decideLockAcquisition, isPidAlive } from '../src/commands/autopilot.ts';
|
||||
|
||||
let tmp: string;
|
||||
let lockPath: string;
|
||||
beforeEach(() => {
|
||||
tmp = mkdtempSync(join(tmpdir(), 'gbrain-autopilot-lock-'));
|
||||
lockPath = join(tmp, 'autopilot.lock');
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
rmSync(tmp, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
describe('isPidAlive', () => {
|
||||
test('returns true for the current process', () => {
|
||||
expect(isPidAlive(process.pid)).toBe(true);
|
||||
});
|
||||
|
||||
test('returns false for invalid process ids', () => {
|
||||
expect(isPidAlive(0)).toBe(false);
|
||||
expect(isPidAlive(-1)).toBe(false);
|
||||
expect(isPidAlive(Number.NaN)).toBe(false);
|
||||
expect(isPidAlive(Number.POSITIVE_INFINITY)).toBe(false);
|
||||
});
|
||||
});
|
||||
|
||||
describe('decideLockAcquisition', () => {
|
||||
test('acquires when no lock exists', () => {
|
||||
expect(decideLockAcquisition(lockPath, process.pid)).toEqual({ action: 'acquire' });
|
||||
});
|
||||
|
||||
test('takes over a lock whose holder is dead', () => {
|
||||
writeFileSync(lockPath, '4194303');
|
||||
expect(decideLockAcquisition(lockPath, process.pid)).toEqual({
|
||||
action: 'takeover',
|
||||
reason: 'dead pid 4194303',
|
||||
});
|
||||
});
|
||||
|
||||
test('keeps a lock whose holder is alive regardless of age', () => {
|
||||
writeFileSync(lockPath, String(process.pid));
|
||||
expect(decideLockAcquisition(lockPath, process.pid + 100_000)).toEqual({
|
||||
action: 'exit',
|
||||
holderPid: process.pid,
|
||||
});
|
||||
});
|
||||
|
||||
test('takes over malformed and empty locks', () => {
|
||||
writeFileSync(lockPath, 'not-a-pid');
|
||||
expect(decideLockAcquisition(lockPath, process.pid).action).toBe('takeover');
|
||||
writeFileSync(lockPath, '');
|
||||
expect(decideLockAcquisition(lockPath, process.pid).action).toBe('takeover');
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,50 @@
|
||||
import { afterEach, describe, expect, test } from 'bun:test';
|
||||
import { existsSync, mkdirSync, mkdtempSync, rmSync, writeFileSync } from 'node:fs';
|
||||
import { tmpdir } from 'node:os';
|
||||
import { join } from 'node:path';
|
||||
import { _testHelpers } from '../../src/commands/schema.ts';
|
||||
import { withEnv } from '../helpers/with-env.ts';
|
||||
|
||||
const tempDirs: string[] = [];
|
||||
|
||||
function tempDir(prefix: string): string {
|
||||
const dir = mkdtempSync(join(tmpdir(), prefix));
|
||||
tempDirs.push(dir);
|
||||
return dir;
|
||||
}
|
||||
|
||||
afterEach(() => {
|
||||
for (const dir of tempDirs.splice(0)) {
|
||||
rmSync(dir, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
describe('schema packPathByName', () => {
|
||||
test('resolves all bundled schema pack names to bundled YAML files', () => {
|
||||
for (const name of ['gbrain-base', 'gbrain-recommended', 'gbrain-base-v2']) {
|
||||
const path = _testHelpers.packPathByName(name);
|
||||
expect(path).toBeTruthy();
|
||||
expect(path!.endsWith(`src/core/schema-pack/base/${name}.yaml`)).toBe(true);
|
||||
expect(existsSync(path!)).toBe(true);
|
||||
}
|
||||
});
|
||||
|
||||
test('returns null for an unknown non-bundled pack name', async () => {
|
||||
const home = tempDir('gbrain-packpath-home-');
|
||||
await withEnv({ GBRAIN_HOME: home }, async () => {
|
||||
expect(_testHelpers.packPathByName('definitely-not-a-pack')).toBeNull();
|
||||
});
|
||||
});
|
||||
|
||||
test('resolves a user-installed pack by name', async () => {
|
||||
const home = tempDir('gbrain-packpath-home-');
|
||||
await withEnv({ GBRAIN_HOME: home }, async () => {
|
||||
const packDir = join(home, '.gbrain', 'schema-packs', 'custom-pack');
|
||||
mkdirSync(packDir, { recursive: true });
|
||||
const packPath = join(packDir, 'pack.yaml');
|
||||
writeFileSync(packPath, 'name: custom-pack\n', 'utf-8');
|
||||
|
||||
expect(_testHelpers.packPathByName('custom-pack')).toBe(packPath);
|
||||
});
|
||||
});
|
||||
});
|
||||
@@ -38,6 +38,15 @@ describe('KNOWN_CONFIG_KEYS', () => {
|
||||
expect(KNOWN_CONFIG_KEYS).toContain('embed.backfill_max_usd');
|
||||
});
|
||||
|
||||
test('includes the gateway-loop toggle and provider API keys the wave wires', () => {
|
||||
// The subagent handler's error message tells users to run
|
||||
// `gbrain config set agent.use_gateway_loop true`; it must be a known key
|
||||
// or `config set` rejects the wave's own enable command without --force.
|
||||
expect(KNOWN_CONFIG_KEYS).toContain('agent.use_gateway_loop');
|
||||
expect(KNOWN_CONFIG_KEYS).toContain('openrouter_api_key');
|
||||
expect(KNOWN_CONFIG_KEYS).toContain('zeroentropy_api_key');
|
||||
});
|
||||
|
||||
test('no duplicate entries', () => {
|
||||
const set = new Set(KNOWN_CONFIG_KEYS);
|
||||
expect(set.size).toBe(KNOWN_CONFIG_KEYS.length);
|
||||
|
||||
@@ -253,3 +253,15 @@ describe('loadConfig — GBRAIN_MAX_MARKUP_RATIO env (v0.42 #1699)', () => {
|
||||
});
|
||||
});
|
||||
});
|
||||
|
||||
describe('KNOWN_CONFIG_KEYS — documented enable commands must be registered', () => {
|
||||
test('Life Chronicle keys are registered (v0.42.56.0 release notes say `config set auto_chronicle true`)', async () => {
|
||||
const { KNOWN_CONFIG_KEYS, KNOWN_CONFIG_KEY_PREFIXES } = await import('../src/core/config.ts');
|
||||
// The flag the chronicle backstop reads (isAutoChronicleEnabled).
|
||||
expect(KNOWN_CONFIG_KEYS).toContain('auto_chronicle');
|
||||
// The takes bootstrap two-gate consent flag (v0.41.18.0 A12).
|
||||
expect(KNOWN_CONFIG_KEYS).toContain('takes.bootstrap_enabled');
|
||||
// chronicle.tz (chronicleTz) + future chronicle.* knobs.
|
||||
expect(KNOWN_CONFIG_KEY_PREFIXES.some(p => 'chronicle.tz'.startsWith(p))).toBe(true);
|
||||
});
|
||||
});
|
||||
|
||||
@@ -136,7 +136,7 @@ describe('D2 — knobsHash differs across cross-modal knob values', () => {
|
||||
return resolveSearchMode({ mode: 'balanced' });
|
||||
}
|
||||
|
||||
test('KNOBS_HASH_VERSION is 11 (cross-modal still appended; 10→11 asymmetric input_type fix)', () => {
|
||||
test('KNOBS_HASH_VERSION is 12 (cross-modal still appended; 11→12 hard-exclude fold #2825)', () => {
|
||||
// v0.35 ladder: 1→2 reranker, 2→3 floor_ratio. v0.36 piggybacks on v=3
|
||||
// with 7 cross-modal knobs + column/provider context. v0.40.4 (salem) +
|
||||
// v0.39 T21 (master) bump to v=4 for graph_signals + schema-pack fields.
|
||||
@@ -145,8 +145,8 @@ describe('D2 — knobsHash differs across cross-modal knob values', () => {
|
||||
// T2: 6→7 title_boost. v0.42.3.0: 7→8 autocut. issue #1777: 8→9 archive/ demote.
|
||||
// v0.43: 9→10 relational recall arm. #1400: 10→11 query-side input_type
|
||||
// finally reaches asymmetric providers — pre-fix rows were keyed on
|
||||
// document-side query vectors.
|
||||
expect(KNOBS_HASH_VERSION).toBe(11);
|
||||
// document-side query vectors. #2825: 11→12 hard-exclude fold (hx=).
|
||||
expect(KNOBS_HASH_VERSION).toBe(12);
|
||||
});
|
||||
|
||||
test('flipping unified_multimodal changes the hash', () => {
|
||||
|
||||
@@ -14,6 +14,7 @@
|
||||
|
||||
import { describe, test, expect, beforeAll, afterAll, beforeEach } from 'bun:test';
|
||||
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
|
||||
import { configureGateway, resetGateway } from '../src/core/ai/gateway.ts';
|
||||
import { resetPgliteState } from './helpers/reset-pglite.ts';
|
||||
import { withEnv } from './helpers/with-env.ts';
|
||||
import { checkHiddenBySearchPolicy } from '../src/commands/doctor.ts';
|
||||
@@ -58,6 +59,23 @@ async function seed(
|
||||
}
|
||||
|
||||
beforeAll(async () => {
|
||||
// Pin the embedding dim to 1536 BEFORE initSchema. basisEmbedding()
|
||||
// hardcodes Float32Array(1536) vectors, but initSchema sizes vector
|
||||
// columns from process-global gateway state (getEmbeddingDimensions(),
|
||||
// default 1280 = zeroentropyai). Whether this file passes therefore
|
||||
// depended on which test files happened to run before it in the shard: a
|
||||
// predecessor that leaves the gateway configured without dims (or a bare
|
||||
// CI env) yields vector(1280) and every upsertChunks here dies with
|
||||
// "expected 1280 dimensions, not 1536". Adding test files to the repo
|
||||
// reshuffles the weight-packed shards, so unrelated PRs trip it (seen on
|
||||
// #2800 CI, test (1)). Same fix + rationale as
|
||||
// engine-find-trajectory.test.ts and cosine-rescore-column.test.ts, which
|
||||
// document this exact class.
|
||||
configureGateway({
|
||||
embedding_model: 'openai:text-embedding-3-large',
|
||||
embedding_dimensions: 1536,
|
||||
env: { OPENAI_API_KEY: 'sk-test-hidden-by-search-policy' },
|
||||
});
|
||||
engine = new PGLiteEngine();
|
||||
await engine.connect({});
|
||||
await engine.initSchema();
|
||||
@@ -65,6 +83,7 @@ beforeAll(async () => {
|
||||
|
||||
afterAll(async () => {
|
||||
await engine.disconnect();
|
||||
resetGateway();
|
||||
});
|
||||
|
||||
beforeEach(async () => {
|
||||
|
||||
@@ -582,6 +582,11 @@ describeE2E('E2E: Files', () => {
|
||||
const files = await callOp('file_list', {}) as any[];
|
||||
expect(files.length).toBe(1);
|
||||
|
||||
// Regression: Postgres BIGINT(size_bytes) returned native BigInt before
|
||||
// v0.22.5 so the MCP serializer threw and CLI listFiles div-by-1024 threw.
|
||||
expect(typeof files[0].size_bytes).toBe('number');
|
||||
expect(() => JSON.stringify(files)).not.toThrow();
|
||||
|
||||
// Verify file_url returns URI format
|
||||
const url = await callOp('file_url', { storage_path: result.storage_path }) as any;
|
||||
expect(url.url).toContain('gbrain:files/');
|
||||
|
||||
@@ -0,0 +1,280 @@
|
||||
/**
|
||||
* E2E: gateway-loop resume reconciliation (fix-wave A).
|
||||
*
|
||||
* The gateway-native subagent loop persists each assistant turn but historically
|
||||
* never persisted the following tool-result user turn. A resumed job therefore
|
||||
* reloaded assistant tool-calls with no matching tool-result, and non-Anthropic
|
||||
* (openai-compat) providers reject that unbalanced history with
|
||||
* AI_MissingToolResultsError — dead-lettering the job. This wave:
|
||||
* 1. forward-persists the tool-result user turn (onToolResultTurn), and
|
||||
* 2. reconciles an already-corrupted transcript on resume by rebuilding the
|
||||
* missing tool-result turns from settled subagent_tool_executions,
|
||||
* re-dispatching idempotent-pending tools and throwing on non-idempotent.
|
||||
*
|
||||
* Hermetic: PGLite in-memory engine, gateway transport stubbed. Seeds use the
|
||||
* sanctioned `$N::text::jsonb` positional bind (NEVER JSON.stringify into a
|
||||
* bare `::jsonb`) so the seed is Postgres-safe too.
|
||||
*
|
||||
* Supersedes the resume/replay work in #1934 #2062 #2065 #2112 #2274 #2487
|
||||
* #2802 #2336 #2257 #2499.
|
||||
*/
|
||||
|
||||
import { describe, it, expect, beforeAll, afterAll, beforeEach } from 'bun:test';
|
||||
import { PGLiteEngine } from '../../src/core/pglite-engine.ts';
|
||||
import { resetPgliteState } from '../helpers/reset-pglite.ts';
|
||||
import { makeSubagentHandler } from '../../src/core/minions/handlers/subagent.ts';
|
||||
import type { MinionJobContext, ToolDef, ToolCtx } from '../../src/core/minions/types.ts';
|
||||
import {
|
||||
__setChatTransportForTests,
|
||||
configureGateway,
|
||||
resetGateway,
|
||||
toModelMessages,
|
||||
type ChatBlock,
|
||||
type ChatMessage,
|
||||
type ChatResult,
|
||||
} from '../../src/core/ai/gateway.ts';
|
||||
|
||||
let engine: PGLiteEngine;
|
||||
|
||||
beforeAll(async () => {
|
||||
engine = new PGLiteEngine();
|
||||
await engine.connect({});
|
||||
await engine.initSchema();
|
||||
});
|
||||
afterAll(async () => {
|
||||
__setChatTransportForTests(null);
|
||||
resetGateway();
|
||||
await engine.disconnect();
|
||||
});
|
||||
beforeEach(async () => {
|
||||
await resetPgliteState(engine);
|
||||
await engine.setConfig('version', '85');
|
||||
await engine.setConfig('agent.use_gateway_loop', 'true');
|
||||
configureGateway({
|
||||
chat_model: 'anthropic:claude-sonnet-4-6',
|
||||
embedding_model: 'openai:text-embedding-3-large',
|
||||
embedding_dimensions: 1536,
|
||||
expansion_model: 'anthropic:claude-haiku-4-5',
|
||||
env: { ANTHROPIC_API_KEY: 'stub', OPENAI_API_KEY: 'stub' },
|
||||
});
|
||||
});
|
||||
|
||||
async function makeJob(prompt: string, model: string): Promise<{ jobId: number; ctx: MinionJobContext }> {
|
||||
const rows = await engine.executeRaw<{ id: number }>(
|
||||
`INSERT INTO minion_jobs (name, status, data, queue, priority, created_at)
|
||||
VALUES ('subagent', 'active', $1::text::jsonb, 'default', 0, now()) RETURNING id`,
|
||||
[JSON.stringify({ prompt, model })],
|
||||
);
|
||||
const jobId = rows[0].id;
|
||||
const ctx: MinionJobContext = {
|
||||
id: jobId, name: 'subagent', data: { prompt, model }, attempts_made: 1,
|
||||
signal: new AbortController().signal, shutdownSignal: new AbortController().signal,
|
||||
updateProgress: async () => {}, updateTokens: async () => {}, log: async () => {},
|
||||
isActive: async () => true, readInbox: async () => [],
|
||||
};
|
||||
return { jobId, ctx };
|
||||
}
|
||||
|
||||
function makeTools(executions: string[]): ToolDef[] {
|
||||
return [
|
||||
{ name: 'search', description: 's', input_schema: { type: 'object' }, idempotent: true,
|
||||
async execute(input: unknown, _c: ToolCtx) { executions.push('search'); return { results: ['fresh'] }; } },
|
||||
{ name: 'put_page', description: 'p', input_schema: { type: 'object' }, idempotent: false,
|
||||
async execute(_input: unknown, _c: ToolCtx) { executions.push('put_page'); return { saved: true }; } },
|
||||
];
|
||||
}
|
||||
|
||||
function buildHandler(toolRegistry: ToolDef[]) {
|
||||
return makeSubagentHandler({
|
||||
engine, config: {} as any, toolRegistry,
|
||||
makeAnthropic: () => ({ messages: { create: async () => { throw new Error('legacy path unused'); } } }) as any,
|
||||
});
|
||||
}
|
||||
|
||||
async function seedMessage(jobId: number, idx: number, role: string, blocks: ChatBlock[]): Promise<void> {
|
||||
await engine.executeRaw(
|
||||
`INSERT INTO subagent_messages (job_id, message_idx, role, content_blocks, schema_version)
|
||||
VALUES ($1, $2, $3, $4::text::jsonb, 2)`,
|
||||
[jobId, idx, role, JSON.stringify(blocks)],
|
||||
);
|
||||
}
|
||||
|
||||
async function seedExec(jobId: number, msgIdx: number, toolUseId: string, name: string, status: string, output: unknown, ordinal: number, error?: string): Promise<void> {
|
||||
await engine.executeRaw(
|
||||
`INSERT INTO subagent_tool_executions
|
||||
(job_id, message_idx, tool_use_id, tool_name, input, status, output, error, schema_version, ordinal)
|
||||
VALUES ($1, $2, $3, $4, $5::text::jsonb, $6, $7::text::jsonb, $8, 2, $9)`,
|
||||
[jobId, msgIdx, toolUseId, name, JSON.stringify({}), status, output == null ? null : JSON.stringify(output), error ?? null, ordinal],
|
||||
);
|
||||
}
|
||||
|
||||
/** Assert every assistant tool-call turn is answered by a following tool-result. */
|
||||
function assertBalanced(messages: ChatMessage[]): void {
|
||||
for (let i = 0; i < messages.length; i++) {
|
||||
const m = messages[i];
|
||||
if (m.role !== 'assistant' || typeof m.content === 'string') continue;
|
||||
const calls = m.content.filter((b): b is Extract<ChatBlock, { type: 'tool-call' }> => b.type === 'tool-call');
|
||||
if (calls.length === 0) continue;
|
||||
const next = messages[i + 1];
|
||||
expect(next, `assistant tool-call turn at ${i} must be followed by a tool-result turn`).toBeDefined();
|
||||
const answered = new Set(
|
||||
(typeof next!.content === 'string' ? [] : next!.content)
|
||||
.filter((b): b is Extract<ChatBlock, { type: 'tool-result' }> => b.type === 'tool-result')
|
||||
.map(b => b.toolCallId),
|
||||
);
|
||||
for (const c of calls) expect(answered.has(c.toolCallId), `tool-call ${c.toolCallId} unanswered`).toBe(true);
|
||||
}
|
||||
// The real provider path: this must not throw the ModelMessage schema error.
|
||||
expect(() => toModelMessages(messages)).not.toThrow();
|
||||
}
|
||||
|
||||
describe('gateway resume reconciliation', () => {
|
||||
it('forward-persists the tool-result user turn (idx 2) in a 2-turn flow', async () => {
|
||||
let turn = 0;
|
||||
__setChatTransportForTests(async () => {
|
||||
turn++;
|
||||
if (turn === 1) return {
|
||||
text: '', blocks: [{ type: 'tool-call', toolCallId: 'tc1', toolName: 'search', input: { q: 'x' } }] as ChatBlock[],
|
||||
stopReason: 'tool_calls', usage: { input_tokens: 1, output_tokens: 1, cache_read_tokens: 0, cache_creation_tokens: 0 },
|
||||
model: 'anthropic:claude-sonnet-4-6', providerId: 'anthropic',
|
||||
} satisfies ChatResult;
|
||||
return {
|
||||
text: 'done', blocks: [{ type: 'text', text: 'done' }] as ChatBlock[],
|
||||
stopReason: 'end', usage: { input_tokens: 1, output_tokens: 1, cache_read_tokens: 0, cache_creation_tokens: 0 },
|
||||
model: 'anthropic:claude-sonnet-4-6', providerId: 'anthropic',
|
||||
} satisfies ChatResult;
|
||||
});
|
||||
const { jobId, ctx } = await makeJob('go', 'anthropic:claude-sonnet-4-6');
|
||||
await buildHandler(makeTools([]))(ctx);
|
||||
|
||||
const msgs = await engine.executeRaw<{ message_idx: number; role: string; content_blocks: unknown }>(
|
||||
`SELECT message_idx, role, content_blocks FROM subagent_messages WHERE job_id = $1 ORDER BY message_idx`, [jobId]);
|
||||
expect(msgs.map(m => [m.message_idx, m.role])).toEqual([[0, 'user'], [1, 'assistant'], [2, 'user'], [3, 'assistant']]);
|
||||
const toolResultTurn = typeof msgs[2].content_blocks === 'string' ? JSON.parse(msgs[2].content_blocks as string) : msgs[2].content_blocks;
|
||||
expect((toolResultTurn as any[])[0].type).toBe('tool-result');
|
||||
expect((toolResultTurn as any[])[0].toolCallId).toBe('tc1');
|
||||
});
|
||||
|
||||
it('self-heals a pre-fix corrupted job from the stored output (no re-execute), transcript balanced', async () => {
|
||||
const { jobId, ctx } = await makeJob('resume me', 'openai:gpt-4o'); // non-Anthropic: strict pairing
|
||||
// Corrupted pre-fix state: seed user + assistant(tool-call), a complete
|
||||
// exec row, but NO tool-result user turn at idx 2.
|
||||
await seedMessage(jobId, 0, 'user', [{ type: 'text', text: 'resume me' }]);
|
||||
await seedMessage(jobId, 1, 'assistant', [{ type: 'tool-call', toolCallId: 'prov-tc-1', toolName: 'search', input: { q: 'x' } }]);
|
||||
await seedExec(jobId, 1, 'prov-tc-1', 'search', 'complete', { results: ['from-prior-run'] }, 0);
|
||||
|
||||
let captured: ChatMessage[] = [];
|
||||
__setChatTransportForTests(async (opts) => {
|
||||
captured = opts.messages;
|
||||
return {
|
||||
text: 'recovered and done', blocks: [{ type: 'text', text: 'recovered and done' }] as ChatBlock[],
|
||||
stopReason: 'end', usage: { input_tokens: 1, output_tokens: 1, cache_read_tokens: 0, cache_creation_tokens: 0 },
|
||||
model: 'openai:gpt-4o', providerId: 'openai',
|
||||
} satisfies ChatResult;
|
||||
});
|
||||
const executions: string[] = [];
|
||||
const result = await buildHandler(makeTools(executions))(ctx);
|
||||
|
||||
expect(result.result).toBe('recovered and done');
|
||||
expect(executions.length).toBe(0); // stored output reused, tool NOT re-run
|
||||
assertBalanced(captured);
|
||||
// The healed tool-result carries the real stored output.
|
||||
const healed = (captured[2].content as ChatBlock[])[0] as Extract<ChatBlock, { type: 'tool-result' }>;
|
||||
expect(healed.output).toEqual({ results: ['from-prior-run'] });
|
||||
|
||||
// Durably persisted so the next resume stays balanced.
|
||||
const msgs = await engine.executeRaw<{ message_idx: number; role: string }>(
|
||||
`SELECT message_idx, role FROM subagent_messages WHERE job_id = $1 ORDER BY message_idx`, [jobId]);
|
||||
expect(msgs.map(m => [m.message_idx, m.role])).toEqual([[0, 'user'], [1, 'assistant'], [2, 'user'], [3, 'assistant']]);
|
||||
});
|
||||
|
||||
it('heals MULTIPLE consecutive dangling assistant turns (pre-fix multi-turn corruption)', async () => {
|
||||
const { jobId, ctx } = await makeJob('multi', 'openai:gpt-4o');
|
||||
// Pre-fix loop persisted assistants at 1 and 3 (gaps at 2 = skipped user idx).
|
||||
await seedMessage(jobId, 0, 'user', [{ type: 'text', text: 'multi' }]);
|
||||
await seedMessage(jobId, 1, 'assistant', [{ type: 'tool-call', toolCallId: 'tc-a', toolName: 'search', input: {} }]);
|
||||
await seedExec(jobId, 1, 'tc-a', 'search', 'complete', { results: ['a'] }, 0);
|
||||
await seedMessage(jobId, 3, 'assistant', [{ type: 'tool-call', toolCallId: 'tc-b', toolName: 'search', input: {} }]);
|
||||
await seedExec(jobId, 3, 'tc-b', 'search', 'complete', { results: ['b'] }, 0);
|
||||
|
||||
let captured: ChatMessage[] = [];
|
||||
__setChatTransportForTests(async (opts) => {
|
||||
captured = opts.messages;
|
||||
return { text: 'ok', blocks: [{ type: 'text', text: 'ok' }] as ChatBlock[], stopReason: 'end',
|
||||
usage: { input_tokens: 1, output_tokens: 1, cache_read_tokens: 0, cache_creation_tokens: 0 },
|
||||
model: 'openai:gpt-4o', providerId: 'openai' } satisfies ChatResult;
|
||||
});
|
||||
const executions: string[] = [];
|
||||
await buildHandler(makeTools(executions))(ctx);
|
||||
|
||||
expect(executions.length).toBe(0);
|
||||
assertBalanced(captured);
|
||||
// Both dangling turns healed and persisted (idx 2 and 4).
|
||||
const msgs = await engine.executeRaw<{ message_idx: number; role: string }>(
|
||||
`SELECT message_idx, role FROM subagent_messages WHERE job_id = $1 ORDER BY message_idx`, [jobId]);
|
||||
expect(msgs.map(m => m.message_idx)).toContain(2);
|
||||
expect(msgs.map(m => m.message_idx)).toContain(4);
|
||||
});
|
||||
|
||||
it('re-dispatches an idempotent tool that was still pending on resume', async () => {
|
||||
const { jobId, ctx } = await makeJob('redispatch', 'openai:gpt-4o');
|
||||
await seedMessage(jobId, 0, 'user', [{ type: 'text', text: 'redispatch' }]);
|
||||
await seedMessage(jobId, 1, 'assistant', [{ type: 'tool-call', toolCallId: 'tc-pending', toolName: 'search', input: {} }]);
|
||||
await seedExec(jobId, 1, 'tc-pending', 'search', 'pending', null, 0);
|
||||
|
||||
__setChatTransportForTests(async () => ({ text: 'ok', blocks: [{ type: 'text', text: 'ok' }] as ChatBlock[], stopReason: 'end',
|
||||
usage: { input_tokens: 1, output_tokens: 1, cache_read_tokens: 0, cache_creation_tokens: 0 }, model: 'openai:gpt-4o', providerId: 'openai' } satisfies ChatResult));
|
||||
const executions: string[] = [];
|
||||
await buildHandler(makeTools(executions))(ctx);
|
||||
expect(executions).toEqual(['search']); // idempotent-pending re-executed once
|
||||
});
|
||||
|
||||
it('throws on a non-idempotent tool still pending on resume', async () => {
|
||||
const { jobId, ctx } = await makeJob('unsafe', 'openai:gpt-4o');
|
||||
await seedMessage(jobId, 0, 'user', [{ type: 'text', text: 'unsafe' }]);
|
||||
await seedMessage(jobId, 1, 'assistant', [{ type: 'tool-call', toolCallId: 'tc-mut', toolName: 'put_page', input: {} }]);
|
||||
await seedExec(jobId, 1, 'tc-mut', 'put_page', 'pending', null, 0);
|
||||
|
||||
__setChatTransportForTests(async () => ({ text: '', blocks: [] as ChatBlock[], stopReason: 'end',
|
||||
usage: { input_tokens: 0, output_tokens: 0, cache_read_tokens: 0, cache_creation_tokens: 0 }, model: 'openai:gpt-4o', providerId: 'openai' } satisfies ChatResult));
|
||||
await expect(buildHandler(makeTools([]))(ctx)).rejects.toThrow(/non-idempotent tool "put_page" pending on resume/i);
|
||||
});
|
||||
|
||||
it('error-stubs a dangling tool-call whose tool is no longer registered', async () => {
|
||||
const { jobId, ctx } = await makeJob('gone tool', 'openai:gpt-4o');
|
||||
await seedMessage(jobId, 0, 'user', [{ type: 'text', text: 'gone tool' }]);
|
||||
await seedMessage(jobId, 1, 'assistant', [{ type: 'tool-call', toolCallId: 'tc-gone', toolName: 'removed_tool', input: {} }]);
|
||||
// No exec row and the tool isn't in the registry (only 'search'/'put_page').
|
||||
|
||||
let captured: ChatMessage[] = [];
|
||||
__setChatTransportForTests(async (opts) => {
|
||||
captured = opts.messages;
|
||||
return { text: 'handled', blocks: [{ type: 'text', text: 'handled' }] as ChatBlock[], stopReason: 'end',
|
||||
usage: { input_tokens: 1, output_tokens: 1, cache_read_tokens: 0, cache_creation_tokens: 0 }, model: 'openai:gpt-4o', providerId: 'openai' } satisfies ChatResult;
|
||||
});
|
||||
const result = await buildHandler(makeTools([]))(ctx);
|
||||
expect(result.result).toBe('handled');
|
||||
assertBalanced(captured);
|
||||
const stub = (captured[2].content as ChatBlock[])[0] as Extract<ChatBlock, { type: 'tool-result' }>;
|
||||
expect(stub.isError).toBe(true);
|
||||
expect(String(stub.output)).toContain('removed_tool');
|
||||
// Persisted as a failed exec so the next resume is stable.
|
||||
const rows = await engine.executeRaw<{ status: string }>(
|
||||
`SELECT status FROM subagent_tool_executions WHERE job_id = $1 AND tool_use_id = 'tc-gone'`, [jobId]);
|
||||
expect(rows[0].status).toBe('failed');
|
||||
});
|
||||
|
||||
it('terminal resume: a completed transcript returns its text without calling the model', async () => {
|
||||
const { jobId, ctx } = await makeJob('already done', 'openai:gpt-4o');
|
||||
await seedMessage(jobId, 0, 'user', [{ type: 'text', text: 'already done' }]);
|
||||
await seedMessage(jobId, 1, 'assistant', [{ type: 'text', text: 'the final answer' }]);
|
||||
|
||||
let chatCalls = 0;
|
||||
__setChatTransportForTests(async () => { chatCalls++; return { text: 'SHOULD NOT RUN', blocks: [] as ChatBlock[], stopReason: 'end',
|
||||
usage: { input_tokens: 0, output_tokens: 0, cache_read_tokens: 0, cache_creation_tokens: 0 }, model: 'openai:gpt-4o', providerId: 'openai' } satisfies ChatResult; });
|
||||
const result = await buildHandler(makeTools([]))(ctx);
|
||||
expect(chatCalls).toBe(0);
|
||||
expect(result.result).toBe('the final answer');
|
||||
expect(result.stop_reason).toBe('end_turn');
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,39 @@
|
||||
/**
|
||||
* Pack-driven extractable-type allowlist (unionExtractableTypes): honors the
|
||||
* schema-pack manifest's `extractable: true` flags while preserving the legacy
|
||||
* hardcoded floor and excluding synthesis outputs. Closes the D2 TODO in
|
||||
* extract-atoms.ts (page discovery was ignoring the pack's extractable flag, so
|
||||
* a type declared extractable — e.g. `note` — never actually extracted).
|
||||
*/
|
||||
import { describe, test, expect } from 'bun:test';
|
||||
import { unionExtractableTypes } from '../src/core/cycle/extract-atoms.ts';
|
||||
|
||||
const LEGACY = ['meeting', 'source', 'article', 'video', 'book', 'original'];
|
||||
|
||||
describe('unionExtractableTypes', () => {
|
||||
test('legacy floor is always present (back-compat)', () => {
|
||||
const r = unionExtractableTypes([]);
|
||||
for (const t of LEGACY) expect(r).toContain(t);
|
||||
});
|
||||
|
||||
test('pack-declared extractable types are added (e.g. note)', () => {
|
||||
const r = unionExtractableTypes(['note', 'writing']);
|
||||
expect(r).toContain('note');
|
||||
expect(r).toContain('writing');
|
||||
for (const t of LEGACY) expect(r).toContain(t);
|
||||
});
|
||||
|
||||
test('synthesis outputs are excluded even when the pack marks them extractable', () => {
|
||||
// gbrain-base declares `concept` extractable:true, but extracting atoms FROM
|
||||
// concepts would loop (concepts are synthesized from atoms).
|
||||
const r = unionExtractableTypes(['note', 'concept', 'atom']);
|
||||
expect(r).toContain('note');
|
||||
expect(r).not.toContain('concept');
|
||||
expect(r).not.toContain('atom');
|
||||
});
|
||||
|
||||
test('no duplicates when the pack repeats a legacy type', () => {
|
||||
const r = unionExtractableTypes(['meeting', 'source']);
|
||||
expect(r.filter((t) => t === 'meeting')).toHaveLength(1);
|
||||
});
|
||||
});
|
||||
@@ -50,7 +50,7 @@ function stubChat(text: string): (o: ChatOpts) => Promise<ChatResult> {
|
||||
|
||||
/**
|
||||
* Stub that returns a unique-title atom on each call so atoms write to
|
||||
* distinct slugs (`atoms/${date}/${slugify(title)}`) instead of upserting
|
||||
* distinct slugs (`atoms/<source-date>/<stem>-<title-hash>`) instead of upserting
|
||||
* into one row. Needed for tests that count atoms after multiple work items.
|
||||
*/
|
||||
function stubChatUnique(): (o: ChatOpts) => Promise<ChatResult> {
|
||||
@@ -108,12 +108,15 @@ async function seedPage(opts: {
|
||||
}
|
||||
|
||||
describe('v0.41.2.1: discoverExtractablePages SQL contract', () => {
|
||||
test('filters by all 6 extractable types', async () => {
|
||||
for (const type of ['meeting', 'source', 'article', 'video', 'book', 'original']) {
|
||||
test('discovers legacy + pack-extractable types, excludes synthesis outputs', async () => {
|
||||
// Legacy floor + `note` (declared extractable:true in gbrain-base, now
|
||||
// honored via the pack manifest — the D2 fix).
|
||||
for (const type of ['meeting', 'source', 'article', 'video', 'book', 'original', 'note']) {
|
||||
await seedPage({ slug: `${type}/x`, type });
|
||||
}
|
||||
// Add a non-extractable page that should NOT appear
|
||||
await seedPage({ slug: 'notes/skip-me', type: 'note' });
|
||||
// `concept` is also extractable:true in gbrain-base, but extracting atoms
|
||||
// FROM concepts would loop — synthesis outputs are always excluded.
|
||||
await seedPage({ slug: 'wiki/concepts/skip-me', type: 'concept' });
|
||||
|
||||
const discovered = await discoverExtractablePages(engine, 'default');
|
||||
const slugs = discovered.map((d) => d.slug).sort();
|
||||
@@ -121,6 +124,7 @@ describe('v0.41.2.1: discoverExtractablePages SQL contract', () => {
|
||||
'article/x',
|
||||
'book/x',
|
||||
'meeting/x',
|
||||
'note/x',
|
||||
'original/x',
|
||||
'source/x',
|
||||
'video/x',
|
||||
@@ -339,6 +343,41 @@ describe('v0.41.2.1: runPhaseExtractAtoms — dual-source merge + idempotency',
|
||||
expect(after[0].count).toBe(before[0].count);
|
||||
});
|
||||
|
||||
test('deterministic slug: source-dated + title-hashed, trailing dash stripped, re-extract upserts (no cross-day twin)', async () => {
|
||||
// 16 three-letter words → slugifySegment output truncates ON a hyphen at the
|
||||
// 60-char cut, exercising the trailing-dash re-strip (Bug A).
|
||||
const title = 'aaa bbb ccc ddd eee fff ggg hhh iii jjj kkk lll mmm nnn ooo ppp';
|
||||
const chat = stubChat(`[{"title":"${title}","atom_type":"insight","body":"b"}]`);
|
||||
// Transcript filename carries a date DIFFERENT from the run date, so a
|
||||
// source-dated slug is observably distinct from the old run-date one.
|
||||
const filePath = '/srv/transcripts/2026-06-12-telegram.md';
|
||||
await runPhaseExtractAtoms(engine, {
|
||||
_transcripts: [{ filePath, content: 'first', contentHash: 'aaaa1111bbbb2222' }],
|
||||
_pages: [],
|
||||
_chat: chat,
|
||||
});
|
||||
// Same file, GROWN content (append-only) → different contentHash, so the
|
||||
// source-hash fast-path does NOT skip and the atom is re-extracted. Pre-fix
|
||||
// this minted a second atom under a new run-date prefix (Bug B); the
|
||||
// source-dated, title-hashed slug must upsert into the same row instead.
|
||||
await runPhaseExtractAtoms(engine, {
|
||||
_transcripts: [{ filePath, content: 'first plus appended', contentHash: 'cccc3333dddd4444' }],
|
||||
_pages: [],
|
||||
_chat: chat,
|
||||
});
|
||||
const rows = await engine.executeRaw<{ slug: string }>(
|
||||
`SELECT slug FROM pages WHERE type = 'atom'`,
|
||||
);
|
||||
expect(rows.length).toBe(1); // upsert, not a cross-day duplicate
|
||||
const slug = rows[0].slug;
|
||||
expect(slug.startsWith('atoms/2026-06-12/')).toBe(true); // SOURCE date, not run date
|
||||
expect(slug).toMatch(/-[0-9a-f]{6}$/); // 6-char title-hash suffix
|
||||
expect(slug).not.toContain('--'); // trailing dash stripped before -<hash>
|
||||
const stem = slug.slice('atoms/2026-06-12/'.length).replace(/-[0-9a-f]{6}$/, '');
|
||||
expect(stem.endsWith('-')).toBe(false);
|
||||
expect(stem.length).toBeLessThanOrEqual(60);
|
||||
});
|
||||
|
||||
test('PhaseResult.details has additive page fields populated', async () => {
|
||||
const chat = stubChat(`[{"title":"x","atom_type":"insight","body":"b"}]`);
|
||||
const result = await runPhaseExtractAtoms(engine, {
|
||||
|
||||
@@ -0,0 +1,87 @@
|
||||
/**
|
||||
* Bootstrap progression regression test.
|
||||
*
|
||||
* extractTakesFromPages selected pages by updated_at DESC + LIMIT with no
|
||||
* exclusion of pages that already hold takes — so on a corpus larger than
|
||||
* one run's cap (the CLI clamps --max-pages to 1000), every re-run rescanned
|
||||
* the same most-recent slice: the older tail could never be bootstrapped,
|
||||
* and each rescan re-spent LLM budget producing upsert-identical rows.
|
||||
* Seen live on a 2,311-eligible-page brain where the second run would have
|
||||
* covered 0 new pages.
|
||||
*
|
||||
* Pins: covered pages are skipped by default (runs progress), and
|
||||
* includeCovered restores the full rescan (refresh semantics).
|
||||
*/
|
||||
import { describe, test, expect, beforeAll, afterAll } from 'bun:test';
|
||||
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
|
||||
import {
|
||||
configureGateway,
|
||||
resetGateway,
|
||||
__setChatTransportForTests,
|
||||
} from '../src/core/ai/gateway.ts';
|
||||
import { extractTakesFromPages } from '../src/core/extract-takes-from-pages.ts';
|
||||
|
||||
let engine: PGLiteEngine;
|
||||
|
||||
beforeAll(async () => {
|
||||
engine = new PGLiteEngine();
|
||||
await engine.connect({});
|
||||
await engine.initSchema();
|
||||
|
||||
configureGateway({
|
||||
chat_model: 'anthropic:claude-haiku-4-5-20251001',
|
||||
env: { ANTHROPIC_API_KEY: 'sk-ant-test-takes-progression' },
|
||||
});
|
||||
__setChatTransportForTests(async () => ({
|
||||
text: '[{"claim":"a stubbed claim","kind":"take","weight":0.7}]',
|
||||
blocks: [{ type: 'text' as const, text: '[{"claim":"a stubbed claim","kind":"take","weight":0.7}]' }],
|
||||
stopReason: 'end' as const,
|
||||
usage: { input_tokens: 1, output_tokens: 1, cache_read_tokens: 0, cache_creation_tokens: 0 },
|
||||
model: 'anthropic:claude-haiku-4-5-20251001',
|
||||
providerId: 'anthropic',
|
||||
}));
|
||||
|
||||
const body = 'An opinion-bearing body long enough to clear the 200-char eligibility floor. '.repeat(5);
|
||||
await engine.putPage('concepts/progression-a', {
|
||||
type: 'concept', title: 'A', compiled_truth: body, frontmatter: {},
|
||||
});
|
||||
await engine.putPage('concepts/progression-b', {
|
||||
type: 'concept', title: 'B', compiled_truth: body, frontmatter: {},
|
||||
});
|
||||
});
|
||||
|
||||
afterAll(async () => {
|
||||
__setChatTransportForTests(null);
|
||||
resetGateway();
|
||||
await engine.disconnect();
|
||||
});
|
||||
|
||||
describe('extractTakesFromPages — bootstrap progression', () => {
|
||||
test('first run covers the eligible pages', async () => {
|
||||
const r1 = await extractTakesFromPages(engine, { bootstrapEnabled: true, maxPages: 50 });
|
||||
expect(r1.pages_scanned).toBe(2);
|
||||
expect(r1.claims_extracted).toBe(2);
|
||||
});
|
||||
|
||||
test('second run skips covered pages — repeat runs progress instead of rescanning', async () => {
|
||||
const r2 = await extractTakesFromPages(engine, { bootstrapEnabled: true, maxPages: 50 });
|
||||
expect(r2.pages_scanned).toBe(0);
|
||||
expect(r2.claims_extracted).toBe(0);
|
||||
});
|
||||
|
||||
test('a page added after the first run is picked up (progression, not a frozen set)', async () => {
|
||||
const body = 'Another opinion-bearing body long enough to clear the eligibility floor. '.repeat(5);
|
||||
await engine.putPage('concepts/progression-c', {
|
||||
type: 'concept', title: 'C', compiled_truth: body, frontmatter: {},
|
||||
});
|
||||
const r3 = await extractTakesFromPages(engine, { bootstrapEnabled: true, maxPages: 50 });
|
||||
expect(r3.pages_scanned).toBe(1);
|
||||
});
|
||||
|
||||
test('includeCovered rescans everything (refresh semantics)', async () => {
|
||||
const r4 = await extractTakesFromPages(engine, {
|
||||
bootstrapEnabled: true, maxPages: 50, includeCovered: true,
|
||||
});
|
||||
expect(r4.pages_scanned).toBe(3);
|
||||
});
|
||||
});
|
||||
@@ -135,6 +135,56 @@ describe('extractTimelineFromContent', () => {
|
||||
const entries = extractTimelineFromContent(content, 'test');
|
||||
expect(entries).toHaveLength(1);
|
||||
});
|
||||
|
||||
it('extracts inline citation format entries', () => {
|
||||
const content = `Closed the seed round with fund-a leading. [Source: board meeting notes, 2025-04-02]`;
|
||||
const entries = extractTimelineFromContent(content, 'deals/acme-seed');
|
||||
expect(entries).toHaveLength(1);
|
||||
expect(entries[0].date).toBe('2025-04-02');
|
||||
expect(entries[0].source).toBe('board meeting notes');
|
||||
expect(entries[0].summary).toBe('Closed the seed round with fund-a leading.');
|
||||
});
|
||||
|
||||
it('keeps commas inside the citation source', () => {
|
||||
const content = `Alice joined as CTO. [Source: email from alice-example re: offer, signed, 2025-05-10]`;
|
||||
const entries = extractTimelineFromContent(content, 'people/alice-example');
|
||||
expect(entries).toHaveLength(1);
|
||||
expect(entries[0].date).toBe('2025-05-10');
|
||||
expect(entries[0].source).toBe('email from alice-example re: offer, signed');
|
||||
});
|
||||
|
||||
it('extracts one entry per citation when a line carries several', () => {
|
||||
const content = `Both sides confirmed the partnership. [Source: call with widget-co, 2025-06-01] [Source: follow-up email, 2025-06-03]`;
|
||||
const entries = extractTimelineFromContent(content, 'companies/widget-co');
|
||||
expect(entries).toHaveLength(2);
|
||||
expect(entries[0].date).toBe('2025-06-01');
|
||||
expect(entries[1].date).toBe('2025-06-03');
|
||||
expect(entries[0].summary).toBe(entries[1].summary);
|
||||
});
|
||||
|
||||
it('does not double-extract a timeline bullet that carries its own citation', () => {
|
||||
const content = `- **2025-03-18** | Meeting — Discussed partnership [Source: meeting notes, 2025-03-18]`;
|
||||
const entries = extractTimelineFromContent(content, 'test');
|
||||
expect(entries).toHaveLength(1); // Format 1 only
|
||||
expect(entries[0].source).toBe('Meeting');
|
||||
});
|
||||
|
||||
it('skips a bare citation with no surrounding text', () => {
|
||||
const content = `[Source: import batch, 2025-07-01]`;
|
||||
expect(extractTimelineFromContent(content, 'test')).toHaveLength(0);
|
||||
});
|
||||
|
||||
it('ignores citations without a date', () => {
|
||||
const content = `Some claim here. [Source: undated memo]`;
|
||||
expect(extractTimelineFromContent(content, 'test')).toHaveLength(0);
|
||||
});
|
||||
|
||||
it('strips list markers from the citation summary', () => {
|
||||
const content = `- Landed the enterprise pilot with acme-example. [Source: CRM update, 2025-08-15]`;
|
||||
const entries = extractTimelineFromContent(content, 'companies/acme-example');
|
||||
expect(entries).toHaveLength(1);
|
||||
expect(entries[0].summary).toBe('Landed the enterprise pilot with acme-example.');
|
||||
});
|
||||
});
|
||||
|
||||
describe('walkMarkdownFiles', () => {
|
||||
|
||||
@@ -136,4 +136,43 @@ echo hi
|
||||
const fenceChunks = chunks.filter(c => c.chunk_source === 'fenced_code');
|
||||
expect(fenceChunks.length).toBe(0);
|
||||
});
|
||||
|
||||
// #2437 — extractFencedChunks skips marked.lexer entirely when the body has
|
||||
// no fence marker (the lexer transiently allocates ~60x the page size, which
|
||||
// OOMs the import worker under memory pressure). These two guard that the
|
||||
// fast-path neither breaks tilde fences nor lexes fence-less pages.
|
||||
test('tilde-fenced (~~~) code is still extracted after the no-fence fast-path (#2437)', async () => {
|
||||
const md = 'Docs.\n\n~~~ts\nexport const x = 1;\n~~~\n';
|
||||
await importFromContent(engine, 'guides/fence-tilde', md, { noEmbed: true });
|
||||
const chunks = await engine.getChunks('guides/fence-tilde');
|
||||
const fenceChunks = chunks.filter(c => c.chunk_source === 'fenced_code');
|
||||
expect(fenceChunks.length).toBeGreaterThan(0);
|
||||
expect(fenceChunks[0]!.language).toBe('typescript');
|
||||
});
|
||||
|
||||
test('CR-only (\\r) line endings still extract a fenced chunk (#2437)', async () => {
|
||||
// marked normalizes \r → \n before lexing, so the fast-path's fence probe
|
||||
// must too; otherwise a classic-Mac line-ended page loses its fenced code.
|
||||
const md = 'intro\r```ts\rexport const x = 1;\r```\r';
|
||||
await importFromContent(engine, 'guides/fence-cr', md, { noEmbed: true });
|
||||
const chunks = await engine.getChunks('guides/fence-cr');
|
||||
const fenceChunks = chunks.filter(c => c.chunk_source === 'fenced_code');
|
||||
expect(fenceChunks.length).toBeGreaterThan(0);
|
||||
expect(fenceChunks[0]!.language).toBe('typescript');
|
||||
});
|
||||
|
||||
test('large fence-less table page imports with zero fenced chunks (no lexer pass) (#2437)', async () => {
|
||||
const cols = 16;
|
||||
const header = '| ' + Array.from({ length: cols }, (_, i) => 'col' + i).join(' | ') + ' |';
|
||||
const sep = '| ' + Array.from({ length: cols }, () => '---').join(' | ') + ' |';
|
||||
const body = Array.from({ length: 2000 }, (_, r) =>
|
||||
'| ' + Array.from({ length: cols }, (_, c) => 'v' + r + '_' + c).join(' | ') + ' |').join('\n');
|
||||
const md = '# Overview\n\n' + header + '\n' + sep + '\n' + body + '\n';
|
||||
// sanity: the page is genuinely fence-less, so the fast-path applies
|
||||
expect(/(^|[\r\n])[ \t]{0,3}(```|~~~)/.test(md)).toBe(false);
|
||||
await importFromContent(engine, 'guides/fence-less-table', md, { noEmbed: true });
|
||||
const chunks = await engine.getChunks('guides/fence-less-table');
|
||||
const fenceChunks = chunks.filter(c => c.chunk_source === 'fenced_code');
|
||||
expect(fenceChunks.length).toBe(0);
|
||||
});
|
||||
});
|
||||
|
||||
+35
-1
@@ -1,10 +1,12 @@
|
||||
import { describe, test, expect, beforeAll, afterAll } from 'bun:test';
|
||||
import { describe, test, expect, beforeAll, afterAll, spyOn } from 'bun:test';
|
||||
import { writeFileSync, mkdirSync, rmSync, symlinkSync, mkdtempSync } from 'fs';
|
||||
import { join, basename } from 'path';
|
||||
import { createHash } from 'crypto';
|
||||
import { extname } from 'path';
|
||||
import { tmpdir } from 'os';
|
||||
import { collectFiles } from '../src/commands/files.ts';
|
||||
import { operationsByName } from '../src/core/operations.ts';
|
||||
import * as db from '../src/core/db.ts';
|
||||
|
||||
const TMP = join(import.meta.dir, '.tmp-files-test');
|
||||
|
||||
@@ -183,6 +185,38 @@ describe('collectFiles (production import)', () => {
|
||||
}
|
||||
});
|
||||
|
||||
test('file_list normalizes BigInt size_bytes for JSON serialization', async () => {
|
||||
// Postgres BIGINT(size_bytes) returns native BigInt under postgres.js's
|
||||
// {bigint: postgres.BigInt} type map. Both JSON.stringify (MCP) and the
|
||||
// CLI's `size_bytes / 1024` divide trip on it. Regression for the bug
|
||||
// openclaw's agent surfaced in v0.22.4.
|
||||
const fakeRows = [
|
||||
{ id: 1, page_slug: 'a', filename: 'f1', storage_path: 'a/f1',
|
||||
mime_type: 'text/plain', size_bytes: 4096n, content_hash: 'h1',
|
||||
created_at: '2026-04-27' },
|
||||
{ id: 2, page_slug: 'a', filename: 'f2', storage_path: 'a/f2',
|
||||
mime_type: null, size_bytes: null, content_hash: 'h2',
|
||||
created_at: '2026-04-27' },
|
||||
];
|
||||
const fakeSql: any = (..._: unknown[]) => Promise.resolve(fakeRows);
|
||||
const spy = spyOn(db, 'getConnection').mockReturnValue(fakeSql);
|
||||
|
||||
try {
|
||||
const op = operationsByName['file_list'];
|
||||
const ctx: any = { engine: null, config: {}, logger: { info() {}, warn() {}, error() {} }, dryRun: false, remote: true };
|
||||
const result = await op.handler(ctx, {}) as Array<Record<string, unknown>>;
|
||||
|
||||
expect(result.length).toBe(2);
|
||||
expect(typeof result[0].size_bytes).toBe('number');
|
||||
expect(result[0].size_bytes).toBe(4096);
|
||||
expect(result[1].size_bytes).toBeNull();
|
||||
// The exact failure mode openclaw reported.
|
||||
expect(() => JSON.stringify(result)).not.toThrow();
|
||||
} finally {
|
||||
spy.mockRestore();
|
||||
}
|
||||
});
|
||||
|
||||
test('collectFiles skips node_modules', () => {
|
||||
const tmpDir = mkdtempSync(join(tmpdir(), 'gbrain-nodemod-'));
|
||||
try {
|
||||
|
||||
@@ -107,6 +107,63 @@ describe('toModelMessages — v6 ModelMessage shape', () => {
|
||||
]);
|
||||
});
|
||||
|
||||
test('Date in tool-result json output serializes to ISO string (Postgres timestamptz)', () => {
|
||||
// node-postgres returns timestamptz columns as JS Date; AI SDK v6's
|
||||
// JSONValue schema rejects a raw Date, dead-lettering the tool loop.
|
||||
const msgs: ChatMessage[] = [
|
||||
{
|
||||
role: 'user',
|
||||
content: [{
|
||||
type: 'tool-result',
|
||||
toolCallId: 'c1',
|
||||
toolName: 'brain_get_page',
|
||||
output: { rows: [{ updated_at: new Date('2026-06-26T06:56:59.000Z'), nested: { created_at: new Date('2026-01-02T03:04:05.000Z') } }] },
|
||||
}],
|
||||
},
|
||||
];
|
||||
const out = toModelMessages(msgs) as any[];
|
||||
const value = out[0].content[0].output.value;
|
||||
expect(out[0].content[0].output.type).toBe('json');
|
||||
expect(value.rows[0].updated_at).toBe('2026-06-26T06:56:59.000Z');
|
||||
expect(value.rows[0].nested.created_at).toBe('2026-01-02T03:04:05.000Z');
|
||||
// No Date instance survives (would throw in AI SDK v6).
|
||||
expect(value.rows[0].updated_at instanceof Date).toBe(false);
|
||||
});
|
||||
|
||||
test('non-string text block is dropped (reasoning-model null-text guard)', () => {
|
||||
// DeepSeek v4 / reasoning models can emit text:null/undefined thinking
|
||||
// parts; AI SDK v6 rejects them. Dropped here; tool-call sibling kept.
|
||||
const msgs: ChatMessage[] = [
|
||||
{
|
||||
role: 'assistant',
|
||||
content: [
|
||||
{ type: 'text', text: null as unknown as string },
|
||||
{ type: 'text', text: undefined as unknown as string },
|
||||
{ type: 'text', text: 'kept' },
|
||||
{ type: 'text', text: '' }, // empty string is valid — kept
|
||||
{ type: 'tool-call', toolCallId: 'c1', toolName: 'search', input: {} },
|
||||
],
|
||||
},
|
||||
];
|
||||
const out = toModelMessages(msgs) as any[];
|
||||
expect(out[0].content).toEqual([
|
||||
{ type: 'text', text: 'kept' },
|
||||
{ type: 'text', text: '' },
|
||||
{ type: 'tool-call', toolCallId: 'c1', toolName: 'search', input: {} },
|
||||
]);
|
||||
});
|
||||
|
||||
test('errored tool-result never throws on circular/bigint output (safeStringify)', () => {
|
||||
const circular: any = {};
|
||||
circular.self = circular;
|
||||
const msgs: ChatMessage[] = [
|
||||
{ role: 'user', content: [{ type: 'tool-result', toolCallId: 'c1', toolName: 'x', output: circular, isError: true }] },
|
||||
];
|
||||
const out = toModelMessages(msgs) as any[];
|
||||
expect(out[0].content[0].output.type).toBe('error-text');
|
||||
expect(typeof out[0].content[0].output.value).toBe('string');
|
||||
});
|
||||
|
||||
test('full multi-turn conversation: user → assistant(tool-call) → tool(result)', () => {
|
||||
const msgs: ChatMessage[] = [
|
||||
{ role: 'user', content: 'find widget' },
|
||||
|
||||
@@ -120,6 +120,29 @@ describe('importImageFile happy path (noEmbed)', () => {
|
||||
expect(r2.status).toBe('skipped');
|
||||
});
|
||||
|
||||
test('routes page, chunks, and file metadata to the requested source (#2706)', async () => {
|
||||
await engine.executeRaw(
|
||||
`INSERT INTO sources (id, name) VALUES ($1, $2) ON CONFLICT (id) DO NOTHING`,
|
||||
['image-source', 'Image Source'],
|
||||
);
|
||||
const target = join(tmpDir, 'source-photo.png');
|
||||
writeFileSync(target, Buffer.from('fake-png-bytes-source-scoped'));
|
||||
|
||||
const result = await importImageFile(engine, target, 'photos/source-photo.png', {
|
||||
noEmbed: true,
|
||||
sourceId: 'image-source',
|
||||
});
|
||||
|
||||
expect(result.status).toBe('imported');
|
||||
expect(await engine.getPage('photos/source-photo.png', { sourceId: 'default' })).toBeNull();
|
||||
const page = await engine.getPage('photos/source-photo.png', { sourceId: 'image-source' });
|
||||
expect(page).not.toBeNull();
|
||||
expect(await engine.getChunks('photos/source-photo.png', { sourceId: 'default' })).toHaveLength(0);
|
||||
expect(await engine.getChunks('photos/source-photo.png', { sourceId: 'image-source' })).toHaveLength(1);
|
||||
expect(await engine.getFile('default', 'photos/source-photo.png')).toBeNull();
|
||||
expect(await engine.getFile('image-source', 'photos/source-photo.png')).not.toBeNull();
|
||||
});
|
||||
|
||||
test('refuses oversized files (>20MB)', async () => {
|
||||
const target = join(tmpDir, 'huge.png');
|
||||
// Write a 21MB file. Buffer.alloc is fast.
|
||||
|
||||
@@ -1265,3 +1265,29 @@ describe("v0.18.0 migration v22 — links_resolution_type", () => {
|
||||
});
|
||||
});
|
||||
|
||||
|
||||
describe('parseTimelineEntries — Format 3: inline [Source: ..., YYYY-MM-DD] citations', () => {
|
||||
test('extracts an entry from a dated citation', () => {
|
||||
const entries = parseTimelineEntries('Closed the seed round. [Source: board notes, 2025-04-02]');
|
||||
expect(entries).toHaveLength(1);
|
||||
expect(entries[0].date).toBe('2025-04-02');
|
||||
expect(entries[0].summary).toBe('Closed the seed round.');
|
||||
expect(entries[0].detail).toBe('Source: board notes');
|
||||
});
|
||||
|
||||
test('keeps commas inside the citation source', () => {
|
||||
const entries = parseTimelineEntries('Alice joined. [Source: email re: offer, signed, 2025-05-10]');
|
||||
expect(entries).toHaveLength(1);
|
||||
expect(entries[0].detail).toBe('Source: email re: offer, signed');
|
||||
});
|
||||
|
||||
test('does not double-extract a timeline bullet carrying its own citation', () => {
|
||||
const entries = parseTimelineEntries('- **2025-03-18** | Meeting notes [Source: notes, 2025-03-18]');
|
||||
expect(entries).toHaveLength(1); // bullet pass only
|
||||
});
|
||||
|
||||
test('skips invalid calendar dates and bare citations', () => {
|
||||
expect(parseTimelineEntries('Claim. [Source: memo, 2026-13-45]')).toHaveLength(0);
|
||||
expect(parseTimelineEntries('[Source: import batch, 2025-07-01]')).toHaveLength(0);
|
||||
});
|
||||
});
|
||||
|
||||
@@ -43,6 +43,14 @@ describe('CANONICAL_PRICING — table integrity', () => {
|
||||
expect(CANONICAL_PRICING['anthropic:claude-opus-4-7']).toEqual({ input: 5.0, output: 25.0 });
|
||||
});
|
||||
|
||||
test('Sonnet 5 present at $3/$15 (standard rate, intro discount not modeled)', () => {
|
||||
expect(CANONICAL_PRICING['anthropic:claude-sonnet-5']).toEqual({ input: 3.0, output: 15.0 });
|
||||
});
|
||||
|
||||
test('Fable 5 present at $10/$50', () => {
|
||||
expect(CANONICAL_PRICING['anthropic:claude-fable-5']).toEqual({ input: 10.0, output: 50.0 });
|
||||
});
|
||||
|
||||
test('Gemini 2.0 Flash reconciled to $0.10/$0.40; legacy alias agrees', () => {
|
||||
expect(CANONICAL_PRICING['google:gemini-2.0-flash']).toEqual({ input: 0.1, output: 0.4 });
|
||||
expect(CANONICAL_PRICING['google:gemini-2-flash']).toEqual(
|
||||
|
||||
@@ -169,6 +169,7 @@ describe('shouldExclude — orphan filter regression (preserve curation)', () =>
|
||||
|
||||
test('raw segment is excluded', () => {
|
||||
expect(shouldExclude('media/x/raw/post')).toBe(true);
|
||||
expect(shouldExclude('raw/chats/claude-code/session')).toBe(true);
|
||||
});
|
||||
|
||||
test('deny-prefixes are excluded', () => {
|
||||
@@ -183,6 +184,8 @@ describe('shouldExclude — orphan filter regression (preserve curation)', () =>
|
||||
expect(shouldExclude('thoughts/today')).toBe(true);
|
||||
expect(shouldExclude('catalog/movies')).toBe(true);
|
||||
expect(shouldExclude('entities/anonymous')).toBe(true);
|
||||
expect(shouldExclude('atoms/fact-123')).toBe(true);
|
||||
expect(shouldExclude('skills/gbrain-operations')).toBe(true);
|
||||
});
|
||||
|
||||
test('regular slugs are NOT excluded', () => {
|
||||
|
||||
@@ -19,6 +19,7 @@ import { PGLiteEngine } from '../src/core/pglite-engine.ts';
|
||||
import { SemanticQueryCache, cacheRowId } from '../src/core/search/query-cache.ts';
|
||||
import type { SearchResult } from '../src/core/types.ts';
|
||||
import { knobsHash, resolveSearchMode } from '../src/core/search/mode.ts';
|
||||
import { resolveHardExcludes } from '../src/core/search/source-boost.ts';
|
||||
import { configureGateway, resetGateway } from '../src/core/ai/gateway.ts';
|
||||
|
||||
let engine: PGLiteEngine;
|
||||
@@ -230,3 +231,49 @@ describe('SemanticQueryCache cross-mode isolation (CDX-4 hotfix)', () => {
|
||||
expect(conservativeHit.hit).toBe(false);
|
||||
});
|
||||
});
|
||||
|
||||
describe('hard-exclude cache isolation (#2825)', () => {
|
||||
// Hashes computed the way hybridSearchCached does: same resolved mode, ctx
|
||||
// carrying the resolved hard-exclude list. A row written by a process
|
||||
// WITHOUT GBRAIN_SEARCH_EXCLUDE (defaults only) must not be served to a
|
||||
// process WITH it, and vice versa.
|
||||
const noEnvHash = knobsHash(resolveSearchMode({ mode: 'balanced' }), {
|
||||
hardExcludes: resolveHardExcludes(undefined, undefined, undefined),
|
||||
});
|
||||
const envExcludeHash = knobsHash(resolveSearchMode({ mode: 'balanced' }), {
|
||||
hardExcludes: resolveHardExcludes(undefined, undefined, 'private/'),
|
||||
});
|
||||
|
||||
test('row written without excludes is NOT served to a lookup with excludes', async () => {
|
||||
const cache = new SemanticQueryCache(engine);
|
||||
const emb = makeEmbedding(6);
|
||||
|
||||
// Simulate a no-exclude process writing results that include a slug the
|
||||
// excluding process must never see.
|
||||
const leaky = makeResults('private', 5);
|
||||
await cache.store('who is alice', emb, leaky, {
|
||||
vector_enabled: true, detail_resolved: null, expansion_applied: false,
|
||||
}, { knobsHash: noEnvHash });
|
||||
|
||||
// Excluding process → MISS (falls through to a fresh, filtered query).
|
||||
const excluded = await cache.lookup(emb, { knobsHash: envExcludeHash });
|
||||
expect(excluded.hit).toBe(false);
|
||||
|
||||
// Original no-exclude process still hits its own row.
|
||||
const original = await cache.lookup(emb, { knobsHash: noEnvHash });
|
||||
expect(original.hit).toBe(true);
|
||||
expect(original.results?.length).toBe(5);
|
||||
});
|
||||
|
||||
test('row written WITH excludes is not served back once excludes are lifted', async () => {
|
||||
const cache = new SemanticQueryCache(engine);
|
||||
const emb = makeEmbedding(7);
|
||||
|
||||
await cache.store('who is alice', emb, makeResults('filtered', 3), {
|
||||
vector_enabled: true, detail_resolved: null, expansion_applied: false,
|
||||
}, { knobsHash: envExcludeHash });
|
||||
|
||||
expect((await cache.lookup(emb, { knobsHash: noEnvHash })).hit).toBe(false);
|
||||
expect((await cache.lookup(emb, { knobsHash: envExcludeHash })).hit).toBe(true);
|
||||
});
|
||||
});
|
||||
|
||||
@@ -89,7 +89,7 @@ describe('alias_resolved boost stage', () => {
|
||||
});
|
||||
|
||||
describe('KNOBS_HASH_VERSION', () => {
|
||||
it('is 11 (10→11 asymmetric input_type fix invalidates document-side query-vector rows, #1400)', () => {
|
||||
expect(KNOBS_HASH_VERSION).toBe(11);
|
||||
it('is 12 (11→12 hard-exclude fold invalidates rows written under a different exclude policy, #2825)', () => {
|
||||
expect(KNOBS_HASH_VERSION).toBe(12);
|
||||
});
|
||||
});
|
||||
|
||||
@@ -407,7 +407,10 @@ describe('knobsHash determinism + cross-mode separation (CDX-4)', () => {
|
||||
// now produces query-side vectors for asymmetric providers (zembed-1,
|
||||
// Voyage v3+), so rows keyed on pre-fix document-side query vectors
|
||||
// must not be served to post-fix lookups.
|
||||
expect(KNOBS_HASH_VERSION).toBe(11);
|
||||
// #2825: bumped 11→12 to fold the resolved hard-exclude prefix list
|
||||
// (hx=) — cached rows leaked GBRAIN_SEARCH_EXCLUDE'd slugs across
|
||||
// processes.
|
||||
expect(KNOBS_HASH_VERSION).toBe(12);
|
||||
});
|
||||
|
||||
test('T1 (codex): floor_ratio set vs unset produces DIFFERENT hashes (cache contamination prevention)', () => {
|
||||
@@ -572,8 +575,8 @@ describe('v0.40.4 — graph_signals knob', () => {
|
||||
});
|
||||
|
||||
describe('v0.42.3.0 — autocut knobs', () => {
|
||||
test('KNOBS_HASH_VERSION is 11 (10→11 asymmetric input_type fix, #1400)', () => {
|
||||
expect(KNOBS_HASH_VERSION).toBe(11);
|
||||
test('KNOBS_HASH_VERSION is 12 (11→12 hard-exclude fold, #2825)', () => {
|
||||
expect(KNOBS_HASH_VERSION).toBe(12);
|
||||
});
|
||||
|
||||
test('bundle defaults: conservative off, balanced/tokenmax on @0.20', () => {
|
||||
|
||||
@@ -27,6 +27,7 @@ import {
|
||||
MODE_BUNDLES,
|
||||
type ResolvedSearchKnobs,
|
||||
} from '../../src/core/search/mode.ts';
|
||||
import { resolveHardExcludes } from '../../src/core/search/source-boost.ts';
|
||||
|
||||
/** Build a baseline resolved knob set with all reranker fields filled. */
|
||||
function baseKnobs(): ResolvedSearchKnobs {
|
||||
@@ -43,7 +44,7 @@ function baseKnobs(): ResolvedSearchKnobs {
|
||||
}
|
||||
|
||||
describe('KNOBS_HASH_VERSION + version invariants', () => {
|
||||
test('version is 11 (…; 8→9 archive-demote #1777; 9→10 relational recall; 10→11 asymmetric input_type #1400)', () => {
|
||||
test('version is 12 (…; 9→10 relational recall; 10→11 asymmetric input_type #1400; 11→12 hard-excludes #2825)', () => {
|
||||
// v0.35.0.0: 1→2 to fold reranker fields. v0.35.6.0: 2→3 to fold
|
||||
// floor_ratio. v0.36 wave: piggybacks on v=3 with 7 cross-modal knobs
|
||||
// (D2) PLUS column + provider context (D8/CDX-2 cross-column isolation).
|
||||
@@ -61,7 +62,9 @@ describe('KNOBS_HASH_VERSION + version invariants', () => {
|
||||
// #1400: 10→11 asymmetric input_type fix — embedQuery() now produces
|
||||
// query-side vectors for asymmetric providers, so rows keyed on
|
||||
// pre-fix document-side query vectors must not be served.
|
||||
expect(KNOBS_HASH_VERSION).toBe(11);
|
||||
// #2825: 11→12 to fold the resolved hard-exclude prefix list (hx=) —
|
||||
// cached rows leaked GBRAIN_SEARCH_EXCLUDE'd slugs across processes.
|
||||
expect(KNOBS_HASH_VERSION).toBe(12);
|
||||
});
|
||||
|
||||
test('hash is 16 hex chars regardless of reranker config', () => {
|
||||
@@ -208,3 +211,36 @@ describe('append-only convention (CDX2-F13)', () => {
|
||||
expect(bare).toBe(explicit);
|
||||
});
|
||||
});
|
||||
|
||||
describe('v=12 hard-exclude participation (#2825)', () => {
|
||||
test('different exclude lists → different hashes', () => {
|
||||
const k = baseKnobs();
|
||||
const noEnv = knobsHash(k, { hardExcludes: resolveHardExcludes(undefined, undefined, undefined) });
|
||||
const withEnv = knobsHash(k, { hardExcludes: resolveHardExcludes(undefined, undefined, 'private/') });
|
||||
expect(noEnv).not.toBe(withEnv);
|
||||
});
|
||||
|
||||
test('include (opt-back-in) changes the hash too', () => {
|
||||
const k = baseKnobs();
|
||||
const a = knobsHash(k, { hardExcludes: resolveHardExcludes(undefined, undefined, undefined) });
|
||||
const b = knobsHash(k, { hardExcludes: resolveHardExcludes(undefined, ['test/'], undefined) });
|
||||
expect(a).not.toBe(b);
|
||||
});
|
||||
|
||||
test('same prefixes in different input order → SAME hash (normalization)', () => {
|
||||
const k = baseKnobs();
|
||||
const a = knobsHash(k, { hardExcludes: ['a/', 'b/', 'test/'] });
|
||||
const b = knobsHash(k, { hardExcludes: ['test/', 'b/', 'a/'] });
|
||||
expect(a).toBe(b);
|
||||
});
|
||||
|
||||
test('undefined hardExcludes is stable (legacy-caller fallback)', () => {
|
||||
const k = baseKnobs();
|
||||
expect(knobsHash(k)).toBe(knobsHash(k));
|
||||
// ...and distinct from an explicit resolved default list — a legacy
|
||||
// caller can never collide with a policy-carrying cache row.
|
||||
expect(knobsHash(k)).not.toBe(
|
||||
knobsHash(k, { hardExcludes: resolveHardExcludes(undefined, undefined, undefined) }),
|
||||
);
|
||||
});
|
||||
});
|
||||
|
||||
@@ -7,7 +7,7 @@
|
||||
* the rule can't drift without the suite catching it.
|
||||
*/
|
||||
import { describe, test, expect } from 'bun:test';
|
||||
import { resolveBootstrapToken } from '../src/commands/serve-http.ts';
|
||||
import { resolveBootstrapToken, shouldSuppressBootstrapPrint } from '../src/commands/serve-http.ts';
|
||||
|
||||
describe('resolveBootstrapToken (v0.36.1.x #1024)', () => {
|
||||
test('unset env → generates a fresh token via the injected RNG', () => {
|
||||
@@ -72,3 +72,27 @@ describe('resolveBootstrapToken (v0.36.1.x #1024)', () => {
|
||||
expect(r.kind).toBe('error');
|
||||
});
|
||||
});
|
||||
|
||||
describe('shouldSuppressBootstrapPrint (#2624 log-leak default)', () => {
|
||||
const base = { suppress: false, fromEnv: false, forcePrint: false, isTty: true };
|
||||
|
||||
test('generated token on non-TTY (container) → hidden by default', () => {
|
||||
expect(shouldSuppressBootstrapPrint({ ...base, isTty: false })).toBe(true);
|
||||
});
|
||||
|
||||
test('generated token on interactive TTY → printed', () => {
|
||||
expect(shouldSuppressBootstrapPrint({ ...base, isTty: true })).toBe(false);
|
||||
});
|
||||
|
||||
test('--print-admin-token forces raw value on non-TTY', () => {
|
||||
expect(shouldSuppressBootstrapPrint({ ...base, isTty: false, forcePrint: true })).toBe(false);
|
||||
});
|
||||
|
||||
test('env-sourced token is never printed', () => {
|
||||
expect(shouldSuppressBootstrapPrint({ ...base, fromEnv: true, isTty: true })).toBe(true);
|
||||
});
|
||||
|
||||
test('--suppress overrides even a forced print', () => {
|
||||
expect(shouldSuppressBootstrapPrint({ ...base, suppress: true, forcePrint: true, isTty: true })).toBe(true);
|
||||
});
|
||||
});
|
||||
|
||||
@@ -0,0 +1,124 @@
|
||||
/**
|
||||
* Test: full-sync reconcile separator normalization + mass-delete safety valve
|
||||
* (#2828 — Windows reconcile mass-delete).
|
||||
*
|
||||
* Pure-helper surface: `planReconcileDeletes` and `massReconcileAllowed` take
|
||||
* plain inputs and read no engine / no ambient env (the env reader is
|
||||
* parameterized), so these run without PGLite and without touching the shared
|
||||
* `process.env` — parallel-loop safe by construction.
|
||||
*/
|
||||
|
||||
import { describe, test, expect } from 'bun:test';
|
||||
import {
|
||||
planReconcileDeletes,
|
||||
massReconcileAllowed,
|
||||
MASS_RECONCILE_RATIO,
|
||||
MASS_RECONCILE_MIN_PAGES,
|
||||
} from '../src/commands/sync.ts';
|
||||
|
||||
/** Build stored page rows from a list of source_paths (slug = `slug-<index>`). */
|
||||
function rows(paths: Array<string | null>): Array<{ slug: string; source_path: string | null }> {
|
||||
return paths.map((p, i) => ({ slug: `slug-${i}`, source_path: p }));
|
||||
}
|
||||
|
||||
/**
|
||||
* Reconcile scenario: `total` file-backed pages, of which the first `stale`
|
||||
* are absent from the working tree (deleted) and the rest are present.
|
||||
*/
|
||||
function scenario(total: number, stale: number) {
|
||||
const stored = rows(Array.from({ length: total }, (_, i) => `p/${i}.md`));
|
||||
const present = stored.slice(stale).map((r) => r.source_path as string);
|
||||
return planReconcileDeletes(stored, present, () => true);
|
||||
}
|
||||
|
||||
describe('planReconcileDeletes — separator normalization (#2828)', () => {
|
||||
test('backslash working-tree paths match forward-slash stored source_path', () => {
|
||||
// Windows `path.relative` yields backslashes; source_path was stored with
|
||||
// forward slashes (e.g. git-derived). Both must compare equal.
|
||||
const stored = rows(['topics/foo.md', 'topics/bar.md', 'notes/baz.md']);
|
||||
const workingTree = ['topics\\foo.md', 'topics\\bar.md', 'notes\\baz.md'];
|
||||
const plan = planReconcileDeletes(stored, workingTree, () => true);
|
||||
expect(plan.staleSlugs).toEqual([]);
|
||||
expect(plan.reconcilableCount).toBe(3);
|
||||
expect(plan.massDelete).toBe(false);
|
||||
});
|
||||
|
||||
test('forward-slash working-tree paths match backslash stored source_path', () => {
|
||||
const stored = rows(['topics\\foo.md', 'topics\\bar.md']);
|
||||
const workingTree = ['topics/foo.md', 'topics/bar.md'];
|
||||
const plan = planReconcileDeletes(stored, workingTree, () => true);
|
||||
expect(plan.staleSlugs).toEqual([]);
|
||||
});
|
||||
|
||||
test('a genuinely removed file is the only stale slug, regardless of separator', () => {
|
||||
const stored = rows(['topics/foo.md', 'topics/bar.md', 'topics/gone.md']);
|
||||
const workingTree = ['topics\\foo.md', 'topics\\bar.md']; // gone.md deleted
|
||||
const plan = planReconcileDeletes(stored, workingTree, () => true);
|
||||
expect(plan.staleSlugs).toEqual(['slug-2']);
|
||||
});
|
||||
|
||||
test('null source_path rows are never reconcilable (manual / put_page pages)', () => {
|
||||
const stored = rows([null, 'x.md']);
|
||||
const plan = planReconcileDeletes(stored, [], () => true);
|
||||
expect(plan.reconcilableCount).toBe(1);
|
||||
expect(plan.staleSlugs).toEqual(['slug-1']);
|
||||
});
|
||||
|
||||
test('the strategy predicate excludes wrong-strategy pages from both stale set and denominator', () => {
|
||||
const stored = rows(['a.md', 'b.md', 'code.ts', 'gone.md']);
|
||||
const onlyMarkdown = (p: string) => p.endsWith('.md');
|
||||
const plan = planReconcileDeletes(stored, [], onlyMarkdown); // nothing present
|
||||
expect(plan.reconcilableCount).toBe(3); // code.ts excluded
|
||||
expect(plan.staleSlugs).toEqual(['slug-0', 'slug-1', 'slug-3']);
|
||||
});
|
||||
});
|
||||
|
||||
describe('planReconcileDeletes — mass-delete safety valve (#2828)', () => {
|
||||
test('trips when > 50% of a > 20-page source would be deleted', () => {
|
||||
const plan = scenario(21, 11); // 11/21 ≈ 52% > 50%, and 21 > 20
|
||||
expect(plan.reconcilableCount).toBe(21);
|
||||
expect(plan.staleSlugs.length).toBe(11);
|
||||
expect(plan.massDelete).toBe(true);
|
||||
});
|
||||
|
||||
test('holds at exactly 50% (threshold is strictly greater)', () => {
|
||||
const plan = scenario(40, 20); // 20/40 == 50%, not > 50%
|
||||
expect(plan.massDelete).toBe(false);
|
||||
});
|
||||
|
||||
test('ignores small sources (<= 20 pages) even at 100% stale', () => {
|
||||
expect(scenario(MASS_RECONCILE_MIN_PAGES, MASS_RECONCILE_MIN_PAGES).massDelete).toBe(false);
|
||||
expect(scenario(15, 15).massDelete).toBe(false);
|
||||
});
|
||||
|
||||
test('trips just past the min-pages boundary with a majority stale', () => {
|
||||
const plan = scenario(21, 20);
|
||||
expect(plan.massDelete).toBe(true);
|
||||
});
|
||||
|
||||
test('thresholds are the documented constants', () => {
|
||||
expect(MASS_RECONCILE_RATIO).toBe(0.5);
|
||||
expect(MASS_RECONCILE_MIN_PAGES).toBe(20);
|
||||
});
|
||||
});
|
||||
|
||||
describe('massReconcileAllowed — GBRAIN_ALLOW_MASS_RECONCILE escape hatch (#2828)', () => {
|
||||
test('=1 restores the old behavior', () => {
|
||||
expect(massReconcileAllowed({ GBRAIN_ALLOW_MASS_RECONCILE: '1' })).toBe(true);
|
||||
});
|
||||
|
||||
test('unset or any other value keeps the valve active', () => {
|
||||
expect(massReconcileAllowed({})).toBe(false);
|
||||
expect(massReconcileAllowed({ GBRAIN_ALLOW_MASS_RECONCILE: '0' })).toBe(false);
|
||||
expect(massReconcileAllowed({ GBRAIN_ALLOW_MASS_RECONCILE: 'true' })).toBe(false);
|
||||
});
|
||||
|
||||
test('effective gate: the valve blocks the delete unless the override is set', () => {
|
||||
const plan = scenario(21, 11);
|
||||
// This mirrors the guard in performFullSync.
|
||||
const blocked = plan.massDelete && !massReconcileAllowed({});
|
||||
const overridden = plan.massDelete && !massReconcileAllowed({ GBRAIN_ALLOW_MASS_RECONCILE: '1' });
|
||||
expect(blocked).toBe(true); // skip delete + loud warning
|
||||
expect(overridden).toBe(false); // old behavior restored
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,76 @@
|
||||
import { afterEach, describe, expect, test } from 'bun:test';
|
||||
import { existsSync, mkdtempSync, readFileSync, rmSync } from 'node:fs';
|
||||
import { tmpdir } from 'node:os';
|
||||
import { join } from 'node:path';
|
||||
import { runTakes } from '../src/commands/takes.ts';
|
||||
import type { BrainEngine, TakeBatchInput } from '../src/core/engine.ts';
|
||||
import { withEnv } from './helpers/with-env.ts';
|
||||
|
||||
const tmpRoots: string[] = [];
|
||||
|
||||
afterEach(() => {
|
||||
for (const root of tmpRoots.splice(0)) {
|
||||
rmSync(root, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
function makeEngine() {
|
||||
const added: TakeBatchInput[][] = [];
|
||||
const pageLookups: unknown[][] = [];
|
||||
const engine = {
|
||||
getConfig: async () => null,
|
||||
executeRaw: async (sql: string, params: unknown[] = []) => {
|
||||
if (sql.includes('FROM sources WHERE id = $1')) {
|
||||
return [{ id: params[0] as string }];
|
||||
}
|
||||
if (sql.includes('FROM pages WHERE slug = $1 AND source_id = $2')) {
|
||||
pageLookups.push(params);
|
||||
if (params[0] === 'shared/page' && params[1] === 'dept') return [{ id: 22 }];
|
||||
if (params[0] === 'shared/page' && params[1] === 'default') return [{ id: 11 }];
|
||||
return [];
|
||||
}
|
||||
if (sql.includes('FROM pages WHERE slug = $1 LIMIT 1')) {
|
||||
pageLookups.push(params);
|
||||
return [{ id: 11 }];
|
||||
}
|
||||
return [];
|
||||
},
|
||||
addTakesBatch: async (rows: TakeBatchInput[]) => {
|
||||
added.push(rows);
|
||||
return rows.length;
|
||||
},
|
||||
} as unknown as BrainEngine;
|
||||
return { engine, added, pageLookups };
|
||||
}
|
||||
|
||||
describe('gbrain takes CLI source scoping', () => {
|
||||
test('add mirrors to the page in GBRAIN_SOURCE, not an arbitrary same-slug page (#2684)', async () => {
|
||||
const brainDir = mkdtempSync(join(tmpdir(), 'gbrain-takes-source-'));
|
||||
const home = mkdtempSync(join(tmpdir(), 'gbrain-takes-home-'));
|
||||
tmpRoots.push(brainDir, home);
|
||||
const { engine, added, pageLookups } = makeEngine();
|
||||
|
||||
await withEnv({ GBRAIN_SOURCE: 'dept', GBRAIN_HOME: home }, async () => {
|
||||
await runTakes(engine, [
|
||||
'add',
|
||||
'shared/page',
|
||||
'--claim',
|
||||
'Dept-scoped claim',
|
||||
'--kind',
|
||||
'take',
|
||||
'--who',
|
||||
'self',
|
||||
'--dir',
|
||||
brainDir,
|
||||
]);
|
||||
});
|
||||
|
||||
expect(pageLookups).toEqual([['shared/page', 'dept']]);
|
||||
expect(added).toHaveLength(1);
|
||||
expect(added[0]![0]!.page_id).toBe(22);
|
||||
|
||||
const written = join(brainDir, 'shared/page.md');
|
||||
expect(existsSync(written)).toBe(true);
|
||||
expect(readFileSync(written, 'utf-8')).toContain('Dept-scoped claim');
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,28 @@
|
||||
/**
|
||||
* Pins `maxOutputTokensFor` — the per-model output-token budget `runThink`
|
||||
* passes to `client.create`. Thinking-by-default Claude 5 models
|
||||
* (`anthropic:claude-*-5`) spend a large share of the budget on internal
|
||||
* reasoning before emitting an answer, so the 4000 default left `think` with
|
||||
* empty/truncated text. They now get 16000; everything else stays 4000.
|
||||
*/
|
||||
import { describe, test, expect } from 'bun:test';
|
||||
import { maxOutputTokensFor } from '../src/core/think/index.ts';
|
||||
|
||||
describe('maxOutputTokensFor — thinking-default headroom', () => {
|
||||
test('Claude 5 family gets 16000', () => {
|
||||
expect(maxOutputTokensFor('anthropic:claude-sonnet-5')).toBe(16000);
|
||||
expect(maxOutputTokensFor('anthropic:claude-opus-5')).toBe(16000);
|
||||
expect(maxOutputTokensFor('anthropic:claude-fable-5')).toBe(16000);
|
||||
expect(maxOutputTokensFor('anthropic:claude-haiku-5')).toBe(16000);
|
||||
expect(maxOutputTokensFor('anthropic/claude-sonnet-5')).toBe(16000); // slash form
|
||||
});
|
||||
|
||||
test('non-Claude-5 and non-Anthropic keep 4000', () => {
|
||||
expect(maxOutputTokensFor('anthropic:claude-opus-4-8')).toBe(4000);
|
||||
expect(maxOutputTokensFor('anthropic:claude-haiku-4-5')).toBe(4000);
|
||||
expect(maxOutputTokensFor('anthropic:claude-sonnet-4-6')).toBe(4000);
|
||||
expect(maxOutputTokensFor('anthropic:claude-3-haiku')).toBe(4000);
|
||||
expect(maxOutputTokensFor('openai:gpt-4o')).toBe(4000);
|
||||
expect(maxOutputTokensFor('deepseek:deepseek-reasoner')).toBe(4000);
|
||||
});
|
||||
});
|
||||
Reference in New Issue
Block a user