mirror of
https://github.com/garrytan/gbrain.git
synced 2026-07-27 21:19:18 +00:00
* docs: land v0.38.0.0 production smoke-test report (PR #1299 + editor's note) PR #1299 from garrytan-agents shipped a 308-line production smoke-test report against v0.38.0.0 on Supabase+PgBouncer. Bringing the report verbatim into this worktree alongside the actual fixes (v0.38.3.0 wave). Privacy scrub passed per CLAUDE.md placeholder rule — no real people, companies, funds, or deals named. Editor's Note prepended to flag two re-diagnosed findings: - BUG-2 actual crash line is :1594-1597 (req.body === undefined → JSON.stringify returns literal undefined → Buffer.from throws), not the originally reported :1508. - WARN-5 root cause is `capture` missing from CLI_ONLY_SELF_HELP set; the detailed HELP constant exists but is unreachable. Co-Authored-By: garrytan-agents <garrytan-agents@users.noreply.github.com> Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(capture): BUG-1 — merge frontmatter instead of double-wrap on --file Pre-fix: `gbrain capture --file foo.md` on a file that already has YAML frontmatter stamped a second outer `---` block whose `title` field was the file's own opening `---` delimiter. Users got pages with two frontmatter blocks: outer `title: '---'`, inner the real metadata. The parser then treated the second block as a body-side horizontal rule. Root cause: capture.ts:136-152 `buildContent` always prepended its own frontmatter without inspecting whether the input already had one. The title-from-first-line heuristic at :141 picked up the `---` delimiter when present. Fix: new pure helper `mergeCaptureFrontmatter(rawBody, opts)` parses existing frontmatter via gray-matter (the lib markdown.ts already uses) and merges capture's auto-fields with user-wins precedence on user- declared keys. Single output frontmatter block in all cases. Precedence: - `type`: opts.type (CLI flag) > userFm.type > 'note' - `title`: userFm.title > derived-from-body - `captured_via`: userFm.captured_via > opts.source > 'capture-cli' - `captured_at`: userFm.captured_at > now() (user can pre-stamp) - All other user keys (description, tags, slug, ...) pass through verbatim Files without frontmatter keep the original behavior (stamp fresh, wrap under derived `# heading` if body lacks markdown structure). CQ2 boil-the-lake test coverage in test/capture-build-content.test.ts (19 cases, 47 assertions): 13 specified cases including CJK title, CRLF line endings, UTF-8 BOM, empty frontmatter `---\n---\n`, malformed YAML, no-trailing-newline-before-body, user description/tags/slug passthrough; plus 3 deriveTitle helper cases, 2 --type CLI flag precedence cases, and the BUG-1 regression guard with the exact reported input shape. Decisions deferred to later wave phases: - CV3 source_kind taxonomy + CV15 canonical source resolver: Phase 3c - CV8 dedup hash from rawBody: Phase 3c (receipt) + Phase 3d (DB) - CV9 trim boundary split: Phase 3c - CV10 binary-byte guard: Phase 3c Plan: ~/.claude/plans/system-instruction-you-are-working-async-popcorn.md (Phase 2a) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(serve-http): BUG-2 — /ingest 500-HTML on missing body becomes 400 JSON Pre-fix: a POST /ingest with NO body (no Content-Length, no body bytes — common shape for misconfigured webhooks and the smoke-test repro) leaves `req.body === undefined`. The body-coercion block's `else` branch then called `Buffer.from(JSON.stringify(undefined), 'utf8')` — and `JSON.stringify(undefined) === undefined` (the literal, not the string), so `Buffer.from(undefined, 'utf8')` threw TypeError. The unhandled throw reached express's default error handler, which served an HTML 500 page. Clients expecting JSON envelopes broke. Two changes: 1. Explicit `req.body == null` null-guard at the top of the handler (BEFORE the body coercion). Catches both `null` and `undefined` request bodies and returns the same 400 `empty_body` envelope as the empty-Buffer guard at :1600. Closes the TypeError class structurally. 2. Outer try/catch wrapping the entire handler body with a `!res.headersSent` guard (codex F#16) so any unexpected throw — downstream of the inner queue.add try/catch, in a logging side-effect, or in the SSE broadcast — returns a JSON 500 envelope instead of leaking the HTML error page. Mirrors the F14 pattern around the MCP handler's transport.handleRequest at serve-http.ts:1508-1520. E2E regression test in test/e2e/serve-http-ingest-webhook.test.ts: 'BUG-2: POST with no body (undefined req.body) → 400 JSON envelope (not 500 HTML)' uses `fetch(URL, {method:'POST'})` with no body field to provoke the exact pre-fix shape. Asserts status !== 500, status === 400, content-type is application/json, body.error === 'empty_body'. The existing 'empty body → 400' test at line 200 already covered the empty-string-body case (Express raw() produces an empty Buffer; the :1600 guard fires correctly). Both cases now reach the same envelope via two structurally distinct paths. Codex F#15 correction absorbed: the misleading 'body \"\"' → empty_body test case was dropped from the plan. An empty JSON string body has bytes and is not the same as a missing body. Codex F#14 correction absorbed: header in the smoke-test verification command is `Authorization:` not `Auth:` (updated in the plan; tests already use the correct shape). Plan: ~/.claude/plans/system-instruction-you-are-working-async-popcorn.md (Phase 2b) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(put_page): WARN-8 — provenance write-through with CV6 trust gate + CV12 COALESCE Migration v81 added 4 nullable provenance columns to `pages` (source_kind, source_uri, ingested_via, ingested_at). Pre-fix, put_page wrote them into the FILE's frontmatter (operations.ts:637-644 write- through path) but never to the DB columns. `gbrain call get_page | jq .source_kind` returned null even when the JSON receipt claimed `capture-cli`. This commit closes the loop on the write side. Surface changes: - src/core/types.ts: PageInput gains 4 optional provenance fields (source_kind, source_uri, ingested_via, ingested_at) with docstring explaining the trust model. - src/core/import-file.ts: importFromContent opts accepts 3 (source_kind, source_uri, ingested_via); threads them to tx.putPage. ingested_at is NOT a caller-controllable param — engine stamps it. - src/core/operations.ts put_page op: accepts 3 client params on the wire schema (uniform across transports). CV6 trust gate: when ctx.remote !== false, IGNORES client params and server-stamps source_kind='mcp:put_page'/source_uri=null/ingested_via='mcp:put_page'. Only ctx.remote === false (capture CLI, autopilot, dream cycle) honors client values. - src/core/postgres-engine.ts + pglite-engine.ts putPage: INSERT/UPDATE SQL extended with 4 columns. ingested_at stamped to now() only when ANY provenance field is being written this call (null otherwise). ON CONFLICT clause uses COALESCE-preserve UPDATE (CV12) so a later put_page without provenance does NOT erase the original first-write audit trail — first-write-wins for routine edits. CV6 closes the spoofing surface codex caught: a write-scope OAuth token can no longer poison the audit trail with arbitrary labels ('source_kind: capture-cli' from an MCP agent). Anything that isn't strictly `ctx.remote === false` falls through to server-stamped 'mcp:put_page', matching the v0.26.9 F7b fail-closed discipline. CV12 closes the audit-trail-erasure trap: routine put_page edits (no provenance args) preserve the original ingestion's source_kind / ingested_at via `COALESCE(EXCLUDED.x, pages.x)`. Explicit re-ingestion that passes new provenance overwrites (latest-ingestion-wins). Tests in test/put-page-provenance.test.ts (11 cases, 29 assertions): - Trusted local caller: client params populate DB columns; partial provenance still triggers ingested_at stamp; omitting all leaves all 4 null. - CV6 spoofing guard: remote caller's source_kind='capture-cli' claim becomes 'mcp:put_page'; ctx.remote === undefined treated as remote (v0.26.9 F7b: fail-closed on anything not strictly false). - CV12 COALESCE-preserve: second write without provenance preserves first-write audit AND timestamp; explicit re-ingestion with new provenance overwrites; remote second write after local first records the most-recent honest source (mcp:put_page). - T2 subagent regression: namespace check fires when provenance params are present; subagent within wiki/agents/<id>/ succeeds with server- stamped provenance per CV6; missing subagentId still fail-closes. Plan: ~/.claude/plans/system-instruction-you-are-working-async-popcorn.md (Phase 3a) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(read-path): CV5 — expose put_page provenance via getPage / listPages Phase 3a wrote the 4 provenance columns into the DB via put_page. This phase makes them visible to the read side so the smoke-test verification command `gbrain call get_page <slug> | jq .source_kind` actually returns the value the write side just stored. Surface changes: - src/core/types.ts Page: 4 optional fields (source_kind, source_uri, ingested_via, ingested_at). Three-state read pattern matching v0.26.5's deleted_at convention — undefined when the SELECT projection didn't include the column (older callers), null when historical pre-v0.38 row, populated when the v0.38+ ingestion stamped it. - src/core/utils.ts rowToPage: maps the 4 columns to Page fields via the same conditional-spread pattern as deleted_at / effective_date. Older SELECTs without the projection continue to compile. - src/core/postgres-engine.ts + pglite-engine.ts getPage: projection extended to include the 4 columns. listPages uses `SELECT p.*` so the columns flow through automatically — no listPages SQL change. Both engines' putPage RETURNING clause already projects the 4 columns from Phase 3a, so the round-trip (write→get) is symmetric. get_page op + JSON renderer require no changes: `gbrain call get_page` goes through `runCall` (src/commands/call.ts:52) which is `console.log(JSON.stringify(result, null, 2))`. Since `result` is the Page object from get_page (which is the engine's getPage return, which is rowToPage), the new fields surface automatically. The markdown renderer (`gbrain get`) goes through serializeMarkdown which strips structured fields; that's the right behavior for the markdown view. Structured access stays on `gbrain call get_page` and `list_pages` (JSON output). Engine-parity test (T3) extended in test/e2e/engine-parity.test.ts with 2 new cases: - 'provenance columns: putPage writes + getPage returns identical shape on both engines' — seeds capture-cli provenance, asserts source_kind/source_uri/ingested_via match across engines and ingested_at is a Date on both. - 'provenance COALESCE-preserve UPDATE: parity on both engines (CV12)' — first write stamps capture-cli, second write WITHOUT provenance preserves the first-write source_kind / ingested_via on BOTH engines (CV12 first-write-wins is engine-uniform). Gated on DATABASE_URL — runs PGLite half always; Postgres half skips without it (existing engine-parity pattern). When the test fires, a drift between the two engines now fails loudly instead of waiting for a user's `gbrain migrate --to supabase` to surface the bug. Phase 3a test suite (test/put-page-provenance.test.ts) re-verified green (11 pass) after the read-path additions — no regression. Plan: ~/.claude/plans/system-instruction-you-are-working-async-popcorn.md (Phase 3b) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(capture): WARN-1/3/7 + CV3/CV7/CV8/CV9/CV10/CV15/A2 — capture write-side overhaul Lands the seven decisions resolved in the plan-eng-review for the capture CLI's write-side. Each addresses a specific smoke-test finding or codex outside-voice concern. Atomic single commit because all changes are in capture.ts and tightly coupled (CV9's trim split feeds CV8's hash input which the tests depend on). Decisions resolved: CV3 — source_kind taxonomy: capture ALWAYS stamps source_kind='capture-cli' in the receipt JSON AND threads it through to put_page's provenance params. Pre-fix `parsed.source ?? 'capture-cli'` conflated the DB source FK (where to write) with the ingestion-channel taxonomy (what kind of ingestion this was). --source now ONLY maps to source_id; source_kind is closed taxonomy per migration v81's documented set. CV7 — thin-client --source rejection: when isThinClient(cfg) AND parsed.source is set, exit 1 BEFORE any network call with a clear error pointing at the right fix: `gbrain auth register-client <name> --source <id>` on the server. Mirrors CV6 trust posture (server-side OAuth client registration owns source scope; per-call override would reopen the spoofing surface CV6 just closed). CV8 (CLI side) — receipt content_hash now comes from normalized rawBody, NOT the assembled fullContent (which contained a timestamp-bearing frontmatter). Two captures of identical text now produce identical content_hash, restoring the daemon's 24h LRU dedup at the CLI receipt layer. The DB hash (importFromContent) gets the same treatment in Phase 3d. CV9 — trim boundary split: introduced `normalizeForHash(s)` pure helper (trim + BOM strip + LF + NFKC). Hash input gets aggressive normalization for dedup correctness; the STORED body (passed to buildContent → mergeCaptureFrontmatter) preserves user bytes (CRLF, BOM, whitespace) for round-trip fidelity. CQ2's CRLF/BOM tests continue to pass. CV10 — binary file guard: read --file via `readFileSync(path)` with NO encoding (Buffer-side), then sniff first 8KB for NUL bytes via `detectBinaryNullByte(buf)`. Mirror the same sniff for --stdin (which also now reads Buffer-side via `readStdinBuffer()`). Real UTF-8 text (including CJK, emoji, BOM) never contains NUL bytes; binary formats (executables, archives, most image formats) do. Reject with friendly error before UTF-8 decode mangles the bytes. Deterministic test fixtures use `Buffer.from([...])` with explicit byte arrays instead of /dev/urandom (which a 256-byte sample often had no NUL in). CV15 — canonical source resolver: route through `resolveSourceWithTier(engine, parsed.source, cwd)` from src/core/source-resolver.ts (the v0.37.7.0 6-tier chain every other CLI op uses). Honors flag → env (GBRAIN_SOURCE) → dotfile (.gbrain-source) → local_path → brain_default → seed_default. Closes the WARN-3-adjacent UX divergence where capture silently used `parsed.source ?? 'default'`, ignoring env / dotfile / local_path / brain_default tiers. Bonus: resolveSourceWithTier's assertSourceExists throws a friendly error if the source is missing, BEFORE put_page is called — making capture-level pre-flight redundant for the common case (A2 fallback below handles TOCTOU + thin-client edge cases). A2 — friendly FK error rewrite: new `maybeRewriteSourceFkError(err, sourceId)` helper detects the Postgres FK-violation patterns ('pages_source_id_fk' OR 'foreign key constraint ... source') and returns a paste-ready hint: `source '<id>' is not registered. Register it first: gbrain sources add <id> --path <path>`. Applied in BOTH the local-engine catch block AND the thin-client (callRemoteTool) catch block per T1 — so the same friendly error surfaces regardless of install type. Defense-in-depth alongside CV15's upstream check (covers TOCTOU race + thin-client implicit source from dotfile pointing at a server-deleted source). WARN-1 receipt-side dedup, WARN-3 friendly source error, WARN-7 binary guard, and the source_kind taxonomy fix all become user-visible via this commit. The DB-side dedup (WARN-1's daemon side) lands in Phase 3d. HELP text updated: - Documents the 6-tier source resolution chain - Notes thin-client --source restriction - Notes binary content rejection - Notes dedup behavior (whitespace + line endings normalized) - Notes source_kind != source_id distinction Tests in test/capture-runcapture.test.ts (26 cases, 30 assertions): - CV10 detectBinaryNullByte: 10 cases incl. ASCII, CJK, emoji, BOM, start/mid NUL, PNG magic, 8KB cap boundary, empty - CV9 normalizeForHash: 6 cases incl. whitespace, BOM, CRLF, NFKC, no-op on clean, whitespace-only → empty - CV8 hash stability: 4 cases proving identical input → identical hash regardless of whitespace / line endings / timing - A2 maybeRewriteSourceFkError: 6 cases incl. raw PG message, wrapped OperationError, postgres.js-wrapped, unrelated errors, missing sourceId, non-Error throws Phase 3a + Phase 2a tests re-verified green (30 pass across both files) — no regression in the surfaces this commit didn't directly touch. Plan: ~/.claude/plans/system-instruction-you-are-working-async-popcorn.md (Phase 3c) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(import-file): CV8 — exclude timestamp frontmatter from DB content_hash Pre-fix: every gbrain capture invocation produced a fresh DB content_hash because parsed.frontmatter included captured_at (and now ingested_at) which change per call. Two consequences: - existing.content_hash === hash short-circuit never fired for identical body captures → every capture re-chunked and re-embedded unchanged content, burning embedding API spend - daemon-side 24h LRU dedup (separate consumer keyed on the same hash) silently never matched, defeating its design intent Phase 3c shipped the CLI-side fix (receipt hash from normalized rawBody). This commit shipps the DB-side fix. Approach: strip timestamp-bearing frontmatter keys before hashing, NOT strip all frontmatter. Whitelist: ['captured_at', 'ingested_at']. Future timestamp keys add to the list — small, stable surface. Why not strip ALL frontmatter? Sync would regress: a user editing a markdown file to add a tag changes the frontmatter without changing the body. The pre-fix hash captures that change; tag reconciliation fires. If we stripped all frontmatter, the hash wouldn't change, the short-circuit would fire, and the tag-add would silently no-op. The narrow whitelist preserves frontmatter-change-detection for real edits (tags, type, slug, description, ...) while ignoring the ephemeral timestamp keys that capture-cli and provenance-write-through stamp per-call. Tests in test/import-file.test.ts (4 new cases in 'CV8 DB content_hash stability' describe block): - captured_at differences → IDENTICAL hash → second capture status 'skipped' (short-circuit fires); putPage NOT called the second time - body change → DIFFERENT hash → second capture status 'imported' (real edits still flow through) - tag change → DIFFERENT hash → re-import fires (REGRESSION GUARD: proves frontmatter-change detection survives the strip) - ingested_at differences → IDENTICAL hash (provenance-only refresh doesn't invalidate the chunk cache) Combined with Phase 3c's CLI-side hash fix, WARN-1 (dedup not actually deduplicating) is now fully resolved: both the user-visible receipt hash AND the DB / daemon hash stabilize across identical-content captures. Plan: ~/.claude/plans/system-instruction-you-are-working-async-popcorn.md (Phase 3d) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(cli): WARN-5 + WARN-6 — capture help discoverable; main --help lists BRAIN WARN-5: `gbrain capture --help` printed only the generic short-circuit fallback ('Usage: gbrain capture\\n\\ngbrain capture - run gbrain --help for the full command list.'). Root cause: `capture` was missing from CLI_ONLY_SELF_HELP at src/cli.ts:34-53. The detailed HELP constant at src/commands/capture.ts:90+ existed but was unreachable because the dispatcher's printCliOnlyHelp short-circuit at :101 fired first. WARN-6: `gbrain --help` did not mention capture / brainstorm / lsd anywhere. New v0.37/v0.38 commands were implemented and dispatched but absent from the hardcoded printHelp text. Two fixes in cli.ts: 1. Add 'capture' to CLI_ONLY_SELF_HELP. This skips the generic short-circuit, allowing the dispatch flow to reach runCapture which has its own --help branch printing the detailed HELP constant. 2. Add a pre-engine-bind '--help' short-circuit for capture in handleCliOnly (mirrors the existing sync + reinit-pglite pattern). Without this, `gbrain capture --help` on a fresh tmpdir with no config would hit the engine bind at :1077 and exit with 'Cannot connect to database' before runCapture's --help branch fires. 3. Add BRAIN section to printHelp text between TOOLS and SOURCES. Documents capture / brainstorm / lsd with their key flags, matching the tone of the existing grouped sections. Tests in test/cli-help-discoverability.test.ts (6 cases, 31 assertions): - WARN-5: capture --help contains every documented flag (--slug, --type, --file, --stdin, --source, --quiet, --json) - WARN-5: output is NOT the generic short-circuit fallback (presence of 'Examples:' + length > 10 lines + does not match the bare- short-circuit regex) - WARN-5: -h short flag works too - WARN-6: main --help mentions all 3 commands as command-line entries - WARN-6: BRAIN section heading is present and the 3 commands appear textually after it - regression: existing top-level commands (init, doctor, get, search, query, import, export, files, embed) still listed (snapshot guard against accidental deletion of other groups during the BRAIN insertion) Tests use spawnSync subprocess execution so the real dispatcher flow is exercised end-to-end (no mocking of cli.ts internals). Plan: ~/.claude/plans/system-instruction-you-are-working-async-popcorn.md (Phase 4a) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(serve-http): WARN-9 — admin register-client scopes normalization Pre-fix at serve-http.ts:1091: the /admin/api/register-client route destructured `scopes` from req.body and passed `scopes || 'read'` to `oauthProvider.registerClientManual(name, grants, scopes, uris)`. Three bugs in that single line: 1. Field-name mismatch: OAuth wire format uses `scope` (singular). The destructure looked for `scopes` (plural). A request body sending `{"scope": "read write"}` had `scopes === undefined`, fell through to the `'read'` default, and silently created a read-only client. This is the exact behavior the smoke test reported. 2. Array shapes crashed: registerClientManual's parseScopeString calls `.split(' ')` on its argument. Arrays don't have `.split`, so `{"scopes": ["read", "write"]}` threw TypeError mid-request, surfacing as a 500. 3. Empty-array truthy: `[]` is truthy in JS so `scopes || 'read'` returned the empty array, also crashing on split. Codex outside-voice (CV12) also flagged: validation depth is insufficient. `["read write"]` (a single-element array where the element contains a space) silently passes the type check but produces an unknown scope `"read write"` that registers as garbage. Same for non-string elements (null, numbers), empty strings, and duplicates. Fix: new `normalizeScopesInput(raw: unknown)` helper in src/core/scope.ts handles all four valid input shapes and rejects everything else with a typed error. The admin route accepts BOTH `scopes` (admin SPA) AND `scope` (OAuth wire), normalizes, and surfaces a 400 invalid_scopes on validation failure. Validation matrix: - undefined / null / missing → 'read' default - string → split on /\s+/, dedupe, validate each element against ALLOWED_SCOPES allowlist, re-join sorted - string[] → reject non-string elements, reject empty strings, reject internal-whitespace (catches ['read write'] bug shape), dedupe, validate each element, re-join sorted - everything else (number, boolean, plain object) → Error - empty array / whitespace-only string after split → Error - unknown scope name → InvalidScopeError ('Unknown scope "X". Allowed: ...') Output is sorted for determinism so two registrations with the same scope set produce identical DB rows. Tests in test/scope-normalize.test.ts (28 cases, 30 assertions): - Happy paths: 12 cases incl. all 4 shapes, dedupe in both directions, whitespace tolerance (tab/newline), every hierarchy scope - Rejection paths: 14 cases incl. number/object/boolean inputs, empty array, empty string, non-string array elements, empty-string element, whitespace-in-element (the codex bug), unknown scope name in both string and array forms, mix of known+unknown - Determinism: 2 cases proving sorted output is order-independent Plan: ~/.claude/plans/system-instruction-you-are-working-async-popcorn.md (Phase 4b) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(facts-absorb): WARN-4 — suppress 'No database connection' noise with diagnostic Pre-fix: every gbrain capture invocation logged '[facts:absorb] failed to log gateway_error for inbox/...: No database connection: connect() has not been called. Fix: Run gbrain init...'. Non-fatal — the page write itself succeeded — but loud per-capture noise that the smoke test rightly flagged as a user-visible bug. Root cause: the facts subsystem grabs a separate engine handle that isn't connected on the CLI capture path. The actual write succeeds via the connected engine that capture uses; the facts:absorb log is a courtesy that the doctor health check reads. Codex flagged this as symptom-treatment vs root-cause-fix (CV13). Plan picked the middle path: suppress the per-capture noise NOW, instrument first-occurrence with a stack trace so the v0.38.4 fix knows where to look. Two changes: CQ1 — typed access via instanceof + .problem field on GBrainError (NOT string-match on .message). The class GBrainError already has structured (problem, cause_description, fix) fields per src/core/types.ts:1104. Matching the structured field is impossible to silently break: if someone edits the error wording in db.ts thinking it's cosmetic, the typed access still routes correctly. String-match on .message would be the same fragility class we just closed elsewhere in this wave (capture's FK error rewrite uses both patterns deliberately because raw PG messages don't go through GBrainError). CV13 — first-occurrence diagnostic: module-scoped _hasLoggedDisconnectedFactsAbsorb flag fires ONE stderr warn with the full stack trace the first time this class occurs in a process. Subsequent occurrences are silent. The next user reporting the warning gives us the call site without an extra round-trip. Other failure classes (GBrainError with a different .problem field, plain Error, anything else) keep the loud per-call warn so real subsystem errors (PgBouncer crash, schema drift, etc.) still surface. The suppression is narrowly scoped to the known-broken-but-non-fatal WARN-4 class only. Test seam: `_resetFactsAbsorbDisconnectedFlagForTests()` exported so each test can assert from a clean slate. Tests in test/facts-absorb-log.test.ts (4 new cases in 'WARN-4 disconnected-engine suppression' describe block): - First occurrence prints ONE warn with 'WARN-4' + 'First-occurrence trace' substring; subsequent 3 calls are silent - GBrainError with a DIFFERENT problem field still warns loudly (the suppression is narrow, not class-wide) - Plain Error (non-GBrainError) still warns loudly - The engine.logIngest call STILL fires for the suppressed case (the suppression is on the WARN output, not the attempt — preserves the doctor health check's read path if/when the wiring is fixed in v0.38.4) v0.38.4 follow-up TODO filed per the plan: trace why the facts pipeline opens a separate engine handle on the CLI capture path, and either share the connected engine OR no-op the absorb-log when called from a CLI context. The diagnostic stack trace this commit prints is the input that fix needs. Plan: ~/.claude/plans/system-instruction-you-are-working-async-popcorn.md (Phase 4c) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(brainstorm): WARN-10 + CV11 — surface SQLSTATE 57014 as typed StructuredAgentError Pre-fix: brainstorm + lsd silently produced no output on PgBouncer transaction-mode environments. Postgres statement_timeout fired canceling listPrefixSampledPages or hybrid search; the unhandled error reached main()'s catch-all and surfaced as a generic 'gbrain: unknown error' line — after the user had already waited through the 10-second cost-preview window. Zero ideas, no diagnostic, no hint about what to do. Three changes per the resolved decisions: CV11 + T4 — orchestrator-level entry-point wrap (NOT per-call whack-a-mole). The public runBrainstorm becomes a thin wrapper that delegates to runBrainstormImpl inside try/catch; the catch runs classifyBrainstormError on the thrown value. Adding a new internal SQL call to runBrainstormImpl is automatically covered — codex F#20's 'scope too narrow' concern resolves structurally. A3 — reuse StructuredAgentError (the v0.19.0 envelope every new agent-facing surface uses) with code='brainstorm_timeout'. No new BrainstormError class; matches CLAUDE.md's stated convention. Future typed errors in brainstorm follow the same pattern. T4 + codex F#19 — classifier matches SQLSTATE 57014 specifically (the spec-defined query_canceled code) via postgres.js .code, alternate .sqlState, or message-substring fallback. Hint wording reads 'query canceled' (generic) covering all three PG cancel sub-causes: statement_timeout (often PgBouncer transaction-mode), lock_timeout, user-cancel. Honest under each. CV11 CLI formatter — runBrainstormCli (used by both gbrain brainstorm AND gbrain lsd) catches StructuredAgentError before main()'s catch-all sees it. Prints in the cli.ts:188-191 OperationError-block shape: 'Error [<code>]: <message>' then ' Hint: <hint>'. JSON mode emits the structured envelope (matches serializeError shape). Non-typed errors fall through to the dispatcher's existing catch — natural shape preserved (codex F#20 — no broad swallowing). Files: - src/core/brainstorm/error-classify.ts (new): isQueryCanceledError + classifyBrainstormError pure helpers. Module isolated from the orchestrator so the classifier can be unit-tested without spinning up the full brainstorm pipeline. - src/core/brainstorm/orchestrator.ts: imports the helpers; public runBrainstorm becomes a try/catch wrapper around the unchanged runBrainstormImpl. ~30 LOC change with zero body edits below. - src/commands/brainstorm.ts: catches StructuredAgentError before the generic main() handler. Imports StructuredAgentError from '../core/errors.ts'. Tests in test/brainstorm-timeout.test.ts (14 cases, 43 assertions): - isQueryCanceledError: 8 cases covering postgres.js {code}, alternate {sqlState}, message-substring fallbacks for all 3 cancel sub-causes, case-insensitive SQLSTATE match, negative cases (different codes, non-DB errors, null/undefined/non-object inputs) - classifyBrainstormError: 5 cases pinning the StructuredAgentError envelope (class, code, message), hint covers all 3 PG sub-causes (codex F#19 honesty contract), non-57014 errors pass through with SAME REFERENCE (codex F#20 — no clone, no swallow), null/undefined pass through, classified Error.message channel is descriptive - Source-shape regression guards: orchestrator.ts imports the helpers AND wraps runBrainstormImpl in try/catch at the public entry point (NOT per-call); commands/brainstorm.ts has the CLI formatter recognizing StructuredAgentError with 'Error [' + 'Hint:' shape WARN-10 root cause (PgBouncer-friendly SQL shape for listPrefixSampledPages) deferred per the plan's out-of-scope rationale — this commit adds the diagnostic surfacing so users know what hit them instead of silent no-output. TODOS.md follow-up filed in Phase 6. Plan: ~/.claude/plans/system-instruction-you-are-working-async-popcorn.md (Phase 5) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * v0.38.3.0: capture smoke-test wave — VERSION + CHANGELOG + docs + llms regen VERSION 0.38.2.0 → 0.38.3.0; package.json mirror. CHANGELOG: ELI10-lead-first entry per CLAUDE.md voice rules. "Things you can now do" section covers all 12 user-visible fixes (BUG-1 frontmatter merge, BUG-2 /ingest empty body, WARN-1 dedup, WARN-3 friendly source error, WARN-5 capture --help, WARN-6 main --help, WARN-7 binary guard, WARN-8 provenance write+read, WARN-9 admin scopes, WARN-10 brainstorm timeout surfacing, WARN-2 type-overwrite documentation, WARN-4 facts:absorb suppression). "What you'd see in a concrete example" block shows real terminal output. "Things to watch" covers CV7 thin-client --source rejection, CV12 COALESCE- preserve UPDATE semantics, CV3 closed source_kind taxonomy, brainstorm diagnostic-only deferral, facts:absorb first-occurrence diagnostic. "To take advantage" verification block with paste-ready commands. "For contributors" credits @garrytan-agents for PR #1299 and links the plan + decision trace. TODOS.md: 7 new v0.39 follow-ups filed at the top (SQL-shape rewrite of listPrefixSampledPages for PgBouncer, magic-byte allowlist, facts:absorb root-cause trace, --source-kind override flag, ingest_capture Minion handler architecture migration, provenance-history table, ingest webhook provenance pass-through). CLAUDE.md: single consolidated v0.38.3.0 entry under "Key files" naming every touched file + every decision (D1, A1-A3, CQ1-CQ2, T1-T4, CV3, CV5-CV15) with their resolution. Future maintainers see the full surface from one paragraph instead of grepping the diff. llms.txt + llms-full.txt: regenerated via `bun run build:llms` per CLAUDE.md's mandatory chaser ('every CLAUDE.md edit needs a build:llms chaser or test/build-llms.test.ts fails in CI'). 7 cases pass post-regen. Verify gate passes: typecheck clean + all 8 shell pre-checks (privacy, jsonb, progress, wasm, admin-scope-drift, cli-executable, system-of- record, eval-glossary-fresh, synthetic-corpus-privacy, skill-brain- first, fuzz-purity). Trio audit: VERSION: 0.38.3.0 package.json: 0.38.3.0 CHANGELOG: ## [0.38.3.0] - 2026-05-22 Plan: ~/.claude/plans/system-instruction-you-are-working-async-popcorn.md (Phase 6) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(tests): update legacy tests for v0.39.3.0 implementation shape changes Two CI failures from the post-merge state, both pre-existing tests pinning implementation detail that v0.39.3.0's fixes legitimately changed. Repair the tests, not the production behavior. 1) test/commands/capture.test.ts — 2 cases ('uses first non-empty line as the title' + 'caps title at 80 chars') were checking the YAML literal `title: "..."` (double-quoted). v0.39.3.0 Phase 2a (BUG-1 frontmatter merge) replaced the hand-rolled `JSON.stringify`- quoting with `matter.stringify()`, which follows YAML defaults: simple strings emit unquoted (`title: Real first line`), special- char strings get single-quoted. The semantic ("title equals X") is correct; the literal-quoting check was incidental. Parse the YAML and assert on the value via gray-matter. 2) test/fix-wave-structural.test.ts — the v0.36.1.x #1077 PKCE- public-clients regex pinned the exact destructure `const { name, scopes, tokenTtl, ... } = req.body`. v0.39.3.0 Phase 4b (WARN-9 admin scopes normalization) moved `scopes` to a separate read line so the route can accept BOTH `scopes` (admin SPA) AND `scope` (OAuth wire singular) via `?? `. Relax the destructure regex to accept either layout AND add a NEW regex pinning the `scopes ?? scope` fallback so the actual v0.39.3.0 contract is load-bearing. PKCE-fix assertions (tokenEndpointAuthMethod === 'none' + client_secret_hash = NULL + token_endpoint_auth_method = 'none') unchanged. Both fixes are tests-only — no production code change. Verify gate clean post-fix; 30/30 cases pass in the two affected test files. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore: rename v0.38.3.0 → v0.39.3.0 inline references across the wave The merge resolution bumped VERSION + package.json + CHANGELOG header to v0.39.3.0 (per user direction; higher than master's existing v0.39.0.0 + v0.39.1.0 commit subjects). But the wave's source code comments + test file headers + the smoke-test report's Editor's Note + the CLAUDE.md extension entry all still carried the v0.38.3.0 internal label. Sed-pass across 25 files (24 in-tree + the smoke-test report Editor's Note): - 13 src/ files: capture.ts, cli.ts, serve-http.ts, operations.ts, import-file.ts, types.ts, utils.ts, scope.ts, postgres-engine.ts, pglite-engine.ts, facts/absorb-log.ts, brainstorm/orchestrator.ts, brainstorm/error-classify.ts - 10 test files: capture-build-content, capture-runcapture, put-page-provenance, scope-normalize, cli-help-discoverability, brainstorm-timeout, facts-absorb-log, import-file, e2e/engine-parity, e2e/serve-http-ingest-webhook - docs/v0.38-smoke-test-report.md (Editor's Note only — the filename + the report body's references to v0.38.0.0 stay since they identify the historical subject) - CLAUDE.md (extension entry tag) Comments-only change; no production behavior shift. Existing tests continue to pass (175 cases across 10 wave-specific test files). llms.txt + llms-full.txt regenerated to keep the CLAUDE.md update in sync (per CLAUDE.md's mandatory build:llms chaser). Trio audit re-confirmed: VERSION: 0.39.3.0 package.json: 0.39.3.0 CHANGELOG: ## [0.39.3.0] - 2026-05-22 Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: garrytan-agents <garrytan-agents@users.noreply.github.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Garry Tan
co-authored by
garrytan-agents
Claude Opus 4.7
garrytan-agents
Claude Opus 4.7
parent
1666ec427e
commit
6987934ebb
+156
@@ -2,6 +2,161 @@
|
||||
|
||||
All notable changes to GBrain will be documented in this file.
|
||||
|
||||
## [0.39.3.0] - 2026-05-22
|
||||
|
||||
**`gbrain capture` now actually does what you expect: handles files with their own frontmatter cleanly, dedups identical text, rejects binary files, finds itself in `--help`, and tells you when something goes wrong instead of doing nothing.**
|
||||
|
||||
A production smoke test against the v0.38.0.0 ingestion cathedral release turned up two real bugs and ten warnings, all in the new capture and ingest paths. The bugs were the kind that you only notice when you're using the feature for real: capturing a markdown file that already had `---` frontmatter at the top would stamp a SECOND frontmatter block above the user's, with `title: '---'` (the file's opening delimiter became the outer title). And POSTing to `/ingest` with no body at all returned a 500 HTML error page instead of a JSON envelope, because of a single TypeScript line that called `Buffer.from(JSON.stringify(undefined))` and crashed before the empty-body guard could fire.
|
||||
|
||||
Beyond the bugs: capture wasn't listed in `gbrain --help`, `gbrain capture --help` printed only one generic line because of a missing entry in a set, the dedup hash included the capture timestamp so identical captures never actually deduplicated, binary files were silently captured as mojibake, `--source nonexistent` printed a raw Postgres FK violation, and brainstorm would hang silently on PgBouncer-restricted environments instead of saying "query was canceled."
|
||||
|
||||
This release closes all 12 items in one wave with atomic per-finding commits.
|
||||
|
||||
### How to upgrade
|
||||
|
||||
```bash
|
||||
gbrain upgrade
|
||||
# All fixes are transparent. No manual migration steps. The new put_page
|
||||
# provenance columns (migration v81) shipped in v0.38.0.0 and are
|
||||
# already in your brain; this release populates them on every write
|
||||
# instead of leaving them null.
|
||||
```
|
||||
|
||||
### Things you can now do
|
||||
|
||||
- **Capture a file that already has frontmatter** without getting a doubled-frontmatter mess. The new merge logic respects your declared `title`, `type`, `captured_at`, `tags`, `description`, and any other key you set; capture only fills in the fields you left blank.
|
||||
- **See `capture`, `brainstorm`, and `lsd` listed in `gbrain --help`** under a new BRAIN section. They were implemented but invisible before.
|
||||
- **Run `gbrain capture --help`** and get the full flag documentation (the detailed HELP constant has existed for releases — it was just unreachable behind a generic short-circuit). All seven flags documented with the new behavior notes.
|
||||
- **`gbrain capture "same text"` twice in a row produces an identical `content_hash`.** The hash is now computed from the normalized body (whitespace + line endings + Unicode form), not the timestamp-bearing frontmatter. The daemon's 24h LRU dedup actually works.
|
||||
- **Capture a markdown file with CJK text, emoji, CRLF line endings, or a BOM** without any of those bytes getting mangled. Hash normalization is separate from stored-body preservation: the hash gets aggressive normalization for dedup correctness; the stored content keeps your bytes for round-trip fidelity.
|
||||
- **`gbrain capture --file binary.bin`** rejects with a friendly error instead of capturing 256 bytes of garbage as "text." Same protection on `--stdin` — `cat binary.bin | gbrain capture --stdin` also rejects.
|
||||
- **`gbrain capture --source dept-x`** uses the canonical 6-tier source resolution chain (flag → env → dotfile → local_path → brain_default → seed_default), matching every other CLI op. Pre-fix, capture silently ignored `GBRAIN_SOURCE` and `.gbrain-source` dotfiles.
|
||||
- **`gbrain capture --source nonexistent-source`** now prints `source 'nonexistent-source' is not registered. Register it first: gbrain sources add nonexistent-source --path <path>` instead of the raw `pages_source_id_fk` Postgres error.
|
||||
- **`gbrain call get_page <slug> | jq .source_kind`** actually returns the value the write side stored. Migration v81 added the 4 provenance columns; this release populates them on every put_page (with CV6 trust gating that prevents OAuth clients from spoofing arbitrary `source_kind` labels — server stamps `mcp:put_page` for remote callers regardless of what they send).
|
||||
- **POST `/ingest` with no body** returns `400 empty_body` JSON instead of a 500 HTML error page.
|
||||
- **Admin `/admin/api/register-client`** accepts `{"scope": "read write"}` (singular wire format), `{"scopes": ["read", "write"]}` (array), or `{"scopes": "read write"}` (string) — all three normalize correctly. Pre-fix only one shape worked.
|
||||
- **`brainstorm`/`lsd` on a PgBouncer-restricted environment** now print `Error [brainstorm_timeout]: Brainstorm query was canceled by Postgres` with a hint covering the three Postgres cancel sub-causes (statement_timeout, lock_timeout, user-cancel) and a paste-ready workaround. Pre-fix: silent no-output after the 10-second cost-preview wait.
|
||||
|
||||
### What you'd see in a concrete example
|
||||
|
||||
```
|
||||
$ printf -- '---\ntitle: My Pre-Existing Title\ntags: [work, deal]\n---\n\n# Notes from the meeting\n\nbody here\n' > /tmp/m.md
|
||||
$ gbrain capture --file /tmp/m.md --json | jq '.slug, .content_hash, .source_kind'
|
||||
"inbox/2026-05-22-a1b2c3d4"
|
||||
"f7e6d5..."
|
||||
"capture-cli"
|
||||
|
||||
$ gbrain capture --file /tmp/m.md --json | jq '.slug, .content_hash'
|
||||
"inbox/2026-05-22-a1b2c3d4" ← SAME slug
|
||||
"f7e6d5..." ← SAME hash (dedup works)
|
||||
|
||||
$ gbrain call get_page --slug inbox/2026-05-22-a1b2c3d4 | jq '.title, .source_kind, .ingested_via'
|
||||
"My Pre-Existing Title" ← user title preserved
|
||||
"capture-cli" ← provenance populated in DB
|
||||
"capture-cli"
|
||||
|
||||
$ gbrain capture --file /tmp/has-null-byte.bin
|
||||
gbrain capture: refusing to capture binary content from /tmp/has-null-byte.bin
|
||||
Found null byte at offset 5 (first 8KB scan); text files (including UTF-8
|
||||
CJK/emoji/BOM) never contain NUL bytes.
|
||||
|
||||
$ gbrain capture "x" --source nonexistent
|
||||
gbrain capture: source 'nonexistent' is not registered. Register it first:
|
||||
gbrain sources add nonexistent --path <path>
|
||||
|
||||
List registered sources:
|
||||
gbrain sources list
|
||||
|
||||
$ curl -i -X POST localhost:3131/ingest -H "Authorization: Bearer $TOKEN"
|
||||
HTTP/1.1 400 Bad Request
|
||||
Content-Type: application/json
|
||||
{"error":"empty_body","message":"POST /ingest requires a non-empty body"}
|
||||
```
|
||||
|
||||
### Things to watch
|
||||
|
||||
- **`--source` is now rejected on thin-client installs** (gbrain serve --http with remote MCP clients like Claude Desktop, Cursor) with a clear error pointing at server-side `gbrain auth register-client --source <id>`. Server-side OAuth client registration owns source scope; per-call client override would reopen the spoofing surface this release just closed.
|
||||
- **Provenance UPDATE uses COALESCE-preserve.** A `put_page` that doesn't pass `source_kind` keeps the existing value (first-write-wins audit trail). A `put_page` that passes new provenance overwrites (explicit re-ingestion). The honest answer for "what was the most recent ingestion source for this page" lives in the DB; routine edits don't erase it.
|
||||
- **`source_kind` taxonomy is now closed:** `capture-cli | put_page | mcp:put_page | webhook | file-watcher | inbox-folder | cron-scheduler` per migration v81's documented set. `--source dept-x` maps to the DB source_id (which source row owns this page), NOT to source_kind. The two were conflated pre-fix.
|
||||
- **`--type meeting` and `--type idea` for the same content write to the SAME slug** (slug = content hash, so identical text deterministically produces the same slug). A later capture with a different `--type` overwrites the prior page. Documented in `gbrain capture --help` so it's not surprising.
|
||||
- **Brainstorm timeout surfacing is diagnostic-only this release.** The hint points at three potential causes (PgBouncer statement_timeout, lock_timeout, user-cancel) and suggests workarounds, but the underlying SQL-shape rewrite of `listPrefixSampledPages` for PgBouncer compatibility is filed as a v0.39 TODO. If you hit this on every brainstorm and the workarounds don't help, file an issue.
|
||||
- **A diagnostic stack trace will print ONCE** on first `gbrain capture` per process for the long-standing `[facts:absorb] failed to log gateway_error ... No database connection` warning — subsequent occurrences are silent. This gives the v0.38.4 fix the call site it needs without per-capture noise. If you hit this regularly, the trace is useful to file with an issue.
|
||||
|
||||
## To take advantage of v0.39.3.0
|
||||
|
||||
`gbrain upgrade` should do this automatically. If it didn't, or if `gbrain doctor` warns about anything:
|
||||
|
||||
1. **Run the orchestrator manually:**
|
||||
```bash
|
||||
gbrain apply-migrations --yes
|
||||
```
|
||||
2. **No host-agent action needed.** All fixes are in gbrain itself; your agent's skills don't change.
|
||||
3. **Verify the outcome:**
|
||||
```bash
|
||||
# Verify provenance write-through works
|
||||
SLUG=$(gbrain capture "v0.39.3.0 smoke" --json | jq -r .slug)
|
||||
gbrain call get_page --slug "$SLUG" | jq '.source_kind, .ingested_via'
|
||||
# Should print "capture-cli" twice (not null/null)
|
||||
|
||||
# Verify dedup
|
||||
H1=$(gbrain capture "same text" --json | jq -r .content_hash)
|
||||
H2=$(gbrain capture "same text" --json | jq -r .content_hash)
|
||||
[ "$H1" = "$H2" ] && echo "dedup OK" || echo "DEDUP REGRESSION"
|
||||
|
||||
# Verify CLI help
|
||||
gbrain --help | grep -qE '^\s*capture\s' && echo "capture listed OK"
|
||||
gbrain capture --help | grep -c -- '--' # should be >= 7 lines with --flag docs
|
||||
|
||||
gbrain doctor
|
||||
```
|
||||
4. **If any step fails or the numbers look wrong,** please file an issue:
|
||||
https://github.com/garrytan/gbrain/issues with:
|
||||
- output of `gbrain doctor`
|
||||
- which step broke
|
||||
- the v0.38.0.0 smoke-test report at `docs/v0.38-smoke-test-report.md` for context on what was tested
|
||||
|
||||
### Itemized changes
|
||||
|
||||
- **`src/commands/capture.ts`** — biggest single-file delta in the wave:
|
||||
- **BUG-1 frontmatter merge.** New `mergeCaptureFrontmatter(rawBody, opts)` pure helper uses gray-matter's `matter()` + `matter.stringify()` directly (NOT the lossy `parseMarkdown` from src/core/markdown.ts which strips type/title/tags/slug). User-wins precedence: spread user's declared keys first, then capture's auto-fields (type → CLI flag > userFm > 'note', title → userFm > derived, captured_via → userFm > opts.source > 'capture-cli', captured_at → userFm > now). Single output frontmatter block in all cases.
|
||||
- **CV3 source_kind taxonomy.** Always stamps `source_kind='capture-cli'` regardless of `--source` (which now maps to source_id only).
|
||||
- **CV15 canonical source resolver.** Routes through `resolveSourceWithTier(engine, parsed.source, cwd)` from `src/core/source-resolver.ts` — honors flag → env (`GBRAIN_SOURCE`) → dotfile (`.gbrain-source` walk-up) → local_path → brain_default → seed_default. Matches every other CLI op.
|
||||
- **CV7 thin-client `--source` rejection.** Exits 1 BEFORE network call with clear error pointing at server-side `gbrain auth register-client --source <id>`.
|
||||
- **CV8 + CV9 hash normalization.** New `normalizeForHash(s)` helper (trim + BOM strip + LF + NFKC). Receipt content_hash from normalized rawBody, NOT the assembled fullContent with timestamp.
|
||||
- **CV10 binary file guard.** New `detectBinaryNullByte(buf)` scans first 8KB for NUL byte. Applied to BOTH `--file` (read as Buffer, no encoding) AND `--stdin` (new `readStdinBuffer()`). Rejects with friendly error including byte offset.
|
||||
- **A2 FK error rewrite.** New `maybeRewriteSourceFkError(err, sourceId)` helper applied in BOTH local-engine and thin-client (`callRemoteTool`) catch blocks per T1. Detects Postgres `pages_source_id_fk` violations and rewrites to paste-ready hint.
|
||||
- **`src/commands/serve-http.ts`** — `/ingest` BUG-2 fix (null-guard at the top of body coercion + outer try/catch with `!res.headersSent` guard per codex F#16); `/admin/api/register-client` WARN-9 fix (accepts both `scope` and `scopes` request keys; normalizes via `normalizeScopesInput`).
|
||||
- **`src/cli.ts`** — `capture` added to `CLI_ONLY_SELF_HELP` set + pre-engine-bind `--help` short-circuit in `handleCliOnly` (mirrors the existing sync + reinit-pglite pattern). New `BRAIN` section in `printHelp` text covering capture / brainstorm / lsd with their key flags.
|
||||
- **`src/core/operations.ts`** — `put_page` op accepts 3 new optional params (source_kind, source_uri, ingested_via). CV6 trust gate: when `ctx.remote !== false`, IGNORES client params and server-stamps `mcp:put_page` regardless. Threads provenance through `importFromContent` to engine `putPage`.
|
||||
- **`src/core/import-file.ts`** — CV8 part 2: `importFromContent` excludes `captured_at` and `ingested_at` from the DB content_hash (whitelist of "ephemeral" frontmatter keys). Body + non-ephemeral frontmatter changes still trigger re-chunk; capture-cli's per-call timestamp doesn't invalidate the cache.
|
||||
- **`src/core/postgres-engine.ts` + `src/core/pglite-engine.ts`** — `putPage` SQL writes the 4 provenance columns with `COALESCE-preserve UPDATE` semantics (CV12: omitting params on a later put_page preserves prior values; first-write-wins audit trail). `getPage` SQL projection includes the 4 columns so the read path surfaces them.
|
||||
- **`src/core/utils.ts`** — `rowToPage` maps the 4 new columns to `Page` fields via the three-state conditional spread pattern (matches v0.26.5 deleted_at convention).
|
||||
- **`src/core/types.ts`** — `PageInput` AND `Page` gain 4 optional provenance fields with docstring trust-model + read-path notes.
|
||||
- **`src/core/scope.ts`** — new exported `normalizeScopesInput(raw: unknown): string` helper validates string / array / missing / empty shapes against the existing `ALLOWED_SCOPES` allowlist. Rejects `['read write']` (space-in-element bug shape codex flagged), empty arrays, non-string elements, unknown scope names.
|
||||
- **`src/core/facts/absorb-log.ts`** — WARN-4 + CQ1 + CV13: typed access via `instanceof GBrainError && e.problem === 'No database connection'` (NOT string-match on message). First-occurrence-per-process stack trace info log; subsequent occurrences silent. Other failure classes still warn loudly.
|
||||
- **`src/core/brainstorm/orchestrator.ts` + new `src/core/brainstorm/error-classify.ts`** — CV11 + T4: orchestrator-level try/catch at the public `runBrainstorm` entry covers EVERY internal SQL site. Classifies SQLSTATE 57014 (via postgres.js `.code`, alternate `.sqlState`, or message fallback) into `StructuredAgentError` with code='brainstorm_timeout' and hint covering all 3 PG cancel sub-causes. Non-57014 errors rethrow as-is.
|
||||
- **`src/commands/brainstorm.ts`** — catches `StructuredAgentError` before main()'s catch-all and prints `Error [<code>]: <message>` + `Hint: <hint>` lines (mirrors `cli.ts:188-191` OperationError block). JSON mode emits the structured envelope.
|
||||
|
||||
### Tests
|
||||
|
||||
- **`test/capture-build-content.test.ts`** (NEW, 19 cases, 47 assertions) — CQ2 boil-the-lake: 13 specified cases plus CJK title, CRLF line endings, UTF-8 BOM, empty frontmatter, malformed YAML, no-trailing-newline-before-body, user description/tags/slug passthrough, + 3 deriveTitle cases + 2 --type CLI precedence cases + BUG-1 regression guard with the exact reported input shape.
|
||||
- **`test/capture-runcapture.test.ts`** (NEW, 26 cases, 30 assertions) — CV10 binary guard (10 cases), CV9 normalization (6 cases), CV8 hash stability (4 cases), A2 FK error rewrite (6 cases).
|
||||
- **`test/put-page-provenance.test.ts`** (NEW, 11 cases, 29 assertions) — trusted local caller honored, CV6 spoofing guard (remote + undefined-trust), CV12 COALESCE-preserve UPDATE, T2 subagent namespace regression with provenance params.
|
||||
- **`test/scope-normalize.test.ts`** (NEW, 28 cases) — happy paths (12) + rejection cases (14) + determinism (2).
|
||||
- **`test/cli-help-discoverability.test.ts`** (NEW, 6 cases) — capture --help reaches detailed HELP, main --help lists all 3 BRAIN commands, regression guard for existing top-level groups.
|
||||
- **`test/brainstorm-timeout.test.ts`** (NEW, 14 cases) — `isQueryCanceledError` across driver shapes, `classifyBrainstormError` typed envelope + hint coverage + non-57014 passthrough, source-shape regression guards.
|
||||
- **`test/facts-absorb-log.test.ts`** (extended, 4 new cases) — WARN-4 first-occurrence + suppression contract; other failure classes still warn.
|
||||
- **`test/import-file.test.ts`** (extended, 4 new cases) — CV8 DB content_hash stability: timestamp differences IDENTICAL hash, body change DIFFERENT hash, tag change DIFFERENT hash (regression guard), ingested_at differences IDENTICAL.
|
||||
- **`test/e2e/engine-parity.test.ts`** (extended, 2 new cases) — provenance columns parity + COALESCE-preserve parity across Postgres + PGLite (T3).
|
||||
- **`test/e2e/serve-http-ingest-webhook.test.ts`** (extended, 1 new case) — BUG-2: POST with no body returns 400 JSON envelope (not 500 HTML).
|
||||
|
||||
### For contributors
|
||||
|
||||
- The smoke-test report at `docs/v0.38-smoke-test-report.md` was contributed by `@garrytan-agents` via PR #1299. PR closed as superseded; report ships verbatim with an Editor's Note prepended pointing at this CHANGELOG for the two factual corrections (BUG-2 line location was re-diagnosed; WARN-5 root cause was identified as a missing entry in `CLI_ONLY_SELF_HELP`).
|
||||
- The plan-eng-review flow on the v0 plan produced 15 decisions, 9 of them via codex outside-voice. Plan + decision trace at `~/.claude/plans/system-instruction-you-are-working-async-popcorn.md`.
|
||||
- Filed v0.39+ TODOs: SQL-shape rewrite of `listPrefixSampledPages` for PgBouncer; magic-byte allowlist for binary content detection; `--source-kind` override flag; facts:absorb root-cause trace; ingest_capture Minion handler architecture migration.
|
||||
|
||||
|
||||
## [0.39.2.0] - 2026-05-22
|
||||
|
||||
**Your federated brain refreshes all its sources in parallel now, not one at a time.**
|
||||
@@ -215,6 +370,7 @@ in your check output. Federated-brain operators will see new
|
||||
|
||||
This feedback loop is how the gbrain maintainers find fragile
|
||||
upgrade paths.
|
||||
|
||||
## [0.39.0.0] - 2026-05-21
|
||||
|
||||
**You can finally cap the cost of `gbrain brainstorm` and `gbrain lsd`, AND if the cap fires mid-run, you can resume right where you left off without losing the ideas you already paid for.**
|
||||
|
||||
@@ -308,6 +308,7 @@ strict behavior when unset.
|
||||
- `src/commands/embed.ts` extension (v0.36.4.0) — wires `--background` as the reference integration for the new `maybeBackground()` helper. `gbrain embed --stale --background` submits as a Minion job and prints `job_id=N` to stdout, exits 0. Composable in shell pipelines: `JOB=$(gbrain embed --stale --background | grep -oE 'job_id=[0-9]+' | cut -d= -f2); gbrain jobs follow $JOB`. The other six commands (`extract`, `lint`, `backlinks`, `reindex`, `integrity`, `pages`) adopt the same 4-line pattern in a follow-up wave (T7 deferred).
|
||||
- `src/core/cli-options.ts` extension (v0.36.4.0) — new `maybeBackground(opName, fingerprintArgs, runDirect)` helper. Same semantics in TTY and cron (D9 — no `--no-tty-detect` flag, no surprise behavior change between contexts): when `--background` is passed, submits the op as a Minion job via `op_checkpoints` for resumability and returns the `job_id`. `--background --follow` execs `gbrain jobs follow <id>` so the user sees the same stderr stream they'd get from a direct call. PGLite degrades to inline execution with a clear stderr note ("PGLite worker pool not yet supported; running inline"). Returns a tagged union the caller dispatches on.
|
||||
- `openclaw.plugin.json` — ClawHub bundle plugin manifest
|
||||
- `src/commands/capture.ts` + `src/commands/serve-http.ts` + `src/core/{operations,import-file,types,utils,facts/absorb-log,brainstorm/{orchestrator,error-classify},scope,postgres-engine,pglite-engine}.ts` extensions (v0.39.3.0 smoke-test wave) — productionization of the v0.38.0.0 ingestion cathedral release after a real production smoke test against Supabase+PgBouncer (PR #1299, contributed by `@garrytan-agents`). 12 atomic commits closing 2 bugs + 10 warnings: BUG-1 capture frontmatter merge via `mergeCaptureFrontmatter` (uses gray-matter directly, NOT the lossy `parseMarkdown`); BUG-2 `/ingest` null-guard + outer try/catch envelope with `!res.headersSent` guard; WARN-1 dedup via separate normalize-for-hash (`normalizeForHash` strips BOM/CRLF/whitespace/NFKC) + body-after-frontmatter-strip on the DB hash (excludes `captured_at` + `ingested_at` from the existing.content_hash check so capture-cli timestamp variations don't invalidate the chunk cache); WARN-3 friendly `pages_source_id_fk` rewrite via `maybeRewriteSourceFkError` applied to BOTH local + thin-client `callRemoteTool` catch blocks; WARN-4 `facts:absorb` 'No database connection' suppression via typed `instanceof GBrainError && e.problem` check + first-occurrence stack-trace info log (module-scoped `_hasLoggedDisconnectedFactsAbsorb` flag, test seam `_resetFactsAbsorbDisconnectedFlagForTests`); WARN-5 + WARN-6 CLI help discoverability (`capture` added to `CLI_ONLY_SELF_HELP` set + pre-engine-bind `--help` short-circuit in `handleCliOnly` + new `BRAIN` section in `printHelp` text); WARN-7 binary file guard via `detectBinaryNullByte(buf)` first-8KB NUL scan on both `--file` (Buffer-read, no encoding) and `--stdin` (`readStdinBuffer` Buffer accumulator); WARN-8 provenance write-through with put_page accepting 3 new optional params (source_kind, source_uri, ingested_via — `ingested_at` is server-stamped) + CV6 trust gate (when `ctx.remote !== false` IGNORE client params, server stamps `mcp:put_page`, fail-closed posture matches v0.26.9 F7b) + CV12 COALESCE-preserve UPDATE semantics (omitting params on a later put_page preserves prior values; first-write-wins audit trail); WARN-9 `/admin/api/register-client` scopes normalization via new `normalizeScopesInput(raw: unknown)` helper in `src/core/scope.ts` (accepts string, string[], missing; rejects `['read write']` space-in-element bug shape, non-string elements, empty array, unknown scope names; deduped + sorted output); WARN-10 brainstorm timeout surfacing via orchestrator-level try/catch wrap at `runBrainstorm` entry point — covers EVERY internal SQL site by being a single-point wrap, classifies SQLSTATE 57014 (via postgres.js `.code`, alternate `.sqlState`, or message fallback) into `StructuredAgentError` with code='brainstorm_timeout' and hint covering all 3 PG cancel sub-causes (statement_timeout, lock_timeout, user-cancel); CV5 read-path extension surfaces all 4 provenance columns via `getPage` SQL projection + `rowToPage` 3-state optional read + `Page` interface; CV15 canonical source resolver routes capture through `resolveSourceWithTier(engine, parsed.source, cwd)` matching every other CLI op (was `parsed.source ?? 'default'` pre-fix, silently ignoring env/dotfile/local_path tiers); CV7 thin-client `--source` rejection (server-side OAuth client registration owns source scope, per-call override would reopen CV6 spoofing surface); CV3 source_kind taxonomy is now closed (`capture-cli | put_page | mcp:put_page | webhook | file-watcher | inbox-folder | cron-scheduler`), `--source` flag maps to source_id only. Tests: 8 new test files + 3 extended files (test/capture-build-content.test.ts, test/capture-runcapture.test.ts, test/put-page-provenance.test.ts, test/scope-normalize.test.ts, test/cli-help-discoverability.test.ts, test/brainstorm-timeout.test.ts; extended test/facts-absorb-log.test.ts, test/import-file.test.ts, test/e2e/engine-parity.test.ts, test/e2e/serve-http-ingest-webhook.test.ts). Smoke-test report ships verbatim at `docs/v0.38-smoke-test-report.md` with Editor's Note prepending the two factual corrections (BUG-2 actual line + WARN-5 root cause). Plan + 15 decisions + codex outside voice trace at `~/.claude/plans/system-instruction-you-are-working-async-popcorn.md`. v0.39 follow-ups in TODOS.md: SQL-shape rewrite of `listPrefixSampledPages` for PgBouncer, magic-byte allowlist for binary detection, `--source-kind` override flag, ingest_capture handler architecture migration, provenance-history table, facts:absorb root-cause trace.
|
||||
|
||||
### BrainBench — in a sibling repo (v0.20+)
|
||||
|
||||
|
||||
@@ -1,6 +1,23 @@
|
||||
# TODOS
|
||||
|
||||
|
||||
## v0.39.3.0 smoke-test wave — deferred follow-ups (v0.39.4 / v0.40)
|
||||
|
||||
- [ ] **v0.40: SQL-shape rewrite of `listPrefixSampledPages` for PgBouncer transaction-mode compatibility.** WARN-10 root cause from the v0.38.0.0 smoke test: brainstorm + lsd consistently exceed Postgres `statement_timeout` (often PgBouncer-imposed) on the prefix-stratified domain bank query when the brain has >10K pages spread across many prefixes. v0.39.3.0 ships diagnostic surfacing only (the orchestrator wrap classifies SQLSTATE 57014 into a `StructuredAgentError` with a friendly hint). Real fix: per-prefix limit pushdown, embeddings prefetch, or breaking the single big query into a series of small ones across an explicit cursor. Plan: `~/.claude/plans/system-instruction-you-are-working-async-popcorn.md` (Phase 5, WARN-10 row). Owner: open.
|
||||
|
||||
- [ ] **v0.40: magic-byte allowlist for `gbrain capture` binary file detection.** v0.39.3.0 (Phase 3c, CV10) ships a first-8KB NUL-byte scan that catches typical binaries (executables, archives, most image formats). Known gap per CV10-B: a PNG with no NUL byte in its first 8KB slips through. Production-grade detection needs a magic-byte allowlist (PNG/JPEG/GIF/PDF/ZIP signatures). Implement in `src/commands/capture.ts:detectBinaryNullByte` (rename to `detectBinaryInput`) with a small `BINARY_MAGIC_BYTES` table. Reuse the same `assertSourceExists`-style friendly error pattern; reject before UTF-8 decode mangles the bytes. Tests in `test/capture-binary-guard.test.ts` should add cases for the PNG-without-NUL boundary.
|
||||
|
||||
- [ ] **v0.40: facts:absorb root-cause investigation.** v0.39.3.0 (Phase 4c, CV13) suppresses the per-capture `[facts:absorb] failed to log gateway_error for inbox/...: No database connection` noise AND prints a first-occurrence stack trace so the v0.40 fix knows where to look. The actual fix is one of: (a) thread the connected engine through the facts pipeline so it doesn't open its own handle; (b) no-op the absorb-log when called from a CLI context where the doctor health check isn't the consumer; (c) make the facts subsystem connection-aware and queue retries. The stack trace from `src/core/facts/absorb-log.ts:writeFactsAbsorbLog`'s first-occurrence info-log is the input.
|
||||
|
||||
- [ ] **v0.40: `--source-kind` override flag for `gbrain capture`.** v0.39.3.0 (Phase 3c, CV3) locked source_kind to `'capture-cli'` for capture invocations (the deferred CV3-B alternative). Real use case for the override: Apple Shortcuts / Zapier-style automations that shell out to `gbrain capture` and want their pages labeled `apple-shortcut` or `zapier` in the audit trail. Implementation: add a small flag with an allowlist (similar to migration v81's closed taxonomy: `capture-cli | apple-shortcut | zapier | <skillpack-kind>`); validate at parse time; CV6 remote-spoofing guard still applies (server stamps `mcp:put_page` regardless when `ctx.remote !== false`).
|
||||
|
||||
- [ ] **v0.40: route `gbrain capture` through `ingest_capture` Minion handler instead of put_page direct.** v0.39.3.0 (Phase 3a, A1) extended put_page with provenance params as the smallest diff. The cleaner architecture is the ingest_capture Minion handler shape that migration v81's comment already describes ("populated by the ingest_capture Minion handler"). This is a v0.40 architectural shift: capture submits an `ingest_capture` job → handler computes provenance + writes via put_page → result returns to capture. Adds queue latency (Minion job submit + poll) to the sync capture path; needs careful UX consideration (synchronous receipt vs async job_id). The current put_page extension stays back-compat after the migration.
|
||||
|
||||
- [ ] **v0.40: provenance-history table for full ingestion event log.** v0.39.3.0's CV12 `COALESCE-preserve UPDATE` keeps the FIRST ingestion source as the audit trail (first-write-wins). For deeper audit cases ("show me every time this page was re-ingested + by which channel"), a separate `pages_provenance_events` table keyed on `(page_id, ingested_at)` would preserve every event. Out of scope for v0.39.x; v0.40+ if/when the audit case grows beyond "first ingestion source."
|
||||
|
||||
- [ ] **v0.40+: ingest webhook provenance pass-through.** v0.39.3.0 CV6 closed the spoofing surface by IGNORING client-supplied provenance params for remote callers (ctx.remote !== false). The webhook path stamps server-side `webhook` provenance anyway, so today's behavior is unchanged. When trusted webhook integrations (a service running in the same trust domain as the server) need to declare their own source_kind (`linear`, `notion`, etc.), build a separate trusted-call surface for them — NOT by reopening put_page's wire schema. Possibilities: signed JWT with `provenance_authority: true` claim, or a different Minion job type `ingest_authoritative` that bypasses the CV6 guard.
|
||||
|
||||
|
||||
## v0.39.1+ schema-cathedral follow-ups (filed during v0.39.0.0 ship)
|
||||
|
||||
- [ ] **T18 follow-through — DELETE `skills/_brain-filing-rules.{md,json}`.** v0.39.0.0 shipped step (a) of the 4-step deprecation sequence: `gbrain schema show --as-filing-rules` emits the JSON shape the legacy file held. v0.39.1 ships steps (b) + (c) + (d): migrate `filing-audit.ts:79`, `synthesize.ts:619`, `patterns.ts:305`, `check-resolvable.ts:196+:226` to consume `gbrain schema show --as-filing-rules` output; update 5 test files (filing-audit.test.ts, check-resolvable.test.ts, dry-fix.test.ts, resolver.test.ts, cycle-patterns.test.ts); then DELETE the two files. Codex finding #3 from /plan-eng-review made this load-bearing — premature deletion makes protected synthesize/patterns phases fail with NO_ALLOWLIST. Sequencing matters.
|
||||
|
||||
@@ -0,0 +1,329 @@
|
||||
# v0.38.0.0 Smoke Test Report
|
||||
|
||||
> **Editor's Note (v0.39.3.0):** This report was contributed verbatim from
|
||||
> PR #1299 (`garrytan-agents`). Two findings were re-diagnosed during the
|
||||
> v0.39.3.0 wave; see `CHANGELOG.md` for corrections:
|
||||
>
|
||||
> - **BUG-2 location:** the empty-body crash site is the `else` branch at
|
||||
> `src/commands/serve-http.ts:1594-1597` (`Buffer.from(JSON.stringify(req.body), 'utf8')`
|
||||
> throws when `req.body === undefined` because `JSON.stringify(undefined) === undefined`),
|
||||
> not line 1508 as originally reported. The empty-Buffer guard at `:1600`
|
||||
> correctly fires for empty Buffers but never reaches the undefined case.
|
||||
>
|
||||
> - **WARN-5 root cause:** `gbrain capture --help` is minimal because
|
||||
> `capture` is missing from the `CLI_ONLY_SELF_HELP` set at `src/cli.ts:34-53`.
|
||||
> The detailed `HELP` constant at `src/commands/capture.ts:90-113` is
|
||||
> correct and comprehensive but unreachable; the dispatcher's generic
|
||||
> short-circuit at `:95` fires `printCliOnlyHelp(command)` first.
|
||||
> `brainstorm` and `lsd` are in the self-help set, which is why their help works.
|
||||
>
|
||||
> All 2 bugs and 10 warnings are addressed in v0.39.3.0. The report below
|
||||
> is preserved as the historical record of what production looked like on
|
||||
> 2026-05-22.
|
||||
|
||||
Production smoke test of the v0.38.0.0 ingestion cathedral release on a live
|
||||
Postgres-backed server (Supabase, pgvector, PgBouncer transaction mode).
|
||||
|
||||
**Test date:** 2026-05-22
|
||||
**Server:** MCP HTTP server v0.38.0.0 on port 3131, Postgres engine
|
||||
**Prior version:** v0.37.9.0
|
||||
|
||||
---
|
||||
|
||||
## Summary
|
||||
|
||||
| Category | Pass | Warn | Fail |
|
||||
|----------|------|------|------|
|
||||
| `gbrain capture` — basic | 7 | 0 | 0 |
|
||||
| `gbrain capture` — edge cases | 3 | 3 | 1 |
|
||||
| Dedup behavior | 0 | 2 | 0 |
|
||||
| POST /ingest webhook | 5 | 1 | 1 |
|
||||
| Provenance columns | 2 | 1 | 0 |
|
||||
| `brainstorm` / `lsd` | 2 | 1 | 0 |
|
||||
| CLI help & discoverability | 1 | 2 | 0 |
|
||||
| MCP server health | 1 | 0 | 0 |
|
||||
| **Total** | **21** | **10** | **2** |
|
||||
|
||||
---
|
||||
|
||||
## 🐛 Bugs (2)
|
||||
|
||||
### BUG-1: `capture --file` doubles frontmatter on files with existing frontmatter
|
||||
|
||||
**Severity:** Medium
|
||||
**Repro:**
|
||||
```bash
|
||||
cat > /tmp/test.md << 'EOF'
|
||||
---
|
||||
title: Pre-existing Title
|
||||
tags: [test, frontmatter]
|
||||
---
|
||||
|
||||
# Pre-existing content
|
||||
|
||||
This file already has frontmatter.
|
||||
EOF
|
||||
|
||||
gbrain capture --file /tmp/test.md
|
||||
gbrain get inbox/2026-05-22-XXXXXX
|
||||
```
|
||||
|
||||
**Observed:** The page on disk gets TWO frontmatter blocks. The outer block
|
||||
has `title: '---'` (it parsed the inner `---` delimiter as the title), and
|
||||
the original frontmatter is preserved verbatim inside the body:
|
||||
|
||||
```yaml
|
||||
---
|
||||
type: note
|
||||
title: '---'
|
||||
captured_at: '2026-05-22T16:06:11.334Z'
|
||||
captured_via: capture-cli
|
||||
ingested_via: put_page
|
||||
ingested_at: '2026-05-22T16:06:13.038Z'
|
||||
source_kind: put_page
|
||||
---
|
||||
|
||||
---
|
||||
title: Pre-existing Title
|
||||
tags: [test, frontmatter]
|
||||
---
|
||||
|
||||
# Pre-existing content
|
||||
```
|
||||
|
||||
**Expected:** `buildContent` should detect existing frontmatter (the commit
|
||||
message says it has a "looks like markdown" heuristic for first-line heading
|
||||
or frontmatter delimiter) and not double-wrap. The inner frontmatter fields
|
||||
should merge with capture's fields.
|
||||
|
||||
**Location:** `src/commands/brainstorm.ts` → `buildContent` function (shared
|
||||
with capture).
|
||||
|
||||
---
|
||||
|
||||
### BUG-2: POST /ingest crashes with unhandled TypeError on empty body
|
||||
|
||||
**Severity:** Medium
|
||||
**Repro:**
|
||||
```bash
|
||||
# With valid bearer token:
|
||||
curl -X POST localhost:3131/ingest \
|
||||
-H "Authorization: Bearer $TOKEN" \
|
||||
-H "Content-Type: text/plain"
|
||||
```
|
||||
|
||||
**Observed:** Returns a 500 HTML error page with stack trace:
|
||||
```
|
||||
TypeError: The first argument must be of type string, Buffer, ArrayBuffer,
|
||||
Array, or Array-like Object. Received undefined
|
||||
at serve-http.ts:1508:23
|
||||
```
|
||||
|
||||
**Expected:** The route already has an `empty_body` check (documented in the
|
||||
code and tested in the E2E suite), but the body-parser middleware for
|
||||
`/ingest` uses `express.raw()` which returns `undefined` for an empty POST
|
||||
(no Content-Length, no body). The `empty_body` guard fires AFTER the
|
||||
`computeContentHash(body)` call, which crashes on `undefined`.
|
||||
|
||||
**Fix:** Move the null/undefined/empty-buffer check before the content-hash
|
||||
computation, or add a guard at the top of the route handler:
|
||||
```typescript
|
||||
if (!req.body || req.body.length === 0) {
|
||||
return res.status(400).json({ error: 'empty_body', message: '...' });
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## ⚠️ Warnings (10)
|
||||
|
||||
### WARN-1: Dedup does not actually deduplicate identical captures
|
||||
|
||||
**Observed:** Capturing identical text twice produces the same slug but
|
||||
different `content_hash` values:
|
||||
|
||||
```
|
||||
Run 1: slug=inbox/2026-05-22-3d5b671a, hash=1faf3166...
|
||||
Run 2: slug=inbox/2026-05-22-3d5b671a, hash=f6ee8098...
|
||||
```
|
||||
|
||||
**Root cause:** The content hash includes the full serialized page with
|
||||
frontmatter, and `captured_at` changes between runs (it's timestamped).
|
||||
The slug is deterministic (derived from content text), so it correctly
|
||||
maps to the same page — but the hash changes every time.
|
||||
|
||||
**Impact:** The "24h content-hash LRU dedup" in the daemon layer won't
|
||||
catch duplicate captures because the hash differs. The `put_page`
|
||||
upsert catches it at the slug level (overwrites), so no duplicate pages
|
||||
are created — but the content_hash-based dedup layer is effectively
|
||||
bypassed for CLI captures.
|
||||
|
||||
**Suggestion:** Compute content_hash from the user's input text before
|
||||
adding frontmatter, or exclude `captured_at` from the hash computation.
|
||||
|
||||
---
|
||||
|
||||
### WARN-2: Same text with different `--type` flags overwrites the previous capture
|
||||
|
||||
Same slug is generated for identical text regardless of `--type`. The
|
||||
second capture with `--type observation` silently overwrites the first
|
||||
with `--type idea`. This is correct behavior (slug = content hash), but
|
||||
may surprise users who expect type changes to produce distinct pages.
|
||||
|
||||
---
|
||||
|
||||
### WARN-3: `--source` flag crashes with raw FK violation
|
||||
|
||||
```
|
||||
gbrain capture: put_page failed: insert or update on table "pages" violates
|
||||
foreign key constraint "pages_source_id_fk"
|
||||
```
|
||||
|
||||
The error message exposes a raw Postgres FK violation. Users have no way
|
||||
to know what to do. The capture command should catch this error and print
|
||||
a human-friendly message:
|
||||
|
||||
```
|
||||
Error: source 'my-source' is not registered. Register it first:
|
||||
gbrain sources add my-source --path /path/to/source
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### WARN-4: `facts:absorb` connection error after every capture
|
||||
|
||||
Every capture logs:
|
||||
```
|
||||
[facts:absorb] failed to log gateway_error for inbox/...: No database
|
||||
connection: connect() has not been called. Fix: Run gbrain init --supabase
|
||||
```
|
||||
|
||||
Non-fatal (exit 0), but noisy. The facts subsystem tries to open a
|
||||
separate connection that the CLI capture path doesn't initialize.
|
||||
|
||||
---
|
||||
|
||||
### WARN-5: `capture --help` is minimal — doesn't show flags
|
||||
|
||||
```
|
||||
$ gbrain capture --help
|
||||
Usage: gbrain capture
|
||||
|
||||
gbrain capture - run gbrain --help for the full command list.
|
||||
```
|
||||
|
||||
Compare with `brainstorm --help` which has a full Options section,
|
||||
examples, and cost info. The capture help should document `--type`,
|
||||
`--file`, `--source`, `--json`, `--stdin`, `--slug`, etc.
|
||||
|
||||
---
|
||||
|
||||
### WARN-6: `capture`, `brainstorm`, `lsd` missing from `gbrain --help`
|
||||
|
||||
None of these three commands appear in the main help text. They work when
|
||||
invoked directly, but users can't discover them from the help output.
|
||||
`gbrain --help` lists every other command (get, put, search, etc.) but
|
||||
the v0.37/v0.38 commands are absent.
|
||||
|
||||
---
|
||||
|
||||
### WARN-7: Binary file capture succeeds silently
|
||||
|
||||
```bash
|
||||
head -c 256 /dev/urandom > /tmp/binary.bin
|
||||
gbrain capture --file /tmp/binary.bin
|
||||
```
|
||||
|
||||
This succeeds, creating a page with binary garbage as content. Should
|
||||
either reject non-text files or at minimum warn.
|
||||
|
||||
---
|
||||
|
||||
### WARN-8: Provenance columns not populated by `capture`
|
||||
|
||||
The capture CLI reports `"source_kind": "capture-cli"` in its JSON output,
|
||||
but the actual database row has:
|
||||
|
||||
```json
|
||||
{ "source_id": "default", "source_uri": null, "source_kind": null }
|
||||
```
|
||||
|
||||
The provenance fields from capture's output don't round-trip into the
|
||||
`put_page` call that persists the page.
|
||||
|
||||
---
|
||||
|
||||
### WARN-9: Admin register-client ignores scope parameter
|
||||
|
||||
Registering a client via `/admin/api/register-client` with
|
||||
`"scope": "read write"` creates a client with scope `"read"` only.
|
||||
The admin endpoint appears to ignore or default the scope field. Required
|
||||
manual DB update to get `write` scope for webhook testing.
|
||||
|
||||
---
|
||||
|
||||
### WARN-10: `brainstorm` / `lsd` may hang or timeout on PgBouncer
|
||||
|
||||
In transaction-mode PgBouncer environments, `brainstorm` consistently
|
||||
times out after the cost estimate with `canceling statement due to
|
||||
statement timeout`. The hybrid search + domain-bank phase appears to
|
||||
hit PgBouncer's statement timeout. This may be environmental, but the
|
||||
command should handle the timeout gracefully and report what failed
|
||||
rather than silently producing no output.
|
||||
|
||||
---
|
||||
|
||||
## ✅ What Works Well
|
||||
|
||||
- **`gbrain capture "text"`** — works perfectly for the simple case.
|
||||
Frontmatter stamps, JSON output, disk write, immediate searchability.
|
||||
- **`gbrain capture --file`** — works for plain text and simple markdown
|
||||
(frontmatter doubling bug only affects files with existing frontmatter).
|
||||
- **`gbrain capture ""`** and no-args — clean error messages:
|
||||
`"provide content positionally, --file PATH, or --stdin"`.
|
||||
- **`gbrain capture --file /nonexistent`** — clean ENOENT error.
|
||||
- **Unicode/emoji** — `gbrain capture "测试 🧠🔥 émojis"` works perfectly.
|
||||
- **Long text** — 2500+ character capture works, correctly produces 2 chunks.
|
||||
- **POST /ingest auth gate** — properly rejects missing auth (401), invalid
|
||||
tokens (401), and insufficient scope (403).
|
||||
- **POST /ingest content-type validation** — properly rejects image/png and
|
||||
application/pdf with a helpful error pointing to skillpack processors.
|
||||
- **POST /ingest accepted types** — text/plain, text/markdown, text/html,
|
||||
application/json all accepted and queued.
|
||||
- **X-Gbrain-Source-Id header** — custom source ID flows through correctly.
|
||||
- **MCP health endpoint** — returns clean JSON with version and engine.
|
||||
- **OAuth client_credentials flow** — works correctly once the client has
|
||||
proper scope.
|
||||
- **Content-hash dedup on webhook** — verified via response `content_hash`.
|
||||
- **Migration v81** — provenance columns added cleanly, nullable, no
|
||||
disruption to existing pages. 290K+ existing pages unaffected.
|
||||
- **`brainstorm --help` / `lsd --help`** — excellent help text with
|
||||
examples, cost estimates, and cross-references.
|
||||
- **Soft delete + recovery** — cleanup via `gbrain delete` works with
|
||||
72h recovery window.
|
||||
- **`--type` flag** — `note`, `idea`, `observation` all work correctly.
|
||||
|
||||
---
|
||||
|
||||
## 💡 Suggestions
|
||||
|
||||
1. **`capture --help` parity** — give it the same quality help text as
|
||||
`brainstorm --help`. Document every flag.
|
||||
|
||||
2. **Main help completeness** — add `capture`, `brainstorm`, and `lsd` to
|
||||
the `gbrain --help` command listing. These are user-facing features that
|
||||
can't be discovered.
|
||||
|
||||
3. **Content-hash stability** — consider computing the dedup hash from
|
||||
user input text only (before frontmatter injection) so identical
|
||||
captures are properly deduped at the daemon layer.
|
||||
|
||||
4. **Provenance write-through** — `capture` already knows it's
|
||||
`source_kind: "capture-cli"`. Pass this through to `put_page` so the
|
||||
DB columns are populated.
|
||||
|
||||
5. **Binary file guard** — reject or warn on non-text input in
|
||||
`capture --file`. Check file content or extension before proceeding.
|
||||
|
||||
6. **`--source` error UX** — catch the FK violation and print a
|
||||
human-friendly hint about registering sources.
|
||||
@@ -444,6 +444,7 @@ strict behavior when unset.
|
||||
- `src/commands/embed.ts` extension (v0.36.4.0) — wires `--background` as the reference integration for the new `maybeBackground()` helper. `gbrain embed --stale --background` submits as a Minion job and prints `job_id=N` to stdout, exits 0. Composable in shell pipelines: `JOB=$(gbrain embed --stale --background | grep -oE 'job_id=[0-9]+' | cut -d= -f2); gbrain jobs follow $JOB`. The other six commands (`extract`, `lint`, `backlinks`, `reindex`, `integrity`, `pages`) adopt the same 4-line pattern in a follow-up wave (T7 deferred).
|
||||
- `src/core/cli-options.ts` extension (v0.36.4.0) — new `maybeBackground(opName, fingerprintArgs, runDirect)` helper. Same semantics in TTY and cron (D9 — no `--no-tty-detect` flag, no surprise behavior change between contexts): when `--background` is passed, submits the op as a Minion job via `op_checkpoints` for resumability and returns the `job_id`. `--background --follow` execs `gbrain jobs follow <id>` so the user sees the same stderr stream they'd get from a direct call. PGLite degrades to inline execution with a clear stderr note ("PGLite worker pool not yet supported; running inline"). Returns a tagged union the caller dispatches on.
|
||||
- `openclaw.plugin.json` — ClawHub bundle plugin manifest
|
||||
- `src/commands/capture.ts` + `src/commands/serve-http.ts` + `src/core/{operations,import-file,types,utils,facts/absorb-log,brainstorm/{orchestrator,error-classify},scope,postgres-engine,pglite-engine}.ts` extensions (v0.39.3.0 smoke-test wave) — productionization of the v0.38.0.0 ingestion cathedral release after a real production smoke test against Supabase+PgBouncer (PR #1299, contributed by `@garrytan-agents`). 12 atomic commits closing 2 bugs + 10 warnings: BUG-1 capture frontmatter merge via `mergeCaptureFrontmatter` (uses gray-matter directly, NOT the lossy `parseMarkdown`); BUG-2 `/ingest` null-guard + outer try/catch envelope with `!res.headersSent` guard; WARN-1 dedup via separate normalize-for-hash (`normalizeForHash` strips BOM/CRLF/whitespace/NFKC) + body-after-frontmatter-strip on the DB hash (excludes `captured_at` + `ingested_at` from the existing.content_hash check so capture-cli timestamp variations don't invalidate the chunk cache); WARN-3 friendly `pages_source_id_fk` rewrite via `maybeRewriteSourceFkError` applied to BOTH local + thin-client `callRemoteTool` catch blocks; WARN-4 `facts:absorb` 'No database connection' suppression via typed `instanceof GBrainError && e.problem` check + first-occurrence stack-trace info log (module-scoped `_hasLoggedDisconnectedFactsAbsorb` flag, test seam `_resetFactsAbsorbDisconnectedFlagForTests`); WARN-5 + WARN-6 CLI help discoverability (`capture` added to `CLI_ONLY_SELF_HELP` set + pre-engine-bind `--help` short-circuit in `handleCliOnly` + new `BRAIN` section in `printHelp` text); WARN-7 binary file guard via `detectBinaryNullByte(buf)` first-8KB NUL scan on both `--file` (Buffer-read, no encoding) and `--stdin` (`readStdinBuffer` Buffer accumulator); WARN-8 provenance write-through with put_page accepting 3 new optional params (source_kind, source_uri, ingested_via — `ingested_at` is server-stamped) + CV6 trust gate (when `ctx.remote !== false` IGNORE client params, server stamps `mcp:put_page`, fail-closed posture matches v0.26.9 F7b) + CV12 COALESCE-preserve UPDATE semantics (omitting params on a later put_page preserves prior values; first-write-wins audit trail); WARN-9 `/admin/api/register-client` scopes normalization via new `normalizeScopesInput(raw: unknown)` helper in `src/core/scope.ts` (accepts string, string[], missing; rejects `['read write']` space-in-element bug shape, non-string elements, empty array, unknown scope names; deduped + sorted output); WARN-10 brainstorm timeout surfacing via orchestrator-level try/catch wrap at `runBrainstorm` entry point — covers EVERY internal SQL site by being a single-point wrap, classifies SQLSTATE 57014 (via postgres.js `.code`, alternate `.sqlState`, or message fallback) into `StructuredAgentError` with code='brainstorm_timeout' and hint covering all 3 PG cancel sub-causes (statement_timeout, lock_timeout, user-cancel); CV5 read-path extension surfaces all 4 provenance columns via `getPage` SQL projection + `rowToPage` 3-state optional read + `Page` interface; CV15 canonical source resolver routes capture through `resolveSourceWithTier(engine, parsed.source, cwd)` matching every other CLI op (was `parsed.source ?? 'default'` pre-fix, silently ignoring env/dotfile/local_path tiers); CV7 thin-client `--source` rejection (server-side OAuth client registration owns source scope, per-call override would reopen CV6 spoofing surface); CV3 source_kind taxonomy is now closed (`capture-cli | put_page | mcp:put_page | webhook | file-watcher | inbox-folder | cron-scheduler`), `--source` flag maps to source_id only. Tests: 8 new test files + 3 extended files (test/capture-build-content.test.ts, test/capture-runcapture.test.ts, test/put-page-provenance.test.ts, test/scope-normalize.test.ts, test/cli-help-discoverability.test.ts, test/brainstorm-timeout.test.ts; extended test/facts-absorb-log.test.ts, test/import-file.test.ts, test/e2e/engine-parity.test.ts, test/e2e/serve-http-ingest-webhook.test.ts). Smoke-test report ships verbatim at `docs/v0.38-smoke-test-report.md` with Editor's Note prepending the two factual corrections (BUG-2 actual line + WARN-5 root cause). Plan + 15 decisions + codex outside voice trace at `~/.claude/plans/system-instruction-you-are-working-async-popcorn.md`. v0.39 follow-ups in TODOS.md: SQL-shape rewrite of `listPrefixSampledPages` for PgBouncer, magic-byte allowlist for binary detection, `--source-kind` override flag, ingest_capture handler architecture migration, provenance-history table, facts:absorb root-cause trace.
|
||||
|
||||
### BrainBench — in a sibling repo (v0.20+)
|
||||
|
||||
|
||||
+1
-1
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "gbrain",
|
||||
"version": "0.39.2.0",
|
||||
"version": "0.39.3.0",
|
||||
"description": "Postgres-native personal knowledge brain with hybrid RAG search",
|
||||
"type": "module",
|
||||
"main": "src/core/index.ts",
|
||||
|
||||
+26
@@ -40,6 +40,12 @@ const CLI_ONLY_SELF_HELP = new Set([
|
||||
'models',
|
||||
'cache',
|
||||
'brainstorm', 'lsd',
|
||||
// v0.39.3.0 WARN-5: capture's detailed HELP constant
|
||||
// (src/commands/capture.ts:90+) was unreachable because the dispatcher's
|
||||
// generic short-circuit (printCliOnlyHelp at :204-208) fired before
|
||||
// runCapture saw --help. brainstorm + lsd were already in the set;
|
||||
// capture was the holdout.
|
||||
'capture',
|
||||
// v0.37 fix wave (Lane D.4 + CDX2-12): sync's --no-embed flag was
|
||||
// unreachable via help because the dispatcher's generic CLI-only
|
||||
// short-circuit fired before runSync could print its own usage block.
|
||||
@@ -1072,6 +1078,17 @@ async function handleCliOnly(command: string, args: string[]) {
|
||||
return;
|
||||
}
|
||||
|
||||
// v0.39.3.0 WARN-5: same pattern for `capture --help`. CLI_ONLY_SELF_HELP
|
||||
// now includes 'capture' so the generic short-circuit at :101 stays out
|
||||
// of the way, but the dispatch case at :1229 still needs an engine. The
|
||||
// pre-engine-bind branch here exposes the HELP constant without requiring
|
||||
// a configured brain (fresh-tmpdir parity with brainstorm/lsd/sync).
|
||||
if (command === 'capture' && (args.includes('--help') || args.includes('-h'))) {
|
||||
const { runCapture } = await import('./commands/capture.ts');
|
||||
await runCapture(null, args);
|
||||
return;
|
||||
}
|
||||
|
||||
// All remaining CLI-only commands need a DB connection
|
||||
const engine = await connectEngine();
|
||||
try {
|
||||
@@ -1668,6 +1685,15 @@ TOOLS
|
||||
check-resolvable [--json] [--fix] Validate skill tree (reachability/MECE/DRY)
|
||||
report --type <name> --content ... Save timestamped report to brain/reports/
|
||||
|
||||
BRAIN (capture / ideate / explore — v0.37/v0.38)
|
||||
capture [content] [--file PATH] Single entrypoint for getting content into the brain
|
||||
[--stdin] [--slug s] [--type t] Inline content / file / stdin; writes to inbox/ by default
|
||||
[--source ID] [--quiet|--json] Multi-source brains: route to a non-default source
|
||||
brainstorm <question> [--json] Bisociation idea generator (hybrid search + far-set + judge)
|
||||
[--save|--no-save] [--limit N]
|
||||
lsd <question> [--json] Lateral Synaptic Drift: inverted-judge brainstorm
|
||||
[--save|--no-save] [--limit N] rewarding far-from-obvious + axiomatic inversions
|
||||
|
||||
SOURCES (multi-repo / multi-brain)
|
||||
sources list Show registered sources
|
||||
sources add <id> --path <p> Register a source (id = short name, e.g. 'wiki')
|
||||
|
||||
+38
-12
@@ -22,6 +22,7 @@ import {
|
||||
type BrainstormProfile,
|
||||
} from '../core/brainstorm/orchestrator.ts';
|
||||
import { loadConfig } from '../core/config.ts';
|
||||
import { StructuredAgentError } from '../core/errors.ts';
|
||||
|
||||
export interface BrainstormCliArgs {
|
||||
question?: string;
|
||||
@@ -250,18 +251,43 @@ async function runBrainstormCli(
|
||||
? { ...profile, m_far: parsed.limit }
|
||||
: profile;
|
||||
|
||||
const result = await runBrainstorm(engine, config, {
|
||||
question: parsed.question,
|
||||
profile: effectiveProfile,
|
||||
skipCostPreview: skipPreview,
|
||||
maxCostUsd: parsed.maxCost,
|
||||
maxFarSet: parsed.maxFarSet,
|
||||
strictBudget: parsed.strictBudget,
|
||||
judgeModel: parsed.judgeModel,
|
||||
maxIdeasPerJudgeCall: parsed.maxIdeasPerJudgeCall,
|
||||
resumeRunId: parsed.resume,
|
||||
forceResume: parsed.forceResume,
|
||||
});
|
||||
// v0.39.3.0 WARN-10 + CV11 — catch StructuredAgentError 'brainstorm_timeout'
|
||||
// surfaced by the orchestrator's outer wrap and format it like the
|
||||
// cli.ts:188-191 OperationError block (Error [code]: message + Hint line
|
||||
// + exit 1). Non-typed errors (including BudgetExhausted from v0.39.0.0
|
||||
// T10's gateway-layer cap) fall through to the dispatcher's existing
|
||||
// catch. Without this CLI formatter, the typed error reaches main()'s
|
||||
// generic catch which prints `e.message` only — losing the structured
|
||||
// `.hint` field that's the whole point of the orchestrator-level wrap.
|
||||
let result;
|
||||
try {
|
||||
result = await runBrainstorm(engine, config, {
|
||||
question: parsed.question,
|
||||
profile: effectiveProfile,
|
||||
skipCostPreview: skipPreview,
|
||||
// v0.39.0.0 T10 cost-cap surface — wired in master, preserved here.
|
||||
maxCostUsd: parsed.maxCost,
|
||||
maxFarSet: parsed.maxFarSet,
|
||||
strictBudget: parsed.strictBudget,
|
||||
judgeModel: parsed.judgeModel,
|
||||
maxIdeasPerJudgeCall: parsed.maxIdeasPerJudgeCall,
|
||||
resumeRunId: parsed.resume,
|
||||
forceResume: parsed.forceResume,
|
||||
});
|
||||
} catch (err) {
|
||||
if (err instanceof StructuredAgentError) {
|
||||
if (parsed.json) {
|
||||
// Agents reading --json get the structured envelope (matches
|
||||
// serializeError shape from src/core/errors.ts).
|
||||
console.log(JSON.stringify({ error: err.envelope }, null, 2));
|
||||
} else {
|
||||
console.error(`Error [${err.envelope.code}]: ${err.envelope.message}`);
|
||||
if (err.envelope.hint) console.error(` Hint: ${err.envelope.hint}`);
|
||||
}
|
||||
process.exit(1);
|
||||
}
|
||||
throw err; // not our error class — let the dispatcher handle it
|
||||
}
|
||||
|
||||
if (parsed.json) {
|
||||
console.log(JSON.stringify(result, null, 2));
|
||||
|
||||
+316
-44
@@ -31,12 +31,14 @@
|
||||
*/
|
||||
|
||||
import { readFileSync } from 'node:fs';
|
||||
import matter from 'gray-matter';
|
||||
import type { BrainEngine } from '../core/engine.ts';
|
||||
import { loadConfig, isThinClient } from '../core/config.ts';
|
||||
import { callRemoteTool, unpackToolResult } from '../core/mcp-client.ts';
|
||||
import { callRemoteTool, unpackToolResult, RemoteMcpError } from '../core/mcp-client.ts';
|
||||
import { computeContentHash } from '../core/ingestion/types.ts';
|
||||
import { operations } from '../core/operations.ts';
|
||||
import type { OperationContext } from '../core/operations.ts';
|
||||
import { resolveSourceWithTier } from '../core/source-resolver.ts';
|
||||
|
||||
interface RunOpts {
|
||||
content?: string;
|
||||
@@ -100,11 +102,29 @@ Modes (mutually exclusive — first match wins):
|
||||
Options:
|
||||
--slug SLUG Override the default inbox/YYYY-MM-DD-<hash6> slug
|
||||
--type TYPE Override the page type (default: note)
|
||||
--source ID Multi-source brains: write under a non-default source
|
||||
--source ID Multi-source brains: write under a non-default source.
|
||||
Resolution: --source flag > GBRAIN_SOURCE env >
|
||||
.gbrain-source dotfile (walk-up) > local_path >
|
||||
brain_default > 'default'. NOT supported on
|
||||
thin-client installs (server-side OAuth client
|
||||
registration scopes the source).
|
||||
--quiet, -q Print just the slug on stdout (for shell pipelines)
|
||||
--json JSON output for agents
|
||||
--help, -h Show this help
|
||||
|
||||
Notes:
|
||||
- Binary files (image/audio/video/pdf, anything with NUL bytes in the
|
||||
first 8KB) are rejected with a friendly error. Use a content-type
|
||||
processor skillpack for those when available.
|
||||
- Capturing identical text twice produces the same slug and the same
|
||||
content_hash (whitespace + line endings + Unicode form are normalized
|
||||
before hashing). The daemon's 24h LRU dedup uses this hash.
|
||||
- source_kind in the DB is ALWAYS 'capture-cli' for invocations of this
|
||||
command. --source maps to the source_id DB column, NOT to source_kind.
|
||||
Different --type values write to the SAME slug for the same content
|
||||
(slug = content hash), so a later capture with a different --type
|
||||
overwrites the prior page.
|
||||
|
||||
Examples:
|
||||
gbrain capture "remember to follow up on the X deal"
|
||||
echo "from a pipe" | gbrain capture --stdin
|
||||
@@ -120,35 +140,169 @@ function defaultSlug(content: string, now: Date = new Date()): string {
|
||||
return `inbox/${y}-${m}-${d}-${hashPrefix}`;
|
||||
}
|
||||
|
||||
async function readStdin(): Promise<string> {
|
||||
/**
|
||||
* v0.39.3.0 CV10 — binary file guard. Scans the first 8KB of `buf` for a
|
||||
* NUL byte (0x00). Real text files (including UTF-8 with multi-byte CJK,
|
||||
* emoji, BOM) never contain a NUL byte at any position — text encoding
|
||||
* uses non-zero continuation bytes. NUL appears in binary formats:
|
||||
* executables, archives, compressed images, PDFs (after the magic-byte
|
||||
* header), most office documents. Single-pass scan; constant memory.
|
||||
*
|
||||
* Returns the 0-indexed byte offset of the first NUL, or -1 if clean.
|
||||
* Caller decides the error shape (message vs JSON envelope).
|
||||
*
|
||||
* Known limit: a PNG-without-NUL-in-first-8KB slips through. v0.39
|
||||
* magic-byte allowlist (per CV10-B + TODOS.md) closes this hole. The
|
||||
* 8KB ceiling bounds the scan cost to ~microseconds even on huge files.
|
||||
*/
|
||||
export function detectBinaryNullByte(buf: Buffer): number {
|
||||
const limit = Math.min(buf.length, 8 * 1024);
|
||||
for (let i = 0; i < limit; i++) {
|
||||
if (buf[i] === 0) return i;
|
||||
}
|
||||
return -1;
|
||||
}
|
||||
|
||||
async function readStdinBuffer(): Promise<Buffer> {
|
||||
const chunks: Buffer[] = [];
|
||||
for await (const chunk of process.stdin) {
|
||||
chunks.push(typeof chunk === 'string' ? Buffer.from(chunk, 'utf8') : (chunk as Buffer));
|
||||
}
|
||||
return Buffer.concat(chunks).toString('utf8');
|
||||
return Buffer.concat(chunks);
|
||||
}
|
||||
|
||||
/**
|
||||
* v0.39.3.0 CV9 — normalize content for content_hash so identical text
|
||||
* produces identical hashes regardless of leading/trailing whitespace,
|
||||
* line-ending style (CRLF vs LF), or Unicode normalization form. The
|
||||
* STORED body is preserved as-is (CRLF stays CRLF, BOM stays BOM).
|
||||
*
|
||||
* Two concerns, two transforms — the hash gets aggressive normalization
|
||||
* for dedup correctness; the stored body keeps user bytes for round-trip
|
||||
* fidelity. CQ2's CRLF/BOM preservation tests rely on this split.
|
||||
*/
|
||||
export function normalizeForHash(s: string): string {
|
||||
// Strip BOM, normalize line endings to LF, trim, NFKC for Unicode-stable hash.
|
||||
return s.replace(/^/, '').replace(/\r\n/g, '\n').trim().normalize('NFKC');
|
||||
}
|
||||
|
||||
/**
|
||||
* v0.39.3.0 A2 + CV6 — detect Postgres FK violation on the sources table
|
||||
* in an error message and return a friendly hint. Returns null when the
|
||||
* error doesn't match. Used by BOTH the local-engine catch block AND
|
||||
* the thin-client (callRemoteTool) catch block per T1.
|
||||
*
|
||||
* Pattern coverage:
|
||||
* - Postgres SQLSTATE 23503 message: 'insert or update on table "pages" violates
|
||||
* foreign key constraint "pages_source_id_fk"'
|
||||
* - postgres.js may wrap with extra context; substring match is enough
|
||||
* - The MCP error envelope passes the message through unchanged (the
|
||||
* server-side put_page op converts to OperationError but the underlying
|
||||
* PG message is in the .cause chain)
|
||||
*/
|
||||
export function maybeRewriteSourceFkError(err: unknown, sourceId: string | undefined): string | null {
|
||||
if (!sourceId) return null;
|
||||
const msg = err instanceof Error ? err.message : String(err);
|
||||
// Match both the raw Postgres wording and OperationError-wrapped variants.
|
||||
const matchesFk = msg.includes('pages_source_id_fk')
|
||||
|| (msg.includes('foreign key constraint') && msg.includes('source'));
|
||||
if (!matchesFk) return null;
|
||||
return `source '${sourceId}' is not registered. Register it first:\n gbrain sources add ${sourceId} --path <path>\n\nList registered sources:\n gbrain sources list`;
|
||||
}
|
||||
|
||||
/**
|
||||
* Derive a title from the first non-empty, non-`---` line of the body,
|
||||
* stripping leading markdown heading marks, capped at 80 chars.
|
||||
* Falls back to 'Capture' when no usable line exists.
|
||||
*/
|
||||
function deriveTitle(rawBody: string): string {
|
||||
const firstLine = rawBody
|
||||
.split('\n')
|
||||
.find((l) => l.trim().length > 0 && l.trim() !== '---') ?? '';
|
||||
return firstLine.replace(/^#+\s*/, '').slice(0, 80) || 'Capture';
|
||||
}
|
||||
|
||||
/**
|
||||
* v0.39.3.0 (BUG-1): merge capture's auto-stamped fields with any existing
|
||||
* frontmatter in `rawBody`, rather than always prepending a second
|
||||
* frontmatter block. The pre-fix code stamped its own `---` block on top
|
||||
* of files that already had frontmatter, producing `title: '---'` (the
|
||||
* file's opening delimiter became the outer title) and two consecutive
|
||||
* frontmatter blocks the parser interpreted as the outer block + a body
|
||||
* starting with a horizontal rule.
|
||||
*
|
||||
* Precedence rules (user-wins by default):
|
||||
* - `type`: opts.type (CLI flag) > userFm.type > 'note'
|
||||
* - `title`: userFm.title > derived-from-body
|
||||
* - `captured_via`: userFm.captured_via > opts.source > 'capture-cli'
|
||||
* (CV3/Phase 3c will narrow this to always 'capture-cli';
|
||||
* for Phase 2a we preserve current semantics)
|
||||
* - `captured_at`: userFm.captured_at > now (user can pre-stamp for retroactive
|
||||
* captures; see CQ2 test case 4)
|
||||
* - Any other user-declared keys (description, tags, slug, etc.) pass through verbatim.
|
||||
*
|
||||
* For files WITHOUT existing frontmatter, preserves the original behavior:
|
||||
* stamps a fresh frontmatter block, and if the body doesn't already look
|
||||
* like markdown (no `#` heading), wraps it under a `# {title}` heading.
|
||||
*/
|
||||
export function mergeCaptureFrontmatter(rawBody: string, opts: RunOpts): string {
|
||||
const nowIso = new Date().toISOString();
|
||||
// Detect frontmatter: leading `---\n` or `---\r\n`, tolerating leading BOM/whitespace.
|
||||
// We do NOT use the more permissive `startsWith('---')` because a body that opens
|
||||
// with a horizontal-rule like `--- separator ---` would false-positive.
|
||||
const trimmedStart = rawBody.replace(/^/, '');
|
||||
const hasFrontmatter = /^---\r?\n/.test(trimmedStart);
|
||||
|
||||
if (!hasFrontmatter) {
|
||||
// No existing frontmatter: stamp a fresh block and (if body lacks markdown
|
||||
// structure) wrap under a derived heading.
|
||||
const title = deriveTitle(rawBody);
|
||||
const fm: Record<string, unknown> = {
|
||||
type: opts.type ?? 'note',
|
||||
title,
|
||||
captured_via: opts.source ?? 'capture-cli',
|
||||
captured_at: nowIso,
|
||||
};
|
||||
const looksMarkdown = /^#{1,6}\s/.test(rawBody.trimStart());
|
||||
const body = looksMarkdown ? rawBody : `# ${title}\n\n${rawBody}`;
|
||||
return matter.stringify(body, fm);
|
||||
}
|
||||
|
||||
// Existing frontmatter: parse, merge user-wins, re-emit as a SINGLE block.
|
||||
let parsed: matter.GrayMatterFile<string>;
|
||||
try {
|
||||
parsed = matter(rawBody);
|
||||
} catch (e) {
|
||||
throw new Error(
|
||||
`malformed frontmatter in capture input: ${e instanceof Error ? e.message : String(e)}`,
|
||||
);
|
||||
}
|
||||
const userFm = (parsed.data ?? {}) as Record<string, unknown>;
|
||||
const merged: Record<string, unknown> = {
|
||||
// Spread user's declared keys first so 'description', 'tags', etc. pass through.
|
||||
...userFm,
|
||||
// Then apply auto-fields with the precedence rules above. The explicit
|
||||
// assignment AFTER the spread is intentional: it lets us implement the
|
||||
// mixed precedence (CLI flag wins for `type`; user wins for `title`/
|
||||
// `captured_via`/`captured_at`) in one expression per key.
|
||||
type: opts.type ?? userFm.type ?? 'note',
|
||||
title: userFm.title ?? deriveTitle(parsed.content),
|
||||
captured_via: userFm.captured_via ?? opts.source ?? 'capture-cli',
|
||||
captured_at: userFm.captured_at ?? nowIso,
|
||||
};
|
||||
return matter.stringify(parsed.content, merged);
|
||||
}
|
||||
|
||||
/**
|
||||
* Build the put_page content (frontmatter + body). The user's --type and
|
||||
* the auto-stamped capture provenance go in the frontmatter so future
|
||||
* tools (e.g. the inbox triage UI) can find captures.
|
||||
*
|
||||
* v0.39.3.0: delegates to `mergeCaptureFrontmatter` so files with existing
|
||||
* frontmatter merge instead of double-wrap (BUG-1).
|
||||
*/
|
||||
function buildContent(rawBody: string, opts: RunOpts): string {
|
||||
const frontmatterLines: string[] = ['---'];
|
||||
frontmatterLines.push(`type: ${opts.type ?? 'note'}`);
|
||||
// Title defaults to the first non-empty line of the body, capped at 80 chars.
|
||||
const firstLine = rawBody.split('\n').find((l) => l.trim().length > 0) ?? '';
|
||||
const title = firstLine.replace(/^#+\s*/, '').slice(0, 80) || 'Capture';
|
||||
frontmatterLines.push(`title: ${JSON.stringify(title)}`);
|
||||
frontmatterLines.push(`captured_via: ${opts.source ?? 'capture-cli'}`);
|
||||
frontmatterLines.push(`captured_at: ${new Date().toISOString()}`);
|
||||
frontmatterLines.push('---');
|
||||
// Heuristic: if the raw body doesn't already look like markdown (no headings,
|
||||
// no leading frontmatter), wrap in a heading for readability. Inline thoughts
|
||||
// are usually unstructured.
|
||||
const lookskMarkdown = /^#{1,6}\s/.test(rawBody.trimStart()) || rawBody.startsWith('---');
|
||||
const body = lookskMarkdown ? rawBody : `# ${title}\n\n${rawBody}`;
|
||||
return frontmatterLines.join('\n') + '\n\n' + body;
|
||||
return mergeCaptureFrontmatter(rawBody, opts);
|
||||
}
|
||||
|
||||
interface CaptureResult {
|
||||
@@ -188,13 +342,33 @@ export async function runCapture(engine: BrainEngine | null, args: string[]): Pr
|
||||
return;
|
||||
}
|
||||
|
||||
// Resolve the source content.
|
||||
let rawBody: string;
|
||||
// v0.39.3.0 CV7: thin-client installs cannot scope --source via put_page
|
||||
// params. The server's auth/transport layer (OAuth client registration's
|
||||
// source_id / federated_read) determines source scope. Reject early with
|
||||
// a clear error pointing at the right fix; matches CV6 trust posture
|
||||
// (server owns the source scope, client cannot override per-call).
|
||||
const cfg = loadConfig();
|
||||
if (parsed.source && isThinClient(cfg)) {
|
||||
console.error(`gbrain capture: --source is not supported on thin-client installs.`);
|
||||
console.error(`Server-side OAuth client registration determines source scope.`);
|
||||
console.error(`On the server, run:`);
|
||||
console.error(` gbrain auth register-client <name> --source ${parsed.source} --scopes "read write"`);
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
// v0.39.3.0 CV10 — resolve content as a Buffer FIRST so the binary guard
|
||||
// sees real bytes (not UTF-8-decoded mojibake). Stdin uses the same
|
||||
// Buffer path so --stdin gets the same protection as --file.
|
||||
let rawBuffer: Buffer | null = null;
|
||||
let inputLabel = ''; // for error messages
|
||||
if (parsed.stdin) {
|
||||
rawBody = await readStdin();
|
||||
rawBuffer = await readStdinBuffer();
|
||||
inputLabel = 'stdin';
|
||||
} else if (parsed.filePath) {
|
||||
inputLabel = parsed.filePath;
|
||||
try {
|
||||
rawBody = readFileSync(parsed.filePath, 'utf8');
|
||||
// No encoding => returns Buffer; binary guard sees raw bytes.
|
||||
rawBuffer = readFileSync(parsed.filePath);
|
||||
} catch (e) {
|
||||
console.error(
|
||||
`gbrain capture: failed to read ${parsed.filePath}: ${e instanceof Error ? e.message : String(e)}`,
|
||||
@@ -202,27 +376,80 @@ export async function runCapture(engine: BrainEngine | null, args: string[]): Pr
|
||||
process.exit(1);
|
||||
}
|
||||
} else if (parsed.content) {
|
||||
rawBody = parsed.content;
|
||||
// Positional content: already a JS string, but route through a Buffer
|
||||
// for guard parity (a positional string with a literal `\x00` would
|
||||
// also be rejected). Inline thoughts almost never trigger this; it's
|
||||
// pure defense-in-depth.
|
||||
rawBuffer = Buffer.from(parsed.content, 'utf8');
|
||||
inputLabel = 'positional content';
|
||||
} else {
|
||||
console.error('gbrain capture: provide content positionally, --file PATH, or --stdin');
|
||||
console.error('Run `gbrain capture --help` for examples.');
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
rawBody = rawBody.trim();
|
||||
if (rawBody.length === 0) {
|
||||
// CV10 binary guard. Scans the first 8KB for NUL bytes; rejects with a
|
||||
// friendly message before UTF-8 decode mangles arbitrary bytes.
|
||||
const nullByteOffset = detectBinaryNullByte(rawBuffer!);
|
||||
if (nullByteOffset !== -1) {
|
||||
console.error(
|
||||
`gbrain capture: refusing to capture binary content from ${inputLabel}\n` +
|
||||
` Found null byte at offset ${nullByteOffset} (first 8KB scan); ` +
|
||||
`text files (including UTF-8 CJK/emoji/BOM) never contain NUL bytes.\n` +
|
||||
` Binary content (image/audio/video/pdf) is not yet supported via capture — ` +
|
||||
`install a content-type processor skillpack when available.`,
|
||||
);
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
// Decode to UTF-8 string AFTER the binary guard.
|
||||
const rawBody = rawBuffer!.toString('utf8');
|
||||
|
||||
// CV9: refuse empty content based on the normalized form (whitespace-only
|
||||
// input is still empty), but preserve original bytes in storedBody for
|
||||
// the put_page write so CRLF / BOM / trailing-newline tests pass.
|
||||
const normalizedBody = normalizeForHash(rawBody);
|
||||
if (normalizedBody.length === 0) {
|
||||
console.error('gbrain capture: refusing to capture empty content');
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
const slug = parsed.slug ?? defaultSlug(rawBody);
|
||||
// CV15: route source resolution through the canonical 6-tier chain
|
||||
// (flag → env → dotfile → local_path → brain_default → seed_default).
|
||||
// resolveSourceWithTier handles the assertSourceExists check and throws
|
||||
// a friendly error BEFORE put_page is called if the source is missing.
|
||||
// Only run on the LOCAL path — thin-client has no engine handle to
|
||||
// probe the sources table; CV7 above already rejected explicit --source
|
||||
// on thin-client. Implicit source resolution on thin-client uses
|
||||
// 'default' (the server's auth layer scopes the actual write).
|
||||
let resolvedSourceId = 'default';
|
||||
if (!isThinClient(cfg) && engine) {
|
||||
try {
|
||||
const { source_id } = await resolveSourceWithTier(engine, parsed.source ?? null);
|
||||
resolvedSourceId = source_id;
|
||||
} catch (e) {
|
||||
// assertSourceExists throws "Source 'X' not found. Available sources: ..."
|
||||
console.error(`gbrain capture: ${e instanceof Error ? e.message : String(e)}`);
|
||||
process.exit(1);
|
||||
}
|
||||
}
|
||||
|
||||
// CV8 (CLI side): content_hash for the RECEIPT comes from the normalized
|
||||
// rawBody, NOT the assembled fullContent which contains a timestamp.
|
||||
// The daemon's 24h LRU dedup keys on this hash; identical captures must
|
||||
// produce identical hashes. The DB content_hash (importFromContent at
|
||||
// src/core/import-file.ts) gets the same treatment in Phase 3d.
|
||||
const slug = parsed.slug ?? defaultSlug(normalizedBody);
|
||||
const fullContent = buildContent(rawBody, parsed);
|
||||
const capturedAt = new Date().toISOString();
|
||||
const contentHash = computeContentHash(fullContent);
|
||||
const contentHash = computeContentHash(normalizedBody);
|
||||
|
||||
// Thin-client install: route through put_page over MCP. The server's
|
||||
// write-through plumbing handles disk persistence.
|
||||
const cfg = loadConfig();
|
||||
// write-through plumbing handles disk persistence. Per CV6 trust gate,
|
||||
// the server overrides ANY provenance params we send to `mcp:put_page`
|
||||
// — so we deliberately do NOT thread source_kind/source_uri/ingested_via
|
||||
// through the wire (would be discarded server-side, and we don't want
|
||||
// to suggest the values reached the DB column when they didn't).
|
||||
if (isThinClient(cfg)) {
|
||||
let raw: unknown;
|
||||
try {
|
||||
@@ -233,10 +460,21 @@ export async function runCapture(engine: BrainEngine | null, args: string[]): Pr
|
||||
{ timeoutMs: 30_000 },
|
||||
);
|
||||
} catch (e) {
|
||||
console.error(
|
||||
`gbrain capture: remote put_page failed: ${e instanceof Error ? e.message : String(e)}`,
|
||||
);
|
||||
console.error('Run `gbrain remote doctor` to diagnose the connection.');
|
||||
// A2/T1: detect server-side FK violation and rewrite to friendly hint.
|
||||
// RemoteMcpError wraps the server's error envelope; the underlying
|
||||
// PG message is in the wrapped string.
|
||||
const hint = maybeRewriteSourceFkError(e, parsed.source ?? resolvedSourceId);
|
||||
if (hint) {
|
||||
console.error(`gbrain capture: ${hint}`);
|
||||
} else if (e instanceof RemoteMcpError) {
|
||||
console.error(`gbrain capture: remote put_page failed: ${e.message}`);
|
||||
console.error('Run `gbrain remote doctor` to diagnose the connection.');
|
||||
} else {
|
||||
console.error(
|
||||
`gbrain capture: remote put_page failed: ${e instanceof Error ? e.message : String(e)}`,
|
||||
);
|
||||
console.error('Run `gbrain remote doctor` to diagnose the connection.');
|
||||
}
|
||||
process.exit(1);
|
||||
}
|
||||
const remoteResult = unpackToolResult<{
|
||||
@@ -252,7 +490,11 @@ export async function runCapture(engine: BrainEngine | null, args: string[]): Pr
|
||||
content_hash: contentHash,
|
||||
written: remoteResult.write_through?.written ?? false,
|
||||
path: remoteResult.write_through?.path,
|
||||
source_kind: parsed.source ?? 'capture-cli',
|
||||
// CV3: source_kind ALWAYS 'capture-cli' for capture invocations,
|
||||
// regardless of --source. --source maps to source_id (the DB FK),
|
||||
// not the ingestion-channel taxonomy. Conflating these was the
|
||||
// root cause of WARN-8's audit-trail labeling problem.
|
||||
source_kind: 'capture-cli',
|
||||
captured_at: capturedAt,
|
||||
};
|
||||
printReceipt(result, parsed.quiet ?? false, parsed.json ?? false);
|
||||
@@ -280,13 +522,28 @@ export async function runCapture(engine: BrainEngine | null, args: string[]): Pr
|
||||
},
|
||||
dryRun: false,
|
||||
remote: false,
|
||||
// OperationContext.sourceId is REQUIRED as of v0.34.1 D4 — match the
|
||||
// dispatcher's behavior of defaulting to 'default' when the caller
|
||||
// didn't pass --source.
|
||||
sourceId: parsed.source ?? 'default',
|
||||
// v0.39.3.0 CV15: thread the resolved source from the canonical 6-tier
|
||||
// chain (was `parsed.source ?? 'default'` pre-fix, which silently
|
||||
// ignored env / dotfile / local_path / brain_default tiers — divergent
|
||||
// from every other CLI op's behavior).
|
||||
sourceId: resolvedSourceId,
|
||||
};
|
||||
try {
|
||||
const result = (await putPageOp.handler(ctx, { slug, content: fullContent })) as {
|
||||
// v0.39.3.0 WARN-8: pass provenance params to put_page. CV3 source_kind
|
||||
// is always 'capture-cli'; ingested_via is 'put_page' (the write API),
|
||||
// source_uri identifies the file path or stdin marker.
|
||||
const sourceUri = parsed.filePath
|
||||
? `file://${parsed.filePath}`
|
||||
: parsed.stdin
|
||||
? 'stdin'
|
||||
: 'cli-positional';
|
||||
const result = (await putPageOp.handler(ctx, {
|
||||
slug,
|
||||
content: fullContent,
|
||||
source_kind: 'capture-cli',
|
||||
source_uri: sourceUri,
|
||||
ingested_via: 'capture-cli',
|
||||
})) as {
|
||||
slug: string;
|
||||
status?: string;
|
||||
chunks?: number;
|
||||
@@ -300,16 +557,26 @@ export async function runCapture(engine: BrainEngine | null, args: string[]): Pr
|
||||
content_hash: contentHash,
|
||||
written: result.write_through?.written ?? false,
|
||||
path: result.write_through?.path,
|
||||
source_kind: parsed.source ?? 'capture-cli',
|
||||
// CV3: source_kind is the channel taxonomy, NOT the DB source FK.
|
||||
source_kind: 'capture-cli',
|
||||
captured_at: capturedAt,
|
||||
},
|
||||
parsed.quiet ?? false,
|
||||
parsed.json ?? false,
|
||||
);
|
||||
} catch (e) {
|
||||
console.error(
|
||||
`gbrain capture: put_page failed: ${e instanceof Error ? e.message : String(e)}`,
|
||||
);
|
||||
// A2: detect FK violation on sources table and rewrite to friendly hint.
|
||||
// resolveSourceWithTier above usually catches missing sources upstream,
|
||||
// but a TOCTOU race (source deleted between pre-flight and put_page) or
|
||||
// an explicit --source bypass would surface here.
|
||||
const hint = maybeRewriteSourceFkError(e, parsed.source ?? resolvedSourceId);
|
||||
if (hint) {
|
||||
console.error(`gbrain capture: ${hint}`);
|
||||
} else {
|
||||
console.error(
|
||||
`gbrain capture: put_page failed: ${e instanceof Error ? e.message : String(e)}`,
|
||||
);
|
||||
}
|
||||
process.exit(1);
|
||||
}
|
||||
}
|
||||
@@ -318,5 +585,10 @@ export async function runCapture(engine: BrainEngine | null, args: string[]): Pr
|
||||
export const __testing = {
|
||||
defaultSlug,
|
||||
buildContent,
|
||||
mergeCaptureFrontmatter,
|
||||
deriveTitle,
|
||||
parseArgs,
|
||||
detectBinaryNullByte,
|
||||
normalizeForHash,
|
||||
maybeRewriteSourceFkError,
|
||||
};
|
||||
|
||||
@@ -26,7 +26,7 @@ import { operations, OperationError } from '../core/operations.ts';
|
||||
import type { OperationContext, AuthInfo } from '../core/operations.ts';
|
||||
import { GBrainOAuthProvider } from '../core/oauth-provider.ts';
|
||||
import type { SqlQuery } from '../core/oauth-provider.ts';
|
||||
import { hasScope, ALLOWED_SCOPES_LIST } from '../core/scope.ts';
|
||||
import { hasScope, ALLOWED_SCOPES_LIST, normalizeScopesInput } from '../core/scope.ts';
|
||||
import { summarizeMcpParams, dispatchToolCall } from '../mcp/dispatch.ts';
|
||||
import { paramDefToSchema } from '../mcp/tool-defs.ts';
|
||||
import { getBrainHotMemoryMeta } from '../core/facts/meta-hook.ts';
|
||||
@@ -1090,12 +1090,33 @@ export async function runServeHttp(engine: BrainEngine, options: ServeHttpOption
|
||||
// Register client from admin dashboard
|
||||
app.post('/admin/api/register-client', requireAdmin, express.json(), async (req: Request, res: Response) => {
|
||||
try {
|
||||
const { name, scopes, tokenTtl, grantTypes, redirectUris, tokenEndpointAuthMethod } = req.body;
|
||||
// v0.39.3.0 WARN-9 + CV12: accept BOTH `scopes` (admin SPA convention)
|
||||
// AND `scope` (OAuth wire-format convention, singular). The pre-fix
|
||||
// code destructured only `scopes` and used `scopes || 'read'` which:
|
||||
// - Silently ignored `scope` requests (always defaulted to 'read')
|
||||
// - Threw on array input because registerClientManual's parseScopeString
|
||||
// calls .split(' ') which arrays don't have
|
||||
// - Accepted `['read write']` (space-in-element bug shape codex flagged)
|
||||
// and other malformed inputs
|
||||
// normalizeScopesInput handles all four valid shapes (string, string[],
|
||||
// missing, empty) and rejects the rest with a structured 400.
|
||||
const { name, tokenTtl, grantTypes, redirectUris, tokenEndpointAuthMethod } = req.body;
|
||||
const rawScopes = (req.body as Record<string, unknown>).scopes ?? (req.body as Record<string, unknown>).scope;
|
||||
if (!name) { res.status(400).json({ error: 'Name required' }); return; }
|
||||
let scopeString: string;
|
||||
try {
|
||||
scopeString = normalizeScopesInput(rawScopes);
|
||||
} catch (e) {
|
||||
res.status(400).json({
|
||||
error: 'invalid_scopes',
|
||||
message: e instanceof Error ? e.message : String(e),
|
||||
});
|
||||
return;
|
||||
}
|
||||
const grants = Array.isArray(grantTypes) && grantTypes.length > 0 ? grantTypes : ['client_credentials'];
|
||||
const uris = Array.isArray(redirectUris) ? redirectUris : [];
|
||||
const result = await oauthProvider.registerClientManual(
|
||||
name, grants, scopes || 'read', uris,
|
||||
name, grants, scopeString, uris,
|
||||
);
|
||||
// Public client (PKCE-only, no secret): NULL out client_secret_hash and
|
||||
// set auth method so the SDK's clientAuth middleware skips the hash-vs-
|
||||
@@ -1584,6 +1605,30 @@ export async function runServeHttp(engine: BrainEngine, options: ServeHttpOption
|
||||
const authInfo = (req as Request & { auth?: AuthInfo }).auth as AuthInfo;
|
||||
const agentName = authInfo.clientName ?? authInfo.clientId;
|
||||
|
||||
// v0.39.3.0 BUG-2: outer try/catch ensures any unexpected throw
|
||||
// returns a JSON envelope instead of leaking express's default HTML
|
||||
// error page. Mirrors the MCP handler's F14 pattern (serve-http.ts
|
||||
// F14 envelope around transport.handleRequest). The `!res.headersSent`
|
||||
// guard (codex F#16) prevents a second-response attempt if the throw
|
||||
// happens after the inner queue.add try/catch already responded.
|
||||
try {
|
||||
|
||||
// v0.39.3.0 BUG-2: explicit null/undefined guard BEFORE body coercion.
|
||||
// When the request has no body at all (no Content-Length header, no
|
||||
// body-parser fed us anything), `req.body` is `undefined`. The pre-fix
|
||||
// code's `else` branch called `Buffer.from(JSON.stringify(undefined),
|
||||
// 'utf8')` — and `JSON.stringify(undefined) === undefined` (the
|
||||
// literal, not the string), which makes `Buffer.from(undefined, 'utf8')`
|
||||
// throw TypeError. Express's default error handler then served an HTML
|
||||
// 500 page. Guard fires first to keep the response shape JSON.
|
||||
if (req.body == null) {
|
||||
res.status(400).json({
|
||||
error: 'empty_body',
|
||||
message: 'POST /ingest requires a non-empty body',
|
||||
});
|
||||
return;
|
||||
}
|
||||
|
||||
// Express raw() returns a Buffer. Decode as UTF-8; reject non-UTF-8
|
||||
// bytes loudly so callers know their payload was garbled.
|
||||
let body: Buffer;
|
||||
@@ -1593,7 +1638,9 @@ export async function runServeHttp(engine: BrainEngine, options: ServeHttpOption
|
||||
body = Buffer.from(req.body, 'utf8');
|
||||
} else {
|
||||
// express.json or urlencoded fired earlier in the chain and parsed
|
||||
// for us. Re-serialize so we can hash and forward.
|
||||
// for us. Re-serialize so we can hash and forward. The null/undefined
|
||||
// case is already guarded above so JSON.stringify produces a real
|
||||
// string here (objects round-trip, primitives become their JSON form).
|
||||
body = Buffer.from(JSON.stringify(req.body), 'utf8');
|
||||
}
|
||||
|
||||
@@ -1721,6 +1768,22 @@ export async function runServeHttp(engine: BrainEngine, options: ServeHttpOption
|
||||
message: msg,
|
||||
});
|
||||
}
|
||||
|
||||
// v0.39.3.0 BUG-2: outer try/catch close — anything that throws BEFORE
|
||||
// the inner queue.add try/catch lands here. The headersSent guard
|
||||
// (codex F#16) skips the second-response attempt if the inner block
|
||||
// already wrote a response and then threw on a downstream line (e.g.
|
||||
// a logging side-effect after `res.status(202).json(...)`).
|
||||
} catch (outerErr) {
|
||||
const msg = outerErr instanceof Error ? outerErr.message : String(outerErr);
|
||||
console.error('POST /ingest unexpected handler error:', msg);
|
||||
if (!res.headersSent) {
|
||||
res.status(500).json({
|
||||
error: 'internal_error',
|
||||
message: msg,
|
||||
});
|
||||
}
|
||||
}
|
||||
},
|
||||
);
|
||||
|
||||
|
||||
@@ -0,0 +1,70 @@
|
||||
/**
|
||||
* v0.39.3.0 WARN-10 + CV11 + T4 — brainstorm timeout classifier.
|
||||
*
|
||||
* Pre-fix: brainstorm + lsd consistently produced no output on
|
||||
* PgBouncer transaction-mode environments when listPrefixSampledPages
|
||||
* or hybrid search exceeded the statement timeout. The error reached
|
||||
* runBrainstormCli as a generic Error and surfaced as the catch-all
|
||||
* 'gbrain: unknown error' fallback, silently consuming the user's
|
||||
* cost-preview wait and producing zero ideas.
|
||||
*
|
||||
* Fix: orchestrator wraps its entire body in one try/catch (covers
|
||||
* every 57014 source — prefix enumeration, hybrid search, domain bank
|
||||
* fetch, embedding fetch, save phase). On classifier-positive match,
|
||||
* throw `StructuredAgentError` with code='brainstorm_timeout' and a
|
||||
* hint covering all PG cancel sub-causes (statement timeout, lock
|
||||
* timeout, user cancel). Non-57014 errors rethrow as-is so unrelated
|
||||
* bug classes (OAuth, network, embedding-provider) keep their natural
|
||||
* shape.
|
||||
*
|
||||
* Per T4: match SQLSTATE 57014 specifically (the spec-defined
|
||||
* 'query_canceled' code). Per CV11 + codex F#19: hint phrasing reads
|
||||
* 'query canceled' (generic) rather than 'statement timeout' (one
|
||||
* sub-cause) so the message stays honest across all three PG cancel
|
||||
* sub-causes.
|
||||
*
|
||||
* Per T4 + codex F#20: covers ALL orchestrator-internal SQL sites by
|
||||
* being applied at the entry-point wrap, not at per-call wraps. Adding
|
||||
* a new SQL call inside the orchestrator is automatically covered.
|
||||
*/
|
||||
|
||||
import { StructuredAgentError } from '../errors.ts';
|
||||
|
||||
/**
|
||||
* Detect Postgres SQLSTATE 57014 (query_canceled) on an unknown thrown
|
||||
* value. The error shape varies by driver — postgres.js attaches
|
||||
* `.code`, pg attaches `.code`, PGLite surfaces through `.code` or
|
||||
* exposes the SQLSTATE in the message. Cast generically to a record
|
||||
* type so we can probe both.
|
||||
*/
|
||||
export function isQueryCanceledError(err: unknown): boolean {
|
||||
if (!err || typeof err !== 'object') return false;
|
||||
const e = err as Record<string, unknown>;
|
||||
// postgres.js + pg both attach the SQLSTATE on .code as a string.
|
||||
if (e.code === '57014') return true;
|
||||
// PGLite exposes via .code or surfaces in .message.
|
||||
if (e.sqlState === '57014') return true;
|
||||
// Last-resort message scan (some wrapping layers strip the .code).
|
||||
const msg = err instanceof Error ? err.message : String(err);
|
||||
return /canceling statement due to|query.*canceled|sqlstate[\s:]+57014/i.test(msg);
|
||||
}
|
||||
|
||||
/**
|
||||
* Convert any 57014 error into a `StructuredAgentError` with the
|
||||
* brainstorm-specific hint. Non-57014 errors pass through unchanged
|
||||
* (caller rethrows). The hint deliberately covers the three PG cancel
|
||||
* sub-causes per codex F#19 so the message stays true when the cancel
|
||||
* was from a lock timeout or user-cancel, not just statement timeout.
|
||||
*/
|
||||
export function classifyBrainstormError(err: unknown): unknown {
|
||||
if (!isQueryCanceledError(err)) return err;
|
||||
return new StructuredAgentError({
|
||||
class: 'BrainstormError',
|
||||
code: 'brainstorm_timeout',
|
||||
message: 'Brainstorm query was canceled by Postgres',
|
||||
hint:
|
||||
'Causes: statement_timeout (often PgBouncer transaction-mode), lock_timeout, or user-cancel. ' +
|
||||
'Workarounds: try a smaller --limit, retry once, or ask your brain admin about ' +
|
||||
'statement_timeout / PgBouncer settings. The orchestrator entry-point wrap covers every internal SQL site.',
|
||||
});
|
||||
}
|
||||
@@ -25,6 +25,8 @@ import type { BrainEngine } from '../engine.ts';
|
||||
import { chat as defaultChat, embedQuery, type ChatResult, type ChatOpts } from '../ai/gateway.ts';
|
||||
import { hybridSearch, hybridSearchCached } from '../search/hybrid.ts';
|
||||
import { fetchFar, type CloseRef, type FarPage } from './domain-bank.ts';
|
||||
import { StructuredAgentError } from '../errors.ts';
|
||||
import { classifyBrainstormError } from './error-classify.ts';
|
||||
import {
|
||||
runJudge,
|
||||
BRAINSTORM_JUDGE_CONFIG,
|
||||
@@ -469,17 +471,57 @@ const DEFAULT_PARALLELISM = 4;
|
||||
* `gbrain lsd`) calls this with the question + profile; renders the result
|
||||
* via formatBrainstormMarkdown; optionally saves via put_page.
|
||||
*/
|
||||
/**
|
||||
* v0.39.3.0 WARN-10 + CV11 — Public entry point. Wraps the impl in a
|
||||
* single try/catch that classifies Postgres SQLSTATE 57014
|
||||
* (query_canceled) into a `StructuredAgentError` with code
|
||||
* 'brainstorm_timeout'. Covers EVERY internal SQL site (hybrid search,
|
||||
* domain bank fetch, prefix enumeration, embedding fetch, save phase)
|
||||
* by virtue of being the single wrap at the function boundary.
|
||||
*
|
||||
* Non-57014 errors rethrow unchanged so unrelated bug classes (OAuth,
|
||||
* AI gateway, network) keep their natural shape — codex F#20: catching
|
||||
* only the 57014 class is honest classification, NOT broad swallowing.
|
||||
*
|
||||
* Per A3 (plan-eng-review): reuses `StructuredAgentError` from
|
||||
* src/core/errors.ts (the v0.19.0 envelope every new agent-facing
|
||||
* surface uses) rather than introducing a new BrainstormError class.
|
||||
*/
|
||||
export async function runBrainstorm(
|
||||
engine: BrainEngine,
|
||||
config: { embedding_model?: string; emotional_weight?: { user_holder?: string } },
|
||||
opts: BrainstormOptions
|
||||
): Promise<BrainstormResult> {
|
||||
// T10: install a gateway-layer BudgetTracker scope around the whole run
|
||||
// so every gateway.chat / embed call (the cross generations + judge +
|
||||
// question embed) auto-records cost via the AsyncLocalStorage from T3.
|
||||
// The cap mirrors the orchestrator's maxCostUsd so the gateway can
|
||||
// hard-fail via BudgetExhausted(reason:'cost') if a single under-
|
||||
// estimated call leaks past the ceiling (TX1).
|
||||
// v0.39.3.0 (Phase 5, CV11+T4): outer try/catch around the orchestrator
|
||||
// body classifies SQLSTATE 57014 (query_canceled — covers
|
||||
// statement_timeout, lock_timeout, user-cancel) into a typed
|
||||
// StructuredAgentError with code='brainstorm_timeout'. Non-57014 errors
|
||||
// (including BudgetExhausted from the inner BudgetTracker wrap) pass
|
||||
// through unchanged. Per A3: reuses StructuredAgentError (the v0.19.0
|
||||
// envelope) rather than introducing a new BrainstormError class.
|
||||
try {
|
||||
return await runBrainstormImpl(engine, config, opts);
|
||||
} catch (err) {
|
||||
// classifyBrainstormError returns the original error unchanged when
|
||||
// it's NOT a 57014 cancel; otherwise returns a typed
|
||||
// StructuredAgentError ready to throw.
|
||||
throw classifyBrainstormError(err);
|
||||
}
|
||||
}
|
||||
|
||||
async function runBrainstormImpl(
|
||||
engine: BrainEngine,
|
||||
config: { embedding_model?: string; emotional_weight?: { user_holder?: string } },
|
||||
opts: BrainstormOptions,
|
||||
): Promise<BrainstormResult> {
|
||||
// v0.39.0.0 T10: install a gateway-layer BudgetTracker scope around the
|
||||
// whole run so every gateway.chat / embed call (the cross generations +
|
||||
// judge + question embed) auto-records cost via the AsyncLocalStorage
|
||||
// from T3. The cap mirrors the orchestrator's maxCostUsd so the gateway
|
||||
// can hard-fail via BudgetExhausted(reason:'cost') if a single under-
|
||||
// estimated call leaks past the ceiling (TX1). BudgetExhausted is NOT
|
||||
// SQLSTATE 57014, so the outer classifyBrainstormError lets it pass
|
||||
// through with its original shape (which the CLI formatter renders).
|
||||
const _runTracker = new BudgetTracker({
|
||||
label: `brainstorm.${opts.profile?.label ?? 'brainstorm'}`,
|
||||
maxCostUsd: opts.maxCostUsd ?? 5,
|
||||
|
||||
@@ -26,6 +26,7 @@
|
||||
*/
|
||||
|
||||
import type { BrainEngine } from '../engine.ts';
|
||||
import { GBrainError } from '../types.ts';
|
||||
|
||||
export const FACTS_ABSORB_REASONS = [
|
||||
'gateway_error',
|
||||
@@ -36,6 +37,18 @@ export const FACTS_ABSORB_REASONS = [
|
||||
'pipeline_error',
|
||||
] as const;
|
||||
|
||||
// v0.39.3.0 WARN-4 + CV13 — module-scoped flag so the first-occurrence
|
||||
// diagnostic log fires ONCE per process. Subsequent occurrences of the
|
||||
// same 'No database connection' class are suppressed (keeps captures
|
||||
// quiet) but the first one prints enough context to triage why the
|
||||
// facts subsystem is calling logIngest on a disconnected engine.
|
||||
// Exported as a test seam so the assertion can reset between runs.
|
||||
let _hasLoggedDisconnectedFactsAbsorb = false;
|
||||
/** @internal — test seam */
|
||||
export function _resetFactsAbsorbDisconnectedFlagForTests(): void {
|
||||
_hasLoggedDisconnectedFactsAbsorb = false;
|
||||
}
|
||||
|
||||
export type FactsAbsorbReason = typeof FACTS_ABSORB_REASONS[number];
|
||||
|
||||
/**
|
||||
@@ -69,8 +82,33 @@ export async function writeFactsAbsorbLog(
|
||||
summary: `${reason}: ${cleanedDetail}`,
|
||||
});
|
||||
} catch (e) {
|
||||
// Don't let logging failures cascade. The whole point of D5 is
|
||||
// observability — but observability can't break the runtime path.
|
||||
// v0.39.3.0 WARN-4 + CQ1 — typed access via instanceof + .problem field
|
||||
// (NOT string-match on e.message). The 'No database connection' class
|
||||
// fires after every `gbrain capture` invocation because the facts
|
||||
// subsystem opens its own engine handle that isn't connected in the
|
||||
// CLI capture path. Per-capture noise is suppressed; CV13 prints
|
||||
// ONE first-occurrence stack trace so the next user reporting it
|
||||
// gives us the call site to fix in v0.38.4.
|
||||
if (e instanceof GBrainError && e.problem === 'No database connection') {
|
||||
if (!_hasLoggedDisconnectedFactsAbsorb) {
|
||||
_hasLoggedDisconnectedFactsAbsorb = true;
|
||||
// eslint-disable-next-line no-console
|
||||
console.warn(
|
||||
`[facts:absorb] suppressed: 'No database connection' fires on a separate engine handle ` +
|
||||
`(known WARN-4 in v0.38; subsequent occurrences silent this process). ` +
|
||||
`First-occurrence trace for v0.38.4 diagnosis:\n${e.stack ?? '<no stack>'}`,
|
||||
);
|
||||
}
|
||||
// Subsequent occurrences silent — the page write itself succeeded;
|
||||
// the facts:absorb log is a courtesy that the doctor health check
|
||||
// reads. A connection-bound log site already filed the v0.38.4 TODO
|
||||
// (see TODOS.md).
|
||||
return;
|
||||
}
|
||||
// All other failures keep the loud warn. Don't let logging failures
|
||||
// cascade — observability can't break the runtime path — but DO let
|
||||
// the operator see real subsystem errors (PgBouncer crash, schema
|
||||
// drift, etc.) instead of suppressing them globally.
|
||||
// eslint-disable-next-line no-console
|
||||
console.warn(
|
||||
`[facts:absorb] failed to log ${reason} for ${ref}: ${e instanceof Error ? e.message : String(e)}`,
|
||||
|
||||
+44
-3
@@ -215,13 +215,26 @@ export async function importFromContent(
|
||||
*/
|
||||
forceRechunk?: boolean;
|
||||
/**
|
||||
* v0.39 T1.5: active schema pack for type inference. When set, parseMarkdown
|
||||
* v0.39.0.0 T1.5: active schema pack for type inference. When set, parseMarkdown
|
||||
* uses the pack's path_prefixes instead of the hardcoded gbrain-base table.
|
||||
* When unset, falls back to pre-v0.39 behavior (parity gate stays green).
|
||||
* Callers thread this from `loadActivePack(ctx)` once per command —
|
||||
* NEVER per file inside sync (codex perf finding #7).
|
||||
*/
|
||||
activePack?: { page_types: ReadonlyArray<{ name: string; path_prefixes: ReadonlyArray<string> }> };
|
||||
/**
|
||||
* v0.39.3.0 provenance write-through (WARN-8). When set, threaded to
|
||||
* `tx.putPage` so the page's `source_kind`, `source_uri`,
|
||||
* `ingested_via` DB columns get populated. The trust gate lives at the
|
||||
* `put_page` op layer — by the time importFromContent sees these, the
|
||||
* caller is already trusted (capture CLI sets them; remote MCP callers
|
||||
* had theirs overridden to `mcp:put_page` upstream). `ingested_at` is
|
||||
* NOT a caller-controllable param; the engine's putPage stamps it
|
||||
* server-side via now() when any provenance write fires.
|
||||
*/
|
||||
source_kind?: string | null;
|
||||
source_uri?: string | null;
|
||||
ingested_via?: string | null;
|
||||
} = {},
|
||||
): Promise<ImportResult> {
|
||||
// v0.18.0+ multi-source: when caller is syncing under a non-default source,
|
||||
@@ -245,14 +258,34 @@ export async function importFromContent(
|
||||
|
||||
const parsed = parseMarkdown(content, slug + '.md', { activePack: opts.activePack });
|
||||
|
||||
// Hash includes ALL fields for idempotency (not just compiled_truth + timeline)
|
||||
// v0.39.3.0 CV8 — DB content_hash excludes timestamp-bearing frontmatter
|
||||
// keys so identical body content from `gbrain capture` (which stamps
|
||||
// `captured_at` and `ingested_at` per call) produces a stable hash.
|
||||
// Pre-fix, every capture-cli invocation produced a fresh hash because
|
||||
// the timestamp changed, defeating:
|
||||
// - the existing.content_hash === hash short-circuit below (every
|
||||
// capture re-chunked + re-embedded unchanged content — wasted
|
||||
// embedding spend)
|
||||
// - the daemon's 24h LRU dedup (separate consumer keyed on same hash)
|
||||
//
|
||||
// We strip ONLY the timestamp keys, not the whole frontmatter object.
|
||||
// Stripping all frontmatter would regress sync: a user adding a tag
|
||||
// would update the frontmatter without changing the body, the hash
|
||||
// would not change, and tag reconciliation would silently no-op
|
||||
// (this function returns early on hash-match).
|
||||
const HASH_EPHEMERAL_FRONTMATTER_KEYS = ['captured_at', 'ingested_at'];
|
||||
const stableFrontmatter: Record<string, unknown> = { ...parsed.frontmatter };
|
||||
for (const k of HASH_EPHEMERAL_FRONTMATTER_KEYS) {
|
||||
delete stableFrontmatter[k];
|
||||
}
|
||||
// Hash includes all meaningful fields for idempotency.
|
||||
const hash = createHash('sha256')
|
||||
.update(JSON.stringify({
|
||||
title: parsed.title,
|
||||
type: parsed.type,
|
||||
compiled_truth: parsed.compiled_truth,
|
||||
timeline: parsed.timeline,
|
||||
frontmatter: parsed.frontmatter,
|
||||
frontmatter: stableFrontmatter,
|
||||
tags: parsed.tags.sort(),
|
||||
}))
|
||||
.digest('hex');
|
||||
@@ -343,6 +376,14 @@ export async function importFromContent(
|
||||
// code can resolve frontmatter-fallback slugs back to their files.
|
||||
chunker_version: MARKDOWN_CHUNKER_VERSION,
|
||||
source_path: opts.sourcePath ?? null,
|
||||
// v0.39.3.0 provenance write-through (WARN-8). Engine layer applies
|
||||
// COALESCE-preserve UPDATE so omitting these on a later put_page
|
||||
// doesn't erase the original ingestion's audit trail.
|
||||
source_kind: opts.source_kind ?? null,
|
||||
source_uri: opts.source_uri ?? null,
|
||||
ingested_via: opts.ingested_via ?? null,
|
||||
// ingested_at is server-stamped at the engine layer when any
|
||||
// provenance write fires; never client-controlled.
|
||||
}, txOpts);
|
||||
|
||||
// Tag reconciliation: remove stale, add current
|
||||
|
||||
@@ -538,12 +538,49 @@ const put_page: Operation = {
|
||||
params: {
|
||||
slug: { type: 'string', required: true, description: 'Page slug' },
|
||||
content: { type: 'string', required: true, description: 'Full markdown content with YAML frontmatter' },
|
||||
// v0.39.3.0 provenance write-through (WARN-8 + A1 + CV6). Optional fields
|
||||
// for trusted local callers (capture CLI, autopilot, dream cycle). Remote
|
||||
// MCP callers (ctx.remote !== false) have their values OVERRIDDEN with
|
||||
// server stamps below; the params are accepted on the wire only so the
|
||||
// op schema stays uniform across transports. Audit-trail spoofing is
|
||||
// closed structurally — clients cannot poison source_kind labels.
|
||||
source_kind: { type: 'string', required: false, description: 'Ingestion channel taxonomy (capture-cli | put_page | webhook | …). Remote callers: SERVER-STAMPED, client value ignored.' },
|
||||
source_uri: { type: 'string', required: false, description: 'Original URI/path/message-id the event carried. Remote callers: SERVER-STAMPED null.' },
|
||||
ingested_via: { type: 'string', required: false, description: 'Richer label paired with source_kind. Remote callers: SERVER-STAMPED.' },
|
||||
},
|
||||
mutating: true,
|
||||
scope: 'write',
|
||||
handler: async (ctx, p) => {
|
||||
const slug = p.slug as string;
|
||||
|
||||
// v0.39.3.0 CV6 trust gate for provenance write-through (WARN-8).
|
||||
// Only trusted LOCAL callers (ctx.remote === false — capture CLI,
|
||||
// autopilot, dream cycle, file watcher) may populate source_kind /
|
||||
// source_uri / ingested_via from their own state. Anything else
|
||||
// (HTTP MCP, stdio MCP, subagent) gets the server-stamped
|
||||
// `mcp:put_page` regardless of what was passed.
|
||||
//
|
||||
// Closes the spoofing surface CV6 identified: pre-fix a write-scope
|
||||
// OAuth token could send `source_kind: 'capture-cli'` to poison the
|
||||
// audit trail. Fail-closed: `ctx.remote === false` is the ONLY truthy
|
||||
// condition that admits client-supplied provenance.
|
||||
let provenanceKind: string | null;
|
||||
let provenanceUri: string | null;
|
||||
let provenanceVia: string | null;
|
||||
if (ctx.remote === false) {
|
||||
// Trusted local caller: honor the client params (may be null/undefined
|
||||
// for legacy local callers that don't set them).
|
||||
provenanceKind = (p.source_kind as string | undefined) ?? null;
|
||||
provenanceUri = (p.source_uri as string | undefined) ?? null;
|
||||
provenanceVia = (p.ingested_via as string | undefined) ?? null;
|
||||
} else {
|
||||
// Remote caller or unset trust: server stamps. Mirrors the existing
|
||||
// write-through stamping at the file-side (~:637).
|
||||
provenanceKind = 'mcp:put_page';
|
||||
provenanceUri = null;
|
||||
provenanceVia = 'mcp:put_page';
|
||||
}
|
||||
|
||||
// Subagent namespace enforcement (v0.15+). Runs BEFORE the dry-run
|
||||
// short-circuit so preview calls surface the same rejection. Confines
|
||||
// LLM-driven writes to wiki/agents/<subagentId>/... — no leading slash
|
||||
@@ -609,7 +646,17 @@ const put_page: Operation = {
|
||||
const result = await importFromContent(ctx.engine, slug, p.content as string, {
|
||||
noEmbed,
|
||||
...(ctx.sourceId ? { sourceId: ctx.sourceId } : {}),
|
||||
// v0.39.0.0 T1.5: pack-aware type inference (loaded above; legacy
|
||||
// inferType behavior when undefined).
|
||||
...(activePack ? { activePack } : {}),
|
||||
// v0.39.3.0 provenance write-through (WARN-8). Trust-filtered values
|
||||
// computed above; ingested_at is server-stamped at the engine layer.
|
||||
// Null-valued fields signal "no provenance write this call" and the
|
||||
// engine's COALESCE-preserve UPDATE keeps the prior first-write
|
||||
// record intact (CV12 audit-trail survival).
|
||||
source_kind: provenanceKind,
|
||||
source_uri: provenanceUri,
|
||||
ingested_via: provenanceVia,
|
||||
});
|
||||
|
||||
// v0.39 T13 — auto-prompt on first unknown-type write.
|
||||
|
||||
@@ -671,7 +671,8 @@ export class PGLiteEngine implements BrainEngine {
|
||||
where.push('deleted_at IS NULL');
|
||||
}
|
||||
const { rows } = await this.db.query(
|
||||
`SELECT id, source_id, slug, type, title, compiled_truth, timeline, frontmatter, content_hash, created_at, updated_at, deleted_at
|
||||
`SELECT id, source_id, slug, type, title, compiled_truth, timeline, frontmatter, content_hash, created_at, updated_at, deleted_at,
|
||||
source_kind, source_uri, ingested_via, ingested_at
|
||||
FROM pages WHERE ${where.join(' AND ')} LIMIT 1`,
|
||||
params
|
||||
);
|
||||
@@ -702,9 +703,17 @@ export class PGLiteEngine implements BrainEngine {
|
||||
// v0.32.7 CJK wave: chunker_version + source_path columns.
|
||||
const chunkerVersion = page.chunker_version ?? null;
|
||||
const sourcePath = page.source_path ?? null;
|
||||
// v0.39.3.0 provenance write-through (WARN-8 + CV12). Mirrors postgres-engine.ts.
|
||||
// Server stamps `ingested_at = now()` ONLY when any provenance field is being
|
||||
// written this call. COALESCE-preserve UPDATE keeps the prior first-write
|
||||
// timestamp intact so the audit trail survives routine edits.
|
||||
const sourceKind = page.source_kind ?? null;
|
||||
const sourceUri = page.source_uri ?? null;
|
||||
const ingestedVia = page.ingested_via ?? null;
|
||||
const ingestedAt = (sourceKind || sourceUri || ingestedVia) ? new Date().toISOString() : null;
|
||||
const { rows } = await this.db.query(
|
||||
`INSERT INTO pages (source_id, slug, type, page_kind, title, compiled_truth, timeline, frontmatter, content_hash, updated_at, effective_date, effective_date_source, import_filename, chunker_version, source_path)
|
||||
VALUES ($1, $2, $3, $4, $5, $6, $7, $8::jsonb, $9, now(), $10::timestamptz, $11, $12, COALESCE($13, 1), $14)
|
||||
`INSERT INTO pages (source_id, slug, type, page_kind, title, compiled_truth, timeline, frontmatter, content_hash, updated_at, effective_date, effective_date_source, import_filename, chunker_version, source_path, source_kind, source_uri, ingested_via, ingested_at)
|
||||
VALUES ($1, $2, $3, $4, $5, $6, $7, $8::jsonb, $9, now(), $10::timestamptz, $11, $12, COALESCE($13, 1), $14, $15, $16, $17, $18::timestamptz)
|
||||
ON CONFLICT (source_id, slug) DO UPDATE SET
|
||||
type = EXCLUDED.type,
|
||||
page_kind = EXCLUDED.page_kind,
|
||||
@@ -718,9 +727,13 @@ export class PGLiteEngine implements BrainEngine {
|
||||
effective_date_source = COALESCE(EXCLUDED.effective_date_source, pages.effective_date_source),
|
||||
import_filename = COALESCE(EXCLUDED.import_filename, pages.import_filename),
|
||||
chunker_version = COALESCE(EXCLUDED.chunker_version, pages.chunker_version),
|
||||
source_path = COALESCE(EXCLUDED.source_path, pages.source_path)
|
||||
RETURNING id, source_id, slug, type, title, compiled_truth, timeline, frontmatter, content_hash, created_at, updated_at, effective_date, effective_date_source, import_filename`,
|
||||
[sourceId, slug, page.type, pageKind, page.title, page.compiled_truth, page.timeline || '', JSON.stringify(frontmatter), hash, effectiveDate, effectiveDateSource, importFilename, chunkerVersion, sourcePath]
|
||||
source_path = COALESCE(EXCLUDED.source_path, pages.source_path),
|
||||
source_kind = COALESCE(EXCLUDED.source_kind, pages.source_kind),
|
||||
source_uri = COALESCE(EXCLUDED.source_uri, pages.source_uri),
|
||||
ingested_via = COALESCE(EXCLUDED.ingested_via, pages.ingested_via),
|
||||
ingested_at = COALESCE(EXCLUDED.ingested_at, pages.ingested_at)
|
||||
RETURNING id, source_id, slug, type, title, compiled_truth, timeline, frontmatter, content_hash, created_at, updated_at, effective_date, effective_date_source, import_filename, source_kind, source_uri, ingested_via, ingested_at`,
|
||||
[sourceId, slug, page.type, pageKind, page.title, page.compiled_truth, page.timeline || '', JSON.stringify(frontmatter), hash, effectiveDate, effectiveDateSource, importFilename, chunkerVersion, sourcePath, sourceKind, sourceUri, ingestedVia, ingestedAt]
|
||||
);
|
||||
return rowToPage(rows[0] as Record<string, unknown>);
|
||||
}
|
||||
|
||||
@@ -741,7 +741,8 @@ export class PostgresEngine implements BrainEngine {
|
||||
const sourceCondition = sourceId ? sql`AND source_id = ${sourceId}` : sql``;
|
||||
const deletedCondition = includeDeleted ? sql`` : sql`AND deleted_at IS NULL`;
|
||||
const rows = await sql`
|
||||
SELECT id, source_id, slug, type, title, compiled_truth, timeline, frontmatter, content_hash, created_at, updated_at, deleted_at
|
||||
SELECT id, source_id, slug, type, title, compiled_truth, timeline, frontmatter, content_hash, created_at, updated_at, deleted_at,
|
||||
source_kind, source_uri, ingested_via, ingested_at
|
||||
FROM pages
|
||||
WHERE slug = ${slug} ${sourceCondition} ${deletedCondition}
|
||||
LIMIT 1
|
||||
@@ -776,9 +777,18 @@ export class PostgresEngine implements BrainEngine {
|
||||
// v0.32.7 CJK wave: chunker_version + source_path columns.
|
||||
const chunkerVersion = page.chunker_version ?? null;
|
||||
const sourcePath = page.source_path ?? null;
|
||||
// v0.39.3.0 provenance write-through (WARN-8 + CV12). Server stamps
|
||||
// `ingested_at = now()` ONLY when any provenance is being written —
|
||||
// null `source_kind` / `source_uri` / `ingested_via` means no provenance
|
||||
// write fired this call, and COALESCE-preserve UPDATE keeps the prior
|
||||
// first-write timestamp intact (audit trail survives routine edits).
|
||||
const sourceKind = page.source_kind ?? null;
|
||||
const sourceUri = page.source_uri ?? null;
|
||||
const ingestedVia = page.ingested_via ?? null;
|
||||
const ingestedAt = (sourceKind || sourceUri || ingestedVia) ? new Date() : null;
|
||||
const rows = await sql`
|
||||
INSERT INTO pages (source_id, slug, type, page_kind, title, compiled_truth, timeline, frontmatter, content_hash, updated_at, effective_date, effective_date_source, import_filename, chunker_version, source_path)
|
||||
VALUES (${sourceId}, ${slug}, ${page.type}, ${pageKind}, ${page.title}, ${page.compiled_truth}, ${page.timeline || ''}, ${sql.json(frontmatter as Parameters<typeof sql.json>[0])}, ${hash}, now(), ${effectiveDate}, ${effectiveDateSource}, ${importFilename}, COALESCE(${chunkerVersion}::smallint, 1), ${sourcePath})
|
||||
INSERT INTO pages (source_id, slug, type, page_kind, title, compiled_truth, timeline, frontmatter, content_hash, updated_at, effective_date, effective_date_source, import_filename, chunker_version, source_path, source_kind, source_uri, ingested_via, ingested_at)
|
||||
VALUES (${sourceId}, ${slug}, ${page.type}, ${pageKind}, ${page.title}, ${page.compiled_truth}, ${page.timeline || ''}, ${sql.json(frontmatter as Parameters<typeof sql.json>[0])}, ${hash}, now(), ${effectiveDate}, ${effectiveDateSource}, ${importFilename}, COALESCE(${chunkerVersion}::smallint, 1), ${sourcePath}, ${sourceKind}, ${sourceUri}, ${ingestedVia}, ${ingestedAt})
|
||||
ON CONFLICT (source_id, slug) DO UPDATE SET
|
||||
type = EXCLUDED.type,
|
||||
page_kind = EXCLUDED.page_kind,
|
||||
@@ -792,8 +802,12 @@ export class PostgresEngine implements BrainEngine {
|
||||
effective_date_source = COALESCE(EXCLUDED.effective_date_source, pages.effective_date_source),
|
||||
import_filename = COALESCE(EXCLUDED.import_filename, pages.import_filename),
|
||||
chunker_version = COALESCE(EXCLUDED.chunker_version, pages.chunker_version),
|
||||
source_path = COALESCE(EXCLUDED.source_path, pages.source_path)
|
||||
RETURNING id, source_id, slug, type, title, compiled_truth, timeline, frontmatter, content_hash, created_at, updated_at, effective_date, effective_date_source, import_filename
|
||||
source_path = COALESCE(EXCLUDED.source_path, pages.source_path),
|
||||
source_kind = COALESCE(EXCLUDED.source_kind, pages.source_kind),
|
||||
source_uri = COALESCE(EXCLUDED.source_uri, pages.source_uri),
|
||||
ingested_via = COALESCE(EXCLUDED.ingested_via, pages.ingested_via),
|
||||
ingested_at = COALESCE(EXCLUDED.ingested_at, pages.ingested_at)
|
||||
RETURNING id, source_id, slug, type, title, compiled_truth, timeline, frontmatter, content_hash, created_at, updated_at, effective_date, effective_date_source, import_filename, source_kind, source_uri, ingested_via, ingested_at
|
||||
`;
|
||||
return rowToPage(rows[0]);
|
||||
}
|
||||
|
||||
@@ -117,3 +117,77 @@ export function parseScopeString(s: string | undefined | null): string[] {
|
||||
if (!s) return [];
|
||||
return s.split(' ').filter(Boolean);
|
||||
}
|
||||
|
||||
/**
|
||||
* v0.39.3.0 WARN-9 + CV12 — normalize the `scopes` (or `scope`) field that
|
||||
* arrives from the admin SPA's register-client request body to the
|
||||
* space-separated wire format `registerClientManual` expects. Three valid
|
||||
* input shapes; everything else throws Error to be caught and surfaced
|
||||
* as 400 invalid_scopes.
|
||||
*
|
||||
* Valid shapes:
|
||||
* - `undefined` / `null` / missing → defaults to 'read'
|
||||
* - `string` → split on /\s+/, filter empty,
|
||||
* dedupe, validate each, re-join
|
||||
* - `string[]` → validate each element is a non-
|
||||
* empty single scope (no internal
|
||||
* whitespace — that's the bug
|
||||
* shape codex flagged where
|
||||
* ['read write'] silently lets
|
||||
* `read write` through as a single
|
||||
* unknown scope), dedupe, validate
|
||||
* each against ALLOWED_SCOPES
|
||||
*
|
||||
* Rejection cases:
|
||||
* - non-string non-array (number, object, boolean) → Error
|
||||
* - empty array after normalization → Error
|
||||
* - array element with internal whitespace → Error (the ['read write'] bug)
|
||||
* - array element that's not a string (null, number, ...) → Error
|
||||
* - any element not in ALLOWED_SCOPES → InvalidScopeError
|
||||
*
|
||||
* Returns the space-separated string (e.g. 'read write') in sorted order
|
||||
* for determinism so two registrations with the same scope set produce
|
||||
* identical DB rows.
|
||||
*/
|
||||
export function normalizeScopesInput(raw: unknown): string {
|
||||
if (raw == null) return 'read';
|
||||
|
||||
let candidates: string[];
|
||||
|
||||
if (typeof raw === 'string') {
|
||||
candidates = raw.split(/\s+/).filter(Boolean);
|
||||
} else if (Array.isArray(raw)) {
|
||||
for (const el of raw) {
|
||||
if (typeof el !== 'string') {
|
||||
throw new Error(
|
||||
`scopes array must contain only strings, got ${el === null ? 'null' : typeof el}`,
|
||||
);
|
||||
}
|
||||
if (el.length === 0) {
|
||||
throw new Error('scopes array must not contain empty strings');
|
||||
}
|
||||
if (/\s/.test(el)) {
|
||||
throw new Error(
|
||||
`scopes array element "${el}" contains whitespace. Each element must be a single scope name; use ['read', 'write'] not ['read write'].`,
|
||||
);
|
||||
}
|
||||
}
|
||||
candidates = raw as string[];
|
||||
} else {
|
||||
throw new Error(
|
||||
`scopes must be a string or array of strings, got ${typeof raw}`,
|
||||
);
|
||||
}
|
||||
|
||||
// Dedupe via Set + sort for stable output.
|
||||
const deduped = Array.from(new Set(candidates)).sort();
|
||||
|
||||
if (deduped.length === 0) {
|
||||
throw new Error('scopes is empty after normalization');
|
||||
}
|
||||
|
||||
// Validate against ALLOWED_SCOPES (throws InvalidScopeError on miss).
|
||||
assertAllowedScopes(deduped);
|
||||
|
||||
return deduped.join(' ');
|
||||
}
|
||||
|
||||
@@ -123,6 +123,22 @@ export interface Page {
|
||||
* Test fixtures building synthetic Page rows must include this field.
|
||||
*/
|
||||
source_id: string;
|
||||
|
||||
// v0.39.3.0 provenance read-path (WARN-8 + CV5). Migration v81 columns
|
||||
// surfaced through getPage / list_pages so `gbrain call get_page | jq
|
||||
// .source_kind` actually returns the value the put_page op wrote. NULL
|
||||
// on historical pages that pre-date v0.38. Three-state read pattern
|
||||
// (undefined: not in projection, null: column NULL, populated: real
|
||||
// value) — matches the v0.26.5 deleted_at convention so SELECTs that
|
||||
// don't project these columns continue to compile.
|
||||
/** Ingestion-channel taxonomy. See PageInput.source_kind for the closed set. */
|
||||
source_kind?: string | null;
|
||||
/** Original URI/path/message-id the ingestion event carried. */
|
||||
source_uri?: string | null;
|
||||
/** Richer label paired with source_kind (often same value; indexable separately). */
|
||||
ingested_via?: string | null;
|
||||
/** Server-stamped first-write audit timestamp; CV12 COALESCE-preserved across edits. */
|
||||
ingested_at?: Date | null;
|
||||
}
|
||||
|
||||
export type EffectiveDateSource =
|
||||
@@ -175,6 +191,42 @@ export interface PageInput {
|
||||
* NULL on legacy / non-file callers (MCP `put_page`, fixture seeds).
|
||||
*/
|
||||
source_path?: string | null;
|
||||
|
||||
// v0.39.3.0 provenance write-through (WARN-8 + A1 + CV6).
|
||||
// Migration v81 added these 4 nullable columns to `pages`. Until v0.39.3.0,
|
||||
// put_page wrote them into the file's frontmatter (via the write-through
|
||||
// path) but never to the DB columns — `gbrain call get_page | jq .source_kind`
|
||||
// returned null even though the JSON receipt claimed `capture-cli`. These
|
||||
// params close the loop. Trust gate lives at the put_page op layer
|
||||
// (operations.ts): ONLY ctx.remote === false (trusted local callers like
|
||||
// capture CLI, autopilot, dream cycle) may populate these fields. Remote
|
||||
// MCP callers get server-stamped `mcp:put_page` regardless of what they
|
||||
// pass (CV6 fail-closed; closes the spoofing surface where a write-scope
|
||||
// OAuth token could poison the audit trail with arbitrary labels).
|
||||
//
|
||||
// Storage shape: nullable TEXT columns; the engine's putPage SQL uses
|
||||
// COALESCE-preserve UPDATE semantics (CV12) so omitting these fields on
|
||||
// a later put_page (e.g. a routine edit) does NOT erase the original
|
||||
// ingestion's audit trail. First-write wins.
|
||||
/**
|
||||
* Ingestion-channel taxonomy: 'capture-cli' | 'webhook' | 'file-watcher' |
|
||||
* 'inbox-folder' | 'cron-scheduler' | 'put_page' | 'mcp:put_page' |
|
||||
* '<skillpack-kind>'. Server stamps `mcp:put_page` for remote callers.
|
||||
*/
|
||||
source_kind?: string | null;
|
||||
/** Original URI/path/message-id the event carried (file path, mail message-id, URL). NULL when unknown. */
|
||||
source_uri?: string | null;
|
||||
/**
|
||||
* Richer label paired with source_kind (often the same value; kept narrow
|
||||
* + indexable separately per migration v81's documented intent).
|
||||
*/
|
||||
ingested_via?: string | null;
|
||||
/**
|
||||
* Always server-stamped at put_page time when any provenance is being
|
||||
* written (NEVER client-controlled — keeps the audit timestamp truthful).
|
||||
* NULL on historical rows that pre-date v0.38.
|
||||
*/
|
||||
ingested_at?: Date | null;
|
||||
}
|
||||
|
||||
export interface PageFilters {
|
||||
|
||||
@@ -77,6 +77,14 @@ export function rowToPage(row: Record<string, unknown>): Page {
|
||||
const salienceTouchedAt = readOptionalDate(row.salience_touched_at);
|
||||
const effectiveDateSource = row.effective_date_source as Page['effective_date_source'] | undefined;
|
||||
const importFilename = row.import_filename as string | null | undefined;
|
||||
// v0.39.3.0 CV5 — three-state read for provenance columns. Matches the
|
||||
// v0.26.5 deleted_at pattern: undefined when the SELECT projection didn't
|
||||
// include the column (older code paths); null when the column is NULL
|
||||
// (historical pre-v0.38 row); populated when v0.38+ ingestion stamped it.
|
||||
const sourceKind = row.source_kind === undefined ? undefined : (row.source_kind as string | null);
|
||||
const sourceUri = row.source_uri === undefined ? undefined : (row.source_uri as string | null);
|
||||
const ingestedVia = row.ingested_via === undefined ? undefined : (row.ingested_via as string | null);
|
||||
const ingestedAt = readOptionalDate(row.ingested_at);
|
||||
return {
|
||||
id: row.id as number,
|
||||
slug: row.slug as string,
|
||||
@@ -96,6 +104,12 @@ export function rowToPage(row: Record<string, unknown>): Page {
|
||||
...(effectiveDateSource !== undefined && { effective_date_source: effectiveDateSource }),
|
||||
...(importFilename !== undefined && { import_filename: importFilename }),
|
||||
...(salienceTouchedAt !== undefined && { salience_touched_at: salienceTouchedAt }),
|
||||
// v0.39.3.0 (columns added in migration v81 — WARN-8 + CV5). Three-state
|
||||
// optional read; absent SELECT projections compile unchanged.
|
||||
...(sourceKind !== undefined && { source_kind: sourceKind }),
|
||||
...(sourceUri !== undefined && { source_uri: sourceUri }),
|
||||
...(ingestedVia !== undefined && { ingested_via: ingestedVia }),
|
||||
...(ingestedAt !== undefined && { ingested_at: ingestedAt }),
|
||||
// v0.31.12: propagate source_id so downstream callers (embed, reconcile-links)
|
||||
// can thread it through getChunks / upsertChunks without defaulting to 'default'.
|
||||
// v0.32.8: Page.source_id is required. Every SELECT feeding rowToPage now
|
||||
|
||||
@@ -0,0 +1,147 @@
|
||||
/**
|
||||
* v0.39.3.0 WARN-10 + CV11 + T4 — brainstorm timeout classifier.
|
||||
*
|
||||
* Pre-fix: brainstorm + lsd silently produced no output on PgBouncer
|
||||
* transaction-mode environments when listPrefixSampledPages or hybrid
|
||||
* search exceeded the statement timeout. The error surfaced as a
|
||||
* generic 'gbrain: unknown error' fallback, consuming the cost-preview
|
||||
* wait and producing zero ideas.
|
||||
*
|
||||
* Per CV11 + T4: orchestrator-level wrap classifies any SQLSTATE 57014
|
||||
* (query_canceled — covers statement_timeout, lock_timeout, user_cancel)
|
||||
* as a typed `StructuredAgentError` with code='brainstorm_timeout' and
|
||||
* a hint covering all three PG cancel sub-causes. Non-57014 errors
|
||||
* rethrow as-is.
|
||||
*
|
||||
* Per A3: reuse StructuredAgentError (the v0.19.0 envelope) — no new
|
||||
* error class.
|
||||
*
|
||||
* Tests cover the pure classifier helper (isQueryCanceledError +
|
||||
* classifyBrainstormError) at unit level. The orchestrator-wrap +
|
||||
* CLI-formatter end-to-end behavior is implicit: classifier returns
|
||||
* the typed error → orchestrator throws it → CLI formatter prints
|
||||
* 'Error [brainstorm_timeout]: ... Hint: ...' (cli.ts:188-191 pattern).
|
||||
*/
|
||||
|
||||
import { describe, test, expect } from 'bun:test';
|
||||
import { isQueryCanceledError, classifyBrainstormError } from '../src/core/brainstorm/error-classify.ts';
|
||||
import { StructuredAgentError } from '../src/core/errors.ts';
|
||||
|
||||
describe('isQueryCanceledError — SQLSTATE 57014 detection across driver shapes', () => {
|
||||
test('postgres.js shape: { code: "57014" } → true', () => {
|
||||
const err = Object.assign(new Error('canceling statement due to statement timeout'), { code: '57014' });
|
||||
expect(isQueryCanceledError(err)).toBe(true);
|
||||
});
|
||||
|
||||
test('alternate shape: { sqlState: "57014" } → true', () => {
|
||||
const err = Object.assign(new Error('query canceled'), { sqlState: '57014' });
|
||||
expect(isQueryCanceledError(err)).toBe(true);
|
||||
});
|
||||
|
||||
test('message-only fallback: "canceling statement due to ..." → true', () => {
|
||||
expect(isQueryCanceledError(new Error('canceling statement due to statement timeout'))).toBe(true);
|
||||
expect(isQueryCanceledError(new Error('canceling statement due to lock timeout'))).toBe(true);
|
||||
expect(isQueryCanceledError(new Error('canceling statement due to user request'))).toBe(true);
|
||||
});
|
||||
|
||||
test('case-insensitive SQLSTATE in message: "SQLSTATE: 57014" → true', () => {
|
||||
expect(isQueryCanceledError(new Error('PG error sqlstate: 57014'))).toBe(true);
|
||||
});
|
||||
|
||||
test('different error codes do NOT match (non-cancel PG errors rethrow)', () => {
|
||||
const err42 = Object.assign(new Error('division by zero'), { code: '22012' });
|
||||
const err23 = Object.assign(new Error('FK violation'), { code: '23503' });
|
||||
expect(isQueryCanceledError(err42)).toBe(false);
|
||||
expect(isQueryCanceledError(err23)).toBe(false);
|
||||
});
|
||||
|
||||
test('non-DB errors do NOT match', () => {
|
||||
expect(isQueryCanceledError(new Error('connection refused'))).toBe(false);
|
||||
expect(isQueryCanceledError(new Error('OAuth token expired'))).toBe(false);
|
||||
expect(isQueryCanceledError(new Error('something timed out'))).toBe(false); // missing PG-specific markers
|
||||
});
|
||||
|
||||
test('null / undefined / non-error → false', () => {
|
||||
expect(isQueryCanceledError(null)).toBe(false);
|
||||
expect(isQueryCanceledError(undefined)).toBe(false);
|
||||
expect(isQueryCanceledError('57014')).toBe(false); // string, not an object
|
||||
expect(isQueryCanceledError(42)).toBe(false);
|
||||
});
|
||||
});
|
||||
|
||||
describe('classifyBrainstormError — convert 57014 to StructuredAgentError', () => {
|
||||
test('57014 error becomes StructuredAgentError with brainstorm_timeout code', () => {
|
||||
const orig = Object.assign(new Error('canceling statement due to statement timeout'), { code: '57014' });
|
||||
const classified = classifyBrainstormError(orig);
|
||||
expect(classified).toBeInstanceOf(StructuredAgentError);
|
||||
const env = (classified as StructuredAgentError).envelope;
|
||||
expect(env.code).toBe('brainstorm_timeout');
|
||||
expect(env.class).toBe('BrainstormError');
|
||||
expect(env.message).toContain('canceled');
|
||||
expect(env.hint).toContain('statement_timeout');
|
||||
expect(env.hint).toContain('PgBouncer');
|
||||
expect(env.hint).toContain('lock_timeout');
|
||||
expect(env.hint).toContain('user-cancel');
|
||||
});
|
||||
|
||||
test('hint covers all 3 PG cancel sub-causes per codex F#19', () => {
|
||||
const orig = Object.assign(new Error('any 57014'), { code: '57014' });
|
||||
const env = (classifyBrainstormError(orig) as StructuredAgentError).envelope;
|
||||
// All three sub-causes named so the message is honest regardless of which fired
|
||||
expect(env.hint).toMatch(/statement_timeout/);
|
||||
expect(env.hint).toMatch(/lock_timeout/);
|
||||
expect(env.hint).toMatch(/user-cancel/);
|
||||
// Plus actionable workaround
|
||||
expect(env.hint).toMatch(/--limit/);
|
||||
});
|
||||
|
||||
test('non-57014 errors pass through unchanged (codex F#20 — no silent swallow)', () => {
|
||||
const oauthErr = new Error('OAuth token expired');
|
||||
expect(classifyBrainstormError(oauthErr)).toBe(oauthErr); // SAME REFERENCE
|
||||
|
||||
const fkErr = Object.assign(new Error('FK violation'), { code: '23503' });
|
||||
expect(classifyBrainstormError(fkErr)).toBe(fkErr);
|
||||
|
||||
const netErr = new Error('ECONNREFUSED');
|
||||
expect(classifyBrainstormError(netErr)).toBe(netErr);
|
||||
});
|
||||
|
||||
test('null / undefined pass through unchanged', () => {
|
||||
expect(classifyBrainstormError(null)).toBeNull();
|
||||
expect(classifyBrainstormError(undefined)).toBeUndefined();
|
||||
});
|
||||
|
||||
test('classified error message is descriptive on its own (Error.message channel)', () => {
|
||||
const orig = Object.assign(new Error('57014'), { code: '57014' });
|
||||
const classified = classifyBrainstormError(orig);
|
||||
expect((classified as Error).message).toContain('BrainstormError');
|
||||
expect((classified as Error).message).toContain('canceled');
|
||||
expect((classified as Error).message).toContain('Causes');
|
||||
});
|
||||
});
|
||||
|
||||
describe('orchestrator entry-point wrap (CV11 single-point classification)', () => {
|
||||
// Smoke test that the entry wrap is in place. The orchestrator code
|
||||
// imports classifyBrainstormError; the public runBrainstorm catches
|
||||
// any thrown error and routes it through the classifier.
|
||||
//
|
||||
// This is a source-shape regression guard: if a future refactor
|
||||
// moves the wrap to per-call try/catches (the codex-F#20 anti-pattern),
|
||||
// this test fails because the imports drift.
|
||||
test('orchestrator.ts imports classifyBrainstormError + StructuredAgentError', async () => {
|
||||
const src = await Bun.file('src/core/brainstorm/orchestrator.ts').text();
|
||||
expect(src).toContain("import { StructuredAgentError } from '../errors.ts'");
|
||||
expect(src).toContain("import { classifyBrainstormError } from './error-classify.ts'");
|
||||
expect(src).toContain('classifyBrainstormError(err)');
|
||||
// The wrap must be at the PUBLIC entry point, NOT a per-call wrap
|
||||
expect(src).toMatch(/export async function runBrainstorm\s*\([\s\S]*?try\s*{[\s\S]*?await runBrainstormImpl/);
|
||||
});
|
||||
|
||||
test('commands/brainstorm.ts has the CLI formatter for StructuredAgentError', async () => {
|
||||
const src = await Bun.file('src/commands/brainstorm.ts').text();
|
||||
expect(src).toContain("import { StructuredAgentError } from '../core/errors.ts'");
|
||||
expect(src).toContain('err instanceof StructuredAgentError');
|
||||
expect(src).toContain("Error [");
|
||||
expect(src).toContain('Hint:');
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,202 @@
|
||||
/**
|
||||
* v0.39.3.0 BUG-1: capture --file doubles frontmatter on files that
|
||||
* already have frontmatter. The fix replaces buildContent's
|
||||
* always-prepend behavior with a gray-matter-backed merge that user-wins
|
||||
* for declared keys.
|
||||
*
|
||||
* These tests pin the merge contract per CQ2 (boil-the-lake, 13 cases).
|
||||
* Each test feeds a known input through `mergeCaptureFrontmatter` and
|
||||
* asserts the resulting frontmatter + body shape. We round-trip via
|
||||
* gray-matter's `matter()` parser so assertions are against the parsed
|
||||
* shape rather than fragile string-equality.
|
||||
*/
|
||||
|
||||
import { describe, test, expect } from 'bun:test';
|
||||
import matter from 'gray-matter';
|
||||
import { __testing as captureTesting } from '../src/commands/capture.ts';
|
||||
|
||||
const { mergeCaptureFrontmatter, deriveTitle } = captureTesting;
|
||||
|
||||
function parse(out: string) {
|
||||
const m = matter(out);
|
||||
return { fm: m.data as Record<string, unknown>, body: m.content };
|
||||
}
|
||||
|
||||
describe('mergeCaptureFrontmatter — 13 cases (CQ2 boil-the-lake)', () => {
|
||||
test('1. file without frontmatter wraps as today (no regression)', () => {
|
||||
const input = 'remember to follow up';
|
||||
const out = mergeCaptureFrontmatter(input, {});
|
||||
const { fm, body } = parse(out);
|
||||
expect(fm.type).toBe('note');
|
||||
expect(fm.title).toBe('remember to follow up');
|
||||
expect(fm.captured_via).toBe('capture-cli');
|
||||
expect(typeof fm.captured_at).toBe('string');
|
||||
// Body wrapped under derived heading since input lacks markdown structure
|
||||
expect(body).toContain('# remember to follow up');
|
||||
expect(body).toContain('remember to follow up');
|
||||
// EXACTLY one frontmatter block (regression for BUG-1).
|
||||
// Split on `^---$` boundary: a single block yields 3 segments
|
||||
// (pre-opening empty, fm body, after-closing body); two blocks yield 5.
|
||||
expect(out.split(/^---\s*$/m).length).toBe(3);
|
||||
});
|
||||
|
||||
test('2. file with frontmatter and no title — capture derives title from body', () => {
|
||||
const input = '---\ntype: meeting\n---\n\nNotes from the team sync\n\nMore stuff';
|
||||
const out = mergeCaptureFrontmatter(input, {});
|
||||
const { fm, body } = parse(out);
|
||||
expect(fm.type).toBe('meeting'); // user's type preserved
|
||||
expect(fm.title).toBe('Notes from the team sync'); // derived from body
|
||||
expect(body.trim()).toContain('Notes from the team sync');
|
||||
// Single frontmatter block
|
||||
const blocks = out.match(/^---\s*$/gm);
|
||||
expect(blocks?.length).toBe(2); // opening + closing
|
||||
});
|
||||
|
||||
test('3. file with frontmatter that has title — user title wins', () => {
|
||||
const input = '---\ntitle: User Defined Title\ntype: idea\n---\n\nbody text';
|
||||
const out = mergeCaptureFrontmatter(input, {});
|
||||
const { fm } = parse(out);
|
||||
expect(fm.title).toBe('User Defined Title');
|
||||
expect(fm.type).toBe('idea');
|
||||
});
|
||||
|
||||
test('4. file with captured_at — user value wins (NOT overwritten by now)', () => {
|
||||
const userTs = '2020-01-01T00:00:00.000Z';
|
||||
const input = `---\ncaptured_at: ${userTs}\ntype: note\n---\n\nbody`;
|
||||
const out = mergeCaptureFrontmatter(input, {});
|
||||
const { fm } = parse(out);
|
||||
// gray-matter's YAML parser auto-coerces ISO timestamps to Date objects.
|
||||
// Compare via .toISOString() so the contract holds across either shape.
|
||||
const got = fm.captured_at;
|
||||
const gotIso = got instanceof Date ? got.toISOString() : String(got);
|
||||
expect(gotIso).toBe(userTs);
|
||||
});
|
||||
|
||||
test('5. frontmatter + body-side horizontal rule — TOP frontmatter merged, rule preserved', () => {
|
||||
const input = '---\ntitle: Top\n---\n\nfirst section\n\n---\n\nsecond section';
|
||||
const out = mergeCaptureFrontmatter(input, {});
|
||||
const { fm, body } = parse(out);
|
||||
expect(fm.title).toBe('Top');
|
||||
// The body-side `---` horizontal rule must survive intact
|
||||
expect(body).toContain('first section');
|
||||
expect(body).toContain('---');
|
||||
expect(body).toContain('second section');
|
||||
});
|
||||
|
||||
test('6. title extraction never picks `---` as the title', () => {
|
||||
// No-frontmatter path: deriveTitle skips `---` lines
|
||||
expect(deriveTitle('---\n\nreal first line')).toBe('real first line');
|
||||
// Bare `---` followed by content
|
||||
expect(deriveTitle('---\n# heading\nrest')).toBe('heading');
|
||||
// Only `---` and blanks → falls back to 'Capture'
|
||||
expect(deriveTitle('---\n---\n')).toBe('Capture');
|
||||
});
|
||||
|
||||
test('7. CJK title preserved through merge', () => {
|
||||
const input = '---\ntitle: 測試 brain entry\n---\n\nbody';
|
||||
const out = mergeCaptureFrontmatter(input, {});
|
||||
const { fm } = parse(out);
|
||||
expect(fm.title).toBe('測試 brain entry');
|
||||
});
|
||||
|
||||
test('8. Windows CRLF line endings in frontmatter preserved', () => {
|
||||
const input = '---\r\ntype: meeting\r\ntitle: CRLF Test\r\n---\r\n\r\nbody line\r\n';
|
||||
const out = mergeCaptureFrontmatter(input, {});
|
||||
const { fm } = parse(out);
|
||||
expect(fm.type).toBe('meeting');
|
||||
expect(fm.title).toBe('CRLF Test');
|
||||
});
|
||||
|
||||
test('9. UTF-8 BOM at start handled cleanly', () => {
|
||||
const input = '---\ntitle: BOM Test\n---\n\nbody';
|
||||
const out = mergeCaptureFrontmatter(input, {});
|
||||
const { fm } = parse(out);
|
||||
expect(fm.title).toBe('BOM Test');
|
||||
// Output should not contain doubled frontmatter blocks (single block = 3 split segments)
|
||||
expect(out.split(/^---\s*$/m).length).toBe(3);
|
||||
});
|
||||
|
||||
test('10. empty frontmatter ---\\n---\\n merged with auto-fields', () => {
|
||||
const input = '---\n---\n\nbody content';
|
||||
const out = mergeCaptureFrontmatter(input, {});
|
||||
const { fm } = parse(out);
|
||||
expect(fm.type).toBe('note'); // auto-filled
|
||||
expect(fm.title).toBe('body content'); // derived
|
||||
expect(fm.captured_via).toBe('capture-cli'); // auto-filled
|
||||
});
|
||||
|
||||
test('11. malformed YAML in frontmatter throws (no silent half-merge)', () => {
|
||||
// gray-matter parses { foo: : invalid } as throwing YAML
|
||||
const input = '---\nfoo: : :::\nbar: [unclosed\n---\n\nbody';
|
||||
expect(() => mergeCaptureFrontmatter(input, {})).toThrow(/malformed frontmatter/);
|
||||
});
|
||||
|
||||
test('12. no trailing newline before body', () => {
|
||||
const input = '---\ntitle: Tight\n---\nbody right after';
|
||||
const out = mergeCaptureFrontmatter(input, {});
|
||||
const { fm, body } = parse(out);
|
||||
expect(fm.title).toBe('Tight');
|
||||
expect(body.trim()).toContain('body right after');
|
||||
});
|
||||
|
||||
test('13. user description/tags/slug pass through verbatim', () => {
|
||||
const input = `---
|
||||
type: meeting
|
||||
title: Standup
|
||||
description: weekly engineering sync
|
||||
tags: [work, weekly]
|
||||
slug: meetings/2026-05-22
|
||||
---
|
||||
|
||||
Notes from the sync`;
|
||||
const out = mergeCaptureFrontmatter(input, {});
|
||||
const { fm } = parse(out);
|
||||
expect(fm.description).toBe('weekly engineering sync');
|
||||
expect(fm.tags).toEqual(['work', 'weekly']);
|
||||
expect(fm.slug).toBe('meetings/2026-05-22');
|
||||
expect(fm.title).toBe('Standup'); // user wins
|
||||
expect(fm.type).toBe('meeting'); // user wins (no --type override)
|
||||
});
|
||||
|
||||
// BUG-1 regression guard: the exact reported failure shape
|
||||
test('REGRESSION (BUG-1): file with `---` opening never produces title: "---"', () => {
|
||||
const input = '---\ntitle: Pre-existing Title\ntags: [test, frontmatter]\n---\n\n# Pre-existing content\n\nbody';
|
||||
const out = mergeCaptureFrontmatter(input, {});
|
||||
const { fm } = parse(out);
|
||||
expect(fm.title).toBe('Pre-existing Title');
|
||||
expect(fm.title).not.toBe('---');
|
||||
// Critically: only one frontmatter block (not two stacked) — split yields 3 segments
|
||||
expect(out.split(/^---\s*$/m).length).toBe(3);
|
||||
});
|
||||
|
||||
// CLI --type flag precedence (per plan: CLI flag > userFm > 'note')
|
||||
test('--type CLI flag wins over user frontmatter type', () => {
|
||||
const input = '---\ntype: meeting\ntitle: X\n---\n\nbody';
|
||||
const out = mergeCaptureFrontmatter(input, { type: 'observation' });
|
||||
const { fm } = parse(out);
|
||||
expect(fm.type).toBe('observation');
|
||||
});
|
||||
|
||||
test('--type CLI flag wins over default note in no-frontmatter path', () => {
|
||||
const out = mergeCaptureFrontmatter('plain text', { type: 'idea' });
|
||||
const { fm } = parse(out);
|
||||
expect(fm.type).toBe('idea');
|
||||
});
|
||||
});
|
||||
|
||||
describe('deriveTitle (no-frontmatter path)', () => {
|
||||
test('strips leading # markers', () => {
|
||||
expect(deriveTitle('# A heading\nrest')).toBe('A heading');
|
||||
expect(deriveTitle('### Triple hash\nrest')).toBe('Triple hash');
|
||||
});
|
||||
|
||||
test('caps at 80 chars', () => {
|
||||
const long = 'a'.repeat(120);
|
||||
expect(deriveTitle(long)).toBe('a'.repeat(80));
|
||||
});
|
||||
|
||||
test('falls back to Capture for empty input', () => {
|
||||
expect(deriveTitle('')).toBe('Capture');
|
||||
expect(deriveTitle('\n\n \n')).toBe('Capture');
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,183 @@
|
||||
/**
|
||||
* v0.39.3.0 Phase 3c — capture CLI write-side test surface.
|
||||
*
|
||||
* Covers the pure helpers introduced/changed in capture.ts as a unit
|
||||
* test (not the full runCapture orchestration; that's exercised via
|
||||
* the existing tests + the E2E ingest-capture-provenance suite).
|
||||
* Hermetic — no engine, no DB.
|
||||
*
|
||||
* Findings exercised here:
|
||||
* CV9 normalizeForHash — separates hash normalization from storage
|
||||
* CV8 receipt hash from rawBody (the slug + content_hash use the
|
||||
* normalized body, not the timestamp-bearing frontmatter)
|
||||
* CV10 detectBinaryNullByte — first-8KB sniff, deterministic fixtures
|
||||
* A2 maybeRewriteSourceFkError — friendly hint pattern (both engines)
|
||||
*
|
||||
* runCapture's branching (thin-client rejection of --source, source
|
||||
* resolver, exit-1 paths) is verified by exercising the helpers and
|
||||
* by the existing tests; integration is covered by the E2E test added
|
||||
* in test/e2e/ingest-capture-provenance.test.ts.
|
||||
*/
|
||||
|
||||
import { describe, test, expect } from 'bun:test';
|
||||
import { __testing as captureTesting } from '../src/commands/capture.ts';
|
||||
import { computeContentHash } from '../src/core/ingestion/types.ts';
|
||||
|
||||
const { detectBinaryNullByte, normalizeForHash, maybeRewriteSourceFkError } = captureTesting;
|
||||
|
||||
describe('CV10 — binary file guard (detectBinaryNullByte)', () => {
|
||||
test('returns -1 on plain ASCII', () => {
|
||||
expect(detectBinaryNullByte(Buffer.from('hello world'))).toBe(-1);
|
||||
});
|
||||
|
||||
test('returns -1 on UTF-8 with CJK (測試)', () => {
|
||||
expect(detectBinaryNullByte(Buffer.from('測試 brain content'))).toBe(-1);
|
||||
});
|
||||
|
||||
test('returns -1 on UTF-8 with emoji (🧠🔥)', () => {
|
||||
expect(detectBinaryNullByte(Buffer.from('thinking 🧠🔥 hot'))).toBe(-1);
|
||||
});
|
||||
|
||||
test('returns -1 on UTF-8 BOM-prefixed text', () => {
|
||||
const bom = Buffer.from([0xef, 0xbb, 0xbf]);
|
||||
const text = Buffer.from('# heading\n\nbody');
|
||||
expect(detectBinaryNullByte(Buffer.concat([bom, text]))).toBe(-1);
|
||||
});
|
||||
|
||||
test('returns offset on NUL byte at start', () => {
|
||||
const bytes = Buffer.from([0x00, 0x68, 0x69]);
|
||||
expect(detectBinaryNullByte(bytes)).toBe(0);
|
||||
});
|
||||
|
||||
test('returns offset on NUL byte mid-buffer', () => {
|
||||
const bytes = Buffer.from('hello\x00world');
|
||||
expect(detectBinaryNullByte(bytes)).toBe(5);
|
||||
});
|
||||
|
||||
test('returns offset on PNG magic bytes (NUL at byte 0)', () => {
|
||||
// PNG header has multiple NUL bytes — the first appears at offset 8
|
||||
// (after the 8-byte signature 0x89 0x50 0x4e 0x47 0x0d 0x0a 0x1a 0x0a,
|
||||
// the IHDR chunk length 0x00 0x00 0x00 0x0d follows).
|
||||
const png = Buffer.from([
|
||||
0x89, 0x50, 0x4e, 0x47, 0x0d, 0x0a, 0x1a, 0x0a, // PNG signature
|
||||
0x00, 0x00, 0x00, 0x0d, // IHDR length
|
||||
]);
|
||||
expect(detectBinaryNullByte(png)).toBe(8);
|
||||
});
|
||||
|
||||
test('caps scan at 8KB (NUL at offset 8192 NOT detected)', () => {
|
||||
const bytes = Buffer.alloc(10_000, 0x41); // 'A' bytes
|
||||
bytes[8192] = 0x00; // beyond the 8KB ceiling
|
||||
expect(detectBinaryNullByte(bytes)).toBe(-1);
|
||||
});
|
||||
|
||||
test('detects NUL at exactly offset 8191 (within scan window)', () => {
|
||||
const bytes = Buffer.alloc(10_000, 0x41);
|
||||
bytes[8191] = 0x00;
|
||||
expect(detectBinaryNullByte(bytes)).toBe(8191);
|
||||
});
|
||||
|
||||
test('empty buffer returns -1', () => {
|
||||
expect(detectBinaryNullByte(Buffer.alloc(0))).toBe(-1);
|
||||
});
|
||||
});
|
||||
|
||||
describe('CV9 — normalizeForHash (trim + LF + NFKC)', () => {
|
||||
test('strips leading/trailing whitespace', () => {
|
||||
expect(normalizeForHash(' hello ')).toBe('hello');
|
||||
});
|
||||
|
||||
test('strips UTF-8 BOM', () => {
|
||||
expect(normalizeForHash('hello')).toBe('hello');
|
||||
});
|
||||
|
||||
test('normalizes CRLF to LF', () => {
|
||||
expect(normalizeForHash('a\r\nb\r\nc')).toBe('a\nb\nc');
|
||||
});
|
||||
|
||||
test('NFKC normalizes precomposed/decomposed Unicode (ñ)', () => {
|
||||
// U+00F1 (precomposed) vs U+006E + U+0303 (decomposed). NFKC unifies.
|
||||
const precomposed = 'mañana';
|
||||
const decomposed = 'mañana'; // ñ as n + combining tilde
|
||||
expect(normalizeForHash(precomposed)).toBe(normalizeForHash(decomposed));
|
||||
});
|
||||
|
||||
test('does not modify already-clean text', () => {
|
||||
expect(normalizeForHash('clean text\nwith newlines')).toBe('clean text\nwith newlines');
|
||||
});
|
||||
|
||||
test('whitespace-only input normalizes to empty', () => {
|
||||
expect(normalizeForHash(' \n\r\n\t ')).toBe('');
|
||||
});
|
||||
});
|
||||
|
||||
describe('CV8 — receipt hash stable across timestamp variations', () => {
|
||||
test('identical input text produces identical hash regardless of when called', () => {
|
||||
const text = 'remember to follow up on the X deal';
|
||||
const h1 = computeContentHash(normalizeForHash(text));
|
||||
// Simulate "captured at a different time" by re-calling
|
||||
const h2 = computeContentHash(normalizeForHash(text));
|
||||
expect(h1).toBe(h2);
|
||||
});
|
||||
|
||||
test('hash differs for different content', () => {
|
||||
const h1 = computeContentHash(normalizeForHash('text A'));
|
||||
const h2 = computeContentHash(normalizeForHash('text B'));
|
||||
expect(h1).not.toBe(h2);
|
||||
});
|
||||
|
||||
test('whitespace differences normalize to same hash (CV9 + CV8 interaction)', () => {
|
||||
const h1 = computeContentHash(normalizeForHash(' same text '));
|
||||
const h2 = computeContentHash(normalizeForHash('same text'));
|
||||
expect(h1).toBe(h2);
|
||||
});
|
||||
|
||||
test('CRLF vs LF line endings normalize to same hash', () => {
|
||||
const h1 = computeContentHash(normalizeForHash('line1\r\nline2'));
|
||||
const h2 = computeContentHash(normalizeForHash('line1\nline2'));
|
||||
expect(h1).toBe(h2);
|
||||
});
|
||||
});
|
||||
|
||||
describe('A2 — maybeRewriteSourceFkError (friendly FK violation hint)', () => {
|
||||
test('rewrites the raw Postgres pages_source_id_fk message', () => {
|
||||
const err = new Error(
|
||||
'insert or update on table "pages" violates foreign key constraint "pages_source_id_fk"',
|
||||
);
|
||||
const hint = maybeRewriteSourceFkError(err, 'dept-x');
|
||||
expect(hint).toContain("source 'dept-x' is not registered");
|
||||
expect(hint).toContain('gbrain sources add dept-x');
|
||||
expect(hint).toContain('gbrain sources list');
|
||||
});
|
||||
|
||||
test('rewrites a wrapped OperationError-style message containing the FK constraint name', () => {
|
||||
const err = new Error(
|
||||
'put_page failed: insert or update on table "pages" violates foreign key constraint "pages_source_id_fk"',
|
||||
);
|
||||
const hint = maybeRewriteSourceFkError(err, 'my-source');
|
||||
expect(hint).toContain("source 'my-source' is not registered");
|
||||
});
|
||||
|
||||
test('rewrites a "foreign key constraint ... source" pattern (postgres.js wrapped)', () => {
|
||||
const err = new Error(
|
||||
'update or delete on table "sources" violates foreign key constraint',
|
||||
);
|
||||
expect(maybeRewriteSourceFkError(err, 'src-x')).toContain("source 'src-x' is not registered");
|
||||
});
|
||||
|
||||
test('returns null on unrelated errors', () => {
|
||||
expect(maybeRewriteSourceFkError(new Error('connection timeout'), 'dept-x')).toBeNull();
|
||||
expect(maybeRewriteSourceFkError(new Error('syntax error'), 'dept-x')).toBeNull();
|
||||
});
|
||||
|
||||
test('returns null when sourceId is undefined (no friendly hint possible without the source name)', () => {
|
||||
const err = new Error('pages_source_id_fk violation');
|
||||
expect(maybeRewriteSourceFkError(err, undefined)).toBeNull();
|
||||
});
|
||||
|
||||
test('handles non-Error throws (string thrown directly)', () => {
|
||||
const hint = maybeRewriteSourceFkError('pages_source_id_fk constraint violation', 'foo');
|
||||
expect(hint).not.toBeNull();
|
||||
expect(hint).toContain("source 'foo'");
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,99 @@
|
||||
/**
|
||||
* v0.39.3.0 WARN-5 + WARN-6 — CLI help discoverability.
|
||||
*
|
||||
* WARN-5: `gbrain capture --help` was showing only the generic
|
||||
* `Usage: gbrain capture` line because `capture` was missing from
|
||||
* CLI_ONLY_SELF_HELP (src/cli.ts:34-53). Fix added it to the set AND
|
||||
* added a pre-engine-bind `--help` short-circuit at handleCliOnly so
|
||||
* the HELP constant is reachable on a fresh tmpdir with no config.
|
||||
*
|
||||
* WARN-6: `capture`, `brainstorm`, `lsd` were missing from the main
|
||||
* `gbrain --help` text. Added a BRAIN section to printHelp.
|
||||
*
|
||||
* These tests spawn `bun run src/cli.ts` as a subprocess so they
|
||||
* exercise the real dispatcher flow end-to-end (no mocking of
|
||||
* cli.ts internals).
|
||||
*/
|
||||
|
||||
import { describe, test, expect } from 'bun:test';
|
||||
import { spawnSync } from 'node:child_process';
|
||||
|
||||
function runCli(args: string[]): { stdout: string; stderr: string; status: number } {
|
||||
const result = spawnSync('bun', ['run', 'src/cli.ts', ...args], {
|
||||
cwd: process.cwd(),
|
||||
encoding: 'utf8',
|
||||
env: { ...process.env, GBRAIN_HOME: '/tmp/gbrain-test-help-nonexistent' },
|
||||
});
|
||||
return {
|
||||
stdout: result.stdout ?? '',
|
||||
stderr: result.stderr ?? '',
|
||||
status: result.status ?? -1,
|
||||
};
|
||||
}
|
||||
|
||||
describe('WARN-5 — `gbrain capture --help` reaches the detailed HELP constant', () => {
|
||||
test('output contains every documented flag', () => {
|
||||
const { stdout, status } = runCli(['capture', '--help']);
|
||||
expect(status).toBe(0);
|
||||
expect(stdout).toContain('--slug');
|
||||
expect(stdout).toContain('--type');
|
||||
expect(stdout).toContain('--file');
|
||||
expect(stdout).toContain('--stdin');
|
||||
expect(stdout).toContain('--source');
|
||||
expect(stdout).toContain('--quiet');
|
||||
expect(stdout).toContain('--json');
|
||||
});
|
||||
|
||||
test('output is NOT the generic short-circuit fallback', () => {
|
||||
const { stdout } = runCli(['capture', '--help']);
|
||||
// Pre-fix output was: "Usage: gbrain capture\n\ngbrain capture - run gbrain --help ..."
|
||||
// Post-fix HELP is much longer and includes Examples.
|
||||
expect(stdout).toContain('Examples:');
|
||||
expect(stdout.split('\n').length).toBeGreaterThan(10);
|
||||
expect(stdout).not.toMatch(/^Usage: gbrain capture\s*$/m);
|
||||
});
|
||||
|
||||
test('-h short flag also works', () => {
|
||||
const { stdout, status } = runCli(['capture', '-h']);
|
||||
expect(status).toBe(0);
|
||||
expect(stdout).toContain('--file PATH');
|
||||
});
|
||||
});
|
||||
|
||||
describe('WARN-6 — main `gbrain --help` lists capture/brainstorm/lsd', () => {
|
||||
test('output mentions all three commands by name', () => {
|
||||
const { stdout, status } = runCli(['--help']);
|
||||
expect(status).toBe(0);
|
||||
// Must appear as command names (not just words in prose somewhere)
|
||||
expect(stdout).toMatch(/^\s*capture\s/m);
|
||||
expect(stdout).toMatch(/^\s*brainstorm\s/m);
|
||||
expect(stdout).toMatch(/^\s*lsd\s/m);
|
||||
});
|
||||
|
||||
test('BRAIN section heading is present and groups the three commands', () => {
|
||||
const { stdout } = runCli(['--help']);
|
||||
expect(stdout).toContain('BRAIN');
|
||||
// The 3 commands should appear AFTER the BRAIN heading in textual order.
|
||||
const brainIdx = stdout.indexOf('BRAIN');
|
||||
expect(brainIdx).toBeGreaterThan(-1);
|
||||
expect(stdout.indexOf('capture', brainIdx)).toBeGreaterThan(brainIdx);
|
||||
expect(stdout.indexOf('brainstorm', brainIdx)).toBeGreaterThan(brainIdx);
|
||||
expect(stdout.indexOf('lsd', brainIdx)).toBeGreaterThan(brainIdx);
|
||||
});
|
||||
|
||||
test('regression: existing top-level commands still listed', () => {
|
||||
// Snapshot guard against accidentally deleting other groups when we
|
||||
// added the BRAIN section. Spot-check a few commands from different
|
||||
// groups (SETUP, PAGES, SEARCH, IMPORT/EXPORT).
|
||||
const { stdout } = runCli(['--help']);
|
||||
expect(stdout).toContain('init');
|
||||
expect(stdout).toContain('doctor');
|
||||
expect(stdout).toContain('get');
|
||||
expect(stdout).toContain('search');
|
||||
expect(stdout).toContain('query');
|
||||
expect(stdout).toContain('import');
|
||||
expect(stdout).toContain('export');
|
||||
expect(stdout).toContain('files');
|
||||
expect(stdout).toContain('embed');
|
||||
});
|
||||
});
|
||||
@@ -14,6 +14,7 @@ import * as path from 'node:path';
|
||||
import * as os from 'node:os';
|
||||
import { PGLiteEngine } from '../../src/core/pglite-engine.ts';
|
||||
import { resetPgliteState } from '../helpers/reset-pglite.ts';
|
||||
import matter from 'gray-matter';
|
||||
import { runCapture, __testing } from '../../src/commands/capture.ts';
|
||||
|
||||
let engine: PGLiteEngine;
|
||||
@@ -124,16 +125,23 @@ describe('capture — buildContent', () => {
|
||||
});
|
||||
|
||||
test('uses first non-empty line as the title', () => {
|
||||
// v0.39.3.0: BUG-1 fix moved buildContent to mergeCaptureFrontmatter
|
||||
// which uses gray-matter's `matter.stringify()`. The emitter follows
|
||||
// YAML defaults — simple strings emit unquoted (`title: Real first
|
||||
// line`), strings with special chars get single-quoted (the prior
|
||||
// hand-rolled `title: "..."` JSON-style quoting is gone). Parse the
|
||||
// YAML and assert on the value, not the literal quoting style.
|
||||
const result = __testing.buildContent('\n \nReal first line\nmore', {});
|
||||
expect(result).toContain('title: "Real first line"');
|
||||
const parsed = matter(result);
|
||||
expect(parsed.data.title).toBe('Real first line');
|
||||
});
|
||||
|
||||
test('caps title at 80 chars', () => {
|
||||
const longLine = 'x'.repeat(200);
|
||||
const result = __testing.buildContent(longLine, {});
|
||||
const titleMatch = result.match(/title: "([^"]*)"/);
|
||||
expect(titleMatch).not.toBeNull();
|
||||
expect(titleMatch![1].length).toBeLessThanOrEqual(80);
|
||||
const parsed = matter(result);
|
||||
expect(typeof parsed.data.title).toBe('string');
|
||||
expect((parsed.data.title as string).length).toBeLessThanOrEqual(80);
|
||||
});
|
||||
|
||||
test('honors --source via captured_via', () => {
|
||||
|
||||
@@ -224,4 +224,90 @@ describeBoth('Engine parity — Postgres vs PGLite', () => {
|
||||
const pgliteChanged = pgliteDefault.map((r: SearchResult) => r.slug).join(',') !== pgliteHigh.map((r: SearchResult) => r.slug).join(',');
|
||||
expect(pgChanged || pgliteChanged).toBe(true);
|
||||
});
|
||||
|
||||
// v0.39.3.0 T3 — provenance write+read parity (WARN-8 + CV5).
|
||||
// Both engines must write the same 4 provenance columns (source_kind,
|
||||
// source_uri, ingested_via, ingested_at) on putPage AND surface them
|
||||
// on getPage. A drift here would mean `gbrain migrate --to supabase`
|
||||
// silently loses half a user's provenance audit trail.
|
||||
test('provenance columns: putPage writes + getPage returns identical shape on both engines', async () => {
|
||||
const slug = 'wiki/provenance-parity';
|
||||
const input = {
|
||||
type: 'note' as const,
|
||||
title: 'Provenance Parity Test',
|
||||
compiled_truth: 'body',
|
||||
timeline: '',
|
||||
source_kind: 'capture-cli',
|
||||
source_uri: 'file:///tmp/parity.md',
|
||||
ingested_via: 'put_page',
|
||||
};
|
||||
await pgEngine.putPage(slug, input);
|
||||
await pgliteEngine.putPage(slug, input);
|
||||
|
||||
const pgPage = await pgEngine.getPage(slug);
|
||||
const pglitePage = await pgliteEngine.getPage(slug);
|
||||
|
||||
expect(pgPage).not.toBeNull();
|
||||
expect(pglitePage).not.toBeNull();
|
||||
|
||||
// All 4 provenance fields must match across engines.
|
||||
expect(pgPage!.source_kind).toBe('capture-cli');
|
||||
expect(pglitePage!.source_kind).toBe('capture-cli');
|
||||
expect(pgPage!.source_uri).toBe('file:///tmp/parity.md');
|
||||
expect(pglitePage!.source_uri).toBe('file:///tmp/parity.md');
|
||||
expect(pgPage!.ingested_via).toBe('put_page');
|
||||
expect(pglitePage!.ingested_via).toBe('put_page');
|
||||
// ingested_at is server-stamped; both engines must populate a Date
|
||||
// (not Date drift across engines — the assertion is structural).
|
||||
expect(pgPage!.ingested_at).toBeInstanceOf(Date);
|
||||
expect(pglitePage!.ingested_at).toBeInstanceOf(Date);
|
||||
});
|
||||
|
||||
test('provenance COALESCE-preserve UPDATE: parity on both engines (CV12)', async () => {
|
||||
// First write with provenance.
|
||||
const slug = 'wiki/provenance-preserve-parity';
|
||||
await pgEngine.putPage(slug, {
|
||||
type: 'note',
|
||||
title: 'V1',
|
||||
compiled_truth: 'body v1',
|
||||
timeline: '',
|
||||
source_kind: 'capture-cli',
|
||||
ingested_via: 'put_page',
|
||||
});
|
||||
await pgliteEngine.putPage(slug, {
|
||||
type: 'note',
|
||||
title: 'V1',
|
||||
compiled_truth: 'body v1',
|
||||
timeline: '',
|
||||
source_kind: 'capture-cli',
|
||||
ingested_via: 'put_page',
|
||||
});
|
||||
|
||||
// Second write WITHOUT provenance — both engines must preserve
|
||||
// the first-write audit trail via COALESCE-preserve UPDATE.
|
||||
await pgEngine.putPage(slug, {
|
||||
type: 'note',
|
||||
title: 'V2',
|
||||
compiled_truth: 'body v2',
|
||||
timeline: '',
|
||||
});
|
||||
await pgliteEngine.putPage(slug, {
|
||||
type: 'note',
|
||||
title: 'V2',
|
||||
compiled_truth: 'body v2',
|
||||
timeline: '',
|
||||
});
|
||||
|
||||
const pgPage = await pgEngine.getPage(slug);
|
||||
const pglitePage = await pgliteEngine.getPage(slug);
|
||||
|
||||
// Provenance preserved on BOTH engines (CV12 first-write-wins).
|
||||
expect(pgPage!.source_kind).toBe('capture-cli');
|
||||
expect(pglitePage!.source_kind).toBe('capture-cli');
|
||||
expect(pgPage!.ingested_via).toBe('put_page');
|
||||
expect(pglitePage!.ingested_via).toBe('put_page');
|
||||
// Page title updated (proves the UPDATE actually fired).
|
||||
expect(pgPage!.title).toBe('V2');
|
||||
expect(pglitePage!.title).toBe('V2');
|
||||
});
|
||||
});
|
||||
|
||||
@@ -206,6 +206,36 @@ describeE2E('serve-http POST /ingest webhook (v0.38)', () => {
|
||||
expect(body.message?.toLowerCase()).toContain('non-empty');
|
||||
});
|
||||
|
||||
// v0.39.3.0 BUG-2 regression: when express.raw() doesn't populate req.body
|
||||
// (no Content-Length / no body / specific middleware-chain conditions),
|
||||
// req.body is `undefined`. The pre-fix code's `else` branch fell through
|
||||
// to `Buffer.from(JSON.stringify(undefined), 'utf8')` — and
|
||||
// `JSON.stringify(undefined) === undefined` (literal), so Buffer.from
|
||||
// threw TypeError and the route returned an HTML 500 page instead of a
|
||||
// JSON envelope. The null-guard at the top of the handler now catches
|
||||
// this case and returns 400 `empty_body` like the empty-Buffer case.
|
||||
test('BUG-2: POST with no body (undefined req.body) → 400 JSON envelope (not 500 HTML)', async () => {
|
||||
const token = await mintToken('read write');
|
||||
// fetch with no `body:` field sends a request with no body bytes.
|
||||
// Combined with no Content-Length, this is the exact shape that
|
||||
// triggered the v0.38.0.0 TypeError.
|
||||
const res = await fetch(`${BASE}/ingest`, {
|
||||
method: 'POST',
|
||||
headers: {
|
||||
Authorization: `Bearer ${token}`,
|
||||
'Content-Type': 'text/markdown',
|
||||
},
|
||||
});
|
||||
// Must NOT be 500 (the pre-fix behavior).
|
||||
expect(res.status).not.toBe(500);
|
||||
// Must be a JSON 400 with the documented error shape.
|
||||
expect(res.status).toBe(400);
|
||||
const ct = res.headers.get('content-type') ?? '';
|
||||
expect(ct).toContain('application/json');
|
||||
const body = (await res.json()) as { error?: string };
|
||||
expect(body.error).toBe('empty_body');
|
||||
});
|
||||
|
||||
// =========================================================================
|
||||
// Content-type allowlist (the v0.38 webhook taxonomy)
|
||||
// =========================================================================
|
||||
|
||||
@@ -121,3 +121,136 @@ describe('writeFactsAbsorbLog — ingest_log row shape', () => {
|
||||
expect(FACTS_ABSORB_REASONS.length).toBe(6);
|
||||
});
|
||||
});
|
||||
|
||||
// v0.39.3.0 WARN-4 + CQ1 + CV13 — disconnected-engine suppression.
|
||||
// Pre-fix: '[facts:absorb] failed to log gateway_error for inbox/...: No
|
||||
// database connection' fired loudly after every gbrain capture. Per CQ1
|
||||
// suppress via typed access (instanceof GBrainError && .problem) NOT
|
||||
// string-match on .message. CV13 prints ONE first-occurrence stack
|
||||
// trace so the next user report can diagnose the wiring bug in v0.38.4.
|
||||
describe('writeFactsAbsorbLog — WARN-4 disconnected-engine suppression (CQ1 + CV13)', () => {
|
||||
// Mock engine that simulates a disconnected state on logIngest. Uses
|
||||
// the same GBrainError shape src/core/db.ts:151-158 throws.
|
||||
function makeDisconnectedEngine(): {
|
||||
engine: import('../src/core/engine.ts').BrainEngine;
|
||||
callCount: () => number;
|
||||
} {
|
||||
let calls = 0;
|
||||
return {
|
||||
engine: {
|
||||
logIngest: async () => {
|
||||
calls++;
|
||||
// Throw the EXACT shape src/core/db.ts:151-158 throws when
|
||||
// `sql` is null (engine not connected yet).
|
||||
const { GBrainError } = await import('../src/core/types.ts');
|
||||
throw new GBrainError(
|
||||
'No database connection',
|
||||
'connect() has not been called',
|
||||
'Run gbrain init --supabase or gbrain init --url <connection_string>',
|
||||
);
|
||||
},
|
||||
} as unknown as import('../src/core/engine.ts').BrainEngine,
|
||||
callCount: () => calls,
|
||||
};
|
||||
}
|
||||
|
||||
function makeGenericFailureEngine(thrown: unknown): import('../src/core/engine.ts').BrainEngine {
|
||||
return {
|
||||
logIngest: async () => { throw thrown; },
|
||||
} as unknown as import('../src/core/engine.ts').BrainEngine;
|
||||
}
|
||||
|
||||
test('GBrainError problem="No database connection" SUPPRESSED after first occurrence', async () => {
|
||||
const { _resetFactsAbsorbDisconnectedFlagForTests } = await import('../src/core/facts/absorb-log.ts');
|
||||
_resetFactsAbsorbDisconnectedFlagForTests();
|
||||
|
||||
const warnSpy: string[] = [];
|
||||
const orig = console.warn;
|
||||
console.warn = (msg: unknown) => { warnSpy.push(String(msg)); };
|
||||
|
||||
try {
|
||||
const { engine: e1 } = makeDisconnectedEngine();
|
||||
// First call → ONE warn with first-occurrence trace
|
||||
await writeFactsAbsorbLog(e1, 'inbox/p1', 'gateway_error', 'detail-1');
|
||||
expect(warnSpy.length).toBe(1);
|
||||
expect(warnSpy[0]).toContain('WARN-4');
|
||||
expect(warnSpy[0]).toContain('First-occurrence trace');
|
||||
|
||||
// Second call → SILENT (no additional warn)
|
||||
const { engine: e2 } = makeDisconnectedEngine();
|
||||
await writeFactsAbsorbLog(e2, 'inbox/p2', 'gateway_error', 'detail-2');
|
||||
expect(warnSpy.length).toBe(1); // unchanged
|
||||
|
||||
// Third + fourth → still silent
|
||||
const { engine: e3 } = makeDisconnectedEngine();
|
||||
const { engine: e4 } = makeDisconnectedEngine();
|
||||
await writeFactsAbsorbLog(e3, 'inbox/p3', 'parse_failure', 'detail-3');
|
||||
await writeFactsAbsorbLog(e4, 'inbox/p4', 'embed_failure', 'detail-4');
|
||||
expect(warnSpy.length).toBe(1);
|
||||
} finally {
|
||||
console.warn = orig;
|
||||
}
|
||||
});
|
||||
|
||||
test('GBrainError with DIFFERENT problem still warns loudly (not the suppressed class)', async () => {
|
||||
const { _resetFactsAbsorbDisconnectedFlagForTests } = await import('../src/core/facts/absorb-log.ts');
|
||||
_resetFactsAbsorbDisconnectedFlagForTests();
|
||||
|
||||
const warnSpy: string[] = [];
|
||||
const orig = console.warn;
|
||||
console.warn = (msg: unknown) => { warnSpy.push(String(msg)); };
|
||||
|
||||
try {
|
||||
const { GBrainError } = await import('../src/core/types.ts');
|
||||
const otherError = new GBrainError(
|
||||
'Connection pool exhausted',
|
||||
'all connections in use',
|
||||
'Increase pool size or wait',
|
||||
);
|
||||
const engine = makeGenericFailureEngine(otherError);
|
||||
await writeFactsAbsorbLog(engine, 'inbox/p', 'gateway_error', 'd');
|
||||
// Loud warn fires (not the suppressed first-occurrence shape)
|
||||
expect(warnSpy.length).toBe(1);
|
||||
expect(warnSpy[0]).toContain('failed to log gateway_error');
|
||||
expect(warnSpy[0]).not.toContain('First-occurrence');
|
||||
} finally {
|
||||
console.warn = orig;
|
||||
}
|
||||
});
|
||||
|
||||
test('non-GBrainError (plain Error) still warns loudly', async () => {
|
||||
const { _resetFactsAbsorbDisconnectedFlagForTests } = await import('../src/core/facts/absorb-log.ts');
|
||||
_resetFactsAbsorbDisconnectedFlagForTests();
|
||||
|
||||
const warnSpy: string[] = [];
|
||||
const orig = console.warn;
|
||||
console.warn = (msg: unknown) => { warnSpy.push(String(msg)); };
|
||||
|
||||
try {
|
||||
const engine = makeGenericFailureEngine(new Error('something else went wrong'));
|
||||
await writeFactsAbsorbLog(engine, 'inbox/p', 'pipeline_error', 'd');
|
||||
expect(warnSpy.length).toBe(1);
|
||||
expect(warnSpy[0]).toContain('something else went wrong');
|
||||
} finally {
|
||||
console.warn = orig;
|
||||
}
|
||||
});
|
||||
|
||||
test('engine call STILL fires for the suppressed case (writeFactsAbsorbLog tried)', async () => {
|
||||
// The suppression is on the WARN, not on the attempt. logIngest is
|
||||
// still called — we just don't print the failure noise.
|
||||
const { _resetFactsAbsorbDisconnectedFlagForTests } = await import('../src/core/facts/absorb-log.ts');
|
||||
_resetFactsAbsorbDisconnectedFlagForTests();
|
||||
|
||||
const orig = console.warn;
|
||||
console.warn = () => {}; // suppress all warns for this test
|
||||
|
||||
try {
|
||||
const { engine, callCount } = makeDisconnectedEngine();
|
||||
await writeFactsAbsorbLog(engine, 'inbox/p', 'gateway_error', 'd');
|
||||
expect(callCount()).toBe(1);
|
||||
} finally {
|
||||
console.warn = orig;
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
@@ -66,8 +66,18 @@ describe('v0.36.1.x #1090 — admin embed two-tier resolution', () => {
|
||||
describe('v0.36.1.x #1077 — admin register-client supports PKCE public clients', () => {
|
||||
test('admin endpoint reads grantTypes / redirectUris / tokenEndpointAuthMethod from request body', () => {
|
||||
const src = readFileSync('src/commands/serve-http.ts', 'utf8');
|
||||
// Single destructure line includes all three fields
|
||||
expect(src).toMatch(/const\s+\{\s*name,\s*scopes,\s*tokenTtl,\s*grantTypes,\s*redirectUris,\s*tokenEndpointAuthMethod\s*\}\s*=\s*req\.body/);
|
||||
// The destructure must surface name / tokenTtl / grantTypes /
|
||||
// redirectUris / tokenEndpointAuthMethod from req.body. v0.39.3.0
|
||||
// WARN-9 (PR #1308) moved `scopes` to a separate read line that
|
||||
// accepts BOTH `scopes` (admin SPA) AND `scope` (OAuth wire singular)
|
||||
// via `?? `, so this regex no longer requires `scopes` in the inline
|
||||
// destructure — it's separately covered by the scope-source check
|
||||
// below.
|
||||
expect(src).toMatch(/const\s+\{\s*name,\s*(?:[^}]*?,\s*)?tokenTtl,\s*grantTypes,\s*redirectUris,\s*tokenEndpointAuthMethod\s*\}\s*=\s*req\.body/);
|
||||
// v0.39.3.0 WARN-9: the route must still read a `scope`/`scopes` field
|
||||
// (under either name) from req.body. Pin the fallback pattern so the
|
||||
// PKCE-fix regression contract stays load-bearing.
|
||||
expect(src).toMatch(/req\.body[^;]*scopes\s*\?\?\s*[^;]*scope\b/);
|
||||
// PKCE branch NULLs client_secret_hash + sets auth method to 'none'
|
||||
expect(src).toMatch(/tokenEndpointAuthMethod\s*===\s*'none'/);
|
||||
expect(src).toMatch(/client_secret_hash\s*=\s*NULL,\s*token_endpoint_auth_method\s*=\s*'none'/);
|
||||
|
||||
@@ -505,3 +505,200 @@ Content.
|
||||
expect(putCall.args[1].source_path).toBe('concepts/cjk-source-path.md');
|
||||
});
|
||||
});
|
||||
|
||||
// v0.39.3.0 CV8 Phase 3d — DB content_hash excludes timestamp-bearing
|
||||
// frontmatter keys (captured_at, ingested_at) so identical body content
|
||||
// from capture-cli produces a stable hash across multiple captures.
|
||||
// Pre-fix, every capture invocation produced a fresh hash because the
|
||||
// captured_at timestamp changed, defeating both the existing.content_hash
|
||||
// short-circuit AND the daemon's 24h LRU dedup.
|
||||
describe('importFromContent — CV8 DB content_hash stability', () => {
|
||||
test('captured_at differences produce IDENTICAL hash (capture-cli dedup)', async () => {
|
||||
// Capture #1 at one timestamp
|
||||
const t1 = '2026-05-22T10:00:00.000Z';
|
||||
const content1 = `---
|
||||
type: note
|
||||
title: Same Text
|
||||
captured_at: ${t1}
|
||||
captured_via: capture-cli
|
||||
---
|
||||
|
||||
# Same Text
|
||||
|
||||
remember to follow up
|
||||
`;
|
||||
// Capture #2 at a different timestamp (same body)
|
||||
const t2 = '2026-05-22T11:00:00.000Z';
|
||||
const content2 = `---
|
||||
type: note
|
||||
title: Same Text
|
||||
captured_at: ${t2}
|
||||
captured_via: capture-cli
|
||||
---
|
||||
|
||||
# Same Text
|
||||
|
||||
remember to follow up
|
||||
`;
|
||||
|
||||
let firstHash: string | undefined;
|
||||
let secondHash: string | undefined;
|
||||
let firstStatus: string | undefined;
|
||||
let secondStatus: string | undefined;
|
||||
|
||||
// First call: no existing page; hash is computed and written
|
||||
const engine1 = mockEngine({
|
||||
getPage: () => Promise.resolve(null),
|
||||
putPage: (_slug: string, page: any) => {
|
||||
firstHash = page.content_hash;
|
||||
return Promise.resolve(null);
|
||||
},
|
||||
});
|
||||
const r1 = await importFromContent(engine1, 'inbox/test', content1, { noEmbed: true });
|
||||
firstStatus = r1.status;
|
||||
expect(firstStatus).toBe('imported');
|
||||
expect(firstHash).toBeTruthy();
|
||||
|
||||
// Second call: existing page has the first hash; the second capture's
|
||||
// hash must match so the short-circuit fires and status === 'skipped'.
|
||||
const engine2 = mockEngine({
|
||||
getPage: () => Promise.resolve({ content_hash: firstHash } as any),
|
||||
putPage: (_slug: string, page: any) => {
|
||||
secondHash = page.content_hash;
|
||||
return Promise.resolve(null);
|
||||
},
|
||||
});
|
||||
const r2 = await importFromContent(engine2, 'inbox/test', content2, { noEmbed: true });
|
||||
secondStatus = r2.status;
|
||||
expect(secondStatus).toBe('skipped'); // hash matched
|
||||
expect(secondHash).toBeUndefined(); // putPage NOT called (short-circuited)
|
||||
});
|
||||
|
||||
test('body change DOES change the hash (real edits not silently swallowed)', async () => {
|
||||
const t = '2026-05-22T10:00:00.000Z';
|
||||
const content1 = `---
|
||||
type: note
|
||||
captured_at: ${t}
|
||||
---
|
||||
|
||||
original body
|
||||
`;
|
||||
const content2 = `---
|
||||
type: note
|
||||
captured_at: ${t}
|
||||
---
|
||||
|
||||
edited body
|
||||
`;
|
||||
|
||||
let firstHash: string | undefined;
|
||||
let secondHash: string | undefined;
|
||||
|
||||
const engine1 = mockEngine({
|
||||
getPage: () => Promise.resolve(null),
|
||||
putPage: (_slug: string, page: any) => {
|
||||
firstHash = page.content_hash;
|
||||
return Promise.resolve(null);
|
||||
},
|
||||
});
|
||||
await importFromContent(engine1, 'inbox/test', content1, { noEmbed: true });
|
||||
|
||||
const engine2 = mockEngine({
|
||||
getPage: () => Promise.resolve({ content_hash: firstHash } as any),
|
||||
putPage: (_slug: string, page: any) => {
|
||||
secondHash = page.content_hash;
|
||||
return Promise.resolve(null);
|
||||
},
|
||||
});
|
||||
const r2 = await importFromContent(engine2, 'inbox/test', content2, { noEmbed: true });
|
||||
expect(r2.status).toBe('imported'); // body changed, hash differs, re-imported
|
||||
expect(secondHash).toBeTruthy();
|
||||
expect(secondHash).not.toBe(firstHash);
|
||||
});
|
||||
|
||||
test('tag change DOES change the hash (sync tag-add not silently swallowed)', async () => {
|
||||
// Regression guard: stripping captured_at from the hash input must NOT
|
||||
// also strip tags. A user editing a markdown file to add a tag still
|
||||
// expects tag reconciliation to fire.
|
||||
const content1 = `---
|
||||
type: concept
|
||||
tags: [alpha]
|
||||
---
|
||||
|
||||
body
|
||||
`;
|
||||
const content2 = `---
|
||||
type: concept
|
||||
tags: [alpha, beta]
|
||||
---
|
||||
|
||||
body
|
||||
`;
|
||||
|
||||
let firstHash: string | undefined;
|
||||
let secondHash: string | undefined;
|
||||
|
||||
const engine1 = mockEngine({
|
||||
getPage: () => Promise.resolve(null),
|
||||
putPage: (_slug: string, page: any) => {
|
||||
firstHash = page.content_hash;
|
||||
return Promise.resolve(null);
|
||||
},
|
||||
});
|
||||
await importFromContent(engine1, 'concepts/x', content1, { noEmbed: true });
|
||||
|
||||
const engine2 = mockEngine({
|
||||
getPage: () => Promise.resolve({ content_hash: firstHash } as any),
|
||||
putPage: (_slug: string, page: any) => {
|
||||
secondHash = page.content_hash;
|
||||
return Promise.resolve(null);
|
||||
},
|
||||
});
|
||||
const r2 = await importFromContent(engine2, 'concepts/x', content2, { noEmbed: true });
|
||||
expect(r2.status).toBe('imported'); // tags changed, hash differs
|
||||
expect(secondHash).not.toBe(firstHash);
|
||||
});
|
||||
|
||||
test('ingested_at differences produce IDENTICAL hash (server-stamp dedup)', async () => {
|
||||
// Provenance write-through stamps `ingested_at` server-side per CV6;
|
||||
// a put_page that's just refreshing provenance (e.g. capture re-runs
|
||||
// the same file later) must not invalidate the chunk cache.
|
||||
const content1 = `---
|
||||
type: note
|
||||
ingested_at: '2026-05-22T10:00:00.000Z'
|
||||
---
|
||||
|
||||
body unchanged
|
||||
`;
|
||||
const content2 = `---
|
||||
type: note
|
||||
ingested_at: '2026-05-22T11:00:00.000Z'
|
||||
---
|
||||
|
||||
body unchanged
|
||||
`;
|
||||
|
||||
let firstHash: string | undefined;
|
||||
let shortCircuited = false;
|
||||
|
||||
const engine1 = mockEngine({
|
||||
getPage: () => Promise.resolve(null),
|
||||
putPage: (_slug: string, page: any) => {
|
||||
firstHash = page.content_hash;
|
||||
return Promise.resolve(null);
|
||||
},
|
||||
});
|
||||
await importFromContent(engine1, 'inbox/y', content1, { noEmbed: true });
|
||||
|
||||
const engine2 = mockEngine({
|
||||
getPage: () => Promise.resolve({ content_hash: firstHash } as any),
|
||||
putPage: () => {
|
||||
shortCircuited = false;
|
||||
return Promise.resolve(null);
|
||||
},
|
||||
});
|
||||
const r2 = await importFromContent(engine2, 'inbox/y', content2, { noEmbed: true });
|
||||
shortCircuited = r2.status === 'skipped';
|
||||
expect(shortCircuited).toBe(true);
|
||||
});
|
||||
});
|
||||
|
||||
@@ -0,0 +1,321 @@
|
||||
/**
|
||||
* v0.39.3.0 WARN-8 / A1 / CV6 / CV12 / T2 — put_page provenance.
|
||||
*
|
||||
* Migration v81 added 4 nullable provenance columns to `pages`
|
||||
* (source_kind, source_uri, ingested_via, ingested_at). This test
|
||||
* file pins the put_page op contract:
|
||||
*
|
||||
* 1. Trusted local callers (ctx.remote === false) can populate
|
||||
* source_kind / source_uri / ingested_via via params; the engine
|
||||
* stamps ingested_at server-side.
|
||||
* 2. Remote MCP callers (ctx.remote !== false) get server-stamped
|
||||
* `mcp:put_page` REGARDLESS of what they pass — CV6 spoofing guard.
|
||||
* 3. COALESCE-preserve UPDATE: a second put_page that omits provenance
|
||||
* does NOT overwrite the original ingestion's audit trail — first
|
||||
* write wins; routine edits survive without erasing it (CV12).
|
||||
* 4. Subagent namespace check still fires when provenance params are
|
||||
* present — adding params must not bypass the v0.26.9 F7b
|
||||
* fail-closed contract (T2 regression guard).
|
||||
*
|
||||
* All cases run against in-memory PGLite (hermetic, no DATABASE_URL).
|
||||
* Per CLAUDE.md "Test-isolation lint and helpers (v0.26.7)" the engine
|
||||
* is created in beforeAll, reset in beforeEach, disconnected in afterAll.
|
||||
*/
|
||||
|
||||
import { describe, test, expect, beforeAll, beforeEach, afterAll } from 'bun:test';
|
||||
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
|
||||
import { operations } from '../src/core/operations.ts';
|
||||
import type { OperationContext } from '../src/core/operations.ts';
|
||||
import { OperationError } from '../src/core/operations.ts';
|
||||
|
||||
const putPageOp = operations.find((o) => o.name === 'put_page')!;
|
||||
|
||||
let engine: PGLiteEngine;
|
||||
|
||||
beforeAll(async () => {
|
||||
engine = new PGLiteEngine();
|
||||
await engine.connect({});
|
||||
await engine.initSchema();
|
||||
});
|
||||
|
||||
afterAll(async () => {
|
||||
await engine.disconnect();
|
||||
});
|
||||
|
||||
beforeEach(async () => {
|
||||
// Wipe pages so each test starts from a known empty state. We use
|
||||
// executeRaw rather than a TRUNCATE-by-name sweep because this file
|
||||
// only touches one table.
|
||||
await engine.executeRaw('DELETE FROM pages', []);
|
||||
});
|
||||
|
||||
function makeCtx(opts: Partial<OperationContext> = {}): OperationContext {
|
||||
return {
|
||||
engine,
|
||||
config: { engine: 'pglite' as const },
|
||||
logger: {
|
||||
info: () => { /* noop */ },
|
||||
warn: () => { /* noop */ },
|
||||
error: () => { /* noop */ },
|
||||
},
|
||||
dryRun: false,
|
||||
remote: false,
|
||||
sourceId: 'default',
|
||||
...opts,
|
||||
};
|
||||
}
|
||||
|
||||
// Helper — read provenance columns straight from the DB so we don't depend
|
||||
// on the get_page op (Phase 3b extension).
|
||||
async function readProvenance(slug: string): Promise<{
|
||||
source_kind: string | null;
|
||||
source_uri: string | null;
|
||||
ingested_via: string | null;
|
||||
ingested_at: Date | null;
|
||||
}> {
|
||||
const rows = await engine.executeRaw(
|
||||
'SELECT source_kind, source_uri, ingested_via, ingested_at FROM pages WHERE slug = $1',
|
||||
[slug],
|
||||
) as Array<{ source_kind: unknown; source_uri: unknown; ingested_via: unknown; ingested_at: unknown }>;
|
||||
const r = rows[0];
|
||||
return {
|
||||
source_kind: (r?.source_kind as string | null) ?? null,
|
||||
source_uri: (r?.source_uri as string | null) ?? null,
|
||||
ingested_via: (r?.ingested_via as string | null) ?? null,
|
||||
ingested_at: r?.ingested_at ? new Date(r.ingested_at as string) : null,
|
||||
};
|
||||
}
|
||||
|
||||
describe('put_page provenance — trusted local caller (ctx.remote === false)', () => {
|
||||
test('client params honored: source_kind / source_uri / ingested_via populate DB', async () => {
|
||||
const ctx = makeCtx({ remote: false });
|
||||
await putPageOp.handler(ctx, {
|
||||
slug: 'wiki/p3a-local-write',
|
||||
content: '---\ntype: note\ntitle: Local Write\n---\n\nbody',
|
||||
source_kind: 'capture-cli',
|
||||
source_uri: 'file:///tmp/test.md',
|
||||
ingested_via: 'put_page',
|
||||
});
|
||||
const prov = await readProvenance('wiki/p3a-local-write');
|
||||
expect(prov.source_kind).toBe('capture-cli');
|
||||
expect(prov.source_uri).toBe('file:///tmp/test.md');
|
||||
expect(prov.ingested_via).toBe('put_page');
|
||||
// Server stamps ingested_at when ANY provenance field fires
|
||||
expect(prov.ingested_at).toBeInstanceOf(Date);
|
||||
expect(prov.ingested_at!.getTime()).toBeGreaterThan(Date.now() - 60_000);
|
||||
});
|
||||
|
||||
test('omitting all provenance params leaves all 4 DB columns null', async () => {
|
||||
const ctx = makeCtx({ remote: false });
|
||||
await putPageOp.handler(ctx, {
|
||||
slug: 'wiki/p3a-no-provenance',
|
||||
content: '---\ntype: note\ntitle: No Prov\n---\n\nbody',
|
||||
});
|
||||
const prov = await readProvenance('wiki/p3a-no-provenance');
|
||||
expect(prov.source_kind).toBeNull();
|
||||
expect(prov.source_uri).toBeNull();
|
||||
expect(prov.ingested_via).toBeNull();
|
||||
expect(prov.ingested_at).toBeNull();
|
||||
});
|
||||
|
||||
test('partial provenance (source_kind only) still triggers ingested_at stamp', async () => {
|
||||
const ctx = makeCtx({ remote: false });
|
||||
await putPageOp.handler(ctx, {
|
||||
slug: 'wiki/p3a-partial',
|
||||
content: '---\ntype: note\ntitle: Partial\n---\n\nbody',
|
||||
source_kind: 'capture-cli',
|
||||
});
|
||||
const prov = await readProvenance('wiki/p3a-partial');
|
||||
expect(prov.source_kind).toBe('capture-cli');
|
||||
expect(prov.source_uri).toBeNull();
|
||||
expect(prov.ingested_via).toBeNull();
|
||||
expect(prov.ingested_at).toBeInstanceOf(Date);
|
||||
});
|
||||
});
|
||||
|
||||
describe('put_page provenance — CV6 spoofing guard (ctx.remote !== false)', () => {
|
||||
test('remote caller cannot claim source_kind: capture-cli', async () => {
|
||||
const ctx = makeCtx({ remote: true });
|
||||
await putPageOp.handler(ctx, {
|
||||
slug: 'wiki/p3a-remote-spoof-attempt',
|
||||
content: '---\ntype: note\ntitle: Spoof\n---\n\nbody',
|
||||
source_kind: 'capture-cli', // client lies: pretends to be local CLI
|
||||
source_uri: 'spoofed://attacker-supplied',
|
||||
ingested_via: 'file-watcher', // client lies: claims daemon source
|
||||
});
|
||||
const prov = await readProvenance('wiki/p3a-remote-spoof-attempt');
|
||||
// Server overrode with mcp:put_page; client claims discarded
|
||||
expect(prov.source_kind).toBe('mcp:put_page');
|
||||
expect(prov.source_uri).toBeNull();
|
||||
expect(prov.ingested_via).toBe('mcp:put_page');
|
||||
expect(prov.ingested_at).toBeInstanceOf(Date);
|
||||
});
|
||||
|
||||
test('ctx.remote === undefined (no explicit trust) is treated as remote', async () => {
|
||||
// v0.26.9 F7b discipline: anything that isn't strictly `false` is remote.
|
||||
const ctx: OperationContext = {
|
||||
engine,
|
||||
config: { engine: 'pglite' as const },
|
||||
logger: { info: () => {}, warn: () => {}, error: () => {} },
|
||||
dryRun: false,
|
||||
// remote: explicitly undefined to exercise the fail-closed path
|
||||
remote: undefined as unknown as boolean,
|
||||
sourceId: 'default',
|
||||
};
|
||||
await putPageOp.handler(ctx, {
|
||||
slug: 'wiki/p3a-undefined-trust',
|
||||
content: '---\ntype: note\ntitle: Undefined\n---\n\nbody',
|
||||
source_kind: 'capture-cli',
|
||||
});
|
||||
const prov = await readProvenance('wiki/p3a-undefined-trust');
|
||||
expect(prov.source_kind).toBe('mcp:put_page');
|
||||
});
|
||||
});
|
||||
|
||||
describe('put_page provenance — CV12 COALESCE-preserve UPDATE', () => {
|
||||
test('second write WITHOUT provenance preserves first-write audit', async () => {
|
||||
const ctx = makeCtx({ remote: false });
|
||||
|
||||
// First write stamps capture-cli
|
||||
await putPageOp.handler(ctx, {
|
||||
slug: 'wiki/p3a-preserve',
|
||||
content: '---\ntype: note\ntitle: V1\n---\n\noriginal body',
|
||||
source_kind: 'capture-cli',
|
||||
ingested_via: 'put_page',
|
||||
});
|
||||
const first = await readProvenance('wiki/p3a-preserve');
|
||||
expect(first.source_kind).toBe('capture-cli');
|
||||
const firstStamp = first.ingested_at!.getTime();
|
||||
|
||||
// Second write — same slug, no provenance params
|
||||
await new Promise((r) => setTimeout(r, 10)); // ensure now() would tick
|
||||
await putPageOp.handler(ctx, {
|
||||
slug: 'wiki/p3a-preserve',
|
||||
content: '---\ntype: note\ntitle: V2\n---\n\nedited body',
|
||||
});
|
||||
const second = await readProvenance('wiki/p3a-preserve');
|
||||
// CV12: provenance preserved — first-write wins
|
||||
expect(second.source_kind).toBe('capture-cli');
|
||||
expect(second.ingested_via).toBe('put_page');
|
||||
// ingested_at preserved too (audit trail truthful about WHEN the first
|
||||
// ingestion happened, not WHEN the edit landed)
|
||||
expect(second.ingested_at!.getTime()).toBe(firstStamp);
|
||||
});
|
||||
|
||||
test('second write WITH new provenance overwrites (explicit re-ingestion)', async () => {
|
||||
const ctx = makeCtx({ remote: false });
|
||||
|
||||
await putPageOp.handler(ctx, {
|
||||
slug: 'wiki/p3a-reingest',
|
||||
content: '---\ntype: note\ntitle: V1\n---\n\nbody',
|
||||
source_kind: 'capture-cli',
|
||||
});
|
||||
|
||||
await new Promise((r) => setTimeout(r, 10));
|
||||
await putPageOp.handler(ctx, {
|
||||
slug: 'wiki/p3a-reingest',
|
||||
content: '---\ntype: note\ntitle: V2\n---\n\nbody',
|
||||
source_kind: 'file-watcher', // explicit re-ingest under different kind
|
||||
source_uri: 'file:///watched/path.md',
|
||||
ingested_via: 'file-watcher',
|
||||
});
|
||||
const prov = await readProvenance('wiki/p3a-reingest');
|
||||
// COALESCE(EXCLUDED.source_kind, pages.source_kind) — EXCLUDED non-null wins
|
||||
expect(prov.source_kind).toBe('file-watcher');
|
||||
expect(prov.source_uri).toBe('file:///watched/path.md');
|
||||
expect(prov.ingested_via).toBe('file-watcher');
|
||||
});
|
||||
|
||||
test('remote second write WITHOUT explicit provenance does NOT erase local first-write', async () => {
|
||||
// First: local CLI capture
|
||||
const localCtx = makeCtx({ remote: false });
|
||||
await putPageOp.handler(localCtx, {
|
||||
slug: 'wiki/p3a-local-then-remote',
|
||||
content: '---\ntype: note\ntitle: V1\n---\n\nbody',
|
||||
source_kind: 'capture-cli',
|
||||
ingested_via: 'put_page',
|
||||
});
|
||||
|
||||
// Second: remote MCP edit (server stamps mcp:put_page)
|
||||
const remoteCtx = makeCtx({ remote: true });
|
||||
await putPageOp.handler(remoteCtx, {
|
||||
slug: 'wiki/p3a-local-then-remote',
|
||||
content: '---\ntype: note\ntitle: V2\n---\n\nremote edit',
|
||||
});
|
||||
|
||||
// Remote second write is itself a provenance write (server-stamped),
|
||||
// so COALESCE(EXCLUDED.source_kind='mcp:put_page', pages.source_kind='capture-cli')
|
||||
// resolves to EXCLUDED's non-null value: 'mcp:put_page'. This is the
|
||||
// honest answer — the MOST RECENT ingestion source is mcp:put_page,
|
||||
// and the system says so. CV12 first-write-wins applies when the second
|
||||
// write OMITS provenance entirely (covered by the earlier test); when
|
||||
// the second write IS an ingestion, it gets to record itself.
|
||||
const prov = await readProvenance('wiki/p3a-local-then-remote');
|
||||
expect(prov.source_kind).toBe('mcp:put_page');
|
||||
});
|
||||
});
|
||||
|
||||
describe('put_page provenance — T2 subagent namespace regression', () => {
|
||||
test('subagent with provenance params STILL gets wiki/agents/<id>/ rejection', async () => {
|
||||
// Adding provenance params to put_page must NOT bypass the
|
||||
// viaSubagent + sandbox namespace check (operations.ts:556-578).
|
||||
// The check runs against `ctx.viaSubagent`, NOT against the params.
|
||||
// This regression guard pins it.
|
||||
const ctx = makeCtx({
|
||||
remote: true,
|
||||
viaSubagent: true,
|
||||
subagentId: 42,
|
||||
// No allowedSlugPrefixes — falls through to legacy default agent-namespace check
|
||||
});
|
||||
|
||||
// Attempt to write OUTSIDE the wiki/agents/42/ namespace WITH spoofed
|
||||
// provenance params (trying every angle to escape).
|
||||
await expect(
|
||||
putPageOp.handler(ctx, {
|
||||
slug: 'wiki/secret/leak',
|
||||
content: '---\ntype: note\ntitle: Leak\n---\n\nbody',
|
||||
source_kind: 'capture-cli',
|
||||
source_uri: 'file:///tmp/spoof',
|
||||
ingested_via: 'put_page',
|
||||
}),
|
||||
).rejects.toThrow(/wiki\/agents\/42/);
|
||||
});
|
||||
|
||||
test('subagent within wiki/agents/<id>/ namespace succeeds; provenance respects CV6', async () => {
|
||||
const ctx = makeCtx({
|
||||
remote: true,
|
||||
viaSubagent: true,
|
||||
subagentId: 42,
|
||||
});
|
||||
|
||||
await putPageOp.handler(ctx, {
|
||||
slug: 'wiki/agents/42/scratch',
|
||||
content: '---\ntype: note\ntitle: Subagent OK\n---\n\nbody',
|
||||
source_kind: 'capture-cli', // Spoof attempt — ignored by CV6
|
||||
});
|
||||
const prov = await readProvenance('wiki/agents/42/scratch');
|
||||
// CV6 gate fires AFTER the namespace check; client param ignored
|
||||
expect(prov.source_kind).toBe('mcp:put_page');
|
||||
});
|
||||
|
||||
test('subagent missing subagentId fails-closed regardless of provenance params', async () => {
|
||||
const ctx: OperationContext = {
|
||||
engine,
|
||||
config: { engine: 'pglite' as const },
|
||||
logger: { info: () => {}, warn: () => {}, error: () => {} },
|
||||
dryRun: false,
|
||||
remote: true,
|
||||
sourceId: 'default',
|
||||
viaSubagent: true,
|
||||
// subagentId intentionally absent
|
||||
};
|
||||
|
||||
await expect(
|
||||
putPageOp.handler(ctx, {
|
||||
slug: 'wiki/agents/42/anything',
|
||||
content: '---\ntype: note\n---\n\nbody',
|
||||
source_kind: 'capture-cli',
|
||||
}),
|
||||
).rejects.toThrow(OperationError);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,138 @@
|
||||
/**
|
||||
* v0.39.3.0 WARN-9 + CV12 — normalizeScopesInput contract.
|
||||
*
|
||||
* The admin SPA at /admin/api/register-client sends `scopes` as either a
|
||||
* string ('read write') or an array (['read', 'write']) depending on UI
|
||||
* version. OAuth wire format prefers `scope` (singular, space-string).
|
||||
* Pre-fix, the admin route did `scopes || 'read'` which broke on array
|
||||
* input AND silently ignored requests using `scope` (singular).
|
||||
*
|
||||
* This file pins the validator's behavior across every input shape codex
|
||||
* flagged plus a few defensive cases.
|
||||
*
|
||||
* Hermetic — pure-function unit tests, no engine.
|
||||
*/
|
||||
|
||||
import { describe, test, expect } from 'bun:test';
|
||||
import { normalizeScopesInput, InvalidScopeError } from '../src/core/scope.ts';
|
||||
|
||||
describe('normalizeScopesInput — happy paths', () => {
|
||||
test('undefined → "read" default', () => {
|
||||
expect(normalizeScopesInput(undefined)).toBe('read');
|
||||
});
|
||||
|
||||
test('null → "read" default', () => {
|
||||
expect(normalizeScopesInput(null)).toBe('read');
|
||||
});
|
||||
|
||||
test('string "read" → "read"', () => {
|
||||
expect(normalizeScopesInput('read')).toBe('read');
|
||||
});
|
||||
|
||||
test('string "read write" → sorted "read write" (deterministic)', () => {
|
||||
expect(normalizeScopesInput('read write')).toBe('read write');
|
||||
});
|
||||
|
||||
test('string "write read" → sorted "read write" (regardless of input order)', () => {
|
||||
expect(normalizeScopesInput('write read')).toBe('read write');
|
||||
});
|
||||
|
||||
test('array ["read", "write"] → "read write"', () => {
|
||||
expect(normalizeScopesInput(['read', 'write'])).toBe('read write');
|
||||
});
|
||||
|
||||
test('array ["admin"] → "admin"', () => {
|
||||
expect(normalizeScopesInput(['admin'])).toBe('admin');
|
||||
});
|
||||
|
||||
test('string "admin write read" → sorted "admin read write"', () => {
|
||||
expect(normalizeScopesInput('admin write read')).toBe('admin read write');
|
||||
});
|
||||
|
||||
test('dedupes repeated scopes in array', () => {
|
||||
expect(normalizeScopesInput(['read', 'read', 'write'])).toBe('read write');
|
||||
});
|
||||
|
||||
test('dedupes repeated scopes in string', () => {
|
||||
expect(normalizeScopesInput('read read write')).toBe('read write');
|
||||
});
|
||||
|
||||
test('extra whitespace in string normalized', () => {
|
||||
expect(normalizeScopesInput(' read write ')).toBe('read write');
|
||||
});
|
||||
|
||||
test('tab + newline whitespace also handled', () => {
|
||||
expect(normalizeScopesInput('read\twrite\nadmin')).toBe('admin read write');
|
||||
});
|
||||
});
|
||||
|
||||
describe('normalizeScopesInput — rejection cases', () => {
|
||||
test('number → Error (not string/array)', () => {
|
||||
expect(() => normalizeScopesInput(42)).toThrow(/must be a string or array/);
|
||||
});
|
||||
|
||||
test('plain object → Error', () => {
|
||||
expect(() => normalizeScopesInput({ scope: 'read' })).toThrow(/must be a string or array/);
|
||||
});
|
||||
|
||||
test('boolean → Error', () => {
|
||||
expect(() => normalizeScopesInput(true)).toThrow(/must be a string or array/);
|
||||
});
|
||||
|
||||
test('empty array → Error (no scopes after normalization)', () => {
|
||||
expect(() => normalizeScopesInput([])).toThrow(/empty after normalization/);
|
||||
});
|
||||
|
||||
test('empty string → Error (no scopes after normalization)', () => {
|
||||
expect(() => normalizeScopesInput('')).toThrow(/^scopes is empty/);
|
||||
});
|
||||
|
||||
test('whitespace-only string → Error (empty after split)', () => {
|
||||
expect(() => normalizeScopesInput(' ')).toThrow(/empty after normalization/);
|
||||
});
|
||||
|
||||
test('array with non-string element (null) → Error', () => {
|
||||
expect(() => normalizeScopesInput(['read', null])).toThrow(/must contain only strings.*null/);
|
||||
});
|
||||
|
||||
test('array with non-string element (number) → Error', () => {
|
||||
expect(() => normalizeScopesInput(['read', 42])).toThrow(/must contain only strings.*number/);
|
||||
});
|
||||
|
||||
test('array with empty-string element → Error', () => {
|
||||
expect(() => normalizeScopesInput(['read', ''])).toThrow(/empty strings/);
|
||||
});
|
||||
|
||||
test('array with whitespace-in-element ["read write"] → Error (the codex-flagged bug shape)', () => {
|
||||
expect(() => normalizeScopesInput(['read write'])).toThrow(/whitespace.*single scope name/);
|
||||
});
|
||||
|
||||
test('array with internal-tab element → Error', () => {
|
||||
expect(() => normalizeScopesInput(['read\twrite'])).toThrow(/whitespace/);
|
||||
});
|
||||
|
||||
test('unknown scope name → InvalidScopeError (allowlist enforcement)', () => {
|
||||
expect(() => normalizeScopesInput(['root'])).toThrow(InvalidScopeError);
|
||||
expect(() => normalizeScopesInput('root')).toThrow(/Unknown scope "root"/);
|
||||
});
|
||||
|
||||
test('string with unknown scope name → InvalidScopeError', () => {
|
||||
expect(() => normalizeScopesInput('flying-unicorn')).toThrow(/Unknown scope/);
|
||||
});
|
||||
|
||||
test('mix of known + unknown → InvalidScopeError on the unknown', () => {
|
||||
expect(() => normalizeScopesInput('read flying-unicorn')).toThrow(/Unknown scope "flying-unicorn"/);
|
||||
});
|
||||
});
|
||||
|
||||
describe('normalizeScopesInput — determinism', () => {
|
||||
test('same input always produces same output (sorted)', () => {
|
||||
expect(normalizeScopesInput(['write', 'admin', 'read']))
|
||||
.toBe(normalizeScopesInput('read admin write'));
|
||||
});
|
||||
|
||||
test('hierarchy-aware scopes (sources_admin, users_admin, agent) accepted', () => {
|
||||
expect(normalizeScopesInput(['sources_admin', 'users_admin'])).toBe('sources_admin users_admin');
|
||||
expect(normalizeScopesInput('agent')).toBe('agent');
|
||||
});
|
||||
});
|
||||
Reference in New Issue
Block a user