mirror of
https://github.com/garrytan/gbrain.git
synced 2026-07-27 08:11:17 +00:00
v0.42.53.0 fix(sync,db): #2339 op_checkpoints jsonb double-encode + bug-class sweep + CI guard (#2375)
* fix(sync): op_checkpoints pin write double-encodes jsonb — every sync aborts (#2339) recordCompleted bound JSON.stringify(array) to a $3::jsonb param via postgres.js .unsafe(), double-encoding it into a jsonb string scalar that violates the v119 op_checkpoints_completed_keys_array CHECK — aborting every multi-source sync on real Postgres at the first checkpoint write. PGLite parses the string silently, which is why unit tests stayed green and it shipped. Cast through $3::text::jsonb so the text->jsonb cast parses a genuine array. Adds a DATABASE_URL-gated parity test + a dedicated Postgres CI job so the guard can never silently skip. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(db): sweep positional jsonb double-encode sites + AST CI guard (#2324) Every executeRaw/.unsafe site that bound JSON.stringify(x) to a bare positional jsonb cast double-encodes on real Postgres (same class as #2339). Sweep them all to the text::jsonb form across query-cache, sources-ops, llm-base, calibration-profile, impact-capture, subagent, receipt-write, traversal-cache, symbol-resolver, and the agent/sources commands. Adds scripts/check-jsonb-params.mjs (AST-lite scanner for the positional form the legacy template grep misses, incl. generic-typed calls), wired into check-jsonb-pattern.sh, with a self-test. PGLite's native db.query is not scanned — it parses text to jsonb natively, so the bug can't occur there. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(search,eval): alias-hop injected results carry page_id (contradiction-probe crash) applyAliasHop injected synthetic SearchResults without page_id (the `as SearchResult` cast hid the missing field), so listActiveTakesForPages bound undefined/NaN into ANY($1::int[]) and crashed the whole contradiction probe on real Postgres. Stamp page_id=page.id at the injection site and add a finite-id filter in generateIntraPagePairs as a defensive backstop (mirrors hybrid.ts:63). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(engines): positional jsonb binding rule (text::jsonb vs the double-encode trap) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * v0.42.53.0 fix(sync,db): #2339 op_checkpoints jsonb double-encode + bug-class sweep + CI guard Bumps VERSION + package.json to 0.42.53.0, adds the CHANGELOG entry, and regenerates llms-full.txt. Ships the #2339 sync-abort hotfix, the repo-wide positional jsonb double-encode sweep, the alias-hop contradiction-probe crash fix, and the new positional-form CI guard. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: post-ship sync — jsonb invariant now covers the positional form + new guard CLAUDE.md JSONB invariant + KEY_FILES (sql-query, check-jsonb-pattern, op-checkpoint) now describe the #2339 positional double-encode class, the $N::text::jsonb fix, and the new check-jsonb-params.mjs guard. Regenerates llms-full.txt. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 4.8
parent
bb2e88c42a
commit
814258dda6
@@ -20,6 +20,49 @@ concurrency:
|
||||
cancel-in-progress: true
|
||||
|
||||
jobs:
|
||||
jsonb-parity:
|
||||
# Dedicated required guard for the JSONB double-encode bug-class (#2339).
|
||||
# PGLite parses a double-encoded jsonb string silently, so this assertion can
|
||||
# ONLY be made on real Postgres — a normal gated e2e file would skip without
|
||||
# DATABASE_URL and let the bug ship green (as #2339 did). This job provisions
|
||||
# Postgres and HARD-FAILS if DATABASE_URL is missing, so the guard can never
|
||||
# silently skip.
|
||||
name: JSONB parity (#2339 regression guard)
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 15
|
||||
services:
|
||||
postgres:
|
||||
image: pgvector/pgvector:pg16
|
||||
env:
|
||||
POSTGRES_USER: postgres
|
||||
POSTGRES_PASSWORD: postgres
|
||||
POSTGRES_DB: gbrain_test
|
||||
ports:
|
||||
- 5432:5432
|
||||
options: >-
|
||||
--health-cmd pg_isready
|
||||
--health-interval 10s
|
||||
--health-timeout 5s
|
||||
--health-retries 5
|
||||
steps:
|
||||
- uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
|
||||
- uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2
|
||||
with:
|
||||
bun-version: 1.3.13
|
||||
- run: bun install
|
||||
- name: Require DATABASE_URL (no silent skip)
|
||||
env:
|
||||
DATABASE_URL: postgresql://postgres:postgres@localhost:5432/gbrain_test
|
||||
run: |
|
||||
if [ -z "$DATABASE_URL" ]; then
|
||||
echo "::error::DATABASE_URL must be set for the jsonb-parity job — the #2339 guard would silently skip (the exact failure PGLite hides). Failing the job." >&2
|
||||
exit 1
|
||||
fi
|
||||
- name: Run JSONB double-encode parity tests on real Postgres
|
||||
env:
|
||||
DATABASE_URL: postgresql://postgres:postgres@localhost:5432/gbrain_test
|
||||
run: bun test test/e2e/op-checkpoint-jsonb-parity.test.ts test/e2e/jsonb-roundtrip.test.ts
|
||||
|
||||
tier1:
|
||||
name: Tier 1 (Mechanical)
|
||||
runs-on: ubuntu-latest
|
||||
|
||||
@@ -2,6 +2,21 @@
|
||||
|
||||
All notable changes to GBrain will be documented in this file.
|
||||
|
||||
## [0.42.53.0] - 2026-06-23
|
||||
|
||||
**`gbrain sync` works again on managed Postgres brains: the durable-checkpoint pin write was encoding its value the wrong way, so every multi-source sync aborted at the very first checkpoint. Fixed, plus a repo-wide sweep of the same JSONB footgun and a new CI guard so it can't come back.** A recent release added a structural check on the sync checkpoint table; the pin write that runs before every drain bound its value as a string rather than a real array, so the check rejected it and the run bailed before importing anything. The bug was invisible on the embedded engine (its driver parses the value either way) and only bit managed Postgres.
|
||||
|
||||
### Fixed
|
||||
- **Multi-source sync no longer aborts at the first checkpoint.** The sync-target pin write now binds its value so Postgres stores a genuine JSONB array instead of a double-encoded string scalar. A dedicated Postgres CI job exercises this on a real database, because the embedded test engine masks the failure — which is exactly why it shipped.
|
||||
- **The same JSONB double-encode footgun is swept across the codebase.** Every raw write that serialized a value into a JSONB column the bug-prone way is corrected to the safe form (search cache, source config, calibration profiles, subagent tool records, eval receipts, code-intel cache, symbol resolver, and others). Readers were already defensive, so existing rows self-heal as each is rewritten.
|
||||
- **`gbrain eval suspected-contradictions` no longer crashes on an exact-alias query.** An alias-matched result was missing its page id, which aborted the whole probe on Postgres; the id is now carried through, with a finite-id filter as a defensive backstop.
|
||||
|
||||
### Added
|
||||
- **A CI guard for the positional JSONB double-encode pattern.** The existing guard caught only the template-string spelling; a new static check (`scripts/check-jsonb-params.mjs`) catches the positional-parameter form — the one behind this wave — across the codebase, with its own self-test. The embedded engine's native path is intentionally not flagged, since the bug can't occur there.
|
||||
|
||||
### To take advantage of v0.42.53.0
|
||||
`gbrain upgrade`. Multi-source Postgres brains that had stopped syncing resume on the next `gbrain sync` — no migration, no manual step. Rows written in the double-encoded form before the fix self-heal as each is rewritten; re-running the affected write (or a sync) repairs them eagerly if you'd rather not wait.
|
||||
|
||||
## [0.42.52.0] - 2026-06-18
|
||||
|
||||
**Autopilot stops manufacturing dead jobs and wedging its own queue, plus four operational rough edges get fixed: minion attempt-accounting, `agent run` flag parsing, honest `sources status`, and a budgeted `gbrain status`.** On a multi-source Postgres brain, autopilot could fan out a continuous stream of dead `autopilot-cycle` jobs while the supervisor periodically wedged the very queue it exists to keep alive. The root cause was one disease with several interacting parts; this wave addresses all of them, then cleans up four smaller reliability bugs found alongside.
|
||||
|
||||
@@ -59,9 +59,14 @@ Per-file detail is in `docs/architecture/KEY_FILES.md`.
|
||||
- **Source isolation.** Every read-side op routes through `sourceScopeOpts(ctx)`; precedence
|
||||
is federated array (`ctx.auth.allowedSources`) > scalar (`ctx.sourceId`) > nothing. Don't
|
||||
hand-roll source filtering — a missed thread is a cross-source data leak.
|
||||
- **JSONB: never `JSON.stringify` into a `::jsonb` cast.** postgres.js double-encodes it;
|
||||
PGLite hides the bug. Pass raw objects to `engine.executeRaw`, or use `executeRawJsonb`.
|
||||
Guarded by `scripts/check-jsonb-pattern.sh`.
|
||||
- **JSONB: never `JSON.stringify` into a `::jsonb` cast.** postgres.js double-encodes it (a jsonb
|
||||
string scalar); PGLite hides the bug. This bites BOTH spellings — the template form
|
||||
(`${JSON.stringify(x)}::jsonb`) AND the positional form (`executeRaw(\`…$N::jsonb\`, [JSON.stringify(x)])`,
|
||||
the #2339 class that aborted every sync). Fix: pass a raw object to `engine.executeRaw` / use
|
||||
`executeRawJsonb` / `sql.json()`; or for the positional path bind through `$N::text::jsonb` (binds as
|
||||
text, the cast parses it). Guarded by `scripts/check-jsonb-pattern.sh` (template grep) +
|
||||
`scripts/check-jsonb-params.mjs` (positional AST scanner); the real backstop is the DATABASE_URL-gated
|
||||
e2e parity tests, since PGLite can't surface the bug. Full rule in `docs/ENGINES.md`.
|
||||
- **Engine parity.** `src/core/postgres-engine.ts` and `src/core/pglite-engine.ts` move in
|
||||
lockstep — a new method/SQL shape lands in BOTH, pinned by `test/e2e/engine-parity.test.ts`.
|
||||
Forward-referenced columns/indexes go in the bootstrap probe set (guarded by
|
||||
|
||||
@@ -176,6 +176,39 @@ RRF fusion, multi-query expansion, and 4-layer dedup are engine-agnostic. They o
|
||||
|
||||
**Migration:** `gbrain migrate --to supabase` exports everything (pages, chunks, embeddings, links, tags, timeline) and imports into Supabase. `gbrain migrate --to pglite` goes the other direction. Bidirectional, lossless.
|
||||
|
||||
## JSONB writes: never double-encode (the #2339 trap)
|
||||
|
||||
Writing a JS value into a `jsonb` column has exactly two correct forms. Get this
|
||||
wrong and the write succeeds on PGLite but stores a **jsonb string scalar** on
|
||||
real Postgres — `col ->> 'k'` returns NULL, `jsonb_array_elements` throws, and a
|
||||
`jsonb_typeof = 'array'` CHECK rejects the row (this aborted every sync in #2339).
|
||||
|
||||
| Form | Verdict |
|
||||
|---|---|
|
||||
| Template tag: `` sql`... ${sql.json(obj)}` `` (postgres-engine only) | ✅ native jsonb serialization |
|
||||
| Positional raw call, raw object: `executeRawJsonb(engine, sql, scalars, [obj])` | ✅ object reaches the wire as jsonb |
|
||||
| Positional raw call, stringified: `executeRaw(\`... $N::text::jsonb\`, [JSON.stringify(x)])` | ✅ binds as text, the cast parses it |
|
||||
| Positional raw call, BARE cast: `executeRaw(\`... $N::jsonb\`, [JSON.stringify(x)])` | ❌ **double-encodes** under postgres.js `.unsafe()` |
|
||||
| Template literal interpolation: `` `... ${JSON.stringify(x)}::jsonb` `` | ❌ double-encodes |
|
||||
|
||||
**Why:** postgres.js `.unsafe(sql, params)` (the path behind `executeRaw` /
|
||||
`executeRawDirect`) binds a JS **string** as a text param. A bare `$N::jsonb`
|
||||
cast then wraps that already-JSON string into a jsonb scalar string instead of
|
||||
parsing it. Casting through `$N::text::jsonb` forces a text→jsonb parse.
|
||||
**PGLite's `db.query` parses text→jsonb natively, so it hides the bug** — which is
|
||||
why a regression only shows up on Postgres (and why the parity test must run there).
|
||||
|
||||
**Two CI guards enforce this, both wired into `scripts/check-jsonb-pattern.sh`:**
|
||||
- the template-tag grep (`${JSON.stringify(x)}::jsonb`), and
|
||||
- `scripts/check-jsonb-params.mjs`, an AST-lite scanner for the positional
|
||||
`$N::jsonb` + `JSON.stringify` form the grep misses. Sanctioned escapes:
|
||||
`$N::text::jsonb`, `$N::text[]`, `executeRawJsonb`, `sql.json`, or an inline
|
||||
`jsonb-guard-ok` comment.
|
||||
|
||||
The real backstop is `test/e2e/op-checkpoint-jsonb-parity.test.ts` +
|
||||
`test/e2e/jsonb-roundtrip.test.ts`, which round-trip writes through real Postgres
|
||||
and assert `jsonb_typeof` — the assertion PGLite cannot make.
|
||||
|
||||
## Adding a new engine
|
||||
|
||||
1. Create `src/core/<name>-engine.ts` implementing `BrainEngine`
|
||||
|
||||
@@ -265,7 +265,7 @@ per-release `**vX.Y.Z:**` narration — CI enforces this
|
||||
- `src/mcp/dispatch.ts` — shared tool-call dispatch consumed by both stdio (`server.ts`) and HTTP transports. Exports `dispatchToolCall(engine, name, params, opts)`, `buildOperationContext(engine, params, opts)`, `validateParams(op, params)`. Single source of truth for `(ctx, params)` handler arg order and the 5-field `OperationContext` shape (engine + config + logger + dryRun + remote). Defaults `remote: true` (untrusted); local CLI callers pass `remote: false`. Also exports `summarizeMcpParams(opName, params)` — privacy-preserving redactor for `mcp_request_log` and the admin SSE feed, returns `{redacted, kind, declared_keys, unknown_key_count, approx_bytes}`. Intersects submitted top-level keys against the operation's declared `params` allow-list (declared keys preserved sorted; unknown keys counted but never named, closing the attacker-controlled-key-name leak). Byte counts bucketed up to nearest 1KB so an attacker can't binary-search secret-content sizes by probing. Raw payload visibility is opt-in via `gbrain serve --http --log-full-params` (loud stderr warning). New logging paths route through this helper, not `JSON.stringify(params)`.
|
||||
- `src/mcp/rate-limit.ts` — Bounded-LRU token-bucket limiter. `buildDefaultLimiters()` returns the two-bucket pipeline: pre-auth IP (30/60s, fires BEFORE the DB lookup so brute-force load against `access_tokens` is capped) + post-auth token-id (60/60s). Tracks `lastTouchedMs` separately from `lastRefillMs` so an exhausted key can't be reset by hammering past the TTL. LRU cap bounds memory under attacker-controlled key growth.
|
||||
- `src/commands/serve-http.ts` — Express 5 HTTP MCP server with OAuth 2.1, admin dashboard, and SSE live activity feed. Started via `gbrain serve --http [--port N] [--token-ttl N] [--enable-dcr] [--public-url URL] [--bind HOST] [--log-full-params]`. Combines MCP SDK's `mcpAuthRouter` (authorize/token/register/revoke), a custom `client_credentials` handler running BEFORE the router (SDK's token endpoint throws `UnsupportedGrantTypeError` for CC; custom handler falls through for `auth_code` / `refresh_token`), `requireBearerAuth` middleware for `/mcp` with scope enforcement + `localOnly` rejection before op dispatch, and `express-rate-limit` at 50 req / 15 min on `/token`. Serves the built admin SPA from `admin/dist/` with SPA fallback. `/admin/events` SSE broadcasts every MCP request. `cookie-parser` wired (Express 5 has no built-in). Startup logging prints port, engine, issuer URL (honors `--public-url`), client count, DCR status, admin bootstrap token. The `/mcp` request handler's OperationContext literal sets `remote: true` explicitly (without it `submit_job`'s protected-name guard at `operations.ts:1391` saw a falsy undefined and a `read+write`-scoped OAuth token could submit `shell` jobs — RCE). `summarizeMcpParams` from `src/mcp/dispatch.ts` feeds both `mcp_request_log` writes and the SSE feed by default (raw via `--log-full-params`). Cookie `Secure` flag set behind HTTPS or a public-URL proxy; magic-link nonce store LRU-bounded; DCR disable routes through the `GBrainOAuthProvider` `dcrDisabled` constructor option (not a router monkey-patch); `transport.handleRequest` wrapped in try/catch to return a JSON-RPC 500 envelope; OperationError + unexpected exceptions unified through `buildError` / `serializeError` so `/mcp` always returns the same envelope. `/health` is liveness-only via `probeLiveness(sql, engineName, version, timeoutMs)` racing `sql\`SELECT 1\`` against the exported `HEALTH_TIMEOUT_MS = 3000` (returns the same `ProbeHealthResult` tagged-union as `probeHealth`, single timer-cleanup site, single 503 envelope); body shape `{status, version, engine}` only. Full stats moved to admin-only `/admin/api/full-stats` (gated by `requireAdmin`, calls `probeHealth(engine, ...)`) — keeps `getStats()`'s 6× count(*) off the public route so a saturated pool doesn't trigger orchestrator restart cascades. Every OAuth/admin/audit SQL call routes through `sqlQueryForEngine(engine)` from `src/core/sql-query.ts` so it works against PGLite; the four `mcp_request_log.params` INSERT sites (success / auth_failed / scope_denied / server-error) go through `executeRawJsonb(engine, ...)` so the column stores real objects (`params->>'op'` returns `search`, not the quoted string). `--bind HOST` defaults `127.0.0.1` (self-hosters pass `--bind 0.0.0.0`); a stderr WARN fires when `--public-url` is set without `--bind`; the banner prints a `Bind:` line. `AuthInfo.sourceId` + `AuthInfo.allowedSources` are the typed source of truth, populated by `oauth-provider.ts:verifyAccessToken` from the `oauth_clients` row. The HTTP MCP `tools/list` handler at `:837-849` uses `paramDefToSchema(v)` from `src/mcp/tool-defs.ts` so array params keep `items` (strict-mode OAuth clients otherwise reject the whole tool list).
|
||||
- `src/core/sql-query.ts` — engine-aware tagged-template SQL adapter for OAuth/admin/auth infrastructure. `sqlQueryForEngine(engine)` returns a `SqlQuery` (`(strings, ...values) => Promise<rows[]>`) that walks the template, builds `$N` positional SQL, asserts every value is a `SqlValue` (string | number | bigint | boolean | Date | null), and routes through `engine.executeRaw(sql, params)` (Postgres via postgres.js `unsafe(sql, params)`, PGLite via `db.query(sql, params)`). Deliberately narrower than postgres.js's `sql` tag: no nested fragments, `sql.json()`, `sql.unsafe()`, `sql.begin()`, or array binding — the narrow scalar-only surface is the feature (keeps it from drifting into a partial postgres.js clone). JSONB writes go through `executeRawJsonb(engine, sql, scalarParams, jsonbParams)` which composes positional `$N::jsonb` casts and passes JS objects through; the double-encode bug class doesn't apply because positional binding through `unsafe()` reaches the wire protocol with the correct type oid (verified by `test/sql-query.test.ts` on PGLite, `test/e2e/auth-permissions.test.ts:67` on Postgres). `scripts/check-jsonb-pattern.sh` doesn't fire because `executeRawJsonb(...)` is a method call, not the banned literal-template-tag interpolation. Consumed by `src/commands/auth.ts`, `src/commands/serve-http.ts`, `src/core/oauth-provider.ts`, `src/commands/files.ts`, `src/mcp/http-transport.ts` so all five work uniformly against PGLite and Postgres.
|
||||
- `src/core/sql-query.ts` — engine-aware tagged-template SQL adapter for OAuth/admin/auth infrastructure. `sqlQueryForEngine(engine)` returns a `SqlQuery` (`(strings, ...values) => Promise<rows[]>`) that walks the template, builds `$N` positional SQL, asserts every value is a `SqlValue` (string | number | bigint | boolean | Date | null), and routes through `engine.executeRaw(sql, params)` (Postgres via postgres.js `unsafe(sql, params)`, PGLite via `db.query(sql, params)`). Deliberately narrower than postgres.js's `sql` tag: no nested fragments, `sql.json()`, `sql.unsafe()`, `sql.begin()`, or array binding — the narrow scalar-only surface is the feature (keeps it from drifting into a partial postgres.js clone). JSONB writes go through `executeRawJsonb(engine, sql, scalarParams, jsonbParams)` which composes positional `$N::jsonb` casts and passes JS **objects** through; an object reaches the wire with the correct type oid, so executeRawJsonb is safe (verified by `test/sql-query.test.ts` on PGLite, `test/e2e/auth-permissions.test.ts:67` on Postgres). Positional binding is NOT universally immune, though: binding a `JSON.stringify(x)` **string** to a bare `$N::jsonb` via `unsafe()` double-encodes it into a jsonb string scalar on real Postgres (the #2339 class; PGLite hides it). Fixes: pass a raw object (executeRawJsonb / `sql.json`), or cast through `$N::text::jsonb`. `scripts/check-jsonb-pattern.sh` (template grep) doesn't fire on `executeRawJsonb(...)` because it passes objects; the positional `$N::jsonb` + `JSON.stringify` form is caught by `scripts/check-jsonb-params.mjs`. Consumed by `src/commands/auth.ts`, `src/commands/serve-http.ts`, `src/core/oauth-provider.ts`, `src/commands/files.ts`, `src/mcp/http-transport.ts` so all five work uniformly against PGLite and Postgres.
|
||||
- `src/commands/serve.ts` — `gbrain serve` stdio MCP entrypoint with idempotent shutdown across every parent-disconnect signal. Stdio EOF, SIGTERM, SIGINT, SIGHUP, and parent-process death (every reparent case — PID 1, launchd subreaper, systemd, tmux, or a parent shell with `PR_SET_CHILD_SUBREAPER`) all funnel into one `cleanup(reason)` that releases the engine and the PGLite write-lock dir within 5 seconds (otherwise the lock is held indefinitely after Claude Desktop / Cursor / launchd-managed gateways disconnect, forcing a 5-minute stale-lock wait on next start). Watchdog reparent check is `getParentPid() !== initialParentPid` (the `=== 1` check missed the subreaper case under launchd/systemd). Bun's `process.ppid` cache is stale across reparenting ([oven-sh/bun#30305](https://github.com/oven-sh/bun/issues/30305)) so `getParentPid()` runs `spawnSync('ps', ['-o', 'ppid=', '-p', PID])` per tick. Startup probe verifies `ps` is on PATH; if not (stripped containers, busybox), the watchdog skips installing AND emits a loud `[gbrain serve] watchdog disabled: ps unavailable ...` stderr line so operators see the degraded mode. Pinned by `test/serve-stdio-lifecycle.test.ts` (22 cases). Credit @Aragorn2046 + @seungsu-kr.
|
||||
- `src/core/oauth-provider.ts` — `GBrainOAuthProvider` implementing the MCP SDK's `OAuthServerProvider` + `OAuthRegisteredClientsStore`. Backed by raw SQL (works on both PGLite and Postgres — OAuth is infrastructure, not a BrainEngine concern). Full OAuth 2.1: `authorize` + `exchangeAuthorizationCode` with PKCE, `client_credentials`, `refresh_token` with rotation, `revokeToken`, `registerClient` (DCR validates redirect_uri is `https://` or loopback per RFC 6749 §3.1.2.1). All tokens + client secrets SHA-256 hashed before storage. Auth codes single-use with 10-minute TTL via atomic `DELETE...RETURNING` (closes RFC 6749 §10.5 TOCTOU); refresh rotation also `DELETE...RETURNING` (§10.4 stolen-token detection). `pgArray()` escapes commas/quotes/braces so a comma-bearing redirect_uri can't smuggle a second array element. Legacy `access_tokens` fallback in `verifyAccessToken` grandfathers pre-v0.26 bearer tokens as `read+write+admin`. `sweepExpiredTokens()` runs on startup in try/catch and returns the count via `RETURNING 1` + array length. RFC hardening: `client_id` folded atomically into the `DELETE WHERE` for both auth-code exchange and refresh rotation (wrong-client paths don't burn the row); refresh-scope-subset enforced against the original grant on the row (RFC 6749 §6, so revoking a scope shrinks existing refresh tokens); `client_id` bound on `revokeToken` (RFC 7009 §2.1); `/token` `redirect_uri` validated against the `/authorize` value (RFC 6749 §4.1.3, empty-string treated as missing not wildcard); bare `catch {}` in `verifyAccessToken`/`getClient` replaced by `isUndefinedColumnError` from `src/core/utils.ts` (only SQLSTATE 42703 falls through to legacy; lock timeouts/network blips throw); `dcrDisabled` constructor option lets `serve-http.ts` disable `/register` without monkey-patching the router. Module-private `coerceTimestamp()` normalizes postgres-driver-as-string BIGINT columns to JS numbers at 5 read sites (`getClient` for RFC 7591 §3.2.1 numeric timestamps, `exchangeRefreshToken` + `verifyAccessToken` for the SDK's `typeof === 'number'` check); throws on NaN/Infinity (fail loud at boundary), returns undefined for SQL NULL (callers treat NULL as expired). Not promoted to `utils.ts` — generic BIGINT precision-loss risk. `registerClient` honors `token_endpoint_auth_method: "none"` (RFC 7591 §3.2.1): public PKCE clients store `client_secret_hash = NULL` and the response omits `client_secret`; confidential clients (`client_secret_post` / `client_secret_basic`) keep their one-time-reveal shape; `getClient` normalizes NULL `client_secret_hash` to JS `undefined` so the SDK's clientAuth path accepts public clients. `verifyAccessToken` JOINs `oauth_clients.source_id` (write scope, scalar) + `oauth_clients.federated_read` (read scope, TEXT[]) onto the returned `AuthInfo`; legacy brains degrade via `isUndefinedColumnError` fallback.
|
||||
- `admin/` — React 19 + Vite + TypeScript admin SPA embedded in the binary via `admin/dist/` served by `serve-http.ts`. 7 screens: Login (bootstrap token → session cookie), Dashboard (metrics + SSE feed + token health), Agents (sortable table + sparklines + Register), Register (modal with scope checkboxes + grant type selector), Credentials reveal (Copy + Download JSON + one-time-only warning), Request Log (filterable paginated), Agent Detail drawer (Details / Activity / Config Export tabs + Revoke). Design tokens: `#0a0a0f` bg, Inter for UI, JetBrains Mono for data, 4-32px spacing scale, rounded pill badges. HTTP-only SameSite=Strict cookie auth. 65KB gzip. Build: `cd admin && bun install && bun run build`; output at `admin/dist/` is committed for self-contained binaries.
|
||||
@@ -326,7 +326,8 @@ per-release `**vX.Y.Z:**` narration — CI enforces this
|
||||
- `scripts/check-progress-to-stdout.sh` — CI guard against regressing to `\r`-on-stdout progress. Wired into `bun run test` via `scripts/check-progress-to-stdout.sh && bun test` in package.json.
|
||||
- `docs/progress-events.md` — Canonical JSON event schema reference. Additive only.
|
||||
- `src/core/markdown.ts` — Frontmatter parsing + body splitter. `coerceFrontmatterString(v)` coerces a non-string `title`/`slug`/`type` to a deterministic string at parse time so a YAML-typed value never reaches `.toLowerCase()` and throws (the #1939 wedge: `title: 2024-06-01` parsed as a `Date`, `title: 1458` as a number, and the throw blocked the sync bookmark from advancing); a `Date` becomes its UTC ISO date (`2024-06-01`, machine-independent and matching the on-disk token, unlike `String(date)`), `null`/`undefined` become `''`, everything else uses `String()`. `splitBody` requires an explicit timeline sentinel (`<!-- timeline -->`, `--- timeline ---`, or `---` immediately before `## Timeline`/`## History`). Plain `---` in body text is a markdown horizontal rule, not a separator. `inferType` auto-types `/wiki/analysis/` → analysis, `/wiki/guides/` → guide, `/wiki/hardware/` → hardware, `/wiki/architecture/` → architecture, `/writing/` → writing (plus existing people/companies/deals/etc heuristics).
|
||||
- `scripts/check-jsonb-pattern.sh` — CI grep guard. Fails the build if anyone reintroduces (a) the `${JSON.stringify(x)}::jsonb` interpolation pattern (postgres.js v3 double-encodes it), or (b) `max_stalled INTEGER NOT NULL DEFAULT 1` in any schema source file (must be DEFAULT 5 to preserve SIGKILL-rescue). Wired into `bun test`.
|
||||
- `scripts/check-jsonb-pattern.sh` — CI grep guard. Fails the build if anyone reintroduces (a) the `${JSON.stringify(x)}::jsonb` interpolation pattern (postgres.js v3 double-encodes it), or (b) `max_stalled INTEGER NOT NULL DEFAULT 1` in any schema source file (must be DEFAULT 5 to preserve SIGKILL-rescue). It also invokes `scripts/check-jsonb-params.mjs` and propagates its exit code. Wired into `bun test`.
|
||||
- `scripts/check-jsonb-params.mjs` — AST-lite CI guard for the POSITIONAL jsonb double-encode form the template grep above misses: an `executeRaw`/`executeRawDirect`/`.unsafe()` call whose balanced arg span binds `JSON.stringify(x)` into a bare `$N::jsonb` cast (the #2339 class). Walks each call's balanced span respecting strings/templates/comments, handles generic-typed calls (`executeRaw<T>(`), and allows the sanctioned forms (`$N::text::jsonb`, `$N::text[]`, `executeRawJsonb`, `sql.json`, an inline `jsonb-guard-ok` comment). PGLite's native `db.query` is deliberately not scanned (it parses text→jsonb, so the bug can't occur there). Heuristic by design (whole-span correlation; can't see a `JSON.stringify` assigned to a variable before the call) — the real backstop is the DATABASE_URL-gated e2e parity tests. Scan roots overridable via argv for its self-test (`test/check-jsonb-params.test.ts`).
|
||||
- `scripts/check-source-id-projection.sh` — CI grep guard for the multi-source bug class. Greps `src/core/postgres-engine.ts` + `src/core/pglite-engine.ts` for `SELECT.*FROM pages` projections matching the `rowToPage` feeder shape (id + slug + type + title) and fails if `source_id` is missing. `Page.source_id` is required at the type level; a projection dropping the column produces `Page` rows with `source_id: undefined` while TypeScript's `: string` lies about it. Wired into `bun run verify` + `bun run check:all`.
|
||||
- `docker-compose.ci.yml` + `scripts/ci-local.sh` — Local CI gate. `bun run ci:local` spins up four `pgvector/pgvector:pg16` services (postgres-1..4) + `oven/bun:1` with named volumes (`gbrain-ci-pg-data-{1..4}`, `gbrain-ci-node-modules`, `gbrain-ci-bun-cache`), runs gitleaks on host, smoke-tests `scripts/run-e2e.sh` argv handling, runs guards + typecheck, then the Tier 1 default: 4-shard parallel unit + E2E (`xargs -P4`, one Postgres per shard; unit phase keeps `DATABASE_URL` unset). `--no-shard` falls back to the legacy unsharded sequential flow (debug aid); `--diff` runs the diff-aware selector unsharded. Also runs a `pgbouncer` service (`edoburu/pgbouncer`, `POOL_MODE: transaction`, `AUTH_TYPE: plain` — pg16 stores SCRAM verifiers, so the userlist must hold the plaintext password; `IGNORE_STARTUP_PARAMETERS` whitelists gbrain's `statement_timeout`/`idle_in_transaction_session_timeout` startup params the way the Supabase pooler does) fronting postgres-1 on host port `GBRAIN_CI_PGBOUNCER_PORT` (default 6543); every E2E invocation exports `GBRAIN_PGBOUNCER_URL` (pooled; dedicated `gbrain_pgbouncer` database so it never races the `gbrain_test` TRUNCATE fixtures) + `GBRAIN_PGBOUNCER_DIRECT_URL`, consumed by `test/e2e/pgbouncer-teardown.test.ts` — the transaction-mode teardown bug class (#1972/#2015/#2084) reproduced in the local gate instead of only in production. `--no-pull` skips upstream pulls; `--clean` nukes named volumes. Postgres host port defaults to 5434; override with `GBRAIN_CI_PG_PORT=NNNN`. Stronger gate than PR CI's 2-file Tier 1 set.
|
||||
- `scripts/select-e2e.ts` + `scripts/e2e-test-map.ts` — Diff-aware E2E test selector. Reads three git sources (committed `origin/master...HEAD`, working-tree `HEAD`, and `git ls-files --others --exclude-standard` for untracked NOT-gitignored files), classifies as EMPTY / DOC_ONLY / SRC. Fail-closed: EMPTY → all files; DOC_ONLY (every path matches the README/CLAUDE/AGENTS/CHANGELOG/TODOS allowlist) → empty stdout; SRC → escape-hatch paths (schema, package.json, skills/) trigger all, else the hand-tuned `E2E_TEST_MAP` glob narrows, and an unmapped src/ change still emits ALL files (never silently nothing). Pure-function exports `selectTests`, `classify`, `matchGlob`. `bun run ci:select-e2e` prints the current selection on stdout. `test/select-e2e.test.ts` covers all 4 branches plus 3 regression guards (skills/, untracked files, unmapped src/) — 24 cases.
|
||||
@@ -383,7 +384,7 @@ per-release `**vX.Y.Z:**` narration — CI enforces this
|
||||
- `src/commands/report.ts` — Structured report saver (audit trail for maintenance/enrichment)
|
||||
- `src/core/destructive-guard.ts` — three-layer protection against accidental data loss. `assessDestructiveImpact(engine, sourceId)` counts pages/chunks/embeddings/files for a source. `checkDestructiveConfirmation(impact, opts)` is the fail-closed gate (`--confirm-destructive` required when data is present; `--yes` alone is rejected). `softDeleteSource` / `restoreSource` / `listArchivedSources` / `purgeExpiredSources` drive the source-level archive lifecycle via `sources.archived BOOLEAN`, `archived_at TIMESTAMPTZ`, `archive_expires_at TIMESTAMPTZ`. Page-level analog: `BrainEngine.softDeletePage` / `restorePage` / `purgeDeletedPages` plus `pages.deleted_at TIMESTAMPTZ` and a partial purge index. The MCP `delete_page` op rewires to `softDeletePage`; ops `restore_page` (`scope: write`) and `purge_deleted_pages` (`scope: admin`, `localOnly: true`) round out the surface. Search visibility (`buildVisibilityClause` in `src/core/search/sql-ranking.ts`) hides soft-deleted pages and archived sources from `searchKeyword` / `searchKeywordChunks` / `searchVector` in both engines. The autopilot cycle's `purge` phase calls `purgeExpiredSources` + `engine.purgeDeletedPages(72)` so the 72h TTL is real.
|
||||
- `src/commands/pages.ts` — `gbrain pages purge-deleted [--older-than HOURS|Nd] [--dry-run] [--json]` operator escape hatch. Mirror of `gbrain sources purge` for the page-level lifecycle. Hard-deletes pages whose `deleted_at` is older than the cutoff; cascades to content_chunks/page_links/chunk_relations.
|
||||
- `src/core/op-checkpoint.ts` — DB-backed checkpoint primitive for long-running ops. Migration v67 introduces `op_checkpoints (op TEXT, fingerprint TEXT, completed_keys JSONB, updated_at TIMESTAMPTZ, PK(op, fingerprint))`. Per-op fingerprint helpers (`embedFingerprint`, `extractFingerprint`, `reindexFingerprint`, `integrityFingerprint`, `purgeFingerprint`) compute `sha8(canonical-JSON(relevant-params))` so re-running with the same params resumes from `completed_keys` and re-running with different params (e.g. `--limit 100` vs `--limit 200`) starts fresh. Cross-worker safe on Postgres (DB row, no file-lock race); PGLite degrades gracefully. Replaces per-op file-backed JSON checkpoints scattered across `import.ts`, `embed.ts`, `reindex.ts`. The 7-day TTL GC runs in the cycle's `purge` phase. All writes (`recordCompleted`, `clearOpCheckpoint`) route through `engine.executeRawDirect` + `withRetry(BULK_RETRY_OPTS)` so they survive Supavisor pool exhaustion, and `recordCompleted` returns `boolean` (banked vs failed-after-retries) — the 9 non-sync consumers keep its REPLACE-into-`completed_keys` semantics. Resumable sync uses the additive `appendCompleted(key, deltaKeys)` / `appendCompletedOnce` (the latter no-retry for the SIGTERM path) which INSERT a delta into the `op_checkpoint_paths` child table (migration v115: `(op, fingerprint, path)` PK, FK to `op_checkpoints` ON DELETE CASCADE) via a single writable-CTE `unnest($3::text[])` write — O(delta), killing the old O(N²) full-set rewrite. `loadOpCheckpoint` returns the `UNION ALL` of legacy `completed_keys` + child-table paths (deduped in JS), so an in-flight upgrade loses nothing. The legacy arm is gated on `jsonb_typeof(completed_keys) = 'array'` so a non-array (scalar) parent row can't make `jsonb_array_elements_text` throw "cannot extract elements from a scalar" and take down the whole union (which would discard the valid child rows and lose all banked progress for the key); a third union arm flags the corruption so the loader logs it once and keeps the child rows. Migration v119 adds the `op_checkpoints_completed_keys_array` CHECK (`jsonb_typeof(completed_keys) = 'array'`) — a DB-enforced, always-on guard that makes the scalar-corruption class structurally impossible going forward; the migration repairs any pre-existing scalar to `'[]'` under `LOCK TABLE ... IN SHARE ROW EXCLUSIVE MODE` and `src/core/schema-embedded.ts` + `src/core/pglite-schema.ts` ship the same CHECK on fresh installs (a loader hit now implies schema drift, a disabled constraint, or an out-of-band writer). `syncFingerprint({sourceId, lastCommit})` keys the sync rows. Pinned by `test/op-checkpoint.test.ts` (incl. delta-append, union read, cascade clear, durable-write boolean, and the scalar-parent guard). `import-checkpoint.ts` was NOT migrated to this primitive — both checkpoint systems coexist without conflict; migrating requires async-propagating 4 sync call sites in `src/commands/import.ts` and rewriting 18 tests, deferred.
|
||||
- `src/core/op-checkpoint.ts` — DB-backed checkpoint primitive for long-running ops. Migration v67 introduces `op_checkpoints (op TEXT, fingerprint TEXT, completed_keys JSONB, updated_at TIMESTAMPTZ, PK(op, fingerprint))`. Per-op fingerprint helpers (`embedFingerprint`, `extractFingerprint`, `reindexFingerprint`, `integrityFingerprint`, `purgeFingerprint`) compute `sha8(canonical-JSON(relevant-params))` so re-running with the same params resumes from `completed_keys` and re-running with different params (e.g. `--limit 100` vs `--limit 200`) starts fresh. Cross-worker safe on Postgres (DB row, no file-lock race); PGLite degrades gracefully. Replaces per-op file-backed JSON checkpoints scattered across `import.ts`, `embed.ts`, `reindex.ts`. The 7-day TTL GC runs in the cycle's `purge` phase. All writes (`recordCompleted`, `clearOpCheckpoint`) route through `engine.executeRawDirect` + `withRetry(BULK_RETRY_OPTS)` so they survive Supavisor pool exhaustion, and `recordCompleted` returns `boolean` (banked vs failed-after-retries) — the 9 non-sync consumers keep its REPLACE-into-`completed_keys` semantics. Resumable sync uses the additive `appendCompleted(key, deltaKeys)` / `appendCompletedOnce` (the latter no-retry for the SIGTERM path) which INSERT a delta into the `op_checkpoint_paths` child table (migration v115: `(op, fingerprint, path)` PK, FK to `op_checkpoints` ON DELETE CASCADE) via a single writable-CTE `unnest($3::text[])` write — O(delta), killing the old O(N²) full-set rewrite. `loadOpCheckpoint` returns the `UNION ALL` of legacy `completed_keys` + child-table paths (deduped in JS), so an in-flight upgrade loses nothing. The legacy arm is gated on `jsonb_typeof(completed_keys) = 'array'` so a non-array (scalar) parent row can't make `jsonb_array_elements_text` throw "cannot extract elements from a scalar" and take down the whole union (which would discard the valid child rows and lose all banked progress for the key); a third union arm flags the corruption so the loader logs it once and keeps the child rows. Migration v119 adds the `op_checkpoints_completed_keys_array` CHECK (`jsonb_typeof(completed_keys) = 'array'`) — a DB-enforced, always-on guard that makes the scalar-corruption class structurally impossible going forward; the migration repairs any pre-existing scalar to `'[]'` under `LOCK TABLE ... IN SHARE ROW EXCLUSIVE MODE` and `src/core/schema-embedded.ts` + `src/core/pglite-schema.ts` ship the same CHECK on fresh installs (a loader hit now implies schema drift, a disabled constraint, or an out-of-band writer). `recordCompleted` binds its array through `$3::text::jsonb` (NOT a bare `$3::jsonb`) so postgres.js `.unsafe()` doesn't double-encode `JSON.stringify(sorted)` into the scalar string that CHECK rejects — the #2339 bug that aborted every multi-source sync at the first pin write (PGLite parsed it silently, so it shipped). A DATABASE_URL-gated `test/e2e/op-checkpoint-jsonb-parity.test.ts` (its own CI job) asserts the array shape on real Postgres. `syncFingerprint({sourceId, lastCommit})` keys the sync rows. Pinned by `test/op-checkpoint.test.ts` (incl. delta-append, union read, cascade clear, durable-write boolean, and the scalar-parent guard). `import-checkpoint.ts` was NOT migrated to this primitive — both checkpoint systems coexist without conflict; migrating requires async-propagating 4 sync call sites in `src/commands/import.ts` and rewriting 18 tests, deferred.
|
||||
- `src/core/brain-score-recommendations.ts` — pure data layer consumed by both `gbrain doctor --remediation-plan` / `--remediate` and `gbrain features`. `computeRecommendations(checks, opts)` returns `Remediation[]` with stable `id`, content-hash `idempotency_key`, `severity`, `est_seconds`, `est_usd_cost`, `depends_on` (references stable ids, not check names — so plan order is reproducible). `classifyChecks(report)` triages every doctor check three-state into `remediable | human_only | blocked` (`human_only` covers RLS warnings and other human-judgment gates; `blocked` covers dependency chains where a parent check failed). `maxReachableScore(checks)` computes the ceiling for empty/under-configured brains (no entity pages → graph_coverage caps at 70; no embedding key → embedding_coverage caps at 60). Cost estimates pull from `anthropic-pricing.ts` (synthesize/patterns/consolidate) and `embedding-pricing.ts` (embed jobs). Pinned by `test/brain-score-recommendations.test.ts` (~27 cases incl. determinism, content-hash idempotency, DB-backed checkpoint provenance, three-state triage).
|
||||
- `src/commands/doctor.ts` extension — `--remediation-plan [--json] [--target-score N]` prints what would run (stable `id`, `idempotency_key`, `severity`, `est_seconds`, `est_usd_cost`, `depends_on`); `--remediate [--yes] [--target-score N] [--max-usd N]` submits each plan step as a Minion job in dependency order, re-checking score between steps. `--target-score N` defaults to 90; refuses to start when target exceeds `maxReachableScore()` and lists what's missing. `--max-usd N` is the cron-safety guard — submission refuses when the plan's `est_total_usd_cost` exceeds the cap. JSON envelope adds a `Check.remediation` field (additive, schema_version unchanged). Pinned by tests in `test/doctor.test.ts`.
|
||||
- `src/commands/jobs.ts` extension — registers 11 Minion handlers: `reindex`, `repair-jsonb`, `orphans`, `integrity`, `purge`, `synthesize` (PROTECTED), `patterns` (PROTECTED), `consolidate` (PROTECTED), `extract_facts`, `resolve_symbol_edges`, `recompute_emotional_weight`. Phase wrappers delegate to `runCycle({phases:[name]})` so `src/core/cycle.ts` stays the single source of truth for phase semantics. The standalone `sync` handler passes `noExtract: true` to match `runPhaseSync`'s contract (doctor's remediation plan emitting `[sync, extract]` would otherwise double-extract).
|
||||
|
||||
+41
-3
@@ -208,9 +208,14 @@ Per-file detail is in `docs/architecture/KEY_FILES.md`.
|
||||
- **Source isolation.** Every read-side op routes through `sourceScopeOpts(ctx)`; precedence
|
||||
is federated array (`ctx.auth.allowedSources`) > scalar (`ctx.sourceId`) > nothing. Don't
|
||||
hand-roll source filtering — a missed thread is a cross-source data leak.
|
||||
- **JSONB: never `JSON.stringify` into a `::jsonb` cast.** postgres.js double-encodes it;
|
||||
PGLite hides the bug. Pass raw objects to `engine.executeRaw`, or use `executeRawJsonb`.
|
||||
Guarded by `scripts/check-jsonb-pattern.sh`.
|
||||
- **JSONB: never `JSON.stringify` into a `::jsonb` cast.** postgres.js double-encodes it (a jsonb
|
||||
string scalar); PGLite hides the bug. This bites BOTH spellings — the template form
|
||||
(`${JSON.stringify(x)}::jsonb`) AND the positional form (`executeRaw(\`…$N::jsonb\`, [JSON.stringify(x)])`,
|
||||
the #2339 class that aborted every sync). Fix: pass a raw object to `engine.executeRaw` / use
|
||||
`executeRawJsonb` / `sql.json()`; or for the positional path bind through `$N::text::jsonb` (binds as
|
||||
text, the cast parses it). Guarded by `scripts/check-jsonb-pattern.sh` (template grep) +
|
||||
`scripts/check-jsonb-params.mjs` (positional AST scanner); the real backstop is the DATABASE_URL-gated
|
||||
e2e parity tests, since PGLite can't surface the bug. Full rule in `docs/ENGINES.md`.
|
||||
- **Engine parity.** `src/core/postgres-engine.ts` and `src/core/pglite-engine.ts` move in
|
||||
lockstep — a new method/SQL shape lands in BOTH, pinned by `test/e2e/engine-parity.test.ts`.
|
||||
Forward-referenced columns/indexes go in the bootstrap probe set (guarded by
|
||||
@@ -2116,6 +2121,39 @@ RRF fusion, multi-query expansion, and 4-layer dedup are engine-agnostic. They o
|
||||
|
||||
**Migration:** `gbrain migrate --to supabase` exports everything (pages, chunks, embeddings, links, tags, timeline) and imports into Supabase. `gbrain migrate --to pglite` goes the other direction. Bidirectional, lossless.
|
||||
|
||||
## JSONB writes: never double-encode (the #2339 trap)
|
||||
|
||||
Writing a JS value into a `jsonb` column has exactly two correct forms. Get this
|
||||
wrong and the write succeeds on PGLite but stores a **jsonb string scalar** on
|
||||
real Postgres — `col ->> 'k'` returns NULL, `jsonb_array_elements` throws, and a
|
||||
`jsonb_typeof = 'array'` CHECK rejects the row (this aborted every sync in #2339).
|
||||
|
||||
| Form | Verdict |
|
||||
|---|---|
|
||||
| Template tag: `` sql`... ${sql.json(obj)}` `` (postgres-engine only) | ✅ native jsonb serialization |
|
||||
| Positional raw call, raw object: `executeRawJsonb(engine, sql, scalars, [obj])` | ✅ object reaches the wire as jsonb |
|
||||
| Positional raw call, stringified: `executeRaw(\`... $N::text::jsonb\`, [JSON.stringify(x)])` | ✅ binds as text, the cast parses it |
|
||||
| Positional raw call, BARE cast: `executeRaw(\`... $N::jsonb\`, [JSON.stringify(x)])` | ❌ **double-encodes** under postgres.js `.unsafe()` |
|
||||
| Template literal interpolation: `` `... ${JSON.stringify(x)}::jsonb` `` | ❌ double-encodes |
|
||||
|
||||
**Why:** postgres.js `.unsafe(sql, params)` (the path behind `executeRaw` /
|
||||
`executeRawDirect`) binds a JS **string** as a text param. A bare `$N::jsonb`
|
||||
cast then wraps that already-JSON string into a jsonb scalar string instead of
|
||||
parsing it. Casting through `$N::text::jsonb` forces a text→jsonb parse.
|
||||
**PGLite's `db.query` parses text→jsonb natively, so it hides the bug** — which is
|
||||
why a regression only shows up on Postgres (and why the parity test must run there).
|
||||
|
||||
**Two CI guards enforce this, both wired into `scripts/check-jsonb-pattern.sh`:**
|
||||
- the template-tag grep (`${JSON.stringify(x)}::jsonb`), and
|
||||
- `scripts/check-jsonb-params.mjs`, an AST-lite scanner for the positional
|
||||
`$N::jsonb` + `JSON.stringify` form the grep misses. Sanctioned escapes:
|
||||
`$N::text::jsonb`, `$N::text[]`, `executeRawJsonb`, `sql.json`, or an inline
|
||||
`jsonb-guard-ok` comment.
|
||||
|
||||
The real backstop is `test/e2e/op-checkpoint-jsonb-parity.test.ts` +
|
||||
`test/e2e/jsonb-roundtrip.test.ts`, which round-trip writes through real Postgres
|
||||
and assert `jsonb_typeof` — the assertion PGLite cannot make.
|
||||
|
||||
## Adding a new engine
|
||||
|
||||
1. Create `src/core/<name>-engine.ts` implementing `BrainEngine`
|
||||
|
||||
+1
-1
@@ -143,5 +143,5 @@
|
||||
"bun": ">=1.3.10"
|
||||
},
|
||||
"license": "MIT",
|
||||
"version": "0.42.52.0"
|
||||
"version": "0.42.53.0"
|
||||
}
|
||||
|
||||
@@ -0,0 +1,128 @@
|
||||
#!/usr/bin/env node
|
||||
/**
|
||||
* CI guard for the POSITIONAL jsonb double-encode footgun (#2339 / #2324 class).
|
||||
*
|
||||
* The legacy scripts/check-jsonb-pattern.sh only catches the template-tag form
|
||||
* (`${JSON.stringify(x)}::jsonb`). It MISSES the positional-param form:
|
||||
*
|
||||
* engine.executeRaw(`... $3::jsonb ...`, [a, b, JSON.stringify(x)])
|
||||
*
|
||||
* Under postgres.js `.unsafe(sql, params)` a JS STRING bound to a `$N::jsonb`
|
||||
* param double-encodes — the text→jsonb cast wraps the already-JSON string into a
|
||||
* jsonb *string scalar*. PGLite parses it silently, so the bug is invisible in
|
||||
* unit tests and only bites on real Postgres (it aborted every sync in #2339).
|
||||
*
|
||||
* This scanner flags any executeRaw / executeRawDirect / .unsafe(...) call whose
|
||||
* balanced argument span contains BOTH a positional `$N::jsonb` cast
|
||||
* (NOT `$N::text::jsonb`, NOT `$N::text[]`) AND a `JSON.stringify(` — the exact
|
||||
* double-encode shape. It is heuristic by design (whole-span correlation); the
|
||||
* real backstop is the DATABASE_URL-gated e2e parity test. Keep both.
|
||||
*
|
||||
* Allowed forms (NOT flagged):
|
||||
* - `$N::text::jsonb` + JSON.stringify (the fix: binds as text, cast parses it)
|
||||
* - `$N::text[]` (the unnest path — arrays bind fine)
|
||||
* - executeRawJsonb(...) (passes raw objects, not strings)
|
||||
* - sql.json(x) (postgres.js native jsonb serializer)
|
||||
* - a `jsonb-guard-ok` comment anywhere in the call span (explicit opt-out)
|
||||
*
|
||||
* Exit 0 = clean, 1 = violations found. Runs under node or bun.
|
||||
*/
|
||||
import { readdirSync, readFileSync, statSync } from 'node:fs';
|
||||
import { join } from 'node:path';
|
||||
|
||||
// Default scan roots; overridable via argv so the guard's own test can point it
|
||||
// at a fixture dir (e.g. `node check-jsonb-params.mjs /tmp/fixtures`).
|
||||
const ROOTS = process.argv.slice(2).length > 0 ? process.argv.slice(2) : ['src', 'scripts'];
|
||||
// executeRawDirect must precede executeRaw in the alternation so the longer name
|
||||
// wins; executeRawJsonb is deliberately excluded (it passes objects). The
|
||||
// optional `<...>` handles generic type args, e.g. `executeRaw<{ id: string }>(`.
|
||||
//
|
||||
// Only the postgres.js raw path is scanned (executeRaw/executeRawDirect/.unsafe).
|
||||
// PGLite's native `this.db.query(...)` is intentionally NOT matched: its driver
|
||||
// parses a text→jsonb cast natively, so the double-encode that bites postgres.js
|
||||
// `.unsafe()` does not occur there (the `pglite-masks` invariant). The engine
|
||||
// parity test pins that the resulting jsonb_typeof agrees across both engines.
|
||||
const CALL_RE = /\b(executeRawDirect|executeRaw|unsafe)\s*(?:<[^>;]*>)?\s*\(/g;
|
||||
|
||||
/** Walk from the '(' at openIdx and return [start,end) of the balanced span,
|
||||
* respecting strings, template literals, and comments. */
|
||||
function findSpan(src, openIdx) {
|
||||
let depth = 0;
|
||||
let mode = 'code'; // code | line | block | sq | dq | tpl
|
||||
for (let i = openIdx; i < src.length; i++) {
|
||||
const c = src[i];
|
||||
const n = src[i + 1];
|
||||
if (mode === 'line') { if (c === '\n') mode = 'code'; continue; }
|
||||
if (mode === 'block') { if (c === '*' && n === '/') { mode = 'code'; i++; } continue; }
|
||||
if (mode === 'sq') { if (c === '\\') { i++; continue; } if (c === "'") mode = 'code'; continue; }
|
||||
if (mode === 'dq') { if (c === '\\') { i++; continue; } if (c === '"') mode = 'code'; continue; }
|
||||
if (mode === 'tpl') { if (c === '\\') { i++; continue; } if (c === '`') mode = 'code'; continue; }
|
||||
// mode === 'code'
|
||||
if (c === '/' && n === '/') { mode = 'line'; i++; continue; }
|
||||
if (c === '/' && n === '*') { mode = 'block'; i++; continue; }
|
||||
if (c === "'") { mode = 'sq'; continue; }
|
||||
if (c === '"') { mode = 'dq'; continue; }
|
||||
if (c === '`') { mode = 'tpl'; continue; }
|
||||
if (c === '(') depth++;
|
||||
else if (c === ')') { depth--; if (depth === 0) return [openIdx + 1, i]; }
|
||||
}
|
||||
return [openIdx + 1, src.length];
|
||||
}
|
||||
|
||||
/** Blank out comments so a commented-out example doesn't trip the JSON.stringify probe. */
|
||||
function stripComments(s) {
|
||||
return s.replace(/\/\/[^\n]*/g, '').replace(/\/\*[\s\S]*?\*\//g, '');
|
||||
}
|
||||
|
||||
const violations = [];
|
||||
|
||||
function scanFile(file) {
|
||||
const src = readFileSync(file, 'utf8');
|
||||
CALL_RE.lastIndex = 0;
|
||||
let m;
|
||||
while ((m = CALL_RE.exec(src))) {
|
||||
const method = m[1];
|
||||
const openIdx = m.index + m[0].length - 1; // index of the '('
|
||||
const [s, e] = findSpan(src, openIdx);
|
||||
const span = src.slice(s, e);
|
||||
if (/jsonb-guard-ok/.test(span)) continue;
|
||||
if (!/JSON\.stringify\s*\(/.test(stripComments(span))) continue;
|
||||
// A positional `$N::jsonb` that is NOT `$N::text::jsonb`.
|
||||
const jsonbRe = /\$\d+\s*::\s*jsonb\b/g;
|
||||
let j;
|
||||
let badText = '';
|
||||
while ((j = jsonbRe.exec(span))) {
|
||||
const pre = span.slice(Math.max(0, j.index - 12), j.index);
|
||||
if (/::\s*text\s*$/.test(pre)) continue; // $N::text::jsonb is the fix — allowed
|
||||
badText = j[0].replace(/\s+/g, '');
|
||||
break;
|
||||
}
|
||||
if (!badText) continue;
|
||||
const line = src.slice(0, s).split('\n').length;
|
||||
violations.push(
|
||||
`${file}:${line} ${method}(...) binds JSON.stringify into ${badText} — use $N::text::jsonb or pass a raw object (executeRawJsonb / sql.json)`,
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
function walk(dir) {
|
||||
let ents;
|
||||
try { ents = readdirSync(dir); } catch { return; }
|
||||
for (const ent of ents) {
|
||||
if (ent === 'node_modules') continue;
|
||||
const p = join(dir, ent);
|
||||
const st = statSync(p);
|
||||
if (st.isDirectory()) walk(p);
|
||||
else if (p.endsWith('.ts') && !p.endsWith('.test.ts')) scanFile(p);
|
||||
}
|
||||
}
|
||||
|
||||
for (const root of ROOTS) walk(root);
|
||||
|
||||
if (violations.length) {
|
||||
console.error('JSONB positional double-encode violations (#2339 class):\n');
|
||||
for (const v of violations) console.error(' ' + v);
|
||||
console.error(`\n${violations.length} violation(s). Fix: bind through $N::text::jsonb (keeping JSON.stringify), or pass a raw object via executeRawJsonb / sql.json. See docs/ENGINES.md.`);
|
||||
process.exit(1);
|
||||
}
|
||||
console.log('check-jsonb-params: clean (no positional $N::jsonb + JSON.stringify double-encodes)');
|
||||
@@ -44,3 +44,17 @@ if grep -rEn "$MAX_STALLED_PATTERN" src/schema.sql src/core/migrate.ts src/core/
|
||||
fi
|
||||
|
||||
echo "OK: max_stalled defaults are 5 in all schema sources"
|
||||
|
||||
# v0.42.x (#2339 / #2324): positional `$N::jsonb` + JSON.stringify double-encode.
|
||||
# The template-string grep above only catches `${JSON.stringify(x)}::jsonb`. It
|
||||
# MISSES the positional-param form — executeRaw(`... $N::jsonb ...`,
|
||||
# [JSON.stringify(x)]) — which is the exact shape that double-encoded the
|
||||
# op_checkpoints pin and aborted every sync in #2339. The AST-lite scanner below
|
||||
# catches it. `set -e` propagates its non-zero exit.
|
||||
if command -v node >/dev/null 2>&1; then
|
||||
node scripts/check-jsonb-params.mjs
|
||||
elif command -v bun >/dev/null 2>&1; then
|
||||
bun scripts/check-jsonb-params.mjs
|
||||
else
|
||||
echo "WARN: neither node nor bun on PATH; skipping check-jsonb-params.mjs" >&2
|
||||
fi
|
||||
|
||||
@@ -310,7 +310,7 @@ async function runFanout(engine: BrainEngine, queue: MinionQueue, flags: RunFlag
|
||||
// do this after submission because each add() returns the committed
|
||||
// row's id; the aggregator's seed started with an empty array.
|
||||
await engine.executeRaw(
|
||||
`UPDATE minion_jobs SET data = jsonb_set(data, '{children_ids}', $1::jsonb) WHERE id = $2`,
|
||||
`UPDATE minion_jobs SET data = jsonb_set(data, '{children_ids}', $1::text::jsonb) WHERE id = $2`,
|
||||
[JSON.stringify(childIds), aggregator.id],
|
||||
);
|
||||
|
||||
|
||||
@@ -692,7 +692,7 @@ async function runFederate(engine: BrainEngine, args: string[], value: boolean):
|
||||
const config = parseConfig(src.config);
|
||||
config.federated = value;
|
||||
await engine.executeRaw(
|
||||
`UPDATE sources SET config = $1::jsonb WHERE id = $2`,
|
||||
`UPDATE sources SET config = $1::text::jsonb WHERE id = $2`,
|
||||
[JSON.stringify(config), id],
|
||||
);
|
||||
console.log(`Source "${id}" is now ${value ? 'federated (appears in cross-source default search)' : 'isolated (only searched when explicitly named)'}.`);
|
||||
@@ -879,7 +879,7 @@ async function runWebhookSet(engine: BrainEngine, args: string[]): Promise<void>
|
||||
cfg.webhook_secret = secret;
|
||||
cfg.github_repo = githubRepo;
|
||||
await engine.executeRaw(
|
||||
`UPDATE sources SET config = $1::jsonb WHERE id = $2`,
|
||||
`UPDATE sources SET config = $1::text::jsonb WHERE id = $2`,
|
||||
[JSON.stringify(cfg), id],
|
||||
);
|
||||
|
||||
@@ -935,7 +935,7 @@ async function runWebhookRotate(engine: BrainEngine, args: string[]): Promise<vo
|
||||
const cfg = parseConfig(src.config);
|
||||
cfg.webhook_secret = secret;
|
||||
await engine.executeRaw(
|
||||
`UPDATE sources SET config = $1::jsonb WHERE id = $2`,
|
||||
`UPDATE sources SET config = $1::text::jsonb WHERE id = $2`,
|
||||
[JSON.stringify(cfg), id],
|
||||
);
|
||||
console.log(`New webhook secret for source "${id}":`);
|
||||
@@ -959,7 +959,7 @@ async function runWebhookClear(engine: BrainEngine, args: string[]): Promise<voi
|
||||
delete cfg.webhook_secret;
|
||||
delete cfg.github_repo;
|
||||
await engine.executeRaw(
|
||||
`UPDATE sources SET config = $1::jsonb WHERE id = $2`,
|
||||
`UPDATE sources SET config = $1::text::jsonb WHERE id = $2`,
|
||||
[JSON.stringify(cfg), id],
|
||||
);
|
||||
console.log(`Webhook configuration cleared for source "${id}".`);
|
||||
@@ -984,7 +984,7 @@ async function runTrackedBranch(engine: BrainEngine, args: string[]): Promise<vo
|
||||
if (setArg) {
|
||||
cfg.tracked_branch = setArg;
|
||||
await engine.executeRaw(
|
||||
`UPDATE sources SET config = $1::jsonb WHERE id = $2`,
|
||||
`UPDATE sources SET config = $1::text::jsonb WHERE id = $2`,
|
||||
[JSON.stringify(cfg), id],
|
||||
);
|
||||
console.log(`Tracked branch for source "${id}" set to "${setArg}".`);
|
||||
@@ -1000,7 +1000,7 @@ async function runTrackedBranch(engine: BrainEngine, args: string[]): Promise<vo
|
||||
const branch = execFileSync('git', ['-C', src.local_path, 'rev-parse', '--abbrev-ref', 'HEAD'], { encoding: 'utf8' }).trim();
|
||||
cfg.tracked_branch = branch;
|
||||
await engine.executeRaw(
|
||||
`UPDATE sources SET config = $1::jsonb WHERE id = $2`,
|
||||
`UPDATE sources SET config = $1::text::jsonb WHERE id = $2`,
|
||||
[JSON.stringify(cfg), id],
|
||||
);
|
||||
console.log(`Detected branch "${branch}" for source "${id}"; persisted to config.tracked_branch.`);
|
||||
|
||||
Binary file not shown.
@@ -123,7 +123,7 @@ export async function putCachedTraversal<T>(
|
||||
`INSERT INTO code_traversal_cache
|
||||
(symbol_qualified, depth, source_id, response_json,
|
||||
max_chunk_updated_at, xmin_max, cluster_generation)
|
||||
VALUES ($1, $2, $3, $4::jsonb, $5::timestamptz, $6, $7)
|
||||
VALUES ($1, $2, $3, $4::text::jsonb, $5::timestamptz, $6, $7)
|
||||
ON CONFLICT (symbol_qualified, depth, source_id)
|
||||
DO UPDATE SET
|
||||
response_json = EXCLUDED.response_json,
|
||||
|
||||
@@ -266,13 +266,13 @@ async function writeDbCache<T>(
|
||||
): Promise<void> {
|
||||
const [, , contentSha] = splitCacheKey(key);
|
||||
if (!contentSha) return;
|
||||
// executeRaw with positional binding for JSONB. Per the sql-query.ts
|
||||
// contract: object values passed via positional params reach the
|
||||
// wire as proper jsonb when cast.
|
||||
// #2339 class: this binds JSON.stringify(value) (a STRING) positionally, so it
|
||||
// must cast through $4::text::jsonb — a bare $4::jsonb double-encodes under
|
||||
// postgres.js .unsafe() (PGLite hides it). Pass a raw object to use $N::jsonb.
|
||||
await engine.executeRaw(
|
||||
`INSERT INTO conversation_parser_llm_cache
|
||||
(content_sha256, model_id, call_shape, value_json)
|
||||
VALUES ($1, $2, $3, $4::jsonb)
|
||||
VALUES ($1, $2, $3, $4::text::jsonb)
|
||||
ON CONFLICT (content_sha256, model_id, call_shape) DO NOTHING`,
|
||||
[contentSha, modelStr, shape, JSON.stringify(value)],
|
||||
);
|
||||
|
||||
@@ -356,7 +356,7 @@ class CalibrationProfilePhase extends BaseCyclePhase {
|
||||
active_bias_tags, model_id, cost_usd, judge_model_agreement
|
||||
) VALUES ($1, $2, now(), false,
|
||||
$3, $4, $5, $6, $7,
|
||||
$8::jsonb, $9::text[],
|
||||
$8::text::jsonb, $9::text[],
|
||||
$10, $11,
|
||||
$12::text[], $13, NULL, NULL)`,
|
||||
[
|
||||
|
||||
@@ -171,11 +171,19 @@ async function generateIntraPagePairs(
|
||||
results: SearchResult[],
|
||||
): Promise<ContradictionPair[]> {
|
||||
if (results.length === 0) return [];
|
||||
// Unique page_ids only.
|
||||
const pageIds = Array.from(new Set(results.map((r) => r.page_id)));
|
||||
// Unique, FINITE page_ids only. Defensive backstop for the alias-hop bug
|
||||
// (#2339 sibling): an alias-injected synthetic result with an undefined/NaN
|
||||
// page_id must never reach `ANY($1::int[])` — postgres.js rejects it with
|
||||
// UNDEFINED_VALUE and aborts the whole probe. Mirrors the hybrid.ts:63 filter.
|
||||
const pageIds = Array.from(
|
||||
new Set(
|
||||
results.map((r) => r.page_id).filter((n): n is number => typeof n === 'number' && Number.isFinite(n)),
|
||||
),
|
||||
);
|
||||
const takesByPage = await engine.listActiveTakesForPages(pageIds);
|
||||
const out: ContradictionPair[] = [];
|
||||
for (const r of results) {
|
||||
if (typeof r.page_id !== 'number' || !Number.isFinite(r.page_id)) continue;
|
||||
const takes = takesByPage.get(r.page_id) ?? [];
|
||||
if (takes.length === 0) continue;
|
||||
const chunkMember = searchResultToMember(r);
|
||||
|
||||
@@ -878,7 +878,7 @@ async function runSubagentViaGateway(args: GatewayRunArgs): Promise<SubagentResu
|
||||
const rows = await engine.executeRaw<{ gbrain_tool_use_id: string }>(
|
||||
`INSERT INTO subagent_tool_executions
|
||||
(job_id, message_idx, tool_use_id, tool_name, input, status, schema_version, ordinal, gbrain_tool_use_id, provider_id)
|
||||
VALUES ($1, $2, $3, $4, $5::jsonb, 'pending', 2, $6, $7, $8)
|
||||
VALUES ($1, $2, $3, $4, $5::text::jsonb, 'pending', 2, $6, $7, $8)
|
||||
ON CONFLICT (job_id, message_idx, ordinal) DO UPDATE
|
||||
SET status = subagent_tool_executions.status
|
||||
RETURNING gbrain_tool_use_id::text AS gbrain_tool_use_id`,
|
||||
@@ -891,7 +891,7 @@ async function runSubagentViaGateway(args: GatewayRunArgs): Promise<SubagentResu
|
||||
onToolCallComplete: async (gbrainToolUseId, output) => {
|
||||
await engine.executeRaw(
|
||||
`UPDATE subagent_tool_executions
|
||||
SET status = 'complete', output = $1::jsonb, ended_at = now()
|
||||
SET status = 'complete', output = $1::text::jsonb, ended_at = now()
|
||||
WHERE gbrain_tool_use_id::text = $2`,
|
||||
[JSON.stringify(output ?? null), gbrainToolUseId],
|
||||
);
|
||||
@@ -1107,7 +1107,7 @@ async function persistMessage(engine: BrainEngine, jobId: number, msg: Persisted
|
||||
await engine.executeRaw(
|
||||
`INSERT INTO subagent_messages (job_id, message_idx, role, content_blocks,
|
||||
tokens_in, tokens_out, tokens_cache_read, tokens_cache_create, model)
|
||||
VALUES ($1, $2, $3, $4::jsonb, $5, $6, $7, $8, $9)
|
||||
VALUES ($1, $2, $3, $4::text::jsonb, $5, $6, $7, $8, $9)
|
||||
ON CONFLICT (job_id, message_idx) DO NOTHING`,
|
||||
[
|
||||
jobId,
|
||||
@@ -1131,13 +1131,15 @@ async function persistToolExecPending(
|
||||
toolName: string,
|
||||
input: unknown,
|
||||
): Promise<void> {
|
||||
// Serialize to JSON string for the ::jsonb cast. When `input` is already a
|
||||
// string (e.g. pre-serialized), avoid double-encoding which produces a jsonb
|
||||
// scalar string instead of a jsonb object — breaking `input->>'key'` lookups.
|
||||
// Serialize to a JSON string, then bind through $5::text::jsonb. The value is
|
||||
// ALWAYS a string here (pre-serialized input, or JSON.stringify) — binding a
|
||||
// string to a bare $5::jsonb double-encodes it into a jsonb scalar string under
|
||||
// postgres.js .unsafe() (#2339 class; PGLite hides it). The ::text cast makes
|
||||
// the text→jsonb parse produce a real jsonb object.
|
||||
const jsonStr = typeof input === 'string' ? input : JSON.stringify(input);
|
||||
await engine.executeRaw(
|
||||
`INSERT INTO subagent_tool_executions (job_id, message_idx, tool_use_id, tool_name, input, status)
|
||||
VALUES ($1, $2, $3, $4, $5::jsonb, 'pending')
|
||||
VALUES ($1, $2, $3, $4, $5::text::jsonb, 'pending')
|
||||
ON CONFLICT (job_id, tool_use_id) DO NOTHING`,
|
||||
[jobId, messageIdx, toolUseId, toolName, jsonStr],
|
||||
);
|
||||
@@ -1151,7 +1153,7 @@ async function persistToolExecComplete(
|
||||
): Promise<void> {
|
||||
await engine.executeRaw(
|
||||
`UPDATE subagent_tool_executions
|
||||
SET status = 'complete', output = $3::jsonb, ended_at = now()
|
||||
SET status = 'complete', output = $3::text::jsonb, ended_at = now()
|
||||
WHERE job_id = $1 AND tool_use_id = $2`,
|
||||
[jobId, toolUseId, typeof output === 'string' ? output : JSON.stringify(output)],
|
||||
);
|
||||
@@ -1170,7 +1172,7 @@ async function persistToolExecFailed(
|
||||
// rejected upfront) and "pending row exists" (tool threw mid-execute).
|
||||
await engine.executeRaw(
|
||||
`INSERT INTO subagent_tool_executions (job_id, message_idx, tool_use_id, tool_name, input, status, error, ended_at)
|
||||
VALUES ($1, $2, $3, $4, $5::jsonb, 'failed', $6, now())
|
||||
VALUES ($1, $2, $3, $4, $5::text::jsonb, 'failed', $6, now())
|
||||
ON CONFLICT (job_id, tool_use_id) DO UPDATE
|
||||
SET status = 'failed', error = EXCLUDED.error, ended_at = now()`,
|
||||
[jobId, messageIdx, toolUseId, toolName, typeof input === 'string' ? input : JSON.stringify(input), error],
|
||||
|
||||
@@ -120,7 +120,7 @@ export async function writeImpactLogRow(
|
||||
remediation_id, metric_name, metric_before, metric_after,
|
||||
job_id, source_id, brain_id, started_at, idempotency_key,
|
||||
applied_by, details
|
||||
) VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11::jsonb)`,
|
||||
) VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11::text::jsonb)`,
|
||||
[
|
||||
attribution.remediation_id,
|
||||
metricName,
|
||||
|
||||
@@ -179,15 +179,22 @@ export async function recordCompleted(
|
||||
// REPLACE semantics (kept deliberately — #1794 V3). Callers like
|
||||
// extract-conversation-facts serialize a MUTABLE map through here and rely on
|
||||
// stale keys being REMOVED; an append would make them unremovable. The full
|
||||
// set lands in the parent `completed_keys` JSONB column via a single UPSERT —
|
||||
// exactly as before. JSON.stringify into `$3::jsonb` is correct (the text→jsonb
|
||||
// cast yields a proper array; NOT the double-encode trap, which is the template
|
||||
// form). Sync uses `appendCompleted` (below) instead, never this.
|
||||
// set lands in the parent `completed_keys` JSONB column via a single UPSERT.
|
||||
// #2339: bind through `$3::text::jsonb`, NOT `$3::jsonb`. Under postgres.js
|
||||
// `.unsafe(sql, params)` (executeRawDirect's path) a JS string bound to a
|
||||
// `$N::jsonb` param double-encodes — the text→jsonb cast wraps the already-JSON
|
||||
// string into a jsonb *string scalar*, which fails the v119
|
||||
// `op_checkpoints_completed_keys_array CHECK (jsonb_typeof = 'array')` and aborts
|
||||
// every sync on real Postgres (PGLite parses it silently, which hid the bug).
|
||||
// Casting through `text` first binds it as a plain text param so the text→jsonb
|
||||
// cast parses it into a genuine jsonb array. This is the positional-param form of
|
||||
// the CLAUDE.md double-encode trap (the grep guard only caught the template form).
|
||||
// Sync uses `appendCompleted` (below, `unnest($3::text[])`) instead, never this.
|
||||
const sorted = [...keys].sort();
|
||||
return durableWrite(engine, key, 'write', () =>
|
||||
engine.executeRawDirect(
|
||||
`INSERT INTO op_checkpoints (op, fingerprint, completed_keys, updated_at)
|
||||
VALUES ($1, $2, $3::jsonb, now())
|
||||
VALUES ($1, $2, $3::text::jsonb, now())
|
||||
ON CONFLICT (op, fingerprint) DO UPDATE
|
||||
SET completed_keys = EXCLUDED.completed_keys,
|
||||
updated_at = now()`,
|
||||
|
||||
@@ -647,6 +647,11 @@ export async function applyAliasHop(
|
||||
if (!page) continue;
|
||||
injectScore += 1e-6;
|
||||
out.push({
|
||||
// #2339-sibling: include page_id. The `as SearchResult` cast hid its
|
||||
// absence, so any consumer reading page_id off an alias-injected result got
|
||||
// undefined — e.g. listActiveTakesForPages bound undefined/NaN into
|
||||
// ANY($1::int[]) and crashed the contradiction probe on real Postgres.
|
||||
page_id: page.id,
|
||||
slug: page.slug,
|
||||
title: page.title,
|
||||
type: page.type,
|
||||
|
||||
@@ -236,7 +236,7 @@ export class SemanticQueryCache {
|
||||
// the v0.40.3.0 IRON-RULE).
|
||||
await this.engine.executeRaw(
|
||||
`INSERT INTO query_cache (id, query_text, source_id, knobs_hash, embedding, results, meta, ttl_seconds, page_generations, max_generation_at_store, created_at)
|
||||
VALUES ($1, $2, $3, $4, $5::vector, $6::jsonb, $7::jsonb, $8, $9::jsonb, $10, now())
|
||||
VALUES ($1, $2, $3, $4, $5::vector, $6::text::jsonb, $7::text::jsonb, $8, $9::text::jsonb, $10, now())
|
||||
ON CONFLICT (id) DO UPDATE SET
|
||||
query_text = EXCLUDED.query_text,
|
||||
knobs_hash = EXCLUDED.knobs_hash,
|
||||
|
||||
@@ -408,7 +408,7 @@ export async function addSource(
|
||||
try {
|
||||
await engine.executeRaw(
|
||||
`INSERT INTO sources (id, name, local_path, config)
|
||||
VALUES ($1, $2, $3, $4::jsonb)`,
|
||||
VALUES ($1, $2, $3, $4::text::jsonb)`,
|
||||
[opts.id, displayName, finalPath, JSON.stringify(config)],
|
||||
);
|
||||
} catch (e) {
|
||||
@@ -454,7 +454,7 @@ export async function addSource(
|
||||
const displayName = opts.name ?? opts.id;
|
||||
await engine.executeRaw(
|
||||
`INSERT INTO sources (id, name, local_path, config)
|
||||
VALUES ($1, $2, $3, $4::jsonb)`,
|
||||
VALUES ($1, $2, $3, $4::text::jsonb)`,
|
||||
[opts.id, displayName, finalPath, JSON.stringify(config)],
|
||||
);
|
||||
}
|
||||
|
||||
+16
-9
@@ -79,15 +79,22 @@ function assertSqlValue(value: unknown): asserts value is SqlValue {
|
||||
* auth/admin surface that a focused helper preserves the contract without
|
||||
* forcing every call site to remember which positions hold JSONB.
|
||||
*
|
||||
* Why this is safe vs the v0.12.0 double-encode bug: the bug was specific
|
||||
* to postgres.js's template-tag auto-stringify path interacting with
|
||||
* sql.json() — not to positional binding through `unsafe()`. JS objects
|
||||
* passed as positional params reach the wire protocol with the correct
|
||||
* type oid (jsonb when cast in the SQL string), so there is no double-
|
||||
* encode. The CI guard (scripts/check-jsonb-pattern.sh) doesn't fire
|
||||
* because the source pattern is a method call (`executeRawJsonb(...)`),
|
||||
* not the banned literal-template-tag interpolation pattern with
|
||||
* JSON.stringify cast to jsonb.
|
||||
* Why this is safe vs the double-encode bug: this helper binds a JS **object**
|
||||
* (not a pre-stringified string) to each `$N::jsonb` position. postgres.js
|
||||
* `unsafe()` and PGLite both serialize a JS object to the jsonb wire type
|
||||
* correctly, so there is no double-encode.
|
||||
*
|
||||
* IMPORTANT (the #2339 distinction): positional binding is NOT universally safe.
|
||||
* Binding `JSON.stringify(x)` (a **string**) to a `$N::jsonb` position via
|
||||
* `unsafe()`/`executeRawDirect` DOES double-encode — the text→jsonb cast wraps
|
||||
* the already-JSON string into a jsonb *string scalar* (PGLite hides it; real
|
||||
* Postgres exposes it, and it broke every sync in #2339). The fixes are: pass a
|
||||
* raw object (this helper), or cast through `$N::text::jsonb` so the string is
|
||||
* parsed, never `$N::jsonb` + JSON.stringify. The legacy grep guard
|
||||
* (scripts/check-jsonb-pattern.sh) only caught the template-tag form; the
|
||||
* positional `$N::jsonb` + JSON.stringify form is caught by the AST guard
|
||||
* scripts/check-jsonb-params.mjs. This helper's `executeRawJsonb(...)` method-call
|
||||
* shape trips neither guard because it passes objects, which is correct.
|
||||
*
|
||||
* Usage:
|
||||
* await executeRawJsonb(
|
||||
|
||||
@@ -32,8 +32,8 @@ export async function writeReceiptToDb(engine: BrainEngine, receipt: TakesQualit
|
||||
receipt_json, receipt_disk_path, created_at
|
||||
) VALUES (
|
||||
$1, $2, $3, $4,
|
||||
$5, $6, $7, $8::jsonb, $9,
|
||||
$10::jsonb, $11, $12::timestamptz
|
||||
$5, $6, $7, $8::text::jsonb, $9,
|
||||
$10::text::jsonb, $11, $12::timestamptz
|
||||
)
|
||||
ON CONFLICT (receipt_sha8_corpus, receipt_sha8_prompt, receipt_sha8_models, receipt_sha8_rubric)
|
||||
DO NOTHING`,
|
||||
|
||||
@@ -0,0 +1,83 @@
|
||||
/**
|
||||
* Self-test for scripts/check-jsonb-params.mjs — the positional jsonb
|
||||
* double-encode guard (#2339 / #2324 class). Verifies it catches the bug shape
|
||||
* (including generic-typed calls and the `jsonStr` variable case is acknowledged
|
||||
* as out of scope) and does NOT false-positive on the sanctioned forms.
|
||||
*
|
||||
* Fixtures are written to a temp dir and the scanner is pointed at it via argv,
|
||||
* so this never touches the real src/ tree.
|
||||
*/
|
||||
import { describe, test, expect, beforeAll, afterAll } from 'bun:test';
|
||||
import { mkdtempSync, writeFileSync, mkdirSync, rmSync } from 'node:fs';
|
||||
import { join } from 'node:path';
|
||||
import { tmpdir } from 'node:os';
|
||||
|
||||
const SCRIPT = join(import.meta.dir, '..', 'scripts', 'check-jsonb-params.mjs');
|
||||
|
||||
let root: string;
|
||||
let badDir: string;
|
||||
let goodDir: string;
|
||||
|
||||
function runGuard(dir: string): { code: number; err: string } {
|
||||
const res = Bun.spawnSync([process.execPath, SCRIPT, dir]);
|
||||
return { code: res.exitCode, err: res.stderr.toString() + res.stdout.toString() };
|
||||
}
|
||||
|
||||
beforeAll(() => {
|
||||
root = mkdtempSync(join(tmpdir(), 'jsonb-guard-'));
|
||||
badDir = join(root, 'bad');
|
||||
goodDir = join(root, 'good');
|
||||
mkdirSync(badDir, { recursive: true });
|
||||
mkdirSync(goodDir, { recursive: true });
|
||||
|
||||
// BAD: positional $1::jsonb bound to a JSON.stringify'd value.
|
||||
writeFileSync(
|
||||
join(badDir, 'bad.ts'),
|
||||
"await engine.executeRaw(`INSERT INTO t (a) VALUES ($1::jsonb)`, [JSON.stringify(x)]);\n",
|
||||
);
|
||||
// BAD: generic-typed executeRaw<T>(...) must still be caught.
|
||||
writeFileSync(
|
||||
join(badDir, 'bad_generic.ts'),
|
||||
"await engine.executeRaw<{ id: string }>(`UPDATE t SET a = $2::jsonb WHERE id = $1`, [id, JSON.stringify(x)]);\n",
|
||||
);
|
||||
|
||||
// GOOD: the fix — $1::text::jsonb.
|
||||
writeFileSync(
|
||||
join(goodDir, 'good_text.ts'),
|
||||
"await engine.executeRaw(`INSERT INTO t (a) VALUES ($1::text::jsonb)`, [JSON.stringify(x)]);\n",
|
||||
);
|
||||
// GOOD: text[] array path (the appendCompleted unnest shape).
|
||||
writeFileSync(
|
||||
join(goodDir, 'good_array.ts'),
|
||||
"await engine.executeRaw(`INSERT INTO t (a) SELECT unnest($1::text[])`, [JSON.stringify(arr)]);\n",
|
||||
);
|
||||
// GOOD: executeRawJsonb passes a raw object, not a string — excluded.
|
||||
writeFileSync(
|
||||
join(goodDir, 'good_helper.ts'),
|
||||
"await executeRawJsonb(engine, `INSERT INTO t (a) VALUES ($1::jsonb)`, [], [JSON.stringify(x)]);\n",
|
||||
);
|
||||
// GOOD: explicit opt-out for a rare legitimate object-binding case.
|
||||
writeFileSync(
|
||||
join(goodDir, 'good_optout.ts'),
|
||||
"await engine.executeRaw(`INSERT INTO t (a) VALUES ($1::jsonb)` /* jsonb-guard-ok */, [JSON.stringify(x)]);\n",
|
||||
);
|
||||
});
|
||||
|
||||
afterAll(() => {
|
||||
rmSync(root, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
describe('check-jsonb-params guard', () => {
|
||||
test('flags positional $N::jsonb + JSON.stringify (incl. generic-typed calls)', () => {
|
||||
const { code, err } = runGuard(badDir);
|
||||
expect(code).toBe(1);
|
||||
expect(err).toContain('bad.ts');
|
||||
expect(err).toContain('bad_generic.ts');
|
||||
});
|
||||
|
||||
test('passes the sanctioned forms (::text::jsonb, ::text[], executeRawJsonb, opt-out)', () => {
|
||||
const { code, err } = runGuard(goodDir);
|
||||
expect(code).toBe(0);
|
||||
expect(err).toContain('clean');
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,80 @@
|
||||
/**
|
||||
* E2E: op_checkpoints.completed_keys JSONB parity — #2339 regression guard.
|
||||
*
|
||||
* #2339: `recordCompleted` bound `JSON.stringify(array)` to a `$3::jsonb` param
|
||||
* via postgres.js `.unsafe()` (executeRawDirect). That double-encodes the value
|
||||
* into a jsonb *string scalar*, which violates the v119
|
||||
* `op_checkpoints_completed_keys_array CHECK (jsonb_typeof = 'array')` and aborts
|
||||
* EVERY sync on real Postgres at the first checkpoint write. PGLite's driver
|
||||
* parses the string silently, which is exactly why the unit suite stayed green
|
||||
* and the bug shipped — so this assertion can ONLY be made on real Postgres.
|
||||
*
|
||||
* This file uses the standard `hasDatabase()` skip gate (consistent with the
|
||||
* other e2e tests). The X2-A guarantee that it actually RUNS lives in a dedicated
|
||||
* CI job (.github/workflows) that provisions a Postgres service so DATABASE_URL
|
||||
* is always present there — rather than a fail-on-skip hack inside this file,
|
||||
* which would red-fail legitimate DB-less local runs.
|
||||
*
|
||||
* Fix under test: `$3::text::jsonb` binds the value as text, so the text→jsonb
|
||||
* cast parses it into a genuine jsonb array.
|
||||
*/
|
||||
import { describe, test, expect, beforeAll, afterAll } from 'bun:test';
|
||||
import { hasDatabase, setupDB, teardownDB, getEngine, getConn } from './helpers.ts';
|
||||
import { recordCompleted, loadOpCheckpoint } from '../../src/core/op-checkpoint.ts';
|
||||
|
||||
const describeE2E = hasDatabase() ? describe : describe.skip;
|
||||
|
||||
describeE2E('E2E: op_checkpoints completed_keys jsonb parity (#2339)', () => {
|
||||
beforeAll(async () => {
|
||||
await setupDB();
|
||||
});
|
||||
afterAll(async () => {
|
||||
await teardownDB();
|
||||
});
|
||||
|
||||
const key = { op: 'sync-target', fingerprint: 'jsonb-parity-2339' };
|
||||
|
||||
test('recordCompleted stores completed_keys as a jsonb ARRAY, not a double-encoded scalar', async () => {
|
||||
const engine = getEngine();
|
||||
|
||||
// Pre-fix this throws SQLSTATE 23514 (the CHECK), durableWrite exhausts its
|
||||
// retries, and recordCompleted returns false. Post-fix it stores a real array.
|
||||
const ok = await recordCompleted(engine, key, ['b/2.md', 'a/1.md', 'c/3.md']);
|
||||
expect(ok).toBe(true);
|
||||
|
||||
const sql = getConn();
|
||||
const [row] = await sql`
|
||||
SELECT jsonb_typeof(completed_keys) AS t,
|
||||
jsonb_array_length(completed_keys) AS len,
|
||||
completed_keys ->> 0 AS first
|
||||
FROM op_checkpoints
|
||||
WHERE op = ${key.op} AND fingerprint = ${key.fingerprint}
|
||||
`;
|
||||
expect(row.t).toBe('array'); // pre-fix: 'string' (scalar) — the bug
|
||||
expect(Number(row.len)).toBe(3);
|
||||
expect(row.first).toBe('a/1.md'); // recordCompleted sorts the set
|
||||
}, 30_000);
|
||||
|
||||
test('loadOpCheckpoint round-trips the recorded set', async () => {
|
||||
const engine = getEngine();
|
||||
const got = await loadOpCheckpoint(engine, key);
|
||||
expect(new Set(got)).toEqual(new Set(['a/1.md', 'b/2.md', 'c/3.md']));
|
||||
}, 30_000);
|
||||
|
||||
test('REPLACE semantics: re-recording a smaller set drops stale keys (stays an array)', async () => {
|
||||
const engine = getEngine();
|
||||
const ok = await recordCompleted(engine, key, ['only/1.md']);
|
||||
expect(ok).toBe(true);
|
||||
|
||||
const got = await loadOpCheckpoint(engine, key);
|
||||
expect(new Set(got)).toEqual(new Set(['only/1.md']));
|
||||
|
||||
const sql = getConn();
|
||||
const [row] = await sql`
|
||||
SELECT jsonb_typeof(completed_keys) AS t
|
||||
FROM op_checkpoints
|
||||
WHERE op = ${key.op} AND fingerprint = ${key.fingerprint}
|
||||
`;
|
||||
expect(row.t).toBe('array');
|
||||
}, 30_000);
|
||||
});
|
||||
Reference in New Issue
Block a user