* fix(sources): recover corrupted config shapes (#3401)
Use one canonical normalizer for nested string and array-shaped source configs across federation reads, config writes, archive/restore, and doctor remediation.\n\nFixes #3401\nFixes #3402\nFixes #3403
Signed-off-by: arisgysel-design <arisgysel-design@users.noreply.github.com>
* fix(sources): bind restoreSource federated patch via ::text::jsonb (#2339 class)
restoreSource bound a JS JSON string to a bare $1::jsonb placeholder;
postgres.js double-encodes that into a jsonb string scalar, so on the
Postgres engine the coerced object || string-scalar concat evaluates as
array-concat and restore RE-CORRUPTS the exact config shape this PR
repairs. PGLite masks the bug (its driver parses the bind natively).
Fix: bind through $1::text::jsonb per the repo JSONB rule.
Adds the DATABASE_URL-gated Postgres regression
(test/e2e/restore-source-config-jsonb-postgres.test.ts): seeds a
corrupted string-scalar config, runs archive -> restore, asserts
jsonb_typeof(config) = 'object' with the federated flag applied and
pre-existing keys preserved. Verified red on the bare ::jsonb bind
(config became a jsonb array) and green on the fix against a real
pgvector Postgres; skips cleanly without DATABASE_URL.
Co-authored-by: arisgysel-design <arisgysel-design@users.noreply.github.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Signed-off-by: arisgysel-design <arisgysel-design@users.noreply.github.com>
Co-authored-by: arisgysel-design <arisgysel-design@users.noreply.github.com>
Co-authored-by: Garry Tan <garrytan@gmail.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
`sources.config` is a jsonb OBJECT column, but a read→write cycle that
JSON.stringify'd an already-stringified value re-wrapped it into a JSON string
scalar ("{}", "\"{}\"", ...) that grew one layer per write. parseSourceConfig
only unwrapped one layer, so the corruption never healed and federation/ACL
reads saw a string instead of the settings object.
- Add normalizeSourceConfig: a bounded (10-iteration) loop that JSON.parses
while the value is a string and returns {} (with a console.warn) when the
result is not a plain object. All six `UPDATE sources SET config` writers run
their config through it before stringify, converging the stored value back to
a jsonb object on the next write.
- parseSourceConfig now does the same bounded unwrap and warns once when more
than one layer was found (one layer is the normal PGLite path).
- Add a `source_config_shape` doctor check that flags any sources row where
jsonb_typeof(config) <> 'object', with the repair path.
- Unit-test the helper (object passthrough, 1-layer, 5-layer nested, garbage
and over-bound inputs) and the doctor check (mock engine).
Co-authored-by: 1alessio <alessio.sulpizi@gmail.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
`sources.config` is a jsonb OBJECT column, but a read→write cycle that
JSON.stringify'd an already-stringified value re-wrapped it into a JSON string
scalar ("{}", "\"{}\"", ...) that grew one layer per write. parseSourceConfig
only unwrapped one layer, so the corruption never healed and federation/ACL
reads saw a string instead of the settings object.
- Add normalizeSourceConfig: a bounded (10-iteration) loop that JSON.parses
while the value is a string and returns {} (with a console.warn) when the
result is not a plain object. All six `UPDATE sources SET config` writers run
their config through it before stringify, converging the stored value back to
a jsonb object on the next write.
- parseSourceConfig now does the same bounded unwrap and warns once when more
than one layer was found (one layer is the normal PGLite path).
- Add a `source_config_shape` doctor check that flags any sources row where
jsonb_typeof(config) <> 'object', with the repair path.
- Unit-test the helper (object passthrough, 1-layer, 5-layer nested, garbage
and over-bound inputs) and the doctor check (mock engine).
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* wip: federated sync v2 pre-merge snapshot
* v0.40.5.0 Federated Sync v2 — parallel source sync + push triggers + per-source health
Bump VERSION + package.json + CHANGELOG header + migration walkthrough filename
to v0.40.5.0 (claiming the next free slot in the v0.40.x patch series after
master's v0.40.1.0).
What ships (6 components, all behind sync.federated_v2 feature flag default-on):
1. Per-source sync lock — syncLockId(sourceId), phantom-redirect parity
2. Parallel sync --all — pMapAllSettled fan-out, --max-sources N cap
3. embed-backfill minion handler — D2 per-source lock + D6 $10/job budget + D15.1
fire-and-forget submission + D19 source-level cooldown + 24h $25 rolling cap
4. sync trigger CLI + POST /webhooks/github — HMAC-verified (60 req/min/IP),
X-GitHub-Event=push + ref filter against tracked_branch
5. sources status + federation_health doctor — batched GROUP BY pipeline
(4 queries instead of 6×N per-source roundtrips)
6. sources federate/unfederate hook — auto-submit embed-backfill on flip
Correctness fixes (unconditional):
- D21: sync.ts:959 facts backstop now passes sourceId to engine.getPage
- D15.4: redactSourceConfig + CI guard prevent webhook_secret leak
- D15.5: safeHexEqual extracted to src/core/timing-safe.ts
Schema:
- Migration v89 (sources_github_repo_index): partial expression index on
config->>'github_repo' for fast webhook source-lookup
Tests:
- 14 new test files, 112 cases. 4 IRON-RULE regressions pinned (SYNC_LOCK_ID
back-compat, phantom per-source lock, embed-backfill kill+resume,
webhook HMAC prefix-strip). All 9449 unit tests pass.
Caught at test-write time: the webhook handler had a Buffer.from('sha256=...',
'hex') truncation bug — without the prefix-strip, every signature would have
"matched" empty buffers. Pinned by a test/sources-webhook.test.ts IRON-RULE.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(check-source-config-leak): tighten regex to source-row patterns only
The v0.40.5.0 wave added scripts/check-source-config-leak.sh with a
too-broad pattern (JSON\.stringify\(.*config) that flagged any variable
named 'config' — catching the GLOBAL gbrain config.json serializers in
src/commands/init.ts (status envelopes) and src/core/config.ts (the
config-file write site). On the CI runner without rg installed, the
grep -rE fallback fired correctly and produced 4 false positives that
broke the `verify` script.
Tightened the patterns to specifically match `(source|src|row|s).config`
property access — the actual risk shape (a sources-table row being
serialized whole). The global gbrain config has a different shape and
threat model (file-mode 0o600 at the write site), so it's safe to
exempt at the regex level rather than per-file whitelist.
Also fixed a latent bug: the rg branch used `--include='*.ts'` (grep's
flag, not rg's). rg silently rejected it and CANDIDATES came back empty,
so the local-dev runs (which have rg) would never have caught a real
leak. Now branches on tool availability: `-g '*.ts'` for rg, `--include`
for grep -rE. Both branches verified against a synthetic leak fixture.
Also added init.ts + config.ts to the whitelist as a belt-and-suspenders
since they handle gbrain-global config (not source rows) and could
otherwise reflect-back via regex iteration.
CI: `bun run verify` exit 0 locally with both the original false-positive
fixture (clean repo) and a synthetic leak fixture (correctly caught,
exit 1).
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
