diff --git a/README.md b/README.md index 9807cadb8..1f87d5883 100644 --- a/README.md +++ b/README.md @@ -258,6 +258,24 @@ The whole loop is described in [`docs/architecture/topologies.md`](docs/architec **Job queue (Minions).** BullMQ-shaped, Postgres-native job queue. Durable subagents (LLM tool loops that survive crashes via two-phase pending→done persistence), shell jobs with audit, child jobs with cascading timeouts, rate leases for outbound providers, attachments via S3/Supabase storage. Replaces "spawn subagent as fire-and-forget Promise" with something that recovers from anything. +**Non-English brains (FTS language config).** The Postgres full-text search tokenizer is configurable via `GBRAIN_FTS_LANGUAGE`. Defaults to `english`. Set it to any text-search configuration that exists in your Postgres instance: + +```bash +export GBRAIN_FTS_LANGUAGE=portuguese # uses built-in portuguese stemmer +export GBRAIN_FTS_LANGUAGE=spanish # built-in spanish stemmer +export GBRAIN_FTS_LANGUAGE=pt_br # custom config (e.g. unaccent + portuguese) +``` + +List available configs: `psql -c "SELECT cfgname FROM pg_ts_config"`. Both the **query side** (`websearch_to_tsquery`) and the **write side** (the trigger functions that populate `pages.search_vector` and `content_chunks.search_vector`) honor `GBRAIN_FTS_LANGUAGE`. On first install (or upgrade), the `configurable_fts_language` schema migration reads the env var and creates trigger functions in the configured language; subsequent inserts/updates tokenize using that setting. To change language on a brain that has already run the migration, use the dedicated CLI command: + +```bash +export GBRAIN_FTS_LANGUAGE=portuguese +gbrain reindex-search-vector --dry-run # preview row counts +gbrain reindex-search-vector --yes # recreate triggers + backfill +``` + +The command is idempotent (re-running with the same language is a no-op for vector content) and uses the same recreate-and-backfill primitives as the migration. For accent-insensitive Portuguese (`pt_br`), see [docs/guides/multi-language-fts.md](docs/guides/multi-language-fts.md) for the `unaccent` + portuguese stemmer recipe. + **43 curated skills.** Routing lives in [`skills/RESOLVER.md`](skills/RESOLVER.md). Covers signal capture, ingest (idea / media / meeting), enrichment, querying, brain ops, citation fixing, daily task management, cron scheduling, reports, voice, soul audit, skill creation, eval framework, and migrations. Skills are markdown files (tool-agnostic), packaged as a single skillpack the installer drops into your agent workspace. **Eval framework.** `gbrain eval longmemeval` runs the public [LongMemEval](https://huggingface.co/datasets/xiaowu0162/longmemeval) benchmark against your hybrid retrieval. `gbrain eval export` + `gbrain eval replay` capture real queries and replay them against code changes (set `GBRAIN_CONTRIBUTOR_MODE=1`). `gbrain eval cross-modal` cross-checks an output against the task using three different-provider frontier models. `gbrain eval retrieval-quality` runs NamedThingBench, which hard-gates the named-thing retrieval families (title-substring, alias-synonym, generic-to-named, multi-chunk-dilution) so a regression in "find the page this query names" fails CI loudly. Full methodology in [`docs/eval/SEARCH_MODE_METHODOLOGY.md`](docs/eval/SEARCH_MODE_METHODOLOGY.md). diff --git a/docs/ENGINES.md b/docs/ENGINES.md index 38e35b3f7..257a6e7cd 100644 --- a/docs/ENGINES.md +++ b/docs/ENGINES.md @@ -148,6 +148,51 @@ RRF fusion, multi-query expansion, and 4-layer dedup are engine-agnostic. They o **Why not self-hosted for v0:** The brain should be infrastructure agents use, not something you maintain. Self-hosted Postgres with Docker is a welcome community PR, but v0 optimizes for zero ops. +### Opt-in RLS source-scope binding (`GBRAIN_RLS_SCOPE_BINDING`) + +Defense-in-depth layer for Postgres deployments that want the database itself +to enforce source isolation, in addition to the mandatory app-layer filters +(`sourceScopeOpts` — layer 1, always on). + +**Mechanism.** With `GBRAIN_RLS_SCOPE_BINDING=1` (or `true`), the engine's +source-scoped read methods wrap their queries in a transaction that first runs +`SELECT set_config('app.scopes', $1, true)` — the value is a bound parameter +(federated `sourceIds` CSV > scalar `sourceId` > `'*'` for unscoped internal +reads), transaction-local (equivalent to `SET LOCAL`, which itself can't take +bound params). An RLS policy can then filter rows by +`current_setting('app.scopes', true)`. + +**Default off.** With the env var unset, reads call through on the shared pool +exactly as before — no per-read transaction, no pool-slot hold (the search +methods keep the transaction they always had for their `SET LOCAL +statement_timeout`). Existing operators see zero behavior change. + +**Enabling it** (operator-managed SQL; gbrain ships no DDL for this): + +```sql +ALTER TABLE pages ENABLE ROW LEVEL SECURITY; +CREATE POLICY pages_scope_filter ON pages + USING (current_setting('app.scopes', true) = '*' + OR source_id = ANY(string_to_array(current_setting('app.scopes', true), ','))); + +-- Required: connections that don't run through the scoped read helper +-- (admin, autopilot, cycle, writes) must default to unscoped, or they +-- see zero rows once the policy exists: +ALTER ROLE SET app.scopes = '*'; + +-- If the runtime role OWNS the table, RLS is skipped for it unless forced: +ALTER TABLE pages FORCE ROW LEVEL SECURITY; +``` + +Safe to enable in either order: the env var without a policy is a no-op +setting; a policy without the env var is enforced only via the role default. + +**Honest caveat:** only read paths routed through the scoped helper carry a +per-request scope binding — unwrapped paths (writes, admin/maintenance reads) +run under the role default and are not backstopped per caller. This is layer 2; +the app-layer source filters remain layer 1 and stay mandatory. Behavioral pins +live in `test/postgres-engine-rls-scope.test.ts`. + ## PGLiteEngine (v0.7, ships) **Dependencies:** `@electric-sql/pglite` (v0.4.4+) diff --git a/docs/architecture/KEY_FILES.md b/docs/architecture/KEY_FILES.md index e4651b99c..a153c2b43 100644 --- a/docs/architecture/KEY_FILES.md +++ b/docs/architecture/KEY_FILES.md @@ -36,13 +36,15 @@ per-release `**vX.Y.Z:**` narration — CI enforces this - `src/core/post-upgrade-reembed.ts` — Pure functions backing the `gbrain upgrade` chunker-bump cost prompt. `computeReembedEstimate(engine, model)` queries real SQL (`COUNT(*)` + `COALESCE(SUM(LENGTH(compiled_truth)) + SUM(LENGTH(timeline)), 0)`) on `pages WHERE chunker_version < MARKDOWN_CHUNKER_VERSION`. `formatReembedPrompt(est, graceSeconds)` is the stderr-line formatter. `runPostUpgradeReembedPrompt(engine, model, opts)` orchestrates the 10-second Ctrl-C window; TTY-only wait (non-TTY auto-proceeds for CI / cron); `GBRAIN_NO_REEMBED=1` bails with a doctor-warning marker; `GBRAIN_REEMBED_GRACE_SECONDS=0` skips the wait. - `src/commands/reindex.ts` — `gbrain reindex --markdown [--limit N] [--dry-run] [--json] [--no-embed] [--repo PATH]`. Walks `pages WHERE page_kind = 'markdown' AND chunker_version < MARKDOWN_CHUNKER_VERSION` in 100-row batches ordered by id. Rows with non-null `source_path` re-import via `importFromFile`; rows without fall back to `importFromContent`. Both paths pass `forceRechunk: true` to bypass `importFromContent`'s `content_hash` short-circuit — without it the chunker version bump never reaches pages whose source content hasn't changed, AND the stripFactsFence privacy strip never applies to pre-strip chunks. Idempotent — partial-completion re-runs pick up via id-ordered batches. Wired into `src/commands/upgrade.ts:runPostUpgrade` after `apply-migrations`. The DB-only fallback (no source file on disk) does NOT pass body-only `compiled_truth` to `importFromContent` (that path re-parses with EMPTY frontmatter and OVERWRITES the page's real frontmatter/title/timeline); it `getPage`+`getTags`, reconstructs FULL markdown via `serializeMarkdown(frontmatter, compiled_truth, timeline, {type, title, tags})`, and re-imports THAT so re-chunking a DB-only page preserves everything while bumping `chunker_version`. Pinned by `test/reindex-preserve-tags.test.ts`. - `src/commands/reindex-code.ts` — `gbrain reindex --code [--source ID] [--dry-run] [--yes] [--json] [--force] [--no-embed]`. Walks `pages WHERE type = 'code'` in 100-row batches, replays through `importCodeFile` for chunk + embed + content_hash folding. Idempotent unless `--force` bypasses the content_hash early-return. Cost-preview model field reads `getEmbeddingModelName()` from the gateway so preview reflects what the gateway will actually embed with. An informational stderr nudge inside `runReindexCode` (so dry-run + execute both surface it): when the configured embedding model isn't code-tuned (allowlist `{'voyage-code-3'}`, case-insensitive bare match), prints a recommendation to switch to `voyage:voyage-code-3`; suppress with `GBRAIN_NO_CODE_MODEL_NUDGE=1`, `--no-embed`, or `--json`. Pure `shouldNudgeCodeModel(bareName)` returns a tagged `NudgeDecision` union (takes the bare model name, emits qualified `voyage:voyage-code-3` for the paste-ready `gbrain config set` line). When `--yes` is absent and the caller is non-TTY or passed `--json`, the cost gate refuses (exit 2, no spend) via the pure exported `buildCostRefusal({json, previewMsg, preview, costUsd, model}): {stdout?, stderr?}` — JSON envelope only when `--json` is explicit, otherwise a human refusal on stderr (the spend guardrail is independent of the output format). `spend.posture=tokenmax` OR an explicit `--max-cost off`/`unlimited` makes the gate informational and proceeds (#2139); `--max-cost off` also disables the runtime BudgetTracker cap. Pinned by `test/ai/voyage-code-3-recipe.test.ts`, `test/reindex-code-nudge.serial.test.ts`, `test/reindex-code-model-source.serial.test.ts` (IRON-RULE regression for the cost-preview fix), `test/reindex-cost-refusal.test.ts`. +- `src/core/fts-language.ts` — Single source for the Postgres text-search configuration name used by FTS. `getFtsLanguage()` resolves `GBRAIN_FTS_LANGUAGE` (default `english`), validates against `/^[a-z][a-z0-9_]*$/` (tsvector config names can't be bound as parameters, so the value is interpolated into raw SQL — the allowlist regex is the injection guard; invalid values warn once and fall back to `english`), and caches on first read (`resetFtsLanguageCache()` is test-only). Consumed by both engines' `searchKeyword`/`searchKeywordChunks` (`websearch_to_tsquery` query side), the `configurable_fts_language` migration, and `reindex-search-vector` (write-side trigger functions). Pinned by `test/fts-language.serial.test.ts` + `test/fts-language-migration.serial.test.ts` (includes the `'; DROP TABLE pages; --` injection cases). +- `src/commands/reindex-search-vector.ts` — `gbrain reindex-search-vector [--dry-run] [--yes] [--json]`. Escape hatch for changing `GBRAIN_FTS_LANGUAGE` after the `configurable_fts_language` migration has run (the migration shows applied and is skipped): recreates `update_page_search_vector` + `update_chunk_search_vector` with the configured language — bodies mirror the migration's and KEEP the `SET search_path = pg_catalog, public` hardening (CREATE OR REPLACE resets proconfig) — then backfills `pages` (UPDATE-to-self re-fires the trigger) and `content_chunks` (direct vector recompute) in id-keyset batches of `BACKFILL_BATCH_SIZE` (5000) via `UPDATE … WHERE id IN (SELECT … LIMIT n) RETURNING id`, streaming phases `reindex_search_vector.pages`/`.chunks` through the shared progress reporter (stderr). Confirmation gate: `--yes`, or an interactive TTY [y/N]; `--json` does NOT bypass the gate (non-TTY without `--yes` refuses with a ConfirmationRequired envelope, exit 2). Idempotent. Pinned by `test/reindex-search-vector.serial.test.ts`. - `src/commands/sync.ts:resolveSlugByPathOrSourcePath` — Resolves a slug by `pages.source_path` first (returns the stored slug for frontmatter-fallback pages whose path doesn't derive a slug), then falls back to `resolveSlugForPath(path)`. Threaded into all 4 delete/rename call sites (`performSync`'s un-syncable cleanup at ~:531, deletes at ~:603, rename oldSlug at ~:622). Without this, emoji-only / Thai / Arabic filenames whose slug came from frontmatter would orphan on delete/rename (the delete path would compute the wrong path-derived slug). Best-effort query — pre-migration brains fall through to the legacy path. - `src/core/sources-ops.ts` — Multi-source registration + clone-lifecycle ops (`addSource`, `recloneIfMissing`, `defaultCloneDir`, `isOwnedClone`, `unownedHint`). **Reclone-ownership invariant (must-never-violate): gbrain may only delete/re-clone a clone it created, NEVER a user working tree.** `recloneIfMissing` deletes `local_path`, so it gates on `isOwnedClone(src)` and throws a `SourceOpError('unmanaged_path', ...)` BEFORE any filesystem op when ownership is unprovable — fail-closed. Ownership is proven by `config.managed_clone === true` (written by `addSource`'s `--url` path, covering default-location and `--clone-dir` clones) OR `local_path === defaultCloneDir(id)` (back-compat for pre-marker clones, via exact normalized-path equality, symlink-free). A row with `remote_url` + an unowned `local_path` (a user-registered working tree, e.g. `sources add --path`) is refused untouched; re-add with `--url` to regain auto-reclone. The reclone is EXDEV-safe: clone into a SIBLING temp of `local_path` (not the shared `clones/.tmp`, which may sit on a different mount than a `--clone-dir` target), then swap (move old aside → move new in → drop old) so `local_path` is never left missing-and-unrecoverable; on swap failure the original is restored, and if restore fails the error names the `aside` path so it's never reflexively deleted. A TOCTOU re-check re-confirms ownership immediately before the destructive move and rejects a symlink leaf swapped in after the entry check (`symlink_escape`). `unownedHint(src, state)` is the shared recovery message used by both the core error and the `gbrain sync --source` CLI error; `gbrain sources restore` special-cases `unmanaged_path` to print "DB row restored; gbrain syncs this path read-only" instead of the misleading "try sync to recover" guidance. `SourceOpErrorCode` includes `unmanaged_path`. Pinned by `test/sources-ops.test.ts`, `test/sources-resync-recovery.test.ts`. - `src/core/utils.ts` — Shared SQL utilities extracted from postgres-engine.ts. Exports `parseEmbedding(value)` (throws on unknown input, used by migration + ingest paths where data integrity matters) and `tryParseEmbedding(value)` (returns `null` + warns once per process, used by search/rescore paths where availability matters more than strictness). `isUndefinedColumnError(err)` predicate — pattern-matches Postgres SQLSTATE 42703 / "column ... does not exist" with engine-driver shape variation tolerated; replaces bare `catch {}` blocks in `oauth-provider.ts` so genuine errors (lock timeout, network blip, permission denied) propagate while column-missing falls through to the legacy fallback. `validateSourceId(id)` throws on anything outside `^[a-z0-9_-]+$`, used by the per-source disk-layout fix in patterns.ts/synthesize.ts before any `join(brainDir, '.sources', source_id, slug+'.md')` so source_id can't traverse out of brainDir. `rowToPage` populates the required `Page.source_id` from the SELECT projection (`scripts/check-source-id-projection.sh` enforces every projection feeding `rowToPage` includes the column). - `src/core/db.ts` — Connection management, schema initialization. `resolveSessionTimeouts()` returns `statement_timeout` + `idle_in_transaction_session_timeout` (defaults 5min each, env-overridable via `GBRAIN_STATEMENT_TIMEOUT`/`GBRAIN_IDLE_TX_TIMEOUT`/`GBRAIN_CLIENT_CHECK_INTERVAL`). Both `connect()` (module singleton) and `PostgresEngine.connect()` (worker pool) consume the result via postgres.js's `connection` option, sending GUCs as startup parameters that survive PgBouncer transaction mode (`setSessionDefaults` kept as a back-compat no-op shim). `connect()` returns `Promise` — `true` iff THIS call created the module singleton, `false` if it joined an existing one; the decision is atomic (no `await` between the `if (sql)` null-check and the synchronous `sql = postgres(...)` assignment), so two concurrent module connects can't both claim creation. `PostgresEngine` stores the return as its `_ownsModuleSingleton` token and only the creating engine may `db.disconnect()` the singleton — a borrower probe engine (lint/doctor config-lift) no-ops its disconnect, so its teardown can't null the connection the long-lived cycle owner is still using (the dream-cycle "connect() has not been called" failure). The module `sql` is only ever nulled by `db.disconnect()` (postgres.js auto-reconnects its own internal pool and never touches our reference). `disconnect()` snapshots + nulls `sql` before awaiting the pool end so a concurrent connect can't join a pool that's already closing. The end routes through `endPoolBounded(pool)` (#1972) — a gbrain-owned `Promise.race` of `pool.end({ timeout: POOL_END_TIMEOUT_SECONDS })` against a hard timer — so a PgBouncer transaction-mode drain that never settles can't hang teardown — the #2084 contract (finishCliTeardown's computed-deadline backstop + flushThenExit's fence-and-grace exit in cli-force-exit.ts) bounds it and delivers pending stdout before exit. `connection-manager.ts` ends its direct + read pools concurrently through the same helper so the per-pool bounds don't stack. - `src/commands/migrate-engine.ts` — Bidirectional engine migration (`gbrain migrate --to supabase/pglite`). Copies the complete source catalog FIRST (`copyMigrationSources` — every `sources` row incl. archived rows and sync/routing metadata, `ON CONFLICT (id) DO UPDATE`, `default` ordered first) so every page write has a valid `pages.source_id` FK parent and the target preserves per-source behavior; pages copy afterward, tracked in the resume manifest by composite `(source_id, slug)` key. The resume manifest is target-aware: `migrationTargetId(config)` hashes `(engine, locator)` (`database_url` for Postgres, resolved `database_path` for PGLite) and `manifestMatchesTarget` requires `schema_version === 2` plus a matching `target_id` — a legacy engine-only manifest, or one from a DIFFERENT target of the same engine kind, starts fresh instead of skipping "completed" pages the new target never received. Pinned by `test/migrate-engine-resume.test.ts` (manifest identity) + `test/e2e/migrate-engine-sources-postgres.test.ts` (source catalog lands before overlapping-slug pages, PGLite → real Postgres). - `src/core/import-file.ts` — importFromFile + importFromContent (chunk + embed + tags). `importFromContent` and `importCodeFile` stamp `pages.embedding_signature` via `setPageEmbeddingSignature(slug, {sourceId, signature: currentEmbeddingSignature()})` when the import actually embedded (not `--no-embed`) so a model/dims swap is detectable as stale; `importCodeFile` only stamps when every chunk was freshly embedded this call (`needsEmbedIndexes.length === chunks.length`), mixed reuse-by-hash pages stay unstamped (`reindex --code --force` / `embed --stale` handle those). `importFromContent`'s tag reconciliation is ADD-ONLY: it only `addTag` (idempotent, ON CONFLICT DO NOTHING). The `tags` table has no provenance column and frontmatter tags are stripped from stored `pages.frontmatter` (markdown.ts:118), so a frontmatter-origin tag can't be distinguished from a DB-enrichment tag (auto-tag / dream synthesize / signal-detector) at re-import — deletion is unsafe (would wipe enrichment under `gbrain reindex --markdown`). Accepted trade-off: removing a tag from frontmatter no longer removes it from the DB on next sync (needs a `tag_source` provenance column). Pinned by `test/reindex-preserve-tags.test.ts` + `test/import-file.test.ts`. -- `src/core/sync.ts` — Pure sync functions (manifest parsing, filtering, slug conversion). Exported `pruneDir(name: string): boolean` is the single source of truth for descent-time directory exclusion across walkers — blocks `node_modules` (no leading dot, so naive walkers slipped through and inflated MISSING_OPEN counts via vendor packages), dot-prefix dirs, `ops/`, and `*.raw` sidecars; `isSyncable` applies it per path segment, and `walkMarkdownFiles` in `src/commands/extract.ts` + `listTextFiles` in `src/core/cycle/transcript-discovery.ts` consult it BEFORE recursing to save the IO of walking thousands of vendor files (closes #923 + #202). `manageGitignore` worktree discriminator matches the gitdir path segment (`/modules/` = submodule, `/worktrees/` = worktree, per Git's documented layout) so Conductor worktrees (first-class repos) get `.gitignore` management for storage-tiering (closes #889). The sync-failure ledger (failure store, error classifier, the shared bookmark gate, and the doctor severity rule) lives in `src/core/sync-failure-ledger.ts`; `sync.ts` re-exports `classifyErrorCode`, `summarizeFailuresByCode`, `loadSyncFailures`, `unacknowledgedSyncFailures`, `acknowledgeSyncFailures`, `recordSyncFailures`, `decideSyncFailureSeverity`, `applySyncFailureGate`, and the `SyncFailure` type for backward-compatible imports — see its entry below. +- `src/core/sync.ts` — Pure sync functions (manifest parsing, filtering, slug conversion). Exported `pruneDir(name: string): boolean` is the single source of truth for descent-time directory exclusion across walkers — blocks `node_modules` (no leading dot, so naive walkers slipped through and inflated MISSING_OPEN counts via vendor packages), `vendor`/`dist`/`build`/`venv`, dot-prefix dirs, and `*.raw` sidecars — NOT `ops/`, which is ordinary user content (#2404; the bundled daily-task-manager stores `ops/tasks` there); `isSyncable` applies it per path segment, and `walkMarkdownFiles` in `src/commands/extract.ts` + `listTextFiles` in `src/core/cycle/transcript-discovery.ts` consult it BEFORE recursing to save the IO of walking thousands of vendor files (closes #923 + #202). `manageGitignore` worktree discriminator matches the gitdir path segment (`/modules/` = submodule, `/worktrees/` = worktree, per Git's documented layout) so Conductor worktrees (first-class repos) get `.gitignore` management for storage-tiering (closes #889). The sync-failure ledger (failure store, error classifier, the shared bookmark gate, and the doctor severity rule) lives in `src/core/sync-failure-ledger.ts`; `sync.ts` re-exports `classifyErrorCode`, `summarizeFailuresByCode`, `loadSyncFailures`, `unacknowledgedSyncFailures`, `acknowledgeSyncFailures`, `recordSyncFailures`, `decideSyncFailureSeverity`, `applySyncFailureGate`, and the `SyncFailure` type for backward-compatible imports — see its entry below. - `src/core/sync-failure-ledger.ts` — the bounded auto-skip sync failure ledger (issue #1939; formerly inline "Bug 9" in `sync.ts`). A LEAF module (imports only fs/path/crypto/config) so `sync.ts` can re-export it without a circular dependency. State lives in `~/.gbrain/sync-failures.jsonl`, one JSON object per line, keyed by `(source_id, path)` with a per-key `attempts` count and a 3-state machine: `open` (fresh/blocking) → `acknowledged` (human resolved via `gbrain sync --skip-failed`) or `auto_skipped` (chronic). `classifyErrorCode(errorMsg)` regex classifier with 12 codes (`SLUG_MISMATCH`, `YAML_PARSE`, `YAML_DUPLICATE_KEY`, `MISSING_OPEN`, `MISSING_CLOSE`, `NESTED_QUOTES`, `EMPTY_FRONTMATTER`, `NULL_BYTES`, `INVALID_UTF8`, `STATEMENT_TIMEOUT`, `FILE_TOO_LARGE`, `SYMLINK_NOT_ALLOWED`) plus `UNKNOWN` (also recognizes `PAGE_JUNK_PATTERN` from the content-sanity gate); `summarizeFailuresByCode(failures)` returns sorted `[{code, count}]`; `MISSING_OPEN`/`MISSING_CLOSE`/`EMPTY_FRONTMATTER` regexes match the `markdown.ts` validator strings, `FILE_TOO_LARGE` covers `import-file.ts:199, 352, 401`, `SYMLINK_NOT_ALLOWED` covers `:347`. All mutations run under `withLedgerLock` (cross-process file lock) with an atomic rename write. The auto-skip threshold resolves via `resolveAutoSkipThreshold()` from `GBRAIN_SYNC_AUTOSKIP_AFTER` (default `DEFAULT_AUTOSKIP_AFTER = 3`; `0` disables the valve = pure fail-closed). Two pure decision functions are the unit-test surface: `decideGateAction({fileFailures, sentinels, attemptsByPath, threshold, skipFailed})` returns `hard_block | block | advance | advance_then_autoskip` (sentinels like `` ALWAYS hard-block, even with `--skip-failed`, so a history rewrite can't auto-skip; any FRESH failure with `attempts < threshold` blocks fail-closed; only when ALL failures are chronic does it `advance_then_autoskip`), and `decideSyncFailureSeverity({entries, nowMs, failHours})` returns the `sync_failures` doctor status (`ok` when zero unresolved; `fail` when ≥10 OPEN-blocking or the oldest OPEN failure has blocked the bookmark past `failHours`; otherwise `warn` — `auto_skipped`-only rows stay WARN-visible regardless of count because the bookmark already advanced). `applySyncFailureGate(input)` is the one orchestrator BOTH sync paths (incremental + full/`runImport`) call: it records/clears ledger rows, runs `decideGateAction`, then executes effects in the crash-safe order (advance the bookmark FIRST via the injected `advance()` callback, THEN auto-skip the chronic set) so a crash can never mark a file skipped while leaving sync wedged. `isSkippablePath` rejects `<…>` sentinels. Pinned by `test/sync-failure-ledger.serial.test.ts` + `test/sync-failures.test.ts`. - `src/core/storage.ts` — Pluggable storage interface (S3, Supabase Storage, local). - `src/core/storage-config.ts` — Storage tiering: `loadStorageConfig` reads `gbrain.yml`, normalizes deprecated keys (`git_tracked`/`supabase_only`) to canonical (`db_tracked`/`db_only`) with once-per-process deprecation warning, and runs `normalizeAndValidateStorageConfig` (auto-fixes missing trailing `/`, throws `StorageConfigError` on tier overlap). Path-segment matcher: `media/x/` does NOT match `media/xerox/foo`. Uses a dedicated parser for the `gbrain.yml` shape rather than gray-matter (broken on delimiter-less YAML). @@ -50,7 +52,7 @@ per-release `**vX.Y.Z:**` narration — CI enforces this - `src/core/git-head.ts` — local git HEAD freshness probe for `gbrain doctor`. `isSourceUnchangedSinceSync(localPath, lastCommit, opts?)` returns true iff `localPath` is a git repo whose current HEAD matches `lastCommit`; when `opts.requireCleanWorkingTree` is true also requires a clean working tree (mirrors `gbrain sync`'s force-walk gate at `sync.ts:1075` so doctor and sync agree on "is there work to do?"). `requireCleanWorkingTree` is `boolean | 'ignore-untracked'` — in `'ignore-untracked'` mode the clean probe runs `git status --porcelain --untracked-files=no` so a quiet repo with stray untracked dirs (`?? companies/`, `?? media/`) is still "unchanged" (sync's incremental path keys off the commit diff and never imports untracked files); `GitCleanProbe` gains an `ignoreUntracked?` second arg. Two probe seams (`_setGitHeadProbeForTests`, `_setGitCleanProbeForTests`) keep unit tests R2-compliant (no `mock.module`). Uses `execFileSync` with array args so shell metachars in `local_path` cannot escape to a shell (the regression test runs real `execFileSync` against `'/nonexistent/$(touch )/repo'` and asserts the sentinel is never created). Fail-open on every error (missing path, not a git repo, git not installed, timeout, NULL inputs, dirty-probe errored → false) preserving the caller's prior time-based behavior. The chunker-version-match check lives in the caller (doctor.ts) because it depends on engine state (`sources.chunker_version` vs `CHUNKER_VERSION` from `src/core/chunkers/code.ts`). Pinned by `test/core/git-head.test.ts` (incl. the shell-injection regression guard). - `src/core/source-health.ts` — per-source health metrics for `gbrain sources status` + doctor's `federation_health`. Commit-relative staleness: `newestCommitMs(localPath)` = HEAD committer time via `git log -1 --format=%ct` (fail-open null; NO working-tree mtime parsing — committed content only, robust against the porcelain-mtime bug farm); pure `lagFromContentMs(contentMs|null, lastSyncMs|null, nowMs)` = remote/column comparator (null lastSync → null; negative wall-clock → skew passthrough; `contentMs <= lastSync` → 0; else/null-content → wall-clock). `computeAllSourceMetrics(engine, sources, {probeContent?})`: LOCAL (`probeContent:true`, `gbrain sources status`) → `isSourceUnchangedSinceSync(..., {requireCleanWorkingTree:'ignore-untracked'}) ? 0 : wall-clock` (live commit-hash catches HEAD moving to an old-dated commit a timestamp compare would miss); REMOTE (default, `federation_health` on the HTTP MCP path) → `lagFromContentMs(row.newest_content_at, ...)`, NO git subprocess (trust boundary). `commitTimeMs(localPath, sha)` is the `newestCommitMs` sibling pinned to an arbitrary commit (committer time via `git show -s --format=%ct `, fail-open null, execFileSync array args) — the resumable sync stamps `newest_content_at` against its pinned target commit, not whatever HEAD raced to. Pinned by `test/source-health.test.ts`. - `src/core/git-remote.ts` — SSRF-hardened git invocations for remote-source `cloneRepo`, `pullRepo`, and `fetchRemote(repoPath, branch)` (the last added for the sync cost-estimator's fetch-first path, #2139, so a cost preview / dry-run fetches through the same hardened flags + `GIT_TERMINAL_PROMPT=0` as real sync rather than a less-protected route). Exports two distinct flag constants because `git`'s argv grammar treats them differently: `GIT_SSRF_FLAGS` (3 `-c` config flags — `protocol.allow=user`, `protocol.file.allow=never`, `http.allowRedirects=false`) is global config, spread BEFORE the subcommand verb; `GIT_SSRF_SUBCOMMAND_FLAGS = ['--no-recurse-submodules']` is subcommand-scoped, spread AFTER the verb (a combined array would spread `--no-recurse-submodules` before the verb where real git rejects it exit 129). `cloneRepo` argv: `git clone --depth=1 [--branch X] -- `. `pullRepo` argv: `git -C pull --ff-only`. Pinned by `test/git-remote.test.ts` position-anchored regression guard (`argv.indexOf('--no-recurse-submodules') > argv.indexOf(verb)`). Also exports the durability-side helpers that power `gbrain sources harden/pull`: `GIT_ENV_AUTH` (the no-prompt env minus the askpass `/bin/false` overrides, so an auth'd push/fetch can consult the repo's configured credential helper while `GIT_TERMINAL_PROMPT=0` still fails fast on a missing credential), `divergenceSafePull(repoPath, branch)` (fetch + `pull --rebase`; returns `skipped_dirty` on a dirty tree, `conflict_aborted` on a rebase conflict after `rebase --abort` so the tree is never left mid-rebase, else `up_to_date`/`advanced`), `detectDefaultBranch` (origin/HEAD → current branch → `main`), `pushProbe(repoPath, branch)` (authenticated `push --dry-run` that proves push access and classifies `auth`/`protected`/`unreachable`), and `isWorkingTreeDirty`. These auth'd paths route their `protocol.file.allow` through `GBRAIN_GIT_ALLOW_FILE_TRANSPORT` (default `never`; set `=1` for self-hosted filesystem remotes), unlike clone/pull which stay strict. -- `src/core/brain-repo-durability.ts` + `src/commands/sources-harden.ts` — brain-repo git durability. `hardenBrainRepo(opts)` makes a brain's working tree durable, idempotently: divergence-safe pull, a LOCAL untracked `.git/hooks/post-commit` auto-push safety net (never committed — a pulled commit can't rewrite executed code next to the token; installed into the active `core.hooksPath` dir and excluded via `.git/info/exclude` when that dir is tracked), a committed `scripts/brain-commit-push.sh` that refuses to exit 0 without a confirmed push (hook + helper render from ONE bash push-retry template — DRY at the TS source, not by the hook sourcing a repo-controlled script), durability rules patched into the active resolver file (`findResolverFile` → RESOLVER.md > AGENTS.md; taxonomy rendered from the bundled `_brain-filing-rules.json`), a minimal DB-free pull cron (launchd/crontab running `gbrain sources pull --path ` so it never opens the PGLite single-writer lock), and a push-probe verify (no heartbeat commit). Credential is REPO-scoped (`acceptPat` from `--pat-file`/`GBRAIN_GITHUB_PAT`, warns on loose perms; reuses an existing repo-local `credential.helper`, else a `0600` store wired via repo-local config); the token is redacted everywhere via `redactSecretsInText` and never enters the repo, remote URL, logs, or `DurabilityReport`. `unhardenBrainRepo` removes the cron/hook/credential wiring (ownership-fingerprinted) and runs before `sources remove`. CLI: `gbrain sources harden ` / `pull |--path ` / `unharden `; auto-harden fires on `sources add --url ... --pat-file` for managed clones (`--no-harden` opts out). `sources pull --path` is dispatched in `src/cli.ts` BEFORE `connectEngine` so the cron stays DB-free. CLI-only (writes executables + an OS cron + a credential helper on the host); never exposed over MCP. Tests: `test/brain-repo-durability.serial.test.ts`, `test/git-remote-durable.serial.test.ts`, `test/brain-durability-hook.serial.test.ts`, `test/durability-cron.test.ts`. +- `src/core/brain-repo-durability.ts` + `src/commands/sources-harden.ts` — brain-repo git durability. `hardenBrainRepo(opts)` makes a brain's working tree durable, idempotently: divergence-safe pull, a LOCAL untracked `.git/hooks/post-commit` auto-push safety net (never committed — a pulled commit can't rewrite executed code next to the token; installed into the active `core.hooksPath` dir and excluded via `.git/info/exclude` when that dir is tracked), a committed `scripts/brain-commit-push.sh` that refuses to exit 0 without a confirmed push and stages+commits BEFORE any pull so a dirty tree of modified pages (the write-through shape) can still be committed — the push-retry's rebase-on-reject handles a remote that advanced (#2426; hook + helper render from ONE bash push-retry template — DRY at the TS source, not by the hook sourcing a repo-controlled script), durability rules patched into the active resolver file (`findResolverFile` → RESOLVER.md > AGENTS.md; taxonomy rendered from the bundled `_brain-filing-rules.json`), a minimal DB-free pull cron (launchd/crontab running `gbrain sources pull --path ` so it never opens the PGLite single-writer lock), and a push-probe verify (no heartbeat commit). Credential is REPO-scoped (`acceptPat` from `--pat-file`/`GBRAIN_GITHUB_PAT`, warns on loose perms; reuses an existing repo-local `credential.helper`, else a `0600` store wired via repo-local config); the token is redacted everywhere via `redactSecretsInText` and never enters the repo, remote URL, logs, or `DurabilityReport`. `unhardenBrainRepo` removes the cron/hook/credential wiring (ownership-fingerprinted) and runs before `sources remove`. CLI: `gbrain sources harden ` / `pull |--path ` / `unharden `; auto-harden fires on `sources add --url ... --pat-file` for managed clones (`--no-harden` opts out). `sources pull --path` is dispatched in `src/cli.ts` BEFORE `connectEngine` so the cron stays DB-free. CLI-only (writes executables + an OS cron + a credential helper on the host); never exposed over MCP. Tests: `test/brain-repo-durability.serial.test.ts`, `test/git-remote-durable.serial.test.ts`, `test/brain-durability-hook.serial.test.ts`, `test/durability-cron.test.ts`. - `src/commands/storage.ts` — `gbrain storage status [--repo P] [--json]`. Split into pure data (`getStorageStatus`) + JSON formatter + human formatter (ASCII-only) matching the `orphans.ts` pattern. `PageCountsByTier` and `DiskUsageByTier` are distinct nominal types so swaps fail at compile time. - `gbrain.yml` (brain repo root) — Optional storage tiering config. Top-level `storage:` section with `db_tracked:` and `db_only:` array-valued keys. `gbrain sync` auto-manages `.gitignore` for `db_only` paths on successful sync (skips on dry-run, blocked-by-failures, submodule context, or `GBRAIN_NO_GITIGNORE=1`). `gbrain export --restore-only [--repo P] [--type T] [--slug-prefix S]` repopulates missing `db_only` files from the database. - `src/core/supabase-admin.ts` — Supabase admin API (project discovery, pgvector check). @@ -151,7 +153,7 @@ per-release `**vX.Y.Z:**` narration — CI enforces this - `test/helpers/cli-pty-runner.ts` — generic real-PTY harness (~470 lines) using pure `Bun.spawn({terminal:})` (Bun 1.3.10+; engines.bun pin in package.json). Generic primitives only, no plan-mode orchestrators. Exports `launchPty`, `resolveBinary`, `stripAnsi`, `parseNumberedOptions`, `optionsSignature`, `isNumberedOptionListVisible`, `isTrustDialogVisible`. Self-tests in `test/cli-pty-runner.test.ts` (24 cases). - `src/core/skillpack/{init-brain-pack,brain-pack-advisory,brain-pack-lint,brain-resident-locate,nag-state}.ts` (#2180) — brain-resident skillpacks. `manifest-v1.ts` gains optional `brain_resident` + `schema_pack` (additive). `runInitBrainPack` scaffolds a pack beside brain content (`brain_resident:true`, exact `gbrain_min_version`, 5-section machine-parseable README); `applyWritePlan` is factored out of `init-scaffold.ts` for the shared refuse-overwrite loop. `brain-pack-lint.lintBrainPackTools` validates each skill's `tools:` against the serving op set (E6 version-skew). Topology A: `src/commands/sources.ts` `runAdd` prints `brain-pack-advisory` to stderr after `opsAddSource`, fail-open; `nag-state.ts` (`~/.gbrain/skillpack-nag-state.json`) keys declines by `(source-repo brain_id, source_id, pack_name)` with pure `decideNagAction` (first/reminder/version-bump/ceiling) — declines count ONLY on CLI-interactive displays. Topology B: `brain-resident-locate.loadResidentPacksForServer` (source-scoped via `sourceScopeOpts`) backs the `list_brain_skillpack` op; `getResidentSkillDetail` backs `get_skill` `source_id`; `scaffold_spec` is the git source, never a server FS path. Tests: `test/skillpack-{init-brain-pack,nag-state,brain-resident-locate}.test.ts` + the brain-resident cases in `test/skillpack-manifest-v1.test.ts`. - `src/core/advisor/{types,run,render,recommended-set,history,apply,collect-*}.ts` + `src/commands/advisor.ts` (#2180) — `gbrain advisor`: read-only ranked actions from brain state. `run.runAdvisor` executes 8 hardcoded collectors (version [cache-only], migration, schema-pack, stalled-jobs [absent-table tolerant], usage-shape, setup-smells, uninstalled-brain-pack, uninstalled-bundled), each in its own try/catch; `rankFindings` orders critical>warn>info then collector order, caps the info tail, and drops `workspace_dependent` findings when `remote` (A1). `render.ts` is the shared `=`-bar renderer used by the advisor AND `post-install-advisory.ts` (generalized to a single current-state `recommended-set.RECOMMENDED`, `install`→`scaffold`). `history.ts` appends bounded `~/.gbrain/advisor-history.jsonl` (no DB migration) for since-last-run deltas; local-only. `apply.resolveApplyTarget` is the allowlist+injection guard for `commands/advisor.ts --apply ` (structured argv, never a shell; local-only). The `advisor` op (`operations.ts`) is read-scoped, NOT localOnly, gated by `mcp.publish_advisor` (config.ts; default off) and strictly read-only on remote. CLI wired in `cli.ts` (`CLI_ONLY` + dispatch). Bundled skill `skills/gbrain-advisor/` + weekly cron recipe. Tests: `test/advisor-{core,apply,op-gate,ranking-eval}.test.ts`. -- `src/core/chronicle/{eligibility,config,backstop,extract-events,ontology,narrative}.ts` + `src/eval/chronicle/harness.ts` + `src/commands/eval-chronicle.ts` (#2390) — Life Chronicle: the temporal spine. `eligibility.isChronicleEligible` decides which pages auto-emit events (meeting/conversation/calendar-event + directory rescue; diary and event pages NEVER eligible — privacy + anti-loop). `backstop.runChronicleBackstop` is the put_page hook body (fires ONLY on `status==='imported'` + the auto-link trust gate + the default-OFF `auto_chronicle` flag; enqueues a `chronicle_extract` minion job — LLM never runs on the write path). `extract-events.runChronicleExtract` is the job body: deterministic when/who, injectable judge (default = chat gateway), an ALL-or-nothing parse barrier (`isValidProposal` requires a real parseable date — a malformed batch writes NOTHING), then content-addressed `life/events/` pages + a `timeline_entries` projection via `engine.upsertEventProjection` (dedup `(event_page_id, date)`; idempotent re-runs). `ontology.ts` carries the deterministic pieces of the bi-temporal per-entity ontology that RIDES THE `facts` TABLE (migration v122 adds `dimension`/`value`/`value_hash`/`dim_status`): `valueHash` (normalized, timestamp-free → crash-retry idempotent), `normalizeDimension` (seed alias lexicon), `isNovelDimension` (novel → quarantined, excluded from resolution/context until confirmed). The engine methods (`mergeOntologyFact` — corroborate on same value, forward-supersede via `valid_until`+`superseded_by` on a new value, backdated conflicts kept + flagged; `getOntology` with `--asof` valid-time travel; `discoverOntologyDimensions`; `findOntologyConflicts` — currently-open rows only) live in BOTH engines; both engines are on the R8 `valid_until` write allow-list (engine-layer, `dimension IS NOT NULL` rows only). Chronicle reads (`getTimelineForDate`/`getSince`/`getLastSeen`/`getOnThisDay`) JOIN the depth page (`deleted_at IS NULL`), hide soft-deleted event projections at READ time, and order by event `effective_date` for intra-day sequence. Ops: `chronicle_day`/`chronicle_since`/`chronicle_last_seen`/`chronicle_on_this_day`/`ontology_*`/`volunteer_chronicle` (agent orientation via `src/core/context/chronicle-context.ts`)/`chronicle_backfill` (admin, localOnly). Diary privacy: diary-sourced ontology + conflict values redacted for `ctx.remote !== false` callers. Search: `applyChronicleTypeBoost` in `search/hybrid.ts` (bounded [1.0,1.25], fires only inside the `recency !== 'off'` post-fusion branch → non-temporal search bit-for-bit unchanged). Advisor collector `collect-chronicle.ts` (conflicts + coverage gap); doctor `chronicle_projection_health` (BRAIN category). Eval: `gbrain eval chronicle` — deterministic, own in-memory PGLite, 6 gold tasks (day order, last-seen, supersession, asof, conflict, isolation), exit 0 iff 6/6. Tests: `test/chronicle-*.test.ts`, `test/eval-chronicle.test.ts`. +- `src/core/chronicle/{eligibility,config,backstop,extract-events,ontology,narrative}.ts` + `src/eval/chronicle/harness.ts` + `src/commands/eval-chronicle.ts` (#2390) — Life Chronicle: the temporal spine. `eligibility.isChronicleEligible` decides which pages auto-emit events (meeting/conversation/calendar-event + directory rescue; diary and event pages NEVER eligible — privacy + anti-loop). `backstop.runChronicleBackstop` is the put_page hook body (fires ONLY on `status==='imported'` + the auto-link trust gate + the default-OFF `auto_chronicle` flag; enqueues a `chronicle_extract` minion job — LLM never runs on the write path). `extract-events.runChronicleExtract` is the job body: deterministic when/who, injectable judge (default = chat gateway; output cap 4000 tokens by default, operator override `chronicle.judge_max_tokens`), an ALL-or-nothing parse barrier (`isValidProposal` requires a real parseable date — a malformed batch writes NOTHING), then content-addressed `life/events/` pages + a `timeline_entries` projection via `engine.upsertEventProjection` (dedup `(event_page_id, date)`; idempotent re-runs). An unusable judge response is never recorded as `no_events` (#2606): a `stopReason: 'length'` truncation or a no-JSON-array response (`parseJudgeJson` returns `null` on parse failure; `[]` only for a legitimate empty array) surfaces as `status: 'skipped'` with reason `judge_truncated` / `judge_parse_failed`. `ontology.ts` carries the deterministic pieces of the bi-temporal per-entity ontology that RIDES THE `facts` TABLE (migration v122 adds `dimension`/`value`/`value_hash`/`dim_status`): `valueHash` (normalized, timestamp-free → crash-retry idempotent), `normalizeDimension` (seed alias lexicon), `isNovelDimension` (novel → quarantined, excluded from resolution/context until confirmed). The engine methods (`mergeOntologyFact` — corroborate on same value, forward-supersede via `valid_until`+`superseded_by` on a new value, backdated conflicts kept + flagged; `getOntology` with `--asof` valid-time travel; `discoverOntologyDimensions`; `findOntologyConflicts` — currently-open rows only) live in BOTH engines; both engines are on the R8 `valid_until` write allow-list (engine-layer, `dimension IS NOT NULL` rows only). Chronicle reads (`getTimelineForDate`/`getSince`/`getLastSeen`/`getOnThisDay`) JOIN the depth page (`deleted_at IS NULL`), hide soft-deleted event projections at READ time, and order by event `effective_date` for intra-day sequence. Ops: `chronicle_day`/`chronicle_since`/`chronicle_last_seen`/`chronicle_on_this_day`/`ontology_*`/`volunteer_chronicle` (agent orientation via `src/core/context/chronicle-context.ts`)/`chronicle_backfill` (admin, localOnly). Diary privacy: diary-sourced ontology + conflict values redacted for `ctx.remote !== false` callers. Search: `applyChronicleTypeBoost` in `search/hybrid.ts` (bounded [1.0,1.25], fires only inside the `recency !== 'off'` post-fusion branch → non-temporal search bit-for-bit unchanged). Advisor collector `collect-chronicle.ts` (conflicts + coverage gap); doctor `chronicle_projection_health` (BRAIN category). Eval: `gbrain eval chronicle` — deterministic, own in-memory PGLite, 6 gold tasks (day order, last-seen, supersession, asof, conflict, isolation), exit 0 iff 6/6. Tests: `test/chronicle-*.test.ts`, `test/eval-chronicle.test.ts`. - `src/core/skill-manifest.ts` — parser for `skill-manifest.json` records. Used by skillpack installer to detect drift between the shipped bundle and the user's local edits, so updates merge instead of overwriting. - `src/commands/routing-eval.ts` + `src/core/routing-eval.ts` — `gbrain routing-eval` catches user phrasings that route to the wrong skill. Reads `skills//routing-eval.jsonl` fixtures (`{intent, expected_skill, ambiguous_with?}`). Structural layer runs in `check-resolvable` by default (zero API cost). `--llm` is a placeholder for a future LLM tie-break layer; today it emits a stderr notice and runs structural only. Uses `autoDetectSkillsDirReadOnly` and the same multi-file resolver merge as `check-resolvable`, so on OpenClaw layouts (`skills/RESOLVER.md` + `../AGENTS.md`) all three commands see the same trigger index. RESOLVER.md rows carry the full frontmatter `triggers:` arrays so the structural matcher sees realistic phrasings; ambiguous-fixture annotations cover deliberate skill chains like `enrich → article-enrichment`. - `src/core/filing-audit.ts` + `skills/_brain-filing-rules.json` — Check 6 of `check-resolvable`. Parses `writes_pages:` / `writes_to:` frontmatter on skills and audits their filing claims against the filing-rules JSON (error severity). Internal `parseFrontmatter` is a thin wrapper over the shared `src/core/skill-frontmatter.ts` parser so both filing-audit and skill-brain-first read the same shape (`tools?`, `triggers?`, `brain_first?: 'exempt'`, typed `brain_first_typo`) from one source of truth. @@ -197,7 +199,7 @@ per-release `**vX.Y.Z:**` narration — CI enforces this - `src/commands/sources.ts` — `gbrain sources {list,add,remove,archive,restore,archived,purge,current,status,audit}`. `current [--json]` calls `resolveSourceWithTier()` and prints `source_id`, `tier` (`flag | env | dotfile | local_path | brain_default | seed_default`), and optional `detail` (decision table in `skills/conventions/brain-routing.md`). `status [--json]` — read-only per-source dashboard (last sync, staleness, page count, embedding coverage, unacked failures); thin wrapper around `buildSyncStatusReport` + `printSyncStatusReport` from `src/commands/sync.ts`; `--json` emits stable `{schema_version: 1, sources, ...}` on stdout; filters input to `local_path IS NOT NULL AND archived IS NOT TRUE`. `audit [--json]` — read-only dry-run disk scan for size distribution + would-blocks + junk-pattern hits WITHOUT touching the DB; walks `sources.local_path`, reads each markdown file, runs `assessContent()` from `src/core/content-sanity.ts`, aggregates by verdict (`ok | warn_oversize | hard_block_junk_pattern`). The live `runStatus` health table gains a `BACKFILL` column between `EMBED` and `FAILS` (`active(N)` beats `queued(N)` beats `idle`, from `SourceMetrics.backfill_active` / `backfill_queued` in `src/core/source-health.ts`) so operators see deferred `embed-backfill` minion work after `sync --all` exits 0; `jobCountsBySource` in `source-health.ts` widens its `minion_jobs` SQL with two `COUNT(*) FILTER (WHERE name = 'embed-backfill' AND ...)` aggregates (best-effort, all-0 on pre-minions brains). Pinned by `test/content-sanity.test.ts`, `test/import-file-content-sanity.test.ts`, `test/source-health.test.ts`. - `src/commands/reindex-frontmatter.ts` — `gbrain reindex-frontmatter`. Query path wrapped in the standard `withEngine(...)` lifecycle so `engine.connect()` runs before the first SQL call. Pinned by `test/reindex-frontmatter-connect.test.ts`. - `src/core/source-resolver.ts` — 6-tier source resolution. `resolveSourceWithTier(engine, explicit, cwd)` returns `{ source_id, tier: SourceTier, detail? }` alongside `resolveSourceId()` (unchanged). `SOURCE_TIER_NAMES = ['flag', 'env', 'dotfile', 'local_path', 'sole_non_default', 'brain_default', 'seed_default']` (7 entries; order matches priority). Tier `sole_non_default` slots between `local_path` and `brain_default`: when NO `sources.default` config is set AND exactly one registered source has `local_path` AND isn't `'default'`, auto-route to it; archived sources excluded (try/catch for pre-v34 brains); private `pickSoleNonDefaultSource(engine)` shared by both resolver entry points so they cannot drift. Exported `formatSoleNonDefaultNudge(sourceId): string | null` builds the user-facing stderr nudge (null when `GBRAIN_NO_SOLE_NON_DEFAULT_NUDGE=1`). `src/commands/sync.ts:1497-1519` calls `resolveSourceWithTier` unconditionally so the tier fires; `src/commands/import.ts:96-128` mirrors with the tier-gated nudge. Consumed by `gbrain sources current`, `import --source-id`, `extract --source-id`, and the `source_routing_health` doctor check. Pinned by `test/source-resolver-with-tier.test.ts` (`withEnv()` per test-isolation lint), `test/source-resolver-sole-non-default.test.ts` (14 cases), `test/sync-sole-non-default-routing.test.ts` (3 PGLite cases driving real `runSync`). -- `src/core/sync.ts` extension — `isSyncable` factored through private `classifySync(path, opts): SyncableReason | null`; exported companion `unsyncableReason(path, opts)` returns the same tagged reason or null when syncable. `SYNC_SKIP_FILES` is a named export (the four canonical metafile basenames `schema.md`, `index.md`, `log.md`, `README.md`). `SyncableReason` union: `'metafile' | 'strategy' | 'pruned-dir' | 'include-glob-miss' | 'exclude-glob-hit'`. `src/commands/sync.ts:772` cleanup loop guards on `unsyncableReason(path) === 'metafile'` so previously-indexed metafile pages survive every re-sync. Does NOT cover `manifest.deleted` (the upstream filter already strips metafiles). Pinned by `test/sync-isSyncable-shape.test.ts` (15 cases, duality contract) + `test/sync-metafile-skip.serial.test.ts` (3 PGLite cases incl. the renamed `.md → .txt` negative). +- `src/core/sync.ts` extension — `isSyncable` factored through private `classifySync(path, opts): SyncableReason | null`; exported companion `unsyncableReason(path, opts)` returns the same tagged reason or null when syncable. `SYNC_SKIP_FILES` is a named export (the four canonical metafile basenames `schema.md`, `index.md`, `log.md`, `README.md`). `SyncableReason` union: `'metafile' | 'strategy' | 'pruned-dir' | 'include-glob-miss' | 'exclude-glob-hit'`. The `commands/sync.ts` cleanup loop guards on `unsyncableReason(path)` being `'metafile'` OR `'pruned-dir'` (#2404) so previously-indexed metafile pages AND deliberately put-created pages under pruned dirs survive every re-sync. Does NOT cover `manifest.deleted` (the upstream filter already strips metafiles). Pinned by `test/sync-isSyncable-shape.test.ts` (15 cases, duality contract) + `test/sync-metafile-skip.serial.test.ts` (3 PGLite cases incl. the renamed `.md → .txt` negative). - `src/core/import-file.ts` extension — identity-based dedup pre-check at `:427-490`. Calls `engine.findDuplicatePage?.(sourceId, {hash, frontmatterId})` (optional `?` so test doubles compile). Posture: SKIP when `frontmatter.id` matches (true external duplicate from overlapping ingest roots), WARN-ALWAYS on content_hash collision with different/missing `frontmatter.id` (templates and daily logs may legitimately share text), FAIL CLOSED on lookup error, bypass via `--force-rechunk`. Soft-deleted pages excluded at the engine layer so tombstones don't block legitimate re-imports under new slugs. Pinned by `test/import-dedup-frontmatter-id.test.ts` (11 cases). - `src/core/engine.ts` extension — two interface members: (1) optional `findDuplicatePage?(sourceId, {hash, frontmatterId?}): Promise<{slug, id} | null>` (identity precedence is content_hash OR frontmatter->>'id', both with `deleted_at IS NULL`); (2) `resolveSlugs(partial, opts?)` extended with `{sourceId?, sourceIds?}` so the MCP fuzzy `get_page` path scopes by source (field names match `sourceScopeOpts(ctx)` output so handlers spread directly; back-compatible — no opts gives prior behavior). Plus a stable tiebreaker `ORDER BY score DESC, page_id ASC, chunk_id ASC` in `searchVector` in both engines: on a score tie (basis-vector eval fixtures) older `page_id` wins, closing the planner-non-determinism class where a new index on `pages` could flip ranking on tied scores. - `src/core/migrate.ts` v95 — `pages_dedup_partial_index` adds `CREATE INDEX pages_dedup_idx ON pages (source_id, content_hash) WHERE deleted_at IS NULL`. Postgres uses `CREATE INDEX CONCURRENTLY` with `transaction: false` + pre-drops any invalid remnant; PGLite uses plain `CREATE INDEX`. Powers `findDuplicatePage` hot path (O(log n) instead of O(n)). @@ -283,7 +285,7 @@ per-release `**vX.Y.Z:**` narration — CI enforces this - `test/fixtures/whoknows-eval.jsonl` — 10-row synthetic placeholder demonstrating the eval-fixture schema (`{query, expected_top_3_slugs, notes?}` JSONL). End users replace with their own real queries; placeholder uses obviously-example slugs (`wiki/people/example-alice`). Drives `test/e2e/whoknows.test.ts` (seeds a matching synthetic brain, asserts the >=80% gate) and the `whoknows_health` doctor check. - `src/core/skillopt/` + `src/commands/skillopt.ts` + `skills/skill-optimizer/` — self-evolving skill optimization grounded in the SkillOpt paper (arXiv 2605.23904). `gbrain skillopt ` treats `SKILL.md` as trainable parameters of a frozen agent: validation-gated (median-of-3 + epsilon=0.05), budget-capped (preflight estimator), per-skill DB-locked (`tryAcquireDbLock('skillopt:', 60min)`), atomic-versioned (history-intent-first 5-step commit), body-only mutations (frontmatter forbidden). Rollouts use `gateway.toolLoop` directly with no-op persistence callbacks (zero `subagent_messages` pollution) + a read-only tool allowlist derived from `BRAIN_TOOL_ALLOWLIST` minus `put_page`/`submit_job`/`file_upload`. Two reflect calls per step; rejected-edit buffer LRU-bounded to 100; bundled-skill gate; bootstrap workflow (sentinel + `--bootstrap-reviewed`); D_sel floor (>=5 with `--split` override); audit JSONL via `audit-writer.ts`. Added to `ALL_PHASES` after `patterns` (default OFF; opt-in via `gbrain config set cycle.skillopt.enabled true`); cycle phase wrapper at `src/core/skillopt/cycle-phase.ts` walks stale skills with per-skill ($0.50) + brain-wide ($2.00) caps. Added to `PROTECTED_JOB_NAMES`. Surface: dream-cycle phase wrapper; `--all` batch mode (`src/core/skillopt/batch.ts:runBatchAll`); `--target-models` fleet (`runFleet` parallel per-model receipts under `skillopt/fleet//`); MCP op `run_skillopt` (admin scope + per-skill `skillopt.allowed_skills` allowlist, NOT localOnly, validates `skill_name` kebab-only + confines caller-supplied benchmark/held-out paths to skillsDir for remote callers); Minion `skillopt` handler + `--background` with `allowProtectedSubmit: true`; write-flavored optimization via `src/core/skillopt/write-capture.ts:buildWriteCaptureRegistry` (virtual `put_page`/`submit_job`/`file_upload` captured in-memory; `--write-capture` flag); held-out real-user test set via `src/core/skillopt/held-out.ts` (capture infra at `~/.gbrain/skillopt-captures//.jsonl`, `--held-out ` flag, `runHeldOutGate` candidate >= baseline). Hermetic via DI seams (`opts.chatFn` for optimizer + judge; `opts.toolLoopFn` for rollouts; no `mock.module`). `--bootstrap-from-skill` → `runBootstrapFromSkill` in `src/core/skillopt/bootstrap-benchmark.ts`: reads `SKILL.md` directly (no `routing-eval.jsonl`), makes ONE LLM call emitting a full starter benchmark (tasks + rule judges) as JSONL, parsed line-by-line with skip-bad-line salvage and a min-2-valid-checks-per-task drop; provider/transport errors PROPAGATE (not collapsed to `bootstrap_empty`). `--bootstrap-tasks N` (default 15, capped 50); `maxTokens` scales `min(8000, max(4000, N*220))`. The stderr REVIEW line prints the literal `gbrain skillopt --bootstrap-reviewed --split 1:1:1` — load-bearing because the default `4:1:5` split makes a 15-task starter's `D_sel = floor(15/10) = 1`, below the `>=5` floor, so a 15-task benchmark needs `--split 1:1:1`. Both bootstrap generators share `assertBenchmarkAbsent` + `readSkillBodyOrThrow`; `--bootstrap-from-skill` is mutually exclusive with `--bootstrap-from-routing`/`--benchmark`/`--all`/`--target-models`/`--resume`. Generated rule judges are explicitly WEAK DRAFTS to be strengthened during the review gate. The F11 held-out gate is wired: `--held-out ` is parsed and threaded through every caller (CLI main + `--background` `held_out_path` + batch/fleet `heldOutPath` + the `run_skillopt` `held_out_path` param), running at CHECKPOINT ACCEPTANCE so no-mutate/fleet paths can't promote a held-out-failing candidate. `assertBundledMutationHeldOut` in bundled-skill-gate.ts: bundled + `--allow-mutate-bundled` requires a NON-EMPTY held-out (`MIN_HELD_OUT_SIZE = D_SEL_MIN_SIZE` = 5, derived so they can't desync) or hard-refuses (exit 2), for ALL callers (they funnel through `runSkillOpt`); held-out must be task_id-DISJOINT from the benchmark (overlap rejected — can't catch overfitting). `receipt.baseline_sel_score` populated + a real final-test eval (`test_score` + `baseline_test_score`) scoring best + baseline on `split.test`; shared `scoreSkillOnTasks` primitive (validate-gate.ts) backs baseline/final-test/held-out scoring. `--no-mutate` writes proposed.md via `writeProposed` in version-store.ts. `maxRuntimeMin` ENFORCED (wall-clock deadline between steps → `skillopt_runtime_exceeded` → outcome aborted). Three eval-internal ablation opts on `SkillOptOpts` (NOT on CLI): `reflectMode` (`'both'`/`'failure-only'`), `disableValidationGate` (greedy-accept), `optimizerMode` (`'reflect'`/`'one-shot-rewrite'`), recorded in `RunReceipt` + audit `run_start` for replayability; `ROLLOUT_SUCCESS_THRESHOLD = 0.5` named constant for the partition; one-shot fence-strip is anchored (`^```...```$`) so an embedded code sample isn't truncated. Budget no-pricing fix: Claude Haiku 4.5's dateless canonical id `claude-haiku-4-5` is in `src/core/anthropic-pricing.ts` (a `BudgetTracker`-capped run on Haiku otherwise threw `no_pricing` on the FIRST `chat()` of every rollout); `runValidationGate` (validate-gate.ts) scans settled results for `isMustAbortError(error)` (from `worker-pool.ts`; `BUDGET_EXHAUSTED` is in `MUST_ABORT_ERROR_TAGS`) and re-throws so the caller aborts loudly instead of recording a hollow `selScore:0` — ordinary non-abort rollout errors still fail-open to `score:0` (judge-hiccup posture preserved). Pinned by 152 tests across 18 files (foundation + adversarial + v2 surface + E2E PGLite serial), `test/skillopt/bootstrap-from-skill.test.ts` (20 cases), `test/skillopt/rollout.test.ts`, `test/skillopt/validate-gate-abort.test.ts` (3 cases), held-out ENFORCE + one-shot-rewrite unit cases, and e2e (F11 block/allow, bundled no-mutate, runtime deadline, receipt honesty, held-out disjointness, no-DB-pollution). Drives the Track B SkillOpt benchmark suite in the sibling `gbrain-evals` repo. - `src/core/brainstorm/{domain-bank,orchestrator,judges}.ts` + `src/commands/{brainstorm,lsd,eval-brainstorm}.ts` + `src/core/last-retrieved.ts` — bisociation-grounded idea generation pair: `gbrain brainstorm ` (defensible, cite-heavy, 4 close × 6 far, judge threshold 4.0/5, save by default) and `gbrain lsd ` (Lateral Synaptic Drift — inverted judge rejecting ideas with resistance >4.5 "too obvious", stale-page bias via `pages.last_retrieved_at`, 2 close × 12 far, axiomatic inversions required, ephemeral by default). The "domain bank" is prefix-stratified sampling from the user's own brain (`SELECT DISTINCT substring(slug from '^[^/]+/[^/]+')` cached 1h-TTL in `config` per source) tiebroken by `JOIN page_links` connection_count, with corpus-sampling fallback when fewer prefixes than M exist. Distance normalized to [0,1] via `1 - clamp(cosine_distance, 0, 2) / 2`. `judges.ts` exports `runJudge(config, ideas)` + two configs (`BRAINSTORM_JUDGE_CONFIG` weighted originality/resistance/thesis_density/concrete_grounding/cognitive_load 0.25/0.20/0.20/0.20/0.15 vs `LSD_JUDGE_CONFIG` cognitive_load 0.50 + inversion rule). Calibration cold-start fallback: when `calibration_profiles.active_bias_tags` is empty, judge runs without anti-bias context AND stderr-warns. Op-layer write-back in `src/core/operations.ts` `search`/`query`/`get_page` handlers fires `bumpLastRetrievedAt(engine, pageIds)` (fire-and-forget, 5-min throttled via SQL clause, default-on with `search.track_retrieval` config escape hatch); internal callers (sync, migrations, dream cycle) bypass the op layer so the LSD stale signal stays clean. The fire-and-forget IIFE is tracked in a module-scoped `Set>`; `awaitPendingLastRetrievedWrites(timeoutMs?: number): Promise<{outcome, pending}>` resolves once all tracked promises settle, bounded by a 5s `Promise.race` timeout that stderr-warns the pending count. `src/cli.ts` awaits the drain unconditionally for every op in the op-dispatch finally block BEFORE `engine.disconnect()`, then a fallback `process.exit(0)` fires ONLY when `outcome === 'timeout'` AND `shouldForceExitAfterMain(argv)` (excludes `serve` so daemons stay alive) — closes the PGLite CLI search/query/get-hang class where the IIFE raced disconnect and PGLite's WASM kept Bun's event loop alive. `pages.last_retrieved_at TIMESTAMPTZ NULL` has a full (NOT partial) B-tree index covering both NULL and range branches; full forward-reference bootstrap probe on both engines. Frontmatter `mode: lsd` makes the dream-cycle synthesize phase skip LSD output via `isLsdOutput()` in `src/core/cycle/transcript-discovery.ts` short-circuiting `isDreamOutput()`. `gbrain eval brainstorm ` is a three-axis conjunctive gate (distance + usefulness + grounding — distance alone is gameable). `gbrain doctor` has a `brainstorm_health` check (migration applied, `search.track_retrieval` setting, calibration cold-start status). `judges.ts` computes the judge token budget via `computeJudgeMaxTokens(ideaCount, modelId)` (named constants `TOKEN_BUDGET_PER_IDEA`, `TOKEN_BUDGET_ENVELOPE`, `LEGACY_MIN_MAX_TOKENS`, `MAX_OUTPUT_TOKENS_CEIL`; `ANTHROPIC_OUTPUT_CAPS` map: Opus 4.7 32K, Sonnet 4.6 / Haiku 4.5 64K, legacy Claude 3.5 8K) so a large multi-call judge doesn't truncate mid-JSON; with no `modelOverride` the cap routes through the gateway's actual configured chat model via `getChatModel()`. `--save` for both commands persists through the canonical ingestion path: `persistSavedIdea(engine, {slug, content, provenanceVia})` calls `importFromContent({noEmbed:true, sourcePath})` (chunked + tagged + content_hash so search finds it, no embedding cost at save) THEN renders the saved row to disk via the shared `writePageThrough` helper (file rendered FROM the row so the two sinks can't diverge and `gbrain sync` doesn't churn it). `formatSaveOutcome(outcome, ctx)` returns an honest per-branch message (both-sinks, DB-only when no `sync.repo_path`/repo-not-a-dir, DB-saved-but-file-errored, total-failure → loud `save FAILED … NOT persisted` on stderr + nonzero exit) — closes the silent-false-success class where `--save` printed "Saved" unconditionally even when the DB write failed. `buildIdeaSlug(question, label, nonce?)` adds a random nonce suffix (injectable for tests) so two same-day runs sharing the first 60 slug chars don't clobber. `--json` callers stay DB-only. `buildBrainstormFrontmatterObject(result)` in orchestrator.ts returns the object form for `serializeMarkdown` (string `buildBrainstormFrontmatter` untouched). Pinned by `test/last-retrieved.test.ts`, `test/e2e/pglite-cli-exit.serial.test.ts` (IRON-RULE: real `bun src/cli.ts` subprocess against a hermetic PGLite tempdir asserts search/get/query exit 0 in <15s + daemon-survival), `test/fix-wave-structural.test.ts` (asserts the drain `await` is textually BEFORE `engine.disconnect`), `test/brainstorm/{distance,lsd-mode-skip,eval-brainstorm,judges-maxtokens,save}.test.ts`. Open Collider source: `github.com/CL-ML/open-collider`. -- `src/core/write-through.ts` — shared atomic disk write-through for the canonical ingestion path. `writePageThrough(engine, slug, {sourceId?, frontmatterOverrides?, logger?})` resolves the disk target from the ASSIGNED source's own working tree (`sources.local_path`), re-reads the just-written DB row (`getPage`), renders it via `serializePageToMarkdown`, and writes the `.md` under that tree's root so the brain has a committable artifact that round-trips through `gbrain sync`. A source with its own `local_path` writes there; a source WITHOUT one falls back to the global `sync.repo_path` ONLY when this is the sole source (then that path is unambiguously this source's tree) and otherwise skips with `source_has_no_local_path` rather than leak into a sibling source's git repo. Rendering FROM the row means file and row cannot diverge. ATOMIC: writes to a unique temp sibling (`.tmp..`) + `renameSync`, cleaning up temp on any failure, so a crash or concurrent `gbrain sync`/autopilot walking the live git tree never reads a half-written `.md` (matches the `.tmp + rename` convention in import-checkpoint.ts / op-checkpoint.ts). Never throws — returns `WriteThroughResult { written, path?, skipped?: 'no_repo_configured' | 'repo_not_found' | 'source_has_no_local_path' | 'page_not_found_after_write', error? }` so the caller decides messaging + exit codes. Trust gating (subagent sandbox, dry-run) stays at the CALLER. Consumers: `put_page` op and `gbrain brainstorm/lsd --save` via `persistSavedIdea`. Pinned by `test/write-through.test.ts`. +- `src/core/write-through.ts` — shared atomic disk write-through for the canonical ingestion path. `writePageThrough(engine, slug, {sourceId?, frontmatterOverrides?, logger?})` resolves the disk target from the ASSIGNED source's own working tree (`sources.local_path`), re-reads the just-written DB row (`getPage`), renders it via `serializePageToMarkdown`, and writes the `.md` under that tree's root so the brain has a committable artifact that round-trips through `gbrain sync`. A source with its own `local_path` writes there; a source WITHOUT one falls back to the global `sync.repo_path` ONLY when this is the sole source (then that path is unambiguously this source's tree) and otherwise skips with `source_has_no_local_path` rather than leak into a sibling source's git repo. Rendering FROM the row means file and row cannot diverge. ATOMIC: writes to a unique temp sibling (`.tmp..`) + `renameSync`, cleaning up temp on any failure, so a crash or concurrent `gbrain sync`/autopilot walking the live git tree never reads a half-written `.md` (matches the `.tmp + rename` convention in import-checkpoint.ts / op-checkpoint.ts). Never throws — returns `WriteThroughResult { written, path?, skipped?: 'no_repo_configured' | 'repo_not_found' | 'source_has_no_local_path' | 'page_not_found_after_write', error? }` so the caller decides messaging + exit codes. Trust gating (subagent sandbox, dry-run) stays at the CALLER. On a durability-hardened repo (`isDurabilityHardened` — the gbrain post-commit hook is installed, i.e. the user ran `gbrain sources harden`), a successful write is best-effort COMMITTED via `commitWriteThroughFile` (path-limited `git commit -- `, never sweeps unrelated edits; the hook then background-pushes) so write-through content reaches git instead of accumulating uncommitted forever (#2426); result carries `committed?: boolean`. Unhardened repos keep write-only behavior. Consumers: `put_page` op and `gbrain brainstorm/lsd --save` via `persistSavedIdea`. Pinned by `test/write-through.test.ts` + `test/write-through-commit.serial.test.ts`. - `src/core/model-id.ts` — `splitProviderModelId(input: string | null | undefined): {provider: string | null, model: string}` shared parser for the pricing side. Splits on `:` first, then `/`. Defensive contract: null/undefined/empty/whitespace returns `{provider: null, model: ''}`. Five sites consume it (`src/core/anthropic-pricing.ts:estimateMaxCostUsd`, `src/core/budget/budget-tracker.ts:lookupPricing`, `src/core/eval-contradictions/cost-tracker.ts:pricingFor`, `src/core/minions/batch-projection.ts` at two call sites, `src/core/model-config.ts:isAnthropicProvider`) so the pricing + classification surface has no parallel re-implementations of `provider:model` splitting — slash-form ids (`anthropic/claude-sonnet-4-6`) classify correctly instead of falling through to "unknown model". Distinct from the gateway-side `parseModelId` in `src/core/ai/model-resolver.ts`, which throws on bare names because routing needs an explicit provider; this one returns `{provider: null, model: 'bare'}` because pricing lookups happen against bare model ids. Pinned by `test/model-id.test.ts`. - `src/core/ai/model-resolver.ts:parseModelId` — gateway-side resolver accepts both colon and slash form (`provider:model` and `provider/model`) so a slash-form id resolves to the same recipe at every gateway entry point (chat / embed / rerank) instead of throwing `AIConfigError: model id must be in format provider:model`. Bare names without ANY separator still throw — gateway routing always needs an explicit provider. Pinned by `test/ai/model-resolver-slash.test.ts` including a `resolveRecipe` round-trip asserting slash form resolves to the same recipe object as colon form. - `src/commands/transcripts.ts` — `gbrain transcripts recent [--days N] [--full] [--json]`: recent raw `.txt` transcripts from the dream-cycle corpus dirs. Imports `listRecentTranscripts` from `src/core/transcripts.ts` (the same library the gated `get_recent_transcripts` MCP op uses). Local-only by construction — the CLI always runs with `ctx.remote=false`. @@ -301,15 +303,14 @@ per-release `**vX.Y.Z:**` narration — CI enforces this - `src/commands/extract-conversation-facts.ts` extension — `--workers N` for LLM-bound fact extraction over conversation pages, with a per-page advisory lock via `src/core/db-lock.ts:withRefreshingLock` (lock id `extract-conversation-facts::`, TTL `PER_PAGE_LOCK_TTL_MINUTES=2` with 20s refresh via `Math.max(15s, 120s/6)`; `LockUnavailableError` triggers skip-and-continue with rate-limited log per (source, minute) + `pages_lock_skipped` counter + CLI exits 3 when non-zero AND no hard failures). `deleteOrphanFactsForPage(engine, sourceId, slug)` provides delete-orphans-first replay safety — wipes facts from a prior crashed run for this (sourceId, slug) before re-extracting, closing the "terminal audit row written after partial insertFacts failure" class. `assertFactsEmbeddingDimMatchesConfig(engine)` is the startup preflight (throws `FactsEmbeddingDimMismatchError` with paste-ready ALTER hint BEFORE the first insert; cached per engine via WeakMap). Result type carries `pages_lock_skipped` + `orphan_facts_cleaned`. Checkpoint state is a shared `cpMap: Map` (NOT a per-page-mutated `cpEntries: string[]`) so atomic `Map.set` survives parallel workers. Minion handler `extract-conversation-facts` in `src/commands/jobs.ts` round-trips `workers` via `job.data.workers` for `--background --workers 20`. Cycle config key `cycle.conversation_facts_backfill.workers` (default 1; opt-in concurrency under brain-wide cost + walltime caps). Pinned by `test/extract-conversation-facts-workers.test.ts` + the existing extract-conversation-facts behavioral tests. - `src/core/embedding-dim-check.ts` extension — facts.embedding dim drift surface. `readFactsEmbeddingDim(engine): Promise` covers both `vector(N)` and `halfvec(N)` shapes (migration v40 falls back to `vector` on pgvector < 0.7); regex ordering is halfvec-before-vector (substring "vec" appears in "halfvec"; naive `/vector/i` would shadow). `buildFactsAlterRecipe(dims, configured, type)` emits the paste-ready `DROP INDEX IF EXISTS idx_facts_embedding_hnsw; ALTER TABLE facts ALTER COLUMN embedding TYPE halfvec(N) USING embedding::halfvec(N); CREATE INDEX idx_facts_embedding_hnsw ON facts USING hnsw (embedding halfvec_cosine_ops) WHERE ...` flow (NOT bare REINDEX, which doesn't rewrite the index after a column-type change). `assertFactsEmbeddingDimMatchesConfig(engine)` is the preflight — throws `FactsEmbeddingDimMismatchError` (tagged `tag: 'FACTS_EMBEDDING_DIM_MISMATCH'` for parity with the worker-pool MUST_ABORT semantics) when configured dim ≠ column width; cached per-engine via `WeakMap`; PGLite engines silently skip. Doctor check `facts_embedding_width_consistency` (registered after `embedding_width_consistency`) reuses the same helpers with an identical ALTER recipe. Pinned by `test/embedding-dim-check-facts.test.ts`. - `src/core/postgres-engine.ts` extension — `insertFact` + `insertFacts` no longer hardcode `tx.unsafe(\`'${embedLit}'::vector\`)` for the embedding column. `resolveFactsEmbeddingCast()` (private) probes `pg_attribute` once per engine instance (cached in `_factsEmbeddingCastSuffix`) and returns `'::halfvec'` when migration v40 created the column as halfvec, else `'::vector'`; both insert paths use the cached suffix so the cast matches the actual column type (works on older pgvector that lacks implicit auto-cast). Test seam `__resetFactsEmbeddingCastCacheForTest()` clears the per-engine cache. -- `src/core/cycle.ts` + `src/core/cycle/extract-atoms.ts` + `src/core/cycle/synthesize-concepts.ts` + `src/commands/extract.ts` + `src/commands/doctor.ts` + `src/core/op-checkpoint.ts` extensions — six daily-driver ops fixes. (1) Batch idempotency: `atomsExistingForHashes(engine, sourceId, hashes[])` (exported from `src/core/cycle/extract-atoms.ts`) replaces the per-hash loop (7K individual queries) with one batched SQL roundtrip returning already-extracted `content_hash16` values; fail-open (SQL error → empty set, extraction proceeds); powered by migration v104 `pages_atom_source_hash_idx` (partial expression index on `frontmatter->>'source_hash'` for atom rows where `deleted_at IS NULL`; Postgres `CREATE INDEX CONCURRENTLY` with invalid-remnant pre-drop, PGLite plain). (2) Shorter cycle lock TTL + active in-phase refresh: `LOCK_TTL_MINUTES = 5` (was 30); `buildYieldDuringPhase(lock, outer)` (exported, with `LockHandle`) calls `lock.refresh()` + any external hook on every fire, throttled to 30s via `maybeYield`, firing both in the main loop AND immediately after every `await chat(...)`; `synthesize_concepts` uses the same throttled hook. A crashed cycle releases its lock 6x faster while a healthy long-running cycle keeps it alive (residual: a single `await chat()` past 5 min can expire the lock mid-await — TODO-OPS-2). (3) Progress wiring: `progress?: ProgressReporter` opt on `ExtractAtomsOpts` and `SynthesizeConceptsOpts`; cycle.ts passes its phase-level reporter down (NOT a child reporter, which would collide on `cycle.extract_atoms.extract_atoms.work`); phases only call `tick()`/`heartbeat()`, cycle.ts owns `start()`/`finish()`. (4) `by-mention` resume: `mentionsFingerprint({source, type, since, gazetteerHash})` in `src/core/op-checkpoint.ts` — the gazetteer hash is load-bearing (adding entity pages mid-pause shifts the hash → new fingerprint → fresh scan against the new gazetteer, never silent skip); `gbrain extract links --by-mention` resumes via `op_checkpoints` with `flushAndCheckpoint` ordering (links flush to DB FIRST, page keys commit to checkpoint SECOND, persist THIRD, so a crash mid-batch leaves the page un-checkpointed and resume re-scans it); persist every 1000 items OR 30s; clean exit clears the checkpoint; `--dry-run` skips both load and write. (5) `sync_consolidation` doctor check (multi-source brains see a paste-ready `gbrain sync --all --parallel 4 --workers 4 --skip-failed`; single-source "not applicable"; SQL errors return `warn` via the check's own try/catch). (6) Test-isolation: `test/cycle-last-full-cycle-at.test.ts` + `test/schema-cli.test.ts` use per-test `GBRAIN_HOME=tempdir`. Pinned by `test/cycle/extract-atoms-batch.test.ts`, `test/cycle/cycle-lock-ttl.test.ts` (regression pin on `LOCK_TTL_MINUTES === 5`), `test/op-checkpoint-mentions-fingerprint.test.ts`, `test/cycle/extract-atoms-progress.test.ts`, `test/cycle/synthesize-concepts-progress.test.ts`, `test/cycle/yield-during-phase-refresh.test.ts`, `test/cycle/yield-during-phase-throttle.test.ts`, `test/extract-by-mention-resume.test.ts`, `test/doctor-sync-consolidation.test.ts`. Companion `sync --all` recipe block in `skills/cron-scheduler/SKILL.md`. -- `src/commands/sync.ts` — `gbrain sync` CLI + the `performSync` / `performFullSync` library entrypoints (consumed by the autopilot cycle and the Minion sync handler). `performSync` runs under a writer lock: per-source `gbrain-sync:` whenever `opts.sourceId` is set, wrapped in `withRefreshingLock` from `src/core/db-lock.ts` so long-running sources (250K+ chunks) don't lose the lock at the 30-min TTL mid-run; the bare no-source path uses the SAME refreshing lock (#1794 — it was previously a non-refreshing `tryAcquireDbLock`, stealable mid-run during an incident); `SyncOpts.lockId?: string` is the explicit override. The lock refresh AND its health probe route through the DIRECT session pool so Supavisor transaction-pool exhaustion (`EMAXCONNSESSION`) can't kill renewal; takeover is heartbeat-aware (it will NOT steal a holder whose `last_refreshed_at` is within `GBRAIN_LOCK_STEAL_GRACE_SECONDS`, defending an alive-but-starved holder); the import loop yields the event loop every `GBRAIN_SYNC_YIELD_EVERY` files (`setTimeout(0)`, not `setImmediate` — Bun starves the timers phase) so the refresh `setInterval` heartbeat fires mid-import. This lock-identity invariant prevents a `sync --all` per-source worker racing `sync --source foo` on the global lock from corrupting the same source. `performSync` throws a typed `SyncLockBusyError` when the writer lock is held; the Minion `sync` handler (`src/commands/jobs.ts`) catches it and marks the job SKIPPED (not failed) so a cron/autopilot tick defers to the holder without polluting crash metrics. `performSyncInner` is RESUMABLE (incremental path): it drains a PINNED target commit (`lastCommit..pin`), banking drained file paths via `appendCompleted` (append-only delta into the `op_checkpoint_paths` child table, migration v115 — one row per path, O(delta) not the old O(N²) full-array rewrite), keyed by `syncFingerprint({sourceId, lastCommit})` from `src/core/op-checkpoint.ts` (paths under `op:'sync'`; the pinned target under `op:'sync-target'`), and advances `last_commit`/`last_sync_at` ONLY at full import completion. Checkpoint writes route through the DIRECT session pool + bounded retry so they survive `EMAXCONNSESSION`; the flush cadence is first-file then every `GBRAIN_SYNC_CHECKPOINT_EVERY` (default 1000) files OR `GBRAIN_SYNC_CHECKPOINT_SECONDS` (default 10s), with a race-safe `pendingCheckpointPaths` delta (single-flight swap, re-merge on failure) under parallel workers; a SIGTERM banks the in-flight delta via a no-retry one-shot (`appendCompletedOnce`, ordered before lock release through `registerCleanup`); and sustained flush failure aborts the run with `reason:'checkpoint_unavailable'` after `GBRAIN_SYNC_MAX_CHECKPOINT_FAILURES` consecutive fails rather than importing work it can never bank (every partial/blocked exit logs the banked-file count). A sync killed mid-import banks its progress, leaves the anchor unmoved (the source stays correctly stale to the autopilot scheduler — `last_sync_at` is never bumped on a partial), and the next run `resumeFilter`s the same fixed diff to skip done files. The pin is the checkpoint's stored target when still reachable from HEAD, else live HEAD (a history rewrite / reset re-pins); completion advances to the pin, NOT live HEAD, so commits landing past the pin are a clean next-sync diff — this closes the cross-run staleness window. After import a pin-reachability gate (`git merge-base --is-ancestor pin HEAD`) replaces the old strict head-drift gate: forward commits on top of the pin (e.g. a background enrich process committing to the same repo every ~2 min) are SAFE and no longer block the run; only a real rewrite (pin not an ancestor of HEAD) blocks and discards the checkpoint. A file added in `lastCommit..pin` but gone from disk (deleted by a commit after the pin) is SKIPPED and marked completed, not recorded as a failure. Downstream extract/facts/embed are size-gated: inline only for `totalChanges <= 100`; large syncs defer to the resumable `extract --stale` watermark + `embed --stale`/backfill + the facts cycle phases, so a 44K-page facts/embed pass never re-blocks import convergence (sync convergence == import convergence). Worker engines wrap in try/finally so disconnect always fires; both PGLite-detection sites use `engine.kind === 'pglite'`. CLI accepts `--workers N` (alias `--concurrency N`) validated via `parseWorkers` (explicit bypasses the file-count floor; auto path defers to `autoConcurrency()`). The newest-first descending-lex order uses `sortNewestFirst(addsAndMods)` from `src/core/sort-newest-first.ts` (shared with `gbrain import`). `gbrain sync --all` runs a continuous worker pool: `parseWorkers`-validated `--parallel N` (default `min(sourceCount, --workers, DEFAULT_PARALLEL_SOURCES=4)`), long-lived async workers pulling from a shared FIFO queue (no head-of-line blocking), per-source `withSourcePrefix(src.id, ...)` so every `slog`/`serr` line carries `[]`; `--skip-failed`/`--retry-failed` are scoped per source (`acknowledgeFailures(sourceId)`; `--all` acks every source, single-source acks only its own) and run UNDER parallel — the #1939 failure ledger is per-`(source_id, path)` and serialized through `withLedgerLock`, so the old "not supported under parallel, re-run with --serial" refusal is lifted (#2139, which also removed the forcing-function that pushed recovery syncs to `--serial` and thus armed the inline cost gate); a connection-budget stderr warning fires when `parallel × workers × 2 > 16` (the `× 2 per-file pool` factor: each per-file worker opens its own `PostgresEngine` with `poolSize=2`). Exports `resolveParallelism`, `syncOneSource`, `buildSyncStatusReport`, `printSyncStatusReport`, `SyncStatusReport` back the `gbrain sources status` dashboard. `--json` envelope `{schema_version: 1, sources, parallel, ok_count, error_count}` on stdout; human banners route to stderr via `humanSink` so `jq` parses cleanly. Exit matrix: 0 all ok, 1 any error. (The non-TTY cost gate no longer exits 2 — it auto-defers; #2139.) The dashboard SQL is `content_chunks ch JOIN pages pg ON pg.id = ch.page_id WHERE pg.deleted_at IS NULL` with `archived = false` at the caller; embedding column resolved via `resolveEmbeddingColumn(undefined, cfg)` from `src/core/search/embedding-column.ts` so Voyage/multimodal/non-default-column brains count against the column they use; errors propagate (no swallow-catch). The sync delete loop is interleaved per-batch resolve+delete using `engine.resolveSlugsByPaths` + `engine.deletePages` from `src/core/engine.ts` (73K-delete commit: ~146K SQL round-trips → ~292, closing the cascade-staleness class where one big-delete commit jammed every other source's sync); per-batch try-catch decomposes batch DELETE failures to per-slug `deletePage` fallback, unrecoverable per-slug failures land in `failedFiles`; `pagesAffected` filters to confirmed-deleted slugs. The rename loop gets the same batched slug-resolve; a rename whose destination is un-syncable folds the source path into the delete set so the old page can't orphan (#1970). An entry-time bookmark-reachability guard distinguishes a gc'd anchor (`cat-file` fails → `performFullSync`) from a history-rewrite anchor that is merely no-longer-an-ancestor but still on disk: the latter is diffed tree-to-tree directly (`git diff lastCommit..pin` is an endpoint-tree compare, ancestry not required) so a force-push / `master`→`main` consolidation imports only the real delta instead of re-walking the whole tree forever (#1970); an oversized or failed diff degrades to `performFullSync`. `performFullSync` is itself authoritative for deletes — after an advancing full import it purges file-backed pages (`source_path != null` AND strategy-aware `isSyncable`) whose source file no longer exists, sparing `put_page`/manual pages (null `source_path`) and metafiles. The stale-file decision routes through the pure, exported `planReconcileDeletes(rows, currentFiles, isSyncablePath)`: it normalizes path separators on both sides of the membership test (a Windows `path.relative` backslash path vs a git-derived forward-slash `source_path` would otherwise mark every page stale and wipe the source) and computes a mass-delete signal — when the reconcile would delete more than `MASS_RECONCILE_RATIO` (50%) of the file-backed pages the strategy manages, on a source holding more than `MASS_RECONCILE_MIN_PAGES` (20) of them, the delete is REFUSED with a loud stderr warning (that shape is almost always a path-comparison bug or the wrong repo path, not a genuine bulk deletion); `GBRAIN_ALLOW_MASS_RECONCILE=1` restores the unguarded delete for genuinely intended bulk removals. Pinned by `test/sync-reconcile-mass-delete.test.ts`. `resolveSlugByPathOrSourcePath` at `sync.ts:267` delegates to `engine.resolveSlugsByPaths` when `sourceId` is set, keeping legacy `executeRaw` fallback for the no-sourceId path. `failedFiles` is hoisted to the top of `performSyncInner` so both delete-decompose and import loops feed the same bookmark gate. The cost gate is the shared `runInlineCostGate` (one implementation on BOTH the `--all` and single-source paths; runs at the command layer, never inside `performSync`), mode-aware via `willEmbedSynchronously` + posture-aware `shouldBlockSync` from `src/core/embedding.ts` (#2139). The DEFERRED path (v2 on, parallel) is INFORMATIONAL (embedding goes to per-source `embed-backfill` jobs with their own `$X/source/24h` cap, default $25 via `SPEND_CAP_CONFIG_KEY` from `embed-backfill-submit.ts`; prints the cap + backlog + queued-job count, NEVER exits 2). The INLINE path (v2 off, or `--serial`) gates on the DELTA estimate vs `sync.cost_gate_min_usd` (default $0.50): below floor proceeds; above floor in a TTY prompts `[y/N]`; above floor in a non-TTY/`--json` session AUTO-DEFERS embeds to capped backfill jobs and exits 0 (NEVER exit 2 — the wedged-cron fix); `spend.posture=tokenmax` makes it informational and proceeds inline. The estimate MIRRORS EXECUTION instead of pricing the whole tree: `estimateInlineNewTokens` routes through the shared `computeSyncDelta` (`src/core/sync-delta.ts`) — fetch-first against `origin/`, prices only the committed delta (a dirty-but-caught-up tree → $0), with a full-tree CEILING only on the fail-open rungs (chunker drift, first sync, git-unavailable) honestly labeled; `--full` adds the stale backlog (full sync sweeps it inline). Return shape carries `estimateKind: 'delta'|'ceiling'|'mixed'|'unchanged'` + `ceilingReasons`. Helpers `resolveCostGateFloorUsd(engine)` + `resolveBackfillCapUsd(engine)` resolve via `parseUsdLimit` (`off`/`unlimited` → `Infinity`; floor accepts `0` = block-on-any-spend). JSON envelopes carry `mode` + `gate` discriminators (`dry_run | deferred_notice | below_floor | auto_deferred_embeds | posture_tokenmax`) + a paste-ready `hint`; `Infinity` floors/caps render as the string `'unlimited'` (never raw, which JSON-serializes to `null`); `SyncStatusReportSource` gains `backfill_queued`/`backfill_active`/`backfill_last_completed_at`; cost previews read `getEmbeddingModelName()` (no hardcoded OpenAI). Format splits on the explicit `--json` flag only (human text otherwise). `SyncOpts.noSchemaPack` (CLI `--no-schema-pack`, threaded through `performSync` AND `syncOneSource`) skips `loadActivePack` so pages fall back to legacy prefix typing — an escape hatch when a suspect pack regex wedges a sync. A per-file BEGIN heartbeat `if (process.env.GBRAIN_SYNC_TRACE) serr('[sync] begin import: ')` fires BEFORE `importFile` (the `progress.tick` fires only AFTER) so a stuck file is a begin-line with no matching completion. Triage doc: `docs/architecture/serve-sync-concurrency.md` (PGLite single-writer serve↔sync contention + the `GBRAIN_SYNC_TRACE` + `--no-schema-pack` recipes). Pinned by `test/e2e/sync-status-pglite.test.ts` (IRON-RULE: PGLite seeds 2 sources × pages × chunks, soft-deletes 1 page, archives 1 source, validates the SQL excludes both AND uses the active embedding column), `test/sync-cost-gate.serial.test.ts`, `test/sync-cost-preview.test.ts`. Runaway-sync protection: `resolveSyncHardDeadline(args, {isTty, env, defaultNonTtySec?})` resolves a wall-clock hard deadline (precedence `--no-hard-deadline` > `--hard-deadline ` > `--timeout `(non-`--all`, which auto-arms the backstop) > `GBRAIN_SYNC_MAX_RUNTIME_SECONDS` env > non-TTY default 3600s > none; `HARD_DEADLINE_GRACE_SEC=30`). `src/cli.ts` installs the out-of-band watchdog (see `src/core/process-watchdog.ts`) for the sync command BEFORE `connectEngine` and disposes it in the dispatch `finally`, so even an event-loop-starved sync — or a connect-phase hang — is SIGTERM-then-SIGKILLed by the deadline instead of orphaning under cron. `runSync` registers a SIGINT handler that aborts an interrupt `AbortController` composed via `composeAbortSignals(...)` (an `AbortSignal.any` wrapper over the defined signals) with the per-source `--timeout` signal, so Ctrl-C returns a clean `partial` and releases the lock through the normal `finally` (process-cleanup.ts owns SIGTERM lock-release; the watchdog owns the hard kill). `withRefreshingLock` `unref()`s its refresh `setInterval`. The spin's own root cause is not yet pinned (leading lead is catastrophic-backtracking in a pack link-inference regex, bounded by the redos-guard); the watchdog heartbeat plus the existing `[gbrain phase]` breadcrumbs are the diagnosis surface. Pinned by `test/sync-hard-deadline.test.ts` (resolution precedence + `composeAbortSignals`). Monorepo subdir sources (#753/#774): `--src-subpath ` (or a repo path that IS a subdir — auto-discovery via `discoverGitRoot`, i.e. `git rev-parse --show-toplevel`) splits the repo path into `gitContextRoot` (all git ops: pull/diff/rev-parse/cat-file) and `syncScopeRoot` (walk/import/delete/rename scope); scoped syncs use git-root-relative slugs + `source_path` (full sync threads `slugRoot` into `runImport`) so full and incremental agree; NAV-1/NAV-2 realpath containment rejects `../`-traversal and symlinked scopes resolving outside the repo BEFORE any git op, and a per-file realpath guard (`isPathSafe`) refuses symlink-escape files in the incremental drain and rename reimport (fail-closed into `failedFiles`, so the bookmark can't advance past an escape); the full-sync reconcile is scope-restricted so a scoped sync never sweeps out-of-scope pages. `--exclude ` (repeatable) filters scope-relative paths in both full and incremental paths; exclusion never deletes previously-imported pages (conservative, matching the #1433 metafile posture); an all-excluded run warns loudly (NAV-4). Both flags are single-invocation only (rejected with `--all`; register the subdir as the source local_path instead). `.gitignore` management resolves to the git root via `manageGitignoreAtGitRoot`. Pinned by `test/sync-monorepo.test.ts`. -- `src/commands/import.ts` — `gbrain import` CLI + `runImport` library entrypoint. Uses a path-set checkpoint via `src/core/import-checkpoint.ts` (the walk still applies `sortNewestFirst()` for embed-cost ordering, but checkpoint correctness no longer depends on sort order). A file enters `completed: Set` only when its `processFile` returns success (including content-hash short-circuit no-ops); failed files never enter the set so the next run retries them automatically with no manual `~/.gbrain/import-checkpoint.json` delete. This closes three classes: parallel-import-with-slow-worker dropping the slow file on crash-resume (the slow file isn't in `completed` until its own `processFile` resolves), failed-file-bumps-counter-past-itself (failures don't add to `completed`), and sort-flip-drops-newest-N-on-cross-version-resume (order is no longer part of the checkpoint). Old positional checkpoints are detected and discarded with a stderr line on first resume (re-walking is cheap because `content_hash` short-circuits unchanged files). Checkpoint persists every 100 successful adds, not every 100 processed files. The `managedBookmark` opt (set by `performFullSync` when `runImport` is the full-sync engine) suppresses `runImport`'s own `sync.last_commit` advance so the shared `applySyncFailureGate` (`src/core/sync-failure-ledger.ts`) owns the bookmark + failure-ledger gating on that path — one gate decides advance/block/auto-skip across both sync paths. Pinned by `test/import-checkpoint.test.ts` + `test/import-resume.test.ts` (incl. the SLUG_MISMATCH retry regression). `runImport` opts also carry `exclude` (glob filter over dir-relative paths, threaded by `performFullSync` for `sync --exclude`; warns when every file is excluded — NAV-4) and `slugRoot` (slug/`source_path` base for monorepo subdir syncs, #753/#774; the resume checkpoint stays dir-relative per `resumeFilter`'s contract). -- `src/core/import-checkpoint.ts` — `loadCheckpoint(brainDir)`, `saveCheckpoint(brainDir, completed)`, `resumeFilter(files, completed, brainDir)`, `clearCheckpoint()`, plus the `ImportCheckpoint` type. Path-set format `{schema_version, brainDir, completed: string[]}`. Atomic write via `.tmp` + `rename()` so a mid-write crash never leaves a partial JSON. `loadCheckpoint` returns `null` on: missing file, malformed JSON, brainDir mismatch (ran against a different brain), and the old positional format (logged to stderr before discard). `resumeFilter` returns `{toProcess, skippedCount}` — pure, no I/O, deterministic. `clearCheckpoint` is no-op-on-missing for clean-exit cleanup. Honors `GBRAIN_HOME` via `gbrainPath()` so `withEnv({GBRAIN_HOME: tmpdir})` test isolation works without monkey-patching fs. Best-effort persistence — `saveCheckpoint` logs warnings on write errors but never throws. +- `src/core/cycle.ts` + `src/core/cycle/extract-atoms.ts` + `src/core/cycle/synthesize-concepts.ts` + `src/commands/extract.ts` + `src/commands/doctor.ts` + `src/core/op-checkpoint.ts` extensions — six daily-driver ops fixes. (1) Batch idempotency: `atomsExistingForHashes(engine, sourceId, hashes[])` (exported from `src/core/cycle/extract-atoms.ts`) replaces the per-hash loop (7K individual queries) with one batched SQL roundtrip returning already-extracted `content_hash16` values; fail-open (SQL error → empty set, extraction proceeds); powered by migration v104 `pages_atom_source_hash_idx` (partial expression index on `frontmatter->>'source_hash'` for atom rows where `deleted_at IS NULL`; Postgres `CREATE INDEX CONCURRENTLY` with invalid-remnant pre-drop, PGLite plain). (2) Shorter cycle lock TTL + active in-phase refresh: `LOCK_TTL_MINUTES = 5` (was 30); `buildYieldDuringPhase(lock, outer)` (exported, with `LockHandle`) calls `lock.refresh()` + any external hook on every fire, throttled to 30s via `maybeYield`, firing both in the main loop AND immediately after every `await chat(...)`; `synthesize_concepts` uses the same throttled hook. A crashed cycle releases its lock 6x faster while a healthy long-running cycle keeps it alive (residual: a single `await chat()` past 5 min can expire the lock mid-await — TODO-OPS-2). (3) Progress wiring: `progress?: ProgressReporter` opt on `ExtractAtomsOpts` and `SynthesizeConceptsOpts`; cycle.ts passes its phase-level reporter down (NOT a child reporter, which would collide on `cycle.extract_atoms.extract_atoms.work`); phases only call `tick()`/`heartbeat()`, cycle.ts owns `start()`/`finish()`. (4) `by-mention` resume: `mentionsFingerprint({source, type, since, gazetteerHash})` in `src/core/op-checkpoint.ts` — the gazetteer hash is load-bearing (adding entity pages mid-pause shifts the hash → new fingerprint → fresh scan against the new gazetteer, never silent skip); `gbrain extract links --by-mention` resumes via `op_checkpoints` with `flushAndCheckpoint` ordering (links flush to DB FIRST, page keys commit to checkpoint SECOND, persist THIRD, so a crash mid-batch leaves the page un-checkpointed and resume re-scans it); persist every 1000 items OR 30s; clean exit clears the checkpoint; `--dry-run` skips both load and write. (5) `sync_consolidation` doctor check (multi-source brains see a paste-ready `gbrain sync --all --parallel 4 --workers 4 --skip-failed`; single-source "not applicable"; SQL errors return `warn` via the check's own try/catch). (6) Test-isolation: `test/cycle-last-full-cycle-at.test.ts` + `test/schema-cli.test.ts` use per-test `GBRAIN_HOME=tempdir`. Pinned by `test/cycle/extract-atoms-batch.test.ts`, `test/cycle/cycle-lock-ttl.test.ts` (regression pin on `LOCK_TTL_MINUTES === 5`), `test/op-checkpoint-mentions-fingerprint.test.ts`, `test/cycle/extract-atoms-progress.test.ts`, `test/cycle/synthesize-concepts-progress.test.ts`, `test/cycle/yield-during-phase-refresh.test.ts`, `test/cycle/yield-during-phase-throttle.test.ts`, `test/extract-by-mention-resume.test.ts`, `test/doctor-sync-consolidation.test.ts`. Companion `sync --all` recipe block in `skills/cron-scheduler/SKILL.md`. `synthesize_concepts` writes concept pages through `importFromContent` (#2163: the same parse→chunk→embed pipeline put_page uses, with put_page's `isAvailable('embedding')` → `noEmbed` gate) so `concepts/` pages carry `content_chunks` + embeddings and are reachable by retrieval (where `source-boost.ts` weights them 1.3×). +- `src/commands/sync.ts` — `gbrain sync` CLI + the `performSync` / `performFullSync` library entrypoints (consumed by the autopilot cycle and the Minion sync handler). `performSync` runs under a writer lock: per-source `gbrain-sync:` whenever `opts.sourceId` is set, wrapped in `withRefreshingLock` from `src/core/db-lock.ts` so long-running sources (250K+ chunks) don't lose the lock at the 30-min TTL mid-run; the bare no-source path uses the SAME refreshing lock (#1794 — it was previously a non-refreshing `tryAcquireDbLock`, stealable mid-run during an incident); `SyncOpts.lockId?: string` is the explicit override. The lock refresh AND its health probe route through the DIRECT session pool so Supavisor transaction-pool exhaustion (`EMAXCONNSESSION`) can't kill renewal; takeover is heartbeat-aware (it will NOT steal a holder whose `last_refreshed_at` is within `GBRAIN_LOCK_STEAL_GRACE_SECONDS`, defending an alive-but-starved holder); the import loop yields the event loop every `GBRAIN_SYNC_YIELD_EVERY` files (`setTimeout(0)`, not `setImmediate` — Bun starves the timers phase) so the refresh `setInterval` heartbeat fires mid-import. This lock-identity invariant prevents a `sync --all` per-source worker racing `sync --source foo` on the global lock from corrupting the same source. `performSync` throws a typed `SyncLockBusyError` when the writer lock is held; the Minion `sync` handler (`src/commands/jobs.ts`) catches it and marks the job SKIPPED (not failed) so a cron/autopilot tick defers to the holder without polluting crash metrics. `performSyncInner` is RESUMABLE (incremental path): it drains a PINNED target commit (`lastCommit..pin`), banking drained file paths via `appendCompleted` (append-only delta into the `op_checkpoint_paths` child table, migration v115 — one row per path, O(delta) not the old O(N²) full-array rewrite), keyed by `syncFingerprint({sourceId, lastCommit})` from `src/core/op-checkpoint.ts` (paths under `op:'sync'`; the pinned target under `op:'sync-target'`), and advances `last_commit`/`last_sync_at` ONLY at full import completion. Checkpoint writes route through the DIRECT session pool + bounded retry so they survive `EMAXCONNSESSION`; the flush cadence is first-file then every `GBRAIN_SYNC_CHECKPOINT_EVERY` (default 1000) files OR `GBRAIN_SYNC_CHECKPOINT_SECONDS` (default 10s), with a race-safe `pendingCheckpointPaths` delta (single-flight swap, re-merge on failure) under parallel workers; a SIGTERM banks the in-flight delta via a no-retry one-shot (`appendCompletedOnce`, ordered before lock release through `registerCleanup`); and sustained flush failure aborts the run with `reason:'checkpoint_unavailable'` after `GBRAIN_SYNC_MAX_CHECKPOINT_FAILURES` consecutive fails rather than importing work it can never bank (every partial/blocked exit logs the banked-file count). A sync killed mid-import banks its progress, leaves the anchor unmoved (the source stays correctly stale to the autopilot scheduler — `last_sync_at` is never bumped on a partial), and the next run `resumeFilter`s the same fixed diff to skip done files. The pin is the checkpoint's stored target when still reachable from HEAD, else live HEAD (a history rewrite / reset re-pins); completion advances to the pin, NOT live HEAD, so commits landing past the pin are a clean next-sync diff — this closes the cross-run staleness window. After import a pin-reachability gate (`git merge-base --is-ancestor pin HEAD`) replaces the old strict head-drift gate: forward commits on top of the pin (e.g. a background enrich process committing to the same repo every ~2 min) are SAFE and no longer block the run; only a real rewrite (pin not an ancestor of HEAD) blocks and discards the checkpoint. A file added in `lastCommit..pin` but gone from disk (deleted by a commit after the pin) is SKIPPED and marked completed, not recorded as a failure. Downstream extract/facts/embed are size-gated: inline only for `totalChanges <= 100`; large syncs defer to the resumable `extract --stale` watermark + `embed --stale`/backfill + the facts cycle phases, so a 44K-page facts/embed pass never re-blocks import convergence (sync convergence == import convergence). Worker engines wrap in try/finally so disconnect always fires; both PGLite-detection sites use `engine.kind === 'pglite'`. CLI accepts `--workers N` (alias `--concurrency N`) validated via `parseWorkers` (explicit bypasses the file-count floor; auto path defers to `autoConcurrency()`). The newest-first descending-lex order uses `sortNewestFirst(addsAndMods)` from `src/core/sort-newest-first.ts` (shared with `gbrain import`). `gbrain sync --all` runs a continuous worker pool: `parseWorkers`-validated `--parallel N` (default `min(sourceCount, --workers, DEFAULT_PARALLEL_SOURCES=4)`), long-lived async workers pulling from a shared FIFO queue (no head-of-line blocking), per-source `withSourcePrefix(src.id, ...)` so every `slog`/`serr` line carries `[]`; `--skip-failed`/`--retry-failed` are scoped per source (`acknowledgeFailures(sourceId)`; `--all` acks every source, single-source acks only its own) and run UNDER parallel — the #1939 failure ledger is per-`(source_id, path)` and serialized through `withLedgerLock`, so the old "not supported under parallel, re-run with --serial" refusal is lifted (#2139, which also removed the forcing-function that pushed recovery syncs to `--serial` and thus armed the inline cost gate); a connection-budget stderr warning fires when `parallel × workers × 2 > 16` (the `× 2 per-file pool` factor: each per-file worker opens its own `PostgresEngine` with `poolSize=2`). Exports `resolveParallelism`, `syncOneSource`, `buildSyncStatusReport`, `printSyncStatusReport`, `SyncStatusReport` back the `gbrain sources status` dashboard. `--json` envelope `{schema_version: 1, sources, parallel, ok_count, error_count}` on stdout; human banners route to stderr via `humanSink` so `jq` parses cleanly. Exit matrix: 0 all ok, 1 any error. (The non-TTY cost gate no longer exits 2 — it auto-defers; #2139.) The dashboard SQL is `content_chunks ch JOIN pages pg ON pg.id = ch.page_id WHERE pg.deleted_at IS NULL` with `archived = false` at the caller; embedding column resolved via `resolveEmbeddingColumn(undefined, cfg)` from `src/core/search/embedding-column.ts` so Voyage/multimodal/non-default-column brains count against the column they use; errors propagate (no swallow-catch). The sync delete loop is interleaved per-batch resolve+delete using `engine.resolveSlugsByPaths` + `engine.deletePages` from `src/core/engine.ts` (73K-delete commit: ~146K SQL round-trips → ~292, closing the cascade-staleness class where one big-delete commit jammed every other source's sync); per-batch try-catch decomposes batch DELETE failures to per-slug `deletePage` fallback, unrecoverable per-slug failures land in `failedFiles`; `pagesAffected` filters to confirmed-deleted slugs. The rename loop gets the same batched slug-resolve; a rename whose destination is un-syncable folds the source path into the delete set so the old page can't orphan (#1970). An entry-time bookmark-reachability guard distinguishes a gc'd anchor (`cat-file` fails → `performFullSync`) from a history-rewrite anchor that is merely no-longer-an-ancestor but still on disk: the latter is diffed tree-to-tree directly (`git diff lastCommit..pin` is an endpoint-tree compare, ancestry not required) so a force-push / `master`→`main` consolidation imports only the real delta instead of re-walking the whole tree forever (#1970); an oversized or failed diff degrades to `performFullSync`. `performFullSync` is itself authoritative for deletes — after an advancing full import it purges file-backed pages (`source_path != null` AND strategy-aware `isSyncable`) whose source file no longer exists, sparing `put_page`/manual pages (null `source_path`) and metafiles. The stale-file decision routes through the pure, exported `planReconcileDeletes(rows, currentFiles, isSyncablePath)`: it normalizes path separators on both sides of the membership test (a Windows `path.relative` backslash path vs a git-derived forward-slash `source_path` would otherwise mark every page stale and wipe the source) and computes a mass-delete signal — when the reconcile would delete more than `MASS_RECONCILE_RATIO` (50%) of the file-backed pages the strategy manages, on a source holding more than `MASS_RECONCILE_MIN_PAGES` (20) of them, the delete is REFUSED with a loud stderr warning (that shape is almost always a path-comparison bug or the wrong repo path, not a genuine bulk deletion); `GBRAIN_ALLOW_MASS_RECONCILE=1` restores the unguarded delete for genuinely intended bulk removals. Pinned by `test/sync-reconcile-mass-delete.test.ts`. Below the valve, stale pages are partitioned by git history via exported `listEverCommittedPaths(repoPath)` (one `git log --all --no-renames --diff-filter=A --name-only` pass; null on non-git dirs → unchanged behavior): a stale path that EVER existed in history was genuinely deleted → reconciled; a path with NO history is DB-only write-through (never committed/pushed, e.g. lost to a fresh clone) → the page is KEPT and its markdown re-exported to the working tree via `writePageThrough`, with a stderr hint to commit it (#2426 — "absent from git" is the symptom of the missing write-through commit, not evidence the content is disposable). Pinned by `test/sync-reconcile-db-only.serial.test.ts`. `resolveSlugByPathOrSourcePath` at `sync.ts:267` delegates to `engine.resolveSlugsByPaths` when `sourceId` is set, keeping legacy `executeRaw` fallback for the no-sourceId path. `failedFiles` is hoisted to the top of `performSyncInner` so both delete-decompose and import loops feed the same bookmark gate. The cost gate is the shared `runInlineCostGate` (one implementation on BOTH the `--all` and single-source paths; runs at the command layer, never inside `performSync`), mode-aware via `willEmbedSynchronously` + posture-aware `shouldBlockSync` from `src/core/embedding.ts` (#2139). The DEFERRED path (v2 on, parallel) is INFORMATIONAL (embedding goes to per-source `embed-backfill` jobs with their own `$X/source/24h` cap, default $25 via `SPEND_CAP_CONFIG_KEY` from `embed-backfill-submit.ts`; prints the cap + backlog + queued-job count, NEVER exits 2). The INLINE path (v2 off, or `--serial`) gates on the DELTA estimate vs `sync.cost_gate_min_usd` (default $0.50): below floor proceeds; above floor in a TTY prompts `[y/N]`; above floor in a non-TTY/`--json` session AUTO-DEFERS embeds to capped backfill jobs and exits 0 (NEVER exit 2 — the wedged-cron fix); `spend.posture=tokenmax` makes it informational and proceeds inline. The estimate MIRRORS EXECUTION instead of pricing the whole tree: `estimateInlineNewTokens` routes through the shared `computeSyncDelta` (`src/core/sync-delta.ts`) — fetch-first against `origin/`, prices only the committed delta (a dirty-but-caught-up tree → $0), with a full-tree CEILING only on the fail-open rungs (chunker drift, first sync, git-unavailable) honestly labeled; `--full` adds the stale backlog (full sync sweeps it inline). Return shape carries `estimateKind: 'delta'|'ceiling'|'mixed'|'unchanged'` + `ceilingReasons`. Helpers `resolveCostGateFloorUsd(engine)` + `resolveBackfillCapUsd(engine)` resolve via `parseUsdLimit` (`off`/`unlimited` → `Infinity`; floor accepts `0` = block-on-any-spend). JSON envelopes carry `mode` + `gate` discriminators (`dry_run | deferred_notice | below_floor | auto_deferred_embeds | posture_tokenmax`) + a paste-ready `hint`; `Infinity` floors/caps render as the string `'unlimited'` (never raw, which JSON-serializes to `null`); `SyncStatusReportSource` gains `backfill_queued`/`backfill_active`/`backfill_last_completed_at`; cost previews read `getEmbeddingModelName()` (no hardcoded OpenAI). Format splits on the explicit `--json` flag only (human text otherwise). `SyncOpts.noSchemaPack` (CLI `--no-schema-pack`, threaded through `performSync` AND `syncOneSource`) skips `loadActivePack` so pages fall back to legacy prefix typing — an escape hatch when a suspect pack regex wedges a sync. A per-file BEGIN heartbeat `if (process.env.GBRAIN_SYNC_TRACE) serr('[sync] begin import: ')` fires BEFORE `importFile` (the `progress.tick` fires only AFTER) so a stuck file is a begin-line with no matching completion. Triage doc: `docs/architecture/serve-sync-concurrency.md` (PGLite single-writer serve↔sync contention + the `GBRAIN_SYNC_TRACE` + `--no-schema-pack` recipes). Pinned by `test/e2e/sync-status-pglite.test.ts` (IRON-RULE: PGLite seeds 2 sources × pages × chunks, soft-deletes 1 page, archives 1 source, validates the SQL excludes both AND uses the active embedding column), `test/sync-cost-gate.serial.test.ts`, `test/sync-cost-preview.test.ts`. Runaway-sync protection: `resolveSyncHardDeadline(args, {isTty, env, defaultNonTtySec?})` resolves a wall-clock hard deadline (precedence `--no-hard-deadline` > `--hard-deadline ` > `--timeout `(non-`--all`, which auto-arms the backstop) > `GBRAIN_SYNC_MAX_RUNTIME_SECONDS` env > non-TTY default 3600s > none; `HARD_DEADLINE_GRACE_SEC=30`). `src/cli.ts` installs the out-of-band watchdog (see `src/core/process-watchdog.ts`) for the sync command BEFORE `connectEngine` and disposes it in the dispatch `finally`, so even an event-loop-starved sync — or a connect-phase hang — is SIGTERM-then-SIGKILLed by the deadline instead of orphaning under cron. `runSync` registers a SIGINT handler that aborts an interrupt `AbortController` composed via `composeAbortSignals(...)` (an `AbortSignal.any` wrapper over the defined signals) with the per-source `--timeout` signal, so Ctrl-C returns a clean `partial` and releases the lock through the normal `finally` (process-cleanup.ts owns SIGTERM lock-release; the watchdog owns the hard kill). `withRefreshingLock` `unref()`s its refresh `setInterval`. The spin's own root cause is not yet pinned (leading lead is catastrophic-backtracking in a pack link-inference regex, bounded by the redos-guard); the watchdog heartbeat plus the existing `[gbrain phase]` breadcrumbs are the diagnosis surface. Pinned by `test/sync-hard-deadline.test.ts` (resolution precedence + `composeAbortSignals`). Monorepo subdir sources (#753/#774): `--src-subpath ` (or a repo path that IS a subdir — auto-discovery via `discoverGitRoot`, i.e. `git rev-parse --show-toplevel`) splits the repo path into `gitContextRoot` (all git ops: pull/diff/rev-parse/cat-file) and `syncScopeRoot` (walk/import/delete/rename scope); scoped syncs use git-root-relative slugs + `source_path` (full sync threads `slugRoot` into `runImport`) so full and incremental agree; NAV-1/NAV-2 realpath containment rejects `../`-traversal and symlinked scopes resolving outside the repo BEFORE any git op, and a per-file realpath guard (`isPathSafe`) refuses symlink-escape files in the incremental drain and rename reimport (fail-closed into `failedFiles`, so the bookmark can't advance past an escape); the full-sync reconcile is scope-restricted so a scoped sync never sweeps out-of-scope pages. `--exclude ` (repeatable) filters scope-relative paths in both full and incremental paths; exclusion never deletes previously-imported pages (conservative, matching the #1433 metafile posture); an all-excluded run warns loudly (NAV-4). +- `src/commands/import.ts` — `gbrain import` CLI + `runImport` library entrypoint. Uses a path-set checkpoint via `src/core/import-checkpoint.ts` (the walk still applies `sortNewestFirst()` for embed-cost ordering, but checkpoint correctness no longer depends on sort order). A file enters `completed: Set` only when its `processFile` returns success (including content-hash short-circuit no-ops); failed files never enter the set so the next run retries them automatically with no manual `~/.gbrain/import-checkpoint.json` delete. This closes three classes: parallel-import-with-slow-worker dropping the slow file on crash-resume (the slow file isn't in `completed` until its own `processFile` resolves), failed-file-bumps-counter-past-itself (failures don't add to `completed`), and sort-flip-drops-newest-N-on-cross-version-resume (order is no longer part of the checkpoint). Old positional checkpoints are detected and discarded with a stderr line on first resume (re-walking is cheap because `content_hash` short-circuits unchanged files). Checkpoint persists every 100 successful adds, not every 100 processed files. The `managedBookmark` opt (set by `performFullSync` when `runImport` is the full-sync engine) suppresses `runImport`'s own `sync.last_commit` advance so the shared `applySyncFailureGate` (`src/core/sync-failure-ledger.ts`) owns the bookmark + failure-ledger gating on that path — one gate decides advance/block/auto-skip across both sync paths. Pinned by `test/import-checkpoint.test.ts` + `test/import-resume.test.ts` (incl. the SLUG_MISMATCH retry regression). `collectSyncableFiles`' shared emit filter `isCollectibleForWalker` applies the SAME segment-level `pruneDir` gate as incremental sync's `classifySync` — load-bearing for the `git ls-files` fast path, which enumerates tracked files under dot-dirs/vendored trees that the FS walk never descends into; without it `sync --full` imported (and resurrected soft-deleted) pages incremental sync excludes (#2607). Pinned by `test/import-git-fastpath-prune.test.ts`. `runImport` opts also carry `exclude` (glob filter over dir-relative paths, threaded by `performFullSync` for `sync --exclude`; warns when every file is excluded — NAV-4) and `slugRoot` (slug/`source_path` base for monorepo subdir syncs, #753/#774; the resume checkpoint stays dir-relative per `resumeFilter`'s contract).- `src/core/import-checkpoint.ts` — `loadCheckpoint(brainDir)`, `saveCheckpoint(brainDir, completed)`, `resumeFilter(files, completed, brainDir)`, `clearCheckpoint()`, plus the `ImportCheckpoint` type. Path-set format `{schema_version, brainDir, completed: string[]}`. Atomic write via `.tmp` + `rename()` so a mid-write crash never leaves a partial JSON. `loadCheckpoint` returns `null` on: missing file, malformed JSON, brainDir mismatch (ran against a different brain), and the old positional format (logged to stderr before discard). `resumeFilter` returns `{toProcess, skippedCount}` — pure, no I/O, deterministic. `clearCheckpoint` is no-op-on-missing for clean-exit cleanup. Honors `GBRAIN_HOME` via `gbrainPath()` so `withEnv({GBRAIN_HOME: tmpdir})` test isolation works without monkey-patching fs. Best-effort persistence — `saveCheckpoint` logs warnings on write errors but never throws. - `src/core/sort-newest-first.ts` — single source of truth for the descending-lex sort that `gbrain import` and `gbrain sync` both apply. Mutates in place (Array.prototype.sort semantics), returns the same array reference for fluent chaining. Empty/single-element inputs short-circuit. Future ordering changes flip one line here instead of touching two CLI commands. Pinned by `test/sort-newest-first.test.ts` (descending order, mixed prefixes, empty, single-element, in-place-mutation contract). -- `src/core/cycle.ts` — brain maintenance cycle primitive (9 phases). `runCycle(engine: BrainEngine | null, opts: CycleOpts): Promise` composes phases in semantic order: lint → backlinks → sync → synthesize → extract → patterns → recompute_emotional_weight → embed → orphans. `synthesize` runs after sync (cross-references see a fresh brain) and before extract (auto-link materializes its writes); `patterns` runs after extract so it reads a fresh graph (subagent put_page sets `ctx.remote=true` and skips auto-link/timeline by default, so extract is the canonical materialization); `recompute_emotional_weight` sees the union of `syncPagesAffected` + `synthesizeWrittenSlugs` incrementally, or all pages when neither anchor is set (full backfill via `gbrain dream --phase recompute_emotional_weight`). `CycleReport.schema_version: "1"` is stable; `totals` is additive (`pages_emotional_weight_recomputed`, `transcripts_processed`, `synth_pages_written`, `patterns_written`). Three callers: `gbrain dream` CLI, `gbrain autopilot` daemon inline path, the Minions `autopilot-cycle` handler. Coordination via `gbrain_cycle_locks` DB table + `~/.gbrain/cycle.lock` file lock with PID-liveness for PGLite. `yieldBetweenPhases` runs between phases; `yieldDuringPhase` is in-phase keepalive (synthesize/patterns renew the cycle-lock TTL during long waits). Engine nullable; lock-skip on read-only phase selections. `CycleOpts.signal?: AbortSignal` propagates the worker's abort signal with `checkAborted()` between every phase. `runPhaseSync` returns `pagesAffected` via `SyncPhaseResult` (threaded to `runPhaseExtract` as the 4th arg) and takes `willRunExtractPhase: boolean` setting `noExtract: phases.includes('extract')` so `gbrain dream --phase sync` doesn't silently lose extraction. `resolveSourceForDir(engine, brainDir)` threads `sourceId` to `performSync()` so sync reads the per-source `sources.last_commit` anchor (not the drift-prone global `config.sync.last_commit`). `CycleOpts.brainDir` is `string | null`; when null (checkout-less postgres/Supabase brain) the 6 filesystem phases (lint/backlinks/sync/synthesize/extract/patterns) skip with `details.reason: 'no_brain_dir'` and the DB-only phases run; `resolveSourceForDir` is null-tolerant. `cycleSourceId = opts.sourceId ?? resolveSourceForDir(engine, brainDir)` is the canonical per-source scope for `extract_facts`/`extract_atoms`/calibration so `gbrain dream --source repo-a` reconciles repo-a's facts even with no checkout (instead of scoping to `'default'` while stamping repo-a fresh). `deriveStatus` counts `edges_resolved`/`edges_ambiguous` as work so an edges-only cycle reports `ok` not `clean`; the `jobs.ts` `autopilot-cycle` + phase-wrapper handlers pass `null` (not `'.'`) when no repo is configured. The cycle is SPLIT for autopilot fan-out (#2194/#2227): `PHASE_SCOPE` partitions `ALL_PHASES` into `GLOBAL_PHASES` (brain-wide: embed/orphans/purge/resolve_symbol_edges/grade_takes/calibration_profile/synthesize_concepts/skillopt) and `NON_GLOBAL_PHASES` (source + mixed). Per-source `autopilot-cycle` jobs run only `NON_GLOBAL_PHASES` and stamp `last_source_cycle_at`; the single `autopilot-global-maintenance` job runs `GLOBAL_PHASES` (no `sourceId`) and stamps the brain-level `autopilot.last_global_at` config key (`LAST_GLOBAL_AT_KEY`). `last_full_cycle_at` is still written alongside `last_source_cycle_at` on a per-source success for doctor/legacy readers (no longer a gate for the brain-wide phases). Pinned by `test/dream-postgres.serial.test.ts` + `test/jobs-autopilot-cycle-braindir.serial.test.ts` + `test/autopilot-global-maintenance.test.ts`. -- `src/core/cycle/synthesize.ts` — Synthesize phase: conversation-transcript-to-brain pipeline. Reads `dream.synthesize.session_corpus_dir`, runs a cheap Haiku verdict (cached in `dream_verdicts`), then fans out one Sonnet subagent per worth-processing transcript with `allowed_slug_prefixes` (sourced from `skills/_brain-filing-rules.json` `dream_synthesize_paths.globs`). Orchestrator collects slugs from `subagent_tool_executions` (NOT `pages.updated_at`) and reverse-renders DB → markdown via `serializeMarkdown`. Cooldown via `dream.synthesize.last_completion_ts`, written ONLY on success. Idempotency key `dream:synth::`. `--dry-run` runs Haiku, skips Sonnet. Subagent never gets fs-write access. `renderPageToMarkdown` (exported) stamps `dream_generated: true` + `dream_cycle_date` into every reverse-write's frontmatter; `writeSummaryPage` does the same on the summary index — this marker is the explicit identity surface `isDreamOutput` checks in `transcript-discovery.ts`. `judgeSignificance` and `JudgeClient` are exported; `judgeSignificance` takes a `verdictModel` param loaded from `dream.synthesize.verdict_model` via `loadSynthConfig`. `splitTranscriptByBudget(content, contentHash, maxChars)` splits oversized transcripts at paragraph boundaries (`## Topic:` → `---` → `\n` ladder) using a deterministic offset seeded from the first 32 bits of `contentHash` so retries chunk identically; per-chunk char budget = `MODEL_CONTEXT_TOKENS[resolvedModel] × 0.9 × 3.5 chars/token` (non-Anthropic ids fall back to a 180K-token safe default + once-per-process stderr warn); operator overrides `dream.synthesize.max_prompt_tokens` (floor 100K, wins) and `dream.synthesize.max_chunks_per_transcript` (default 24); per-chunk subagent job/wait timeouts are `dream.synthesize.subagent_timeout_ms` / `dream.synthesize.subagent_wait_timeout_ms` (defaults 30/35 min). Per-chunk idempotency keys `dream:synth:::cof`; single-chunk transcripts preserve the legacy `dream:synth::` key byte-for-byte so existing brains skip with `already_synthesized_legacy_single_chunk` instead of re-spending Sonnet. `collectChildPutPageSlugs` raw-fetches every (job_id, slug) pair (not `SELECT DISTINCT`) and rewrites bare-hash6 slugs to `-c` for chunked children (orchestrator-side, zero Sonnet trust). Cap-hit skips don't write to `dream_verdicts` so raising the cap on next run re-attempts cleanly. Bounds INITIAL prompt size only; tool-loop turn-N accumulation is caught by terminal-error classification in `subagent.ts`. Verdict routing is gateway-routed: `makeJudgeClient(verdictModel)` (exported, replacing `makeHaikuClient()`) mirrors `tryBuildGatewayClient` in `src/core/think/index.ts` — a construction-time provider/key probe returns `null` on a clear miss (unknown provider id via `resolveRecipe` AIConfigError, or Anthropic provider with no key via `hasAnthropicKey()`). The verdict loop wraps `judgeSignificance` in try/catch for `AIConfigError` so mid-run provider failures surface as per-transcript `worth=false, reasons=['gateway error: ...']` instead of crashing the phase. Canonical config key `models.dream.synthesize_verdict` (per `PER_TASK_KEYS` in `src/core/model-config.ts`); `JudgeClient` signature preserved verbatim for test-seam stability; CI guard `scripts/check-gateway-routed-no-direct-anthropic.sh` prevents reintroducing `new Anthropic()` here or in `think/index.ts`. At the queue.add boundary (lines 395-404) a conditional `anthropic:` prefix is applied ONLY when the resolved model has no colon AND starts with `claude-` (because `resolveModel` returns bare ids from `TIER_DEFAULTS`/`DEFAULT_ALIASES` and the subagent validator requires `provider:model` form) — avoids changing the shared constants which would ripple across every `resolveModel` caller. Pinned by `test/cycle/synthesize-gateway-adapter.test.ts`, `test/e2e/dream-synthesize-pglite.test.ts` (gateway-adapter mid-run AIConfigError catch), `test/cycle/regression-pr-wave-r1-r2-r4.test.ts`. +- `src/core/cycle.ts` — brain maintenance cycle primitive (9 phases). `runCycle(engine: BrainEngine | null, opts: CycleOpts): Promise` composes phases in semantic order: lint → backlinks → sync → synthesize → extract → patterns → recompute_emotional_weight → embed → orphans. `synthesize` runs after sync (cross-references see a fresh brain) and before extract (auto-link materializes its writes); `patterns` runs after extract so it reads a fresh graph (subagent put_page sets `ctx.remote=true` and skips auto-link/timeline by default, so extract is the canonical materialization); `recompute_emotional_weight` sees the union of `syncPagesAffected` + `synthesizeWrittenSlugs` incrementally, or all pages when neither anchor is set (full backfill via `gbrain dream --phase recompute_emotional_weight`). `CycleReport.schema_version: "1"` is stable; `totals` is additive (`pages_emotional_weight_recomputed`, `transcripts_processed`, `synth_pages_written`, `patterns_written`). Three callers: `gbrain dream` CLI, `gbrain autopilot` daemon inline path, the Minions `autopilot-cycle` handler. Coordination via `gbrain_cycle_locks` DB table + `~/.gbrain/cycle.lock` file lock with PID-liveness for PGLite. `yieldBetweenPhases` runs between phases; `yieldDuringPhase` is in-phase keepalive (synthesize/patterns renew the cycle-lock TTL during long waits). Engine nullable; lock-skip on read-only phase selections. `CycleOpts.signal?: AbortSignal` propagates the worker's abort signal with `checkAborted()` between every phase. `runPhaseSync` returns `pagesAffected` via `SyncPhaseResult` (threaded to `runPhaseExtract` as the 4th arg) and takes `willRunExtractPhase: boolean` setting `noExtract: phases.includes('extract')` so `gbrain dream --phase sync` doesn't silently lose extraction. `resolveSourceForDir(engine, brainDir)` threads `sourceId` to `performSync()` so sync reads the per-source `sources.last_commit` anchor (not the drift-prone global `config.sync.last_commit`). `CycleOpts.brainDir` is `string | null`; when null (checkout-less postgres/Supabase brain) the 6 filesystem phases (lint/backlinks/sync/synthesize/extract/patterns) skip with `details.reason: 'no_brain_dir'` and the DB-only phases run; `resolveSourceForDir` is null-tolerant. `cycleSourceId = opts.sourceId ?? resolveSourceForDir(engine, brainDir)` is the canonical per-source scope for `extract_facts`/`extract_atoms`/calibration — and for `synthesize` (#1586: threaded as `SynthesizePhaseOpts.sourceId` so synthesized pages land in the cycle's resolved source, not `'default'`) — so `gbrain dream --source repo-a` reconciles repo-a's facts even with no checkout (instead of scoping to `'default'` while stamping repo-a fresh). `deriveStatus` counts `edges_resolved`/`edges_ambiguous` as work so an edges-only cycle reports `ok` not `clean`; the `jobs.ts` `autopilot-cycle` + phase-wrapper handlers pass `null` (not `'.'`) when no repo is configured. The cycle is SPLIT for autopilot fan-out (#2194/#2227): `PHASE_SCOPE` partitions `ALL_PHASES` into `GLOBAL_PHASES` (brain-wide: embed/orphans/purge/resolve_symbol_edges/grade_takes/calibration_profile/synthesize_concepts/skillopt) and `NON_GLOBAL_PHASES` (source + mixed). Per-source `autopilot-cycle` jobs run only `NON_GLOBAL_PHASES` and stamp `last_source_cycle_at`; the single `autopilot-global-maintenance` job runs `GLOBAL_PHASES` (no `sourceId`) and stamps the brain-level `autopilot.last_global_at` config key (`LAST_GLOBAL_AT_KEY`). `last_full_cycle_at` is still written alongside `last_source_cycle_at` on a per-source success for doctor/legacy readers (no longer a gate for the brain-wide phases). Pinned by `test/dream-postgres.serial.test.ts` + `test/jobs-autopilot-cycle-braindir.serial.test.ts` + `test/autopilot-global-maintenance.test.ts`. +- `src/core/cycle/synthesize.ts` — Synthesize phase: conversation-transcript-to-brain pipeline. Reads `dream.synthesize.session_corpus_dir`, runs a cheap Haiku verdict (cached in `dream_verdicts`), then fans out one Sonnet subagent per worth-processing transcript with `allowed_slug_prefixes` (sourced from `skills/_brain-filing-rules.json` `dream_synthesize_paths.globs`; when `dream.synthesize.output_root` is set, `loadAllowedSlugPrefixes(outputRoot)` remaps the `wiki/`-rooted globs to the configured namespace — #2415 — and the same root drives the prompt slug templates; default 'wiki', validated against the slug grammar via the exported `loadOutputRoot`). The phase is source-scoped (#1586): cycle.ts threads `cycleSourceId` as `opts.sourceId` → each child's `SubagentHandlerData.source_id` → the subagent tool registry's `OperationContext.sourceId`, so put_page writes, collected refs, the summary page, and reverse-writes all target the cycle's resolved source ('default' when unscoped; reverse-writes for the cycle's own source land at `brainDir/.md`, foreign sources under `brainDir/.sources//`). Orchestrator collects slugs from `subagent_tool_executions` (NOT `pages.updated_at`) and reverse-renders DB → markdown via `serializeMarkdown`. Cooldown via `dream.synthesize.last_completion_ts`, written ONLY on success. Idempotency key `dream:synth::`. `--dry-run` runs Haiku, skips Sonnet. Subagent never gets fs-write access. `renderPageToMarkdown` (exported) stamps `dream_generated: true` + `dream_cycle_date` into every reverse-write's frontmatter; `writeSummaryPage` does the same on the summary index — this marker is the explicit identity surface `isDreamOutput` checks in `transcript-discovery.ts`. `stampDreamProvenance` (#2569) additionally persists the same marker into the `pages.frontmatter` JSONB row (merge via `executeRawJsonb`, raw object bound to `$N::jsonb`) for every child-written page BEFORE reverse-rendering, so generated pages are DB-queryable and a later put_page write-through (which re-renders from the DB row) can't erase the stamp. `judgeSignificance` and `JudgeClient` are exported; `judgeSignificance` takes a `verdictModel` param loaded from `dream.synthesize.verdict_model` via `loadSynthConfig`. `splitTranscriptByBudget(content, contentHash, maxChars)` splits oversized transcripts at paragraph boundaries (`## Topic:` → `---` → `\n` ladder) using a deterministic offset seeded from the first 32 bits of `contentHash` so retries chunk identically; per-chunk char budget = `MODEL_CONTEXT_TOKENS[resolvedModel] × 0.9 × 3.5 chars/token` (non-Anthropic ids fall back to a 180K-token safe default + once-per-process stderr warn); operator overrides `dream.synthesize.max_prompt_tokens` (floor 100K, wins) and `dream.synthesize.max_chunks_per_transcript` (default 24); per-chunk subagent job/wait timeouts are `dream.synthesize.subagent_timeout_ms` / `dream.synthesize.subagent_wait_timeout_ms` (defaults 30/35 min). Per-chunk idempotency keys `dream:synth:::cof`; single-chunk transcripts preserve the legacy `dream:synth::` key byte-for-byte so existing brains skip with `already_synthesized_legacy_single_chunk` instead of re-spending Sonnet. `collectChildPutPageSlugs` raw-fetches every (job_id, slug) pair (not `SELECT DISTINCT`) and rewrites bare-hash6 slugs to `-c` for chunked children (orchestrator-side, zero Sonnet trust). Cap-hit skips don't write to `dream_verdicts` so raising the cap on next run re-attempts cleanly. Bounds INITIAL prompt size only; tool-loop turn-N accumulation is caught by terminal-error classification in `subagent.ts`. Verdict routing is gateway-routed: `makeJudgeClient(verdictModel)` (exported, replacing `makeHaikuClient()`) mirrors `tryBuildGatewayClient` in `src/core/think/index.ts` — a construction-time provider/key probe returns `null` on a clear miss (unknown provider id via `resolveRecipe` AIConfigError, or Anthropic provider with no key via `hasAnthropicKey()`). The verdict loop wraps `judgeSignificance` in try/catch for `AIConfigError` so mid-run provider failures surface as per-transcript `worth=false, reasons=['gateway error: ...']` instead of crashing the phase. Canonical config key `models.dream.synthesize_verdict` (per `PER_TASK_KEYS` in `src/core/model-config.ts`); `JudgeClient` signature preserved verbatim for test-seam stability; CI guard `scripts/check-gateway-routed-no-direct-anthropic.sh` prevents reintroducing `new Anthropic()` here or in `think/index.ts`. At the queue.add boundary (lines 395-404) a conditional `anthropic:` prefix is applied ONLY when the resolved model has no colon AND starts with `claude-` (because `resolveModel` returns bare ids from `TIER_DEFAULTS`/`DEFAULT_ALIASES` and the subagent validator requires `provider:model` form) — avoids changing the shared constants which would ripple across every `resolveModel` caller. Pinned by `test/cycle/synthesize-gateway-adapter.test.ts`, `test/e2e/dream-synthesize-pglite.test.ts` (gateway-adapter mid-run AIConfigError catch), `test/cycle/regression-pr-wave-r1-r2-r4.test.ts`. - `scripts/check-gateway-routed-no-direct-anthropic.sh` — CI guard that fails the build if `src/core/cycle/synthesize.ts` or `src/core/think/index.ts` reintroduces a runtime `new Anthropic()` constructor call or a value-shaped `import Anthropic from '@anthropic-ai/sdk'` import. Type-only imports (`import type Anthropic from '@anthropic-ai/sdk'`) stay allowed for adapter types; comment lines (`//` or ` *` prefixes) are excluded so historical JSDoc doesn't false-fire. Mirrors `scripts/check-jsonb-pattern.sh`. Wired into `bun run verify` and `bun run check:all`. Extend `GUARDED_FILES` when migrating another file off direct SDK construction. -- `src/core/cycle/patterns.ts` — Patterns phase: cross-session theme detection over reflections within `dream.patterns.lookback_days` (default 30). Names a pattern only when ≥`dream.patterns.min_evidence` (default 3) reflections support it. Single Sonnet subagent; same allow-list path as synthesize. Subagent job/wait timeouts are config keys `dream.patterns.subagent_timeout_ms` / `dream.patterns.subagent_wait_timeout_ms` (defaults 30/35 min, mirroring the `dream.synthesize.*` pair). The phase status reflects the child outcome: non-`complete` outcome with zero writes → `fail` (error code `PATTERNS_CHILD_`); non-`complete` with partial writes → `warn`. Runs AFTER `extract` so the graph is fresh. +- `src/core/cycle/patterns.ts` — Patterns phase: cross-session theme detection over reflections within `dream.patterns.lookback_days` (default 30). Names a pattern only when ≥`dream.patterns.min_evidence` (default 3) reflections support it. Single Sonnet subagent; same allow-list path as synthesize (imports `loadAllowedSlugPrefixes` + `loadOutputRoot` from synthesize.ts — #2415: the reflections lookup, prompt slug templates, and allow-list all honor `dream.synthesize.output_root`, default 'wiki'). Subagent job/wait timeouts are config keys `dream.patterns.subagent_timeout_ms` / `dream.patterns.subagent_wait_timeout_ms` (defaults 30/35 min, mirroring the `dream.synthesize.*` pair). The phase status reflects the child outcome: non-`complete` outcome with zero writes → `fail` (error code `PATTERNS_CHILD_`); non-`complete` with partial writes → `warn`. Runs AFTER `extract` so the graph is fresh. - `src/core/cycle/extract-facts.ts` — extract_facts cycle phase. Fence is canonical: per-page wipe (`deleteFactsForPage`) + reinsert from `parseFactsFence` + `extractFactsFromFenceText` + `engine.insertFacts`. #1928: the per-page wipe passes `excludeSourcePrefixes: ['cli:']` so conversation facts (written by `extract-conversation-facts`, on pages with NO `## Facts` fence to recreate them from) survive the reconcile instead of being deleted-with-nothing-to-reinsert. The destructive phase no longer inherits a failed sync's full-brain walk: `slugs: []` (a real incremental no-op) is distinguished from `slugs: undefined` (full-walk intent) by presence, not length. `runPhaseExtractFacts` (cycle.ts) surfaces a `warn` (`net_fact_deletion`) when the reconcile deletes at least `NET_DELETION_WARN_FLOOR` (50) more facts than it reinserts — the exact signature of the conversation-facts wipe, which previously read as a silent `ok`. Empty-fence guard refuses when legacy rows (`row_num IS NULL AND entity_slug IS NOT NULL`) pend backfill (status: warn, hint: `gbrain apply-migrations --yes`). A phantom-redirect pre-pass runs AFTER the legacy-row guard, BEFORE the main reconcile loop: when `opts.brainDir` is set, `runPhantomRedirectPass(engine, brainDir, sourceId, dryRun)` walks unprefixed-slug pages capped by `GBRAIN_PHANTOM_REDIRECT_LIMIT` (default 50). The pass returns `touched_canonicals` — canonical slugs whose disk fence merged with phantom rows; `runExtractFacts` UNIONs them into the main reconcile slug set so canonical's DB facts derive from the merged fence in the same cycle (handles phantom-had-only-on-disk-fence). `ExtractFactsResult` gains six phantom fields: `phantomsScanned`, `phantomsRedirected`, `phantomsAmbiguous`, `phantomsSkippedDrift`, `phantomsLockBusy`, `phantomsMorePending`. Three bubble to `CycleReport.totals` (`phantoms_redirected`, `phantoms_ambiguous`, `phantoms_skipped_drift`). - `src/core/fence-shared.ts` — shared pipe-table primitives for the `## Takes` (`takes-fence.ts`) and `## Facts` (`facts-fence.ts`) fences: `parseRowCells`, `isSeparatorRow`, `stripStrikethrough`, `parseStringCell`, `escapeFenceCell`. `parseRowCells` is escape-aware: `\|` stays inside its cell and decodes back to a literal `|` (exact inverse of `escapeFenceCell`), while any other backslash passes through verbatim so existing fence text (e.g. Windows paths) is byte-stable across a render → parse cycle. This matters because the fence is canonical and reconcile is wipe-and-reinsert — a pipe inside a cell shifting the column layout would corrupt every row behind it on the next reconcile. Pinned by the escape cases in `test/facts-fence.test.ts` + the full render → parse → reconcile round-trip in `test/e2e/facts-fence-reconcile-postgres.test.ts`. - `src/core/entities/resolve.ts` — Free-form entity name → canonical slug resolution. `resolveEntitySlug(engine, source_id, raw)`: exact slug → unambiguous bare-name prefix expansion across `people/-%` + `companies/-%` → high-specificity fuzzy match for multi-token input (pg_trgm @ 0.7 threshold) → deterministic `slugify` holding fallback. Bare-name collisions never use popularity as confidence; shared-token company names below the threshold remain unresolved. Two helpers for the phantom-redirect pass: `resolvePhantomCanonical(engine, sourceId, phantomSlug)` SKIPS the exact-slug step (a phantom slug `'alice'` would exact-match itself and no-op the redirect); returns the canonical only when non-null AND contains `/`. `findPrefixCandidates(engine, sourceId, token)` is a standalone SQL query returning ALL candidates across `PREFIX_EXPANSION_DIRS` (hardcoded `['people', 'companies']`) via `slug LIKE ANY($N::text[])` over patterns `dir/token` + `dir/token-%`, cap of 10 ordered by `connection_count DESC, slug ASC`. Pinned by `test/entity-resolve.test.ts` (explicit, unique, ambiguous-person, and shared-token-company cases) plus `test/phantom-redirect.test.ts` (resolvePhantomCanonical 3 cases + findPrefixCandidates 6 cases incl. multi-dir ambiguity and the `people/aliceberg`-doesn't-match-`alice` false-positive guard). diff --git a/docs/guides/multi-language-fts.md b/docs/guides/multi-language-fts.md new file mode 100644 index 000000000..1e5a2fe03 --- /dev/null +++ b/docs/guides/multi-language-fts.md @@ -0,0 +1,97 @@ +# Multi-language full-text search + +GBrain's keyword search arm uses Postgres full-text search (tsvector/tsquery). +The tokenizer language is configurable via the `GBRAIN_FTS_LANGUAGE` +environment variable. Default: `english`. + +## How it works + +Postgres text-search configurations control stemming and stop-word removal. +`GBRAIN_FTS_LANGUAGE` is read by `src/core/fts-language.ts` and applied on +both sides of the search: + +- **Query side** — `websearch_to_tsquery('', $query)` in both engines + (Postgres and PGLite). +- **Write side** — the `update_page_search_vector` and + `update_chunk_search_vector` trigger functions that populate + `pages.search_vector` and `content_chunks.search_vector`. + +The value is validated against `/^[a-z][a-z0-9_]*$/` before it is ever +interpolated into SQL (tsvector functions don't accept parameterized config +names). Invalid values fall back to `english` with a warning. + +## Built-in languages + +Set the env var to any configuration your Postgres instance ships: + +```bash +export GBRAIN_FTS_LANGUAGE=portuguese +export GBRAIN_FTS_LANGUAGE=spanish +export GBRAIN_FTS_LANGUAGE=german +``` + +List what's available: + +```sql +SELECT cfgname FROM pg_ts_config; +``` + +PGLite (the embedded default engine) ships the same built-in snowball +configurations as stock Postgres. + +## First install vs. changing language later + +On first install (or upgrade), the `configurable_fts_language` schema +migration reads `GBRAIN_FTS_LANGUAGE` and stamps the trigger functions with +that language. After the migration has run, changing the env var alone does +NOT retokenize existing rows — the migration shows as applied and is skipped. +Use the explicit command: + +```bash +export GBRAIN_FTS_LANGUAGE=portuguese +gbrain reindex-search-vector --dry-run # preview: language + row counts +gbrain reindex-search-vector --yes # recreate triggers + backfill +``` + +The command recreates both trigger functions under the new language and +backfills every existing `pages` and `content_chunks` row in batches, +streaming progress to stderr. It is idempotent: re-running with the same +language produces identical vectors. `--json` prints a machine-readable +result envelope but still requires `--yes` (or an interactive confirm). + +## Recipe: accent-insensitive Portuguese (`pt_br`) + +Brazilian Portuguese content often mixes accented and unaccented spellings +("São Paulo" vs "Sao Paulo"). Build a custom config that folds accents via +the `unaccent` extension, then stems with the portuguese snowball dictionary: + +```sql +CREATE EXTENSION IF NOT EXISTS unaccent; + +CREATE TEXT SEARCH CONFIGURATION pt_br (COPY = portuguese); + +ALTER TEXT SEARCH CONFIGURATION pt_br + ALTER MAPPING FOR hword, hword_part, word + WITH unaccent, portuguese_stem; +``` + +Then point GBrain at it: + +```bash +export GBRAIN_FTS_LANGUAGE=pt_br +gbrain reindex-search-vector --yes +``` + +Note: custom configurations require a real Postgres instance (e.g. the +Supabase engine). The config must exist BEFORE the migration or the reindex +command runs, or Postgres will reject the trigger recreation with +`text search configuration "pt_br" does not exist`. + +## Caveats + +- One language per brain: the setting is global to the database, not + per-source. Mixed-language brains should pick the dominant language (the + vector-search arm is language-agnostic and covers the rest). +- Keep `GBRAIN_FTS_LANGUAGE` set consistently in every environment that + writes to the brain (CLI shells, MCP server, cron jobs) — a writer without + the env var tokenizes new rows in `english` until the next reindex. diff --git a/llms-full.txt b/llms-full.txt index 1a9710038..4de2b11e4 100644 --- a/llms-full.txt +++ b/llms-full.txt @@ -1752,6 +1752,24 @@ The whole loop is described in [`docs/architecture/topologies.md`](docs/architec **Job queue (Minions).** BullMQ-shaped, Postgres-native job queue. Durable subagents (LLM tool loops that survive crashes via two-phase pending→done persistence), shell jobs with audit, child jobs with cascading timeouts, rate leases for outbound providers, attachments via S3/Supabase storage. Replaces "spawn subagent as fire-and-forget Promise" with something that recovers from anything. +**Non-English brains (FTS language config).** The Postgres full-text search tokenizer is configurable via `GBRAIN_FTS_LANGUAGE`. Defaults to `english`. Set it to any text-search configuration that exists in your Postgres instance: + +```bash +export GBRAIN_FTS_LANGUAGE=portuguese # uses built-in portuguese stemmer +export GBRAIN_FTS_LANGUAGE=spanish # built-in spanish stemmer +export GBRAIN_FTS_LANGUAGE=pt_br # custom config (e.g. unaccent + portuguese) +``` + +List available configs: `psql -c "SELECT cfgname FROM pg_ts_config"`. Both the **query side** (`websearch_to_tsquery`) and the **write side** (the trigger functions that populate `pages.search_vector` and `content_chunks.search_vector`) honor `GBRAIN_FTS_LANGUAGE`. On first install (or upgrade), the `configurable_fts_language` schema migration reads the env var and creates trigger functions in the configured language; subsequent inserts/updates tokenize using that setting. To change language on a brain that has already run the migration, use the dedicated CLI command: + +```bash +export GBRAIN_FTS_LANGUAGE=portuguese +gbrain reindex-search-vector --dry-run # preview row counts +gbrain reindex-search-vector --yes # recreate triggers + backfill +``` + +The command is idempotent (re-running with the same language is a no-op for vector content) and uses the same recreate-and-backfill primitives as the migration. For accent-insensitive Portuguese (`pt_br`), see [docs/guides/multi-language-fts.md](docs/guides/multi-language-fts.md) for the `unaccent` + portuguese stemmer recipe. + **43 curated skills.** Routing lives in [`skills/RESOLVER.md`](skills/RESOLVER.md). Covers signal capture, ingest (idea / media / meeting), enrichment, querying, brain ops, citation fixing, daily task management, cron scheduling, reports, voice, soul audit, skill creation, eval framework, and migrations. Skills are markdown files (tool-agnostic), packaged as a single skillpack the installer drops into your agent workspace. **Eval framework.** `gbrain eval longmemeval` runs the public [LongMemEval](https://huggingface.co/datasets/xiaowu0162/longmemeval) benchmark against your hybrid retrieval. `gbrain eval export` + `gbrain eval replay` capture real queries and replay them against code changes (set `GBRAIN_CONTRIBUTOR_MODE=1`). `gbrain eval cross-modal` cross-checks an output against the task using three different-provider frontier models. `gbrain eval retrieval-quality` runs NamedThingBench, which hard-gates the named-thing retrieval families (title-substring, alias-synonym, generic-to-named, multi-chunk-dilution) so a regression in "find the page this query names" fails CI loudly. Full methodology in [`docs/eval/SEARCH_MODE_METHODOLOGY.md`](docs/eval/SEARCH_MODE_METHODOLOGY.md). @@ -2095,6 +2113,51 @@ RRF fusion, multi-query expansion, and 4-layer dedup are engine-agnostic. They o **Why not self-hosted for v0:** The brain should be infrastructure agents use, not something you maintain. Self-hosted Postgres with Docker is a welcome community PR, but v0 optimizes for zero ops. +### Opt-in RLS source-scope binding (`GBRAIN_RLS_SCOPE_BINDING`) + +Defense-in-depth layer for Postgres deployments that want the database itself +to enforce source isolation, in addition to the mandatory app-layer filters +(`sourceScopeOpts` — layer 1, always on). + +**Mechanism.** With `GBRAIN_RLS_SCOPE_BINDING=1` (or `true`), the engine's +source-scoped read methods wrap their queries in a transaction that first runs +`SELECT set_config('app.scopes', $1, true)` — the value is a bound parameter +(federated `sourceIds` CSV > scalar `sourceId` > `'*'` for unscoped internal +reads), transaction-local (equivalent to `SET LOCAL`, which itself can't take +bound params). An RLS policy can then filter rows by +`current_setting('app.scopes', true)`. + +**Default off.** With the env var unset, reads call through on the shared pool +exactly as before — no per-read transaction, no pool-slot hold (the search +methods keep the transaction they always had for their `SET LOCAL +statement_timeout`). Existing operators see zero behavior change. + +**Enabling it** (operator-managed SQL; gbrain ships no DDL for this): + +```sql +ALTER TABLE pages ENABLE ROW LEVEL SECURITY; +CREATE POLICY pages_scope_filter ON pages + USING (current_setting('app.scopes', true) = '*' + OR source_id = ANY(string_to_array(current_setting('app.scopes', true), ','))); + +-- Required: connections that don't run through the scoped read helper +-- (admin, autopilot, cycle, writes) must default to unscoped, or they +-- see zero rows once the policy exists: +ALTER ROLE SET app.scopes = '*'; + +-- If the runtime role OWNS the table, RLS is skipped for it unless forced: +ALTER TABLE pages FORCE ROW LEVEL SECURITY; +``` + +Safe to enable in either order: the env var without a policy is a no-op +setting; a policy without the env var is enforced only via the role default. + +**Honest caveat:** only read paths routed through the scoped helper carry a +per-request scope binding — unwrapped paths (writes, admin/maintenance reads) +run under the role default and are not backstopped per caller. This is layer 2; +the app-layer source filters remain layer 1 and stay mandatory. Behavioral pins +live in `test/postgres-engine-rls-scope.test.ts`. + ## PGLiteEngine (v0.7, ships) **Dependencies:** `@electric-sql/pglite` (v0.4.4+) diff --git a/src/cli.ts b/src/cli.ts index 3e14c39f9..53622bb05 100755 --- a/src/cli.ts +++ b/src/cli.ts @@ -54,7 +54,7 @@ export function bigintToStringReplacer(_key: string, value: unknown): unknown { } // CLI-only commands that bypass the operation layer -export const CLI_ONLY = new Set(['init', 'reinit-pglite', 'upgrade', 'post-upgrade', 'check-update', 'integrations', 'publish', 'check-backlinks', 'lint', 'report', 'import', 'export', 'files', 'embed', 'serve', 'call', 'config', 'doctor', 'migrate', 'eval', 'sync', 'extract', 'extract-conversation-facts', 'enrich', 'features', 'autopilot', 'graph-query', 'jobs', 'agent', 'apply-migrations', 'skillpack-check', 'skillpack', 'resolvers', 'integrity', 'repair-jsonb', 'orphans', 'sources', 'mounts', 'dream', 'check-resolvable', 'routing-eval', 'skillify', 'smoke-test', 'providers', 'storage', 'repos', 'code-def', 'code-refs', 'reindex', 'reindex-code', 'reindex-frontmatter', 'code-callers', 'code-callees', 'frontmatter', 'auth', 'friction', 'claw-test', 'book-mirror', 'takes', 'think', 'salience', 'anomalies', 'calibration', 'transcripts', 'models', 'remote', 'recall', 'forget', 'edges-backfill', 'cache', 'ze-switch', 'founder', 'brainstorm', 'lsd', 'schema', 'capture', 'onboard', 'conversation-parser', 'status', 'connect', 'skillopt', 'quarantine', 'self-upgrade', 'advisor', 'watch']); +export const CLI_ONLY = new Set(['init', 'reinit-pglite', 'upgrade', 'post-upgrade', 'check-update', 'integrations', 'publish', 'check-backlinks', 'lint', 'report', 'import', 'export', 'files', 'embed', 'serve', 'call', 'config', 'doctor', 'migrate', 'eval', 'sync', 'extract', 'extract-conversation-facts', 'enrich', 'features', 'autopilot', 'graph-query', 'jobs', 'agent', 'apply-migrations', 'skillpack-check', 'skillpack', 'resolvers', 'integrity', 'repair-jsonb', 'orphans', 'sources', 'mounts', 'dream', 'check-resolvable', 'routing-eval', 'skillify', 'smoke-test', 'providers', 'storage', 'repos', 'code-def', 'code-refs', 'reindex', 'reindex-code', 'reindex-frontmatter', 'code-callers', 'code-callees', 'frontmatter', 'auth', 'friction', 'claw-test', 'book-mirror', 'takes', 'think', 'salience', 'anomalies', 'calibration', 'transcripts', 'models', 'remote', 'recall', 'forget', 'edges-backfill', 'cache', 'ze-switch', 'founder', 'brainstorm', 'lsd', 'schema', 'capture', 'onboard', 'conversation-parser', 'status', 'connect', 'skillopt', 'quarantine', 'self-upgrade', 'advisor', 'watch', 'reindex-search-vector']); // CLI-only commands whose handlers print their own --help text. These are // excluded from the generic short-circuit so detailed per-command and // per-subcommand usage stays reachable. @@ -2001,6 +2001,15 @@ async function handleCliOnly(command: string, args: string[]) { await runReindexCodeCli(engine, args); break; } + case 'reindex-search-vector': { + // Explicit recreate of FTS trigger functions + batched backfill, + // honoring GBRAIN_FTS_LANGUAGE. Use after changing the language + // env var on a brain that already ran the configurable_fts_language + // migration. + const { runReindexSearchVectorCli } = await import('./commands/reindex-search-vector.ts'); + await runReindexSearchVectorCli(engine, args); + break; + } case 'reindex-frontmatter': { // v0.29.1: recovery / explicit-rebuild path for pages.effective_date. // Mirror of reindex-code shape. Wraps the shared library function in @@ -2336,6 +2345,9 @@ CODE INDEXING (v0.19.0 / v0.20.0 Cathedral II) query --symbol-kind Filter to symbol type (function|class|method|...) (v0.20.0) reconcile-links [--dry-run] Batch-recompute doc↔impl edges (v0.20.0) reindex-code [--source id] [--yes] Explicit code-page reindex (v0.20.0) + reindex-search-vector [--dry-run] [--yes] [--json] + Recreate FTS triggers + backfill under + $GBRAIN_FTS_LANGUAGE (default 'english') sync --strategy code Sync code files into the brain JOBS (Minions) diff --git a/src/commands/extract.ts b/src/commands/extract.ts index 9b2ff4fbc..21eeaaef5 100644 --- a/src/commands/extract.ts +++ b/src/commands/extract.ts @@ -188,7 +188,7 @@ export function walkMarkdownFiles(dir: string): { path: string; relPath: string // Pre-fix, this walker had only an ad-hoc dot-prefix exclusion and didn't // call isSyncable at all — so it descended into `node_modules/`, emitted // markdown files from there, AND ignored the canonical exclusion list - // (`.raw/`, `ops/`, README.md, etc.). Now: pruneDir skips entire vendor + // (`.raw/`, README.md, etc.). Now: pruneDir skips entire vendor // subtrees before recursion (saving IO), and isSyncable filters the emit // set against the canonical markdown-strategy rules. const files: { path: string; relPath: string }[] = []; diff --git a/src/commands/import.ts b/src/commands/import.ts index 9e792a384..704970cd2 100644 --- a/src/commands/import.ts +++ b/src/commands/import.ts @@ -569,10 +569,21 @@ function isCollectibleForWalker( strategy: SyncStrategy, multimodalOn: boolean, ): boolean { + // #2607: apply the SAME segment-level prune gate as incremental sync's + // `classifySync` (core/sync.ts). The FS walk below prunes at descent time, + // but the git fast path enumerates via `git ls-files` and historically + // filtered only by extension — so `sync --full` imported (and resurrected + // previously-deleted) pages under dot-dirs / vendored trees that incremental + // sync excludes. Full and incremental must agree on the exclusion set. + // (In the FS-walk route `path` is a basename, so this is the same dot-file + // check pruneDir already applied there — no behavior change on that route.) + const segments = path.split('/'); + if (segments.some((seg) => !pruneDir(seg))) return false; + // Metafiles are directory scaffolding (READMEs / index / log / schema / // resolver), not typed brain pages — same exclusion `sync`'s `isSyncable` // applies. Guards both the FS-walk and the git-fast-path collection routes. - const basename = path.split('/').pop() || ''; + const basename = segments[segments.length - 1] || ''; if ((SYNC_SKIP_FILES as readonly string[]).includes(basename)) return false; switch (strategy) { diff --git a/src/commands/reindex-search-vector.ts b/src/commands/reindex-search-vector.ts new file mode 100644 index 000000000..59524511e --- /dev/null +++ b/src/commands/reindex-search-vector.ts @@ -0,0 +1,277 @@ +/** + * `gbrain reindex-search-vector` — recreate FTS trigger functions and + * backfill existing rows under the language configured via + * GBRAIN_FTS_LANGUAGE. + * + * Why this command exists: schema migration v123 (configurable_fts_language) + * stamps the trigger functions with the configured language at first apply. + * After that, changing the env var has no effect on the write side because + * v123 already shows as "applied" — the migrations runner will skip it. + * This command is the documented escape hatch: it re-runs the same + * recreate-and-backfill logic v123 uses, gated on an explicit user + * action so the operation is intentional and visible (writes touch + * every row in pages and content_chunks). + * + * Idempotent: running twice with the same GBRAIN_FTS_LANGUAGE produces + * the same trigger function bodies and the same tokenized vectors. + * + * Flags: + * --dry-run Show what would happen, exit 0 without touching DB. + * --yes Skip interactive [y/N]. Required for non-TTY (including --json). + * --json Machine-readable result envelope. Does NOT imply --yes. + * + * Backfill runs in id-keyset batches (BACKFILL_BATCH_SIZE rows per UPDATE) + * so a large brain never holds one giant row lock, and streams progress + * through the shared reporter (stderr; stdout stays clean for --json). + * + * Cost: trigger recreate is sub-millisecond. Backfill is one tsvector + * rebuild per page + per chunk. On a 20K-page brain with 80K chunks, + * expect ~5-15s depending on Postgres CPU and content size. + */ + +import type { BrainEngine } from '../core/engine.ts'; +import { getFtsLanguage } from '../core/fts-language.ts'; +import { createInterface } from 'readline'; +import { createProgress } from '../core/progress.ts'; +import { getCliOptions, cliOptsToProgressOptions } from '../core/cli-options.ts'; + +export interface ReindexSearchVectorOpts { + dryRun?: boolean; + yes?: boolean; + json?: boolean; +} + +export interface ReindexSearchVectorResult { + status: 'ok' | 'dry_run' | 'cancelled'; + language: string; + pagesUpdated: number; + chunksUpdated: number; + triggersRecreated: number; + durationMs: number; +} + +interface CountRow { + pages: number; + chunks: number; +} + +/** Rows per backfill UPDATE. Keyset-batched so one statement never locks the whole table. */ +export const BACKFILL_BATCH_SIZE = 5000; + +/** + * Keyset-batched UPDATE: applies `setClause` to `table` rows where + * search_vector IS NOT NULL, BACKFILL_BATCH_SIZE ids at a time, ticking + * the shared progress reporter after each batch. Terminates when a batch + * returns fewer rows than the batch size (or none). + */ +async function batchedBackfill( + engine: BrainEngine, + table: 'pages' | 'content_chunks', + setClause: string, + tick: (n: number) => void +): Promise { + let cursor = 0; + for (;;) { + const rows = await engine.executeRaw<{ id: number }>(` + UPDATE ${table} SET ${setClause} + WHERE id IN ( + SELECT id FROM ${table} + WHERE search_vector IS NOT NULL AND id > ${cursor} + ORDER BY id + LIMIT ${BACKFILL_BATCH_SIZE} + ) + RETURNING id + `); + if (rows.length === 0) break; + tick(rows.length); + cursor = rows.reduce((m, r) => Math.max(m, Number(r.id)), cursor); + if (rows.length < BACKFILL_BATCH_SIZE) break; + } +} + +/** + * Programmatic entrypoint — takes a typed opts object. Used by tests and + * future internal callers. The CLI wrapper is `runReindexSearchVectorCli` + * defined at the bottom of this file. + */ +export async function runReindexSearchVector( + engine: BrainEngine, + opts: ReindexSearchVectorOpts +): Promise { + const lang = getFtsLanguage(); + const startedAt = Date.now(); + + // Inventory: how many rows will the backfill touch? + const counts = await engine.executeRaw( + `SELECT + (SELECT COUNT(*)::int FROM pages WHERE search_vector IS NOT NULL) AS pages, + (SELECT COUNT(*)::int FROM content_chunks WHERE search_vector IS NOT NULL) AS chunks` + ); + const pagesCount = counts[0]?.pages ?? 0; + const chunksCount = counts[0]?.chunks ?? 0; + + if (opts.dryRun) { + const result: ReindexSearchVectorResult = { + status: 'dry_run', + language: lang, + pagesUpdated: pagesCount, + chunksUpdated: chunksCount, + triggersRecreated: 0, + durationMs: Date.now() - startedAt, + }; + if (opts.json) { + console.log(JSON.stringify(result, null, 2)); + } else { + console.log(`[dry-run] Would recreate 2 trigger functions with language='${lang}'`); + console.log(`[dry-run] Would backfill ${pagesCount} pages + ${chunksCount} chunks`); + console.log(`[dry-run] Skipping all DB writes. Pass --yes to apply.`); + } + return result; + } + + // Confirm unless --yes. --json does NOT bypass the gate — a machine + // caller must pass --yes explicitly (mirrors reindex-code, #1784). + if (!opts.yes) { + if (!process.stdin.isTTY) { + if (opts.json) { + console.log(JSON.stringify({ + error: { + class: 'ConfirmationRequired', + code: 'reindex_requires_yes', + message: `Refusing to recreate FTS triggers + backfill ${pagesCount} pages + ${chunksCount} chunks without --yes in a non-TTY environment.`, + hint: 'Pass --yes to proceed, or --dry-run to preview.', + }, + language: lang, + pages: pagesCount, + chunks: chunksCount, + })); + } else { + console.error('Refusing to run without --yes in non-TTY environment.'); + } + process.exit(2); + } + + const rl = createInterface({ input: process.stdin, output: process.stdout }); + const answer = await new Promise(resolve => { + rl.question( + `Recreate FTS triggers with language='${lang}' and backfill ${pagesCount} pages + ${chunksCount} chunks? [y/N]: `, + resolve + ); + }); + rl.close(); + + if (!/^y(es)?$/i.test(answer.trim())) { + const result: ReindexSearchVectorResult = { + status: 'cancelled', + language: lang, + pagesUpdated: 0, + chunksUpdated: 0, + triggersRecreated: 0, + durationMs: Date.now() - startedAt, + }; + if (opts.json) { + console.log(JSON.stringify(result, null, 2)); + } else { + console.log('Cancelled.'); + } + return result; + } + } + + // Recreate trigger functions. The strings are intentionally identical to + // the v123 migration body — keeping them in lockstep is the contract. + // `SET search_path = pg_catalog, public` mirrors the v120/#1647 hardening: + // CREATE OR REPLACE resets proconfig, so omitting it here would strip the + // hardening from every brain that runs this command. + const recreatePagesFn = ` + CREATE OR REPLACE FUNCTION update_page_search_vector() RETURNS trigger SET search_path = pg_catalog, public AS $fn$ + DECLARE + timeline_text TEXT; + BEGIN + SELECT coalesce(string_agg(summary || ' ' || detail, ' '), '') + INTO timeline_text + FROM timeline_entries + WHERE page_id = NEW.id; + + NEW.search_vector := + setweight(to_tsvector('${lang}', coalesce(NEW.title, '')), 'A') || + setweight(to_tsvector('${lang}', coalesce(NEW.compiled_truth, '')), 'B') || + setweight(to_tsvector('${lang}', coalesce(NEW.timeline, '')), 'C') || + setweight(to_tsvector('${lang}', coalesce(timeline_text, '')), 'C'); + + RETURN NEW; + END; + $fn$ LANGUAGE plpgsql; + `; + + const recreateChunksFn = ` + CREATE OR REPLACE FUNCTION update_chunk_search_vector() RETURNS TRIGGER SET search_path = pg_catalog, public AS $fn$ + BEGIN + NEW.search_vector := + setweight(to_tsvector('${lang}', COALESCE(NEW.doc_comment, '')), 'A') || + setweight(to_tsvector('${lang}', COALESCE(NEW.symbol_name_qualified, '')), 'A') || + setweight(to_tsvector('${lang}', COALESCE(NEW.chunk_text, '')), 'B'); + RETURN NEW; + END; + $fn$ LANGUAGE plpgsql; + `; + + await engine.executeRaw(recreatePagesFn); + await engine.executeRaw(recreateChunksFn); + + const progress = createProgress(cliOptsToProgressOptions(getCliOptions())); + + // Backfill: UPDATE-to-self forces the pages trigger to re-fire + // (Postgres re-fires on UPDATE-to-same-value); content_chunks gets a + // direct vector compute since the column itself is what we want. + progress.start('reindex_search_vector.pages', pagesCount); + await batchedBackfill(engine, 'pages', 'id = id', n => progress.tick(n)); + progress.finish(); + + progress.start('reindex_search_vector.chunks', chunksCount); + await batchedBackfill( + engine, + 'content_chunks', + `search_vector = + setweight(to_tsvector('${lang}', COALESCE(doc_comment, '')), 'A') || + setweight(to_tsvector('${lang}', COALESCE(symbol_name_qualified, '')), 'A') || + setweight(to_tsvector('${lang}', COALESCE(chunk_text, '')), 'B')`, + n => progress.tick(n) + ); + progress.finish(); + + const result: ReindexSearchVectorResult = { + status: 'ok', + language: lang, + pagesUpdated: pagesCount, + chunksUpdated: chunksCount, + triggersRecreated: 2, + durationMs: Date.now() - startedAt, + }; + + if (opts.json) { + console.log(JSON.stringify(result, null, 2)); + } else { + console.log(`✅ Recreated 2 trigger functions with language='${lang}'`); + console.log(`✅ Backfilled ${pagesCount} pages + ${chunksCount} chunks (${result.durationMs}ms)`); + } + + return result; +} + +/** + * CLI entrypoint. Parses argv flags and dispatches to runReindexSearchVector. + * Matches the style of `reindex-code`: --dry-run, --yes/-y, --json. + * + * Exit codes: 0 success/dry-run/cancelled, 2 if non-TTY without --yes. + */ +export async function runReindexSearchVectorCli( + engine: BrainEngine, + args: string[] +): Promise { + const dryRun = args.includes('--dry-run'); + const yes = args.includes('--yes') || args.includes('-y'); + const json = args.includes('--json'); + + await runReindexSearchVector(engine, { dryRun, yes, json }); +} diff --git a/src/commands/sync.ts b/src/commands/sync.ts index 0c2861934..ca54dd29a 100644 --- a/src/commands/sync.ts +++ b/src/commands/sync.ts @@ -2010,7 +2010,13 @@ async function performSyncInner(engine: BrainEngine, opts: SyncOpts): Promise 0) { + // #2426: a stale page whose source_path was NEVER committed to git is + // DB-only write-through (the file was written into the clone but never + // committed/pushed, then lost — e.g. a fresh clone). "Absent from git" + // is the SYMPTOM of that bug, not evidence the content is disposable. + // Keep those pages and re-export their markdown to the working tree so + // they're file-backed again; only pages whose file once existed in git + // history (i.e. was genuinely deleted) are reconcile-deleted. + const everCommitted = listEverCommittedPaths(repoPath); + const pathBySlug = new Map(rows.map(r => [r.slug, r.source_path])); + let deletableSlugs = plan.staleSlugs; + const dbOnlySlugs: string[] = []; + if (everCommitted) { + deletableSlugs = []; + for (const slug of plan.staleSlugs) { + const sp = pathBySlug.get(slug); + if (sp && !everCommitted.has(sp.replace(/\\/g, '/'))) dbOnlySlugs.push(slug); + else deletableSlugs.push(slug); + } + } + if (dbOnlySlugs.length > 0) { + let reExported = 0; + try { + const { writePageThrough } = await import('../core/write-through.ts'); + for (const slug of dbOnlySlugs) { + const r = await writePageThrough(engine, slug, { sourceId: sid }); + if (r.written) reExported++; + } + } catch { /* best-effort — pages are preserved either way */ } + serr( + `\n Kept ${dbOnlySlugs.length} page(s) whose markdown was never committed to git ` + + `(DB-only write-through — not deleting).` + + (reExported > 0 ? ` Re-exported ${reExported} of them to the working tree.` : '') + + `\n Commit + push them (e.g. scripts/brain-commit-push.sh, or 'gbrain sources harden') ` + + `so the next sync sees them as file-backed.`, + ); + } const deleteScopedOpts = { sourceId: sid }; - for (let i = 0; i < plan.staleSlugs.length; i += DELETE_BATCH_SIZE) { - const batch = plan.staleSlugs.slice(i, i + DELETE_BATCH_SIZE); + for (let i = 0; i < deletableSlugs.length; i += DELETE_BATCH_SIZE) { + const batch = deletableSlugs.slice(i, i + DELETE_BATCH_SIZE); try { const deleted = await engine.deletePages(batch, deleteScopedOpts); reconciledDeletes += deleted.length; @@ -3442,6 +3484,34 @@ export function planReconcileDeletes( return { staleSlugs, reconcilableCount: reconcilable.length, massDelete }; } +/** + * #2426: every repo-relative path that ever appeared as an ADD in git history + * (rename detection off, so a `git mv` destination still counts as an add). + * Used by the full-sync reconcile to distinguish "file was committed and later + * deleted" (genuine delete → reconcile) from "file was NEVER committed" + * (DB-only write-through → preserve). Returns null when `repoPath` isn't a git + * work tree or git is unavailable — callers keep the plain-directory behavior. + * Forward-slash-normalized to match `normalizeReconcilePath` membership tests. + */ +export function listEverCommittedPaths(repoPath: string): Set | null { + let stdout: string; + try { + stdout = execFileSync( + 'git', + ['-C', repoPath, '-c', 'core.quotepath=off', 'log', '--all', '--no-renames', + '--diff-filter=A', '--format=', '--name-only'], + { encoding: 'utf8', maxBuffer: 512 * 1024 * 1024, stdio: ['ignore', 'pipe', 'ignore'] }, + ); + } catch { + return null; + } + const set = new Set(); + for (const line of stdout.split('\n')) { + if (line) set.add(line.replace(/\\/g, '/')); + } + return set; +} + /** * #2828 escape hatch: `GBRAIN_ALLOW_MASS_RECONCILE=1` restores the pre-valve * behavior for the rare intentional bulk removal. Env-only (an incident-time diff --git a/src/core/ai/recipes/index.ts b/src/core/ai/recipes/index.ts index 383350fff..098691b46 100644 --- a/src/core/ai/recipes/index.ts +++ b/src/core/ai/recipes/index.ts @@ -23,6 +23,7 @@ import { zhipu } from './zhipu.ts'; import { azureOpenAI } from './azure-openai.ts'; import { zeroentropyai } from './zeroentropyai.ts'; import { llamaServerReranker } from './llama-server-reranker.ts'; +import { moonshot } from './moonshot.ts'; const ALL: Recipe[] = [ openai, @@ -42,6 +43,7 @@ const ALL: Recipe[] = [ zhipu, azureOpenAI, zeroentropyai, + moonshot, ]; /** Map from `provider:id` key to recipe. */ diff --git a/src/core/ai/recipes/moonshot.ts b/src/core/ai/recipes/moonshot.ts new file mode 100644 index 000000000..859588c76 --- /dev/null +++ b/src/core/ai/recipes/moonshot.ts @@ -0,0 +1,42 @@ +import type { Recipe } from '../types.ts'; + +/** + * Moonshot AI / Kimi Open Platform. Kimi exposes an OpenAI-compatible + * /v1/chat/completions API at https://api.moonshot.ai/v1. + * + * Verified against Kimi API docs and live /v1/models on 2026-06-23. + * The recipe is local-production glue until upstream GBrain carries a native + * Moonshot recipe; keep it registered in the local patch registry. + */ +export const moonshot: Recipe = { + id: 'moonshot', + name: 'Moonshot AI / Kimi', + tier: 'openai-compat', + implementation: 'openai-compatible', + base_url_default: 'https://api.moonshot.ai/v1', + auth_env: { + required: ['MOONSHOT_API_KEY'], + setup_url: 'https://platform.kimi.ai/console/api-keys', + }, + touchpoints: { + expansion: { + models: ['kimi-k2.7-code', 'kimi-k2.7-code-highspeed', 'kimi-k2.6', 'kimi-k2.5'], + // Kimi pricing varies by current promotional/account terms; do not use + // this advisory field for budget enforcement. Canonical budget pricing + // belongs in src/core/model-pricing.ts when verified for the account. + price_last_verified: '2026-06-23', + }, + chat: { + models: ['kimi-k2.7-code', 'kimi-k2.7-code-highspeed', 'kimi-k2.6', 'kimi-k2.5'], + supports_tools: true, + // Kimi tool calling is enough for ordinary chat/tool calls. GBrain's + // subagent loop remains Anthropic-pinned because upstream requires stable + // Anthropic-style tool_use_id behavior across crashes/replays. + supports_subagent_loop: false, + supports_prompt_cache: false, + max_context_tokens: 256000, + price_last_verified: '2026-06-23', + }, + }, + setup_hint: 'Get an API key at https://platform.kimi.ai/console/api-keys, then `export MOONSHOT_API_KEY=...` and use `moonshot:kimi-k2.7-code`.', +}; diff --git a/src/core/brain-repo-durability.ts b/src/core/brain-repo-durability.ts index d4b77c09f..7fa1685aa 100644 --- a/src/core/brain-repo-durability.ts +++ b/src/core/brain-repo-durability.ts @@ -183,15 +183,17 @@ if [ "\${1:-}" = "--push-only" ]; then fi _msg="\${1:?usage: brain-commit-push.sh [paths...]}"; shift || true -# Pull first so the local tree is current before we stage. -git fetch origin >/dev/null 2>&1 || true -git pull --rebase origin "$_branch" || { git rebase --abort >/dev/null 2>&1 || true; echo "rebase conflict: manual attention needed" >&2; exit 3; } # EXPLICIT paths only — never a blind 'git add -A' (would risk committing # secrets, temp files, or unrelated edits). if [ "$#" -eq 0 ]; then echo "refusing blind 'git add -A' — pass explicit path(s) to commit" >&2; exit 2 fi +# COMMIT BEFORE PULL (#2426): the old order (fetch + pull --rebase, THEN stage) +# aborted on any dirty tree — 'cannot pull with rebase: You have unstaged +# changes' — so the helper could never commit a MODIFIED page (exactly the +# write-through case). Stage + commit first; brain_push below already handles +# a remote that advanced (push -> rejected -> pull --rebase -> push). git add -- "$@" if git diff --cached --quiet; then echo "nothing to commit"; exit 0; fi git commit -m "$_msg" @@ -337,6 +339,47 @@ function uninstallLocalHook(repoPath: string): boolean { return true; } +/** + * True when the gbrain durability post-commit hook is installed — i.e. the + * user opted this repo into push-durability via `gbrain sources harden`. + * Cheap (one git-config read + one file read); used as the gate for + * write-through auto-commit (#2426). + */ +export function isDurabilityHardened(repoPath: string): boolean { + try { + const { dir } = resolveHooksDir(repoPath); + const hookPath = join(dir, 'post-commit'); + return existsSync(hookPath) && readFileSync(hookPath, 'utf-8').includes(HOOK_BANNER); + } catch { + return false; + } +} + +/** + * #2426: best-effort commit of a single write-through artifact so DB writes + * reach git (the post-commit hook then background-pushes). Pre-fix, + * write-through `.md` accumulated uncommitted forever: it never reached the + * remote, froze `last_sync_at` (HEAD never moved), and a later `sync --full` + * delete-reconcile treated the never-committed pages as disposable. + * + * Path-limited (`git commit -- `) so unrelated staged/dirty edits are + * never swept into the commit. Never throws; returns false on any failure + * (index.lock contention, nothing changed, detached states) — the DB row and + * the on-disk file remain the durable sinks either way. + */ +export function commitWriteThroughFile(repoPath: string, absPath: string, slug: string): boolean { + try { + const rel = relative(repoPath, absPath); + if (!rel || rel.startsWith('..') || isAbsolute(rel)) return false; + const gitOpts = { stdio: 'ignore', timeout: 30_000, env: { ...process.env, ...GIT_ENV } } as const; + execFileSync('git', ['-C', repoPath, 'add', '--', rel], gitOpts); + execFileSync('git', ['-C', repoPath, 'commit', '-m', `gbrain: write-through ${slug}`, '--', rel], gitOpts); + return true; + } catch { + return false; + } +} + // ── Committed helper ──────────────────────────────────────────────────────── function installHelper(repoPath: string, dryRun: boolean): { status: StepStatus; detail: string } { diff --git a/src/core/chronicle/extract-events.ts b/src/core/chronicle/extract-events.ts index 4354614ea..a1324118d 100644 --- a/src/core/chronicle/extract-events.ts +++ b/src/core/chronicle/extract-events.ts @@ -26,7 +26,17 @@ export interface ChronicleJudgeInput { effectiveDate: string | null; // depth page effective_date (deterministic when) attendees: string[]; // deterministic who from frontmatter } -export interface ChronicleJudgeResult { events: ChronicleEventProposal[] } +export interface ChronicleJudgeResult { + events: ChronicleEventProposal[]; + /** + * #2606 — distinct judge-failure signal so an unusable response is never + * recorded as a legitimate `no_events`: + * - 'truncated': the model hit the output-token cap (stopReason 'length'); + * the JSON array was cut mid-stream and must not be parsed as complete. + * - 'parse_failed': the model returned text but no valid JSON array. + */ + failure?: 'truncated' | 'parse_failed'; +} export type ChronicleJudge = (input: ChronicleJudgeInput) => Promise; export interface ChronicleExtractResult { @@ -126,6 +136,12 @@ export async function runChronicleExtract( return { slug: opts.slug, status: 'skipped', events_written: 0, reason: 'judge_error' }; } + // #2606: a truncated or unparseable judge response is a FAILURE, not an + // empty page. Record it as a distinct skipped reason so operators (and + // retries) can tell it apart from a genuine no_events. + if (result?.failure) { + return { slug: opts.slug, status: 'skipped', events_written: 0, reason: `judge_${result.failure}` }; + } const proposals = Array.isArray(result?.events) ? result.events : []; if (proposals.length === 0) return { slug: opts.slug, status: 'no_events', events_written: 0 }; // PARSE BARRIER — reject the WHOLE batch on any malformed proposal; no partial writes. @@ -167,11 +183,25 @@ const JUDGE_SYSTEM = `You segment a meeting/transcript page into discrete timeli Return ONLY a JSON array. Each element: {"when": ISO datetime or YYYY-MM-DD, "who": [entity slugs/names], "what": one-clause summary, "where": optional string, "kind": one of meeting|call|meal|solo|travel|work|commitment|decision|intro|conflict|milestone|event}. Prefer the page's known date for "when" when the text gives no explicit time. Use the provided attendee slugs for "who" when the text does not name participants. No prose, no markdown — just the JSON array.`; +/** + * #2606: default output-token cap for the judge. Raised from the original + * 1500 (which event-dense pages overflowed, silently truncating the JSON + * array). Override via `chronicle.judge_max_tokens`. + */ +const DEFAULT_JUDGE_MAX_TOKENS = 4000; + function defaultJudge(engine: BrainEngine): ChronicleJudge { return async (input) => { const { isAvailable, chat } = await import('../ai/gateway.ts'); if (!isAvailable('chat')) return { events: [] }; const body = (input.body || '').slice(0, 12_000); + // #2606: configurable cap so event-dense pages have headroom. + let maxTokens = DEFAULT_JUDGE_MAX_TOKENS; + const capRaw = await engine.getConfig('chronicle.judge_max_tokens').catch(() => null); + if (capRaw) { + const n = parseInt(capRaw, 10); + if (Number.isFinite(n) && n > 0) maxTokens = n; + } let text: string; try { const res = await chat({ @@ -183,32 +213,44 @@ function defaultJudge(engine: BrainEngine): ChronicleJudge { `${input.title}\n\n${body}\n\n\n` + `Known attendees: ${input.attendees.slice(0, 10).join(', ') || '(none)'}.\nExtract the events.`, }], - maxTokens: 1500, + maxTokens, }); if (res.stopReason === 'refusal' || res.stopReason === 'content_filter') return { events: [] }; + // #2606: output hit the token cap — the JSON array is cut mid-stream. + // Do NOT feed it to the parser as if complete; surface the truncation. + if (res.stopReason === 'length') return { events: [], failure: 'truncated' }; text = res.text; } catch (err) { if ((err as Error)?.name === 'AbortError') throw err; return { events: [] }; } const parsed = parseJudgeJson(text); + // #2606: non-empty model text with no parseable JSON array is a parse + // failure, distinct from the model legitimately answering `[]`. + if (parsed === null) return { events: [], failure: 'parse_failed' }; return { events: parsed }; }; } -/** Tolerant JSON-array extraction from a model response (mirrors facts parser). */ -export function parseJudgeJson(text: string): ChronicleEventProposal[] { - if (!text) return []; +/** + * Tolerant JSON-array extraction from a model response (mirrors facts parser). + * + * #2606: returns `null` on parse FAILURE (empty text, no `[...]` found, + * JSON.parse throw, non-array result) so callers can distinguish "the model + * said no events" (a legitimate `[]`) from "the response was unusable". + */ +export function parseJudgeJson(text: string): ChronicleEventProposal[] | null { + if (!text) return null; let s = text.trim(); const fence = s.match(/```(?:json)?\s*([\s\S]*?)```/i); if (fence) s = fence[1].trim(); const start = s.indexOf('['); const end = s.lastIndexOf(']'); - if (start === -1 || end === -1 || end < start) return []; + if (start === -1 || end === -1 || end < start) return null; try { const arr = JSON.parse(s.slice(start, end + 1)); - return Array.isArray(arr) ? arr : []; + return Array.isArray(arr) ? arr : null; } catch { - return []; + return null; } } diff --git a/src/core/config.ts b/src/core/config.ts index 89493e040..4ca00cdc9 100644 --- a/src/core/config.ts +++ b/src/core/config.ts @@ -915,6 +915,8 @@ export const KNOWN_CONFIG_KEYS: readonly string[] = [ 'dream.synthesize.verdict_model', 'dream.synthesize.max_prompt_tokens', 'dream.synthesize.max_chunks_per_transcript', + // #2415: top-level namespace for synthesize/patterns output (default 'wiki'). + 'dream.synthesize.output_root', 'dream.synthesize.subagent_timeout_ms', 'dream.synthesize.subagent_wait_timeout_ms', 'dream.patterns.lookback_days', @@ -971,6 +973,10 @@ export const KNOWN_CONFIG_KEYS: readonly string[] = [ // operator had to discover --force by reading source. Same class as the // spend-controls registration above. 'auto_chronicle', + // #2606: chronicle judge output-token cap (default 4000). Event-dense + // pages overflowed the old hardcoded 1500 and were misrecorded as + // no_events; the cap is now configurable and truncation is surfaced. + 'chronicle.judge_max_tokens', // Takes bootstrap (v0.41.18.0, A12). The onboard remediation's two-gate // consent reads this key, and enabling it is the documented path to // `gbrain takes extract --from-pages` — same unregistered-key class. diff --git a/src/core/cycle.ts b/src/core/cycle.ts index d7410868d..a52efcd89 100644 --- a/src/core/cycle.ts +++ b/src/core/cycle.ts @@ -1682,6 +1682,9 @@ export async function runCycle( from: opts.synthFrom, to: opts.synthTo, bypassDreamGuard: opts.synthBypassDreamGuard, + // #1586: scope synthesized writes to the cycle's resolved source + // (explicit --source wins, else derived from the checkout dir). + sourceId: cycleSourceId, })); result.duration_ms = duration_ms; phaseResults.push(result); diff --git a/src/core/cycle/extract-facts.ts b/src/core/cycle/extract-facts.ts index 39b15d9ad..d805fdec4 100644 --- a/src/core/cycle/extract-facts.ts +++ b/src/core/cycle/extract-facts.ts @@ -11,15 +11,17 @@ * 1. Reads the markdown body (DB-side fetch via engine.getPage). * 2. Parses the `## Facts` fence with parseFactsFence. * 3. Maps ParsedFact → FenceExtractedFact via extractFactsFromFenceText. - * 4. Wipes the page's DB index via deleteFactsForPage. - * 5. Re-inserts via engine.insertFacts batch. + * 4. De-dupes rows by canonical (claim, source) content key. + * 5. Reconciles the page-scoped DB index: no-op when already in sync, + * insert only missing keys when possible, or wipe/reinsert when stale + * DB rows need cleanup (#1781 — the unconditional wipe-and-reinsert + * made every cycle non-idempotent, re-appending duplicate rows). * - * After the phase, the DB index for every affected page byte-matches - * the fence (modulo embeddings + runtime-derived fields). Pages with - * no fence go through delete-then-empty-insert — DB rows for that - * page coordinate are wiped; legacy NULL-source_markdown_slug rows - * survive because deleteFactsForPage targets source_markdown_slug = - * slug only. + * After the phase, the DB index for every affected page matches the + * fence's canonical (claim, source) row set (modulo embeddings + + * runtime-derived fields). Pages with no fence wipe DB rows for that + * page coordinate only; legacy NULL-source_markdown_slug rows survive + * because deleteFactsForPage targets source_markdown_slug = slug only. * * Empty-fence guard (Codex R2-#7): the phase refuses to do its * destructive reconciliation pass when legacy rows (row_num IS NULL, @@ -35,7 +37,11 @@ import type { BrainEngine } from '../engine.ts'; import { writeReceipt } from '../extract/receipt-writer.ts'; import { upsertExtractRollup } from '../extract/rollup-writer.ts'; import { parseFactsFence } from '../facts-fence.ts'; -import { extractFactsFromFenceText } from '../facts/extract-from-fence.ts'; +import { + extractFactsFromFenceText, + FENCE_SOURCE_DEFAULT, + type FenceExtractedFact, +} from '../facts/extract-from-fence.ts'; import { runPhantomRedirectPass, emptyPhantomPassResult, @@ -44,6 +50,51 @@ import { import { embed, isAvailable } from '../ai/gateway.ts'; import { isAborted } from '../abort-check.ts'; +interface ExistingPageFact { + fact: string; + source: string | null; + row_num: number | string | null; +} + +function factContentKey(fact: string, source: string | null | undefined): string { + return `${fact}\u0000${source ?? FENCE_SOURCE_DEFAULT}`; +} + +function dedupeFactsByContentKey(facts: FenceExtractedFact[]): FenceExtractedFact[] { + const seen = new Set(); + const deduped: FenceExtractedFact[] = []; + for (const fact of facts) { + const key = factContentKey(fact.fact, fact.source); + if (seen.has(key)) continue; + seen.add(key); + deduped.push(fact); + } + return deduped; +} + +/** + * Fence-owned DB rows for one page coordinate. Excludes `cli:`-origin + * conversation facts (#1928) — they are not fence-owned, so they must + * neither count as "stale" (which would force a wipe every cycle) nor + * be compared against the fence's row set. Mirrors the + * excludeSourcePrefixes filter deleteFactsForPage applies on the wipe. + */ +async function listExistingFactsForPage( + engine: BrainEngine, + slug: string, + sourceId: string, +): Promise { + return engine.executeRaw( + `SELECT fact, source, row_num + FROM facts + WHERE source_id = $1 + AND source_markdown_slug = $2 + AND COALESCE(source, '') NOT LIKE 'cli:%' + ORDER BY row_num ASC, id ASC`, + [sourceId, slug], + ); +} + export interface ExtractFactsOpts { /** Subset of slugs to reconcile. undefined = walk every page in the brain. */ slugs?: string[]; @@ -220,28 +271,70 @@ export async function runExtractFacts( if (parsed.facts.length > 0) result.pagesWithFacts += 1; - if (opts.dryRun) continue; - - // Wipe-and-reinsert per page. The delete targets source_markdown_slug = - // slug only, so NULL-source_markdown_slug legacy rows survive (the - // partial-UNIQUE-index keyspace). #1928: `cli:`-origin facts (conversation - // facts from extract-conversation-facts) are NOT fence-owned — the page - // carries no `## Facts` fence to recreate them — so they MUST survive this - // reconcile. Exclude them from the wipe. - const deleted = await engine.deleteFactsForPage(slug, sourceId, { - excludeSourcePrefixes: ['cli:'], - }); - result.factsDeleted += deleted.deleted; - - if (parsed.facts.length === 0) continue; - // v0.35.4 (D-ENG-1) — thread page.effective_date as the fallback // valid_from. Without this, fence rows without explicit `validFrom:` // land with `valid_from = now()` (import timestamp) and every // trajectory query against the page returns import dates instead of // claim dates. const pageEffectiveDate = page.effective_date ? new Date(page.effective_date) : null; - const extracted = extractFactsFromFenceText(parsed.facts, slug, sourceId, { pageEffectiveDate }); + const extracted = dedupeFactsByContentKey( + extractFactsFromFenceText(parsed.facts, slug, sourceId, { pageEffectiveDate }), + ); + + if (opts.dryRun) continue; + + // #1781 — reconcile instead of unconditional wipe-and-reinsert. Compare + // the fence's canonical (claim, source) row set against the page's + // fence-owned DB rows: no-op when already in sync, insert only missing + // keys when possible, wipe/reinsert only when stale rows need cleanup. + const existing = await listExistingFactsForPage(engine, slug, sourceId); + const existingKeys = new Set(existing.map(f => factContentKey(f.fact, f.source))); + const desiredByKey = new Map(extracted.map(f => [factContentKey(f.fact, f.source), f])); + + if (extracted.length === 0) { + if (existing.length > 0) { + // The delete targets source_markdown_slug = slug only, so + // NULL-source_markdown_slug legacy rows survive (the + // partial-UNIQUE-index keyspace). #1928: `cli:`-origin facts + // (conversation facts from extract-conversation-facts) are NOT + // fence-owned — the page carries no `## Facts` fence to recreate + // them — so they MUST survive this reconcile. + const deleted = await engine.deleteFactsForPage(slug, sourceId, { + excludeSourcePrefixes: ['cli:'], + }); + result.factsDeleted += deleted.deleted; + } + continue; + } + + const hasStaleExisting = existing.some(f => !desiredByKey.has(factContentKey(f.fact, f.source))); + const hasDuplicateExisting = existing.length !== existingKeys.size; + const hasRowNumDrift = existing.some(f => { + const desired = desiredByKey.get(factContentKey(f.fact, f.source)); + return desired !== undefined && Number(f.row_num) !== desired.row_num; + }); + + if ( + existing.length === extracted.length && + !hasStaleExisting && + !hasDuplicateExisting && + !hasRowNumDrift + ) { + continue; + } + + let toInsert = extracted.filter(f => !existingKeys.has(factContentKey(f.fact, f.source))); + if (hasStaleExisting || hasDuplicateExisting || hasRowNumDrift) { + // Fall back to the legacy page-level reconcile when old DB rows must + // be removed. Same delete scoping as above: legacy + // NULL-source_markdown_slug rows and `cli:`-origin conversation + // facts (#1928) survive. + const deleted = await engine.deleteFactsForPage(slug, sourceId, { + excludeSourcePrefixes: ['cli:'], + }); + result.factsDeleted += deleted.deleted; + toInsert = extracted; + } // v0.35.4 (D-CDX-3) — batch-embed before insert. Without this, // cycle-inserted facts land with `embedding = NULL`, which breaks @@ -250,17 +343,17 @@ export async function runExtractFacts( // unavailable (no API key configured), facts still insert with // NULL embeddings — drift_score gracefully returns null and // clustering falls back to recency. - if (isAvailable('embedding') && extracted.length > 0) { + if (isAvailable('embedding') && toInsert.length > 0) { try { - const texts = extracted.map(e => e.fact); + const texts = toInsert.map(e => e.fact); // #1972: forward the abort signal so a cancelled cycle's in-flight // batch embed (a network call) is itself abortable, not just the loop. const embeddings = await embed(texts, { abortSignal: opts.signal }); // Defensive: embed should return one vector per input; if the // gateway returns a partial array (provider partial-batch retry // returning fewer than requested), only fill what we have. - for (let i = 0; i < extracted.length && i < embeddings.length; i++) { - extracted[i].embedding = embeddings[i]; + for (let i = 0; i < toInsert.length && i < embeddings.length; i++) { + toInsert[i].embedding = embeddings[i]; } } catch (err) { // Embedding failure is non-fatal — facts still get inserted, just @@ -271,7 +364,9 @@ export async function runExtractFacts( } } - const inserted = await engine.insertFacts(extracted, { source_id: sourceId }); // gbrain-allow-direct-insert: extract_facts cycle phase reconciles fence → DB + if (toInsert.length === 0) continue; + + const inserted = await engine.insertFacts(toInsert, { source_id: sourceId }); // gbrain-allow-direct-insert: extract_facts cycle phase reconciles fence → DB result.factsInserted += inserted.inserted; } diff --git a/src/core/cycle/patterns.ts b/src/core/cycle/patterns.ts index 4a39779e1..d96584dad 100644 --- a/src/core/cycle/patterns.ts +++ b/src/core/cycle/patterns.ts @@ -19,7 +19,7 @@ */ import { join, dirname } from 'node:path'; -import { mkdirSync, writeFileSync, existsSync, readFileSync } from 'node:fs'; +import { mkdirSync, writeFileSync } from 'node:fs'; import type { BrainEngine } from '../engine.ts'; import type { PhaseResult, PhaseError } from '../cycle.ts'; import { MinionQueue } from '../minions/queue.ts'; @@ -27,6 +27,9 @@ import { waitForCompletion, TimeoutError } from '../minions/wait-for-completion. import type { MinionJobInput, SubagentHandlerData } from '../minions/types.ts'; import { serializeMarkdown } from '../markdown.ts'; import type { Page, PageType } from '../types.ts'; +// #2415: allow-list + output-root resolution shared with the synthesize +// phase — both phases must agree on the configured namespace. +import { loadAllowedSlugPrefixes, loadOutputRoot } from './synthesize.ts'; import { probeChatModel } from '../ai/gateway.ts'; import { normalizeModelId } from '../model-id.ts'; @@ -49,7 +52,7 @@ export async function runPhasePatterns( } // Gather reflections within lookback window. - const reflections = await gatherReflections(engine, config.lookbackDays); + const reflections = await gatherReflections(engine, config.lookbackDays, config.outputRoot); if (reflections.length < config.minEvidence) { return skipped( 'insufficient_evidence', @@ -81,7 +84,7 @@ export async function runPhasePatterns( return skipped('no_provider', `pattern detection skipped: ${probe.detail}`); } - const allowedSlugPrefixes = await loadAllowedSlugPrefixes(); + const allowedSlugPrefixes = await loadAllowedSlugPrefixes(config.outputRoot); if (allowedSlugPrefixes.length === 0) { return failed(makeError('InternalError', 'NO_ALLOWLIST', 'skills/_brain-filing-rules.json missing dream_synthesize_paths.globs')); @@ -89,7 +92,7 @@ export async function runPhasePatterns( const queue = new MinionQueue(engine); const data: SubagentHandlerData = { - prompt: buildPatternsPrompt(reflections, config.minEvidence), + prompt: buildPatternsPrompt(reflections, config.minEvidence, config.outputRoot), model: config.model, max_turns: 30, allowed_slug_prefixes: allowedSlugPrefixes, @@ -182,6 +185,8 @@ interface PatternsConfig { lookbackDays: number; minEvidence: number; model: string; + /** #2415: shared output namespace (dream.synthesize.output_root, default 'wiki'). */ + outputRoot: string; /** #1594-family: subagent job timeout, config `dream.patterns.subagent_timeout_ms`. */ subagentTimeoutMs: number; /** #1594-family: waitForCompletion timeout, config `dream.patterns.subagent_wait_timeout_ms`. */ @@ -216,6 +221,7 @@ async function loadPatternsConfig(engine: BrainEngine): Promise lookbackDays: lookbackStr ? Math.max(1, parseInt(lookbackStr, 10) || 30) : 30, minEvidence: minEvidenceStr ? Math.max(1, parseInt(minEvidenceStr, 10) || 3) : 3, model, + outputRoot: await loadOutputRoot(engine), subagentTimeoutMs: await getNumberConfig( engine, 'dream.patterns.subagent_timeout_ms', DEFAULT_PATTERNS_SUBAGENT_TIMEOUT_MS, ), @@ -236,16 +242,19 @@ interface ReflectionRef { async function gatherReflections( engine: BrainEngine, lookbackDays: number, + outputRoot = 'wiki', ): Promise { const since = new Date(Date.now() - lookbackDays * 24 * 60 * 60 * 1000).toISOString(); + // #2415: reflections live under the configured output root (bound as a + // parameter; outputRoot is slug-grammar-validated by loadOutputRoot). const rows = await engine.executeRaw<{ slug: string; title: string | null; compiled_truth: string | null }>( `SELECT slug, title, compiled_truth FROM pages - WHERE slug LIKE 'wiki/personal/reflections/%' + WHERE slug LIKE $2 AND updated_at >= $1::timestamptz ORDER BY updated_at DESC LIMIT 100`, - [since], + [since, `${outputRoot}/personal/reflections/%`], ); return rows.map(r => ({ slug: r.slug, @@ -256,7 +265,7 @@ async function gatherReflections( // ── Prompt ──────────────────────────────────────────────────────────── -function buildPatternsPrompt(reflections: ReflectionRef[], minEvidence: number): string { +function buildPatternsPrompt(reflections: ReflectionRef[], minEvidence: number, outputRoot = 'wiki'): string { const today = new Date().toISOString().slice(0, 10); const corpus = reflections .map((r, i) => `### ${i + 1}. [[${r.slug}]] — ${r.title}\n${r.excerpt}`) @@ -266,15 +275,15 @@ function buildPatternsPrompt(reflections: ReflectionRef[], minEvidence: number): OUTPUT POLICY - Only name a pattern if it appears in at least ${minEvidence} DISTINCT reflections. -- Each pattern page MUST cite the reflections that constitute its evidence (use [[wiki/personal/reflections/...]] wikilinks). +- Each pattern page MUST cite the reflections that constitute its evidence (use [[${outputRoot}/personal/reflections/...]] wikilinks). - Use \`search\` to check whether a similar pattern page already exists; if yes, update it (use the same slug). If no, create a new one. -- Pattern slug format: \`wiki/personal/patterns/\` (lowercase alphanumeric + hyphens; no underscores, no extension, no date). +- Pattern slug format: \`${outputRoot}/personal/patterns/\` (lowercase alphanumeric + hyphens; no underscores, no extension, no date). - A "pattern" is a recurring theme, anxiety, decision pattern, relationship dynamic, or self-knowledge motif. NOT a single insight. NOT a list of unrelated topics. DO NOT WRITE - A "patterns from today" digest (that's the dream-cycle-summaries page; not your job). - Patterns with <${minEvidence} reflections cited. -- Anything outside wiki/personal/patterns/. +- Anything outside ${outputRoot}/personal/patterns/. CONTEXT - Today: ${today} @@ -365,27 +374,6 @@ function renderPageToMarkdown(page: Page, tags: string[]): string { ); } -// ── Allow-list (shared with synthesize.ts) ─────────────────────────── - -async function loadAllowedSlugPrefixes(): Promise { - const candidates = [ - join(process.cwd(), 'skills', '_brain-filing-rules.json'), - join(__dirname, '..', '..', '..', 'skills', '_brain-filing-rules.json'), - ]; - for (const path of candidates) { - if (!existsSync(path)) continue; - try { - const raw = readFileSync(path, 'utf8'); - const parsed = JSON.parse(raw) as { dream_synthesize_paths?: { globs?: unknown } }; - const globs = parsed?.dream_synthesize_paths?.globs; - if (Array.isArray(globs) && globs.every(g => typeof g === 'string')) { - return globs as string[]; - } - } catch { /* try next */ } - } - return []; -} - // ── Status helpers ─────────────────────────────────────────────────── function ok(summary: string, details: Record = {}): PhaseResult { diff --git a/src/core/cycle/synthesize-concepts.ts b/src/core/cycle/synthesize-concepts.ts index dbc352b7b..b88cc06d5 100644 --- a/src/core/cycle/synthesize-concepts.ts +++ b/src/core/cycle/synthesize-concepts.ts @@ -23,7 +23,13 @@ import type { PhaseResult } from '../cycle.ts'; import type { ProgressReporter } from '../progress.ts'; import { writeReceipt } from '../extract/receipt-writer.ts'; import { upsertExtractRollup } from '../extract/rollup-writer.ts'; -import { chat as gatewayChat } from '../ai/gateway.ts'; +import { chat as gatewayChat, isAvailable } from '../ai/gateway.ts'; +// #2163: concept pages route through importFromContent (the same +// parse→chunk→embed pipeline put_page uses) instead of a bare engine.putPage, +// so they land in the retrieval surface (content_chunks + embeddings) where +// source-boost's 1.3× 'concepts/' weighting can actually reach them. +import { importFromContent } from '../import-file.ts'; +import { serializeMarkdown } from '../markdown.ts'; const DEFAULT_BUDGET_USD = 1.5; const TIER_T1_MIN = 10; @@ -216,19 +222,23 @@ export async function runPhaseSynthesizeConcepts( if (!opts.dryRun) { const title = group.conceptSlug.split('/').pop() ?? group.conceptSlug; - await engine.putPage(`concepts/${title}`, { - title: title.replace(/-/g, ' '), - type: 'concept', - compiled_truth: narrative, - frontmatter: { - type: 'concept', + // #2163: serialize to markdown and import via the canonical pipeline so + // the page is chunked (+ embedded when a provider is configured) — + // mirrors put_page's isAvailable('embedding') → noEmbed gate. + const md = serializeMarkdown( + { tier: group.tier, mention_count: group.atomTitles.length, composite_score: group.atomTitles.length, synthesized_at: new Date().toISOString(), synthesized_by: 'synthesize_concepts-v0.41', }, - timeline: '', + narrative, + '', + { type: 'concept', title: title.replace(/-/g, ' '), tags: [] }, + ); + await importFromContent(engine, `concepts/${title}`, md, { + noEmbed: !isAvailable('embedding'), }); } conceptsWritten++; diff --git a/src/core/cycle/synthesize.ts b/src/core/cycle/synthesize.ts index 94564379e..59aad6630 100644 --- a/src/core/cycle/synthesize.ts +++ b/src/core/cycle/synthesize.ts @@ -244,6 +244,14 @@ export interface SynthesizePhaseOpts { * the synthesize loop. Caller must opt in explicitly. */ bypassDreamGuard?: boolean; + /** + * #1586: the cycle's resolved brain source (cycleSourceId from cycle.ts — + * explicit --source wins, else derived from the checkout dir). Threaded to + * every subagent child as `source_id` so put_page writes land in this + * source, and stamped onto collected refs so reverse-writes read the + * correct (source_id, slug) row. Unset → legacy 'default'. + */ + sourceId?: string; } export async function runPhaseSynthesize( @@ -399,7 +407,7 @@ export async function runPhaseSynthesize( // Fan-out: submit one subagent per worth-processing transcript (or one // per chunk for transcripts that exceed the model's per-prompt budget). - const allowedSlugPrefixes = await loadAllowedSlugPrefixes(); + const allowedSlugPrefixes = await loadAllowedSlugPrefixes(config.outputRoot); if (allowedSlugPrefixes.length === 0) { return failed(makeError('InternalError', 'NO_ALLOWLIST', 'skills/_brain-filing-rules.json missing dream_synthesize_paths.globs')); @@ -462,10 +470,13 @@ export async function runPhaseSynthesize( : config.model; for (let i = 0; i < chunks.length; i++) { const childData: SubagentHandlerData = { - prompt: buildSynthesisPrompt(t, chunks[i], i, chunks.length, priorContradictionsBlock), + prompt: buildSynthesisPrompt(t, chunks[i], i, chunks.length, priorContradictionsBlock, config.outputRoot), model: subagentModel, max_turns: 30, allowed_slug_prefixes: allowedSlugPrefixes, + // #1586: scope every child tool call to the cycle's resolved source + // so put_page writes land there instead of the hardcoded 'default'. + ...(opts.sourceId ? { source_id: opts.sourceId } : {}), }; // Idempotency key parity: // - single-chunk → legacy `dream:synth::` (byte- @@ -524,20 +535,29 @@ export async function runPhaseSynthesize( // bare-hash slugs to `-c` so chunked siblings can't collide // even if Sonnet drops the chunk suffix. // v0.32.8: refs carry source_id so reverseWriteRefs picks the correct - // (source, slug) row (currently always 'default' from subagent put_page). - const writtenRefs = await collectChildPutPageSlugs(engine, childIds, chunkInfo); + // (source, slug) row. #1586: refs are stamped with the cycle's resolved + // source (children write there via SubagentHandlerData.source_id). + const cycleSourceId = opts.sourceId ?? 'default'; + const writtenRefs = await collectChildPutPageSlugs(engine, childIds, chunkInfo, cycleSourceId); + + const summaryDate = opts.date ?? today(); + + // #2569: persist the dream-output identity marker into the DB frontmatter + // of every child-written page BEFORE reverse-rendering, so generated pages + // are queryable (`frontmatter->>'dream_generated'`) and a later put_page + // write-through (which re-renders from the DB row) can't erase the stamp. + await stampDreamProvenance(engine, writtenRefs, summaryDate); // Dual-write: reverse-render each DB row → markdown file. - const reverseWriteCount = await reverseWriteRefs(engine, opts.brainDir, writtenRefs); + const reverseWriteCount = await reverseWriteRefs(engine, opts.brainDir, writtenRefs, cycleSourceId); // Summary index page (deterministic; orchestrator-written via direct // engine.putPage so no allow-list path needed). - const summaryDate = opts.date ?? today(); const summarySlug = `dream-cycle-summaries/${summaryDate}`; // Back-compat: writeSummaryPage takes string[] for display; map refs back to slugs. const writtenSlugs = writtenRefs.map(r => r.slug); if (SUMMARY_SLUG_RE.test(summarySlug)) { - await writeSummaryPage(engine, opts.brainDir, summarySlug, summaryDate, writtenSlugs, childOutcomes); + await writeSummaryPage(engine, opts.brainDir, summarySlug, summaryDate, writtenSlugs, childOutcomes, cycleSourceId); } // Write completion timestamp ON SUCCESS only. @@ -595,10 +615,29 @@ interface SynthConfig { * `dream.synthesize.max_chunks_per_transcript`. */ maxChunksPerTranscript: number; + /** + * #2415: top-level namespace for synthesized output (reflections, originals, + * patterns). Config key `dream.synthesize.output_root`; default 'wiki' — + * zero behavior change unless set. No trailing slash. Must satisfy the slug + * grammar; invalid values fall back to 'wiki' with a stderr warning. + */ + outputRoot: string; subagentTimeoutMs: number; subagentWaitTimeoutMs: number; } +/** #2415: shared output-root resolution (synthesize + patterns phases). */ +export async function loadOutputRoot(engine: BrainEngine): Promise { + const raw = await engine.getConfig('dream.synthesize.output_root'); + if (!raw) return 'wiki'; + const trimmed = raw.trim().replace(/^\/+|\/+$/g, ''); + if (SUMMARY_SLUG_RE.test(trimmed)) return trimmed; + process.stderr.write( + `[dream] dream.synthesize.output_root "${raw}" is not a valid slug prefix; falling back to "wiki".\n`, + ); + return 'wiki'; +} + async function loadSynthConfig(engine: BrainEngine): Promise { const enabledRaw = await engine.getConfig('dream.synthesize.enabled'); const corpusDir = await engine.getConfig('dream.synthesize.session_corpus_dir'); @@ -672,6 +711,7 @@ async function loadSynthConfig(engine: BrainEngine): Promise { cooldownHours: cooldownHoursStr ? Math.max(0, parseInt(cooldownHoursStr, 10) || 12) : 12, maxPromptTokens, maxChunksPerTranscript, + outputRoot: await loadOutputRoot(engine), subagentTimeoutMs, subagentWaitTimeoutMs, }; @@ -704,7 +744,13 @@ async function checkCooldown( // ── Allow-list source of truth ─────────────────────────────────────── -async function loadAllowedSlugPrefixes(): Promise { +/** + * #2415: `outputRoot` remaps the canonical `wiki/`-rooted globs to the + * configured namespace (e.g. `notes/personal/reflections/*`). Default 'wiki' + * returns the globs verbatim. Shared by the patterns phase (imported there — + * the two phases must enforce the same allow-list). + */ +export async function loadAllowedSlugPrefixes(outputRoot = 'wiki'): Promise { // Search a few known locations relative to the binary / repo. The first // hit wins; if none found, return []. const candidates = [ @@ -718,7 +764,10 @@ async function loadAllowedSlugPrefixes(): Promise { const parsed = JSON.parse(raw) as { dream_synthesize_paths?: { globs?: unknown } }; const globs = parsed?.dream_synthesize_paths?.globs; if (Array.isArray(globs) && globs.every(g => typeof g === 'string')) { - return globs as string[]; + if (outputRoot === 'wiki') return globs as string[]; + return (globs as string[]).map(g => + g.startsWith('wiki/') ? `${outputRoot}/${g.slice('wiki/'.length)}` : g, + ); } } catch { /* try next */ } } @@ -966,6 +1015,7 @@ function buildSynthesisPrompt( chunkIdx: number, chunkTotal: number, priorContradictionsBlock = '', + outputRoot = 'wiki', ): string { const dateHint = t.inferredDate ?? today(); const baseSlugSegment = sanitizeForSlug(t.basename) || `session-${dateHint}`; @@ -994,10 +1044,10 @@ OUTPUT POLICY (ALL of these are required) TASKS A. Reflections (self-knowledge, pattern recognition, emotional processing): - slug: \`wiki/personal/reflections/${dateHint}--${hashSuffix}\` + slug: \`${outputRoot}/personal/reflections/${dateHint}--${hashSuffix}\` B. Originals (new ideas, frames, theses, mental models): - slug: \`wiki/originals/ideas/${dateHint}--${hashSuffix}\` + slug: \`${outputRoot}/originals/ideas/${dateHint}--${hashSuffix}\` C. People mentions: search first; if a page exists, do not put_page over it (the orchestrator handles people enrichment via timeline entries — your job is the reflection/original synthesis, NOT modifying existing person pages). @@ -1038,6 +1088,7 @@ async function collectChildPutPageSlugs( engine: BrainEngine, childIds: number[], chunkInfo: Map, + sourceId = 'default', ): Promise> { if (childIds.length === 0) return []; // Raw fetch — NO SELECT DISTINCT. Preserves per-child slug duplicates so @@ -1047,10 +1098,10 @@ async function collectChildPutPageSlugs( // // v0.32.8: returns Array<{slug, source_id}> instead of string[]. Subagent // put_page tool schema doesn't expose source_id (subagents are scoped to - // a single source); default to 'default' for the current dream-cycle - // product behavior. Threading the source_id through reverseWriteRefs - // guarantees getPage targets the correct (source, slug) row instead of - // the first DB match. + // a single source). #1586: the orchestrator scopes each child to the + // cycle's resolved source via SubagentHandlerData.source_id, and stamps + // the SAME source here so reverseWriteRefs / provenance reads target the + // correct (source_id, slug) row. Unset → legacy 'default'. const rows = await engine.executeRaw<{ job_id: number; slug: string }>( `SELECT job_id, COALESCE(input->>'slug', (input #>> '{}')::jsonb->>'slug') AS slug @@ -1066,7 +1117,7 @@ async function collectChildPutPageSlugs( const ci = chunkInfo.get(r.job_id); rewritten.add(ci ? rewriteChunkedSlug(r.slug, ci.hash6, ci.idx) : r.slug); } - return Array.from(rewritten).sort().map(slug => ({ slug, source_id: 'default' })); + return Array.from(rewritten).sort().map(slug => ({ slug, source_id: sourceId })); } /** @@ -1095,12 +1146,52 @@ async function hasLegacySingleChunkCompletion( return rows.length > 0; } +// ── Dream-provenance DB stamp (#2569) ──────────────────────────────── + +/** + * Persist the dream-output identity marker (`dream_generated: true` + + * `dream_cycle_date`) into the `pages.frontmatter` JSONB row for every page + * a synthesize child wrote. Render-time `frontmatterOverrides` alone only + * reach the markdown FILE — the DB row stayed unstamped, so DB consumers + * couldn't enumerate generated pages and a later put_page write-through + * (which re-renders from the DB row) silently erased the marker. + * + * Plain UPDATE through executeRawJsonb (raw object bound to $3::jsonb — + * never JSON.stringify into a ::jsonb cast; engine-parity safe, no new + * engine method). Best-effort per row: a stamp failure never kills the + * phase (the render-time override still covers the file). + */ +async function stampDreamProvenance( + engine: BrainEngine, + refs: Array<{ slug: string; source_id: string }>, + cycleDate: string, +): Promise { + if (refs.length === 0) return; + const { executeRawJsonb } = await import('../sql-query.ts'); + for (const { slug, source_id } of refs) { + try { + await executeRawJsonb( + engine, + `UPDATE pages + SET frontmatter = COALESCE(frontmatter, '{}'::jsonb) || $3::jsonb + WHERE slug = $1 AND source_id = $2`, + [slug, source_id], + [{ dream_generated: true, dream_cycle_date: cycleDate }], + ); + } catch (e) { + const msg = e instanceof Error ? e.message : String(e); + process.stderr.write(`[dream] provenance stamp ${slug}@${source_id} failed: ${msg}\n`); + } + } +} + // ── Reverse-write DB rows → markdown files ─────────────────────────── async function reverseWriteRefs( engine: BrainEngine, brainDir: string, refs: Array<{ slug: string; source_id: string }>, + nativeSourceId = 'default', ): Promise { let count = 0; for (const { slug, source_id } of refs) { @@ -1111,10 +1202,11 @@ async function reverseWriteRefs( const tags = await engine.getTags(slug, { sourceId: source_id }); try { const md = renderPageToMarkdown(page, tags); - // v0.32.8 F6: non-default sources land at brainDir/.sources//.md - // so same-slug-different-source pages don't collide. Default-source - // pages stay at brainDir/.md so single-source brains see no change. - const filePath = source_id === 'default' + // v0.32.8 F6: foreign-source pages land at brainDir/.sources//.md + // so same-slug-different-source pages don't collide. Pages belonging to + // the cycle's own source (#1586: brainDir IS that source's checkout — + // legacy 'default' when unscoped) stay at brainDir/.md. + const filePath = source_id === nativeSourceId ? join(brainDir, `${slug}.md`) : join(brainDir, '.sources', source_id, `${slug}.md`); mkdirSync(dirname(filePath), { recursive: true }); @@ -1161,6 +1253,7 @@ async function writeSummaryPage( summaryDate: string, writtenSlugs: string[], childOutcomes: Array<{ jobId: number; status: string }>, + sourceId = 'default', ): Promise { const completed = childOutcomes.filter(c => c.status === 'completed').length; const failed = childOutcomes.length - completed; @@ -1198,13 +1291,15 @@ async function writeSummaryPage( // unnecessarily; we go straight to the engine. const { parseMarkdown } = await import('../markdown.ts'); const parsed = parseMarkdown(fullMarkdown); + // #1586: summary lands in the cycle's resolved source too — otherwise the + // children live in the named source while the index drifts to 'default'. await engine.putPage(summarySlug, { type: parsed.type, title: parsed.title, compiled_truth: parsed.compiled_truth, timeline: parsed.timeline, frontmatter: parsed.frontmatter, - }); + }, { sourceId }); // Also write to disk (orchestrator dual-write). try { @@ -1269,4 +1364,7 @@ function makeError(cls: string, code: string, message: string, hint?: string): P // double-encoded jsonb regression). Not part of the runtime contract. export const __testing = { collectChildPutPageSlugs, + buildSynthesisPrompt, + stampDreamProvenance, + reverseWriteRefs, }; diff --git a/src/core/fts-language.ts b/src/core/fts-language.ts new file mode 100644 index 000000000..569c0410f --- /dev/null +++ b/src/core/fts-language.ts @@ -0,0 +1,69 @@ +/** + * Full-text search language configuration. + * + * Postgres tsvector/tsquery require a text search configuration name (e.g. + * 'english', 'portuguese', 'spanish'). Historically GBrain hardcoded + * 'english' across engines and trigger functions, which broke search + * quality for non-English brains (no stemming, no stop-word removal). + * + * This helper centralizes the choice. Default stays 'english' for backward + * compatibility — only users who set GBRAIN_FTS_LANGUAGE see different + * behavior. + * + * Custom configs (e.g. accent-insensitive 'pt_br' built with unaccent + + * portuguese stemmer) are supported as long as the configuration exists + * in the target Postgres instance. See docs/guides/multi-language-fts.md + * for setup instructions. + * + * Validation: only allow lowercase letters, digits, and underscores. This + * prevents SQL injection when the value is interpolated into queries + * (Postgres tsvector functions don't accept parameterized config names — + * they must be literals or identifiers). + */ + +const VALID_CONFIG_NAME = /^[a-z][a-z0-9_]*$/; +const DEFAULT_LANGUAGE = 'english'; + +let cachedLanguage: string | null = null; + +/** + * Returns the configured Postgres text search configuration name. + * + * Resolution order: + * 1. process.env.GBRAIN_FTS_LANGUAGE (if set and valid) + * 2. 'english' (default — preserves existing behavior) + * + * The return value is safe to interpolate directly into SQL because it + * passes the VALID_CONFIG_NAME guard. If validation fails, falls back to + * the default and emits a one-time warning. + * + * Cached on first call; reset with `resetFtsLanguageCache()` (test only). + */ +export function getFtsLanguage(): string { + if (cachedLanguage !== null) return cachedLanguage; + + const raw = process.env.GBRAIN_FTS_LANGUAGE?.trim(); + if (!raw) { + cachedLanguage = DEFAULT_LANGUAGE; + return cachedLanguage; + } + + if (!VALID_CONFIG_NAME.test(raw)) { + console.warn( + `[gbrain] Invalid GBRAIN_FTS_LANGUAGE='${raw}' — must match /^[a-z][a-z0-9_]*$/. ` + + `Falling back to '${DEFAULT_LANGUAGE}'.` + ); + cachedLanguage = DEFAULT_LANGUAGE; + return cachedLanguage; + } + + cachedLanguage = raw; + return cachedLanguage; +} + +/** + * Resets the cached language. Tests only — don't use in production code. + */ +export function resetFtsLanguageCache(): void { + cachedLanguage = null; +} diff --git a/src/core/migrate.ts b/src/core/migrate.ts index c9c3a1581..7b95a5767 100644 --- a/src/core/migrate.ts +++ b/src/core/migrate.ts @@ -1,5 +1,6 @@ import type { BrainEngine } from './engine.ts'; import { slugifyPath } from './sync.ts'; +import { getFtsLanguage } from './fts-language.ts'; /** * Schema migrations — run automatically on initSchema(). @@ -5505,6 +5506,96 @@ export const MIGRATIONS: Migration[] = [ WHERE dimension IS NOT NULL; `, }, + { + version: 123, + name: 'configurable_fts_language', + // Recreate the two search_vector trigger functions using the language + // configured via GBRAIN_FTS_LANGUAGE (default 'english'). Idempotent: + // CREATE OR REPLACE swaps the function body atomically; no trigger + // recreation needed since the trigger references the function by name. + // + // Why a handler instead of a static SQL string: Postgres tsvector + // functions don't accept parameterized config names — the language + // must be a literal in the SQL. getFtsLanguage() validates the value + // (lowercase letters/digits/underscores only) before interpolation. + // + // Function bodies mirror schema.sql / pglite-schema.ts exactly — + // INCLUDING the `SET search_path = pg_catalog, public` hardening from + // v120/#1647 (CREATE OR REPLACE resets proconfig, so omitting it here + // would silently strip the hardening on every upgraded brain). Only + // the text-search config name is parameterized. Keep all copies in + // sync when the trigger logic changes. + // + // Backfill: after recreating the functions, re-tokenize existing rows + // under the new language. Skipped when the configured language is + // 'english' (trigger output identical — re-tokenizing is wasted I/O). + // To change language after this migration has run, use + // `gbrain reindex-search-vector`. + sql: '', + handler: async (engine) => { + const lang = getFtsLanguage(); + + const recreatePagesFn = ` + CREATE OR REPLACE FUNCTION update_page_search_vector() RETURNS trigger SET search_path = pg_catalog, public AS $fn$ + DECLARE + timeline_text TEXT; + BEGIN + SELECT coalesce(string_agg(summary || ' ' || detail, ' '), '') + INTO timeline_text + FROM timeline_entries + WHERE page_id = NEW.id; + + NEW.search_vector := + setweight(to_tsvector('${lang}', coalesce(NEW.title, '')), 'A') || + setweight(to_tsvector('${lang}', coalesce(NEW.compiled_truth, '')), 'B') || + setweight(to_tsvector('${lang}', coalesce(NEW.timeline, '')), 'C') || + setweight(to_tsvector('${lang}', coalesce(timeline_text, '')), 'C'); + + RETURN NEW; + END; + $fn$ LANGUAGE plpgsql; + `; + + const recreateChunksFn = ` + CREATE OR REPLACE FUNCTION update_chunk_search_vector() RETURNS TRIGGER SET search_path = pg_catalog, public AS $fn$ + BEGIN + NEW.search_vector := + setweight(to_tsvector('${lang}', COALESCE(NEW.doc_comment, '')), 'A') || + setweight(to_tsvector('${lang}', COALESCE(NEW.symbol_name_qualified, '')), 'A') || + setweight(to_tsvector('${lang}', COALESCE(NEW.chunk_text, '')), 'B'); + RETURN NEW; + END; + $fn$ LANGUAGE plpgsql; + `; + + await engine.executeRaw(recreatePagesFn); + await engine.executeRaw(recreateChunksFn); + + if (lang === 'english') { + console.log(` v123: trigger functions recreated with language='english' (default — no backfill needed)`); + return; + } + + // Backfill existing rows under the new tokenizer. UPDATE-to-same-value + // re-fires the pages trigger; chunks are rewritten directly with the + // same expression as the trigger. + await engine.executeRaw(` + UPDATE pages SET id = id + WHERE search_vector IS NOT NULL; + `); + + await engine.executeRaw(` + UPDATE content_chunks + SET search_vector = + setweight(to_tsvector('${lang}', COALESCE(doc_comment, '')), 'A') || + setweight(to_tsvector('${lang}', COALESCE(symbol_name_qualified, '')), 'A') || + setweight(to_tsvector('${lang}', COALESCE(chunk_text, '')), 'B') + WHERE search_vector IS NOT NULL; + `); + + console.log(` v123: trigger functions recreated with language='${lang}' + backfilled existing rows`); + }, + }, ]; export const LATEST_VERSION = MIGRATIONS.length > 0 diff --git a/src/core/minions/handlers/subagent.ts b/src/core/minions/handlers/subagent.ts index 53ef86433..3852469ed 100644 --- a/src/core/minions/handlers/subagent.ts +++ b/src/core/minions/handlers/subagent.ts @@ -272,6 +272,8 @@ export function makeSubagentHandler(deps: SubagentDeps) { config, brainId: data.brain_id, allowedSlugPrefixes: data.allowed_slug_prefixes, + // #1586: cycle-resolved source scope for tool-call OperationContexts. + sourceId: data.source_id, }); const toolDefs = data.allowed_tools && data.allowed_tools.length > 0 ? filterAllowedTools(registry, data.allowed_tools) diff --git a/src/core/minions/tools/brain-allowlist.ts b/src/core/minions/tools/brain-allowlist.ts index ffcce65e3..2b1a0f195 100644 --- a/src/core/minions/tools/brain-allowlist.ts +++ b/src/core/minions/tools/brain-allowlist.ts @@ -27,6 +27,7 @@ import type { GBrainConfig } from '../../config.ts'; import { operations } from '../../operations.ts'; import type { Operation, OperationContext } from '../../operations.ts'; import { paramDefToSchema } from '../../../mcp/tool-defs.ts'; +import { validateSourceId } from '../../utils.ts'; import type { ToolCtx, ToolDef } from '../types.ts'; /** @@ -201,6 +202,13 @@ export interface BuildBrainToolsOpts { * SubagentHandlerData.allowed_slug_prefixes via the handler. */ allowedSlugPrefixes?: readonly string[]; + /** + * Brain source every tool-call OperationContext is scoped to (#1586). + * Trusted (flows from SubagentHandlerData.source_id, which only + * PROTECTED_JOB_NAMES-gated submitters can set); validated at build time. + * Unset → legacy 'default'. + */ + sourceId?: string; } interface OpContextDeps { @@ -211,6 +219,7 @@ interface OpContextDeps { signal?: AbortSignal; brainId?: string; allowedSlugPrefixes?: readonly string[]; + sourceId?: string; } function buildOpContext(deps: OpContextDeps): OperationContext { @@ -224,7 +233,8 @@ function buildOpContext(deps: OpContextDeps): OperationContext { }, dryRun: false, remote: true, // match MCP trust boundary for auto-link skip - sourceId: 'default', // v0.34 D4: required; subagent tools default to host source + // #1586: cycle-resolved source when provided; legacy host default else. + sourceId: deps.sourceId ?? 'default', jobId: deps.jobId, subagentId: deps.subagentId, viaSubagent: true, // FAIL-CLOSED: put_page etc. enforce namespace @@ -248,6 +258,11 @@ export function buildBrainTools(opts: BuildBrainToolsOpts): ToolDef[] { op => BRAIN_TOOL_ALLOWLIST.has(op.name) && filter.has(op.name), ); + // #1586: fail fast on a malformed source id before any tool executes + // (defense-in-depth — the seam is trusted, but the value round-trips + // through the job payload). + if (opts.sourceId !== undefined) validateSourceId(opts.sourceId); + return picked.map(op => { const schema = op.name === 'put_page' ? namespacedPutPageSchema(op, opts.subagentId, opts.allowedSlugPrefixes) @@ -277,6 +292,7 @@ export function buildBrainTools(opts: BuildBrainToolsOpts): ToolDef[] { signal: ctx.signal, brainId: opts.brainId, allowedSlugPrefixes: opts.allowedSlugPrefixes, + sourceId: opts.sourceId, }); const params = (input && typeof input === 'object') ? input as Record : {}; return op.handler(opCtx, params); diff --git a/src/core/minions/types.ts b/src/core/minions/types.ts index a24d5446d..73877bb99 100644 --- a/src/core/minions/types.ts +++ b/src/core/minions/types.ts @@ -455,6 +455,17 @@ export interface SubagentHandlerData { * and direct CLI submitters set it. */ allowed_slug_prefixes?: string[]; + /** + * Brain source the subagent's tool calls are scoped to (#1586). + * + * When set, every tool-call `OperationContext.sourceId` uses this value + * instead of the legacy 'default', so put_page writes land in the cycle's + * resolved source. Same trust story as `allowed_slug_prefixes`: + * PROTECTED_JOB_NAMES gates subagent submission, so only cycle.ts and + * direct CLI submitters can set it. Validated via `validateSourceId` at + * tool-registry build time. + */ + source_id?: string; /** * v0.41 Approach C: opt out of the auto-generated tool-usage preamble * that `buildSystemPrompt()` splices into `system`. Default behavior diff --git a/src/core/pglite-engine.ts b/src/core/pglite-engine.ts index d30738f24..8239ec354 100644 --- a/src/core/pglite-engine.ts +++ b/src/core/pglite-engine.ts @@ -24,6 +24,7 @@ import { PGLITE_SCHEMA_SQL, getPGLiteSchema } from './pglite-schema.ts'; import { DEFAULT_EMBEDDING_MODEL, DEFAULT_EMBEDDING_DIMENSIONS } from './ai/defaults.ts'; import { DELETE_BATCH_SIZE } from './engine-constants.ts'; import { acquireLock, releaseLock, type LockHandle } from './pglite-lock.ts'; +import { getFtsLanguage } from './fts-language.ts'; import type { Page, PageInput, PageFilters, PageType, Chunk, ChunkInput, StaleChunkRow, StalePageRow, @@ -1625,20 +1626,24 @@ export class PGLiteEngine implements BrainEngine { extraFilter += ` AND p.source_id = $${params.length}`; } + // FTS config name (e.g. 'english', 'pt_br'). Validated by getFtsLanguage() + // — safe to interpolate into raw SQL. + const ftsLang = getFtsLanguage(); + const { rows } = await this.db.query( `WITH ranked AS ( SELECT p.slug, p.id as page_id, p.title, p.type, p.source_id, p.effective_date, p.effective_date_source, cc.id as chunk_id, cc.chunk_index, cc.chunk_text, cc.chunk_source, - ts_rank(cc.search_vector, websearch_to_tsquery('english', $1)) * ${sourceFactorCase} AS score, + ts_rank(cc.search_vector, websearch_to_tsquery('${ftsLang}', $1)) * ${sourceFactorCase} AS score, CASE WHEN p.updated_at < ( SELECT MAX(te.created_at) FROM timeline_entries te WHERE te.page_id = p.id ) THEN true ELSE false END AS stale FROM content_chunks cc JOIN pages p ON p.id = cc.page_id JOIN sources s ON s.id = p.source_id - WHERE cc.search_vector @@ websearch_to_tsquery('english', $1) ${detailFilter}${extraFilter} ${hardExcludeClause} ${visibilityClause} + WHERE cc.search_vector @@ websearch_to_tsquery('${ftsLang}', $1) ${detailFilter}${extraFilter} ${hardExcludeClause} ${visibilityClause} -- v0.27.1: hide image rows from default text-keyword search so -- OCR text doesn't drown text-page hits. Image-similarity queries -- run a separate vector path on embedding_image. @@ -1857,20 +1862,23 @@ export class PGLiteEngine implements BrainEngine { } // visibilityClause already declared above (v0.32.7: hoisted so CJK branch can reuse). + // FTS config name (e.g. 'english', 'pt_br'). Validated by getFtsLanguage() + // — safe to interpolate into raw SQL. + const ftsLang = getFtsLanguage(); const { rows } = await this.db.query( `SELECT p.slug, p.id as page_id, p.title, p.type, p.source_id, p.effective_date, p.effective_date_source, cc.id as chunk_id, cc.chunk_index, cc.chunk_text, cc.chunk_source, - ts_rank(cc.search_vector, websearch_to_tsquery('english', $1)) * ${sourceFactorCase} AS score, + ts_rank(cc.search_vector, websearch_to_tsquery('${ftsLang}', $1)) * ${sourceFactorCase} AS score, CASE WHEN p.updated_at < ( SELECT MAX(te.created_at) FROM timeline_entries te WHERE te.page_id = p.id ) THEN true ELSE false END AS stale FROM content_chunks cc JOIN pages p ON p.id = cc.page_id JOIN sources s ON s.id = p.source_id - WHERE cc.search_vector @@ websearch_to_tsquery('english', $1) ${detailFilter}${extraFilter} ${hardExcludeClause} ${visibilityClause} + WHERE cc.search_vector @@ websearch_to_tsquery('${ftsLang}', $1) ${detailFilter}${extraFilter} ${hardExcludeClause} ${visibilityClause} ORDER BY score DESC LIMIT $2 OFFSET $3`, params diff --git a/src/core/postgres-engine.ts b/src/core/postgres-engine.ts index 2fe57e517..eafaf8bfa 100644 --- a/src/core/postgres-engine.ts +++ b/src/core/postgres-engine.ts @@ -34,6 +34,7 @@ import { COLUMN_NAME_REGEX, EmbeddingColumnNotRegisteredError, } from './search/embedding-column.ts'; +import { getFtsLanguage } from './fts-language.ts'; import type { Page, PageInput, PageFilters, PageType, Chunk, ChunkInput, StaleChunkRow, StalePageRow, @@ -160,6 +161,88 @@ export class PostgresEngine implements BrainEngine { return db.getConnection(); } + // Source-scope binding for Postgres RLS — opt-in via env var. + // + // When `GBRAIN_RLS_SCOPE_BINDING` is set to `1` / `true`, source-scoped + // query methods (listPages, search*, getChunks, etc.) wrap their queries + // in a transaction that begins with + // SELECT set_config('app.scopes', '', true) + // (equivalent to `SET LOCAL app.scopes = ''`, but works through + // parameterised SQL — `SET LOCAL` itself doesn't accept parameters) + // so Postgres RLS policies on source-scoped tables can filter rows by + // `current_setting('app.scopes', true)`. The expected policy shape: + // + // USING (current_setting('app.scopes', true) = '*' + // OR source_id = ANY(string_to_array( + // current_setting('app.scopes', true), ','))) + // + // Recommended runtime-role default: + // ALTER ROLE SET app.scopes = '*'; + // so admin / autopilot / cycle queries that don't pass scope info still + // see all rows. OAuth-scoped requests override the default per + // transaction with their allowed-source CSV. + // + // Default behavior (env var unset): the helper is a TRUE pass-through — + // it calls `callback(this.sql)` with no transaction wrap and no + // set_config, byte-identical to not having this helper at all. The only + // exception is callers that pass `alwaysTransaction: true` (the search + // methods, whose `SET LOCAL statement_timeout` already required a + // transaction on master) — they keep exactly the `sql.begin()` wrap + // they had before this helper existed. No read gains a new per-read + // pool-hold when the flag is off (the #1794 PgBouncer-exhaustion class). + // + // Honest caveat: only the read paths that route through this helper are + // backstopped by RLS. This is defense-in-depth layer 2; the app-layer + // source filters (sourceScopeOpts) remain layer 1 and stay mandatory. + private get rlsScopeBindingEnabled(): boolean { + const v = process.env.GBRAIN_RLS_SCOPE_BINDING; + return v === '1' || v === 'true'; + } + + private async withScopedReadTransaction( + sourceIds: string[] | undefined, + sourceId: string | undefined, + callback: (tx: ReturnType) => Promise, + opts?: { alwaysTransaction?: boolean }, + ): Promise { + // Flag off + no pre-existing transaction need: call through on the + // shared pool exactly as master does. No tx round-trip, no pool slot + // held for the duration of the read. + if (!this.rlsScopeBindingEnabled && !opts?.alwaysTransaction) { + return await callback(this.sql); + } + // Precedence matches sourceScopeOpts: federated array > scalar > '*' + // (unscoped — relies on the recommended `ALTER ROLE ... SET + // app.scopes = '*'` default, or on no policy being installed). + let scopesValue = '*'; + if (sourceIds && sourceIds.length > 0) { + scopesValue = sourceIds.join(','); + } else if (sourceId) { + scopesValue = sourceId; + } + // Note on nesting: a postgres.js transaction handle exposes + // `.savepoint()` not `.begin()`, so callbacks must not try to open + // their own `tx.begin()` inside this wrap — they'd fail with + // `tx.begin is not a function`. Callbacks that need SET LOCAL emit it + // directly on the handle (it shares this transaction). + // + // `sql.begin(...)` returns `UnwrapPromiseArray` in postgres.js's typings + // — TypeScript strict-generics can't narrow that back to `T` for arbitrary + // callback return shapes (TS2322). The unwrap is a no-op when the callback + // returns a single value (not an array of promises), so the cast is safe. + return (await this.sql.begin(async (tx: any) => { + if (this.rlsScopeBindingEnabled) { + // `SET LOCAL` doesn't accept parameters in PostgreSQL — using + // `tx\`SET LOCAL ... = ${val}\`` binds val as $1 and errors with + // `syntax error at or near "$1"`. set_config() is a regular function + // and accepts a parameterised value; passing `true` as the third + // argument makes it transaction-local (same scope as SET LOCAL). + await tx`SELECT set_config('app.scopes', ${scopesValue}, true)`; + } + return await callback(tx as ReturnType); + })) as T; + } + // Lifecycle async connect(config: EngineConfig & { poolSize?: number; parentConnectionManager?: ConnectionManager }): Promise { this._savedConfig = config; @@ -920,30 +1003,36 @@ export class PostgresEngine implements BrainEngine { // Pages CRUD async getPage(slug: string, opts?: { sourceId?: string; sourceIds?: string[]; includeDeleted?: boolean }): Promise { - const sql = this.sql; const includeDeleted = opts?.includeDeleted === true; const sourceId = opts?.sourceId; const sourceIds = opts?.sourceIds; - // v0.26.5: default hides soft-deleted rows. Compose with optional source - // filter via fragment chaining (postgres.js supports sql`` composition). - // #1393: a federated grant (sourceIds[]) takes precedence over scalar - // sourceId so the exact-match read honors allowedSources, not just one source. - const sourceCondition = - sourceIds && sourceIds.length > 0 - ? sql`AND source_id = ANY(${sourceIds}::text[])` - : sourceId - ? sql`AND source_id = ${sourceId}` - : sql``; - const deletedCondition = includeDeleted ? sql`` : sql`AND deleted_at IS NULL`; - const rows = await sql` - SELECT id, source_id, slug, type, title, compiled_truth, timeline, frontmatter, content_hash, created_at, updated_at, deleted_at, - source_kind, source_uri, ingested_via, ingested_at - FROM pages - WHERE slug = ${slug} ${sourceCondition} ${deletedCondition} - LIMIT 1 - `; - if (rows.length === 0) return null; - return rowToPage(rows[0]); + // Two layers of defense: + // 1. RLS scope binding (opt-in via GBRAIN_RLS_SCOPE_BINDING): wraps the + // query in a transaction that sets `app.scopes` so the row-level + // policy on `pages` filters at the SQL layer. Pass-through when off. + // 2. App-layer source filter (#1393): a federated grant (sourceIds[]) + // takes precedence over scalar sourceId so the exact-match read + // honors allowedSources, not just one source. + return await this.withScopedReadTransaction(sourceIds, sourceId, async (tx) => { + // v0.26.5: default hides soft-deleted rows. Compose with optional source + // filter via fragment chaining (postgres.js supports sql`` composition). + const sourceCondition = + sourceIds && sourceIds.length > 0 + ? tx`AND source_id = ANY(${sourceIds}::text[])` + : sourceId + ? tx`AND source_id = ${sourceId}` + : tx``; + const deletedCondition = includeDeleted ? tx`` : tx`AND deleted_at IS NULL`; + const rows = await tx` + SELECT id, source_id, slug, type, title, compiled_truth, timeline, frontmatter, content_hash, created_at, updated_at, deleted_at, + source_kind, source_uri, ingested_via, ingested_at + FROM pages + WHERE slug = ${slug} ${sourceCondition} ${deletedCondition} + LIMIT 1 + `; + if (rows.length === 0) return null; + return rowToPage(rows[0]); + }); } /** @@ -954,19 +1043,21 @@ export class PostgresEngine implements BrainEngine { sourceId: string, opts: { hash: string; frontmatterId?: string | null }, ): Promise<{ slug: string; id: number } | null> { - const sql = this.sql; const fmId = opts.frontmatterId ?? null; - const rows = await sql` - SELECT id, slug FROM pages - WHERE source_id = ${sourceId} - AND deleted_at IS NULL - AND (content_hash = ${opts.hash} OR (frontmatter->>'id' = ${fmId} AND ${fmId}::text IS NOT NULL)) - ORDER BY id - LIMIT 1 - `; - if (rows.length === 0) return null; - const r = rows[0] as { id: number | string; slug: string }; - return { slug: r.slug, id: Number(r.id) }; + // RLS scope binding: sourceId is positional here. + return await this.withScopedReadTransaction(undefined, sourceId, async (tx) => { + const rows = await tx` + SELECT id, slug FROM pages + WHERE source_id = ${sourceId} + AND deleted_at IS NULL + AND (content_hash = ${opts.hash} OR (frontmatter->>'id' = ${fmId} AND ${fmId}::text IS NOT NULL)) + ORDER BY id + LIMIT 1 + `; + if (rows.length === 0) return null; + const r = rows[0] as { id: number | string; slug: string }; + return { slug: r.slug, id: Number(r.id) }; + }); } async putPage(slug: string, page: PageInput, opts?: { sourceId?: string }): Promise { @@ -1235,25 +1326,31 @@ export class PostgresEngine implements BrainEngine { const sortKey = filters?.sort && PAGE_SORT_SQL[filters.sort] ? filters.sort : 'updated_desc'; const orderBy = sql.unsafe(PAGE_SORT_SQL[sortKey]); - const rows = await sql` - SELECT p.* FROM pages p - ${tagJoin} - WHERE 1=1 ${typeCondition} ${tagCondition} ${updatedCondition} ${slugCondition} ${sourceCondition} ${deletedCondition} - ORDER BY ${orderBy} LIMIT ${limit} OFFSET ${offset} - `; - - return rows.map(rowToPage); + // RLS scope binding (opt-in via GBRAIN_RLS_SCOPE_BINDING): when + // enabled, this wraps the query in a transaction that sets + // `app.scopes` from filters; when disabled, it's a pass-through. + return await this.withScopedReadTransaction(filters?.sourceIds, filters?.sourceId, async (tx) => { + const rows = await tx` + SELECT p.* FROM pages p + ${tagJoin} + WHERE 1=1 ${typeCondition} ${tagCondition} ${updatedCondition} ${slugCondition} ${sourceCondition} ${deletedCondition} + ORDER BY ${orderBy} LIMIT ${limit} OFFSET ${offset} + `; + return rows.map(rowToPage); + }); } async getAllSlugs(opts?: { sourceId?: string }): Promise> { - const sql = this.sql; - // v0.31.8 (D12): two-branch. See pglite-engine.ts:getAllSlugs for context. - if (opts?.sourceId) { - const rows = await sql`SELECT slug FROM pages WHERE source_id = ${opts.sourceId}`; - return new Set(rows.map((r) => r.slug as string)); - } - const rows = await sql`SELECT slug FROM pages`; - return new Set(rows.map((r) => r.slug as string)); + // RLS scope binding (opt-in via GBRAIN_RLS_SCOPE_BINDING). + return await this.withScopedReadTransaction(undefined, opts?.sourceId, async (tx) => { + // v0.31.8 (D12): two-branch. See pglite-engine.ts:getAllSlugs for context. + if (opts?.sourceId) { + const rows = await tx`SELECT slug FROM pages WHERE source_id = ${opts.sourceId}`; + return new Set(rows.map((r: Record) => r.slug as string)); + } + const rows = await tx`SELECT slug FROM pages`; + return new Set(rows.map((r: Record) => r.slug as string)); + }); } async listAllPageRefs(): Promise> { @@ -1368,7 +1465,6 @@ export class PostgresEngine implements BrainEngine { // 2. connection_count DESC — structural-centrality tiebreaker (D10) // 3. slug ASC — deterministic for tests async listPrefixSampledPages(opts: DomainBankSampleOpts): Promise { - const sql = this.sql; if (opts.prefixes.length === 0) return []; const exclude = opts.excludeSlugs ?? []; const staleBias = opts.staleBias === true; @@ -1376,7 +1472,9 @@ export class PostgresEngine implements BrainEngine { // Source scoping (D5, codex r2 #2 — federated array wins over scalar). const sourceIds = opts.sourceIds ?? null; const sourceId = opts.sourceId ?? null; - const rows = await sql` + // RLS scope binding (opt-in via GBRAIN_RLS_SCOPE_BINDING). + return await this.withScopedReadTransaction(opts.sourceIds, opts.sourceId, async (tx) => { + const rows = await tx` WITH prefix_pages AS ( SELECT p.id AS page_id, @@ -1443,36 +1541,42 @@ export class PostgresEngine implements BrainEngine { FROM with_chunk ORDER BY prefix `; - return rows.map((r): DomainBankRow => ({ - slug: r.slug as string, - source_id: r.source_id as string, - prefix: r.prefix as string | null, - page_id: Number(r.page_id), - title: r.title as string | null, - compiled_truth: (r.compiled_truth as string | null) ?? '', - connection_count: Number(r.connection_count), - last_retrieved_at: r.last_retrieved_at as Date | null, - representative_chunk_id: r.representative_chunk_id == null ? null : Number(r.representative_chunk_id), - })); + return rows.map((r: Record): DomainBankRow => ({ + slug: r.slug as string, + source_id: r.source_id as string, + prefix: r.prefix as string | null, + page_id: Number(r.page_id), + title: r.title as string | null, + compiled_truth: (r.compiled_truth as string | null) ?? '', + connection_count: Number(r.connection_count), + last_retrieved_at: r.last_retrieved_at as Date | null, + representative_chunk_id: r.representative_chunk_id == null ? null : Number(r.representative_chunk_id), + })); + }); } // v0.37.0 — corpus-sampling fallback when prefix-stratified can't fill M. // Deterministic with opts.seed (setseed before SELECT); random otherwise. async listCorpusSample(opts: CorpusSampleOpts): Promise { - const sql = this.sql; if (opts.n <= 0) return []; const exclude = opts.excludeSlugs ?? []; const sourceIds = opts.sourceIds ?? null; const sourceId = opts.sourceId ?? null; - // setseed deterministic path: use SELECT setseed($1) + RANDOM(). PGLite/Postgres - // both honor setseed for the same session/transaction. For tests this gives - // identical ordering across runs. - if (typeof opts.seed === 'number') { - // Clamp to [-1, 1] required by setseed. - const clamped = Math.max(-1, Math.min(1, opts.seed)); - await sql`SELECT setseed(${clamped}::float8)`; - } - const rows = await sql` + // RLS scope binding (opt-in via GBRAIN_RLS_SCOPE_BINDING). + // alwaysTransaction when seeded: setseed() only affects RANDOM() on the + // SAME connection. On a pool, a bare `sql\`SELECT setseed(...)\`` and the + // subsequent SELECT can land on different connections, silently breaking + // the deterministic path — the transaction pins both to one connection. + return await this.withScopedReadTransaction(opts.sourceIds, opts.sourceId, async (tx) => { + // setseed deterministic path: use SELECT setseed($1) + RANDOM(). PGLite/Postgres + // both honor setseed for the same session/transaction. For tests this gives + // identical ordering across runs. + if (typeof opts.seed === 'number') { + // Clamp to [-1, 1] required by setseed. + const clamped = Math.max(-1, Math.min(1, opts.seed)); + await tx`SELECT setseed(${clamped}::float8)`; + } + const rows = await tx` WITH sampled AS ( SELECT p.id AS page_id, @@ -1504,17 +1608,18 @@ export class PostgresEngine implements BrainEngine { ) AS representative_chunk_id FROM sampled s `; - return rows.map((r): DomainBankRow => ({ - slug: r.slug as string, - source_id: r.source_id as string, - prefix: r.prefix as string | null, - page_id: Number(r.page_id), - title: r.title as string | null, - compiled_truth: (r.compiled_truth as string | null) ?? '', - connection_count: Number(r.connection_count), - last_retrieved_at: r.last_retrieved_at as Date | null, - representative_chunk_id: r.representative_chunk_id == null ? null : Number(r.representative_chunk_id), - })); + return rows.map((r: Record): DomainBankRow => ({ + slug: r.slug as string, + source_id: r.source_id as string, + prefix: r.prefix as string | null, + page_id: Number(r.page_id), + title: r.title as string | null, + compiled_truth: (r.compiled_truth as string | null) ?? '', + connection_count: Number(r.connection_count), + last_retrieved_at: r.last_retrieved_at as Date | null, + representative_chunk_id: r.representative_chunk_id == null ? null : Number(r.representative_chunk_id), + })); + }, { alwaysTransaction: typeof opts.seed === 'number' }); } async resolveSlugs(partial: string, opts?: { sourceId?: string; sourceIds?: string[] }): Promise { @@ -1555,7 +1660,6 @@ export class PostgresEngine implements BrainEngine { // list_pages etc. see zero breaking changes. A2 two-pass (Layer 7) // consumes searchKeywordChunks for the raw chunk-grain primitive. async searchKeyword(query: string, opts?: SearchOpts): Promise { - const sql = this.sql; const limit = clampSearchLimit(opts?.limit); const offset = opts?.offset || 0; const type = opts?.type; @@ -1653,6 +1757,9 @@ export class PostgresEngine implements BrainEngine { // column lookup. NOT bypassed by detail=high — soft-delete is a contract, // not a temporal preference. const visibilityClause = buildVisibilityClause('p', 's'); + // FTS config name (e.g. 'english', 'pt_br'). Validated by getFtsLanguage() + // — safe to interpolate into raw SQL. + const ftsLang = getFtsLanguage(); const rawQuery = ` WITH ranked_chunks AS ( @@ -1660,11 +1767,11 @@ export class PostgresEngine implements BrainEngine { p.slug, p.id as page_id, p.title, p.type, p.source_id, p.effective_date, p.effective_date_source, cc.id as chunk_id, cc.chunk_index, cc.chunk_text, cc.chunk_source, - ts_rank(cc.search_vector, websearch_to_tsquery('english', $1)) * ${sourceFactorCase} AS score + ts_rank(cc.search_vector, websearch_to_tsquery('${ftsLang}', $1)) * ${sourceFactorCase} AS score FROM content_chunks cc JOIN pages p ON p.id = cc.page_id JOIN sources s ON s.id = p.source_id - WHERE cc.search_vector @@ websearch_to_tsquery('english', $1) + WHERE cc.search_vector @@ websearch_to_tsquery('${ftsLang}', $1) ${typeClause} ${typesClause} ${excludeSlugsClause} @@ -1694,12 +1801,16 @@ export class PostgresEngine implements BrainEngine { OFFSET ${offsetParam} `; - // Search-only timeout. SET LOCAL inside sql.begin() scopes the GUC - // to the transaction so it can never leak onto a pooled connection. - const rows = await sql.begin(async sql => { - await sql`SET LOCAL statement_timeout = '8s'`; - return await sql.unsafe(rawQuery, params as Parameters[1]); - }); + // RLS scope binding (opt-in via GBRAIN_RLS_SCOPE_BINDING) + search-only + // timeout. alwaysTransaction: this method needed sql.begin() on master + // already (SET LOCAL statement_timeout must be transaction-scoped so + // the GUC can never leak onto a pooled connection). Flag off → the + // wrap is identical to master's; flag on → set_config('app.scopes') + // shares the same transaction as the timeout. + const rows = await this.withScopedReadTransaction(opts?.sourceIds, opts?.sourceId, async (tx) => { + await tx`SET LOCAL statement_timeout = '8s'`; + return await tx.unsafe(rawQuery, params as Parameters[1]); + }, { alwaysTransaction: true }); return rows.map(rowToSearchResult); } @@ -1713,7 +1824,6 @@ export class PostgresEngine implements BrainEngine { * contract). This is intentionally a narrow internal knob. */ async searchKeywordChunks(query: string, opts?: SearchOpts): Promise { - const sql = this.sql; const limit = clampSearchLimit(opts?.limit); const offset = opts?.offset || 0; const type = opts?.type; @@ -1792,18 +1902,21 @@ export class PostgresEngine implements BrainEngine { // v0.26.5: visibility filter for searchKeywordChunks (anchor primitive). const visibilityClause = buildVisibilityClause('p', 's'); + // FTS config name (e.g. 'english', 'pt_br'). Validated by getFtsLanguage() + // — safe to interpolate into raw SQL. + const ftsLang = getFtsLanguage(); const rawQuery = ` SELECT p.slug, p.id as page_id, p.title, p.type, p.source_id, p.effective_date, p.effective_date_source, cc.id as chunk_id, cc.chunk_index, cc.chunk_text, cc.chunk_source, - ts_rank(cc.search_vector, websearch_to_tsquery('english', $1)) * ${sourceFactorCase} AS score, + ts_rank(cc.search_vector, websearch_to_tsquery('${ftsLang}', $1)) * ${sourceFactorCase} AS score, false AS stale FROM content_chunks cc JOIN pages p ON p.id = cc.page_id JOIN sources s ON s.id = p.source_id - WHERE cc.search_vector @@ websearch_to_tsquery('english', $1) + WHERE cc.search_vector @@ websearch_to_tsquery('${ftsLang}', $1) ${typeClause} ${typesClause} ${excludeSlugsClause} @@ -1820,15 +1933,17 @@ export class PostgresEngine implements BrainEngine { OFFSET ${offsetParam} `; - const rows = await sql.begin(async sql => { - await sql`SET LOCAL statement_timeout = '8s'`; - return await sql.unsafe(rawQuery, params as Parameters[1]); - }); + // RLS scope binding + search-only timeout. alwaysTransaction: master + // already wrapped this in sql.begin() for the SET LOCAL; flag off is + // identical to that wrap, flag on adds set_config in the same tx. + const rows = await this.withScopedReadTransaction(opts?.sourceIds, opts?.sourceId, async (tx) => { + await tx`SET LOCAL statement_timeout = '8s'`; + return await tx.unsafe(rawQuery, params as Parameters[1]); + }, { alwaysTransaction: true }); return rows.map(rowToSearchResult); } async searchVector(embedding: Float32Array, opts?: SearchOpts): Promise { - const sql = this.sql; const limit = clampSearchLimit(opts?.limit); const offset = opts?.offset || 0; const type = opts?.type; @@ -1992,10 +2107,13 @@ export class PostgresEngine implements BrainEngine { OFFSET ${offsetParam} `; - const rows = await sql.begin(async sql => { - await sql`SET LOCAL statement_timeout = '8s'`; - return await sql.unsafe(rawQuery, params as Parameters[1]); - }); + // RLS scope binding + search-only timeout. alwaysTransaction: master + // already wrapped this in sql.begin() for the SET LOCAL; flag off is + // identical to that wrap, flag on adds set_config in the same tx. + const rows = await this.withScopedReadTransaction(opts?.sourceIds, opts?.sourceId, async (tx) => { + await tx`SET LOCAL statement_timeout = '8s'`; + return await tx.unsafe(rawQuery, params as Parameters[1]); + }, { alwaysTransaction: true }); return rows.map(rowToSearchResult); } @@ -2238,15 +2356,17 @@ export class PostgresEngine implements BrainEngine { } async getChunks(slug: string, opts?: { sourceId?: string }): Promise { - const sql = this.sql; const sourceId = opts?.sourceId ?? 'default'; - const rows = await sql` - SELECT cc.* FROM content_chunks cc - JOIN pages p ON p.id = cc.page_id - WHERE p.slug = ${slug} AND p.source_id = ${sourceId} - ORDER BY cc.chunk_index - `; - return rows.map((r) => rowToChunk(r as Record)); + // RLS scope binding (opt-in via GBRAIN_RLS_SCOPE_BINDING). + return await this.withScopedReadTransaction(undefined, sourceId, async (tx) => { + const rows = await tx` + SELECT cc.* FROM content_chunks cc + JOIN pages p ON p.id = cc.page_id + WHERE p.slug = ${slug} AND p.source_id = ${sourceId} + ORDER BY cc.chunk_index + `; + return rows.map((r: Record) => rowToChunk(r)); + }); } /** @@ -2277,14 +2397,17 @@ export class PostgresEngine implements BrainEngine { // D7: source_id scoping. v0.41.31: optional signature widens staleness // to embedding_signature drift (NULL grandfathered). const { where, params } = this.buildStaleChunkWhere(opts); - const rows = await this.sql.unsafe( - `SELECT count(*)::int AS count - FROM content_chunks cc - JOIN pages p ON p.id = cc.page_id - WHERE ${where}`, - params as Parameters[1], - ); - return Number((rows[0] as { count?: number } | undefined)?.count ?? 0); + // RLS scope binding (opt-in via GBRAIN_RLS_SCOPE_BINDING). + return await this.withScopedReadTransaction(undefined, opts?.sourceId, async (tx) => { + const rows = await tx.unsafe( + `SELECT count(*)::int AS count + FROM content_chunks cc + JOIN pages p ON p.id = cc.page_id + WHERE ${where}`, + params as Parameters[1], + ); + return Number((rows[0] as { count?: number } | undefined)?.count ?? 0); + }); } async sumStaleChunkChars(opts?: { sourceId?: string; signature?: string }): Promise { @@ -2341,41 +2464,67 @@ export class PostgresEngine implements BrainEngine { orderBy?: 'page_id' | 'updated_desc'; afterUpdatedAt?: string | null; }): Promise { - const sql = this.sql; const limit = opts?.batchSize ?? 2000; const afterPid = opts?.afterPageId ?? 0; const afterIdx = opts?.afterChunkIndex ?? -1; const orderBy = opts?.orderBy ?? 'page_id'; - // v0.41.18.0 (A13, codex #9): --priority recent path. Composite cursor - // (updated_at DESC NULLS LAST, page_id ASC, chunk_index ASC). Backed by - // idx_pages_updated_at_desc + content_chunks_stale_idx partial. - // "Next row" semantic with DESC NULLS LAST + ASC tiebreakers is: - // (updated_at < prev) OR - // (updated_at = prev AND page_id > prev_page_id) OR - // (updated_at = prev AND page_id = prev_page_id AND chunk_index > prev_chunk_index) - // First call: afterUpdatedAt undefined → returns the highest updated_at rows. - if (orderBy === 'updated_desc') { - const afterUpdated = opts?.afterUpdatedAt ?? null; - const isFirstPage = afterUpdated === null && afterPid === 0; - if (opts?.sourceId === undefined) { - const rows = isFirstPage ? await sql` + // RLS scope binding (opt-in via GBRAIN_RLS_SCOPE_BINDING). + return await this.withScopedReadTransaction(undefined, opts?.sourceId, async (tx) => { + // v0.41.18.0 (A13, codex #9): --priority recent path. Composite cursor + // (updated_at DESC NULLS LAST, page_id ASC, chunk_index ASC). Backed by + // idx_pages_updated_at_desc + content_chunks_stale_idx partial. + if (orderBy === 'updated_desc') { + const afterUpdated = opts?.afterUpdatedAt ?? null; + const isFirstPage = afterUpdated === null && afterPid === 0; + if (opts?.sourceId === undefined) { + const rows = isFirstPage ? await tx` + SELECT p.slug, cc.chunk_index, cc.chunk_text, cc.chunk_source, + cc.model, cc.token_count, p.source_id, cc.page_id, + p.updated_at + FROM content_chunks cc + JOIN pages p ON p.id = cc.page_id + WHERE cc.embedding IS NULL + AND NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'embed_skip') + ORDER BY p.updated_at DESC NULLS LAST, p.id ASC, cc.chunk_index ASC + LIMIT ${limit} + ` : await tx` + SELECT p.slug, cc.chunk_index, cc.chunk_text, cc.chunk_source, + cc.model, cc.token_count, p.source_id, cc.page_id, + p.updated_at + FROM content_chunks cc + JOIN pages p ON p.id = cc.page_id + WHERE cc.embedding IS NULL + AND NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'embed_skip') + AND ( + p.updated_at < ${afterUpdated}::timestamptz + OR (p.updated_at = ${afterUpdated}::timestamptz AND p.id > ${afterPid}) + OR (p.updated_at = ${afterUpdated}::timestamptz AND p.id = ${afterPid} AND cc.chunk_index > ${afterIdx}) + ) + ORDER BY p.updated_at DESC NULLS LAST, p.id ASC, cc.chunk_index ASC + LIMIT ${limit} + `; + return rows as unknown as StaleChunkRow[]; + } + const rows = isFirstPage ? await tx` SELECT p.slug, cc.chunk_index, cc.chunk_text, cc.chunk_source, cc.model, cc.token_count, p.source_id, cc.page_id, p.updated_at FROM content_chunks cc JOIN pages p ON p.id = cc.page_id WHERE cc.embedding IS NULL + AND p.source_id = ${opts.sourceId} AND NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'embed_skip') ORDER BY p.updated_at DESC NULLS LAST, p.id ASC, cc.chunk_index ASC LIMIT ${limit} - ` : await sql` + ` : await tx` SELECT p.slug, cc.chunk_index, cc.chunk_text, cc.chunk_source, cc.model, cc.token_count, p.source_id, cc.page_id, p.updated_at FROM content_chunks cc JOIN pages p ON p.id = cc.page_id WHERE cc.embedding IS NULL + AND p.source_id = ${opts.sourceId} AND NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'embed_skip') AND ( p.updated_at < ${afterUpdated}::timestamptz @@ -2387,77 +2536,35 @@ export class PostgresEngine implements BrainEngine { `; return rows as unknown as StaleChunkRow[]; } - const rows = isFirstPage ? await sql` - SELECT p.slug, cc.chunk_index, cc.chunk_text, cc.chunk_source, - cc.model, cc.token_count, p.source_id, cc.page_id, - p.updated_at - FROM content_chunks cc - JOIN pages p ON p.id = cc.page_id - WHERE cc.embedding IS NULL - AND p.source_id = ${opts.sourceId} - AND NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'embed_skip') - ORDER BY p.updated_at DESC NULLS LAST, p.id ASC, cc.chunk_index ASC - LIMIT ${limit} - ` : await sql` - SELECT p.slug, cc.chunk_index, cc.chunk_text, cc.chunk_source, - cc.model, cc.token_count, p.source_id, cc.page_id, - p.updated_at - FROM content_chunks cc - JOIN pages p ON p.id = cc.page_id - WHERE cc.embedding IS NULL - AND p.source_id = ${opts.sourceId} - AND NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'embed_skip') - AND ( - p.updated_at < ${afterUpdated}::timestamptz - OR (p.updated_at = ${afterUpdated}::timestamptz AND p.id > ${afterPid}) - OR (p.updated_at = ${afterUpdated}::timestamptz AND p.id = ${afterPid} AND cc.chunk_index > ${afterIdx}) - ) - ORDER BY p.updated_at DESC NULLS LAST, p.id ASC, cc.chunk_index ASC - LIMIT ${limit} - `; - return rows as unknown as StaleChunkRow[]; - } - // orderBy === 'page_id' — legacy stable cursor (unchanged below). - // Cursor-paginated: keyset pagination on (page_id, chunk_index). - // The partial index idx_chunks_embedding_null makes the WHERE fast; - // LIMIT keeps each round-trip well within statement_timeout. - // - // D7: optional source_id filter. NULL/undefined = scan all sources - // (pre-existing behavior); a value scopes to that source so - // `gbrain embed --stale --source X` actually does what it says. - // - // v0.41 (D4+D8): NOT (frontmatter ? 'embed_skip') filter applied via - // the always-JOINed pages row. Soft-blocked pages won't surface in - // the stale list; their chunks were deleted at ingest time anyway - // (D9 transition invariant), but the filter is defense-in-depth for - // pre-fix inventory that might still have orphan chunks. - if (opts?.sourceId === undefined) { - const rows = await sql` + // orderBy === 'page_id' — legacy stable cursor. + if (opts?.sourceId === undefined) { + const rows = await tx` + SELECT p.slug, cc.chunk_index, cc.chunk_text, cc.chunk_source, + cc.model, cc.token_count, p.source_id, cc.page_id + FROM content_chunks cc + JOIN pages p ON p.id = cc.page_id + WHERE cc.embedding IS NULL + AND NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'embed_skip') + AND (cc.page_id, cc.chunk_index) > (${afterPid}, ${afterIdx}) + ORDER BY cc.page_id, cc.chunk_index + LIMIT ${limit} + `; + return rows as unknown as StaleChunkRow[]; + } + const rows = await tx` SELECT p.slug, cc.chunk_index, cc.chunk_text, cc.chunk_source, cc.model, cc.token_count, p.source_id, cc.page_id FROM content_chunks cc JOIN pages p ON p.id = cc.page_id WHERE cc.embedding IS NULL + AND p.source_id = ${opts.sourceId} AND NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'embed_skip') AND (cc.page_id, cc.chunk_index) > (${afterPid}, ${afterIdx}) ORDER BY cc.page_id, cc.chunk_index LIMIT ${limit} `; return rows as unknown as StaleChunkRow[]; - } - const rows = await sql` - SELECT p.slug, cc.chunk_index, cc.chunk_text, cc.chunk_source, - cc.model, cc.token_count, p.source_id, cc.page_id - FROM content_chunks cc - JOIN pages p ON p.id = cc.page_id - WHERE cc.embedding IS NULL - AND p.source_id = ${opts.sourceId} - AND NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'embed_skip') - AND (cc.page_id, cc.chunk_index) > (${afterPid}, ${afterIdx}) - ORDER BY cc.page_id, cc.chunk_index - LIMIT ${limit} - `; - return rows as unknown as StaleChunkRow[]; + }); } async deleteChunks(slug: string, opts?: { sourceId?: string }): Promise { @@ -2490,11 +2597,14 @@ export class PostgresEngine implements BrainEngine { async countStalePagesForExtraction(opts?: { sourceId?: string; versionTs?: string }): Promise { const { where, params } = this.buildStalePagesWhere(opts); - const rows = await this.sql.unsafe( - `SELECT count(*)::int AS count FROM pages WHERE ${where}`, - params as Parameters[1], - ); - return Number((rows[0] as { count?: number } | undefined)?.count ?? 0); + // RLS scope binding (opt-in via GBRAIN_RLS_SCOPE_BINDING). + return await this.withScopedReadTransaction(undefined, opts?.sourceId, async (tx) => { + const rows = await tx.unsafe( + `SELECT count(*)::int AS count FROM pages WHERE ${where}`, + params as Parameters[1], + ); + return Number((rows[0] as { count?: number } | undefined)?.count ?? 0); + }); } async listStalePagesForExtraction(opts: { @@ -2511,19 +2621,22 @@ export class PostgresEngine implements BrainEngine { } params.push(opts.batchSize); const limitIdx = params.length; - const rows = await this.sql.unsafe( - // #1768: project a deterministic full-µs UTC string alongside updated_at. - // to_char (not ::text — DateStyle-fragile) so extractStaleFromDB can stamp - // links_extracted_at = the exact updated_at and the staleness predicate clears. - `SELECT id, slug, source_id, type, title, compiled_truth, timeline, frontmatter, updated_at, - to_char(updated_at AT TIME ZONE 'UTC', 'YYYY-MM-DD"T"HH24:MI:SS.US"Z"') AS updated_at_iso - FROM pages - WHERE ${where}${afterClause} - ORDER BY id - LIMIT $${limitIdx}`, - params as Parameters[1], - ); - return (rows as Record[]).map(rowToStalePage); + // RLS scope binding (opt-in via GBRAIN_RLS_SCOPE_BINDING). + return await this.withScopedReadTransaction(undefined, opts.sourceId, async (tx) => { + const rows = await tx.unsafe( + // #1768: project a deterministic full-µs UTC string alongside updated_at. + // to_char (not ::text — DateStyle-fragile) so extractStaleFromDB can stamp + // links_extracted_at = the exact updated_at and the staleness predicate clears. + `SELECT id, slug, source_id, type, title, compiled_truth, timeline, frontmatter, updated_at, + to_char(updated_at AT TIME ZONE 'UTC', 'YYYY-MM-DD"T"HH24:MI:SS.US"Z"') AS updated_at_iso + FROM pages + WHERE ${where}${afterClause} + ORDER BY id + LIMIT $${limitIdx}`, + params as Parameters[1], + ); + return (rows as Record[]).map(rowToStalePage); + }); } async markPagesExtractedBatch(refs: Array<{ slug: string; source_id: string; extractedAt?: string }>, defaultExtractedAt: string): Promise { @@ -2671,32 +2784,47 @@ export class PostgresEngine implements BrainEngine { } async getLinks(slug: string, opts?: { sourceId?: string; sourceIds?: string[] }): Promise { - const sql = this.sql; - // #2200: federated grant scopes ALL THREE page endpoints — from, to, AND the - // origin (the page that authored the edge, surfaced as origin_slug). Scoping - // only from+to would still leak an out-of-grant origin's slug; the origin - // LEFT JOIN carries the same ANY($) filter so origin_slug nulls out of grant. - // Remote MCP clients always land here. - if (opts?.sourceIds && opts.sourceIds.length > 0) { - const ids = opts.sourceIds; - const rows = await sql` - SELECT f.slug as from_slug, t.slug as to_slug, - l.link_type, l.context, l.link_source, - o.slug as origin_slug, l.origin_field - FROM links l - JOIN pages f ON f.id = l.from_page_id - JOIN pages t ON t.id = l.to_page_id - LEFT JOIN pages o ON o.id = l.origin_page_id AND o.source_id = ANY(${ids}::text[]) - WHERE f.slug = ${slug} AND f.source_id = ANY(${ids}::text[]) AND t.source_id = ANY(${ids}::text[]) - `; - return rows as unknown as Link[]; - } - // v0.31.8 (D16) + #2200: the federated arm above is the first branch; the - // two below preserve pre-v0.31.8 semantics. Without opts.sourceId, no source - // filter (cross-source view for internal callers). With opts.sourceId, scope - // the from-page lookup. See pglite-engine.ts:getLinks for context. - if (opts?.sourceId) { - const rows = await sql` + // Two layers of defense (see getPage for the full pattern): + // 1. RLS scope binding (opt-in via GBRAIN_RLS_SCOPE_BINDING) + // 2. App-layer source filter (#2200 federated) + return await this.withScopedReadTransaction(opts?.sourceIds, opts?.sourceId, async (tx) => { + // #2200: federated grant scopes ALL THREE page endpoints — from, to, AND + // the origin (the page that authored the edge, surfaced as origin_slug). + // Scoping only from+to would still leak an out-of-grant origin's slug; the + // origin LEFT JOIN carries the same ANY($) filter so origin_slug nulls + // out of grant. Remote MCP clients always land here. + if (opts?.sourceIds && opts.sourceIds.length > 0) { + const ids = opts.sourceIds; + const rows = await tx` + SELECT f.slug as from_slug, t.slug as to_slug, + l.link_type, l.context, l.link_source, + o.slug as origin_slug, l.origin_field + FROM links l + JOIN pages f ON f.id = l.from_page_id + JOIN pages t ON t.id = l.to_page_id + LEFT JOIN pages o ON o.id = l.origin_page_id AND o.source_id = ANY(${ids}::text[]) + WHERE f.slug = ${slug} AND f.source_id = ANY(${ids}::text[]) AND t.source_id = ANY(${ids}::text[]) + `; + return rows as unknown as Link[]; + } + // v0.31.8 (D16) + #2200: the federated arm above is the first branch; the + // two below preserve pre-v0.31.8 semantics. Without opts.sourceId, no + // source filter (cross-source view for internal callers). With + // opts.sourceId, scope the from-page lookup. + if (opts?.sourceId) { + const rows = await tx` + SELECT f.slug as from_slug, t.slug as to_slug, + l.link_type, l.context, l.link_source, + o.slug as origin_slug, l.origin_field + FROM links l + JOIN pages f ON f.id = l.from_page_id + JOIN pages t ON t.id = l.to_page_id + LEFT JOIN pages o ON o.id = l.origin_page_id + WHERE f.slug = ${slug} AND f.source_id = ${opts.sourceId} + `; + return rows as unknown as Link[]; + } + const rows = await tx` SELECT f.slug as from_slug, t.slug as to_slug, l.link_type, l.context, l.link_source, o.slug as origin_slug, l.origin_field @@ -2704,45 +2832,49 @@ export class PostgresEngine implements BrainEngine { JOIN pages f ON f.id = l.from_page_id JOIN pages t ON t.id = l.to_page_id LEFT JOIN pages o ON o.id = l.origin_page_id - WHERE f.slug = ${slug} AND f.source_id = ${opts.sourceId} + WHERE f.slug = ${slug} `; return rows as unknown as Link[]; - } - const rows = await sql` - SELECT f.slug as from_slug, t.slug as to_slug, - l.link_type, l.context, l.link_source, - o.slug as origin_slug, l.origin_field - FROM links l - JOIN pages f ON f.id = l.from_page_id - JOIN pages t ON t.id = l.to_page_id - LEFT JOIN pages o ON o.id = l.origin_page_id - WHERE f.slug = ${slug} - `; - return rows as unknown as Link[]; + }); } async getBacklinks(slug: string, opts?: { sourceId?: string; sourceIds?: string[] }): Promise { - const sql = this.sql; - // #2200: federated grant scopes all three endpoints (mirrors getLinks) — the - // referrer (from), the queried page (to), AND the origin — so neither a - // foreign referrer nor a foreign origin slug is disclosed to the caller. - if (opts?.sourceIds && opts.sourceIds.length > 0) { - const ids = opts.sourceIds; - const rows = await sql` - SELECT f.slug as from_slug, t.slug as to_slug, - l.link_type, l.context, l.link_source, - o.slug as origin_slug, l.origin_field - FROM links l - JOIN pages f ON f.id = l.from_page_id - JOIN pages t ON t.id = l.to_page_id - LEFT JOIN pages o ON o.id = l.origin_page_id AND o.source_id = ANY(${ids}::text[]) - WHERE t.slug = ${slug} AND t.source_id = ANY(${ids}::text[]) AND f.source_id = ANY(${ids}::text[]) - `; - return rows as unknown as Link[]; - } - // v0.31.8 (D16) + #2200: federated arm above is first; two below mirror getLinks. - if (opts?.sourceId) { - const rows = await sql` + // Two layers of defense (see getPage for the full pattern): + // 1. RLS scope binding (opt-in via GBRAIN_RLS_SCOPE_BINDING) + // 2. App-layer source filter (#2200 federated) + return await this.withScopedReadTransaction(opts?.sourceIds, opts?.sourceId, async (tx) => { + // #2200: federated grant scopes all three endpoints (mirrors getLinks) — + // the referrer (from), the queried page (to), AND the origin — so neither + // a foreign referrer nor a foreign origin slug is disclosed to the caller. + if (opts?.sourceIds && opts.sourceIds.length > 0) { + const ids = opts.sourceIds; + const rows = await tx` + SELECT f.slug as from_slug, t.slug as to_slug, + l.link_type, l.context, l.link_source, + o.slug as origin_slug, l.origin_field + FROM links l + JOIN pages f ON f.id = l.from_page_id + JOIN pages t ON t.id = l.to_page_id + LEFT JOIN pages o ON o.id = l.origin_page_id AND o.source_id = ANY(${ids}::text[]) + WHERE t.slug = ${slug} AND t.source_id = ANY(${ids}::text[]) AND f.source_id = ANY(${ids}::text[]) + `; + return rows as unknown as Link[]; + } + // v0.31.8 (D16) + #2200: federated arm above is first; two below mirror getLinks. + if (opts?.sourceId) { + const rows = await tx` + SELECT f.slug as from_slug, t.slug as to_slug, + l.link_type, l.context, l.link_source, + o.slug as origin_slug, l.origin_field + FROM links l + JOIN pages f ON f.id = l.from_page_id + JOIN pages t ON t.id = l.to_page_id + LEFT JOIN pages o ON o.id = l.origin_page_id + WHERE t.slug = ${slug} AND t.source_id = ${opts.sourceId} + `; + return rows as unknown as Link[]; + } + const rows = await tx` SELECT f.slug as from_slug, t.slug as to_slug, l.link_type, l.context, l.link_source, o.slug as origin_slug, l.origin_field @@ -2750,45 +2882,36 @@ export class PostgresEngine implements BrainEngine { JOIN pages f ON f.id = l.from_page_id JOIN pages t ON t.id = l.to_page_id LEFT JOIN pages o ON o.id = l.origin_page_id - WHERE t.slug = ${slug} AND t.source_id = ${opts.sourceId} + WHERE t.slug = ${slug} `; return rows as unknown as Link[]; - } - const rows = await sql` - SELECT f.slug as from_slug, t.slug as to_slug, - l.link_type, l.context, l.link_source, - o.slug as origin_slug, l.origin_field - FROM links l - JOIN pages f ON f.id = l.from_page_id - JOIN pages t ON t.id = l.to_page_id - LEFT JOIN pages o ON o.id = l.origin_page_id - WHERE t.slug = ${slug} - `; - return rows as unknown as Link[]; + }); } async listLinkSources( opts?: { sourceId?: string; sourceIds?: string[] }, ): Promise<{ link_source: string | null; count: number }[]> { - const sql = this.sql; - // v114 (#1941): distinct provenances + counts for `gbrain link-sources`. - // Scope by the FROM page's source (consistent with getLinks). Federated - // {sourceIds} takes precedence over scalar {sourceId}; neither = unscoped. - const sourceCondition = - opts?.sourceIds && opts.sourceIds.length > 0 - ? sql`WHERE f.source_id = ANY(${opts.sourceIds}::text[])` - : opts?.sourceId - ? sql`WHERE f.source_id = ${opts.sourceId}` - : sql``; - const rows = await sql` - SELECT l.link_source, COUNT(*)::int AS count - FROM links l - JOIN pages f ON f.id = l.from_page_id - ${sourceCondition} - GROUP BY l.link_source - ORDER BY count DESC, l.link_source ASC NULLS LAST - `; - return rows as unknown as { link_source: string | null; count: number }[]; + // RLS scope binding (opt-in via GBRAIN_RLS_SCOPE_BINDING). + return await this.withScopedReadTransaction(opts?.sourceIds, opts?.sourceId, async (tx) => { + // v114 (#1941): distinct provenances + counts for `gbrain link-sources`. + // Scope by the FROM page's source (consistent with getLinks). Federated + // {sourceIds} takes precedence over scalar {sourceId}; neither = unscoped. + const sourceCondition = + opts?.sourceIds && opts.sourceIds.length > 0 + ? tx`WHERE f.source_id = ANY(${opts.sourceIds}::text[])` + : opts?.sourceId + ? tx`WHERE f.source_id = ${opts.sourceId}` + : tx``; + const rows = await tx` + SELECT l.link_source, COUNT(*)::int AS count + FROM links l + JOIN pages f ON f.id = l.from_page_id + ${sourceCondition} + GROUP BY l.link_source + ORDER BY count DESC, l.link_source ASC NULLS LAST + `; + return rows as unknown as { link_source: string | null; count: number }[]; + }); } async findByTitleFuzzy( diff --git a/src/core/sync.ts b/src/core/sync.ts index 455607a6e..af6ff1ba3 100644 --- a/src/core/sync.ts +++ b/src/core/sync.ts @@ -255,7 +255,12 @@ const PRUNE_DIR_NAMES = new Set([ // with the first-sync walker in commands/import.ts. 'venv', '.raw', - 'ops', + // NOTE (#2404): `'ops'` used to be in this list (a v0.2.0-era carve-out for + // one brain layout). Matching the bare segment pruned EVERY user `ops/` + // directory at any depth — sync silently deleted `ops/*` pages and never + // imported `ops/*` files, while the bundled daily-task-manager skill + // prescribes `ops/tasks` as its canonical page. `ops/` is ordinary content; + // do NOT re-add it. Only generated/vendored trees belong here. ]); /** @@ -352,8 +357,8 @@ function classifySync(path: string, opts: SyncableOptions = {}): SyncableReason if (!isAllowedByStrategy(path, strategy)) return 'strategy'; // Skip every path segment that pruneDir would block walkers from descending - // into. Catches hidden dirs (`.git`, `.obsidian`), `.raw/` sidecars, - // `node_modules/` (latent bug fix), and `ops/` at any depth. + // into. Catches hidden dirs (`.git`, `.obsidian`), `.raw/` sidecars, and + // vendor/generated trees (`node_modules/`, `vendor/`, …) at any depth. const segments = path.split('/'); if (segments.some(p => !pruneDir(p))) return 'pruned-dir'; diff --git a/src/core/write-through.ts b/src/core/write-through.ts index 5ca2b565c..02f792a3f 100644 --- a/src/core/write-through.ts +++ b/src/core/write-through.ts @@ -27,6 +27,7 @@ import { randomBytes } from 'crypto'; import type { BrainEngine } from './engine.ts'; import { serializePageToMarkdown, resolvePageFilePath } from './markdown.ts'; import { isWriteTargetContained } from './path-confine.ts'; +import { isDurabilityHardened, commitWriteThroughFile } from './brain-repo-durability.ts'; /** Minimal logger surface — structurally compatible with operations.ts `Logger`. */ export interface WriteThroughLogger { @@ -36,6 +37,13 @@ export interface WriteThroughLogger { export interface WriteThroughResult { written: boolean; path?: string; + /** + * True when the write was also committed to git (#2426). Only attempted on + * repos hardened via `gbrain sources harden` (durability hook installed); + * the hook then background-pushes the commit. Best-effort — a false/absent + * value never blocks the write. + */ + committed?: boolean; /** * Non-error reasons the file was not written: * - no_repo_configured: the resolved target (source `local_path` or, for a @@ -157,7 +165,20 @@ export async function writePageThrough( throw writeErr; } - return { written: true, path: filePath }; + // #2426: on a durability-hardened repo (user ran `gbrain sources harden`), + // commit the artifact so it reaches git — pre-fix, write-through content + // stayed uncommitted forever: never pushed, `last_sync_at` frozen, and + // silently deleted by a later `sync --full` delete-reconcile. The local + // post-commit hook background-pushes the commit. Best-effort: a commit + // failure never fails the write (the DB row + file are the durable sinks). + let committed = false; + try { + if (isDurabilityHardened(writeRoot)) { + committed = commitWriteThroughFile(writeRoot, filePath, slug); + } + } catch { /* best-effort */ } + + return { written: true, path: filePath, ...(committed ? { committed } : {}) }; } catch (e) { const msg = e instanceof Error ? e.message : String(e); opts.logger?.warn(`[write-through] failed for ${slug}: ${msg}`); diff --git a/test/ai/recipe-moonshot.test.ts b/test/ai/recipe-moonshot.test.ts new file mode 100644 index 000000000..0af8844d3 --- /dev/null +++ b/test/ai/recipe-moonshot.test.ts @@ -0,0 +1,53 @@ +/** + * Moonshot/Kimi local recipe smoke. + * + * This pins the governed production exception GBrain-Local-003: GBrain can + * route configured Kimi chat/expansion IDs through Moonshot's OpenAI-compatible + * endpoint without treating `moonshot` as an unknown provider. + */ + +import { describe, expect, test } from 'bun:test'; +import { getRecipe } from '../../src/core/ai/recipes/index.ts'; +import { defaultResolveAuth } from '../../src/core/ai/gateway.ts'; +import { assertTouchpoint } from '../../src/core/ai/model-resolver.ts'; +import { AIConfigError } from '../../src/core/ai/errors.ts'; + +describe('recipe: moonshot', () => { + test('registered with expected OpenAI-compatible shape', () => { + const r = getRecipe('moonshot'); + expect(r).toBeDefined(); + expect(r!.id).toBe('moonshot'); + expect(r!.tier).toBe('openai-compat'); + expect(r!.implementation).toBe('openai-compatible'); + expect(r!.base_url_default).toBe('https://api.moonshot.ai/v1'); + expect(r!.auth_env?.required).toEqual(['MOONSHOT_API_KEY']); + }); + + test('chat and expansion touchpoints include Kimi K2.7 Code', () => { + const r = getRecipe('moonshot')!; + expect(r.touchpoints.chat).toBeDefined(); + expect(r.touchpoints.expansion).toBeDefined(); + expect(r.touchpoints.chat!.models).toContain('kimi-k2.7-code'); + expect(r.touchpoints.expansion!.models).toContain('kimi-k2.7-code'); + expect(r.touchpoints.chat!.supports_tools).toBe(true); + expect(r.touchpoints.chat!.supports_subagent_loop).toBe(false); + }); + + test('configured Kimi model is accepted for chat and expansion', () => { + const r = getRecipe('moonshot')!; + expect(() => assertTouchpoint(r, 'chat', 'kimi-k2.7-code')).not.toThrow(); + expect(() => assertTouchpoint(r, 'expansion', 'kimi-k2.7-code')).not.toThrow(); + }); + + test('default auth: MOONSHOT_API_KEY set -> Bearer token', () => { + const r = getRecipe('moonshot')!; + const auth = defaultResolveAuth(r, { MOONSHOT_API_KEY: 'fake-moonshot-key' }, 'chat'); + expect(auth.headerName).toBe('Authorization'); + expect(auth.token).toBe('Bearer fake-moonshot-key'); + }); + + test('default auth: missing MOONSHOT_API_KEY -> AIConfigError', () => { + const r = getRecipe('moonshot')!; + expect(() => defaultResolveAuth(r, {}, 'chat')).toThrow(AIConfigError); + }); +}); diff --git a/test/brain-allowlist.serial.test.ts b/test/brain-allowlist.serial.test.ts index 60e74bba9..7cfd5c2a7 100644 --- a/test/brain-allowlist.serial.test.ts +++ b/test/brain-allowlist.serial.test.ts @@ -146,6 +146,41 @@ describe('buildBrainTools', () => { ), ).rejects.toBeInstanceOf(OperationError); }); + + // #1586: sourceId threads through buildBrainTools → buildOpContext → + // put_page → importFromContent, so subagent writes land in the cycle's + // resolved source instead of the hardcoded 'default'. + test('execute() on put_page writes to the configured sourceId (#1586)', async () => { + await engine.executeRaw( + `INSERT INTO sources (id, name, local_path, config, archived, created_at) + VALUES ('mybrain', 'My Brain', '/tmp/mybrain', '{}'::jsonb, false, now()) + ON CONFLICT (id) DO NOTHING`, + ); + const tools = buildBrainTools({ + subagentId: 42, + engine, + config, + allowedSlugPrefixes: ['wiki/personal/reflections/*'], + sourceId: 'mybrain', + }); + const putPage = tools.find(t => t.name === 'brain_put_page'); + const ctx: ToolCtx = { engine, jobId: 1, remote: true }; + await putPage!.execute( + { slug: 'wiki/personal/reflections/2026-07-17-scoped', content: '---\ntitle: Scoped\n---\nbody' }, + ctx, + ); + const rows = await engine.executeRaw<{ source_id: string }>( + `SELECT source_id FROM pages WHERE slug = 'wiki/personal/reflections/2026-07-17-scoped'`, + ); + expect(rows.length).toBe(1); + expect(rows[0].source_id).toBe('mybrain'); + }); + + test('buildBrainTools rejects a malformed sourceId at build time (#1586)', () => { + expect(() => + buildBrainTools({ subagentId: 1, engine, config, sourceId: '../evil' }), + ).toThrow(); + }); }); describe('filterAllowedTools', () => { diff --git a/test/brain-durability-hook.serial.test.ts b/test/brain-durability-hook.serial.test.ts index b676b5eff..38aff4df9 100644 --- a/test/brain-durability-hook.serial.test.ts +++ b/test/brain-durability-hook.serial.test.ts @@ -90,6 +90,36 @@ describe('brain-commit-push.sh (D13 guarantee)', () => { } catch (e: any) { code = e.status ?? 1; } expect(code).toBe(2); }); + + test('#2426 — commits a MODIFIED tracked file even when the remote advanced (commit before pull)', () => { + // Pre-fix, the helper ran `git pull --rebase` BEFORE staging, so any dirty + // tree (a modified/enriched page — exactly the write-through case) aborted + // with 'cannot pull with rebase: You have unstaged changes' (exit 3). The + // helper could only ever commit untracked-NEW files, never modifications. + // Remove the post-commit hook so its background push can't race the + // helper's own push (macOS has no flock to serialize them) — this test + // targets the HELPER's ordering; hook behavior is covered below. + rmSync(join(work, '.git', 'hooks', 'post-commit')); + // Advance the remote from a second clone so a pull is genuinely needed. + const other = mkdtempSync(join(root, 'other-')); + execFileSync('git', ['-c', 'protocol.file.allow=always', 'clone', '-q', bare, other], { stdio: 'ignore' }); + git(other, 'config', 'user.email', 'o@o.o'); git(other, 'config', 'user.name', 'other'); + writeFileSync(join(other, 'remote.md'), 'from other\n'); + git(other, 'add', 'remote.md'); git(other, 'commit', '-qm', 'remote change'); git(other, 'push', '-q', 'origin', 'main'); + + // Dirty MODIFICATION of a tracked file in the hardened clone (write-through shape). + writeFileSync(join(work, 'README.md'), 'modified by write-through\n'); + execFileSync('bash', [join(work, 'scripts', 'brain-commit-push.sh'), 'wt: README', 'README.md'], { + cwd: work, stdio: ['ignore', 'pipe', 'pipe'], env: process.env, + }); + + // Both the remote's commit and ours are on origin/main. + const subjects = git(bare, 'log', '--format=%s', 'main'); + expect(subjects).toContain('wt: README'); + expect(subjects).toContain('remote change'); + // Working tree is clean — the modification was committed, not stranded. + expect(git(work, 'status', '--porcelain', 'README.md')).toBe(''); + }); }); describe('post-commit hook (D9 local, D7 self-contained)', () => { diff --git a/test/brain-writer-walk-prune.test.ts b/test/brain-writer-walk-prune.test.ts index 1cd3540a0..9b455ca25 100644 --- a/test/brain-writer-walk-prune.test.ts +++ b/test/brain-writer-walk-prune.test.ts @@ -43,8 +43,10 @@ beforeAll(() => { writeFileSync(join(root, '.obsidian', 'workspace.json'), '{}'); mkdirSync(join(root, 'people', 'pedro.raw'), { recursive: true }); writeFileSync(join(root, 'people', 'pedro.raw', 'source.md'), '---\ntitle: should not visit\n---\n'); + // ops/ is ORDINARY content (#2404) — walker MUST descend (it used to be + // wrongly pruned, silently excluding user runbooks / ops/tasks). mkdirSync(join(root, 'ops', 'logs'), { recursive: true }); - writeFileSync(join(root, 'ops', 'logs', 'run.md'), '# nope\n'); + writeFileSync(join(root, 'ops', 'logs', 'run.md'), '---\ntitle: Run\n---\n\nbody\n'); // Nested node_modules — must also be pruned, not just at the root. mkdirSync(join(root, 'people', 'tools', 'node_modules', 'inner'), { recursive: true }); writeFileSync(join(root, 'people', 'tools', 'node_modules', 'inner', 'a.md'), '---\ntitle: nope\n---\n'); @@ -95,8 +97,11 @@ describe('walkDir (brain-writer.ts) — descent-time pruning', () => { walkDir(root, (f) => { files.push(f); }, (dir) => visited.push(dir)); expect(visited.some(d => d.endsWith('/people'))).toBe(true); expect(visited.some(d => d.endsWith('/concepts/subdir'))).toBe(true); + // ops/ is ordinary content — descended, not pruned (#2404). + expect(visited.some(d => d.endsWith('/ops/logs'))).toBe(true); expect(files.some(f => f.endsWith('/people/alice.md'))).toBe(true); expect(files.some(f => f.endsWith('/concepts/subdir/thing.md'))).toBe(true); + expect(files.some(f => f.endsWith('/ops/logs/run.md'))).toBe(true); // And explicitly does NOT visit the file under node_modules. expect(files.some(f => f.includes('/node_modules/'))).toBe(false); }); @@ -107,7 +112,7 @@ describe('walkDir (brain-writer.ts) — descent-time pruning', () => { // visitDir would be called with node_modules paths. const descents: string[] = []; walkDir(root, () => {}, (d) => descents.push(d)); - const vendor = descents.filter(d => /\/(node_modules|\.git|\.obsidian|ops)(\/|$)/.test(d) || /\.raw$/.test(d)); + const vendor = descents.filter(d => /\/(node_modules|\.git|\.obsidian)(\/|$)/.test(d) || /\.raw$/.test(d)); expect(vendor).toEqual([]); }); }); @@ -119,13 +124,20 @@ describe('collectFiles (frontmatter.ts) — descent-time pruning parity', () => expect(visited.some(d => d.includes('/node_modules'))).toBe(false); }); - test('does NOT descend into .git, .obsidian, *.raw, or ops', () => { + test('does NOT descend into .git, .obsidian, or *.raw', () => { const visited: string[] = []; collectFiles(root, (dir) => visited.push(dir)); expect(visited.some(d => d.includes('/.git'))).toBe(false); expect(visited.some(d => d.includes('/.obsidian'))).toBe(false); expect(visited.some(d => d.endsWith('.raw'))).toBe(false); - expect(visited.some(d => d.endsWith('/ops') || d.includes('/ops/'))).toBe(false); + }); + + test('DOES descend into ops/ — ordinary content, not a vendor tree (#2404)', () => { + const visited: string[] = []; + collectFiles(root, (dir) => visited.push(dir)); + expect(visited.some(d => d.endsWith('/ops') || d.includes('/ops/'))).toBe(true); + const files = collectFiles(root); + expect(files.some(f => f.endsWith('/ops/logs/run.md'))).toBe(true); }); test('does NOT descend into git submodule directories', () => { diff --git a/test/chronicle-extract.test.ts b/test/chronicle-extract.test.ts index 404177af4..4152a568d 100644 --- a/test/chronicle-extract.test.ts +++ b/test/chronicle-extract.test.ts @@ -8,7 +8,7 @@ import { describe, test, expect, beforeAll, afterAll, beforeEach } from 'bun:test'; import { PGLiteEngine } from '../src/core/pglite-engine.ts'; import { isChronicleEligible } from '../src/core/chronicle/eligibility.ts'; -import { runChronicleExtract, type ChronicleJudge } from '../src/core/chronicle/extract-events.ts'; +import { runChronicleExtract, parseJudgeJson, type ChronicleJudge } from '../src/core/chronicle/extract-events.ts'; import { runChronicleBackstop } from '../src/core/chronicle/backstop.ts'; let engine: PGLiteEngine; @@ -111,6 +111,44 @@ describe('runChronicleExtract', () => { const r = await runChronicleExtract(engine, { slug: 'meetings/2026-06-18-sync', judge: none }); expect(r.status).toBe('no_events'); }); + + // #2606: a truncated or unparseable judge response must NOT be recorded as + // a legitimate no_events — it gets a distinct skipped reason. + test('truncated judge output → skipped/judge_truncated, not no_events (#2606)', async () => { + const truncated: ChronicleJudge = async () => ({ events: [], failure: 'truncated' }); + const r = await runChronicleExtract(engine, { slug: 'meetings/2026-06-18-sync', judge: truncated }); + expect(r.status).toBe('skipped'); + expect(r.reason).toBe('judge_truncated'); + expect(await countEvents()).toBe(0); + }); + + test('unparseable judge output → skipped/judge_parse_failed (#2606)', async () => { + const parseFailed: ChronicleJudge = async () => ({ events: [], failure: 'parse_failed' }); + const r = await runChronicleExtract(engine, { slug: 'meetings/2026-06-18-sync', judge: parseFailed }); + expect(r.status).toBe('skipped'); + expect(r.reason).toBe('judge_parse_failed'); + }); +}); + +describe('parseJudgeJson failure signalling (#2606)', () => { + test('a legitimate empty array parses to []', () => { + expect(parseJudgeJson('[]')).toEqual([]); + expect(parseJudgeJson('```json\n[]\n```')).toEqual([]); + }); + + test('a valid array round-trips', () => { + const arr = parseJudgeJson('[{"when":"2026-06-18","who":[],"what":"x","kind":"meeting"}]'); + expect(Array.isArray(arr)).toBe(true); + expect(arr!.length).toBe(1); + }); + + test('empty / no-array / truncated / non-array responses return null', () => { + expect(parseJudgeJson('')).toBeNull(); + expect(parseJudgeJson('I found no events worth extracting.')).toBeNull(); + // Truncated mid-array (the maxTokens-cap shape from the issue). + expect(parseJudgeJson('[{"when":"2026-06-18","who":["a"],"what":"long ev')).toBeNull(); + expect(parseJudgeJson('{"events": 1}')).toBeNull(); + }); }); describe('runChronicleBackstop gating', () => { diff --git a/test/cycle-dream-output-root.test.ts b/test/cycle-dream-output-root.test.ts new file mode 100644 index 000000000..384a7a64d --- /dev/null +++ b/test/cycle-dream-output-root.test.ts @@ -0,0 +1,112 @@ +/** + * #2415 — configurable dream output namespace (`dream.synthesize.output_root`). + * + * The synthesize + patterns phases previously hardcoded `wiki/` in the + * subagent prompt slug templates, the patterns reflection lookup, and the + * trusted-workspace allow-list loaded from skills/_brain-filing-rules.json. + * This suite pins: + * - default 'wiki' → byte-identical prompt + verbatim filing-rule globs + * (zero behavior change unless the key is set); + * - a custom root remaps prompt slug templates and the allow-list globs; + * - loadOutputRoot validates against the slug grammar (bad values fall + * back to 'wiki'); + * - the patterns phase gathers reflections under the configured root. + */ +import { describe, test, expect, beforeAll, afterAll } from 'bun:test'; +import { PGLiteEngine } from '../src/core/pglite-engine.ts'; +import { __testing, loadAllowedSlugPrefixes, loadOutputRoot } from '../src/core/cycle/synthesize.ts'; +import { runPhasePatterns } from '../src/core/cycle/patterns.ts'; +import type { DiscoveredTranscript } from '../src/core/cycle/transcript-discovery.ts'; + +const { buildSynthesisPrompt } = __testing; + +const transcript: DiscoveredTranscript = { + filePath: '/tmp/t.txt', + basename: 't', + content: 'User: hello world', + contentHash: 'abcdef0123456789', + inferredDate: '2026-07-17', +} as DiscoveredTranscript; + +describe('#2415: buildSynthesisPrompt output root', () => { + test('defaults to wiki/ slug templates', () => { + const prompt = buildSynthesisPrompt(transcript, 'chunk', 0, 1); + expect(prompt).toContain('wiki/personal/reflections/2026-07-17-'); + expect(prompt).toContain('wiki/originals/ideas/2026-07-17-'); + }); + + test('custom root replaces wiki/ in both slug templates', () => { + const prompt = buildSynthesisPrompt(transcript, 'chunk', 0, 1, '', 'notes'); + expect(prompt).toContain('notes/personal/reflections/2026-07-17-'); + expect(prompt).toContain('notes/originals/ideas/2026-07-17-'); + expect(prompt).not.toContain('wiki/personal/reflections/'); + expect(prompt).not.toContain('wiki/originals/ideas/'); + }); +}); + +describe('#2415: loadAllowedSlugPrefixes remap', () => { + // Runs from the repo root, so skills/_brain-filing-rules.json resolves. + test("default 'wiki' returns the filing-rule globs verbatim", async () => { + const globs = await loadAllowedSlugPrefixes(); + expect(globs).toContain('wiki/personal/reflections/*'); + expect(globs).toContain('dream-cycle-summaries/*'); + }); + + test('custom root remaps only wiki/-rooted globs', async () => { + const globs = await loadAllowedSlugPrefixes('notes'); + expect(globs).toContain('notes/personal/reflections/*'); + expect(globs).toContain('notes/originals/*'); + expect(globs).toContain('notes/personal/patterns/*'); + // Non-wiki globs pass through untouched. + expect(globs).toContain('dream-cycle-summaries/*'); + expect(globs.some(g => g.startsWith('wiki/'))).toBe(false); + }); +}); + +describe('#2415: loadOutputRoot validation + patterns gather scope', () => { + let engine: PGLiteEngine; + + beforeAll(async () => { + engine = new PGLiteEngine(); + await engine.connect({}); + await engine.initSchema(); + }); + + afterAll(async () => { + await engine.disconnect(); + }); + + test('unset → wiki; trailing slash trimmed; invalid → wiki fallback', async () => { + expect(await loadOutputRoot(engine)).toBe('wiki'); + await engine.setConfig('dream.synthesize.output_root', 'notes/'); + expect(await loadOutputRoot(engine)).toBe('notes'); + await engine.setConfig('dream.synthesize.output_root', '../escape'); + expect(await loadOutputRoot(engine)).toBe('wiki'); + await engine.setConfig('dream.synthesize.output_root', 'Bad_Root'); + expect(await loadOutputRoot(engine)).toBe('wiki'); + }); + + test('patterns phase gathers reflections under the configured root', async () => { + await engine.setConfig('dream.synthesize.output_root', 'notes'); + for (let i = 0; i < 3; i++) { + await engine.putPage(`notes/personal/reflections/2026-07-17-r${i}`, { + type: 'note', + title: `R${i}`, + compiled_truth: `reflection ${i}`, + timeline: '', + frontmatter: {}, + }); + } + // A wiki/-rooted reflection must NOT be counted under the custom root. + await engine.putPage('wiki/personal/reflections/2026-07-17-old', { + type: 'note', + title: 'Old', + compiled_truth: 'legacy reflection', + timeline: '', + frontmatter: {}, + }); + const result = await runPhasePatterns(engine, { brainDir: '/tmp', dryRun: true }); + expect(result.status).toBe('ok'); + expect(result.details?.reflections_considered).toBe(3); + }); +}); diff --git a/test/cycle-patterns.test.ts b/test/cycle-patterns.test.ts index 1ea368d89..f50892348 100644 --- a/test/cycle-patterns.test.ts +++ b/test/cycle-patterns.test.ts @@ -74,8 +74,12 @@ describe('patterns phase wiring', () => { }); describe('patterns scope filter', () => { - test('filters reflections by slug LIKE wiki/personal/reflections/%', () => { - expect(patternsSrc).toContain("slug LIKE 'wiki/personal/reflections/%'"); + test('filters reflections by slug LIKE /personal/reflections/%', () => { + // #2415: the namespace root is configurable (dream.synthesize.output_root, + // default 'wiki') and bound as a parameter — the scope filter itself and + // the reflections sub-path stay pinned. + expect(patternsSrc).toContain('slug LIKE $2'); + expect(patternsSrc).toContain('/personal/reflections/%'); }); test('orders by updated_at DESC for recency-bias', () => { diff --git a/test/cycle-synthesize-slug-collection.test.ts b/test/cycle-synthesize-slug-collection.test.ts index bdbba6c4c..1ccbaa27e 100644 --- a/test/cycle-synthesize-slug-collection.test.ts +++ b/test/cycle-synthesize-slug-collection.test.ts @@ -21,7 +21,7 @@ import { describe, test, expect, beforeAll, afterAll } from 'bun:test'; import { PGLiteEngine } from '../src/core/pglite-engine.ts'; import { __testing } from '../src/core/cycle/synthesize.ts'; -const { collectChildPutPageSlugs } = __testing; +const { collectChildPutPageSlugs, stampDreamProvenance } = __testing; let engine: PGLiteEngine; @@ -103,4 +103,52 @@ describe('C6: collectChildPutPageSlugs survives double-encoded jsonb (#745)', () // Function silently drops rows whose slug resolves to null/empty. expect(refs.map((r: { slug: string }) => r.slug)).not.toContain('no-slug'); }); + + // #1586: refs are stamped with the cycle's resolved source, not a + // hardcoded 'default'. + test('stamps refs with the provided cycle sourceId (#1586)', async () => { + const refs = await collectChildPutPageSlugs(engine as any, [1001], new Map(), 'mybrain'); + expect(refs.length).toBeGreaterThan(0); + for (const r of refs) expect(r.source_id).toBe('mybrain'); + }); + + test('defaults to source_id=default when no sourceId is passed (legacy)', async () => { + const refs = await collectChildPutPageSlugs(engine as any, [1001], new Map()); + expect(refs.length).toBeGreaterThan(0); + for (const r of refs) expect(r.source_id).toBe('default'); + }); +}); + +describe('#2569: stampDreamProvenance persists the marker into DB frontmatter', () => { + test('merges dream_generated + dream_cycle_date into pages.frontmatter', async () => { + await engine.putPage('wiki/originals/ideas/2026-07-17-stamp-me-abc123', { + type: 'note', + title: 'Stamp me', + compiled_truth: 'body', + timeline: '', + frontmatter: { keep_me: 'yes' }, + }); + await stampDreamProvenance( + engine as any, + [{ slug: 'wiki/originals/ideas/2026-07-17-stamp-me-abc123', source_id: 'default' }], + '2026-07-17', + ); + const rows = await engine.executeRaw<{ fm: Record }>( + `SELECT frontmatter AS fm FROM pages WHERE slug = 'wiki/originals/ideas/2026-07-17-stamp-me-abc123'`, + ); + expect(rows.length).toBe(1); + const fm = rows[0].fm as Record; + // The stamp lands as real JSONB values (queryable via ->>), not a + // double-encoded string scalar. + expect(fm.dream_generated).toBe(true); + expect(fm.dream_cycle_date).toBe('2026-07-17'); + // Merge, not replace: pre-existing frontmatter keys survive. + expect(fm.keep_me).toBe('yes'); + }); + + test('is idempotent and never throws for a missing page', async () => { + const refs = [{ slug: 'wiki/originals/ideas/does-not-exist', source_id: 'default' }]; + await stampDreamProvenance(engine as any, refs, '2026-07-17'); // no throw + await stampDreamProvenance(engine as any, refs, '2026-07-17'); // idempotent + }); }); diff --git a/test/cycle/extract-atoms-synthesize-concepts.test.ts b/test/cycle/extract-atoms-synthesize-concepts.test.ts index a22874110..d14102495 100644 --- a/test/cycle/extract-atoms-synthesize-concepts.test.ts +++ b/test/cycle/extract-atoms-synthesize-concepts.test.ts @@ -316,4 +316,32 @@ describe('v0.41 T6: runPhaseSynthesizeConcepts via stubbed chat', () => { ); expect(rows[0].compiled_truth).toContain('Custom synthesized narrative'); }); + + // #2163: concept pages must enter the retrieval surface. The write routes + // through importFromContent (the same parse→chunk pipeline put_page uses), + // so content_chunks rows exist and source-boost's 1.3× 'concepts/' weight + // has something to boost. (Embeddings are skipped in this env — no + // provider — but chunks + search_vector land regardless.) + test('concept pages are chunked (#2163)', async () => { + const atoms = Array.from({ length: 12 }, (_, i) => ({ + slug: `c${i}`, + title: `Chunk atom ${i}`, + body: `Chunky body ${i}.`, + concept_refs: ['chunked-concept'], + })); + const chat = stubChat('A concept narrative long enough to produce at least one chunk.'); + await runPhaseSynthesizeConcepts(engine, { _atoms: atoms, _chat: chat }); + const rows = await engine.executeRaw<{ n: number }>( + `SELECT count(*)::int AS n + FROM content_chunks c JOIN pages p ON p.id = c.page_id + WHERE p.slug = 'concepts/chunked-concept'`, + ); + expect(Number(rows[0].n)).toBeGreaterThan(0); + // Page metadata survives the importFromContent round-trip. + const page = await engine.executeRaw<{ type: string; fm: Record }>( + `SELECT type, frontmatter AS fm FROM pages WHERE slug = 'concepts/chunked-concept'`, + ); + expect(page[0].type).toBe('concept'); + expect((page[0].fm as Record).tier).toBe('T1'); + }); }); diff --git a/test/e2e/sync.test.ts b/test/e2e/sync.test.ts index e8cc62631..3f02767c2 100644 --- a/test/e2e/sync.test.ts +++ b/test/e2e/sync.test.ts @@ -224,7 +224,7 @@ describeE2E('E2E: Git-to-DB Sync Pipeline', () => { expect(bob).toBeNull(); }); - test('sync skips non-syncable files (README, hidden, .raw)', async () => { + test('sync skips non-syncable files (README, hidden, .raw) but imports ops/ (#2404)', async () => { const { performSync } = await import('../../src/commands/sync.ts'); const engine = getEngine(); @@ -249,8 +249,9 @@ describeE2E('E2E: Git-to-DB Sync Pipeline', () => { const raw = await engine.getPage('.raw/data'); expect(raw).toBeNull(); + // ops/ is ordinary content and DOES sync (#2404). const ops = await engine.getPage('ops/deploy'); - expect(ops).toBeNull(); + expect(ops).not.toBeNull(); }); test('sync stores last_commit and last_run in config', async () => { diff --git a/test/extract-facts-phase.test.ts b/test/extract-facts-phase.test.ts index 3ddfc0bb8..1fdd24ef5 100644 --- a/test/extract-facts-phase.test.ts +++ b/test/extract-facts-phase.test.ts @@ -12,6 +12,7 @@ import { describe, test, expect, beforeAll, afterAll, beforeEach } from 'bun:test'; import { PGLiteEngine } from '../src/core/pglite-engine.ts'; import { runExtractFacts } from '../src/core/cycle/extract-facts.ts'; +import { parseFactsFence } from '../src/core/facts-fence.ts'; let engine: PGLiteEngine; @@ -99,11 +100,114 @@ describe('runExtractFacts — happy path', () => { ); expect(r2.guardTriggered).toBe(false); + expect(r2.factsInserted).toBe(0); + expect(r2.factsDeleted).toBe(0); expect(after2.rows.map((r: { fact: string }) => r.fact)) .toEqual(after1.rows.map((r: { fact: string }) => r.fact)); expect(after2.rows).toHaveLength(2); }); + test('dedupes duplicate fence rows by claim and source without rewriting the fence', async () => { + const body = FACT_FENCE( + `| 1 | A | fact | 1.0 | world | medium | 2026-01-01 | | s | | +| 2 | A | fact | 1.0 | world | medium | 2026-01-01 | | s | |`, + ); + await putPage('people/alice', body); + + const r1 = await runExtractFacts(engine, { slugs: ['people/alice'] }); + const r2 = await runExtractFacts(engine, { slugs: ['people/alice'] }); + + expect(r1.factsInserted).toBe(1); + expect(r2.factsInserted).toBe(0); + expect(r2.factsDeleted).toBe(0); + + // The cycle dedups the derived DB index; it does not destructively + // rewrite user-authored markdown fence rows. + const page = await engine.getPage('people/alice', { sourceId: 'default' }); + expect(parseFactsFence(page?.compiled_truth ?? '').facts).toHaveLength(2); + + // eslint-disable-next-line @typescript-eslint/no-explicit-any + const rows = await (engine as any).db.query( + `SELECT fact, source FROM facts WHERE source_markdown_slug = 'people/alice'`, + ); + expect(rows.rows).toHaveLength(1); + expect(rows.rows[0]).toMatchObject({ fact: 'A', source: 's' }); + }); + + test('same claim with a different source is not treated as duplicate', async () => { + await putPage('people/alice', FACT_FENCE( + `| 1 | Same claim | fact | 1.0 | world | medium | 2026-01-01 | | source-a | |`, + )); + await runExtractFacts(engine, { slugs: ['people/alice'] }); + + await putPage('people/alice', FACT_FENCE( + `| 1 | Same claim | fact | 1.0 | world | medium | 2026-01-01 | | source-a | | +| 2 | Same claim | fact | 1.0 | world | medium | 2026-01-01 | | source-b | |`, + )); + const r = await runExtractFacts(engine, { slugs: ['people/alice'] }); + + expect(r.factsInserted).toBe(1); + expect(r.factsDeleted).toBe(0); + + // eslint-disable-next-line @typescript-eslint/no-explicit-any + const rows = await (engine as any).db.query( + `SELECT fact, source FROM facts WHERE source_markdown_slug = 'people/alice' ORDER BY row_num`, + ); + expect(rows.rows).toEqual([ + expect.objectContaining({ fact: 'Same claim', source: 'source-a' }), + expect.objectContaining({ fact: 'Same claim', source: 'source-b' }), + ]); + }); + + test('new fact added to the fence is inserted once without re-appending existing facts', async () => { + await putPage('people/alice', FACT_FENCE( + `| 1 | Existing | fact | 1.0 | world | medium | 2026-01-01 | | s | |`, + )); + await runExtractFacts(engine, { slugs: ['people/alice'] }); + + await putPage('people/alice', FACT_FENCE( + `| 1 | Existing | fact | 1.0 | world | medium | 2026-01-01 | | s | | +| 2 | New | fact | 1.0 | world | medium | 2026-01-01 | | s | |`, + )); + + const r = await runExtractFacts(engine, { slugs: ['people/alice'] }); + expect(r.factsInserted).toBe(1); + expect(r.factsDeleted).toBe(0); + + // eslint-disable-next-line @typescript-eslint/no-explicit-any + const rows = await (engine as any).db.query( + `SELECT fact FROM facts WHERE source_markdown_slug = 'people/alice' ORDER BY row_num`, + ); + expect(rows.rows.map((row: { fact: string }) => row.fact)).toEqual(['Existing', 'New']); + }); + + test('cli:-origin conversation facts (#1928) neither break idempotency nor get wiped', async () => { + await putPage('people/alice', FACT_FENCE( + `| 1 | Fence fact | fact | 1.0 | world | medium | 2026-01-01 | | s | |`, + )); + // A conversation fact on the same page coordinate — NOT fence-owned. + await engine.insertFacts( + [{ fact: 'conversation fact', kind: 'fact', source: 'cli:extract-conversation-facts', row_num: 99, source_markdown_slug: 'people/alice' }], + { source_id: 'default' }, + ); + + const r1 = await runExtractFacts(engine, { slugs: ['people/alice'] }); + const r2 = await runExtractFacts(engine, { slugs: ['people/alice'] }); + + // The cli: row must not count as "stale" — a wipe/reinsert every cycle + // would defeat idempotency (and churn factsDeleted/factsInserted). + expect(r1.factsInserted).toBe(1); + expect(r2.factsInserted).toBe(0); + expect(r2.factsDeleted).toBe(0); + + // eslint-disable-next-line @typescript-eslint/no-explicit-any + const rows = await (engine as any).db.query( + `SELECT fact FROM facts WHERE source_markdown_slug = 'people/alice' ORDER BY row_num`, + ); + expect(rows.rows.map((row: { fact: string }) => row.fact)) + .toEqual(['Fence fact', 'conversation fact']); + }); + test('removed-from-fence row is deleted from DB (wipe-and-reinsert pattern)', async () => { // Seed: 2 facts. await putPage('people/alice', FACT_FENCE( diff --git a/test/fts-language-migration.serial.test.ts b/test/fts-language-migration.serial.test.ts new file mode 100644 index 000000000..da7841da0 --- /dev/null +++ b/test/fts-language-migration.serial.test.ts @@ -0,0 +1,119 @@ +import { describe, test, expect, beforeEach, afterEach } from 'bun:test'; +import type { BrainEngine } from '../src/core/engine.ts'; +import { MIGRATIONS, LATEST_VERSION } from '../src/core/migrate.ts'; +import { resetFtsLanguageCache } from '../src/core/fts-language.ts'; + +const ENV_KEY = 'GBRAIN_FTS_LANGUAGE'; +const originalLang = process.env[ENV_KEY]; + +beforeEach(() => { + delete process.env[ENV_KEY]; + resetFtsLanguageCache(); +}); + +afterEach(() => { + delete process.env[ENV_KEY]; + if (originalLang !== undefined) process.env[ENV_KEY] = originalLang; + resetFtsLanguageCache(); +}); + +describe('configurable_fts_language migration', () => { + test('migration is registered', () => { + const ftsMig = MIGRATIONS.find(m => m.name === 'configurable_fts_language'); + expect(ftsMig).toBeDefined(); + expect(ftsMig?.version).toBeGreaterThan(115); + }); + + test('fts migration is the latest migration', () => { + expect(MIGRATIONS.find(m => m.name === 'configurable_fts_language')?.version).toBe(LATEST_VERSION); + }); + + test('ftsMig uses handler (not static SQL) because language interpolation is dynamic', () => { + const ftsMig = MIGRATIONS.find(m => m.name === 'configurable_fts_language'); + expect(ftsMig?.sql).toBe(''); + expect(ftsMig?.handler).toBeTypeOf('function'); + }); + + test('ftsMig handler is async', () => { + const ftsMig = MIGRATIONS.find(m => m.name === 'configurable_fts_language'); + // Async function check: the constructor name is 'AsyncFunction' + expect(ftsMig?.handler?.constructor.name).toBe('AsyncFunction'); + }); + + test('migration handler issues recreate-function calls (smoke check via mock engine)', async () => { + const ftsMig = MIGRATIONS.find(m => m.name === 'configurable_fts_language'); + const calls: string[] = []; + + const mockEngine = { + executeRaw: async (sql: string) => { + calls.push(sql); + return []; + }, + } as unknown as BrainEngine; + + process.env[ENV_KEY] = 'english'; + resetFtsLanguageCache(); + + await ftsMig?.handler?.(mockEngine); + + // Default 'english' \u2014 no backfill, only 2 CREATE OR REPLACE calls. + expect(calls.length).toBe(2); + expect(calls[0]).toContain('CREATE OR REPLACE FUNCTION update_page_search_vector'); + expect(calls[0]).toContain("to_tsvector('english'"); + expect(calls[1]).toContain('CREATE OR REPLACE FUNCTION update_chunk_search_vector'); + expect(calls[1]).toContain("to_tsvector('english'"); + // v120/#1647 hardening must survive the CREATE OR REPLACE (which resets + // proconfig): both recreated bodies pin search_path. + expect(calls[0]).toContain('SET search_path = pg_catalog, public'); + expect(calls[1]).toContain('SET search_path = pg_catalog, public'); + }); + + test('non-english language triggers backfill', async () => { + const ftsMig = MIGRATIONS.find(m => m.name === 'configurable_fts_language'); + const calls: string[] = []; + + const mockEngine = { + executeRaw: async (sql: string) => { + calls.push(sql); + return []; + }, + } as unknown as BrainEngine; + + process.env[ENV_KEY] = 'pt_br'; + resetFtsLanguageCache(); + + await ftsMig?.handler?.(mockEngine); + + // pt_br \u2014 2 CREATE + 2 backfill UPDATEs = 4 calls + expect(calls.length).toBe(4); + expect(calls[0]).toContain("to_tsvector('pt_br'"); + expect(calls[1]).toContain("to_tsvector('pt_br'"); + expect(calls[2]).toMatch(/UPDATE pages/); + expect(calls[3]).toContain("to_tsvector('pt_br'"); + expect(calls[3]).toMatch(/UPDATE content_chunks/); + }); + + test('invalid language falls back to english (no SQL injection)', async () => { + const ftsMig = MIGRATIONS.find(m => m.name === 'configurable_fts_language'); + const calls: string[] = []; + + const mockEngine = { + executeRaw: async (sql: string) => { + calls.push(sql); + return []; + }, + } as unknown as BrainEngine; + + process.env[ENV_KEY] = "english'; DROP TABLE pages; --"; + resetFtsLanguageCache(); + + await ftsMig?.handler?.(mockEngine); + + // Falls back to english: 2 CREATE OR REPLACE only, no DROP TABLE in any SQL. + expect(calls.length).toBe(2); + for (const sql of calls) { + expect(sql).not.toContain('DROP TABLE'); + expect(sql).toContain("to_tsvector('english'"); + } + }); +}); diff --git a/test/fts-language.serial.test.ts b/test/fts-language.serial.test.ts new file mode 100644 index 000000000..ea6c4cfcb --- /dev/null +++ b/test/fts-language.serial.test.ts @@ -0,0 +1,93 @@ +import { describe, test, expect, beforeEach, afterEach } from 'bun:test'; +import { getFtsLanguage, resetFtsLanguageCache } from '../src/core/fts-language.ts'; + +const ENV_KEY = 'GBRAIN_FTS_LANGUAGE'; + +beforeEach(() => { + delete process.env[ENV_KEY]; + resetFtsLanguageCache(); +}); + +afterEach(() => { + delete process.env[ENV_KEY]; + resetFtsLanguageCache(); +}); + +describe('getFtsLanguage', () => { + test('defaults to english when env is unset', () => { + expect(getFtsLanguage()).toBe('english'); + }); + + test('defaults to english when env is empty string', () => { + process.env[ENV_KEY] = ''; + expect(getFtsLanguage()).toBe('english'); + }); + + test('defaults to english when env is whitespace', () => { + process.env[ENV_KEY] = ' '; + expect(getFtsLanguage()).toBe('english'); + }); + + test('reads valid pt_br config', () => { + process.env[ENV_KEY] = 'pt_br'; + expect(getFtsLanguage()).toBe('pt_br'); + }); + + test('reads valid simple language name', () => { + process.env[ENV_KEY] = 'spanish'; + expect(getFtsLanguage()).toBe('spanish'); + }); + + test('reads name with underscores and digits', () => { + process.env[ENV_KEY] = 'custom_lang_v2'; + expect(getFtsLanguage()).toBe('custom_lang_v2'); + }); + + test('rejects names with quotes (SQL injection guard)', () => { + process.env[ENV_KEY] = "english'; DROP TABLE pages; --"; + expect(getFtsLanguage()).toBe('english'); + }); + + test('rejects names with spaces', () => { + process.env[ENV_KEY] = 'pt br'; + expect(getFtsLanguage()).toBe('english'); + }); + + test('rejects names with hyphens', () => { + process.env[ENV_KEY] = 'pt-br'; + expect(getFtsLanguage()).toBe('english'); + }); + + test('rejects names starting with digit', () => { + process.env[ENV_KEY] = '1lang'; + expect(getFtsLanguage()).toBe('english'); + }); + + test('rejects uppercase (Postgres config names are lowercase)', () => { + process.env[ENV_KEY] = 'English'; + expect(getFtsLanguage()).toBe('english'); + }); + + test('caches after first read', () => { + process.env[ENV_KEY] = 'pt_br'; + expect(getFtsLanguage()).toBe('pt_br'); + + // Mutate env after first read \u2014 cached value wins. + process.env[ENV_KEY] = 'spanish'; + expect(getFtsLanguage()).toBe('pt_br'); + }); + + test('resetFtsLanguageCache clears cache', () => { + process.env[ENV_KEY] = 'pt_br'; + expect(getFtsLanguage()).toBe('pt_br'); + + resetFtsLanguageCache(); + process.env[ENV_KEY] = 'spanish'; + expect(getFtsLanguage()).toBe('spanish'); + }); + + test('trims surrounding whitespace from valid value', () => { + process.env[ENV_KEY] = ' pt_br '; + expect(getFtsLanguage()).toBe('pt_br'); + }); +}); diff --git a/test/import-git-fastpath-prune.test.ts b/test/import-git-fastpath-prune.test.ts new file mode 100644 index 000000000..f591646e2 --- /dev/null +++ b/test/import-git-fastpath-prune.test.ts @@ -0,0 +1,90 @@ +/** + * #2607 — the `sync --full` git fast path applies the same prune gate as + * incremental sync. + * + * Bug class: `collectSyncableFiles` on a git work tree takes the + * `git ls-files` fast path, which historically filtered ONLY by + * strategy/extension + .gitignore — no `pruneDir`, so `sync --full` + * imported (and resurrected previously-soft-deleted) pages under dot-dirs + * and vendored trees that incremental sync's `isSyncable` excludes. The two + * enumeration modes cycled content in and out depending on which ran last. + * + * Fix: `isCollectibleForWalker` (shared by the git fast path AND the FS-walk + * emit filter) now rejects any path with a segment `pruneDir` would block — + * the same segment rule `classifySync` applies on the incremental path. + * + * No PGLite needed: `collectSyncableFiles` is pure filesystem + git. + */ + +import { describe, test, expect, beforeAll, afterAll } from 'bun:test'; +import { mkdtempSync, writeFileSync, rmSync, mkdirSync } from 'fs'; +import { execSync } from 'child_process'; +import { tmpdir } from 'os'; +import { join, relative } from 'path'; +import { collectSyncableFiles } from '../src/commands/import.ts'; +import { isSyncable } from '../src/core/sync.ts'; + +let repo: string; + +function rel(files: string[]): string[] { + return files.map((f) => relative(repo, f)); +} + +beforeAll(() => { + repo = mkdtempSync(join(tmpdir(), 'gbrain-fastpath-')); + execSync('git init', { cwd: repo, stdio: 'pipe' }); + execSync('git config user.email "t@t.t"', { cwd: repo, stdio: 'pipe' }); + execSync('git config user.name "T"', { cwd: repo, stdio: 'pipe' }); + + // Ordinary content — must be collected. + mkdirSync(join(repo, 'notes'), { recursive: true }); + writeFileSync(join(repo, 'notes/real.md'), '---\ntitle: Real\n---\nbody\n'); + mkdirSync(join(repo, 'ops'), { recursive: true }); + writeFileSync(join(repo, 'ops/tasks.md'), '---\ntitle: Tasks\n---\nbody\n'); + + // TRACKED files under excluded trees — `git ls-files` returns these, so + // only the prune gate keeps them out (this is the #2607 divergence). + mkdirSync(join(repo, '.obsidian'), { recursive: true }); + writeFileSync(join(repo, '.obsidian/plugin-notes.md'), 'not a page\n'); + mkdirSync(join(repo, 'vendor/pkg'), { recursive: true }); + writeFileSync(join(repo, 'vendor/pkg/notes.md'), 'vendored\n'); + mkdirSync(join(repo, 'node_modules/dep'), { recursive: true }); + writeFileSync(join(repo, 'node_modules/dep/CHANGELOG.md'), 'dep changelog\n'); + mkdirSync(join(repo, 'people/pedro.raw'), { recursive: true }); + writeFileSync(join(repo, 'people/pedro.raw/source.md'), 'raw sidecar\n'); + + // Metafiles — excluded on both routes (pre-existing #345 behavior). + writeFileSync(join(repo, 'README.md'), '# repo\n'); + writeFileSync(join(repo, 'notes/index.md'), '# index\n'); + + execSync('git add -A -f && git commit -m "fixture"', { cwd: repo, stdio: 'pipe' }); +}); + +afterAll(() => { + if (repo) rmSync(repo, { recursive: true, force: true }); +}); + +describe('#2607 — git fast path excludes what incremental sync excludes', () => { + test('tracked files under pruned dirs are NOT collected', () => { + const files = rel(collectSyncableFiles(repo, { strategy: 'markdown' })); + expect(files).toContain('notes/real.md'); + expect(files).toContain('ops/tasks.md'); // ordinary content (#2404) + expect(files).not.toContain('.obsidian/plugin-notes.md'); + expect(files).not.toContain('vendor/pkg/notes.md'); + expect(files).not.toContain('node_modules/dep/CHANGELOG.md'); + expect(files).not.toContain('people/pedro.raw/source.md'); + // Metafiles stay excluded too. + expect(files).not.toContain('README.md'); + expect(files).not.toContain('notes/index.md'); + }); + + test('full-sync enumeration agrees with incremental isSyncable for every collected file', () => { + // The single-source-of-truth contract: nothing the full path collects may + // be something the incremental path would refuse to sync. + const files = rel(collectSyncableFiles(repo, { strategy: 'markdown' })); + for (const f of files) { + expect({ path: f, syncable: isSyncable(f) }).toEqual({ path: f, syncable: true }); + } + expect(files.length).toBeGreaterThan(0); + }); +}); diff --git a/test/postgres-engine-rls-scope.test.ts b/test/postgres-engine-rls-scope.test.ts new file mode 100644 index 000000000..ade5a3227 --- /dev/null +++ b/test/postgres-engine-rls-scope.test.ts @@ -0,0 +1,181 @@ +/** + * withScopedReadTransaction — opt-in Postgres RLS source-scope binding + * (GBRAIN_RLS_SCOPE_BINDING, lands community PR #2387). + * + * Behavioral pins, no real DB (fake postgres.js sql handle): + * - flag OFF (default): TRUE pass-through — callback receives the shared + * pool handle directly, no sql.begin(), no set_config. This is the + * #1794-class guard: reads must not gain a per-read pool hold. + * - flag OFF + alwaysTransaction (the search methods' SET LOCAL path): + * sql.begin() opens, still no set_config — identical to master's wrap. + * - flag ON: sql.begin() + SELECT set_config('app.scopes', $1, true) + * with federated-array > scalar > '*' precedence, and the CSV value + * carried as a BOUND PARAMETER, never interpolated into the SQL text. + */ + +import { describe, test, expect } from 'bun:test'; +import { PostgresEngine } from '../src/core/postgres-engine.ts'; +import { withEnv } from './helpers/with-env.ts'; + +type Recorded = { text: string; params: unknown[] }; + +function makeFakeSql() { + const queries: Recorded[] = []; + let beginCalls = 0; + const record = (strings: TemplateStringsArray, ...params: unknown[]) => { + // Join the literal segments with a placeholder marker so the test can + // assert the exact SQL text shape around each bound parameter. + queries.push({ text: strings.join('${}'), params }); + return Promise.resolve([]); + }; + const sql = ((strings: TemplateStringsArray, ...params: unknown[]) => + record(strings, ...params)) as unknown as Record & { + (strings: TemplateStringsArray, ...params: unknown[]): Promise; + begin: (cb: (tx: unknown) => Promise) => Promise; + }; + const tx = ((strings: TemplateStringsArray, ...params: unknown[]) => + record(strings, ...params)) as unknown as Record; + sql.begin = async (cb: (t: unknown) => Promise) => { + beginCalls++; + return await cb(tx); + }; + return { sql, tx, queries, beginCalls: () => beginCalls }; +} + +function makeEngine(fake: ReturnType) { + const e = new PostgresEngine(); + (e as unknown as { _sql: unknown })._sql = fake.sql; + (e as unknown as { _connectionStyle: string })._connectionStyle = 'instance'; + // private method, invoked directly for the pin + return e as unknown as { + withScopedReadTransaction( + sourceIds: string[] | undefined, + sourceId: string | undefined, + cb: (tx: unknown) => Promise, + opts?: { alwaysTransaction?: boolean }, + ): Promise; + }; +} + +function setConfigQueries(queries: Recorded[]): Recorded[] { + return queries.filter((q) => q.text.includes('set_config')); +} + +describe('withScopedReadTransaction / flag off (default)', () => { + test('true pass-through: callback gets the shared pool handle, no begin, no set_config', async () => { + await withEnv({ GBRAIN_RLS_SCOPE_BINDING: undefined }, async () => { + const fake = makeFakeSql(); + const engine = makeEngine(fake); + let received: unknown; + const result = await engine.withScopedReadTransaction(undefined, 'src-a', async (tx) => { + received = tx; + return 42; + }); + expect(result).toBe(42); + expect(received).toBe(fake.sql); // the pool handle itself, not a tx + expect(fake.beginCalls()).toBe(0); + expect(setConfigQueries(fake.queries)).toHaveLength(0); + }); + }); + + test('explicit "0" is off too', async () => { + await withEnv({ GBRAIN_RLS_SCOPE_BINDING: '0' }, async () => { + const fake = makeFakeSql(); + const engine = makeEngine(fake); + await engine.withScopedReadTransaction(['a', 'b'], undefined, async () => null); + expect(fake.beginCalls()).toBe(0); + expect(setConfigQueries(fake.queries)).toHaveLength(0); + }); + }); + + test('alwaysTransaction keeps master\'s sql.begin() wrap, still no set_config', async () => { + await withEnv({ GBRAIN_RLS_SCOPE_BINDING: undefined }, async () => { + const fake = makeFakeSql(); + const engine = makeEngine(fake); + let received: unknown; + await engine.withScopedReadTransaction( + undefined, + 'src-a', + async (tx) => { + received = tx; + return null; + }, + { alwaysTransaction: true }, + ); + expect(fake.beginCalls()).toBe(1); + expect(received).toBe(fake.tx); // a transaction handle this time + expect(setConfigQueries(fake.queries)).toHaveLength(0); + }); + }); +}); + +describe('withScopedReadTransaction / flag on', () => { + test('emits set_config(\'app.scopes\', ...) inside a transaction, before the callback', async () => { + await withEnv({ GBRAIN_RLS_SCOPE_BINDING: '1' }, async () => { + const fake = makeFakeSql(); + const engine = makeEngine(fake); + let queriesAtCallback = -1; + await engine.withScopedReadTransaction(undefined, 'src-a', async () => { + queriesAtCallback = fake.queries.length; + return null; + }); + expect(fake.beginCalls()).toBe(1); + const sc = setConfigQueries(fake.queries); + expect(sc).toHaveLength(1); + expect(sc[0].params).toEqual(['src-a']); + // set_config was emitted before the callback ran + expect(queriesAtCallback).toBe(1); + expect(fake.queries[0]).toBe(sc[0]); + }); + }); + + test('"true" also enables', async () => { + await withEnv({ GBRAIN_RLS_SCOPE_BINDING: 'true' }, async () => { + const fake = makeFakeSql(); + const engine = makeEngine(fake); + await engine.withScopedReadTransaction(undefined, 'src-a', async () => null); + expect(setConfigQueries(fake.queries)).toHaveLength(1); + }); + }); + + test('federated array wins over scalar: CSV of sourceIds', async () => { + await withEnv({ GBRAIN_RLS_SCOPE_BINDING: '1' }, async () => { + const fake = makeFakeSql(); + const engine = makeEngine(fake); + await engine.withScopedReadTransaction(['a', 'b', 'c'], 'ignored-scalar', async () => null); + expect(setConfigQueries(fake.queries)[0].params).toEqual(['a,b,c']); + }); + }); + + test('empty federated array falls back to scalar', async () => { + await withEnv({ GBRAIN_RLS_SCOPE_BINDING: '1' }, async () => { + const fake = makeFakeSql(); + const engine = makeEngine(fake); + await engine.withScopedReadTransaction([], 'src-b', async () => null); + expect(setConfigQueries(fake.queries)[0].params).toEqual(['src-b']); + }); + }); + + test("unscoped (no sourceIds, no sourceId) binds '*'", async () => { + await withEnv({ GBRAIN_RLS_SCOPE_BINDING: '1' }, async () => { + const fake = makeFakeSql(); + const engine = makeEngine(fake); + await engine.withScopedReadTransaction(undefined, undefined, async () => null); + expect(setConfigQueries(fake.queries)[0].params).toEqual(['*']); + }); + }); + + test('the scopes CSV is a BOUND PARAMETER, never interpolated into SQL text', async () => { + await withEnv({ GBRAIN_RLS_SCOPE_BINDING: '1' }, async () => { + const fake = makeFakeSql(); + const engine = makeEngine(fake); + const hostile = "x','y'); DROP TABLE pages; --"; + await engine.withScopedReadTransaction(undefined, hostile, async () => null); + const sc = setConfigQueries(fake.queries)[0]; + // Exact literal-segment shape: the value slot is the tagged-template hole. + expect(sc.text).toBe("SELECT set_config('app.scopes', ${}, true)"); + expect(sc.params).toEqual([hostile]); + expect(sc.text).not.toContain(hostile); + }); + }); +}); diff --git a/test/postgres-engine.test.ts b/test/postgres-engine.test.ts index e750f9d5f..8d51fc894 100644 --- a/test/postgres-engine.test.ts +++ b/test/postgres-engine.test.ts @@ -48,14 +48,34 @@ describe('postgres-engine / search path timeout isolation', () => { expect(bare).toBeNull(); }); - test('searchKeyword wraps its query in sql.begin()', () => { + test('searchKeyword wraps its query in a transaction (via withScopedReadTransaction alwaysTransaction)', () => { + // Post-RLS-scope-binding invariant: the search methods route through + // withScopedReadTransaction with alwaysTransaction: true, which + // guarantees a sql.begin() wrap in BOTH modes — flag off (identical to + // master's pre-helper wrap) and flag on (scoped transaction with + // set_config). See the helper tests in + // test/postgres-engine-rls-scope.test.ts for the behavioral pins. const fn = extractMethod(SRC, 'searchKeyword'); - expect(fn).toMatch(/sql\.begin\s*\(\s*async\s+sql\s*=>/); + expect(fn).toMatch(/withScopedReadTransaction\s*\(/); + expect(fn).toMatch(/alwaysTransaction:\s*true/); }); - test('searchVector wraps its query in sql.begin()', () => { + test('searchVector wraps its query in a transaction (via withScopedReadTransaction alwaysTransaction)', () => { const fn = extractMethod(SRC, 'searchVector'); - expect(fn).toMatch(/sql\.begin\s*\(\s*async\s+sql\s*=>/); + expect(fn).toMatch(/withScopedReadTransaction\s*\(/); + expect(fn).toMatch(/alwaysTransaction:\s*true/); + }); + + test('withScopedReadTransaction owns the sql.begin() wrap (and only opens it when needed)', () => { + // (extractMethod can't grab this one: `private async ...(`.) + const stripped = stripComments(SRC); + // The transaction lives in the helper... + expect(stripped).toMatch(/this\.sql\.begin\s*\(/); + // ...and the flag-off / non-alwaysTransaction path is a true + // pass-through on the shared pool — no per-read transaction hold. + expect(stripped).toMatch( + /if\s*\(!this\.rlsScopeBindingEnabled\s*&&\s*!opts\?\.alwaysTransaction\)\s*\{\s*return\s+await\s+callback\(this\.sql\);/, + ); }); test('both search methods use SET LOCAL for the timeout', () => { diff --git a/test/reindex-search-vector.serial.test.ts b/test/reindex-search-vector.serial.test.ts new file mode 100644 index 000000000..e8dd3a1a9 --- /dev/null +++ b/test/reindex-search-vector.serial.test.ts @@ -0,0 +1,152 @@ +import { describe, test, expect, beforeEach, afterEach } from 'bun:test'; +import type { BrainEngine } from '../src/core/engine.ts'; +import { runReindexSearchVector } from '../src/commands/reindex-search-vector.ts'; +import { resetFtsLanguageCache } from '../src/core/fts-language.ts'; + +const ENV_KEY = 'GBRAIN_FTS_LANGUAGE'; +const originalLang = process.env[ENV_KEY]; + +interface MockState { + calls: string[]; + rowsToReturn: { pages: number; chunks: number }; +} + +function makeMockEngine(state: MockState): BrainEngine { + return { + executeRaw: async (sql: string) => { + state.calls.push(sql); + // Inventory query — return the configured counts + if (sql.includes('SELECT') && sql.includes('FROM pages WHERE search_vector')) { + return [{ pages: state.rowsToReturn.pages, chunks: state.rowsToReturn.chunks }]; + } + return []; + }, + } as unknown as BrainEngine; +} + +beforeEach(() => { + delete process.env[ENV_KEY]; + resetFtsLanguageCache(); +}); + +afterEach(() => { + delete process.env[ENV_KEY]; + if (originalLang !== undefined) process.env[ENV_KEY] = originalLang; + resetFtsLanguageCache(); +}); + +describe('runReindexSearchVector', () => { + test('--dry-run does not issue any DDL or backfill SQL', async () => { + const state: MockState = { calls: [], rowsToReturn: { pages: 100, chunks: 500 } }; + const engine = makeMockEngine(state); + + process.env[ENV_KEY] = 'pt_br'; + resetFtsLanguageCache(); + + const result = await runReindexSearchVector(engine, { dryRun: true, json: true }); + + expect(result.status).toBe('dry_run'); + expect(result.language).toBe('pt_br'); + expect(result.pagesUpdated).toBe(100); + expect(result.chunksUpdated).toBe(500); + expect(result.triggersRecreated).toBe(0); + + // Only the inventory query — no CREATE OR REPLACE, no UPDATE. + expect(state.calls.length).toBe(1); + expect(state.calls[0]).toContain('SELECT'); + expect(state.calls[0]).not.toContain('CREATE OR REPLACE'); + expect(state.calls[0]).not.toContain('UPDATE'); + }); + + test('--yes recreates triggers + backfills with configured language', async () => { + const state: MockState = { calls: [], rowsToReturn: { pages: 50, chunks: 200 } }; + const engine = makeMockEngine(state); + + process.env[ENV_KEY] = 'pt_br'; + resetFtsLanguageCache(); + + const result = await runReindexSearchVector(engine, { yes: true, json: true }); + + expect(result.status).toBe('ok'); + expect(result.language).toBe('pt_br'); + expect(result.triggersRecreated).toBe(2); + expect(result.pagesUpdated).toBe(50); + expect(result.chunksUpdated).toBe(200); + + // 1 inventory + 2 CREATE + 2 backfill batches (mock returns no rows, so + // the keyset loop terminates after the first batch per table) = 5 calls + expect(state.calls.length).toBe(5); + expect(state.calls[1]).toContain('CREATE OR REPLACE FUNCTION update_page_search_vector'); + expect(state.calls[1]).toContain("to_tsvector('pt_br'"); + expect(state.calls[2]).toContain('CREATE OR REPLACE FUNCTION update_chunk_search_vector'); + expect(state.calls[2]).toContain("to_tsvector('pt_br'"); + expect(state.calls[3]).toMatch(/UPDATE pages/); + expect(state.calls[4]).toMatch(/UPDATE content_chunks/); + expect(state.calls[4]).toContain("to_tsvector('pt_br'"); + // v120/#1647 hardening must survive the CREATE OR REPLACE (which resets + // proconfig): both recreated bodies pin search_path. + expect(state.calls[1]).toContain('SET search_path = pg_catalog, public'); + expect(state.calls[2]).toContain('SET search_path = pg_catalog, public'); + }); + + test('default english language still recreates + backfills (no shortcut here)', async () => { + // Note: unlike the configurable_fts_language migration, the CLI command + // intentionally backfills even for english. The user explicitly asked for + // it, so we honor it. The migration skips backfill for english because it + // auto-runs on first apply. + const state: MockState = { calls: [], rowsToReturn: { pages: 10, chunks: 30 } }; + const engine = makeMockEngine(state); + + const result = await runReindexSearchVector(engine, { yes: true, json: true }); + + expect(result.status).toBe('ok'); + expect(result.language).toBe('english'); + expect(state.calls.length).toBe(5); + + // Trigger recreates (calls 1, 2) and chunks backfill (call 4) embed the + // language literal. Pages backfill (call 3) is UPDATE-to-self that + // re-fires the trigger, so the language literal lives in the trigger + // function body — not in the UPDATE statement. + expect(state.calls[1]).toContain("'english'"); + expect(state.calls[2]).toContain("'english'"); + expect(state.calls[3]).toMatch(/UPDATE pages/); + expect(state.calls[4]).toContain("'english'"); + }); + + test('SQL injection attempt falls back to english', async () => { + const state: MockState = { calls: [], rowsToReturn: { pages: 10, chunks: 30 } }; + const engine = makeMockEngine(state); + + process.env[ENV_KEY] = "english'; DROP TABLE pages; --"; + resetFtsLanguageCache(); + + const result = await runReindexSearchVector(engine, { yes: true, json: true }); + + expect(result.language).toBe('english'); + for (const sql of state.calls) { + expect(sql).not.toContain('DROP TABLE'); + } + }); + + test('empty inventory still completes successfully', async () => { + const state: MockState = { calls: [], rowsToReturn: { pages: 0, chunks: 0 } }; + const engine = makeMockEngine(state); + + const result = await runReindexSearchVector(engine, { yes: true, json: true }); + + expect(result.status).toBe('ok'); + expect(result.pagesUpdated).toBe(0); + expect(result.chunksUpdated).toBe(0); + expect(result.triggersRecreated).toBe(2); + }); + + test('result includes durationMs', async () => { + const state: MockState = { calls: [], rowsToReturn: { pages: 1, chunks: 1 } }; + const engine = makeMockEngine(state); + + const result = await runReindexSearchVector(engine, { yes: true, json: true }); + + expect(typeof result.durationMs).toBe('number'); + expect(result.durationMs).toBeGreaterThanOrEqual(0); + }); +}); diff --git a/test/sync-isSyncable-shape.test.ts b/test/sync-isSyncable-shape.test.ts index 44b29c2c8..7c715e590 100644 --- a/test/sync-isSyncable-shape.test.ts +++ b/test/sync-isSyncable-shape.test.ts @@ -28,7 +28,8 @@ describe('#1433 — isSyncable / unsyncableReason are duals of one classifier', { path: 'RESOLVER.md', expected: 'metafile', note: 'top-level master routing config (closes #345)' }, { path: 'brain/RESOLVER.md', expected: 'metafile', note: 'RESOLVER.md anywhere is metafile (closes #345)' }, { path: 'people/alice.txt', expected: 'strategy', note: '.txt rejected by markdown strategy' }, - { path: 'ops/scratch/note.md', expected: 'pruned-dir', note: 'ops/ is pruned' }, + { path: 'ops/scratch/note.md', expected: null, note: 'ops/ is ordinary content, not pruned (#2404)' }, + { path: 'vendor/pkg/note.md', expected: 'pruned-dir', note: 'vendor/ is pruned' }, { path: '.git/notes.md', expected: 'pruned-dir', note: 'hidden dir pruned' }, { path: 'node_modules/foo/README.md', expected: 'pruned-dir', note: 'node_modules pruned' }, ]; diff --git a/test/sync-ops-pages.serial.test.ts b/test/sync-ops-pages.serial.test.ts new file mode 100644 index 000000000..cda7bc12b --- /dev/null +++ b/test/sync-ops-pages.serial.test.ts @@ -0,0 +1,129 @@ +/** + * #2404 — `ops/` is ordinary content: sync imports `ops/*` files and never + * deletes `ops/*` pages. + * + * Bug class: `'ops'` was hardcoded in PRUNE_DIR_NAMES (a v0.2.0-era carve-out), + * so `classifySync` treated ANY path with an `ops` segment as 'pruned-dir': + * - committed `ops/*.md` files were never imported (even by `sync --full`); + * - a modified `ops/*` file fell into the unsyncableModified delete loop, + * whose #1433 guard only skipped 'metafile' — so put-created `ops/*` pages + * (e.g. the bundled daily-task-manager's canonical `ops/tasks`) were + * silently deleted on every sync. + * + * Fix: remove `'ops'` from PRUNE_DIR_NAMES, and harden the delete loop to also + * skip 'pruned-dir' classifications (a page under a genuinely-pruned dir can + * only exist via a deliberate put_page — never delete it on a file edit). + * + * Modeled on test/sync-metafile-skip.serial.test.ts (the #1433 iron rule). + */ + +import { describe, test, expect, beforeAll, afterAll, beforeEach, afterEach } from 'bun:test'; +import { mkdtempSync, writeFileSync, rmSync, mkdirSync } from 'fs'; +import { execSync } from 'child_process'; +import { tmpdir } from 'os'; +import { join } from 'path'; +import { PGLiteEngine } from '../src/core/pglite-engine.ts'; +import { resetPgliteState } from './helpers/reset-pglite.ts'; + +let engine: PGLiteEngine; +let repoPath: string; + +function gitInit(repo: string): void { + execSync('git init', { cwd: repo, stdio: 'pipe' }); + execSync('git config user.email "test@test.com"', { cwd: repo, stdio: 'pipe' }); + execSync('git config user.name "Test"', { cwd: repo, stdio: 'pipe' }); +} + +describe('#2404 — ops/ pages sync like any other content', () => { + beforeAll(async () => { + engine = new PGLiteEngine(); + await engine.connect({}); + await engine.initSchema(); + }, 60_000); + + afterAll(async () => { + if (engine) await engine.disconnect(); + }, 60_000); + + beforeEach(async () => { + await resetPgliteState(engine); + repoPath = mkdtempSync(join(tmpdir(), 'gbrain-ops-')); + gitInit(repoPath); + mkdirSync(join(repoPath, 'topics'), { recursive: true }); + writeFileSync(join(repoPath, 'topics/foo.md'), [ + '---', 'type: concept', 'title: Foo', '---', '', 'Baseline content.', + ].join('\n')); + execSync('git add -A && git commit -m "initial"', { cwd: repoPath, stdio: 'pipe' }); + }); + + afterEach(() => { + if (repoPath) rmSync(repoPath, { recursive: true, force: true }); + }); + + test('a committed ops/*.md file is imported by sync', async () => { + const { performSync } = await import('../src/commands/sync.ts'); + mkdirSync(join(repoPath, 'ops'), { recursive: true }); + writeFileSync(join(repoPath, 'ops/tasks.md'), [ + '---', 'type: concept', 'title: Tasks', '---', '', 'Open tasks live here.', + ].join('\n')); + execSync('git add -A && git commit -m "add ops/tasks"', { cwd: repoPath, stdio: 'pipe' }); + + const result = await performSync(engine, { repoPath, full: true, noPull: true, noEmbed: true }); + expect(['first_sync', 'synced']).toContain(result.status); + + // Pre-fix: ops/* was 'pruned-dir' → imported=0 for it, even on --full. + const page = await engine.getPage('ops/tasks'); + expect(page).not.toBeNull(); + expect(page?.compiled_truth).toContain('Open tasks'); + }, 60_000); + + test('an edited ops/*.md updates its page instead of deleting it', async () => { + const { performSync } = await import('../src/commands/sync.ts'); + mkdirSync(join(repoPath, 'ops'), { recursive: true }); + writeFileSync(join(repoPath, 'ops/tasks.md'), [ + '---', 'type: concept', 'title: Tasks', '---', '', 'v1', + ].join('\n')); + execSync('git add -A && git commit -m "add ops/tasks"', { cwd: repoPath, stdio: 'pipe' }); + await performSync(engine, { repoPath, full: true, noPull: true, noEmbed: true }); + + writeFileSync(join(repoPath, 'ops/tasks.md'), [ + '---', 'type: concept', 'title: Tasks', '---', '', 'v2 with a new task', + ].join('\n')); + execSync('git add -A && git commit -m "edit ops/tasks"', { cwd: repoPath, stdio: 'pipe' }); + + // Pre-fix: this incremental sync hit the unsyncableModified delete loop + // ("Deleted un-syncable page: ops/tasks" — the autopilot kill-loop). + const second = await performSync(engine, { repoPath, noPull: true, noEmbed: true }); + expect(['synced', 'up_to_date', 'first_sync']).toContain(second.status); + + const page = await engine.getPage('ops/tasks'); + expect(page).not.toBeNull(); + expect(page?.compiled_truth).toContain('v2'); + }, 60_000); + + test('hardening: a put-created page under a STILL-pruned dir survives a file edit (pruned-dir delete guard)', async () => { + const { performSync } = await import('../src/commands/sync.ts'); + // node_modules stays in PRUNE_DIR_NAMES. Commit a file there, then seed a + // same-path page via putPage — the deliberate-put precondition. + mkdirSync(join(repoPath, 'node_modules/pkg'), { recursive: true }); + writeFileSync(join(repoPath, 'node_modules/pkg/notes.md'), 'v1\n'); + execSync('git add -A -f && git commit -m "vendor file"', { cwd: repoPath, stdio: 'pipe' }); + await performSync(engine, { repoPath, full: true, noPull: true, noEmbed: true }); + + await engine.putPage('node_modules/pkg/notes', { + type: 'concept', + title: 'Deliberate put page', + compiled_truth: 'Created via put_page; must survive sync.', + timeline: '', + frontmatter: { type: 'concept' }, + }); + + writeFileSync(join(repoPath, 'node_modules/pkg/notes.md'), 'v2\n'); + execSync('git add -A -f && git commit -m "edit vendor file"', { cwd: repoPath, stdio: 'pipe' }); + await performSync(engine, { repoPath, noPull: true, noEmbed: true }); + + // Pre-fix: reason 'pruned-dir' was not guarded → page deleted. + const survivor = await engine.getPage('node_modules/pkg/notes'); + expect(survivor).not.toBeNull(); + }, 60_000); +}); diff --git a/test/sync-reconcile-db-only.serial.test.ts b/test/sync-reconcile-db-only.serial.test.ts new file mode 100644 index 000000000..a531a7dca --- /dev/null +++ b/test/sync-reconcile-db-only.serial.test.ts @@ -0,0 +1,142 @@ +/** + * #2426 (bug 3) — `sync --full` delete-reconcile preserves DB-only pages. + * + * Bug class: the full-sync reconcile soft-deleted ANY file-backed page whose + * `source_path` was absent from the working tree — including pages whose + * markdown was NEVER committed to git (write-through that never made it to + * the remote, then a fresh clone). "Absent from git" is the SYMPTOM of the + * missing write-through commit, not evidence the content is disposable; one + * production pass soft-deleted thousands of genuine pages this way. + * + * Fix: the reconcile partitions stale pages by git history — a path that ever + * appeared as an ADD was genuinely deleted (reconcile as before); a path with + * NO history is DB-only write-through: keep the page and re-export its + * markdown to the working tree so it's file-backed again. + * + * Builds on the #2828 mass-delete valve (this guard covers the below-valve + * cases the ratio check can't see). + */ + +import { describe, test, expect, beforeAll, afterAll, beforeEach, afterEach } from 'bun:test'; +import { mkdtempSync, writeFileSync, rmSync, mkdirSync, existsSync } from 'fs'; +import { execSync } from 'child_process'; +import { tmpdir } from 'os'; +import { join } from 'path'; +import { PGLiteEngine } from '../src/core/pglite-engine.ts'; +import { resetPgliteState } from './helpers/reset-pglite.ts'; +import { listEverCommittedPaths } from '../src/commands/sync.ts'; + +let engine: PGLiteEngine; +let repoPath: string; + +function gitInit(repo: string): void { + execSync('git init', { cwd: repo, stdio: 'pipe' }); + execSync('git config user.email "t@t.t"', { cwd: repo, stdio: 'pipe' }); + execSync('git config user.name "T"', { cwd: repo, stdio: 'pipe' }); +} + +describe('listEverCommittedPaths (#2426)', () => { + test('returns every path ever added, including later-deleted ones; null for non-git dirs', () => { + const repo = mkdtempSync(join(tmpdir(), 'gbrain-ecp-')); + try { + gitInit(repo); + writeFileSync(join(repo, 'kept.md'), 'kept\n'); + writeFileSync(join(repo, 'gone.md'), 'gone\n'); + execSync('git add -A && git commit -m add', { cwd: repo, stdio: 'pipe' }); + execSync('git rm -q gone.md && git commit -m rm', { cwd: repo, stdio: 'pipe' }); + + const set = listEverCommittedPaths(repo); + expect(set).not.toBeNull(); + expect(set!.has('kept.md')).toBe(true); + expect(set!.has('gone.md')).toBe(true); // deleted, but WAS committed + expect(set!.has('never-committed.md')).toBe(false); + + const plain = mkdtempSync(join(tmpdir(), 'gbrain-ecp-plain-')); + try { + expect(listEverCommittedPaths(plain)).toBeNull(); + } finally { + rmSync(plain, { recursive: true, force: true }); + } + } finally { + rmSync(repo, { recursive: true, force: true }); + } + }); +}); + +describe('#2426 — full-sync reconcile keeps never-committed (DB-only) pages', () => { + beforeAll(async () => { + engine = new PGLiteEngine(); + await engine.connect({}); + await engine.initSchema(); + }, 60_000); + + afterAll(async () => { + if (engine) await engine.disconnect(); + }, 60_000); + + beforeEach(async () => { + await resetPgliteState(engine); + repoPath = mkdtempSync(join(tmpdir(), 'gbrain-dbonly-')); + gitInit(repoPath); + mkdirSync(join(repoPath, 'topics'), { recursive: true }); + writeFileSync(join(repoPath, 'topics/keep.md'), [ + '---', 'type: concept', 'title: Keep', '---', '', 'still here', + ].join('\n')); + writeFileSync(join(repoPath, 'topics/gone.md'), [ + '---', 'type: concept', 'title: Gone', '---', '', 'will be git-rm-ed', + ].join('\n')); + execSync('git add -A && git commit -m initial', { cwd: repoPath, stdio: 'pipe' }); + }); + + afterEach(() => { + if (repoPath) rmSync(repoPath, { recursive: true, force: true }); + }); + + test('genuinely-deleted pages reconcile; never-committed pages are kept and re-exported', async () => { + const { performSync } = await import('../src/commands/sync.ts'); + + // Full sync #1: both file-backed pages land. + const first = await performSync(engine, { + repoPath, full: true, sourceId: 'default', noPull: true, noEmbed: true, + }); + expect(['first_sync', 'synced']).toContain(first.status); + expect(await engine.getPage('topics/keep')).not.toBeNull(); + expect(await engine.getPage('topics/gone')).not.toBeNull(); + + // A DB-only write-through casualty: the page row exists with a + // source_path, but its file was never committed and is absent from the + // clone (e.g. write-through was never pushed, then the repo was re-cloned). + await engine.putPage('memories/lost', { + type: 'concept', + title: 'Lost write-through', + compiled_truth: 'Years of content that must not be reconciled away.', + timeline: '', + frontmatter: { type: 'concept' }, + }); + await engine.executeRaw( + `UPDATE pages SET source_path = $1 WHERE slug = $2 AND source_id = $3`, + ['memories/lost.md', 'memories/lost', 'default'], + ); + + // A genuine deletion: topics/gone.md removed via git. + execSync('git rm -q topics/gone.md && git commit -m "rm gone"', { cwd: repoPath, stdio: 'pipe' }); + await engine.setConfig('sync.repo_path', repoPath); + + // Full sync #2 runs the delete-reconcile. + const second = await performSync(engine, { + repoPath, full: true, sourceId: 'default', noPull: true, noEmbed: true, + }); + expect(['first_sync', 'synced']).toContain(second.status); + + // The genuinely-deleted page is reconciled away… + expect(await engine.getPage('topics/gone')).toBeNull(); + // …the still-present page survives… + expect(await engine.getPage('topics/keep')).not.toBeNull(); + // …and the DB-only page is PRESERVED (pre-fix: soft-deleted here)… + const lost = await engine.getPage('memories/lost'); + expect(lost).not.toBeNull(); + expect(lost?.compiled_truth).toContain('must not be reconciled'); + // …and re-exported to the working tree so it is file-backed again. + expect(existsSync(join(repoPath, 'memories/lost.md'))).toBe(true); + }, 120_000); +}); diff --git a/test/sync-strategy.test.ts b/test/sync-strategy.test.ts index fb2fc893b..37477ea16 100644 --- a/test/sync-strategy.test.ts +++ b/test/sync-strategy.test.ts @@ -56,8 +56,10 @@ describe('isSyncable with strategy', () => { expect(isSyncable('.git/config.js', { strategy: 'code' })).toBe(false); // README.md is skipped under markdown expect(isSyncable('README.md', { strategy: 'markdown' })).toBe(false); - // ops/ directory always skipped - expect(isSyncable('ops/migrate.py', { strategy: 'code' })).toBe(false); + // ops/ is ordinary content — NOT skipped (#2404) + expect(isSyncable('ops/migrate.py', { strategy: 'code' })).toBe(true); + // vendored trees always skipped + expect(isSyncable('vendor/pkg/migrate.py', { strategy: 'code' })).toBe(false); // .raw/ sidecar always skipped expect(isSyncable('dir/.raw/code.ts', { strategy: 'code' })).toBe(false); }); diff --git a/test/sync.test.ts b/test/sync.test.ts index ae7901585..204408f02 100644 --- a/test/sync.test.ts +++ b/test/sync.test.ts @@ -95,9 +95,10 @@ describe('isSyncable', () => { expect(isSyncable('people/README.md')).toBe(false); }); - test('rejects ops/ directory', () => { - expect(isSyncable('ops/deploy-log.md')).toBe(false); - expect(isSyncable('ops/config.md')).toBe(false); + test('accepts ops/ — ordinary content directory, not pruned (#2404)', () => { + expect(isSyncable('ops/deploy-log.md')).toBe(true); + expect(isSyncable('ops/config.md')).toBe(true); + expect(isSyncable('ops/tasks.md')).toBe(true); }); // ──────────────────────────────────────────────────────────────── @@ -128,8 +129,15 @@ describe('pruneDir', () => { expect(pruneDir('.vscode')).toBe(false); }); - test('blocks ops (gbrain operational dir)', () => { - expect(pruneDir('ops')).toBe(false); + test('allows ops — ordinary content dir, not a vendor tree (#2404)', () => { + expect(pruneDir('ops')).toBe(true); + }); + + test('blocks vendored / generated trees', () => { + expect(pruneDir('vendor')).toBe(false); + expect(pruneDir('dist')).toBe(false); + expect(pruneDir('build')).toBe(false); + expect(pruneDir('venv')).toBe(false); }); test('blocks *.raw sidecar dirs (gbrain convention)', () => { diff --git a/test/write-through-commit.serial.test.ts b/test/write-through-commit.serial.test.ts new file mode 100644 index 000000000..23b985639 --- /dev/null +++ b/test/write-through-commit.serial.test.ts @@ -0,0 +1,115 @@ +/** + * #2426 (bug 1) — write-through reaches git on durability-hardened repos. + * + * Bug class: `put_page` / capture / enrichment wrote `.md` into + * `sync.repo_path` but NOTHING ever committed it. The post-commit hook only + * fires after a commit — and write-through never made one — so write-through + * content accumulated uncommitted forever: never pushed, `last_sync_at` + * frozen (HEAD never moved), and silently deleted by a later `sync --full` + * delete-reconcile. + * + * Fix: `writePageThrough` best-effort commits the artifact (path-limited) + * when the repo carries the gbrain durability post-commit hook (i.e. the + * user opted in via `gbrain sources harden`); the hook then background-pushes. + * Unhardened repos keep the old write-only behavior. + */ + +import { describe, test, expect, beforeAll, afterAll, beforeEach, afterEach } from 'bun:test'; +import { mkdtempSync, writeFileSync, rmSync, mkdirSync, chmodSync } from 'fs'; +import { execSync, execFileSync } from 'child_process'; +import { tmpdir } from 'os'; +import { join } from 'path'; +import { PGLiteEngine } from '../src/core/pglite-engine.ts'; +import { resetPgliteState } from './helpers/reset-pglite.ts'; +import { writePageThrough } from '../src/core/write-through.ts'; + +let engine: PGLiteEngine; +let repo: string; + +function git(cwd: string, ...args: string[]): string { + return execFileSync('git', ['-C', cwd, ...args], { + stdio: ['ignore', 'pipe', 'pipe'], encoding: 'utf-8', + }).trim(); +} + +/** Install a hook file carrying the gbrain durability banner (the detection + * key `isDurabilityHardened` looks for) with a no-op body so tests never + * attempt a real push. */ +function installFakeDurabilityHook(repoPath: string): void { + const hooksDir = join(repoPath, '.git', 'hooks'); + mkdirSync(hooksDir, { recursive: true }); + const hookPath = join(hooksDir, 'post-commit'); + writeFileSync(hookPath, [ + '#!/usr/bin/env bash', + '# gbrain brain-durability post-commit hook (v0.42.44+)', + 'exit 0', + '', + ].join('\n')); + chmodSync(hookPath, 0o755); +} + +async function seedPage(slug: string): Promise { + await engine.putPage(slug, { + type: 'concept', + title: 'Write-through page', + compiled_truth: 'Content that must reach git.', + timeline: '', + frontmatter: { type: 'concept' }, + }); +} + +describe('#2426 — writePageThrough auto-commit', () => { + beforeAll(async () => { + engine = new PGLiteEngine(); + await engine.connect({}); + await engine.initSchema(); + }, 60_000); + + afterAll(async () => { + if (engine) await engine.disconnect(); + }, 60_000); + + beforeEach(async () => { + await resetPgliteState(engine); + repo = mkdtempSync(join(tmpdir(), 'gbrain-wt-')); + execSync('git init', { cwd: repo, stdio: 'pipe' }); + execSync('git config user.email "t@t.t"', { cwd: repo, stdio: 'pipe' }); + execSync('git config user.name "T"', { cwd: repo, stdio: 'pipe' }); + writeFileSync(join(repo, 'seed.md'), 'seed\n'); + execSync('git add -A && git commit -m init', { cwd: repo, stdio: 'pipe' }); + await engine.setConfig('sync.repo_path', repo); + }); + + afterEach(() => { + if (repo) rmSync(repo, { recursive: true, force: true }); + }); + + test('on a hardened repo, the write-through artifact is committed (path-limited)', async () => { + installFakeDurabilityHook(repo); + // Unrelated dirty edit — must NOT be swept into the write-through commit. + writeFileSync(join(repo, 'seed.md'), 'dirty unrelated edit\n'); + + await seedPage('notes/hello'); + const result = await writePageThrough(engine, 'notes/hello'); + + expect(result.written).toBe(true); + expect(result.committed).toBe(true); + // The artifact is committed… + expect(git(repo, 'log', '-1', '--format=%s')).toBe('gbrain: write-through notes/hello'); + expect(git(repo, 'log', '-1', '--name-only', '--format=')).toBe('notes/hello.md'); + expect(git(repo, 'status', '--porcelain', 'notes/hello.md')).toBe(''); + // …and the unrelated edit stays uncommitted (explicit-path discipline). + expect(git(repo, 'status', '--porcelain', 'seed.md')).not.toBe(''); + }, 60_000); + + test('on an unhardened repo, the file is written but NOT committed (no behavior change)', async () => { + await seedPage('notes/plain'); + const result = await writePageThrough(engine, 'notes/plain'); + + expect(result.written).toBe(true); + expect(result.committed).toBeUndefined(); + // Untracked, uncommitted — the pre-existing contract. + expect(git(repo, 'status', '--porcelain', 'notes/plain.md')).toContain('?? notes/plain.md'); + expect(git(repo, 'log', '-1', '--format=%s')).toBe('init'); + }, 60_000); +});