mirror of
https://github.com/garrytan/gbrain.git
synced 2026-07-27 22:15:33 +00:00
Merge remote-tracking branch 'origin/master' into tmp-2942
# Conflicts: # docs/architecture/KEY_FILES.md
This commit is contained in:
Garry Tan
@@ -258,6 +258,24 @@ The whole loop is described in [`docs/architecture/topologies.md`](docs/architec
|
||||
|
||||
**Job queue (Minions).** BullMQ-shaped, Postgres-native job queue. Durable subagents (LLM tool loops that survive crashes via two-phase pending→done persistence), shell jobs with audit, child jobs with cascading timeouts, rate leases for outbound providers, attachments via S3/Supabase storage. Replaces "spawn subagent as fire-and-forget Promise" with something that recovers from anything.
|
||||
|
||||
**Non-English brains (FTS language config).** The Postgres full-text search tokenizer is configurable via `GBRAIN_FTS_LANGUAGE`. Defaults to `english`. Set it to any text-search configuration that exists in your Postgres instance:
|
||||
|
||||
```bash
|
||||
export GBRAIN_FTS_LANGUAGE=portuguese # uses built-in portuguese stemmer
|
||||
export GBRAIN_FTS_LANGUAGE=spanish # built-in spanish stemmer
|
||||
export GBRAIN_FTS_LANGUAGE=pt_br # custom config (e.g. unaccent + portuguese)
|
||||
```
|
||||
|
||||
List available configs: `psql -c "SELECT cfgname FROM pg_ts_config"`. Both the **query side** (`websearch_to_tsquery`) and the **write side** (the trigger functions that populate `pages.search_vector` and `content_chunks.search_vector`) honor `GBRAIN_FTS_LANGUAGE`. On first install (or upgrade), the `configurable_fts_language` schema migration reads the env var and creates trigger functions in the configured language; subsequent inserts/updates tokenize using that setting. To change language on a brain that has already run the migration, use the dedicated CLI command:
|
||||
|
||||
```bash
|
||||
export GBRAIN_FTS_LANGUAGE=portuguese
|
||||
gbrain reindex-search-vector --dry-run # preview row counts
|
||||
gbrain reindex-search-vector --yes # recreate triggers + backfill
|
||||
```
|
||||
|
||||
The command is idempotent (re-running with the same language is a no-op for vector content) and uses the same recreate-and-backfill primitives as the migration. For accent-insensitive Portuguese (`pt_br`), see [docs/guides/multi-language-fts.md](docs/guides/multi-language-fts.md) for the `unaccent` + portuguese stemmer recipe.
|
||||
|
||||
**43 curated skills.** Routing lives in [`skills/RESOLVER.md`](skills/RESOLVER.md). Covers signal capture, ingest (idea / media / meeting), enrichment, querying, brain ops, citation fixing, daily task management, cron scheduling, reports, voice, soul audit, skill creation, eval framework, and migrations. Skills are markdown files (tool-agnostic), packaged as a single skillpack the installer drops into your agent workspace.
|
||||
|
||||
**Eval framework.** `gbrain eval longmemeval` runs the public [LongMemEval](https://huggingface.co/datasets/xiaowu0162/longmemeval) benchmark against your hybrid retrieval. `gbrain eval export` + `gbrain eval replay` capture real queries and replay them against code changes (set `GBRAIN_CONTRIBUTOR_MODE=1`). `gbrain eval cross-modal` cross-checks an output against the task using three different-provider frontier models. `gbrain eval retrieval-quality` runs NamedThingBench, which hard-gates the named-thing retrieval families (title-substring, alias-synonym, generic-to-named, multi-chunk-dilution) so a regression in "find the page this query names" fails CI loudly. Full methodology in [`docs/eval/SEARCH_MODE_METHODOLOGY.md`](docs/eval/SEARCH_MODE_METHODOLOGY.md).
|
||||
|
||||
@@ -148,6 +148,51 @@ RRF fusion, multi-query expansion, and 4-layer dedup are engine-agnostic. They o
|
||||
|
||||
**Why not self-hosted for v0:** The brain should be infrastructure agents use, not something you maintain. Self-hosted Postgres with Docker is a welcome community PR, but v0 optimizes for zero ops.
|
||||
|
||||
### Opt-in RLS source-scope binding (`GBRAIN_RLS_SCOPE_BINDING`)
|
||||
|
||||
Defense-in-depth layer for Postgres deployments that want the database itself
|
||||
to enforce source isolation, in addition to the mandatory app-layer filters
|
||||
(`sourceScopeOpts` — layer 1, always on).
|
||||
|
||||
**Mechanism.** With `GBRAIN_RLS_SCOPE_BINDING=1` (or `true`), the engine's
|
||||
source-scoped read methods wrap their queries in a transaction that first runs
|
||||
`SELECT set_config('app.scopes', $1, true)` — the value is a bound parameter
|
||||
(federated `sourceIds` CSV > scalar `sourceId` > `'*'` for unscoped internal
|
||||
reads), transaction-local (equivalent to `SET LOCAL`, which itself can't take
|
||||
bound params). An RLS policy can then filter rows by
|
||||
`current_setting('app.scopes', true)`.
|
||||
|
||||
**Default off.** With the env var unset, reads call through on the shared pool
|
||||
exactly as before — no per-read transaction, no pool-slot hold (the search
|
||||
methods keep the transaction they always had for their `SET LOCAL
|
||||
statement_timeout`). Existing operators see zero behavior change.
|
||||
|
||||
**Enabling it** (operator-managed SQL; gbrain ships no DDL for this):
|
||||
|
||||
```sql
|
||||
ALTER TABLE pages ENABLE ROW LEVEL SECURITY;
|
||||
CREATE POLICY pages_scope_filter ON pages
|
||||
USING (current_setting('app.scopes', true) = '*'
|
||||
OR source_id = ANY(string_to_array(current_setting('app.scopes', true), ',')));
|
||||
|
||||
-- Required: connections that don't run through the scoped read helper
|
||||
-- (admin, autopilot, cycle, writes) must default to unscoped, or they
|
||||
-- see zero rows once the policy exists:
|
||||
ALTER ROLE <runtime-role> SET app.scopes = '*';
|
||||
|
||||
-- If the runtime role OWNS the table, RLS is skipped for it unless forced:
|
||||
ALTER TABLE pages FORCE ROW LEVEL SECURITY;
|
||||
```
|
||||
|
||||
Safe to enable in either order: the env var without a policy is a no-op
|
||||
setting; a policy without the env var is enforced only via the role default.
|
||||
|
||||
**Honest caveat:** only read paths routed through the scoped helper carry a
|
||||
per-request scope binding — unwrapped paths (writes, admin/maintenance reads)
|
||||
run under the role default and are not backstopped per caller. This is layer 2;
|
||||
the app-layer source filters remain layer 1 and stay mandatory. Behavioral pins
|
||||
live in `test/postgres-engine-rls-scope.test.ts`.
|
||||
|
||||
## PGLiteEngine (v0.7, ships)
|
||||
|
||||
**Dependencies:** `@electric-sql/pglite` (v0.4.4+)
|
||||
|
||||
File diff suppressed because one or more lines are too long
@@ -0,0 +1,97 @@
|
||||
# Multi-language full-text search
|
||||
|
||||
GBrain's keyword search arm uses Postgres full-text search (tsvector/tsquery).
|
||||
The tokenizer language is configurable via the `GBRAIN_FTS_LANGUAGE`
|
||||
environment variable. Default: `english`.
|
||||
|
||||
## How it works
|
||||
|
||||
Postgres text-search configurations control stemming and stop-word removal.
|
||||
`GBRAIN_FTS_LANGUAGE` is read by `src/core/fts-language.ts` and applied on
|
||||
both sides of the search:
|
||||
|
||||
- **Query side** — `websearch_to_tsquery('<lang>', $query)` in both engines
|
||||
(Postgres and PGLite).
|
||||
- **Write side** — the `update_page_search_vector` and
|
||||
`update_chunk_search_vector` trigger functions that populate
|
||||
`pages.search_vector` and `content_chunks.search_vector`.
|
||||
|
||||
The value is validated against `/^[a-z][a-z0-9_]*$/` before it is ever
|
||||
interpolated into SQL (tsvector functions don't accept parameterized config
|
||||
names). Invalid values fall back to `english` with a warning.
|
||||
|
||||
## Built-in languages
|
||||
|
||||
Set the env var to any configuration your Postgres instance ships:
|
||||
|
||||
```bash
|
||||
export GBRAIN_FTS_LANGUAGE=portuguese
|
||||
export GBRAIN_FTS_LANGUAGE=spanish
|
||||
export GBRAIN_FTS_LANGUAGE=german
|
||||
```
|
||||
|
||||
List what's available:
|
||||
|
||||
```sql
|
||||
SELECT cfgname FROM pg_ts_config;
|
||||
```
|
||||
|
||||
PGLite (the embedded default engine) ships the same built-in snowball
|
||||
configurations as stock Postgres.
|
||||
|
||||
## First install vs. changing language later
|
||||
|
||||
On first install (or upgrade), the `configurable_fts_language` schema
|
||||
migration reads `GBRAIN_FTS_LANGUAGE` and stamps the trigger functions with
|
||||
that language. After the migration has run, changing the env var alone does
|
||||
NOT retokenize existing rows — the migration shows as applied and is skipped.
|
||||
Use the explicit command:
|
||||
|
||||
```bash
|
||||
export GBRAIN_FTS_LANGUAGE=portuguese
|
||||
gbrain reindex-search-vector --dry-run # preview: language + row counts
|
||||
gbrain reindex-search-vector --yes # recreate triggers + backfill
|
||||
```
|
||||
|
||||
The command recreates both trigger functions under the new language and
|
||||
backfills every existing `pages` and `content_chunks` row in batches,
|
||||
streaming progress to stderr. It is idempotent: re-running with the same
|
||||
language produces identical vectors. `--json` prints a machine-readable
|
||||
result envelope but still requires `--yes` (or an interactive confirm).
|
||||
|
||||
## Recipe: accent-insensitive Portuguese (`pt_br`)
|
||||
|
||||
Brazilian Portuguese content often mixes accented and unaccented spellings
|
||||
("São Paulo" vs "Sao Paulo"). Build a custom config that folds accents via
|
||||
the `unaccent` extension, then stems with the portuguese snowball dictionary:
|
||||
|
||||
```sql
|
||||
CREATE EXTENSION IF NOT EXISTS unaccent;
|
||||
|
||||
CREATE TEXT SEARCH CONFIGURATION pt_br (COPY = portuguese);
|
||||
|
||||
ALTER TEXT SEARCH CONFIGURATION pt_br
|
||||
ALTER MAPPING FOR hword, hword_part, word
|
||||
WITH unaccent, portuguese_stem;
|
||||
```
|
||||
|
||||
Then point GBrain at it:
|
||||
|
||||
```bash
|
||||
export GBRAIN_FTS_LANGUAGE=pt_br
|
||||
gbrain reindex-search-vector --yes
|
||||
```
|
||||
|
||||
Note: custom configurations require a real Postgres instance (e.g. the
|
||||
Supabase engine). The config must exist BEFORE the migration or the reindex
|
||||
command runs, or Postgres will reject the trigger recreation with
|
||||
`text search configuration "pt_br" does not exist`.
|
||||
|
||||
## Caveats
|
||||
|
||||
- One language per brain: the setting is global to the database, not
|
||||
per-source. Mixed-language brains should pick the dominant language (the
|
||||
vector-search arm is language-agnostic and covers the rest).
|
||||
- Keep `GBRAIN_FTS_LANGUAGE` set consistently in every environment that
|
||||
writes to the brain (CLI shells, MCP server, cron jobs) — a writer without
|
||||
the env var tokenizes new rows in `english` until the next reindex.
|
||||
@@ -1752,6 +1752,24 @@ The whole loop is described in [`docs/architecture/topologies.md`](docs/architec
|
||||
|
||||
**Job queue (Minions).** BullMQ-shaped, Postgres-native job queue. Durable subagents (LLM tool loops that survive crashes via two-phase pending→done persistence), shell jobs with audit, child jobs with cascading timeouts, rate leases for outbound providers, attachments via S3/Supabase storage. Replaces "spawn subagent as fire-and-forget Promise" with something that recovers from anything.
|
||||
|
||||
**Non-English brains (FTS language config).** The Postgres full-text search tokenizer is configurable via `GBRAIN_FTS_LANGUAGE`. Defaults to `english`. Set it to any text-search configuration that exists in your Postgres instance:
|
||||
|
||||
```bash
|
||||
export GBRAIN_FTS_LANGUAGE=portuguese # uses built-in portuguese stemmer
|
||||
export GBRAIN_FTS_LANGUAGE=spanish # built-in spanish stemmer
|
||||
export GBRAIN_FTS_LANGUAGE=pt_br # custom config (e.g. unaccent + portuguese)
|
||||
```
|
||||
|
||||
List available configs: `psql -c "SELECT cfgname FROM pg_ts_config"`. Both the **query side** (`websearch_to_tsquery`) and the **write side** (the trigger functions that populate `pages.search_vector` and `content_chunks.search_vector`) honor `GBRAIN_FTS_LANGUAGE`. On first install (or upgrade), the `configurable_fts_language` schema migration reads the env var and creates trigger functions in the configured language; subsequent inserts/updates tokenize using that setting. To change language on a brain that has already run the migration, use the dedicated CLI command:
|
||||
|
||||
```bash
|
||||
export GBRAIN_FTS_LANGUAGE=portuguese
|
||||
gbrain reindex-search-vector --dry-run # preview row counts
|
||||
gbrain reindex-search-vector --yes # recreate triggers + backfill
|
||||
```
|
||||
|
||||
The command is idempotent (re-running with the same language is a no-op for vector content) and uses the same recreate-and-backfill primitives as the migration. For accent-insensitive Portuguese (`pt_br`), see [docs/guides/multi-language-fts.md](docs/guides/multi-language-fts.md) for the `unaccent` + portuguese stemmer recipe.
|
||||
|
||||
**43 curated skills.** Routing lives in [`skills/RESOLVER.md`](skills/RESOLVER.md). Covers signal capture, ingest (idea / media / meeting), enrichment, querying, brain ops, citation fixing, daily task management, cron scheduling, reports, voice, soul audit, skill creation, eval framework, and migrations. Skills are markdown files (tool-agnostic), packaged as a single skillpack the installer drops into your agent workspace.
|
||||
|
||||
**Eval framework.** `gbrain eval longmemeval` runs the public [LongMemEval](https://huggingface.co/datasets/xiaowu0162/longmemeval) benchmark against your hybrid retrieval. `gbrain eval export` + `gbrain eval replay` capture real queries and replay them against code changes (set `GBRAIN_CONTRIBUTOR_MODE=1`). `gbrain eval cross-modal` cross-checks an output against the task using three different-provider frontier models. `gbrain eval retrieval-quality` runs NamedThingBench, which hard-gates the named-thing retrieval families (title-substring, alias-synonym, generic-to-named, multi-chunk-dilution) so a regression in "find the page this query names" fails CI loudly. Full methodology in [`docs/eval/SEARCH_MODE_METHODOLOGY.md`](docs/eval/SEARCH_MODE_METHODOLOGY.md).
|
||||
@@ -2095,6 +2113,51 @@ RRF fusion, multi-query expansion, and 4-layer dedup are engine-agnostic. They o
|
||||
|
||||
**Why not self-hosted for v0:** The brain should be infrastructure agents use, not something you maintain. Self-hosted Postgres with Docker is a welcome community PR, but v0 optimizes for zero ops.
|
||||
|
||||
### Opt-in RLS source-scope binding (`GBRAIN_RLS_SCOPE_BINDING`)
|
||||
|
||||
Defense-in-depth layer for Postgres deployments that want the database itself
|
||||
to enforce source isolation, in addition to the mandatory app-layer filters
|
||||
(`sourceScopeOpts` — layer 1, always on).
|
||||
|
||||
**Mechanism.** With `GBRAIN_RLS_SCOPE_BINDING=1` (or `true`), the engine's
|
||||
source-scoped read methods wrap their queries in a transaction that first runs
|
||||
`SELECT set_config('app.scopes', $1, true)` — the value is a bound parameter
|
||||
(federated `sourceIds` CSV > scalar `sourceId` > `'*'` for unscoped internal
|
||||
reads), transaction-local (equivalent to `SET LOCAL`, which itself can't take
|
||||
bound params). An RLS policy can then filter rows by
|
||||
`current_setting('app.scopes', true)`.
|
||||
|
||||
**Default off.** With the env var unset, reads call through on the shared pool
|
||||
exactly as before — no per-read transaction, no pool-slot hold (the search
|
||||
methods keep the transaction they always had for their `SET LOCAL
|
||||
statement_timeout`). Existing operators see zero behavior change.
|
||||
|
||||
**Enabling it** (operator-managed SQL; gbrain ships no DDL for this):
|
||||
|
||||
```sql
|
||||
ALTER TABLE pages ENABLE ROW LEVEL SECURITY;
|
||||
CREATE POLICY pages_scope_filter ON pages
|
||||
USING (current_setting('app.scopes', true) = '*'
|
||||
OR source_id = ANY(string_to_array(current_setting('app.scopes', true), ',')));
|
||||
|
||||
-- Required: connections that don't run through the scoped read helper
|
||||
-- (admin, autopilot, cycle, writes) must default to unscoped, or they
|
||||
-- see zero rows once the policy exists:
|
||||
ALTER ROLE <runtime-role> SET app.scopes = '*';
|
||||
|
||||
-- If the runtime role OWNS the table, RLS is skipped for it unless forced:
|
||||
ALTER TABLE pages FORCE ROW LEVEL SECURITY;
|
||||
```
|
||||
|
||||
Safe to enable in either order: the env var without a policy is a no-op
|
||||
setting; a policy without the env var is enforced only via the role default.
|
||||
|
||||
**Honest caveat:** only read paths routed through the scoped helper carry a
|
||||
per-request scope binding — unwrapped paths (writes, admin/maintenance reads)
|
||||
run under the role default and are not backstopped per caller. This is layer 2;
|
||||
the app-layer source filters remain layer 1 and stay mandatory. Behavioral pins
|
||||
live in `test/postgres-engine-rls-scope.test.ts`.
|
||||
|
||||
## PGLiteEngine (v0.7, ships)
|
||||
|
||||
**Dependencies:** `@electric-sql/pglite` (v0.4.4+)
|
||||
|
||||
+13
-1
@@ -54,7 +54,7 @@ export function bigintToStringReplacer(_key: string, value: unknown): unknown {
|
||||
}
|
||||
|
||||
// CLI-only commands that bypass the operation layer
|
||||
export const CLI_ONLY = new Set(['init', 'reinit-pglite', 'upgrade', 'post-upgrade', 'check-update', 'integrations', 'publish', 'check-backlinks', 'lint', 'report', 'import', 'export', 'files', 'embed', 'serve', 'call', 'config', 'doctor', 'migrate', 'eval', 'sync', 'extract', 'extract-conversation-facts', 'enrich', 'features', 'autopilot', 'graph-query', 'jobs', 'agent', 'apply-migrations', 'skillpack-check', 'skillpack', 'resolvers', 'integrity', 'repair-jsonb', 'orphans', 'sources', 'mounts', 'dream', 'check-resolvable', 'routing-eval', 'skillify', 'smoke-test', 'providers', 'storage', 'repos', 'code-def', 'code-refs', 'reindex', 'reindex-code', 'reindex-frontmatter', 'code-callers', 'code-callees', 'frontmatter', 'auth', 'friction', 'claw-test', 'book-mirror', 'takes', 'think', 'salience', 'anomalies', 'calibration', 'transcripts', 'models', 'remote', 'recall', 'forget', 'edges-backfill', 'cache', 'ze-switch', 'founder', 'brainstorm', 'lsd', 'schema', 'capture', 'onboard', 'conversation-parser', 'status', 'connect', 'skillopt', 'quarantine', 'self-upgrade', 'advisor', 'watch']);
|
||||
export const CLI_ONLY = new Set(['init', 'reinit-pglite', 'upgrade', 'post-upgrade', 'check-update', 'integrations', 'publish', 'check-backlinks', 'lint', 'report', 'import', 'export', 'files', 'embed', 'serve', 'call', 'config', 'doctor', 'migrate', 'eval', 'sync', 'extract', 'extract-conversation-facts', 'enrich', 'features', 'autopilot', 'graph-query', 'jobs', 'agent', 'apply-migrations', 'skillpack-check', 'skillpack', 'resolvers', 'integrity', 'repair-jsonb', 'orphans', 'sources', 'mounts', 'dream', 'check-resolvable', 'routing-eval', 'skillify', 'smoke-test', 'providers', 'storage', 'repos', 'code-def', 'code-refs', 'reindex', 'reindex-code', 'reindex-frontmatter', 'code-callers', 'code-callees', 'frontmatter', 'auth', 'friction', 'claw-test', 'book-mirror', 'takes', 'think', 'salience', 'anomalies', 'calibration', 'transcripts', 'models', 'remote', 'recall', 'forget', 'edges-backfill', 'cache', 'ze-switch', 'founder', 'brainstorm', 'lsd', 'schema', 'capture', 'onboard', 'conversation-parser', 'status', 'connect', 'skillopt', 'quarantine', 'self-upgrade', 'advisor', 'watch', 'reindex-search-vector']);
|
||||
// CLI-only commands whose handlers print their own --help text. These are
|
||||
// excluded from the generic short-circuit so detailed per-command and
|
||||
// per-subcommand usage stays reachable.
|
||||
@@ -2001,6 +2001,15 @@ async function handleCliOnly(command: string, args: string[]) {
|
||||
await runReindexCodeCli(engine, args);
|
||||
break;
|
||||
}
|
||||
case 'reindex-search-vector': {
|
||||
// Explicit recreate of FTS trigger functions + batched backfill,
|
||||
// honoring GBRAIN_FTS_LANGUAGE. Use after changing the language
|
||||
// env var on a brain that already ran the configurable_fts_language
|
||||
// migration.
|
||||
const { runReindexSearchVectorCli } = await import('./commands/reindex-search-vector.ts');
|
||||
await runReindexSearchVectorCli(engine, args);
|
||||
break;
|
||||
}
|
||||
case 'reindex-frontmatter': {
|
||||
// v0.29.1: recovery / explicit-rebuild path for pages.effective_date.
|
||||
// Mirror of reindex-code shape. Wraps the shared library function in
|
||||
@@ -2336,6 +2345,9 @@ CODE INDEXING (v0.19.0 / v0.20.0 Cathedral II)
|
||||
query <q> --symbol-kind <k> Filter to symbol type (function|class|method|...) (v0.20.0)
|
||||
reconcile-links [--dry-run] Batch-recompute doc↔impl edges (v0.20.0)
|
||||
reindex-code [--source id] [--yes] Explicit code-page reindex (v0.20.0)
|
||||
reindex-search-vector [--dry-run] [--yes] [--json]
|
||||
Recreate FTS triggers + backfill under
|
||||
$GBRAIN_FTS_LANGUAGE (default 'english')
|
||||
sync --strategy code Sync code files into the brain
|
||||
|
||||
JOBS (Minions)
|
||||
|
||||
@@ -188,7 +188,7 @@ export function walkMarkdownFiles(dir: string): { path: string; relPath: string
|
||||
// Pre-fix, this walker had only an ad-hoc dot-prefix exclusion and didn't
|
||||
// call isSyncable at all — so it descended into `node_modules/`, emitted
|
||||
// markdown files from there, AND ignored the canonical exclusion list
|
||||
// (`.raw/`, `ops/`, README.md, etc.). Now: pruneDir skips entire vendor
|
||||
// (`.raw/`, README.md, etc.). Now: pruneDir skips entire vendor
|
||||
// subtrees before recursion (saving IO), and isSyncable filters the emit
|
||||
// set against the canonical markdown-strategy rules.
|
||||
const files: { path: string; relPath: string }[] = [];
|
||||
|
||||
+12
-1
@@ -569,10 +569,21 @@ function isCollectibleForWalker(
|
||||
strategy: SyncStrategy,
|
||||
multimodalOn: boolean,
|
||||
): boolean {
|
||||
// #2607: apply the SAME segment-level prune gate as incremental sync's
|
||||
// `classifySync` (core/sync.ts). The FS walk below prunes at descent time,
|
||||
// but the git fast path enumerates via `git ls-files` and historically
|
||||
// filtered only by extension — so `sync --full` imported (and resurrected
|
||||
// previously-deleted) pages under dot-dirs / vendored trees that incremental
|
||||
// sync excludes. Full and incremental must agree on the exclusion set.
|
||||
// (In the FS-walk route `path` is a basename, so this is the same dot-file
|
||||
// check pruneDir already applied there — no behavior change on that route.)
|
||||
const segments = path.split('/');
|
||||
if (segments.some((seg) => !pruneDir(seg))) return false;
|
||||
|
||||
// Metafiles are directory scaffolding (READMEs / index / log / schema /
|
||||
// resolver), not typed brain pages — same exclusion `sync`'s `isSyncable`
|
||||
// applies. Guards both the FS-walk and the git-fast-path collection routes.
|
||||
const basename = path.split('/').pop() || '';
|
||||
const basename = segments[segments.length - 1] || '';
|
||||
if ((SYNC_SKIP_FILES as readonly string[]).includes(basename)) return false;
|
||||
|
||||
switch (strategy) {
|
||||
|
||||
@@ -0,0 +1,277 @@
|
||||
/**
|
||||
* `gbrain reindex-search-vector` — recreate FTS trigger functions and
|
||||
* backfill existing rows under the language configured via
|
||||
* GBRAIN_FTS_LANGUAGE.
|
||||
*
|
||||
* Why this command exists: schema migration v123 (configurable_fts_language)
|
||||
* stamps the trigger functions with the configured language at first apply.
|
||||
* After that, changing the env var has no effect on the write side because
|
||||
* v123 already shows as "applied" — the migrations runner will skip it.
|
||||
* This command is the documented escape hatch: it re-runs the same
|
||||
* recreate-and-backfill logic v123 uses, gated on an explicit user
|
||||
* action so the operation is intentional and visible (writes touch
|
||||
* every row in pages and content_chunks).
|
||||
*
|
||||
* Idempotent: running twice with the same GBRAIN_FTS_LANGUAGE produces
|
||||
* the same trigger function bodies and the same tokenized vectors.
|
||||
*
|
||||
* Flags:
|
||||
* --dry-run Show what would happen, exit 0 without touching DB.
|
||||
* --yes Skip interactive [y/N]. Required for non-TTY (including --json).
|
||||
* --json Machine-readable result envelope. Does NOT imply --yes.
|
||||
*
|
||||
* Backfill runs in id-keyset batches (BACKFILL_BATCH_SIZE rows per UPDATE)
|
||||
* so a large brain never holds one giant row lock, and streams progress
|
||||
* through the shared reporter (stderr; stdout stays clean for --json).
|
||||
*
|
||||
* Cost: trigger recreate is sub-millisecond. Backfill is one tsvector
|
||||
* rebuild per page + per chunk. On a 20K-page brain with 80K chunks,
|
||||
* expect ~5-15s depending on Postgres CPU and content size.
|
||||
*/
|
||||
|
||||
import type { BrainEngine } from '../core/engine.ts';
|
||||
import { getFtsLanguage } from '../core/fts-language.ts';
|
||||
import { createInterface } from 'readline';
|
||||
import { createProgress } from '../core/progress.ts';
|
||||
import { getCliOptions, cliOptsToProgressOptions } from '../core/cli-options.ts';
|
||||
|
||||
export interface ReindexSearchVectorOpts {
|
||||
dryRun?: boolean;
|
||||
yes?: boolean;
|
||||
json?: boolean;
|
||||
}
|
||||
|
||||
export interface ReindexSearchVectorResult {
|
||||
status: 'ok' | 'dry_run' | 'cancelled';
|
||||
language: string;
|
||||
pagesUpdated: number;
|
||||
chunksUpdated: number;
|
||||
triggersRecreated: number;
|
||||
durationMs: number;
|
||||
}
|
||||
|
||||
interface CountRow {
|
||||
pages: number;
|
||||
chunks: number;
|
||||
}
|
||||
|
||||
/** Rows per backfill UPDATE. Keyset-batched so one statement never locks the whole table. */
|
||||
export const BACKFILL_BATCH_SIZE = 5000;
|
||||
|
||||
/**
|
||||
* Keyset-batched UPDATE: applies `setClause` to `table` rows where
|
||||
* search_vector IS NOT NULL, BACKFILL_BATCH_SIZE ids at a time, ticking
|
||||
* the shared progress reporter after each batch. Terminates when a batch
|
||||
* returns fewer rows than the batch size (or none).
|
||||
*/
|
||||
async function batchedBackfill(
|
||||
engine: BrainEngine,
|
||||
table: 'pages' | 'content_chunks',
|
||||
setClause: string,
|
||||
tick: (n: number) => void
|
||||
): Promise<void> {
|
||||
let cursor = 0;
|
||||
for (;;) {
|
||||
const rows = await engine.executeRaw<{ id: number }>(`
|
||||
UPDATE ${table} SET ${setClause}
|
||||
WHERE id IN (
|
||||
SELECT id FROM ${table}
|
||||
WHERE search_vector IS NOT NULL AND id > ${cursor}
|
||||
ORDER BY id
|
||||
LIMIT ${BACKFILL_BATCH_SIZE}
|
||||
)
|
||||
RETURNING id
|
||||
`);
|
||||
if (rows.length === 0) break;
|
||||
tick(rows.length);
|
||||
cursor = rows.reduce((m, r) => Math.max(m, Number(r.id)), cursor);
|
||||
if (rows.length < BACKFILL_BATCH_SIZE) break;
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Programmatic entrypoint — takes a typed opts object. Used by tests and
|
||||
* future internal callers. The CLI wrapper is `runReindexSearchVectorCli`
|
||||
* defined at the bottom of this file.
|
||||
*/
|
||||
export async function runReindexSearchVector(
|
||||
engine: BrainEngine,
|
||||
opts: ReindexSearchVectorOpts
|
||||
): Promise<ReindexSearchVectorResult> {
|
||||
const lang = getFtsLanguage();
|
||||
const startedAt = Date.now();
|
||||
|
||||
// Inventory: how many rows will the backfill touch?
|
||||
const counts = await engine.executeRaw<CountRow>(
|
||||
`SELECT
|
||||
(SELECT COUNT(*)::int FROM pages WHERE search_vector IS NOT NULL) AS pages,
|
||||
(SELECT COUNT(*)::int FROM content_chunks WHERE search_vector IS NOT NULL) AS chunks`
|
||||
);
|
||||
const pagesCount = counts[0]?.pages ?? 0;
|
||||
const chunksCount = counts[0]?.chunks ?? 0;
|
||||
|
||||
if (opts.dryRun) {
|
||||
const result: ReindexSearchVectorResult = {
|
||||
status: 'dry_run',
|
||||
language: lang,
|
||||
pagesUpdated: pagesCount,
|
||||
chunksUpdated: chunksCount,
|
||||
triggersRecreated: 0,
|
||||
durationMs: Date.now() - startedAt,
|
||||
};
|
||||
if (opts.json) {
|
||||
console.log(JSON.stringify(result, null, 2));
|
||||
} else {
|
||||
console.log(`[dry-run] Would recreate 2 trigger functions with language='${lang}'`);
|
||||
console.log(`[dry-run] Would backfill ${pagesCount} pages + ${chunksCount} chunks`);
|
||||
console.log(`[dry-run] Skipping all DB writes. Pass --yes to apply.`);
|
||||
}
|
||||
return result;
|
||||
}
|
||||
|
||||
// Confirm unless --yes. --json does NOT bypass the gate — a machine
|
||||
// caller must pass --yes explicitly (mirrors reindex-code, #1784).
|
||||
if (!opts.yes) {
|
||||
if (!process.stdin.isTTY) {
|
||||
if (opts.json) {
|
||||
console.log(JSON.stringify({
|
||||
error: {
|
||||
class: 'ConfirmationRequired',
|
||||
code: 'reindex_requires_yes',
|
||||
message: `Refusing to recreate FTS triggers + backfill ${pagesCount} pages + ${chunksCount} chunks without --yes in a non-TTY environment.`,
|
||||
hint: 'Pass --yes to proceed, or --dry-run to preview.',
|
||||
},
|
||||
language: lang,
|
||||
pages: pagesCount,
|
||||
chunks: chunksCount,
|
||||
}));
|
||||
} else {
|
||||
console.error('Refusing to run without --yes in non-TTY environment.');
|
||||
}
|
||||
process.exit(2);
|
||||
}
|
||||
|
||||
const rl = createInterface({ input: process.stdin, output: process.stdout });
|
||||
const answer = await new Promise<string>(resolve => {
|
||||
rl.question(
|
||||
`Recreate FTS triggers with language='${lang}' and backfill ${pagesCount} pages + ${chunksCount} chunks? [y/N]: `,
|
||||
resolve
|
||||
);
|
||||
});
|
||||
rl.close();
|
||||
|
||||
if (!/^y(es)?$/i.test(answer.trim())) {
|
||||
const result: ReindexSearchVectorResult = {
|
||||
status: 'cancelled',
|
||||
language: lang,
|
||||
pagesUpdated: 0,
|
||||
chunksUpdated: 0,
|
||||
triggersRecreated: 0,
|
||||
durationMs: Date.now() - startedAt,
|
||||
};
|
||||
if (opts.json) {
|
||||
console.log(JSON.stringify(result, null, 2));
|
||||
} else {
|
||||
console.log('Cancelled.');
|
||||
}
|
||||
return result;
|
||||
}
|
||||
}
|
||||
|
||||
// Recreate trigger functions. The strings are intentionally identical to
|
||||
// the v123 migration body — keeping them in lockstep is the contract.
|
||||
// `SET search_path = pg_catalog, public` mirrors the v120/#1647 hardening:
|
||||
// CREATE OR REPLACE resets proconfig, so omitting it here would strip the
|
||||
// hardening from every brain that runs this command.
|
||||
const recreatePagesFn = `
|
||||
CREATE OR REPLACE FUNCTION update_page_search_vector() RETURNS trigger SET search_path = pg_catalog, public AS $fn$
|
||||
DECLARE
|
||||
timeline_text TEXT;
|
||||
BEGIN
|
||||
SELECT coalesce(string_agg(summary || ' ' || detail, ' '), '')
|
||||
INTO timeline_text
|
||||
FROM timeline_entries
|
||||
WHERE page_id = NEW.id;
|
||||
|
||||
NEW.search_vector :=
|
||||
setweight(to_tsvector('${lang}', coalesce(NEW.title, '')), 'A') ||
|
||||
setweight(to_tsvector('${lang}', coalesce(NEW.compiled_truth, '')), 'B') ||
|
||||
setweight(to_tsvector('${lang}', coalesce(NEW.timeline, '')), 'C') ||
|
||||
setweight(to_tsvector('${lang}', coalesce(timeline_text, '')), 'C');
|
||||
|
||||
RETURN NEW;
|
||||
END;
|
||||
$fn$ LANGUAGE plpgsql;
|
||||
`;
|
||||
|
||||
const recreateChunksFn = `
|
||||
CREATE OR REPLACE FUNCTION update_chunk_search_vector() RETURNS TRIGGER SET search_path = pg_catalog, public AS $fn$
|
||||
BEGIN
|
||||
NEW.search_vector :=
|
||||
setweight(to_tsvector('${lang}', COALESCE(NEW.doc_comment, '')), 'A') ||
|
||||
setweight(to_tsvector('${lang}', COALESCE(NEW.symbol_name_qualified, '')), 'A') ||
|
||||
setweight(to_tsvector('${lang}', COALESCE(NEW.chunk_text, '')), 'B');
|
||||
RETURN NEW;
|
||||
END;
|
||||
$fn$ LANGUAGE plpgsql;
|
||||
`;
|
||||
|
||||
await engine.executeRaw(recreatePagesFn);
|
||||
await engine.executeRaw(recreateChunksFn);
|
||||
|
||||
const progress = createProgress(cliOptsToProgressOptions(getCliOptions()));
|
||||
|
||||
// Backfill: UPDATE-to-self forces the pages trigger to re-fire
|
||||
// (Postgres re-fires on UPDATE-to-same-value); content_chunks gets a
|
||||
// direct vector compute since the column itself is what we want.
|
||||
progress.start('reindex_search_vector.pages', pagesCount);
|
||||
await batchedBackfill(engine, 'pages', 'id = id', n => progress.tick(n));
|
||||
progress.finish();
|
||||
|
||||
progress.start('reindex_search_vector.chunks', chunksCount);
|
||||
await batchedBackfill(
|
||||
engine,
|
||||
'content_chunks',
|
||||
`search_vector =
|
||||
setweight(to_tsvector('${lang}', COALESCE(doc_comment, '')), 'A') ||
|
||||
setweight(to_tsvector('${lang}', COALESCE(symbol_name_qualified, '')), 'A') ||
|
||||
setweight(to_tsvector('${lang}', COALESCE(chunk_text, '')), 'B')`,
|
||||
n => progress.tick(n)
|
||||
);
|
||||
progress.finish();
|
||||
|
||||
const result: ReindexSearchVectorResult = {
|
||||
status: 'ok',
|
||||
language: lang,
|
||||
pagesUpdated: pagesCount,
|
||||
chunksUpdated: chunksCount,
|
||||
triggersRecreated: 2,
|
||||
durationMs: Date.now() - startedAt,
|
||||
};
|
||||
|
||||
if (opts.json) {
|
||||
console.log(JSON.stringify(result, null, 2));
|
||||
} else {
|
||||
console.log(`✅ Recreated 2 trigger functions with language='${lang}'`);
|
||||
console.log(`✅ Backfilled ${pagesCount} pages + ${chunksCount} chunks (${result.durationMs}ms)`);
|
||||
}
|
||||
|
||||
return result;
|
||||
}
|
||||
|
||||
/**
|
||||
* CLI entrypoint. Parses argv flags and dispatches to runReindexSearchVector.
|
||||
* Matches the style of `reindex-code`: --dry-run, --yes/-y, --json.
|
||||
*
|
||||
* Exit codes: 0 success/dry-run/cancelled, 2 if non-TTY without --yes.
|
||||
*/
|
||||
export async function runReindexSearchVectorCli(
|
||||
engine: BrainEngine,
|
||||
args: string[]
|
||||
): Promise<void> {
|
||||
const dryRun = args.includes('--dry-run');
|
||||
const yes = args.includes('--yes') || args.includes('-y');
|
||||
const json = args.includes('--json');
|
||||
|
||||
await runReindexSearchVector(engine, { dryRun, yes, json });
|
||||
}
|
||||
+73
-3
@@ -2010,7 +2010,13 @@ async function performSyncInner(engine: BrainEngine, opts: SyncOpts): Promise<Sy
|
||||
const pageOpts = opts.sourceId ? { sourceId: opts.sourceId } : undefined;
|
||||
for (const path of unsyncableModified) {
|
||||
// v0.41.13 #1433: never delete on metafile classification.
|
||||
if (unsyncableReason(path, syncOpts) === 'metafile') continue;
|
||||
// #2404 hardening: same for 'pruned-dir' — a page under a pruned
|
||||
// directory can only exist via a deliberate put_page (sync never
|
||||
// imports those paths), so "the file was modified" is not evidence
|
||||
// the page is stale. Deleting here silently destroyed put-created
|
||||
// pages every time their materialized file landed in a commit.
|
||||
const reason = unsyncableReason(path, syncOpts);
|
||||
if (reason === 'metafile' || reason === 'pruned-dir') continue;
|
||||
const slug = await resolveSlugByPathOrSourcePath(engine, path, opts.sourceId);
|
||||
try {
|
||||
const existing = await engine.getPage(slug, pageOpts);
|
||||
@@ -3321,9 +3327,45 @@ async function performFullSync(
|
||||
`GBRAIN_ALLOW_MASS_RECONCILE=1 to restore the old behavior.`,
|
||||
);
|
||||
} else if (plan.staleSlugs.length > 0) {
|
||||
// #2426: a stale page whose source_path was NEVER committed to git is
|
||||
// DB-only write-through (the file was written into the clone but never
|
||||
// committed/pushed, then lost — e.g. a fresh clone). "Absent from git"
|
||||
// is the SYMPTOM of that bug, not evidence the content is disposable.
|
||||
// Keep those pages and re-export their markdown to the working tree so
|
||||
// they're file-backed again; only pages whose file once existed in git
|
||||
// history (i.e. was genuinely deleted) are reconcile-deleted.
|
||||
const everCommitted = listEverCommittedPaths(repoPath);
|
||||
const pathBySlug = new Map(rows.map(r => [r.slug, r.source_path]));
|
||||
let deletableSlugs = plan.staleSlugs;
|
||||
const dbOnlySlugs: string[] = [];
|
||||
if (everCommitted) {
|
||||
deletableSlugs = [];
|
||||
for (const slug of plan.staleSlugs) {
|
||||
const sp = pathBySlug.get(slug);
|
||||
if (sp && !everCommitted.has(sp.replace(/\\/g, '/'))) dbOnlySlugs.push(slug);
|
||||
else deletableSlugs.push(slug);
|
||||
}
|
||||
}
|
||||
if (dbOnlySlugs.length > 0) {
|
||||
let reExported = 0;
|
||||
try {
|
||||
const { writePageThrough } = await import('../core/write-through.ts');
|
||||
for (const slug of dbOnlySlugs) {
|
||||
const r = await writePageThrough(engine, slug, { sourceId: sid });
|
||||
if (r.written) reExported++;
|
||||
}
|
||||
} catch { /* best-effort — pages are preserved either way */ }
|
||||
serr(
|
||||
`\n Kept ${dbOnlySlugs.length} page(s) whose markdown was never committed to git ` +
|
||||
`(DB-only write-through — not deleting).` +
|
||||
(reExported > 0 ? ` Re-exported ${reExported} of them to the working tree.` : '') +
|
||||
`\n Commit + push them (e.g. scripts/brain-commit-push.sh, or 'gbrain sources harden') ` +
|
||||
`so the next sync sees them as file-backed.`,
|
||||
);
|
||||
}
|
||||
const deleteScopedOpts = { sourceId: sid };
|
||||
for (let i = 0; i < plan.staleSlugs.length; i += DELETE_BATCH_SIZE) {
|
||||
const batch = plan.staleSlugs.slice(i, i + DELETE_BATCH_SIZE);
|
||||
for (let i = 0; i < deletableSlugs.length; i += DELETE_BATCH_SIZE) {
|
||||
const batch = deletableSlugs.slice(i, i + DELETE_BATCH_SIZE);
|
||||
try {
|
||||
const deleted = await engine.deletePages(batch, deleteScopedOpts);
|
||||
reconciledDeletes += deleted.length;
|
||||
@@ -3442,6 +3484,34 @@ export function planReconcileDeletes(
|
||||
return { staleSlugs, reconcilableCount: reconcilable.length, massDelete };
|
||||
}
|
||||
|
||||
/**
|
||||
* #2426: every repo-relative path that ever appeared as an ADD in git history
|
||||
* (rename detection off, so a `git mv` destination still counts as an add).
|
||||
* Used by the full-sync reconcile to distinguish "file was committed and later
|
||||
* deleted" (genuine delete → reconcile) from "file was NEVER committed"
|
||||
* (DB-only write-through → preserve). Returns null when `repoPath` isn't a git
|
||||
* work tree or git is unavailable — callers keep the plain-directory behavior.
|
||||
* Forward-slash-normalized to match `normalizeReconcilePath` membership tests.
|
||||
*/
|
||||
export function listEverCommittedPaths(repoPath: string): Set<string> | null {
|
||||
let stdout: string;
|
||||
try {
|
||||
stdout = execFileSync(
|
||||
'git',
|
||||
['-C', repoPath, '-c', 'core.quotepath=off', 'log', '--all', '--no-renames',
|
||||
'--diff-filter=A', '--format=', '--name-only'],
|
||||
{ encoding: 'utf8', maxBuffer: 512 * 1024 * 1024, stdio: ['ignore', 'pipe', 'ignore'] },
|
||||
);
|
||||
} catch {
|
||||
return null;
|
||||
}
|
||||
const set = new Set<string>();
|
||||
for (const line of stdout.split('\n')) {
|
||||
if (line) set.add(line.replace(/\\/g, '/'));
|
||||
}
|
||||
return set;
|
||||
}
|
||||
|
||||
/**
|
||||
* #2828 escape hatch: `GBRAIN_ALLOW_MASS_RECONCILE=1` restores the pre-valve
|
||||
* behavior for the rare intentional bulk removal. Env-only (an incident-time
|
||||
|
||||
@@ -23,6 +23,7 @@ import { zhipu } from './zhipu.ts';
|
||||
import { azureOpenAI } from './azure-openai.ts';
|
||||
import { zeroentropyai } from './zeroentropyai.ts';
|
||||
import { llamaServerReranker } from './llama-server-reranker.ts';
|
||||
import { moonshot } from './moonshot.ts';
|
||||
|
||||
const ALL: Recipe[] = [
|
||||
openai,
|
||||
@@ -42,6 +43,7 @@ const ALL: Recipe[] = [
|
||||
zhipu,
|
||||
azureOpenAI,
|
||||
zeroentropyai,
|
||||
moonshot,
|
||||
];
|
||||
|
||||
/** Map from `provider:id` key to recipe. */
|
||||
|
||||
@@ -0,0 +1,42 @@
|
||||
import type { Recipe } from '../types.ts';
|
||||
|
||||
/**
|
||||
* Moonshot AI / Kimi Open Platform. Kimi exposes an OpenAI-compatible
|
||||
* /v1/chat/completions API at https://api.moonshot.ai/v1.
|
||||
*
|
||||
* Verified against Kimi API docs and live /v1/models on 2026-06-23.
|
||||
* The recipe is local-production glue until upstream GBrain carries a native
|
||||
* Moonshot recipe; keep it registered in the local patch registry.
|
||||
*/
|
||||
export const moonshot: Recipe = {
|
||||
id: 'moonshot',
|
||||
name: 'Moonshot AI / Kimi',
|
||||
tier: 'openai-compat',
|
||||
implementation: 'openai-compatible',
|
||||
base_url_default: 'https://api.moonshot.ai/v1',
|
||||
auth_env: {
|
||||
required: ['MOONSHOT_API_KEY'],
|
||||
setup_url: 'https://platform.kimi.ai/console/api-keys',
|
||||
},
|
||||
touchpoints: {
|
||||
expansion: {
|
||||
models: ['kimi-k2.7-code', 'kimi-k2.7-code-highspeed', 'kimi-k2.6', 'kimi-k2.5'],
|
||||
// Kimi pricing varies by current promotional/account terms; do not use
|
||||
// this advisory field for budget enforcement. Canonical budget pricing
|
||||
// belongs in src/core/model-pricing.ts when verified for the account.
|
||||
price_last_verified: '2026-06-23',
|
||||
},
|
||||
chat: {
|
||||
models: ['kimi-k2.7-code', 'kimi-k2.7-code-highspeed', 'kimi-k2.6', 'kimi-k2.5'],
|
||||
supports_tools: true,
|
||||
// Kimi tool calling is enough for ordinary chat/tool calls. GBrain's
|
||||
// subagent loop remains Anthropic-pinned because upstream requires stable
|
||||
// Anthropic-style tool_use_id behavior across crashes/replays.
|
||||
supports_subagent_loop: false,
|
||||
supports_prompt_cache: false,
|
||||
max_context_tokens: 256000,
|
||||
price_last_verified: '2026-06-23',
|
||||
},
|
||||
},
|
||||
setup_hint: 'Get an API key at https://platform.kimi.ai/console/api-keys, then `export MOONSHOT_API_KEY=...` and use `moonshot:kimi-k2.7-code`.',
|
||||
};
|
||||
@@ -183,15 +183,17 @@ if [ "\${1:-}" = "--push-only" ]; then
|
||||
fi
|
||||
|
||||
_msg="\${1:?usage: brain-commit-push.sh <message> <path> [paths...]}"; shift || true
|
||||
# Pull first so the local tree is current before we stage.
|
||||
git fetch origin >/dev/null 2>&1 || true
|
||||
git pull --rebase origin "$_branch" || { git rebase --abort >/dev/null 2>&1 || true; echo "rebase conflict: manual attention needed" >&2; exit 3; }
|
||||
|
||||
# EXPLICIT paths only — never a blind 'git add -A' (would risk committing
|
||||
# secrets, temp files, or unrelated edits).
|
||||
if [ "$#" -eq 0 ]; then
|
||||
echo "refusing blind 'git add -A' — pass explicit path(s) to commit" >&2; exit 2
|
||||
fi
|
||||
# COMMIT BEFORE PULL (#2426): the old order (fetch + pull --rebase, THEN stage)
|
||||
# aborted on any dirty tree — 'cannot pull with rebase: You have unstaged
|
||||
# changes' — so the helper could never commit a MODIFIED page (exactly the
|
||||
# write-through case). Stage + commit first; brain_push below already handles
|
||||
# a remote that advanced (push -> rejected -> pull --rebase -> push).
|
||||
git add -- "$@"
|
||||
if git diff --cached --quiet; then echo "nothing to commit"; exit 0; fi
|
||||
git commit -m "$_msg"
|
||||
@@ -337,6 +339,47 @@ function uninstallLocalHook(repoPath: string): boolean {
|
||||
return true;
|
||||
}
|
||||
|
||||
/**
|
||||
* True when the gbrain durability post-commit hook is installed — i.e. the
|
||||
* user opted this repo into push-durability via `gbrain sources harden`.
|
||||
* Cheap (one git-config read + one file read); used as the gate for
|
||||
* write-through auto-commit (#2426).
|
||||
*/
|
||||
export function isDurabilityHardened(repoPath: string): boolean {
|
||||
try {
|
||||
const { dir } = resolveHooksDir(repoPath);
|
||||
const hookPath = join(dir, 'post-commit');
|
||||
return existsSync(hookPath) && readFileSync(hookPath, 'utf-8').includes(HOOK_BANNER);
|
||||
} catch {
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* #2426: best-effort commit of a single write-through artifact so DB writes
|
||||
* reach git (the post-commit hook then background-pushes). Pre-fix,
|
||||
* write-through `.md` accumulated uncommitted forever: it never reached the
|
||||
* remote, froze `last_sync_at` (HEAD never moved), and a later `sync --full`
|
||||
* delete-reconcile treated the never-committed pages as disposable.
|
||||
*
|
||||
* Path-limited (`git commit -- <path>`) so unrelated staged/dirty edits are
|
||||
* never swept into the commit. Never throws; returns false on any failure
|
||||
* (index.lock contention, nothing changed, detached states) — the DB row and
|
||||
* the on-disk file remain the durable sinks either way.
|
||||
*/
|
||||
export function commitWriteThroughFile(repoPath: string, absPath: string, slug: string): boolean {
|
||||
try {
|
||||
const rel = relative(repoPath, absPath);
|
||||
if (!rel || rel.startsWith('..') || isAbsolute(rel)) return false;
|
||||
const gitOpts = { stdio: 'ignore', timeout: 30_000, env: { ...process.env, ...GIT_ENV } } as const;
|
||||
execFileSync('git', ['-C', repoPath, 'add', '--', rel], gitOpts);
|
||||
execFileSync('git', ['-C', repoPath, 'commit', '-m', `gbrain: write-through ${slug}`, '--', rel], gitOpts);
|
||||
return true;
|
||||
} catch {
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
// ── Committed helper ────────────────────────────────────────────────────────
|
||||
|
||||
function installHelper(repoPath: string, dryRun: boolean): { status: StepStatus; detail: string } {
|
||||
|
||||
@@ -26,7 +26,17 @@ export interface ChronicleJudgeInput {
|
||||
effectiveDate: string | null; // depth page effective_date (deterministic when)
|
||||
attendees: string[]; // deterministic who from frontmatter
|
||||
}
|
||||
export interface ChronicleJudgeResult { events: ChronicleEventProposal[] }
|
||||
export interface ChronicleJudgeResult {
|
||||
events: ChronicleEventProposal[];
|
||||
/**
|
||||
* #2606 — distinct judge-failure signal so an unusable response is never
|
||||
* recorded as a legitimate `no_events`:
|
||||
* - 'truncated': the model hit the output-token cap (stopReason 'length');
|
||||
* the JSON array was cut mid-stream and must not be parsed as complete.
|
||||
* - 'parse_failed': the model returned text but no valid JSON array.
|
||||
*/
|
||||
failure?: 'truncated' | 'parse_failed';
|
||||
}
|
||||
export type ChronicleJudge = (input: ChronicleJudgeInput) => Promise<ChronicleJudgeResult>;
|
||||
|
||||
export interface ChronicleExtractResult {
|
||||
@@ -126,6 +136,12 @@ export async function runChronicleExtract(
|
||||
return { slug: opts.slug, status: 'skipped', events_written: 0, reason: 'judge_error' };
|
||||
}
|
||||
|
||||
// #2606: a truncated or unparseable judge response is a FAILURE, not an
|
||||
// empty page. Record it as a distinct skipped reason so operators (and
|
||||
// retries) can tell it apart from a genuine no_events.
|
||||
if (result?.failure) {
|
||||
return { slug: opts.slug, status: 'skipped', events_written: 0, reason: `judge_${result.failure}` };
|
||||
}
|
||||
const proposals = Array.isArray(result?.events) ? result.events : [];
|
||||
if (proposals.length === 0) return { slug: opts.slug, status: 'no_events', events_written: 0 };
|
||||
// PARSE BARRIER — reject the WHOLE batch on any malformed proposal; no partial writes.
|
||||
@@ -167,11 +183,25 @@ const JUDGE_SYSTEM = `You segment a meeting/transcript page into discrete timeli
|
||||
Return ONLY a JSON array. Each element: {"when": ISO datetime or YYYY-MM-DD, "who": [entity slugs/names], "what": one-clause summary, "where": optional string, "kind": one of meeting|call|meal|solo|travel|work|commitment|decision|intro|conflict|milestone|event}.
|
||||
Prefer the page's known date for "when" when the text gives no explicit time. Use the provided attendee slugs for "who" when the text does not name participants. No prose, no markdown — just the JSON array.`;
|
||||
|
||||
/**
|
||||
* #2606: default output-token cap for the judge. Raised from the original
|
||||
* 1500 (which event-dense pages overflowed, silently truncating the JSON
|
||||
* array). Override via `chronicle.judge_max_tokens`.
|
||||
*/
|
||||
const DEFAULT_JUDGE_MAX_TOKENS = 4000;
|
||||
|
||||
function defaultJudge(engine: BrainEngine): ChronicleJudge {
|
||||
return async (input) => {
|
||||
const { isAvailable, chat } = await import('../ai/gateway.ts');
|
||||
if (!isAvailable('chat')) return { events: [] };
|
||||
const body = (input.body || '').slice(0, 12_000);
|
||||
// #2606: configurable cap so event-dense pages have headroom.
|
||||
let maxTokens = DEFAULT_JUDGE_MAX_TOKENS;
|
||||
const capRaw = await engine.getConfig('chronicle.judge_max_tokens').catch(() => null);
|
||||
if (capRaw) {
|
||||
const n = parseInt(capRaw, 10);
|
||||
if (Number.isFinite(n) && n > 0) maxTokens = n;
|
||||
}
|
||||
let text: string;
|
||||
try {
|
||||
const res = await chat({
|
||||
@@ -183,32 +213,44 @@ function defaultJudge(engine: BrainEngine): ChronicleJudge {
|
||||
`${input.title}\n\n${body}\n</page>\n\n` +
|
||||
`Known attendees: ${input.attendees.slice(0, 10).join(', ') || '(none)'}.\nExtract the events.`,
|
||||
}],
|
||||
maxTokens: 1500,
|
||||
maxTokens,
|
||||
});
|
||||
if (res.stopReason === 'refusal' || res.stopReason === 'content_filter') return { events: [] };
|
||||
// #2606: output hit the token cap — the JSON array is cut mid-stream.
|
||||
// Do NOT feed it to the parser as if complete; surface the truncation.
|
||||
if (res.stopReason === 'length') return { events: [], failure: 'truncated' };
|
||||
text = res.text;
|
||||
} catch (err) {
|
||||
if ((err as Error)?.name === 'AbortError') throw err;
|
||||
return { events: [] };
|
||||
}
|
||||
const parsed = parseJudgeJson(text);
|
||||
// #2606: non-empty model text with no parseable JSON array is a parse
|
||||
// failure, distinct from the model legitimately answering `[]`.
|
||||
if (parsed === null) return { events: [], failure: 'parse_failed' };
|
||||
return { events: parsed };
|
||||
};
|
||||
}
|
||||
|
||||
/** Tolerant JSON-array extraction from a model response (mirrors facts parser). */
|
||||
export function parseJudgeJson(text: string): ChronicleEventProposal[] {
|
||||
if (!text) return [];
|
||||
/**
|
||||
* Tolerant JSON-array extraction from a model response (mirrors facts parser).
|
||||
*
|
||||
* #2606: returns `null` on parse FAILURE (empty text, no `[...]` found,
|
||||
* JSON.parse throw, non-array result) so callers can distinguish "the model
|
||||
* said no events" (a legitimate `[]`) from "the response was unusable".
|
||||
*/
|
||||
export function parseJudgeJson(text: string): ChronicleEventProposal[] | null {
|
||||
if (!text) return null;
|
||||
let s = text.trim();
|
||||
const fence = s.match(/```(?:json)?\s*([\s\S]*?)```/i);
|
||||
if (fence) s = fence[1].trim();
|
||||
const start = s.indexOf('[');
|
||||
const end = s.lastIndexOf(']');
|
||||
if (start === -1 || end === -1 || end < start) return [];
|
||||
if (start === -1 || end === -1 || end < start) return null;
|
||||
try {
|
||||
const arr = JSON.parse(s.slice(start, end + 1));
|
||||
return Array.isArray(arr) ? arr : [];
|
||||
return Array.isArray(arr) ? arr : null;
|
||||
} catch {
|
||||
return [];
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
@@ -915,6 +915,8 @@ export const KNOWN_CONFIG_KEYS: readonly string[] = [
|
||||
'dream.synthesize.verdict_model',
|
||||
'dream.synthesize.max_prompt_tokens',
|
||||
'dream.synthesize.max_chunks_per_transcript',
|
||||
// #2415: top-level namespace for synthesize/patterns output (default 'wiki').
|
||||
'dream.synthesize.output_root',
|
||||
'dream.synthesize.subagent_timeout_ms',
|
||||
'dream.synthesize.subagent_wait_timeout_ms',
|
||||
'dream.patterns.lookback_days',
|
||||
@@ -971,6 +973,10 @@ export const KNOWN_CONFIG_KEYS: readonly string[] = [
|
||||
// operator had to discover --force by reading source. Same class as the
|
||||
// spend-controls registration above.
|
||||
'auto_chronicle',
|
||||
// #2606: chronicle judge output-token cap (default 4000). Event-dense
|
||||
// pages overflowed the old hardcoded 1500 and were misrecorded as
|
||||
// no_events; the cap is now configurable and truncation is surfaced.
|
||||
'chronicle.judge_max_tokens',
|
||||
// Takes bootstrap (v0.41.18.0, A12). The onboard remediation's two-gate
|
||||
// consent reads this key, and enabling it is the documented path to
|
||||
// `gbrain takes extract --from-pages` — same unregistered-key class.
|
||||
|
||||
@@ -1682,6 +1682,9 @@ export async function runCycle(
|
||||
from: opts.synthFrom,
|
||||
to: opts.synthTo,
|
||||
bypassDreamGuard: opts.synthBypassDreamGuard,
|
||||
// #1586: scope synthesized writes to the cycle's resolved source
|
||||
// (explicit --source wins, else derived from the checkout dir).
|
||||
sourceId: cycleSourceId,
|
||||
}));
|
||||
result.duration_ms = duration_ms;
|
||||
phaseResults.push(result);
|
||||
|
||||
+125
-30
@@ -11,15 +11,17 @@
|
||||
* 1. Reads the markdown body (DB-side fetch via engine.getPage).
|
||||
* 2. Parses the `## Facts` fence with parseFactsFence.
|
||||
* 3. Maps ParsedFact → FenceExtractedFact via extractFactsFromFenceText.
|
||||
* 4. Wipes the page's DB index via deleteFactsForPage.
|
||||
* 5. Re-inserts via engine.insertFacts batch.
|
||||
* 4. De-dupes rows by canonical (claim, source) content key.
|
||||
* 5. Reconciles the page-scoped DB index: no-op when already in sync,
|
||||
* insert only missing keys when possible, or wipe/reinsert when stale
|
||||
* DB rows need cleanup (#1781 — the unconditional wipe-and-reinsert
|
||||
* made every cycle non-idempotent, re-appending duplicate rows).
|
||||
*
|
||||
* After the phase, the DB index for every affected page byte-matches
|
||||
* the fence (modulo embeddings + runtime-derived fields). Pages with
|
||||
* no fence go through delete-then-empty-insert — DB rows for that
|
||||
* page coordinate are wiped; legacy NULL-source_markdown_slug rows
|
||||
* survive because deleteFactsForPage targets source_markdown_slug =
|
||||
* slug only.
|
||||
* After the phase, the DB index for every affected page matches the
|
||||
* fence's canonical (claim, source) row set (modulo embeddings +
|
||||
* runtime-derived fields). Pages with no fence wipe DB rows for that
|
||||
* page coordinate only; legacy NULL-source_markdown_slug rows survive
|
||||
* because deleteFactsForPage targets source_markdown_slug = slug only.
|
||||
*
|
||||
* Empty-fence guard (Codex R2-#7): the phase refuses to do its
|
||||
* destructive reconciliation pass when legacy rows (row_num IS NULL,
|
||||
@@ -35,7 +37,11 @@ import type { BrainEngine } from '../engine.ts';
|
||||
import { writeReceipt } from '../extract/receipt-writer.ts';
|
||||
import { upsertExtractRollup } from '../extract/rollup-writer.ts';
|
||||
import { parseFactsFence } from '../facts-fence.ts';
|
||||
import { extractFactsFromFenceText } from '../facts/extract-from-fence.ts';
|
||||
import {
|
||||
extractFactsFromFenceText,
|
||||
FENCE_SOURCE_DEFAULT,
|
||||
type FenceExtractedFact,
|
||||
} from '../facts/extract-from-fence.ts';
|
||||
import {
|
||||
runPhantomRedirectPass,
|
||||
emptyPhantomPassResult,
|
||||
@@ -44,6 +50,51 @@ import {
|
||||
import { embed, isAvailable } from '../ai/gateway.ts';
|
||||
import { isAborted } from '../abort-check.ts';
|
||||
|
||||
interface ExistingPageFact {
|
||||
fact: string;
|
||||
source: string | null;
|
||||
row_num: number | string | null;
|
||||
}
|
||||
|
||||
function factContentKey(fact: string, source: string | null | undefined): string {
|
||||
return `${fact}\u0000${source ?? FENCE_SOURCE_DEFAULT}`;
|
||||
}
|
||||
|
||||
function dedupeFactsByContentKey(facts: FenceExtractedFact[]): FenceExtractedFact[] {
|
||||
const seen = new Set<string>();
|
||||
const deduped: FenceExtractedFact[] = [];
|
||||
for (const fact of facts) {
|
||||
const key = factContentKey(fact.fact, fact.source);
|
||||
if (seen.has(key)) continue;
|
||||
seen.add(key);
|
||||
deduped.push(fact);
|
||||
}
|
||||
return deduped;
|
||||
}
|
||||
|
||||
/**
|
||||
* Fence-owned DB rows for one page coordinate. Excludes `cli:`-origin
|
||||
* conversation facts (#1928) — they are not fence-owned, so they must
|
||||
* neither count as "stale" (which would force a wipe every cycle) nor
|
||||
* be compared against the fence's row set. Mirrors the
|
||||
* excludeSourcePrefixes filter deleteFactsForPage applies on the wipe.
|
||||
*/
|
||||
async function listExistingFactsForPage(
|
||||
engine: BrainEngine,
|
||||
slug: string,
|
||||
sourceId: string,
|
||||
): Promise<ExistingPageFact[]> {
|
||||
return engine.executeRaw<ExistingPageFact>(
|
||||
`SELECT fact, source, row_num
|
||||
FROM facts
|
||||
WHERE source_id = $1
|
||||
AND source_markdown_slug = $2
|
||||
AND COALESCE(source, '') NOT LIKE 'cli:%'
|
||||
ORDER BY row_num ASC, id ASC`,
|
||||
[sourceId, slug],
|
||||
);
|
||||
}
|
||||
|
||||
export interface ExtractFactsOpts {
|
||||
/** Subset of slugs to reconcile. undefined = walk every page in the brain. */
|
||||
slugs?: string[];
|
||||
@@ -220,28 +271,70 @@ export async function runExtractFacts(
|
||||
|
||||
if (parsed.facts.length > 0) result.pagesWithFacts += 1;
|
||||
|
||||
if (opts.dryRun) continue;
|
||||
|
||||
// Wipe-and-reinsert per page. The delete targets source_markdown_slug =
|
||||
// slug only, so NULL-source_markdown_slug legacy rows survive (the
|
||||
// partial-UNIQUE-index keyspace). #1928: `cli:`-origin facts (conversation
|
||||
// facts from extract-conversation-facts) are NOT fence-owned — the page
|
||||
// carries no `## Facts` fence to recreate them — so they MUST survive this
|
||||
// reconcile. Exclude them from the wipe.
|
||||
const deleted = await engine.deleteFactsForPage(slug, sourceId, {
|
||||
excludeSourcePrefixes: ['cli:'],
|
||||
});
|
||||
result.factsDeleted += deleted.deleted;
|
||||
|
||||
if (parsed.facts.length === 0) continue;
|
||||
|
||||
// v0.35.4 (D-ENG-1) — thread page.effective_date as the fallback
|
||||
// valid_from. Without this, fence rows without explicit `validFrom:`
|
||||
// land with `valid_from = now()` (import timestamp) and every
|
||||
// trajectory query against the page returns import dates instead of
|
||||
// claim dates.
|
||||
const pageEffectiveDate = page.effective_date ? new Date(page.effective_date) : null;
|
||||
const extracted = extractFactsFromFenceText(parsed.facts, slug, sourceId, { pageEffectiveDate });
|
||||
const extracted = dedupeFactsByContentKey(
|
||||
extractFactsFromFenceText(parsed.facts, slug, sourceId, { pageEffectiveDate }),
|
||||
);
|
||||
|
||||
if (opts.dryRun) continue;
|
||||
|
||||
// #1781 — reconcile instead of unconditional wipe-and-reinsert. Compare
|
||||
// the fence's canonical (claim, source) row set against the page's
|
||||
// fence-owned DB rows: no-op when already in sync, insert only missing
|
||||
// keys when possible, wipe/reinsert only when stale rows need cleanup.
|
||||
const existing = await listExistingFactsForPage(engine, slug, sourceId);
|
||||
const existingKeys = new Set(existing.map(f => factContentKey(f.fact, f.source)));
|
||||
const desiredByKey = new Map(extracted.map(f => [factContentKey(f.fact, f.source), f]));
|
||||
|
||||
if (extracted.length === 0) {
|
||||
if (existing.length > 0) {
|
||||
// The delete targets source_markdown_slug = slug only, so
|
||||
// NULL-source_markdown_slug legacy rows survive (the
|
||||
// partial-UNIQUE-index keyspace). #1928: `cli:`-origin facts
|
||||
// (conversation facts from extract-conversation-facts) are NOT
|
||||
// fence-owned — the page carries no `## Facts` fence to recreate
|
||||
// them — so they MUST survive this reconcile.
|
||||
const deleted = await engine.deleteFactsForPage(slug, sourceId, {
|
||||
excludeSourcePrefixes: ['cli:'],
|
||||
});
|
||||
result.factsDeleted += deleted.deleted;
|
||||
}
|
||||
continue;
|
||||
}
|
||||
|
||||
const hasStaleExisting = existing.some(f => !desiredByKey.has(factContentKey(f.fact, f.source)));
|
||||
const hasDuplicateExisting = existing.length !== existingKeys.size;
|
||||
const hasRowNumDrift = existing.some(f => {
|
||||
const desired = desiredByKey.get(factContentKey(f.fact, f.source));
|
||||
return desired !== undefined && Number(f.row_num) !== desired.row_num;
|
||||
});
|
||||
|
||||
if (
|
||||
existing.length === extracted.length &&
|
||||
!hasStaleExisting &&
|
||||
!hasDuplicateExisting &&
|
||||
!hasRowNumDrift
|
||||
) {
|
||||
continue;
|
||||
}
|
||||
|
||||
let toInsert = extracted.filter(f => !existingKeys.has(factContentKey(f.fact, f.source)));
|
||||
if (hasStaleExisting || hasDuplicateExisting || hasRowNumDrift) {
|
||||
// Fall back to the legacy page-level reconcile when old DB rows must
|
||||
// be removed. Same delete scoping as above: legacy
|
||||
// NULL-source_markdown_slug rows and `cli:`-origin conversation
|
||||
// facts (#1928) survive.
|
||||
const deleted = await engine.deleteFactsForPage(slug, sourceId, {
|
||||
excludeSourcePrefixes: ['cli:'],
|
||||
});
|
||||
result.factsDeleted += deleted.deleted;
|
||||
toInsert = extracted;
|
||||
}
|
||||
|
||||
// v0.35.4 (D-CDX-3) — batch-embed before insert. Without this,
|
||||
// cycle-inserted facts land with `embedding = NULL`, which breaks
|
||||
@@ -250,17 +343,17 @@ export async function runExtractFacts(
|
||||
// unavailable (no API key configured), facts still insert with
|
||||
// NULL embeddings — drift_score gracefully returns null and
|
||||
// clustering falls back to recency.
|
||||
if (isAvailable('embedding') && extracted.length > 0) {
|
||||
if (isAvailable('embedding') && toInsert.length > 0) {
|
||||
try {
|
||||
const texts = extracted.map(e => e.fact);
|
||||
const texts = toInsert.map(e => e.fact);
|
||||
// #1972: forward the abort signal so a cancelled cycle's in-flight
|
||||
// batch embed (a network call) is itself abortable, not just the loop.
|
||||
const embeddings = await embed(texts, { abortSignal: opts.signal });
|
||||
// Defensive: embed should return one vector per input; if the
|
||||
// gateway returns a partial array (provider partial-batch retry
|
||||
// returning fewer than requested), only fill what we have.
|
||||
for (let i = 0; i < extracted.length && i < embeddings.length; i++) {
|
||||
extracted[i].embedding = embeddings[i];
|
||||
for (let i = 0; i < toInsert.length && i < embeddings.length; i++) {
|
||||
toInsert[i].embedding = embeddings[i];
|
||||
}
|
||||
} catch (err) {
|
||||
// Embedding failure is non-fatal — facts still get inserted, just
|
||||
@@ -271,7 +364,9 @@ export async function runExtractFacts(
|
||||
}
|
||||
}
|
||||
|
||||
const inserted = await engine.insertFacts(extracted, { source_id: sourceId }); // gbrain-allow-direct-insert: extract_facts cycle phase reconciles fence → DB
|
||||
if (toInsert.length === 0) continue;
|
||||
|
||||
const inserted = await engine.insertFacts(toInsert, { source_id: sourceId }); // gbrain-allow-direct-insert: extract_facts cycle phase reconciles fence → DB
|
||||
result.factsInserted += inserted.inserted;
|
||||
}
|
||||
|
||||
|
||||
+19
-31
@@ -19,7 +19,7 @@
|
||||
*/
|
||||
|
||||
import { join, dirname } from 'node:path';
|
||||
import { mkdirSync, writeFileSync, existsSync, readFileSync } from 'node:fs';
|
||||
import { mkdirSync, writeFileSync } from 'node:fs';
|
||||
import type { BrainEngine } from '../engine.ts';
|
||||
import type { PhaseResult, PhaseError } from '../cycle.ts';
|
||||
import { MinionQueue } from '../minions/queue.ts';
|
||||
@@ -27,6 +27,9 @@ import { waitForCompletion, TimeoutError } from '../minions/wait-for-completion.
|
||||
import type { MinionJobInput, SubagentHandlerData } from '../minions/types.ts';
|
||||
import { serializeMarkdown } from '../markdown.ts';
|
||||
import type { Page, PageType } from '../types.ts';
|
||||
// #2415: allow-list + output-root resolution shared with the synthesize
|
||||
// phase — both phases must agree on the configured namespace.
|
||||
import { loadAllowedSlugPrefixes, loadOutputRoot } from './synthesize.ts';
|
||||
import { probeChatModel } from '../ai/gateway.ts';
|
||||
import { normalizeModelId } from '../model-id.ts';
|
||||
|
||||
@@ -49,7 +52,7 @@ export async function runPhasePatterns(
|
||||
}
|
||||
|
||||
// Gather reflections within lookback window.
|
||||
const reflections = await gatherReflections(engine, config.lookbackDays);
|
||||
const reflections = await gatherReflections(engine, config.lookbackDays, config.outputRoot);
|
||||
if (reflections.length < config.minEvidence) {
|
||||
return skipped(
|
||||
'insufficient_evidence',
|
||||
@@ -81,7 +84,7 @@ export async function runPhasePatterns(
|
||||
return skipped('no_provider', `pattern detection skipped: ${probe.detail}`);
|
||||
}
|
||||
|
||||
const allowedSlugPrefixes = await loadAllowedSlugPrefixes();
|
||||
const allowedSlugPrefixes = await loadAllowedSlugPrefixes(config.outputRoot);
|
||||
if (allowedSlugPrefixes.length === 0) {
|
||||
return failed(makeError('InternalError', 'NO_ALLOWLIST',
|
||||
'skills/_brain-filing-rules.json missing dream_synthesize_paths.globs'));
|
||||
@@ -89,7 +92,7 @@ export async function runPhasePatterns(
|
||||
|
||||
const queue = new MinionQueue(engine);
|
||||
const data: SubagentHandlerData = {
|
||||
prompt: buildPatternsPrompt(reflections, config.minEvidence),
|
||||
prompt: buildPatternsPrompt(reflections, config.minEvidence, config.outputRoot),
|
||||
model: config.model,
|
||||
max_turns: 30,
|
||||
allowed_slug_prefixes: allowedSlugPrefixes,
|
||||
@@ -182,6 +185,8 @@ interface PatternsConfig {
|
||||
lookbackDays: number;
|
||||
minEvidence: number;
|
||||
model: string;
|
||||
/** #2415: shared output namespace (dream.synthesize.output_root, default 'wiki'). */
|
||||
outputRoot: string;
|
||||
/** #1594-family: subagent job timeout, config `dream.patterns.subagent_timeout_ms`. */
|
||||
subagentTimeoutMs: number;
|
||||
/** #1594-family: waitForCompletion timeout, config `dream.patterns.subagent_wait_timeout_ms`. */
|
||||
@@ -216,6 +221,7 @@ async function loadPatternsConfig(engine: BrainEngine): Promise<PatternsConfig>
|
||||
lookbackDays: lookbackStr ? Math.max(1, parseInt(lookbackStr, 10) || 30) : 30,
|
||||
minEvidence: minEvidenceStr ? Math.max(1, parseInt(minEvidenceStr, 10) || 3) : 3,
|
||||
model,
|
||||
outputRoot: await loadOutputRoot(engine),
|
||||
subagentTimeoutMs: await getNumberConfig(
|
||||
engine, 'dream.patterns.subagent_timeout_ms', DEFAULT_PATTERNS_SUBAGENT_TIMEOUT_MS,
|
||||
),
|
||||
@@ -236,16 +242,19 @@ interface ReflectionRef {
|
||||
async function gatherReflections(
|
||||
engine: BrainEngine,
|
||||
lookbackDays: number,
|
||||
outputRoot = 'wiki',
|
||||
): Promise<ReflectionRef[]> {
|
||||
const since = new Date(Date.now() - lookbackDays * 24 * 60 * 60 * 1000).toISOString();
|
||||
// #2415: reflections live under the configured output root (bound as a
|
||||
// parameter; outputRoot is slug-grammar-validated by loadOutputRoot).
|
||||
const rows = await engine.executeRaw<{ slug: string; title: string | null; compiled_truth: string | null }>(
|
||||
`SELECT slug, title, compiled_truth
|
||||
FROM pages
|
||||
WHERE slug LIKE 'wiki/personal/reflections/%'
|
||||
WHERE slug LIKE $2
|
||||
AND updated_at >= $1::timestamptz
|
||||
ORDER BY updated_at DESC
|
||||
LIMIT 100`,
|
||||
[since],
|
||||
[since, `${outputRoot}/personal/reflections/%`],
|
||||
);
|
||||
return rows.map(r => ({
|
||||
slug: r.slug,
|
||||
@@ -256,7 +265,7 @@ async function gatherReflections(
|
||||
|
||||
// ── Prompt ────────────────────────────────────────────────────────────
|
||||
|
||||
function buildPatternsPrompt(reflections: ReflectionRef[], minEvidence: number): string {
|
||||
function buildPatternsPrompt(reflections: ReflectionRef[], minEvidence: number, outputRoot = 'wiki'): string {
|
||||
const today = new Date().toISOString().slice(0, 10);
|
||||
const corpus = reflections
|
||||
.map((r, i) => `### ${i + 1}. [[${r.slug}]] — ${r.title}\n${r.excerpt}`)
|
||||
@@ -266,15 +275,15 @@ function buildPatternsPrompt(reflections: ReflectionRef[], minEvidence: number):
|
||||
|
||||
OUTPUT POLICY
|
||||
- Only name a pattern if it appears in at least ${minEvidence} DISTINCT reflections.
|
||||
- Each pattern page MUST cite the reflections that constitute its evidence (use [[wiki/personal/reflections/...]] wikilinks).
|
||||
- Each pattern page MUST cite the reflections that constitute its evidence (use [[${outputRoot}/personal/reflections/...]] wikilinks).
|
||||
- Use \`search\` to check whether a similar pattern page already exists; if yes, update it (use the same slug). If no, create a new one.
|
||||
- Pattern slug format: \`wiki/personal/patterns/<topic-slug>\` (lowercase alphanumeric + hyphens; no underscores, no extension, no date).
|
||||
- Pattern slug format: \`${outputRoot}/personal/patterns/<topic-slug>\` (lowercase alphanumeric + hyphens; no underscores, no extension, no date).
|
||||
- A "pattern" is a recurring theme, anxiety, decision pattern, relationship dynamic, or self-knowledge motif. NOT a single insight. NOT a list of unrelated topics.
|
||||
|
||||
DO NOT WRITE
|
||||
- A "patterns from today" digest (that's the dream-cycle-summaries page; not your job).
|
||||
- Patterns with <${minEvidence} reflections cited.
|
||||
- Anything outside wiki/personal/patterns/.
|
||||
- Anything outside ${outputRoot}/personal/patterns/.
|
||||
|
||||
CONTEXT
|
||||
- Today: ${today}
|
||||
@@ -365,27 +374,6 @@ function renderPageToMarkdown(page: Page, tags: string[]): string {
|
||||
);
|
||||
}
|
||||
|
||||
// ── Allow-list (shared with synthesize.ts) ───────────────────────────
|
||||
|
||||
async function loadAllowedSlugPrefixes(): Promise<string[]> {
|
||||
const candidates = [
|
||||
join(process.cwd(), 'skills', '_brain-filing-rules.json'),
|
||||
join(__dirname, '..', '..', '..', 'skills', '_brain-filing-rules.json'),
|
||||
];
|
||||
for (const path of candidates) {
|
||||
if (!existsSync(path)) continue;
|
||||
try {
|
||||
const raw = readFileSync(path, 'utf8');
|
||||
const parsed = JSON.parse(raw) as { dream_synthesize_paths?: { globs?: unknown } };
|
||||
const globs = parsed?.dream_synthesize_paths?.globs;
|
||||
if (Array.isArray(globs) && globs.every(g => typeof g === 'string')) {
|
||||
return globs as string[];
|
||||
}
|
||||
} catch { /* try next */ }
|
||||
}
|
||||
return [];
|
||||
}
|
||||
|
||||
// ── Status helpers ───────────────────────────────────────────────────
|
||||
|
||||
function ok(summary: string, details: Record<string, unknown> = {}): PhaseResult {
|
||||
|
||||
@@ -23,7 +23,13 @@ import type { PhaseResult } from '../cycle.ts';
|
||||
import type { ProgressReporter } from '../progress.ts';
|
||||
import { writeReceipt } from '../extract/receipt-writer.ts';
|
||||
import { upsertExtractRollup } from '../extract/rollup-writer.ts';
|
||||
import { chat as gatewayChat } from '../ai/gateway.ts';
|
||||
import { chat as gatewayChat, isAvailable } from '../ai/gateway.ts';
|
||||
// #2163: concept pages route through importFromContent (the same
|
||||
// parse→chunk→embed pipeline put_page uses) instead of a bare engine.putPage,
|
||||
// so they land in the retrieval surface (content_chunks + embeddings) where
|
||||
// source-boost's 1.3× 'concepts/' weighting can actually reach them.
|
||||
import { importFromContent } from '../import-file.ts';
|
||||
import { serializeMarkdown } from '../markdown.ts';
|
||||
|
||||
const DEFAULT_BUDGET_USD = 1.5;
|
||||
const TIER_T1_MIN = 10;
|
||||
@@ -216,19 +222,23 @@ export async function runPhaseSynthesizeConcepts(
|
||||
|
||||
if (!opts.dryRun) {
|
||||
const title = group.conceptSlug.split('/').pop() ?? group.conceptSlug;
|
||||
await engine.putPage(`concepts/${title}`, {
|
||||
title: title.replace(/-/g, ' '),
|
||||
type: 'concept',
|
||||
compiled_truth: narrative,
|
||||
frontmatter: {
|
||||
type: 'concept',
|
||||
// #2163: serialize to markdown and import via the canonical pipeline so
|
||||
// the page is chunked (+ embedded when a provider is configured) —
|
||||
// mirrors put_page's isAvailable('embedding') → noEmbed gate.
|
||||
const md = serializeMarkdown(
|
||||
{
|
||||
tier: group.tier,
|
||||
mention_count: group.atomTitles.length,
|
||||
composite_score: group.atomTitles.length,
|
||||
synthesized_at: new Date().toISOString(),
|
||||
synthesized_by: 'synthesize_concepts-v0.41',
|
||||
},
|
||||
timeline: '',
|
||||
narrative,
|
||||
'',
|
||||
{ type: 'concept', title: title.replace(/-/g, ' '), tags: [] },
|
||||
);
|
||||
await importFromContent(engine, `concepts/${title}`, md, {
|
||||
noEmbed: !isAvailable('embedding'),
|
||||
});
|
||||
}
|
||||
conceptsWritten++;
|
||||
|
||||
+119
-21
@@ -244,6 +244,14 @@ export interface SynthesizePhaseOpts {
|
||||
* the synthesize loop. Caller must opt in explicitly.
|
||||
*/
|
||||
bypassDreamGuard?: boolean;
|
||||
/**
|
||||
* #1586: the cycle's resolved brain source (cycleSourceId from cycle.ts —
|
||||
* explicit --source wins, else derived from the checkout dir). Threaded to
|
||||
* every subagent child as `source_id` so put_page writes land in this
|
||||
* source, and stamped onto collected refs so reverse-writes read the
|
||||
* correct (source_id, slug) row. Unset → legacy 'default'.
|
||||
*/
|
||||
sourceId?: string;
|
||||
}
|
||||
|
||||
export async function runPhaseSynthesize(
|
||||
@@ -399,7 +407,7 @@ export async function runPhaseSynthesize(
|
||||
|
||||
// Fan-out: submit one subagent per worth-processing transcript (or one
|
||||
// per chunk for transcripts that exceed the model's per-prompt budget).
|
||||
const allowedSlugPrefixes = await loadAllowedSlugPrefixes();
|
||||
const allowedSlugPrefixes = await loadAllowedSlugPrefixes(config.outputRoot);
|
||||
if (allowedSlugPrefixes.length === 0) {
|
||||
return failed(makeError('InternalError', 'NO_ALLOWLIST',
|
||||
'skills/_brain-filing-rules.json missing dream_synthesize_paths.globs'));
|
||||
@@ -462,10 +470,13 @@ export async function runPhaseSynthesize(
|
||||
: config.model;
|
||||
for (let i = 0; i < chunks.length; i++) {
|
||||
const childData: SubagentHandlerData = {
|
||||
prompt: buildSynthesisPrompt(t, chunks[i], i, chunks.length, priorContradictionsBlock),
|
||||
prompt: buildSynthesisPrompt(t, chunks[i], i, chunks.length, priorContradictionsBlock, config.outputRoot),
|
||||
model: subagentModel,
|
||||
max_turns: 30,
|
||||
allowed_slug_prefixes: allowedSlugPrefixes,
|
||||
// #1586: scope every child tool call to the cycle's resolved source
|
||||
// so put_page writes land there instead of the hardcoded 'default'.
|
||||
...(opts.sourceId ? { source_id: opts.sourceId } : {}),
|
||||
};
|
||||
// Idempotency key parity:
|
||||
// - single-chunk → legacy `dream:synth:<filePath>:<hash16>` (byte-
|
||||
@@ -524,20 +535,29 @@ export async function runPhaseSynthesize(
|
||||
// bare-hash slugs to `<hash6>-c<idx>` so chunked siblings can't collide
|
||||
// even if Sonnet drops the chunk suffix.
|
||||
// v0.32.8: refs carry source_id so reverseWriteRefs picks the correct
|
||||
// (source, slug) row (currently always 'default' from subagent put_page).
|
||||
const writtenRefs = await collectChildPutPageSlugs(engine, childIds, chunkInfo);
|
||||
// (source, slug) row. #1586: refs are stamped with the cycle's resolved
|
||||
// source (children write there via SubagentHandlerData.source_id).
|
||||
const cycleSourceId = opts.sourceId ?? 'default';
|
||||
const writtenRefs = await collectChildPutPageSlugs(engine, childIds, chunkInfo, cycleSourceId);
|
||||
|
||||
const summaryDate = opts.date ?? today();
|
||||
|
||||
// #2569: persist the dream-output identity marker into the DB frontmatter
|
||||
// of every child-written page BEFORE reverse-rendering, so generated pages
|
||||
// are queryable (`frontmatter->>'dream_generated'`) and a later put_page
|
||||
// write-through (which re-renders from the DB row) can't erase the stamp.
|
||||
await stampDreamProvenance(engine, writtenRefs, summaryDate);
|
||||
|
||||
// Dual-write: reverse-render each DB row → markdown file.
|
||||
const reverseWriteCount = await reverseWriteRefs(engine, opts.brainDir, writtenRefs);
|
||||
const reverseWriteCount = await reverseWriteRefs(engine, opts.brainDir, writtenRefs, cycleSourceId);
|
||||
|
||||
// Summary index page (deterministic; orchestrator-written via direct
|
||||
// engine.putPage so no allow-list path needed).
|
||||
const summaryDate = opts.date ?? today();
|
||||
const summarySlug = `dream-cycle-summaries/${summaryDate}`;
|
||||
// Back-compat: writeSummaryPage takes string[] for display; map refs back to slugs.
|
||||
const writtenSlugs = writtenRefs.map(r => r.slug);
|
||||
if (SUMMARY_SLUG_RE.test(summarySlug)) {
|
||||
await writeSummaryPage(engine, opts.brainDir, summarySlug, summaryDate, writtenSlugs, childOutcomes);
|
||||
await writeSummaryPage(engine, opts.brainDir, summarySlug, summaryDate, writtenSlugs, childOutcomes, cycleSourceId);
|
||||
}
|
||||
|
||||
// Write completion timestamp ON SUCCESS only.
|
||||
@@ -595,10 +615,29 @@ interface SynthConfig {
|
||||
* `dream.synthesize.max_chunks_per_transcript`.
|
||||
*/
|
||||
maxChunksPerTranscript: number;
|
||||
/**
|
||||
* #2415: top-level namespace for synthesized output (reflections, originals,
|
||||
* patterns). Config key `dream.synthesize.output_root`; default 'wiki' —
|
||||
* zero behavior change unless set. No trailing slash. Must satisfy the slug
|
||||
* grammar; invalid values fall back to 'wiki' with a stderr warning.
|
||||
*/
|
||||
outputRoot: string;
|
||||
subagentTimeoutMs: number;
|
||||
subagentWaitTimeoutMs: number;
|
||||
}
|
||||
|
||||
/** #2415: shared output-root resolution (synthesize + patterns phases). */
|
||||
export async function loadOutputRoot(engine: BrainEngine): Promise<string> {
|
||||
const raw = await engine.getConfig('dream.synthesize.output_root');
|
||||
if (!raw) return 'wiki';
|
||||
const trimmed = raw.trim().replace(/^\/+|\/+$/g, '');
|
||||
if (SUMMARY_SLUG_RE.test(trimmed)) return trimmed;
|
||||
process.stderr.write(
|
||||
`[dream] dream.synthesize.output_root "${raw}" is not a valid slug prefix; falling back to "wiki".\n`,
|
||||
);
|
||||
return 'wiki';
|
||||
}
|
||||
|
||||
async function loadSynthConfig(engine: BrainEngine): Promise<SynthConfig> {
|
||||
const enabledRaw = await engine.getConfig('dream.synthesize.enabled');
|
||||
const corpusDir = await engine.getConfig('dream.synthesize.session_corpus_dir');
|
||||
@@ -672,6 +711,7 @@ async function loadSynthConfig(engine: BrainEngine): Promise<SynthConfig> {
|
||||
cooldownHours: cooldownHoursStr ? Math.max(0, parseInt(cooldownHoursStr, 10) || 12) : 12,
|
||||
maxPromptTokens,
|
||||
maxChunksPerTranscript,
|
||||
outputRoot: await loadOutputRoot(engine),
|
||||
subagentTimeoutMs,
|
||||
subagentWaitTimeoutMs,
|
||||
};
|
||||
@@ -704,7 +744,13 @@ async function checkCooldown(
|
||||
|
||||
// ── Allow-list source of truth ───────────────────────────────────────
|
||||
|
||||
async function loadAllowedSlugPrefixes(): Promise<string[]> {
|
||||
/**
|
||||
* #2415: `outputRoot` remaps the canonical `wiki/`-rooted globs to the
|
||||
* configured namespace (e.g. `notes/personal/reflections/*`). Default 'wiki'
|
||||
* returns the globs verbatim. Shared by the patterns phase (imported there —
|
||||
* the two phases must enforce the same allow-list).
|
||||
*/
|
||||
export async function loadAllowedSlugPrefixes(outputRoot = 'wiki'): Promise<string[]> {
|
||||
// Search a few known locations relative to the binary / repo. The first
|
||||
// hit wins; if none found, return [].
|
||||
const candidates = [
|
||||
@@ -718,7 +764,10 @@ async function loadAllowedSlugPrefixes(): Promise<string[]> {
|
||||
const parsed = JSON.parse(raw) as { dream_synthesize_paths?: { globs?: unknown } };
|
||||
const globs = parsed?.dream_synthesize_paths?.globs;
|
||||
if (Array.isArray(globs) && globs.every(g => typeof g === 'string')) {
|
||||
return globs as string[];
|
||||
if (outputRoot === 'wiki') return globs as string[];
|
||||
return (globs as string[]).map(g =>
|
||||
g.startsWith('wiki/') ? `${outputRoot}/${g.slice('wiki/'.length)}` : g,
|
||||
);
|
||||
}
|
||||
} catch { /* try next */ }
|
||||
}
|
||||
@@ -966,6 +1015,7 @@ function buildSynthesisPrompt(
|
||||
chunkIdx: number,
|
||||
chunkTotal: number,
|
||||
priorContradictionsBlock = '',
|
||||
outputRoot = 'wiki',
|
||||
): string {
|
||||
const dateHint = t.inferredDate ?? today();
|
||||
const baseSlugSegment = sanitizeForSlug(t.basename) || `session-${dateHint}`;
|
||||
@@ -994,10 +1044,10 @@ OUTPUT POLICY (ALL of these are required)
|
||||
|
||||
TASKS
|
||||
A. Reflections (self-knowledge, pattern recognition, emotional processing):
|
||||
slug: \`wiki/personal/reflections/${dateHint}-<topic-slug>-${hashSuffix}\`
|
||||
slug: \`${outputRoot}/personal/reflections/${dateHint}-<topic-slug>-${hashSuffix}\`
|
||||
|
||||
B. Originals (new ideas, frames, theses, mental models):
|
||||
slug: \`wiki/originals/ideas/${dateHint}-<idea-slug>-${hashSuffix}\`
|
||||
slug: \`${outputRoot}/originals/ideas/${dateHint}-<idea-slug>-${hashSuffix}\`
|
||||
|
||||
C. People mentions: search first; if a page exists, do not put_page over it (the orchestrator handles people enrichment via timeline entries — your job is the reflection/original synthesis, NOT modifying existing person pages).
|
||||
|
||||
@@ -1038,6 +1088,7 @@ async function collectChildPutPageSlugs(
|
||||
engine: BrainEngine,
|
||||
childIds: number[],
|
||||
chunkInfo: Map<number, { idx: number; hash6: string }>,
|
||||
sourceId = 'default',
|
||||
): Promise<Array<{ slug: string; source_id: string }>> {
|
||||
if (childIds.length === 0) return [];
|
||||
// Raw fetch — NO SELECT DISTINCT. Preserves per-child slug duplicates so
|
||||
@@ -1047,10 +1098,10 @@ async function collectChildPutPageSlugs(
|
||||
//
|
||||
// v0.32.8: returns Array<{slug, source_id}> instead of string[]. Subagent
|
||||
// put_page tool schema doesn't expose source_id (subagents are scoped to
|
||||
// a single source); default to 'default' for the current dream-cycle
|
||||
// product behavior. Threading the source_id through reverseWriteRefs
|
||||
// guarantees getPage targets the correct (source, slug) row instead of
|
||||
// the first DB match.
|
||||
// a single source). #1586: the orchestrator scopes each child to the
|
||||
// cycle's resolved source via SubagentHandlerData.source_id, and stamps
|
||||
// the SAME source here so reverseWriteRefs / provenance reads target the
|
||||
// correct (source_id, slug) row. Unset → legacy 'default'.
|
||||
const rows = await engine.executeRaw<{ job_id: number; slug: string }>(
|
||||
`SELECT job_id,
|
||||
COALESCE(input->>'slug', (input #>> '{}')::jsonb->>'slug') AS slug
|
||||
@@ -1066,7 +1117,7 @@ async function collectChildPutPageSlugs(
|
||||
const ci = chunkInfo.get(r.job_id);
|
||||
rewritten.add(ci ? rewriteChunkedSlug(r.slug, ci.hash6, ci.idx) : r.slug);
|
||||
}
|
||||
return Array.from(rewritten).sort().map(slug => ({ slug, source_id: 'default' }));
|
||||
return Array.from(rewritten).sort().map(slug => ({ slug, source_id: sourceId }));
|
||||
}
|
||||
|
||||
/**
|
||||
@@ -1095,12 +1146,52 @@ async function hasLegacySingleChunkCompletion(
|
||||
return rows.length > 0;
|
||||
}
|
||||
|
||||
// ── Dream-provenance DB stamp (#2569) ────────────────────────────────
|
||||
|
||||
/**
|
||||
* Persist the dream-output identity marker (`dream_generated: true` +
|
||||
* `dream_cycle_date`) into the `pages.frontmatter` JSONB row for every page
|
||||
* a synthesize child wrote. Render-time `frontmatterOverrides` alone only
|
||||
* reach the markdown FILE — the DB row stayed unstamped, so DB consumers
|
||||
* couldn't enumerate generated pages and a later put_page write-through
|
||||
* (which re-renders from the DB row) silently erased the marker.
|
||||
*
|
||||
* Plain UPDATE through executeRawJsonb (raw object bound to $3::jsonb —
|
||||
* never JSON.stringify into a ::jsonb cast; engine-parity safe, no new
|
||||
* engine method). Best-effort per row: a stamp failure never kills the
|
||||
* phase (the render-time override still covers the file).
|
||||
*/
|
||||
async function stampDreamProvenance(
|
||||
engine: BrainEngine,
|
||||
refs: Array<{ slug: string; source_id: string }>,
|
||||
cycleDate: string,
|
||||
): Promise<void> {
|
||||
if (refs.length === 0) return;
|
||||
const { executeRawJsonb } = await import('../sql-query.ts');
|
||||
for (const { slug, source_id } of refs) {
|
||||
try {
|
||||
await executeRawJsonb(
|
||||
engine,
|
||||
`UPDATE pages
|
||||
SET frontmatter = COALESCE(frontmatter, '{}'::jsonb) || $3::jsonb
|
||||
WHERE slug = $1 AND source_id = $2`,
|
||||
[slug, source_id],
|
||||
[{ dream_generated: true, dream_cycle_date: cycleDate }],
|
||||
);
|
||||
} catch (e) {
|
||||
const msg = e instanceof Error ? e.message : String(e);
|
||||
process.stderr.write(`[dream] provenance stamp ${slug}@${source_id} failed: ${msg}\n`);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// ── Reverse-write DB rows → markdown files ───────────────────────────
|
||||
|
||||
async function reverseWriteRefs(
|
||||
engine: BrainEngine,
|
||||
brainDir: string,
|
||||
refs: Array<{ slug: string; source_id: string }>,
|
||||
nativeSourceId = 'default',
|
||||
): Promise<number> {
|
||||
let count = 0;
|
||||
for (const { slug, source_id } of refs) {
|
||||
@@ -1111,10 +1202,11 @@ async function reverseWriteRefs(
|
||||
const tags = await engine.getTags(slug, { sourceId: source_id });
|
||||
try {
|
||||
const md = renderPageToMarkdown(page, tags);
|
||||
// v0.32.8 F6: non-default sources land at brainDir/.sources/<id>/<slug>.md
|
||||
// so same-slug-different-source pages don't collide. Default-source
|
||||
// pages stay at brainDir/<slug>.md so single-source brains see no change.
|
||||
const filePath = source_id === 'default'
|
||||
// v0.32.8 F6: foreign-source pages land at brainDir/.sources/<id>/<slug>.md
|
||||
// so same-slug-different-source pages don't collide. Pages belonging to
|
||||
// the cycle's own source (#1586: brainDir IS that source's checkout —
|
||||
// legacy 'default' when unscoped) stay at brainDir/<slug>.md.
|
||||
const filePath = source_id === nativeSourceId
|
||||
? join(brainDir, `${slug}.md`)
|
||||
: join(brainDir, '.sources', source_id, `${slug}.md`);
|
||||
mkdirSync(dirname(filePath), { recursive: true });
|
||||
@@ -1161,6 +1253,7 @@ async function writeSummaryPage(
|
||||
summaryDate: string,
|
||||
writtenSlugs: string[],
|
||||
childOutcomes: Array<{ jobId: number; status: string }>,
|
||||
sourceId = 'default',
|
||||
): Promise<void> {
|
||||
const completed = childOutcomes.filter(c => c.status === 'completed').length;
|
||||
const failed = childOutcomes.length - completed;
|
||||
@@ -1198,13 +1291,15 @@ async function writeSummaryPage(
|
||||
// unnecessarily; we go straight to the engine.
|
||||
const { parseMarkdown } = await import('../markdown.ts');
|
||||
const parsed = parseMarkdown(fullMarkdown);
|
||||
// #1586: summary lands in the cycle's resolved source too — otherwise the
|
||||
// children live in the named source while the index drifts to 'default'.
|
||||
await engine.putPage(summarySlug, {
|
||||
type: parsed.type,
|
||||
title: parsed.title,
|
||||
compiled_truth: parsed.compiled_truth,
|
||||
timeline: parsed.timeline,
|
||||
frontmatter: parsed.frontmatter,
|
||||
});
|
||||
}, { sourceId });
|
||||
|
||||
// Also write to disk (orchestrator dual-write).
|
||||
try {
|
||||
@@ -1269,4 +1364,7 @@ function makeError(cls: string, code: string, message: string, hint?: string): P
|
||||
// double-encoded jsonb regression). Not part of the runtime contract.
|
||||
export const __testing = {
|
||||
collectChildPutPageSlugs,
|
||||
buildSynthesisPrompt,
|
||||
stampDreamProvenance,
|
||||
reverseWriteRefs,
|
||||
};
|
||||
|
||||
@@ -0,0 +1,69 @@
|
||||
/**
|
||||
* Full-text search language configuration.
|
||||
*
|
||||
* Postgres tsvector/tsquery require a text search configuration name (e.g.
|
||||
* 'english', 'portuguese', 'spanish'). Historically GBrain hardcoded
|
||||
* 'english' across engines and trigger functions, which broke search
|
||||
* quality for non-English brains (no stemming, no stop-word removal).
|
||||
*
|
||||
* This helper centralizes the choice. Default stays 'english' for backward
|
||||
* compatibility — only users who set GBRAIN_FTS_LANGUAGE see different
|
||||
* behavior.
|
||||
*
|
||||
* Custom configs (e.g. accent-insensitive 'pt_br' built with unaccent +
|
||||
* portuguese stemmer) are supported as long as the configuration exists
|
||||
* in the target Postgres instance. See docs/guides/multi-language-fts.md
|
||||
* for setup instructions.
|
||||
*
|
||||
* Validation: only allow lowercase letters, digits, and underscores. This
|
||||
* prevents SQL injection when the value is interpolated into queries
|
||||
* (Postgres tsvector functions don't accept parameterized config names —
|
||||
* they must be literals or identifiers).
|
||||
*/
|
||||
|
||||
const VALID_CONFIG_NAME = /^[a-z][a-z0-9_]*$/;
|
||||
const DEFAULT_LANGUAGE = 'english';
|
||||
|
||||
let cachedLanguage: string | null = null;
|
||||
|
||||
/**
|
||||
* Returns the configured Postgres text search configuration name.
|
||||
*
|
||||
* Resolution order:
|
||||
* 1. process.env.GBRAIN_FTS_LANGUAGE (if set and valid)
|
||||
* 2. 'english' (default — preserves existing behavior)
|
||||
*
|
||||
* The return value is safe to interpolate directly into SQL because it
|
||||
* passes the VALID_CONFIG_NAME guard. If validation fails, falls back to
|
||||
* the default and emits a one-time warning.
|
||||
*
|
||||
* Cached on first call; reset with `resetFtsLanguageCache()` (test only).
|
||||
*/
|
||||
export function getFtsLanguage(): string {
|
||||
if (cachedLanguage !== null) return cachedLanguage;
|
||||
|
||||
const raw = process.env.GBRAIN_FTS_LANGUAGE?.trim();
|
||||
if (!raw) {
|
||||
cachedLanguage = DEFAULT_LANGUAGE;
|
||||
return cachedLanguage;
|
||||
}
|
||||
|
||||
if (!VALID_CONFIG_NAME.test(raw)) {
|
||||
console.warn(
|
||||
`[gbrain] Invalid GBRAIN_FTS_LANGUAGE='${raw}' — must match /^[a-z][a-z0-9_]*$/. ` +
|
||||
`Falling back to '${DEFAULT_LANGUAGE}'.`
|
||||
);
|
||||
cachedLanguage = DEFAULT_LANGUAGE;
|
||||
return cachedLanguage;
|
||||
}
|
||||
|
||||
cachedLanguage = raw;
|
||||
return cachedLanguage;
|
||||
}
|
||||
|
||||
/**
|
||||
* Resets the cached language. Tests only — don't use in production code.
|
||||
*/
|
||||
export function resetFtsLanguageCache(): void {
|
||||
cachedLanguage = null;
|
||||
}
|
||||
@@ -1,5 +1,6 @@
|
||||
import type { BrainEngine } from './engine.ts';
|
||||
import { slugifyPath } from './sync.ts';
|
||||
import { getFtsLanguage } from './fts-language.ts';
|
||||
|
||||
/**
|
||||
* Schema migrations — run automatically on initSchema().
|
||||
@@ -5505,6 +5506,96 @@ export const MIGRATIONS: Migration[] = [
|
||||
WHERE dimension IS NOT NULL;
|
||||
`,
|
||||
},
|
||||
{
|
||||
version: 123,
|
||||
name: 'configurable_fts_language',
|
||||
// Recreate the two search_vector trigger functions using the language
|
||||
// configured via GBRAIN_FTS_LANGUAGE (default 'english'). Idempotent:
|
||||
// CREATE OR REPLACE swaps the function body atomically; no trigger
|
||||
// recreation needed since the trigger references the function by name.
|
||||
//
|
||||
// Why a handler instead of a static SQL string: Postgres tsvector
|
||||
// functions don't accept parameterized config names — the language
|
||||
// must be a literal in the SQL. getFtsLanguage() validates the value
|
||||
// (lowercase letters/digits/underscores only) before interpolation.
|
||||
//
|
||||
// Function bodies mirror schema.sql / pglite-schema.ts exactly —
|
||||
// INCLUDING the `SET search_path = pg_catalog, public` hardening from
|
||||
// v120/#1647 (CREATE OR REPLACE resets proconfig, so omitting it here
|
||||
// would silently strip the hardening on every upgraded brain). Only
|
||||
// the text-search config name is parameterized. Keep all copies in
|
||||
// sync when the trigger logic changes.
|
||||
//
|
||||
// Backfill: after recreating the functions, re-tokenize existing rows
|
||||
// under the new language. Skipped when the configured language is
|
||||
// 'english' (trigger output identical — re-tokenizing is wasted I/O).
|
||||
// To change language after this migration has run, use
|
||||
// `gbrain reindex-search-vector`.
|
||||
sql: '',
|
||||
handler: async (engine) => {
|
||||
const lang = getFtsLanguage();
|
||||
|
||||
const recreatePagesFn = `
|
||||
CREATE OR REPLACE FUNCTION update_page_search_vector() RETURNS trigger SET search_path = pg_catalog, public AS $fn$
|
||||
DECLARE
|
||||
timeline_text TEXT;
|
||||
BEGIN
|
||||
SELECT coalesce(string_agg(summary || ' ' || detail, ' '), '')
|
||||
INTO timeline_text
|
||||
FROM timeline_entries
|
||||
WHERE page_id = NEW.id;
|
||||
|
||||
NEW.search_vector :=
|
||||
setweight(to_tsvector('${lang}', coalesce(NEW.title, '')), 'A') ||
|
||||
setweight(to_tsvector('${lang}', coalesce(NEW.compiled_truth, '')), 'B') ||
|
||||
setweight(to_tsvector('${lang}', coalesce(NEW.timeline, '')), 'C') ||
|
||||
setweight(to_tsvector('${lang}', coalesce(timeline_text, '')), 'C');
|
||||
|
||||
RETURN NEW;
|
||||
END;
|
||||
$fn$ LANGUAGE plpgsql;
|
||||
`;
|
||||
|
||||
const recreateChunksFn = `
|
||||
CREATE OR REPLACE FUNCTION update_chunk_search_vector() RETURNS TRIGGER SET search_path = pg_catalog, public AS $fn$
|
||||
BEGIN
|
||||
NEW.search_vector :=
|
||||
setweight(to_tsvector('${lang}', COALESCE(NEW.doc_comment, '')), 'A') ||
|
||||
setweight(to_tsvector('${lang}', COALESCE(NEW.symbol_name_qualified, '')), 'A') ||
|
||||
setweight(to_tsvector('${lang}', COALESCE(NEW.chunk_text, '')), 'B');
|
||||
RETURN NEW;
|
||||
END;
|
||||
$fn$ LANGUAGE plpgsql;
|
||||
`;
|
||||
|
||||
await engine.executeRaw(recreatePagesFn);
|
||||
await engine.executeRaw(recreateChunksFn);
|
||||
|
||||
if (lang === 'english') {
|
||||
console.log(` v123: trigger functions recreated with language='english' (default — no backfill needed)`);
|
||||
return;
|
||||
}
|
||||
|
||||
// Backfill existing rows under the new tokenizer. UPDATE-to-same-value
|
||||
// re-fires the pages trigger; chunks are rewritten directly with the
|
||||
// same expression as the trigger.
|
||||
await engine.executeRaw(`
|
||||
UPDATE pages SET id = id
|
||||
WHERE search_vector IS NOT NULL;
|
||||
`);
|
||||
|
||||
await engine.executeRaw(`
|
||||
UPDATE content_chunks
|
||||
SET search_vector =
|
||||
setweight(to_tsvector('${lang}', COALESCE(doc_comment, '')), 'A') ||
|
||||
setweight(to_tsvector('${lang}', COALESCE(symbol_name_qualified, '')), 'A') ||
|
||||
setweight(to_tsvector('${lang}', COALESCE(chunk_text, '')), 'B')
|
||||
WHERE search_vector IS NOT NULL;
|
||||
`);
|
||||
|
||||
console.log(` v123: trigger functions recreated with language='${lang}' + backfilled existing rows`);
|
||||
},
|
||||
},
|
||||
];
|
||||
|
||||
export const LATEST_VERSION = MIGRATIONS.length > 0
|
||||
|
||||
@@ -272,6 +272,8 @@ export function makeSubagentHandler(deps: SubagentDeps) {
|
||||
config,
|
||||
brainId: data.brain_id,
|
||||
allowedSlugPrefixes: data.allowed_slug_prefixes,
|
||||
// #1586: cycle-resolved source scope for tool-call OperationContexts.
|
||||
sourceId: data.source_id,
|
||||
});
|
||||
const toolDefs = data.allowed_tools && data.allowed_tools.length > 0
|
||||
? filterAllowedTools(registry, data.allowed_tools)
|
||||
|
||||
@@ -27,6 +27,7 @@ import type { GBrainConfig } from '../../config.ts';
|
||||
import { operations } from '../../operations.ts';
|
||||
import type { Operation, OperationContext } from '../../operations.ts';
|
||||
import { paramDefToSchema } from '../../../mcp/tool-defs.ts';
|
||||
import { validateSourceId } from '../../utils.ts';
|
||||
import type { ToolCtx, ToolDef } from '../types.ts';
|
||||
|
||||
/**
|
||||
@@ -201,6 +202,13 @@ export interface BuildBrainToolsOpts {
|
||||
* SubagentHandlerData.allowed_slug_prefixes via the handler.
|
||||
*/
|
||||
allowedSlugPrefixes?: readonly string[];
|
||||
/**
|
||||
* Brain source every tool-call OperationContext is scoped to (#1586).
|
||||
* Trusted (flows from SubagentHandlerData.source_id, which only
|
||||
* PROTECTED_JOB_NAMES-gated submitters can set); validated at build time.
|
||||
* Unset → legacy 'default'.
|
||||
*/
|
||||
sourceId?: string;
|
||||
}
|
||||
|
||||
interface OpContextDeps {
|
||||
@@ -211,6 +219,7 @@ interface OpContextDeps {
|
||||
signal?: AbortSignal;
|
||||
brainId?: string;
|
||||
allowedSlugPrefixes?: readonly string[];
|
||||
sourceId?: string;
|
||||
}
|
||||
|
||||
function buildOpContext(deps: OpContextDeps): OperationContext {
|
||||
@@ -224,7 +233,8 @@ function buildOpContext(deps: OpContextDeps): OperationContext {
|
||||
},
|
||||
dryRun: false,
|
||||
remote: true, // match MCP trust boundary for auto-link skip
|
||||
sourceId: 'default', // v0.34 D4: required; subagent tools default to host source
|
||||
// #1586: cycle-resolved source when provided; legacy host default else.
|
||||
sourceId: deps.sourceId ?? 'default',
|
||||
jobId: deps.jobId,
|
||||
subagentId: deps.subagentId,
|
||||
viaSubagent: true, // FAIL-CLOSED: put_page etc. enforce namespace
|
||||
@@ -248,6 +258,11 @@ export function buildBrainTools(opts: BuildBrainToolsOpts): ToolDef[] {
|
||||
op => BRAIN_TOOL_ALLOWLIST.has(op.name) && filter.has(op.name),
|
||||
);
|
||||
|
||||
// #1586: fail fast on a malformed source id before any tool executes
|
||||
// (defense-in-depth — the seam is trusted, but the value round-trips
|
||||
// through the job payload).
|
||||
if (opts.sourceId !== undefined) validateSourceId(opts.sourceId);
|
||||
|
||||
return picked.map<ToolDef>(op => {
|
||||
const schema = op.name === 'put_page'
|
||||
? namespacedPutPageSchema(op, opts.subagentId, opts.allowedSlugPrefixes)
|
||||
@@ -277,6 +292,7 @@ export function buildBrainTools(opts: BuildBrainToolsOpts): ToolDef[] {
|
||||
signal: ctx.signal,
|
||||
brainId: opts.brainId,
|
||||
allowedSlugPrefixes: opts.allowedSlugPrefixes,
|
||||
sourceId: opts.sourceId,
|
||||
});
|
||||
const params = (input && typeof input === 'object') ? input as Record<string, unknown> : {};
|
||||
return op.handler(opCtx, params);
|
||||
|
||||
@@ -455,6 +455,17 @@ export interface SubagentHandlerData {
|
||||
* and direct CLI submitters set it.
|
||||
*/
|
||||
allowed_slug_prefixes?: string[];
|
||||
/**
|
||||
* Brain source the subagent's tool calls are scoped to (#1586).
|
||||
*
|
||||
* When set, every tool-call `OperationContext.sourceId` uses this value
|
||||
* instead of the legacy 'default', so put_page writes land in the cycle's
|
||||
* resolved source. Same trust story as `allowed_slug_prefixes`:
|
||||
* PROTECTED_JOB_NAMES gates subagent submission, so only cycle.ts and
|
||||
* direct CLI submitters can set it. Validated via `validateSourceId` at
|
||||
* tool-registry build time.
|
||||
*/
|
||||
source_id?: string;
|
||||
/**
|
||||
* v0.41 Approach C: opt out of the auto-generated tool-usage preamble
|
||||
* that `buildSystemPrompt()` splices into `system`. Default behavior
|
||||
|
||||
@@ -24,6 +24,7 @@ import { PGLITE_SCHEMA_SQL, getPGLiteSchema } from './pglite-schema.ts';
|
||||
import { DEFAULT_EMBEDDING_MODEL, DEFAULT_EMBEDDING_DIMENSIONS } from './ai/defaults.ts';
|
||||
import { DELETE_BATCH_SIZE } from './engine-constants.ts';
|
||||
import { acquireLock, releaseLock, type LockHandle } from './pglite-lock.ts';
|
||||
import { getFtsLanguage } from './fts-language.ts';
|
||||
import type {
|
||||
Page, PageInput, PageFilters, PageType,
|
||||
Chunk, ChunkInput, StaleChunkRow, StalePageRow,
|
||||
@@ -1625,20 +1626,24 @@ export class PGLiteEngine implements BrainEngine {
|
||||
extraFilter += ` AND p.source_id = $${params.length}`;
|
||||
}
|
||||
|
||||
// FTS config name (e.g. 'english', 'pt_br'). Validated by getFtsLanguage()
|
||||
// — safe to interpolate into raw SQL.
|
||||
const ftsLang = getFtsLanguage();
|
||||
|
||||
const { rows } = await this.db.query(
|
||||
`WITH ranked AS (
|
||||
SELECT
|
||||
p.slug, p.id as page_id, p.title, p.type, p.source_id,
|
||||
p.effective_date, p.effective_date_source,
|
||||
cc.id as chunk_id, cc.chunk_index, cc.chunk_text, cc.chunk_source,
|
||||
ts_rank(cc.search_vector, websearch_to_tsquery('english', $1)) * ${sourceFactorCase} AS score,
|
||||
ts_rank(cc.search_vector, websearch_to_tsquery('${ftsLang}', $1)) * ${sourceFactorCase} AS score,
|
||||
CASE WHEN p.updated_at < (
|
||||
SELECT MAX(te.created_at) FROM timeline_entries te WHERE te.page_id = p.id
|
||||
) THEN true ELSE false END AS stale
|
||||
FROM content_chunks cc
|
||||
JOIN pages p ON p.id = cc.page_id
|
||||
JOIN sources s ON s.id = p.source_id
|
||||
WHERE cc.search_vector @@ websearch_to_tsquery('english', $1) ${detailFilter}${extraFilter} ${hardExcludeClause} ${visibilityClause}
|
||||
WHERE cc.search_vector @@ websearch_to_tsquery('${ftsLang}', $1) ${detailFilter}${extraFilter} ${hardExcludeClause} ${visibilityClause}
|
||||
-- v0.27.1: hide image rows from default text-keyword search so
|
||||
-- OCR text doesn't drown text-page hits. Image-similarity queries
|
||||
-- run a separate vector path on embedding_image.
|
||||
@@ -1857,20 +1862,23 @@ export class PGLiteEngine implements BrainEngine {
|
||||
}
|
||||
|
||||
// visibilityClause already declared above (v0.32.7: hoisted so CJK branch can reuse).
|
||||
// FTS config name (e.g. 'english', 'pt_br'). Validated by getFtsLanguage()
|
||||
// — safe to interpolate into raw SQL.
|
||||
const ftsLang = getFtsLanguage();
|
||||
|
||||
const { rows } = await this.db.query(
|
||||
`SELECT
|
||||
p.slug, p.id as page_id, p.title, p.type, p.source_id,
|
||||
p.effective_date, p.effective_date_source,
|
||||
cc.id as chunk_id, cc.chunk_index, cc.chunk_text, cc.chunk_source,
|
||||
ts_rank(cc.search_vector, websearch_to_tsquery('english', $1)) * ${sourceFactorCase} AS score,
|
||||
ts_rank(cc.search_vector, websearch_to_tsquery('${ftsLang}', $1)) * ${sourceFactorCase} AS score,
|
||||
CASE WHEN p.updated_at < (
|
||||
SELECT MAX(te.created_at) FROM timeline_entries te WHERE te.page_id = p.id
|
||||
) THEN true ELSE false END AS stale
|
||||
FROM content_chunks cc
|
||||
JOIN pages p ON p.id = cc.page_id
|
||||
JOIN sources s ON s.id = p.source_id
|
||||
WHERE cc.search_vector @@ websearch_to_tsquery('english', $1) ${detailFilter}${extraFilter} ${hardExcludeClause} ${visibilityClause}
|
||||
WHERE cc.search_vector @@ websearch_to_tsquery('${ftsLang}', $1) ${detailFilter}${extraFilter} ${hardExcludeClause} ${visibilityClause}
|
||||
ORDER BY score DESC
|
||||
LIMIT $2 OFFSET $3`,
|
||||
params
|
||||
|
||||
+428
-305
@@ -34,6 +34,7 @@ import {
|
||||
COLUMN_NAME_REGEX,
|
||||
EmbeddingColumnNotRegisteredError,
|
||||
} from './search/embedding-column.ts';
|
||||
import { getFtsLanguage } from './fts-language.ts';
|
||||
import type {
|
||||
Page, PageInput, PageFilters, PageType,
|
||||
Chunk, ChunkInput, StaleChunkRow, StalePageRow,
|
||||
@@ -160,6 +161,88 @@ export class PostgresEngine implements BrainEngine {
|
||||
return db.getConnection();
|
||||
}
|
||||
|
||||
// Source-scope binding for Postgres RLS — opt-in via env var.
|
||||
//
|
||||
// When `GBRAIN_RLS_SCOPE_BINDING` is set to `1` / `true`, source-scoped
|
||||
// query methods (listPages, search*, getChunks, etc.) wrap their queries
|
||||
// in a transaction that begins with
|
||||
// SELECT set_config('app.scopes', '<csv-of-allowed-source-ids>', true)
|
||||
// (equivalent to `SET LOCAL app.scopes = '<value>'`, but works through
|
||||
// parameterised SQL — `SET LOCAL` itself doesn't accept parameters)
|
||||
// so Postgres RLS policies on source-scoped tables can filter rows by
|
||||
// `current_setting('app.scopes', true)`. The expected policy shape:
|
||||
//
|
||||
// USING (current_setting('app.scopes', true) = '*'
|
||||
// OR source_id = ANY(string_to_array(
|
||||
// current_setting('app.scopes', true), ',')))
|
||||
//
|
||||
// Recommended runtime-role default:
|
||||
// ALTER ROLE <runtime-role> SET app.scopes = '*';
|
||||
// so admin / autopilot / cycle queries that don't pass scope info still
|
||||
// see all rows. OAuth-scoped requests override the default per
|
||||
// transaction with their allowed-source CSV.
|
||||
//
|
||||
// Default behavior (env var unset): the helper is a TRUE pass-through —
|
||||
// it calls `callback(this.sql)` with no transaction wrap and no
|
||||
// set_config, byte-identical to not having this helper at all. The only
|
||||
// exception is callers that pass `alwaysTransaction: true` (the search
|
||||
// methods, whose `SET LOCAL statement_timeout` already required a
|
||||
// transaction on master) — they keep exactly the `sql.begin()` wrap
|
||||
// they had before this helper existed. No read gains a new per-read
|
||||
// pool-hold when the flag is off (the #1794 PgBouncer-exhaustion class).
|
||||
//
|
||||
// Honest caveat: only the read paths that route through this helper are
|
||||
// backstopped by RLS. This is defense-in-depth layer 2; the app-layer
|
||||
// source filters (sourceScopeOpts) remain layer 1 and stay mandatory.
|
||||
private get rlsScopeBindingEnabled(): boolean {
|
||||
const v = process.env.GBRAIN_RLS_SCOPE_BINDING;
|
||||
return v === '1' || v === 'true';
|
||||
}
|
||||
|
||||
private async withScopedReadTransaction<T>(
|
||||
sourceIds: string[] | undefined,
|
||||
sourceId: string | undefined,
|
||||
callback: (tx: ReturnType<typeof postgres>) => Promise<T>,
|
||||
opts?: { alwaysTransaction?: boolean },
|
||||
): Promise<T> {
|
||||
// Flag off + no pre-existing transaction need: call through on the
|
||||
// shared pool exactly as master does. No tx round-trip, no pool slot
|
||||
// held for the duration of the read.
|
||||
if (!this.rlsScopeBindingEnabled && !opts?.alwaysTransaction) {
|
||||
return await callback(this.sql);
|
||||
}
|
||||
// Precedence matches sourceScopeOpts: federated array > scalar > '*'
|
||||
// (unscoped — relies on the recommended `ALTER ROLE ... SET
|
||||
// app.scopes = '*'` default, or on no policy being installed).
|
||||
let scopesValue = '*';
|
||||
if (sourceIds && sourceIds.length > 0) {
|
||||
scopesValue = sourceIds.join(',');
|
||||
} else if (sourceId) {
|
||||
scopesValue = sourceId;
|
||||
}
|
||||
// Note on nesting: a postgres.js transaction handle exposes
|
||||
// `.savepoint()` not `.begin()`, so callbacks must not try to open
|
||||
// their own `tx.begin()` inside this wrap — they'd fail with
|
||||
// `tx.begin is not a function`. Callbacks that need SET LOCAL emit it
|
||||
// directly on the handle (it shares this transaction).
|
||||
//
|
||||
// `sql.begin<T>(...)` returns `UnwrapPromiseArray<T>` in postgres.js's typings
|
||||
// — TypeScript strict-generics can't narrow that back to `T` for arbitrary
|
||||
// callback return shapes (TS2322). The unwrap is a no-op when the callback
|
||||
// returns a single value (not an array of promises), so the cast is safe.
|
||||
return (await this.sql.begin(async (tx: any) => {
|
||||
if (this.rlsScopeBindingEnabled) {
|
||||
// `SET LOCAL` doesn't accept parameters in PostgreSQL — using
|
||||
// `tx\`SET LOCAL ... = ${val}\`` binds val as $1 and errors with
|
||||
// `syntax error at or near "$1"`. set_config() is a regular function
|
||||
// and accepts a parameterised value; passing `true` as the third
|
||||
// argument makes it transaction-local (same scope as SET LOCAL).
|
||||
await tx`SELECT set_config('app.scopes', ${scopesValue}, true)`;
|
||||
}
|
||||
return await callback(tx as ReturnType<typeof postgres>);
|
||||
})) as T;
|
||||
}
|
||||
|
||||
// Lifecycle
|
||||
async connect(config: EngineConfig & { poolSize?: number; parentConnectionManager?: ConnectionManager }): Promise<void> {
|
||||
this._savedConfig = config;
|
||||
@@ -920,30 +1003,36 @@ export class PostgresEngine implements BrainEngine {
|
||||
|
||||
// Pages CRUD
|
||||
async getPage(slug: string, opts?: { sourceId?: string; sourceIds?: string[]; includeDeleted?: boolean }): Promise<Page | null> {
|
||||
const sql = this.sql;
|
||||
const includeDeleted = opts?.includeDeleted === true;
|
||||
const sourceId = opts?.sourceId;
|
||||
const sourceIds = opts?.sourceIds;
|
||||
// v0.26.5: default hides soft-deleted rows. Compose with optional source
|
||||
// filter via fragment chaining (postgres.js supports sql`` composition).
|
||||
// #1393: a federated grant (sourceIds[]) takes precedence over scalar
|
||||
// sourceId so the exact-match read honors allowedSources, not just one source.
|
||||
const sourceCondition =
|
||||
sourceIds && sourceIds.length > 0
|
||||
? sql`AND source_id = ANY(${sourceIds}::text[])`
|
||||
: sourceId
|
||||
? sql`AND source_id = ${sourceId}`
|
||||
: sql``;
|
||||
const deletedCondition = includeDeleted ? sql`` : sql`AND deleted_at IS NULL`;
|
||||
const rows = await sql`
|
||||
SELECT id, source_id, slug, type, title, compiled_truth, timeline, frontmatter, content_hash, created_at, updated_at, deleted_at,
|
||||
source_kind, source_uri, ingested_via, ingested_at
|
||||
FROM pages
|
||||
WHERE slug = ${slug} ${sourceCondition} ${deletedCondition}
|
||||
LIMIT 1
|
||||
`;
|
||||
if (rows.length === 0) return null;
|
||||
return rowToPage(rows[0]);
|
||||
// Two layers of defense:
|
||||
// 1. RLS scope binding (opt-in via GBRAIN_RLS_SCOPE_BINDING): wraps the
|
||||
// query in a transaction that sets `app.scopes` so the row-level
|
||||
// policy on `pages` filters at the SQL layer. Pass-through when off.
|
||||
// 2. App-layer source filter (#1393): a federated grant (sourceIds[])
|
||||
// takes precedence over scalar sourceId so the exact-match read
|
||||
// honors allowedSources, not just one source.
|
||||
return await this.withScopedReadTransaction(sourceIds, sourceId, async (tx) => {
|
||||
// v0.26.5: default hides soft-deleted rows. Compose with optional source
|
||||
// filter via fragment chaining (postgres.js supports sql`` composition).
|
||||
const sourceCondition =
|
||||
sourceIds && sourceIds.length > 0
|
||||
? tx`AND source_id = ANY(${sourceIds}::text[])`
|
||||
: sourceId
|
||||
? tx`AND source_id = ${sourceId}`
|
||||
: tx``;
|
||||
const deletedCondition = includeDeleted ? tx`` : tx`AND deleted_at IS NULL`;
|
||||
const rows = await tx`
|
||||
SELECT id, source_id, slug, type, title, compiled_truth, timeline, frontmatter, content_hash, created_at, updated_at, deleted_at,
|
||||
source_kind, source_uri, ingested_via, ingested_at
|
||||
FROM pages
|
||||
WHERE slug = ${slug} ${sourceCondition} ${deletedCondition}
|
||||
LIMIT 1
|
||||
`;
|
||||
if (rows.length === 0) return null;
|
||||
return rowToPage(rows[0]);
|
||||
});
|
||||
}
|
||||
|
||||
/**
|
||||
@@ -954,19 +1043,21 @@ export class PostgresEngine implements BrainEngine {
|
||||
sourceId: string,
|
||||
opts: { hash: string; frontmatterId?: string | null },
|
||||
): Promise<{ slug: string; id: number } | null> {
|
||||
const sql = this.sql;
|
||||
const fmId = opts.frontmatterId ?? null;
|
||||
const rows = await sql`
|
||||
SELECT id, slug FROM pages
|
||||
WHERE source_id = ${sourceId}
|
||||
AND deleted_at IS NULL
|
||||
AND (content_hash = ${opts.hash} OR (frontmatter->>'id' = ${fmId} AND ${fmId}::text IS NOT NULL))
|
||||
ORDER BY id
|
||||
LIMIT 1
|
||||
`;
|
||||
if (rows.length === 0) return null;
|
||||
const r = rows[0] as { id: number | string; slug: string };
|
||||
return { slug: r.slug, id: Number(r.id) };
|
||||
// RLS scope binding: sourceId is positional here.
|
||||
return await this.withScopedReadTransaction(undefined, sourceId, async (tx) => {
|
||||
const rows = await tx`
|
||||
SELECT id, slug FROM pages
|
||||
WHERE source_id = ${sourceId}
|
||||
AND deleted_at IS NULL
|
||||
AND (content_hash = ${opts.hash} OR (frontmatter->>'id' = ${fmId} AND ${fmId}::text IS NOT NULL))
|
||||
ORDER BY id
|
||||
LIMIT 1
|
||||
`;
|
||||
if (rows.length === 0) return null;
|
||||
const r = rows[0] as { id: number | string; slug: string };
|
||||
return { slug: r.slug, id: Number(r.id) };
|
||||
});
|
||||
}
|
||||
|
||||
async putPage(slug: string, page: PageInput, opts?: { sourceId?: string }): Promise<Page> {
|
||||
@@ -1235,25 +1326,31 @@ export class PostgresEngine implements BrainEngine {
|
||||
const sortKey = filters?.sort && PAGE_SORT_SQL[filters.sort] ? filters.sort : 'updated_desc';
|
||||
const orderBy = sql.unsafe(PAGE_SORT_SQL[sortKey]);
|
||||
|
||||
const rows = await sql`
|
||||
SELECT p.* FROM pages p
|
||||
${tagJoin}
|
||||
WHERE 1=1 ${typeCondition} ${tagCondition} ${updatedCondition} ${slugCondition} ${sourceCondition} ${deletedCondition}
|
||||
ORDER BY ${orderBy} LIMIT ${limit} OFFSET ${offset}
|
||||
`;
|
||||
|
||||
return rows.map(rowToPage);
|
||||
// RLS scope binding (opt-in via GBRAIN_RLS_SCOPE_BINDING): when
|
||||
// enabled, this wraps the query in a transaction that sets
|
||||
// `app.scopes` from filters; when disabled, it's a pass-through.
|
||||
return await this.withScopedReadTransaction(filters?.sourceIds, filters?.sourceId, async (tx) => {
|
||||
const rows = await tx`
|
||||
SELECT p.* FROM pages p
|
||||
${tagJoin}
|
||||
WHERE 1=1 ${typeCondition} ${tagCondition} ${updatedCondition} ${slugCondition} ${sourceCondition} ${deletedCondition}
|
||||
ORDER BY ${orderBy} LIMIT ${limit} OFFSET ${offset}
|
||||
`;
|
||||
return rows.map(rowToPage);
|
||||
});
|
||||
}
|
||||
|
||||
async getAllSlugs(opts?: { sourceId?: string }): Promise<Set<string>> {
|
||||
const sql = this.sql;
|
||||
// v0.31.8 (D12): two-branch. See pglite-engine.ts:getAllSlugs for context.
|
||||
if (opts?.sourceId) {
|
||||
const rows = await sql`SELECT slug FROM pages WHERE source_id = ${opts.sourceId}`;
|
||||
return new Set(rows.map((r) => r.slug as string));
|
||||
}
|
||||
const rows = await sql`SELECT slug FROM pages`;
|
||||
return new Set(rows.map((r) => r.slug as string));
|
||||
// RLS scope binding (opt-in via GBRAIN_RLS_SCOPE_BINDING).
|
||||
return await this.withScopedReadTransaction(undefined, opts?.sourceId, async (tx) => {
|
||||
// v0.31.8 (D12): two-branch. See pglite-engine.ts:getAllSlugs for context.
|
||||
if (opts?.sourceId) {
|
||||
const rows = await tx`SELECT slug FROM pages WHERE source_id = ${opts.sourceId}`;
|
||||
return new Set(rows.map((r: Record<string, unknown>) => r.slug as string));
|
||||
}
|
||||
const rows = await tx`SELECT slug FROM pages`;
|
||||
return new Set(rows.map((r: Record<string, unknown>) => r.slug as string));
|
||||
});
|
||||
}
|
||||
|
||||
async listAllPageRefs(): Promise<Array<{ slug: string; source_id: string }>> {
|
||||
@@ -1368,7 +1465,6 @@ export class PostgresEngine implements BrainEngine {
|
||||
// 2. connection_count DESC — structural-centrality tiebreaker (D10)
|
||||
// 3. slug ASC — deterministic for tests
|
||||
async listPrefixSampledPages(opts: DomainBankSampleOpts): Promise<DomainBankRow[]> {
|
||||
const sql = this.sql;
|
||||
if (opts.prefixes.length === 0) return [];
|
||||
const exclude = opts.excludeSlugs ?? [];
|
||||
const staleBias = opts.staleBias === true;
|
||||
@@ -1376,7 +1472,9 @@ export class PostgresEngine implements BrainEngine {
|
||||
// Source scoping (D5, codex r2 #2 — federated array wins over scalar).
|
||||
const sourceIds = opts.sourceIds ?? null;
|
||||
const sourceId = opts.sourceId ?? null;
|
||||
const rows = await sql`
|
||||
// RLS scope binding (opt-in via GBRAIN_RLS_SCOPE_BINDING).
|
||||
return await this.withScopedReadTransaction(opts.sourceIds, opts.sourceId, async (tx) => {
|
||||
const rows = await tx`
|
||||
WITH prefix_pages AS (
|
||||
SELECT
|
||||
p.id AS page_id,
|
||||
@@ -1443,36 +1541,42 @@ export class PostgresEngine implements BrainEngine {
|
||||
FROM with_chunk
|
||||
ORDER BY prefix
|
||||
`;
|
||||
return rows.map((r): DomainBankRow => ({
|
||||
slug: r.slug as string,
|
||||
source_id: r.source_id as string,
|
||||
prefix: r.prefix as string | null,
|
||||
page_id: Number(r.page_id),
|
||||
title: r.title as string | null,
|
||||
compiled_truth: (r.compiled_truth as string | null) ?? '',
|
||||
connection_count: Number(r.connection_count),
|
||||
last_retrieved_at: r.last_retrieved_at as Date | null,
|
||||
representative_chunk_id: r.representative_chunk_id == null ? null : Number(r.representative_chunk_id),
|
||||
}));
|
||||
return rows.map((r: Record<string, unknown>): DomainBankRow => ({
|
||||
slug: r.slug as string,
|
||||
source_id: r.source_id as string,
|
||||
prefix: r.prefix as string | null,
|
||||
page_id: Number(r.page_id),
|
||||
title: r.title as string | null,
|
||||
compiled_truth: (r.compiled_truth as string | null) ?? '',
|
||||
connection_count: Number(r.connection_count),
|
||||
last_retrieved_at: r.last_retrieved_at as Date | null,
|
||||
representative_chunk_id: r.representative_chunk_id == null ? null : Number(r.representative_chunk_id),
|
||||
}));
|
||||
});
|
||||
}
|
||||
|
||||
// v0.37.0 — corpus-sampling fallback when prefix-stratified can't fill M.
|
||||
// Deterministic with opts.seed (setseed before SELECT); random otherwise.
|
||||
async listCorpusSample(opts: CorpusSampleOpts): Promise<DomainBankRow[]> {
|
||||
const sql = this.sql;
|
||||
if (opts.n <= 0) return [];
|
||||
const exclude = opts.excludeSlugs ?? [];
|
||||
const sourceIds = opts.sourceIds ?? null;
|
||||
const sourceId = opts.sourceId ?? null;
|
||||
// setseed deterministic path: use SELECT setseed($1) + RANDOM(). PGLite/Postgres
|
||||
// both honor setseed for the same session/transaction. For tests this gives
|
||||
// identical ordering across runs.
|
||||
if (typeof opts.seed === 'number') {
|
||||
// Clamp to [-1, 1] required by setseed.
|
||||
const clamped = Math.max(-1, Math.min(1, opts.seed));
|
||||
await sql`SELECT setseed(${clamped}::float8)`;
|
||||
}
|
||||
const rows = await sql`
|
||||
// RLS scope binding (opt-in via GBRAIN_RLS_SCOPE_BINDING).
|
||||
// alwaysTransaction when seeded: setseed() only affects RANDOM() on the
|
||||
// SAME connection. On a pool, a bare `sql\`SELECT setseed(...)\`` and the
|
||||
// subsequent SELECT can land on different connections, silently breaking
|
||||
// the deterministic path — the transaction pins both to one connection.
|
||||
return await this.withScopedReadTransaction(opts.sourceIds, opts.sourceId, async (tx) => {
|
||||
// setseed deterministic path: use SELECT setseed($1) + RANDOM(). PGLite/Postgres
|
||||
// both honor setseed for the same session/transaction. For tests this gives
|
||||
// identical ordering across runs.
|
||||
if (typeof opts.seed === 'number') {
|
||||
// Clamp to [-1, 1] required by setseed.
|
||||
const clamped = Math.max(-1, Math.min(1, opts.seed));
|
||||
await tx`SELECT setseed(${clamped}::float8)`;
|
||||
}
|
||||
const rows = await tx`
|
||||
WITH sampled AS (
|
||||
SELECT
|
||||
p.id AS page_id,
|
||||
@@ -1504,17 +1608,18 @@ export class PostgresEngine implements BrainEngine {
|
||||
) AS representative_chunk_id
|
||||
FROM sampled s
|
||||
`;
|
||||
return rows.map((r): DomainBankRow => ({
|
||||
slug: r.slug as string,
|
||||
source_id: r.source_id as string,
|
||||
prefix: r.prefix as string | null,
|
||||
page_id: Number(r.page_id),
|
||||
title: r.title as string | null,
|
||||
compiled_truth: (r.compiled_truth as string | null) ?? '',
|
||||
connection_count: Number(r.connection_count),
|
||||
last_retrieved_at: r.last_retrieved_at as Date | null,
|
||||
representative_chunk_id: r.representative_chunk_id == null ? null : Number(r.representative_chunk_id),
|
||||
}));
|
||||
return rows.map((r: Record<string, unknown>): DomainBankRow => ({
|
||||
slug: r.slug as string,
|
||||
source_id: r.source_id as string,
|
||||
prefix: r.prefix as string | null,
|
||||
page_id: Number(r.page_id),
|
||||
title: r.title as string | null,
|
||||
compiled_truth: (r.compiled_truth as string | null) ?? '',
|
||||
connection_count: Number(r.connection_count),
|
||||
last_retrieved_at: r.last_retrieved_at as Date | null,
|
||||
representative_chunk_id: r.representative_chunk_id == null ? null : Number(r.representative_chunk_id),
|
||||
}));
|
||||
}, { alwaysTransaction: typeof opts.seed === 'number' });
|
||||
}
|
||||
|
||||
async resolveSlugs(partial: string, opts?: { sourceId?: string; sourceIds?: string[] }): Promise<string[]> {
|
||||
@@ -1555,7 +1660,6 @@ export class PostgresEngine implements BrainEngine {
|
||||
// list_pages etc. see zero breaking changes. A2 two-pass (Layer 7)
|
||||
// consumes searchKeywordChunks for the raw chunk-grain primitive.
|
||||
async searchKeyword(query: string, opts?: SearchOpts): Promise<SearchResult[]> {
|
||||
const sql = this.sql;
|
||||
const limit = clampSearchLimit(opts?.limit);
|
||||
const offset = opts?.offset || 0;
|
||||
const type = opts?.type;
|
||||
@@ -1653,6 +1757,9 @@ export class PostgresEngine implements BrainEngine {
|
||||
// column lookup. NOT bypassed by detail=high — soft-delete is a contract,
|
||||
// not a temporal preference.
|
||||
const visibilityClause = buildVisibilityClause('p', 's');
|
||||
// FTS config name (e.g. 'english', 'pt_br'). Validated by getFtsLanguage()
|
||||
// — safe to interpolate into raw SQL.
|
||||
const ftsLang = getFtsLanguage();
|
||||
|
||||
const rawQuery = `
|
||||
WITH ranked_chunks AS (
|
||||
@@ -1660,11 +1767,11 @@ export class PostgresEngine implements BrainEngine {
|
||||
p.slug, p.id as page_id, p.title, p.type, p.source_id,
|
||||
p.effective_date, p.effective_date_source,
|
||||
cc.id as chunk_id, cc.chunk_index, cc.chunk_text, cc.chunk_source,
|
||||
ts_rank(cc.search_vector, websearch_to_tsquery('english', $1)) * ${sourceFactorCase} AS score
|
||||
ts_rank(cc.search_vector, websearch_to_tsquery('${ftsLang}', $1)) * ${sourceFactorCase} AS score
|
||||
FROM content_chunks cc
|
||||
JOIN pages p ON p.id = cc.page_id
|
||||
JOIN sources s ON s.id = p.source_id
|
||||
WHERE cc.search_vector @@ websearch_to_tsquery('english', $1)
|
||||
WHERE cc.search_vector @@ websearch_to_tsquery('${ftsLang}', $1)
|
||||
${typeClause}
|
||||
${typesClause}
|
||||
${excludeSlugsClause}
|
||||
@@ -1694,12 +1801,16 @@ export class PostgresEngine implements BrainEngine {
|
||||
OFFSET ${offsetParam}
|
||||
`;
|
||||
|
||||
// Search-only timeout. SET LOCAL inside sql.begin() scopes the GUC
|
||||
// to the transaction so it can never leak onto a pooled connection.
|
||||
const rows = await sql.begin(async sql => {
|
||||
await sql`SET LOCAL statement_timeout = '8s'`;
|
||||
return await sql.unsafe(rawQuery, params as Parameters<typeof sql.unsafe>[1]);
|
||||
});
|
||||
// RLS scope binding (opt-in via GBRAIN_RLS_SCOPE_BINDING) + search-only
|
||||
// timeout. alwaysTransaction: this method needed sql.begin() on master
|
||||
// already (SET LOCAL statement_timeout must be transaction-scoped so
|
||||
// the GUC can never leak onto a pooled connection). Flag off → the
|
||||
// wrap is identical to master's; flag on → set_config('app.scopes')
|
||||
// shares the same transaction as the timeout.
|
||||
const rows = await this.withScopedReadTransaction(opts?.sourceIds, opts?.sourceId, async (tx) => {
|
||||
await tx`SET LOCAL statement_timeout = '8s'`;
|
||||
return await tx.unsafe(rawQuery, params as Parameters<typeof tx.unsafe>[1]);
|
||||
}, { alwaysTransaction: true });
|
||||
return rows.map(rowToSearchResult);
|
||||
}
|
||||
|
||||
@@ -1713,7 +1824,6 @@ export class PostgresEngine implements BrainEngine {
|
||||
* contract). This is intentionally a narrow internal knob.
|
||||
*/
|
||||
async searchKeywordChunks(query: string, opts?: SearchOpts): Promise<SearchResult[]> {
|
||||
const sql = this.sql;
|
||||
const limit = clampSearchLimit(opts?.limit);
|
||||
const offset = opts?.offset || 0;
|
||||
const type = opts?.type;
|
||||
@@ -1792,18 +1902,21 @@ export class PostgresEngine implements BrainEngine {
|
||||
|
||||
// v0.26.5: visibility filter for searchKeywordChunks (anchor primitive).
|
||||
const visibilityClause = buildVisibilityClause('p', 's');
|
||||
// FTS config name (e.g. 'english', 'pt_br'). Validated by getFtsLanguage()
|
||||
// — safe to interpolate into raw SQL.
|
||||
const ftsLang = getFtsLanguage();
|
||||
|
||||
const rawQuery = `
|
||||
SELECT
|
||||
p.slug, p.id as page_id, p.title, p.type, p.source_id,
|
||||
p.effective_date, p.effective_date_source,
|
||||
cc.id as chunk_id, cc.chunk_index, cc.chunk_text, cc.chunk_source,
|
||||
ts_rank(cc.search_vector, websearch_to_tsquery('english', $1)) * ${sourceFactorCase} AS score,
|
||||
ts_rank(cc.search_vector, websearch_to_tsquery('${ftsLang}', $1)) * ${sourceFactorCase} AS score,
|
||||
false AS stale
|
||||
FROM content_chunks cc
|
||||
JOIN pages p ON p.id = cc.page_id
|
||||
JOIN sources s ON s.id = p.source_id
|
||||
WHERE cc.search_vector @@ websearch_to_tsquery('english', $1)
|
||||
WHERE cc.search_vector @@ websearch_to_tsquery('${ftsLang}', $1)
|
||||
${typeClause}
|
||||
${typesClause}
|
||||
${excludeSlugsClause}
|
||||
@@ -1820,15 +1933,17 @@ export class PostgresEngine implements BrainEngine {
|
||||
OFFSET ${offsetParam}
|
||||
`;
|
||||
|
||||
const rows = await sql.begin(async sql => {
|
||||
await sql`SET LOCAL statement_timeout = '8s'`;
|
||||
return await sql.unsafe(rawQuery, params as Parameters<typeof sql.unsafe>[1]);
|
||||
});
|
||||
// RLS scope binding + search-only timeout. alwaysTransaction: master
|
||||
// already wrapped this in sql.begin() for the SET LOCAL; flag off is
|
||||
// identical to that wrap, flag on adds set_config in the same tx.
|
||||
const rows = await this.withScopedReadTransaction(opts?.sourceIds, opts?.sourceId, async (tx) => {
|
||||
await tx`SET LOCAL statement_timeout = '8s'`;
|
||||
return await tx.unsafe(rawQuery, params as Parameters<typeof tx.unsafe>[1]);
|
||||
}, { alwaysTransaction: true });
|
||||
return rows.map(rowToSearchResult);
|
||||
}
|
||||
|
||||
async searchVector(embedding: Float32Array, opts?: SearchOpts): Promise<SearchResult[]> {
|
||||
const sql = this.sql;
|
||||
const limit = clampSearchLimit(opts?.limit);
|
||||
const offset = opts?.offset || 0;
|
||||
const type = opts?.type;
|
||||
@@ -1992,10 +2107,13 @@ export class PostgresEngine implements BrainEngine {
|
||||
OFFSET ${offsetParam}
|
||||
`;
|
||||
|
||||
const rows = await sql.begin(async sql => {
|
||||
await sql`SET LOCAL statement_timeout = '8s'`;
|
||||
return await sql.unsafe(rawQuery, params as Parameters<typeof sql.unsafe>[1]);
|
||||
});
|
||||
// RLS scope binding + search-only timeout. alwaysTransaction: master
|
||||
// already wrapped this in sql.begin() for the SET LOCAL; flag off is
|
||||
// identical to that wrap, flag on adds set_config in the same tx.
|
||||
const rows = await this.withScopedReadTransaction(opts?.sourceIds, opts?.sourceId, async (tx) => {
|
||||
await tx`SET LOCAL statement_timeout = '8s'`;
|
||||
return await tx.unsafe(rawQuery, params as Parameters<typeof tx.unsafe>[1]);
|
||||
}, { alwaysTransaction: true });
|
||||
return rows.map(rowToSearchResult);
|
||||
}
|
||||
|
||||
@@ -2238,15 +2356,17 @@ export class PostgresEngine implements BrainEngine {
|
||||
}
|
||||
|
||||
async getChunks(slug: string, opts?: { sourceId?: string }): Promise<Chunk[]> {
|
||||
const sql = this.sql;
|
||||
const sourceId = opts?.sourceId ?? 'default';
|
||||
const rows = await sql`
|
||||
SELECT cc.* FROM content_chunks cc
|
||||
JOIN pages p ON p.id = cc.page_id
|
||||
WHERE p.slug = ${slug} AND p.source_id = ${sourceId}
|
||||
ORDER BY cc.chunk_index
|
||||
`;
|
||||
return rows.map((r) => rowToChunk(r as Record<string, unknown>));
|
||||
// RLS scope binding (opt-in via GBRAIN_RLS_SCOPE_BINDING).
|
||||
return await this.withScopedReadTransaction(undefined, sourceId, async (tx) => {
|
||||
const rows = await tx`
|
||||
SELECT cc.* FROM content_chunks cc
|
||||
JOIN pages p ON p.id = cc.page_id
|
||||
WHERE p.slug = ${slug} AND p.source_id = ${sourceId}
|
||||
ORDER BY cc.chunk_index
|
||||
`;
|
||||
return rows.map((r: Record<string, unknown>) => rowToChunk(r));
|
||||
});
|
||||
}
|
||||
|
||||
/**
|
||||
@@ -2277,14 +2397,17 @@ export class PostgresEngine implements BrainEngine {
|
||||
// D7: source_id scoping. v0.41.31: optional signature widens staleness
|
||||
// to embedding_signature drift (NULL grandfathered).
|
||||
const { where, params } = this.buildStaleChunkWhere(opts);
|
||||
const rows = await this.sql.unsafe(
|
||||
`SELECT count(*)::int AS count
|
||||
FROM content_chunks cc
|
||||
JOIN pages p ON p.id = cc.page_id
|
||||
WHERE ${where}`,
|
||||
params as Parameters<typeof this.sql.unsafe>[1],
|
||||
);
|
||||
return Number((rows[0] as { count?: number } | undefined)?.count ?? 0);
|
||||
// RLS scope binding (opt-in via GBRAIN_RLS_SCOPE_BINDING).
|
||||
return await this.withScopedReadTransaction(undefined, opts?.sourceId, async (tx) => {
|
||||
const rows = await tx.unsafe(
|
||||
`SELECT count(*)::int AS count
|
||||
FROM content_chunks cc
|
||||
JOIN pages p ON p.id = cc.page_id
|
||||
WHERE ${where}`,
|
||||
params as Parameters<typeof tx.unsafe>[1],
|
||||
);
|
||||
return Number((rows[0] as { count?: number } | undefined)?.count ?? 0);
|
||||
});
|
||||
}
|
||||
|
||||
async sumStaleChunkChars(opts?: { sourceId?: string; signature?: string }): Promise<number> {
|
||||
@@ -2341,41 +2464,67 @@ export class PostgresEngine implements BrainEngine {
|
||||
orderBy?: 'page_id' | 'updated_desc';
|
||||
afterUpdatedAt?: string | null;
|
||||
}): Promise<StaleChunkRow[]> {
|
||||
const sql = this.sql;
|
||||
const limit = opts?.batchSize ?? 2000;
|
||||
const afterPid = opts?.afterPageId ?? 0;
|
||||
const afterIdx = opts?.afterChunkIndex ?? -1;
|
||||
const orderBy = opts?.orderBy ?? 'page_id';
|
||||
|
||||
// v0.41.18.0 (A13, codex #9): --priority recent path. Composite cursor
|
||||
// (updated_at DESC NULLS LAST, page_id ASC, chunk_index ASC). Backed by
|
||||
// idx_pages_updated_at_desc + content_chunks_stale_idx partial.
|
||||
// "Next row" semantic with DESC NULLS LAST + ASC tiebreakers is:
|
||||
// (updated_at < prev) OR
|
||||
// (updated_at = prev AND page_id > prev_page_id) OR
|
||||
// (updated_at = prev AND page_id = prev_page_id AND chunk_index > prev_chunk_index)
|
||||
// First call: afterUpdatedAt undefined → returns the highest updated_at rows.
|
||||
if (orderBy === 'updated_desc') {
|
||||
const afterUpdated = opts?.afterUpdatedAt ?? null;
|
||||
const isFirstPage = afterUpdated === null && afterPid === 0;
|
||||
if (opts?.sourceId === undefined) {
|
||||
const rows = isFirstPage ? await sql`
|
||||
// RLS scope binding (opt-in via GBRAIN_RLS_SCOPE_BINDING).
|
||||
return await this.withScopedReadTransaction(undefined, opts?.sourceId, async (tx) => {
|
||||
// v0.41.18.0 (A13, codex #9): --priority recent path. Composite cursor
|
||||
// (updated_at DESC NULLS LAST, page_id ASC, chunk_index ASC). Backed by
|
||||
// idx_pages_updated_at_desc + content_chunks_stale_idx partial.
|
||||
if (orderBy === 'updated_desc') {
|
||||
const afterUpdated = opts?.afterUpdatedAt ?? null;
|
||||
const isFirstPage = afterUpdated === null && afterPid === 0;
|
||||
if (opts?.sourceId === undefined) {
|
||||
const rows = isFirstPage ? await tx`
|
||||
SELECT p.slug, cc.chunk_index, cc.chunk_text, cc.chunk_source,
|
||||
cc.model, cc.token_count, p.source_id, cc.page_id,
|
||||
p.updated_at
|
||||
FROM content_chunks cc
|
||||
JOIN pages p ON p.id = cc.page_id
|
||||
WHERE cc.embedding IS NULL
|
||||
AND NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'embed_skip')
|
||||
ORDER BY p.updated_at DESC NULLS LAST, p.id ASC, cc.chunk_index ASC
|
||||
LIMIT ${limit}
|
||||
` : await tx`
|
||||
SELECT p.slug, cc.chunk_index, cc.chunk_text, cc.chunk_source,
|
||||
cc.model, cc.token_count, p.source_id, cc.page_id,
|
||||
p.updated_at
|
||||
FROM content_chunks cc
|
||||
JOIN pages p ON p.id = cc.page_id
|
||||
WHERE cc.embedding IS NULL
|
||||
AND NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'embed_skip')
|
||||
AND (
|
||||
p.updated_at < ${afterUpdated}::timestamptz
|
||||
OR (p.updated_at = ${afterUpdated}::timestamptz AND p.id > ${afterPid})
|
||||
OR (p.updated_at = ${afterUpdated}::timestamptz AND p.id = ${afterPid} AND cc.chunk_index > ${afterIdx})
|
||||
)
|
||||
ORDER BY p.updated_at DESC NULLS LAST, p.id ASC, cc.chunk_index ASC
|
||||
LIMIT ${limit}
|
||||
`;
|
||||
return rows as unknown as StaleChunkRow[];
|
||||
}
|
||||
const rows = isFirstPage ? await tx`
|
||||
SELECT p.slug, cc.chunk_index, cc.chunk_text, cc.chunk_source,
|
||||
cc.model, cc.token_count, p.source_id, cc.page_id,
|
||||
p.updated_at
|
||||
FROM content_chunks cc
|
||||
JOIN pages p ON p.id = cc.page_id
|
||||
WHERE cc.embedding IS NULL
|
||||
AND p.source_id = ${opts.sourceId}
|
||||
AND NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'embed_skip')
|
||||
ORDER BY p.updated_at DESC NULLS LAST, p.id ASC, cc.chunk_index ASC
|
||||
LIMIT ${limit}
|
||||
` : await sql`
|
||||
` : await tx`
|
||||
SELECT p.slug, cc.chunk_index, cc.chunk_text, cc.chunk_source,
|
||||
cc.model, cc.token_count, p.source_id, cc.page_id,
|
||||
p.updated_at
|
||||
FROM content_chunks cc
|
||||
JOIN pages p ON p.id = cc.page_id
|
||||
WHERE cc.embedding IS NULL
|
||||
AND p.source_id = ${opts.sourceId}
|
||||
AND NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'embed_skip')
|
||||
AND (
|
||||
p.updated_at < ${afterUpdated}::timestamptz
|
||||
@@ -2387,77 +2536,35 @@ export class PostgresEngine implements BrainEngine {
|
||||
`;
|
||||
return rows as unknown as StaleChunkRow[];
|
||||
}
|
||||
const rows = isFirstPage ? await sql`
|
||||
SELECT p.slug, cc.chunk_index, cc.chunk_text, cc.chunk_source,
|
||||
cc.model, cc.token_count, p.source_id, cc.page_id,
|
||||
p.updated_at
|
||||
FROM content_chunks cc
|
||||
JOIN pages p ON p.id = cc.page_id
|
||||
WHERE cc.embedding IS NULL
|
||||
AND p.source_id = ${opts.sourceId}
|
||||
AND NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'embed_skip')
|
||||
ORDER BY p.updated_at DESC NULLS LAST, p.id ASC, cc.chunk_index ASC
|
||||
LIMIT ${limit}
|
||||
` : await sql`
|
||||
SELECT p.slug, cc.chunk_index, cc.chunk_text, cc.chunk_source,
|
||||
cc.model, cc.token_count, p.source_id, cc.page_id,
|
||||
p.updated_at
|
||||
FROM content_chunks cc
|
||||
JOIN pages p ON p.id = cc.page_id
|
||||
WHERE cc.embedding IS NULL
|
||||
AND p.source_id = ${opts.sourceId}
|
||||
AND NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'embed_skip')
|
||||
AND (
|
||||
p.updated_at < ${afterUpdated}::timestamptz
|
||||
OR (p.updated_at = ${afterUpdated}::timestamptz AND p.id > ${afterPid})
|
||||
OR (p.updated_at = ${afterUpdated}::timestamptz AND p.id = ${afterPid} AND cc.chunk_index > ${afterIdx})
|
||||
)
|
||||
ORDER BY p.updated_at DESC NULLS LAST, p.id ASC, cc.chunk_index ASC
|
||||
LIMIT ${limit}
|
||||
`;
|
||||
return rows as unknown as StaleChunkRow[];
|
||||
}
|
||||
// orderBy === 'page_id' — legacy stable cursor (unchanged below).
|
||||
// Cursor-paginated: keyset pagination on (page_id, chunk_index).
|
||||
// The partial index idx_chunks_embedding_null makes the WHERE fast;
|
||||
// LIMIT keeps each round-trip well within statement_timeout.
|
||||
//
|
||||
// D7: optional source_id filter. NULL/undefined = scan all sources
|
||||
// (pre-existing behavior); a value scopes to that source so
|
||||
// `gbrain embed --stale --source X` actually does what it says.
|
||||
//
|
||||
// v0.41 (D4+D8): NOT (frontmatter ? 'embed_skip') filter applied via
|
||||
// the always-JOINed pages row. Soft-blocked pages won't surface in
|
||||
// the stale list; their chunks were deleted at ingest time anyway
|
||||
// (D9 transition invariant), but the filter is defense-in-depth for
|
||||
// pre-fix inventory that might still have orphan chunks.
|
||||
if (opts?.sourceId === undefined) {
|
||||
const rows = await sql`
|
||||
// orderBy === 'page_id' — legacy stable cursor.
|
||||
if (opts?.sourceId === undefined) {
|
||||
const rows = await tx`
|
||||
SELECT p.slug, cc.chunk_index, cc.chunk_text, cc.chunk_source,
|
||||
cc.model, cc.token_count, p.source_id, cc.page_id
|
||||
FROM content_chunks cc
|
||||
JOIN pages p ON p.id = cc.page_id
|
||||
WHERE cc.embedding IS NULL
|
||||
AND NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'embed_skip')
|
||||
AND (cc.page_id, cc.chunk_index) > (${afterPid}, ${afterIdx})
|
||||
ORDER BY cc.page_id, cc.chunk_index
|
||||
LIMIT ${limit}
|
||||
`;
|
||||
return rows as unknown as StaleChunkRow[];
|
||||
}
|
||||
const rows = await tx`
|
||||
SELECT p.slug, cc.chunk_index, cc.chunk_text, cc.chunk_source,
|
||||
cc.model, cc.token_count, p.source_id, cc.page_id
|
||||
FROM content_chunks cc
|
||||
JOIN pages p ON p.id = cc.page_id
|
||||
WHERE cc.embedding IS NULL
|
||||
AND p.source_id = ${opts.sourceId}
|
||||
AND NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'embed_skip')
|
||||
AND (cc.page_id, cc.chunk_index) > (${afterPid}, ${afterIdx})
|
||||
ORDER BY cc.page_id, cc.chunk_index
|
||||
LIMIT ${limit}
|
||||
`;
|
||||
return rows as unknown as StaleChunkRow[];
|
||||
}
|
||||
const rows = await sql`
|
||||
SELECT p.slug, cc.chunk_index, cc.chunk_text, cc.chunk_source,
|
||||
cc.model, cc.token_count, p.source_id, cc.page_id
|
||||
FROM content_chunks cc
|
||||
JOIN pages p ON p.id = cc.page_id
|
||||
WHERE cc.embedding IS NULL
|
||||
AND p.source_id = ${opts.sourceId}
|
||||
AND NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'embed_skip')
|
||||
AND (cc.page_id, cc.chunk_index) > (${afterPid}, ${afterIdx})
|
||||
ORDER BY cc.page_id, cc.chunk_index
|
||||
LIMIT ${limit}
|
||||
`;
|
||||
return rows as unknown as StaleChunkRow[];
|
||||
});
|
||||
}
|
||||
|
||||
async deleteChunks(slug: string, opts?: { sourceId?: string }): Promise<void> {
|
||||
@@ -2490,11 +2597,14 @@ export class PostgresEngine implements BrainEngine {
|
||||
|
||||
async countStalePagesForExtraction(opts?: { sourceId?: string; versionTs?: string }): Promise<number> {
|
||||
const { where, params } = this.buildStalePagesWhere(opts);
|
||||
const rows = await this.sql.unsafe(
|
||||
`SELECT count(*)::int AS count FROM pages WHERE ${where}`,
|
||||
params as Parameters<typeof this.sql.unsafe>[1],
|
||||
);
|
||||
return Number((rows[0] as { count?: number } | undefined)?.count ?? 0);
|
||||
// RLS scope binding (opt-in via GBRAIN_RLS_SCOPE_BINDING).
|
||||
return await this.withScopedReadTransaction(undefined, opts?.sourceId, async (tx) => {
|
||||
const rows = await tx.unsafe(
|
||||
`SELECT count(*)::int AS count FROM pages WHERE ${where}`,
|
||||
params as Parameters<typeof tx.unsafe>[1],
|
||||
);
|
||||
return Number((rows[0] as { count?: number } | undefined)?.count ?? 0);
|
||||
});
|
||||
}
|
||||
|
||||
async listStalePagesForExtraction(opts: {
|
||||
@@ -2511,19 +2621,22 @@ export class PostgresEngine implements BrainEngine {
|
||||
}
|
||||
params.push(opts.batchSize);
|
||||
const limitIdx = params.length;
|
||||
const rows = await this.sql.unsafe(
|
||||
// #1768: project a deterministic full-µs UTC string alongside updated_at.
|
||||
// to_char (not ::text — DateStyle-fragile) so extractStaleFromDB can stamp
|
||||
// links_extracted_at = the exact updated_at and the staleness predicate clears.
|
||||
`SELECT id, slug, source_id, type, title, compiled_truth, timeline, frontmatter, updated_at,
|
||||
to_char(updated_at AT TIME ZONE 'UTC', 'YYYY-MM-DD"T"HH24:MI:SS.US"Z"') AS updated_at_iso
|
||||
FROM pages
|
||||
WHERE ${where}${afterClause}
|
||||
ORDER BY id
|
||||
LIMIT $${limitIdx}`,
|
||||
params as Parameters<typeof this.sql.unsafe>[1],
|
||||
);
|
||||
return (rows as Record<string, unknown>[]).map(rowToStalePage);
|
||||
// RLS scope binding (opt-in via GBRAIN_RLS_SCOPE_BINDING).
|
||||
return await this.withScopedReadTransaction(undefined, opts.sourceId, async (tx) => {
|
||||
const rows = await tx.unsafe(
|
||||
// #1768: project a deterministic full-µs UTC string alongside updated_at.
|
||||
// to_char (not ::text — DateStyle-fragile) so extractStaleFromDB can stamp
|
||||
// links_extracted_at = the exact updated_at and the staleness predicate clears.
|
||||
`SELECT id, slug, source_id, type, title, compiled_truth, timeline, frontmatter, updated_at,
|
||||
to_char(updated_at AT TIME ZONE 'UTC', 'YYYY-MM-DD"T"HH24:MI:SS.US"Z"') AS updated_at_iso
|
||||
FROM pages
|
||||
WHERE ${where}${afterClause}
|
||||
ORDER BY id
|
||||
LIMIT $${limitIdx}`,
|
||||
params as Parameters<typeof tx.unsafe>[1],
|
||||
);
|
||||
return (rows as Record<string, unknown>[]).map(rowToStalePage);
|
||||
});
|
||||
}
|
||||
|
||||
async markPagesExtractedBatch(refs: Array<{ slug: string; source_id: string; extractedAt?: string }>, defaultExtractedAt: string): Promise<void> {
|
||||
@@ -2671,32 +2784,47 @@ export class PostgresEngine implements BrainEngine {
|
||||
}
|
||||
|
||||
async getLinks(slug: string, opts?: { sourceId?: string; sourceIds?: string[] }): Promise<Link[]> {
|
||||
const sql = this.sql;
|
||||
// #2200: federated grant scopes ALL THREE page endpoints — from, to, AND the
|
||||
// origin (the page that authored the edge, surfaced as origin_slug). Scoping
|
||||
// only from+to would still leak an out-of-grant origin's slug; the origin
|
||||
// LEFT JOIN carries the same ANY($) filter so origin_slug nulls out of grant.
|
||||
// Remote MCP clients always land here.
|
||||
if (opts?.sourceIds && opts.sourceIds.length > 0) {
|
||||
const ids = opts.sourceIds;
|
||||
const rows = await sql`
|
||||
SELECT f.slug as from_slug, t.slug as to_slug,
|
||||
l.link_type, l.context, l.link_source,
|
||||
o.slug as origin_slug, l.origin_field
|
||||
FROM links l
|
||||
JOIN pages f ON f.id = l.from_page_id
|
||||
JOIN pages t ON t.id = l.to_page_id
|
||||
LEFT JOIN pages o ON o.id = l.origin_page_id AND o.source_id = ANY(${ids}::text[])
|
||||
WHERE f.slug = ${slug} AND f.source_id = ANY(${ids}::text[]) AND t.source_id = ANY(${ids}::text[])
|
||||
`;
|
||||
return rows as unknown as Link[];
|
||||
}
|
||||
// v0.31.8 (D16) + #2200: the federated arm above is the first branch; the
|
||||
// two below preserve pre-v0.31.8 semantics. Without opts.sourceId, no source
|
||||
// filter (cross-source view for internal callers). With opts.sourceId, scope
|
||||
// the from-page lookup. See pglite-engine.ts:getLinks for context.
|
||||
if (opts?.sourceId) {
|
||||
const rows = await sql`
|
||||
// Two layers of defense (see getPage for the full pattern):
|
||||
// 1. RLS scope binding (opt-in via GBRAIN_RLS_SCOPE_BINDING)
|
||||
// 2. App-layer source filter (#2200 federated)
|
||||
return await this.withScopedReadTransaction(opts?.sourceIds, opts?.sourceId, async (tx) => {
|
||||
// #2200: federated grant scopes ALL THREE page endpoints — from, to, AND
|
||||
// the origin (the page that authored the edge, surfaced as origin_slug).
|
||||
// Scoping only from+to would still leak an out-of-grant origin's slug; the
|
||||
// origin LEFT JOIN carries the same ANY($) filter so origin_slug nulls
|
||||
// out of grant. Remote MCP clients always land here.
|
||||
if (opts?.sourceIds && opts.sourceIds.length > 0) {
|
||||
const ids = opts.sourceIds;
|
||||
const rows = await tx`
|
||||
SELECT f.slug as from_slug, t.slug as to_slug,
|
||||
l.link_type, l.context, l.link_source,
|
||||
o.slug as origin_slug, l.origin_field
|
||||
FROM links l
|
||||
JOIN pages f ON f.id = l.from_page_id
|
||||
JOIN pages t ON t.id = l.to_page_id
|
||||
LEFT JOIN pages o ON o.id = l.origin_page_id AND o.source_id = ANY(${ids}::text[])
|
||||
WHERE f.slug = ${slug} AND f.source_id = ANY(${ids}::text[]) AND t.source_id = ANY(${ids}::text[])
|
||||
`;
|
||||
return rows as unknown as Link[];
|
||||
}
|
||||
// v0.31.8 (D16) + #2200: the federated arm above is the first branch; the
|
||||
// two below preserve pre-v0.31.8 semantics. Without opts.sourceId, no
|
||||
// source filter (cross-source view for internal callers). With
|
||||
// opts.sourceId, scope the from-page lookup.
|
||||
if (opts?.sourceId) {
|
||||
const rows = await tx`
|
||||
SELECT f.slug as from_slug, t.slug as to_slug,
|
||||
l.link_type, l.context, l.link_source,
|
||||
o.slug as origin_slug, l.origin_field
|
||||
FROM links l
|
||||
JOIN pages f ON f.id = l.from_page_id
|
||||
JOIN pages t ON t.id = l.to_page_id
|
||||
LEFT JOIN pages o ON o.id = l.origin_page_id
|
||||
WHERE f.slug = ${slug} AND f.source_id = ${opts.sourceId}
|
||||
`;
|
||||
return rows as unknown as Link[];
|
||||
}
|
||||
const rows = await tx`
|
||||
SELECT f.slug as from_slug, t.slug as to_slug,
|
||||
l.link_type, l.context, l.link_source,
|
||||
o.slug as origin_slug, l.origin_field
|
||||
@@ -2704,45 +2832,49 @@ export class PostgresEngine implements BrainEngine {
|
||||
JOIN pages f ON f.id = l.from_page_id
|
||||
JOIN pages t ON t.id = l.to_page_id
|
||||
LEFT JOIN pages o ON o.id = l.origin_page_id
|
||||
WHERE f.slug = ${slug} AND f.source_id = ${opts.sourceId}
|
||||
WHERE f.slug = ${slug}
|
||||
`;
|
||||
return rows as unknown as Link[];
|
||||
}
|
||||
const rows = await sql`
|
||||
SELECT f.slug as from_slug, t.slug as to_slug,
|
||||
l.link_type, l.context, l.link_source,
|
||||
o.slug as origin_slug, l.origin_field
|
||||
FROM links l
|
||||
JOIN pages f ON f.id = l.from_page_id
|
||||
JOIN pages t ON t.id = l.to_page_id
|
||||
LEFT JOIN pages o ON o.id = l.origin_page_id
|
||||
WHERE f.slug = ${slug}
|
||||
`;
|
||||
return rows as unknown as Link[];
|
||||
});
|
||||
}
|
||||
|
||||
async getBacklinks(slug: string, opts?: { sourceId?: string; sourceIds?: string[] }): Promise<Link[]> {
|
||||
const sql = this.sql;
|
||||
// #2200: federated grant scopes all three endpoints (mirrors getLinks) — the
|
||||
// referrer (from), the queried page (to), AND the origin — so neither a
|
||||
// foreign referrer nor a foreign origin slug is disclosed to the caller.
|
||||
if (opts?.sourceIds && opts.sourceIds.length > 0) {
|
||||
const ids = opts.sourceIds;
|
||||
const rows = await sql`
|
||||
SELECT f.slug as from_slug, t.slug as to_slug,
|
||||
l.link_type, l.context, l.link_source,
|
||||
o.slug as origin_slug, l.origin_field
|
||||
FROM links l
|
||||
JOIN pages f ON f.id = l.from_page_id
|
||||
JOIN pages t ON t.id = l.to_page_id
|
||||
LEFT JOIN pages o ON o.id = l.origin_page_id AND o.source_id = ANY(${ids}::text[])
|
||||
WHERE t.slug = ${slug} AND t.source_id = ANY(${ids}::text[]) AND f.source_id = ANY(${ids}::text[])
|
||||
`;
|
||||
return rows as unknown as Link[];
|
||||
}
|
||||
// v0.31.8 (D16) + #2200: federated arm above is first; two below mirror getLinks.
|
||||
if (opts?.sourceId) {
|
||||
const rows = await sql`
|
||||
// Two layers of defense (see getPage for the full pattern):
|
||||
// 1. RLS scope binding (opt-in via GBRAIN_RLS_SCOPE_BINDING)
|
||||
// 2. App-layer source filter (#2200 federated)
|
||||
return await this.withScopedReadTransaction(opts?.sourceIds, opts?.sourceId, async (tx) => {
|
||||
// #2200: federated grant scopes all three endpoints (mirrors getLinks) —
|
||||
// the referrer (from), the queried page (to), AND the origin — so neither
|
||||
// a foreign referrer nor a foreign origin slug is disclosed to the caller.
|
||||
if (opts?.sourceIds && opts.sourceIds.length > 0) {
|
||||
const ids = opts.sourceIds;
|
||||
const rows = await tx`
|
||||
SELECT f.slug as from_slug, t.slug as to_slug,
|
||||
l.link_type, l.context, l.link_source,
|
||||
o.slug as origin_slug, l.origin_field
|
||||
FROM links l
|
||||
JOIN pages f ON f.id = l.from_page_id
|
||||
JOIN pages t ON t.id = l.to_page_id
|
||||
LEFT JOIN pages o ON o.id = l.origin_page_id AND o.source_id = ANY(${ids}::text[])
|
||||
WHERE t.slug = ${slug} AND t.source_id = ANY(${ids}::text[]) AND f.source_id = ANY(${ids}::text[])
|
||||
`;
|
||||
return rows as unknown as Link[];
|
||||
}
|
||||
// v0.31.8 (D16) + #2200: federated arm above is first; two below mirror getLinks.
|
||||
if (opts?.sourceId) {
|
||||
const rows = await tx`
|
||||
SELECT f.slug as from_slug, t.slug as to_slug,
|
||||
l.link_type, l.context, l.link_source,
|
||||
o.slug as origin_slug, l.origin_field
|
||||
FROM links l
|
||||
JOIN pages f ON f.id = l.from_page_id
|
||||
JOIN pages t ON t.id = l.to_page_id
|
||||
LEFT JOIN pages o ON o.id = l.origin_page_id
|
||||
WHERE t.slug = ${slug} AND t.source_id = ${opts.sourceId}
|
||||
`;
|
||||
return rows as unknown as Link[];
|
||||
}
|
||||
const rows = await tx`
|
||||
SELECT f.slug as from_slug, t.slug as to_slug,
|
||||
l.link_type, l.context, l.link_source,
|
||||
o.slug as origin_slug, l.origin_field
|
||||
@@ -2750,45 +2882,36 @@ export class PostgresEngine implements BrainEngine {
|
||||
JOIN pages f ON f.id = l.from_page_id
|
||||
JOIN pages t ON t.id = l.to_page_id
|
||||
LEFT JOIN pages o ON o.id = l.origin_page_id
|
||||
WHERE t.slug = ${slug} AND t.source_id = ${opts.sourceId}
|
||||
WHERE t.slug = ${slug}
|
||||
`;
|
||||
return rows as unknown as Link[];
|
||||
}
|
||||
const rows = await sql`
|
||||
SELECT f.slug as from_slug, t.slug as to_slug,
|
||||
l.link_type, l.context, l.link_source,
|
||||
o.slug as origin_slug, l.origin_field
|
||||
FROM links l
|
||||
JOIN pages f ON f.id = l.from_page_id
|
||||
JOIN pages t ON t.id = l.to_page_id
|
||||
LEFT JOIN pages o ON o.id = l.origin_page_id
|
||||
WHERE t.slug = ${slug}
|
||||
`;
|
||||
return rows as unknown as Link[];
|
||||
});
|
||||
}
|
||||
|
||||
async listLinkSources(
|
||||
opts?: { sourceId?: string; sourceIds?: string[] },
|
||||
): Promise<{ link_source: string | null; count: number }[]> {
|
||||
const sql = this.sql;
|
||||
// v114 (#1941): distinct provenances + counts for `gbrain link-sources`.
|
||||
// Scope by the FROM page's source (consistent with getLinks). Federated
|
||||
// {sourceIds} takes precedence over scalar {sourceId}; neither = unscoped.
|
||||
const sourceCondition =
|
||||
opts?.sourceIds && opts.sourceIds.length > 0
|
||||
? sql`WHERE f.source_id = ANY(${opts.sourceIds}::text[])`
|
||||
: opts?.sourceId
|
||||
? sql`WHERE f.source_id = ${opts.sourceId}`
|
||||
: sql``;
|
||||
const rows = await sql`
|
||||
SELECT l.link_source, COUNT(*)::int AS count
|
||||
FROM links l
|
||||
JOIN pages f ON f.id = l.from_page_id
|
||||
${sourceCondition}
|
||||
GROUP BY l.link_source
|
||||
ORDER BY count DESC, l.link_source ASC NULLS LAST
|
||||
`;
|
||||
return rows as unknown as { link_source: string | null; count: number }[];
|
||||
// RLS scope binding (opt-in via GBRAIN_RLS_SCOPE_BINDING).
|
||||
return await this.withScopedReadTransaction(opts?.sourceIds, opts?.sourceId, async (tx) => {
|
||||
// v114 (#1941): distinct provenances + counts for `gbrain link-sources`.
|
||||
// Scope by the FROM page's source (consistent with getLinks). Federated
|
||||
// {sourceIds} takes precedence over scalar {sourceId}; neither = unscoped.
|
||||
const sourceCondition =
|
||||
opts?.sourceIds && opts.sourceIds.length > 0
|
||||
? tx`WHERE f.source_id = ANY(${opts.sourceIds}::text[])`
|
||||
: opts?.sourceId
|
||||
? tx`WHERE f.source_id = ${opts.sourceId}`
|
||||
: tx``;
|
||||
const rows = await tx`
|
||||
SELECT l.link_source, COUNT(*)::int AS count
|
||||
FROM links l
|
||||
JOIN pages f ON f.id = l.from_page_id
|
||||
${sourceCondition}
|
||||
GROUP BY l.link_source
|
||||
ORDER BY count DESC, l.link_source ASC NULLS LAST
|
||||
`;
|
||||
return rows as unknown as { link_source: string | null; count: number }[];
|
||||
});
|
||||
}
|
||||
|
||||
async findByTitleFuzzy(
|
||||
|
||||
+8
-3
@@ -255,7 +255,12 @@ const PRUNE_DIR_NAMES = new Set<string>([
|
||||
// with the first-sync walker in commands/import.ts.
|
||||
'venv',
|
||||
'.raw',
|
||||
'ops',
|
||||
// NOTE (#2404): `'ops'` used to be in this list (a v0.2.0-era carve-out for
|
||||
// one brain layout). Matching the bare segment pruned EVERY user `ops/`
|
||||
// directory at any depth — sync silently deleted `ops/*` pages and never
|
||||
// imported `ops/*` files, while the bundled daily-task-manager skill
|
||||
// prescribes `ops/tasks` as its canonical page. `ops/` is ordinary content;
|
||||
// do NOT re-add it. Only generated/vendored trees belong here.
|
||||
]);
|
||||
|
||||
/**
|
||||
@@ -352,8 +357,8 @@ function classifySync(path: string, opts: SyncableOptions = {}): SyncableReason
|
||||
if (!isAllowedByStrategy(path, strategy)) return 'strategy';
|
||||
|
||||
// Skip every path segment that pruneDir would block walkers from descending
|
||||
// into. Catches hidden dirs (`.git`, `.obsidian`), `.raw/` sidecars,
|
||||
// `node_modules/` (latent bug fix), and `ops/` at any depth.
|
||||
// into. Catches hidden dirs (`.git`, `.obsidian`), `.raw/` sidecars, and
|
||||
// vendor/generated trees (`node_modules/`, `vendor/`, …) at any depth.
|
||||
const segments = path.split('/');
|
||||
if (segments.some(p => !pruneDir(p))) return 'pruned-dir';
|
||||
|
||||
|
||||
@@ -27,6 +27,7 @@ import { randomBytes } from 'crypto';
|
||||
import type { BrainEngine } from './engine.ts';
|
||||
import { serializePageToMarkdown, resolvePageFilePath } from './markdown.ts';
|
||||
import { isWriteTargetContained } from './path-confine.ts';
|
||||
import { isDurabilityHardened, commitWriteThroughFile } from './brain-repo-durability.ts';
|
||||
|
||||
/** Minimal logger surface — structurally compatible with operations.ts `Logger`. */
|
||||
export interface WriteThroughLogger {
|
||||
@@ -36,6 +37,13 @@ export interface WriteThroughLogger {
|
||||
export interface WriteThroughResult {
|
||||
written: boolean;
|
||||
path?: string;
|
||||
/**
|
||||
* True when the write was also committed to git (#2426). Only attempted on
|
||||
* repos hardened via `gbrain sources harden` (durability hook installed);
|
||||
* the hook then background-pushes the commit. Best-effort — a false/absent
|
||||
* value never blocks the write.
|
||||
*/
|
||||
committed?: boolean;
|
||||
/**
|
||||
* Non-error reasons the file was not written:
|
||||
* - no_repo_configured: the resolved target (source `local_path` or, for a
|
||||
@@ -157,7 +165,20 @@ export async function writePageThrough(
|
||||
throw writeErr;
|
||||
}
|
||||
|
||||
return { written: true, path: filePath };
|
||||
// #2426: on a durability-hardened repo (user ran `gbrain sources harden`),
|
||||
// commit the artifact so it reaches git — pre-fix, write-through content
|
||||
// stayed uncommitted forever: never pushed, `last_sync_at` frozen, and
|
||||
// silently deleted by a later `sync --full` delete-reconcile. The local
|
||||
// post-commit hook background-pushes the commit. Best-effort: a commit
|
||||
// failure never fails the write (the DB row + file are the durable sinks).
|
||||
let committed = false;
|
||||
try {
|
||||
if (isDurabilityHardened(writeRoot)) {
|
||||
committed = commitWriteThroughFile(writeRoot, filePath, slug);
|
||||
}
|
||||
} catch { /* best-effort */ }
|
||||
|
||||
return { written: true, path: filePath, ...(committed ? { committed } : {}) };
|
||||
} catch (e) {
|
||||
const msg = e instanceof Error ? e.message : String(e);
|
||||
opts.logger?.warn(`[write-through] failed for ${slug}: ${msg}`);
|
||||
|
||||
@@ -0,0 +1,53 @@
|
||||
/**
|
||||
* Moonshot/Kimi local recipe smoke.
|
||||
*
|
||||
* This pins the governed production exception GBrain-Local-003: GBrain can
|
||||
* route configured Kimi chat/expansion IDs through Moonshot's OpenAI-compatible
|
||||
* endpoint without treating `moonshot` as an unknown provider.
|
||||
*/
|
||||
|
||||
import { describe, expect, test } from 'bun:test';
|
||||
import { getRecipe } from '../../src/core/ai/recipes/index.ts';
|
||||
import { defaultResolveAuth } from '../../src/core/ai/gateway.ts';
|
||||
import { assertTouchpoint } from '../../src/core/ai/model-resolver.ts';
|
||||
import { AIConfigError } from '../../src/core/ai/errors.ts';
|
||||
|
||||
describe('recipe: moonshot', () => {
|
||||
test('registered with expected OpenAI-compatible shape', () => {
|
||||
const r = getRecipe('moonshot');
|
||||
expect(r).toBeDefined();
|
||||
expect(r!.id).toBe('moonshot');
|
||||
expect(r!.tier).toBe('openai-compat');
|
||||
expect(r!.implementation).toBe('openai-compatible');
|
||||
expect(r!.base_url_default).toBe('https://api.moonshot.ai/v1');
|
||||
expect(r!.auth_env?.required).toEqual(['MOONSHOT_API_KEY']);
|
||||
});
|
||||
|
||||
test('chat and expansion touchpoints include Kimi K2.7 Code', () => {
|
||||
const r = getRecipe('moonshot')!;
|
||||
expect(r.touchpoints.chat).toBeDefined();
|
||||
expect(r.touchpoints.expansion).toBeDefined();
|
||||
expect(r.touchpoints.chat!.models).toContain('kimi-k2.7-code');
|
||||
expect(r.touchpoints.expansion!.models).toContain('kimi-k2.7-code');
|
||||
expect(r.touchpoints.chat!.supports_tools).toBe(true);
|
||||
expect(r.touchpoints.chat!.supports_subagent_loop).toBe(false);
|
||||
});
|
||||
|
||||
test('configured Kimi model is accepted for chat and expansion', () => {
|
||||
const r = getRecipe('moonshot')!;
|
||||
expect(() => assertTouchpoint(r, 'chat', 'kimi-k2.7-code')).not.toThrow();
|
||||
expect(() => assertTouchpoint(r, 'expansion', 'kimi-k2.7-code')).not.toThrow();
|
||||
});
|
||||
|
||||
test('default auth: MOONSHOT_API_KEY set -> Bearer token', () => {
|
||||
const r = getRecipe('moonshot')!;
|
||||
const auth = defaultResolveAuth(r, { MOONSHOT_API_KEY: 'fake-moonshot-key' }, 'chat');
|
||||
expect(auth.headerName).toBe('Authorization');
|
||||
expect(auth.token).toBe('Bearer fake-moonshot-key');
|
||||
});
|
||||
|
||||
test('default auth: missing MOONSHOT_API_KEY -> AIConfigError', () => {
|
||||
const r = getRecipe('moonshot')!;
|
||||
expect(() => defaultResolveAuth(r, {}, 'chat')).toThrow(AIConfigError);
|
||||
});
|
||||
});
|
||||
@@ -146,6 +146,41 @@ describe('buildBrainTools', () => {
|
||||
),
|
||||
).rejects.toBeInstanceOf(OperationError);
|
||||
});
|
||||
|
||||
// #1586: sourceId threads through buildBrainTools → buildOpContext →
|
||||
// put_page → importFromContent, so subagent writes land in the cycle's
|
||||
// resolved source instead of the hardcoded 'default'.
|
||||
test('execute() on put_page writes to the configured sourceId (#1586)', async () => {
|
||||
await engine.executeRaw(
|
||||
`INSERT INTO sources (id, name, local_path, config, archived, created_at)
|
||||
VALUES ('mybrain', 'My Brain', '/tmp/mybrain', '{}'::jsonb, false, now())
|
||||
ON CONFLICT (id) DO NOTHING`,
|
||||
);
|
||||
const tools = buildBrainTools({
|
||||
subagentId: 42,
|
||||
engine,
|
||||
config,
|
||||
allowedSlugPrefixes: ['wiki/personal/reflections/*'],
|
||||
sourceId: 'mybrain',
|
||||
});
|
||||
const putPage = tools.find(t => t.name === 'brain_put_page');
|
||||
const ctx: ToolCtx = { engine, jobId: 1, remote: true };
|
||||
await putPage!.execute(
|
||||
{ slug: 'wiki/personal/reflections/2026-07-17-scoped', content: '---\ntitle: Scoped\n---\nbody' },
|
||||
ctx,
|
||||
);
|
||||
const rows = await engine.executeRaw<{ source_id: string }>(
|
||||
`SELECT source_id FROM pages WHERE slug = 'wiki/personal/reflections/2026-07-17-scoped'`,
|
||||
);
|
||||
expect(rows.length).toBe(1);
|
||||
expect(rows[0].source_id).toBe('mybrain');
|
||||
});
|
||||
|
||||
test('buildBrainTools rejects a malformed sourceId at build time (#1586)', () => {
|
||||
expect(() =>
|
||||
buildBrainTools({ subagentId: 1, engine, config, sourceId: '../evil' }),
|
||||
).toThrow();
|
||||
});
|
||||
});
|
||||
|
||||
describe('filterAllowedTools', () => {
|
||||
|
||||
@@ -90,6 +90,36 @@ describe('brain-commit-push.sh (D13 guarantee)', () => {
|
||||
} catch (e: any) { code = e.status ?? 1; }
|
||||
expect(code).toBe(2);
|
||||
});
|
||||
|
||||
test('#2426 — commits a MODIFIED tracked file even when the remote advanced (commit before pull)', () => {
|
||||
// Pre-fix, the helper ran `git pull --rebase` BEFORE staging, so any dirty
|
||||
// tree (a modified/enriched page — exactly the write-through case) aborted
|
||||
// with 'cannot pull with rebase: You have unstaged changes' (exit 3). The
|
||||
// helper could only ever commit untracked-NEW files, never modifications.
|
||||
// Remove the post-commit hook so its background push can't race the
|
||||
// helper's own push (macOS has no flock to serialize them) — this test
|
||||
// targets the HELPER's ordering; hook behavior is covered below.
|
||||
rmSync(join(work, '.git', 'hooks', 'post-commit'));
|
||||
// Advance the remote from a second clone so a pull is genuinely needed.
|
||||
const other = mkdtempSync(join(root, 'other-'));
|
||||
execFileSync('git', ['-c', 'protocol.file.allow=always', 'clone', '-q', bare, other], { stdio: 'ignore' });
|
||||
git(other, 'config', 'user.email', 'o@o.o'); git(other, 'config', 'user.name', 'other');
|
||||
writeFileSync(join(other, 'remote.md'), 'from other\n');
|
||||
git(other, 'add', 'remote.md'); git(other, 'commit', '-qm', 'remote change'); git(other, 'push', '-q', 'origin', 'main');
|
||||
|
||||
// Dirty MODIFICATION of a tracked file in the hardened clone (write-through shape).
|
||||
writeFileSync(join(work, 'README.md'), 'modified by write-through\n');
|
||||
execFileSync('bash', [join(work, 'scripts', 'brain-commit-push.sh'), 'wt: README', 'README.md'], {
|
||||
cwd: work, stdio: ['ignore', 'pipe', 'pipe'], env: process.env,
|
||||
});
|
||||
|
||||
// Both the remote's commit and ours are on origin/main.
|
||||
const subjects = git(bare, 'log', '--format=%s', 'main');
|
||||
expect(subjects).toContain('wt: README');
|
||||
expect(subjects).toContain('remote change');
|
||||
// Working tree is clean — the modification was committed, not stranded.
|
||||
expect(git(work, 'status', '--porcelain', 'README.md')).toBe('');
|
||||
});
|
||||
});
|
||||
|
||||
describe('post-commit hook (D9 local, D7 self-contained)', () => {
|
||||
|
||||
@@ -43,8 +43,10 @@ beforeAll(() => {
|
||||
writeFileSync(join(root, '.obsidian', 'workspace.json'), '{}');
|
||||
mkdirSync(join(root, 'people', 'pedro.raw'), { recursive: true });
|
||||
writeFileSync(join(root, 'people', 'pedro.raw', 'source.md'), '---\ntitle: should not visit\n---\n');
|
||||
// ops/ is ORDINARY content (#2404) — walker MUST descend (it used to be
|
||||
// wrongly pruned, silently excluding user runbooks / ops/tasks).
|
||||
mkdirSync(join(root, 'ops', 'logs'), { recursive: true });
|
||||
writeFileSync(join(root, 'ops', 'logs', 'run.md'), '# nope\n');
|
||||
writeFileSync(join(root, 'ops', 'logs', 'run.md'), '---\ntitle: Run\n---\n\nbody\n');
|
||||
// Nested node_modules — must also be pruned, not just at the root.
|
||||
mkdirSync(join(root, 'people', 'tools', 'node_modules', 'inner'), { recursive: true });
|
||||
writeFileSync(join(root, 'people', 'tools', 'node_modules', 'inner', 'a.md'), '---\ntitle: nope\n---\n');
|
||||
@@ -95,8 +97,11 @@ describe('walkDir (brain-writer.ts) — descent-time pruning', () => {
|
||||
walkDir(root, (f) => { files.push(f); }, (dir) => visited.push(dir));
|
||||
expect(visited.some(d => d.endsWith('/people'))).toBe(true);
|
||||
expect(visited.some(d => d.endsWith('/concepts/subdir'))).toBe(true);
|
||||
// ops/ is ordinary content — descended, not pruned (#2404).
|
||||
expect(visited.some(d => d.endsWith('/ops/logs'))).toBe(true);
|
||||
expect(files.some(f => f.endsWith('/people/alice.md'))).toBe(true);
|
||||
expect(files.some(f => f.endsWith('/concepts/subdir/thing.md'))).toBe(true);
|
||||
expect(files.some(f => f.endsWith('/ops/logs/run.md'))).toBe(true);
|
||||
// And explicitly does NOT visit the file under node_modules.
|
||||
expect(files.some(f => f.includes('/node_modules/'))).toBe(false);
|
||||
});
|
||||
@@ -107,7 +112,7 @@ describe('walkDir (brain-writer.ts) — descent-time pruning', () => {
|
||||
// visitDir would be called with node_modules paths.
|
||||
const descents: string[] = [];
|
||||
walkDir(root, () => {}, (d) => descents.push(d));
|
||||
const vendor = descents.filter(d => /\/(node_modules|\.git|\.obsidian|ops)(\/|$)/.test(d) || /\.raw$/.test(d));
|
||||
const vendor = descents.filter(d => /\/(node_modules|\.git|\.obsidian)(\/|$)/.test(d) || /\.raw$/.test(d));
|
||||
expect(vendor).toEqual([]);
|
||||
});
|
||||
});
|
||||
@@ -119,13 +124,20 @@ describe('collectFiles (frontmatter.ts) — descent-time pruning parity', () =>
|
||||
expect(visited.some(d => d.includes('/node_modules'))).toBe(false);
|
||||
});
|
||||
|
||||
test('does NOT descend into .git, .obsidian, *.raw, or ops', () => {
|
||||
test('does NOT descend into .git, .obsidian, or *.raw', () => {
|
||||
const visited: string[] = [];
|
||||
collectFiles(root, (dir) => visited.push(dir));
|
||||
expect(visited.some(d => d.includes('/.git'))).toBe(false);
|
||||
expect(visited.some(d => d.includes('/.obsidian'))).toBe(false);
|
||||
expect(visited.some(d => d.endsWith('.raw'))).toBe(false);
|
||||
expect(visited.some(d => d.endsWith('/ops') || d.includes('/ops/'))).toBe(false);
|
||||
});
|
||||
|
||||
test('DOES descend into ops/ — ordinary content, not a vendor tree (#2404)', () => {
|
||||
const visited: string[] = [];
|
||||
collectFiles(root, (dir) => visited.push(dir));
|
||||
expect(visited.some(d => d.endsWith('/ops') || d.includes('/ops/'))).toBe(true);
|
||||
const files = collectFiles(root);
|
||||
expect(files.some(f => f.endsWith('/ops/logs/run.md'))).toBe(true);
|
||||
});
|
||||
|
||||
test('does NOT descend into git submodule directories', () => {
|
||||
|
||||
@@ -8,7 +8,7 @@
|
||||
import { describe, test, expect, beforeAll, afterAll, beforeEach } from 'bun:test';
|
||||
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
|
||||
import { isChronicleEligible } from '../src/core/chronicle/eligibility.ts';
|
||||
import { runChronicleExtract, type ChronicleJudge } from '../src/core/chronicle/extract-events.ts';
|
||||
import { runChronicleExtract, parseJudgeJson, type ChronicleJudge } from '../src/core/chronicle/extract-events.ts';
|
||||
import { runChronicleBackstop } from '../src/core/chronicle/backstop.ts';
|
||||
|
||||
let engine: PGLiteEngine;
|
||||
@@ -111,6 +111,44 @@ describe('runChronicleExtract', () => {
|
||||
const r = await runChronicleExtract(engine, { slug: 'meetings/2026-06-18-sync', judge: none });
|
||||
expect(r.status).toBe('no_events');
|
||||
});
|
||||
|
||||
// #2606: a truncated or unparseable judge response must NOT be recorded as
|
||||
// a legitimate no_events — it gets a distinct skipped reason.
|
||||
test('truncated judge output → skipped/judge_truncated, not no_events (#2606)', async () => {
|
||||
const truncated: ChronicleJudge = async () => ({ events: [], failure: 'truncated' });
|
||||
const r = await runChronicleExtract(engine, { slug: 'meetings/2026-06-18-sync', judge: truncated });
|
||||
expect(r.status).toBe('skipped');
|
||||
expect(r.reason).toBe('judge_truncated');
|
||||
expect(await countEvents()).toBe(0);
|
||||
});
|
||||
|
||||
test('unparseable judge output → skipped/judge_parse_failed (#2606)', async () => {
|
||||
const parseFailed: ChronicleJudge = async () => ({ events: [], failure: 'parse_failed' });
|
||||
const r = await runChronicleExtract(engine, { slug: 'meetings/2026-06-18-sync', judge: parseFailed });
|
||||
expect(r.status).toBe('skipped');
|
||||
expect(r.reason).toBe('judge_parse_failed');
|
||||
});
|
||||
});
|
||||
|
||||
describe('parseJudgeJson failure signalling (#2606)', () => {
|
||||
test('a legitimate empty array parses to []', () => {
|
||||
expect(parseJudgeJson('[]')).toEqual([]);
|
||||
expect(parseJudgeJson('```json\n[]\n```')).toEqual([]);
|
||||
});
|
||||
|
||||
test('a valid array round-trips', () => {
|
||||
const arr = parseJudgeJson('[{"when":"2026-06-18","who":[],"what":"x","kind":"meeting"}]');
|
||||
expect(Array.isArray(arr)).toBe(true);
|
||||
expect(arr!.length).toBe(1);
|
||||
});
|
||||
|
||||
test('empty / no-array / truncated / non-array responses return null', () => {
|
||||
expect(parseJudgeJson('')).toBeNull();
|
||||
expect(parseJudgeJson('I found no events worth extracting.')).toBeNull();
|
||||
// Truncated mid-array (the maxTokens-cap shape from the issue).
|
||||
expect(parseJudgeJson('[{"when":"2026-06-18","who":["a"],"what":"long ev')).toBeNull();
|
||||
expect(parseJudgeJson('{"events": 1}')).toBeNull();
|
||||
});
|
||||
});
|
||||
|
||||
describe('runChronicleBackstop gating', () => {
|
||||
|
||||
@@ -0,0 +1,112 @@
|
||||
/**
|
||||
* #2415 — configurable dream output namespace (`dream.synthesize.output_root`).
|
||||
*
|
||||
* The synthesize + patterns phases previously hardcoded `wiki/` in the
|
||||
* subagent prompt slug templates, the patterns reflection lookup, and the
|
||||
* trusted-workspace allow-list loaded from skills/_brain-filing-rules.json.
|
||||
* This suite pins:
|
||||
* - default 'wiki' → byte-identical prompt + verbatim filing-rule globs
|
||||
* (zero behavior change unless the key is set);
|
||||
* - a custom root remaps prompt slug templates and the allow-list globs;
|
||||
* - loadOutputRoot validates against the slug grammar (bad values fall
|
||||
* back to 'wiki');
|
||||
* - the patterns phase gathers reflections under the configured root.
|
||||
*/
|
||||
import { describe, test, expect, beforeAll, afterAll } from 'bun:test';
|
||||
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
|
||||
import { __testing, loadAllowedSlugPrefixes, loadOutputRoot } from '../src/core/cycle/synthesize.ts';
|
||||
import { runPhasePatterns } from '../src/core/cycle/patterns.ts';
|
||||
import type { DiscoveredTranscript } from '../src/core/cycle/transcript-discovery.ts';
|
||||
|
||||
const { buildSynthesisPrompt } = __testing;
|
||||
|
||||
const transcript: DiscoveredTranscript = {
|
||||
filePath: '/tmp/t.txt',
|
||||
basename: 't',
|
||||
content: 'User: hello world',
|
||||
contentHash: 'abcdef0123456789',
|
||||
inferredDate: '2026-07-17',
|
||||
} as DiscoveredTranscript;
|
||||
|
||||
describe('#2415: buildSynthesisPrompt output root', () => {
|
||||
test('defaults to wiki/ slug templates', () => {
|
||||
const prompt = buildSynthesisPrompt(transcript, 'chunk', 0, 1);
|
||||
expect(prompt).toContain('wiki/personal/reflections/2026-07-17-');
|
||||
expect(prompt).toContain('wiki/originals/ideas/2026-07-17-');
|
||||
});
|
||||
|
||||
test('custom root replaces wiki/ in both slug templates', () => {
|
||||
const prompt = buildSynthesisPrompt(transcript, 'chunk', 0, 1, '', 'notes');
|
||||
expect(prompt).toContain('notes/personal/reflections/2026-07-17-');
|
||||
expect(prompt).toContain('notes/originals/ideas/2026-07-17-');
|
||||
expect(prompt).not.toContain('wiki/personal/reflections/');
|
||||
expect(prompt).not.toContain('wiki/originals/ideas/');
|
||||
});
|
||||
});
|
||||
|
||||
describe('#2415: loadAllowedSlugPrefixes remap', () => {
|
||||
// Runs from the repo root, so skills/_brain-filing-rules.json resolves.
|
||||
test("default 'wiki' returns the filing-rule globs verbatim", async () => {
|
||||
const globs = await loadAllowedSlugPrefixes();
|
||||
expect(globs).toContain('wiki/personal/reflections/*');
|
||||
expect(globs).toContain('dream-cycle-summaries/*');
|
||||
});
|
||||
|
||||
test('custom root remaps only wiki/-rooted globs', async () => {
|
||||
const globs = await loadAllowedSlugPrefixes('notes');
|
||||
expect(globs).toContain('notes/personal/reflections/*');
|
||||
expect(globs).toContain('notes/originals/*');
|
||||
expect(globs).toContain('notes/personal/patterns/*');
|
||||
// Non-wiki globs pass through untouched.
|
||||
expect(globs).toContain('dream-cycle-summaries/*');
|
||||
expect(globs.some(g => g.startsWith('wiki/'))).toBe(false);
|
||||
});
|
||||
});
|
||||
|
||||
describe('#2415: loadOutputRoot validation + patterns gather scope', () => {
|
||||
let engine: PGLiteEngine;
|
||||
|
||||
beforeAll(async () => {
|
||||
engine = new PGLiteEngine();
|
||||
await engine.connect({});
|
||||
await engine.initSchema();
|
||||
});
|
||||
|
||||
afterAll(async () => {
|
||||
await engine.disconnect();
|
||||
});
|
||||
|
||||
test('unset → wiki; trailing slash trimmed; invalid → wiki fallback', async () => {
|
||||
expect(await loadOutputRoot(engine)).toBe('wiki');
|
||||
await engine.setConfig('dream.synthesize.output_root', 'notes/');
|
||||
expect(await loadOutputRoot(engine)).toBe('notes');
|
||||
await engine.setConfig('dream.synthesize.output_root', '../escape');
|
||||
expect(await loadOutputRoot(engine)).toBe('wiki');
|
||||
await engine.setConfig('dream.synthesize.output_root', 'Bad_Root');
|
||||
expect(await loadOutputRoot(engine)).toBe('wiki');
|
||||
});
|
||||
|
||||
test('patterns phase gathers reflections under the configured root', async () => {
|
||||
await engine.setConfig('dream.synthesize.output_root', 'notes');
|
||||
for (let i = 0; i < 3; i++) {
|
||||
await engine.putPage(`notes/personal/reflections/2026-07-17-r${i}`, {
|
||||
type: 'note',
|
||||
title: `R${i}`,
|
||||
compiled_truth: `reflection ${i}`,
|
||||
timeline: '',
|
||||
frontmatter: {},
|
||||
});
|
||||
}
|
||||
// A wiki/-rooted reflection must NOT be counted under the custom root.
|
||||
await engine.putPage('wiki/personal/reflections/2026-07-17-old', {
|
||||
type: 'note',
|
||||
title: 'Old',
|
||||
compiled_truth: 'legacy reflection',
|
||||
timeline: '',
|
||||
frontmatter: {},
|
||||
});
|
||||
const result = await runPhasePatterns(engine, { brainDir: '/tmp', dryRun: true });
|
||||
expect(result.status).toBe('ok');
|
||||
expect(result.details?.reflections_considered).toBe(3);
|
||||
});
|
||||
});
|
||||
@@ -74,8 +74,12 @@ describe('patterns phase wiring', () => {
|
||||
});
|
||||
|
||||
describe('patterns scope filter', () => {
|
||||
test('filters reflections by slug LIKE wiki/personal/reflections/%', () => {
|
||||
expect(patternsSrc).toContain("slug LIKE 'wiki/personal/reflections/%'");
|
||||
test('filters reflections by slug LIKE <output_root>/personal/reflections/%', () => {
|
||||
// #2415: the namespace root is configurable (dream.synthesize.output_root,
|
||||
// default 'wiki') and bound as a parameter — the scope filter itself and
|
||||
// the reflections sub-path stay pinned.
|
||||
expect(patternsSrc).toContain('slug LIKE $2');
|
||||
expect(patternsSrc).toContain('/personal/reflections/%');
|
||||
});
|
||||
|
||||
test('orders by updated_at DESC for recency-bias', () => {
|
||||
|
||||
@@ -21,7 +21,7 @@ import { describe, test, expect, beforeAll, afterAll } from 'bun:test';
|
||||
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
|
||||
import { __testing } from '../src/core/cycle/synthesize.ts';
|
||||
|
||||
const { collectChildPutPageSlugs } = __testing;
|
||||
const { collectChildPutPageSlugs, stampDreamProvenance } = __testing;
|
||||
|
||||
let engine: PGLiteEngine;
|
||||
|
||||
@@ -103,4 +103,52 @@ describe('C6: collectChildPutPageSlugs survives double-encoded jsonb (#745)', ()
|
||||
// Function silently drops rows whose slug resolves to null/empty.
|
||||
expect(refs.map((r: { slug: string }) => r.slug)).not.toContain('no-slug');
|
||||
});
|
||||
|
||||
// #1586: refs are stamped with the cycle's resolved source, not a
|
||||
// hardcoded 'default'.
|
||||
test('stamps refs with the provided cycle sourceId (#1586)', async () => {
|
||||
const refs = await collectChildPutPageSlugs(engine as any, [1001], new Map(), 'mybrain');
|
||||
expect(refs.length).toBeGreaterThan(0);
|
||||
for (const r of refs) expect(r.source_id).toBe('mybrain');
|
||||
});
|
||||
|
||||
test('defaults to source_id=default when no sourceId is passed (legacy)', async () => {
|
||||
const refs = await collectChildPutPageSlugs(engine as any, [1001], new Map());
|
||||
expect(refs.length).toBeGreaterThan(0);
|
||||
for (const r of refs) expect(r.source_id).toBe('default');
|
||||
});
|
||||
});
|
||||
|
||||
describe('#2569: stampDreamProvenance persists the marker into DB frontmatter', () => {
|
||||
test('merges dream_generated + dream_cycle_date into pages.frontmatter', async () => {
|
||||
await engine.putPage('wiki/originals/ideas/2026-07-17-stamp-me-abc123', {
|
||||
type: 'note',
|
||||
title: 'Stamp me',
|
||||
compiled_truth: 'body',
|
||||
timeline: '',
|
||||
frontmatter: { keep_me: 'yes' },
|
||||
});
|
||||
await stampDreamProvenance(
|
||||
engine as any,
|
||||
[{ slug: 'wiki/originals/ideas/2026-07-17-stamp-me-abc123', source_id: 'default' }],
|
||||
'2026-07-17',
|
||||
);
|
||||
const rows = await engine.executeRaw<{ fm: Record<string, unknown> }>(
|
||||
`SELECT frontmatter AS fm FROM pages WHERE slug = 'wiki/originals/ideas/2026-07-17-stamp-me-abc123'`,
|
||||
);
|
||||
expect(rows.length).toBe(1);
|
||||
const fm = rows[0].fm as Record<string, unknown>;
|
||||
// The stamp lands as real JSONB values (queryable via ->>), not a
|
||||
// double-encoded string scalar.
|
||||
expect(fm.dream_generated).toBe(true);
|
||||
expect(fm.dream_cycle_date).toBe('2026-07-17');
|
||||
// Merge, not replace: pre-existing frontmatter keys survive.
|
||||
expect(fm.keep_me).toBe('yes');
|
||||
});
|
||||
|
||||
test('is idempotent and never throws for a missing page', async () => {
|
||||
const refs = [{ slug: 'wiki/originals/ideas/does-not-exist', source_id: 'default' }];
|
||||
await stampDreamProvenance(engine as any, refs, '2026-07-17'); // no throw
|
||||
await stampDreamProvenance(engine as any, refs, '2026-07-17'); // idempotent
|
||||
});
|
||||
});
|
||||
|
||||
@@ -316,4 +316,32 @@ describe('v0.41 T6: runPhaseSynthesizeConcepts via stubbed chat', () => {
|
||||
);
|
||||
expect(rows[0].compiled_truth).toContain('Custom synthesized narrative');
|
||||
});
|
||||
|
||||
// #2163: concept pages must enter the retrieval surface. The write routes
|
||||
// through importFromContent (the same parse→chunk pipeline put_page uses),
|
||||
// so content_chunks rows exist and source-boost's 1.3× 'concepts/' weight
|
||||
// has something to boost. (Embeddings are skipped in this env — no
|
||||
// provider — but chunks + search_vector land regardless.)
|
||||
test('concept pages are chunked (#2163)', async () => {
|
||||
const atoms = Array.from({ length: 12 }, (_, i) => ({
|
||||
slug: `c${i}`,
|
||||
title: `Chunk atom ${i}`,
|
||||
body: `Chunky body ${i}.`,
|
||||
concept_refs: ['chunked-concept'],
|
||||
}));
|
||||
const chat = stubChat('A concept narrative long enough to produce at least one chunk.');
|
||||
await runPhaseSynthesizeConcepts(engine, { _atoms: atoms, _chat: chat });
|
||||
const rows = await engine.executeRaw<{ n: number }>(
|
||||
`SELECT count(*)::int AS n
|
||||
FROM content_chunks c JOIN pages p ON p.id = c.page_id
|
||||
WHERE p.slug = 'concepts/chunked-concept'`,
|
||||
);
|
||||
expect(Number(rows[0].n)).toBeGreaterThan(0);
|
||||
// Page metadata survives the importFromContent round-trip.
|
||||
const page = await engine.executeRaw<{ type: string; fm: Record<string, unknown> }>(
|
||||
`SELECT type, frontmatter AS fm FROM pages WHERE slug = 'concepts/chunked-concept'`,
|
||||
);
|
||||
expect(page[0].type).toBe('concept');
|
||||
expect((page[0].fm as Record<string, unknown>).tier).toBe('T1');
|
||||
});
|
||||
});
|
||||
|
||||
@@ -224,7 +224,7 @@ describeE2E('E2E: Git-to-DB Sync Pipeline', () => {
|
||||
expect(bob).toBeNull();
|
||||
});
|
||||
|
||||
test('sync skips non-syncable files (README, hidden, .raw)', async () => {
|
||||
test('sync skips non-syncable files (README, hidden, .raw) but imports ops/ (#2404)', async () => {
|
||||
const { performSync } = await import('../../src/commands/sync.ts');
|
||||
const engine = getEngine();
|
||||
|
||||
@@ -249,8 +249,9 @@ describeE2E('E2E: Git-to-DB Sync Pipeline', () => {
|
||||
const raw = await engine.getPage('.raw/data');
|
||||
expect(raw).toBeNull();
|
||||
|
||||
// ops/ is ordinary content and DOES sync (#2404).
|
||||
const ops = await engine.getPage('ops/deploy');
|
||||
expect(ops).toBeNull();
|
||||
expect(ops).not.toBeNull();
|
||||
});
|
||||
|
||||
test('sync stores last_commit and last_run in config', async () => {
|
||||
|
||||
@@ -12,6 +12,7 @@
|
||||
import { describe, test, expect, beforeAll, afterAll, beforeEach } from 'bun:test';
|
||||
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
|
||||
import { runExtractFacts } from '../src/core/cycle/extract-facts.ts';
|
||||
import { parseFactsFence } from '../src/core/facts-fence.ts';
|
||||
|
||||
let engine: PGLiteEngine;
|
||||
|
||||
@@ -99,11 +100,114 @@ describe('runExtractFacts — happy path', () => {
|
||||
);
|
||||
|
||||
expect(r2.guardTriggered).toBe(false);
|
||||
expect(r2.factsInserted).toBe(0);
|
||||
expect(r2.factsDeleted).toBe(0);
|
||||
expect(after2.rows.map((r: { fact: string }) => r.fact))
|
||||
.toEqual(after1.rows.map((r: { fact: string }) => r.fact));
|
||||
expect(after2.rows).toHaveLength(2);
|
||||
});
|
||||
|
||||
test('dedupes duplicate fence rows by claim and source without rewriting the fence', async () => {
|
||||
const body = FACT_FENCE(
|
||||
`| 1 | A | fact | 1.0 | world | medium | 2026-01-01 | | s | |
|
||||
| 2 | A | fact | 1.0 | world | medium | 2026-01-01 | | s | |`,
|
||||
);
|
||||
await putPage('people/alice', body);
|
||||
|
||||
const r1 = await runExtractFacts(engine, { slugs: ['people/alice'] });
|
||||
const r2 = await runExtractFacts(engine, { slugs: ['people/alice'] });
|
||||
|
||||
expect(r1.factsInserted).toBe(1);
|
||||
expect(r2.factsInserted).toBe(0);
|
||||
expect(r2.factsDeleted).toBe(0);
|
||||
|
||||
// The cycle dedups the derived DB index; it does not destructively
|
||||
// rewrite user-authored markdown fence rows.
|
||||
const page = await engine.getPage('people/alice', { sourceId: 'default' });
|
||||
expect(parseFactsFence(page?.compiled_truth ?? '').facts).toHaveLength(2);
|
||||
|
||||
// eslint-disable-next-line @typescript-eslint/no-explicit-any
|
||||
const rows = await (engine as any).db.query(
|
||||
`SELECT fact, source FROM facts WHERE source_markdown_slug = 'people/alice'`,
|
||||
);
|
||||
expect(rows.rows).toHaveLength(1);
|
||||
expect(rows.rows[0]).toMatchObject({ fact: 'A', source: 's' });
|
||||
});
|
||||
|
||||
test('same claim with a different source is not treated as duplicate', async () => {
|
||||
await putPage('people/alice', FACT_FENCE(
|
||||
`| 1 | Same claim | fact | 1.0 | world | medium | 2026-01-01 | | source-a | |`,
|
||||
));
|
||||
await runExtractFacts(engine, { slugs: ['people/alice'] });
|
||||
|
||||
await putPage('people/alice', FACT_FENCE(
|
||||
`| 1 | Same claim | fact | 1.0 | world | medium | 2026-01-01 | | source-a | |
|
||||
| 2 | Same claim | fact | 1.0 | world | medium | 2026-01-01 | | source-b | |`,
|
||||
));
|
||||
const r = await runExtractFacts(engine, { slugs: ['people/alice'] });
|
||||
|
||||
expect(r.factsInserted).toBe(1);
|
||||
expect(r.factsDeleted).toBe(0);
|
||||
|
||||
// eslint-disable-next-line @typescript-eslint/no-explicit-any
|
||||
const rows = await (engine as any).db.query(
|
||||
`SELECT fact, source FROM facts WHERE source_markdown_slug = 'people/alice' ORDER BY row_num`,
|
||||
);
|
||||
expect(rows.rows).toEqual([
|
||||
expect.objectContaining({ fact: 'Same claim', source: 'source-a' }),
|
||||
expect.objectContaining({ fact: 'Same claim', source: 'source-b' }),
|
||||
]);
|
||||
});
|
||||
|
||||
test('new fact added to the fence is inserted once without re-appending existing facts', async () => {
|
||||
await putPage('people/alice', FACT_FENCE(
|
||||
`| 1 | Existing | fact | 1.0 | world | medium | 2026-01-01 | | s | |`,
|
||||
));
|
||||
await runExtractFacts(engine, { slugs: ['people/alice'] });
|
||||
|
||||
await putPage('people/alice', FACT_FENCE(
|
||||
`| 1 | Existing | fact | 1.0 | world | medium | 2026-01-01 | | s | |
|
||||
| 2 | New | fact | 1.0 | world | medium | 2026-01-01 | | s | |`,
|
||||
));
|
||||
|
||||
const r = await runExtractFacts(engine, { slugs: ['people/alice'] });
|
||||
expect(r.factsInserted).toBe(1);
|
||||
expect(r.factsDeleted).toBe(0);
|
||||
|
||||
// eslint-disable-next-line @typescript-eslint/no-explicit-any
|
||||
const rows = await (engine as any).db.query(
|
||||
`SELECT fact FROM facts WHERE source_markdown_slug = 'people/alice' ORDER BY row_num`,
|
||||
);
|
||||
expect(rows.rows.map((row: { fact: string }) => row.fact)).toEqual(['Existing', 'New']);
|
||||
});
|
||||
|
||||
test('cli:-origin conversation facts (#1928) neither break idempotency nor get wiped', async () => {
|
||||
await putPage('people/alice', FACT_FENCE(
|
||||
`| 1 | Fence fact | fact | 1.0 | world | medium | 2026-01-01 | | s | |`,
|
||||
));
|
||||
// A conversation fact on the same page coordinate — NOT fence-owned.
|
||||
await engine.insertFacts(
|
||||
[{ fact: 'conversation fact', kind: 'fact', source: 'cli:extract-conversation-facts', row_num: 99, source_markdown_slug: 'people/alice' }],
|
||||
{ source_id: 'default' },
|
||||
);
|
||||
|
||||
const r1 = await runExtractFacts(engine, { slugs: ['people/alice'] });
|
||||
const r2 = await runExtractFacts(engine, { slugs: ['people/alice'] });
|
||||
|
||||
// The cli: row must not count as "stale" — a wipe/reinsert every cycle
|
||||
// would defeat idempotency (and churn factsDeleted/factsInserted).
|
||||
expect(r1.factsInserted).toBe(1);
|
||||
expect(r2.factsInserted).toBe(0);
|
||||
expect(r2.factsDeleted).toBe(0);
|
||||
|
||||
// eslint-disable-next-line @typescript-eslint/no-explicit-any
|
||||
const rows = await (engine as any).db.query(
|
||||
`SELECT fact FROM facts WHERE source_markdown_slug = 'people/alice' ORDER BY row_num`,
|
||||
);
|
||||
expect(rows.rows.map((row: { fact: string }) => row.fact))
|
||||
.toEqual(['Fence fact', 'conversation fact']);
|
||||
});
|
||||
|
||||
test('removed-from-fence row is deleted from DB (wipe-and-reinsert pattern)', async () => {
|
||||
// Seed: 2 facts.
|
||||
await putPage('people/alice', FACT_FENCE(
|
||||
|
||||
@@ -0,0 +1,119 @@
|
||||
import { describe, test, expect, beforeEach, afterEach } from 'bun:test';
|
||||
import type { BrainEngine } from '../src/core/engine.ts';
|
||||
import { MIGRATIONS, LATEST_VERSION } from '../src/core/migrate.ts';
|
||||
import { resetFtsLanguageCache } from '../src/core/fts-language.ts';
|
||||
|
||||
const ENV_KEY = 'GBRAIN_FTS_LANGUAGE';
|
||||
const originalLang = process.env[ENV_KEY];
|
||||
|
||||
beforeEach(() => {
|
||||
delete process.env[ENV_KEY];
|
||||
resetFtsLanguageCache();
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
delete process.env[ENV_KEY];
|
||||
if (originalLang !== undefined) process.env[ENV_KEY] = originalLang;
|
||||
resetFtsLanguageCache();
|
||||
});
|
||||
|
||||
describe('configurable_fts_language migration', () => {
|
||||
test('migration is registered', () => {
|
||||
const ftsMig = MIGRATIONS.find(m => m.name === 'configurable_fts_language');
|
||||
expect(ftsMig).toBeDefined();
|
||||
expect(ftsMig?.version).toBeGreaterThan(115);
|
||||
});
|
||||
|
||||
test('fts migration is the latest migration', () => {
|
||||
expect(MIGRATIONS.find(m => m.name === 'configurable_fts_language')?.version).toBe(LATEST_VERSION);
|
||||
});
|
||||
|
||||
test('ftsMig uses handler (not static SQL) because language interpolation is dynamic', () => {
|
||||
const ftsMig = MIGRATIONS.find(m => m.name === 'configurable_fts_language');
|
||||
expect(ftsMig?.sql).toBe('');
|
||||
expect(ftsMig?.handler).toBeTypeOf('function');
|
||||
});
|
||||
|
||||
test('ftsMig handler is async', () => {
|
||||
const ftsMig = MIGRATIONS.find(m => m.name === 'configurable_fts_language');
|
||||
// Async function check: the constructor name is 'AsyncFunction'
|
||||
expect(ftsMig?.handler?.constructor.name).toBe('AsyncFunction');
|
||||
});
|
||||
|
||||
test('migration handler issues recreate-function calls (smoke check via mock engine)', async () => {
|
||||
const ftsMig = MIGRATIONS.find(m => m.name === 'configurable_fts_language');
|
||||
const calls: string[] = [];
|
||||
|
||||
const mockEngine = {
|
||||
executeRaw: async (sql: string) => {
|
||||
calls.push(sql);
|
||||
return [];
|
||||
},
|
||||
} as unknown as BrainEngine;
|
||||
|
||||
process.env[ENV_KEY] = 'english';
|
||||
resetFtsLanguageCache();
|
||||
|
||||
await ftsMig?.handler?.(mockEngine);
|
||||
|
||||
// Default 'english' \u2014 no backfill, only 2 CREATE OR REPLACE calls.
|
||||
expect(calls.length).toBe(2);
|
||||
expect(calls[0]).toContain('CREATE OR REPLACE FUNCTION update_page_search_vector');
|
||||
expect(calls[0]).toContain("to_tsvector('english'");
|
||||
expect(calls[1]).toContain('CREATE OR REPLACE FUNCTION update_chunk_search_vector');
|
||||
expect(calls[1]).toContain("to_tsvector('english'");
|
||||
// v120/#1647 hardening must survive the CREATE OR REPLACE (which resets
|
||||
// proconfig): both recreated bodies pin search_path.
|
||||
expect(calls[0]).toContain('SET search_path = pg_catalog, public');
|
||||
expect(calls[1]).toContain('SET search_path = pg_catalog, public');
|
||||
});
|
||||
|
||||
test('non-english language triggers backfill', async () => {
|
||||
const ftsMig = MIGRATIONS.find(m => m.name === 'configurable_fts_language');
|
||||
const calls: string[] = [];
|
||||
|
||||
const mockEngine = {
|
||||
executeRaw: async (sql: string) => {
|
||||
calls.push(sql);
|
||||
return [];
|
||||
},
|
||||
} as unknown as BrainEngine;
|
||||
|
||||
process.env[ENV_KEY] = 'pt_br';
|
||||
resetFtsLanguageCache();
|
||||
|
||||
await ftsMig?.handler?.(mockEngine);
|
||||
|
||||
// pt_br \u2014 2 CREATE + 2 backfill UPDATEs = 4 calls
|
||||
expect(calls.length).toBe(4);
|
||||
expect(calls[0]).toContain("to_tsvector('pt_br'");
|
||||
expect(calls[1]).toContain("to_tsvector('pt_br'");
|
||||
expect(calls[2]).toMatch(/UPDATE pages/);
|
||||
expect(calls[3]).toContain("to_tsvector('pt_br'");
|
||||
expect(calls[3]).toMatch(/UPDATE content_chunks/);
|
||||
});
|
||||
|
||||
test('invalid language falls back to english (no SQL injection)', async () => {
|
||||
const ftsMig = MIGRATIONS.find(m => m.name === 'configurable_fts_language');
|
||||
const calls: string[] = [];
|
||||
|
||||
const mockEngine = {
|
||||
executeRaw: async (sql: string) => {
|
||||
calls.push(sql);
|
||||
return [];
|
||||
},
|
||||
} as unknown as BrainEngine;
|
||||
|
||||
process.env[ENV_KEY] = "english'; DROP TABLE pages; --";
|
||||
resetFtsLanguageCache();
|
||||
|
||||
await ftsMig?.handler?.(mockEngine);
|
||||
|
||||
// Falls back to english: 2 CREATE OR REPLACE only, no DROP TABLE in any SQL.
|
||||
expect(calls.length).toBe(2);
|
||||
for (const sql of calls) {
|
||||
expect(sql).not.toContain('DROP TABLE');
|
||||
expect(sql).toContain("to_tsvector('english'");
|
||||
}
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,93 @@
|
||||
import { describe, test, expect, beforeEach, afterEach } from 'bun:test';
|
||||
import { getFtsLanguage, resetFtsLanguageCache } from '../src/core/fts-language.ts';
|
||||
|
||||
const ENV_KEY = 'GBRAIN_FTS_LANGUAGE';
|
||||
|
||||
beforeEach(() => {
|
||||
delete process.env[ENV_KEY];
|
||||
resetFtsLanguageCache();
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
delete process.env[ENV_KEY];
|
||||
resetFtsLanguageCache();
|
||||
});
|
||||
|
||||
describe('getFtsLanguage', () => {
|
||||
test('defaults to english when env is unset', () => {
|
||||
expect(getFtsLanguage()).toBe('english');
|
||||
});
|
||||
|
||||
test('defaults to english when env is empty string', () => {
|
||||
process.env[ENV_KEY] = '';
|
||||
expect(getFtsLanguage()).toBe('english');
|
||||
});
|
||||
|
||||
test('defaults to english when env is whitespace', () => {
|
||||
process.env[ENV_KEY] = ' ';
|
||||
expect(getFtsLanguage()).toBe('english');
|
||||
});
|
||||
|
||||
test('reads valid pt_br config', () => {
|
||||
process.env[ENV_KEY] = 'pt_br';
|
||||
expect(getFtsLanguage()).toBe('pt_br');
|
||||
});
|
||||
|
||||
test('reads valid simple language name', () => {
|
||||
process.env[ENV_KEY] = 'spanish';
|
||||
expect(getFtsLanguage()).toBe('spanish');
|
||||
});
|
||||
|
||||
test('reads name with underscores and digits', () => {
|
||||
process.env[ENV_KEY] = 'custom_lang_v2';
|
||||
expect(getFtsLanguage()).toBe('custom_lang_v2');
|
||||
});
|
||||
|
||||
test('rejects names with quotes (SQL injection guard)', () => {
|
||||
process.env[ENV_KEY] = "english'; DROP TABLE pages; --";
|
||||
expect(getFtsLanguage()).toBe('english');
|
||||
});
|
||||
|
||||
test('rejects names with spaces', () => {
|
||||
process.env[ENV_KEY] = 'pt br';
|
||||
expect(getFtsLanguage()).toBe('english');
|
||||
});
|
||||
|
||||
test('rejects names with hyphens', () => {
|
||||
process.env[ENV_KEY] = 'pt-br';
|
||||
expect(getFtsLanguage()).toBe('english');
|
||||
});
|
||||
|
||||
test('rejects names starting with digit', () => {
|
||||
process.env[ENV_KEY] = '1lang';
|
||||
expect(getFtsLanguage()).toBe('english');
|
||||
});
|
||||
|
||||
test('rejects uppercase (Postgres config names are lowercase)', () => {
|
||||
process.env[ENV_KEY] = 'English';
|
||||
expect(getFtsLanguage()).toBe('english');
|
||||
});
|
||||
|
||||
test('caches after first read', () => {
|
||||
process.env[ENV_KEY] = 'pt_br';
|
||||
expect(getFtsLanguage()).toBe('pt_br');
|
||||
|
||||
// Mutate env after first read \u2014 cached value wins.
|
||||
process.env[ENV_KEY] = 'spanish';
|
||||
expect(getFtsLanguage()).toBe('pt_br');
|
||||
});
|
||||
|
||||
test('resetFtsLanguageCache clears cache', () => {
|
||||
process.env[ENV_KEY] = 'pt_br';
|
||||
expect(getFtsLanguage()).toBe('pt_br');
|
||||
|
||||
resetFtsLanguageCache();
|
||||
process.env[ENV_KEY] = 'spanish';
|
||||
expect(getFtsLanguage()).toBe('spanish');
|
||||
});
|
||||
|
||||
test('trims surrounding whitespace from valid value', () => {
|
||||
process.env[ENV_KEY] = ' pt_br ';
|
||||
expect(getFtsLanguage()).toBe('pt_br');
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,90 @@
|
||||
/**
|
||||
* #2607 — the `sync --full` git fast path applies the same prune gate as
|
||||
* incremental sync.
|
||||
*
|
||||
* Bug class: `collectSyncableFiles` on a git work tree takes the
|
||||
* `git ls-files` fast path, which historically filtered ONLY by
|
||||
* strategy/extension + .gitignore — no `pruneDir`, so `sync --full`
|
||||
* imported (and resurrected previously-soft-deleted) pages under dot-dirs
|
||||
* and vendored trees that incremental sync's `isSyncable` excludes. The two
|
||||
* enumeration modes cycled content in and out depending on which ran last.
|
||||
*
|
||||
* Fix: `isCollectibleForWalker` (shared by the git fast path AND the FS-walk
|
||||
* emit filter) now rejects any path with a segment `pruneDir` would block —
|
||||
* the same segment rule `classifySync` applies on the incremental path.
|
||||
*
|
||||
* No PGLite needed: `collectSyncableFiles` is pure filesystem + git.
|
||||
*/
|
||||
|
||||
import { describe, test, expect, beforeAll, afterAll } from 'bun:test';
|
||||
import { mkdtempSync, writeFileSync, rmSync, mkdirSync } from 'fs';
|
||||
import { execSync } from 'child_process';
|
||||
import { tmpdir } from 'os';
|
||||
import { join, relative } from 'path';
|
||||
import { collectSyncableFiles } from '../src/commands/import.ts';
|
||||
import { isSyncable } from '../src/core/sync.ts';
|
||||
|
||||
let repo: string;
|
||||
|
||||
function rel(files: string[]): string[] {
|
||||
return files.map((f) => relative(repo, f));
|
||||
}
|
||||
|
||||
beforeAll(() => {
|
||||
repo = mkdtempSync(join(tmpdir(), 'gbrain-fastpath-'));
|
||||
execSync('git init', { cwd: repo, stdio: 'pipe' });
|
||||
execSync('git config user.email "t@t.t"', { cwd: repo, stdio: 'pipe' });
|
||||
execSync('git config user.name "T"', { cwd: repo, stdio: 'pipe' });
|
||||
|
||||
// Ordinary content — must be collected.
|
||||
mkdirSync(join(repo, 'notes'), { recursive: true });
|
||||
writeFileSync(join(repo, 'notes/real.md'), '---\ntitle: Real\n---\nbody\n');
|
||||
mkdirSync(join(repo, 'ops'), { recursive: true });
|
||||
writeFileSync(join(repo, 'ops/tasks.md'), '---\ntitle: Tasks\n---\nbody\n');
|
||||
|
||||
// TRACKED files under excluded trees — `git ls-files` returns these, so
|
||||
// only the prune gate keeps them out (this is the #2607 divergence).
|
||||
mkdirSync(join(repo, '.obsidian'), { recursive: true });
|
||||
writeFileSync(join(repo, '.obsidian/plugin-notes.md'), 'not a page\n');
|
||||
mkdirSync(join(repo, 'vendor/pkg'), { recursive: true });
|
||||
writeFileSync(join(repo, 'vendor/pkg/notes.md'), 'vendored\n');
|
||||
mkdirSync(join(repo, 'node_modules/dep'), { recursive: true });
|
||||
writeFileSync(join(repo, 'node_modules/dep/CHANGELOG.md'), 'dep changelog\n');
|
||||
mkdirSync(join(repo, 'people/pedro.raw'), { recursive: true });
|
||||
writeFileSync(join(repo, 'people/pedro.raw/source.md'), 'raw sidecar\n');
|
||||
|
||||
// Metafiles — excluded on both routes (pre-existing #345 behavior).
|
||||
writeFileSync(join(repo, 'README.md'), '# repo\n');
|
||||
writeFileSync(join(repo, 'notes/index.md'), '# index\n');
|
||||
|
||||
execSync('git add -A -f && git commit -m "fixture"', { cwd: repo, stdio: 'pipe' });
|
||||
});
|
||||
|
||||
afterAll(() => {
|
||||
if (repo) rmSync(repo, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
describe('#2607 — git fast path excludes what incremental sync excludes', () => {
|
||||
test('tracked files under pruned dirs are NOT collected', () => {
|
||||
const files = rel(collectSyncableFiles(repo, { strategy: 'markdown' }));
|
||||
expect(files).toContain('notes/real.md');
|
||||
expect(files).toContain('ops/tasks.md'); // ordinary content (#2404)
|
||||
expect(files).not.toContain('.obsidian/plugin-notes.md');
|
||||
expect(files).not.toContain('vendor/pkg/notes.md');
|
||||
expect(files).not.toContain('node_modules/dep/CHANGELOG.md');
|
||||
expect(files).not.toContain('people/pedro.raw/source.md');
|
||||
// Metafiles stay excluded too.
|
||||
expect(files).not.toContain('README.md');
|
||||
expect(files).not.toContain('notes/index.md');
|
||||
});
|
||||
|
||||
test('full-sync enumeration agrees with incremental isSyncable for every collected file', () => {
|
||||
// The single-source-of-truth contract: nothing the full path collects may
|
||||
// be something the incremental path would refuse to sync.
|
||||
const files = rel(collectSyncableFiles(repo, { strategy: 'markdown' }));
|
||||
for (const f of files) {
|
||||
expect({ path: f, syncable: isSyncable(f) }).toEqual({ path: f, syncable: true });
|
||||
}
|
||||
expect(files.length).toBeGreaterThan(0);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,181 @@
|
||||
/**
|
||||
* withScopedReadTransaction — opt-in Postgres RLS source-scope binding
|
||||
* (GBRAIN_RLS_SCOPE_BINDING, lands community PR #2387).
|
||||
*
|
||||
* Behavioral pins, no real DB (fake postgres.js sql handle):
|
||||
* - flag OFF (default): TRUE pass-through — callback receives the shared
|
||||
* pool handle directly, no sql.begin(), no set_config. This is the
|
||||
* #1794-class guard: reads must not gain a per-read pool hold.
|
||||
* - flag OFF + alwaysTransaction (the search methods' SET LOCAL path):
|
||||
* sql.begin() opens, still no set_config — identical to master's wrap.
|
||||
* - flag ON: sql.begin() + SELECT set_config('app.scopes', $1, true)
|
||||
* with federated-array > scalar > '*' precedence, and the CSV value
|
||||
* carried as a BOUND PARAMETER, never interpolated into the SQL text.
|
||||
*/
|
||||
|
||||
import { describe, test, expect } from 'bun:test';
|
||||
import { PostgresEngine } from '../src/core/postgres-engine.ts';
|
||||
import { withEnv } from './helpers/with-env.ts';
|
||||
|
||||
type Recorded = { text: string; params: unknown[] };
|
||||
|
||||
function makeFakeSql() {
|
||||
const queries: Recorded[] = [];
|
||||
let beginCalls = 0;
|
||||
const record = (strings: TemplateStringsArray, ...params: unknown[]) => {
|
||||
// Join the literal segments with a placeholder marker so the test can
|
||||
// assert the exact SQL text shape around each bound parameter.
|
||||
queries.push({ text: strings.join('${}'), params });
|
||||
return Promise.resolve([]);
|
||||
};
|
||||
const sql = ((strings: TemplateStringsArray, ...params: unknown[]) =>
|
||||
record(strings, ...params)) as unknown as Record<string, unknown> & {
|
||||
(strings: TemplateStringsArray, ...params: unknown[]): Promise<unknown[]>;
|
||||
begin: (cb: (tx: unknown) => Promise<unknown>) => Promise<unknown>;
|
||||
};
|
||||
const tx = ((strings: TemplateStringsArray, ...params: unknown[]) =>
|
||||
record(strings, ...params)) as unknown as Record<string, unknown>;
|
||||
sql.begin = async (cb: (t: unknown) => Promise<unknown>) => {
|
||||
beginCalls++;
|
||||
return await cb(tx);
|
||||
};
|
||||
return { sql, tx, queries, beginCalls: () => beginCalls };
|
||||
}
|
||||
|
||||
function makeEngine(fake: ReturnType<typeof makeFakeSql>) {
|
||||
const e = new PostgresEngine();
|
||||
(e as unknown as { _sql: unknown })._sql = fake.sql;
|
||||
(e as unknown as { _connectionStyle: string })._connectionStyle = 'instance';
|
||||
// private method, invoked directly for the pin
|
||||
return e as unknown as {
|
||||
withScopedReadTransaction<T>(
|
||||
sourceIds: string[] | undefined,
|
||||
sourceId: string | undefined,
|
||||
cb: (tx: unknown) => Promise<T>,
|
||||
opts?: { alwaysTransaction?: boolean },
|
||||
): Promise<T>;
|
||||
};
|
||||
}
|
||||
|
||||
function setConfigQueries(queries: Recorded[]): Recorded[] {
|
||||
return queries.filter((q) => q.text.includes('set_config'));
|
||||
}
|
||||
|
||||
describe('withScopedReadTransaction / flag off (default)', () => {
|
||||
test('true pass-through: callback gets the shared pool handle, no begin, no set_config', async () => {
|
||||
await withEnv({ GBRAIN_RLS_SCOPE_BINDING: undefined }, async () => {
|
||||
const fake = makeFakeSql();
|
||||
const engine = makeEngine(fake);
|
||||
let received: unknown;
|
||||
const result = await engine.withScopedReadTransaction(undefined, 'src-a', async (tx) => {
|
||||
received = tx;
|
||||
return 42;
|
||||
});
|
||||
expect(result).toBe(42);
|
||||
expect(received).toBe(fake.sql); // the pool handle itself, not a tx
|
||||
expect(fake.beginCalls()).toBe(0);
|
||||
expect(setConfigQueries(fake.queries)).toHaveLength(0);
|
||||
});
|
||||
});
|
||||
|
||||
test('explicit "0" is off too', async () => {
|
||||
await withEnv({ GBRAIN_RLS_SCOPE_BINDING: '0' }, async () => {
|
||||
const fake = makeFakeSql();
|
||||
const engine = makeEngine(fake);
|
||||
await engine.withScopedReadTransaction(['a', 'b'], undefined, async () => null);
|
||||
expect(fake.beginCalls()).toBe(0);
|
||||
expect(setConfigQueries(fake.queries)).toHaveLength(0);
|
||||
});
|
||||
});
|
||||
|
||||
test('alwaysTransaction keeps master\'s sql.begin() wrap, still no set_config', async () => {
|
||||
await withEnv({ GBRAIN_RLS_SCOPE_BINDING: undefined }, async () => {
|
||||
const fake = makeFakeSql();
|
||||
const engine = makeEngine(fake);
|
||||
let received: unknown;
|
||||
await engine.withScopedReadTransaction(
|
||||
undefined,
|
||||
'src-a',
|
||||
async (tx) => {
|
||||
received = tx;
|
||||
return null;
|
||||
},
|
||||
{ alwaysTransaction: true },
|
||||
);
|
||||
expect(fake.beginCalls()).toBe(1);
|
||||
expect(received).toBe(fake.tx); // a transaction handle this time
|
||||
expect(setConfigQueries(fake.queries)).toHaveLength(0);
|
||||
});
|
||||
});
|
||||
});
|
||||
|
||||
describe('withScopedReadTransaction / flag on', () => {
|
||||
test('emits set_config(\'app.scopes\', ...) inside a transaction, before the callback', async () => {
|
||||
await withEnv({ GBRAIN_RLS_SCOPE_BINDING: '1' }, async () => {
|
||||
const fake = makeFakeSql();
|
||||
const engine = makeEngine(fake);
|
||||
let queriesAtCallback = -1;
|
||||
await engine.withScopedReadTransaction(undefined, 'src-a', async () => {
|
||||
queriesAtCallback = fake.queries.length;
|
||||
return null;
|
||||
});
|
||||
expect(fake.beginCalls()).toBe(1);
|
||||
const sc = setConfigQueries(fake.queries);
|
||||
expect(sc).toHaveLength(1);
|
||||
expect(sc[0].params).toEqual(['src-a']);
|
||||
// set_config was emitted before the callback ran
|
||||
expect(queriesAtCallback).toBe(1);
|
||||
expect(fake.queries[0]).toBe(sc[0]);
|
||||
});
|
||||
});
|
||||
|
||||
test('"true" also enables', async () => {
|
||||
await withEnv({ GBRAIN_RLS_SCOPE_BINDING: 'true' }, async () => {
|
||||
const fake = makeFakeSql();
|
||||
const engine = makeEngine(fake);
|
||||
await engine.withScopedReadTransaction(undefined, 'src-a', async () => null);
|
||||
expect(setConfigQueries(fake.queries)).toHaveLength(1);
|
||||
});
|
||||
});
|
||||
|
||||
test('federated array wins over scalar: CSV of sourceIds', async () => {
|
||||
await withEnv({ GBRAIN_RLS_SCOPE_BINDING: '1' }, async () => {
|
||||
const fake = makeFakeSql();
|
||||
const engine = makeEngine(fake);
|
||||
await engine.withScopedReadTransaction(['a', 'b', 'c'], 'ignored-scalar', async () => null);
|
||||
expect(setConfigQueries(fake.queries)[0].params).toEqual(['a,b,c']);
|
||||
});
|
||||
});
|
||||
|
||||
test('empty federated array falls back to scalar', async () => {
|
||||
await withEnv({ GBRAIN_RLS_SCOPE_BINDING: '1' }, async () => {
|
||||
const fake = makeFakeSql();
|
||||
const engine = makeEngine(fake);
|
||||
await engine.withScopedReadTransaction([], 'src-b', async () => null);
|
||||
expect(setConfigQueries(fake.queries)[0].params).toEqual(['src-b']);
|
||||
});
|
||||
});
|
||||
|
||||
test("unscoped (no sourceIds, no sourceId) binds '*'", async () => {
|
||||
await withEnv({ GBRAIN_RLS_SCOPE_BINDING: '1' }, async () => {
|
||||
const fake = makeFakeSql();
|
||||
const engine = makeEngine(fake);
|
||||
await engine.withScopedReadTransaction(undefined, undefined, async () => null);
|
||||
expect(setConfigQueries(fake.queries)[0].params).toEqual(['*']);
|
||||
});
|
||||
});
|
||||
|
||||
test('the scopes CSV is a BOUND PARAMETER, never interpolated into SQL text', async () => {
|
||||
await withEnv({ GBRAIN_RLS_SCOPE_BINDING: '1' }, async () => {
|
||||
const fake = makeFakeSql();
|
||||
const engine = makeEngine(fake);
|
||||
const hostile = "x','y'); DROP TABLE pages; --";
|
||||
await engine.withScopedReadTransaction(undefined, hostile, async () => null);
|
||||
const sc = setConfigQueries(fake.queries)[0];
|
||||
// Exact literal-segment shape: the value slot is the tagged-template hole.
|
||||
expect(sc.text).toBe("SELECT set_config('app.scopes', ${}, true)");
|
||||
expect(sc.params).toEqual([hostile]);
|
||||
expect(sc.text).not.toContain(hostile);
|
||||
});
|
||||
});
|
||||
});
|
||||
@@ -48,14 +48,34 @@ describe('postgres-engine / search path timeout isolation', () => {
|
||||
expect(bare).toBeNull();
|
||||
});
|
||||
|
||||
test('searchKeyword wraps its query in sql.begin()', () => {
|
||||
test('searchKeyword wraps its query in a transaction (via withScopedReadTransaction alwaysTransaction)', () => {
|
||||
// Post-RLS-scope-binding invariant: the search methods route through
|
||||
// withScopedReadTransaction with alwaysTransaction: true, which
|
||||
// guarantees a sql.begin() wrap in BOTH modes — flag off (identical to
|
||||
// master's pre-helper wrap) and flag on (scoped transaction with
|
||||
// set_config). See the helper tests in
|
||||
// test/postgres-engine-rls-scope.test.ts for the behavioral pins.
|
||||
const fn = extractMethod(SRC, 'searchKeyword');
|
||||
expect(fn).toMatch(/sql\.begin\s*\(\s*async\s+sql\s*=>/);
|
||||
expect(fn).toMatch(/withScopedReadTransaction\s*\(/);
|
||||
expect(fn).toMatch(/alwaysTransaction:\s*true/);
|
||||
});
|
||||
|
||||
test('searchVector wraps its query in sql.begin()', () => {
|
||||
test('searchVector wraps its query in a transaction (via withScopedReadTransaction alwaysTransaction)', () => {
|
||||
const fn = extractMethod(SRC, 'searchVector');
|
||||
expect(fn).toMatch(/sql\.begin\s*\(\s*async\s+sql\s*=>/);
|
||||
expect(fn).toMatch(/withScopedReadTransaction\s*\(/);
|
||||
expect(fn).toMatch(/alwaysTransaction:\s*true/);
|
||||
});
|
||||
|
||||
test('withScopedReadTransaction owns the sql.begin() wrap (and only opens it when needed)', () => {
|
||||
// (extractMethod can't grab this one: `private async ...<T>(`.)
|
||||
const stripped = stripComments(SRC);
|
||||
// The transaction lives in the helper...
|
||||
expect(stripped).toMatch(/this\.sql\.begin\s*\(/);
|
||||
// ...and the flag-off / non-alwaysTransaction path is a true
|
||||
// pass-through on the shared pool — no per-read transaction hold.
|
||||
expect(stripped).toMatch(
|
||||
/if\s*\(!this\.rlsScopeBindingEnabled\s*&&\s*!opts\?\.alwaysTransaction\)\s*\{\s*return\s+await\s+callback\(this\.sql\);/,
|
||||
);
|
||||
});
|
||||
|
||||
test('both search methods use SET LOCAL for the timeout', () => {
|
||||
|
||||
@@ -0,0 +1,152 @@
|
||||
import { describe, test, expect, beforeEach, afterEach } from 'bun:test';
|
||||
import type { BrainEngine } from '../src/core/engine.ts';
|
||||
import { runReindexSearchVector } from '../src/commands/reindex-search-vector.ts';
|
||||
import { resetFtsLanguageCache } from '../src/core/fts-language.ts';
|
||||
|
||||
const ENV_KEY = 'GBRAIN_FTS_LANGUAGE';
|
||||
const originalLang = process.env[ENV_KEY];
|
||||
|
||||
interface MockState {
|
||||
calls: string[];
|
||||
rowsToReturn: { pages: number; chunks: number };
|
||||
}
|
||||
|
||||
function makeMockEngine(state: MockState): BrainEngine {
|
||||
return {
|
||||
executeRaw: async (sql: string) => {
|
||||
state.calls.push(sql);
|
||||
// Inventory query — return the configured counts
|
||||
if (sql.includes('SELECT') && sql.includes('FROM pages WHERE search_vector')) {
|
||||
return [{ pages: state.rowsToReturn.pages, chunks: state.rowsToReturn.chunks }];
|
||||
}
|
||||
return [];
|
||||
},
|
||||
} as unknown as BrainEngine;
|
||||
}
|
||||
|
||||
beforeEach(() => {
|
||||
delete process.env[ENV_KEY];
|
||||
resetFtsLanguageCache();
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
delete process.env[ENV_KEY];
|
||||
if (originalLang !== undefined) process.env[ENV_KEY] = originalLang;
|
||||
resetFtsLanguageCache();
|
||||
});
|
||||
|
||||
describe('runReindexSearchVector', () => {
|
||||
test('--dry-run does not issue any DDL or backfill SQL', async () => {
|
||||
const state: MockState = { calls: [], rowsToReturn: { pages: 100, chunks: 500 } };
|
||||
const engine = makeMockEngine(state);
|
||||
|
||||
process.env[ENV_KEY] = 'pt_br';
|
||||
resetFtsLanguageCache();
|
||||
|
||||
const result = await runReindexSearchVector(engine, { dryRun: true, json: true });
|
||||
|
||||
expect(result.status).toBe('dry_run');
|
||||
expect(result.language).toBe('pt_br');
|
||||
expect(result.pagesUpdated).toBe(100);
|
||||
expect(result.chunksUpdated).toBe(500);
|
||||
expect(result.triggersRecreated).toBe(0);
|
||||
|
||||
// Only the inventory query — no CREATE OR REPLACE, no UPDATE.
|
||||
expect(state.calls.length).toBe(1);
|
||||
expect(state.calls[0]).toContain('SELECT');
|
||||
expect(state.calls[0]).not.toContain('CREATE OR REPLACE');
|
||||
expect(state.calls[0]).not.toContain('UPDATE');
|
||||
});
|
||||
|
||||
test('--yes recreates triggers + backfills with configured language', async () => {
|
||||
const state: MockState = { calls: [], rowsToReturn: { pages: 50, chunks: 200 } };
|
||||
const engine = makeMockEngine(state);
|
||||
|
||||
process.env[ENV_KEY] = 'pt_br';
|
||||
resetFtsLanguageCache();
|
||||
|
||||
const result = await runReindexSearchVector(engine, { yes: true, json: true });
|
||||
|
||||
expect(result.status).toBe('ok');
|
||||
expect(result.language).toBe('pt_br');
|
||||
expect(result.triggersRecreated).toBe(2);
|
||||
expect(result.pagesUpdated).toBe(50);
|
||||
expect(result.chunksUpdated).toBe(200);
|
||||
|
||||
// 1 inventory + 2 CREATE + 2 backfill batches (mock returns no rows, so
|
||||
// the keyset loop terminates after the first batch per table) = 5 calls
|
||||
expect(state.calls.length).toBe(5);
|
||||
expect(state.calls[1]).toContain('CREATE OR REPLACE FUNCTION update_page_search_vector');
|
||||
expect(state.calls[1]).toContain("to_tsvector('pt_br'");
|
||||
expect(state.calls[2]).toContain('CREATE OR REPLACE FUNCTION update_chunk_search_vector');
|
||||
expect(state.calls[2]).toContain("to_tsvector('pt_br'");
|
||||
expect(state.calls[3]).toMatch(/UPDATE pages/);
|
||||
expect(state.calls[4]).toMatch(/UPDATE content_chunks/);
|
||||
expect(state.calls[4]).toContain("to_tsvector('pt_br'");
|
||||
// v120/#1647 hardening must survive the CREATE OR REPLACE (which resets
|
||||
// proconfig): both recreated bodies pin search_path.
|
||||
expect(state.calls[1]).toContain('SET search_path = pg_catalog, public');
|
||||
expect(state.calls[2]).toContain('SET search_path = pg_catalog, public');
|
||||
});
|
||||
|
||||
test('default english language still recreates + backfills (no shortcut here)', async () => {
|
||||
// Note: unlike the configurable_fts_language migration, the CLI command
|
||||
// intentionally backfills even for english. The user explicitly asked for
|
||||
// it, so we honor it. The migration skips backfill for english because it
|
||||
// auto-runs on first apply.
|
||||
const state: MockState = { calls: [], rowsToReturn: { pages: 10, chunks: 30 } };
|
||||
const engine = makeMockEngine(state);
|
||||
|
||||
const result = await runReindexSearchVector(engine, { yes: true, json: true });
|
||||
|
||||
expect(result.status).toBe('ok');
|
||||
expect(result.language).toBe('english');
|
||||
expect(state.calls.length).toBe(5);
|
||||
|
||||
// Trigger recreates (calls 1, 2) and chunks backfill (call 4) embed the
|
||||
// language literal. Pages backfill (call 3) is UPDATE-to-self that
|
||||
// re-fires the trigger, so the language literal lives in the trigger
|
||||
// function body — not in the UPDATE statement.
|
||||
expect(state.calls[1]).toContain("'english'");
|
||||
expect(state.calls[2]).toContain("'english'");
|
||||
expect(state.calls[3]).toMatch(/UPDATE pages/);
|
||||
expect(state.calls[4]).toContain("'english'");
|
||||
});
|
||||
|
||||
test('SQL injection attempt falls back to english', async () => {
|
||||
const state: MockState = { calls: [], rowsToReturn: { pages: 10, chunks: 30 } };
|
||||
const engine = makeMockEngine(state);
|
||||
|
||||
process.env[ENV_KEY] = "english'; DROP TABLE pages; --";
|
||||
resetFtsLanguageCache();
|
||||
|
||||
const result = await runReindexSearchVector(engine, { yes: true, json: true });
|
||||
|
||||
expect(result.language).toBe('english');
|
||||
for (const sql of state.calls) {
|
||||
expect(sql).not.toContain('DROP TABLE');
|
||||
}
|
||||
});
|
||||
|
||||
test('empty inventory still completes successfully', async () => {
|
||||
const state: MockState = { calls: [], rowsToReturn: { pages: 0, chunks: 0 } };
|
||||
const engine = makeMockEngine(state);
|
||||
|
||||
const result = await runReindexSearchVector(engine, { yes: true, json: true });
|
||||
|
||||
expect(result.status).toBe('ok');
|
||||
expect(result.pagesUpdated).toBe(0);
|
||||
expect(result.chunksUpdated).toBe(0);
|
||||
expect(result.triggersRecreated).toBe(2);
|
||||
});
|
||||
|
||||
test('result includes durationMs', async () => {
|
||||
const state: MockState = { calls: [], rowsToReturn: { pages: 1, chunks: 1 } };
|
||||
const engine = makeMockEngine(state);
|
||||
|
||||
const result = await runReindexSearchVector(engine, { yes: true, json: true });
|
||||
|
||||
expect(typeof result.durationMs).toBe('number');
|
||||
expect(result.durationMs).toBeGreaterThanOrEqual(0);
|
||||
});
|
||||
});
|
||||
@@ -28,7 +28,8 @@ describe('#1433 — isSyncable / unsyncableReason are duals of one classifier',
|
||||
{ path: 'RESOLVER.md', expected: 'metafile', note: 'top-level master routing config (closes #345)' },
|
||||
{ path: 'brain/RESOLVER.md', expected: 'metafile', note: 'RESOLVER.md anywhere is metafile (closes #345)' },
|
||||
{ path: 'people/alice.txt', expected: 'strategy', note: '.txt rejected by markdown strategy' },
|
||||
{ path: 'ops/scratch/note.md', expected: 'pruned-dir', note: 'ops/ is pruned' },
|
||||
{ path: 'ops/scratch/note.md', expected: null, note: 'ops/ is ordinary content, not pruned (#2404)' },
|
||||
{ path: 'vendor/pkg/note.md', expected: 'pruned-dir', note: 'vendor/ is pruned' },
|
||||
{ path: '.git/notes.md', expected: 'pruned-dir', note: 'hidden dir pruned' },
|
||||
{ path: 'node_modules/foo/README.md', expected: 'pruned-dir', note: 'node_modules pruned' },
|
||||
];
|
||||
|
||||
@@ -0,0 +1,129 @@
|
||||
/**
|
||||
* #2404 — `ops/` is ordinary content: sync imports `ops/*` files and never
|
||||
* deletes `ops/*` pages.
|
||||
*
|
||||
* Bug class: `'ops'` was hardcoded in PRUNE_DIR_NAMES (a v0.2.0-era carve-out),
|
||||
* so `classifySync` treated ANY path with an `ops` segment as 'pruned-dir':
|
||||
* - committed `ops/*.md` files were never imported (even by `sync --full`);
|
||||
* - a modified `ops/*` file fell into the unsyncableModified delete loop,
|
||||
* whose #1433 guard only skipped 'metafile' — so put-created `ops/*` pages
|
||||
* (e.g. the bundled daily-task-manager's canonical `ops/tasks`) were
|
||||
* silently deleted on every sync.
|
||||
*
|
||||
* Fix: remove `'ops'` from PRUNE_DIR_NAMES, and harden the delete loop to also
|
||||
* skip 'pruned-dir' classifications (a page under a genuinely-pruned dir can
|
||||
* only exist via a deliberate put_page — never delete it on a file edit).
|
||||
*
|
||||
* Modeled on test/sync-metafile-skip.serial.test.ts (the #1433 iron rule).
|
||||
*/
|
||||
|
||||
import { describe, test, expect, beforeAll, afterAll, beforeEach, afterEach } from 'bun:test';
|
||||
import { mkdtempSync, writeFileSync, rmSync, mkdirSync } from 'fs';
|
||||
import { execSync } from 'child_process';
|
||||
import { tmpdir } from 'os';
|
||||
import { join } from 'path';
|
||||
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
|
||||
import { resetPgliteState } from './helpers/reset-pglite.ts';
|
||||
|
||||
let engine: PGLiteEngine;
|
||||
let repoPath: string;
|
||||
|
||||
function gitInit(repo: string): void {
|
||||
execSync('git init', { cwd: repo, stdio: 'pipe' });
|
||||
execSync('git config user.email "test@test.com"', { cwd: repo, stdio: 'pipe' });
|
||||
execSync('git config user.name "Test"', { cwd: repo, stdio: 'pipe' });
|
||||
}
|
||||
|
||||
describe('#2404 — ops/ pages sync like any other content', () => {
|
||||
beforeAll(async () => {
|
||||
engine = new PGLiteEngine();
|
||||
await engine.connect({});
|
||||
await engine.initSchema();
|
||||
}, 60_000);
|
||||
|
||||
afterAll(async () => {
|
||||
if (engine) await engine.disconnect();
|
||||
}, 60_000);
|
||||
|
||||
beforeEach(async () => {
|
||||
await resetPgliteState(engine);
|
||||
repoPath = mkdtempSync(join(tmpdir(), 'gbrain-ops-'));
|
||||
gitInit(repoPath);
|
||||
mkdirSync(join(repoPath, 'topics'), { recursive: true });
|
||||
writeFileSync(join(repoPath, 'topics/foo.md'), [
|
||||
'---', 'type: concept', 'title: Foo', '---', '', 'Baseline content.',
|
||||
].join('\n'));
|
||||
execSync('git add -A && git commit -m "initial"', { cwd: repoPath, stdio: 'pipe' });
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
if (repoPath) rmSync(repoPath, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
test('a committed ops/*.md file is imported by sync', async () => {
|
||||
const { performSync } = await import('../src/commands/sync.ts');
|
||||
mkdirSync(join(repoPath, 'ops'), { recursive: true });
|
||||
writeFileSync(join(repoPath, 'ops/tasks.md'), [
|
||||
'---', 'type: concept', 'title: Tasks', '---', '', 'Open tasks live here.',
|
||||
].join('\n'));
|
||||
execSync('git add -A && git commit -m "add ops/tasks"', { cwd: repoPath, stdio: 'pipe' });
|
||||
|
||||
const result = await performSync(engine, { repoPath, full: true, noPull: true, noEmbed: true });
|
||||
expect(['first_sync', 'synced']).toContain(result.status);
|
||||
|
||||
// Pre-fix: ops/* was 'pruned-dir' → imported=0 for it, even on --full.
|
||||
const page = await engine.getPage('ops/tasks');
|
||||
expect(page).not.toBeNull();
|
||||
expect(page?.compiled_truth).toContain('Open tasks');
|
||||
}, 60_000);
|
||||
|
||||
test('an edited ops/*.md updates its page instead of deleting it', async () => {
|
||||
const { performSync } = await import('../src/commands/sync.ts');
|
||||
mkdirSync(join(repoPath, 'ops'), { recursive: true });
|
||||
writeFileSync(join(repoPath, 'ops/tasks.md'), [
|
||||
'---', 'type: concept', 'title: Tasks', '---', '', 'v1',
|
||||
].join('\n'));
|
||||
execSync('git add -A && git commit -m "add ops/tasks"', { cwd: repoPath, stdio: 'pipe' });
|
||||
await performSync(engine, { repoPath, full: true, noPull: true, noEmbed: true });
|
||||
|
||||
writeFileSync(join(repoPath, 'ops/tasks.md'), [
|
||||
'---', 'type: concept', 'title: Tasks', '---', '', 'v2 with a new task',
|
||||
].join('\n'));
|
||||
execSync('git add -A && git commit -m "edit ops/tasks"', { cwd: repoPath, stdio: 'pipe' });
|
||||
|
||||
// Pre-fix: this incremental sync hit the unsyncableModified delete loop
|
||||
// ("Deleted un-syncable page: ops/tasks" — the autopilot kill-loop).
|
||||
const second = await performSync(engine, { repoPath, noPull: true, noEmbed: true });
|
||||
expect(['synced', 'up_to_date', 'first_sync']).toContain(second.status);
|
||||
|
||||
const page = await engine.getPage('ops/tasks');
|
||||
expect(page).not.toBeNull();
|
||||
expect(page?.compiled_truth).toContain('v2');
|
||||
}, 60_000);
|
||||
|
||||
test('hardening: a put-created page under a STILL-pruned dir survives a file edit (pruned-dir delete guard)', async () => {
|
||||
const { performSync } = await import('../src/commands/sync.ts');
|
||||
// node_modules stays in PRUNE_DIR_NAMES. Commit a file there, then seed a
|
||||
// same-path page via putPage — the deliberate-put precondition.
|
||||
mkdirSync(join(repoPath, 'node_modules/pkg'), { recursive: true });
|
||||
writeFileSync(join(repoPath, 'node_modules/pkg/notes.md'), 'v1\n');
|
||||
execSync('git add -A -f && git commit -m "vendor file"', { cwd: repoPath, stdio: 'pipe' });
|
||||
await performSync(engine, { repoPath, full: true, noPull: true, noEmbed: true });
|
||||
|
||||
await engine.putPage('node_modules/pkg/notes', {
|
||||
type: 'concept',
|
||||
title: 'Deliberate put page',
|
||||
compiled_truth: 'Created via put_page; must survive sync.',
|
||||
timeline: '',
|
||||
frontmatter: { type: 'concept' },
|
||||
});
|
||||
|
||||
writeFileSync(join(repoPath, 'node_modules/pkg/notes.md'), 'v2\n');
|
||||
execSync('git add -A -f && git commit -m "edit vendor file"', { cwd: repoPath, stdio: 'pipe' });
|
||||
await performSync(engine, { repoPath, noPull: true, noEmbed: true });
|
||||
|
||||
// Pre-fix: reason 'pruned-dir' was not guarded → page deleted.
|
||||
const survivor = await engine.getPage('node_modules/pkg/notes');
|
||||
expect(survivor).not.toBeNull();
|
||||
}, 60_000);
|
||||
});
|
||||
@@ -0,0 +1,142 @@
|
||||
/**
|
||||
* #2426 (bug 3) — `sync --full` delete-reconcile preserves DB-only pages.
|
||||
*
|
||||
* Bug class: the full-sync reconcile soft-deleted ANY file-backed page whose
|
||||
* `source_path` was absent from the working tree — including pages whose
|
||||
* markdown was NEVER committed to git (write-through that never made it to
|
||||
* the remote, then a fresh clone). "Absent from git" is the SYMPTOM of the
|
||||
* missing write-through commit, not evidence the content is disposable; one
|
||||
* production pass soft-deleted thousands of genuine pages this way.
|
||||
*
|
||||
* Fix: the reconcile partitions stale pages by git history — a path that ever
|
||||
* appeared as an ADD was genuinely deleted (reconcile as before); a path with
|
||||
* NO history is DB-only write-through: keep the page and re-export its
|
||||
* markdown to the working tree so it's file-backed again.
|
||||
*
|
||||
* Builds on the #2828 mass-delete valve (this guard covers the below-valve
|
||||
* cases the ratio check can't see).
|
||||
*/
|
||||
|
||||
import { describe, test, expect, beforeAll, afterAll, beforeEach, afterEach } from 'bun:test';
|
||||
import { mkdtempSync, writeFileSync, rmSync, mkdirSync, existsSync } from 'fs';
|
||||
import { execSync } from 'child_process';
|
||||
import { tmpdir } from 'os';
|
||||
import { join } from 'path';
|
||||
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
|
||||
import { resetPgliteState } from './helpers/reset-pglite.ts';
|
||||
import { listEverCommittedPaths } from '../src/commands/sync.ts';
|
||||
|
||||
let engine: PGLiteEngine;
|
||||
let repoPath: string;
|
||||
|
||||
function gitInit(repo: string): void {
|
||||
execSync('git init', { cwd: repo, stdio: 'pipe' });
|
||||
execSync('git config user.email "t@t.t"', { cwd: repo, stdio: 'pipe' });
|
||||
execSync('git config user.name "T"', { cwd: repo, stdio: 'pipe' });
|
||||
}
|
||||
|
||||
describe('listEverCommittedPaths (#2426)', () => {
|
||||
test('returns every path ever added, including later-deleted ones; null for non-git dirs', () => {
|
||||
const repo = mkdtempSync(join(tmpdir(), 'gbrain-ecp-'));
|
||||
try {
|
||||
gitInit(repo);
|
||||
writeFileSync(join(repo, 'kept.md'), 'kept\n');
|
||||
writeFileSync(join(repo, 'gone.md'), 'gone\n');
|
||||
execSync('git add -A && git commit -m add', { cwd: repo, stdio: 'pipe' });
|
||||
execSync('git rm -q gone.md && git commit -m rm', { cwd: repo, stdio: 'pipe' });
|
||||
|
||||
const set = listEverCommittedPaths(repo);
|
||||
expect(set).not.toBeNull();
|
||||
expect(set!.has('kept.md')).toBe(true);
|
||||
expect(set!.has('gone.md')).toBe(true); // deleted, but WAS committed
|
||||
expect(set!.has('never-committed.md')).toBe(false);
|
||||
|
||||
const plain = mkdtempSync(join(tmpdir(), 'gbrain-ecp-plain-'));
|
||||
try {
|
||||
expect(listEverCommittedPaths(plain)).toBeNull();
|
||||
} finally {
|
||||
rmSync(plain, { recursive: true, force: true });
|
||||
}
|
||||
} finally {
|
||||
rmSync(repo, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
describe('#2426 — full-sync reconcile keeps never-committed (DB-only) pages', () => {
|
||||
beforeAll(async () => {
|
||||
engine = new PGLiteEngine();
|
||||
await engine.connect({});
|
||||
await engine.initSchema();
|
||||
}, 60_000);
|
||||
|
||||
afterAll(async () => {
|
||||
if (engine) await engine.disconnect();
|
||||
}, 60_000);
|
||||
|
||||
beforeEach(async () => {
|
||||
await resetPgliteState(engine);
|
||||
repoPath = mkdtempSync(join(tmpdir(), 'gbrain-dbonly-'));
|
||||
gitInit(repoPath);
|
||||
mkdirSync(join(repoPath, 'topics'), { recursive: true });
|
||||
writeFileSync(join(repoPath, 'topics/keep.md'), [
|
||||
'---', 'type: concept', 'title: Keep', '---', '', 'still here',
|
||||
].join('\n'));
|
||||
writeFileSync(join(repoPath, 'topics/gone.md'), [
|
||||
'---', 'type: concept', 'title: Gone', '---', '', 'will be git-rm-ed',
|
||||
].join('\n'));
|
||||
execSync('git add -A && git commit -m initial', { cwd: repoPath, stdio: 'pipe' });
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
if (repoPath) rmSync(repoPath, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
test('genuinely-deleted pages reconcile; never-committed pages are kept and re-exported', async () => {
|
||||
const { performSync } = await import('../src/commands/sync.ts');
|
||||
|
||||
// Full sync #1: both file-backed pages land.
|
||||
const first = await performSync(engine, {
|
||||
repoPath, full: true, sourceId: 'default', noPull: true, noEmbed: true,
|
||||
});
|
||||
expect(['first_sync', 'synced']).toContain(first.status);
|
||||
expect(await engine.getPage('topics/keep')).not.toBeNull();
|
||||
expect(await engine.getPage('topics/gone')).not.toBeNull();
|
||||
|
||||
// A DB-only write-through casualty: the page row exists with a
|
||||
// source_path, but its file was never committed and is absent from the
|
||||
// clone (e.g. write-through was never pushed, then the repo was re-cloned).
|
||||
await engine.putPage('memories/lost', {
|
||||
type: 'concept',
|
||||
title: 'Lost write-through',
|
||||
compiled_truth: 'Years of content that must not be reconciled away.',
|
||||
timeline: '',
|
||||
frontmatter: { type: 'concept' },
|
||||
});
|
||||
await engine.executeRaw(
|
||||
`UPDATE pages SET source_path = $1 WHERE slug = $2 AND source_id = $3`,
|
||||
['memories/lost.md', 'memories/lost', 'default'],
|
||||
);
|
||||
|
||||
// A genuine deletion: topics/gone.md removed via git.
|
||||
execSync('git rm -q topics/gone.md && git commit -m "rm gone"', { cwd: repoPath, stdio: 'pipe' });
|
||||
await engine.setConfig('sync.repo_path', repoPath);
|
||||
|
||||
// Full sync #2 runs the delete-reconcile.
|
||||
const second = await performSync(engine, {
|
||||
repoPath, full: true, sourceId: 'default', noPull: true, noEmbed: true,
|
||||
});
|
||||
expect(['first_sync', 'synced']).toContain(second.status);
|
||||
|
||||
// The genuinely-deleted page is reconciled away…
|
||||
expect(await engine.getPage('topics/gone')).toBeNull();
|
||||
// …the still-present page survives…
|
||||
expect(await engine.getPage('topics/keep')).not.toBeNull();
|
||||
// …and the DB-only page is PRESERVED (pre-fix: soft-deleted here)…
|
||||
const lost = await engine.getPage('memories/lost');
|
||||
expect(lost).not.toBeNull();
|
||||
expect(lost?.compiled_truth).toContain('must not be reconciled');
|
||||
// …and re-exported to the working tree so it is file-backed again.
|
||||
expect(existsSync(join(repoPath, 'memories/lost.md'))).toBe(true);
|
||||
}, 120_000);
|
||||
});
|
||||
@@ -56,8 +56,10 @@ describe('isSyncable with strategy', () => {
|
||||
expect(isSyncable('.git/config.js', { strategy: 'code' })).toBe(false);
|
||||
// README.md is skipped under markdown
|
||||
expect(isSyncable('README.md', { strategy: 'markdown' })).toBe(false);
|
||||
// ops/ directory always skipped
|
||||
expect(isSyncable('ops/migrate.py', { strategy: 'code' })).toBe(false);
|
||||
// ops/ is ordinary content — NOT skipped (#2404)
|
||||
expect(isSyncable('ops/migrate.py', { strategy: 'code' })).toBe(true);
|
||||
// vendored trees always skipped
|
||||
expect(isSyncable('vendor/pkg/migrate.py', { strategy: 'code' })).toBe(false);
|
||||
// .raw/ sidecar always skipped
|
||||
expect(isSyncable('dir/.raw/code.ts', { strategy: 'code' })).toBe(false);
|
||||
});
|
||||
|
||||
+13
-5
@@ -95,9 +95,10 @@ describe('isSyncable', () => {
|
||||
expect(isSyncable('people/README.md')).toBe(false);
|
||||
});
|
||||
|
||||
test('rejects ops/ directory', () => {
|
||||
expect(isSyncable('ops/deploy-log.md')).toBe(false);
|
||||
expect(isSyncable('ops/config.md')).toBe(false);
|
||||
test('accepts ops/ — ordinary content directory, not pruned (#2404)', () => {
|
||||
expect(isSyncable('ops/deploy-log.md')).toBe(true);
|
||||
expect(isSyncable('ops/config.md')).toBe(true);
|
||||
expect(isSyncable('ops/tasks.md')).toBe(true);
|
||||
});
|
||||
|
||||
// ────────────────────────────────────────────────────────────────
|
||||
@@ -128,8 +129,15 @@ describe('pruneDir', () => {
|
||||
expect(pruneDir('.vscode')).toBe(false);
|
||||
});
|
||||
|
||||
test('blocks ops (gbrain operational dir)', () => {
|
||||
expect(pruneDir('ops')).toBe(false);
|
||||
test('allows ops — ordinary content dir, not a vendor tree (#2404)', () => {
|
||||
expect(pruneDir('ops')).toBe(true);
|
||||
});
|
||||
|
||||
test('blocks vendored / generated trees', () => {
|
||||
expect(pruneDir('vendor')).toBe(false);
|
||||
expect(pruneDir('dist')).toBe(false);
|
||||
expect(pruneDir('build')).toBe(false);
|
||||
expect(pruneDir('venv')).toBe(false);
|
||||
});
|
||||
|
||||
test('blocks *.raw sidecar dirs (gbrain convention)', () => {
|
||||
|
||||
@@ -0,0 +1,115 @@
|
||||
/**
|
||||
* #2426 (bug 1) — write-through reaches git on durability-hardened repos.
|
||||
*
|
||||
* Bug class: `put_page` / capture / enrichment wrote `.md` into
|
||||
* `sync.repo_path` but NOTHING ever committed it. The post-commit hook only
|
||||
* fires after a commit — and write-through never made one — so write-through
|
||||
* content accumulated uncommitted forever: never pushed, `last_sync_at`
|
||||
* frozen (HEAD never moved), and silently deleted by a later `sync --full`
|
||||
* delete-reconcile.
|
||||
*
|
||||
* Fix: `writePageThrough` best-effort commits the artifact (path-limited)
|
||||
* when the repo carries the gbrain durability post-commit hook (i.e. the
|
||||
* user opted in via `gbrain sources harden`); the hook then background-pushes.
|
||||
* Unhardened repos keep the old write-only behavior.
|
||||
*/
|
||||
|
||||
import { describe, test, expect, beforeAll, afterAll, beforeEach, afterEach } from 'bun:test';
|
||||
import { mkdtempSync, writeFileSync, rmSync, mkdirSync, chmodSync } from 'fs';
|
||||
import { execSync, execFileSync } from 'child_process';
|
||||
import { tmpdir } from 'os';
|
||||
import { join } from 'path';
|
||||
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
|
||||
import { resetPgliteState } from './helpers/reset-pglite.ts';
|
||||
import { writePageThrough } from '../src/core/write-through.ts';
|
||||
|
||||
let engine: PGLiteEngine;
|
||||
let repo: string;
|
||||
|
||||
function git(cwd: string, ...args: string[]): string {
|
||||
return execFileSync('git', ['-C', cwd, ...args], {
|
||||
stdio: ['ignore', 'pipe', 'pipe'], encoding: 'utf-8',
|
||||
}).trim();
|
||||
}
|
||||
|
||||
/** Install a hook file carrying the gbrain durability banner (the detection
|
||||
* key `isDurabilityHardened` looks for) with a no-op body so tests never
|
||||
* attempt a real push. */
|
||||
function installFakeDurabilityHook(repoPath: string): void {
|
||||
const hooksDir = join(repoPath, '.git', 'hooks');
|
||||
mkdirSync(hooksDir, { recursive: true });
|
||||
const hookPath = join(hooksDir, 'post-commit');
|
||||
writeFileSync(hookPath, [
|
||||
'#!/usr/bin/env bash',
|
||||
'# gbrain brain-durability post-commit hook (v0.42.44+)',
|
||||
'exit 0',
|
||||
'',
|
||||
].join('\n'));
|
||||
chmodSync(hookPath, 0o755);
|
||||
}
|
||||
|
||||
async function seedPage(slug: string): Promise<void> {
|
||||
await engine.putPage(slug, {
|
||||
type: 'concept',
|
||||
title: 'Write-through page',
|
||||
compiled_truth: 'Content that must reach git.',
|
||||
timeline: '',
|
||||
frontmatter: { type: 'concept' },
|
||||
});
|
||||
}
|
||||
|
||||
describe('#2426 — writePageThrough auto-commit', () => {
|
||||
beforeAll(async () => {
|
||||
engine = new PGLiteEngine();
|
||||
await engine.connect({});
|
||||
await engine.initSchema();
|
||||
}, 60_000);
|
||||
|
||||
afterAll(async () => {
|
||||
if (engine) await engine.disconnect();
|
||||
}, 60_000);
|
||||
|
||||
beforeEach(async () => {
|
||||
await resetPgliteState(engine);
|
||||
repo = mkdtempSync(join(tmpdir(), 'gbrain-wt-'));
|
||||
execSync('git init', { cwd: repo, stdio: 'pipe' });
|
||||
execSync('git config user.email "t@t.t"', { cwd: repo, stdio: 'pipe' });
|
||||
execSync('git config user.name "T"', { cwd: repo, stdio: 'pipe' });
|
||||
writeFileSync(join(repo, 'seed.md'), 'seed\n');
|
||||
execSync('git add -A && git commit -m init', { cwd: repo, stdio: 'pipe' });
|
||||
await engine.setConfig('sync.repo_path', repo);
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
if (repo) rmSync(repo, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
test('on a hardened repo, the write-through artifact is committed (path-limited)', async () => {
|
||||
installFakeDurabilityHook(repo);
|
||||
// Unrelated dirty edit — must NOT be swept into the write-through commit.
|
||||
writeFileSync(join(repo, 'seed.md'), 'dirty unrelated edit\n');
|
||||
|
||||
await seedPage('notes/hello');
|
||||
const result = await writePageThrough(engine, 'notes/hello');
|
||||
|
||||
expect(result.written).toBe(true);
|
||||
expect(result.committed).toBe(true);
|
||||
// The artifact is committed…
|
||||
expect(git(repo, 'log', '-1', '--format=%s')).toBe('gbrain: write-through notes/hello');
|
||||
expect(git(repo, 'log', '-1', '--name-only', '--format=')).toBe('notes/hello.md');
|
||||
expect(git(repo, 'status', '--porcelain', 'notes/hello.md')).toBe('');
|
||||
// …and the unrelated edit stays uncommitted (explicit-path discipline).
|
||||
expect(git(repo, 'status', '--porcelain', 'seed.md')).not.toBe('');
|
||||
}, 60_000);
|
||||
|
||||
test('on an unhardened repo, the file is written but NOT committed (no behavior change)', async () => {
|
||||
await seedPage('notes/plain');
|
||||
const result = await writePageThrough(engine, 'notes/plain');
|
||||
|
||||
expect(result.written).toBe(true);
|
||||
expect(result.committed).toBeUndefined();
|
||||
// Untracked, uncommitted — the pre-existing contract.
|
||||
expect(git(repo, 'status', '--porcelain', 'notes/plain.md')).toContain('?? notes/plain.md');
|
||||
expect(git(repo, 'log', '-1', '--format=%s')).toBe('init');
|
||||
}, 60_000);
|
||||
});
|
||||
Reference in New Issue
Block a user