v0.41.26.0 fix: dream --source + ingest junk titles + emoji-crash (supersedes #1559, #1561) (#1571)

* fix: dream --source/--source-id plumbs sourceId to runCycle (supersedes #1559)

Closes the silent-no-op class where `gbrain dream --source <id>` ran
the cycle but never wrote `last_full_cycle_at`, leaving
`gbrain doctor`'s cycle_freshness check stuck red forever.

Changes to src/commands/dream.ts:
- DreamArgs.source field; parseArgs recognizes --source <id> AND the
  --source-id alias (matches v0.37.7.0 #1167 naming across
  import/extract/graph-query)
- Argv validation: missing value → exit 2; repeated different values
  → exit 2; --source X --source-id Y conflict → exit 2; same-value
  repetition → accepted
- --help short-circuit ordering preserved with IRON-RULE comment +
  structural test guard
- runDream engine-null guard: --source requires a connected brain
- runDream resolveSourceId → archived-source guard via fetchSource
  from src/core/sources-load.ts (single-row SELECT that projects
  archived + handles pre-v0.26.5 schema via isUndefinedColumnError)
- Typed-error try/catch via isResolverUserError predicate: only
  swallows known resolver-user errors; TypeError / postgres errors
  propagate uncaught with stack trace so genuine programmer bugs
  aren't hidden behind operator-error UX
- Forwarded sourceId to runCycle; existing v0.38 writeback at
  cycle.ts:1947-1967 now actually fires
- --help text documents both flag names

Tests:
- test/dream-cli-flags.test.ts: structural assertions for new flags,
  help text, IRON-RULE comment guard, resolver/predicate wiring
- test/dream.test.ts: 13 PGLite integration cases covering happy
  path (the regression that closes PR #1559), back-compat, alias
  equivalence, all argv edge cases, engine-null, archived,
  --help short-circuit ordering, T3 typed-error propagation, and
  D5 end-to-end dream→checkCycleFreshness column-name drift guard

Plan + 11 decisions: ~/.claude/plans/system-instruction-you-are-working-starry-papert.md

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix: judgeSignificance uses canonical safeSplitIndex (closes #1559/#1561 emoji crash)

Closes the 2026-05-24 production SYNTH_PHASE_FAIL: 🤖 (U+1F916,
surrogate pair U+D83E U+DD16) at offset 3999 in a long telegram
transcript made the raw 4000-char slice produce a lone high
surrogate; Anthropic's JSON parser rejected the payload with "no
low surrogate in string"; the synthesize phase failed.

Changes to src/core/cycle/synthesize.ts:
- judgeSignificance head+tail slice routed through safeSplitIndex
  from src/core/text-safe.ts (already imported)
- Did NOT introduce safeSliceEnd from PRs #1559+#1561 — that helper
  re-introduces the case-3 bug src/core/text-safe.ts:18-21 documents
- Did NOT touch findBoundary — master already routes through
  safeSplitIndex per the v0.42.0.0 wave

Tests in test/cycle-synthesize.test.ts:
- New describe('judgeSignificance — UTF-16 safety') block
- test.each over head boundaries (offsets 3998-4001) AND tail
  boundaries (offsets 3999-4002) for an 8001-char content with
  the robot emoji placed at each
- Primary assertion: explicit unpaired-surrogate scan over the
  captured prompt (NOT JSON.stringify per codex C-11 — V8/JSCore
  do not throw on lone surrogates, so that assertion was weak)
- Sub-8000 short-content branch case: no slicing, emoji passes
  through unchanged

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix: expand error_page_title + add cloudflare_challenge_title (supersedes #1561)

Closes the bug class where scraper error pages with titles like
"Forbidden", "Access Denied", "Service Unavailable", "Robot Check",
and "Just a moment..." were slipping through the ingest gate
because the matcher only caught bare numeric codes (403/404/500...)
and "page not found". 232+ pages observed (202+ from straylight-
brain) were inflating page counts and tripping
content_sanity_audit_recent on every doctor run.

Changes to src/core/content-sanity.ts BUILT_IN_JUNK_PATTERNS:
- Expanded error_page_title regex to also catch forbidden,
  access denied, service unavailable, robot check, verify you are
  human (case-insensitive, anchored — so long-form essays about
  these topics still ingest fine)
- New cloudflare_challenge_title pattern with DISTINCT name from
  error_page_title (PR #1561 collapsed both into one name and lost
  audit signal — the new name preserves diagnosability in
  ~/.gbrain/audit/content-sanity-YYYY-Www.jsonl and doctor's
  content_sanity_audit_recent aggregation)
- Dropped PR #1561's bare-`error` matcher — too aggressive on
  legitimate concept/taxonomy pages titled exactly "Error"

Tests:
- test/content-sanity.test.ts: pattern-count locked at 7, new
  matches via test.each, over-match regression guard (legitimate
  prose titled "How to Handle Access Denied Errors" / "Error
  Boundary in React" etc. must pass), audit-name distinctness
  pinned
- test/import-file-content-sanity.test.ts: end-to-end
  ContentSanityBlockError via importFromContent for each new
  pattern family (D6 — assessor wiring coverage, not just regex)

Out of scope, filed in TODOS.md as TODO-V13-C: gbrain pages
audit-junk-titles legacy-cleanup command. Dropped from this PR
per codex outside-voice tension (T1) for ship-and-validate-
matchers-first discipline. The 200+ pre-existing scraper pages
already in the DB will get the destructive-cleanup operator
surface after ~1 week of production observation against this
matcher.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* chore: bump v0.41.23.0 + CHANGELOG + follow-up TODOs

VERSION + package.json bump to 0.41.23.0.

CHANGELOG voice: ELI10 lead naming the bug ("`gbrain dream --source
<id>` finally counts as a cycle"), then per-fix detail, then a
"To take advantage of v0.41.23.0" operator-action block and itemized
changes.

TODOS.md v0.41.23.x follow-ups:
- TODO-V13-A (P2): --max-pages plumbing (PR #1559's flag, deferred
  because CycleOpts has no maxPages field today)
- TODO-V13-B (P3): --source vs --source-id flag-name unification
  across all CLI commands
- TODO-V13-C (P2): gbrain pages audit-junk-titles legacy cleanup
  (deferred for ~1 week of matcher production observation)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* chore: bump v0.41.25.0 → v0.41.26.0 (leave headroom for in-flight PR)

Master shipped v0.41.23.0 + v0.41.24.0 mid-review; this branch
originally bumped to v0.41.25.0 post-merge. User flagged v0.41.26.0
to leave a slot open for another in-flight PR. No code changes;
VERSION + package.json + CHANGELOG header + "To take advantage"
section updated in lockstep.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Garry Tan
2026-05-27 08:52:36 -07:00
committed by GitHub
co-authored by Claude Opus 4.7
parent ff32fcaa78
commit 42d99b6fca
12 changed files with 1120 additions and 10 deletions
+187
View File
@@ -2,6 +2,193 @@
All notable changes to GBrain will be documented in this file.
## [0.41.26.0] - 2026-05-27
**`gbrain dream --source <id>` finally counts as a cycle.**
If you have more than one brain source — a personal brain plus
media-corpus plus a team brain — `gbrain dream --source media-corpus`
used to run the cycle but never tell anyone it ran. The doctor would
say "media-corpus cycle is 60 hours stale" right after you re-ran
dream. You re-ran it. Still stale. Forever. Cron jobs scoped to one
source kept looping with no way to make the freshness gate go green.
Fixed. Now `gbrain dream --source <id>` records the completion
timestamp on the source, so `gbrain doctor`'s `cycle_freshness` check
goes green the moment the cycle finishes. The flag also accepts
`--source-id` (matches the naming every other v0.37.7.0+ command uses
`gbrain import --source-id`, `gbrain extract --source-id`,
`gbrain graph-query --source-id`). Both names work; pick whichever
you already remember.
Two safety rails came along for the ride:
- `gbrain dream --source <unknown>` errors clean ("Source not found.
Run `gbrain sources list`...") instead of silently passing
through and writing nothing.
- `gbrain dream --source <archived>` refuses with a paste-ready
`gbrain sources restore <id>` hint instead of stamping a fresh
cycle timestamp on a hidden source (which would mask data
staleness if you later restored it).
**Bonus housekeeping: ingest now blocks more scraper junk.**
Cloudflare and WAF challenge pages with titles like `Just a
moment...`, `Forbidden`, `Access Denied`, `Service Unavailable`, and
`Robot Check` used to slip through ingest because the matcher only
caught bare numeric codes (`403`, `404`, `500`...). They now hit the
same gate that's always caught the numeric ones. Real essays titled
"How to handle Access Denied errors" or "Forbidden Knowledge" still
ingest fine — the gate only fires when the title is exclusively the
error phrase (with optional trailing whitespace), so longer-form
prose passes through.
The Cloudflare challenge title gets its own audit name
(`cloudflare_challenge_title`) instead of being grouped under
`error_page_title`, so operators reading the `~/.gbrain/audit/
content-sanity-YYYY-Www.jsonl` log or running `gbrain doctor` can
distinguish "this was a 403" from "this was a Cloudflare challenge".
Cleanup for the 200+ pre-existing scraper pages already in your DB
ships in a follow-up PR (`gbrain pages audit-junk-titles`) after
this matcher has a week of production observation. The ingest gate
is reversible (won't accept the page); the cleanup is destructive
(soft-delete) and deserves its own observation window before it
gains authority to delete.
**Third smaller fix: dream cycle no longer crashes on
emoji-at-offset-4000.**
The `judgeSignificance` step of the dream cycle was using a raw 4000-
character slice that could split the UTF-16 surrogate pair encoding
a non-BMP emoji (`🤖` and friends) right down the middle. Anthropic's
JSON parser would then reject the prompt with `"no low surrogate in
string"`, the whole synthesize phase would fail (`SYNTH_PHASE_FAIL`),
and your transcript wouldn't get processed. It happened on a real
telegram transcript on 2026-05-24.
`judgeSignificance` now routes both head and tail slices through the
canonical `safeSplitIndex` helper from `src/core/text-safe.ts` (the
same helper `findBoundary` has used since v0.42.0.0). Same content
budget, same trimming, just safe at the surrogate boundary. No more
emoji-class crashes.
This release supersedes community PRs #1559 and #1561 (both by
`@garrytan-agents`). PR #1559's `--max-pages` flag was dropped from
this release because the cycle plumbing doesn't exist to honor it
yet (shipping the flag would be a lying flag); filed as a v0.42+
follow-up.
### To take advantage of v0.41.26.0
`gbrain upgrade` should do this automatically. If you target a
specific source via cron or by hand:
1. **Replace** any `gbrain dream` cron line that was meant to scope
to a source but didn't have the flag working. Add `--source <id>`:
```bash
# Old (silently brain-wide, never wrote per-source freshness):
0 2 * * * gbrain dream --json >> ~/gbrain.log
# New (per-source freshness keeps doctor green):
0 2 * * * gbrain dream --source media-corpus --json >> ~/gbrain.log
```
2. **Verify** the freshness gate flips. After a successful run:
```bash
gbrain doctor --json | jq '.checks[] | select(.name=="cycle_freshness")'
# Should show status="ok" for the source you cycled.
```
3. **If any step fails or doctor doesn't go green,** file an issue:
https://github.com/garrytan/gbrain/issues with:
- output of `gbrain doctor`
- the exact `gbrain dream` command you ran
- the source's row from `sources` table (config column)
### Itemized changes
**dream --source / --source-id (supersedes PR #1559):**
- `DreamArgs.source` field added; `parseArgs` recognizes `--source
<id>` AND the alias `--source-id <id>`.
- Argv validation: missing value → exit 2; repeated with different
values → exit 2; `--source X --source-id Y` (conflict) → exit 2;
same value repeated is accepted.
- `--help` short-circuit ordering preserved (IRON RULE comment + test
guard); `gbrain dream --help --source whatever` always prints help
and exits 0.
- `runDream` engine-null guard: `--source` requires a connected brain
(matches the v0.37.7.0 #1167 pattern across import/extract/
graph-query).
- `runDream` archived-source guard: refuses with `gbrain sources
restore <id>` hint instead of stamping `last_full_cycle_at` on a
hidden source.
- `runDream` typed-error try/catch via `isResolverUserError`
predicate: only matches the resolver's known user-facing throws;
TypeError / postgres errors propagate with stack trace so genuine
bugs aren't hidden as operator errors.
- Forwarded `sourceId` to `runCycle`; the existing v0.38 writeback in
`cycle.ts:1947-1967` now actually fires.
- `--help` text documents both flag names and the `cycle_freshness`
unlock.
**Surrogate-safe trimming in synthesize.ts (PR #1559+#1561 wave commit
`78b93f3`, productionized correctly):**
- `judgeSignificance` slice routed through `safeSplitIndex` from
`src/core/text-safe.ts` (canonical helper).
- Did NOT introduce the duplicate `safeSliceEnd` helper from the
original PRs (it re-introduces the case-3 bug `text-safe.ts:18-21`
documents).
- Did NOT touch `findBoundary` — already routes through
`safeSplitIndex` in master.
**Expanded `error_page_title` regex (supersedes PR #1561):**
- Caught: `forbidden`, `access denied`, `service unavailable`,
`robot check`, `verify you are human` (case-insensitive,
anchored).
- Dropped PR #1561's bare-`error` matcher — too aggressive on
legitimate taxonomy pages titled "Error".
- New `cloudflare_challenge_title` pattern (separate name from
`error_page_title`) for the title-scoped Cloudflare challenge —
preserves audit-log diagnosability (PR #1561 collapsed both into
one name).
**Tests:**
- `test/dream-cli-flags.test.ts` — structural assertions for new
flags, help text, IRON-RULE comment guard.
- `test/dream.test.ts` — 12 PGLite integration cases covering
--source happy path (the bug fix regression), back-compat,
--source-id alias, repetition/conflict/missing-value, engine-null,
archived, --help short-circuit ordering, TypeError propagation
(the typed-error catch can't hide programmer bugs).
- `test/dream.test.ts` end-to-end dream→checkCycleFreshness case
(D5): closes column-rename drift bug class.
- `test/cycle-synthesize.test.ts` — UTF-16 safety describe block
with `test.each` over head + tail surrogate-boundary offsets;
primary assertion is an explicit unpaired-surrogate scan (NOT
`JSON.stringify`, which doesn't throw on lone surrogates in V8).
- `test/content-sanity.test.ts` — new pattern matches +
over-match regression guard ("How to Handle Access Denied
Errors" must pass) + audit-name distinctness.
- `test/import-file-content-sanity.test.ts` — end-to-end
`importFromContent` integration via `test.each` for each new
pattern family.
**Out of scope, filed in TODOS.md:**
- `gbrain dream --max-pages <n>` plumbing (PR #1559 dropped — needs
cycle-phase iteration refactor).
- `--source` vs `--source-id` flag-name unification across the CLI.
- `gbrain pages audit-junk-titles` legacy cleanup command (deferred
for ~1 week of matcher production observation before destructive
cleanup ships).
### For contributors
- Codex outside-voice review surfaced 18 findings against the plan;
11 absorbed inline (typed-error try/catch tightening, doctor check
separation, help-ordering documentation, repetition rules,
surrogate-safety contract, dropped weak `JSON.stringify`
assertion). Two substantive cross-model tensions resolved via
explicit user decision (T1 drop Fix 4, T3 typed-error catch).
- Plan + 11 AskUserQuestion decision points captured at
`~/.claude/plans/system-instruction-you-are-working-starry-papert.md`.
## [0.41.25.0] - 2026-05-27
+91
View File
@@ -1,5 +1,96 @@
# TODOS
## v0.41.20.x dream-source-ingest-titles follow-ups (v0.42+)
- **TODO-V13-A (P2): `gbrain dream --max-pages <n>` plumbing.**
PR #1559 included a `--max-pages` flag for cost-bounded cycles on
large brains. v0.41.20 dropped it because `CycleOpts` has no `maxPages`
field and no cycle phase consults page-count limits — shipping the flag
would have been a lying flag.
- **What:** extend `CycleOpts` with `maxPages: number | undefined` and
thread it through extract phases (extract.ts, extract-facts.ts,
recompute-emotional-weight.ts) so per-source cost-bounded cycles
become real.
- **Why:** straylight-brain-class corpora (100K+ pages) benefit from
capping each cycle's work. Today operators have to wait full
extract sweeps regardless of cost.
- **Pros:** closes the lying-flag class; real cost brake.
- **Cons:** real refactor — extract phases iterate all pages today,
not page-count-bounded.
- **Context:** PR #1559 commit 67f98ca had the flag; the v0.41.20
plan dropped it under "Out of scope" with this TODO as the
forwarding pointer.
- **Depends on:** CycleOpts type extension + extract page-iteration
refactor + decision on per-phase vs per-cycle cap semantics.
- **TODO-V13-B (P3): `--source` / `--source-id` flag-name unification.**
Current drift: `dream`, `recall`, `sync` accept `--source`;
`import`, `extract`, `graph-query`, `sources` accept `--source-id`.
v0.41.20 added `--source-id` as an alias for dream's `--source` so
both work, but the codebase still ships two surface names.
- **What:** pick one canonical flag name across all CLI commands;
deprecate the other with a stderr warning; update doctor.ts
hint to match.
- **Why:** ergonomic consistency. Users who learned `--source-id`
via import shouldn't trip on `--source` in dream.
- **Pros:** ends a real user-facing confusion.
- **Cons:** low-priority polish; both names work today via alias.
- **Context:** doctor.ts historically pinned `--source`; v0.37.7.0
#1167 standardized `--source-id` across new commands. Recommend
picking `--source-id` for v0.37.7.0+ consistency and deprecating
`--source` over one minor.
- **Depends on:** nothing technical.
- **TODO-V13-C (P2): `gbrain pages audit-junk-titles` legacy cleanup.**
v0.41.20 widened the `error_page_title` matcher to catch Cloudflare /
WAF challenge titles ("Forbidden", "Access Denied", "Service
Unavailable", "Robot Check", "Just a moment...") at ingest. But the
200+ scraper pages already in production DBs (202+ from
straylight-brain) are NOT cleaned up by the matcher widening.
Dropped from v0.41.20 per codex outside-voice tension (T1) for
ship-and-validate-matchers-first discipline.
- **What:** new operator command for soft-deleting pre-existing
scraper-junk pages whose titles match the expanded
`BUILT_IN_JUNK_PATTERNS`. Full spec preserved:
- Signature: `gbrain pages audit-junk-titles [--source <id>]
[--dry-run|--apply] [--confirm-destructive] [--json]`
- Default `--dry-run`. Prints `{pattern_name: count, sample_slugs}`.
- `--apply` requires `--confirm-destructive` when match count
exceeds `DESTRUCTIVE_THRESHOLD` (reuse v0.26.5 constant).
- `--source <id>` scopes; without it, audits all non-archived
sources (filter via `listAllSources().filter(s => !s.archived)`).
- Soft-delete via existing `engine.softDeletePage(slug, sourceId)`.
- Audit JSONL via `logContentSanityEvent` with event kind
`junk_title_soft_deleted`.
- Idempotent.
- **Hybrid SQL+JS scanner**: pure
`buildJunkTitleSqlClause(patterns)` +
`scanForJunkTitles(rows, patterns)`. SQL pre-filter avoids
streaming all rows over the wire (perf rationale: even seq-scan
ILIKE beats JS regex per-row via the postgres driver).
- **`cleanup_safe: boolean` flag** per JunkPattern (codex C-13):
only patterns flagged `cleanup_safe: true` are eligible for
destructive cleanup. Stops future matcher widening from
automatically expanding destructive scope. Initial allowlist:
`cloudflare_attention_required`, `cloudflare_just_a_moment`,
`cloudflare_ray_id`, `access_denied`, `captcha_required`,
`error_page_title` (only the literal-numeric parts; the new
word-titles get `cleanup_safe: false` until the matcher proves
itself further), `cloudflare_challenge_title`.
- New doctor check `scraper_junk_pages_legacy` (separate from
`content_sanity_audit_recent` per codex C-5 — audit-log reader
vs live DB scan are different concerns).
- Tests: `test/pages-audit-junk-titles.test.ts` (hermetic PGLite),
`test/doctor.test.ts` extension.
- **Why:** ingest gate alone leaves 200+ existing junk pages
inflating page counts; this command closes the data-debt gap.
- **Pros:** finishes the cleanup story.
- **Cons:** destructive surface (soft-delete + audit JSONL).
- **Depends on:** ~1 week of production observation against
v0.41.20's new ingest matchers. If real-world reports surface
false-positive blocks, refine the matcher AND the `cleanup_safe`
allowlist before shipping the destructive command.
## v0.41.22.1 brainstorm judge fix-wave follow-ups (v0.42+)
Filed from the v0.41.22.1 plan-eng-review per cross-model-tension D13c.
+1 -1
View File
@@ -1 +1 @@
0.41.25.0
0.41.26.0
+1 -1
View File
@@ -140,5 +140,5 @@
"bun": ">=1.3.10"
},
"license": "MIT",
"version": "0.41.25.0"
"version": "0.41.26.0"
}
+146
View File
@@ -30,6 +30,8 @@ import {
type CyclePhase,
type CycleReport,
} from '../core/cycle.ts';
import { resolveSourceId } from '../core/source-resolver.ts';
import { fetchSource } from '../core/sources-load.ts';
import { existsSync } from 'fs';
import { resolve } from 'node:path';
@@ -54,10 +56,40 @@ interface DreamArgs {
* Never auto-applied for --input (codex finding #3).
*/
bypassDreamGuard: boolean;
/**
* v0.41.13: per-source cycle scoping. Threaded into runCycle as
* `sourceId` so `cycle.ts:1947-1967` writes `last_full_cycle_at`
* to `sources.config` on success — without it, `gbrain doctor`'s
* `cycle_freshness` check stays stale forever. Accepts `--source
* <id>` and the alias `--source-id <id>` (the v0.37.7.0 #1167
* canonical name across import/extract/graph-query); both work
* until a follow-up CLI cleanup picks one. Supersedes PR #1559.
*/
source: string | null;
}
const ISO_DATE_RE = /^\d{4}-\d{2}-\d{2}$/;
/**
* Collect every occurrence of `--<flag> <value>` in argv. Used to
* detect repeated flags with different values (e.g.
* `--source X --source Y`) and to surface a clean usage error
* instead of silently last-wins. Repeated identical values are
* collapsed to one (no-op). Missing values (flag at end of argv)
* return null to let the caller raise an explicit usage error
* rather than fall through with `undefined`.
*/
function collectFlagValues(args: string[], flag: string): string[] | null {
const values: string[] = [];
for (let i = 0; i < args.length; i++) {
if (args[i] !== flag) continue;
const v = args[i + 1];
if (v === undefined) return null; // flag at end of argv
values.push(v);
}
return values;
}
function parseArgs(args: string[]): DreamArgs {
const phaseIdx = args.indexOf('--phase');
const rawPhase = phaseIdx !== -1 ? args[phaseIdx + 1] : null;
@@ -110,6 +142,43 @@ function parseArgs(args: string[]): DreamArgs {
// --input implies --phase synthesize.
if (inputFile && !phase) phase = 'synthesize';
// v0.41.13: --source <id> (and the --source-id alias) drives per-source
// cycle scoping. Resolution rules:
// - missing value (flag at end of argv) → exit 2 with usage
// - repeated with different values (e.g. --source X --source Y) → exit 2
// - --source X --source-id Y (conflicting flag aliases) → exit 2
// - --source X --source X (or --source-id repeated with same value) → accepted
// - --help short-circuits BEFORE this block fires (see runDream).
// Closes the PR #1559 silent-no-op class through a clean argv contract.
const sourceValues = collectFlagValues(args, '--source');
const sourceIdValues = collectFlagValues(args, '--source-id');
if (sourceValues === null) {
console.error('--source <id>: missing value. Usage: gbrain dream --source <source-id>');
process.exit(2);
}
if (sourceIdValues === null) {
console.error('--source-id <id>: missing value. Usage: gbrain dream --source-id <source-id>');
process.exit(2);
}
const uniqSource = Array.from(new Set(sourceValues));
const uniqSourceId = Array.from(new Set(sourceIdValues));
if (uniqSource.length > 1) {
console.error(`specify --source once; got [${uniqSource.map(v => `"${v}"`).join(', ')}]`);
process.exit(2);
}
if (uniqSourceId.length > 1) {
console.error(`specify --source-id once; got [${uniqSourceId.map(v => `"${v}"`).join(', ')}]`);
process.exit(2);
}
if (uniqSource.length === 1 && uniqSourceId.length === 1 && uniqSource[0] !== uniqSourceId[0]) {
console.error(
`use --source OR --source-id, not both (different values): ` +
`--source="${uniqSource[0]}" vs --source-id="${uniqSourceId[0]}"`,
);
process.exit(2);
}
const source = uniqSource[0] ?? uniqSourceId[0] ?? null;
return {
json: args.includes('--json'),
dryRun: args.includes('--dry-run'),
@@ -122,6 +191,7 @@ function parseArgs(args: string[]): DreamArgs {
from,
to,
bypassDreamGuard: args.includes('--unsafe-bypass-dream-guard'),
source,
};
}
@@ -183,6 +253,14 @@ Options:
--pull git pull the brain repo before syncing (default: no pull)
--dir <path> Brain directory (default: configured brain)
--source <id> Scope the cycle to one source so doctor's
cycle_freshness check sees a fresh stamp on
completion. Without this, gbrain dream's
timestamp never lands and federated brains
see "stale cycle" forever.
--source-id <id> Alias for --source. Matches the v0.37.7.0+
naming used by import/extract/graph-query.
--input <file> Synthesize a specific transcript file (implies
--phase synthesize). Bypasses corpus-dir scan.
--date YYYY-MM-DD Synthesize transcripts dated for one specific day.
@@ -267,14 +345,81 @@ function printHuman(report: CycleReport) {
// ─── CLI entry ─────────────────────────────────────────────────────
/**
* Predicate: is this error one of the resolver's user-facing throws
* we want to surface as a clean stderr line + exit 1?
*
* Matches the message prefixes thrown from
* `src/core/source-resolver.ts:resolveSourceId` and
* `assertSourceExists`. Anything else (TypeError / ReferenceError /
* postgres connection failures / unexpected bugs) is intentionally
* NOT caught — those propagate to Bun's default unhandled handler
* with a stack trace so genuine programmer bugs aren't hidden as
* if they were operator errors. (Plan D-T3, codex C-7.)
*/
function isResolverUserError(e: unknown): boolean {
if (!(e instanceof Error)) return false;
const m = e.message;
return (m.startsWith('Source "') && m.includes(' not found.'))
|| m.startsWith('Invalid --source value')
|| m.startsWith('Invalid GBRAIN_SOURCE value');
}
export async function runDream(engine: BrainEngine | null, args: string[]): Promise<CycleReport | void> {
const opts = parseArgs(args);
// ─── IRON RULE: --help short-circuits BEFORE any engine-bearing work ─
// Tests pin this ordering so `gbrain dream --help --source whatever`
// ALWAYS prints help and exits 0, never reaching the engine-null gate
// below. If you reorder this, dream-cli-flags.test.ts will fail.
if (opts.help) {
printHelp();
return;
}
// v0.41.13: --source <id> resolution. Three guards in order:
// 1. engine null → exit 1 (the writeback in cycle.ts requires a
// DB connection; without engine we'd silently fail the same way
// PR #1559 was created to fix)
// 2. resolveSourceId throws on unknown id → typed-error catch
// surfaces clean message; non-resolver throws propagate
// 3. archived source → exit 1 with restore hint (writing
// last_full_cycle_at to an archived source would mask data
// staleness when the source is later restored)
let resolvedSourceId: string | undefined;
if (opts.source !== null) {
if (engine === null) {
console.error(
'gbrain dream --source <id> requires a connected brain ' +
'(no engine available); omit --source or run `gbrain init` first',
);
process.exit(1);
}
try {
resolvedSourceId = await resolveSourceId(engine, opts.source);
} catch (e) {
if (isResolverUserError(e)) {
console.error((e as Error).message);
process.exit(1);
}
throw e; // genuine bugs propagate with stack trace
}
// Archived-source guard via fetchSource from sources-load.ts
// (single-row SELECT that projects `archived` and falls back to
// pre-v0.26.5 schemas via isUndefinedColumnError catch — same
// legacy-safety net the rest of the codebase uses). engine's
// built-in listAllSources defaults to includeArchived=false AND
// doesn't project the archived column, so it cannot be used here.
const src = await fetchSource(engine, resolvedSourceId);
if (src?.archived === true) {
console.error(
`source ${resolvedSourceId} is archived; restore with ` +
`\`gbrain sources restore ${resolvedSourceId}\` before cycling`,
);
process.exit(1);
}
}
const brainDir = await resolveBrainDir(engine, opts.dir);
const phases: CyclePhase[] | undefined = opts.phase ? [opts.phase] : undefined;
@@ -283,6 +428,7 @@ export async function runDream(engine: BrainEngine | null, args: string[]): Prom
dryRun: opts.dryRun,
pull: opts.pull,
phases,
sourceId: resolvedSourceId, // undefined when --source not set → legacy back-compat
synthInputFile: opts.inputFile ?? undefined,
synthDate: opts.date ?? undefined,
synthFrom: opts.from ?? undefined,
+26 -2
View File
@@ -163,10 +163,34 @@ export const BUILT_IN_JUNK_PATTERNS: ReadonlyArray<JunkPattern> = Object.freeze(
applies_to: 'both',
},
// Bare error-page titles. Anchored so the title is exclusively the
// error code — a thoughtful page ABOUT 404 errors won't trip.
// error phrase — a thoughtful page ABOUT 404 errors or one titled
// "How to Handle Access Denied Errors" won't trip.
//
// v0.41.13 (supersedes PR #1561): expanded from bare numeric codes
// + "page not found" to also catch Cloudflare/WAF challenge
// titles ("Forbidden", "Access Denied", "Service Unavailable",
// "Robot Check", "Verify You Are Human"). Deliberately drops PR
// #1561's bare-`error` matcher (would false-positive on
// legitimate taxonomy pages titled "Error"). 232+ scraper pages
// motivating this change (202+ from straylight-brain).
{
name: 'error_page_title',
pattern: /^(403|404|500|502|503|error \d{3}|page not found)\s*$/i,
pattern: /^(403|404|500|502|503|error \d{3}|page not found|forbidden|access denied|service unavailable|robot check|verify you are human)\s*$/i,
applies_to: 'title',
},
// Cloudflare challenge title (companion to the body-scoped
// `cloudflare_just_a_moment` pattern above, which requires both
// phrase + URL). The title alone is a sufficient signal because
// legitimate pages don't title themselves "Just a moment...".
//
// v0.41.13: distinct name from `error_page_title` so audit JSONL
// (`~/.gbrain/audit/content-sanity-YYYY-Www.jsonl`) and doctor's
// `content_sanity_audit_recent` aggregation stay diagnosable.
// PR #1561 reused the `error_page_title` name and collapsed audit
// signal; we don't.
{
name: 'cloudflare_challenge_title',
pattern: /^just a moment\.{0,3}$/i,
applies_to: 'title',
},
]);
+20 -3
View File
@@ -832,9 +832,26 @@ export async function judgeSignificance(
// Truncate the transcript at 8K chars for cost control. Haiku's verdict
// doesn't need the full body; the opening + closing sections are usually
// representative of significance.
const trimmed = t.content.length > 8000
? t.content.slice(0, 4000) + '\n[...truncated...]\n' + t.content.slice(-4000)
: t.content;
//
// v0.41.13 surrogate-safety (supersedes PRs #1559+#1561's safeSliceEnd
// helper; see text-safe.ts:18-21 module docstring for why that helper
// re-introduces the case-3 bug the canonical safeSplitIndex was written
// to fix). Routes head + tail slicing through safeSplitIndex so an emoji
// at offset 4000 (or length-4000) never produces a lone surrogate that
// Anthropic's JSON parser rejects ("no low surrogate in string", caught
// 2026-05-24 on telegram).
//
// Contract: this branch only runs when content.length > 8000, so
// length - 4000 > 4000 > 0 — safeSplitIndex never sees an out-of-range
// maxChars here. (Codex C-10 documented contract.)
let trimmed: string;
if (t.content.length > 8000) {
const headEnd = safeSplitIndex(t.content, 4000);
const tailStart = safeSplitIndex(t.content, t.content.length - 4000);
trimmed = t.content.slice(0, headEnd) + '\n[...truncated...]\n' + t.content.slice(tailStart);
} else {
trimmed = t.content;
}
const sys = `You judge whether a conversation transcript is worth synthesizing into a personal knowledge brain.
+119 -3
View File
@@ -106,8 +106,8 @@ describe('assessContentSanity — size boundaries', () => {
// ─── 6 BUILT-IN PATTERNS ──────────────────────────────────────
describe('assessContentSanity — built-in junk patterns', () => {
test('built-in pattern count is locked at 6 (D3 dropped empty_body_with_source_url)', () => {
expect(BUILT_IN_JUNK_PATTERNS.length).toBe(6);
test('built-in pattern count is locked at 7 (v0.41.13 added cloudflare_challenge_title)', () => {
expect(BUILT_IN_JUNK_PATTERNS.length).toBe(7);
const names = BUILT_IN_JUNK_PATTERNS.map((p) => p.name);
expect(names).toContain('cloudflare_attention_required');
expect(names).toContain('cloudflare_just_a_moment');
@@ -115,8 +115,10 @@ describe('assessContentSanity — built-in junk patterns', () => {
expect(names).toContain('access_denied');
expect(names).toContain('captcha_required');
expect(names).toContain('error_page_title');
// v0.41.13: distinct name from error_page_title (audit-name distinctness).
expect(names).toContain('cloudflare_challenge_title');
// D3 regression: this rule was dropped. If it ever returns, the test
// count above bumps to 7 deliberately.
// count above bumps deliberately.
expect(names).not.toContain('empty_body_with_source_url');
});
@@ -218,6 +220,120 @@ describe('assessContentSanity — built-in junk patterns', () => {
});
expect(r.junk_pattern_matches).toContain('cloudflare_attention_required');
});
// ─── v0.41.13: expanded error_page_title + cloudflare_challenge_title ─
//
// Supersedes PR #1561. Scraper titles like "Forbidden", "Access Denied",
// "Service Unavailable", "Robot Check" were slipping through the
// bare-numeric-codes-only regex; the expanded matcher catches them
// without false-positiving on longer-form essays about those topics.
describe('error_page_title — v0.41.13 expanded matches', () => {
// Phrases that MUST match (exact-title scraper junk).
test.each([
'Forbidden',
'Access Denied',
'Service Unavailable',
'Robot Check',
'Verify You Are Human',
// Existing matches still work
'403',
'404',
'Error 500',
'Page Not Found',
// Anchored with optional trailing whitespace
'Forbidden ',
'access denied ',
// Case-insensitive
'forbidden',
'ROBOT CHECK',
])('matches scraper title %j', (title) => {
const r = assessContentSanity({ compiled_truth: '', timeline: '', title });
expect(r.junk_pattern_matches).toContain('error_page_title');
expect(r.shouldHardBlock).toBe(true);
});
// Over-match regression guard: these must NOT trip (the gate
// motivating the PR #1561 review-and-reshape).
test.each([
'How to Handle Access Denied Errors',
'Error Boundary in React',
'Service Unavailable Pattern',
'Forbidden Knowledge',
'Forbidden City', // legitimate place name
'Designing the Perfect Robot Check', // long-form essay
'Verify You Are Human (a poem)',
])('does NOT match legitimate prose title %j (over-match regression)', (title) => {
const r = assessContentSanity({ compiled_truth: '', timeline: '', title });
expect(r.junk_pattern_matches).not.toContain('error_page_title');
});
// Bare-`error` matcher was DELIBERATELY dropped from PR #1561's
// expansion. A page titled just "Error" (e.g. a programming
// taxonomy node) must NOT be hard-blocked.
test('bare title "Error" does NOT match (PR #1561 bare-`error` matcher dropped)', () => {
const r = assessContentSanity({ compiled_truth: '', timeline: '', title: 'Error' });
expect(r.junk_pattern_matches).not.toContain('error_page_title');
expect(r.shouldHardBlock).toBe(false);
});
});
describe('cloudflare_challenge_title — v0.41.13 distinct-name pattern', () => {
test.each([
'Just a moment...',
'Just a moment', // zero dots
'Just a moment.', // one dot
'Just a moment..', // two dots
'just a moment...', // case-insensitive
'JUST A MOMENT...',
])('matches Cloudflare title %j', (title) => {
const r = assessContentSanity({ compiled_truth: '', timeline: '', title });
expect(r.junk_pattern_matches).toContain('cloudflare_challenge_title');
expect(r.shouldHardBlock).toBe(true);
});
test('regex is strict-anchored — trailing whitespace does NOT match (no \\s*$ in the new pattern)', () => {
// Distinct from error_page_title which allows trailing whitespace
// (\s*$). The Cloudflare title is observed in the wild as exactly
// "Just a moment...", so we keep strict anchoring to avoid
// accidentally trapping titles with trailing content.
const r = assessContentSanity({ compiled_truth: '', timeline: '', title: 'Just a moment... ' });
expect(r.junk_pattern_matches).not.toContain('cloudflare_challenge_title');
});
test('does NOT match longer prose with "Just a moment..." prefix', () => {
const r = assessContentSanity({
compiled_truth: '',
timeline: '',
title: 'Just a moment, please — checking with the team',
});
expect(r.junk_pattern_matches).not.toContain('cloudflare_challenge_title');
});
test('records as cloudflare_challenge_title, NOT error_page_title (audit-name distinctness)', () => {
const r = assessContentSanity({
compiled_truth: '',
timeline: '',
title: 'Just a moment...',
});
expect(r.junk_pattern_matches).toContain('cloudflare_challenge_title');
expect(r.junk_pattern_matches).not.toContain('error_page_title');
});
test('body-scoped cloudflare_just_a_moment is independent (BOTH may fire on same content)', () => {
// Body needs both phrase + cdn-cgi/challenge-platform URL; title needs
// exactly "Just a moment[...]". A real Cloudflare interstitial would
// trip both (different scopes, different names — operator sees both
// in the audit log).
const r = assessContentSanity({
compiled_truth: 'Just a moment... please wait\ncdn-cgi/challenge-platform/h/blah',
timeline: '',
title: 'Just a moment...',
});
expect(r.junk_pattern_matches).toContain('cloudflare_challenge_title');
expect(r.junk_pattern_matches).toContain('cloudflare_just_a_moment');
});
});
});
// ─── REASON ORDERING + MESSAGES ────────────────────────────────
+124
View File
@@ -335,3 +335,127 @@ describe('judgeSignificance', () => {
expect(r.reasons[0]).toContain('unparseable');
});
});
// ─── v0.41.13: UTF-16 safety in judgeSignificance ─────────────────────
//
// Reproduces the 2026-05-24 production SYNTH_PHASE_FAIL: `🤖` (U+1F916,
// encoded as surrogate pair U+D83E U+DD16) at offset 3999 in a long
// telegram transcript made the 4000-char slice produce a lone high
// surrogate. Anthropic's JSON parser rejected the payload with "no low
// surrogate in string". Fix routes both head + tail slices through the
// canonical safeSplitIndex helper from text-safe.ts.
//
// Primary assertion: scan the captured prompt for unpaired surrogates.
// (NOT JSON.stringify, which doesn't throw on lone surrogates in V8/
// JSCore — codex C-11.)
describe('judgeSignificance — UTF-16 safety (v0.41.13)', () => {
const HIGH = '\uD83E'; // high surrogate of 🤖
const LOW = '\uDD16'; // low surrogate of 🤖
const ROBOT = HIGH + LOW; // U+1F916, 🤖, two UTF-16 code units
function isHighSurrogate(c: number): boolean { return c >= 0xD800 && c <= 0xDBFF; }
function isLowSurrogate(c: number): boolean { return c >= 0xDC00 && c <= 0xDFFF; }
/**
* Build content of exactly `length` chars with `🤖` placed so that
* the high surrogate sits at `emojiHighOffset`. The pair occupies
* positions [emojiHighOffset, emojiHighOffset+1]. Filler is plain
* ASCII so surrogate-scanning has no other true positives.
*/
function buildContentWithEmojiAt(length: number, emojiHighOffset: number): string {
if (emojiHighOffset < 0 || emojiHighOffset > length - 2) {
throw new Error(`emojiHighOffset ${emojiHighOffset} out of range for length ${length}`);
}
const head = 'a'.repeat(emojiHighOffset);
const tail = 'b'.repeat(length - emojiHighOffset - 2);
return head + ROBOT + tail;
}
/** Stub client that captures the user-message string for inspection. */
function makeCapturingClient(): { client: JudgeClient; captured: { userMessage: string | null } } {
const captured: { userMessage: string | null } = { userMessage: null };
const client: JudgeClient = {
create: async (p: any) => {
// judgeSignificance posts `Transcript ${basename}:\n\n${trimmed}` as
// the user message. Capture for post-call scanning.
const userMsg = p.messages[0]?.content;
captured.userMessage = typeof userMsg === 'string' ? userMsg : null;
return { content: [{ type: 'text', text: '{"worth_processing": false, "reasons": ["stub"]}' }] } as any;
},
};
return { client, captured };
}
/**
* Extract the trimmed payload from the captured prompt by stripping
* the prompt prefix and the `\n[...truncated...]\n` separator. We
* scan the WHOLE captured message for unpaired surrogates anyway
* (prompt prefix is pure ASCII), so the extraction is defense-in-
* depth, not the primary signal.
*/
function scanForUnpairedSurrogates(s: string): { index: number; kind: 'lone-high' | 'lone-low' } | null {
for (let i = 0; i < s.length; i++) {
const c = s.charCodeAt(i);
if (isHighSurrogate(c)) {
const next = i + 1 < s.length ? s.charCodeAt(i + 1) : -1;
if (!isLowSurrogate(next)) return { index: i, kind: 'lone-high' };
} else if (isLowSurrogate(c)) {
const prev = i > 0 ? s.charCodeAt(i - 1) : -1;
if (!isHighSurrogate(prev)) return { index: i, kind: 'lone-low' };
}
}
return null;
}
function makeLongTranscript(content: string): import('../src/core/cycle/transcript-discovery.ts').DiscoveredTranscript {
return {
filePath: '/tmp/long.txt',
contentHash: 'utf16-test',
content,
basename: 'long',
inferredDate: null,
};
}
// ─── Head-boundary cases (offset around 4000) ──────────────────────
test.each([3998, 3999, 4000, 4001])(
'emoji at head offset %i: captured prompt has zero unpaired surrogates',
async (offset) => {
const content = buildContentWithEmojiAt(8001, offset);
const { client, captured } = makeCapturingClient();
await judgeSignificance(client, makeLongTranscript(content));
expect(captured.userMessage).not.toBeNull();
const result = scanForUnpairedSurrogates(captured.userMessage!);
expect(result).toBeNull();
},
);
// ─── Tail-boundary cases (offset around length-4000 = 4001) ────────
test.each([3999, 4000, 4001, 4002])(
'emoji at tail offset %i: captured prompt has zero unpaired surrogates',
async (offset) => {
// 8001 - 4000 = 4001 is the tail boundary; we test around it.
const content = buildContentWithEmojiAt(8001, offset);
const { client, captured } = makeCapturingClient();
await judgeSignificance(client, makeLongTranscript(content));
expect(captured.userMessage).not.toBeNull();
const result = scanForUnpairedSurrogates(captured.userMessage!);
expect(result).toBeNull();
},
);
// ─── Sub-8000 short-content branch: no slicing, no risk ────────────
test('content <= 8000 chars: no slicing applied, emoji passes through unchanged', async () => {
const content = 'a'.repeat(100) + ROBOT + 'b'.repeat(100); // 202 chars total
const { client, captured } = makeCapturingClient();
await judgeSignificance(client, makeLongTranscript(content));
expect(captured.userMessage).not.toBeNull();
expect(scanForUnpairedSurrogates(captured.userMessage!)).toBeNull();
// Emoji's full pair must appear at least once.
expect(captured.userMessage!).toContain(ROBOT);
});
});
+47
View File
@@ -58,4 +58,51 @@ describe('dream CLI flag wiring', () => {
expect(dreamSrc).toContain('skips the Sonnet');
expect(dreamSrc.toLowerCase()).toContain('zero llm calls');
});
// v0.41.13: --source / --source-id flag wiring (supersedes PR #1559).
// Structural-only tests; behavioral tests live in test/dream.test.ts.
describe('--source / --source-id wiring (v0.41.13)', () => {
test('declares --source flag in argv parsing', () => {
expect(dreamSrc).toContain("'--source'");
});
test('declares --source-id alias in argv parsing', () => {
expect(dreamSrc).toContain("'--source-id'");
});
test('forwards resolved sourceId to runCycle', () => {
// The runCycle call must pass sourceId; gate name "sourceId"
// not "source" because CycleOpts.sourceId is the contract.
expect(dreamSrc).toMatch(/sourceId:\s*resolvedSourceId/);
});
test('imports resolveSourceId from canonical source-resolver helper', () => {
expect(dreamSrc).toContain("from '../core/source-resolver.ts'");
expect(dreamSrc).toContain('resolveSourceId');
});
test('declares isResolverUserError predicate for typed-error catch (T3 from eng review)', () => {
expect(dreamSrc).toContain('function isResolverUserError');
});
test('documents --source in --help output', () => {
expect(dreamSrc).toContain('--source <id>');
expect(dreamSrc).toContain('--source-id <id>');
});
test('preserves --help short-circuit ordering comment (IRON RULE)', () => {
// The comment lives in runDream BEFORE the engine-null gate.
// Future refactors that reorder these blocks will trip this guard.
expect(dreamSrc).toContain('IRON RULE: --help short-circuits BEFORE');
});
test('declares engine-null guard for --source', () => {
expect(dreamSrc).toContain('requires a connected brain');
});
test('declares archived-source guard', () => {
expect(dreamSrc).toMatch(/source.*is archived/);
expect(dreamSrc).toContain('gbrain sources restore');
});
});
});
+310
View File
@@ -241,3 +241,313 @@ describe('runDream — exit-code semantics', () => {
spy.mockRestore();
});
});
// ─── v0.41.13: --source / --source-id wiring (supersedes PR #1559) ────
//
// Covers:
// - argv repetition + conflict rules (parseArgs path)
// - engine-null guard (D1 from eng review)
// - assertSourceExists propagation (resolveSourceId path)
// - archived-source guard (D2)
// - --source-id alias equivalence (D3)
// - --help short-circuit ordering (C-8)
// - typed-error propagation (T3: TypeError must NOT be swallowed)
// - end-to-end dream→doctor (D5): writeback flips cycle_freshness
// - back-compat regression: bare `gbrain dream` writes no per-source stamp
describe('runDream — --source / --source-id (v0.41.13)', () => {
let repo: string;
let engine: InstanceType<typeof PGLiteEngine>;
async function seedSource(id: string, archived: boolean = false): Promise<void> {
await engine.executeRaw(
`INSERT INTO sources (id, name, local_path, config, archived, created_at)
VALUES ($1, $2, $3, '{}'::jsonb, $4, NOW())
ON CONFLICT (id) DO UPDATE SET local_path = EXCLUDED.local_path, archived = EXCLUDED.archived`,
[id, id, repo, archived],
);
}
async function readLastFullCycleAt(sourceId: string): Promise<string | null> {
const sources = await engine.listAllSources();
const s = sources.find(x => x.id === sourceId);
if (!s) return null;
const raw = (s.config as any)?.last_full_cycle_at;
return typeof raw === 'string' ? raw : null;
}
beforeEach(async () => {
repo = makeGitRepo();
engine = await makePGLite();
}, 60_000);
afterEach(async () => {
if (engine) await engine.disconnect();
rmSync(repo, { recursive: true, force: true });
}, 60_000);
// ─── parseArgs: --source missing / conflict / repetition ────────────
test('--source with no value exits 2 with usage hint', async () => {
const exitSpy = spyOn(process, 'exit').mockImplementation(() => { throw new Error('EXIT'); });
const errSpy = spyOn(console, 'error').mockImplementation(() => {});
try {
await runDream(engine, ['--source']);
} catch (e: any) {
expect(e.message).toBe('EXIT');
}
expect(exitSpy).toHaveBeenCalledWith(2);
expect(errSpy.mock.calls.flat().join(' ')).toMatch(/--source.*missing value/);
exitSpy.mockRestore();
errSpy.mockRestore();
});
test('--source-id with no value exits 2 with usage hint', async () => {
const exitSpy = spyOn(process, 'exit').mockImplementation(() => { throw new Error('EXIT'); });
const errSpy = spyOn(console, 'error').mockImplementation(() => {});
try {
await runDream(engine, ['--source-id']);
} catch (e: any) {
expect(e.message).toBe('EXIT');
}
expect(exitSpy).toHaveBeenCalledWith(2);
expect(errSpy.mock.calls.flat().join(' ')).toMatch(/--source-id.*missing value/);
exitSpy.mockRestore();
errSpy.mockRestore();
});
test('--source X --source Y (repeated, different values) exits 2', async () => {
const exitSpy = spyOn(process, 'exit').mockImplementation(() => { throw new Error('EXIT'); });
const errSpy = spyOn(console, 'error').mockImplementation(() => {});
try {
await runDream(engine, ['--source', 'foo', '--source', 'bar']);
} catch (e: any) {
expect(e.message).toBe('EXIT');
}
expect(exitSpy).toHaveBeenCalledWith(2);
expect(errSpy.mock.calls.flat().join(' ')).toMatch(/specify --source once/);
exitSpy.mockRestore();
errSpy.mockRestore();
});
test('--source X --source X (repeated, same value) is accepted', async () => {
await seedSource('alpha');
const report = await runDream(engine, ['--dir', repo, '--phase', 'lint', '--source', 'alpha', '--source', 'alpha', '--json']);
expect(report).toBeTruthy();
if (report) expect(['ok', 'clean']).toContain(report.status);
}, 60_000);
test('--source X --source-id Y (conflict) exits 2', async () => {
const exitSpy = spyOn(process, 'exit').mockImplementation(() => { throw new Error('EXIT'); });
const errSpy = spyOn(console, 'error').mockImplementation(() => {});
try {
await runDream(engine, ['--source', 'foo', '--source-id', 'bar']);
} catch (e: any) {
expect(e.message).toBe('EXIT');
}
expect(exitSpy).toHaveBeenCalledWith(2);
expect(errSpy.mock.calls.flat().join(' ')).toMatch(/use --source OR --source-id, not both/);
exitSpy.mockRestore();
errSpy.mockRestore();
});
// ─── Help short-circuit ordering (C-8 IRON RULE) ────────────────────
test('--help --source whatever prints help and exits 0 (no engine-null error)', async () => {
// engine null + --source set would normally exit 1, but --help short-circuits first.
const exitSpy = spyOn(process, 'exit').mockImplementation(() => { throw new Error('EXIT'); });
const logSpy = spyOn(console, 'log').mockImplementation(() => {});
try {
const result = await runDream(null, ['--help', '--source', 'anything']);
expect(result).toBeUndefined();
} catch (e: any) {
throw new Error('--help with --source should NOT exit; got: ' + e.message);
}
expect(exitSpy).not.toHaveBeenCalled();
expect(logSpy.mock.calls.flat().join(' ')).toMatch(/Usage: gbrain dream/);
exitSpy.mockRestore();
logSpy.mockRestore();
});
// ─── Engine-null guard (D1) ─────────────────────────────────────────
test('engine=null + --source set exits 1 with "requires a connected brain"', async () => {
const exitSpy = spyOn(process, 'exit').mockImplementation(() => { throw new Error('EXIT'); });
const errSpy = spyOn(console, 'error').mockImplementation(() => {});
try {
await runDream(null, ['--source', 'whatever']);
} catch (e: any) {
expect(e.message).toBe('EXIT');
}
expect(exitSpy).toHaveBeenCalledWith(1);
expect(errSpy.mock.calls.flat().join(' ')).toMatch(/requires a connected brain/);
exitSpy.mockRestore();
errSpy.mockRestore();
});
// ─── Unknown source (resolveSourceId throw) ─────────────────────────
test('--source <unknown> exits 1 with assertSourceExists hint', async () => {
const exitSpy = spyOn(process, 'exit').mockImplementation(() => { throw new Error('EXIT'); });
const errSpy = spyOn(console, 'error').mockImplementation(() => {});
try {
await runDream(engine, ['--source', 'no-such-source']);
} catch (e: any) {
expect(e.message).toBe('EXIT');
}
expect(exitSpy).toHaveBeenCalledWith(1);
const errOut = errSpy.mock.calls.flat().join(' ');
expect(errOut).toMatch(/Source "no-such-source" not found/);
expect(errOut).toMatch(/gbrain sources list/);
exitSpy.mockRestore();
errSpy.mockRestore();
});
// ─── Archived source guard (D2) ─────────────────────────────────────
test('--source <archived> exits 1 and leaves last_full_cycle_at untouched', async () => {
await seedSource('archived-thing', /* archived = */ true);
const before = await readLastFullCycleAt('archived-thing');
expect(before).toBeNull();
const exitSpy = spyOn(process, 'exit').mockImplementation(() => { throw new Error('EXIT'); });
const errSpy = spyOn(console, 'error').mockImplementation(() => {});
try {
await runDream(engine, ['--source', 'archived-thing']);
} catch (e: any) {
expect(e.message).toBe('EXIT');
}
expect(exitSpy).toHaveBeenCalledWith(1);
const errOut = errSpy.mock.calls.flat().join(' ');
expect(errOut).toMatch(/source archived-thing is archived/);
expect(errOut).toMatch(/gbrain sources restore archived-thing/);
const after = await readLastFullCycleAt('archived-thing');
expect(after).toBeNull(); // archived guard prevents writeback
exitSpy.mockRestore();
errSpy.mockRestore();
});
// ─── Happy path: --source writes last_full_cycle_at (the bug fix) ───
test('--source <existing> writes last_full_cycle_at on success (PR #1559 regression)', async () => {
await seedSource('media-corpus');
const before = await readLastFullCycleAt('media-corpus');
expect(before).toBeNull();
const t0 = Date.now();
const report = await runDream(engine, ['--dir', repo, '--source', 'media-corpus', '--phase', 'lint', '--json']);
expect(report).toBeTruthy();
if (report) expect(['ok', 'clean']).toContain(report.status);
const after = await readLastFullCycleAt('media-corpus');
expect(after).not.toBeNull();
const writtenMs = new Date(after!).getTime();
expect(writtenMs).toBeGreaterThanOrEqual(t0);
expect(writtenMs).toBeLessThanOrEqual(Date.now() + 1000);
}, 60_000);
// ─── Back-compat: bare `gbrain dream` does NOT write per-source stamp ─
test('gbrain dream (no --source) leaves all sources untouched (back-compat regression)', async () => {
await seedSource('alpha');
await seedSource('beta');
const report = await runDream(engine, ['--dir', repo, '--phase', 'lint', '--json']);
expect(report).toBeTruthy();
expect(await readLastFullCycleAt('alpha')).toBeNull();
expect(await readLastFullCycleAt('beta')).toBeNull();
}, 60_000);
// ─── --source-id alias equivalence (D3) ─────────────────────────────
test('--source-id <existing> is equivalent to --source (writes timestamp)', async () => {
await seedSource('beta');
const before = await readLastFullCycleAt('beta');
expect(before).toBeNull();
const report = await runDream(engine, ['--dir', repo, '--source-id', 'beta', '--phase', 'lint', '--json']);
expect(report).toBeTruthy();
if (report) expect(['ok', 'clean']).toContain(report.status);
const after = await readLastFullCycleAt('beta');
expect(after).not.toBeNull();
}, 60_000);
// ─── T3: TypeError MUST propagate (not swallowed by predicate-gated catch) ─
test('non-resolver-user errors propagate uncaught (T3)', async () => {
// Monkey-patch executeRaw to throw a synthetic TypeError on the
// assertSourceExists SELECT. The typed-error catch in runDream only
// matches resolver-user-error message shapes; a TypeError thrown
// from any source-resolution path must bubble up with its original
// stack trace (proving real programmer bugs are NOT hidden behind
// operator-error UX).
await seedSource('gamma');
const original = (engine as any).executeRaw.bind(engine);
let restored = false;
try {
// Throw a TypeError on the source-lookup SELECT that assertSourceExists runs.
// Other executeRaw calls (used by engine internals during cycle) keep
// working so the test exercises ONLY the resolution-path failure.
(engine as any).executeRaw = async (sql: string, params?: unknown[]) => {
if (typeof sql === 'string' && /FROM\s+sources\s+WHERE\s+id\s*=/i.test(sql)) {
throw new TypeError('synthetic-test-bug');
}
return original(sql, params);
};
await expect(
runDream(engine, ['--dir', repo, '--source', 'gamma', '--phase', 'lint', '--json'])
).rejects.toThrow(/synthetic-test-bug/);
} finally {
(engine as any).executeRaw = original;
restored = true;
}
expect(restored).toBe(true);
});
});
// ─── v0.41.13 D5: end-to-end dream → checkCycleFreshness parity ───────
//
// Closes the column-rename drift class: if a future PR renames
// last_full_cycle_at on one side but not the other, both isolated
// tests stay green but production breaks. This exercises the FULL
// chain through the exact seam both sides consume.
describe('runDream → checkCycleFreshness end-to-end (D5)', () => {
let repo: string;
let engine: InstanceType<typeof PGLiteEngine>;
beforeEach(async () => {
repo = makeGitRepo();
engine = await makePGLite();
}, 60_000);
afterEach(async () => {
if (engine) await engine.disconnect();
rmSync(repo, { recursive: true, force: true });
}, 60_000);
test('stale source becomes fresh after dream --source (column-name drift guard)', async () => {
// Seed source with last_full_cycle_at backdated 25h (above warn floor).
const stale = new Date(Date.now() - 25 * 60 * 60 * 1000).toISOString();
await engine.executeRaw(
`INSERT INTO sources (id, name, local_path, config, archived, created_at)
VALUES ('gamma', 'gamma', $1, jsonb_build_object('last_full_cycle_at', $2::text), false, NOW())`,
[repo, stale],
);
// Doctor sees stale (warn or fail; we only care that it's NOT ok)
const { checkCycleFreshness } = await import('../src/commands/doctor.ts');
const beforeCheck = await checkCycleFreshness(engine);
expect(beforeCheck.status).not.toBe('ok');
// Run dream against the stale source.
const report = await runDream(engine, ['--dir', repo, '--source', 'gamma', '--phase', 'lint', '--json']);
expect(report).toBeTruthy();
if (report) expect(['ok', 'clean']).toContain(report.status);
// Doctor now sees fresh.
const afterCheck = await checkCycleFreshness(engine);
expect(afterCheck.status).toBe('ok');
}, 60_000);
});
+48
View File
@@ -101,6 +101,54 @@ created: 2026-05-24
expect(page).toBeNull();
});
});
// ─── v0.41.13: end-to-end coverage for the expanded patterns ────────
// Exercises the assessor wiring (not just the regex) per D6.
test.each([
['Forbidden', 'error_page_title'],
['Access Denied', 'error_page_title'],
['Service Unavailable', 'error_page_title'],
['Robot Check', 'error_page_title'],
['Just a moment...', 'cloudflare_challenge_title'],
])('v0.41.13: title %j → ContentSanityBlockError (matches %s)', async (title, expectedPattern) => {
await withIsolatedHome(async () => {
const content = `---
title: '${title}'
type: note
created: 2026-05-24
---
scraper junk body`;
let caught: ContentSanityBlockError | undefined;
try {
await importFromContent(engine, 'test/v04113-' + title.toLowerCase().replace(/[^a-z]/g, '-'), content, { noEmbed: true });
} catch (e) {
if (e instanceof ContentSanityBlockError) caught = e;
else throw e;
}
expect(caught).toBeDefined();
expect(caught!.result.junk_pattern_matches).toContain(expectedPattern);
expect(caught!.message).toContain('PAGE_JUNK_PATTERN');
});
});
test('v0.41.13: over-match regression — "How to Handle Access Denied Errors" imports cleanly', async () => {
await withIsolatedHome(async () => {
const content = `---
title: 'How to Handle Access Denied Errors'
type: note
created: 2026-05-24
---
A legitimate essay about handling access-denied errors in your app.`;
// Should NOT throw.
const result = await importFromContent(engine, 'test/v04113-essay', content, { noEmbed: true });
expect(result.status).not.toBe('error');
const page = await engine.getPage('test/v04113-essay');
expect(page).not.toBeNull();
});
});
});
describe('importFromContent — soft-block (D9 transition + embed_skip)', () => {