diff --git a/CHANGELOG.md b/CHANGELOG.md index b80d6ead7..b3654bb4d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,193 @@ All notable changes to GBrain will be documented in this file. +## [0.41.26.0] - 2026-05-27 + +**`gbrain dream --source ` finally counts as a cycle.** + +If you have more than one brain source β€” a personal brain plus +media-corpus plus a team brain β€” `gbrain dream --source media-corpus` +used to run the cycle but never tell anyone it ran. The doctor would +say "media-corpus cycle is 60 hours stale" right after you re-ran +dream. You re-ran it. Still stale. Forever. Cron jobs scoped to one +source kept looping with no way to make the freshness gate go green. + +Fixed. Now `gbrain dream --source ` records the completion +timestamp on the source, so `gbrain doctor`'s `cycle_freshness` check +goes green the moment the cycle finishes. The flag also accepts +`--source-id` (matches the naming every other v0.37.7.0+ command uses +β€” `gbrain import --source-id`, `gbrain extract --source-id`, +`gbrain graph-query --source-id`). Both names work; pick whichever +you already remember. + +Two safety rails came along for the ride: + +- `gbrain dream --source ` errors clean ("Source not found. + Run `gbrain sources list`...") instead of silently passing + through and writing nothing. +- `gbrain dream --source ` refuses with a paste-ready + `gbrain sources restore ` hint instead of stamping a fresh + cycle timestamp on a hidden source (which would mask data + staleness if you later restored it). + +**Bonus housekeeping: ingest now blocks more scraper junk.** + +Cloudflare and WAF challenge pages with titles like `Just a +moment...`, `Forbidden`, `Access Denied`, `Service Unavailable`, and +`Robot Check` used to slip through ingest because the matcher only +caught bare numeric codes (`403`, `404`, `500`...). They now hit the +same gate that's always caught the numeric ones. Real essays titled +"How to handle Access Denied errors" or "Forbidden Knowledge" still +ingest fine β€” the gate only fires when the title is exclusively the +error phrase (with optional trailing whitespace), so longer-form +prose passes through. + +The Cloudflare challenge title gets its own audit name +(`cloudflare_challenge_title`) instead of being grouped under +`error_page_title`, so operators reading the `~/.gbrain/audit/ +content-sanity-YYYY-Www.jsonl` log or running `gbrain doctor` can +distinguish "this was a 403" from "this was a Cloudflare challenge". + +Cleanup for the 200+ pre-existing scraper pages already in your DB +ships in a follow-up PR (`gbrain pages audit-junk-titles`) after +this matcher has a week of production observation. The ingest gate +is reversible (won't accept the page); the cleanup is destructive +(soft-delete) and deserves its own observation window before it +gains authority to delete. + +**Third smaller fix: dream cycle no longer crashes on +emoji-at-offset-4000.** + +The `judgeSignificance` step of the dream cycle was using a raw 4000- +character slice that could split the UTF-16 surrogate pair encoding +a non-BMP emoji (`πŸ€–` and friends) right down the middle. Anthropic's +JSON parser would then reject the prompt with `"no low surrogate in +string"`, the whole synthesize phase would fail (`SYNTH_PHASE_FAIL`), +and your transcript wouldn't get processed. It happened on a real +telegram transcript on 2026-05-24. + +`judgeSignificance` now routes both head and tail slices through the +canonical `safeSplitIndex` helper from `src/core/text-safe.ts` (the +same helper `findBoundary` has used since v0.42.0.0). Same content +budget, same trimming, just safe at the surrogate boundary. No more +emoji-class crashes. + +This release supersedes community PRs #1559 and #1561 (both by +`@garrytan-agents`). PR #1559's `--max-pages` flag was dropped from +this release because the cycle plumbing doesn't exist to honor it +yet (shipping the flag would be a lying flag); filed as a v0.42+ +follow-up. + +### To take advantage of v0.41.26.0 + +`gbrain upgrade` should do this automatically. If you target a +specific source via cron or by hand: + +1. **Replace** any `gbrain dream` cron line that was meant to scope + to a source but didn't have the flag working. Add `--source `: + ```bash + # Old (silently brain-wide, never wrote per-source freshness): + 0 2 * * * gbrain dream --json >> ~/gbrain.log + # New (per-source freshness keeps doctor green): + 0 2 * * * gbrain dream --source media-corpus --json >> ~/gbrain.log + ``` +2. **Verify** the freshness gate flips. After a successful run: + ```bash + gbrain doctor --json | jq '.checks[] | select(.name=="cycle_freshness")' + # Should show status="ok" for the source you cycled. + ``` +3. **If any step fails or doctor doesn't go green,** file an issue: + https://github.com/garrytan/gbrain/issues with: + - output of `gbrain doctor` + - the exact `gbrain dream` command you ran + - the source's row from `sources` table (config column) + +### Itemized changes + +**dream --source / --source-id (supersedes PR #1559):** +- `DreamArgs.source` field added; `parseArgs` recognizes `--source + ` AND the alias `--source-id `. +- Argv validation: missing value β†’ exit 2; repeated with different + values β†’ exit 2; `--source X --source-id Y` (conflict) β†’ exit 2; + same value repeated is accepted. +- `--help` short-circuit ordering preserved (IRON RULE comment + test + guard); `gbrain dream --help --source whatever` always prints help + and exits 0. +- `runDream` engine-null guard: `--source` requires a connected brain + (matches the v0.37.7.0 #1167 pattern across import/extract/ + graph-query). +- `runDream` archived-source guard: refuses with `gbrain sources + restore ` hint instead of stamping `last_full_cycle_at` on a + hidden source. +- `runDream` typed-error try/catch via `isResolverUserError` + predicate: only matches the resolver's known user-facing throws; + TypeError / postgres errors propagate with stack trace so genuine + bugs aren't hidden as operator errors. +- Forwarded `sourceId` to `runCycle`; the existing v0.38 writeback in + `cycle.ts:1947-1967` now actually fires. +- `--help` text documents both flag names and the `cycle_freshness` + unlock. + +**Surrogate-safe trimming in synthesize.ts (PR #1559+#1561 wave commit +`78b93f3`, productionized correctly):** +- `judgeSignificance` slice routed through `safeSplitIndex` from + `src/core/text-safe.ts` (canonical helper). +- Did NOT introduce the duplicate `safeSliceEnd` helper from the + original PRs (it re-introduces the case-3 bug `text-safe.ts:18-21` + documents). +- Did NOT touch `findBoundary` β€” already routes through + `safeSplitIndex` in master. + +**Expanded `error_page_title` regex (supersedes PR #1561):** +- Caught: `forbidden`, `access denied`, `service unavailable`, + `robot check`, `verify you are human` (case-insensitive, + anchored). +- Dropped PR #1561's bare-`error` matcher β€” too aggressive on + legitimate taxonomy pages titled "Error". +- New `cloudflare_challenge_title` pattern (separate name from + `error_page_title`) for the title-scoped Cloudflare challenge β€” + preserves audit-log diagnosability (PR #1561 collapsed both into + one name). + +**Tests:** +- `test/dream-cli-flags.test.ts` β€” structural assertions for new + flags, help text, IRON-RULE comment guard. +- `test/dream.test.ts` β€” 12 PGLite integration cases covering + --source happy path (the bug fix regression), back-compat, + --source-id alias, repetition/conflict/missing-value, engine-null, + archived, --help short-circuit ordering, TypeError propagation + (the typed-error catch can't hide programmer bugs). +- `test/dream.test.ts` end-to-end dreamβ†’checkCycleFreshness case + (D5): closes column-rename drift bug class. +- `test/cycle-synthesize.test.ts` β€” UTF-16 safety describe block + with `test.each` over head + tail surrogate-boundary offsets; + primary assertion is an explicit unpaired-surrogate scan (NOT + `JSON.stringify`, which doesn't throw on lone surrogates in V8). +- `test/content-sanity.test.ts` β€” new pattern matches + + over-match regression guard ("How to Handle Access Denied + Errors" must pass) + audit-name distinctness. +- `test/import-file-content-sanity.test.ts` β€” end-to-end + `importFromContent` integration via `test.each` for each new + pattern family. + +**Out of scope, filed in TODOS.md:** +- `gbrain dream --max-pages ` plumbing (PR #1559 dropped β€” needs + cycle-phase iteration refactor). +- `--source` vs `--source-id` flag-name unification across the CLI. +- `gbrain pages audit-junk-titles` legacy cleanup command (deferred + for ~1 week of matcher production observation before destructive + cleanup ships). + +### For contributors + +- Codex outside-voice review surfaced 18 findings against the plan; + 11 absorbed inline (typed-error try/catch tightening, doctor check + separation, help-ordering documentation, repetition rules, + surrogate-safety contract, dropped weak `JSON.stringify` + assertion). Two substantive cross-model tensions resolved via + explicit user decision (T1 drop Fix 4, T3 typed-error catch). +- Plan + 11 AskUserQuestion decision points captured at + `~/.claude/plans/system-instruction-you-are-working-starry-papert.md`. ## [0.41.25.0] - 2026-05-27 diff --git a/TODOS.md b/TODOS.md index 554b20e13..63f8e790f 100644 --- a/TODOS.md +++ b/TODOS.md @@ -1,5 +1,96 @@ # TODOS +## v0.41.20.x dream-source-ingest-titles follow-ups (v0.42+) + +- **TODO-V13-A (P2): `gbrain dream --max-pages ` plumbing.** + PR #1559 included a `--max-pages` flag for cost-bounded cycles on + large brains. v0.41.20 dropped it because `CycleOpts` has no `maxPages` + field and no cycle phase consults page-count limits β€” shipping the flag + would have been a lying flag. + - **What:** extend `CycleOpts` with `maxPages: number | undefined` and + thread it through extract phases (extract.ts, extract-facts.ts, + recompute-emotional-weight.ts) so per-source cost-bounded cycles + become real. + - **Why:** straylight-brain-class corpora (100K+ pages) benefit from + capping each cycle's work. Today operators have to wait full + extract sweeps regardless of cost. + - **Pros:** closes the lying-flag class; real cost brake. + - **Cons:** real refactor β€” extract phases iterate all pages today, + not page-count-bounded. + - **Context:** PR #1559 commit 67f98ca had the flag; the v0.41.20 + plan dropped it under "Out of scope" with this TODO as the + forwarding pointer. + - **Depends on:** CycleOpts type extension + extract page-iteration + refactor + decision on per-phase vs per-cycle cap semantics. + +- **TODO-V13-B (P3): `--source` / `--source-id` flag-name unification.** + Current drift: `dream`, `recall`, `sync` accept `--source`; + `import`, `extract`, `graph-query`, `sources` accept `--source-id`. + v0.41.20 added `--source-id` as an alias for dream's `--source` so + both work, but the codebase still ships two surface names. + - **What:** pick one canonical flag name across all CLI commands; + deprecate the other with a stderr warning; update doctor.ts + hint to match. + - **Why:** ergonomic consistency. Users who learned `--source-id` + via import shouldn't trip on `--source` in dream. + - **Pros:** ends a real user-facing confusion. + - **Cons:** low-priority polish; both names work today via alias. + - **Context:** doctor.ts historically pinned `--source`; v0.37.7.0 + #1167 standardized `--source-id` across new commands. Recommend + picking `--source-id` for v0.37.7.0+ consistency and deprecating + `--source` over one minor. + - **Depends on:** nothing technical. + +- **TODO-V13-C (P2): `gbrain pages audit-junk-titles` legacy cleanup.** + v0.41.20 widened the `error_page_title` matcher to catch Cloudflare / + WAF challenge titles ("Forbidden", "Access Denied", "Service + Unavailable", "Robot Check", "Just a moment...") at ingest. But the + 200+ scraper pages already in production DBs (202+ from + straylight-brain) are NOT cleaned up by the matcher widening. + Dropped from v0.41.20 per codex outside-voice tension (T1) for + ship-and-validate-matchers-first discipline. + - **What:** new operator command for soft-deleting pre-existing + scraper-junk pages whose titles match the expanded + `BUILT_IN_JUNK_PATTERNS`. Full spec preserved: + - Signature: `gbrain pages audit-junk-titles [--source ] + [--dry-run|--apply] [--confirm-destructive] [--json]` + - Default `--dry-run`. Prints `{pattern_name: count, sample_slugs}`. + - `--apply` requires `--confirm-destructive` when match count + exceeds `DESTRUCTIVE_THRESHOLD` (reuse v0.26.5 constant). + - `--source ` scopes; without it, audits all non-archived + sources (filter via `listAllSources().filter(s => !s.archived)`). + - Soft-delete via existing `engine.softDeletePage(slug, sourceId)`. + - Audit JSONL via `logContentSanityEvent` with event kind + `junk_title_soft_deleted`. + - Idempotent. + - **Hybrid SQL+JS scanner**: pure + `buildJunkTitleSqlClause(patterns)` + + `scanForJunkTitles(rows, patterns)`. SQL pre-filter avoids + streaming all rows over the wire (perf rationale: even seq-scan + ILIKE beats JS regex per-row via the postgres driver). + - **`cleanup_safe: boolean` flag** per JunkPattern (codex C-13): + only patterns flagged `cleanup_safe: true` are eligible for + destructive cleanup. Stops future matcher widening from + automatically expanding destructive scope. Initial allowlist: + `cloudflare_attention_required`, `cloudflare_just_a_moment`, + `cloudflare_ray_id`, `access_denied`, `captcha_required`, + `error_page_title` (only the literal-numeric parts; the new + word-titles get `cleanup_safe: false` until the matcher proves + itself further), `cloudflare_challenge_title`. + - New doctor check `scraper_junk_pages_legacy` (separate from + `content_sanity_audit_recent` per codex C-5 β€” audit-log reader + vs live DB scan are different concerns). + - Tests: `test/pages-audit-junk-titles.test.ts` (hermetic PGLite), + `test/doctor.test.ts` extension. + - **Why:** ingest gate alone leaves 200+ existing junk pages + inflating page counts; this command closes the data-debt gap. + - **Pros:** finishes the cleanup story. + - **Cons:** destructive surface (soft-delete + audit JSONL). + - **Depends on:** ~1 week of production observation against + v0.41.20's new ingest matchers. If real-world reports surface + false-positive blocks, refine the matcher AND the `cleanup_safe` + allowlist before shipping the destructive command. + ## v0.41.22.1 brainstorm judge fix-wave follow-ups (v0.42+) Filed from the v0.41.22.1 plan-eng-review per cross-model-tension D13c. diff --git a/VERSION b/VERSION index 1a46e86c5..a54d656e5 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -0.41.25.0 +0.41.26.0 \ No newline at end of file diff --git a/package.json b/package.json index a47a44b51..2c4aa7a20 100644 --- a/package.json +++ b/package.json @@ -140,5 +140,5 @@ "bun": ">=1.3.10" }, "license": "MIT", - "version": "0.41.25.0" + "version": "0.41.26.0" } diff --git a/src/commands/dream.ts b/src/commands/dream.ts index a82e67ca4..9746b4e6c 100644 --- a/src/commands/dream.ts +++ b/src/commands/dream.ts @@ -30,6 +30,8 @@ import { type CyclePhase, type CycleReport, } from '../core/cycle.ts'; +import { resolveSourceId } from '../core/source-resolver.ts'; +import { fetchSource } from '../core/sources-load.ts'; import { existsSync } from 'fs'; import { resolve } from 'node:path'; @@ -54,10 +56,40 @@ interface DreamArgs { * Never auto-applied for --input (codex finding #3). */ bypassDreamGuard: boolean; + /** + * v0.41.13: per-source cycle scoping. Threaded into runCycle as + * `sourceId` so `cycle.ts:1947-1967` writes `last_full_cycle_at` + * to `sources.config` on success β€” without it, `gbrain doctor`'s + * `cycle_freshness` check stays stale forever. Accepts `--source + * ` and the alias `--source-id ` (the v0.37.7.0 #1167 + * canonical name across import/extract/graph-query); both work + * until a follow-up CLI cleanup picks one. Supersedes PR #1559. + */ + source: string | null; } const ISO_DATE_RE = /^\d{4}-\d{2}-\d{2}$/; +/** + * Collect every occurrence of `-- ` in argv. Used to + * detect repeated flags with different values (e.g. + * `--source X --source Y`) and to surface a clean usage error + * instead of silently last-wins. Repeated identical values are + * collapsed to one (no-op). Missing values (flag at end of argv) + * return null to let the caller raise an explicit usage error + * rather than fall through with `undefined`. + */ +function collectFlagValues(args: string[], flag: string): string[] | null { + const values: string[] = []; + for (let i = 0; i < args.length; i++) { + if (args[i] !== flag) continue; + const v = args[i + 1]; + if (v === undefined) return null; // flag at end of argv + values.push(v); + } + return values; +} + function parseArgs(args: string[]): DreamArgs { const phaseIdx = args.indexOf('--phase'); const rawPhase = phaseIdx !== -1 ? args[phaseIdx + 1] : null; @@ -110,6 +142,43 @@ function parseArgs(args: string[]): DreamArgs { // --input implies --phase synthesize. if (inputFile && !phase) phase = 'synthesize'; + // v0.41.13: --source (and the --source-id alias) drives per-source + // cycle scoping. Resolution rules: + // - missing value (flag at end of argv) β†’ exit 2 with usage + // - repeated with different values (e.g. --source X --source Y) β†’ exit 2 + // - --source X --source-id Y (conflicting flag aliases) β†’ exit 2 + // - --source X --source X (or --source-id repeated with same value) β†’ accepted + // - --help short-circuits BEFORE this block fires (see runDream). + // Closes the PR #1559 silent-no-op class through a clean argv contract. + const sourceValues = collectFlagValues(args, '--source'); + const sourceIdValues = collectFlagValues(args, '--source-id'); + if (sourceValues === null) { + console.error('--source : missing value. Usage: gbrain dream --source '); + process.exit(2); + } + if (sourceIdValues === null) { + console.error('--source-id : missing value. Usage: gbrain dream --source-id '); + process.exit(2); + } + const uniqSource = Array.from(new Set(sourceValues)); + const uniqSourceId = Array.from(new Set(sourceIdValues)); + if (uniqSource.length > 1) { + console.error(`specify --source once; got [${uniqSource.map(v => `"${v}"`).join(', ')}]`); + process.exit(2); + } + if (uniqSourceId.length > 1) { + console.error(`specify --source-id once; got [${uniqSourceId.map(v => `"${v}"`).join(', ')}]`); + process.exit(2); + } + if (uniqSource.length === 1 && uniqSourceId.length === 1 && uniqSource[0] !== uniqSourceId[0]) { + console.error( + `use --source OR --source-id, not both (different values): ` + + `--source="${uniqSource[0]}" vs --source-id="${uniqSourceId[0]}"`, + ); + process.exit(2); + } + const source = uniqSource[0] ?? uniqSourceId[0] ?? null; + return { json: args.includes('--json'), dryRun: args.includes('--dry-run'), @@ -122,6 +191,7 @@ function parseArgs(args: string[]): DreamArgs { from, to, bypassDreamGuard: args.includes('--unsafe-bypass-dream-guard'), + source, }; } @@ -183,6 +253,14 @@ Options: --pull git pull the brain repo before syncing (default: no pull) --dir Brain directory (default: configured brain) + --source Scope the cycle to one source so doctor's + cycle_freshness check sees a fresh stamp on + completion. Without this, gbrain dream's + timestamp never lands and federated brains + see "stale cycle" forever. + --source-id Alias for --source. Matches the v0.37.7.0+ + naming used by import/extract/graph-query. + --input Synthesize a specific transcript file (implies --phase synthesize). Bypasses corpus-dir scan. --date YYYY-MM-DD Synthesize transcripts dated for one specific day. @@ -267,14 +345,81 @@ function printHuman(report: CycleReport) { // ─── CLI entry ───────────────────────────────────────────────────── +/** + * Predicate: is this error one of the resolver's user-facing throws + * we want to surface as a clean stderr line + exit 1? + * + * Matches the message prefixes thrown from + * `src/core/source-resolver.ts:resolveSourceId` and + * `assertSourceExists`. Anything else (TypeError / ReferenceError / + * postgres connection failures / unexpected bugs) is intentionally + * NOT caught β€” those propagate to Bun's default unhandled handler + * with a stack trace so genuine programmer bugs aren't hidden as + * if they were operator errors. (Plan D-T3, codex C-7.) + */ +function isResolverUserError(e: unknown): boolean { + if (!(e instanceof Error)) return false; + const m = e.message; + return (m.startsWith('Source "') && m.includes(' not found.')) + || m.startsWith('Invalid --source value') + || m.startsWith('Invalid GBRAIN_SOURCE value'); +} + export async function runDream(engine: BrainEngine | null, args: string[]): Promise { const opts = parseArgs(args); + // ─── IRON RULE: --help short-circuits BEFORE any engine-bearing work ─ + // Tests pin this ordering so `gbrain dream --help --source whatever` + // ALWAYS prints help and exits 0, never reaching the engine-null gate + // below. If you reorder this, dream-cli-flags.test.ts will fail. if (opts.help) { printHelp(); return; } + // v0.41.13: --source resolution. Three guards in order: + // 1. engine null β†’ exit 1 (the writeback in cycle.ts requires a + // DB connection; without engine we'd silently fail the same way + // PR #1559 was created to fix) + // 2. resolveSourceId throws on unknown id β†’ typed-error catch + // surfaces clean message; non-resolver throws propagate + // 3. archived source β†’ exit 1 with restore hint (writing + // last_full_cycle_at to an archived source would mask data + // staleness when the source is later restored) + let resolvedSourceId: string | undefined; + if (opts.source !== null) { + if (engine === null) { + console.error( + 'gbrain dream --source requires a connected brain ' + + '(no engine available); omit --source or run `gbrain init` first', + ); + process.exit(1); + } + try { + resolvedSourceId = await resolveSourceId(engine, opts.source); + } catch (e) { + if (isResolverUserError(e)) { + console.error((e as Error).message); + process.exit(1); + } + throw e; // genuine bugs propagate with stack trace + } + // Archived-source guard via fetchSource from sources-load.ts + // (single-row SELECT that projects `archived` and falls back to + // pre-v0.26.5 schemas via isUndefinedColumnError catch β€” same + // legacy-safety net the rest of the codebase uses). engine's + // built-in listAllSources defaults to includeArchived=false AND + // doesn't project the archived column, so it cannot be used here. + const src = await fetchSource(engine, resolvedSourceId); + if (src?.archived === true) { + console.error( + `source ${resolvedSourceId} is archived; restore with ` + + `\`gbrain sources restore ${resolvedSourceId}\` before cycling`, + ); + process.exit(1); + } + } + const brainDir = await resolveBrainDir(engine, opts.dir); const phases: CyclePhase[] | undefined = opts.phase ? [opts.phase] : undefined; @@ -283,6 +428,7 @@ export async function runDream(engine: BrainEngine | null, args: string[]): Prom dryRun: opts.dryRun, pull: opts.pull, phases, + sourceId: resolvedSourceId, // undefined when --source not set β†’ legacy back-compat synthInputFile: opts.inputFile ?? undefined, synthDate: opts.date ?? undefined, synthFrom: opts.from ?? undefined, diff --git a/src/core/content-sanity.ts b/src/core/content-sanity.ts index 3005e45e4..ea36d77ec 100644 --- a/src/core/content-sanity.ts +++ b/src/core/content-sanity.ts @@ -163,10 +163,34 @@ export const BUILT_IN_JUNK_PATTERNS: ReadonlyArray = Object.freeze( applies_to: 'both', }, // Bare error-page titles. Anchored so the title is exclusively the - // error code β€” a thoughtful page ABOUT 404 errors won't trip. + // error phrase β€” a thoughtful page ABOUT 404 errors or one titled + // "How to Handle Access Denied Errors" won't trip. + // + // v0.41.13 (supersedes PR #1561): expanded from bare numeric codes + // + "page not found" to also catch Cloudflare/WAF challenge + // titles ("Forbidden", "Access Denied", "Service Unavailable", + // "Robot Check", "Verify You Are Human"). Deliberately drops PR + // #1561's bare-`error` matcher (would false-positive on + // legitimate taxonomy pages titled "Error"). 232+ scraper pages + // motivating this change (202+ from straylight-brain). { name: 'error_page_title', - pattern: /^(403|404|500|502|503|error \d{3}|page not found)\s*$/i, + pattern: /^(403|404|500|502|503|error \d{3}|page not found|forbidden|access denied|service unavailable|robot check|verify you are human)\s*$/i, + applies_to: 'title', + }, + // Cloudflare challenge title (companion to the body-scoped + // `cloudflare_just_a_moment` pattern above, which requires both + // phrase + URL). The title alone is a sufficient signal because + // legitimate pages don't title themselves "Just a moment...". + // + // v0.41.13: distinct name from `error_page_title` so audit JSONL + // (`~/.gbrain/audit/content-sanity-YYYY-Www.jsonl`) and doctor's + // `content_sanity_audit_recent` aggregation stay diagnosable. + // PR #1561 reused the `error_page_title` name and collapsed audit + // signal; we don't. + { + name: 'cloudflare_challenge_title', + pattern: /^just a moment\.{0,3}$/i, applies_to: 'title', }, ]); diff --git a/src/core/cycle/synthesize.ts b/src/core/cycle/synthesize.ts index 78c05ff0a..773f9d021 100644 --- a/src/core/cycle/synthesize.ts +++ b/src/core/cycle/synthesize.ts @@ -832,9 +832,26 @@ export async function judgeSignificance( // Truncate the transcript at 8K chars for cost control. Haiku's verdict // doesn't need the full body; the opening + closing sections are usually // representative of significance. - const trimmed = t.content.length > 8000 - ? t.content.slice(0, 4000) + '\n[...truncated...]\n' + t.content.slice(-4000) - : t.content; + // + // v0.41.13 surrogate-safety (supersedes PRs #1559+#1561's safeSliceEnd + // helper; see text-safe.ts:18-21 module docstring for why that helper + // re-introduces the case-3 bug the canonical safeSplitIndex was written + // to fix). Routes head + tail slicing through safeSplitIndex so an emoji + // at offset 4000 (or length-4000) never produces a lone surrogate that + // Anthropic's JSON parser rejects ("no low surrogate in string", caught + // 2026-05-24 on telegram). + // + // Contract: this branch only runs when content.length > 8000, so + // length - 4000 > 4000 > 0 β€” safeSplitIndex never sees an out-of-range + // maxChars here. (Codex C-10 documented contract.) + let trimmed: string; + if (t.content.length > 8000) { + const headEnd = safeSplitIndex(t.content, 4000); + const tailStart = safeSplitIndex(t.content, t.content.length - 4000); + trimmed = t.content.slice(0, headEnd) + '\n[...truncated...]\n' + t.content.slice(tailStart); + } else { + trimmed = t.content; + } const sys = `You judge whether a conversation transcript is worth synthesizing into a personal knowledge brain. diff --git a/test/content-sanity.test.ts b/test/content-sanity.test.ts index 3e93e1623..66842fac4 100644 --- a/test/content-sanity.test.ts +++ b/test/content-sanity.test.ts @@ -106,8 +106,8 @@ describe('assessContentSanity β€” size boundaries', () => { // ─── 6 BUILT-IN PATTERNS ────────────────────────────────────── describe('assessContentSanity β€” built-in junk patterns', () => { - test('built-in pattern count is locked at 6 (D3 dropped empty_body_with_source_url)', () => { - expect(BUILT_IN_JUNK_PATTERNS.length).toBe(6); + test('built-in pattern count is locked at 7 (v0.41.13 added cloudflare_challenge_title)', () => { + expect(BUILT_IN_JUNK_PATTERNS.length).toBe(7); const names = BUILT_IN_JUNK_PATTERNS.map((p) => p.name); expect(names).toContain('cloudflare_attention_required'); expect(names).toContain('cloudflare_just_a_moment'); @@ -115,8 +115,10 @@ describe('assessContentSanity β€” built-in junk patterns', () => { expect(names).toContain('access_denied'); expect(names).toContain('captcha_required'); expect(names).toContain('error_page_title'); + // v0.41.13: distinct name from error_page_title (audit-name distinctness). + expect(names).toContain('cloudflare_challenge_title'); // D3 regression: this rule was dropped. If it ever returns, the test - // count above bumps to 7 deliberately. + // count above bumps deliberately. expect(names).not.toContain('empty_body_with_source_url'); }); @@ -218,6 +220,120 @@ describe('assessContentSanity β€” built-in junk patterns', () => { }); expect(r.junk_pattern_matches).toContain('cloudflare_attention_required'); }); + + // ─── v0.41.13: expanded error_page_title + cloudflare_challenge_title ─ + // + // Supersedes PR #1561. Scraper titles like "Forbidden", "Access Denied", + // "Service Unavailable", "Robot Check" were slipping through the + // bare-numeric-codes-only regex; the expanded matcher catches them + // without false-positiving on longer-form essays about those topics. + + describe('error_page_title β€” v0.41.13 expanded matches', () => { + // Phrases that MUST match (exact-title scraper junk). + test.each([ + 'Forbidden', + 'Access Denied', + 'Service Unavailable', + 'Robot Check', + 'Verify You Are Human', + // Existing matches still work + '403', + '404', + 'Error 500', + 'Page Not Found', + // Anchored with optional trailing whitespace + 'Forbidden ', + 'access denied ', + // Case-insensitive + 'forbidden', + 'ROBOT CHECK', + ])('matches scraper title %j', (title) => { + const r = assessContentSanity({ compiled_truth: '', timeline: '', title }); + expect(r.junk_pattern_matches).toContain('error_page_title'); + expect(r.shouldHardBlock).toBe(true); + }); + + // Over-match regression guard: these must NOT trip (the gate + // motivating the PR #1561 review-and-reshape). + test.each([ + 'How to Handle Access Denied Errors', + 'Error Boundary in React', + 'Service Unavailable Pattern', + 'Forbidden Knowledge', + 'Forbidden City', // legitimate place name + 'Designing the Perfect Robot Check', // long-form essay + 'Verify You Are Human (a poem)', + ])('does NOT match legitimate prose title %j (over-match regression)', (title) => { + const r = assessContentSanity({ compiled_truth: '', timeline: '', title }); + expect(r.junk_pattern_matches).not.toContain('error_page_title'); + }); + + // Bare-`error` matcher was DELIBERATELY dropped from PR #1561's + // expansion. A page titled just "Error" (e.g. a programming + // taxonomy node) must NOT be hard-blocked. + test('bare title "Error" does NOT match (PR #1561 bare-`error` matcher dropped)', () => { + const r = assessContentSanity({ compiled_truth: '', timeline: '', title: 'Error' }); + expect(r.junk_pattern_matches).not.toContain('error_page_title'); + expect(r.shouldHardBlock).toBe(false); + }); + }); + + describe('cloudflare_challenge_title β€” v0.41.13 distinct-name pattern', () => { + test.each([ + 'Just a moment...', + 'Just a moment', // zero dots + 'Just a moment.', // one dot + 'Just a moment..', // two dots + 'just a moment...', // case-insensitive + 'JUST A MOMENT...', + ])('matches Cloudflare title %j', (title) => { + const r = assessContentSanity({ compiled_truth: '', timeline: '', title }); + expect(r.junk_pattern_matches).toContain('cloudflare_challenge_title'); + expect(r.shouldHardBlock).toBe(true); + }); + + test('regex is strict-anchored β€” trailing whitespace does NOT match (no \\s*$ in the new pattern)', () => { + // Distinct from error_page_title which allows trailing whitespace + // (\s*$). The Cloudflare title is observed in the wild as exactly + // "Just a moment...", so we keep strict anchoring to avoid + // accidentally trapping titles with trailing content. + const r = assessContentSanity({ compiled_truth: '', timeline: '', title: 'Just a moment... ' }); + expect(r.junk_pattern_matches).not.toContain('cloudflare_challenge_title'); + }); + + test('does NOT match longer prose with "Just a moment..." prefix', () => { + const r = assessContentSanity({ + compiled_truth: '', + timeline: '', + title: 'Just a moment, please β€” checking with the team', + }); + expect(r.junk_pattern_matches).not.toContain('cloudflare_challenge_title'); + }); + + test('records as cloudflare_challenge_title, NOT error_page_title (audit-name distinctness)', () => { + const r = assessContentSanity({ + compiled_truth: '', + timeline: '', + title: 'Just a moment...', + }); + expect(r.junk_pattern_matches).toContain('cloudflare_challenge_title'); + expect(r.junk_pattern_matches).not.toContain('error_page_title'); + }); + + test('body-scoped cloudflare_just_a_moment is independent (BOTH may fire on same content)', () => { + // Body needs both phrase + cdn-cgi/challenge-platform URL; title needs + // exactly "Just a moment[...]". A real Cloudflare interstitial would + // trip both (different scopes, different names β€” operator sees both + // in the audit log). + const r = assessContentSanity({ + compiled_truth: 'Just a moment... please wait\ncdn-cgi/challenge-platform/h/blah', + timeline: '', + title: 'Just a moment...', + }); + expect(r.junk_pattern_matches).toContain('cloudflare_challenge_title'); + expect(r.junk_pattern_matches).toContain('cloudflare_just_a_moment'); + }); + }); }); // ─── REASON ORDERING + MESSAGES ──────────────────────────────── diff --git a/test/cycle-synthesize.test.ts b/test/cycle-synthesize.test.ts index 87819462c..2e60bbf2f 100644 --- a/test/cycle-synthesize.test.ts +++ b/test/cycle-synthesize.test.ts @@ -335,3 +335,127 @@ describe('judgeSignificance', () => { expect(r.reasons[0]).toContain('unparseable'); }); }); + +// ─── v0.41.13: UTF-16 safety in judgeSignificance ───────────────────── +// +// Reproduces the 2026-05-24 production SYNTH_PHASE_FAIL: `πŸ€–` (U+1F916, +// encoded as surrogate pair U+D83E U+DD16) at offset 3999 in a long +// telegram transcript made the 4000-char slice produce a lone high +// surrogate. Anthropic's JSON parser rejected the payload with "no low +// surrogate in string". Fix routes both head + tail slices through the +// canonical safeSplitIndex helper from text-safe.ts. +// +// Primary assertion: scan the captured prompt for unpaired surrogates. +// (NOT JSON.stringify, which doesn't throw on lone surrogates in V8/ +// JSCore β€” codex C-11.) + +describe('judgeSignificance β€” UTF-16 safety (v0.41.13)', () => { + const HIGH = '\uD83E'; // high surrogate of πŸ€– + const LOW = '\uDD16'; // low surrogate of πŸ€– + const ROBOT = HIGH + LOW; // U+1F916, πŸ€–, two UTF-16 code units + + function isHighSurrogate(c: number): boolean { return c >= 0xD800 && c <= 0xDBFF; } + function isLowSurrogate(c: number): boolean { return c >= 0xDC00 && c <= 0xDFFF; } + + /** + * Build content of exactly `length` chars with `πŸ€–` placed so that + * the high surrogate sits at `emojiHighOffset`. The pair occupies + * positions [emojiHighOffset, emojiHighOffset+1]. Filler is plain + * ASCII so surrogate-scanning has no other true positives. + */ + function buildContentWithEmojiAt(length: number, emojiHighOffset: number): string { + if (emojiHighOffset < 0 || emojiHighOffset > length - 2) { + throw new Error(`emojiHighOffset ${emojiHighOffset} out of range for length ${length}`); + } + const head = 'a'.repeat(emojiHighOffset); + const tail = 'b'.repeat(length - emojiHighOffset - 2); + return head + ROBOT + tail; + } + + /** Stub client that captures the user-message string for inspection. */ + function makeCapturingClient(): { client: JudgeClient; captured: { userMessage: string | null } } { + const captured: { userMessage: string | null } = { userMessage: null }; + const client: JudgeClient = { + create: async (p: any) => { + // judgeSignificance posts `Transcript ${basename}:\n\n${trimmed}` as + // the user message. Capture for post-call scanning. + const userMsg = p.messages[0]?.content; + captured.userMessage = typeof userMsg === 'string' ? userMsg : null; + return { content: [{ type: 'text', text: '{"worth_processing": false, "reasons": ["stub"]}' }] } as any; + }, + }; + return { client, captured }; + } + + /** + * Extract the trimmed payload from the captured prompt by stripping + * the prompt prefix and the `\n[...truncated...]\n` separator. We + * scan the WHOLE captured message for unpaired surrogates anyway + * (prompt prefix is pure ASCII), so the extraction is defense-in- + * depth, not the primary signal. + */ + function scanForUnpairedSurrogates(s: string): { index: number; kind: 'lone-high' | 'lone-low' } | null { + for (let i = 0; i < s.length; i++) { + const c = s.charCodeAt(i); + if (isHighSurrogate(c)) { + const next = i + 1 < s.length ? s.charCodeAt(i + 1) : -1; + if (!isLowSurrogate(next)) return { index: i, kind: 'lone-high' }; + } else if (isLowSurrogate(c)) { + const prev = i > 0 ? s.charCodeAt(i - 1) : -1; + if (!isHighSurrogate(prev)) return { index: i, kind: 'lone-low' }; + } + } + return null; + } + + function makeLongTranscript(content: string): import('../src/core/cycle/transcript-discovery.ts').DiscoveredTranscript { + return { + filePath: '/tmp/long.txt', + contentHash: 'utf16-test', + content, + basename: 'long', + inferredDate: null, + }; + } + + // ─── Head-boundary cases (offset around 4000) ────────────────────── + + test.each([3998, 3999, 4000, 4001])( + 'emoji at head offset %i: captured prompt has zero unpaired surrogates', + async (offset) => { + const content = buildContentWithEmojiAt(8001, offset); + const { client, captured } = makeCapturingClient(); + await judgeSignificance(client, makeLongTranscript(content)); + expect(captured.userMessage).not.toBeNull(); + const result = scanForUnpairedSurrogates(captured.userMessage!); + expect(result).toBeNull(); + }, + ); + + // ─── Tail-boundary cases (offset around length-4000 = 4001) ──────── + + test.each([3999, 4000, 4001, 4002])( + 'emoji at tail offset %i: captured prompt has zero unpaired surrogates', + async (offset) => { + // 8001 - 4000 = 4001 is the tail boundary; we test around it. + const content = buildContentWithEmojiAt(8001, offset); + const { client, captured } = makeCapturingClient(); + await judgeSignificance(client, makeLongTranscript(content)); + expect(captured.userMessage).not.toBeNull(); + const result = scanForUnpairedSurrogates(captured.userMessage!); + expect(result).toBeNull(); + }, + ); + + // ─── Sub-8000 short-content branch: no slicing, no risk ──────────── + + test('content <= 8000 chars: no slicing applied, emoji passes through unchanged', async () => { + const content = 'a'.repeat(100) + ROBOT + 'b'.repeat(100); // 202 chars total + const { client, captured } = makeCapturingClient(); + await judgeSignificance(client, makeLongTranscript(content)); + expect(captured.userMessage).not.toBeNull(); + expect(scanForUnpairedSurrogates(captured.userMessage!)).toBeNull(); + // Emoji's full pair must appear at least once. + expect(captured.userMessage!).toContain(ROBOT); + }); +}); diff --git a/test/dream-cli-flags.test.ts b/test/dream-cli-flags.test.ts index 5d8b82714..26b3ac3cf 100644 --- a/test/dream-cli-flags.test.ts +++ b/test/dream-cli-flags.test.ts @@ -58,4 +58,51 @@ describe('dream CLI flag wiring', () => { expect(dreamSrc).toContain('skips the Sonnet'); expect(dreamSrc.toLowerCase()).toContain('zero llm calls'); }); + + // v0.41.13: --source / --source-id flag wiring (supersedes PR #1559). + // Structural-only tests; behavioral tests live in test/dream.test.ts. + describe('--source / --source-id wiring (v0.41.13)', () => { + test('declares --source flag in argv parsing', () => { + expect(dreamSrc).toContain("'--source'"); + }); + + test('declares --source-id alias in argv parsing', () => { + expect(dreamSrc).toContain("'--source-id'"); + }); + + test('forwards resolved sourceId to runCycle', () => { + // The runCycle call must pass sourceId; gate name "sourceId" + // not "source" because CycleOpts.sourceId is the contract. + expect(dreamSrc).toMatch(/sourceId:\s*resolvedSourceId/); + }); + + test('imports resolveSourceId from canonical source-resolver helper', () => { + expect(dreamSrc).toContain("from '../core/source-resolver.ts'"); + expect(dreamSrc).toContain('resolveSourceId'); + }); + + test('declares isResolverUserError predicate for typed-error catch (T3 from eng review)', () => { + expect(dreamSrc).toContain('function isResolverUserError'); + }); + + test('documents --source in --help output', () => { + expect(dreamSrc).toContain('--source '); + expect(dreamSrc).toContain('--source-id '); + }); + + test('preserves --help short-circuit ordering comment (IRON RULE)', () => { + // The comment lives in runDream BEFORE the engine-null gate. + // Future refactors that reorder these blocks will trip this guard. + expect(dreamSrc).toContain('IRON RULE: --help short-circuits BEFORE'); + }); + + test('declares engine-null guard for --source', () => { + expect(dreamSrc).toContain('requires a connected brain'); + }); + + test('declares archived-source guard', () => { + expect(dreamSrc).toMatch(/source.*is archived/); + expect(dreamSrc).toContain('gbrain sources restore'); + }); + }); }); diff --git a/test/dream.test.ts b/test/dream.test.ts index a773e51a8..0d4510218 100644 --- a/test/dream.test.ts +++ b/test/dream.test.ts @@ -241,3 +241,313 @@ describe('runDream β€” exit-code semantics', () => { spy.mockRestore(); }); }); + +// ─── v0.41.13: --source / --source-id wiring (supersedes PR #1559) ──── +// +// Covers: +// - argv repetition + conflict rules (parseArgs path) +// - engine-null guard (D1 from eng review) +// - assertSourceExists propagation (resolveSourceId path) +// - archived-source guard (D2) +// - --source-id alias equivalence (D3) +// - --help short-circuit ordering (C-8) +// - typed-error propagation (T3: TypeError must NOT be swallowed) +// - end-to-end dreamβ†’doctor (D5): writeback flips cycle_freshness +// - back-compat regression: bare `gbrain dream` writes no per-source stamp + +describe('runDream β€” --source / --source-id (v0.41.13)', () => { + let repo: string; + let engine: InstanceType; + + async function seedSource(id: string, archived: boolean = false): Promise { + await engine.executeRaw( + `INSERT INTO sources (id, name, local_path, config, archived, created_at) + VALUES ($1, $2, $3, '{}'::jsonb, $4, NOW()) + ON CONFLICT (id) DO UPDATE SET local_path = EXCLUDED.local_path, archived = EXCLUDED.archived`, + [id, id, repo, archived], + ); + } + + async function readLastFullCycleAt(sourceId: string): Promise { + const sources = await engine.listAllSources(); + const s = sources.find(x => x.id === sourceId); + if (!s) return null; + const raw = (s.config as any)?.last_full_cycle_at; + return typeof raw === 'string' ? raw : null; + } + + beforeEach(async () => { + repo = makeGitRepo(); + engine = await makePGLite(); + }, 60_000); + + afterEach(async () => { + if (engine) await engine.disconnect(); + rmSync(repo, { recursive: true, force: true }); + }, 60_000); + + // ─── parseArgs: --source missing / conflict / repetition ──────────── + + test('--source with no value exits 2 with usage hint', async () => { + const exitSpy = spyOn(process, 'exit').mockImplementation(() => { throw new Error('EXIT'); }); + const errSpy = spyOn(console, 'error').mockImplementation(() => {}); + try { + await runDream(engine, ['--source']); + } catch (e: any) { + expect(e.message).toBe('EXIT'); + } + expect(exitSpy).toHaveBeenCalledWith(2); + expect(errSpy.mock.calls.flat().join(' ')).toMatch(/--source.*missing value/); + exitSpy.mockRestore(); + errSpy.mockRestore(); + }); + + test('--source-id with no value exits 2 with usage hint', async () => { + const exitSpy = spyOn(process, 'exit').mockImplementation(() => { throw new Error('EXIT'); }); + const errSpy = spyOn(console, 'error').mockImplementation(() => {}); + try { + await runDream(engine, ['--source-id']); + } catch (e: any) { + expect(e.message).toBe('EXIT'); + } + expect(exitSpy).toHaveBeenCalledWith(2); + expect(errSpy.mock.calls.flat().join(' ')).toMatch(/--source-id.*missing value/); + exitSpy.mockRestore(); + errSpy.mockRestore(); + }); + + test('--source X --source Y (repeated, different values) exits 2', async () => { + const exitSpy = spyOn(process, 'exit').mockImplementation(() => { throw new Error('EXIT'); }); + const errSpy = spyOn(console, 'error').mockImplementation(() => {}); + try { + await runDream(engine, ['--source', 'foo', '--source', 'bar']); + } catch (e: any) { + expect(e.message).toBe('EXIT'); + } + expect(exitSpy).toHaveBeenCalledWith(2); + expect(errSpy.mock.calls.flat().join(' ')).toMatch(/specify --source once/); + exitSpy.mockRestore(); + errSpy.mockRestore(); + }); + + test('--source X --source X (repeated, same value) is accepted', async () => { + await seedSource('alpha'); + const report = await runDream(engine, ['--dir', repo, '--phase', 'lint', '--source', 'alpha', '--source', 'alpha', '--json']); + expect(report).toBeTruthy(); + if (report) expect(['ok', 'clean']).toContain(report.status); + }, 60_000); + + test('--source X --source-id Y (conflict) exits 2', async () => { + const exitSpy = spyOn(process, 'exit').mockImplementation(() => { throw new Error('EXIT'); }); + const errSpy = spyOn(console, 'error').mockImplementation(() => {}); + try { + await runDream(engine, ['--source', 'foo', '--source-id', 'bar']); + } catch (e: any) { + expect(e.message).toBe('EXIT'); + } + expect(exitSpy).toHaveBeenCalledWith(2); + expect(errSpy.mock.calls.flat().join(' ')).toMatch(/use --source OR --source-id, not both/); + exitSpy.mockRestore(); + errSpy.mockRestore(); + }); + + // ─── Help short-circuit ordering (C-8 IRON RULE) ──────────────────── + + test('--help --source whatever prints help and exits 0 (no engine-null error)', async () => { + // engine null + --source set would normally exit 1, but --help short-circuits first. + const exitSpy = spyOn(process, 'exit').mockImplementation(() => { throw new Error('EXIT'); }); + const logSpy = spyOn(console, 'log').mockImplementation(() => {}); + try { + const result = await runDream(null, ['--help', '--source', 'anything']); + expect(result).toBeUndefined(); + } catch (e: any) { + throw new Error('--help with --source should NOT exit; got: ' + e.message); + } + expect(exitSpy).not.toHaveBeenCalled(); + expect(logSpy.mock.calls.flat().join(' ')).toMatch(/Usage: gbrain dream/); + exitSpy.mockRestore(); + logSpy.mockRestore(); + }); + + // ─── Engine-null guard (D1) ───────────────────────────────────────── + + test('engine=null + --source set exits 1 with "requires a connected brain"', async () => { + const exitSpy = spyOn(process, 'exit').mockImplementation(() => { throw new Error('EXIT'); }); + const errSpy = spyOn(console, 'error').mockImplementation(() => {}); + try { + await runDream(null, ['--source', 'whatever']); + } catch (e: any) { + expect(e.message).toBe('EXIT'); + } + expect(exitSpy).toHaveBeenCalledWith(1); + expect(errSpy.mock.calls.flat().join(' ')).toMatch(/requires a connected brain/); + exitSpy.mockRestore(); + errSpy.mockRestore(); + }); + + // ─── Unknown source (resolveSourceId throw) ───────────────────────── + + test('--source exits 1 with assertSourceExists hint', async () => { + const exitSpy = spyOn(process, 'exit').mockImplementation(() => { throw new Error('EXIT'); }); + const errSpy = spyOn(console, 'error').mockImplementation(() => {}); + try { + await runDream(engine, ['--source', 'no-such-source']); + } catch (e: any) { + expect(e.message).toBe('EXIT'); + } + expect(exitSpy).toHaveBeenCalledWith(1); + const errOut = errSpy.mock.calls.flat().join(' '); + expect(errOut).toMatch(/Source "no-such-source" not found/); + expect(errOut).toMatch(/gbrain sources list/); + exitSpy.mockRestore(); + errSpy.mockRestore(); + }); + + // ─── Archived source guard (D2) ───────────────────────────────────── + + test('--source exits 1 and leaves last_full_cycle_at untouched', async () => { + await seedSource('archived-thing', /* archived = */ true); + const before = await readLastFullCycleAt('archived-thing'); + expect(before).toBeNull(); + + const exitSpy = spyOn(process, 'exit').mockImplementation(() => { throw new Error('EXIT'); }); + const errSpy = spyOn(console, 'error').mockImplementation(() => {}); + try { + await runDream(engine, ['--source', 'archived-thing']); + } catch (e: any) { + expect(e.message).toBe('EXIT'); + } + expect(exitSpy).toHaveBeenCalledWith(1); + const errOut = errSpy.mock.calls.flat().join(' '); + expect(errOut).toMatch(/source archived-thing is archived/); + expect(errOut).toMatch(/gbrain sources restore archived-thing/); + + const after = await readLastFullCycleAt('archived-thing'); + expect(after).toBeNull(); // archived guard prevents writeback + exitSpy.mockRestore(); + errSpy.mockRestore(); + }); + + // ─── Happy path: --source writes last_full_cycle_at (the bug fix) ─── + + test('--source writes last_full_cycle_at on success (PR #1559 regression)', async () => { + await seedSource('media-corpus'); + const before = await readLastFullCycleAt('media-corpus'); + expect(before).toBeNull(); + + const t0 = Date.now(); + const report = await runDream(engine, ['--dir', repo, '--source', 'media-corpus', '--phase', 'lint', '--json']); + expect(report).toBeTruthy(); + if (report) expect(['ok', 'clean']).toContain(report.status); + + const after = await readLastFullCycleAt('media-corpus'); + expect(after).not.toBeNull(); + const writtenMs = new Date(after!).getTime(); + expect(writtenMs).toBeGreaterThanOrEqual(t0); + expect(writtenMs).toBeLessThanOrEqual(Date.now() + 1000); + }, 60_000); + + // ─── Back-compat: bare `gbrain dream` does NOT write per-source stamp ─ + + test('gbrain dream (no --source) leaves all sources untouched (back-compat regression)', async () => { + await seedSource('alpha'); + await seedSource('beta'); + const report = await runDream(engine, ['--dir', repo, '--phase', 'lint', '--json']); + expect(report).toBeTruthy(); + expect(await readLastFullCycleAt('alpha')).toBeNull(); + expect(await readLastFullCycleAt('beta')).toBeNull(); + }, 60_000); + + // ─── --source-id alias equivalence (D3) ───────────────────────────── + + test('--source-id is equivalent to --source (writes timestamp)', async () => { + await seedSource('beta'); + const before = await readLastFullCycleAt('beta'); + expect(before).toBeNull(); + + const report = await runDream(engine, ['--dir', repo, '--source-id', 'beta', '--phase', 'lint', '--json']); + expect(report).toBeTruthy(); + if (report) expect(['ok', 'clean']).toContain(report.status); + + const after = await readLastFullCycleAt('beta'); + expect(after).not.toBeNull(); + }, 60_000); + + // ─── T3: TypeError MUST propagate (not swallowed by predicate-gated catch) ─ + + test('non-resolver-user errors propagate uncaught (T3)', async () => { + // Monkey-patch executeRaw to throw a synthetic TypeError on the + // assertSourceExists SELECT. The typed-error catch in runDream only + // matches resolver-user-error message shapes; a TypeError thrown + // from any source-resolution path must bubble up with its original + // stack trace (proving real programmer bugs are NOT hidden behind + // operator-error UX). + await seedSource('gamma'); + const original = (engine as any).executeRaw.bind(engine); + let restored = false; + try { + // Throw a TypeError on the source-lookup SELECT that assertSourceExists runs. + // Other executeRaw calls (used by engine internals during cycle) keep + // working so the test exercises ONLY the resolution-path failure. + (engine as any).executeRaw = async (sql: string, params?: unknown[]) => { + if (typeof sql === 'string' && /FROM\s+sources\s+WHERE\s+id\s*=/i.test(sql)) { + throw new TypeError('synthetic-test-bug'); + } + return original(sql, params); + }; + await expect( + runDream(engine, ['--dir', repo, '--source', 'gamma', '--phase', 'lint', '--json']) + ).rejects.toThrow(/synthetic-test-bug/); + } finally { + (engine as any).executeRaw = original; + restored = true; + } + expect(restored).toBe(true); + }); +}); + +// ─── v0.41.13 D5: end-to-end dream β†’ checkCycleFreshness parity ─────── +// +// Closes the column-rename drift class: if a future PR renames +// last_full_cycle_at on one side but not the other, both isolated +// tests stay green but production breaks. This exercises the FULL +// chain through the exact seam both sides consume. + +describe('runDream β†’ checkCycleFreshness end-to-end (D5)', () => { + let repo: string; + let engine: InstanceType; + + beforeEach(async () => { + repo = makeGitRepo(); + engine = await makePGLite(); + }, 60_000); + + afterEach(async () => { + if (engine) await engine.disconnect(); + rmSync(repo, { recursive: true, force: true }); + }, 60_000); + + test('stale source becomes fresh after dream --source (column-name drift guard)', async () => { + // Seed source with last_full_cycle_at backdated 25h (above warn floor). + const stale = new Date(Date.now() - 25 * 60 * 60 * 1000).toISOString(); + await engine.executeRaw( + `INSERT INTO sources (id, name, local_path, config, archived, created_at) + VALUES ('gamma', 'gamma', $1, jsonb_build_object('last_full_cycle_at', $2::text), false, NOW())`, + [repo, stale], + ); + + // Doctor sees stale (warn or fail; we only care that it's NOT ok) + const { checkCycleFreshness } = await import('../src/commands/doctor.ts'); + const beforeCheck = await checkCycleFreshness(engine); + expect(beforeCheck.status).not.toBe('ok'); + + // Run dream against the stale source. + const report = await runDream(engine, ['--dir', repo, '--source', 'gamma', '--phase', 'lint', '--json']); + expect(report).toBeTruthy(); + if (report) expect(['ok', 'clean']).toContain(report.status); + + // Doctor now sees fresh. + const afterCheck = await checkCycleFreshness(engine); + expect(afterCheck.status).toBe('ok'); + }, 60_000); +}); diff --git a/test/import-file-content-sanity.test.ts b/test/import-file-content-sanity.test.ts index 960bc3ec9..19b78cad1 100644 --- a/test/import-file-content-sanity.test.ts +++ b/test/import-file-content-sanity.test.ts @@ -101,6 +101,54 @@ created: 2026-05-24 expect(page).toBeNull(); }); }); + + // ─── v0.41.13: end-to-end coverage for the expanded patterns ──────── + // Exercises the assessor wiring (not just the regex) per D6. + + test.each([ + ['Forbidden', 'error_page_title'], + ['Access Denied', 'error_page_title'], + ['Service Unavailable', 'error_page_title'], + ['Robot Check', 'error_page_title'], + ['Just a moment...', 'cloudflare_challenge_title'], + ])('v0.41.13: title %j β†’ ContentSanityBlockError (matches %s)', async (title, expectedPattern) => { + await withIsolatedHome(async () => { + const content = `--- +title: '${title}' +type: note +created: 2026-05-24 +--- + +scraper junk body`; + let caught: ContentSanityBlockError | undefined; + try { + await importFromContent(engine, 'test/v04113-' + title.toLowerCase().replace(/[^a-z]/g, '-'), content, { noEmbed: true }); + } catch (e) { + if (e instanceof ContentSanityBlockError) caught = e; + else throw e; + } + expect(caught).toBeDefined(); + expect(caught!.result.junk_pattern_matches).toContain(expectedPattern); + expect(caught!.message).toContain('PAGE_JUNK_PATTERN'); + }); + }); + + test('v0.41.13: over-match regression β€” "How to Handle Access Denied Errors" imports cleanly', async () => { + await withIsolatedHome(async () => { + const content = `--- +title: 'How to Handle Access Denied Errors' +type: note +created: 2026-05-24 +--- + +A legitimate essay about handling access-denied errors in your app.`; + // Should NOT throw. + const result = await importFromContent(engine, 'test/v04113-essay', content, { noEmbed: true }); + expect(result.status).not.toBe('error'); + const page = await engine.getPage('test/v04113-essay'); + expect(page).not.toBeNull(); + }); + }); }); describe('importFromContent β€” soft-block (D9 transition + embed_skip)', () => {