mirror of
https://github.com/garrytan/gbrain.git
synced 2026-07-27 22:15:33 +00:00
v0.42.32.0 fix(sync): coerce non-string frontmatter titles + bounded auto-skip failure ledger (#1939) (#1956)
* fix(import): coerce non-string frontmatter title/slug/type (#1939) YAML `title: 2024-06-01` parses to a Date and `title: 1458` to a number; the old `(frontmatter.X as string)` cast was a compile-time lie, so downstream `.toLowerCase()` threw and (via the importer failure gate) could wedge sync indefinitely. parseMarkdown now coerces via coerceFrontmatterString (Date -> UTC ISO date, deterministic), and the pure assessContentSanity self-protects against a non-string title. * feat(sync): bounded auto-skip failure ledger; poison file can't wedge indexing (#1939) New src/core/sync-failure-ledger.ts owns the failure store + a crash-safe, multi-source, concurrent bounded auto-skip valve. A file that fails N consecutive syncs (GBRAIN_SYNC_AUTOSKIP_AFTER, default 3) auto-skips so it can't freeze all indexing forever, while fresh failures still fail-closed and a `<head>` history-rewrite sentinel hard-blocks even with --skip-failed. - (source_id, path) keying — failures never merge across sources - success clears a path so attempts are truly consecutive - advance-before-ack ordering (a crash can't mark a file skipped while wedged) - shared applySyncFailureGate used by BOTH the incremental and full-sync gates - legacy-row normalization + duplicate collapse on load - cross-process lock + atomic temp-rename, age-based stale-lock break sync.ts re-exports the ledger for existing callers; import.ts records source-scoped and defers the bookmark to the gate under managedBookmark. * fix(doctor): sync_failures severity via one shared decision on both surfaces (#1939) Local buildChecks and remote doctorReportRemote now both route through decideSyncFailureSeverity, so a stuck bookmark escalates WARN -> FAIL consistently (oldest-open age > fail cadence, or large unresolved count), auto-skipped pages stay visible (WARN, not hidden), and the acknowledged/acknowledged_at field-split that caused drift is gone. The remote surface stays subprocess-free (file read + Date.parse only). * chore(test): add trailing newline to e5-lease-cap-ab baseline fixture * fix(sync): address adversarial review findings on the failure ledger (#1939) - #1: a parse-failed file that is later deleted/renamed-away no longer leaves a permanent open ledger row. Removed paths (filtered.deleted, renamed-from, and the "gone from disk" forward-delete skip branch) are treated as resolved so the ledger self-heals instead of aging doctor to a stuck FAIL. - #3: decideSyncFailureSeverity escalates to FAIL on OPEN (blocking) failures only — auto_skipped rows already advanced the bookmark, so they stay WARN-visible regardless of count, matching the state-machine contract. * chore: bump version and changelog (v0.42.30.0) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: document sync-failure ledger + auto-skip valve for v0.42.30.0 KEY_FILES.md: new src/core/sync-failure-ledger.ts entry (bounded auto-skip state machine, decideGateAction/decideSyncFailureSeverity/applySyncFailureGate, GBRAIN_SYNC_AUTOSKIP_AFTER); update sync.ts (failure store moved to ledger, re-exported), doctor.ts (sync_failures severity via shared rule on both surfaces), markdown.ts (coerceFrontmatterString), import.ts (managedBookmark). live-sync.md: poison-file auto-skip tricky-spot. Regenerated llms-full.txt. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore: re-bump to v0.42.31.0 (queue collision on 0.42.30.0) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore: re-bump to v0.42.32.0 (queue collision) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Garry Tan
co-authored by
Claude Opus 4.8
Claude Opus 4.8
parent
f401d7407e
commit
5a06af5a57
@@ -2,6 +2,24 @@
|
||||
|
||||
All notable changes to GBrain will be documented in this file.
|
||||
|
||||
## [0.42.32.0] - 2026-06-07
|
||||
|
||||
**A single un-parseable note can no longer silently stop your brain from indexing anything new.** A page whose YAML frontmatter `title:` was a bare date (`title: 2024-06-01`) or number (`title: 1458`) parsed as a Date/number, not text — and the importer threw when it tried to lowercase it. That throw blocked the sync bookmark from advancing, so every later `gbrain sync` re-walked the whole repo, never reached HEAD, and quietly stopped indexing new commits. The page was committed and on GitHub, but `gbrain get` returned `page_not_found` with no surfaced error.
|
||||
|
||||
Two fixes. First, a non-string title/slug/type now coerces deterministically at parse time — a YAML date becomes its UTC ISO string (`2024-06-01`), so the same page reads the same on every machine and the import never throws. Second, the importer gained a **bounded auto-skip safety valve**: a file that fails to import N consecutive syncs (default 3, `GBRAIN_SYNC_AUTOSKIP_AFTER`) is recorded and skipped so it can't wedge all indexing forever — while a *fresh* failure still fails closed (the bookmark holds and you're told what broke), and a repository history rewrite still hard-blocks even with `--skip-failed`. Skipped pages stay visible: `gbrain doctor` keeps warning until you fix them, and escalates to a hard failure when a real failure has blocked the bookmark past the staleness window.
|
||||
|
||||
`gbrain doctor` now decides sync-failure severity through one shared rule on both the local and remote surfaces, so a stuck bookmark surfaces identically whether you run doctor on your own machine or against a remote brain.
|
||||
|
||||
### Added
|
||||
- **Bounded auto-skip sync ledger.** A file that fails N consecutive syncs (`GBRAIN_SYNC_AUTOSKIP_AFTER`, default 3; set `0` to disable) is auto-skipped so one poison file can't freeze indexing for the whole brain. Skips are per-source, survive crashes (the bookmark advances before anything is marked skipped), and self-heal — fix or delete the file and the next sync clears it. `gbrain doctor` lists what was skipped and why.
|
||||
|
||||
### Fixed
|
||||
- **Non-string frontmatter titles no longer wedge indexing (#1939).** `title: 2024-06-01` / `title: 1458` (and date/number `slug`/`type`) coerce to deterministic strings at parse time instead of throwing, so a handful of date-named notes can't silently stop your brain from indexing new commits.
|
||||
- **`gbrain doctor` sync-failure severity is now consistent across surfaces (#1939).** Local and remote doctor share one decision: a stuck bookmark escalates to FAIL once it has blocked past the staleness window (or many files are blocking), while already-skipped pages stay a visible warning.
|
||||
|
||||
### To take advantage of v0.42.32.0
|
||||
Upgrade and run `gbrain sync` once. Any pages that previously wedged the importer (bare date/number titles) now import on their own. If a file still genuinely can't parse, sync tells you which one; fix it, or let the auto-skip valve move past it after a few runs and watch for it in `gbrain doctor`. Tune the threshold with `GBRAIN_SYNC_AUTOSKIP_AFTER` (set `0` to keep the strict fail-closed behavior).
|
||||
|
||||
## [0.42.31.0] - 2026-06-07
|
||||
|
||||
**You can now write typed graph edges with your own provenance straight from the CLI — `gbrain link-add a b --link-type relies-on --link-source citation-graph` — and an external edge-writer (a citation-graph ingester, an importer, a classifier) no longer needs a gbrain schema migration to register a new provenance.** Two ergonomics gaps for tools that compute edges out-of-band, filed by a downstream agent building a citation-graph ingester (#1941).
|
||||
|
||||
@@ -1828,22 +1828,18 @@ at plan time and got carved out:
|
||||
|
||||
## v0.40.3.0 follow-ups (v0.41+)
|
||||
|
||||
- [ ] **v0.41+: source-scope the `sync-failures.jsonl` log so `--skip-failed` works under `--parallel > 1`.**
|
||||
v0.40.3.0 shipped `gbrain sync --all --parallel N` as a continuous worker pool
|
||||
with per-source DB locks. The remaining unsafe path: `recordSyncFailures()` /
|
||||
`acknowledgeSyncFailures()` in `src/core/sync.ts` write to a brain-global JSONL
|
||||
file at `~/.gbrain/sync-failures.jsonl` with no per-source scope. Under parallel
|
||||
sync, source A's `--skip-failed` ack can swallow source B's failures recorded
|
||||
while B was still running. v0.40.3.0's safe interim: refuse to combine
|
||||
`--skip-failed` / `--retry-failed` with `--parallel > 1` (loud error, paste-ready
|
||||
hint pointing at `--parallel 1`). The proper fix: (1) extend the JSONL row
|
||||
schema with a `source_id` field; (2) `recordSyncFailures(failures, sourceId)`
|
||||
stamps the field; (3) `acknowledgeSyncFailures({sourceId})` filters acks to
|
||||
one source's rows; (4) `unacknowledgedSyncFailures({sourceId})` reads the
|
||||
subset. Drop the v0.40.3.0 restriction once source-scoped acks are
|
||||
deterministic. Estimate: ~1-2 days. Filed during v0.40.3.0 plan review by
|
||||
Codex outside-voice (decision D15 → B in the eng-review plan at
|
||||
`~/.claude/plans/system-instruction-you-are-working-fluttering-grove.md`).
|
||||
- [ ] **v0.41+: drop the `--skip-failed` / `--retry-failed` + `--parallel > 1` restriction now that the failure log is source-scoped.**
|
||||
**Priority:** P3
|
||||
v0.42.32.0 (#1939) landed the source-scoping infrastructure this TODO asked
|
||||
for: `src/core/sync-failure-ledger.ts` keys every row by `(source_id, path)`,
|
||||
`recordFailures(sourceId, …)` stamps it, `acknowledgeFailures(sourceId)` /
|
||||
`autoSkipFailures(sourceId, …)` filter to one source, and a cross-process
|
||||
lock + atomic temp-rename (`withLedgerLock`) makes concurrent read-modify-write
|
||||
safe. The remaining work is just to LIFT the v0.40.3.0 interim guard at
|
||||
`src/commands/sync.ts:3078` (`parallelEligible && (skipFailed || retryFailed)`
|
||||
→ loud refuse) after adding a test that proves source-scoped acks stay
|
||||
deterministic under `--all --parallel N`. Estimate: ~0.5 day. Originally filed
|
||||
during the v0.40.3.0 plan review (Codex outside-voice, decision D15 → B).
|
||||
|
||||
- [ ] **v0.41+ (optional): extend `checkSyncFreshness` to include `embedding_coverage_pct`
|
||||
per source.** v0.40.3.0 plan originally proposed adding a NEW doctor check
|
||||
|
||||
File diff suppressed because one or more lines are too long
@@ -120,6 +120,17 @@ hashes match. If both a cron and `--watch` fire simultaneously, no conflict.
|
||||
server is down when a push happens, that sync is missed. Pair webhooks
|
||||
with a cron fallback that catches anything the webhook missed.
|
||||
|
||||
4. **A single un-parseable file can't wedge all indexing.** When a file fails
|
||||
to import (malformed YAML frontmatter, an unquoted colon, etc.), sync holds
|
||||
the bookmark and tells you exactly which file broke — a *fresh* failure
|
||||
fails closed so nothing is silently dropped. But a file that fails the same
|
||||
way `GBRAIN_SYNC_AUTOSKIP_AFTER` consecutive syncs (default 3, set `0` to
|
||||
disable) is auto-skipped so the rest of the brain keeps indexing past it.
|
||||
Skipped files don't disappear: `gbrain doctor` keeps warning until you fix
|
||||
or delete them, and fixing the file clears it on the next sync. A repository
|
||||
history rewrite still hard-blocks even with `--skip-failed`. Run
|
||||
`gbrain sync --skip-failed` to acknowledge a known-bad set yourself.
|
||||
|
||||
## How to Verify
|
||||
|
||||
1. **Edit a file and search for the change.** Edit a brain markdown file,
|
||||
|
||||
@@ -2617,6 +2617,17 @@ hashes match. If both a cron and `--watch` fire simultaneously, no conflict.
|
||||
server is down when a push happens, that sync is missed. Pair webhooks
|
||||
with a cron fallback that catches anything the webhook missed.
|
||||
|
||||
4. **A single un-parseable file can't wedge all indexing.** When a file fails
|
||||
to import (malformed YAML frontmatter, an unquoted colon, etc.), sync holds
|
||||
the bookmark and tells you exactly which file broke — a *fresh* failure
|
||||
fails closed so nothing is silently dropped. But a file that fails the same
|
||||
way `GBRAIN_SYNC_AUTOSKIP_AFTER` consecutive syncs (default 3, set `0` to
|
||||
disable) is auto-skipped so the rest of the brain keeps indexing past it.
|
||||
Skipped files don't disappear: `gbrain doctor` keeps warning until you fix
|
||||
or delete them, and fixing the file clears it on the next sync. A repository
|
||||
history rewrite still hard-blocks even with `--skip-failed`. Run
|
||||
`gbrain sync --skip-failed` to acknowledge a known-bad set yourself.
|
||||
|
||||
## How to Verify
|
||||
|
||||
1. **Edit a file and search for the change.** Edit a brain markdown file,
|
||||
|
||||
+1
-1
@@ -143,5 +143,5 @@
|
||||
"bun": ">=1.3.10"
|
||||
},
|
||||
"license": "MIT",
|
||||
"version": "0.42.31.0"
|
||||
"version": "0.42.32.0"
|
||||
}
|
||||
|
||||
+37
-34
@@ -569,29 +569,24 @@ export async function doctorReportRemote(engine: BrainEngine): Promise<DoctorRep
|
||||
// remote doctor.
|
||||
}
|
||||
|
||||
// 4. Sync failures (file-plane state, not in-DB; see src/core/sync.ts).
|
||||
// Read the JSONL file directly at the canonical path; cheap and engine-agnostic.
|
||||
// 4. Sync failures (file-plane ledger; see src/core/sync-failure-ledger.ts).
|
||||
// issue #1939: read via the shared loader + severity decision so this remote
|
||||
// surface agrees with the local buildChecks emitter by construction. Stays
|
||||
// subprocess-free (file read + Date.parse only, no git), preserving the remote
|
||||
// trust boundary. Escalates to FAIL when a stuck bookmark has blocked past the
|
||||
// sync-freshness fail cadence or unresolved count is large.
|
||||
try {
|
||||
const { readFileSync, existsSync } = await import('fs');
|
||||
const { gbrainPath } = await import('../core/config.ts');
|
||||
const path = gbrainPath('sync-failures.jsonl');
|
||||
let unacked = 0;
|
||||
if (existsSync(path)) {
|
||||
const lines = readFileSync(path, 'utf-8').split('\n').filter(l => l.trim());
|
||||
for (const line of lines) {
|
||||
try {
|
||||
const entry = JSON.parse(line) as { acknowledged_at?: string | null };
|
||||
if (!entry.acknowledged_at) unacked++;
|
||||
} catch { /* skip malformed line */ }
|
||||
}
|
||||
}
|
||||
checks.push({
|
||||
name: 'sync_failures',
|
||||
status: unacked === 0 ? 'ok' : 'warn',
|
||||
message: unacked === 0
|
||||
? 'No unacked failures'
|
||||
: `${unacked} unacked failure(s) — run \`gbrain sync --skip-failed\` on the host to acknowledge`,
|
||||
});
|
||||
const { loadSyncFailures, decideSyncFailureSeverity } = await import('../core/sync.ts');
|
||||
const entries = loadSyncFailures();
|
||||
const failHours = _resolveSyncFreshnessHours('GBRAIN_SYNC_FRESHNESS_FAIL_HOURS', 72);
|
||||
const sev = decideSyncFailureSeverity({ entries, nowMs: Date.now(), failHours });
|
||||
const msg =
|
||||
sev.unresolved === 0
|
||||
? 'No unresolved sync failures'
|
||||
: `${sev.unresolved} unresolved sync failure(s)` +
|
||||
(sev.auto_skipped > 0 ? ` (${sev.auto_skipped} auto-skipped — pages NOT indexed)` : '') +
|
||||
` — run \`gbrain sync --skip-failed\` on the host to acknowledge`;
|
||||
checks.push({ name: 'sync_failures', status: sev.status, message: msg });
|
||||
} catch {
|
||||
checks.push({ name: 'sync_failures', status: 'ok', message: 'No failures recorded' });
|
||||
}
|
||||
@@ -4396,19 +4391,25 @@ export async function buildChecks(
|
||||
// Without this doctor check, users see "sync blocked" and have no
|
||||
// surface showing which files to fix.
|
||||
try {
|
||||
const { unacknowledgedSyncFailures, loadSyncFailures, summarizeFailuresByCode } = await import('../core/sync.ts');
|
||||
const unacked = unacknowledgedSyncFailures();
|
||||
const { unacknowledgedSyncFailures, loadSyncFailures, summarizeFailuresByCode, decideSyncFailureSeverity } = await import('../core/sync.ts');
|
||||
const all = loadSyncFailures();
|
||||
if (unacked.length > 0) {
|
||||
const codeSummary = summarizeFailuresByCode(unacked);
|
||||
// issue #1939: "unresolved" = open + auto_skipped. Severity (ok/warn/fail)
|
||||
// comes from the SAME shared decision the remote surface uses, so a stuck
|
||||
// bookmark blocked past the fail cadence (or a large unresolved count)
|
||||
// escalates to FAIL instead of staying a quiet WARN forever.
|
||||
const unresolved = unacknowledgedSyncFailures();
|
||||
if (unresolved.length > 0) {
|
||||
const failHours = _resolveSyncFreshnessHours('GBRAIN_SYNC_FRESHNESS_FAIL_HOURS', 72);
|
||||
const sev = decideSyncFailureSeverity({ entries: all, nowMs: Date.now(), failHours });
|
||||
const codeSummary = summarizeFailuresByCode(unresolved);
|
||||
const codeBreakdown = codeSummary.map(s => `${s.code}=${s.count}`).join(', ');
|
||||
const preview = unacked.slice(0, 3).map(f => `${f.path} (${f.error.slice(0, 60)})`).join('; ');
|
||||
const preview = unresolved.slice(0, 3).map(f => `${f.path} (${f.error.slice(0, 60)})`).join('; ');
|
||||
// v0.40.3.0 T8b (D8 + D12 Bug 3): emit a single sync-retry-failed
|
||||
// step. sync-skip-failed is DELIBERATELY NOT emitted as a remediation
|
||||
// — auto-skipping failed syncs hides data loss. Operators can still
|
||||
// run `gbrain sync --skip-failed` manually.
|
||||
const { makeRemediationStep } = await import('../core/remediation-step.ts');
|
||||
const oldestTs = unacked.reduce(
|
||||
const oldestTs = unresolved.reduce(
|
||||
(acc, f) => (acc === '' || f.ts < acc ? f.ts : acc),
|
||||
'',
|
||||
);
|
||||
@@ -4417,18 +4418,20 @@ export async function buildChecks(
|
||||
job: 'sync-retry-failed',
|
||||
// Content-stable per codex D12 Bug 2: count + oldest_ts captures
|
||||
// the relevant state without using a real timestamp.
|
||||
params: { failure_count: unacked.length, oldest_failure: oldestTs },
|
||||
severity: unacked.length >= 10 ? 'high' : 'medium',
|
||||
params: { failure_count: unresolved.length, oldest_failure: oldestTs },
|
||||
severity: sev.status === 'fail' ? 'high' : 'medium',
|
||||
est_seconds: 30,
|
||||
est_usd_cost: 0,
|
||||
rationale: `Retry ${unacked.length} unacked sync failure(s) (codes: ${codeBreakdown})`,
|
||||
rationale: `Retry ${unresolved.length} unresolved sync failure(s) (codes: ${codeBreakdown})`,
|
||||
});
|
||||
checks.push({
|
||||
name: 'sync_failures',
|
||||
status: 'warn',
|
||||
status: sev.status,
|
||||
message:
|
||||
`${unacked.length} unacknowledged sync failure(s) [${codeBreakdown}]. ${preview}` +
|
||||
`${unacked.length > 3 ? `, and ${unacked.length - 3} more` : ''}. ` +
|
||||
`${unresolved.length} unresolved sync failure(s) [${codeBreakdown}]` +
|
||||
(sev.auto_skipped > 0 ? ` — ${sev.auto_skipped} auto-skipped (pages NOT indexed)` : '') +
|
||||
`. ${preview}` +
|
||||
`${unresolved.length > 3 ? `, and ${unresolved.length - 3} more` : ''}. ` +
|
||||
`Fix the file(s) and re-run 'gbrain sync', or use 'gbrain sync --skip-failed' to acknowledge.`,
|
||||
remediation: [retryStep],
|
||||
remediation_status: 'remediable',
|
||||
|
||||
@@ -44,7 +44,7 @@ export interface RunImportResult {
|
||||
export async function runImport(
|
||||
engine: BrainEngine,
|
||||
args: string[],
|
||||
opts: { commit?: string; strategy?: SyncStrategy; sourceId?: string } = {},
|
||||
opts: { commit?: string; strategy?: SyncStrategy; sourceId?: string; managedBookmark?: boolean } = {},
|
||||
): Promise<RunImportResult> {
|
||||
const noEmbed = args.includes('--no-embed');
|
||||
const fresh = args.includes('--fresh');
|
||||
@@ -438,13 +438,17 @@ export async function runImport(
|
||||
// Not a git repo or git not available
|
||||
}
|
||||
|
||||
if (gitHead) {
|
||||
// issue #1939: when performFullSync drives runImport it owns the failure
|
||||
// ledger + bookmark via the shared gate (applySyncFailureGate). Skipping the
|
||||
// internal handling here prevents double-recording (which would double-count
|
||||
// the auto-skip `attempts` streak) and a competing bookmark write.
|
||||
if (gitHead && !opts.managedBookmark) {
|
||||
// Record failures into the central JSONL so doctor can surface them.
|
||||
// Use gitHead as the commit so a later sync can tell "same broken
|
||||
// state as last time" from "new broken state."
|
||||
// state as last time" from "new broken state." Source-scoped (#1939 #2).
|
||||
if (failures.length > 0) {
|
||||
const { recordSyncFailures } = await import('../core/sync.ts');
|
||||
recordSyncFailures(failures, gitHead);
|
||||
const { recordFailures } = await import('../core/sync.ts');
|
||||
recordFailures(opts.sourceId ?? 'default', failures, gitHead);
|
||||
}
|
||||
if (failures.length === 0) {
|
||||
await engine.setConfig('sync.last_commit', gitHead);
|
||||
|
||||
+173
-107
@@ -11,10 +11,14 @@ import {
|
||||
isSyncable,
|
||||
unsyncableReason,
|
||||
resolveSlugForPath,
|
||||
recordSyncFailures,
|
||||
unacknowledgedSyncFailures,
|
||||
acknowledgeSyncFailures,
|
||||
loadSyncFailures,
|
||||
formatCodeBreakdown,
|
||||
applySyncFailureGate,
|
||||
isSkippablePath,
|
||||
resolveAutoSkipThreshold,
|
||||
DEFAULT_SOURCE_ID,
|
||||
} from '../core/sync.ts';
|
||||
import { estimateTokens, CHUNKER_VERSION } from '../core/chunkers/code.ts';
|
||||
import {
|
||||
@@ -1472,6 +1476,10 @@ async function performSyncInner(engine: BrainEngine, opts: SyncOpts): Promise<Sy
|
||||
};
|
||||
|
||||
const pagesAffected: string[] = [];
|
||||
// issue #1939: file paths that imported cleanly this run. The failure-ledger
|
||||
// gate clears these so a previously-failing file's `attempts` streak resets
|
||||
// on success (consecutive-failure semantics for the auto-skip valve).
|
||||
const succeededPaths: string[] = [];
|
||||
let chunksCreated = 0;
|
||||
// v0.41.13.0 (T2): tracks add+modify files actually persisted so far.
|
||||
// Only bumped from inside importOnePath's success path. partial() reports
|
||||
@@ -1811,6 +1819,11 @@ async function performSyncInner(engine: BrainEngine, opts: SyncOpts): Promise<Sy
|
||||
// correct only when the gate compared HEAD == captured; under pinning a
|
||||
// forward delete must not block.
|
||||
await markCompleted(path);
|
||||
// issue #1939 adversarial finding #1: a file that previously failed to
|
||||
// parse (open ledger row) and is now gone from disk is resolved — clear
|
||||
// its row so it can't age doctor to a permanent FAIL. (This covers the
|
||||
// net-zero add-then-delete range where the path isn't in filtered.deleted.)
|
||||
succeededPaths.push(path);
|
||||
progress.tick(1, `skip:${path}`);
|
||||
return;
|
||||
}
|
||||
@@ -1831,6 +1844,9 @@ async function performSyncInner(engine: BrainEngine, opts: SyncOpts): Promise<Sy
|
||||
if (result.status === 'imported') {
|
||||
chunksCreated += result.chunks;
|
||||
pagesAffected.push(result.slug);
|
||||
// issue #1939: record the file path (not slug) so the gate clears any
|
||||
// prior failure-ledger row — success resets the auto-skip attempt streak.
|
||||
succeededPaths.push(path);
|
||||
// v0.41.13.0 (T2): bump filesImported on every successful
|
||||
// persist. partial() reports this so cron operators see how
|
||||
// much actually landed before --timeout fired.
|
||||
@@ -1993,79 +2009,104 @@ async function performSyncInner(engine: BrainEngine, opts: SyncOpts): Promise<Sy
|
||||
|
||||
const elapsed = Date.now() - start;
|
||||
|
||||
// Bug 9 — gate the sync bookmark on success. If any per-file parse
|
||||
// failed, record it to ~/.gbrain/sync-failures.jsonl and DO NOT advance
|
||||
// sync.last_commit. The next sync re-walks the same diff and re-attempts
|
||||
// the failed files. Escape hatches: --skip-failed acknowledges the
|
||||
// current set, --retry-failed re-parses before running the normal sync.
|
||||
if (failedFiles.length > 0) {
|
||||
recordSyncFailures(failedFiles, pin);
|
||||
// Emit structured summary grouped by error code so the operator
|
||||
// can see *why* files failed, not just how many.
|
||||
// issue #1939 — gate the bookmark through the shared failure ledger.
|
||||
// • Fresh failures still BLOCK (fail-closed): the next sync re-walks the
|
||||
// diff and re-attempts. Escape hatch: --skip-failed.
|
||||
// • A file that fails >= threshold consecutive syncs AUTO-SKIPS so a poison
|
||||
// file can't wedge all indexing forever (recorded, surfaced by doctor).
|
||||
// • A `<head>` SENTINEL (history rewrite) HARD-BLOCKS even with
|
||||
// --skip-failed — advancing would record a commit that no longer matches
|
||||
// the indexed tree.
|
||||
// `advance` is the bookmark write; the gate runs it ONLY when advancing, and
|
||||
// ALWAYS before marking anything auto-skipped/acknowledged (crash-atomic).
|
||||
const advance = async (): Promise<void> => {
|
||||
// v0.42.x (#1794): advance to the PINNED target (not live HEAD) — commits
|
||||
// past the pin are the next sync's pin..HEAD diff. `commitTimeMs(pin)` stamps
|
||||
// newest_content_at against the commit we drained to. `last_sync_at` is bumped
|
||||
// HERE and ONLY here so the autopilot scheduler never sees a stuck source as
|
||||
// "fresh". The checkpoint rows clear here — CONVERGENCE CONTRACT: sync
|
||||
// convergence == IMPORT convergence; downstream extract/facts/embed is
|
||||
// decoupled (its own resumable stale sweeps).
|
||||
await writeSyncAnchor(engine, opts.sourceId, 'last_commit', pin, commitTimeMs(repoPath, pin));
|
||||
await engine.setConfig('sync.last_run', new Date().toISOString());
|
||||
await writeSyncAnchor(engine, opts.sourceId, 'repo_path', repoPath);
|
||||
await writeChunkerVersion(engine, opts.sourceId, String(CHUNKER_VERSION));
|
||||
await clearOpCheckpoint(engine, ckpt.paths);
|
||||
await clearOpCheckpoint(engine, ckpt.target);
|
||||
};
|
||||
|
||||
// issue #1939 adversarial finding #1: a file that failed to parse (open ledger
|
||||
// row) and is then deleted/renamed-away never re-enters failedFiles and never
|
||||
// imports, so its row would never clear and would age doctor to a permanent
|
||||
// FAIL. Treat removed paths as resolved so the ledger self-heals.
|
||||
const resolvedPaths = [
|
||||
...succeededPaths,
|
||||
...filtered.deleted,
|
||||
...filtered.renamed.map(r => r.from),
|
||||
];
|
||||
|
||||
const gate = await applySyncFailureGate({
|
||||
sourceId: opts.sourceId ?? DEFAULT_SOURCE_ID,
|
||||
failedFiles,
|
||||
succeededPaths: resolvedPaths,
|
||||
commit: pin,
|
||||
skipFailed: opts.skipFailed === true,
|
||||
advance,
|
||||
});
|
||||
|
||||
if (!gate.advanced) {
|
||||
const codeBreakdown = formatCodeBreakdown(failedFiles);
|
||||
if (!opts.skipFailed) {
|
||||
if (gate.sentinelBlocked) {
|
||||
serr(
|
||||
`\nSync blocked: ${failedFiles.length} file(s) failed to parse:\n` +
|
||||
`\nSync blocked: repository history changed during sync (force-push / reset).\n` +
|
||||
`${codeBreakdown}\n\n` +
|
||||
`Fix the YAML frontmatter in the files above and re-run, or use ` +
|
||||
`'gbrain sync --skip-failed' to acknowledge and move on.`,
|
||||
`The pinned target is no longer an ancestor of HEAD; advancing would record ` +
|
||||
`a commit that doesn't match the indexed tree. Re-run sync to re-pin against ` +
|
||||
`current HEAD.`,
|
||||
);
|
||||
// Update last_run + repo_path (progress on infra) but NOT last_commit.
|
||||
// v0.42.x (#1794): the checkpoint is INTENTIONALLY left in place — the
|
||||
// banked `completed` set (flushed above) lets the next run skip the files
|
||||
// already drained and re-attempt only the failures.
|
||||
await engine.setConfig('sync.last_run', new Date().toISOString());
|
||||
await writeSyncAnchor(engine, opts.sourceId, 'repo_path', repoPath);
|
||||
return {
|
||||
status: 'blocked_by_failures',
|
||||
fromCommit: lastCommit,
|
||||
toCommit: pin,
|
||||
added: filtered.added.length,
|
||||
modified: filtered.modified.length,
|
||||
deleted: filtered.deleted.length,
|
||||
renamed: filtered.renamed.length,
|
||||
chunksCreated,
|
||||
embedded: 0,
|
||||
pagesAffected,
|
||||
failedFiles: failedFiles.length,
|
||||
};
|
||||
}
|
||||
// --skip-failed: acknowledge the now-recorded set and proceed.
|
||||
const acked = acknowledgeSyncFailures();
|
||||
if (acked.count > 0) {
|
||||
} else {
|
||||
const fileFailCount = failedFiles.filter(f => isSkippablePath(f.path)).length;
|
||||
serr(
|
||||
` Acknowledged ${acked.count} failure(s) and advancing past them:\n` +
|
||||
`${formatCodeBreakdown(acked.summary)}`,
|
||||
`\nSync blocked: ${fileFailCount} file(s) failed to parse:\n` +
|
||||
`${codeBreakdown}\n\n` +
|
||||
`Fix the frontmatter and re-run, or use 'gbrain sync --skip-failed' to ` +
|
||||
`acknowledge and move on. A file that keeps failing auto-skips after ` +
|
||||
`${resolveAutoSkipThreshold()} consecutive syncs.`,
|
||||
);
|
||||
}
|
||||
// Update last_run + repo_path (progress on infra) but NOT last_commit. The
|
||||
// checkpoint is INTENTIONALLY left in place — the banked completed set lets
|
||||
// the next run skip the drained files and re-attempt only the failures.
|
||||
await engine.setConfig('sync.last_run', new Date().toISOString());
|
||||
await writeSyncAnchor(engine, opts.sourceId, 'repo_path', repoPath);
|
||||
return {
|
||||
status: 'blocked_by_failures',
|
||||
fromCommit: lastCommit,
|
||||
toCommit: pin,
|
||||
added: filtered.added.length,
|
||||
modified: filtered.modified.length,
|
||||
deleted: filtered.deleted.length,
|
||||
renamed: filtered.renamed.length,
|
||||
chunksCreated,
|
||||
embedded: 0,
|
||||
pagesAffected,
|
||||
failedFiles: failedFiles.length,
|
||||
};
|
||||
}
|
||||
|
||||
// Update sync state AFTER all changes succeed (source-scoped when
|
||||
// opts.sourceId is set, global config otherwise). v0.42.x (#1794): advance to
|
||||
// the PINNED target (not live HEAD) — commits past the pin are the next
|
||||
// sync's pin..HEAD diff. `commitTimeMs(pin)` stamps newest_content_at against
|
||||
// the commit we actually drained to. `last_sync_at` is bumped HERE and ONLY
|
||||
// here (inside writeSyncAnchor) — never on a killed partial — so the
|
||||
// autopilot scheduler never sees a stuck source as "fresh".
|
||||
await writeSyncAnchor(engine, opts.sourceId, 'last_commit', pin, commitTimeMs(repoPath, pin));
|
||||
await engine.setConfig('sync.last_run', new Date().toISOString());
|
||||
await writeSyncAnchor(engine, opts.sourceId, 'repo_path', repoPath);
|
||||
// v0.20.0 Cathedral II Layer 12: persist the chunker version we just
|
||||
// finished with so the next sync's up_to_date gate respects it. Only
|
||||
// source-scoped syncs track this (see readChunkerVersion for rationale).
|
||||
await writeChunkerVersion(engine, opts.sourceId, String(CHUNKER_VERSION));
|
||||
// v0.42.x (#1794): import drained the full lastCommit..pin range — the anchor
|
||||
// is advanced and both checkpoint rows clear here. CONVERGENCE CONTRACT: sync
|
||||
// convergence == IMPORT convergence. Downstream (extract/facts/embed below)
|
||||
// is DELIBERATELY decoupled from the anchor: it is size-gated (inline only for
|
||||
// small syncs) and otherwise handled by its own resumable sweeps
|
||||
// (extract --stale watermark, embed --stale / embed-backfill, the extract_facts
|
||||
// + conversation_facts_backfill cycle phases). Coupling a 44k-page facts/embed
|
||||
// pass into the anchor gate would re-introduce the exact non-convergence this
|
||||
// fix exists to kill. A kill mid-downstream just means the banked pages get
|
||||
// their links/embeddings/facts from the next stale sweep.
|
||||
await clearOpCheckpoint(engine, ckpt.paths);
|
||||
await clearOpCheckpoint(engine, ckpt.target);
|
||||
// Advanced. Surface what the gate did past the failures.
|
||||
if (gate.acknowledged > 0) {
|
||||
serr(` Acknowledged ${gate.acknowledged} failure(s) and advanced past them.`);
|
||||
}
|
||||
if (gate.autoSkipped.length > 0) {
|
||||
serr(
|
||||
`\n Auto-skipped ${gate.autoSkipped.length} file(s) that failed >= ` +
|
||||
`${resolveAutoSkipThreshold()} consecutive syncs:\n` +
|
||||
gate.autoSkipped.map(p => ` ${p}`).join('\n') + '\n' +
|
||||
` Bookmark advanced; these pages are NOT indexed and remain in ` +
|
||||
`sync-failures.jsonl. 'gbrain doctor' will warn until they're fixed.`,
|
||||
);
|
||||
}
|
||||
|
||||
// Log ingest
|
||||
await engine.logIngest({
|
||||
@@ -2263,55 +2304,80 @@ async function performFullSync(
|
||||
commit: headCommit,
|
||||
strategy: opts.strategy,
|
||||
sourceId: opts.sourceId,
|
||||
// issue #1939: performFullSync owns the failure ledger + bookmark via the
|
||||
// shared gate below; don't let runImport double-record or write its own.
|
||||
managedBookmark: true,
|
||||
});
|
||||
serr(
|
||||
`[gbrain phase] sync.fullsync.import done ${Date.now() - _fullImportT0}ms ` +
|
||||
`imported=${result.imported} skipped=${result.skipped} errors=${result.errors}`,
|
||||
);
|
||||
|
||||
// Bug 9 — gate the full-sync bookmark on success. runImport already
|
||||
// writes its own sync.last_commit conditionally (import.ts), but
|
||||
// performFullSync is called on first-sync + force-full paths where
|
||||
// the sync module owns the last_commit write. Respect the same gate.
|
||||
if (result.failures.length > 0) {
|
||||
recordSyncFailures(result.failures, headCommit);
|
||||
const codeBreakdown = formatCodeBreakdown(result.failures);
|
||||
if (!opts.skipFailed) {
|
||||
serr(
|
||||
`\nFull sync blocked: ${result.failures.length} file(s) failed:\n` +
|
||||
`${codeBreakdown}\n\n` +
|
||||
`Fix the YAML in those files and re-run, or use '--skip-failed'.`,
|
||||
);
|
||||
await engine.setConfig('sync.last_run', new Date().toISOString());
|
||||
await writeSyncAnchor(engine, opts.sourceId, 'repo_path', repoPath);
|
||||
return {
|
||||
status: 'blocked_by_failures',
|
||||
fromCommit: null,
|
||||
toCommit: headCommit,
|
||||
added: 0, modified: 0, deleted: 0, renamed: 0,
|
||||
chunksCreated: result.chunksCreated,
|
||||
embedded: 0,
|
||||
pagesAffected: [],
|
||||
failedFiles: result.failures.length,
|
||||
};
|
||||
}
|
||||
const acked = acknowledgeSyncFailures();
|
||||
if (acked.count > 0) {
|
||||
serr(
|
||||
` Acknowledged ${acked.count} failure(s) and advancing past them:\n` +
|
||||
`${formatCodeBreakdown(acked.summary)}`,
|
||||
);
|
||||
}
|
||||
}
|
||||
// issue #1939 — gate the full-sync bookmark through the SAME shared ledger as
|
||||
// the incremental path (Codex #6: a wedge here on first/forced sync was
|
||||
// previously unreachable by the valve). A full re-import is authoritative for
|
||||
// the whole tree, so any previously-tracked failing path that ISN'T failing
|
||||
// now has been resolved → clear it (resets its auto-skip streak); current
|
||||
// failures still climb their attempts.
|
||||
const fullSourceId = opts.sourceId ?? DEFAULT_SOURCE_ID;
|
||||
const fullFailureSet = new Set(result.failures.map(f => f.path));
|
||||
const fullSucceeded = loadSyncFailures()
|
||||
.filter(e => e.source_id === fullSourceId && isSkippablePath(e.path) && !fullFailureSet.has(e.path))
|
||||
.map(e => e.path);
|
||||
const advanceFull = async (): Promise<void> => {
|
||||
// Persist sync state so the next sync is incremental. Routed through
|
||||
// writeSyncAnchor so --source pins the right sources row.
|
||||
await writeSyncAnchor(engine, opts.sourceId, 'last_commit', headCommit, newestCommitMs(repoPath));
|
||||
await engine.setConfig('sync.last_run', new Date().toISOString());
|
||||
await writeSyncAnchor(engine, opts.sourceId, 'repo_path', repoPath);
|
||||
await writeChunkerVersion(engine, opts.sourceId, String(CHUNKER_VERSION));
|
||||
};
|
||||
|
||||
// Persist sync state so next sync is incremental (C1 fix: was missing).
|
||||
// v0.18.0 Step 5: routed through writeSyncAnchor so --source pins it
|
||||
// to the right sources row rather than the global config.
|
||||
await writeSyncAnchor(engine, opts.sourceId, 'last_commit', headCommit, newestCommitMs(repoPath));
|
||||
await engine.setConfig('sync.last_run', new Date().toISOString());
|
||||
await writeSyncAnchor(engine, opts.sourceId, 'repo_path', repoPath);
|
||||
// v0.20.0 Cathedral II Layer 12: persist chunker version for the gate.
|
||||
await writeChunkerVersion(engine, opts.sourceId, String(CHUNKER_VERSION));
|
||||
const fullGate = await applySyncFailureGate({
|
||||
sourceId: fullSourceId,
|
||||
failedFiles: result.failures,
|
||||
succeededPaths: fullSucceeded,
|
||||
commit: headCommit,
|
||||
skipFailed: opts.skipFailed === true,
|
||||
advance: advanceFull,
|
||||
});
|
||||
|
||||
if (!fullGate.advanced) {
|
||||
const codeBreakdown = formatCodeBreakdown(result.failures);
|
||||
if (fullGate.sentinelBlocked) {
|
||||
serr(`\nFull sync blocked: repository history changed during sync.\n${codeBreakdown}`);
|
||||
} else {
|
||||
const fileFailCount = result.failures.filter(f => isSkippablePath(f.path)).length;
|
||||
serr(
|
||||
`\nFull sync blocked: ${fileFailCount} file(s) failed:\n` +
|
||||
`${codeBreakdown}\n\n` +
|
||||
`Fix the YAML in those files and re-run, or use '--skip-failed'. A file ` +
|
||||
`that keeps failing auto-skips after ${resolveAutoSkipThreshold()} consecutive syncs.`,
|
||||
);
|
||||
}
|
||||
await engine.setConfig('sync.last_run', new Date().toISOString());
|
||||
await writeSyncAnchor(engine, opts.sourceId, 'repo_path', repoPath);
|
||||
return {
|
||||
status: 'blocked_by_failures',
|
||||
fromCommit: null,
|
||||
toCommit: headCommit,
|
||||
added: 0, modified: 0, deleted: 0, renamed: 0,
|
||||
chunksCreated: result.chunksCreated,
|
||||
embedded: 0,
|
||||
pagesAffected: [],
|
||||
failedFiles: result.failures.length,
|
||||
};
|
||||
}
|
||||
if (fullGate.acknowledged > 0) {
|
||||
serr(` Acknowledged ${fullGate.acknowledged} failure(s) and advanced past them.`);
|
||||
}
|
||||
if (fullGate.autoSkipped.length > 0) {
|
||||
serr(
|
||||
`\n Auto-skipped ${fullGate.autoSkipped.length} file(s) that failed >= ` +
|
||||
`${resolveAutoSkipThreshold()} consecutive syncs. These pages are NOT indexed; ` +
|
||||
`'gbrain doctor' will warn until they're fixed.`,
|
||||
);
|
||||
}
|
||||
|
||||
// Full sync doesn't track pagesAffected, so fall back to embed --stale.
|
||||
// v0.37 fix wave (Lane D.3 + CDX2-8): switched to runEmbedCore for the
|
||||
|
||||
@@ -376,14 +376,18 @@ export function assessContentSanity(opts: {
|
||||
// doesn't repeat the lowercase per literal.
|
||||
const bodyHead = body.slice(0, SCAN_HEAD_BYTES);
|
||||
const bodyHeadLower = bodyHead.toLowerCase();
|
||||
const titleLower = opts.title.toLowerCase();
|
||||
// Defensive coercion (issue #1939): this is a pure exported fn; lint.ts and
|
||||
// import-file both pass `parsed.title`, which a malformed YAML date/number
|
||||
// title could make non-string. Never throw on a bad title.
|
||||
const title = String(opts.title ?? '');
|
||||
const titleLower = title.toLowerCase();
|
||||
|
||||
const junk_pattern_matches: string[] = [];
|
||||
for (const p of BUILT_IN_JUNK_PATTERNS) {
|
||||
const scope = p.applies_to ?? 'both';
|
||||
let matched = false;
|
||||
if (scope === 'title' || scope === 'both') {
|
||||
if (p.pattern.test(opts.title)) matched = true;
|
||||
if (p.pattern.test(title)) matched = true;
|
||||
}
|
||||
if (!matched && (scope === 'body' || scope === 'both')) {
|
||||
if (p.pattern.test(bodyHead)) matched = true;
|
||||
|
||||
@@ -112,7 +112,8 @@ function nonRedundancy(page: Page): number {
|
||||
}
|
||||
|
||||
function hasTitle(page: Page): number {
|
||||
return page.title && page.title.trim().length > 0 ? 1 : 0;
|
||||
// Coerce (issue #1939): a malformed YAML date/number title could be non-string.
|
||||
return String(page.title ?? '').trim().length > 0 ? 1 : 0;
|
||||
}
|
||||
|
||||
function hasBody(page: Page): number {
|
||||
|
||||
+22
-3
@@ -49,6 +49,25 @@ export interface ParsedMarkdown {
|
||||
errors?: ParseValidationError[];
|
||||
}
|
||||
|
||||
/**
|
||||
* Coerce a raw YAML frontmatter value into a string.
|
||||
*
|
||||
* js-yaml parses unquoted scalars by type: `title: 2024-06-01` becomes a JS
|
||||
* `Date`, `title: 1458` becomes a `number`. The old `(frontmatter.X as string)`
|
||||
* cast was a compile-time lie — at runtime the value stayed a Date/number, so
|
||||
* any downstream `.toLowerCase()` / `.trim()` threw and (via the importer's
|
||||
* failure gate) could wedge sync indefinitely (issue #1939).
|
||||
*
|
||||
* Dates coerce to their UTC ISO date (`2024-06-01`) — deterministic across
|
||||
* machines and matching the on-disk source token, unlike `String(date)` which
|
||||
* renders a timezone-dependent long form. Everything else uses `String()`.
|
||||
*/
|
||||
export function coerceFrontmatterString(v: unknown): string {
|
||||
if (v == null) return '';
|
||||
if (v instanceof Date) return v.toISOString().slice(0, 10);
|
||||
return String(v);
|
||||
}
|
||||
|
||||
/**
|
||||
* Parse a markdown file with YAML frontmatter into its components.
|
||||
*
|
||||
@@ -105,12 +124,12 @@ export function parseMarkdown(
|
||||
|
||||
const { compiled_truth, timeline } = splitBody(body);
|
||||
|
||||
const type = (frontmatter.type as string) || (
|
||||
const type = coerceFrontmatterString(frontmatter.type) || (
|
||||
opts?.activePack ? inferTypeFromPack(filePath, opts.activePack) : inferType(filePath)
|
||||
);
|
||||
const title = (frontmatter.title as string) || inferTitle(filePath);
|
||||
const title = coerceFrontmatterString(frontmatter.title).trim() || inferTitle(filePath);
|
||||
const tags = extractTags(frontmatter);
|
||||
const slug = (frontmatter.slug as string) || inferSlug(filePath);
|
||||
const slug = coerceFrontmatterString(frontmatter.slug) || inferSlug(filePath);
|
||||
|
||||
const cleanFrontmatter = { ...frontmatter };
|
||||
delete cleanFrontmatter.type;
|
||||
|
||||
@@ -0,0 +1,764 @@
|
||||
// ─────────────────────────────────────────────────────────────────
|
||||
// Sync failure ledger — issue #1939 (was "Bug 9" in src/core/sync.ts)
|
||||
// ─────────────────────────────────────────────────────────────────
|
||||
//
|
||||
// When a sync run catches a per-file parse error (YAML with unquoted
|
||||
// colons, malformed frontmatter, a non-string title, etc.), we record it
|
||||
// here instead of just logging and moving on. Goals:
|
||||
// 1. Gate the sync.last_commit bookmark advance in BOTH sync gates
|
||||
// (incremental + full/runImport) through one shared policy.
|
||||
// 2. Give a visible, per-(source,path) record of what failed, with the
|
||||
// commit hash to re-attempt after fixing the source file.
|
||||
// 3. `gbrain sync --skip-failed` acknowledges a known-bad set.
|
||||
// 4. BOUNDED AUTO-SKIP: a file that fails N consecutive syncs is
|
||||
// auto-skipped so a single poison file can't wedge ALL indexing
|
||||
// forever — without ever silently dropping a FRESH failure, and
|
||||
// never auto-skipping an infra sentinel like `<head>`.
|
||||
//
|
||||
// This module is a LEAF (imports only fs/path/crypto/config) so it can be
|
||||
// re-exported from sync.ts without a circular dependency. The state lives
|
||||
// in `~/.gbrain/sync-failures.jsonl`, one JSON object per line.
|
||||
//
|
||||
// State machine (per (source_id, path)):
|
||||
//
|
||||
// recordFailures (file fails a sync run)
|
||||
// │ attempts++ each consecutive run
|
||||
// ▼
|
||||
// ┌──────┐ clearFailures (file imports OK) ┌─────────┐
|
||||
// │ open │ ───────────────────────────────────────│ removed │
|
||||
// └──────┘ └─────────┘
|
||||
// │ │
|
||||
// --skip-│ │ attempts >= threshold AND no fresh failures
|
||||
// failed │ │ (after bookmark advance)
|
||||
// ▼ ▼
|
||||
// ┌────────────┐ ┌──────────────┐
|
||||
// │acknowledged│ │ auto_skipped │ (still UNRESOLVED → doctor WARN
|
||||
// │ (resolved) │ │ visible │ until the file imports cleanly)
|
||||
// └────────────┘ └──────────────┘
|
||||
//
|
||||
// `acknowledged` = ok (human resolved). `open` + `auto_skipped` = unresolved
|
||||
// (doctor surfaces them). `<head>` and any `<…>` sentinel is recorded but
|
||||
// NEVER auto-skipped/acknowledged-to-advance — a history rewrite must hard-block.
|
||||
|
||||
import {
|
||||
existsSync as _existsSync,
|
||||
readFileSync as _readFileSync,
|
||||
writeFileSync as _writeFileSync,
|
||||
mkdirSync as _mkdirSync,
|
||||
renameSync as _renameSync,
|
||||
openSync as _openSync,
|
||||
closeSync as _closeSync,
|
||||
unlinkSync as _unlinkSync,
|
||||
statSync as _statSync,
|
||||
} from 'fs';
|
||||
import { join as _joinPath } from 'path';
|
||||
import { gbrainPath as _gbrainPath } from './config.ts';
|
||||
import { createHash as _createHash } from 'crypto';
|
||||
|
||||
export const DEFAULT_SOURCE_ID = 'default';
|
||||
/** Reserved sentinel paths (e.g. `<head>`) start with this; never file paths. */
|
||||
export const SENTINEL_PREFIX = '<';
|
||||
export const DEFAULT_AUTOSKIP_AFTER = 3;
|
||||
const LOCK_STALE_MS = 30_000;
|
||||
const LOCK_SPIN_MS = 50;
|
||||
const LOCK_TIMEOUT_MS = 5_000;
|
||||
|
||||
export type SyncFailureState = 'open' | 'acknowledged' | 'auto_skipped';
|
||||
|
||||
export interface SyncFailure {
|
||||
/** Owning source (#1939 Codex #2 — failures must not merge across sources). */
|
||||
source_id: string;
|
||||
path: string;
|
||||
error: string;
|
||||
/** Structured error code extracted from the error message. */
|
||||
code: string;
|
||||
/** Most recent commit this path failed on. */
|
||||
commit: string;
|
||||
line?: number;
|
||||
/** ISO — start of the current unresolved streak. */
|
||||
first_seen: string;
|
||||
/** ISO — last update. */
|
||||
ts: string;
|
||||
/** Consecutive failed sync runs for (source_id, path). */
|
||||
attempts: number;
|
||||
state: SyncFailureState;
|
||||
resolved_at?: string;
|
||||
// Legacy MIRROR fields, still WRITTEN for one release so any pre-#1939
|
||||
// reader of `acknowledged_at` keeps working. Derived from `state`.
|
||||
acknowledged?: boolean;
|
||||
acknowledged_at?: string | null;
|
||||
}
|
||||
|
||||
export interface AcknowledgeResult {
|
||||
count: number;
|
||||
summary: Array<{ code: string; count: number }>;
|
||||
}
|
||||
|
||||
/** A real importable file (not an infra sentinel like `<head>`). */
|
||||
export function isSkippablePath(path: string): boolean {
|
||||
return !path.startsWith(SENTINEL_PREFIX);
|
||||
}
|
||||
|
||||
/**
|
||||
* Resolve the auto-skip threshold from `GBRAIN_SYNC_AUTOSKIP_AFTER`
|
||||
* (default 3). `0` disables the valve entirely (pure fail-closed).
|
||||
*/
|
||||
export function resolveAutoSkipThreshold(): number {
|
||||
const raw = process.env.GBRAIN_SYNC_AUTOSKIP_AFTER;
|
||||
if (raw === undefined || raw === '') return DEFAULT_AUTOSKIP_AFTER;
|
||||
const n = Number(raw);
|
||||
if (!Number.isFinite(n) || n < 0) return DEFAULT_AUTOSKIP_AFTER;
|
||||
return Math.floor(n);
|
||||
}
|
||||
|
||||
/**
|
||||
* Best-effort extraction of a structured error code from a sync failure
|
||||
* message. Order matters: DB-layer errors are checked BEFORE YAML-layer
|
||||
* ones so Postgres `duplicate key` isn't mislabeled as a YAML duplicate-key.
|
||||
*/
|
||||
export function classifyErrorCode(errorMsg: string): string {
|
||||
// SLUG_MISMATCH: thrown by importFromFile() at src/core/import-file.ts.
|
||||
if (/slug.*does not match|SLUG_MISMATCH/i.test(errorMsg)) return 'SLUG_MISMATCH';
|
||||
|
||||
// DB-layer errors come BEFORE the YAML duplicate-key check.
|
||||
if (/duplicate key value violates unique constraint|DB_DUPLICATE_KEY/i.test(errorMsg)) {
|
||||
return 'DB_DUPLICATE_KEY';
|
||||
}
|
||||
if (/canceling statement due to statement timeout|STATEMENT_TIMEOUT/i.test(errorMsg)) {
|
||||
return 'STATEMENT_TIMEOUT';
|
||||
}
|
||||
|
||||
// YAML / frontmatter patterns.
|
||||
if (/YAML parse failed|YAML_PARSE/i.test(errorMsg)) return 'YAML_PARSE';
|
||||
if (/YAMLException|duplicated mapping key|YAML_DUPLICATE_KEY/i.test(errorMsg)) {
|
||||
return 'YAML_DUPLICATE_KEY';
|
||||
}
|
||||
if (/File is empty or whitespace-only|Frontmatter must start with ---|MISSING_OPEN/i.test(errorMsg)) {
|
||||
return 'MISSING_OPEN';
|
||||
}
|
||||
if (/No closing --- delimiter|Heading at line .* found inside frontmatter|MISSING_CLOSE/i.test(errorMsg)) {
|
||||
return 'MISSING_CLOSE';
|
||||
}
|
||||
if (/Frontmatter block is empty|EMPTY_FRONTMATTER/i.test(errorMsg)) return 'EMPTY_FRONTMATTER';
|
||||
if (/Content contains null bytes|NULL_BYTES|null byte/i.test(errorMsg)) return 'NULL_BYTES';
|
||||
if (/Nested double quotes|NESTED_QUOTES/i.test(errorMsg)) return 'NESTED_QUOTES';
|
||||
|
||||
// Generic fallbacks.
|
||||
if (/invalid UTF-?8|INVALID_UTF8/i.test(errorMsg)) return 'INVALID_UTF8';
|
||||
if (/file too large|content too large|FILE_TOO_LARGE/i.test(errorMsg)) return 'FILE_TOO_LARGE';
|
||||
if (/skipping symlink|symlink|SYMLINK_NOT_ALLOWED/i.test(errorMsg)) return 'SYMLINK_NOT_ALLOWED';
|
||||
|
||||
// takes-v2 fence + holder grammar failures.
|
||||
if (/TAKES_TABLE_MALFORMED|TAKES_ROW_NUM_COLLISION|TAKES_FENCE_UNBALANCED/i.test(errorMsg)) {
|
||||
return 'TAKES_TABLE_MALFORMED';
|
||||
}
|
||||
if (/TAKES_HOLDER_INVALID/i.test(errorMsg)) return 'TAKES_HOLDER_INVALID';
|
||||
|
||||
// Embedding error classification.
|
||||
if (/embedding requires [A-Z][A-Z0-9_]+_API_KEY|EMBEDDING_NO_CREDS/i.test(errorMsg)) {
|
||||
return 'EMBEDDING_NO_CREDS';
|
||||
}
|
||||
if (/Anthropic has no embedding model|EMBEDDING_NO_TOUCHPOINT/i.test(errorMsg)) {
|
||||
return 'EMBEDDING_NO_TOUCHPOINT';
|
||||
}
|
||||
if (/\brate.?limit|\b429\b|too many requests|rate_limited|RateLimit/i.test(errorMsg)) {
|
||||
return 'EMBEDDING_RATE_LIMIT';
|
||||
}
|
||||
if (/insufficient_quota|quota exceeded|exceeded.*quota|credit balance is too low|billing|EMBEDDING_QUOTA/i.test(errorMsg)) {
|
||||
return 'EMBEDDING_QUOTA';
|
||||
}
|
||||
if (/maximum context length|max_tokens|context length|input too long|input length exceeds|tokens? exceed|too many tokens|EMBEDDING_OVERSIZE/i.test(errorMsg)) {
|
||||
return 'EMBEDDING_OVERSIZE';
|
||||
}
|
||||
|
||||
// content-sanity reject disposition.
|
||||
if (/PAGE_JUNK_PATTERN/i.test(errorMsg)) return 'PAGE_JUNK_PATTERN';
|
||||
|
||||
return 'UNKNOWN';
|
||||
}
|
||||
|
||||
/** Group failures by error code and return a sorted summary. */
|
||||
export function summarizeFailuresByCode(
|
||||
failures: Array<{ error: string; code?: string }>,
|
||||
): Array<{ code: string; count: number }> {
|
||||
const counts: Record<string, number> = {};
|
||||
for (const f of failures) {
|
||||
const code = f.code ?? classifyErrorCode(f.error);
|
||||
counts[code] = (counts[code] ?? 0) + 1;
|
||||
}
|
||||
return Object.entries(counts)
|
||||
.sort(([, a], [, b]) => b - a)
|
||||
.map(([code, count]) => ({ code, count }));
|
||||
}
|
||||
|
||||
/**
|
||||
* Format a code-grouped summary as a human-readable multi-line string.
|
||||
* Accepts either raw failures (summarized internally) or an already-
|
||||
* summarized `{code, count}[]`. Empty input → empty string.
|
||||
*/
|
||||
export function formatCodeBreakdown(
|
||||
input: Array<{ error: string; code?: string }> | Array<{ code: string; count: number }>,
|
||||
): string {
|
||||
const summary =
|
||||
input.length > 0 && typeof (input[0] as { count?: unknown }).count === 'number'
|
||||
? (input as Array<{ code: string; count: number }>)
|
||||
: summarizeFailuresByCode(input as Array<{ error: string; code?: string }>);
|
||||
return summary.map(s => ` ${s.code}: ${s.count}`).join('\n');
|
||||
}
|
||||
|
||||
function _failuresDir(): string {
|
||||
return _gbrainPath();
|
||||
}
|
||||
|
||||
export function syncFailuresPath(): string {
|
||||
return _joinPath(_failuresDir(), 'sync-failures.jsonl');
|
||||
}
|
||||
|
||||
function _ledgerKey(f: { source_id: string; path: string }): string {
|
||||
// NUL separator can't appear in a source id or path.
|
||||
return `${f.source_id} ${f.path}`;
|
||||
}
|
||||
|
||||
// ─── State mirror ───────────────────────────────────────────────────
|
||||
|
||||
/**
|
||||
* Keep the legacy `acknowledged`/`acknowledged_at` fields consistent with
|
||||
* `state`. `auto_skipped` is intentionally NOT acknowledged (it is still an
|
||||
* unindexed page) so even a pre-#1939 reader counting `!acknowledged_at`
|
||||
* keeps surfacing it.
|
||||
*/
|
||||
function _applyMirror(f: SyncFailure): SyncFailure {
|
||||
if (f.state === 'acknowledged') {
|
||||
f.acknowledged = true;
|
||||
f.acknowledged_at = f.resolved_at ?? f.ts;
|
||||
} else {
|
||||
f.acknowledged = false;
|
||||
f.acknowledged_at = null;
|
||||
}
|
||||
return f;
|
||||
}
|
||||
|
||||
// ─── Load + normalize ────────────────────────────────────────────────
|
||||
|
||||
function _normalizeRow(raw: Record<string, unknown>): SyncFailure {
|
||||
const source_id =
|
||||
typeof raw.source_id === 'string' && raw.source_id ? raw.source_id : DEFAULT_SOURCE_ID;
|
||||
const error = String(raw.error ?? '');
|
||||
const code = typeof raw.code === 'string' && raw.code ? raw.code : classifyErrorCode(error);
|
||||
const ts = typeof raw.ts === 'string' && raw.ts ? raw.ts : new Date(0).toISOString();
|
||||
const first_seen =
|
||||
typeof raw.first_seen === 'string' && raw.first_seen ? raw.first_seen : ts;
|
||||
let state: SyncFailureState;
|
||||
if (raw.state === 'open' || raw.state === 'acknowledged' || raw.state === 'auto_skipped') {
|
||||
state = raw.state;
|
||||
} else {
|
||||
state = raw.acknowledged === true || raw.acknowledged_at ? 'acknowledged' : 'open';
|
||||
}
|
||||
const attempts =
|
||||
typeof raw.attempts === 'number' && Number.isFinite(raw.attempts) && raw.attempts > 0
|
||||
? Math.floor(raw.attempts)
|
||||
: 1;
|
||||
const row: SyncFailure = {
|
||||
source_id,
|
||||
path: String(raw.path ?? ''),
|
||||
error,
|
||||
code,
|
||||
commit: String(raw.commit ?? ''),
|
||||
line: typeof raw.line === 'number' ? raw.line : undefined,
|
||||
first_seen,
|
||||
ts,
|
||||
attempts,
|
||||
state,
|
||||
resolved_at:
|
||||
typeof raw.resolved_at === 'string'
|
||||
? raw.resolved_at
|
||||
: typeof raw.acknowledged_at === 'string'
|
||||
? raw.acknowledged_at
|
||||
: undefined,
|
||||
};
|
||||
return _applyMirror(row);
|
||||
}
|
||||
|
||||
/** Merge legacy duplicate rows for one (source_id, path) into a single row. */
|
||||
function _mergeGroup(group: SyncFailure[]): SyncFailure {
|
||||
const sorted = [...group].sort((a, b) => (a.ts < b.ts ? -1 : a.ts > b.ts ? 1 : 0));
|
||||
const latest = sorted[sorted.length - 1];
|
||||
const first_seen = sorted.reduce(
|
||||
(m, r) => (r.first_seen && r.first_seen < m ? r.first_seen : m),
|
||||
sorted[0].first_seen,
|
||||
);
|
||||
const hasOpen = group.some(r => r.state === 'open');
|
||||
const hasAuto = group.some(r => r.state === 'auto_skipped');
|
||||
const state: SyncFailureState = hasOpen ? 'open' : hasAuto ? 'auto_skipped' : 'acknowledged';
|
||||
// attempts reconstruction: distinct commits is a proxy for distinct runs;
|
||||
// never under-count below the largest recorded attempts.
|
||||
const distinctCommits = new Set(group.map(r => r.commit)).size;
|
||||
const maxAttempts = group.reduce((m, r) => Math.max(m, r.attempts), 0);
|
||||
const attempts = Math.max(distinctCommits, maxAttempts, 1);
|
||||
return _applyMirror({ ...latest, first_seen, state, attempts });
|
||||
}
|
||||
|
||||
/**
|
||||
* Read the ledger, normalizing every row (backfill source_id/state/attempts/
|
||||
* first_seen) and collapsing duplicate (source_id, path) rows. Skips malformed
|
||||
* lines with a warning. Empty array if the file doesn't exist.
|
||||
*/
|
||||
export function loadSyncFailures(): SyncFailure[] {
|
||||
const path = syncFailuresPath();
|
||||
if (!_existsSync(path)) return [];
|
||||
const raw = _readFileSync(path, 'utf-8');
|
||||
const rows: SyncFailure[] = [];
|
||||
for (const line of raw.split('\n')) {
|
||||
const trimmed = line.trim();
|
||||
if (!trimmed) continue;
|
||||
try {
|
||||
rows.push(_normalizeRow(JSON.parse(trimmed) as Record<string, unknown>));
|
||||
} catch {
|
||||
console.warn(`[sync-failures] skipping malformed line: ${trimmed.slice(0, 120)}`);
|
||||
}
|
||||
}
|
||||
const byKey = new Map<string, SyncFailure[]>();
|
||||
for (const r of rows) {
|
||||
const k = _ledgerKey(r);
|
||||
const arr = byKey.get(k);
|
||||
if (arr) arr.push(r);
|
||||
else byKey.set(k, [r]);
|
||||
}
|
||||
const out: SyncFailure[] = [];
|
||||
for (const group of byKey.values()) {
|
||||
out.push(group.length === 1 ? group[0] : _mergeGroup(group));
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
/** Unresolved failures (open + auto_skipped). */
|
||||
export function unacknowledgedSyncFailures(): SyncFailure[] {
|
||||
return loadSyncFailures().filter(e => e.state !== 'acknowledged');
|
||||
}
|
||||
|
||||
// ─── Concurrency: cross-process lock + atomic write ──────────────────
|
||||
|
||||
function _sleepSync(ms: number): void {
|
||||
// Synchronous sleep without busy-spin; works in Node + Bun.
|
||||
Atomics.wait(new Int32Array(new SharedArrayBuffer(4)), 0, 0, ms);
|
||||
}
|
||||
|
||||
function _acquireLock(lockPath: string): boolean {
|
||||
const deadline = Date.now() + LOCK_TIMEOUT_MS;
|
||||
// eslint-disable-next-line no-constant-condition
|
||||
while (true) {
|
||||
try {
|
||||
_closeSync(_openSync(lockPath, 'wx'));
|
||||
return true;
|
||||
} catch (e) {
|
||||
if ((e as NodeJS.ErrnoException)?.code !== 'EEXIST') return false; // best-effort
|
||||
// Age-based stale break (prefer age over PID liveness, per db-lock learning).
|
||||
try {
|
||||
const st = _statSync(lockPath);
|
||||
if (Date.now() - st.mtimeMs > LOCK_STALE_MS) {
|
||||
try { _unlinkSync(lockPath); } catch { /* raced; retry */ }
|
||||
continue;
|
||||
}
|
||||
} catch {
|
||||
continue; // lock vanished underfoot; retry acquire
|
||||
}
|
||||
if (Date.now() >= deadline) return false;
|
||||
_sleepSync(LOCK_SPIN_MS);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Serialize a read-modify-write of the ledger across processes
|
||||
* (`sync --all`, per-source cron). Falls back to best-effort (no lock) on
|
||||
* acquire timeout so a sync is never deadlocked by a wedged lock holder.
|
||||
*/
|
||||
export function withLedgerLock<T>(fn: () => T): T {
|
||||
_mkdirSync(_failuresDir(), { recursive: true });
|
||||
const lockPath = syncFailuresPath() + '.lock';
|
||||
const got = _acquireLock(lockPath);
|
||||
if (!got) {
|
||||
console.warn('[sync-failures] could not acquire ledger lock; proceeding best-effort');
|
||||
}
|
||||
try {
|
||||
return fn();
|
||||
} finally {
|
||||
if (got) {
|
||||
try { _unlinkSync(lockPath); } catch { /* already gone */ }
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
function _writeAll(entries: SyncFailure[]): void {
|
||||
_mkdirSync(_failuresDir(), { recursive: true });
|
||||
const target = syncFailuresPath();
|
||||
const tmp = `${target}.tmp-${process.pid}`;
|
||||
const body = entries.map(e => JSON.stringify(e)).join('\n');
|
||||
_writeFileSync(tmp, entries.length ? body + '\n' : '');
|
||||
_renameSync(tmp, target); // atomic on POSIX
|
||||
}
|
||||
|
||||
// ─── Mutations ───────────────────────────────────────────────────────
|
||||
|
||||
/**
|
||||
* Record this run's failures AND clear succeeded paths in ONE locked
|
||||
* transaction. Returns post-state attempts for each failing path.
|
||||
*
|
||||
* Upsert is keyed by (source_id, path) over OPEN rows: an existing open row
|
||||
* increments `attempts` (consecutive); anything else (no row, or an
|
||||
* acknowledged/auto_skipped row) starts a fresh open row at attempts 1 — a
|
||||
* re-failure after a fix is a new streak (#1939 Codex #4).
|
||||
*/
|
||||
function _recordAndClear(
|
||||
sourceId: string,
|
||||
succeededPaths: string[],
|
||||
failures: Array<{ path: string; error: string; line?: number }>,
|
||||
commit: string,
|
||||
): Map<string, number> {
|
||||
return withLedgerLock(() => {
|
||||
const entries = loadSyncFailures();
|
||||
const byKey = new Map<string, SyncFailure>();
|
||||
for (const e of entries) byKey.set(_ledgerKey(e), e);
|
||||
let mutated = false;
|
||||
|
||||
// Resolve succeeded paths (success fully clears the row — #1939 Codex #4).
|
||||
for (const p of succeededPaths) {
|
||||
if (byKey.delete(_ledgerKey({ source_id: sourceId, path: p }))) mutated = true;
|
||||
}
|
||||
|
||||
const now = new Date().toISOString();
|
||||
for (const f of failures) {
|
||||
const key = _ledgerKey({ source_id: sourceId, path: f.path });
|
||||
const ex = byKey.get(key);
|
||||
const code = classifyErrorCode(f.error);
|
||||
if (ex && ex.state === 'open') {
|
||||
ex.attempts += 1;
|
||||
ex.ts = now;
|
||||
ex.commit = commit;
|
||||
ex.error = f.error;
|
||||
ex.code = code;
|
||||
ex.line = f.line;
|
||||
_applyMirror(ex);
|
||||
} else {
|
||||
byKey.set(
|
||||
key,
|
||||
_applyMirror({
|
||||
source_id: sourceId,
|
||||
path: f.path,
|
||||
error: f.error,
|
||||
code,
|
||||
commit,
|
||||
line: f.line,
|
||||
first_seen: now,
|
||||
ts: now,
|
||||
attempts: 1,
|
||||
state: 'open',
|
||||
}),
|
||||
);
|
||||
}
|
||||
mutated = true;
|
||||
}
|
||||
|
||||
// Skip the write (and avoid creating an empty ledger file) when a clean run
|
||||
// touched nothing — e.g. succeeded paths that had no prior failure rows.
|
||||
if (mutated) _writeAll([...byKey.values()]);
|
||||
|
||||
const attempts = new Map<string, number>();
|
||||
for (const f of failures) {
|
||||
attempts.set(
|
||||
f.path,
|
||||
byKey.get(_ledgerKey({ source_id: sourceId, path: f.path }))?.attempts ?? 1,
|
||||
);
|
||||
}
|
||||
return attempts;
|
||||
});
|
||||
}
|
||||
|
||||
/**
|
||||
* Public single-purpose recorder (no clear). Used by callers outside the
|
||||
* sync gate (e.g. `gbrain import`). Increments attempts like the gate.
|
||||
*/
|
||||
export function recordFailures(
|
||||
sourceId: string,
|
||||
failures: Array<{ path: string; error: string; line?: number }>,
|
||||
commit: string,
|
||||
): void {
|
||||
if (failures.length === 0) return;
|
||||
_recordAndClear(sourceId, [], failures, commit);
|
||||
}
|
||||
|
||||
/** Remove ledger rows for the given (sourceId, paths) — used on success. */
|
||||
export function clearFailures(sourceId: string, paths: string[]): void {
|
||||
if (paths.length === 0) return;
|
||||
withLedgerLock(() => {
|
||||
const entries = loadSyncFailures();
|
||||
const remove = new Set(paths.map(p => _ledgerKey({ source_id: sourceId, path: p })));
|
||||
const kept = entries.filter(e => !remove.has(_ledgerKey(e)));
|
||||
if (kept.length !== entries.length) _writeAll(kept);
|
||||
});
|
||||
}
|
||||
|
||||
/**
|
||||
* Acknowledge OPEN file failures (human `--skip-failed`). Scoped to one
|
||||
* source when `sourceId` is given (never acks another source — #1939 Codex
|
||||
* #2). Sentinels (`<head>`) are NEVER acknowledged this way.
|
||||
*/
|
||||
export function acknowledgeFailures(sourceId?: string): AcknowledgeResult {
|
||||
return withLedgerLock(() => {
|
||||
const entries = loadSyncFailures();
|
||||
const now = new Date().toISOString();
|
||||
let changed = 0;
|
||||
const acked: SyncFailure[] = [];
|
||||
for (const e of entries) {
|
||||
if (e.state !== 'open') continue;
|
||||
if (sourceId !== undefined && e.source_id !== sourceId) continue;
|
||||
if (!isSkippablePath(e.path)) continue;
|
||||
e.state = 'acknowledged';
|
||||
e.resolved_at = now;
|
||||
_applyMirror(e);
|
||||
changed++;
|
||||
acked.push(e);
|
||||
}
|
||||
if (changed > 0) _writeAll(entries);
|
||||
return { count: changed, summary: summarizeFailuresByCode(acked) };
|
||||
});
|
||||
}
|
||||
|
||||
/**
|
||||
* Mark the given chronic file paths `auto_skipped` (valve fired). Only OPEN,
|
||||
* non-sentinel rows transition. Auto-skipped rows stay UNRESOLVED so doctor
|
||||
* keeps warning until the file imports cleanly.
|
||||
*/
|
||||
export function autoSkipFailures(sourceId: string, paths: string[]): AcknowledgeResult {
|
||||
if (paths.length === 0) return { count: 0, summary: [] };
|
||||
return withLedgerLock(() => {
|
||||
const entries = loadSyncFailures();
|
||||
const target = new Set(
|
||||
paths.filter(isSkippablePath).map(p => _ledgerKey({ source_id: sourceId, path: p })),
|
||||
);
|
||||
const now = new Date().toISOString();
|
||||
let changed = 0;
|
||||
const skipped: SyncFailure[] = [];
|
||||
for (const e of entries) {
|
||||
if (!target.has(_ledgerKey(e))) continue;
|
||||
if (e.state !== 'open') continue;
|
||||
e.state = 'auto_skipped';
|
||||
e.resolved_at = now;
|
||||
_applyMirror(e);
|
||||
changed++;
|
||||
skipped.push(e);
|
||||
}
|
||||
if (changed > 0) _writeAll(entries);
|
||||
return { count: changed, summary: summarizeFailuresByCode(skipped) };
|
||||
});
|
||||
}
|
||||
|
||||
// ─── Legacy shims (re-exported from sync.ts for existing callers) ─────
|
||||
|
||||
/** @deprecated use recordFailures(sourceId, …). Defaults to the host source. */
|
||||
export function recordSyncFailures(
|
||||
failures: Array<{ path: string; error: string; line?: number }>,
|
||||
commit: string,
|
||||
): void {
|
||||
recordFailures(DEFAULT_SOURCE_ID, failures, commit);
|
||||
}
|
||||
|
||||
/** @deprecated use acknowledgeFailures(sourceId). Acks ALL sources' open files. */
|
||||
export function acknowledgeSyncFailures(): AcknowledgeResult {
|
||||
return acknowledgeFailures(undefined);
|
||||
}
|
||||
|
||||
// ─── Pure decisions (no side effects — the unit-test surface) ─────────
|
||||
|
||||
export interface GateDecision {
|
||||
action: 'hard_block' | 'block' | 'advance' | 'advance_then_autoskip';
|
||||
autoSkipPaths: string[];
|
||||
}
|
||||
|
||||
/**
|
||||
* Decide what the sync gate should do, given this run's failures and the
|
||||
* current attempt counts. Pure — the caller executes effects in the safe
|
||||
* order (advance THEN ack, so a crash can't mark a file skipped while the
|
||||
* sync stays wedged — #1939 Codex #5).
|
||||
*
|
||||
* sentinels present → hard_block (ALWAYS, even --skip-failed)
|
||||
* no file failures → advance
|
||||
* --skip-failed → advance (ack handled post-advance)
|
||||
* valve disabled (threshold<=0) → block (pure fail-closed) if failures
|
||||
* any fresh (attempts<threshold) → block
|
||||
* all chronic (attempts>=threshold) → advance_then_autoskip
|
||||
*/
|
||||
export function decideGateAction(args: {
|
||||
fileFailures: Array<{ path: string }>;
|
||||
sentinels: Array<{ path: string }>;
|
||||
attemptsByPath: Map<string, number>;
|
||||
threshold: number;
|
||||
skipFailed: boolean;
|
||||
}): GateDecision {
|
||||
if (args.sentinels.length > 0) return { action: 'hard_block', autoSkipPaths: [] };
|
||||
if (args.fileFailures.length === 0) return { action: 'advance', autoSkipPaths: [] };
|
||||
if (args.skipFailed) return { action: 'advance', autoSkipPaths: [] };
|
||||
if (args.threshold <= 0) return { action: 'block', autoSkipPaths: [] };
|
||||
|
||||
const chronic: string[] = [];
|
||||
let fresh = 0;
|
||||
for (const f of args.fileFailures) {
|
||||
const a = args.attemptsByPath.get(f.path) ?? 1;
|
||||
if (a >= args.threshold) chronic.push(f.path);
|
||||
else fresh++;
|
||||
}
|
||||
if (fresh > 0) return { action: 'block', autoSkipPaths: [] };
|
||||
if (chronic.length > 0) return { action: 'advance_then_autoskip', autoSkipPaths: chronic };
|
||||
return { action: 'block', autoSkipPaths: [] };
|
||||
}
|
||||
|
||||
export interface SeverityResult {
|
||||
status: 'ok' | 'warn' | 'fail';
|
||||
unresolved: number;
|
||||
open: number;
|
||||
auto_skipped: number;
|
||||
}
|
||||
|
||||
/**
|
||||
* Decide the `sync_failures` doctor severity from ledger entries. Pure;
|
||||
* `nowMs` is injected for deterministic boundary tests. Both doctor surfaces
|
||||
* (local + remote) call this so they can never drift (#1939 Codex #1).
|
||||
*
|
||||
* unresolved (open|auto_skipped) == 0 → ok
|
||||
* oldest OPEN older than failHours, OR
|
||||
* total unresolved >= 10 → fail
|
||||
* else (incl. auto_skipped-only) → warn (stays visible)
|
||||
* malformed ts → treated as not-old (never crashes doctor)
|
||||
*/
|
||||
export function decideSyncFailureSeverity(args: {
|
||||
entries: SyncFailure[];
|
||||
nowMs: number;
|
||||
failHours: number;
|
||||
}): SeverityResult {
|
||||
const unresolved = args.entries.filter(
|
||||
e => e.state === 'open' || e.state === 'auto_skipped',
|
||||
);
|
||||
const autoSkipped = unresolved.filter(e => e.state === 'auto_skipped').length;
|
||||
const open = unresolved.length - autoSkipped;
|
||||
if (unresolved.length === 0) {
|
||||
return { status: 'ok', unresolved: 0, open: 0, auto_skipped: 0 };
|
||||
}
|
||||
let oldestOpenMs = Infinity;
|
||||
for (const e of unresolved) {
|
||||
if (e.state !== 'open') continue;
|
||||
const ms = Date.parse(e.ts);
|
||||
if (Number.isFinite(ms)) oldestOpenMs = Math.min(oldestOpenMs, ms);
|
||||
}
|
||||
const blockedTooLong =
|
||||
Number.isFinite(oldestOpenMs) && args.nowMs - oldestOpenMs > args.failHours * 3_600_000;
|
||||
// FAIL keys off OPEN (blocking) failures only — many open, or one blocking the
|
||||
// bookmark past the fail cadence. `auto_skipped` rows already advanced the
|
||||
// bookmark (indexing is NOT wedged) so they stay WARN-visible regardless of
|
||||
// count, matching the state-machine contract. (#1939 adversarial finding #3.)
|
||||
const status: 'warn' | 'fail' = open >= 10 || blockedTooLong ? 'fail' : 'warn';
|
||||
return { status, unresolved: unresolved.length, open, auto_skipped: autoSkipped };
|
||||
}
|
||||
|
||||
// ─── Shared gate orchestrator (incremental + full sync) ──────────────
|
||||
|
||||
export interface SyncGateInput {
|
||||
sourceId: string;
|
||||
/** All per-file failures this run, including sentinels like `<head>`. */
|
||||
failedFiles: Array<{ path: string; error: string; line?: number }>;
|
||||
/** File paths that imported successfully this run (clears their rows). */
|
||||
succeededPaths: string[];
|
||||
/** Pin commit the run drained to (stamped on recorded failures). */
|
||||
commit: string;
|
||||
skipFailed: boolean;
|
||||
threshold?: number;
|
||||
/** Advances the bookmark + clears checkpoints. Awaited BEFORE any ack. */
|
||||
advance: () => Promise<void> | void;
|
||||
}
|
||||
|
||||
export interface SyncGateOutcome {
|
||||
advanced: boolean;
|
||||
sentinelBlocked: boolean;
|
||||
fresh: number;
|
||||
chronic: number;
|
||||
autoSkipped: string[];
|
||||
acknowledged: number;
|
||||
}
|
||||
|
||||
/**
|
||||
* The one gate both sync paths share (#1939 Codex #6). Records/clears the
|
||||
* ledger under a single lock, decides, then on advance runs `advance()`
|
||||
* FIRST and only marks auto_skipped/acknowledged afterwards (#1939 Codex #5).
|
||||
*/
|
||||
export async function applySyncFailureGate(input: SyncGateInput): Promise<SyncGateOutcome> {
|
||||
const threshold = input.threshold ?? resolveAutoSkipThreshold();
|
||||
const sentinels = input.failedFiles.filter(f => !isSkippablePath(f.path));
|
||||
const fileFailures = input.failedFiles.filter(f => isSkippablePath(f.path));
|
||||
|
||||
// Fast path: clean run touched no failures and no successes — nothing to
|
||||
// reconcile in the ledger, just advance.
|
||||
if (input.failedFiles.length === 0 && input.succeededPaths.length === 0) {
|
||||
await input.advance();
|
||||
return {
|
||||
advanced: true,
|
||||
sentinelBlocked: false,
|
||||
fresh: 0,
|
||||
chronic: 0,
|
||||
autoSkipped: [],
|
||||
acknowledged: 0,
|
||||
};
|
||||
}
|
||||
|
||||
const attemptsByPath = _recordAndClear(
|
||||
input.sourceId,
|
||||
input.succeededPaths,
|
||||
input.failedFiles,
|
||||
input.commit,
|
||||
);
|
||||
|
||||
const decision = decideGateAction({
|
||||
fileFailures,
|
||||
sentinels,
|
||||
attemptsByPath,
|
||||
threshold,
|
||||
skipFailed: input.skipFailed,
|
||||
});
|
||||
|
||||
let fresh = 0;
|
||||
let chronic = 0;
|
||||
for (const f of fileFailures) {
|
||||
if ((attemptsByPath.get(f.path) ?? 1) >= threshold && threshold > 0) chronic++;
|
||||
else fresh++;
|
||||
}
|
||||
|
||||
if (decision.action === 'hard_block' || decision.action === 'block') {
|
||||
return {
|
||||
advanced: false,
|
||||
sentinelBlocked: decision.action === 'hard_block',
|
||||
fresh,
|
||||
chronic,
|
||||
autoSkipped: [],
|
||||
acknowledged: 0,
|
||||
};
|
||||
}
|
||||
|
||||
// ATOMICITY: advance the bookmark BEFORE marking anything skipped/acked.
|
||||
await input.advance();
|
||||
|
||||
let autoSkipped: string[] = [];
|
||||
let acknowledged = 0;
|
||||
if (input.skipFailed) {
|
||||
acknowledged = acknowledgeFailures(input.sourceId).count;
|
||||
} else if (decision.action === 'advance_then_autoskip') {
|
||||
autoSkipped = decision.autoSkipPaths;
|
||||
autoSkipFailures(input.sourceId, autoSkipped);
|
||||
}
|
||||
|
||||
return {
|
||||
advanced: true,
|
||||
sentinelBlocked: false,
|
||||
fresh,
|
||||
chronic,
|
||||
autoSkipped,
|
||||
acknowledged,
|
||||
};
|
||||
}
|
||||
+37
-277
@@ -468,282 +468,42 @@ export function resolveSlugForPath(filePath: string, repoPrefix?: string): strin
|
||||
}
|
||||
|
||||
// ─────────────────────────────────────────────────────────────────
|
||||
// Sync failure tracking — Bug 9
|
||||
// Sync failure ledger — moved to ./sync-failure-ledger.ts (issue #1939)
|
||||
// ─────────────────────────────────────────────────────────────────
|
||||
//
|
||||
// When a sync run catches a per-file parse error (YAML with unquoted
|
||||
// colons, malformed frontmatter, etc.), we record it here instead of just
|
||||
// logging and moving on. Three goals:
|
||||
// 1. Gate the sync.last_commit bookmark advance in all three sync paths
|
||||
// (incremental, full/runImport, `gbrain import` git continuity).
|
||||
// 2. Give users a visible record of what failed, with the commit hash
|
||||
// they can use to re-attempt after fixing the source file.
|
||||
// 3. Let `gbrain sync --skip-failed` acknowledge a known-bad set so
|
||||
// repos with many broken files aren't permanently stuck.
|
||||
|
||||
import { existsSync as _existsSync, readFileSync as _readFileSync, appendFileSync as _appendFileSync, mkdirSync as _mkdirSync } from 'fs';
|
||||
import { join as _joinPath } from 'path';
|
||||
import { gbrainPath as _gbrainPath } from './config.ts';
|
||||
import { createHash as _createHash } from 'crypto';
|
||||
|
||||
export interface SyncFailure {
|
||||
path: string;
|
||||
error: string;
|
||||
/** Structured error code extracted from the error message. */
|
||||
code?: string;
|
||||
commit: string;
|
||||
line?: number;
|
||||
ts: string;
|
||||
acknowledged?: boolean;
|
||||
acknowledged_at?: string;
|
||||
}
|
||||
|
||||
/**
|
||||
* Best-effort extraction of a structured error code from a sync failure
|
||||
* message. Matches known ParseValidationCode patterns (SLUG_MISMATCH,
|
||||
* YAML_PARSE, etc.) and common DB / timeout errors. Returns 'UNKNOWN'
|
||||
* when no pattern matches.
|
||||
*
|
||||
* Order matters: DB-layer errors are checked BEFORE YAML-layer ones so
|
||||
* Postgres `duplicate key value violates unique constraint` doesn't get
|
||||
* mislabeled as a YAML duplicate-key. Frontmatter patterns key off the
|
||||
* canonical messages emitted by `collectValidationErrors()` in markdown.ts.
|
||||
*/
|
||||
export function classifyErrorCode(errorMsg: string): string {
|
||||
// SLUG_MISMATCH: thrown by importFromFile() at src/core/import-file.ts:374.
|
||||
if (/slug.*does not match|SLUG_MISMATCH/i.test(errorMsg)) return 'SLUG_MISMATCH';
|
||||
|
||||
// DB-layer errors come BEFORE the YAML duplicate-key check. Postgres unique-
|
||||
// constraint violations contain "duplicate key" but are not a YAML problem.
|
||||
if (/duplicate key value violates unique constraint|DB_DUPLICATE_KEY/i.test(errorMsg)) {
|
||||
return 'DB_DUPLICATE_KEY';
|
||||
}
|
||||
if (/canceling statement due to statement timeout|STATEMENT_TIMEOUT/i.test(errorMsg)) {
|
||||
return 'STATEMENT_TIMEOUT';
|
||||
}
|
||||
|
||||
// YAML / frontmatter patterns. These match either the canonical message
|
||||
// strings in src/core/markdown.ts (collectValidationErrors) or the literal
|
||||
// ParseValidationCode token, so they fire whether the caller stores the
|
||||
// message or just the code.
|
||||
if (/YAML parse failed|YAML_PARSE/i.test(errorMsg)) return 'YAML_PARSE';
|
||||
if (/YAMLException|duplicated mapping key|YAML_DUPLICATE_KEY/i.test(errorMsg)) {
|
||||
return 'YAML_DUPLICATE_KEY';
|
||||
}
|
||||
if (/File is empty or whitespace-only|Frontmatter must start with ---|MISSING_OPEN/i.test(errorMsg)) {
|
||||
return 'MISSING_OPEN';
|
||||
}
|
||||
if (/No closing --- delimiter|Heading at line .* found inside frontmatter|MISSING_CLOSE/i.test(errorMsg)) {
|
||||
return 'MISSING_CLOSE';
|
||||
}
|
||||
if (/Frontmatter block is empty|EMPTY_FRONTMATTER/i.test(errorMsg)) return 'EMPTY_FRONTMATTER';
|
||||
if (/Content contains null bytes|NULL_BYTES|null byte/i.test(errorMsg)) return 'NULL_BYTES';
|
||||
if (/Nested double quotes|NESTED_QUOTES/i.test(errorMsg)) return 'NESTED_QUOTES';
|
||||
|
||||
// Generic fallbacks.
|
||||
if (/invalid UTF-?8|INVALID_UTF8/i.test(errorMsg)) return 'INVALID_UTF8';
|
||||
|
||||
// v0.22.12 additions: covers the four real production sites in src/core/import-file.ts
|
||||
// (lines 199, 347, 352, 401) that previously bucketed to UNKNOWN.
|
||||
if (/file too large|content too large|FILE_TOO_LARGE/i.test(errorMsg)) return 'FILE_TOO_LARGE';
|
||||
if (/skipping symlink|symlink|SYMLINK_NOT_ALLOWED/i.test(errorMsg)) return 'SYMLINK_NOT_ALLOWED';
|
||||
|
||||
// v0.32 takes-v2 additions: malformed fence rows + holder-grammar failures.
|
||||
// TAKES_TABLE_MALFORMED and TAKES_ROW_NUM_COLLISION are produced by
|
||||
// parseTakesFence (src/core/takes-fence.ts); TAKES_HOLDER_INVALID lands
|
||||
// in v0.32 (EXP-4) when a holder doesn't match the world|brain|people/...|
|
||||
// companies/... grammar. Wired into sync-failures.jsonl by the v0_28_0
|
||||
// migration's phaseBBackfill (one-time backfill emission).
|
||||
if (/TAKES_TABLE_MALFORMED|TAKES_ROW_NUM_COLLISION|TAKES_FENCE_UNBALANCED/i.test(errorMsg)) {
|
||||
return 'TAKES_TABLE_MALFORMED';
|
||||
}
|
||||
if (/TAKES_HOLDER_INVALID/i.test(errorMsg)) return 'TAKES_HOLDER_INVALID';
|
||||
|
||||
// v0.41.6.0 D2: embedding error classification. Per-recipe verbatim shapes:
|
||||
// native-openai → "OpenAI embedding requires OPENAI_API_KEY."
|
||||
// native-google → "Google embedding requires GOOGLE_GENERATIVE_AI_API_KEY."
|
||||
// openai-compat → "${recipe.name} embedding requires ${REQUIRED_ENV}."
|
||||
// (Voyage AI / ZeroEntropy / DeepSeek / Together AI /
|
||||
// DashScope / MiniMax / Zhipu AI all use this shape via
|
||||
// defaultResolveAuth at src/core/ai/gateway.ts:250)
|
||||
// EMBEDDING_NO_CREDS catches the missing-env case for every provider. The
|
||||
// anthropic-no-touchpoint case ("Anthropic has no embedding model") is a
|
||||
// misconfig, not a creds issue — bucketed separately so users don't get
|
||||
// pointed at setting a key for a provider that doesn't offer embeddings.
|
||||
if (/embedding requires [A-Z][A-Z0-9_]+_API_KEY|EMBEDDING_NO_CREDS/i.test(errorMsg)) {
|
||||
return 'EMBEDDING_NO_CREDS';
|
||||
}
|
||||
if (/Anthropic has no embedding model|EMBEDDING_NO_TOUCHPOINT/i.test(errorMsg)) {
|
||||
return 'EMBEDDING_NO_TOUCHPOINT';
|
||||
}
|
||||
// 429 status + textual rate-limit signals. AI SDK normalizes provider 429s
|
||||
// into messages containing "rate limit" / "rate_limited" / "429".
|
||||
if (/\brate.?limit|\b429\b|too many requests|rate_limited|RateLimit/i.test(errorMsg)) {
|
||||
return 'EMBEDDING_RATE_LIMIT';
|
||||
}
|
||||
// OpenAI: insufficient_quota / "exceeded your current quota". Anthropic:
|
||||
// "credit balance is too low". Catch-all token: EMBEDDING_QUOTA.
|
||||
if (/insufficient_quota|quota exceeded|exceeded.*quota|credit balance is too low|billing|EMBEDDING_QUOTA/i.test(errorMsg)) {
|
||||
return 'EMBEDDING_QUOTA';
|
||||
}
|
||||
// OpenAI: "maximum context length" / "too many tokens in request". Voyage:
|
||||
// "input length exceeds". General: "max_tokens" / "context length".
|
||||
if (/maximum context length|max_tokens|context length|input too long|input length exceeds|tokens? exceed|too many tokens|EMBEDDING_OVERSIZE/i.test(errorMsg)) {
|
||||
return 'EMBEDDING_OVERSIZE';
|
||||
}
|
||||
|
||||
// v0.41/v0.42 content-sanity gate. The ONLY throw path is junk under the
|
||||
// `reject` disposition (the v0.42 default is `quarantine`, which lands the
|
||||
// page hidden and never enters this classifier). The thrown
|
||||
// ContentSanityBlockError embeds `PAGE_JUNK_PATTERN:` (see
|
||||
// src/core/content-sanity.ts PAGE_JUNK_PATTERN_CODE). Soft-block (oversize)
|
||||
// and flag (markup-heavy) also never throw — the page lands. So a
|
||||
// PAGE_JUNK_PATTERN row in sync-failures.jsonl on a v0.42 brain means the
|
||||
// operator opted into reject mode.
|
||||
if (/PAGE_JUNK_PATTERN/i.test(errorMsg)) return 'PAGE_JUNK_PATTERN';
|
||||
|
||||
return 'UNKNOWN';
|
||||
}
|
||||
|
||||
/** Group failures by error code and return a sorted summary. */
|
||||
export function summarizeFailuresByCode(
|
||||
failures: Array<{ error: string; code?: string }>,
|
||||
): Array<{ code: string; count: number }> {
|
||||
const counts: Record<string, number> = {};
|
||||
for (const f of failures) {
|
||||
const code = f.code ?? classifyErrorCode(f.error);
|
||||
counts[code] = (counts[code] ?? 0) + 1;
|
||||
}
|
||||
return Object.entries(counts)
|
||||
.sort(([, a], [, b]) => b - a)
|
||||
.map(([code, count]) => ({ code, count }));
|
||||
}
|
||||
|
||||
/**
|
||||
* Format a code-grouped summary as a human-readable multi-line string for
|
||||
* stderr / doctor output. Accepts either raw failures (which are summarized
|
||||
* internally) or an already-summarized `{code, count}[]` shape (the return
|
||||
* value of `summarizeFailuresByCode` or `AcknowledgeResult.summary`).
|
||||
* Returns an empty string when the input is empty.
|
||||
*/
|
||||
export function formatCodeBreakdown(
|
||||
input: Array<{ error: string; code?: string }> | Array<{ code: string; count: number }>,
|
||||
): string {
|
||||
// Distinguish by shape: summary entries have a numeric `count`. Empty array
|
||||
// returns '' from either branch — both paths produce a 0-length join.
|
||||
const summary =
|
||||
input.length > 0 && typeof (input[0] as { count?: unknown }).count === 'number'
|
||||
? (input as Array<{ code: string; count: number }>)
|
||||
: summarizeFailuresByCode(input as Array<{ error: string; code?: string }>);
|
||||
return summary.map(s => ` ${s.code}: ${s.count}`).join('\n');
|
||||
}
|
||||
|
||||
function _failuresDir(): string {
|
||||
return _gbrainPath();
|
||||
}
|
||||
|
||||
export function syncFailuresPath(): string {
|
||||
return _joinPath(_failuresDir(), 'sync-failures.jsonl');
|
||||
}
|
||||
|
||||
function _hashError(msg: string): string {
|
||||
return _createHash('sha256').update(msg).digest('hex').slice(0, 12);
|
||||
}
|
||||
|
||||
function _dedupKey(f: { path: string; commit: string; error: string }): string {
|
||||
return `${f.path}|${f.commit}|${_hashError(f.error)}`;
|
||||
}
|
||||
|
||||
/**
|
||||
* Read the failures JSONL, skipping malformed lines with a warning to stderr.
|
||||
* Returns empty array if the file doesn't exist.
|
||||
*/
|
||||
export function loadSyncFailures(): SyncFailure[] {
|
||||
const path = syncFailuresPath();
|
||||
if (!_existsSync(path)) return [];
|
||||
const raw = _readFileSync(path, 'utf-8');
|
||||
const out: SyncFailure[] = [];
|
||||
for (const line of raw.split('\n')) {
|
||||
const trimmed = line.trim();
|
||||
if (!trimmed) continue;
|
||||
try {
|
||||
out.push(JSON.parse(trimmed) as SyncFailure);
|
||||
} catch {
|
||||
console.warn(`[sync-failures] skipping malformed line: ${trimmed.slice(0, 120)}`);
|
||||
}
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
/**
|
||||
* Append failure entries to the JSONL. Dedups by (path, commit, error-hash) —
|
||||
* the same file failing with the same error on the same commit writes ONCE
|
||||
* to the log, not once per sync run.
|
||||
*/
|
||||
export function recordSyncFailures(
|
||||
failures: Array<{ path: string; error: string; line?: number }>,
|
||||
commit: string,
|
||||
): void {
|
||||
if (failures.length === 0) return;
|
||||
const existing = loadSyncFailures();
|
||||
const seen = new Set(existing.map(f => _dedupKey(f)));
|
||||
|
||||
_mkdirSync(_failuresDir(), { recursive: true });
|
||||
const now = new Date().toISOString();
|
||||
for (const f of failures) {
|
||||
const entry: SyncFailure = {
|
||||
path: f.path,
|
||||
error: f.error,
|
||||
code: classifyErrorCode(f.error),
|
||||
commit,
|
||||
line: f.line,
|
||||
ts: now,
|
||||
};
|
||||
if (seen.has(_dedupKey(entry))) continue;
|
||||
_appendFileSync(syncFailuresPath(), JSON.stringify(entry) + '\n');
|
||||
seen.add(_dedupKey(entry));
|
||||
}
|
||||
}
|
||||
|
||||
export interface AcknowledgeResult {
|
||||
count: number;
|
||||
summary: Array<{ code: string; count: number }>;
|
||||
}
|
||||
|
||||
/**
|
||||
* Mark all unacknowledged failures as acknowledged. Used by
|
||||
* `gbrain sync --skip-failed`. Returns count and a structured summary
|
||||
* grouped by error code so the operator can see *why* files were skipped.
|
||||
*
|
||||
* We do not delete — acknowledged entries stay as historical record so
|
||||
* doctor can still show them under a "previously skipped" bucket.
|
||||
*/
|
||||
export function acknowledgeSyncFailures(): AcknowledgeResult {
|
||||
const entries = loadSyncFailures();
|
||||
if (entries.length === 0) return { count: 0, summary: [] };
|
||||
const now = new Date().toISOString();
|
||||
let changed = 0;
|
||||
const newlyAcked: SyncFailure[] = [];
|
||||
const updated = entries.map(e => {
|
||||
if (e.acknowledged) return e;
|
||||
changed++;
|
||||
// Backfill code for entries that predate the code field.
|
||||
const code = e.code ?? classifyErrorCode(e.error);
|
||||
const acked = { ...e, code, acknowledged: true, acknowledged_at: now };
|
||||
newlyAcked.push(acked);
|
||||
return acked;
|
||||
});
|
||||
if (changed === 0) return { count: 0, summary: [] };
|
||||
_mkdirSync(_failuresDir(), { recursive: true });
|
||||
const fd = require('fs').writeFileSync;
|
||||
fd(syncFailuresPath(), updated.map(e => JSON.stringify(e)).join('\n') + '\n');
|
||||
return {
|
||||
count: changed,
|
||||
summary: summarizeFailuresByCode(newlyAcked),
|
||||
};
|
||||
}
|
||||
|
||||
/** Return only unacknowledged failures. */
|
||||
export function unacknowledgedSyncFailures(): SyncFailure[] {
|
||||
return loadSyncFailures().filter(f => !f.acknowledged);
|
||||
}
|
||||
// The failure store + bounded auto-skip valve now live in a leaf module so
|
||||
// they can be unit-tested in isolation and shared by both sync gates without
|
||||
// a circular import. Re-exported here so existing callers that
|
||||
// `await import('../core/sync.ts')` for these symbols keep working.
|
||||
export {
|
||||
classifyErrorCode,
|
||||
summarizeFailuresByCode,
|
||||
formatCodeBreakdown,
|
||||
syncFailuresPath,
|
||||
loadSyncFailures,
|
||||
unacknowledgedSyncFailures,
|
||||
recordSyncFailures,
|
||||
acknowledgeSyncFailures,
|
||||
recordFailures,
|
||||
clearFailures,
|
||||
acknowledgeFailures,
|
||||
autoSkipFailures,
|
||||
withLedgerLock,
|
||||
resolveAutoSkipThreshold,
|
||||
isSkippablePath,
|
||||
decideGateAction,
|
||||
decideSyncFailureSeverity,
|
||||
applySyncFailureGate,
|
||||
DEFAULT_SOURCE_ID,
|
||||
SENTINEL_PREFIX,
|
||||
DEFAULT_AUTOSKIP_AFTER,
|
||||
} from './sync-failure-ledger.ts';
|
||||
export type {
|
||||
SyncFailure,
|
||||
SyncFailureState,
|
||||
AcknowledgeResult,
|
||||
GateDecision,
|
||||
SeverityResult,
|
||||
SyncGateInput,
|
||||
SyncGateOutcome,
|
||||
} from './sync-failure-ledger.ts';
|
||||
|
||||
@@ -640,3 +640,28 @@ describe('assessContentSanity — confidence split (Q1=A)', () => {
|
||||
expect(r.shouldFlag).toBe(false);
|
||||
});
|
||||
});
|
||||
|
||||
// issue #1939 — assessContentSanity is a pure exported fn; lint.ts and
|
||||
// import-file both pass `parsed.title`, which a malformed YAML date/number title
|
||||
// could make non-string. It must coerce defensively and never throw.
|
||||
describe('issue #1939 — non-string title defensive coercion', () => {
|
||||
const base = { compiled_truth: 'some prose body here', timeline: '' };
|
||||
|
||||
test('Date title does not throw', () => {
|
||||
expect(() =>
|
||||
assessContentSanity({ ...base, title: new Date('2024-06-01') as unknown as string }),
|
||||
).not.toThrow();
|
||||
});
|
||||
|
||||
test('number title does not throw', () => {
|
||||
expect(() =>
|
||||
assessContentSanity({ ...base, title: 1458 as unknown as string }),
|
||||
).not.toThrow();
|
||||
});
|
||||
|
||||
test('null/undefined title does not throw and yields a normal result', () => {
|
||||
const r = assessContentSanity({ ...base, title: undefined as unknown as string });
|
||||
expect(r).toBeDefined();
|
||||
expect(typeof r.shouldQuarantine).toBe('boolean');
|
||||
});
|
||||
});
|
||||
|
||||
@@ -553,4 +553,105 @@ describeE2E('E2E: sync --skip-failed structured summary loop (v0.22.12, issue #5
|
||||
const finalSummary = summarizeFailuresByCode(failures);
|
||||
expect(finalSummary).toEqual([{ code: 'SLUG_MISMATCH', count: 2 }]);
|
||||
});
|
||||
|
||||
// issue #1939 CRITICAL REGRESSION: a page whose YAML title parses to a Date
|
||||
// (or number) must import cleanly — pre-fix it threw in assessContentSanity
|
||||
// and wedged the bookmark. This mirrors the apple-notes repro
|
||||
// (sources/apple-notes/.../2024-06-01 8189165238.md).
|
||||
test('date/number-titled page imports cleanly; bookmark advances; get returns it', async () => {
|
||||
const { performSync } = await import('../../src/commands/sync.ts');
|
||||
const { loadSyncFailures } = await import('../../src/core/sync.ts');
|
||||
const engine = getEngine();
|
||||
|
||||
const beforeCommit = await engine.getConfig('sync.last_commit');
|
||||
// Bare-date title (→ Date) and bare-number title (→ number).
|
||||
writeFileSync(join(repoPath, 'people/datey.md'), [
|
||||
'---', 'type: note', 'title: 2024-06-01', '---', '', 'Apple note body.',
|
||||
].join('\n'));
|
||||
writeFileSync(join(repoPath, 'people/numbery.md'), [
|
||||
'---', 'type: note', 'title: 1458', '---', '', 'Another note.',
|
||||
].join('\n'));
|
||||
execSync('git add -A && git commit -m "add date/number titled notes"', { cwd: repoPath, stdio: 'pipe' });
|
||||
|
||||
const result = await performSync(engine, { repoPath, noPull: true, noEmbed: true });
|
||||
expect(result.status).not.toBe('blocked_by_failures');
|
||||
|
||||
// Neither file landed as a failure.
|
||||
const fails = loadSyncFailures().filter(f => f.path.includes('datey') || f.path.includes('numbery'));
|
||||
expect(fails.length).toBe(0);
|
||||
|
||||
// Bookmark advanced past the broken-but-now-fixed commit.
|
||||
const afterCommit = await engine.getConfig('sync.last_commit');
|
||||
expect(afterCommit).not.toBe(beforeCommit);
|
||||
|
||||
// The pages are retrievable, with deterministic coerced titles.
|
||||
const datey = await engine.getPage('people/datey');
|
||||
expect(datey).not.toBeNull();
|
||||
expect(datey!.title).toBe('2024-06-01');
|
||||
const numbery = await engine.getPage('people/numbery');
|
||||
expect(numbery).not.toBeNull();
|
||||
expect(numbery!.title).toBe('1458');
|
||||
});
|
||||
|
||||
// issue #1939 valve: a genuinely un-importable file blocks for (threshold-1)
|
||||
// syncs, then auto-skips on the Nth so it can't wedge indexing forever.
|
||||
test('bounded auto-skip: poison file blocks then auto-skips, advancing the bookmark', async () => {
|
||||
const { performSync } = await import('../../src/commands/sync.ts');
|
||||
const { loadSyncFailures } = await import('../../src/core/sync.ts');
|
||||
const engine = getEngine();
|
||||
const prevThreshold = process.env.GBRAIN_SYNC_AUTOSKIP_AFTER;
|
||||
process.env.GBRAIN_SYNC_AUTOSKIP_AFTER = '2';
|
||||
try {
|
||||
const beforeCommit = await engine.getConfig('sync.last_commit');
|
||||
// slug-mismatch = a reliable per-file import failure.
|
||||
writeFileSync(join(repoPath, 'people/poison.md'), [
|
||||
'---', 'type: person', 'title: Poison', 'slug: not-the-path-slug', '---', '', 'Body.',
|
||||
].join('\n'));
|
||||
execSync('git add -A && git commit -m "add poison file"', { cwd: repoPath, stdio: 'pipe' });
|
||||
|
||||
// Attempt 1: blocks (attempts=1 < 2).
|
||||
let result = await performSync(engine, { repoPath, noPull: true, noEmbed: true });
|
||||
expect(result.status).toBe('blocked_by_failures');
|
||||
expect(await engine.getConfig('sync.last_commit')).toBe(beforeCommit);
|
||||
let poison = loadSyncFailures().find(f => f.path.includes('poison'))!;
|
||||
expect(poison.state).toBe('open');
|
||||
expect(poison.attempts).toBe(1);
|
||||
|
||||
// Attempt 2: attempts hits threshold, fresh==0 → auto-skip + advance.
|
||||
result = await performSync(engine, { repoPath, noPull: true, noEmbed: true });
|
||||
expect(result.status).toBe('synced');
|
||||
expect(await engine.getConfig('sync.last_commit')).not.toBe(beforeCommit);
|
||||
poison = loadSyncFailures().find(f => f.path.includes('poison'))!;
|
||||
expect(poison.state).toBe('auto_skipped');
|
||||
} finally {
|
||||
if (prevThreshold === undefined) delete process.env.GBRAIN_SYNC_AUTOSKIP_AFTER;
|
||||
else process.env.GBRAIN_SYNC_AUTOSKIP_AFTER = prevThreshold;
|
||||
}
|
||||
});
|
||||
|
||||
// issue #1939 adversarial finding #1: a parse-failed file that is later DELETED
|
||||
// from the repo must not leave a permanent open ledger row (which would age
|
||||
// doctor to FAIL forever). The incremental gate treats removed paths as resolved.
|
||||
test('deleting a failed file clears its ledger row (self-heal, no stuck FAIL)', async () => {
|
||||
const { performSync } = await import('../../src/commands/sync.ts');
|
||||
const { loadSyncFailures } = await import('../../src/core/sync.ts');
|
||||
const engine = getEngine();
|
||||
|
||||
writeFileSync(join(repoPath, 'people/gonepoison.md'), [
|
||||
'---', 'type: person', 'title: Gone', 'slug: wrong-derived-slug', '---', '', 'Body.',
|
||||
].join('\n'));
|
||||
execSync('git add -A && git commit -m "add a file that fails to parse"', { cwd: repoPath, stdio: 'pipe' });
|
||||
|
||||
// Sync blocks; an open ledger row exists for the bad file.
|
||||
let result = await performSync(engine, { repoPath, noPull: true, noEmbed: true });
|
||||
expect(result.status).toBe('blocked_by_failures');
|
||||
expect(loadSyncFailures().some(f => f.path.includes('gonepoison'))).toBe(true);
|
||||
|
||||
// Delete the file and sync. The removed path is treated as resolved, so the
|
||||
// ledger row is cleared and the bookmark advances.
|
||||
execSync('git rm people/gonepoison.md && git commit -m "delete the bad file"', { cwd: repoPath, stdio: 'pipe' });
|
||||
result = await performSync(engine, { repoPath, noPull: true, noEmbed: true });
|
||||
expect(result.status).not.toBe('blocked_by_failures');
|
||||
expect(loadSyncFailures().some(f => f.path.includes('gonepoison'))).toBe(false);
|
||||
});
|
||||
});
|
||||
|
||||
@@ -40,4 +40,4 @@
|
||||
"pr_gate_pass": false,
|
||||
"note": "controller does NOT meet PR gate; defaults stay OFF"
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -302,3 +302,44 @@ Some content.`;
|
||||
expect(parseMarkdown('', 'projects/blog/writing/essay.md').type).toBe('writing');
|
||||
});
|
||||
});
|
||||
|
||||
// issue #1939 — js-yaml parses `title: 2024-06-01` as a Date and `title: 1458`
|
||||
// as a number. The old `(frontmatter.title as string)` cast was a compile-time
|
||||
// lie; at runtime downstream `.toLowerCase()` threw and wedged sync. Coercion
|
||||
// must be non-throwing AND deterministic (UTC ISO for dates, no timezone drift).
|
||||
describe('issue #1939 — non-string frontmatter coercion', () => {
|
||||
test('date title coerces to its UTC ISO date string', () => {
|
||||
const parsed = parseMarkdown('---\ntitle: 2024-06-01\n---\nbody\n', 'apple-notes/x.md');
|
||||
expect(parsed.title).toBe('2024-06-01');
|
||||
expect(typeof parsed.title).toBe('string');
|
||||
});
|
||||
|
||||
test('number title coerces to its string form', () => {
|
||||
const parsed = parseMarkdown('---\ntitle: 1458\n---\nbody\n', 'apple-notes/x.md');
|
||||
expect(parsed.title).toBe('1458');
|
||||
});
|
||||
|
||||
test('date title is timezone-independent (UTC) — repro file shape', () => {
|
||||
// sources/apple-notes/YC/Talks YC/2023-04-25 1458.md style page.
|
||||
const parsed = parseMarkdown('---\ntitle: 2023-04-25\n---\nnotes\n', 'apple-notes/2023-04-25 1458.md');
|
||||
expect(parsed.title).toBe('2023-04-25'); // never "Mon Apr 24 2023 ...GMT-0700"
|
||||
});
|
||||
|
||||
test('date/number slug + type coerce without throwing', () => {
|
||||
const parsed = parseMarkdown('---\nslug: 2024-06-01\ntype: 2024\n---\nbody\n', 'x.md');
|
||||
expect(typeof parsed.slug).toBe('string');
|
||||
expect(parsed.slug).toBe('2024-06-01');
|
||||
expect(typeof parsed.type).toBe('string');
|
||||
});
|
||||
|
||||
test('missing/empty title falls back to inferred title (no throw)', () => {
|
||||
const parsed = parseMarkdown('---\ntype: note\n---\nbody\n', 'people/alice-example.md');
|
||||
expect(typeof parsed.title).toBe('string');
|
||||
expect(parsed.title.length).toBeGreaterThan(0);
|
||||
});
|
||||
|
||||
test('string title still passes through unchanged', () => {
|
||||
const parsed = parseMarkdown('---\ntitle: A Normal Title\n---\nbody\n', 'x.md');
|
||||
expect(parsed.title).toBe('A Normal Title');
|
||||
});
|
||||
});
|
||||
|
||||
@@ -0,0 +1,377 @@
|
||||
/**
|
||||
* issue #1939 — sync failure ledger + bounded auto-skip valve.
|
||||
*
|
||||
* Covers the correctness gates the /codex outside-voice review identified:
|
||||
* #1 auto-skipped entries stay UNRESOLVED (doctor WARN), not hidden
|
||||
* #2 (source_id, path) keying — failures never merge across sources
|
||||
* #3 `<head>` sentinel never auto-skips; always hard-blocks
|
||||
* #4 success clears a path so `attempts` is truly consecutive
|
||||
* #5 advance-before-ack atomicity (a throwing advance marks nothing)
|
||||
* #7 legacy rows normalize + duplicates collapse deterministically
|
||||
* #8 cross-process lock + atomic write (stale-lock break, no partial file)
|
||||
*/
|
||||
|
||||
import { describe, test, expect, beforeEach, afterEach } from 'bun:test';
|
||||
import { mkdtempSync, rmSync, readFileSync, writeFileSync, mkdirSync, existsSync, utimesSync, openSync, closeSync } from 'fs';
|
||||
import { join, dirname } from 'path';
|
||||
import { tmpdir } from 'os';
|
||||
|
||||
let tmpHome: string;
|
||||
const originalHome = process.env.HOME;
|
||||
const originalThreshold = process.env.GBRAIN_SYNC_AUTOSKIP_AFTER;
|
||||
|
||||
beforeEach(async () => {
|
||||
tmpHome = mkdtempSync(join(tmpdir(), 'gbrain-ledger-'));
|
||||
process.env.HOME = tmpHome;
|
||||
delete process.env.GBRAIN_SYNC_AUTOSKIP_AFTER;
|
||||
const { syncFailuresPath } = await import('../src/core/sync-failure-ledger.ts');
|
||||
try { rmSync(syncFailuresPath(), { force: true }); } catch { /* none */ }
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
if (originalHome) process.env.HOME = originalHome; else delete process.env.HOME;
|
||||
if (originalThreshold === undefined) delete process.env.GBRAIN_SYNC_AUTOSKIP_AFTER;
|
||||
else process.env.GBRAIN_SYNC_AUTOSKIP_AFTER = originalThreshold;
|
||||
try { rmSync(tmpHome, { recursive: true, force: true }); } catch { /* ignore */ }
|
||||
});
|
||||
|
||||
async function L() {
|
||||
return import('../src/core/sync-failure-ledger.ts');
|
||||
}
|
||||
|
||||
describe('#2 multi-source keying', () => {
|
||||
test('same relative path in two sources keeps independent counters', async () => {
|
||||
const { recordFailures, loadSyncFailures } = await L();
|
||||
recordFailures('alpha', [{ path: 'people/x.md', error: 'YAML parse failed' }], 'c1');
|
||||
recordFailures('alpha', [{ path: 'people/x.md', error: 'YAML parse failed' }], 'c2');
|
||||
recordFailures('beta', [{ path: 'people/x.md', error: 'YAML parse failed' }], 'c1');
|
||||
|
||||
const rows = loadSyncFailures();
|
||||
expect(rows.length).toBe(2);
|
||||
const alpha = rows.find(r => r.source_id === 'alpha')!;
|
||||
const beta = rows.find(r => r.source_id === 'beta')!;
|
||||
expect(alpha.attempts).toBe(2);
|
||||
expect(beta.attempts).toBe(1);
|
||||
});
|
||||
|
||||
test('acknowledgeFailures(sourceId) only acks that source', async () => {
|
||||
const { recordFailures, acknowledgeFailures, loadSyncFailures } = await L();
|
||||
recordFailures('alpha', [{ path: 'a.md', error: 'e' }], 'c1');
|
||||
recordFailures('beta', [{ path: 'b.md', error: 'e' }], 'c1');
|
||||
|
||||
const res = acknowledgeFailures('alpha');
|
||||
expect(res.count).toBe(1);
|
||||
const rows = loadSyncFailures();
|
||||
expect(rows.find(r => r.source_id === 'alpha')!.state).toBe('acknowledged');
|
||||
expect(rows.find(r => r.source_id === 'beta')!.state).toBe('open');
|
||||
});
|
||||
});
|
||||
|
||||
describe('#4 success clears → consecutive attempts', () => {
|
||||
test('clearFailures removes a path; a later failure restarts at 1', async () => {
|
||||
const { recordFailures, clearFailures, loadSyncFailures } = await L();
|
||||
recordFailures('s', [{ path: 'a.md', error: 'e' }], 'c1');
|
||||
recordFailures('s', [{ path: 'a.md', error: 'e' }], 'c2');
|
||||
expect(loadSyncFailures()[0].attempts).toBe(2);
|
||||
|
||||
clearFailures('s', ['a.md']);
|
||||
expect(loadSyncFailures().length).toBe(0);
|
||||
|
||||
recordFailures('s', [{ path: 'a.md', error: 'e' }], 'c3');
|
||||
expect(loadSyncFailures()[0].attempts).toBe(1);
|
||||
});
|
||||
|
||||
test('clearFailures is source-scoped', async () => {
|
||||
const { recordFailures, clearFailures, loadSyncFailures } = await L();
|
||||
recordFailures('s1', [{ path: 'a.md', error: 'e' }], 'c1');
|
||||
recordFailures('s2', [{ path: 'a.md', error: 'e' }], 'c1');
|
||||
clearFailures('s1', ['a.md']);
|
||||
const rows = loadSyncFailures();
|
||||
expect(rows.length).toBe(1);
|
||||
expect(rows[0].source_id).toBe('s2');
|
||||
});
|
||||
});
|
||||
|
||||
describe('#3 sentinel never auto-skips', () => {
|
||||
test('decideGateAction hard-blocks on a sentinel even when chronic + skipFailed', async () => {
|
||||
const { decideGateAction } = await L();
|
||||
const d = decideGateAction({
|
||||
fileFailures: [],
|
||||
sentinels: [{ path: '<head>' }],
|
||||
attemptsByPath: new Map(),
|
||||
threshold: 3,
|
||||
skipFailed: true,
|
||||
});
|
||||
expect(d.action).toBe('hard_block');
|
||||
expect(d.autoSkipPaths).toEqual([]);
|
||||
});
|
||||
|
||||
test('autoSkipFailures refuses to skip a sentinel path', async () => {
|
||||
const { recordFailures, autoSkipFailures, loadSyncFailures } = await L();
|
||||
recordFailures('s', [{ path: '<head>', error: 'history rewrite' }], 'c1');
|
||||
const res = autoSkipFailures('s', ['<head>']);
|
||||
expect(res.count).toBe(0);
|
||||
expect(loadSyncFailures()[0].state).toBe('open');
|
||||
});
|
||||
|
||||
test('isSkippablePath', async () => {
|
||||
const { isSkippablePath } = await L();
|
||||
expect(isSkippablePath('people/x.md')).toBe(true);
|
||||
expect(isSkippablePath('<head>')).toBe(false);
|
||||
});
|
||||
});
|
||||
|
||||
describe('#7 legacy normalization + dup collapse', () => {
|
||||
test('backfills source_id/state/attempts/first_seen on legacy rows', async () => {
|
||||
const { loadSyncFailures, syncFailuresPath } = await L();
|
||||
mkdirSync(dirname(syncFailuresPath()), { recursive: true });
|
||||
writeFileSync(
|
||||
syncFailuresPath(),
|
||||
JSON.stringify({ path: 'a.md', error: 'YAML parse failed', commit: 'old', ts: '2025-01-01T00:00:00Z' }) + '\n' +
|
||||
JSON.stringify({ path: 'b.md', error: 'e', commit: 'old', ts: '2025-01-02T00:00:00Z', acknowledged_at: '2025-02-01T00:00:00Z' }) + '\n',
|
||||
);
|
||||
const rows = loadSyncFailures();
|
||||
const a = rows.find(r => r.path === 'a.md')!;
|
||||
const b = rows.find(r => r.path === 'b.md')!;
|
||||
expect(a.source_id).toBe('default');
|
||||
expect(a.state).toBe('open');
|
||||
expect(a.attempts).toBe(1);
|
||||
expect(a.first_seen).toBe('2025-01-01T00:00:00Z');
|
||||
expect(b.state).toBe('acknowledged');
|
||||
});
|
||||
|
||||
test('collapses duplicate open rows for one (source,path) into one with attempts = distinct commits', async () => {
|
||||
const { loadSyncFailures, syncFailuresPath } = await L();
|
||||
mkdirSync(dirname(syncFailuresPath()), { recursive: true });
|
||||
// Three legacy rows, same path, two distinct commits.
|
||||
writeFileSync(
|
||||
syncFailuresPath(),
|
||||
[
|
||||
{ path: 'a.md', error: 'e', commit: 'c1', ts: '2025-01-01T00:00:00Z' },
|
||||
{ path: 'a.md', error: 'e', commit: 'c2', ts: '2025-01-02T00:00:00Z' },
|
||||
{ path: 'a.md', error: 'e2', commit: 'c2', ts: '2025-01-03T00:00:00Z' },
|
||||
].map(r => JSON.stringify(r)).join('\n') + '\n',
|
||||
);
|
||||
const rows = loadSyncFailures();
|
||||
expect(rows.length).toBe(1);
|
||||
expect(rows[0].attempts).toBe(2); // distinct commits c1, c2
|
||||
expect(rows[0].commit).toBe('c2'); // latest by ts
|
||||
expect(rows[0].first_seen).toBe('2025-01-01T00:00:00Z');
|
||||
});
|
||||
|
||||
test('skips malformed lines', async () => {
|
||||
const { loadSyncFailures, recordFailures, syncFailuresPath } = await L();
|
||||
recordFailures('s', [{ path: 'a.md', error: 'e' }], 'c1');
|
||||
writeFileSync(syncFailuresPath(), readFileSync(syncFailuresPath(), 'utf-8') + 'NOT-JSON\n');
|
||||
expect(loadSyncFailures().length).toBe(1);
|
||||
});
|
||||
});
|
||||
|
||||
describe('#1 severity — auto_skipped stays visible (WARN)', () => {
|
||||
test('decideSyncFailureSeverity branches', async () => {
|
||||
const { decideSyncFailureSeverity } = await L();
|
||||
const base = { source_id: 's', path: 'a.md', error: 'e', code: 'X', commit: 'c', first_seen: '', ts: '', attempts: 1 };
|
||||
const now = Date.parse('2026-06-07T00:00:00Z');
|
||||
const failHours = 72;
|
||||
|
||||
// empty → ok
|
||||
expect(decideSyncFailureSeverity({ entries: [], nowMs: now, failHours }).status).toBe('ok');
|
||||
|
||||
// one recent open → warn
|
||||
expect(decideSyncFailureSeverity({
|
||||
entries: [{ ...base, state: 'open', ts: '2026-06-06T18:00:00Z' }],
|
||||
nowMs: now, failHours,
|
||||
}).status).toBe('warn');
|
||||
|
||||
// one OLD open (>72h) → fail
|
||||
expect(decideSyncFailureSeverity({
|
||||
entries: [{ ...base, state: 'open', ts: '2026-06-01T00:00:00Z' }],
|
||||
nowMs: now, failHours,
|
||||
}).status).toBe('fail');
|
||||
|
||||
// 10 recent OPEN → fail (count of blocking failures)
|
||||
const ten = Array.from({ length: 10 }, (_, i) => ({ ...base, path: `a${i}.md`, state: 'open' as const, ts: '2026-06-06T23:00:00Z' }));
|
||||
expect(decideSyncFailureSeverity({ entries: ten, nowMs: now, failHours }).status).toBe('fail');
|
||||
|
||||
// 10 recent AUTO_SKIPPED → still warn (#3): the valve already advanced the
|
||||
// bookmark, so indexing is not wedged. Visible, not gating.
|
||||
const tenSkipped = Array.from({ length: 10 }, (_, i) => ({ ...base, path: `s${i}.md`, state: 'auto_skipped' as const, ts: '2026-06-06T23:00:00Z' }));
|
||||
const skSev = decideSyncFailureSeverity({ entries: tenSkipped, nowMs: now, failHours });
|
||||
expect(skSev.status).toBe('warn');
|
||||
expect(skSev.auto_skipped).toBe(10);
|
||||
|
||||
// auto_skipped only → warn (still visible), counted
|
||||
const sev = decideSyncFailureSeverity({
|
||||
entries: [{ ...base, state: 'auto_skipped', ts: '2026-06-01T00:00:00Z' }],
|
||||
nowMs: now, failHours,
|
||||
});
|
||||
expect(sev.status).toBe('warn');
|
||||
expect(sev.auto_skipped).toBe(1);
|
||||
expect(sev.unresolved).toBe(1);
|
||||
|
||||
// acknowledged only → ok
|
||||
expect(decideSyncFailureSeverity({
|
||||
entries: [{ ...base, state: 'acknowledged', ts: '2026-06-01T00:00:00Z' }],
|
||||
nowMs: now, failHours,
|
||||
}).status).toBe('ok');
|
||||
});
|
||||
|
||||
test('malformed ts never crashes (treated as not-old)', async () => {
|
||||
const { decideSyncFailureSeverity } = await L();
|
||||
const sev = decideSyncFailureSeverity({
|
||||
entries: [{ source_id: 's', path: 'a.md', error: 'e', code: 'X', commit: 'c', first_seen: '', ts: 'not-a-date', attempts: 1, state: 'open' }],
|
||||
nowMs: Date.now(), failHours: 72,
|
||||
});
|
||||
expect(sev.status).toBe('warn');
|
||||
});
|
||||
|
||||
test('auto_skipped row keeps acknowledged_at null so legacy !acknowledged_at readers still count it', async () => {
|
||||
const { recordFailures, autoSkipFailures, loadSyncFailures } = await L();
|
||||
recordFailures('s', [{ path: 'a.md', error: 'e' }], 'c1');
|
||||
autoSkipFailures('s', ['a.md']);
|
||||
const row = loadSyncFailures()[0];
|
||||
expect(row.state).toBe('auto_skipped');
|
||||
expect(row.acknowledged).toBe(false);
|
||||
expect(row.acknowledged_at).toBeNull();
|
||||
});
|
||||
});
|
||||
|
||||
describe('decideGateAction — full branch table', () => {
|
||||
test('all branches', async () => {
|
||||
const { decideGateAction } = await L();
|
||||
const mk = (over: Partial<Parameters<typeof decideGateAction>[0]>) => decideGateAction({
|
||||
fileFailures: [], sentinels: [], attemptsByPath: new Map(), threshold: 3, skipFailed: false, ...over,
|
||||
});
|
||||
|
||||
expect(mk({}).action).toBe('advance'); // no failures
|
||||
expect(mk({ fileFailures: [{ path: 'a' }], skipFailed: true }).action).toBe('advance');
|
||||
expect(mk({ fileFailures: [{ path: 'a' }], threshold: 0 }).action).toBe('block'); // valve off
|
||||
expect(mk({ fileFailures: [{ path: 'a' }], attemptsByPath: new Map([['a', 1]]) }).action).toBe('block'); // fresh
|
||||
|
||||
const chronic = mk({ fileFailures: [{ path: 'a' }], attemptsByPath: new Map([['a', 3]]) });
|
||||
expect(chronic.action).toBe('advance_then_autoskip');
|
||||
expect(chronic.autoSkipPaths).toEqual(['a']);
|
||||
|
||||
// mixed fresh + chronic → block (don't silently drop the fresh one)
|
||||
expect(mk({
|
||||
fileFailures: [{ path: 'a' }, { path: 'b' }],
|
||||
attemptsByPath: new Map([['a', 3], ['b', 1]]),
|
||||
}).action).toBe('block');
|
||||
});
|
||||
});
|
||||
|
||||
describe('#5 + #6 applySyncFailureGate orchestration', () => {
|
||||
test('advance_then_autoskip after threshold; advance runs before auto-skip mark', async () => {
|
||||
process.env.GBRAIN_SYNC_AUTOSKIP_AFTER = '3';
|
||||
const { applySyncFailureGate, loadSyncFailures } = await L();
|
||||
let advanceCalls = 0;
|
||||
const run = () => applySyncFailureGate({
|
||||
sourceId: 's', failedFiles: [{ path: 'poison.md', error: 'YAML parse failed' }],
|
||||
succeededPaths: [], commit: 'c', skipFailed: false,
|
||||
advance: async () => { advanceCalls++; },
|
||||
});
|
||||
|
||||
let r = await run();
|
||||
expect(r.advanced).toBe(false); // attempts 1 → block
|
||||
r = await run();
|
||||
expect(r.advanced).toBe(false); // attempts 2 → block
|
||||
r = await run();
|
||||
expect(r.advanced).toBe(true); // attempts 3 → advance + auto-skip
|
||||
expect(r.autoSkipped).toEqual(['poison.md']);
|
||||
expect(advanceCalls).toBe(1);
|
||||
expect(loadSyncFailures()[0].state).toBe('auto_skipped');
|
||||
});
|
||||
|
||||
test('a throwing advance() marks nothing auto_skipped (atomicity)', async () => {
|
||||
process.env.GBRAIN_SYNC_AUTOSKIP_AFTER = '1';
|
||||
const { applySyncFailureGate, loadSyncFailures } = await L();
|
||||
await expect(applySyncFailureGate({
|
||||
sourceId: 's', failedFiles: [{ path: 'poison.md', error: 'e' }],
|
||||
succeededPaths: [], commit: 'c', skipFailed: false,
|
||||
advance: async () => { throw new Error('db write failed'); },
|
||||
})).rejects.toThrow('db write failed');
|
||||
// Recorded as open, NOT auto_skipped — next run can retry.
|
||||
const row = loadSyncFailures()[0];
|
||||
expect(row.state).toBe('open');
|
||||
});
|
||||
|
||||
test('sentinel hard-blocks even with skipFailed; no advance', async () => {
|
||||
const { applySyncFailureGate } = await L();
|
||||
let advanced = false;
|
||||
const r = await applySyncFailureGate({
|
||||
sourceId: 's', failedFiles: [{ path: '<head>', error: 'history rewrite' }],
|
||||
succeededPaths: [], commit: 'c', skipFailed: true,
|
||||
advance: async () => { advanced = true; },
|
||||
});
|
||||
expect(r.advanced).toBe(false);
|
||||
expect(r.sentinelBlocked).toBe(true);
|
||||
expect(advanced).toBe(false);
|
||||
});
|
||||
|
||||
test('succeeded paths clear prior failures even with no new failures', async () => {
|
||||
const { recordFailures, applySyncFailureGate, loadSyncFailures } = await L();
|
||||
recordFailures('s', [{ path: 'a.md', error: 'e' }], 'c1');
|
||||
expect(loadSyncFailures().length).toBe(1);
|
||||
const r = await applySyncFailureGate({
|
||||
sourceId: 's', failedFiles: [], succeededPaths: ['a.md'], commit: 'c2',
|
||||
skipFailed: false, advance: async () => {},
|
||||
});
|
||||
expect(r.advanced).toBe(true);
|
||||
expect(loadSyncFailures().length).toBe(0);
|
||||
});
|
||||
|
||||
test('skipFailed acknowledges and advances', async () => {
|
||||
const { applySyncFailureGate, loadSyncFailures } = await L();
|
||||
const r = await applySyncFailureGate({
|
||||
sourceId: 's', failedFiles: [{ path: 'a.md', error: 'e' }],
|
||||
succeededPaths: [], commit: 'c', skipFailed: true, advance: async () => {},
|
||||
});
|
||||
expect(r.advanced).toBe(true);
|
||||
expect(r.acknowledged).toBe(1);
|
||||
expect(loadSyncFailures()[0].state).toBe('acknowledged');
|
||||
});
|
||||
});
|
||||
|
||||
describe('#8 concurrency: lock + atomic write', () => {
|
||||
test('atomic rewrite leaves valid JSON lines (no partial)', async () => {
|
||||
const { recordFailures, syncFailuresPath } = await L();
|
||||
recordFailures('s', [{ path: 'a.md', error: 'e' }, { path: 'b.md', error: 'e' }], 'c1');
|
||||
const raw = readFileSync(syncFailuresPath(), 'utf-8');
|
||||
for (const line of raw.split('\n').filter(Boolean)) {
|
||||
expect(() => JSON.parse(line)).not.toThrow();
|
||||
}
|
||||
expect(existsSync(syncFailuresPath() + '.lock')).toBe(false); // lock released
|
||||
});
|
||||
|
||||
test('a stale lock (old mtime) is broken so a mutation still proceeds', async () => {
|
||||
const { recordFailures, syncFailuresPath, withLedgerLock } = await L();
|
||||
// Pre-create the data dir + a stale lock file.
|
||||
mkdirSync(dirname(syncFailuresPath()), { recursive: true });
|
||||
const lockPath = syncFailuresPath() + '.lock';
|
||||
closeSync(openSync(lockPath, 'w'));
|
||||
const old = new Date(Date.now() - 120_000); // 2 min ago > 30s stale window
|
||||
utimesSync(lockPath, old, old);
|
||||
|
||||
// Mutation should break the stale lock and succeed.
|
||||
recordFailures('s', [{ path: 'a.md', error: 'e' }], 'c1');
|
||||
const { loadSyncFailures } = await L();
|
||||
expect(loadSyncFailures().length).toBe(1);
|
||||
|
||||
// withLedgerLock returns the callback's value.
|
||||
expect(withLedgerLock(() => 42)).toBe(42);
|
||||
});
|
||||
});
|
||||
|
||||
describe('resolveAutoSkipThreshold', () => {
|
||||
test('default 3, env override, 0 disables, invalid → default', async () => {
|
||||
const { resolveAutoSkipThreshold } = await L();
|
||||
delete process.env.GBRAIN_SYNC_AUTOSKIP_AFTER;
|
||||
expect(resolveAutoSkipThreshold()).toBe(3);
|
||||
process.env.GBRAIN_SYNC_AUTOSKIP_AFTER = '5';
|
||||
expect(resolveAutoSkipThreshold()).toBe(5);
|
||||
process.env.GBRAIN_SYNC_AUTOSKIP_AFTER = '0';
|
||||
expect(resolveAutoSkipThreshold()).toBe(0);
|
||||
process.env.GBRAIN_SYNC_AUTOSKIP_AFTER = 'nonsense';
|
||||
expect(resolveAutoSkipThreshold()).toBe(3);
|
||||
});
|
||||
});
|
||||
+39
-14
@@ -40,7 +40,11 @@ afterEach(() => {
|
||||
});
|
||||
|
||||
describe('Bug 9 — sync-failures JSONL helpers', () => {
|
||||
test('recordSyncFailures appends one line per failure with dedup', async () => {
|
||||
// issue #1939: recordSyncFailures now upserts by (source_id, path) and
|
||||
// increments `attempts` (consecutive failed runs) instead of appending a row
|
||||
// per (path, commit, error). One row per failing path; the attempt counter
|
||||
// drives the bounded auto-skip valve.
|
||||
test('recordSyncFailures upserts per path, incrementing attempts', async () => {
|
||||
const { recordSyncFailures, loadSyncFailures, syncFailuresPath } = await import('../src/core/sync.ts');
|
||||
|
||||
recordSyncFailures([
|
||||
@@ -51,21 +55,28 @@ describe('Bug 9 — sync-failures JSONL helpers', () => {
|
||||
expect(existsSync(syncFailuresPath())).toBe(true);
|
||||
const entries = loadSyncFailures();
|
||||
expect(entries.length).toBe(2);
|
||||
expect(entries[0].path).toBe('people/alice.md');
|
||||
expect(entries[0].commit).toBe('abc123def456');
|
||||
expect(entries[0].acknowledged).toBeUndefined();
|
||||
const alice = entries.find(e => e.path === 'people/alice.md')!;
|
||||
expect(alice.commit).toBe('abc123def456');
|
||||
expect(alice.state).toBe('open');
|
||||
expect(alice.attempts).toBe(1);
|
||||
expect(alice.acknowledged).toBe(false);
|
||||
|
||||
// Same failure on same commit should NOT re-append.
|
||||
// Same path again (same commit) → still ONE row, attempts climbs.
|
||||
recordSyncFailures([
|
||||
{ path: 'people/alice.md', error: 'YAML: unexpected colon in title' },
|
||||
], 'abc123def456');
|
||||
expect(loadSyncFailures().length).toBe(2);
|
||||
expect(loadSyncFailures().find(e => e.path === 'people/alice.md')!.attempts).toBe(2);
|
||||
|
||||
// Different commit → new entry.
|
||||
// Different commit → still upserted (not appended), attempts keeps climbing.
|
||||
recordSyncFailures([
|
||||
{ path: 'people/alice.md', error: 'YAML: unexpected colon in title' },
|
||||
], 'zzz999');
|
||||
expect(loadSyncFailures().length).toBe(3);
|
||||
const after = loadSyncFailures();
|
||||
expect(after.length).toBe(2);
|
||||
const alice2 = after.find(e => e.path === 'people/alice.md')!;
|
||||
expect(alice2.attempts).toBe(3);
|
||||
expect(alice2.commit).toBe('zzz999');
|
||||
});
|
||||
|
||||
test('acknowledgeSyncFailures marks unacked entries, leaves acked alone', async () => {
|
||||
@@ -129,6 +140,18 @@ describe('Bug 9 — doctor surfaces sync failures', () => {
|
||||
expect(source).toContain('unacknowledgedSyncFailures');
|
||||
expect(source).toContain("'gbrain sync --skip-failed'");
|
||||
});
|
||||
|
||||
// issue #1939: BOTH doctor surfaces (local buildChecks + remote
|
||||
// doctorReportRemote) must decide severity through the one shared helper so
|
||||
// they can never drift. The remote surface must no longer hand-roll an
|
||||
// `acknowledged_at` count (the old field-split that caused drift).
|
||||
test('both doctor surfaces use the shared decideSyncFailureSeverity', async () => {
|
||||
const source = await Bun.file(new URL('../src/commands/doctor.ts', import.meta.url)).text();
|
||||
const occurrences = source.split('decideSyncFailureSeverity').length - 1;
|
||||
expect(occurrences).toBeGreaterThanOrEqual(2); // local + remote
|
||||
// remote no longer counts `!entry.acknowledged_at` by hand.
|
||||
expect(source).not.toContain('if (!entry.acknowledged_at) unacked++');
|
||||
});
|
||||
});
|
||||
|
||||
describe('Bug 9 — sync.ts CLI flag wiring', () => {
|
||||
@@ -169,23 +192,25 @@ describe('Bug 9 — sync.ts CLI flag wiring', () => {
|
||||
expect(unacknowledgedSyncFailures().length).toBe(0);
|
||||
});
|
||||
|
||||
test('performSync gates sync.last_commit on failedFiles.length', async () => {
|
||||
test('performSync gates the bookmark through the shared failure ledger', async () => {
|
||||
const source = await Bun.file(new URL('../src/commands/sync.ts', import.meta.url)).text();
|
||||
// The gate exists and references the failure set.
|
||||
expect(source).toContain('failedFiles.length > 0');
|
||||
// issue #1939: the gate is now the shared applySyncFailureGate orchestrator.
|
||||
expect(source).toContain('applySyncFailureGate');
|
||||
expect(source).toContain('blocked_by_failures');
|
||||
});
|
||||
|
||||
test('performFullSync gates on result.failures from runImport', async () => {
|
||||
test('performFullSync routes failures through the same shared gate', async () => {
|
||||
const source = await Bun.file(new URL('../src/commands/sync.ts', import.meta.url)).text();
|
||||
expect(source).toContain('result.failures.length > 0');
|
||||
expect(source).toContain('result.failures');
|
||||
expect(source).toContain('applySyncFailureGate');
|
||||
});
|
||||
|
||||
test('runImport returns RunImportResult with failures list', async () => {
|
||||
test('runImport returns RunImportResult and records via the ledger', async () => {
|
||||
const source = await Bun.file(new URL('../src/commands/import.ts', import.meta.url)).text();
|
||||
expect(source).toContain('RunImportResult');
|
||||
expect(source).toContain('failures: Array<{ path: string; error: string }>');
|
||||
expect(source).toContain('recordSyncFailures');
|
||||
// issue #1939: import records source-scoped via recordFailures (sourceId, …).
|
||||
expect(source).toContain('recordFailures');
|
||||
});
|
||||
});
|
||||
|
||||
|
||||
Reference in New Issue
Block a user