Files
gbrain/test/extract/status.test.ts
T
48e1000306 v0.41.23.0 feat: extract operator surfaces + pack-driven extractables (#1541)
* Wave A: schema + receipts foundation for v0.42 extract operator surfaces

Foundation layer for the pack-driven extractables + receipt-as-brain-memory
+ operator-discoverability cathedral. Five atomic pieces ship together
because their schema + helpers + module dependencies are tight-coupled:

A1. Widen pack manifest's `extractable` from `boolean` to
    `boolean | ExtractableSpec`. ExtractableSpec carries prompt_template,
    fixture_corpus, eval_dimensions, benchmark_min_recall, and reserves
    verifier_path for v0.43+ pack-shipped verifier code (REFUSE at
    runtime in v0.42 per plan D-EXTRACT-37). Back-compat: every pre-v0.42
    pack with `extractable: true` continues parsing unchanged. Three new
    helpers: extractableSpecsFromPack(), getExtractableSpec(),
    refuseVerifierPathInV042().

A2. New page type `extract_receipt` in ALL_PAGE_TYPES. Source-boost map
    adds `extracts/` prefix at factor 0.3 — receipts surface in search
    when extraction-relevant but never dominate user content (D-EXTRACT-42).

A3. New module src/core/extract/receipt-writer.ts (~190 LOC) exporting
    writeReceipt(engine, input). Canonical slug shape
    extracts/{date}/{kind}/{source_id}/{run_id_short}/round-{N} per
    D-EXTRACT-17. Frontmatter belt+suspenders per D-EXTRACT-19: BOTH
    type:extract_receipt AND dream_generated:true stamped on every
    receipt, regardless of caller, so the eligibility predicate's
    anti-loop guards reject the receipt page from any future extraction
    sweep (single-flag bypass requires breaking two unrelated checks).
    Idempotent on resume — same run_id+round overwrites cleanly.

A4. Migration v104 creates extract_rollup_7d table (per-day rollup of
    extract events keyed on kind+source_id+day). Audit JSONL stays the
    SOURCE OF TRUTH per F-OUT-19; this table is a best-effort cache for
    doctor's <100ms read budget. Per-day rows mean the 7-day window
    auto-evicts on every read. v100 was deliberately skipped on master
    (renumbered out during a prior wave); v101/v102/v103 also taken;
    v104 is the next clean slot.

A5. Doctor `extract_health` check reads extract_rollup_7d for last 7
    days and emits per-kind aggregates: cost_7d_usd, eval_pass_count,
    eval_fail_count, halt_count, round_completed_count, halt_rate.
    3-state: OK when rollup empty (pre-v0.42 brain or fresh init), WARN
    when any per-kind halt rate > 10% (top-3 named in message), WARN
    when rollup_write_failures > 0 (audit JSONL is SoT but operator
    deserves to know the DB cache is degraded). Pre-v104 brains stay
    quiet — the missing-table error path is caught and treated as
    OK so doctor doesn't warn during the upgrade window.

Tests added:
- test/extractable-spec-widening.test.ts (22 cases) — back-compat with
  boolean shape, new struct parsing, verifier_path REFUSE contract.
- test/extract/receipt-writer.test.ts (12 cases) — slug shape, frontmatter
  belt+suspenders, idempotent resume, body human-readability.
- test/doctor-extract-health.test.ts (8 cases) — empty rollup OK, halt
  rate WARN, rollup_write_failures WARN, 7-day window inclusion at
  boundary, multi-kind top-3 message ordering.

Plus the canonical bootstrap-coverage test passes with the new v104
migration cleanly applied through both engines.

Plan: ~/.claude/plans/system-instruction-you-are-working-stateless-dragonfly.md
Wave A scope. Wave B (hook receipts into existing extractors) follows.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* Wave B: hook receipts + rollup row into the 5 shipped extractors

Each LLM-backed extractor surface now records its run in two places when
something actually happened:

1. An extract receipt PAGE at extracts/{date}/{kind}/{source_id}/{run_id_short}/round-{N}
   (queryable via gbrain search, citable, surfaces in cross-modal
   contradiction probes per the Wave A foundation). Only written when
   `total_rows > 0` so no-op runs don't bloat the brain.

2. An UPSERT row in extract_rollup_7d (DB-backed best-effort cache
   per F-OUT-19) so the doctor extract_health check from Wave A reads
   per-kind aggregates without scanning JSONL.

New module src/core/extract/rollup-writer.ts (~120 LOC) exports
upsertExtractRollup() with PostgreSQL ON CONFLICT DO UPDATE on the
(kind, source_id, day) PK. Concurrency-safe per F-OUT-14 design.
Failure path is best-effort — bumps rollup_write_failures in the
table itself, stderr-warns once per (kind, day, error-class), and
NEVER fails the parent extraction operation. JSONL remains source
of truth.

Wired into 5 extractors:
- extract-conversation-facts (kind: facts.conversation) — both
  success path AND BudgetExhausted halt path write receipt+rollup
  so partial runs are still observable.
- extract_atoms cycle phase (kind: atoms)
- synthesize_concepts cycle phase (kind: concepts, source_id: default
  because concepts are brain-global)
- propose_takes cycle phase (kind: takes.proposed) — scope-aware
  source_id from the read scope.
- extract_facts cycle phase (kind: facts.fence) — deterministic
  (no LLM cost) but still records reconcile activity so doctor sees
  the cycle is alive.

Receipt frontmatter belt+suspenders (D-EXTRACT-19) reused from
Wave A: every receipt stamps BOTH `type: extract_receipt` AND
`dream_generated: true` so the eligibility predicate's anti-loop
guards reject the receipt page from any future extraction sweep.

Test surgery in test/propose-takes.test.ts — one existing assertion
tightened from "no INSERTs" to "no INSERT INTO take_proposals" so
the new rollup UPSERT doesn't falsely fail the cache-hit case test.

Run regression: 85/85 tests pass across extract-conversation-facts,
extract-atoms-synthesize-concepts, extract-facts-phase, propose-takes.

Plan: ~/.claude/plans/system-instruction-you-are-working-stateless-dragonfly.md
Wave B scope. Wave C (pack-author scaffolding + benchmark) follows.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* Wave C+D: pack-author scaffolding + operator surfaces for v0.42 extract

Wave C: pack-author authoring loop
  - scaffold-extractable mutation primitive declares a kind as extractable
    on a pack manifest in one verb (wires through updateTypeOnPack from
    the v0.41 mutate library); generates 5 placeholder fixtures + a
    pack-supplied prompt template stub
  - schema CLI wires gbrain schema scaffold-extractable <type> --pack <pack>
  - extract benchmark CLI loads a pack's fixture corpus through strict
    D-EXTRACT-21 path validation (rejects absolute paths, .. traversal,
    null bytes, symlinks resolving outside pack root); v0.42 ships as a
    stub reporter (LLM dispatch deferred to Wave E)

Wave D: operator surfaces
  - extract status CLI reads extract_rollup_7d for the last 7 days,
    sorts by (halt_rate desc, cost desc); kubectl-style right-aligned
    table, top-5 + "more rows" hint by default, --verbose shows all;
    stable schema_version: 1 JSON envelope for monitoring pipelines
  - extract --explain <kind> CLI prints the active pack's resolution
    chain: declaration source (pack-declared vs built-in cycle phase),
    prompt_template + fixture_corpus paths with existence checks,
    eval_dimensions, benchmark_min_recall, and the last 7d rollup
  - extract.ts gains a lifecycle-grouped help text (Extraction /
    Inspection / Status) per the original D3 plan goal

Tests:
  - test/schema-pack/scaffold-extractable.test.ts (15 cases) including
    explicit privacy-rule assertions guarding against real-name leakage
  - test/extract/benchmark.test.ts (17 cases) covering path validation
    rejections + JSONL fixture parsing
  - test/extract/status.test.ts (15 cases) over pure aggregation +
    formatting

Housekeeping:
  - test/extract/receipt-writer.test.ts refactored to the canonical
    PGLite block (beforeAll/afterAll/resetPgliteState in beforeEach)
    per CLAUDE.md test-isolation R3+R4; runtime drops from ~30s of
    99-migration replay per test to <6s for all 12 cases together

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* v0.42.0.0: extract operator surfaces + pack-driven extractables

Bump VERSION + package.json to 0.42.0.0. CHANGELOG entry covers the
three-wave shipped scope (receipts + rollup + doctor check; receipts
hooked into all 5 shipped extractors; pack-author scaffolding +
benchmark stub-reporter; status + --explain dashboards + lifecycle
help). CLAUDE.md Key Files gains a v0.42 cluster annotation. llms.txt
regenerated.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* v0.41.23.0: re-tag from 0.42.0.0 (patch-channel slot, no scope change)

VERSION + package.json + CHANGELOG header + CLAUDE.md cluster annotation
all moved from 0.42.0.0 to 0.41.23.0. Body text updated in-place: every
"v0.42" / "v0.43+" reference inside this entry's release notes now reads
"v0.41.23" or "follow-up release" as appropriate.

Same scope shipping — the three-wave extract operator surface stays
intact. Just lands in the patch-channel queue (.20/.21/.23 free; .22 is
PR #1542's type-unification cathedral) instead of the minor-channel bump.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix: add extract_receipt to gbrain-base.yaml page_types (CI parity gate)

CI shard 5 caught the drift: test/regressions/gbrain-base-equivalence.test.ts
asserts every ALL_PAGE_TYPES seed has a matching page_type entry in the
gbrain-base.yaml pack. Wave A added `extract_receipt` to ALL_PAGE_TYPES
but didn't seed it in the base pack manifest.

Adds the entry under the `annotation` primitive with `extracts/` path
prefix (matches the source-boost demote site) and `extractable: false`
(receipts are written by the framework, never extracted from). Comment
documents the belt+suspenders D-EXTRACT-19 invariant so future readers
understand why receipts carry both `dream_generated: true` AND
`type: extract_receipt`.

Closes the CI gate without changing runtime behavior — the pack-aware
read paths already had the prefix demote wired in src/core/search/source-boost.ts.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix: bump gbrain-base page-type count 24→25 in schema-cli test

CI shard 4 caught the second drift from the same root cause as the
prior parity-gate fix: v0.41.23's `extract_receipt` addition bumped
gbrain-base.yaml from 24 to 25 page types. The schema-cli smoke test
was pinned at 24 (the count after v0.41.11.0 added `conversation` +
`atom`); update to 25 and note v0.41.23's contribution alongside the
prior version stamp.

Verified hermetic: running test/schema-cli.test.ts with a clean
GBRAIN_HOME tempdir produces 12/12 pass (the local-machine 'schema
active' fail is from a real ~/.gbrain pinning gbrain-base-v2; not a
shipped-code issue, doesn't repro on CI).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(test): pack-locator stub leak between shard 6 test files

CI shard 6 caught three flaky failures in test/onboard-pack-upgrade-checks.test.ts:
  - checkPackUpgradeAvailable > fires on gbrain-base brain with gbrain-base-v2
  - checkPackUpgradeAvailable > manual_only routing via render.ts allowlist (D17)
  - checkTypeProliferation > warns when distinct types exceed declared+5

Root cause: test/schema-pack-sync.test.ts calls
`__setPackLocatorForTests(...)` to stub the disk-loader, but doesn't
restore in afterAll. Bun's CI shard 6 loads multiple test files into
one process; when sync.test.ts runs before onboard-pack-upgrade-checks.test.ts,
the stubbed locator persists at module scope. `loadActivePack` for
gbrain-base / gbrain-base-v2 then returns null and:
  - findPackSuccessors returns [] → status='ok' instead of 'warn' (F1+F2)
  - declared falls back to 15 → fail threshold becomes 30, 32 > 30 → 'fail'
    instead of 'warn' (F3)

Local single-file runs pass because the locator starts at its default.

Two-layer fix:
  1. test/schema-pack-sync.test.ts afterAll calls
     `_resetPackLocatorForTests()` to undo the mutation (the canonical
     fix at the source).
  2. test/onboard-pack-upgrade-checks.test.ts beforeEach calls the same
     reset (defense-in-depth against any future test file in the shard
     that forgets to restore).

Reproduced locally: running the three shard-6 schema-pack files together
fails 3 tests pre-fix and passes 30/30 post-fix. Full shard 6 sweep
(77 files, 1232 tests) now green; bun run verify still 28/28.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(test): pglite-engine — dim-agnostic chunk-embedding test data

CI shard 6 caught two flaky failures in test/pglite-engine.test.ts:
  - PGLiteEngine: Chunks > getChunksWithEmbeddings returns embedding data
  - PGLiteEngine: stale chunk pagination > countStaleChunks counts chunks
    with NULL embedding only

Both failed with `expected 1280 dimensions, not 1536` at the upsert site.

Root cause: pglite-engine.ts:287 initSchema() reads embedding dim from
gw.getEmbeddingDimensions() if the gateway is configured (potentially
left in that state by another shard-6 test file in the same bun process),
falling back to DEFAULT_EMBEDDING_DIMENSIONS otherwise — which is 1280
since v0.36+ when the ZE default landed (zeroentropyai:zembed-1).
Pre-v0.36 defaults were OpenAI's 1536; my test data was pinned to that
stale literal.

The two outcomes that pass:
  - gateway happens to be configured for 1536-dim (e.g. master shard 6
    run 26515999465 — these tests passed at 20ms + 24ms with no
    "dimensions" error)
  - gateway happens to be configured for 1280-dim AND test data is 1280

The outcome that fails:
  - gateway configured for 1280-dim AND test data hardcoded to 1536

Fix: capture the actual column width after initSchema (probe
pg_attribute.atttypmod for content_chunks.embedding) and use that
captured `CHUNK_EMBED_DIM` constant at the three Float32Array sites.
Test data now matches whatever width the column was created at,
regardless of which shard-6 file ran first.

Local repro: full shard 6 (77 files, 1232 tests, ~6min) green; this
file standalone (100 tests) green; bun run verify 28/28.

Broader pattern: 9 other test files use the same Float32Array(1536)
literal. None land in shard 6 today (so they don't flake), but the
fix shape here can be lifted into a shared helper if the bug class
surfaces elsewhere — filed as a v0.42+ follow-up rather than a
preemptive sweep, since each file's setup shape is slightly different.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-27 08:27:05 -07:00

207 lines
6.2 KiB
TypeScript

// v0.42 Wave D1 — extract status CLI unit tests.
//
// Pins:
// - Sort order: halt_rate desc, cost desc — most-troubled first
// - Non-verbose: top 5; verbose: all
// - Empty rollup → "No extract events" message
// - JSON envelope schema_version: 1
import { describe, expect, test } from 'bun:test';
import {
buildStatusReport,
formatStatusTable,
type ExtractStatusRow,
} from '../../src/commands/extract-status.ts';
const SAMPLE_ROWS = [
{
kind: 'facts.conversation',
source_id: 'default',
cost_7d_usd: 1.50,
eval_pass_count: 5,
eval_fail_count: 0,
halt_count: 0,
round_completed_count: 10,
last_updated_at: '2026-05-27T14:00:00Z',
},
{
kind: 'atoms',
source_id: 'default',
cost_7d_usd: 0.30,
eval_pass_count: 3,
eval_fail_count: 0,
halt_count: 5,
round_completed_count: 5,
last_updated_at: '2026-05-27T13:00:00Z',
},
{
kind: 'concepts',
source_id: 'default',
cost_7d_usd: 0.10,
eval_pass_count: 1,
eval_fail_count: 0,
halt_count: 1,
round_completed_count: 9,
last_updated_at: '2026-05-27T12:00:00Z',
},
];
describe('buildStatusReport — pure aggregation', () => {
test('schema_version stamped', () => {
const report = buildStatusReport([], {});
expect(report.schema_version).toBe(1);
});
test('halt_rate computed correctly from halt + completed counts', () => {
const report = buildStatusReport(SAMPLE_ROWS, {});
const atoms = report.rows.find(r => r.kind === 'atoms')!;
// 5 halts + 5 completed = 50% halt rate
expect(atoms.halt_rate).toBe(0.5);
const fc = report.rows.find(r => r.kind === 'facts.conversation')!;
// 0 halts + 10 completed = 0% halt rate
expect(fc.halt_rate).toBe(0);
});
test('sorts by halt_rate desc, then cost desc', () => {
const report = buildStatusReport(SAMPLE_ROWS, {});
// atoms (50% halt) should come first; then concepts (10%); then facts.conv (0%)
expect(report.rows.map(r => r.kind)).toEqual(['atoms', 'concepts', 'facts.conversation']);
});
test('zero-completed + zero-halt rows have halt_rate 0 (not NaN)', () => {
const report = buildStatusReport(
[{
kind: 'empty', source_id: 'default',
cost_7d_usd: 0, eval_pass_count: 0, eval_fail_count: 0,
halt_count: 0, round_completed_count: 0,
last_updated_at: null,
}],
{},
);
expect(report.rows[0].halt_rate).toBe(0);
});
test('coerces string-typed counts (postgres returns SUM() as string)', () => {
const report = buildStatusReport(
[{
kind: 'atoms', source_id: 'default',
cost_7d_usd: '0.50' as unknown as number,
eval_pass_count: '3' as unknown as number,
eval_fail_count: '0' as unknown as number,
halt_count: '2' as unknown as number,
round_completed_count: '8' as unknown as number,
last_updated_at: null,
}],
{},
);
expect(report.rows[0].cost_7d_usd).toBe(0.5);
expect(report.rows[0].halt_count).toBe(2);
expect(report.rows[0].halt_rate).toBe(0.2);
});
test('last_updated_at coerces to ISO string (engine returns Date)', () => {
const dateObj = new Date('2026-05-27T10:00:00.000Z');
const report = buildStatusReport(
[{
kind: 'atoms', source_id: 'default',
cost_7d_usd: 0, eval_pass_count: 0, eval_fail_count: 0,
halt_count: 0, round_completed_count: 1,
last_updated_at: dateObj,
}],
{},
);
expect(report.rows[0].last_updated_at).toBe('2026-05-27T10:00:00.000Z');
});
test('null last_updated_at stays null', () => {
const report = buildStatusReport(
[{
kind: 'a', source_id: 'b',
cost_7d_usd: 0, eval_pass_count: 0, eval_fail_count: 0,
halt_count: 0, round_completed_count: 0,
last_updated_at: null,
}],
{},
);
expect(report.rows[0].last_updated_at).toBeNull();
});
test('filters propagated', () => {
const report = buildStatusReport(SAMPLE_ROWS, {
source_id: 'media-corpus',
kind: 'atoms',
});
expect(report.filters.source_id).toBe('media-corpus');
expect(report.filters.kind).toBe('atoms');
});
});
describe('formatStatusTable — human output', () => {
test('empty rows returns informative message', () => {
const report = buildStatusReport([], {});
expect(formatStatusTable(report, false)).toContain('No extract events');
});
test('empty with filters mentions the filters', () => {
const report = buildStatusReport([], { source_id: 'media', kind: 'atoms' });
const out = formatStatusTable(report, false);
expect(out).toContain('source=media');
expect(out).toContain('kind=atoms');
});
test('header row contains expected columns', () => {
const report = buildStatusReport(SAMPLE_ROWS, {});
const out = formatStatusTable(report, true);
expect(out).toContain('KIND');
expect(out).toContain('SOURCE');
expect(out).toContain('COST_7D_USD');
expect(out).toContain('HALT_RATE');
});
test('non-verbose truncates to top 5 with "more rows" hint', () => {
const manyRows: ExtractStatusRow[] = Array.from({ length: 8 }, (_, i) => ({
kind: `kind${i}`,
source_id: 'default',
cost_7d_usd: 0,
eval_pass_count: 0,
eval_fail_count: 0,
halt_count: 0,
round_completed_count: 1,
halt_rate: 0,
last_updated_at: null,
}));
const report = {
schema_version: 1 as const,
rows: manyRows,
filters: {},
};
const out = formatStatusTable(report, false);
expect(out).toContain('kind0');
expect(out).toContain('kind4');
expect(out).not.toContain('kind5'); // truncated
expect(out).toContain('+3 more rows');
});
test('verbose shows all rows', () => {
const manyRows: ExtractStatusRow[] = Array.from({ length: 8 }, (_, i) => ({
kind: `kind${i}`,
source_id: 'default',
cost_7d_usd: 0,
eval_pass_count: 0,
eval_fail_count: 0,
halt_count: 0,
round_completed_count: 1,
halt_rate: 0,
last_updated_at: null,
}));
const report = {
schema_version: 1 as const,
rows: manyRows,
filters: {},
};
const out = formatStatusTable(report, true);
expect(out).toContain('kind7');
expect(out).not.toContain('more rows');
});
});