Files
gbrain/test/conversation-parser/parse.test.ts
T
041d89babe v0.41.29.0 feat(conversation-parser): bold-name-no-time builtin + fix(orphans): source-scoped orphan_ratio (supersedes #1613) (#1620)
* feat(conversation-parser): add bold-name-no-time builtin (Circleback/Granola/Zoom, no timestamp)

The 14th built-in pattern parses `**Speaker:** text` transcripts with NO
per-line timestamp — the shape Circleback / Granola / Zoom emit. Every prior
builtin required a time anchor, so this shape matched nothing: a production
brain had 104 conversation pages + 3,423 eligible pages silently extracting
zero facts. Messages anchor at T00:00:00Z of the frontmatter date (no
fabricated wall-clock; line order preserves sequence), same convention as
irc-classic.

Hardening beyond the original community proposal:
- regex `/^\*\*(?!\[)(.+?):\*\*\s*(.*)$/`: the colon-inside-bold (NOT
  declaration order) is what prevents shadowing bold-paren-time; the `(?!\[)`
  lookahead rejects telegram-bracket `**[18:37] Name:**` so disabling
  telegram-bracket yields an honest no_match instead of speaker="[18:37] Name".
- new optional PatternEntry.score_full_body: `**Label:** text` is a common
  prose idiom, so a notes page with bold labels clustered in its first 10
  lines scored 0.3 on the head pass (NOT < SCORING_HEAD_TRIGGER_THRESHOLD, so
  the full-body fallback never fired) and cleared the 0.05 floor. parse.ts now
  recomputes the winner's score over the full body before the floor, so such a
  page drops to its true low density and stays no_match.
- scrubbed pre-existing real names from bold-paren-time test_positive samples
  (privacy rule).

Fixtures use placeholder names only. Pinned by new bold-name-no-time +
clustered-head no_match cases in parse.test.ts and the eval corpus.

Co-Authored-By: garrytan-agents <noreply@github.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(orphans): scope orphan_ratio + find_orphans by source; fix total_linkable denominator

`gbrain doctor --source <id>` and `gbrain orphans --source <id>` now scope
the orphan scan to that source instead of reporting brain-wide. Three fixes:

- findOrphanPages(opts?: { sourceId?, sourceIds? }) on both engines scopes the
  CANDIDATE set (scalar `= $1` or federated `= ANY($1::text[])`). Inbound links
  from ANY source still count, so a page in source X linked FROM source Y is
  reachable and NOT an orphan of X (the deliberate, less-surprising definition).
- corrected the total_linkable denominator in findOrphans: it now enumerates
  all live pages (scoped) and subtracts every excluded-by-slug page, not just
  excluded orphans. The old `total - excludedOrphans` left excluded NON-orphan
  pages (templates/, scratch/) with inbound links in the denominator, inflating
  it and suppressing warnings. Changes orphan_ratio output for every brain, in
  the accurate direction.
- the find_orphans MCP op threads sourceScopeOpts(ctx), closing a cross-source
  read leak where a source-bound OAuth client saw brain-wide orphans (v0.34.1
  source-isolation class).

doctor uses an explicit `--source` flag parse (NOT resolveSourceWithTier, which
would scope bare invocations to a default), and under explicit --source reports
the ratio with a low-scale caveat below 100 entity pages instead of a vacuous
"ok". Thin-client doctor --source orphan_ratio deferred (TODOS.md).

Pinned by test/orphans-source-scope.test.ts (PGLite: scoping, cross-source
inbound, denominator, find_orphans op scope) + a Postgres↔PGLite parity case
in test/e2e/engine-parity.test.ts (scalar + federated binding).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* docs: v0.41.29.0 — bold-name-no-time + orphan source scoping

VERSION + package.json → 0.41.29.0; CHANGELOG entry; CLAUDE.md conversation-parser
(13→14 patterns) + orphans source-scoping notes; regenerated llms bundles; TODOS
for thin-client doctor --source + check-test-real-names widening.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: garrytan-agents <noreply@github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-05-29 07:06:08 -07:00

716 lines
30 KiB
TypeScript
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
/**
* v0.41.16.0 — Conversation parser orchestrator tests.
*
* Covers:
* - PR #1461's 6 telegram-bracket cases verbatim (REGRESSION pin)
* - All 12 built-in patterns hit their test_positive samples
* - Date derivation precedence (D8)
* - Pattern priority scoring (D18) — overlap resolution
* - Quick-reject fast path (D11)
* - Multi-line continuation (D5)
* - Disabled-builtin honored
* - Timezone warning (D19) emitted when frontmatter timezone missing
*
* Pure-function tests; no PGLite, no LLM. The LLM polish/fallback
* tests live in `llm-base.test.ts`, `llm-polish.test.ts`,
* `llm-fallback.test.ts` (T4).
*/
import { describe, expect, test } from 'bun:test';
import {
parseConversation,
deriveDateContext,
applyPattern,
scorePattern,
scorePatternFull,
} from '../../src/core/conversation-parser/parse.ts';
import { BUILTIN_PATTERNS } from '../../src/core/conversation-parser/builtins.ts';
import type { Page } from '../../src/core/types.ts';
// Helper to construct a minimal Page for date-derivation tests.
function makePage(
frontmatter: Record<string, unknown> = {},
effective_date?: Date,
): Page {
return {
id: 1,
slug: 'test/page',
type: 'conversation',
title: 'Test',
compiled_truth: '',
timeline: '',
frontmatter,
content_hash: undefined,
created_at: new Date(),
updated_at: new Date(),
effective_date: effective_date ?? null,
} as Page;
}
// ---------------------------------------------------------------------------
// REGRESSION: PR #1461's 6 telegram-bracket cases verbatim
// ---------------------------------------------------------------------------
describe('parseConversation — REGRESSION PR #1461 (telegram-bracket)', () => {
test('bracket-time with 👤 emoji speaker prefix', () => {
const body = '**[18:37] \u{1f464} G T:** hello world';
const r = parseConversation(body, { fallbackDate: '2026-05-24' });
expect(r.messages).toHaveLength(1);
expect(r.messages[0].speaker).toBe('G T');
expect(r.messages[0].text).toBe('hello world');
expect(r.messages[0].timestamp).toBe('2026-05-24T18:37:00Z');
expect(r.matched_pattern_id).toBe('telegram-bracket');
});
test('bracket-time with 🤖 robot emoji', () => {
const body = '**[06:00] \u{1f916} Zion:** On it.';
const r = parseConversation(body, { fallbackDate: '2026-05-25' });
expect(r.messages).toHaveLength(1);
expect(r.messages[0].speaker).toBe('Zion');
expect(r.messages[0].text).toBe('On it.');
expect(r.messages[0].timestamp).toBe('2026-05-25T06:00:00Z');
});
test('bracket-time multi-line continuation', () => {
const body = [
'**[09:00] \u{1f464} Alice Example:** first line',
'second line of same message',
'**[09:05] \u{1f464} Bob Example:** separate message',
].join('\n');
const r = parseConversation(body, { fallbackDate: '2026-05-20' });
expect(r.messages).toHaveLength(2);
expect(r.messages[0].text).toBe('first line\nsecond line of same message');
expect(r.messages[1].text).toBe('separate message');
});
test('bracket-time falls back to 1970-01-01 without fallbackDate', () => {
const body = '**[14:30] \u{1f464} Alice Example:** test';
const r = parseConversation(body);
expect(r.messages).toHaveLength(1);
expect(r.messages[0].timestamp).toBe('1970-01-01T14:30:00Z');
});
test('mixed formats in one body: iMessage + bracket-time', () => {
const body = [
'**Alice Example** (2024-03-15 9:00 AM): format 1',
'**[10:30] \u{1f464} Bob Example:** format 2',
].join('\n');
const r = parseConversation(body, { fallbackDate: '2024-03-15' });
// D18 scoring picks the dominant pattern. Both have one hit; ties
// resolve to declared priority (imessage-slack=0, telegram-bracket=1).
// The imessage line matches; the telegram line becomes a
// continuation. This is a known D18 tradeoff for very-mixed bodies.
// In practice every chat export is homogeneous, so this is a
// degenerate test case.
expect(r.messages.length).toBeGreaterThanOrEqual(1);
expect(r.matched_pattern_id).toBe('imessage-slack');
});
test('bracket-time without emoji prefix', () => {
const body = '**[22:15] Plain Name:** no emoji';
const r = parseConversation(body, { fallbackDate: '2026-01-01' });
expect(r.messages).toHaveLength(1);
expect(r.messages[0].speaker).toBe('Plain Name');
expect(r.messages[0].text).toBe('no emoji');
});
});
// ---------------------------------------------------------------------------
// All 12 built-ins must parse their test_positive samples
// ---------------------------------------------------------------------------
describe('parseConversation — every built-in matches its test_positive sample', () => {
for (const entry of BUILTIN_PATTERNS) {
test(`pattern ${entry.id}: first test_positive parses`, () => {
const body = entry.test_positive[0];
// Multi-line patterns need a body line on the next line.
const fullBody =
entry.multi_line && entry.captures.text_group === 0
? `${body}\nsome body text`
: body;
const r = parseConversation(fullBody, {
fallbackDate: '2024-03-15',
});
// Either matches a message OR (for some multi-line patterns) the
// first line is the anchor and the body is consumed as text.
expect(r.messages.length).toBeGreaterThanOrEqual(1);
expect(r.matched_pattern_id).toBe(entry.id);
});
}
});
// ---------------------------------------------------------------------------
// Date derivation precedence (D8)
// ---------------------------------------------------------------------------
describe('deriveDateContext (D8 precedence chain)', () => {
test('explicit fallbackDate wins', () => {
const page = makePage({ date: '2024-01-01' }, new Date('2023-06-15'));
const ctx = deriveDateContext({ fallbackDate: '2025-12-25', page });
expect(ctx.fallbackDate).toBe('2025-12-25');
expect(ctx.source).toBe('explicit');
});
test('frontmatter.date wins over effective_date', () => {
const page = makePage({ date: '2024-01-01' }, new Date('2023-06-15'));
const ctx = deriveDateContext({ page });
expect(ctx.fallbackDate).toBe('2024-01-01');
expect(ctx.source).toBe('frontmatter_date');
});
test('effective_date wins when no frontmatter.date', () => {
const page = makePage({}, new Date('2023-06-15T00:00:00Z'));
const ctx = deriveDateContext({ page });
expect(ctx.fallbackDate).toBe('2023-06-15');
expect(ctx.source).toBe('effective_date');
});
test('epoch_default when nothing set', () => {
const ctx = deriveDateContext({});
expect(ctx.fallbackDate).toBe('1970-01-01');
expect(ctx.source).toBe('epoch_default');
});
test('frontmatter.timezone surfaces', () => {
const page = makePage({
date: '2024-01-01',
timezone: 'America/Los_Angeles',
});
const ctx = deriveDateContext({ page });
expect(ctx.timezone).toBe('America/Los_Angeles');
});
test('invalid frontmatter.date falls through', () => {
const page = makePage({ date: 'not-a-date' });
const ctx = deriveDateContext({ page });
expect(ctx.source).toBe('epoch_default');
});
test('frontmatter.date slices full ISO to YYYY-MM-DD', () => {
const page = makePage({ date: '2024-03-15T18:37:00Z' });
const ctx = deriveDateContext({ page });
expect(ctx.fallbackDate).toBe('2024-03-15');
});
});
// ---------------------------------------------------------------------------
// Pattern priority scoring (D18)
// ---------------------------------------------------------------------------
describe('scorePattern (D18 priority scoring)', () => {
test('telegram-bracket scores 1.0 on a pure telegram body', () => {
const body = [
'**[18:37] \u{1f464} Alice:** one',
'**[18:38] \u{1f464} Bob:** two',
'**[18:39] \u{1f464} Alice:** three',
].join('\n');
const tg = BUILTIN_PATTERNS.find((p) => p.id === 'telegram-bracket')!;
expect(scorePattern(body, tg)).toBe(1);
});
test('imessage-slack scores 0 on pure telegram body', () => {
const body = [
'**[18:37] \u{1f464} Alice:** one',
'**[18:38] \u{1f464} Bob:** two',
].join('\n');
const im = BUILTIN_PATTERNS.find((p) => p.id === 'imessage-slack')!;
expect(scorePattern(body, im)).toBe(0);
});
test('mixed body: D18 picks the higher-scoring pattern', () => {
// 3 telegram lines + 1 imessage line. Telegram should win.
const body = [
'**[18:37] \u{1f464} Alice:** one',
'**[18:38] \u{1f464} Bob:** two',
'**[18:39] \u{1f464} Alice:** three',
'**Charlie** (2024-03-15 9:00 AM): one imessage',
].join('\n');
const r = parseConversation(body, { fallbackDate: '2024-03-15' });
expect(r.matched_pattern_id).toBe('telegram-bracket');
});
});
// ---------------------------------------------------------------------------
// Disabled-builtin honored
// ---------------------------------------------------------------------------
describe('parseConversation — disabledBuiltinIds', () => {
test('disabling top pattern falls through to next', () => {
const body = '**[18:37] \u{1f464} Alice:** hello';
const rDefault = parseConversation(body, { fallbackDate: '2024-03-15' });
expect(rDefault.matched_pattern_id).toBe('telegram-bracket');
const rDisabled = parseConversation(body, {
fallbackDate: '2024-03-15',
disabledBuiltinIds: ['telegram-bracket'],
});
// No other built-in matches this exact shape → no_match.
expect(rDisabled.phase).toBe('no_match');
});
});
// ---------------------------------------------------------------------------
// Multi-line continuation (D5)
// ---------------------------------------------------------------------------
describe('parseConversation — multi-line continuation (D5)', () => {
test('iMessage continuation absorbs orphan lines', () => {
const body = [
'**Alice Example** (2024-03-15 9:00 AM): first line',
'continuation line',
'another continuation',
'**Bob Example** (2024-03-15 9:05 AM): second message',
].join('\n');
const r = parseConversation(body);
expect(r.messages).toHaveLength(2);
expect(r.messages[0].text).toBe(
'first line\ncontinuation line\nanother continuation',
);
expect(r.messages[1].text).toBe('second message');
});
});
// ---------------------------------------------------------------------------
// Timezone warning (D19)
// ---------------------------------------------------------------------------
describe('parseConversation — timezone warning (D19)', () => {
test('telegram-bracket emits warning when no timezone in frontmatter', () => {
const body = '**[18:37] \u{1f464} Alice:** hello';
const r = parseConversation(body, { fallbackDate: '2024-03-15' });
expect(r.timezone_warning).toBeDefined();
expect(r.timezone_warning).toContain('telegram-bracket');
expect(r.timezone_warning).toContain('UTC');
});
test('telegram-bracket does NOT warn when timezone is present', () => {
const body = '**[18:37] \u{1f464} Alice:** hello';
const page = makePage({
date: '2024-03-15',
timezone: 'America/Los_Angeles',
});
const r = parseConversation(body, { page });
expect(r.timezone_warning).toBeUndefined();
});
test('imessage-slack does NOT warn (inline_utc policy)', () => {
const body = '**Alice Example** (2024-03-15 9:00 AM): hello';
const r = parseConversation(body);
expect(r.timezone_warning).toBeUndefined();
});
});
// ---------------------------------------------------------------------------
// Empty body + degenerate cases
// ---------------------------------------------------------------------------
describe('parseConversation — degenerate inputs', () => {
test('empty body returns no_match + empty messages', () => {
expect(parseConversation('')).toEqual({
messages: [],
phase: 'no_match',
});
});
test('non-conversational text returns no_match', () => {
const r = parseConversation('This is just prose with no chat shape.');
expect(r.phase).toBe('no_match');
expect(r.messages).toHaveLength(0);
});
});
// ---------------------------------------------------------------------------
// applyPattern — direct unit tests for the matcher
// ---------------------------------------------------------------------------
describe('applyPattern — quick_reject fast path (D11)', () => {
test('telegram quick_reject skips iMessage lines fast', () => {
const body = '**Alice Example** (2024-03-15 9:00 AM): hello';
const tg = BUILTIN_PATTERNS.find((p) => p.id === 'telegram-bracket')!;
const r = applyPattern(body, tg, {
fallbackDate: '2024-03-15',
source: 'explicit',
});
// Quick_reject /^\*\*\[/ rejects '**Alice' (no `[`). Zero matches.
expect(r).toHaveLength(0);
});
});
// ---------------------------------------------------------------------------
// scorePattern boundary cases
// ---------------------------------------------------------------------------
describe('scorePattern — boundary', () => {
test('empty body scores 0', () => {
const tg = BUILTIN_PATTERNS.find((p) => p.id === 'telegram-bracket')!;
expect(scorePattern('', tg)).toBe(0);
});
test('only blank lines scores 0', () => {
const tg = BUILTIN_PATTERNS.find((p) => p.id === 'telegram-bracket')!;
expect(scorePattern('\n\n \n', tg)).toBe(0);
});
// T5 reshape (Codex P2 #6): pins BEHAVIOR not the constant value.
// The prior test ("100 matching lines score 1.0") would pass with
// head=10 or head=1000 — it didn't prove anything about the cap.
test('head cap ignores lines past line 10 (10 match + 1 non-match scores 1.0)', () => {
const tg = BUILTIN_PATTERNS.find((p) => p.id === 'telegram-bracket')!;
const matching = Array.from(
{ length: 10 },
(_, i) => `**[18:${String(i).padStart(2, '0')}] \u{1f464} Alice:** msg ${i}`,
);
const body = [...matching, 'plain text outside the head window'].join('\n');
// First 10 lines all match → 10/10. Line 11 was ignored.
expect(scorePattern(body, tg)).toBe(1);
});
test('head cap stops at line 10 (9 non-match + 1 match at line 10 + 100 match after scores 0.1)', () => {
const tg = BUILTIN_PATTERNS.find((p) => p.id === 'telegram-bracket')!;
const nonMatches = Array.from({ length: 9 }, (_, i) => `non-matching prose line ${i}`);
const matchingLate = Array.from(
{ length: 100 },
(_, i) => `**[18:${String(i).padStart(2, '0')}] \u{1f464} Alice:** msg ${i}`,
);
// Line 10 (index 9 in the matching array) IS a match; lines 11-109 are too
// but are past the head cap and don't count.
const body = [...nonMatches, matchingLate[0], ...matchingLate.slice(1)].join('\n');
// Head sees 9 non-matches + 1 match = 1/10 = 0.1. Pre-fix: same result.
// Post-fix: same result (this test pins head-cap behavior, not the new
// fallback path — that's tested separately below).
expect(scorePattern(body, tg)).toBe(0.1);
});
});
// ---------------------------------------------------------------------------
// scorePatternFull — direct unit tests (v0.41.18+ T3 #5)
// ---------------------------------------------------------------------------
describe('scorePatternFull — full-body scoring (v0.41.18+ Codex P1 #1)', () => {
test('empty body scores 0', () => {
const im = BUILTIN_PATTERNS.find((p) => p.id === 'imessage-slack')!;
expect(scorePatternFull('', im)).toBe(0);
});
test('preamble + 20 matching lines scores 20/(preamble + 20)', () => {
const im = BUILTIN_PATTERNS.find((p) => p.id === 'imessage-slack')!;
const preamble = ['## Summary', 'Three sentences.', '> Source: ref', '## Transcript'];
const matches = Array.from(
{ length: 20 },
(_, i) => `**Garry Tan** (2026-01-29 12:00 PM): message ${i}`,
);
const body = [...preamble, ...matches].join('\n');
// 24 total non-blank, 20 match → 20/24 ≈ 0.833
expect(scorePatternFull(body, im)).toBeCloseTo(20 / 24, 5);
});
test('preamble-only-no-match scores 0', () => {
const im = BUILTIN_PATTERNS.find((p) => p.id === 'imessage-slack')!;
const body = '## Summary\nProse paragraph.\n> Blockquote\n## Heading';
expect(scorePatternFull(body, im)).toBe(0);
});
});
// ---------------------------------------------------------------------------
// bold-paren-time pattern (v0.41.18+ D-FOLLOWUP-1.B; closes user-facing
// half of #1533 — the 112 Circleback meeting files at
// ~/git/brain/meetings/*.md with `source: circleback` frontmatter)
// ---------------------------------------------------------------------------
describe('bold-paren-time pattern (Circleback meeting transcripts)', () => {
test('matches **Speaker** (HH:MM): text with frontmatter date', () => {
const body = [
'**Alice Example** (00:00): Hey, can you hear me?',
'**Participant 2** (02:22): Yeah, just joined.',
'**Alice Example** (15:09): That makes sense.',
].join('\n');
const r = parseConversation(body, { fallbackDate: '2026-03-19' });
expect(r.phase).toBe('regex_match');
expect(r.matched_pattern_id).toBe('bold-paren-time');
expect(r.messages).toHaveLength(3);
expect(r.messages[0]).toEqual({
speaker: 'Alice Example',
timestamp: '2026-03-19T00:00:00Z',
text: 'Hey, can you hear me?',
});
expect(r.messages[2]).toEqual({
speaker: 'Alice Example',
timestamp: '2026-03-19T15:09:00Z',
text: 'That makes sense.',
});
});
test('matches **Speaker** (HH:MM:SS): text shape (Circleback seconds variant)', () => {
const body = [
'**Participant 1** (00:00:00): opening line',
'**Participant 2** (00:00:19): quick reply',
'**Participant 1** (01:23:45): later in the meeting',
].join('\n');
const r = parseConversation(body, { fallbackDate: '2026-04-01' });
expect(r.phase).toBe('regex_match');
expect(r.matched_pattern_id).toBe('bold-paren-time');
expect(r.messages).toHaveLength(3);
// Seconds segment is non-capturing; minute_group still captures the
// minutes component. Time-format is wall-clock 24h on frontmatter date.
expect(r.messages[0].timestamp).toBe('2026-04-01T00:00:00Z');
expect(r.messages[1].timestamp).toBe('2026-04-01T00:00:00Z');
expect(r.messages[2].timestamp).toBe('2026-04-01T01:23:00Z');
});
test('imessage-slack shape still wins over bold-paren-time on overlap', () => {
// Both patterns start with `**` and have parens. The imessage-
// slack regex requires a full date+time inside; bold-paren-time
// requires just `(HH:MM)`. The dates-with-AM/PM shape MUST fall
// through to imessage-slack, not bold-paren-time.
const body = '**Alice Example** (2024-03-15 9:00 AM): hello world';
const r = parseConversation(body);
expect(r.phase).toBe('regex_match');
expect(r.matched_pattern_id).toBe('imessage-slack');
expect(r.messages).toHaveLength(1);
expect(r.messages[0].text).toBe('hello world');
});
test('meeting page with preamble + bold-paren-time transcript hits fallback', () => {
// Real Circleback shape: ## Summary + blockquote + ## Transcript
// before the bold-paren-time chat. Same fallback gate that
// closes #1533 must work for this pattern too.
const preamble = [
'## Summary',
'Meeting covered Q1 roadmap discussion.',
'> Source: circleback meeting #7411053',
'## Topics Discussed',
'- Roadmap',
'- Hiring',
'## Transcript',
];
const transcript = Array.from(
{ length: 20 },
(_, i) => `**Participant 2** (${String(Math.floor(i / 6)).padStart(2, '0')}:${String((i * 11) % 60).padStart(2, '0')}): transcript line ${i}`,
);
const body = [...preamble, ...transcript].join('\n');
const r = parseConversation(body, { fallbackDate: '2026-03-19' });
expect(r.phase).toBe('regex_match');
expect(r.matched_pattern_id).toBe('bold-paren-time');
expect(r.messages).toHaveLength(20);
});
});
// ---------------------------------------------------------------------------
// bold-name-no-time pattern (Circleback / Granola / Zoom transcripts with NO
// per-line timestamp — `**Speaker:** text`). Additive pattern; the colon
// inside the bold markers + the `(?!\[)` lookahead are what keep it from
// shadowing bold-paren-time / telegram-bracket (NOT declaration order).
// ---------------------------------------------------------------------------
describe('bold-name-no-time pattern (Circleback/Granola/Zoom, no timestamp)', () => {
test('parses **Speaker:** text transcript with frontmatter date anchor', () => {
const body = [
'**Alice Example:** Okay, start on. And then weirdly like zoom doesnt...',
'**Participant 2:** he tried to reset it remotely the other night. Let me ask him.',
'**Alice Example:** I mean its really just like we need to get zoom to fix this.',
'**Participant 2:** Okay, let me.',
].join('\n');
const r = parseConversation(body, { fallbackDate: '2026-05-28' });
expect(r.phase).toBe('regex_match');
expect(r.matched_pattern_id).toBe('bold-name-no-time');
expect(r.messages).toHaveLength(4);
expect(r.messages[0]).toEqual({
speaker: 'Alice Example',
timestamp: '2026-05-28T00:00:00Z',
text: 'Okay, start on. And then weirdly like zoom doesnt...',
});
expect(r.messages[1].speaker).toBe('Participant 2');
expect(r.messages[1].text).toBe(
'he tried to reset it remotely the other night. Let me ask him.',
);
expect(r.messages[3]).toEqual({
speaker: 'Participant 2',
timestamp: '2026-05-28T00:00:00Z',
text: 'Okay, let me.',
});
// No-time pattern anchors at 00:00:00 of the frontmatter date
// (same convention as irc-classic). No wall-clock time fabricated.
});
test('scores above the 0.05 floor on a pure bold-name transcript (epoch default)', () => {
const body = [
'**Alice Example:** line one',
'**Participant 2:** line two',
'**Alice Example:** line three',
].join('\n');
const r = parseConversation(body);
expect(r.phase).toBe('regex_match');
expect(r.matched_pattern_id).toBe('bold-name-no-time');
expect(r.messages).toHaveLength(3);
// No fallbackDate → epoch default, still anchors at 00:00:00.
expect(r.messages[0].timestamp).toBe('1970-01-01T00:00:00Z');
});
// REGRESSION: must NOT shadow bold-paren-time. A `**Name** (00:00): text`
// line has its colon OUTSIDE the bold markers, so it must still parse via
// bold-paren-time (the safety is the regex, not declaration order).
test('REGRESSION: **Speaker** (HH:MM): text still matches bold-paren-time', () => {
const body = [
'**Alice Example** (00:00): Hey, can you hear me?',
'**Participant 2** (02:22): Yeah, just joined.',
].join('\n');
const r = parseConversation(body, { fallbackDate: '2026-03-19' });
expect(r.phase).toBe('regex_match');
expect(r.matched_pattern_id).toBe('bold-paren-time');
expect(r.matched_pattern_id).not.toBe('bold-name-no-time');
expect(r.messages).toHaveLength(2);
expect(r.messages[0].timestamp).toBe('2026-03-19T00:00:00Z');
});
// REGRESSION: a pure telegram-bracket transcript has the colon inside the
// bold markers BUT the speaker starts with `[`, so the `(?!\[)` lookahead
// rejects it AND telegram-bracket (lower declaration index) wins anyway.
test('REGRESSION: telegram-bracket transcript still matches telegram-bracket', () => {
const body = [
'**[18:37] \u{1f464} Alice:** one',
'**[18:38] \u{1f464} Bob:** two',
].join('\n');
const r = parseConversation(body, { fallbackDate: '2024-03-15' });
expect(r.matched_pattern_id).toBe('telegram-bracket');
});
// F1 (Codex): the broad regex matches any `**Label:** text`. A prose notes
// page with bold labels CLUSTERED in its first 10 lines scores 0.3 on the
// head pass (3/10) — NOT < SCORING_HEAD_TRIGGER_THRESHOLD (0.3), so the
// full-body fallback never fires — and that 0.3 clears the 0.05 floor.
// Without score_full_body this page would mis-parse as a conversation.
// score_full_body recomputes the winner over the FULL body: 3 labels /
// 83 lines ≈ 0.036 < 0.05 → no_match. This is the exact bug Codex caught
// and the test the naive "0.05 floor protects us" assumption would fail.
test('F1: prose with bold labels clustered in the head returns no_match', () => {
const lines = [
'**Attendees:** Alice Example, Bob Example, Participant 2',
'**Date:** 2026-05-28',
'**Goal:** decide on the Q3 roadmap and unblock the vendor migration',
];
// 80 plain prose lines (no bold-label shape) → 83 total, only 3 match.
// First 10 lines = 3 labels + 7 prose → head score 0.3 (skips rescore).
// Full body = 3/83 ≈ 0.036 < 0.05 floor.
for (let i = 0; i < 80; i++) {
lines.push(
`This is an ordinary prose sentence number ${i} describing the meeting in detail.`,
);
}
const body = lines.join('\n');
const r = parseConversation(body, { fallbackDate: '2026-05-28' });
expect(r.phase).toBe('no_match');
});
});
// ---------------------------------------------------------------------------
// parseConversation — full-body fallback (v0.41.18+ #1533 + Codex P1 #1, #2, #8)
// ---------------------------------------------------------------------------
describe('parseConversation — full-body fallback', () => {
// T3 #1: IRON-RULE regression pin for #1533. Pre-fix this returns
// no_match because head 10 sees only preamble.
test('#1533: meeting page with ## Summary + blockquote + ## Transcript before chat hits fallback', () => {
const preamble = [
'## Summary',
'This meeting covered Q1 roadmap discussion.',
'Three engineers participated in the call.',
'Action items were captured during the conversation.',
'> Source: [meeting recording](https://example.com/rec/123)',
'## Topics Discussed',
'- Product roadmap for Q1',
'- Engineering team allocation',
'- Customer feedback synthesis',
'## Transcript',
];
const transcript = Array.from(
{ length: 20 },
(_, i) => `**Garry Tan** (2026-01-29 12:00 PM): line ${i}`,
);
const body = [...preamble, ...transcript].join('\n');
const r = parseConversation(body);
expect(r.phase).toBe('regex_match');
expect(r.matched_pattern_id).toBe('imessage-slack');
expect(r.messages).toHaveLength(20);
});
// T3 #2: diagnostic now reports total_non_blank - matched, not total.
test('#1533: unmatched_line_count subtracts matched messages after fallback', () => {
const preamble = [
'## Summary',
'Prose A.',
'Prose B.',
'> Blockquote',
'## Transcript',
];
const transcript = Array.from(
{ length: 20 },
(_, i) => `**Garry Tan** (2026-01-29 12:00 PM): line ${i}`,
);
const body = [...preamble, ...transcript].join('\n');
const r = parseConversation(body, { diagnostic: true });
expect(r.phase).toBe('regex_match');
expect(r.unmatched_line_count).toBe(5); // 25 total non-blank - 20 messages = 5
});
// T3 #3: a 50-line essay with no chat shape stays no_match.
test('pure-prose 50-line essay stays no_match (fallback found nothing to anchor)', () => {
const body = Array.from(
{ length: 50 },
(_, i) => `This is the ${i + 1}th paragraph of a pure-prose article.`,
).join('\n');
const r = parseConversation(body);
expect(r.phase).toBe('no_match');
expect(r.messages).toHaveLength(0);
});
// T3 #4: proves "full-body" not just "wider window" — 300-line preamble
// far exceeds any reasonable head-bump alternative.
test('300-line preamble + 50 chat lines hits fallback (any preamble length)', () => {
const preamble = Array.from(
{ length: 300 },
(_, i) => `Preamble paragraph ${i + 1} with prose content here.`,
);
const transcript = Array.from(
{ length: 50 },
(_, i) => `**Garry Tan** (2026-01-29 12:00 PM): chat line ${i}`,
);
const body = [...preamble, ...transcript].join('\n');
const r = parseConversation(body);
expect(r.phase).toBe('regex_match');
expect(r.matched_pattern_id).toBe('imessage-slack');
expect(r.messages).toHaveLength(50);
});
// T3 #6 (Codex P1 #1 + #8): stray-head-match doesn't suppress fallback.
// Pre-fix: irc-classic 0.1 in head → no fallback → irc-classic wins with 1
// message. Post-fix: 0.1 < 0.3 trigger → fallback re-scores → imessage-slack
// wins (50/60 ≈ 0.83 vs irc-classic 1/60 ≈ 0.017).
test('Codex P1 #1: stray irc-classic match in head does not suppress fallback', () => {
const preamble = [
'## Meeting Notes',
'<presenter> Garry Tan opening remarks', // stray irc-classic match
'- agenda item 1',
'- agenda item 2',
'- agenda item 3',
'- agenda item 4',
'- agenda item 5',
'- agenda item 6',
'- agenda item 7',
'## Transcript',
];
const transcript = Array.from(
{ length: 50 },
(_, i) => `**Garry Tan** (2024-01-29 12:00 PM): real transcript line ${i}`,
);
const body = [...preamble, ...transcript].join('\n');
const r = parseConversation(body);
expect(r.phase).toBe('regex_match');
// The critical assertion: imessage-slack wins, NOT irc-classic.
expect(r.matched_pattern_id).toBe('imessage-slack');
expect(r.messages).toHaveLength(50);
});
// T3 #7 (Codex P1 #2): essay with one stray chat-shape line stays
// no_match. 1/301 ≈ 0.003, below SCORING_MIN_ACCEPTANCE (0.05).
test('Codex P1 #2: 300-line essay with one stray chat line stays no_match (acceptance floor)', () => {
const prose = Array.from(
{ length: 150 },
(_, i) => `Essay paragraph ${i + 1} of pure prose with no chat shape.`,
);
const strayChatLine = '**Author Name** (2024-01-01 9:00 AM): stray quoted snippet';
const morePros = Array.from(
{ length: 150 },
(_, i) => `Essay continuation paragraph ${i + 151}.`,
);
const body = [...prose, strayChatLine, ...morePros].join('\n');
const r = parseConversation(body);
// Pre-fix: regex_match with messages.length === 1.
// Post-fix: no_match because 1/301 < 0.05 acceptance floor.
expect(r.phase).toBe('no_match');
expect(r.messages).toHaveLength(0);
});
});