mirror of
https://github.com/garrytan/gbrain.git
synced 2026-07-29 19:01:39 +00:00
38cc7198b7
Adds the bold-time-dash built-in pattern: **Speaker** HH:MM <dash> text (em dash, en dash, or ASCII hyphen), valid 24-hour times only, date from page frontmatter/date headings, multi-line continuation bodies. Opt-in score_continuations_as_body scoring keeps long multiline messages parseable while preserving the sparse-prose false-positive floor (needs two anchors or a first-line anchor before candidate-only scoring kicks in). Hardens validatePatternEntry to reject non-integer / out-of-range capture indexes including text_group. Adds maintainer doc, JSONL fixtures, and adversarial coverage. Takeover of #3289 (fork branch went CONFLICTING against master on the version trio); code applied 3-way, version/CHANGELOG bump dropped per fleet release convention. Co-authored-by: Garry Tan <garrytan@gmail.com> Co-authored-by: FloridaStyle <daniel.wiggins@gmail.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
147 lines
6.2 KiB
Markdown
147 lines
6.2 KiB
Markdown
# Conversation parser patterns
|
||
|
||
The conversation parser turns exported chat and meeting transcripts into a
|
||
common message stream without requiring an LLM call for known formats. This
|
||
document describes the built-in pattern contract and the checks required when
|
||
adding or changing a format.
|
||
|
||
## Data flow
|
||
|
||
`parseConversation` uses this sequence:
|
||
|
||
1. Resolve the page date and timezone context.
|
||
2. Score every enabled built-in and user pattern against the first ten
|
||
non-blank lines.
|
||
3. Re-score the full body when the head score is inconclusive, or when a broad
|
||
pattern explicitly requires full-body scoring.
|
||
4. Reject the winner when its acceptance score is below the false-positive
|
||
floor.
|
||
5. Apply the winning pattern to every line and attach continuation lines to the
|
||
preceding message.
|
||
6. Optionally run LLM polish or fallback when those features are enabled.
|
||
|
||
Pattern order is only a tie-breaker. A new regex must be structurally distinct
|
||
from neighboring formats; moving it earlier in the registry is not a valid
|
||
non-shadowing strategy.
|
||
|
||
## Built-in pattern contract
|
||
|
||
Every `PatternEntry` in `builtins.ts` declares:
|
||
|
||
- A stable, kebab-case `id`.
|
||
- A hand-vetted line regex and explicit capture-group indexes.
|
||
- Where the date comes from and how the time is represented.
|
||
- A timezone policy.
|
||
- Whether the format supports multi-line message bodies.
|
||
- Positive and negative samples that run during module initialization.
|
||
- A documentation pointer describing the source format.
|
||
|
||
The registry refuses to load when a positive sample stops matching, a negative
|
||
sample starts matching, or a capture map becomes invalid. This catches local
|
||
regex mistakes before extraction can silently produce empty conversations.
|
||
|
||
### Date and timezone rules
|
||
|
||
Formats with an inline date should capture it from each message. Time-only
|
||
formats use an explicit caller fallback first, then the page frontmatter date,
|
||
then the page effective date. If none is available, the parser uses
|
||
`1970-01-01` so the missing date remains visible instead of inventing a current
|
||
date.
|
||
|
||
Time-only formats normally use `utc_assumed_with_warn`. The parser constructs a
|
||
UTC timestamp and returns a timezone warning when the page does not provide a
|
||
timezone. A new pattern should not imply local-time precision that the source
|
||
format does not contain.
|
||
|
||
### Multi-line messages
|
||
|
||
An anchor regex identifies the first line of a message. Subsequent non-anchor
|
||
lines are appended to that message until another anchor appears. Set
|
||
`multi_line: true` when continuation content is part of the documented format,
|
||
such as Markdown bullets, blockquotes, or an exported message body on the next
|
||
line.
|
||
|
||
Tests for a multi-line format should assert the complete message text, including
|
||
newlines. A message-count assertion alone will not detect lost bullets or a
|
||
continuation attached to the wrong speaker.
|
||
|
||
### Scoring and false positives
|
||
|
||
The score compares matched anchors with the pattern's relevant candidate lines.
|
||
The first pass uses the head of the page for speed. Low-confidence pages are
|
||
re-scored across the full body before the parser accepts a winner.
|
||
|
||
Multi-line formats may opt into `score_continuations_as_body` when their anchor
|
||
grammar is distinctive. Candidate-only scoring activates only after two anchors
|
||
match, or when the first non-blank line is an anchor. This evidence threshold
|
||
lets a single long message keep its continuation body without turning one stray
|
||
anchor in a prose page into a conversation. Candidate anchor lines that fail the
|
||
full regex still lower the score. Other patterns continue to use all non-blank
|
||
lines in their density score.
|
||
|
||
Use `score_full_body: true` for a broad grammar that also occurs in ordinary
|
||
prose. For example, `**Label:** text` can be either a transcript line or a bold
|
||
label in meeting notes. Narrow formats with a timestamp and a distinctive
|
||
separator generally do not need this override.
|
||
|
||
`quick_reject` is a performance hint, not an acceptance rule. It should cheaply
|
||
exclude obviously unrelated lines while admitting every string accepted by the
|
||
main regex.
|
||
|
||
## Normalized Slack Markdown
|
||
|
||
The `bold-time-dash` pattern parses message anchors shaped like:
|
||
|
||
```text
|
||
**Alice Example** 09:15 — first message
|
||
- supporting detail
|
||
**Bob Example** 09:18 — second message
|
||
```
|
||
|
||
Its grammar is:
|
||
|
||
```text
|
||
**speaker** H:MM <dash> text
|
||
```
|
||
|
||
where:
|
||
|
||
- `H:MM` is a valid 24-hour time from `0:00` through `23:59`.
|
||
- `<dash>` may be an em dash (`—`), en dash (`–`), or ASCII hyphen (`-`).
|
||
- The date comes from the resolved page date context.
|
||
- Continuation lines belong to the preceding message.
|
||
- The captured clock value is emitted with `Z`. Timezone metadata suppresses
|
||
the missing-timezone warning but is not currently used for IANA conversion.
|
||
|
||
The required time and dash distinguish it from all existing bold-speaker
|
||
formats:
|
||
|
||
- `**Speaker** (09:15): text` uses `bold-paren-time`.
|
||
- `**Speaker** (9:15 AM): text` uses `bold-paren-time-12h`.
|
||
- `**Speaker:** text` uses `bold-name-no-time`.
|
||
- `**Speaker** (2026-04-09 9:15 AM): text` uses `imessage-slack`.
|
||
|
||
Keeping these examples in both `test_negative` and parser regression tests makes
|
||
the non-shadowing contract executable.
|
||
|
||
## Adding a built-in format
|
||
|
||
1. Collect multiple anonymized examples, including separator and timestamp
|
||
variants that occur in the same export family.
|
||
2. Choose the narrowest grammar that represents the format. Constrain numeric
|
||
fields such as hours and minutes when possible.
|
||
3. Add at least two positive module-load samples and negative samples for every
|
||
neighboring pattern that could plausibly overlap.
|
||
4. Add parser tests that verify speakers, timestamps, text, continuation
|
||
handling, and non-shadowing behavior.
|
||
5. Add a dedicated JSONL fixture and include the same cases in
|
||
`test/fixtures/conversation-formats/all.jsonl`.
|
||
6. Run the focused parser tests and the fixture evaluator.
|
||
7. Run the repository verification and full test suites before submission.
|
||
8. Update `docs/architecture/KEY_FILES.md` when the registry count or supported
|
||
format inventory changes.
|
||
|
||
Use generic fixture identities such as `Alice Example`, `Bob Example`, and
|
||
`Summary Bot`. Never copy real transcript names or private content into source,
|
||
tests, documentation, commits, or pull-request descriptions.
|