Files
gbrain/test
0413c93e72 feat: CJK entity extraction for Chinese/Japanese/Korean names (#1637)
* feat: CJK entity extraction for Chinese/Japanese/Korean names

- Add hasCJK() / cjkCharCount() detection helpers
- Lower min name length for CJK entities from 4 to 2 chars
- Fix tokenizeTitle() to handle pure CJK titles as single tokens
  (was returning [] for CJK-only titles, excluding them from gazetteer)
- Add CJK substring matching pass in findMentionedEntities
- NER extraction works without schema pack (plain mentions fallback)

Verified: gbrain extract links --by-mention creates 27 links from
456 pages with 3 CJK entity pages in gazetteer.

* feat: Chinese link type inference + timeline date formats

Link types:
- CN_FOUNDED_RE: 创立/创办/成立/创建 → founded
- CN_INVESTED_RE: 投资/入股/融资 → invested_in
- CN_ADVISES_RE: 顾问/咨询/指导 → advises
- CN_WORKS_AT_RE: 任职/就职/担任 → works_at
- CN_CITED_RE: 引用/提到/提及 → cited

Timeline:
- TIMELINE_LINE_RE_CN: YYYY年M月D日 | event
- Auto-normalizes to YYYY-MM-DD format
- Falls through to English format if CN doesn't match

* fix: CJK tokenizer uses char-level tokens (reviewer feedback)

Addresses all 4 concerns from review of PR #1637:

1. tokenizeForScan now emits CJK characters as individual tokens
   — normal scan path reaches CJK gazetteer entries naturally,
     eliminating the separate O(P×C×N) substring fallback pass.

2. tokenizeTitle splits pure CJK titles into individual chars
   — e.g. '纳瓦尔' → ['纳','瓦','尔'], matching body-level CJK tokens.

3. Removed O(P×C×N) CJK substring pass — no longer needed.
   Performance now O(P × N_tokens) for both ASCII and CJK.

4. Renamed CN_*_RE → ZH_*_RE in link-extraction.ts with a comment
   clarifying these are Chinese-only (entity NAME extraction in
   by-mention.ts covers CJK scripts, link TYPE extraction is zh only).

Added 12 CJK-specific tests (10 pure + 2 engine integration).
All 51 existing + new tests pass.

* review-repair(#1637): scope CN timeline regex to 年月日, revert off-scope extract-ner no-pack change, cosmetics

- TIMELINE_LINE_RE_CN required only [年-] separators, so non-bold ASCII
  dates (- 2020-01-02 - text) started parsing as timeline entries — an
  English-default regression. Now requires the 年/月 markers.
- Dropped the dead 'm = cm as any' assignment.
- src/core/extract-ner.ts reverted to origin/master: the no-pack →
  plain-mentions walk was off-scope for a CJK PR, duplicated the
  existing --by-mention pass, and hardcoded pack_unavailable:false
  (breaking the CLI hint).
- by-mention.ts: fixed stray indentation + restored EOF newline.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Garry Tan <garrytan@gmail.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Time Attakc <89218912+time-attack@users.noreply.github.com>
2026-07-27 18:03:31 -07:00
..