Metadata-Version: 2.4
Name: gram-recall
Version: 0.1.0
Summary: Zero-embedding rare-gram keyword recall/ranking for CJK text (sliding 2/3-char grams, df filter, precision-over-recall floor)
Project-URL: Homepage, https://github.com/felixchaos/gram-recall
Project-URL: Issues, https://github.com/felixchaos/gram-recall/issues
Author: Stellatrix Labs
License-Expression: MIT
License-File: LICENSE
Keywords: chinese,cjk,keyword,ngram,no-embedding,ranking,recall,retrieval
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Natural Language :: Chinese (Simplified)
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Software Development :: Libraries
Classifier: Topic :: Text Processing :: Linguistic
Classifier: Typing :: Typed
Requires-Python: >=3.10
Description-Content-Type: text/markdown

# gram-recall

Zero-embedding, rare-gram keyword recall and ranking for **CJK** text — pull the few relevant items out of a long history (chat turns, event logs, notes) with nothing but the standard library.

- **No dependencies, no embeddings, no model.** Pure `re` + `dataclasses`. Runs anywhere, offline, deterministically.
- **Built for Chinese.** A sliding 2/3-character window over runs of CJK ideographs — no tokenizer, no word-segmentation dependency.
- **Precision over recall.** A query-side stop set and a corpus-side document-frequency filter suppress common grams; a score floor means a single weak match returns *nothing* rather than a false memory.
- **One pool, one ranking.** All candidate documents are scored together, so provenance never beats relevance.

Extracted from the episodic-recall subsystem of a production roleplay platform, where it is the *main* retrieval path: a corpus of 77k events had only 0.65% embedding coverage (region-locked embedders, almost nobody configures a BYOK one), so the deterministic keyword fallback carries real traffic.

## Install

```bash
pip install gram-recall
```

Requires Python 3.10+. No runtime dependencies.

## Quickstart

```python
from gram_recall import recall

docs = [
    "很久以前我把祖传的银怀表交给菲奥娜保管,叮嘱她别打开表盖",
    "今天天气不错,我们在集市上闲逛",
    "康拉德在石桥渡提起了纹章的来历",
]

hits = recall("我想起交给菲奥娜的那块银怀表", docs, top_k=3)
for h in hits:
    print(h.score, h.id, h.excerpt)
# 12 0 很久以前我把祖传的银怀表交给菲奥娜保管,叮嘱她别打开表盖
```

(Only document 0 clears the floor — the weather-chat and the heraldry line share no rare gram with the query, so they are not returned at all.)

`recall` returns a list of `ScoredDoc`, highest score first. Each carries the caller's document `id`, the matched `text`, the integer `score`, the `matched_grams` that contributed, and (unless you pass `with_excerpt=False`) an `excerpt` windowed around the first match.

### Bring your own ids

A document is either a bare string (its id becomes its list position) or an `(id, text)` pair with any id you like:

```python
docs = [
    (1001, "康拉德在石桥渡提起了纹章的来历"),
    (1002, "无关紧要的日常对话"),
]
hits = recall("康拉德和石桥渡的纹章", docs)
assert hits[0].id == 1001
```

### Flatten your own fields

The library searches one text string per document — you decide what goes into it. To make a structured record's fields all searchable, join them yourself:

```python
def flatten(event: dict) -> str:
    parts = [event["summary"]]
    if event.get("location"):
        parts.append(event["location"])
    parts.extend(event.get("participants", []))
    return " ".join(parts)

docs = [(e["id"], flatten(e)) for e in events]
hits = recall(query, docs)
```

## API

| Function | Signature | Description |
| --- | --- | --- |
| `grams` | `grams(text: str, *, config=DEFAULT_CONFIG) -> set[str]` | Candidate 2/3-character grams from every run of CJK characters, minus the stop set. |
| `score` | `score(query: str, docs: Sequence[str \| tuple[Any, str]], *, config=DEFAULT_CONFIG) -> list[ScoredDoc]` | Rank all documents in a single pool by rare-gram overlap. Empty when nothing clears the floor. |
| `recall` | `recall(query, docs, *, top_k=5, with_excerpt=True, config=DEFAULT_CONFIG) -> list[ScoredDoc]` | The top `top_k` of `score`, with excerpts filled in. |
| `excerpt` | `excerpt(text: str, query: str, *, window=None, config=DEFAULT_CONFIG) -> str` | A window of `text` around its first (longest, then lexicographically-first) query-gram match. |

`ScoredDoc` is a frozen dataclass: `id`, `text`, `score`, `matched_grams: tuple[str, ...]`, `excerpt: str | None`.

## How scoring works

1. **Query grams.** Slide a 2- and 3-character window over each run of CJK ideographs in the query; drop grams in the stop set (high-frequency function words). No gram spans a latin/punctuation/emoji boundary.
2. **Document-frequency filter (poor-man's IDF).** A gram present in more than `df_ratio` (default 25%) of the corpus carries no discriminating signal and is ignored. The cap is floored at `df_cap_floor` (default 2) so tiny corpora are not over-filtered.
3. **Per-document score.** Sum the lengths of the distinct matched grams. A gram nested inside a longer matched gram is **not** counted twice — a matched `康拉德` (3) absorbs its own `康拉`/`拉德`, so the score is 3, not 3 + 2 + 2.
4. **Floor.** Only documents scoring at least `score_floor` (default 3) are returned. So a lone two-character gram (score 2) never recalls, but a lone three-character gram — a typical name length — does. **Below the floor the result is empty**: this is a recall system that would rather say nothing than surface a spurious match.
5. **Rank.** Sort by score descending; ties break by input order (earlier documents first — pass your most-preferred documents first). The full pipeline is deterministic across runs and Python hash seeds.

## When to use it (and when not)

**Use it** for CJK recall where you can't or won't run embeddings: an offline or privacy-constrained deployment, a long-tail corpus most of which will never be embedded, a deterministic fallback beneath a semantic path, or a cheap first-pass filter.

**Reach for embeddings instead** when you need synonymy or paraphrase matching (this is literal gram overlap — it will not connect 医生 to 大夫), when your text is primarily non-CJK (Latin scripts have word boundaries and are better served by classic tokenized BM25/TF-IDF), or when recall matters more than precision (this design deliberately drops weak matches).

## Tuning

Every threshold lives on a frozen `Config`; override with `dataclasses.replace` and pass it in:

```python
from dataclasses import replace
from gram_recall import DEFAULT_CONFIG, recall
import re

cfg = replace(
    DEFAULT_CONFIG,
    score_floor=2,                                   # admit lone bigrams
    df_ratio=0.5,                                    # keep grams up to 50% of corpus
    stop_grams=DEFAULT_CONFIG.stop_grams | {"魔王"},  # add a domain stop word
    cjk_pattern=re.compile(r"[぀-ヿ一-鿿]{2,}"),      # also cover kana
)
hits = recall(query, docs, config=cfg)
```

The default `cjk_pattern` matches only the base CJK Unified Ideographs block (U+4E00–U+9FFF); supply your own to include CJK Extension A/B, kana, or Hangul.

## License

MIT — see [LICENSE](LICENSE). Copyright (c) 2026 Stellatrix Labs.

中文文档见 [README.zh-CN.md](README.zh-CN.md)。
