Metadata-Version: 2.4
Name: patternminer
Version: 0.1.0
Summary: Mine recurrent surface text patterns in a corpus (built on generalized suffix trees)
Author: Guillaume Dubuisson Duplessis
License-Expression: MIT
Project-URL: Homepage, https://github.com/GuillaumeDD/patternminer
Project-URL: Source, https://github.com/GuillaumeDD/patternminer
Project-URL: Changelog, https://github.com/GuillaumeDD/patternminer/blob/main/CHANGELOG.md
Project-URL: Issues, https://github.com/GuillaumeDD/patternminer/issues
Keywords: pattern mining,text patterns,repetition,corpus linguistics,suffix tree,computational linguistics
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Science/Research
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Information Analysis
Classifier: Topic :: Text Processing :: Linguistic
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE.txt
Requires-Dist: pygstlib>=0.1
Provides-Extra: analysis
Requires-Dist: pandas>=2.0; extra == "analysis"
Provides-Extra: dev
Requires-Dist: pytest; extra == "dev"
Requires-Dist: pytest-cov; extra == "dev"
Requires-Dist: ruff; extra == "dev"
Requires-Dist: pandas>=2.0; extra == "dev"
Dynamic: license-file

# patternminer: Mining Recurrent Text Patterns in a Corpus

[![PyPI version](https://img.shields.io/pypi/v/patternminer.svg)](https://pypi.org/project/patternminer/)
[![CI](https://github.com/GuillaumeDD/patternminer/actions/workflows/ci.yml/badge.svg)](https://github.com/GuillaumeDD/patternminer/actions/workflows/ci.yml)

patternminer makes it easy to mine **recurrent surface text patterns** in a
corpus of texts: which token subsequences recur, where, how often, and how
they nest inside each other.

It is built on [pygstlib](https://github.com/GuillaumeDD/pygstlib)
(generalized suffix trees — mining is linear-time) and generalizes the
pattern-mining approach of [dialign](https://github.com/GuillaumeDD/dialign) /
[pydialign](https://github.com/GuillaumeDD/pydialign) beyond dyadic dialogue.

Features:

- **cross-document mining**: patterns present in ≥ *k* documents, with every
  occurrence located as `(doc_id, unit_id, start)` and both absolute
  (`doc_freq`, `unit_freq`) and relative (`doc_support`, `unit_support`)
  frequencies;
- **within-document mining**: patterns a document repeats across its own
  units (sentences, lines, utterances…);
- **hierarchy**: regroup subpatterns under their parent patterns (e.g.
  parent `hello world !` groups `hello` and `world !`);
- **filtering**: by size, frequency, arbitrary predicates, or down to the
  maximal patterns;
- **text in, patterns out**: built-in basic tokenizers and unit splitters —
  or bring your own tokens;
- **pandas layer** (optional): patterns and occurrences as DataFrames.

## Installation

```sh
uv add patternminer               # or: pip install patternminer
uv add 'patternminer[analysis]'   # with the pandas layer
```

Requires Python ≥ 3.10. The only required runtime dependency is
[pygstlib](https://pypi.org/project/pygstlib/).

## Quickstart

A corpus is a list of **documents**; each document is a list of **units**
(the granularity at which repetition is observed — by default, one unit per
line); each unit is a tuple of string tokens.

```python
from patternminer import Corpus, mine_corpus

corpus = Corpus.from_texts([
    "hello world !\nhow are you today ?",
    "hello world ! how are you doing ?",
])
patterns = mine_corpus(corpus)  # patterns shared by >= 2 documents

for p in patterns:
    print(f"{p.surface!r}: doc_freq={p.doc_freq}, occurrences={list(p.occurrences)}")

print()
print(patterns.hierarchy().render())
```

Output:

```text
'hello world !': doc_freq=2, occurrences=[(0, 0, 0), (1, 0, 0)]
'how are you': doc_freq=2, occurrences=[(0, 1, 0), (1, 0, 3)]
'?': doc_freq=2, occurrences=[(0, 1, 4), (1, 0, 7)]

hello world !  [doc_freq=2, unit_freq=2]
how are you  [doc_freq=2, unit_freq=2]
?  [doc_freq=2, unit_freq=2]
```

Subpatterns that never occur independently (`are you`, `world !`, `you`,
`!` — always enclosed in `hello world !` / `how are you`) are discarded;
they would appear if they occurred free somewhere in the corpus.

## Tour of the API

```python
from patternminer import Corpus, mine_corpus, mine_document, mine_within, nlp

# Corpus building — raw text with basic helpers…
corpus = Corpus.from_texts(texts, tokenizer=nlp.word_punct_tokenizer,
                           unit_splitter=nlp.split_sentences, lowercase=True)
corpus = Corpus.from_files(paths)
# …or pre-tokenized input:
corpus = Corpus.from_token_lists(list_of_token_lists)   # 1 unit per document
corpus = Corpus.from_token_units(docs_units_tokens)     # nested

# Mining — threshold by absolute freq or relative support (or both)
patterns = mine_corpus(corpus, min_doc_freq=2, is_valid=nlp.has_alphabetic)
patterns = mine_corpus(corpus, min_doc_support=0.5)   # >= half the documents
repetitions = mine_document(corpus[0])       # within one document
per_doc = mine_within(corpus, min_unit_support=0.1)   # doc_id -> PatternSet

# Pattern: frequencies, supports, free/constrained classification
p = patterns.get("world !")
p.doc_freq; p.unit_freq    # absolute counts (documents / units)
p.doc_support              # doc_freq / total documents in the corpus
p.unit_support             # unit_freq / total units in the corpus
p.occurrences              # every occurrence, (doc_id, unit_id, start)
p.free_occurrences         # the subset not enclosed in a larger pattern
p.constrained_occurrences  # the enclosed complement

# PatternSet
patterns.filter(min_size=2, min_unit_freq=3, min_doc_support=0.5, ...)
patterns.maximal()          # patterns contained in no other
patterns.get("world !")     # lookup by surface or token tuple; None if the
                            # key is absent (e.g. a pattern that never
                            # occurs free — see caveats below)
patterns.to_dataframe()     # pandas ([analysis] extra)

# Hierarchy (containment DAG)
hier = patterns.hierarchy()
hier.roots; hier.children(p); hier.parents(p); hier.descendants(p)
print(hier.render())
```

**➡ Tutorials:** the [quickstart notebook](examples/notebook/quickstart.ipynb)
and a real [corpus study](examples/notebook/corpus-study.ipynb) on three
chapters of *David Copperfield*.

## Semantics & caveats

- **Free patterns only.** The inventory holds the *right-maximal* repeats
  that occur **free** at least once; a pattern whose every occurrence is
  enclosed in a free occurrence of a larger pattern is discarded (e.g.
  `are you` is dropped when it only ever appears inside `how are you`).
  Kept patterns still report *all* their occurrences, enclosed instances
  included; `Pattern.free_occurrences` / `constrained_occurrences` expose
  the per-occurrence split. Details in
  [docs/architecture.md](docs/architecture.md).
- **Repetition is counted across units.** A pattern occurring twice inside
  a *single* unit (and nowhere else) is not detected. Choose the unit
  granularity accordingly (`nlp.split_sentences`, `nlp.split_lines`, …).
- **Surface-level.** patternminer sees exactly the tokens you give it —
  normalize (case, tokenization) upstream, or use the `lowercase=True` and
  tokenizer options. The built-in tokenizers are deliberately simple.
- **Determinism.** Results come in a canonical order (size desc, then
  tokens) and are identical across processes.
- **Concurrency.** Like pygstlib, mining structures are not thread-safe;
  the returned `Pattern`/`PatternSet`/`PatternHierarchy` objects are
  immutable and safe to share once built.

## Development

```sh
uv venv && uv pip install -e '.[dev]' --group notebook
uv run pytest          # unit, property, randomized naive-reference and notebook tests
uv run ruff check .
```

The project is managed BMAD-style: see [`docs/prd.md`](docs/prd.md),
[`docs/architecture.md`](docs/architecture.md) and the story backlog in
[`docs/stories/`](docs/stories/).

## Contributors

- Guillaume Dubuisson Duplessis (2026)

## Usage for Research Purposes

If you use this library for research purposes, please make reference to it
by citing the following paper on recurrent surface text patterns:

- Dubuisson Duplessis, G.; Charras, F.; Letard, V.; Ligozat, A.-L.; Rosset, S.,
  **Utterance Retrieval based on Recurrent Surface Text Patterns**, 39th
  European Conference on Information Retrieval (ECIR), 2017, pp. 199--211
  \[[More](https://hal.archives-ouvertes.fr/hal-01436052)
  [DOI](https://dx.doi.org/10.1007/978-3-319-56608-5_16)\]

See also [dialign](https://github.com/GuillaumeDD/dialign) and
[pydialign](https://github.com/GuillaumeDD/pydialign) for the
dialogue-specific measures built on the same mining approach.

## License

MIT — see the [LICENSE.txt](LICENSE.txt) file.
