Metadata-Version: 2.4
Name: termxtract
Version: 0.1.0
Summary: Automatic term extraction (ATE) from plain text or Teanga corpora, with unsupervised and supervised methods.
License-Expression: Apache-2.0
License-File: LICENSE
Keywords: terminology extraction,automatic term extraction,ATE,NLP,Teanga
Author: Shubhanker Banerjee
Author-email: shubcode1@gmail.com
Maintainer: John McCrae
Maintainer-email: john@mccr.ae
Requires-Python: >=3.10,<4.0
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Text Processing :: Linguistic
Provides-Extra: transformers
Requires-Dist: accelerate (>=1.5.2) ; extra == "transformers"
Requires-Dist: numpy (>=1.24.0,<2.0.0)
Requires-Dist: sacremoses (>=0.0.53) ; extra == "transformers"
Requires-Dist: scikit-learn (>=1.3.0,<2.0.0)
Requires-Dist: teanga (>=0.1.0,<0.2.0)
Requires-Dist: torch (>=1.13.0) ; extra == "transformers"
Requires-Dist: transformers (>=4.41.0,<5.0.0) ; extra == "transformers"
Requires-Dist: wn (>=1.1,<2.0)
Project-URL: Documentation, https://teanga.io/TermXtract/
Project-URL: Homepage, https://teanga.io/TermXtract/
Project-URL: Issues, https://github.com/TeangaNLP/TermXtract/issues
Project-URL: Repository, https://github.com/TeangaNLP/TermXtract
Description-Content-Type: text/markdown

# TermXtract

[![CI](https://github.com/TeangaNLP/TermXtract/actions/workflows/ci-cd.yml/badge.svg)](https://github.com/TeangaNLP/TermXtract/actions/workflows/ci-cd.yml)
[![Docs](https://img.shields.io/badge/docs-teanga.io%2FTermXtract-blue)](https://teanga.io/TermXtract/)
[![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](LICENSE)

**TermXtract** is a Python library for **automatic term extraction (ATE)**: finding the domain-specific terms in a collection of documents. It offers a range of classic **unsupervised** scoring methods (TF-IDF, C-Value, Weirdness, RAKE, topic models, …) and **supervised** methods that learn from labelled examples (AdaBoost, Roger, and transformer-based token classification).

Every method takes either a plain list of strings or a [Teanga](https://github.com/TeangaNLP/teanga) `Corpus`, and returns the same `ATEResults` object.

📖 **Documentation:** <https://teanga.io/TermXtract/>  
📝 **Changelog:** [CHANGELOG.md](CHANGELOG.md). TermXtract follows [Semantic Versioning](https://semver.org/); before 1.0, minor releases may contain breaking changes.

## Installation

TermXtract needs **Python 3.10 or later**. It is not yet on PyPI, so install it from GitHub:

```bash
pip install git+https://github.com/TeangaNLP/TermXtract.git
```

The transformer-based `token-classification` extractor needs PyTorch and Hugging Face Transformers, which are large, so they are an optional extra. Install them with:

```bash
pip install "termxtract[transformers] @ git+https://github.com/TeangaNLP/TermXtract.git"
```

All other methods work with the base install.

## Quickstart

### Unsupervised extraction

```python
from termxtract import TermExtractor

corpus = [
    "Machine learning is a field of artificial intelligence.",
    "Deep learning is a subfield of machine learning.",
    "Natural language processing applies machine learning to text.",
]

extractor = TermExtractor(method="cvalue", n=2)  # consider unigrams and bigrams
results = extractor.extract(corpus)

for doc in results.terms:
    print(doc["doc_id"], doc["terms"][:3])
# doc_0 [{'term': 'machine learning', 'score': 1.0}, {'term': 'artificial intelligence', 'score': 1.0}, {'term': 'machine', 'score': 0.0}]
# ...
```

Each document's terms are ranked from highest to lowest score.

Pass `threshold=...` to keep only terms scoring at or above a minimum.

Candidates that start or end with a stopword, such as `is a` or `of machine`, are dropped by default, using a built-in list of English function words. Stopwords inside a term, as in `degree of freedom`, are kept. Pass `stopwords=None` to keep every candidate, or a list of words to use your own stopwords.

If a Teanga corpus has part-of-speech tags, `pos_pattern="noun_phrase"` keeps only noun phrases, such as `machine learning` or `degree of freedom` (see [Using a Teanga corpus](#using-a-teanga-corpus)).

### Methods that need a reference corpus

`weirdness`, `domainpertinence` and `relevance` compare the target corpus against a general-language reference corpus:

```python
reference = [
    "The weather was pleasant and people walked in the park.",
    "She bought bread and milk at the shop on the corner.",
]

extractor = TermExtractor(method="weirdness", reference_corpus=reference, n=2)
results = extractor.extract(corpus)
```

### Supervised extraction

Supervised methods are trained from a dictionary of candidate terms labelled `1` (term) or `0` (not a term):

```python
from termxtract import SupervisedTermExtractor

labels = {
    "machine learning": 1,
    "deep learning": 1,
    "natural language processing": 1,
    "field": 0,
    "text": 0,
}

extractor = SupervisedTermExtractor(method="adaboost", n=3)
results = extractor.extract(corpus, labels)
```

The AdaBoost extractor uses [Open English WordNet](https://en-word.net/) through the [`wn`](https://pypi.org/project/wn/) package. The first time it needs WordNet it downloads the `oewn:2025+` lexicon into `~/.wn_data` (set `WN_DATA_DIR` to change this). To install it ahead of time, e.g. for offline use, run `python -m wn download oewn:2025+`. Pass `wordnet_lexicon=...` to use a different lexicon.

### Using a Teanga corpus

Any extractor also accepts a Teanga `Corpus` with a `words` span layer:

```python
from teanga import Corpus

corpus = Corpus()
corpus.add_layer_meta("text")
corpus.add_layer_meta("words", layer_type="span", base="text")
doc = corpus.add_doc("Deep learning is a subfield of machine learning.")
doc.words = [(0, 4), (5, 13), (14, 16), (17, 18), (19, 27), (28, 30), (31, 38), (39, 47)]

results = TermExtractor(method="basic", n=2).extract(corpus)
```

Words are read from the `words` layer or, if there is none, from a `tokens` layer, as written by Teanga's CoNLL-U reader and spaCy service; pass `words_layer=...` to use another layer. If the corpus has part-of-speech tags in a `upos` or `pos` layer based on the words, pass `pos_pattern` to keep only the terms whose tags match a pattern, either `"noun_phrase"` or your own pattern such as `"<ADJ|NOUN>*<NOUN>"`. See the [TF-IDF tutorial](https://teanga.io/TermXtract/tutorials/tfidf_tutorial/#part-of-speech-patterns).

## Available methods

`TermExtractor.METHODS` and `SupervisedTermExtractor.METHODS` map each method name to its implementation, so `list(TermExtractor.METHODS)` lists the available methods. Passing an unknown `method` raises a `ValueError` that lists them too.

### Unsupervised — `TermExtractor(method=...)`

| `method`           | Description                                         | Extra parameters                      |
|--------------------|-----------------------------------------------------|---------------------------------------|
| `tfidf`            | Term frequency–inverse document frequency           |                                       |
| `ridf`             | Residual IDF                                        |                                       |
| `cvalue`           | C-Value (nested multi-word terms)                   |                                       |
| `basic`            | Basic                                               | `alpha`                               |
| `combobasic`       | ComboBasic                                          | `alpha`, `beta`                       |
| `rake`             | Rapid Automatic Keyword Extraction                  | `stoplist`, `phrase_delimiters` (required) |
| `domaincoherence`  | Domain coherence                                    | `window_size`                         |
| `domainpertinence` | Domain pertinence                                   | `reference_corpus` (required)         |
| `weirdness`        | Weirdness                                           | `reference_corpus` (required)         |
| `relevance`        | Relevance                                           | `reference_corpus` (required)         |
| `topicmodeling`    | LDA topic-model-based scoring                       | `num_topics`                          |
| `nmf`              | NMF topic-model-based scoring                       | `n_topics`, `max_features`            |

All methods accept `n` (maximum n-gram length), `threshold` (minimum score), `stopwords` (default `"english"`), and, for Teanga input, `words_layer`, `pos_pattern` and `pos_layer`. `rake` does not use `n`, `stopwords` or `words_layer`; it splits candidates at its own `stoplist`.

### Supervised — `SupervisedTermExtractor(method=...)`

| `method`               | Description                                              | Extra parameters                                         |
|------------------------|----------------------------------------------------------|----------------------------------------------------------|
| `adaboost`             | AdaBoost over linguistic and statistical features        | `threshold`, `max_depth`, `n_estimators`, `estimator`, `wordnet_lexicon` |
| `roger`                | Ranking function of association measures, learned by evolution (multi-word terms only) | `population_size`, `offspring_size`, `generations`, `runs`, `random_state` |
| `token-classification` | Fine-tuned Hugging Face token-classification model (needs the `transformers` extra) | `model_name`, `epochs`, `batch_size`, `learning_rate`, `max_length`, `language` |

All supervised methods accept `n` (maximum n-gram length, default 6), and `stopwords` and `pos_pattern`, which filter the extracted terms but not the training candidates, and `words_layer` and `pos_layer` for Teanga input.

## Results

`extract()` returns an `ATEResults` object with:

- `results.terms`: one entry per document, `{"doc_id": ..., "terms": [{"term": str, "score": float}, ...]}`, with terms ranked from highest to lowest score. Every method uses this format.
- `results.offsets`: for Teanga input, one entry per document, `{"doc_id": ..., "offsets": [{"term": str, "offset": (start, end)}, ...]}`, giving the character offsets of every occurrence of each extracted term. It is `None` for plain-text input, and for `nmf` and `domaincoherence`, which score terms over the whole corpus rather than per document.
- `results.corpus`: the input corpus.

## Development

```bash
git clone https://github.com/TeangaNLP/TermXtract.git
cd TermXtract
poetry install --with dev --extras transformers

poetry run pytest                              # run the test suite
poetry run pytest --cov                        # ...with a coverage report
poetry run ruff check termxtract tests         # lint
poetry run ruff format termxtract tests        # format
poetry run mypy termxtract                     # type check
poetry run mkdocs serve                        # preview the docs locally
```

CI runs all of these on every pull request, with the tests on Python 3.10–3.12, on Linux, macOS and Windows, and against the oldest dependency versions `pyproject.toml` allows. The codebase was reformatted in one commit, which `git blame` can skip with `git config blame.ignoreRevsFile .git-blame-ignore-revs`.

### Releasing

Version numbers follow [Semantic Versioning](https://semver.org/): bump the patch version for bug fixes, the minor version for new features, and the major version for breaking changes (before 1.0, breaking changes bump the minor version).

1. In `CHANGELOG.md`, rename `## [Unreleased]` to `## [X.Y.Z] - YYYY-MM-DD`, add a new empty `## [Unreleased]` section above it, and update the comparison links at the bottom of the file.
2. Set the version with `poetry version X.Y.Z` (or `poetry version patch`, `minor` or `major`).
3. Commit both files as `Release X.Y.Z` and get the commit onto `main`.
4. On GitHub, draft a release with a new tag `vX.Y.Z` on that commit, paste the changelog section as the release notes, and publish it. The release workflow checks that the tag matches the version, builds the package and uploads it to PyPI.

`pyproject.toml` is still at 0.1.0 and nothing has been published, so the first release can be 0.1.0 without a version bump.

## License

TermXtract is licensed under the [Apache License 2.0](LICENSE), the same license as [Teanga](https://github.com/TeangaNLP/teanga).

