Metadata-Version: 2.3
Name: PyDistintoX
Version: 0.2.4
Summary: A Python tool for contrastive text analysis using 16 statistical distinctiveness measures and based on Pydistinto. It can be used both as a CLI and as a library.
Author: Leon Glüsing, Stefan Walter-Heßbrüggen
Author-email: Leon Glüsing <leongluesing@uni-muenster.de>
License: CC0-1.0
Requires-Dist: altair>=6.0.0
Requires-Dist: argparse>=1.4.0
Requires-Dist: gensim>=4.3.3
Requires-Dist: logging>=0.4.9.6
Requires-Dist: numpy>=1.26.4
Requires-Dist: pandas>=2.2.3
Requires-Dist: pip>=25.2
Requires-Dist: scikit-learn>=1.6.1
Requires-Dist: scipy>=1.13.1
Requires-Dist: spacy>=3.1
Maintainer: Leon Glüsing
Maintainer-email: Leon Glüsing <leongluesing@posteo.de>
Requires-Python: >=3.11.1, <3.13
Project-URL: Repository, https://gitlab.com/leongluesing/pydistintox
Project-URL: API/Library Documentation, https://pydistintox-993c39.gitlab.io/api/
Project-URL: Contribute, https://pydistintox-993c39.gitlab.io/CONTRIBUTOR/
Project-URL: Homepage of Original Project, https://www.uni-muenster.de/Wissenschaftstheorie/Forschung/PRODATPHIL/
Description-Content-Type: text/markdown

# PyDistintoX


A Python tool for **contrastive text analysis** using 16 statistical distinctiveness measures (based on [Pydistinto](https://github.com/Zeta-and-Company/pydistinto), see [Project Status](https://gitlab.com/leongluesing/PyDistintoX/#project-status) for details).


<table style="width: 100%;">
  <tr>
    <td style="width: 33%; text-align: center;">
      <img src="https://gitlab.com/leongluesing/PyDistintoX/-/raw/main/docs/screenshots/overview.png" style="max-width: 100%; height: auto;"/>
      <p><em>Overview</em></p>
    </td>
    <td style="width: 33%; text-align: center;">
      <img src="https://gitlab.com/leongluesing/PyDistintoX/-/raw/main/docs/screenshots/zeta_sd2.png" style="max-width: 100%; height: auto;"/>
      <p><em>Zeta SD2</em></p>
    </td>
    <td style="width: 33%; text-align: center;">
      <img src="https://gitlab.com/leongluesing/PyDistintoX/-/raw/main/docs/screenshots/nfc.png" style="max-width: 100%; height: auto;"/>
      <p><em>NFC</em></p>
    </td>
  </tr>
  <tr>
    <td colspan="3" style="text-align: center;">
      <img src="https://gitlab.com/leongluesing/PyDistintoX/-/raw/main/docs/screenshots/heatmap.png" style="max-width: 300px; height: auto;"/>
      <p><em>Heatmap</em></p>
    </td>
  </tr>
</table>


PyDistintoX is designed to compare and analyze **two text corpora** using a comprehensive set of 16 statistical measures (e.g., TF-IDF, Zeta, Chi-squared, LLR, Eta). Built as a **command-line interface (CLI) tool**, it enables researchers, linguists, and data analysts to efficiently quantify and visualize lexical distinctiveness between corpora.

**Key Features:**

- **16 distinctiveness measures** for robust contrastive analysis.
- **CLInterface** for quick, scriptable workflows.
- **Python Library**  for custom workflows (see [documentation](https://pydistintox-993c39.gitlab.io/api/))
- **Visualization support** (e.g., heatmaps) for intuitive interpretation.
- **Lightweight and scalable**—optimized for large corpora using Gensim and NumPy/SciPy.sparse.

---

## Overview
PyDistintoX can be used in two ways:
- [Standalone Application (CLI)](https://gitlab.com/leongluesing/PyDistintoX/#standalone-application-cli): Run analyses directly from the command line.
- [Python library](https://gitlab.com/leongluesing/PyDistintoX/#python-library): Import functions for custom workflows.

Furthermore, we support the installation via `uv` as well via `pip`.


### Prerequisites
Choose either:
| uv (recommended) | Pure Python |
|---|---|
| install [here](https://docs.astral.sh/uv/getting-started/installation/)| >= 3.11.1, < 3.13|

> **Note**: Replace `python` in the commands below with your specific version if needed.

The two corpora you want to compare must be stored in separate directories.

## Standalone Application (CLI)
Run analyses directly from the command line.


### Installation
Choose either: 

#### uv (recommended)

```bash
uv tool install pydistintox
```


#### pip
```bash
pip install pipx
pipx install pydistintox
```

### Setup
Download spacy [model of your choice](https://spacy.io/models). You need to know the model name when using the [CLI](https://gitlab.com/leongluesing/PyDistintoX/#command-line-options) later as well.

Then run
```bash
pydistintox --download <MODEL_NAME>
```

For an english language model run for example:
```bash
pydistintox --download en_core_web_sm
```

An **Example Script** can be found [here](https://pydistintox-993c39.gitlab.io/examples/standalone/)

---

### Usage

Base Command: `pydistintox [OPTIONS]`. The results will be saved in the current working directory.

#### Command Line Options

**Required**
* `--tar path/to/target/corpus` specify the directory for target corpus. Give the directory, not a path to a file! (Escape spaces in the path if necessary)
* `--ref path/to/reference/corpus` specify the directory for reference corpus. Give the directory, not a path to a file! (Escape spaces in the path if necessary)
* `--model spacy_model_name` spaCy model name in the format: `{lang}_core_{dataset}_{size}`. Example: `en_core_web_sm` (lang=en, dataset=web, size=sm). Available sizes: sm, md, lg, trf. [Find models here](https://spacy.io/models).

**Optional**
* `--load-nlp path/to/json/dir` skips tokenization and lemmatization, and load nlp results from path. Make sure `path/to/json/dir` contains the folders `tar` and `ref`. If no path is specified, the default `data/interim/json` is used.
* `--raw-scores` Scores will not be scaled to -1,1.
* `--verbose` or `-v` changes logging mode to verbose

#### Example Command
```bash
pydistintox -v --model en_core_web_sm --tar ./data/example_texts/tar --ref ./data/example_texts/ref
```
#### Workflow Overview

The application processes your texts in three steps:
1. **NLP Processing:** Texts are tokenized and lemmatized. If `--save-nlp` flag is set, the results are saved as JSON in the specified directory.
2. **Statistical Analysis:** Various distinctiveness measures are calculated.
3. **Results Visualization:** The results are automatically displayed in your default browser.

If you already created the JSON files (f.e. from a previous run), you can skip the first step by setting the `--load-nlp` flag (see [Command Line Options](#command-line-options) above).


---


## Python library

Import [functions](https://pydistintox-993c39.gitlab.io/api/) for custom workflows.

<details>
<summary>click to expand</summary>


### Installation
- **If you are not using a virtual environment yet:**
  Create one:
  - `uv venv`
  - `python -m venv .venv`

  Activate it with:
  - `source .venv/bin/activate` (Linux/macOS)
  - `.\.venv\Scripts\activate` (Windows)

Then run either:
|   | uv (recommended)                  | pip                                      |
|---|-----------------------------------|------------------------------------------|
|Install | `uv pip install pydistintox` | `pip install pydistintox`                |
|Check Installation|You can see the place of installation via `uv pip show pydistintox`.|You can see the place of installation via `pip show pydistintox`.|

**Example Implementation:**

```bash
# Assuming you do not have an active virtual environment
python3.11 -m venv .venv
source .venv/bin/activate
# and you are in the root directory of the pydistintox
# project. You may install it via
pip install pydistintox
# You can check your installation via 
pip show pydistintox
```

### Usage

```python
from pathlib import Path
from pydistintox import *

spacy_download('en_core_web_sm')
# Caution: Keep in mind that the download command installs a Python package into your environment. 
# In order for it to be found after installation, you MAY need to restart or reload your Python process so that new packages are recognized.
# Have a look at https://gitlab.com/leongluesing/PyDistintoX/#troubleshooting

config = Config(
    skip_nlp = False,
    save_nlp= JSON,
    spacy_model_name='en_core_web_sm',
    target=INPUT_TAR,
    reference=INPUT_REF,
    verbose = True,
    source='YOUR COMMENT HERE',
    scaling= True,
    measures_to_calculate = ['zeta_sd2', 'afn']
)

...

```

**For a complete example, see the script:** [library.py](https://gitlab.com/leongluesing/PyDistintoX/docs/examples/library.py)


### Available Import Functions

Key functions for programmatic use (import via `from pydistintox import ...`).
**Full documentation** [here](https://pydistintox-993c39.gitlab.io/api/)

</details>

---

## Remove PyDistintoX

|   | uv                  | pip                                      |
|---|-----------------------------------|------------------------------------------|
|uninstall CLI | `uv tool uninstall pydistintox`| `pipx uninstall pydistintox` |
|uninstall library|`uv pip uninstall pydistintox`|`pip uninstall pydistintox`|
|clean cache|`uv cache clean`|`pip cache purge`|
|remove model|`uv pip uninstall en_core_web_sm`|`pip uninstall en_core_web_sm`|

You may delete the virtual environment completely. Thereby all dependencies and downloaded models will be deleted as well: First deactivate the virtual environment (`deactivate`) and then delete its folder (`rm -r .venv` or `rmdir /s .venv` on windows).

---

## Troubleshooting

### Installation of spaCy Model

The spaCy documentation warns against downloading and loading a model in the same script ([see here](https://spacy.io/usage/models)):

> Keep in mind that the download command installs a Python package into your environment. In order for it to be found after installation, you will need to restart or reload your Python process so that new packages are recognized.

However, in most cases it works nevertheless. If it is not working in your case, you need to preinstal the model manually with:

```bash
(uv) pip install $(spacy info <MODEL_NAME> --url) 
```

If the installtion of a spacy model via `(uv) pip install <MODEL_NAME>` still fails, you may download the model by some other means and run `(uv) pip install path/to/model` instead.

---

## Technical Reference

The distinctiveness measures provided are scaled to the range of [-1, 1] by default for visualization, but this scaling does not make them statistically comparable. Each measure is based on different underlying assumptions, mathematical formulations, and distributions (e.g., frequency-based vs. dispersion-based). Direct comparison of values across measures is not meaningful, unless one explicitly accounts for the differing mathematical foundations.

### Non-TF-IDF Measures

Using the gensim tf-idf-model as helper function, `PyDistintoX` calculates absolute and binary frequencies. This is tf-idf divided by 1, i. e. disregarding document frequency, so strictly speaking tf-no-idf. Based on this data, the following distinct measures are calculated.

The following table is adapted from the foundational article on which this software is based: ["Evaluation of Measures of Distinctiveness", by Keli Du, Julia Dudar, Christof Schöch](https://doi.org/10.48694/jcls.102) ([Wayback Machine](https://web.archive.org/web/20260116101348/https://jcls.io/article/id/102/)). For more information see [here](https://zeta-project.eu/de/distinktivitaetsmasse/) ([Wayback Machine](https://web.archive.org/web/20260113084856/https://zeta-project.eu/de/distinktivitaetsmasse/))

| Name                              | Type of measure      | References                                      | Evaluated in                                                                 | Implementation Key   |
|-----------------------------------|----------------------|-------------------------------------------------|------------------------------------------------------------------------------|----------------------|
| TF-IDF                            | Term weighting       | Luhn 1957; Spärck Jones 1972                    | Salton and Buckley 1988                                                      | –                    |
| Ratio of relative frequencies (RRF) | Frequency-based      | Damerau 1993                                    | Gries 2010                                                                   | `rrf_dr0`            |
| Chi-squared test (χ²)             | Frequency-based      | Dunning 1993                                    | Lijffijt et al. 2014                                                         | `chi_square_value`   |
| Log-likelihood ratio test (LLR)  | Frequency-based      | Dunning 1993                                    | Egbert and Biber 2019; Paquot and Bestgen 2009; Lijffijt et al. 2014         | `LLR_value`          |
| Welch’s t-test (Welch)           | Distribution-based   | Welch 1947                                      | Paquot and Bestgen 2009; Lijffijt et al. 2014                                | `welch_t_value`      |
| Wilcoxon rank sum test (Wilcoxon) | Dispersion-based     | Wilcoxon 1945; Mann and Whitney 1947            | Paquot and Bestgen 2009; Lijffijt et al. 2014                                | `ranksumtest_value`  |
| Burrows Zeta (Zeta_orig)          | Dispersion-based     | Burrows 2007; Craig and Kinney 2009             | Schöch 2018                                                                  | `zeta_sd0`           |
| logarithmic Zeta (Zeta_log)       | Dispersion-based     | Schöch 2018                                     | Schöch 2018; Du et al. 2021                                                  | `zeta_sd2`           |
| Eta                               | Dispersion-based     | Du et al. 2021                                  | Du et al. 2021                                                               | `eta_sg0`            |


### TF-IDF Measures
`PyDistintoX` uses the gensim tf-idf model for calculating distinctness scores. The following parameters are used:

- 'nfn',
- 'nfc',
- 'bfn',
- 'afn',
- 'lfn',
- 'ltc'

For more information on the meaning of these parameters see [wikipedia](https://en.wikipedia.org/wiki/SMART_Information_Retrieval_System) [Wayback Machine](https://web.archive.org/web/20260203181537/https://en.wikipedia.org/wiki/SMART_Information_Retrieval_System), or the following paragraph:

<details>

<summary>
Explanation from the Gensim Documentation
</summary>

From the [gensim documentation](https://radimrehurek.com/gensim/models/tfidfmodel.html#gensim.models.tfidfmodel.TfidfModel): 

>**smartirs (str, optional) –**
>
>SMART (System for the Mechanical Analysis and Retrieval of Text) Information Retrieval System, a mnemonic scheme for denoting tf-idf weighting variants in the vector space model. The mnemonic for representing a combination of weights takes the form XYZ, for example ‘ntc’, ‘bpn’ and so on, where the letters represents the term weighting of the document vector.
>
>Term frequency weighing:
> - b - binary,
> - t or n - raw,
> - a - augmented,
> - l - logarithm,
> - d - double logarithm,
> - L - log average.
>
>Document frequency weighting:
> - x or n - none,
> - f - idf,
> - t - zero-corrected idf,
> - p - probabilistic idf.
>
>Document normalization:
> - x or n - none,
> - c - cosine,
> - u - pivoted unique,
> - b - pivoted character length.
>
>Default is ‘nfc’. For more information visit SMART Information Retrieval System.

</details>

### SpaCy and Gensim

While gensim features its own rule-based [tokenizer](https://radimrehurek.com/gensim/utils.html#gensim.utils.simple_tokenize), we provide the opportunity to plug-in spaCy models for tokenization and lemmatization. Results of this process are saved as json files in spaCy's own format. We then extract lemmas from these data and recast them as Gensim input documents (basically list of words forming sentences which in turn are combined in a list to form a document).


## Security

If you discover a security vulnerability in PyDistintoX, please report it to [leongluesing@posteo.de](mailto:leongluesing@posteo.de) with details (version, reproduction steps, impact) and allow 48–72 hours for a response before public disclosure.

## Project Status

PyDistintoX was developed by Leon Glüsing and Stefan Heßbrüggen-Walter. Original development was funded by the Deutsche Forschungsgemeinschaft via the project "[Prodatphil: Science and Logic(https://www.uni-muenster.de/Wissenschaftstheorie/en/Forschung/PRODATPHIL/index.html)]", project no. 537184692. Since Leon Glüsing’s (my) contract has ended in March 2026, the project is now voluntarily maintained and further developed by Leon Glüsing. While no major updates are currently planned, the project’s future trajectory could evolve with increased community involvement or active contributions.  **So feel free to get involved**!

This is a reimplementation of fundamental functionalities of "**Pydistinto**", a software project developed at the [TCDH](https://tcdh.uni-trier.de/). The original script can be found [here](https://github.com/Zeta-and-Company/pydistinto). 

The following major changes are worth mentioning:

* using Gensim and NumPy instead of Pandas for the construction of term-document matrices; Gensim can parse corpora that do not fit into RAM, NumPy/SciPy is used to process sparse matrices with lower memory requirements,

* using the inbuilt tf-idf functions of Gensim for adding more tf-idf related measures of distinctiveness (see [below](https://gitlab.com/leongluesing/PyDistintoX/#tf-idf-measures)),

This is a *lite* version of the original Pydistinto since we offer less features, f.e. no randomization of texts. However, if your corpus reaches a size that makes it difficult for the original Pydistinto to cope, it may be worth to try out our solution. Furthermore, the CLI Version of PyDistintoX is easier to use than the original Pydistinto.


## Contribute

This project thrives on community contributions. Please feel free to submit pull requests or open issues! You will find more detailed information for developers [here](https://pydistintox-993c39.gitlab.io/CONTRIBUTOR/)