Metadata-Version: 2.4
Name: textdirectory
Version: 0.4.1
Summary: Aggregate multiple text files into one, with filters and transformations.
Project-URL: Homepage, https://github.com/IngoKl/textdirectory
Project-URL: Documentation, https://textdirectory.readthedocs.io
Project-URL: Repository, https://github.com/IngoKl/textdirectory
Project-URL: Issues, https://github.com/IngoKl/textdirectory/issues
Author-email: Ingo Kleiber <ingo@kleiber.me>
License-Expression: MIT
License-File: LICENSE
Keywords: aggregation,corpus,nlp,text,textdirectory
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Natural Language :: English
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Text Processing :: Linguistic
Requires-Python: >=3.10
Requires-Dist: beautifulsoup4>=4.11
Requires-Dist: click>=8.1
Requires-Dist: ftfy>=6.1
Requires-Dist: lxml>=5.0
Requires-Dist: requests>=2.32
Requires-Dist: tqdm>=4.64
Provides-Extra: nlp
Requires-Dist: psutil>=5.9; extra == 'nlp'
Requires-Dist: spacy<4,>=3.8; extra == 'nlp'
Description-Content-Type: text/markdown

# TextDirectory

[![PyPI](https://img.shields.io/pypi/v/textdirectory.svg)](https://pypi.org/project/textdirectory/)
[![CI](https://github.com/IngoKl/textdirectory/actions/workflows/ci.yml/badge.svg)](https://github.com/IngoKl/textdirectory/actions/workflows/ci.yml)
[![Documentation Status](https://readthedocs.org/projects/textdirectory/badge/?version=latest)](https://textdirectory.readthedocs.io/en/latest/?badge=latest)
[![Python Versions](https://img.shields.io/pypi/pyversions/textdirectory.svg)](https://pypi.org/project/textdirectory/)

![TextDirectory](https://user-images.githubusercontent.com/16179317/39367680-cd409a00-4a37-11e8-8d42-0bed5a4e814b.png)

*TextDirectory* allows you to combine multiple text files into one aggregated file. TextDirectory also supports matching
files for certain criteria and applying transformations to the aggregated text.

*TextDirectory* can be used as a mere tool (via the CLI) and as a Python library.

Of course, everything *TextDirectory* does could be achieved in bash or PowerShell. However, there are certain
use cases (e.g., when used as a library) in which it might be useful.

* Free software: MIT license
* Documentation: <https://textdirectory.readthedocs.io>

## Features

* Aggregating multiple text files
* Filtering documents/texts based on various parameters such as length, content, and random sampling
* Filtering and transforming text files
* Transforming the aggregated text (e.g., transforming the text to lowercase)

| Version | Filters | Transformations |
| --- | --- | --- |
| 0.1.0 | filter_by_max_chars(n int); filter_by_min_chars(n int); filter_by_max_tokens(n int); filter_by_min_tokens(n int); filter_by_contains(str); filter_by_not_contains(str); filter_by_random_sampling(n int; replace=False) | transformation_lowercase |
| 0.1.1 | filter_by_chars_outliers(n sigmas int) | transformation_remove_nl |
| 0.1.2 | filter_by_filename_contains(contains str) | transformation_usas_en_semtag; transformation_uppercase; transformation_postag(spacy_model str) |
| 0.1.3 | filter_by_similar_documents(reference_file str; threshold float) | transformation_remove_non_ascii; transformation_remove_non_alphanumerical |
| 0.2.0 | filter_by_max_filesize(max_kb int); filter_by_min_filesize(min_kb int) | transformation_to_leetspeak; transformation_crude_spellchecker(language model str) |
| 0.2.1 | None | transformation_remove_stopwords(stopwords_source str; stopwords str [en]; spacy_model str; custom_stopwords str); transformation_remove_htmltags |
| 0.3.0 | None | transformation_remove_weird_tokens(spaCy model; remove_double_space=False); transformation_lemmatize(spaCy model) |
| 0.3.2 | None | transformation_expand_english_contractions |
| 0.3.3 | filter_by_filenames(filenames list); filter_by_filename_not_contains(not_contains str) | transformation_eebop4_to_plaintext; transformation_replace_digits(replacement_character str); transformation_ftfy |
| 0.3.4 | filter_by_type_token_ratio(min_ttr float; max_ttr float) | transformation_replace_string(replace str; replace_with str) |
| 0.4.0 | None (modernization release) | None (modernization release) |

## Installation

Install *TextDirectory* via pip:

```bash
pip install textdirectory
```

The spaCy-based transformations (`transformation_postag`, `transformation_lemmatize`,
`transformation_remove_stopwords`, `transformation_remove_weird_tokens`) are optional. To use them, install the
`nlp` extra and download a spaCy model:

```bash
pip install 'textdirectory[nlp]'
python -m spacy download en_core_web_sm
```

## Quickstart

*TextDirectory*, as exemplified below, works with a two-stage model. After loading in your data (directory) you can,
by applying *filters*, iteratively select the files you want to process.
In a second step you can perform transformations on the text before finally aggregating it to either a file or memory.

![TextDirectory](https://user-images.githubusercontent.com/16179317/39367589-7f774116-4a37-11e8-9a09-5cbdf5f3311b.png)

### As a Command-Line Tool

*TextDirectory* comes equipped with a CLI (`textdirectory`, also available as `python -m textdirectory`).

The syntax for both the *filters* and *transformations* works similarly. They are chained by adding slashes (/) and
parameters are passed via commas (,): `filter_by_min_tokens,5/filter_by_random_sampling,2`.

#### Example 1: A Very Simple Aggregation

```bash
textdirectory --directory testdata --output_file aggregated.txt
```

This will take all files (.txt) in *testdata* and then aggregates the files into a file called *aggregated.txt*.

You could also use '*' as a wildcard for filetype if you need to include all files and not just .txt:

```bash
textdirectory --directory testdata --output_file aggregated.txt --filetype *
```

#### Example 2: Applying Filters and Transformations

In this example, we want to filter the files based on their token count, perform a random sampling and finally
transform all text to lowercase:

```bash
textdirectory --directory testdata --output_file aggregated.txt --filters filter_by_min_tokens,5/filter_by_random_sampling,2 --transformations transformation_lowercase
```

After passing two filters (*filter_by_min_tokens* and *filter_by_random_sampling*) we've applied the
*transformation_lowercase* transformation.

The resulting file will contain the content of two files that each have at least five tokens.

The same transformation can also be staged multiple times. For example, we could stage two simple string replacements
like this:

```bash
textdirectory --directory testdata --output_file aggregated.txt --transformations transformation_replace_string,lorem,x/transformation_replace_string,ipsum,y
```

If the `--output_file` is missing, the CLI will output the aggregated text to the console. In this scenario, there is
no progress bar visible (can be controlled using `--disable_tqdm True/False`).

### As a Python Library

In order to demonstrate *TextDirectory* as a Python library, we'll recreate the second example from above:

```python
import textdirectory

td = textdirectory.TextDirectory(directory='testdata')
td.load_files(recursive=False, filetype='txt', sort=True)
td.filter_by_min_tokens(5)
td.filter_by_random_sampling(2)
td.stage_transformation(['transformation_lowercase'])
td.aggregate_to_file('aggregated.txt')
```

If we don't have special requirements, we can also call
`td = textdirectory.TextDirectory(directory='testdata', autoload=True)` to skip manually calling `load_files`.
If we wanted to keep working with the actual aggregated text, we could have called `text = td.aggregate_to_memory()`
instead of `aggregate_to_file`.

```python
import textdirectory

td = textdirectory.TextDirectory(directory='testdata', autoload=True)
td.get_text(0)
```

Sometimes we might want to get the actual text of a given file. This can be achieved as seen above. The `get_text`
method will return the transformed text if it is available. Otherwise, it will simply read the file and return the
text.

Every applied filter will create a *state* (i.e., a checkpoint). If we want to go back to a previous state, we can
print all states by calling `td.print_saved_states()`. Previous states can then be loaded by calling
`td.load_aggregation_state(state=0)`.

It's also possible to pass arguments to the individual transformations. In order to do this (at the moment) you have
to adhere to the correct order of arguments:

```python
# def transformation_remove_stopwords(text, stopwords_source='internal', stopwords='en', spacy_model='en_core_web_sm', custom_stopwords=None, *args)
td.stage_transformation(['transformation_remove_stopwords', 'internal', 'en', 'en_core_web_sm', 'dolor'])
```

In the above example, we are adding additional custom stopwords to the transformer.

You also might not always want to aggregate texts into one file in many cases but filter and transform them:

```python
import textdirectory

td = textdirectory.TextDirectory(directory='input')
td.load_files()

td.filter_by_max_chars(480)
td.stage_transformation(['transformation_to_leetspeak'])

td.transform_to_files('output')
```

In the example above, we are loading all files in `input`. After filtering and transforming, the modified files will
be written to `output`.

In addition, there are a few [simple examples](https://github.com/IngoKl/textdirectory/tree/master/examples) in the
repository for you to look at.

## Special Transformations

### transformation_crude_spellchecker

This is a very crude spellchecker implemented following Peter Norvig's excellent
[tutorial](http://norvig.com/spell-correct.html). It's really more of an experiment and not production ready in any
sense of the word.

### transformation_eebop4_to_plaintext

This is a highly specific transformation that will extract the plain text from an EEBO-TCP P4 corpus file.
Both the header as well as all XML tags will be removed during this transformation.

### transformation_ftfy

This transformation simply applies `ftfy.fix_text` to the text.
It is highly recommended to use [ftfy](https://pypi.org/project/ftfy/) as a first transformation if you are working
with messy Unicode text.

## Notes for Developers

The project uses [uv](https://docs.astral.sh/uv/) for development:

```bash
uv sync --group dev --extra nlp
uv pip install https://github.com/explosion/spacy-models/releases/download/en_core_web_sm-3.8.0/en_core_web_sm-3.8.0-py3-none-any.whl

# Later syncs: use --inexact, otherwise the spaCy model is removed again
uv sync --inexact --group dev --extra nlp

uv run pytest              # run the test suite (network tests excluded by default)
uv run pytest -m network   # run the tests that hit live external services
uv run ruff check src tests
uv run mypy
```

See [CONTRIBUTING.md](CONTRIBUTING.md) for the full development guide and
[the releasing documentation](https://textdirectory.readthedocs.io/en/latest/releasing.html) for how releases are
built and published.

## Behavior

We are not holding the actual texts in memory. This leads to much more disk read activity (and time inefficiency),
but saves memory. Of course, this is not the case when using `aggregate_to_memory`.

`transformation_usas_en_semtag` relies on the web version of
[Paul Rayson's USAS Tagger](http://ucrel.lancs.ac.uk/usas/). **This transformation uploads the full text of every
processed file to a third-party server operated by Lancaster University. Do not use it with confidential, personal,
or licensed data.** Don't use this transformation for large amounts of text, give credit, and consider using their
commercial product [Wmatrix](http://ucrel.lancs.ac.uk/wmatrix/).

If you are working with a lot of files, it might be wise to use `load_files(fast=True, skip_checkpoint=True)`.
This will load files much quicker but skip collecting metadata. Filters that need that metadata (character and
token counts) will then raise a `ValueError`.

## Credits

This package was originally based on the
[audreyr/cookiecutter-pypackage](https://github.com/audreyr/cookiecutter-pypackage) cookiecutter template.
