Metadata-Version: 2.4
Name: opusfilter
Version: 3.4.0
Summary: Toolbox for filtering and processing parallel corpora
Author-email: Mikko Aulamo <mikko.aulamo@helsinki.fi>, Sami Virpioja <sami.virpioja@helsinki.fi>
Maintainer-email: Mikko Aulamo <mikko.aulamo@helsinki.fi>, Sami Virpioja <sami.virpioja@helsinki.fi>
License-Expression: MIT
Project-URL: Homepage, https://github.com/Helsinki-NLP/OpusFilter
Project-URL: Documentation, https://helsinki-nlp.github.io/OpusFilter/
Project-URL: Repository, https://github.com/Helsinki-NLP/OpusFilter.git
Project-URL: Bug Tracker, https://github.com/Helsinki-NLP/OpusFilter/issues
Project-URL: Changelog, https://github.com/Helsinki-NLP/OpusFilter/blob/main/docs/CHANGELOG.md
Project-URL: Paper (ACL 2020), https://www.aclweb.org/anthology/2020.acl-demos.20
Keywords: parallel-corpora,corpus-filtering,natural-language-processing,machine-translation,data-cleaning,nlp,text-processing
Classifier: Development Status :: 5 - Production/Stable
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Text Processing :: Linguistic
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Operating System :: OS Independent
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: setuptools
Requires-Dist: opustools>=1.10.0
Requires-Dist: beautifulsoup4>=4.8.0
Requires-Dist: graphviz
Requires-Dist: py3langid>=0.2.2
Requires-Dist: matplotlib
Requires-Dist: morfessor
Requires-Dist: opus-fast-mosestokenizer>=0.0.8.11
Requires-Dist: pandas>=1.0.0
Requires-Dist: xxhash>=3.2.0
Requires-Dist: sentence-splitter
Requires-Dist: rapidfuzz
Requires-Dist: ruamel.yaml>=0.15.0
Requires-Dist: regex
Requires-Dist: requests
Requires-Dist: scikit-learn
Requires-Dist: subword_nmt
Requires-Dist: tqdm
Requires-Dist: lingua-language-detector>=2.2.0; python_version >= "3.12"
Requires-Dist: lingua-language-detector<2.2,>=1.3.0; python_version >= "3.10" and python_version < "3.12"
Requires-Dist: lingua-language-detector<2.1,>=1.3.0; python_version < "3.10"
Requires-Dist: iso639-lang
Provides-Extra: pycld2
Requires-Dist: pycld2; extra == "pycld2"
Provides-Extra: fasttext
Requires-Dist: py3langid<0.3.0; extra == "fasttext"
Requires-Dist: fasttext; extra == "fasttext"
Provides-Extra: eflomal
Requires-Dist: eflomal>=2.0.0; extra == "eflomal"
Provides-Extra: jieba
Requires-Dist: jieba>=0.42; extra == "jieba"
Provides-Extra: mecab
Requires-Dist: mecab-python3>=1.0.8; extra == "mecab"
Requires-Dist: unidic-lite; extra == "mecab"
Provides-Extra: laser
Requires-Dist: laserembeddings; extra == "laser"
Provides-Extra: varikn
Requires-Dist: varikn; extra == "varikn"
Provides-Extra: heliport
Requires-Dist: heliport>=0.10.0; extra == "heliport"
Provides-Extra: huggingface
Requires-Dist: datasets>=2.14.0; extra == "huggingface"
Provides-Extra: test
Requires-Dist: pytest; extra == "test"
Provides-Extra: docs
Requires-Dist: myst-parser; extra == "docs"
Requires-Dist: sphinx; extra == "docs"
Requires-Dist: sphinx-rtd-theme; extra == "docs"
Requires-Dist: sphinxcontrib-bibtex; extra == "docs"
Provides-Extra: all
Requires-Dist: opusfilter[docs,eflomal,fasttext,heliport,jieba,laser,mecab,pycld2,test,varikn]; extra == "all"
Provides-Extra: dev
Requires-Dist: pytest; extra == "dev"
Requires-Dist: pytest-cov; extra == "dev"
Requires-Dist: black; extra == "dev"
Requires-Dist: isort; extra == "dev"
Requires-Dist: pylint; extra == "dev"
Requires-Dist: mypy; extra == "dev"
Dynamic: license-file

# OpusFilter

OpusFilter is a tool for filtering and combining parallel corpora.

Features:

* Corpus preprocessing pipelines configured with [YAML](https://yaml.org/)
* Simple downloading of parallel corpora from [OPUS](http://opus.nlpl.eu/) with [OpusTools](https://github.com/Helsinki-NLP/OpusTools)
* Implementations for many common text file operations on parallel files
* Memory-efficient processing of large files
* Implemented filters based e.g. on language identification, word aligment, n-gram language models, and multilingual sentence embeddings
* Extendable with your own filters written in Python

OpusFilter has been presented in [ACL 2020 system demonstrations](https://www.aclweb.org/anthology/2020.acl-demos.20).

## Installing

Install the latest release from PyPI:

* `pip install opusfilter` or `pip install opusfilter[all]` (include optional Python libraries)

Install from source:

* `pip install .` or `python setup.py install`

### Troubleshooting

OpusFilter should generally work fine on Python 3.9 to 3.14. In the case of troubles, try installing the exact versions in `requirements.txt`:

* `pip install -r requirements.txt`

## Documentation

The complete OpusFilter documentation is available from [helsinki-nlp.github.io/OpusFilter](https://helsinki-nlp.github.io/OpusFilter/).

You can also build the documents from the source:

* `pip install -r docs/requirements.txt` or  `pip install .[docs]`
* `sphinx-build docs docs-html`

## Changelog

A changelog is available in [docs/CHANGELOG.md](docs/CHANGELOG.md).

## Citing

If you use OpusFilter in your research, please cite our [ACL 2020 paper](https://www.aclweb.org/anthology/2020.acl-demos.20):

```bibtex
@inproceedings{aulamo-etal-2020-opusfilter,
    title = "{O}pus{F}ilter: A Configurable Parallel Corpus Filtering Toolbox",
    author = {Aulamo, Mikko and Virpioja, Sami and Tiedemann, J{\"o}rg},
    booktitle = "Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations",
    month = jul,
    year = "2020",
    publisher = "Association for Computational Linguistics",
    url = "https://www.aclweb.org/anthology/2020.acl-demos.20",
    doi = "10.18653/v1/2020.acl-demos.20",
    pages = "150--156"
}
```

A full bibliography of papers cited in the documentation and code can be found from [docs/references.bib](docs/references.bib).

## Contributing

See [docs/CONTRIBUTING.md](docs/CONTRIBUTING.md).


## Acknowledgements

OpusFilter has partially been supported by various projects such as:

* [MeMAD](https://memad.eu/) - Methods for Managing Audiovisual Data (EU Horizon 2020)
* [FoTran](https://www.helsinki.fi/en/researchgroups/natural-language-understanding) - Found in Translation (EU ERC)
* [HPLT](https://hplt-project.org/) - High-Performance Language Technologies (EU Horizon)
* [GreenNLP](https://greennlp.github.io/) - Controlling the Carbon Footprint in Sustainable Language Technology (Academy of Finland ICT 2023)
