Metadata-Version: 2.4
Name: removedup
Version: 1.1.0
Summary: Remove duplicates from parallel corpora
Home-page: https://github.com/LibreTranslate/RemoveDUP
Author: Piero Toffanin
Author-email: pt@masseranolabs.com
Requires-Python: >=3.7
Description-Content-Type: text/markdown
License-File: LICENSE
Provides-Extra: test
Requires-Dist: pytest>=6.0; extra == "test"
Dynamic: author
Dynamic: author-email
Dynamic: description
Dynamic: description-content-type
Dynamic: home-page
Dynamic: license-file
Dynamic: provides-extra
Dynamic: requires-python
Dynamic: summary

# RemoveDup

A fast, memory efficient Python module to remove duplicates from parallel text corpora.

It's useful for cleaning up datasets that contain duplicate entries for training language models.

## Installation

```bash
pip install removedup
```

## Usage

```python
from removedup import rdup

src, tgt, removed = rdup("source.txt", "target.txt")
print(src, tgt, removed)
# source.txt.dedup
# target.txt.dedup
# <num lines removed>
```

## Notes

Source and target must have the same number of lines. No validation checks are made.

Duplication checks are made both on the source and target content. E.g. duplication needs to appear on both source and target.

This is a breaking change from versions < `1.1.0`.

## Build

```bash
git clone https://github.com/LibreTranslate/RemoveDup
cd RemoveDup
python setup.py build
```

## Standalone Binary

You can also use removedup as a standalone Windows, macOS or Linux application (but you currently need to build from source, we don't provide binaries).

```
mkdir build
cd build && cmake .. && make -j4
./rdup source.txt target.txt
```

## Contributing

We welcome pull requests!

## License

AGPLv3
