Metadata-Version: 2.4
Name: purism
Version: 1.0.2
Summary: Automatic data filtering library specialized in Korean data purification
Author: Lumia101
Project-URL: Homepage, https://github.com/Lumia101/purism/tree/main
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: torch
Requires-Dist: transformers
Requires-Dist: bitsandbytes
Requires-Dist: lingua-language-detector
Requires-Dist: datasketch[redis]
Requires-Dist: ftfy
Dynamic: license-file

# Purism: Automatic data filtering library specialized in Korean data purification
> **Puri**fy **s**yste**m**

[View in Korean](https://github.com/Lumia101/Purism/blob/main/KO_README.md)

## Summary
This repository is an automatic data filtering library specialized in Korean data purification.
 
# How to use

## Installation
**Installation using pip**
```bash
pip install purism
```

## Quickstart
[C4 dataset](https://huggingface.co/datasets/allenai/c4) has **"Clean"** in its name, but it is **NOT clean** at all (especially the Korean subset). This is a code that performs **additional filtering** on the C4 dataset using this library.

```Python
from purism import PurifyConfig, UnicodeCleaner, UICleaner, TextCleaner, HarmfulWordsFilter, SpamWordsFilter
from datasets import load_dataset
from tqdm.auto import tqdm

# Load dataset
ds = load_dataset(
    "allenai/c4",
    "ko",
    split="train",
    streaming=True
).take(100)

# Load normalization techniques
norms = [
    UnicodeCleaner("NFC"),
    UICleaner(),
    TextCleaner()
]

# Load filters
filters = [
    HarmfulWordsFilter(3),
    SpamWordsFilter(4)
]

# Load the previously defined filters and normalization techniques into PurifyConfig.
purifier = PurifyConfig(filters, norms)

passed = []
filtered = []

for text in tqdm(ds, desc="Filtering...", total=100):
    result = purifier.purify(text["text"]) # Apply the normalization techniques and filters loaded in PurifyConfig.
    if result["passed"]:
        passed.append(result["raw_text"])
    else:
        filtered.append(result["raw_text"])

print("=" * 200)

# Uncensored corpora
for i in range(10):
    print(f"Sample {i + 1} (passed): {passed[i]}")
    print("=" * 200)

# Censored corpora
for i in range(10):
    print(f"Sample {i + 1} (filtered): {filtered[i]}")
    print("=" * 200)
```
You can see that corpus marked as "passed" are better than corpus marked as "filtered".

## API
This library contains many more types of filters in addition to the two mentioned earlier. If you would like to see more features, please visit [this page.](https://github.com/Lumia101/Purism/blob/main/API.md)

# Limitations

* This library can accurately filter only Korean text. Modification of the source code is required to use other languages.
* This library is not always accurate. It can filter out non-harmful corpora, but may fail to filter out some harmful corpora.
