Metadata-Version: 2.4
Name: candle-deduplicator
Version: 1.0.0
Summary: ONNX-based Arabic Consecutive-Character Deduplicator (Candle model by Abjad AI)
Home-page: https://github.com/abjadai/candle
Author: Faris Alasmary
Author-email: farisalasmary@gmail.com
Classifier: Programming Language :: Python :: 3
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Operating System :: OS Independent
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Natural Language :: Arabic
Requires-Python: >=3.10
Description-Content-Type: text/markdown
Requires-Dist: onnxruntime>=1.16.0
Requires-Dist: numpy>=1.19.0
Requires-Dist: tqdm>=4.60.0
Provides-Extra: gpu
Requires-Dist: onnxruntime-gpu>=1.16.0; extra == "gpu"
Dynamic: author
Dynamic: author-email
Dynamic: classifier
Dynamic: description
Dynamic: description-content-type
Dynamic: home-page
Dynamic: provides-extra
Dynamic: requires-dist
Dynamic: requires-python
Dynamic: summary

# candle-deduplicator

[![License](https://img.shields.io/badge/License-Apache_2.0-blue.svg)](https://opensource.org/licenses/Apache-2.0)

ONNX-based Arabic consecutive-character deduplication powered by the
[CANDLE](https://github.com/abjadai/candle) model from Abjad AI.

The model removes unintentional character elongation that appears in
user-generated Arabic text — for example:

| Input | Output |
|---|---|
| `الممملكككة الأررررردنييية الللهههااااششميية` | `المملكة الأردنية الهاشمية` |
| ` الممملكككة الأررررردنييية ررراااننيييااا` | `الملكة الأردنية رانيا` |
| `مررررحبببببا` | `مرحبا` |

---

## Installation

```bash
pip install candle-deduplicator
```

For GPU inference, uninstall `onnxruntime` and install the GPU variant instead:

```bash
pip uninstall onnxruntime
pip install candle-deduplicator onnxruntime-gpu
```

ONNX model weights are downloaded automatically on first use and cached
inside the package directory.

---

## Quick start

### Single string

```python
from candle_deduplicator import CandleDeduplicator

model = CandleDeduplicator()
print(model.deduplicate('الممملكككة الأررررردنييية الللهههااااششميية'))
# المملكة الأردنية الهاشمية
```

### Batch inference

```python
texts = [
    'الممملكككة الأررررردنييية',
    'مررررحبببببا بببالعررربييية',
]
results = model.deduplicate_batch(texts, batch_size=32)
print(results)
```

### Distilled (faster) variant

```python
from candle_deduplicator import CandleDeduplicatorDistilled

model = CandleDeduplicatorDistilled()
print(model.deduplicate('مررررحبببببا'))
# مرحبا
```

### Load a local ONNX file

Pass `model_path` to skip the auto-download and load directly from disk:

```python
model = CandleDeduplicator(model_path="full_model.onnx")
model = CandleDeduplicatorDistilled(model_path="distilled_model.onnx")
```

### GPU inference

```python
model = CandleDeduplicator(model_path="full_model.onnx", device="cuda")
# ✅ Loaded model successfully! (provider: CUDAExecutionProvider)
```

If `onnxruntime-gpu` is not installed or CUDA is unavailable, ONNX Runtime
falls back to CPU automatically and the printed provider will reflect that.

---

## API reference

`CandleDeduplicator(model_path=None, auto_preprocess=True, device="cpu")`

Full 6-layer encoder model. Higher accuracy, recommended for offline
batch processing.

`CandleDeduplicatorDistilled(model_path=None, auto_preprocess=True, device="cpu")`

Distilled 2-layer encoder model. Faster inference, recommended for
latency-sensitive applications.

Both classes share the same interface:

| Method | Description |
|---|---|
| `deduplicate(text)` | Deduplicate a single string. |
| `deduplicate_batch(texts, batch_size=16, verbose=True)` | Deduplicate a list of strings. |

**Constructor parameters**

| Parameter | Type | Default | Description |
|---|---|---|---|
| `model_path` | `str` or `None` | `None` | Path to a local `.onnx` file. When `None` the model is downloaded automatically from GitHub Releases. |
| `auto_preprocess` | `bool` | `True` | Strip non-Arabic characters before running the model. |
| `device` | `str` | `"cpu"` | Execution device: `"cpu"` or `"cuda"`. GPU requires `onnxruntime-gpu`. |

---

## How it works

The CANDLE model uses an encoder-only Transformer trained with CTC loss.
During inference:

1. Each input string is checked for consecutive duplicate characters.
   Strings that are already clean skip the model entirely.
2. Strings with duplicates are normalised to at most two consecutive
   occurrences of each character and tokenised at the character level.
3. The encoder produces per-token logits; greedy argmax selects the most
   likely token at each position.
4. CTC greedy decoding collapses repeated tokens and removes blank
   symbols to recover the deduplicated text.
5. A word-level guard ensures that words without duplicates in the
   original text are never altered by the model.

---

## License

Apache 2.0 — see [LICENSE](LICENSE).
