Metadata-Version: 2.5
Name: near-dupes
Version: 0.1.0
Summary: Find near-duplicate text, records and images with one call, then dedupe keeping the best copy
Project-URL: Homepage, https://pypi.org/project/near-dupes/
Project-URL: Author, https://pypi.org/user/pranaymahendrakar/
Author: Pranay Mahendrakar
License-Expression: MIT
License-File: LICENSE
Keywords: deduplication,dhash,duplicate-detection,fuzzy-matching,jaccard,lsh,minhash,near-duplicate,pandas
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.9
Requires-Dist: numpy>=1.23
Requires-Dist: pandas>=1.5
Provides-Extra: dev
Requires-Dist: pillow>=9; extra == 'dev'
Requires-Dist: pyarrow>=12; extra == 'dev'
Requires-Dist: pytest>=7; extra == 'dev'
Provides-Extra: images
Requires-Dist: pillow>=9; extra == 'images'
Provides-Extra: parquet
Requires-Dist: pyarrow>=12; extra == 'parquet'
Description-Content-Type: text/markdown

# near-dupes

Find near-duplicate texts, table rows and images with one call, then drop the copies and keep the best one.

## Install

```
pip install near-dupes
```

Image support needs Pillow: `pip install "near-dupes[images]"`.

## Quickstart

```python
import near_dupes

texts = ["The quick brown fox jumps over the lazy dog.", "the quick brown fox jumps over the lazy dog", "Something else entirely."]
result = near_dupes.find_duplicates(texts)   # or a DataFrame, or a list of image paths
print(result.summary())                       # 1 duplicate group: [0, 1]
print(result.dedupe(texts))                   # keeps the longest copy plus the unrelated text
```

Rows of a DataFrame work the same way: `near_dupes.dedupe(df, key=["name", "city"])` compares rows on
those columns and returns the frame without the copies.

## What it does

- **Text** (`list[str]`): character n-gram shingles, MinHash (128 permutations, fixed seed) and LSH banding
  propose candidate pairs; every candidate is then scored with the exact Jaccard similarity of its shingle
  sets, so the scores you get back are exact. With 2000 or fewer distinct texts there is no MinHash or
  banding at all: every pair is scored exactly (only when the texts are very long is each pair first screened
  by a MinHash estimate with a wide safety margin). Case, Unicode form (NFKC) and whitespace are normalized first.
- **Records** (`pandas.DataFrame`, a `.csv`/`.parquet` path, or a list of dicts): the key columns of each row
  (default: all columns) are normalized for case, whitespace and punctuation, joined into one string and run
  through the text pipeline. Exact duplicates are always found; NaN cells count as empty.
- **Images** (paths or `PIL.Image` objects): 64-bit difference hash (dHash); similarity is `1 - hamming / 64`.
- `threshold` is the minimum similarity in `(0, 1]`; `1.0` means exact duplicates only.
- Duplicate groups are the connected components of the pairs found. One representative per group is kept:
  the longest text, the first row, or the first image.
- Blank items (empty strings, rows whose key cells are all missing) are never treated as duplicates of each other.
- Result indices are 0-based positions in the input, so they line up with `list` positions and `df.iloc`.
- Everything is deterministic: the MinHash permutations come from a fixed `random_state`.

## API

```python
near_dupes.find_duplicates(items, *, kind="auto", threshold=0.85, key=None, n_gram=3) -> DuplicateResult
near_dupes.dedupe(items, **kw)   # find_duplicates(items, **kw).dedupe(items) in one line
```

`items` is a list of strings, a `pandas.DataFrame` (or a `.csv`/`.parquet` path, or a list of dicts), or a
list of image paths / `PIL.Image` objects. `kind` is detected from the input unless given (`"text"`,
`"records"`, `"images"`). `key` names the DataFrame column(s) to compare rows on; `n_gram` is the character
shingle length.

```python
import pandas as pd
df = pd.DataFrame({"name": ["Acme Corp", "ACME Corp.", "Globex"], "city": ["New York", "new york", "Springfield"]})
near_dupes.find_duplicates(df, key=["name", "city"]).groups   # [[0, 1]]
near_dupes.dedupe(df, key=["name", "city"])                   # rows 0 and 2, original index kept
near_dupes.dedupe(["a.jpg", "a_copy.jpg", "b.jpg"])           # image paths; needs near-dupes[images]
```

`DuplicateFinder(kind="auto", threshold=0.85, key=None, n_gram=3, num_perm=128, random_state=0, normalize=True, all_pairs_max=2000)`
is the class underneath, with `.find(items)` and `.dedupe(items)`, for the extra knobs.
`text_similarity(a, b, n_gram=3)` is the exact shingle Jaccard between two strings.

`DuplicateResult`

- `.groups` - `list[list[int]]`, each group of near-duplicate indices (size >= 2), ascending
- `.pairs` - `list[(i, j, score)]`, the edges found (`i < j`); exact duplicates score `1.0`
- `.representatives` - the index kept for each group; `.keep_indices` - all singletons plus one
  representative per group, ascending; `.drop_indices` - what `dedupe` removes
- `.n_duplicates`, `.n_groups`, `.n_items`, `.kind`, `.threshold`
- `.summary()` - human-readable text; `.to_dict()` - JSON-safe dict
- `.dedupe(items)` - `items` with duplicates removed, same type as the input
  (a DataFrame keeps its original index labels)

## CLI

```
near-dupes INPUT [INPUT ...] [--kind auto|text|records|images] [--threshold 0.85]
           [--key COL [COL ...]] [--n-gram 3] [--json] [--output PATH]
```

- `near-dupes contacts.csv --key name email` prints the summary for a table.
- `near-dupes tickets.txt` treats each line as one text.
- `near-dupes photos/` or `near-dupes a.jpg b.jpg` compares images.
- `--json` prints `to_dict()` as JSON; `--output PATH` writes the deduplicated items
  (`.csv`/`.parquet` for tables, one item per line otherwise).

## License

MIT
