Metadata-Version: 2.4
Name: scrubkit
Version: 0.1.0
Summary: Safe, deterministic cleaner for scraped data: fixes broken encoding, HTML and invisible characters without ever destroying a legitimate value.
Author-email: AI Data Tools <aidatatools@proton.me>
License-Expression: Apache-2.0
Project-URL: Homepage, https://aidatatools.dev
Project-URL: Documentation, https://github.com/aidatatools-dev/scrubkit#readme
Project-URL: Source, https://github.com/aidatatools-dev/scrubkit
Project-URL: Issues, https://github.com/aidatatools-dev/scrubkit/issues
Project-URL: Changelog, https://github.com/aidatatools-dev/scrubkit/blob/main/CHANGELOG.md
Keywords: scraping,web-scraping,data-cleaning,clean-scraped-data,encoding,mojibake,unicode,html,text-cleaning,etl,data-quality,deterministic,csv
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Information Technology
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Text Processing :: General
Classifier: Topic :: Text Processing :: Markup :: HTML
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Topic :: Internet :: WWW/HTTP :: Indexing/Search
Classifier: Typing :: Typed
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
License-File: NOTICE
Provides-Extra: duplicates
Requires-Dist: rapidfuzz>=3.0.0; extra == "duplicates"
Requires-Dist: numpy>=1.24.0; extra == "duplicates"
Provides-Extra: test
Requires-Dist: pytest>=7.0; extra == "test"
Dynamic: license-file

# scrubkit

**Clean scraped data safely.** Fixes broken encoding, leftover HTML and invisible
characters — and never destroys a legitimate value.

[![PyPI](https://img.shields.io/pypi/v/scrubkit.svg)](https://pypi.org/project/scrubkit/)
[![Python](https://img.shields.io/pypi/pyversions/scrubkit.svg)](https://pypi.org/project/scrubkit/)
[![License](https://img.shields.io/pypi/l/scrubkit.svg)](LICENSE)
![Dependencies](https://img.shields.io/badge/dependencies-none-brightgreen)

You scraped 40,000 rows. Some titles say `CafÃ©`. Some still have `<b>` tags in
them. Some rows say *"Please wait, verifying you are human"*. And some cells
contain a zero-width character that quietly breaks every join you run.

scrubkit fixes all of that in one call, tells you what it refused to touch and
why, and gives you a reversible record of every change it made.

```bash
pip install scrubkit
```

```python
from scrubkit import clean

clean("CafÃ© <b>Noir</b>&nbsp;").data
# 'Café Noir'
```

No dependencies. No LLM. No network. Same input always gives the same bytes out.

---

## Clean a scrape in one line

Pass whatever your scraper gave you — a list of rows, a single object, a JSON
string, a CSV string, or a plain string. You get the same shape back, cleaned.

```python
from scrubkit import clean

rows = [
    {"title": "CafÃ© <b>Noir</b>&nbsp;", "price": "US $59.99", "stock": "N/A"},
    {"title": "ThÃ© Vert​",         "price": "US $9.00",  "stock": "12"},
]

result = clean(rows)

result.data
# [{'title': 'Café Noir', 'price': 'US $59.99', 'stock': 'N/A'},
#  {'title': 'Thé Vert',  'price': 'US $9.00',  'stock': '12'}]
```

Nested lists and dicts are walked to any depth, so scraped `specs` and
`variants` arrays are cleaned too.

## What it fixes

| Input | Output | |
|---|---|---|
| `CafÃ©` | `Café` | mojibake (UTF-8 read as cp1252) |
| `CafÃƒÂ©` | `Café` | double- and triple-encoded mojibake |
| `<p>Warm light.</p><p>Fits any room.</p>` | `Warm light. Fits any room.` | HTML tags, with the block break kept |
| `Solid oak &mdash; seats&nbsp;two.` | `Solid oak — seats two.` | HTML entities, decoded once |
| `Tokyo​﻿ Ltd­` | `Tokyo Ltd` | zero-width space, BOM, soft hyphen |
| `Bar Stool` | `Bar Stool` | non-breaking and exotic spaces |
| `"  Lumina  "` | `"Lumina"` | leading and trailing whitespace |

## What it refuses to do

This is the part that matters, and it is why scrubkit exists.

| Input | Left exactly as it was, because |
|---|---|
| `"None"` | it is a real surname |
| `"NA"` | it is Namibia's ISO country code |
| `"captcha"` | it is a real word in real product data |
| `"Please wait, verifying you are human"` | your scrape **failed** — deleting the row would hide that |
| `می‌تواند` (Persian ZWNJ) | the zero-width non-joiner is a required letter-shaping character |
| `👨‍👩‍👧` (emoji ZWJ) | removing the joiner splits one family into three people |
| `AT&T`, `5 < 7` | a real HTML entity ends in `;` — these are not markup |
| `&amp;amp;` | decoding once gives `&amp;`, which is ambiguous — so it is reported, not guessed |
| `3 m²`, `½ cm` | Unicode NFKC would rewrite these to `3 m2` and `1/2` |
| `<div class='cta'>…</div>` in a `description_html` field | the field's name says it holds markup on purpose |

> A cleaner that tells you *"row 42 says 'Access denied', your scrape failed"* is
> worth more than one that silently deletes row 42 and reports a 100% clean
> dataset.

## The boundary: AUTO / opt-in / flag

Every rule sits in exactly one of three tiers, and the tier is part of the API,
not a promise in a README.

| Tier | Meaning | Rules |
|---|---|---|
| **AUTO** | The removed bytes carry no meaning the surviving bytes do not already carry. Applied silently, always recorded. | mojibake repair, HTML tag stripping, entity decoding, invisible-character removal, whitespace normalising, trimming |
| **OPT-IN** | Correct and useful, but it changes something you may depend on — your row count, a value's type, your schema. Never applied unless you ask by name. | nulling placeholders, dropping exact duplicates, coercing numeric text, repairing keys |
| **FLAG** | There is a realistic dataset where applying this destroys real data, and no deterministic rule can tell it apart from the one where the fix is right. Reported with a concrete proposal. **No option turns these on.** | failed-scrape detection, near-duplicates, ambiguous placeholders, NFKC normalisation, HTML-bearing fields, impossible values |

With no options at all, scrubkit **cannot** change your row count, any value's
type, or your schema. Read the full ruleset — every rule with the argument for
its placement — straight from the package:

```python
from scrubkit import RULES

for rule in RULES:
    print(rule.action, rule.rule_id, "—", rule.what)
```

## Reading the flags

```python
result = clean(rows)

for flag in result.flags:
    print(f"{flag['path']}  [{flag['rule']}]")
    print(f"   {flag['reason']}")
```

```
[0].desc  [boilerplate.flag]
   value contains site furniture ('please wait'), which means the extractor
   missed the real content. Nothing here can be repaired -- the content was
   never captured. Re-scrape this record.

[0].stock  [placeholder.null_high_confidence]
   'N/A' is an unambiguous masked-missing marker that every other check counts
   as a present, valid value. Nulling it would change this field's type, so it
   is proposed rather than applied.
   enable_with: placeholder_policy='null_high_confidence'
```

Every flag names the option that would act on it, so you can decide once and
switch it on.

## Opt-ins

```python
result = clean(
    rows,
    coerce_numeric_text=True,                    # "$12.50" -> 12.5
    drop_exact_duplicates=True,                  # remove byte-identical rows
    placeholder_policy="null_high_confidence",   # "N/A" -> None, "None" stays
    repair_keys=True,                            # clean the dict keys too
)
```

| Option | Default | What it changes |
|---|---|---|
| `placeholder_policy` | `"flag"` | `"null_high_confidence"` nulls only unambiguous placeholders; `"null_all"` also nulls `"None"`, `"NA"`, `"-"` — correct only if you know your data has no such legitimate values |
| `drop_exact_duplicates` | `False` | Removes rows byte-identical to an earlier row *after* cleaning. Off by default: dropping rows changes every count and sum downstream |
| `coerce_numeric_text` | `False` | Turns numeric strings into numbers, all-or-nothing per field |
| `repair_keys` | `False` | Cleans dict keys as well as values (changes your schema) |
| `trim_whitespace` | `True` | Trims the edges only — interior line breaks are kept |
| `detect_duplicates` | `True` | Scans for duplicate and near-duplicate rows |

Coercion refuses rather than guesses:

```python
clean([{"price": "12,50"}, {"price": "9,00"}, {"price": "7,25"}],
      coerce_numeric_text=True).data
# unchanged — a lone ',' is a decimal separator in fr/de/es and a malformed
# thousands group in en. The refusal, with that reason, is in .flags
```

## Is it actually deterministic?

Don't take our word for it. The test suite ships **inside** the installed
package:

```bash
pip install "scrubkit[test]"
python -m pytest --pyargs scrubkit
```

It asserts, on realistic scraper fixtures:

- **byte-identical output** — ten runs of the same input produce the same bytes,
  same `output_sha256`, same `replay_id`
- **idempotence** — cleaning the output again changes nothing
- **the control set is untouched** — ten rows of legitimate data that a careless
  cleaner would damage come back byte-for-byte identical, with zero changes
- **reversibility** — the change record alone rebuilds your exact input
- **no new defects** — a repair that would introduce a defect is reverted
- **the boundary matches the code** — every rule in `RULES` is implemented, no
  `FLAG` rule can be switched on by any combination of options

In your own code, the whole claim is two lines:

```python
assert clean(data).output_sha256 == clean(data).output_sha256   # deterministic
assert clean(clean(data).data).data == clean(data).data         # idempotent
```

## Undo anything

```python
result = clean(rows)

result.changes[0]
# {'path': '[0].title', 'path_steps': [0, 'title'],
#  'rule': 'mojibake.roundtrip', 'action': 'AUTO',
#  'before': 'CafÃ© <b>Noir</b>&nbsp;', 'after': 'Café <b>Noir</b>&nbsp;',
#  'detail': {'iterations': 1, 'residual_mojibake': False}}

result.revert() == rows     # True — reconstructed from the record alone
```

The change list is a complete inverse patch, not a log. Store it as JSON and
undo later with `scrubkit.revert(data, changes)`.

## scrubkit vs ftfy

[ftfy](https://github.com/rspeer/python-ftfy) is excellent, and **if your only
problem is broken Unicode, use ftfy** — it is the reference implementation for
mojibake repair and it goes deeper on that one job than scrubkit does.

scrubkit solves the next problem up. Scraped data is not only mis-encoded: it
has HTML left in it, invisible characters, error pages that look like content,
placeholder strings you must not confuse with real values, and rows that are
duplicates of each other. scrubkit handles that whole pipeline, tells you what
it refused to touch and why, and hands you a reversible record.

| | ftfy | scrubkit |
|---|---|---|
| Mojibake / broken encoding | ✅ deeper | ✅ |
| HTML entities | ✅ | ✅ decoded once, refused when ambiguous |
| Control & invisible characters | ✅ | ✅ context-gated for Persian, Arabic, Indic, emoji |
| Unicode normalisation, curly quotes, ligatures | ✅ by default | reported, never applied automatically |
| Stripping HTML **tags** | — | ✅ |
| Detecting a failed scrape (captcha / cookie wall / error page) | — | ✅ flagged |
| Placeholder handling that knows `None` is a surname | — | ✅ |
| Duplicate and near-duplicate rows | — | ✅ |
| Structured rows, nested objects, CSV | — | ✅ |
| Report of what was refused, and why | — | ✅ |
| Reversible change record | — | ✅ |
| Dependencies | 1 | 0 |

They compose. Nothing stops you running ftfy for the Unicode layer and scrubkit
for the pipeline around it.

## Also works on

Not just scraping — anything where text arrived from somewhere you don't
control:

- **Apify / Scrapy / Playwright output** — `clean(dataset_items)`
- **CSV exports from legacy systems** — `clean(open("export.csv").read())`
  returns parsed, cleaned rows
- **Before loading into a database** — invisible characters are what make two
  "identical" keys fail to join
- **Before sending text to an LLM** — mojibake and markup burn tokens and
  confuse extraction

## API

```python
clean(
    data,                              # rows, object, JSON text, CSV text, or str
    *,
    placeholder_policy="flag",         # "flag" | "null_high_confidence" | "null_all"
    drop_exact_duplicates=False,
    coerce_numeric_text=False,
    repair_keys=False,
    trim_whitespace=True,
    detect_duplicates=True,
) -> CleanResult
```

`CleanResult` carries `.data`, `.flags`, `.changes`, `.summary`, `.duplicates`,
`.replay_id`, `.input_sha256`, `.output_sha256`, `.dropped_rows`, plus
`.revert()` and `.to_dict()`.

Bad **options** raise `ValueError` — that is a typo in your code. Bad **data**
never raises: a malformed row comes back with a warning in
`.summary["input"]["warnings"]`, because you wanted your data cleaned, not an
exception in the middle of a scraping run.

### Optional extra

Near-duplicate detection needs `pip install "scrubkit[duplicates]"`. Without it,
exact duplicates still work and the near-duplicate scan degrades to a stated
reason in `.duplicates` — nothing else changes.

```python
clean(rows).duplicates["near"]
# {'skipped': True, 'reason': "near-duplicate matching unavailable in this
#  environment (ModuleNotFoundError: No module named 'rapidfuzz'); exact-
#  duplicate detection above is unaffected"}
```

A scan that quietly found nothing is indistinguishable from a clean dataset, so
it says so instead.

### Known limitation

Combining `placeholder_policy="null_*"` with `coerce_numeric_text=True` is the
one case where cleaning twice differs from cleaning once: nulling placeholders
removes the non-numeric values that were correctly blocking coercion, so a field
refused on the first pass can qualify on the second. It settles immediately, it
cannot happen with the default options, and it is asserted in the test suite so
it cannot get worse silently. Clean once — which is what a pipeline does anyway.

## Hosted API

scrubkit is the library. The same engine runs as a hosted service at
[aidatatools.dev](https://aidatatools.dev) — no install, callable from any
language, plus dataset-level quality scoring that the library does not include.
There is an agent-payable endpoint (x402) and a subscription for teams. The
library is free and stays free.

## License

Apache-2.0. See [LICENSE](LICENSE).
