Metadata-Version: 2.4
Name: localscrub
Version: 0.1.0
Summary: Local-first PHI/PII de-identification: rule/NER cascade + local LLM. Your data never leaves your machine.
Project-URL: Repository, https://github.com/valbarov/localscrub
Project-URL: Documentation, https://github.com/valbarov/localscrub/blob/main/docs/quickstart.md
Project-URL: Changelog, https://github.com/valbarov/localscrub/blob/main/ROADMAP.md
Author-email: Vadim Albarov <albarov.vadim@gmail.com>
License-Expression: Apache-2.0
License-File: LICENSE
Keywords: de-identification,deidentification,hipaa,local-llm,phi,pii,privacy,redaction
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Healthcare Industry
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Security
Classifier: Topic :: Text Processing :: Filters
Requires-Python: >=3.12
Requires-Dist: typer>=0.12
Provides-Extra: ner
Requires-Dist: torch>=2.13.0; extra == 'ner'
Requires-Dist: transformers>=4.40; extra == 'ner'
Description-Content-Type: text/markdown

# localscrub

Local-first PHI/PII de-identification for clinical and free-form text.

Your data never leaves your machine: a fast rule/NER pass catches the obvious
identifiers, and a **local LLM** handles the context-dependent remainder — no
cloud APIs, no data crossing your trust boundary.

> **Status:** v0.1 — usable, honest about its limits, pre-1.0 API.
> New here? Start with the **[quickstart](docs/quickstart.md)**. Roadmap: [ROADMAP.md](ROADMAP.md).

## Why

Existing options force a bad trade:

- **Cloud de-id APIs** — accurate, but you send the sensitive data out to get it scrubbed.
- **Local rule/NER tools** — private, but miss context-dependent PHI in free text
  ("the patient's sister works at the bakery on Elm Street").

localscrub runs a two-stage cascade entirely on your hardware: cheap detection for
the 90%+, a small fine-tuned local model for the ambiguous rest.

## Usage

```bash
localscrub scrub notes.txt                      # print redacted text
localscrub scrub notes.txt --json               # print the JSON entity report
localscrub scrub notes.txt -o clean.txt --report report.json
localscrub scrub --report-only < notes.txt      # detect without rewriting
localscrub scrub notes.txt --fail-on-detect     # CI gate: exit 3 if anything found
localscrub scrub notes.txt --llm                # + stage 2: local LLM via Ollama
localscrub scrub notes.txt --ner                # + pretrained clinical NER (CPU)

localscrub synth -n 500 --seed 42 -o data/corpora/dev.jsonl   # synthetic eval corpus
localscrub synth -n 500 --seed 42 --diversify -o dev.jsonl    # + local-LLM paraphrase (Ollama)
localscrub mtsamples --fetch -n 100 --seed 42 -o mts.jsonl    # injection benchmark: real prose
localscrub eval data/corpora/dev.jsonl                        # per-entity precision/recall
```

Current accuracy numbers (including what stage 1 honestly cannot catch yet)
live in [docs/results.md](docs/results.md). On our benchmark of authentic
medical-transcription prose, localscrub with the `[ner]` extra fully redacts
**99.9%** of planted identifiers on CPU, vs. **75%** for Presidio, the
standard rules-only baseline — measured by the same harness, reproducible
locally: [docs/benchmark.md](docs/benchmark.md).

Deploying on real clinical text? Read the trust-boundary and residual-risk
discussion first: [docs/threat-model.md](docs/threat-model.md).

Or from Python:

```python
from localscrub import scrub

result = scrub("Reach the patient at test@example.com.")
result.redacted      # 'Reach the patient at [EMAIL_1].'
result.entities      # (Entity(type=EMAIL, start=21, end=37, confidence=0.95, ...),)
result.escalations   # spans stage 1 couldn't settle — stage 2's work queue
```

Stage 1 (rules) currently detects: emails, phone/fax numbers, SSNs, credit
cards (Luhn-validated), IP addresses, URLs, dates and ages over 89, and
labeled MRN / health-plan / account identifiers. Uncertain or ambiguous spans
are redacted fail-closed *and* flagged for stage-2 adjudication. The `[ner]`
extra (`pip install 'localscrub[ner]'`, then `--ner`) adds a pretrained
clinical NER model for person/place names on CPU. Stage 2 (`--llm`, or
`scrub(text, stage2=LlmExtractor())`) runs a local Ollama model for
narrative identifiers and escalation adjudication; see
[ADR 0002](docs/decisions/0002-stage2-llm-interface.md) for its contract.

## Entity coverage

All 18 HIPAA Safe Harbor identifier categories (45 CFR 164.514(b)(2)), plus
generic PII that matters in practice: credit cards, passports, national IDs,
and leaked credentials/secrets. See `src/localscrub/entities.py`.

## License

Apache-2.0 — see [LICENSE](LICENSE).
