Metadata-Version: 2.4
Name: privyscope-en
Version: 0.1.2
Summary: privyscope-en — English language pack for the privyscope PII engine
Author: privyscope Core Team
License: Apache-2.0
Project-URL: Homepage, https://github.com/zafrem/privyscope-en
Project-URL: Documentation, https://github.com/zafrem/privyscope-en/blob/main/README.md
Keywords: pii,ner,redaction,privacy,english,onnx
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Text Processing :: Linguistic
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: privyscope>=0.1.0
Provides-Extra: train
Requires-Dist: privyscope[train]>=0.1.0; extra == "train"
Provides-Extra: dev
Requires-Dist: pytest>=7.0; extra == "dev"
Requires-Dist: pytest-cov>=4.0; extra == "dev"
Dynamic: license-file

# privyscope-en

English **PII detection & masking** engine — part of the
[privyscope](https://github.com/zafrem) series. Detects and masks person names,
phone numbers, national IDs, emails, addresses, financial info, private dates,
and credentials in English text.

> ⚠️ privyscope is a redaction **aid**, not an anonymization or compliance
> guarantee. See [Limitations](#limitations).

## Install

```bash
pip install privyscope-en
```

## 60-second quickstart

**Python**

```python
from privyscope_en import Privyscope

engine = Privyscope.from_pretrained()                 # downloads ONNX weights on first run
result = engine.redact("John Smith's number is 555-123-4567")

result.masked_text        # "<PER>'s number is <PHONE>"
result.detected_spans     # [DetectedSpan(label="PER", start=0, end=10, ...), ...]
result.summary            # {"span_count": 2, "by_label": {"PER": 1, "PHONE": 1}, ...}
```

**CLI**

```bash
privyscope redact "John Smith's number is 555-123-4567"
cat notes.txt | privyscope redact --operating-point high_recall
```

## Documentation

Full guides live in [`docs/`](docs/) — organised by what you want to do:

| I want to… | Guide |
|---|---|
| Run it from the terminal | [CLI Reference](docs/CLI.md) |
| Call it from Python | [Python API Reference](docs/API.md) |
| Understand the JSON output | [Output Schemas](docs/OUTPUT_SCHEMAS.md) |
| Score it on my labelled data | [Evaluation & Output Modes](docs/EVAL_AND_OUTPUT_MODES.md) |
| Trade precision vs recall | [Operating Points](docs/OPERATING_POINTS.md) |
| Run it offline / air-gapped | [Offline Usage](docs/OFFLINE.md) |
| Fine-tune on my own data | [Fine-tuning](docs/FINETUNING.md) |

## Entities

`PER` · `PHONE` · `ID_NUM` · `EMAIL` · `LOC` · `BANK` · `DATE` · `SECRET`.
Regex patterns live in [`privyscope/regex_rules.yaml`](privyscope/regex_rules.yaml)
(user-extensible, no code change); EN-specific extended entities in
[`privyscope/entity_config.yaml`](privyscope/entity_config.yaml).

## How it works

A **two-stage hybrid pipeline** (SRS §3.4), results merged via Union:

1. **Regex filter** — structurally obvious PII (phone, email, IDs, cards, secrets).
2. **ONNX NER** — a BIOES token classifier with a constrained Viterbi decoder for
   contextual PII (names, addresses, private dates).

Inference is **ONNX Runtime only — no PyTorch at runtime**. Recall-first, with
runtime [operating-point](docs/OPERATING_POINTS.md) tuning (no retraining).
PyTorch is needed only to [fine-tune](docs/FINETUNING.md).

## Model & performance

- **Architecture** — `roberta-base` encoder → BIOES token-classification head →
  constrained Viterbi decoder.
- **Runtime artifact** — INT8-quantized ONNX, **~120 MB** (under the ≤ 150 MB
  budget), max sequence length 256. Weights download from Hugging Face Hub on
  first use, with a SHA-256 `checksum.txt` for integrity verification.
- **Accuracy** — entity-level **strict micro-F1 = 0.997** on a held-out
  validation set (2,000 sentences, disjoint from training; `typed`/strict
  scoring over the full regex + NER pipeline). Per-entity strict F1:

  | PER | LOC | DATE | ID_NUM | BANK | PHONE | SECRET |
  |-----|-----|------|--------|------|-------|--------|
  | 1.00 | 1.00 | 1.00 | 0.99 | 0.99 | 0.98 | 1.00 |
  (I think it is overfit...)

  (`EMAIL` is matched deterministically by the regex stage; the sample contained
  no `EMAIL` instances.) The set is disjoint from training, so the score reflects
  generalization rather than memorization — out-of-distribution text (unusual
  names or contexts) will score lower. Reproduce with `privyscope eval --lang en
  your_val.jsonl`; see [Evaluation & Output Modes](docs/EVAL_AND_OUTPUT_MODES.md).

## Limitations

- Not an anonymization/compliance guarantee; use as one layer of privacy-by-design.
- Known failure modes: under-detection of uncommon/regional names; over-redaction
  of public entities in ambiguous contexts; fragmented spans in heavily
  mixed-format text; missed `SECRET` for novel credential formats.
- Extra human review recommended for medical/legal/financial/government workflows.

## License

Apache-2.0. Weights are distributed on Hugging Face Hub under Apache-2.0 with a
`checksum.txt` (SHA-256) for integrity verification. Contributions welcome — see
[CONTRIBUTING.md](CONTRIBUTING.md).
