Metadata-Version: 2.4
Name: presidio-redefiner
Version: 0.1.0
Summary: Unofficial community plugin: use ReDefiNER, a rule-following span extractor, as a Presidio recognizer.
Author: Sethuraman T A, Jungroo AI Labs
Author-email: Cibe Hariharan K <cibe@jungroo.com>
License-Expression: Apache-2.0
Project-URL: Homepage, https://github.com/Jungroo-AI/presidio-redefiner
Project-URL: Repository, https://github.com/Jungroo-AI/presidio-redefiner
Project-URL: Issues, https://github.com/Jungroo-AI/presidio-redefiner/issues
Project-URL: ReDefiNER PII, https://huggingface.co/jungroo-ai/redefiner-pii
Keywords: presidio,pii,redaction,named-entity-recognition,redefiner
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: presidio-analyzer>=2.2
Requires-Dist: redefiner>=1.0.9
Provides-Extra: anonymizer
Requires-Dist: presidio-anonymizer>=2.2; extra == "anonymizer"
Provides-Extra: dev
Requires-Dist: pytest; extra == "dev"
Requires-Dist: presidio-anonymizer>=2.2; extra == "dev"
Dynamic: license-file

# presidio-redefiner

A [Presidio](https://github.com/data-privacy-stack/presidio) recognizer that uses
[ReDefiNER](https://github.com/jungroo-ai/redefiner) to find personal data. ReDefiNER is a 0.8B model that extracts
the spans a plain-English rulebook defines, so Presidio can find what your own written policy describes, or what the
default PII rulebook of `jungroo-ai/redefiner-pii` describes.

This is an unofficial community plugin. It is not part of Presidio, and it is not maintained or endorsed by the
Presidio maintainers or by Microsoft.

## Install

```bash
pip install "presidio-redefiner[anonymizer]"
python -m spacy download en_core_web_sm    # for Presidio's slim NLP engine in the example below
```

The first run downloads ReDefiNER's weights (about 0.8 GB) from Hugging Face.

The model weights (`jungroo-ai/redefiner-0.8b`, about 770 MB) are downloaded from Hugging Face the first time the
recognizer analyzes a text, unless they are already in your Hugging Face cache. The model runs on a CPU; an NVIDIA
GPU is used when PyTorch can use CUDA.

## Usage

```python
from presidio_analyzer import AnalyzerEngine
from presidio_analyzer.nlp_engine import NlpEngineProvider
from presidio_anonymizer import AnonymizerEngine
from presidio_redefiner import ReDefiNERRecognizer

nlp = NlpEngineProvider(nlp_configuration={"nlp_engine_name": "slim", "models": [{"lang_code": "en", "model_name": "en_core_web_sm"}]}).create_engine()
analyzer = AnalyzerEngine(nlp_engine=nlp)
analyzer.registry.add_recognizer(ReDefiNERRecognizer())
text = "Please call Rosa Diaz on +44 20 7946 0958 or email rdiaz@example.net."
results = analyzer.analyze(text=text, language="en")
print(AnonymizerEngine().anonymize(text=text, analyzer_results=results).text)
# Please call <PERSON> on <PHONE_NUMBER> or email <EMAIL_ADDRESS>.
```

ReDefiNER runs beside Presidio's own recognizers (email, phone, credit card and other patterns), and Presidio combines
their results. The slim NLP engine gives tokens and lemmas only, without spaCy's named entities, so names and
addresses come from ReDefiNER. With Presidio's default NLP engine, spaCy's entities are added as well.

## Use your own rulebook

Write the types you want found, with a one-line description each, and anything that should never be found under
`NOT_AN_ENTITY`:

```python
# continues the example above
from presidio_analyzer import RecognizerRegistry
from presidio_analyzer.nlp_engine import NoOpNlpEngine

policy = """[RULES]
G1 Extract every piece of personal data of the supplied types. Return exact spans from the text.

[TYPE] PERSON
[DESCRIPTION] A person's name, without titles such as "Dr." or "Ms".

[TYPE] EMPLOYEE_ID
[DESCRIPTION] An employee number, such as "E-48213".

[TYPE] SALARY
[DESCRIPTION] A person's pay: salary, bonus or hourly rate, with its currency.

[TYPE] NOT_AN_ENTITY
NE1 Names of teams, departments and products, and the company's own name.
"""
recognizer = ReDefiNERRecognizer(rulebook=policy, entity_mapping={"EMPLOYEE_ID": "ID"})

registry = RecognizerRegistry()
registry.add_recognizer(recognizer)
analyzer = AnalyzerEngine(registry=registry, nlp_engine=NoOpNlpEngine(models=[{"lang_code": "en", "model_name": "no_op"}]))
text = "Dr. Maria Okafor (E-40917) joins the Payments team at Acme on a salary of $96,500."
results = analyzer.analyze(text=text, language="en")
print(AnonymizerEngine().anonymize(text=text, analyzer_results=results).text)
# Dr. <PERSON> (<ID>) joins the Payments team at Acme on a salary of <SALARY>.
```

- The registry above holds only this recognizer, so only the rulebook decides what is found. If you add the
  recognizer to Presidio's default registry instead, Presidio's own recognizers still mark what they find (every email
  address, date, URL and so on), whatever the rulebook says. The no-op NLP engine needs no spaCy model.
- `rulebook` also takes a `pathlib.Path` to a rulebook file, or a rulebook dict (see `redefiner.to_text`).
- Types come back under their own names unless `entity_mapping` renames them (see below).
- The rulebook format, and which kinds of rules the model follows, are described in the
  [ReDefiNER README](https://github.com/jungroo-ai/redefiner#rulebooks). Rules that go against what the model already
  knows, such as calling a person's name something other than a person, are mostly ignored. Try a rulebook on a few
  dozen of your own texts before you rely on it.

Recognizers that use the same model and settings share it, so a default PII recognizer and several policy recognizers
load the weights once.

## Options

| Argument | Default | Meaning |
|---|---|---|
| `model` | `"jungroo-ai/redefiner-pii"` | A local folder or Hugging Face repo. The default holds a PII rulebook and takes its weights from `jungroo-ai/redefiner-0.8b`. |
| `rulebook` | `None` | Your rulebook as text, a dict or a `pathlib.Path`. `None` uses the model's default rulebook. |
| `entity_mapping` | `None` | Rulebook type to Presidio entity name. Entries add to or override the default mapping; a type mapped to `None` is dropped. |
| `min_confidence` | `0.0` | Drop spans below this ReDefiNER confidence. 0 keeps every span the model returns, as `redefiner redact` does; raise it to drop low-confidence spans. |
| `device` | `None` | `"cpu"`, `"cuda"`, `"cuda:1"`, ... `None` uses CUDA when PyTorch can use it, else the CPU. |
| `max_span_tokens` | `None` | The longest span in model tokens. `None` uses the model's setting. |
| `revision` | `None` | The Hugging Face revision of `model`. |
| `name`, `supported_language`, `version` | `"ReDefiNERRecognizer"`, `"en"`, `"0.1.0"` | As for any Presidio recognizer. The model reads English only. |

## Entity names

Rulebook types are renamed to Presidio's entity names with `presidio_redefiner.DEFAULT_ENTITY_MAPPING`. For the 20
types of the default PII rulebook:

| Rulebook type | Presidio entity |
|---|---|
| `EMAIL` | `EMAIL_ADDRESS` |
| `PHONE` | `PHONE_NUMBER` |
| `ADDRESS` | `LOCATION` |
| `GOVERNMENT_ID`, `ID_NUMBER` | `ID` |
| `PERSON`, `LOCATION`, `DATE_TIME`, `IP_ADDRESS`, `URL`, `ORGANIZATION` | unchanged (already Presidio names) |
| `TITLE`, `USERNAME`, `COORDINATE`, `OCCUPATION`, `AGE`, `GENDER`, `DEMOGRAPHIC`, `FINANCIAL`, `SECRET` | unchanged |

`FINANCIAL` covers card numbers and bank account numbers alike, so it is not renamed to Presidio's `CREDIT_CARD`;
remap it with `entity_mapping` if your pipeline needs another name. The mapping also renames a few names other
rulebooks may use: `NAME` to `PERSON`; `DATE`, `TIME` and `DATE_OF_BIRTH` to `DATE_TIME`; `CARD_NUMBER` and
`CREDIT_CARD_NUMBER` to `CREDIT_CARD`; `IBAN` to `IBAN_CODE`. Any other type keeps its own name.

The recognizer's supported entities are its rulebook's types after this mapping, read from the rulebook when the
recognizer is created. Each result keeps the original rulebook type in
`result.recognition_metadata["redefiner_label"]`, so a `GOVERNMENT_ID` and an `ID_NUMBER`, both reported as `ID`, can
still be told apart.

## How it runs

- Creating the recognizer reads only the rulebook (for the default rulebook, the model's `config.json` and rulebook
  file). The weights load on the first `analyze()` call; call `recognizer.load_model()` to load them earlier.
- In one process, recognizers with the same `model`, `device`, `max_span_tokens` and `revision` share one loaded
  model. A loaded model handles one text at a time, so calls from several threads wait for each other.
- The first text after a new rulebook takes longer, because the rulebook is read by the model once and then cached.
- A result's score is ReDefiNER's confidence for the span. By default every span the model returns is kept, which
  favours finding more personal data over marking less text. To drop low-confidence spans, raise `min_confidence`
  (or pass `score_threshold` to `analyze()`, which filters every recognizer); choose the cutoff on a few dozen of your
  own texts.
- Long texts are split into overlapping windows by ReDefiNER itself, so Presidio's text chunkers are not needed.

## Do not rely on it alone

Automatic redaction should not be your only safeguard. The model can miss personal data, mark text that is not
personal data, or give a span the wrong type, and its confidences are model scores, not guarantees. Check the
output, keep people in the loop for sensitive documents, and combine it with other controls such as access limits
and Presidio's pattern recognizers.

The model can give a number-like value the wrong type (for example, an IP address reported as `PHONE_NUMBER`). The
value is still masked, but filtering by entity type can then miss it, so for redaction keep all types on.

## Tests

```bash
pip install -e ".[dev]"
pytest                 # fast tests, with a fake model
pytest -m slow -s      # loads the real model on CPU from the Hugging Face cache
```

The slow test loads `jungroo-ai/redefiner-pii` (about 0.8 GB of weights on the first run, then cached); set
`REDEFINER_PII_POINTER` to a local pointer folder or another repo to test that instead.

## Licence

Apache License 2.0. See [LICENSE](LICENSE).
