Metadata-Version: 2.4
Name: lightspeed-data-anonymizer
Version: 0.1.0
Summary: PII and sensitive-data redaction for Lightspeed assistants, built on Presidio.
License: Apache-2.0
License-File: LICENSE
License-File: NOTICE
Author: UXE Data Foundations and AI Solutions
Author-email: xe-support-ai-notifications@redhat.com
Maintainer: Anik Bhattacharjee
Maintainer-email: anbhatta@redhat.com
Requires-Python: >=3.12,<3.14
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Requires-Dist: presidio-analyzer (>=2.2.364,<3.0.0)
Requires-Dist: presidio-anonymizer (>=2.2.364,<3.0.0)
Project-URL: Repository, https://github.com/anik120/lightspeed-data-anonymizer
Description-Content-Type: text/markdown

# Lightspeed Data Anonymizer

A PII and sensitive-data redaction library for Lightspeed assistants and the
Lightspeed AI observability pipeline, maintained by the lightspeed-core team.

> **Derived from `data-anonymizer`.** This project is seeded from
> [`data-anonymizer`](https://gitlab.cee.redhat.com/uxe-data-ai-solutions/data-anonymizer),
> built by Red Hat's UXE Data & AI Solutions team for the Ask Red Hat agent, and
> reused with the authors' permission. The lightspeed-core team maintains this
> fork and owns its tuning, testing, accuracy, and support for Lightspeed
> workloads. See [NOTICE](./NOTICE) for attribution.

## Introduction

Lightspeed Data Anonymizer is a standalone PII/sensitive-data redaction library
for Red Hat AI use cases, implemented as a wrapper around the Presidio SDK.

PII entities that can be processed using data-anonymizer:

- Email Address
- Hostname
- IP Address (IPv4, IPv6)
- Location
- Organization (off by default)
- Person
- Phone Number
- URL

Organization names are exempt from anonymization and are left in the text. Detecting them
also flags product and technology names such as OpenShift, RHEL and Sosreport, so enable it
only if you need it:

```
anonymizer = PIIAnonymizer(organization=True, allow_list=["RHEL", "OpenShift"])
```

Locations written in all caps are detected by a second pass over a title cased copy of the
text. What it costs depends on the text: roughly 2% on prose with no all caps words, 14% on
a typical mixed sentence, and up to 80% on an address block, which is the case that needs
it. Turn it off to trade that recall for latency:

```
anonymizer = PIIAnonymizer(case_normalization=False)
```

### Upgrading from 0.2.0

`organization` now defaults to `False`, so **organization names that 0.2.0 redacted are left
in the text**. This is deliberate, for the reason above, but it changes what leaves the
library. Callers that relied on the old behaviour have to ask for it:

```
anonymizer = PIIAnonymizer(organization=True)
```

### Choosing a model

Detection quality is set by the spaCy model behind it. Install `en_core_web_lg` as the
default; `en_core_web_trf` detects substantially more locations and people, but depends on
torch, which adds roughly 500 MB to an image:

| Model | Locations | People | Median latency | Peak memory | Extra dependency |
| --- | --- | --- | --- | --- | --- |
| `en_core_web_lg` (default) | 7/23 | 0/4 | ~13 ms | ~0.9 GB | none |
| `en_core_web_trf` | 19/23 | 4/4 | ~64 ms | ~2.7 GB | torch |

Memory is usually the deciding factor rather than latency. Check that the environment has
the headroom for the transformer before enabling it.

Detection of emails, hostnames, IP addresses and phone numbers is by regex and is identical
on both models.

Install models with spaCy (they are not PyPI packages, so they are not pulled in
automatically):

```
python -m spacy download en_core_web_lg      # default
python -m spacy download en_core_web_trf     # also: pip install torch
```

Both can be installed side by side, so an environment can switch between them without
reinstalling. Select the model with the `DATA_ANONYMIZER_SPACY_MODEL` environment variable,
so a deployment can switch without a code change in the calling service:

```
export DATA_ANONYMIZER_SPACY_MODEL=en_core_web_trf
```

The `spacy_model` argument overrides the environment variable:

```
anonymizer = PIIAnonymizer(spacy_model="en_core_web_trf")
```

Naming a model that is not installed raises on construction, so a misconfigured environment
fails at startup rather than silently anonymizing less than it should.

To always redact specific terms, regardless of what the model detects, pass a deny list.
Naming a term here enables its entity type even when that type is otherwise off:

```
anonymizer = PIIAnonymizer(deny_list={"ORGANIZATION": ["Acme Corp"], "LOCATION": ["Novo mesto"]})
```

## Getting Started

1. Install the library and a spaCy model.

```
pip install lightspeed-data-anonymizer
python -m spacy download en_core_web_lg
```

(From source instead: `pip install "git+https://github.com/anik120/lightspeed-data-anonymizer.git"`.)

2. Import the PIIAnonymizer

```
from lightspeed_data_anonymizer.anonymizer import PIIAnonymizer
```

3. Instantiate and use the anonymizer.

```
anonymizer = PIIAnonymizer()

To anonymize a list of strings:
anonymized_list = anonymizer.anonymize_list([str1, str2, str3])

To anonymize a text/string:
anonymized_text = anonymizer.anonymize_text(text)
```
