Metadata-Version: 2.5
Name: euredact
Version: 0.5.0
Summary: European PII redaction SDK — rule engine for 31 countries
Project-URL: Homepage, https://euredact.dev
Project-URL: Documentation, https://euredact.dev/docs
Project-URL: Repository, https://github.com/euredact/euredact
Project-URL: Changelog, https://github.com/euredact/euredact/blob/main/euredact-python/CHANGELOG.md
Project-URL: Issues, https://github.com/euredact/euredact/issues
Author: JNJS
License-Expression: Apache-2.0
License-File: LICENSE
Keywords: anonymization,europe,gdpr,masking,pii,privacy,redaction,referential-integrity
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Security
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Topic :: Text Processing
Classifier: Typing :: Typed
Requires-Python: >=3.10
Provides-Extra: cloud
Requires-Dist: httpx>=0.27; extra == 'cloud'
Provides-Extra: coref
Provides-Extra: dev
Requires-Dist: google-re2>=1.1; extra == 'dev'
Requires-Dist: httpx>=0.27; extra == 'dev'
Requires-Dist: mypy; extra == 'dev'
Requires-Dist: pyahocorasick>=2.0.0; extra == 'dev'
Requires-Dist: pytest-asyncio>=0.21; extra == 'dev'
Requires-Dist: pytest-benchmark; extra == 'dev'
Requires-Dist: pytest>=7.0; extra == 'dev'
Requires-Dist: ruff; extra == 'dev'
Provides-Extra: fast
Requires-Dist: google-re2>=1.1; extra == 'fast'
Requires-Dist: pyahocorasick>=2.0.0; extra == 'fast'
Description-Content-Type: text/markdown

# EuRedact

**European PII redaction SDK -- rule engine**

---

EuRedact is a pure-Python SDK for detecting and redacting personally identifiable
information (PII) in European text data. It covers 31 countries with a two-pass
architecture: liberal pattern matching in the first pass, followed by suppression
filters and checksum validation in the second. The library has zero required
dependencies, is thread-safe, and produces immutable detection objects.

## Quick Start

```bash
pip install euredact
```

```python
import euredact

result = euredact.redact("Mijn BSN is 111222333 en IBAN NL91ABNA0417164300.")
print(result.redacted_text)
# "Mijn BSN is [NATIONAL_ID] en IBAN [BANK_ACCOUNT]."

print(result.detections)
# [Detection(entity_type=<EntityType.NATIONAL_ID>, ...), Detection(entity_type=<EntityType.BANK_ACCOUNT>, ...)]
```

## Features

- **31 European countries** (see list below)
- **41 PII entity types** (30 from the rules engine, 11 from the
  [cloud tier](#cloud-tier)): national IDs, IBANs, phone numbers, email addresses,
  VAT numbers, license plates, VIN, credit cards, BIC/SWIFT, IMEI, GPS
  coordinates, UUIDs, social handles, MAC addresses, IP/IPv6 addresses, health
  insurance numbers, passport numbers, driver's licenses, secrets/API keys, and more
- **Secret/API key detection:** known-prefix patterns for AWS, GitHub, Stripe,
  OpenAI, Slack, JWT, SendGrid, plus Shannon entropy-based detection for generic
  high-entropy tokens near context keywords
- **Custom patterns:** register your own regex patterns for domain-specific PII
  types at runtime via `add_custom_pattern()`
- **Reversible tokenization:** `tokenize=True` swaps values for `EMAIL_K7Q2`-style
  tokens and `restore()` puts them back — for prompts that go to an LLM and
  come back
- **Allowlist:** values that are never redacted, such as your own organisation's
  name and addresses, per call or per instance
- **Checksum validation:** IBAN mod-97, Luhn (credit cards), and 30+ country-specific
  national ID checksums (e.g., Dutch BSN 11-proof, Belgian national number modulo)
- **Priority-aware deduplication:** when matches overlap, validated patterns
  (with passing checksums, corroborated by the document's country) win over
  custom patterns, which win over regex-only patterns; a failed checksum demotes
  same-type matches rather than deleting them, so it can never silence a
  detection
- **Two-pass detection:** liberal regex matching followed by suppression filters
  that eliminate false positives
- **Context-aware:** keyword proximity checks and structural detection (JSON field
  names, CSV headers) for ambiguous patterns like dates of birth
- **Country self-detection:** infers a document's countries from the entities
  that carry one, so an ambiguous value resolves without the caller naming a
  country — and `countries=` never gates what is looked for
- **Fast:** ~1.1 ms for a short record, ~9 ms for a 3,500-character document
  with `[fast]` installed — though cost tracks identifier density more than
  length (see [Performance](#performance))
- **Zero required dependencies** (`pip install euredact[fast]` adds optional acceleration)
- **Thread-safe,** immutable `Detection` objects (frozen dataclasses)

### Supported Countries

| Region | Countries |
|---|---|
| Western Europe | AT, BE, CH, DE, FR, LU, NL |
| Southern Europe | CY, EL, ES, IT, MT, PT |
| Northern Europe | DK, EE, FI, IS, LT, LV, NO, SE |
| Eastern Europe | BG, CZ, HR, HU, PL, RO, SI, SK |
| British Isles | IE, UK |

## API Reference

### Which option do I need?

| I want to… | Use |
|---|---|
| redact a document with no further setup | `redact(text)` — all 31 countries, `[ENTITY_TYPE]` output |
| restrict scope to the countries I operate in | `countries=["NL", "BE"]` |
| keep detection wide but resolve ambiguity | `country_hint=["DE"]` |
| send a prompt to an LLM and restore the reply | `tokenize=True`, then [`restore()`](#euredactrestore) |
| keep relationships visible across a whole session | `referential_integrity=True` |
| never redact my own company name or addresses | `allowlist=[...]`, or `EuRedact(allowlist=[...])` |
| never redact anything at my own domain | `allowlist_domains=["acme.be"]` |
| audit what the allowlist kept in the document | `result.exempted` |
| catch person names, employers, job titles, diagnoses | `mode="cloud"` — see [Cloud tier](#cloud-tier) |
| include dates of birth | `detect_dates=True` |
| redact a document too large for one call | `context=` + `chunk_offset=` — see [Chunked documents](#chunked-documents) |
| detect an identifier the engine does not know | [`add_custom_pattern()`](#euredactadd_custom_pattern) |
| isolate tenants from each other | one `EuRedact()` instance each — see [Instance Isolation](#instance-isolation) |
| free PII held in memory | `clear()` |


EuRedact provides both module-level functions (using a shared singleton) and an
instance-based `EuRedact` class. The module-level API is the easiest way to get
started; the class-based API gives you isolated instances with separate caches
and custom pattern registrations.

### Module-Level Functions

#### `euredact.redact()`

```python
euredact.redact(
    text: str,
    *,
    countries: list[str] | None = None,
    country_hint: list[str] | None = None,
    context: DocumentContext | None = None,
    chunk_offset: int = 0,
    mode: str = "rules",
    referential_integrity: bool = False,
    tokenize: bool = False,
    allowlist: list[str] | None = None,
    detect_dates: bool = False,
    coref: bool = False,
    coref_model: str = "default",
    cache: bool = True,
) -> RedactResult
```

Main entry point. Detects and redacts PII in the given text. Every option is
keyword-only.

**What is looked for**

| Parameter | Default | Description |
|---|---|---|
| `text` | — | Input text to scan. Longer than `max_input_length` raises `ValueError`. |
| `countries` | `None` | ISO 3166-1 alpha-2 codes that define **scope** (e.g. `["NL", "BE"]`). `None` loads all 31. This never gates what is looked for: a detection attributed elsewhere is flagged `out_of_scope`, never dropped. Passing a bare string raises `TypeError`. See [Country codes](#country-codes). |
| `country_hint` | `None` | A **prior only**. Helps resolve an ambiguous value without narrowing scope or flagging anything out of scope. See [Country Hints](#country-hints). |
| `detect_dates` | `False` | Include `DOB` and `DATE_OF_DEATH`. Off by default: a bare date without keyword or structural context is deferred to the cloud tier. When on, the engine applies keyword and JSON/CSV-header checks. |

**How the output looks**

The three output styles are mutually exclusive in practice — pick at most one.

| Parameter | Default | Description |
|---|---|---|
| *(none)* | — | Default: each span becomes `[ENTITY_TYPE]`. |
| `referential_integrity` | `False` | Replace each distinct value with a consistent label (`EMAIL_1`). Labels persist **on the instance** across calls, so two documents sharing a value get the same label. See [Referential Integrity](#referential-integrity). |
| `tokenize` | `False` | Replace each value with a reversible token (`EMAIL_K7Q2`) and return the token → value mapping in `result.tokens`; pass it to [`restore()`](#euredactrestore). Tokens are unique to the **call**, so nothing is retained. Cannot be combined with `referential_integrity`. See [Reversible tokenization](#reversible-tokenization). |
| `allowlist` | `None` | Values never to redact, matched whole and case-insensitively, merged with the instance's list. Structured identifiers (IBAN, phone, VAT, …) also match across spacing and hyphenation. Applies to cloud-tier types too. See [Allowlist](#allowlist). |
| `allowlist_domains` | `None` | Domains whose addresses are never redacted, e.g. `["acme.be"]`. Applies to `EMAIL` and `URL` only, and covers subdomains. See [Allowlist](#allowlist). |

**Long documents and tiers**

| Parameter | Default | Description |
|---|---|---|
| `mode` | `"rules"` | `"rules"` runs locally. `"cloud"` sends the document to the euRedact service, which adds the model-only types (person names, organisations, job titles, diagnoses). See [Cloud tier](#cloud-tier). |
| `context` | `None` | Share country evidence across the chunks of one document, so a chunk with no country signal of its own is still scored against the rest. Pass the same [`DocumentContext`](#documentcontext) to every chunk. Disables the cache. See [Chunked documents](#chunked-documents). |
| `chunk_offset` | `0` | Where this chunk starts in the whole document. Used only to rebase spans recorded in `context`; returned detections are always relative to `text`. |
| `cache` | `True` | Reuse the result for an identical input and configuration. Set `False` for one-off calls on sensitive text, or when timing the engine. |

**Not yet implemented**

| Parameter | Default | Description |
|---|---|---|
| `coref` | `False` | Reserved for pronoun/coreference resolution. Accepted and ignored in rules mode; raises in cloud mode. |
| `coref_model` | `"default"` | Reserved, with `coref`. |

#### `euredact.redact_batch()`

```python
euredact.redact_batch(
    texts: list[str],
    *,
    countries: list[str] | None = None,
    mode: str = "rules",
    referential_integrity: bool = False,
    tokenize: bool = False,
    allowlist: list[str] | None = None,
    detect_dates: bool = False,
    cache: bool = True,
) -> list[RedactResult]
```

Redact PII from multiple texts at once. More efficient than calling `redact()` in
a loop because country configs are loaded once. Returns results in the same order
as the input.

#### `euredact.aredact()`

```python
async euredact.aredact(
    text: str,
    **kwargs,
) -> RedactResult
```

Async version of `redact()`. Offloads CPU-bound work to a thread pool so it
doesn't block the event loop. Accepts the same keyword arguments.

#### `euredact.aredact_batch()`

```python
async euredact.aredact_batch(
    texts: list[str],
    *,
    max_concurrency: int = 4,
    **kwargs,
) -> list[RedactResult]
```

Async batch redaction with controlled concurrency. Processes texts concurrently in
a thread pool. `max_concurrency` limits parallel threads (default 4). Returns
results in input order.

#### `euredact.redact_iter()`

```python
euredact.redact_iter(
    texts: Iterator[str],
    **kwargs,
) -> Iterator[RedactResult]
```

Lazy iterator that yields results one at a time. Useful for processing large
datasets without loading all results into memory. Loads country configs once on
the first item.

#### `euredact.add_custom_pattern()`

```python
euredact.add_custom_pattern(name: str, pattern: str) -> None
```

Register a custom regex pattern. Matches are reported with `name` as the entity
type. See [Custom Patterns](#custom-patterns) below for details and examples.

#### `euredact.restore()`

```python
euredact.restore(text: str, tokens: Mapping[str, str]) -> str
```

Put the original values back into text that derives from a `tokenize=True`
result — typically an LLM's reply to the tokenized prompt. `tokens` is
`result.tokens`. See [Reversible tokenization](#reversible-tokenization).

#### `euredact.available_countries()`

```python
euredact.available_countries() -> list[str]
```

Returns a sorted list of supported ISO country codes (e.g. `["AT", "BE", "BG", ...]`).

### Instance-Based API (`EuRedact` Class)

For applications that need isolated instances (separate caches, separate custom
patterns), use the `EuRedact` class directly:

```python
from euredact import EuRedact

instance = EuRedact()

# Register custom patterns on this instance only
instance.add_custom_pattern("CASE_REF", r"CASE-\d{8}")

# Redact using this instance's configuration
result = instance.redact("See CASE-20260401 for details", countries=["NL", "BE"])
print(result.redacted_text)
# "See [CASE_REF] for details"
```

The `EuRedact` class exposes the same methods as the module-level API: `redact()`,
`redact_batch()`, `aredact()`, `aredact_batch()`, `redact_iter()`, `clear()` and
`add_custom_pattern()`.

```python
EuRedact(
    *,
    max_input_length: int = 10_485_760,
    allowlist: list[str] | None = None,
)
```

| Parameter | Default | Description |
|---|---|---|
| `max_input_length` | `10_485_760` (~10 MB) | Longest document `redact()` accepts, in characters. Above it, `ValueError` — split the input or raise the ceiling. |
| `allowlist` | `None` | Values never to redact, for every call on this instance: your own organisation's name, its own addresses. Merged with the per-call `allowlist`. See [Allowlist](#allowlist). |
| `allowlist_domains` | `None` | Domains never to redact, for every call on this instance. Merged with the per-call value. |

State held on the instance — the result cache, referential-integrity labels and
custom patterns — is per instance, which is what makes one instance per tenant
the right default. `clear()` releases the cache and the label mapping.

### Return Types

#### `DocumentContext`

```python
from euredact import DocumentContext

ctx = DocumentContext()
for offset, chunk in chunks:
    result = euredact.redact(chunk, context=ctx, chunk_offset=offset)
```

Shares country evidence across the chunks of one document, so a chunk carrying
no country signal of its own is still scored against what the rest of the
document showed. Pass the same object to every chunk, with `chunk_offset` set
to where the chunk starts. Passing a context disables the result cache, because
the result then depends on evidence from other chunks rather than on the text
alone.

| Method | Description |
|---|---|
| `add(evidence, chunk_offset=0)` | Record a chunk's evidence. `redact()` calls this for you. |
| `evidence()` | Every signal gathered so far, spans rebased onto the whole document. |

Not supported in cloud mode: the model has never seen a chunk boundary, so the
service rejects oversized input rather than splitting it.

#### `RedactResult`

Returned by `redact()` and all batch/async variants.

```python
@dataclass
class RedactResult:
    redacted_text: str          # The input text with PII replaced
    detections: list[Detection] # All PII spans found
    source: str = "rules"       # Detection backend ("rules")
    degraded: bool = False      # True if the engine fell back to a simpler mode

    # Country inference — see "Country-independent detection"
    inferred_countries: tuple[tuple[str, float], ...] = ()  # (country, confidence), strongest first
    evidence: tuple[CountryEvidence, ...] = ()              # every signal, with the span behind it
    detection_mode: str = "declared"                        # "declared" if countries= was passed,
                                                            # "inferred" otherwise
    tokens: dict[str, str] = {}                             # token -> original value; only with tokenize=True
    exempted: list[Exemption] = []                          # spans the allowlist kept, with the rule that matched
```

#### `Detection`

A single PII span. Frozen dataclass (immutable, hashable).

```python
@dataclass(frozen=True)
class Detection:
    entity_type: EntityType | str  # PII category (EntityType enum or custom name)
    start: int                     # Start offset in the original text
    end: int                       # End offset (exclusive) in the original text
    text: str                      # The matched substring
    source: DetectionSource        # "rules" or "cloud"
    country: str | None            # ISO code of the matched country, or None for shared/custom patterns
    confidence: str = "high"       # "high" | "medium" | "low" — see below
    country_confidence: float = 0.0  # How strongly the document supports `country`, in [0, 1].
                                     # 0.0 means the attribution rests on a checksum alone.
    out_of_scope: bool = False       # True when attributed outside the declared `countries`.
                                     # Flagged, never dropped.
```

**`confidence`** describes how the *type* was arrived at. It never says anything
about whether the span is masked — every detection is, at every level.

| value | meaning |
|---|---|
| `"high"` | a pattern matched and, where one exists, its checksum passed |
| `"medium"` | the type comes from a label touching the span, because no pattern of that type claimed it — `Αρ. Ταυτότητας: 00892341` is a `NATIONAL_ID` although nothing can checksum it |
| `"low"` | a pattern matched, its checksum *failed*, and the document labels the span as that very type — `Rijksregisternummer: 85.03.19-284.73` is a national number with a bad check digit |

Filter on it when you need only checksum-backed detections:

```python
strict = [d for d in result.detections if d.confidence == "high"]
```

A `"low"` detection is the honest description of a mistyped, OCR'd or invented
identifier: the shape and the label agree, the check digit does not. Earlier
versions dropped these, which meant a redaction library printed in full an
identifier it had recognised and rejected.

#### `EntityType`

String enum with all supported PII categories:

```
PERSON_NAME       ADDRESS           BANK_ACCOUNT      BIC
CREDIT_CARD       PHONE             EMAIL             DOB
DATE_OF_DEATH     NATIONAL_ID       SSN               TAX_ID
PASSPORT          DRIVERS_LICENSE   RESIDENCE_PERMIT  LICENSE_PLATE
VIN               VAT               POSTAL_CODE       IP_ADDRESS
IPV6_ADDRESS      MAC_ADDRESS       HEALTH_INSURANCE  HEALTHCARE_PROVIDER
CHAMBER_OF_COMMERCE  IMEI          GPS_COORDINATES   UUID
SOCIAL_HANDLE     SECRET            INTERNAL_ID       OTHER

Cloud tier only — the rule engine never emits these, because there is no shape
to match on. That is precisely why the model exists:

ORGANISATION_NAME JOB_TITLE         MEDICAL_CONDITION SENSITIVE_ATTRIBUTE
BIOMETRIC_REF     FINANCIAL_AMOUNT  QUASI_IDENTIFIER  CREDENTIAL
URL
```

`INTERNAL_ID` — an employee, badge or customer number tied to a person — is
emitted **only** when an explicit label names it (`medarbejdernummer:`,
`Personalnummer:`, `Employee No:`, `Betriebsstättennr.`, `Badge`). There is no
pattern for one, because there is no shape for one: without the label, a digit
run is not distinguishable from any other. The type exists so that a labelled
employee number is filed correctly instead of being claimed by the phone
pattern.

#### What a label in front of a value decides

A label touching the left edge of a value decides what that value is called.
The label may be the abbreviation or the word it abbreviates — `BSN:` and
`Burgerservicenummer:` both reach `NATIONAL_ID` — and it may be the official
name in the document's own language:

| label | type |
|---|---|
| `Companies House Registration:`, `Company Registration Number:` | `CHAMBER_OF_COMMERCE` |
| `Sozialversicherungsnummer:` | `SSN` |
| `Passport No.:`, `Paspoortnummer:` | `PASSPORT` |
| `AGB-code:`, `LANR`, `GMC Number:` | `HEALTHCARE_PROVIDER` |
| `Medical Card No.:` | `HEALTH_INSURANCE` |
| `sort code`, `account number` | `BANK_ACCOUNT` |
| `TAN-activatiecode`, `activation code` | `SECRET` |

A label can also *rule a type out*. A four-digit run is not a postal code when
a founding or payment participle introduces it (`Opgericht in 2016`,
`Fondée en 2017`) or when it sits in a telephone parenthetical (`(toest. 3841)`,
`(ext. 2219)`). Postal codes that merely look like years are unaffected —
Antwerp's `2018` in `rustige ligging in 2018` is still a `POSTAL_CODE`.

A label never moves a span; it only decides the label. Which characters are
masked is unchanged either way.

For custom patterns registered via `add_custom_pattern()`, `entity_type` is a
plain string (e.g. `"EMPLOYEE_ID"`) rather than an `EntityType` enum member.

#### `DetectionSource`

String enum: `"rules"` or `"cloud"`.

## Cloud tier

> **Status: private alpha.** The cloud tier is in closed testing — it is **not**
> in public beta and is not generally available. Keys are issued to alpha
> participants only, and the request/response surface may still change between
> releases. The rules engine below is unaffected and is the supported path:
> `mode="rules"` is the default and carries no alpha caveat.

The rule engine catches what has a shape: IBANs, national IDs, phone numbers,
anything with a checksum. It cannot catch what does not — a person's name, an
employer, a diagnosis, a job title. The cloud tier adds a fine-tuned model
asked only *what did the rules miss?*

```bash
pip install 'euredact[cloud]'
```

```python
import euredact

euredact.configure(api_key="erk_...")          # or set EUREDACT_API_KEY
result = euredact.redact(text, countries=["BE"], mode="cloud")

result.source                                   # "cloud"
[(d.entity_type, d.text) for d in result.detections]
# [(<EntityType.PERSON_NAME>, 'Bas Verhoeven'), (<EntityType.PHONE>, '+32 ...')]
```

`mode="cloud"` **raises** `NotConfiguredError` when the tier is not configured.
It never falls back to rules-only output: a caller who believes names and
diagnoses were checked, and ships a document that only had its phone numbers
masked, is the one failure this library must not have.

An async client is available for the same contract:

```python
from euredact.cloud import AsyncCloudClient

async with AsyncCloudClient() as client:
    result = await client.redact(text, country="BE")
```

Retries carry an `Idempotency-Key`, so a retry after a timeout cannot bill
twice. `Retry-After` is obeyed. A document that outlives the service's sync
window is polled transparently — callers never write that branch. Oversized
input raises `TooLargeError` (413): the service refuses it rather than
chunking, because the model has never seen a chunk boundary.

### `euredact.configure()`

```python
euredact.configure(
    api_key: str | None = None,
    *,
    base_url: str | None = None,
    timeout_s: float = 30.0,
    poll_timeout_s: float = 300.0,
    max_retries: int = 3,
    headers: dict[str, str] | None = None,
) -> CloudConfig
```

| Parameter | Default | Environment variable | Description |
|---|---|---|---|
| `api_key` | — | `EUREDACT_API_KEY` | Your alpha key. Required; `configure()` raises without one. |
| `base_url` | `https://api.euredact.dev` | `EUREDACT_BASE_URL` | Service endpoint. |
| `timeout_s` | `30.0` | — | Per-request timeout. |
| `poll_timeout_s` | `300.0` | — | Ceiling for polling a document that outlives the synchronous window. |
| `max_retries` | `3` | — | Retries for `429` and `5xx`. Each carries an `Idempotency-Key`, so a retry after a timeout cannot bill twice, and `Retry-After` is obeyed. |
| `headers` | `None` | — | Extra headers sent with every request. |

`configure()` reads the two environment variables itself, so it must still be
called — but with no arguments if the environment is set. Errors are
`NotConfiguredError`, `QuotaExceededError`, `TooLargeError` (413, the service
refuses oversized input rather than chunking it) and `CloudError`.

Options the service cannot honour raise rather than being ignored: multiple
`countries`, `country_hint`, `context`/`chunk_offset`, `referential_integrity`
and `coref`. `tokenize`, `allowlist` and `allowlist_domains` are honoured: the SDK applies them to
the spans the service returns and rebuilds the text from those.

## `NAME` is now `PERSON_NAME`

The canonical type name is `PERSON_NAME`; `NAME` is a legacy alias, exactly as
`IBAN` aliases `BANK_ACCOUNT`. The placeholder written into redacted text is
`[PERSON_NAME]`.

Nothing could have depended on the old value: the type is cloud-only and the
cloud tier was stubbed until this release, so it was never emitted. Code
matching the *string* `"NAME"` should be updated; `LEGACY_TYPE_ALIASES`
publishes the mapping, and `STREET_ADDRESS` → `ADDRESS` and
`NATIONALITY_ETHNICITY` → `SENSITIVE_ATTRIBUTE` are recognised the same way.

## Country codes

`countries=[...]` accepts **ISO 3166-1 alpha-2** codes. The two EU/VAT
spellings are accepted as equivalents:

| ISO 3166-1 | EU/VAT | |
|---|---|---|
| `GB` | `UK` | United Kingdom |
| `GR` | `EL` | Greece |

Codes are case-insensitive and whitespace-tolerant. An **unrecognised** code
does not raise — it emits an `UnknownCountryWarning` and detection continues
with the shared, country-independent patterns:

```python
euredact.redact(text, countries=["ZZ"])
# UnknownCountryWarning: Unknown country code: 'ZZ'. Continuing with shared
# country-independent patterns only (email, IBAN, international phone, ...).
```

This is deliberate. Raising on an unknown locale invites callers to wrap the
call in `try/except` and skip redaction entirely — failing open, with
unredacted PII in the output.

## Country-independent detection

**`countries=[...]` never gates detection.** Every pattern runs on every
document, whatever you pass. The country you declare decides *how a match is
labelled*, never *whether it is found*.

This is the engine's central invariant, enforced by
`tests/test_invariant_generation.py`: no value of `countries=` may change which
spans are detected. A wrong or missing country cannot cause a miss — silent
recall loss is invisible in testing and surfaces in a breach report, whereas a
false positive is recoverable.

It was not always so. `countries=["BE"]` used to make a valid Dutch BSN vanish
entirely, because the Dutch patterns were never run:

```python
euredact.redact("Werknemer met BSN 111222333", countries=["BE"])
# before: 'Werknemer met BSN 111222333'   <- leaked
# now:    'Werknemer met BSN [NATIONAL_ID]'
```

Entities found outside the countries you declared are **flagged, not dropped**:

```python
det = euredact.redact("BSN 111222333", countries=["BE"]).detections[0]
det.out_of_scope   # True — detected, masked, and marked as outside your scope
```

So a Belgian IBAN in a document processed with `countries=["AT"]` is still
detected — cross-border traffic (a foreign invoice in a local file, an employee
paid to a foreign account) does not leak:

```python
euredact.redact("Rekening: BE68 5390 0754 7034", countries=["AT"])
# -> 'Rekening: [BANK_ACCOUNT]'
```

### Which country a value belongs to

Because every pattern runs, the same digits often match several countries'
schemes. Of the national-ID values in our corpus that pass any country's checksum,
34.7% pass more than one country's (32,827 of 94,528), so the digits alone
cannot decide it — the *document* does.

The engine infers the document's countries from entities that carry their
country in the string, then uses that to resolve the ambiguity:

```python
r = euredact.redact("Bereikbaar op telefoon 0612345678, mail jan@test.nl")
r.inferred_countries          # (('NL', 0.98),)
r.detections[0].entity_type   # PHONE

r = euredact.redact("Kontakt: 0612345678, e-mail jens@test.dk")
r.inferred_countries          # (('DK', 0.98),)
r.detections[0].entity_type   # NATIONAL_ID
```

Identical digits, different answer — `0612345678` is both a valid Dutch mobile
number and a valid Danish CPR. Only the surrounding document distinguishes them.

Every inference is auditable: `result.evidence` lists each signal, its weight,
and the span that produced it.

| Signal | Weight (log-odds) | Measured reliability |
|---|---:|---|
| `e164_prefix` | 4.00 (capped) | 41,402 / 41,402 |
| `bic_country` | 4.00 (capped) | 2,588 / 2,588 |
| `email_tld` | 4.00 (capped) | 97,865 / 98,949 |
| `vat_prefix` | 2.84 | 19,022 / 20,136 |
| `iban_prefix` | 1.94 | 110,572 / 126,428 |

Weights are derived from the corpus, not chosen by hand. The IBAN prefix being
weakest is real and worth knowing: a Belgian IBAN in a Dutch invoice is
ordinary, so an account's country is only weak evidence about the document's.

Confidences are per-country and do **not** sum to 1 — document countries are
not mutually exclusive. A Belgian supplier invoicing a German customer is
genuinely both.

International phone numbers are matched by a generic E.164 pattern (`+`
followed by 8-15 digits, any grouping or separators, including `(0)` trunk
prefixes) that runs alongside the per-country patterns.

## BIC detection

BIC is the only bank identifier in the engine with **no check digit** — IBAN
has mod-97, VAT has country-specific checksums. ISO 9362 structure alone
cannot decide a match, because characters 5-6 of ordinary uppercase words are
frequently valid ISO 3166 country codes (`DRINGEND` → `GE`, `HOSPITAL` →
`IT`). Detection is therefore gated:

| Stage | Condition | Result |
|---|---|---|
| Gate 0 | the token also occurs as an ordinary lowercase word in the same document | never emitted |
| Tier 1 | registry hit on the BIC6 institution+country prefix | emitted |
| Gate 2 | heading / shouted-word shape | never emitted |
| Tier 2 | `BIC`/`SWIFT` keyword, an IBAN, or a bank block in the enclosing line, record or paragraph | emitted |
| — | none of the above | never emitted |

The context window is the enclosing **line, record or paragraph**, not a
character count — a banking cue often sits several fields away in the same
CSV row.

### Supplying your own BIC registry

The package bundles **no licensed BIC data**. The authoritative SWIFTRef BIC
Directory is a commercial product, and redistributing it inside a package
requires a specific redistribution licence. What ships is a small seed list of
BIC6 prefixes for major European banks, compiled from publicly published bank
data.

Deployments holding a licensed directory install it at startup:

```python
import euredact

# A path to a newline-delimited file of BICs...
euredact.set_bic_registry("/etc/euredact/swiftref-bics.txt")

# ...an iterable...
euredact.set_bic_registry({"ABNANL2A", "INGBNL2A", "BBRUBE"})

# ...or any membership callable.
euredact.set_bic_registry(lambda bic: bic in my_directory)

# Remove it again:
euredact.set_bic_registry(None)
```

Entries may be full BIC8/BIC11 codes or bare BIC6 prefixes; both are matched,
case-insensitively and ignoring spaces.

The registry is an **accept** signal, never a filter. A code missing from it
falls through to the context gate and is still detected when banking context
is present, so a stale list costs a little recall on bare, contextless BICs —
it never causes a leak. Annual review is sufficient.

## Custom Patterns

Register domain-specific PII patterns at runtime. Custom patterns are detected
alongside built-in patterns and participate in the same deduplication pipeline.

```python
import euredact

# Register patterns
euredact.add_custom_pattern("EMPLOYEE_ID", r"EMP-\d{6}")
euredact.add_custom_pattern("CASE_REF", r"CASE-\d{8}")

# They are detected alongside built-in PII
result = euredact.redact(
    "Employee EMP-123456, email jan@example.com, ref CASE-20260401"
)
print(result.redacted_text)
# "Employee [EMPLOYEE_ID], email [EMAIL], ref [CASE_REF]"

# Check detections
for d in result.detections:
    print(f"  {d.entity_type}: {d.text}")
# EMPLOYEE_ID: EMP-123456
# EMAIL: jan@example.com
# CASE_REF: CASE-20260401
```

### How Custom Patterns Work

- `name` becomes the entity type reported in detections and used in replacement
  tags (e.g. `[EMPLOYEE_ID]`)
- `pattern` is a Python regular expression (same syntax as `re` module)
- Custom patterns are always active regardless of the `countries` parameter
- Custom patterns have no validator (they are purely regex-based)
- In overlap resolution, custom patterns have higher priority than built-in
  regex-only patterns but lower priority than built-in patterns with a passing
  checksum validator

### Instance Isolation

Custom patterns registered on the module-level function apply to the shared
singleton. For isolated pattern registrations, use separate `EuRedact` instances:

```python
from euredact import EuRedact

# Instance A detects employee IDs
a = EuRedact()
a.add_custom_pattern("EMPLOYEE_ID", r"EMP-\d{6}")

# Instance B detects case references
b = EuRedact()
b.add_custom_pattern("CASE_REF", r"CASE-\d{8}")

# Each instance only detects its own custom patterns
result_a = a.redact("EMP-123456 CASE-20260401")
result_b = b.redact("EMP-123456 CASE-20260401")
```

## Secret and API Key Detection

EuRedact includes built-in detection for API keys, tokens, and passwords. This is
always active -- no configuration required.

### Known-Prefix Patterns

The following token formats are detected with high confidence based on their
distinctive prefixes:

| Pattern | Description |
|---|---|
| `AKIA...` | AWS Access Key ID |
| `ghp_`, `gho_`, `ghs_`, `github_pat_` | GitHub tokens (PAT, OAuth, app, server) |
| `sk_live_`, `pk_live_`, `sk_test_`, `pk_test_` | Stripe secret and publishable keys |
| `sk-`, `sk-ant-` | OpenAI and Anthropic API keys |
| `xoxb-`, `xoxp-`, `xoxa-`, `xoxs-` | Slack tokens |
| `eyJ...` (3-part base64url) | JWT tokens |
| `SG.` | SendGrid API keys |

```python
result = euredact.redact("My API key is sk-proj-abc123def456ghi789jkl0")
print(result.redacted_text)
# "My API key is [SECRET]"
```

### Entropy-Based Detection

For secrets that don't have a recognizable prefix, EuRedact uses Shannon entropy
analysis. A high-entropy string (32+ characters of alphanumeric/base64 content) is
flagged as `SECRET` when it appears near context keywords like `key`, `token`,
`secret`, `password`, `credential`, `auth`, or `bearer` (including translations in
12 European languages).

```python
result = euredact.redact("The api_key is xK9mPqR7vLnW2bFjY8cGhT4sDfAeU6iO")
print(result.redacted_text)
# "The api_key is [SECRET]"

# Without a context keyword, the same string is not flagged:
result = euredact.redact("identifier: xK9mPqR7vLnW2bFjY8cGhT4sDfAeU6iO")
print(result.redacted_text)
# "identifier: xK9mPqR7vLnW2bFjY8cGhT4sDfAeU6iO"  (unchanged)

# Low-entropy strings are also not flagged, even with context:
result = euredact.redact("The password is aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa")
print(result.redacted_text)
# "The password is aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"  (unchanged)
```

## Country Hints

Two arguments tell the engine about country, and neither restricts what is
looked for:

| Argument | Meaning |
|---|---|
| `countries=[...]` | **Scope.** Resolves ambiguity, and flags anything attributed elsewhere as `out_of_scope`. |
| `country_hint=[...]` | **Prior only.** Resolves ambiguity without narrowing scope or flagging anything. |

```python
# You know this batch is Swedish, but do not want foreign PII marked out of scope:
euredact.redact(text, country_hint=["SE"])

# You want anything non-Swedish flagged for review:
euredact.redact(text, countries=["SE"])
```

Declaring a country helps where a value is genuinely ambiguous and the document
carries no other signal:

```python
euredact.redact("Telefon: 0708787668", country_hint=["SE"]).detections[0]
# PHONE / SE  — without the hint this is a valid Danish CPR and nothing says otherwise
```

`result.detection_mode` reports which happened: `"declared"` if you passed
`countries=`, `"inferred"` otherwise.

Custom patterns are always active regardless of either parameter.

Passing neither is safe, and is the right default for mixed-origin data: the
engine infers what it can and reports it in `result.inferred_countries`. On our
152,300-document corpus, blind detection scores 99.6% precision against 99.8%
with country hints — a 0.2-point gap, down from 3.4 points before inference.

## Chunked documents

A long document is usually redacted in pieces. Each piece is scanned
independently, so each infers its own country — and a chunk that happens to
contain no IBAN, no `+CC` number and no ccTLD infers nothing at all, even when
page 1 identified the document beyond doubt:

```python
euredact.redact("Telefoon 0612345678")
# -> NATIONAL_ID (DK)   <- a valid Danish CPR, and nothing here says otherwise
```

Pass a `DocumentContext` to carry evidence forward across chunks:

```python
from euredact import DocumentContext

ctx = DocumentContext()
offset = 0
for page in pages:
    result = euredact.redact(page, context=ctx, chunk_offset=offset)
    offset += len(page)

# page 1: "Factuur — IBAN NL91ABNA0417164300, info@example.nl"
# page 7: "Telefoon 0612345678"  -> PHONE (NL)
```

`chunk_offset` rebases spans recorded in the context so they point into the
whole document. Returned detections stay relative to the chunk you passed in.

A context is thread-safe — `aredact_batch` fans chunks across a thread pool.
Caching is disabled automatically while one is in use, because the result then
depends on evidence the text alone does not determine.

Reuse a context only for chunks of the **same** document. Sharing one across
unrelated documents mixes their countries together. That cannot hide anything
— the invariant still holds — but it will attribute values to the wrong
national scheme.

## Referential Integrity

When `referential_integrity=True`, each unique PII value is mapped to a consistent
label within the session. The same input always produces the same label:

```python
import euredact

text = "BSN 111222333 en later weer 111222333, IBAN NL91ABNA0417164300."
result = euredact.redact(text, referential_integrity=True)
print(result.redacted_text)
# "BSN NATIONAL_ID_1 en later weer NATIONAL_ID_1, IBAN BANK_ACCOUNT_1."
```

The mapping is scoped to the `EuRedact` instance. The module-level `redact()`
function uses a shared singleton, so labels are consistent across calls within
the same process.

## Reversible tokenization

`tokenize=True` is for text that has to come back. Each value is replaced by a
token that names its type and nothing else, and the result carries the mapping
that turns tokens back into values:

```python
import euredact

prompt = "Write an email to Joren at joren.janssens@euredact.be or call +32 475 12 34 56 about the invoice."
result = euredact.redact(prompt, countries=["BE"], tokenize=True)
print(result.redacted_text)
# "Write an email to Joren at EMAIL_K7Q2 or call PHONE_P4RT about the invoice."
print(result.tokens)
# {'EMAIL_K7Q2': 'joren.janssens@euredact.be', 'PHONE_P4RT': '+32 475 12 34 56'}

reply = call_your_llm(result.redacted_text)   # sees tokens, never the values
print(euredact.restore(reply, result.tokens))
# the reply, with the real address and number back in it
```

The suffixes are random; yours will differ. With the [cloud tier](#cloud-tier)
the name is tokenized too (`PERSON_NAME_W3NB`), since person names have no
shape for the rules tier to match on.

Within one call the same value gets the same token, so a prompt that names
someone twice still reads as one person. Across calls it gets a different
token: nothing is retained on the instance, `result.tokens` is the only copy,
and two tokenized documents never reveal that they share a value. That is the
opposite retention model from `referential_integrity`, which is why the two
cannot be combined. Batch and iterator variants tokenize each text on its own.

A token is `TYPE_` plus four characters from `ABCDEFGHJKLMNPQRSTUVWXYZ23456789`
(no vowels, no `0`/`1`/`I`/`O`). Tokens are kept clear of any token-shaped
string already in the document, so an LLM's reply to a tokenized prompt can
itself be redacted without `restore()` putting the wrong value back.
`restore()` replaces every occurrence, including a token an LLM glued to other
characters (`EMAIL_P4RTs`) — leaving a token behind is the worse failure.

Works in cloud mode: the SDK rebuilds the text from the spans the service
returns, which is exactly what the service built its own output from.

## Allowlist

Values a caller declares are not PII to them — their own organisation's name,
their own addresses. Set it per call, on the instance, or both; the two merge:

```python
import euredact

sdk = euredact.EuRedact(allowlist=["ACME NV", "info@acme.be"])

text = "ACME NV: mail info@acme.be or jan@acme.be about IBAN NL91 ABNA 0417 1643 00."
print(sdk.redact(text, countries=["NL"]).redacted_text)
# "ACME NV: mail info@acme.be or [EMAIL] about IBAN [BANK_ACCOUNT]."

print(sdk.redact(text, countries=["NL"], allowlist=["jan@acme.be"]).redacted_text)
# "ACME NV: mail info@acme.be or jan@acme.be about IBAN [BANK_ACCOUNT]."
```

Matching is whole-span and case-insensitive. A bare string
(`allowlist="ACME NV"`) raises `TypeError` rather than being iterated into
single letters that exempt nothing. Works in cloud mode — including on types
only the model finds, such as `ORGANISATION_NAME` — and together with
`tokenize`.

### Spacing and punctuation

For **structured identifiers** separators are presentational, so an allowlisted
value matches however the document writes it:

```python
sdk = euredact.EuRedact(allowlist=["NL91ABNA0417164300"])
sdk.redact("Pay to NL91 ABNA 0417 1643 00.", countries=["NL"]).redacted_text
# 'Pay to NL91 ABNA 0417 1643 00.'   — exempt, despite the spacing
```

Applies to `BANK_ACCOUNT`, `BIC`, `CREDIT_CARD`, `PHONE`, `VAT`, `NATIONAL_ID`,
`SSN`, `TAX_ID`, `PASSPORT`, `DRIVERS_LICENSE`, `RESIDENCE_PERMIT`,
`HEALTH_INSURANCE`, `CHAMBER_OF_COMMERCE`, `IMEI` and `VIN`. It widens the
*spelling*, never the *scope*: a different account is still redacted.

Free-text types (`EMAIL`, `PERSON_NAME`, `ORGANISATION_NAME`, …) stay literal,
because folding would exempt values you never listed —
`jan.devries@acme.be` and `jandevries@acme.be` are different mailboxes at most
providers, and `ACME NV` folded would also match `ACMENV`. Case still does not
matter, so list a company's legal spellings if they vary:
`["ACME NV", "ACME N.V."]`.

### Exempting a whole domain

Enumerating every mailbox does not scale — a new hire's address is redacted in
your own documents until someone updates the list. `allowlist_domains` takes
the domain instead:

```python
sdk = euredact.EuRedact(allowlist_domains=["acme.be"])
sdk.redact("Mail jan@acme.be or piet@acme.be", countries=["NL"]).redacted_text
# 'Mail jan@acme.be or piet@acme.be'
```

It applies to `EMAIL` and `URL` only and covers subdomains (`mail.acme.be`).
The match is on a label boundary, so `acme.be` does **not** exempt
`evilacme.be`. Entries may be written `acme.be`, `@acme.be` or `.acme.be`.

There are deliberately **no wildcards**. The allowlist is the only option that
turns redaction *off*, so an over-broad entry fails toward under-redaction and
does so silently: `*@*` would disable email redaction entirely and `* Maes` a
family's names. A domain rule cannot be widened that way — it structurally
cannot match a person's name or an IBAN.

### What was exempted

Every exemption is reported, so a document that keeps a direct identifier can be
audited rather than taken on trust:

```python
r = euredact.redact("Mail jan@acme.be", countries=["NL"], allowlist_domains=["acme.be"])
for e in r.exempted:
    print(e.entity_type, e.text, e.rule, e.rule_kind)
# EntityType.EMAIL jan@acme.be acme.be domain
```

`Exemption` carries `entity_type`, `start`, `end`, `text`, the `rule` that
matched as you wrote it, and `rule_kind` (`"value"` or `"domain"`). An exempted
span is absent from `detections`, so `exempted` is the only record that it was
found at all.

## Architecture

```
Input text
    |
    v
[Normalizer] -- Unicode normalization, whitespace cleanup
    |
    v
[Pass 1: Pattern Matching] -- All country + shared + custom regexes
    |                          via MultiPatternMatcher (Aho-Corasick optional)
    v
[Pass 2a: Validation] -- Checksum validators (mod-97, Luhn, entropy, ...)
    |                     Failed spans are recorded, per entity type
    v
[Evidence]     -- Which countries does this document belong to? From IBAN
    |              prefixes, +CC codes, VAT prefixes, BIC, email ccTLDs
    v
[Pass 2b: Suppression] -- Remove false positives (currency amounts, units,
    |                      references). Failed checksums demote same-type
    |                      matches rather than deleting them
    v
[Deduplication] -- Priority-aware, country-evidence-weighted
    |               Longer span outranks declared country
    v
[Replacement] -- Right-to-left substitution with [ENTITY_TYPE] labels
    |               or labels
    v
RedactResult
```

The engine is **thread-safe**: a `threading.Lock` guards country loading and
custom-pattern registration, and all detection state is local to each `detect()`
call. `Detection` objects are frozen dataclasses and can be safely shared across
threads.

Registering a pattern while other threads are detecting is safe too:
`add_custom_pattern` builds a whole new scan plan and publishes it in one
assignment, so a scan in flight keeps using the plan it started with and sees
the pattern set as of either before or after the registration — never a mixture.
It will not observe the new pattern until it next calls `detect()`.

Two caveats that are about *sharing*, not locking:

- **Referential integrity is per instance.** Labels (`EMAIL_1`, …) come from a
  mapper owned by the `EuRedact` instance, so every caller of the module-level
  `euredact.redact()` shares one namespace. A label appearing in two documents
  tells you they contain the same underlying value. Give each tenant its own
  `EuRedact()` instance.
- **The mapper is never evicted**, because evicting would give a previously seen
  value a second label. It grows until you call `clear()`, which is the right
  thing to do between workloads in a long-running process.

### Suppression Zones

When a regex matches a pattern that has a checksum validator but the checksum
fails, the matched span becomes a "suppression zone." Any purely regex-based
detection fully contained within that zone is suppressed as a false positive.

For example, the text `BE71 0012 3456 7890` matches the Belgian IBAN regex but
fails mod-97 validation. Without suppression zones, sub-parts of this span might
be incorrectly detected as a license plate (`BE71`) or a phone number
(`0012 3456`). The suppression zone prevents these false positives while
correctly not reporting an invalid IBAN.

### Deduplication Priority

When multiple patterns match overlapping spans, the engine resolves conflicts
using a priority system:

| Tier | What |
|---:|---|
| 3 | **Validated** — a checksum validator passes *and* the document corroborates its country |
| 2 | **Custom patterns** registered via `add_custom_pattern()` |
| 1 | **Regex-only**, and validated patterns whose country the document does not corroborate |
| 0 | **Postal codes** — a bare digit run, the weakest evidence in the engine |
| -1 | **Demoted** — a validator-less match inside a failed checksum of *its own type* |

Within a tier, ranking is: longer span, then stronger country evidence, then
whether the country was declared. **Span length outranks country** deliberately:
preferring the declared country over the longest match truncates entities — with
`countries=["BE"]` the Belgian phone pattern claimed 11 of the 14 characters of
`06 12 34 56 78` and left three digits exposed. Country can change *which*
country is attributed, never *what* is masked.

Two tiers are less obvious than they look:

- **A passing checksum does not automatically win.** A weak checksum fits by
  luck — a mod-11 scheme accepts a random number about one time in eleven — so a
  validated candidate from a country the document shows no trace of drops to
  tier 1. Entities that carry their own country vouch for themselves (an IBAN
  emits evidence for its own country), so a foreign IBAN in a domestic invoice
  keeps tier 3.
- **Nothing is deleted for failing a checksum.** A failed checksum demotes
  rather than removes, and only candidates of *the same entity type*: it is
  evidence against that type, not against the span. Deleting instead removed 454
  detections across the corpus, of which 454 overlapped real labelled PII.

## Adding a New Country

Each country is a single Python file in `src/euredact/rules/countries/`. The
registry discovers new countries automatically -- no manual registration required.

1. Create a file, e.g. `src/euredact/rules/countries/gr.py`.
2. Define a `CountryConfig` subclass with patterns:

```python
"""Greece (GR) PII patterns."""

from euredact.rules.countries._base import CountryConfig, PatternDef
from euredact.types import EntityType


class GRConfig(CountryConfig):
    def __post_init__(self) -> None:
        self.code = "GR"
        self.name = "Greece"
        self.patterns = [
            PatternDef(
                entity_type=EntityType.NATIONAL_ID,
                pattern=r"\b[A-Z]{2}[0-9]{6}\b",
                validator=None,
                description="Greek national ID (example)",
            ),
        ]
```

That is all. The `CountryRegistry` scans the `countries/` package at startup and
picks up any module that defines a `CountryConfig` subclass with a non-empty
`code`. Files prefixed with `_` (like `_base.py` and `_shared.py`) receive
special treatment and are not treated as standalone countries.

Each `PatternDef` can specify:
- `entity_type` -- which `EntityType` this pattern detects
- `pattern` -- a regular expression
- `validator` -- an optional named validator (e.g. `"bsn"`, `"luhn"`, `"iban"`)
- `context_keywords` -- proximity keywords that boost confidence
- `requires_context` -- if `True`, the match is discarded without a nearby keyword

## Performance

Measured on one core (Apple Silicon M3 Pro, CPython 3.12.13), all 31 countries
loaded, `detect_dates=True`, cache off, over two cohorts sampled evenly from the
corpus: 3,000 distinct short records (~190 chars) and 611 distinct real
documents (~3,450 chars).

**Median** per document, 10th–90th percentile in brackets:

| Input | Pure Python | Aho-Corasick | With `[fast]` |
|---|---:|---:|---:|
| Short record (~190 chars) | 1,612 µs <br><sub>1,072 – 2,417</sub> | 1,507 µs <br><sub>956 – 2,293</sub> | **1,141 µs — 876/s** <br><sub>599 – 1,958</sub> |
| Real document (~3,450 chars) | 18.88 ms <br><sub>15.28 – 25.54</sub> | 15.95 ms <br><sub>12.51 – 22.70</sub> | **9.36 ms — 107/s** <br><sub>6.22 – 15.83</sub> |
| Memory per country | ~50 KB | ~50 KB | ~50 KB |

The spread matters more than the median: **cost tracks identifier density, not
length.** At an identical 3,424 characters, the cheapest document in the corpus
takes 4.8 ms on `[fast]` and the dearest 9.5 ms; across the whole long cohort
the range is 6.2–15.8 ms. A profile shows why — the dominant cost is not the
pattern scan but the cue and suppressor checks that run per candidate match, so
a document dense in identifiers pays for every one of them. Quote a figure for
*your* documents, not this table.

Making `\b` catch identifiers glued to a non-ASCII letter costs something, but
only where it must: next to a digit the ASCII reading alone is exactly
equivalent, so one lookaround replaces a three-way alternation, and plain `\b`
stays everywhere else. 0.3.3 applied the union to all 303 patterns and was
1.8×/2.8× slower for it.

```bash
pip install euredact[fast]
```

The `fast` extra installs two optional accelerators. Neither changes what is
detected — both are covered by `tests/test_scan_path_parity.py`, which runs
every available scan path against the plain-Python one and requires them to
agree:

- **`google-re2`** builds a prefilter over every pattern it can express (334 of
  345). One DFA pass per 1 KB window reports which patterns match anywhere in
  it — typically 42 of 314 for a real document — and only those are then run.
  Patterns RE2 cannot express, such as the lookbehind-based `SECRET` rules,
  always run.

  Asking the question per window matters: over a long document nearly every
  pattern matches *somewhere*, so a whole-document prefilter stops filtering
  (2.48× on a short record, decaying to 1.11× at 12 KB).

- **`pyahocorasick`** indexes patterns that begin with a literal and runs them
  only near a prefix hit. Used when `google-re2` is unavailable.

Detection cost is dominated by how much prose surrounds the PII, not by
document size alone: the same engine runs 1.4× faster on dense records and
~3× faster on prose-heavy documents where whole windows can be skipped.

The TypeScript SDK needs no such extra — V8's regex engine has literal
prefilters CPython lacks, and measured on the same documents it runs roughly 3×
faster than the accelerated Python path.

## License

Apache 2.0
