Metadata-Version: 2.4
Name: wardcat
Version: 1.0.1
Summary: PII detection and anonymization for LLM inputs — regex, NER and on-prem LLM hybrid engine
Project-URL: Homepage, https://github.com/oguzhantopcu0/wardcat
Project-URL: Bug Tracker, https://github.com/oguzhantopcu0/wardcat/issues
Project-URL: Changelog, https://github.com/oguzhantopcu0/wardcat/blob/main/CHANGELOG.md
Author-email: Oğuzhan Topçu <oguzhantopcu000@gmail.com>
License: MIT License
        
        Copyright (c) 2024-2026 wardcat contributors
        
        Permission is hereby granted, free of charge, to any person obtaining a copy
        of this software and associated documentation files (the "Software"), to deal
        in the Software without restriction, including without limitation the rights
        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
        copies of the Software, and to permit persons to whom the Software is
        furnished to do so, subject to the following conditions:
        
        The above copyright notice and this permission notice shall be included in all
        copies or substantial portions of the Software.
        
        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
        SOFTWARE.
License-File: LICENSE
Keywords: anonymization,data-protection,gdpr,llm,ner,nlp,ollama,pii,privacy,regex
Classifier: Development Status :: 5 - Production/Stable
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Security
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Topic :: Text Processing :: Linguistic
Classifier: Typing :: Typed
Requires-Python: >=3.11
Requires-Dist: httpx<1,>=0.28.1
Requires-Dist: pyyaml<7,>=6.0
Provides-Extra: all
Requires-Dist: accelerate>=1.14.0; extra == 'all'
Requires-Dist: bitsandbytes>=0.43; extra == 'all'
Requires-Dist: protobuf>=7.35.1; extra == 'all'
Requires-Dist: sentencepiece>=0.1.99; extra == 'all'
Requires-Dist: spacy<4,>=3.8.14; extra == 'all'
Requires-Dist: torch>=2.12.1; extra == 'all'
Requires-Dist: transformers<6,>=5.13.0; extra == 'all'
Provides-Extra: ner
Requires-Dist: spacy<4,>=3.8.14; extra == 'ner'
Provides-Extra: transformers
Requires-Dist: accelerate>=1.14.0; extra == 'transformers'
Requires-Dist: bitsandbytes>=0.43; extra == 'transformers'
Requires-Dist: protobuf>=7.35.1; extra == 'transformers'
Requires-Dist: sentencepiece>=0.1.99; extra == 'transformers'
Requires-Dist: torch>=2.12.1; extra == 'transformers'
Requires-Dist: transformers<6,>=5.13.0; extra == 'transformers'
Description-Content-Type: text/markdown

<p align="center">
  <img src="https://raw.githubusercontent.com/oguzhantopcu0/wardcat/main/docs/assets/logo.png" alt="wardcat" width="240">
</p>

<div align="center">
<pre>
██╗    ██╗ █████╗ ██████╗ ██████╗  ██████╗ █████╗ ████████╗
██║    ██║██╔══██╗██╔══██╗██╔══██╗██╔════╝██╔══██╗╚══██╔══╝
██║ █╗ ██║███████║██████╔╝██║  ██║██║     ███████║   ██║   
██║███╗██║██╔══██║██╔══██╗██║  ██║██║     ██╔══██║   ██║   
╚███╔███╔╝██║  ██║██║  ██║██████╔╝╚██████╗██║  ██║   ██║   
 ╚══╝╚══╝ ╚═╝  ╚═╝╚═╝  ╚═╝╚═════╝  ╚═════╝╚═╝  ╚═╝   ╚═╝   
</pre>
</div>

[![CI](https://github.com/oguzhantopcu0/wardcat/actions/workflows/ci.yml/badge.svg)](https://github.com/oguzhantopcu0/wardcat/actions/workflows/ci.yml)
[![Release](https://img.shields.io/github/v/release/oguzhantopcu0/wardcat?label=release)](https://github.com/oguzhantopcu0/wardcat/releases)
![Python](https://img.shields.io/badge/python-3.11%20%7C%203.12%20%7C%203.13-blue)
[![License: MIT](https://img.shields.io/badge/license-MIT-green.svg)](LICENSE)
[![Ruff](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/ruff/main/assets/badge/v2.json)](https://github.com/astral-sh/ruff)

**PII detection and anonymization for LLM inputs** — hybrid regex + NER + on-prem LLM engine.

`wardcat` scans text for personally identifiable information (PII) before it reaches an LLM, and either warns about or replaces the sensitive data with salted SHA-256 hashes. It supports Turkish, English, German, and French out of the box.

> **📖 Documentation** lives in [`docs/`](docs/) (MkDocs + Material, with an auto-generated API reference). Preview it locally with `uv run --group docs mkdocs serve`.

**Detection is opt-in:** a bare `Wardcat()` detects nothing — you configure the
layers and entities you want. Here is the full picture — all three layers, a
per-entity action policy, the semantic guardrail, and anonymization:

```python
import os
from wardcat import Wardcat, Backend, Entity, Action

# One guard, all three detection layers — regex + SpaCy NER + on-prem LLM.
guard = (
    Wardcat(salt=os.environ["WARDCAT_SALT"])               # your app supplies the salt
    .with_ner(language="tr")                               # names / orgs  (needs wardcat[ner])
    .with_llm(backend=Backend.OLLAMA, model="gemma3:12b")  # contextual + semantic (needs Ollama)
    # An action per entity: hash IDs, mask contact details, redact names, flag IPs.
    .add_entities([Entity.CREDIT_CARD, Entity.IBAN, Entity.TC_ID], action=Action.HASH)
    .add_entities([Entity.EMAIL, Entity.PHONE], action=Action.MASK)
    .add_entity(Entity.PERSON, Action.REDACT)
    .add_entity(Entity.IP_ADDRESS, Action.WARN)
)

text = "Ben Ahmet Yılmaz, kartım 4111 1111 1111 1111, e-posta ahmet@example.com."

# 1) Semantic guardrail: does the text contain sensitive info at all? (holistic LLM yes/no)
if guard.is_sensitive(text):

    # 2) Anonymize the PII before the text is stored, logged, or forwarded.
    result = guard.scan(text)
    print(result.sanitized_text)
    # Ben [PERSON], kartım [CREDIT_CARD:b22b36262d8d2769], e-posta a****@example.com.

    print(result.redacted())   # PII-free dict, safe for logs / APIs
```

> The NER and LLM layers are optional. **Regex-only detection needs no models** —
> just `Wardcat(salt=...).add_entity(Entity.CREDIT_CARD, Action.HASH)` and
> `.scan(...)`. Add `with_ner(...)` / `with_llm(...)` when you want names or
> contextual/semantic detection. See [Installation](#installation).

---

## Features

- **Hybrid detection** — Regex + SpaCy NER + on-prem LLM (Ollama, vLLM, OpenAI-compatible, HuggingFace Transformers)
- **Semantic sensitivity gate** — `is_sensitive(text) → bool`: a holistic yes/no on whether text is safe to send onward (LLM-only), catching confidential content the typed detectors miss (unreleased financials, deal terms, a confidential project); optional per-language prompt
- **Ensemble adjudication** (optional) — the LLM verifies/relabels/drops regex & NER candidates and adds what they missed, in one call; deterministic regex results are always protected
- **Four actions** — `warn` (keep text, report only), `hash` (`[TYPE:16hex]` via SHA-256 + salt; the default when `action` is omitted), `redact` (`[TYPE]` label, no hash), `mask` (entity-aware partial masking)
- **Checksum validation** — TC_ID (Nüfus İdaresi algorithm), IBAN (mod-97), and CREDIT_CARD (Luhn mod-10) validated before flagging — eliminates false positives
- **Rainbow table protection** — user-defined salt for all hashes
- **Two APIs** — method chaining (programmatic) and YAML (declarative)
- **Async & batch** — `scan_async` / `scan_batch` / `is_sensitive_async`; concurrent requests overlap (native async LLM I/O), one shared guard is safe to reuse across scans
- **Multilingual support** — Turkish, English, German, and French for names, addresses, birth dates, and phone numbers; plus Spanish, Italian, Dutch address patterns; TC_ID, IBAN, SSN, NIN, DNI/NIE, UK postcodes, US ZIP+4, EU VAT numbers and more
- **Secret detection** — API keys and tokens (OpenAI, Anthropic, Stripe, AWS, Google, GitHub, GitLab, Slack, Twilio, SendGrid, npm) and PEM private keys
- **Passport detection** — contextual passport number detection (regex keyword-based + LLM) for any country
- **Evasion resistance** — confusable/homoglyph folding catches lookalike-character tricks (Cyrillic/fullwidth/Arabic-Indic) that fool ASCII-oriented regex
- **Fault tolerance** — a layer that can't run (e.g. LLM backend down) degrades gracefully and is surfaced on `ScanResult.warnings` instead of crashing the scan
- **DoS protection** — inputs exceeding 500 KB are rejected
- **Safe logging API** — `result.redacted()` returns a PII-free dict for logs and APIs

> **Disclaimer.** wardcat is a **best-effort** PII detector — it does not catch everything (false negatives and positives are expected) and is **not legal advice or a substitute for compliance review** (e.g. GDPR/KVKK); using it does not by itself make a system compliant. Validate it against your own data and requirements. Provided "as is" (MIT — see [LICENSE](LICENSE)).

---

## Contents

[Installation](#installation) · [Quick Start](#quick-start) · [Supported Entity Types](#supported-entity-types) · [Output Structure](#output-structure) · [Security](#security) · [Configuration](#configuration) · [Project Structure](#project-structure) · [Testing](#testing) · [Known Limitations](#known-limitations) · [Development](#development)

---

## Installation

```bash
# Base install — regex detection + Ollama/OpenAI-compatible LLM backend
pip install wardcat

# + SpaCy NER (PERSON, ORG, ADDRESS detection)
pip install "wardcat[ner]"

# Everything at once (SpaCy + Transformers)
pip install "wardcat[all]"
```

Or, for development, install from source with [uv](https://github.com/astral-sh/uv):

```bash
git clone https://github.com/oguzhantopcu0/wardcat.git
cd wardcat
uv sync                 # base: regex + Ollama/OpenAI-compatible LLM backend
uv sync --extra ner     # + SpaCy NER (PERSON, ORG, ADDRESS)
uv sync --extra all     # + HuggingFace Transformers backend
```

To use SpaCy NER (PERSON, ORG, ADDRESS detection) you need a language model.
The simplest path is to let wardcat resolve and download it for you via the
`language=` builder:

```python
from wardcat import Wardcat, Language

# Turns NER on, picks the right model from the catalog, and downloads it if missing
guard = Wardcat().with_ner(language=Language.EN)                   # → en_core_web_sm
guard = Wardcat().with_ner(language=Language.TR, spacy_size="md")  # → tr_core_news_md
```

Or download a model yourself with SpaCy's own CLI:

```bash
uv run python -m spacy download en_core_web_sm     # English (recommended)
uv run python -m spacy download tr_core_news_md    # Turkish (recommended)
```

> If a requested SpaCy model is not installed, wardcat automatically falls back to any installed model of the same language and logs a warning. SpaCy is not required if you only need regex-based detection.

---

## Quick Start

> **Upgrading from a pre-1.0 version?** Each breaking change is listed with
> migration steps in the [CHANGELOG](CHANGELOG.md) — e.g. NER is now configured
> only via `with_ner(...)` (0.7.0) and LLM backends are no longer user-extensible
> (0.9.0). As of 1.0, the public API is stable (semantic versioning).

### Programmatic API

```python
from wardcat import Wardcat, Entity, Action

guard = (
    Wardcat(salt="my-secret-salt")
    .add_entity(Entity.CREDIT_CARD, Action.HASH)
    .add_entity(Entity.EMAIL,       Action.WARN)
    .add_entity(Entity.TC_ID,       Action.HASH)
)

result = guard.scan("""
  Customer: Ali Veli, TC: 12345678950
  Card: 4111 1111 1111 1111
  Email: ali.veli@example.com
""")

print(result.sanitized_text)
# Customer: Ali Veli, TC: [TC_ID:86349f34a1bc2d5e]
# Card: [CREDIT_CARD:b22b36262d8d2769]
# Email: ali.veli@example.com   ← warn: text is kept

for v in result.violations:
    print(f"[{v.action}] {v.entity_type}: {v.original!r}")
# [hash] TC_ID: '12345678950'
# [hash] CREDIT_CARD: '4111 1111 1111 1111'
# [warn] EMAIL: 'ali.veli@example.com'
```

#### Typo-proof config with `Entity` and `Action` constants

Prefer constants over bare strings — your IDE autocompletes them and a typo is
caught at edit time instead of becoming a silent runtime warning. They are fully
interchangeable with the string forms (`Entity.EMAIL == "EMAIL"`):

```python
from wardcat import Wardcat, Entity, Action

guard = (
    Wardcat(salt="my-secret-salt")
    .add_entity(Entity.CREDIT_CARD, action=Action.HASH)
    .add_entity(Entity.EMAIL,       action=Action.REDACT)
    .add_entity(Entity.PHONE,       action=Action.MASK)
)

# Batch form — also accepts Entity keys and Action values:
guard.add_entities({
    Entity.CREDIT_CARD: Action.HASH,
    Entity.EMAIL:       Action.REDACT,
})
```

### Choosing which filters run, and on which layer

Each entity can be detected by one or more of three layers — `regex`
(deterministic), `ner` (SpaCy), and `llm` (contextual/semantic). When you enable
an entity it runs on every layer that
supports it; pass `layers=[...]` to target a specific layer — for example, keep a
semantic-only entity off the regex/NER path:

```python
# Detect EMAIL with regex only; leave SPECIAL_CATEGORY (GDPR Art. 9) to the LLM
guard.add_entity("EMAIL", action="redact", layers=["regex"])
guard.add_entity("SPECIAL_CATEGORY", action="redact", layers=["llm"])
```

To turn on many filters at once, use `add_entities()`. It accepts a list,
a `{name: action}` mapping, or a `{name: {...}}` mapping for per-entity control,
and applies them in a single rebuild:

```python
from wardcat import Wardcat, turkish_entities

guard = Wardcat()

# a) a whole predefined group with one action
guard.add_entities(turkish_entities(), action="hash")

# b) an explicit list
guard.add_entities(["EMAIL", "CREDIT_CARD", "IBAN"], action="redact")

# c) per-entity actions and layers in one call
guard.add_entities({
    "CREDIT_CARD":      "hash",
    "EMAIL":            {"action": "mask"},
    "SPECIAL_CATEGORY": {"action": "redact", "layers": ["llm"]},
})
```

Predefined groups (`core_entities`, `financial_entities`, `turkish_entities`,
`european_entities`, `uk_entities`, `us_entities`, `network_entities`,
`identity_entities`, `all_entities`) are importable from `wardcat` and pair
naturally with `add_entities()`.

#### Enable everything, then prune

`Entity.ALL` turns on **every** known entity in one call; `remove_entity()` (and
`remove_entities()`) then disables the ones you do not want. This "allow-list by
exclusion" pattern is often the quickest way to a strict policy:

```python
from wardcat import Wardcat, Entity

guard = (
    Wardcat(salt="my-secret-salt")
    .add_entity(Entity.ALL, action="hash")   # everything on, hashed
    .remove_entity(Entity.ORG)               # …except organisation names
    .remove_entities([Entity.UUID, Entity.MAC_ADDRESS])
)

# remove_entity(Entity.ALL) disables everything again.
```

To **change the action** of an entity that is already enabled — without touching
which layers it runs on — use `change_entity_action()`:

```python
guard.change_entity_action(Entity.EMAIL, Action.HASH)      # warn → hash
guard.change_entity_action(Entity.ALL, Action.REDACT)      # every enabled entity → redact
```

`change_entity_action()` never silently re-enables an entity: changing the action
of a removed or never-added entity raises `ConfigError` — add it first with
`add_entity()`.

To **inspect** the current policy at any point:

```python
guard.enabled_entities()            # {"CREDIT_CARD", "EMAIL", ...} — what's on
guard.get_entity_action("EMAIL")    # "hash"  (None if the entity is not enabled)
guard.entity_policy()               # {"CREDIT_CARD": "hash", "EMAIL": "warn", ...}

# Discover what wardcat *can* detect (and on which layer):
Wardcat.supported_entities()        # every known entity type
Wardcat.supported_entities("ner")   # {"PERSON", "ORG", "ADDRESS"}
Wardcat.supported_entities("llm")   # contextual/semantic types
```

> Removing an entity that was never enabled is a no-op (an unknown *name* logs a
> warning, to catch typos). Passing a bare string to `add_entities()` /
> `remove_entities()` (instead of a list) raises `ConfigError` — use the singular
> `add_entity()` / `remove_entity()` for one entity. `Entity.All` is a deprecated
> alias of `Entity.ALL`.

### Declarative API (YAML)

```python
from wardcat import Wardcat

guard = Wardcat(config_path="config/my_policy.yaml")
result = guard.scan(text)
```

```yaml
# config/my_policy.yaml
salt: ""          # read from env in production
use_ner: false   # NER is off by default; set true AND provide a model below
# spacy_model: "en_core_web_sm"      # one model
# spacy_models: ["en_core_web_sm", "de_core_news_sm"]   # or several (multilingual)

entities:
  CREDIT_CARD: { enabled: true,  action: hash }
  EMAIL:       { enabled: true,  action: warn }
  TC_ID:       { enabled: true,  action: hash }
  IBAN:        { enabled: true,  action: hash }
  PERSON:      { enabled: true,  action: hash }
  ORG:         { enabled: false, action: warn }
```

### Batch Scanning

```python
guard = Wardcat(salt="s").add_entities(["EMAIL", "CREDIT_CARD"])
results = guard.scan_batch([
    "ali@example.com",
    "Card: 4111 1111 1111 1111",
    "Clean text.",
])

for r in results:
    print(r.is_clean, len(r.violations))
# False 1
# False 1
# True  0
```

### Async & concurrency

Every scanning entry point has an async twin — `scan_async`, `scan_batch_async`,
`is_sensitive_async`. CPU layers (regex, NER) run in a thread pool; the LLM layer
uses native async I/O (`httpx.AsyncClient`), so concurrent requests **overlap**
instead of blocking each other:

```python
import asyncio

guard = Wardcat(salt="s").with_llm(model="llama3.1:8b").add_entity("EMAIL")

texts = ["mail me at a@example.com", "card 4111 1111 1111 1111", "clean text"]

async def main():
    # one async scan
    result = await guard.scan_async(texts[0])

    # many at once — their LLM calls overlap (asyncio.gather)
    results = await asyncio.gather(*(guard.scan_async(t) for t in texts))

asyncio.run(main())
```

**Concurrency notes**
- **Build one guard and share it.** A `Wardcat` instance is safe to reuse across
  concurrent scans (the SpaCy model cache, LLM cache, and detectors are shared and
  lock-protected). Don't *reconfigure* it (`add_entity` / `with_ner` / `with_llm`)
  while it is serving scans — configure once at startup.
- **`scan_async` is non-blocking, but the LLM server is the real ceiling.** Your
  requests overlap in Python, yet a single Ollama on one GPU may still process them
  near-sequentially. For genuine parallel LLM throughput use **vLLM** (continuous
  batching) or raise `OLLAMA_NUM_PARALLEL` with enough VRAM.
- Regex/NER-only scans are already sub-millisecond to ~100 ms, so concurrency there
  is rarely the bottleneck.

**FastAPI (async handler):**

```python
from fastapi import FastAPI
from wardcat import Wardcat

guard = Wardcat(salt="s").add_entities(["EMAIL", "CREDIT_CARD", "TC_ID"])
app = FastAPI()

@app.post("/scan")
async def scan(text: str):
    result = await guard.scan_async(text)      # await — does not block the loop
    return {"sanitized": result.sanitized_text, "clean": result.is_clean}
```

### Catching every occurrence (value propagation)

Model-based layers (SpaCy NER, the LLM) sometimes report a value that
repeats in a document only **once**, which would leave the other occurrences
unredacted. Enable `with_propagation()` so that once *any* layer detects a value,
**every** whole-token occurrence of it is anonymized — with that value's entity
type and action:

```python
guard = (
    Wardcat(salt="s")
    .with_ner(language="en")
    .add_entity(Entity.PERSON, Action.REDACT)
    .with_propagation()          # a name caught once → redacted everywhere
)
```

It is **off by default** because it can over-redact (e.g. a short, common name).
Only exact, token-bounded matches at least `min_length` chars long (default 3)
are propagated, and deterministic regex spans still win overlaps — a propagated
match never displaces a checksum-validated one. Structural PII (email, phone,
IBAN…) is already caught exhaustively by the regex layer, so propagation mainly
helps names and other model-only entities.

### On-prem LLM

> **Fluent setup (recommended).** Instead of passing a dozen `llm_*` / `spacy_*`
> constructor arguments, use the chainable builders — they keep each layer's
> config in one place and read top-to-bottom:
>
> ```python
> from wardcat import Wardcat, Backend, Language
>
> guard = (
>     Wardcat(salt="s")
>     .with_ner(language=Language.TR)                       # NER layer
>     .with_llm(backend=Backend.OLLAMA, model="llama3.2",   # LLM layer
>               adjudicate=True)
> )
> ```
>
> `with_ner()` and `with_llm()` both return `self`, so they chain back-to-back
> (regex + NER + LLM all active).

The LLM detector is configured **only** via `with_llm()` (or a YAML `config_path`)
— it is not a set of constructor arguments. `backend` is the backend **type**
(use the `Backend` constants — `Backend.OLLAMA`, `Backend.VLLM`,
`Backend.OPENAI_COMPATIBLE`, `Backend.TRANSFORMERS`; plain strings work too);
the **address** goes to `base_url`.

**Option 1 — Run locally with Ollama:**

```bash
# Install Ollama: https://ollama.com
ollama pull llama3.2
```

```python
from wardcat import Wardcat, Backend

guard = Wardcat(salt="s").with_llm(
    backend=Backend.OLLAMA,
    model="llama3.2",
    base_url="http://localhost:11434",
)
```

**Option 2 — vLLM server:**

```bash
vllm serve meta-llama/Llama-3.1-8B-Instruct   # default port 8000
```

```python
from wardcat import Wardcat, Backend

guard = Wardcat(salt="s").with_llm(
    backend=Backend.VLLM,
    model="meta-llama/Llama-3.1-8B-Instruct",   # the served model name
    base_url="http://localhost:8000/v1",         # vLLM's default endpoint
    # api_key="..."   # only if vLLM was started with --api-key
)
```

`Backend.VLLM` sends the full chat message array natively and defaults to
vLLM's `http://localhost:8000/v1`. For other OpenAI-compatible servers (LM
Studio, LocalAI, LiteLLM), use `Backend.OPENAI_COMPATIBLE` with your endpoint's
`base_url`.

**Option 3 — HuggingFace Transformers (GPU/CPU):**

```python
from wardcat import Wardcat, Backend

guard = Wardcat(salt="s").with_llm(
    backend=Backend.TRANSFORMERS,
    model="meta-llama/Llama-3.1-8B-Instruct",
    load_in_8bit=True,  # optional: reduce VRAM usage
)
```

> **Note:** Loopback HTTP (`localhost` / `127.0.0.1` / `::1`) is allowed with no warning — it never leaves the machine, so the common local-Ollama setup needs no `allow_http`. HTTP to a **remote** host is blocked (pass `allow_http=True` to override); use HTTPS in production via a reverse proxy (nginx, Caddy).

#### Custom actions (extensible)

Actions (`hash`/`redact`/`mask`/`warn`) come from a registry, so you can add your
own — `tokenize`, `encrypt`, format-preserving masking — **without changing
wardcat**. An action maps a span to its replacement (or `None` to keep the text):

```python
from wardcat import Wardcat, register_action

# ctx carries the salt; the span has .entity_type, .text, .start, .end
register_action("tokenize", lambda span, ctx: f"<{span.entity_type}:{vault.put(span.text)}>")

guard = Wardcat(salt="s").add_entity("EMAIL", "tokenize")   # use it like any built-in
```

> Detection and anonymization are separate stages: `DetectionEngine` finds the
> spans, and a standalone `Anonymizer` applies the actions — so you can reuse the
> action registry or the `Anonymizer` independently.

> **LLM backends are not user-extensible.** wardcat ships four — `ollama`,
> `openai_compatible`, `vllm`, `transformers` — selected via the `Backend` enum.
> A third-party backend would sit outside wardcat's safety checks (the
> plaintext-HTTP-to-remote guard, PII handling), so point a built-in at your
> endpoint instead: `openai_compatible` covers most OpenAI-style gateways.

#### Ensemble adjudication

By default the three layers run independently and their findings are merged (union). With `with_llm(adjudicate=True)`, the engine instead sends the regex/NER candidates to the LLM, which — in a **single call** — verifies each one (keep / relabel / drop) and adds any PII the other layers missed. Deterministic, checksum-validated regex spans are always kept regardless of the LLM verdict. This sharply reduces NER noise (e.g. job titles mislabeled as names, or a model run on the wrong language):

```python
guard = Wardcat().with_ner(spacy_model="de_core_news_sm").with_llm(
    model="gemma3:12b",
    adjudicate=True,   # LLM acts as detector + arbiter in one call
)
```

> Adjudication has no effect in LLM-only deployments (no regex/NER candidates to judge) — the LLM simply runs as a pure detector.

### Is this text sensitive? (semantic gate)

Sometimes you don't need *what* the PII is — just a yes/no on whether a text is safe to send onward. `is_sensitive()` asks the configured LLM for a single **holistic** judgement (a general semantic decision, not the per-entity extraction of `scan()`), so it also catches things the enumerated detectors don't — unreleased financials, deal terms, a confidential project:

```python
from wardcat import Wardcat, Backend

guard = Wardcat().with_llm(backend=Backend.OLLAMA, model="gemma3:12b")

guard.is_sensitive("Ödeme için IBAN TR58 0000 0011 1111 1111 1111 11")   # True
guard.is_sensitive("Aksa Teknoloji'nin açıklanmamış Q2 net kârı 47,3M TL")  # True (confidential)
guard.is_sensitive("What's the weather like tomorrow?")                    # False

# guardrail before calling an external service
if guard.is_sensitive(user_text):
    raise ValueError("won't forward sensitive text")
```

- **LLM-only** — configured through `with_llm(...)`; no entities to enable, no regex/NER runs. Raises `ConfigError` if the LLM layer isn't set up.
- **Fail-closed** — if the backend is unreachable the error propagates (a guardrail never silently treats sensitive text as safe). Empty text is `False`.
- **Localized prompt** — `with_llm(language="tr")` (or `de`/`fr`) uses a system prompt written in that language, which can help smaller models; any other value uses the English, multilingual-aware prompt. (This affects only `is_sensitive()`, not the `scan()` detection prompt.)
- **Long inputs** are chunked at paragraph boundaries — any sensitive chunk makes the whole text sensitive — and oversized input is rejected (`max_text_bytes`).
- Async: `await guard.is_sensitive_async(text)`.

---

### Examples

Runnable scripts in [`examples/`](examples/):

| File | Shows |
|---|---|
| `demo.py` | Programmatic + YAML APIs |
| `batch_and_async.py` | `scan_batch` and the async API (regex-only, no services) |
| `llm_hybrid.py` | regex + NER + LLM with ensemble adjudication (needs Ollama) |
| `asgi_middleware.py` | Copy-paste ASGI middleware (FastAPI/Starlette) that scans request bodies — wardcat ships no web-framework code; this is a self-contained example |

---

## Supported Entity Types

### Regex (built-in, no dependencies)

#### Financial & Identity

| Entity | Default Action | Description |
|---|---|---|
| `CREDIT_CARD` | `hash` | Visa, MC, Amex, Discover — with or without separators; Luhn (mod-10) validated |
| `IBAN` | `hash` | International IBAN — mod-97 checksum validated |
| `SSN` | `hash` | US Social Security Number (123-45-6789) |
| `NIN` | `hash` | UK National Insurance Number (AB123456C) |
| `TC_ID` | `hash` | Turkish national ID — 11 digits, Nüfus İdaresi checksum validated |
| `EU_NATIONAL_ID` | `hash` | Spanish DNI (12345678Z) / NIE (X1234567L), French INSEE (15 digits) |
| `CODICE_FISCALE` | `hash` | Italian tax code (RSSMRA85T10A562S) |
| `VAT_NUMBER` | `warn` | EU VAT (DE/FR/GB/IT/ES/AT/NL prefixed) + Turkish Vergi No (keyword-based) |
| `PASSPORT` | `hash` | Passport numbers — keyword-based (`passport no:`, `pasaport`, `Reisepass`, `passeport`) |
| `DATE_OF_BIRTH` | `hash` | Birth dates — month names (TR/EN/DE/FR) or keyword + numeric/ISO date |
| `VEHICLE_PLATE` | `warn` | Turkish vehicle plates (34 ABC 123) |
| `FINANCIAL_AMOUNT` | `redact` | Monetary amounts (₺/$/€/£, `45.000 TL`, `2.1 milyon TL`) — **off by default** |

#### Contact & Location

| Entity | Default Action | Description |
|---|---|---|
| `EMAIL` | `warn` | RFC-compliant email addresses |
| `PHONE` | `warn` | Turkish (`0`, `+90`), French national (`01 23 45 67 89`), German mobile (`0151 …`), and international E.164 (`+1`, `+44`, …) |
| `ADDRESS` | `warn` | Turkish (Cad., Sok., Mah.), English (Street, Avenue, Road…), French (Rue, Allée…), Spanish (Calle, Avenida…), Italian (Piazza, Corso…), Dutch (straat, gracht…), German (Straße, Weg, Platz…) |
| `POSTAL_CODE` | `warn` | Turkish postal codes (01000–81999) |
| `UK_POSTAL_CODE` | `warn` | British postcodes (SW1A 1AA, GU21 6TH, M1 1AE) |
| `US_ZIP_CODE` | `warn` | US ZIP+4 codes (12345-6789) |

#### Network & Technical

| Entity | Default Action | Description |
|---|---|---|
| `IP_ADDRESS` | `warn` | IPv4 addresses |
| `IPv6` | `warn` | IPv6 addresses (full and compressed forms) |
| `MAC_ADDRESS` | `warn` | Network hardware address (00:1A:2B:3C:4D:5E) |
| `UUID` | `warn` | RFC 4122 UUID / GUID |
| `JWT` | `hash` | JSON Web Token (starts with `eyJ`) |
| `CUSTOM_SECRET` | `hash` | API keys & tokens: OpenAI/Anthropic (`sk-`, `sk-ant-`), Stripe (`sk_live_`), AWS (`AKIA`), Google (`AIza`, `ya29.`), GitHub (`ghp_`), GitLab (`glpat-`), Slack (`xoxb-`, webhook URLs), Twilio (`SK`/`AC`), SendGrid (`SG.`), npm (`npm_`), and PEM private-key blocks |

### SpaCy NER (requires `spacy` + language model)

| Entity | Default Action | Description |
|---|---|---|
| `PERSON` | `hash` | Person names (first + last) — cross-language |
| `ORG` | `warn` | Organization / company names |
| `ADDRESS` | `warn` | Location entities (complements regex) |

> **NER requires an explicit model — there is no default.** NER is **off by default**; calling `with_ner()` without a model (or language) raises `ConfigError`. Choose a model in a documented way via `language=` (recommended) or `spacy_model=`. Running, say, the Turkish model on German text produces noisy results, so pick the model per language (or rely on the LLM layer for cross-language names). A multilingual gazetteer filters out job titles, HR terms, and abbreviations (EN/DE/FR/TR) that NER models commonly mislabel.

**Pick a language, not a model name.** Use the `Language` constants (or their ISO codes) and an optional size tier — wardcat resolves the right model from its catalog and **downloads it automatically if it is missing**:

```python
from wardcat import Wardcat, Language

# Selecting a language turns NER on and implies auto-download (off with auto_download=False)
guard = Wardcat().with_ner(language=Language.DE)                  # → de_core_news_sm
guard = Wardcat().with_ner(language=Language.FR, spacy_size="md") # → fr_core_news_md (sm/md/lg/trf)
guard = Wardcat().with_ner(language=Language.TR, spacy_size="lg") # → tr_core_news_lg

# Or name the SpaCy package(s) explicitly — one or several:
guard = Wardcat().with_ner(spacy_model="en_core_web_sm")
guard = Wardcat().with_ner(spacy_model=["en_core_web_sm", "de_core_news_sm"])
```

Supported languages: `Language.EN`, `DE`, `FR`, `ES`, `IT`, `NL`, `PT`, `TR` (plain ISO codes like `"de"` are also accepted). If the requested size is unavailable for a language, the recommended model is used.

#### Multi-language text

For text that mixes several languages you have two options:

1. **LLM layer (recommended, no extra models).** The LLM prompt is multilingual, so a single LLM-enabled guard detects names across languages without loading any SpaCy model:

   ```python
   guard = Wardcat().with_llm(model="llama3.1:8b")  # handles EN+DE+FR+TR in one call
   ```

2. **Multiple SpaCy models.** Pass a list of languages — wardcat loads one NER model per language and the engine merges their results. Each model adds RAM and roughly multiplies NER scan time, so this is opt-in:

   ```python
   from wardcat import Wardcat, Language
   guard = Wardcat().with_ner(language=[Language.EN, Language.DE, Language.FR])  # 3 NER models, auto-downloaded
   ```

   This is **explicit** — you list the languages; wardcat does not guess the language of the input. The regex layer is multilingual regardless of these choices.

### LLM (requires on-prem LLM backend)

| Entity | Default Action | Description |
|---|---|---|
| `PASSPORT` | `hash` | Passport numbers of any country — contextual detection (e.g. `Passport: A12345678`) |
| `EU_NATIONAL_ID` | `hash` | French INSEE, German Personalausweis — contextual variants not caught by regex |
| `UK_POSTAL_CODE` | `warn` | Postcodes in ambiguous contexts |
| `US_ZIP_CODE` | `warn` | ZIP codes in ambiguous contexts |
| `CUSTOM_SECRET` | `hash` | Contextual secrets — `password=VALUE`, `api_key=VALUE`, access codes |
| `SPECIAL_CATEGORY` | `redact` | GDPR Art.9 special-category data — health, religion, ethnicity, political opinion, sexual orientation, trade-union, genetic/biometric. **Off by default** (semantic, LLM-only) |
| *(any above)* | — | LLM supplements and verifies all regex/NER entity types |

> **GDPR special categories:** `SPECIAL_CATEGORY` flags sensitive statements that have no pattern — a stated health condition, religious or political affiliation, or trade-union membership — which only the LLM can detect. It is off by default; enable it under `llm_detector.entities`. Because it is semantic and subjective, expect lower precision than structural entities.

---

## Output Structure

`scan()` and `scan_batch()` return a `ScanResult` per text:

```python
@dataclass
class ScanResult:
    original_text:  str               # unmodified input — contains raw PII
    sanitized_text: str               # anonymized output — safe to forward to LLM
    violations:     List[Violation]   # all detected violations
    is_clean:       bool              # True if no violations found
    warnings:       List[str]         # non-fatal issues — a layer that could not run

    def redacted(self) -> dict:       # PII-free dict — safe for logging and APIs
        ...

@dataclass
class Violation:
    entity_type: str          # e.g. "CREDIT_CARD", "EMAIL"
    original:    str          # raw value from input — contains PII
    start:       int          # start index in original text
    end:         int          # end index in original text
    action:      Action       # WARN | HASH | REDACT | MASK
    replacement: str | None   # "[TYPE:16hex]" for hash, "[TYPE]" for redact, masked value for mask, None for warn
    confidence:  float        # checksum 1.0 · structural regex 0.97 · fuzzy regex 0.90 · NER/LLM 0.85
```

> **Degraded scans — check `warnings`.** If a detector layer cannot run (most
> commonly the LLM backend being unreachable), the scan still returns the other
> layers' results but records the failure in `result.warnings`. A non-empty
> `warnings` means detection was **degraded** — some PII may have been missed —
> so you are not silently misled into thinking every layer ran:
>
> ```python
> result = guard.scan(text)
> if result.warnings:
>     logger.warning("PII scan degraded: %s", result.warnings)
> ```

---

## Security

### Hashing

Sensitive values are replaced in the sanitized text using the format `[TYPE:16hex]` (64-bit entropy):

```
4111 1111 1111 1111  →  [CREDIT_CARD:b22b36262d8d2769]
12345678950          →  [TC_ID:86349f34a1bc2d5e]
```

> **Limitation — deterministic hashing is not anonymization.** The same value always hashes to the same token (this is intentional: it lets you correlate records). But because the hash is deterministic and SHA-256 is fast, **low-entropy PII (phone numbers, SSNs, TC IDs) can be recovered by brute force** by anyone who has the salt and knows the value format. Treat hashed output as *pseudonymized*, not anonymized: keep the salt secret, and use `redact`/`mask` when you need values that cannot be reversed.

### Salt

Salt prevents rainbow table attacks. Set it via environment variable in production:

```python
import os
guard = Wardcat(salt=os.environ["WARDCAT_SALT"]).add_entity("CREDIT_CARD", "hash")
```

> Never hardcode the salt in source code or config files.

### Safe Logging

`ScanResult.original_text` and `violations[].original` contain raw PII. Use `result.redacted()` for logs and API responses:

```python
result = guard.scan(text)

# ✗ Leaks PII to logs
logger.info("result: %s", result.original_text)

# ✓ Safe — no raw PII
logger.info("result: %s", result.redacted())
```

### Input Size Limit

Inputs exceeding **500 KB** raise a `ValueError`. Split large documents into smaller chunks before scanning.

---

## Configuration

> **The library never reads environment variables.** Pass everything explicitly —
> `Wardcat(salt=...).with_llm(base_url=..., allow_http=...)` or a YAML
> `config_path`. Read secrets from the environment in *your* application and hand
> them to the constructor; wardcat itself does no implicit environment lookup.

To allow plaintext HTTP to a **remote** LLM (blocked by default), pass
`with_llm(allow_http=True)`.

---

## Project Structure

```
wardcat/
├── src/wardcat/
│   ├── guard.py              # Wardcat — main interface
│   ├── core/
│   │   ├── engine.py         # DetectionEngine — overlap resolution, action application
│   │   └── models.py         # Action, Violation, ScanResult
│   ├── detectors/
│   │   ├── base.py           # BaseDetector ABC
│   │   ├── regex_detector.py # 25+ regex patterns with checksum/Luhn validation
│   │   ├── ner_detector.py   # SpaCy NER (multilingual) + gazetteer FP filter
│   │   └── llm_detector.py   # LLM-based detection with hallucination filter
│   ├── llm/
│   │   ├── backends/         # ollama, openai_compat, transformers
│   │   ├── model_catalog.py  # Supported model list
│   │   └── prompt.py         # PII detection prompt builder
│   ├── config/
│   │   └── loader.py         # YAML loader, env var overrides
│   └── utils/
│       └── hashing.py        # SHA-256 + salt
├── tests/
│   ├── unit/                 # Component-level tests
│   └── integration/          # Scenario and adversarial tests
├── config/
│   └── default.yaml          # Example policy file
└── pyproject.toml
```

---

## Testing

```bash
# All tests
uv run pytest tests/unit/ tests/integration/

# Unit tests only
uv run pytest tests/unit/

# NER tests (requires SpaCy model)
uv run pytest -m ner

# Live LLM tests against a real Ollama model (auto-skipped if unavailable)
uv run pytest -m slow tests/integration/test_llm_live.py
# Pick the model: WARDCAT_TEST_LLM_MODEL=llama3.2:1b uv run pytest -m slow ...

# Skip slow tests (fast run)
uv run pytest -m "not slow"

# Coverage report
uv run pytest --cov=src/wardcat --cov-report=term-missing
```

---

## Known Limitations

| Case | Description | Mitigation |
|---|---|---|
| Homoglyph domain | Confusable folding (`normalize_confusables`, on by default) catches Cyrillic/Greek lookalikes and fullwidth/Arabic digits (`ali@tеst.com`, `４111…`), but it is a curated skeleton, not the full Unicode table — an exotic lookalike may slip through | Add an allowlist/extra validation for high-stakes fields; the LLM layer can flag suspicious text |
| `US_ZIP_CODE` | A bare 5-digit ZIP is matched only with a `ZIP:` keyword; otherwise ZIP+4 (`12345-6789`) is required, to avoid false positives | Enable `POSTAL_CODE` (catches any bare 5-digit), or the LLM layer for contextual ZIPs |
| `EU_NATIONAL_ID` | Spanish DNI/NIE and French INSEE are matched by regex; German IDs are not | Enable the LLM layer — it catches German (and other) national IDs contextually |
| `PASSPORT` | Regex needs a passport keyword (`passport no:`, `pasaport`, `Reisepass`, `passeport`) — bare passport numbers are too generic to match safely | Enable the LLM layer — it catches unlabeled passport numbers |
| `FINANCIAL_AMOUNT` / `VAT_NUMBER` | `FINANCIAL_AMOUNT` is off by default; a bare Turkish Vergi No needs a keyword | Enable `FINANCIAL_AMOUNT` for confidential docs; use a `Vergi No`/`VKN` keyword or the LLM layer for VAT |
| Multilingual NER | One SpaCy model loads per language; wardcat bundles no language *detection* by design (keeps the core dependency-light) | Detect the language yourself (check `supported_languages()`) and pass several models via `language=[...]`, or use the language-agnostic LLM layer |
| European addresses | Regex needs a street-type keyword (Straße, Rue, Calle…); unnumbered informal addresses may be missed | Use the NER `ADDRESS` / LLM layer, or add a `custom_patterns` rule for your address format |
| Turkish NER (`tr_core_news_trf`) | The transformer model is incompatible with SpaCy 3.5+ | Use `tr_core_news_md` or `tr_core_news_lg` |
| Turkish NER quality | `tr_core_news_md/lg` are news-trained and may miss names in non-standard contexts | Combine with `.with_llm(...)` — the LLM catches names NER misses |

---

## Development

```bash
uv sync --dev
uv run pytest tests/unit/ tests/integration/ tests/benchmark/
```

Requires Python 3.11+.

**Detection quality.** `tests/benchmark/` scores the detectors two ways: a
false-positive suite (detectors must stay quiet on clean text) and a
precision/recall harness over a labelled, checksum-valid corpus. Both run in CI.
Print the P/R report:

```bash
uv run python -m tests.benchmark.eval_harness
```

Widen coverage by adding rows to `CORPUS` in `tests/benchmark/eval_harness.py`.
