Metadata-Version: 2.4
Name: datafog
Version: 4.8.0b6
Summary: Lightning-fast PII detection and anonymization library, 100x+ faster than NER-based detection (see benchmarks/)
Author: Sid Mohan
Author-email: sid@datafog.ai
Project-URL: Homepage, https://datafog.ai
Project-URL: Documentation, https://docs.datafog.ai
Project-URL: Discord, https://discord.gg/bzDth394R4
Project-URL: Twitter, https://twitter.com/datafoginc
Project-URL: GitHub, https://github.com/datafog/datafog-python
Keywords: pii detection anonymization privacy regex performance
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Topic :: Text Processing
Classifier: Topic :: Security
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: pydantic<3.0,>=2.0
Requires-Dist: pydantic-settings>=2.0.0
Requires-Dist: typing-extensions>=4.0
Provides-Extra: nlp
Requires-Dist: click<9.0,>=8.0; extra == "nlp"
Requires-Dist: spacy<4.0,>=3.7.0; extra == "nlp"
Provides-Extra: nlp-advanced
Requires-Dist: gliner>=0.2.5; extra == "nlp-advanced"
Requires-Dist: torch<2.7,>=2.1.0; extra == "nlp-advanced"
Requires-Dist: transformers>=4.20.0; extra == "nlp-advanced"
Requires-Dist: huggingface-hub>=0.16.0; extra == "nlp-advanced"
Provides-Extra: ocr
Requires-Dist: numpy>=1.24.0; extra == "ocr"
Requires-Dist: pytesseract>=0.3.0; extra == "ocr"
Requires-Dist: Pillow>=12.2.0; extra == "ocr"
Requires-Dist: sentencepiece>=0.2.0; extra == "ocr"
Requires-Dist: protobuf>=4.0.0; extra == "ocr"
Provides-Extra: distributed
Requires-Dist: pandas>=2.0.0; extra == "distributed"
Requires-Dist: numpy>=1.24.0; extra == "distributed"
Requires-Dist: pyspark>=3.5.0; extra == "distributed"
Provides-Extra: web
Requires-Dist: fastapi>=0.100.0; extra == "web"
Requires-Dist: aiohttp>=3.13.4; extra == "web"
Requires-Dist: certifi>=2025.4.26; extra == "web"
Requires-Dist: requests>=2.33.0; extra == "web"
Provides-Extra: cli
Requires-Dist: click<9.0,>=8.0; extra == "cli"
Requires-Dist: typer>=0.12.0; extra == "cli"
Requires-Dist: pydantic-settings>=2.0.0; extra == "cli"
Provides-Extra: crypto
Requires-Dist: cryptography>=46.0.7; extra == "crypto"
Provides-Extra: test
Requires-Dist: pytest>=9.0.3; extra == "test"
Requires-Dist: pytest-asyncio>=1.3.0; extra == "test"
Requires-Dist: pytest-cov>=7.1.0; extra == "test"
Provides-Extra: docs
Requires-Dist: sphinx>=7.2.6; extra == "docs"
Provides-Extra: benchmark
Requires-Dist: pytest-benchmark>=4.0.0; extra == "benchmark"
Provides-Extra: dev
Requires-Dist: pytest>=9.0.3; extra == "dev"
Requires-Dist: pytest-asyncio>=1.3.0; extra == "dev"
Requires-Dist: pytest-cov>=7.1.0; extra == "dev"
Requires-Dist: sphinx>=7.2.6; extra == "dev"
Provides-Extra: all
Requires-Dist: click<9.0,>=8.0; extra == "all"
Requires-Dist: spacy<4.0,>=3.7.0; extra == "all"
Requires-Dist: gliner>=0.2.5; extra == "all"
Requires-Dist: torch<2.7,>=2.1.0; extra == "all"
Requires-Dist: transformers>=4.20.0; extra == "all"
Requires-Dist: huggingface-hub>=0.16.0; extra == "all"
Requires-Dist: numpy>=1.24.0; extra == "all"
Requires-Dist: pytesseract>=0.3.0; extra == "all"
Requires-Dist: Pillow>=12.2.0; extra == "all"
Requires-Dist: sentencepiece>=0.2.0; extra == "all"
Requires-Dist: protobuf>=4.0.0; extra == "all"
Requires-Dist: pandas>=2.0.0; extra == "all"
Requires-Dist: numpy>=1.24.0; extra == "all"
Requires-Dist: pyspark>=3.5.0; extra == "all"
Requires-Dist: fastapi>=0.100.0; extra == "all"
Requires-Dist: aiohttp>=3.13.4; extra == "all"
Requires-Dist: certifi>=2025.4.26; extra == "all"
Requires-Dist: requests>=2.33.0; extra == "all"
Requires-Dist: click<9.0,>=8.0; extra == "all"
Requires-Dist: typer>=0.12.0; extra == "all"
Requires-Dist: pydantic-settings>=2.0.0; extra == "all"
Requires-Dist: cryptography>=46.0.7; extra == "all"
Dynamic: author
Dynamic: author-email
Dynamic: classifier
Dynamic: description
Dynamic: description-content-type
Dynamic: keywords
Dynamic: license-file
Dynamic: project-url
Dynamic: provides-extra
Dynamic: requires-dist
Dynamic: requires-python
Dynamic: summary

﻿# DataFog Python

DataFog is a Python library for detecting and redacting personally identifiable information (PII).

It provides:

- Fast structured PII detection via regex
- An offline PII firewall for AI agents: a Claude Code hook and a LiteLLM
  gateway guardrail (introduced in 4.6)
- Optional NER support via spaCy and GLiNER
- A simple agent-oriented API for LLM applications
- Backward-compatible `DataFog` and `TextService` classes

## Agent & Gateway Firewall

DataFog includes two ready-made enforcement points that catch PII at the
moment it would leave your machine — offline, in microseconds, with matched
values never echoed into logs or transcripts:

- **Claude Code hook** (`datafog-hook`): gates agent tool calls (shell
  commands, web requests, file writes, MCP tools) and warns the model when
  prompts or tool results carry PII. ~70–90ms per invocation including
  process startup. Easiest install is the
  [Claude Code plugin](https://github.com/DataFog/datafog-claude-plugin):

  ```
  /plugin marketplace add DataFog/datafog-claude-plugin
  /plugin install datafog@datafog
  ```

  Manual hook setup and limitations: [examples/claude_code_hook/](examples/claude_code_hook/).

- **LiteLLM guardrail** (`DataFogGuardrail`): redacts or blocks PII in
  requests and responses at the gateway, for any LiteLLM-proxied provider.
  In-process (~40µs per message scanned; a request clears the guardrail in
  well under a millisecond), no sidecar service. Setup:
  [examples/litellm_guardrail/](examples/litellm_guardrail/).

Both default to the high-precision entity set (`EMAIL`, `PHONE`,
`CREDIT_CARD`, `SSN`); noisier types are opt-in. Known-safe values can be
exempted with an allowlist: `scan(text, allowlist=[...])` for exact values,
`allowlist_patterns=[...]` for full-match regexes (e.g. `^\d{10}$` to stop
unix timestamps matching as phone numbers) — available in both adapters and
the API. Presidio-style entity names (`EMAIL_ADDRESS`, `PHONE_NUMBER`,
`US_SSN`) are accepted as aliases for easy migration.

Every performance number above is reproducible with one command —
methodology, pinned payloads, and comparisons against Presidio and spaCy
NER live in [benchmarks/](benchmarks/).

## Installation

```bash
# Core install (regex engine)
pip install datafog

# Add spaCy support
pip install datafog[nlp]

# Add GLiNER + spaCy support
pip install datafog[nlp-advanced]

# Add local OCR support
pip install datafog[ocr]

# Add Spark/distributed support
pip install datafog[distributed]

# Everything
pip install datafog[all]
```

Python 3.13 support is certified for the core SDK, CLI, `nlp`,
`nlp-advanced`, and `ocr` install profiles. Donut OCR still requires a model
that is available locally before runtime use. `distributed` and `all` remain
optional, heavier profiles and are not part of the lightweight core path.

Python 3.14 support is certified for the core SDK and CLI. Optional profiles
remain dependent on upstream Python 3.14 package availability until they are
covered by the corresponding CI install-profile checks.

## Quick Start

```python
import datafog

text = "Contact john@example.com or call (555) 123-4567"
clean = datafog.sanitize(text, engine="regex")
print(clean)
# Contact [EMAIL_1] or call [PHONE_1]
```

## For LLM Applications

```python
import datafog

# 1) Scan prompt text before sending to an LLM
prompt = "My SSN is 123-45-6789"
scan_result = datafog.scan_prompt(prompt, engine="regex")
if scan_result.entities:
    print(f"Detected {len(scan_result.entities)} PII entities")

# 2) Redact model output before returning it
output = "Email me at jane.doe@example.com"
safe_result = datafog.filter_output(output, engine="regex")
print(safe_result.redacted_text)
# Email me at [EMAIL_1]

# 3) One-liner redaction
print(datafog.sanitize("Card: 4111-1111-1111-1111", engine="regex"))
# Card: [CREDIT_CARD_1]
```

## German Structured PII

German structured PII is country-specific and opt-in. Use explicit locale
selection or entity-type filtering when you want German VAT IDs, German IBANs,
tax IDs, postal codes, passports, or residence permits.

```python
import datafog

text = "Steuer-ID 12345678901 liegt vor."

print(datafog.scan(text, engine="regex").entities)
# []

print(datafog.scan(text, engine="regex", locales=["de"]).entities)
# [Entity(type='DE_TAX_ID', text='12345678901', ...)]
```

### Guardrails

```python
import datafog

# Reusable guardrail object
guard = datafog.create_guardrail(engine="regex", on_detect="redact")

@guard
def call_llm() -> str:
    return "Send to admin@example.com"

print(call_llm())
# Send to [EMAIL_1]
```

## Engines

Use the engine that matches your accuracy and dependency constraints:

- `regex`:
  - Fastest and always available.
  - Best for default structured entities: `EMAIL`, `PHONE`, `SSN`, `CREDIT_CARD`, `IP_ADDRESS`, `DATE`, `ZIP_CODE` (`DOB` and `ZIP` are accepted as input aliases).
  - Use `locales=["de"]` for German structured IDs such as `DE_VAT_ID`, `DE_IBAN`, `DE_TAX_ID`, `DE_POSTAL_CODE`, and passport or residence permit numbers.
- `spacy`:
  - Requires `pip install datafog[nlp]`.
  - Useful for unstructured entities like person and organization names.
- `gliner`:
  - Requires `pip install datafog[nlp-advanced]`.
  - Stronger NER coverage than regex for unstructured text.
- `smart`:
  - Cascades regex with optional NER engines.
  - If optional deps are missing, it degrades gracefully and warns.

## Optional OCR And Spark Surfaces

The 4.x line keeps the main package story centered on lightweight text PII
screening. OCR and Spark remain supported optional surfaces for users who
already rely on them, but they are not required for the core import, default
scan/redact helpers, or guardrail helpers.

- OCR:
  - Install `datafog[ocr]` for local image OCR helpers.
  - URL-based image downloading also needs `datafog[web,ocr]`.
  - Tesseract usage requires the system `tesseract` binary.
  - Python 3.13 is validated for the OCR install profile, Pillow,
    pytesseract, and system Tesseract smoke checks.
  - Donut OCR requires `datafog[nlp-advanced,ocr]` and a model already available
    locally.
- Spark:
  - Install `datafog[distributed]` for `SparkService`.
  - Spark PII UDF helpers also require `datafog[nlp]` and an installed spaCy
    model.
  - A Java runtime is required by PySpark.

OCR and Spark are not deprecated. Their broader API and packaging overhaul is
deferred; the 4.x goal is to keep them explicit, documented, and isolated from
the lightweight core path.

## Backward-Compatible APIs

The existing public API remains available.

### `DataFog` class

```python
from datafog import DataFog

result = DataFog().scan_text("Email john@example.com")
print(result["EMAIL"])
```

### `TextService` class

```python
from datafog.services import TextService

service = TextService(engine="regex")
result = service.annotate_text_sync("Call (555) 123-4567")
print(result["PHONE"])
```

## CLI

```bash
# Scan text
datafog scan-text "john@example.com"

# Redact text
datafog redact-text "john@example.com"

# Replace text with pseudonyms
datafog replace-text "john@example.com"

# Hash detected entities
datafog hash-text "john@example.com"

# Enable German regex identifiers
datafog redact-text "Steuer-ID 12345678901" --locale de
```

## Telemetry

DataFog telemetry is disabled by default.

To opt in:

```bash
export DATAFOG_TELEMETRY=1
```

To force telemetry off:

```bash
export DATAFOG_NO_TELEMETRY=1
# or
export DO_NOT_TRACK=1
```

Telemetry does not include input text or detected PII values.

## Development

```bash
git clone https://github.com/datafog/datafog-python
cd datafog-python
python -m venv .venv
source .venv/bin/activate  # Windows: .venv\Scripts\activate
pip install -e ".[all,dev]"
pip install -r requirements-dev.txt
pytest tests/
```
