Metadata-Version: 2.4
Name: veil-llm
Version: 0.1.0
Summary: Local DLP reverse proxy for LLM traffic
Project-URL: Homepage, https://github.com/veilllm/veil-llm
Author: VeilLLM Contributors
License: MIT
License-File: LICENSE
Keywords: anonymization,api,dlp,llm,masking,openai,pii,privacy,proxy,security
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Information Technology
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Internet :: Proxy Servers
Classifier: Topic :: Security
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Requires-Python: >=3.12
Requires-Dist: fastapi>=0.115.0
Requires-Dist: httpx>=0.28.0
Requires-Dist: presidio-analyzer<3,>=2.2.0
Requires-Dist: uvicorn[standard]>=0.34.0
Description-Content-Type: text/markdown

# VeilLLM

A local DLP reverse proxy for LLM traffic. Mask PII, API keys, and secrets before they leave your machine — restore them on the way back.

![banner](docs/banner.png)



https://github.com/user-attachments/assets/dec6fb5d-64d2-4731-a4b3-40260ff8f461



## Quickstart

```bash
# Install
pip install veil-llm

# Start the gateway
veil-llm serve --port 4000
```

On first run, VeilLLM creates a `./config.toml` with a random salt and default settings. The spaCy model (`en_core_web_lg`, ~500MB) downloads automatically if missing.

### CLI reference

```
veil-llm serve [options]
veil-llm config            # show resolved settings
veil-llm --help            # full help
```

| Flag | Env var | Default | Description |
|---|---|---|---|
| `--port` | `GATEWAY_PORT` | `4000` | Listen port |
| `--host` | — | `127.0.0.1` | Listen host |
| `--upstream-url` | `GATEWAY_UPSTREAM_URL` | `https://openrouter.ai/api/v1` | Upstream API base URL |
| `--model-name` | `GATEWAY_MODEL_NAME` | `en_core_web_lg` | spaCy model for PII detection |
| `--debug` | `GATEWAY_DEBUG=1` | `false` | Log placeholder→original map to stderr (⚠️ leaks secrets) |
| `--high-entropy` | `GATEWAY_HIGH_ENTROPY=1` | `false` | Enable high-entropy string detection |
| `--entropy-threshold` | `GATEWAY_ENTROPY_THRESHOLD` | `4.5` | Shannon entropy threshold for high-entropy detection |
| `--enabled-entities` | `GATEWAY_ENABLED_ENTITIES` | `PERSON,EMAIL_ADDRESS,…` | Comma-separated list of PII entity types to detect |

Priority: `config.toml defaults` < `config.toml values` < `env vars` < `CLI flags`.

### Available spaCy models

| Model | Size | Vectors | Accuracy | Best for |
|---|---|---|---|---|
| `en_core_web_sm` | 12 MB | ❌ | Low | Minimal footprint, quick tests |
| `en_core_web_md` | 40 MB | ✅ | Medium | Balanced — smaller than lg, better NER than sm |
| `en_core_web_lg` | 560 MB | ✅ | High | Best CPU accuracy (default) |
| `en_core_web_trf` | 460 MB | ✅ | Highest | Transformer-based; slowest, needs GPU for practical use |

Use `--model-name` or config to switch:

```bash
veil-llm serve --model-name en_core_web_md
```

## Usage with any agentic harness

VeilLLM speaks the OpenAI-compatible API. Any harness that can point to a custom base URL — Hermes, Cline, Continue, Aider, OpenHands, or your own scripts — works with one config change:

```yaml
# Before: harness hits provider directly
base_url: https://api.openai.com/v1

# After: harness hits VeilLLM, which masks then forwards
base_url: http://127.0.0.1:4000/v1
```

Your `Authorization` header is forwarded through unchanged. VeilLLM masks sensitive data in your prompts and restores it in responses transparently — the harness never knows it's there.

> **Important:** VeilLLM v0.1 does not support streaming (`stream: true`). Configure your harness for non-streaming mode.

## Configuration

VeilLLM auto-creates `./config.toml` on first run with sensible defaults:

```toml
[gateway]
upstream_url = "https://openrouter.ai/api/v1"
port = 4000
salt = "<auto-generated>"
model_name = "en_core_web_lg"

[detection]
enabled_entities = [
    "PERSON",
    "EMAIL_ADDRESS",
    "PHONE_NUMBER",
    "CREDIT_CARD",
    "IBAN_CODE",
    "IP_ADDRESS",
    "LOCATION",
    "US_SSN",
]

[recognizers]
high_entropy_enabled = false
entropy_threshold = 4.5
```

Every setting can be overridden via environment variable or CLI flag (see table above). Run `gateway config` to see the resolved settings your gateway will use.

## What gets masked

| Category | Examples |
|---|---|
| PII (Presidio) | Names, emails, phone numbers, credit cards, IBANs, IPs, locations, SSNs |
| LLM API keys | `sk-...`, `sk-ant-...`, `sk-or-...` |
| AWS keys | `AKIA...` access keys and secret keys |
| GitHub tokens | `ghp_...`, `gho_...`, `github_pat_...` |
| Google API keys | `AIza...` |
| JWTs | `eyJ...` three-segment tokens |
| Private keys | PEM `-----BEGIN ... PRIVATE KEY-----` blocks |
| High-entropy strings | Opt-in: strings with Shannon entropy > threshold |

## Gradio playground

For a hands-on feel, launch the Gradio UI to see masking live — no upstream API key needed for the detection tab:

```bash
uv run --with gradio --with python-dotenv python playground.py
```

Open `http://127.0.0.1:7860`. The **PII Detection** tab lets you paste text and see exactly what gets masked. The **Chat** tab sends masked prompts to an LLM and shows a transparency log of every entity detected, masked, and restored — the same thing the proxy does silently.

`playground.py` is a self-contained demo; read it alongside this README to understand the full flow.

## How it works

VeilLLM replaces detected values with deterministic placeholders:

```
"Jane" → <PERSON_a3f2>
"jane@example.com" → <EMAIL_9c1d4e>
"sk-abc123..." → <API_KEY_77b0af>
```

The same value always maps to the same placeholder (salted hash), so agent conversation history stays coherent across turns. The salt is a random value in `./config.toml` — never sent over the network.

On the response, placeholders are restored using tolerant regex matching (handles LLM-mangled casing, spaces, and punctuation).

## Endpoints

| Endpoint | Behavior |
|---|---|
| `POST /v1/chat/completions` | Mask → forward → unmask |
| `GET /v1/models` | Pass-through |
| `GET /healthz` | Liveness check |
| Any other `/v1/*` | Pass-through (no masking, warning logged) |

## Telemetry

Per-request log line (stderr):
```
[req a3f2bc8d] masked 3 entities: API_KEY×1, EMAIL_ADDRESS×1, PERSON×1 | detect 35ms | upstream 1850ms | unmask 2ms
```

Set `GATEWAY_DEBUG=1` to see the placeholder→original map (with a prominent warning).

## Docker

```bash
docker build -t veilllm .
docker run -p 4000:4000 -v ./config.toml:/app/config.toml veilllm
```

## Limitations (v0.1)

- No streaming (SSE) — returns 400 if `stream: true`
- English-only PII detection
- Single upstream URL (defaults to OpenRouter; any OpenAI-compatible API works via `GATEWAY_UPSTREAM_URL` or config.toml)
- Not a full proxy — only the `/v1/*` paths are forwarded

## License

MIT
