Metadata-Version: 2.4
Name: llm-shield-proxy
Version: 1.0.3
Summary: Enterprise Zero-Egress Privacy Redaction Proxy Engine for LLMs
Author-email: Ninad Phalak <ninadphalak@gmail.com>
License-Expression: Apache-2.0
Project-URL: Homepage, https://github.com/ninadphalak/LLM-Shield-Proxy
Project-URL: Repository, https://github.com/ninadphalak/LLM-Shield-Proxy
Project-URL: Bug Tracker, https://github.com/ninadphalak/LLM-Shield-Proxy/issues
Keywords: llm,pii,redaction,privacy,proxy,fastapi,streaming,soc2,hipaa
Classifier: Development Status :: 5 - Production/Stable
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Security
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: fastapi>=0.100.0
Requires-Dist: uvicorn>=0.22.0
Requires-Dist: httpx>=0.24.0
Requires-Dist: pydantic>=2.0.0
Requires-Dist: pydantic-settings>=2.0.0
Provides-Extra: dev
Requires-Dist: pytest>=7.0.0; extra == "dev"
Requires-Dist: pytest-asyncio>=0.21.0; extra == "dev"
Requires-Dist: pytest-httpx>=0.24.0; extra == "dev"
Requires-Dist: pytest-mock>=3.10.0; extra == "dev"
Requires-Dist: openai>=1.0.0; extra == "dev"
Requires-Dist: build; extra == "dev"
Requires-Dist: twine; extra == "dev"
Dynamic: license-file

# LLM-Shield-Proxy - Enterprise Privacy Redaction Engine

[![PyPI version](https://badge.fury.io/py/llm-shield-proxy.svg)](https://pypi.org/project/llm-shield-proxy/)
[![License](https://img.shields.io/badge/License-Apache_2.0-blue.svg)](https://opensource.org/licenses/Apache-2.0)
[![Python Version](https://img.shields.io/badge/python-3.9%2B-blue)](https://www.python.org/)

A zero-latency, zero-egress, streaming-safe PII redaction proxy for Enterprise LLMs.

**LLM-Shield-Proxy** is an open-source, zero-egress middleware proxy that intercepts OpenAI-compatible LLM API requests, redacts Personally Identifiable Information (PII) before it leaves your local infrastructure, and deterministically re-hydrates real-time SSE streaming responses without breaking stream latency.

Designed to unblock enterprise privacy compliance (**SOC 2 / HIPAA**).

Author & Core Maintainer: **Ninad Phalak** (`ninadphalak@gmail.com`)

---

## 🏗️ Architecture & Data Flow

```mermaid
flowchart TD
    classDef client fill:#e0f2fe,stroke:#0284c7,stroke-width:2px,color:#0369a1,font-weight:bold;
    classDef proxyEngine fill:#f8fafc,stroke:#475569,stroke-width:2px,color:#0f172a,font-weight:bold;
    classDef piiSecurity fill:#fef2f2,stroke:#ef4444,stroke-width:2px,color:#991b1b,font-weight:bold;
    classDef vault fill:#fffbebe,stroke:#f59e0b,stroke-width:2px,color:#92400e,font-weight:bold;
    classDef upstream fill:#f3e8ff,stroke:#9333ea,stroke-width:2px,color:#6b21a8,font-weight:bold;

    UserApp["👤 User Application\n(OpenAI / LangChain SDK)"]:::client

    subgraph SecurityMoat ["🛡️ Zero-Egress Local Environment (Apache 2.0 Licensed)"]
        direction TD
        FastAPIProxy["⚡ FastAPI Catch-All Proxy\n(/{path:path})"]:::proxyEngine

        subgraph CascadeEngine ["🔒 Two-Tier PII Cascade Engine"]
            Tier1["Tier 1: Compiled Regex"]:::piiSecurity
            Tier2["Tier 2: Quantized ONNX NER"]:::piiSecurity
            Tier1 --> Tier2
        end

        VaultStore[("🔑 Session Vault Store\n(Deterministic Tokens)")]:::vault
        LookaheadBuffer["⏱️ Sliding-Window Lookahead Buffer\n(Prevent SSE Tag Leaks)"]:::proxyEngine
        Rehydrator["🔄 Stream Re-hydrator\n(Token -> Original Value)"]:::proxyEngine
    end

    UpstreamLLM["☁️ Upstream LLM Provider\n(OpenAI / Anthropic / vLLM)"]:::upstream

    %% Inbound Flow (Prompt Sanitization)
    UserApp -- "1. Inbound Raw Prompt Payload" --> FastAPIProxy
    FastAPIProxy -- "2. Scan Payload" --> Tier1
    Tier2 -- "3. Store Vault Keys" --> VaultStore
    Tier2 -- "4. Redacted JSON Payload" --> UpstreamLLM

    %% Outbound Flow (Streaming De-redaction)
    UpstreamLLM -. "5. Raw SSE Stream Deltas" .-> LookaheadBuffer
    LookaheadBuffer -- "6. Tag-Safe Assembly" --> Rehydrator
    Rehydrator <--> VaultStore
    Rehydrator -. "7. Sanitized Real-Time Stream" .-> UserApp

    style SecurityMoat fill:#f8fafc,stroke:#0284c7,stroke-width:2px,stroke-dasharray: 5 5,color:#0f172a
    style CascadeEngine fill:#ffffff,stroke:#cbd5e1,stroke-width:1px
```

### How It Works (The Data Flow)

#### 📥 Inbound (Prompt Sanitization)
1. **Intercept:** Your application sends a standard OpenAI / LangChain payload to `localhost:8000`.
2. **Cascade Redaction:** The proxy intercepts the JSON and routes text through a high-speed compiled Regex engine (SSNs, emails, credit cards), falling back to a local ONNX model for unstructured names.
3. **Vault Storage:** The original PII is mapped to a deterministic tag (e.g., `[PERSON_1]`) and stored locally in a TTL-backed session vault.
4. **Clean Egress:** A 100% sanitized payload is forwarded to OpenAI. OpenAI never sees your raw sensitive data.

#### 📤 Outbound (Streaming De-redaction)
1. **SSE Stream Intercept:** OpenAI streams the response back chunk-by-chunk via Server-Sent Events (SSE).
2. **Lookahead Buffer:** Because tags can be split across SSE chunks (e.g., `[PER` in chunk N and `SON_1]` in chunk N+1), the proxy's sliding-window buffer holds back unclosed brackets to prevent tag leaks.
3. **Re-hydration:** Once a tag is fully assembled, the proxy swaps the real data back from the local vault and streams the final, un-redacted text to the user's application in real-time.

---

## 🔒 Why LLM-Shield-Proxy? (Architectural Moats)

Why not just write a basic regex script? Basic regex scripts break on streaming responses, leak split tokens, and slow down your application. LLM-Shield-Proxy is purpose-built for production:

- **Streaming Safety:** Includes a proprietary **Sliding-Window Lookahead Buffer** that prevents Server-Sent Event (SSE) chunking leaks across split tokens (e.g., holding back `[PER` until `SON_1]` arrives).
- **Speed & Accuracy:** Leverages a **Two-Tier Cascade Engine** (sub-millisecond compiled Regex + Quantized ONNX NER) that catches unstructured names in ~10ms with zero cloud dependencies.
- **Context Preservation:** Uses a **Local TTL Session Vault** that maps PII to session-bound tokens (`[PERSON_1]`), enabling the LLM to retain full conversational context without receiving raw sensitive data.
- **Zero-Egress Security:** Runs 100% on your local infrastructure. Your raw data never leaves your server; only sanitized, redacted payloads reach upstream LLM providers.

---

## ⚡ Core Features

- **Zero Latency Streaming:** Sliding-window tag-safety buffer intercepts SSE streams delta-by-delta without buffering full requests or responses.
- **Zero Cloud / Zero Egress:** 100% local processing. No external API calls for PII detection.
- **Two-Tier PII Cascade Engine:**
  - **Tier 1 (Sub-millisecond Regex):** SSNs, Credit Cards, Email Addresses, Phone Numbers, IPv4/IPv6, API Keys.
  - **Tier 2 (NER Engine):** Person Names and unstructured entities.
- **Deterministic Re-Hydration Vault:** Swaps PII with session-bound tokens (e.g., `Sarah` -> `[PERSON_1]`). Maps back deterministically when the LLM streams responses. Supports request-scoped and session-scoped (`X-Session-ID`) vaults.
- **SOC 2 Structured Audit Logging:** Emits JSON structured audit logs for compliance monitoring.
- **Tier 2 (NER Engine):** A lightweight, quantized ONNX Named Entity Recognition (NER) model. It uses local AI to catch unstructured names and entities that slip past standard regex, executing in ~10ms without requiring heavy Python frameworks or cloud APIs.

---

## 🛠️ Quickstart

### Installation

Install the package from PyPI:

```bash
pip install llm-shield-proxy
```

### Configuration
Create a `.env` file in the root directory before starting the server.

```env
# Required upstream provider key (or pass via Authorization header)
OPENAI_API_KEY=sk-your-openai-key-here

# Optional configuration
PORT=8000
TELEMETRY_ENABLED=false
```

#### 1. Start the Proxy

Run the proxy locally via Docker or Uvicorn. No external database required for the open-source core.

```bash
uvicorn app.main:app --host 0.0.0.0 --port 8000
```
or via Docker Compose:
```bash
docker-compose up -d
```

#### 2. Update your Application (1-Line Change)

Point your existing OpenAI SDK `base_url` to your local LLM-Shield-Proxy instance.

```python
from openai import OpenAI

client = OpenAI(
    api_key="your-openai-api-key",
    base_url="http://localhost:8000/v1" # Point to LLM-Shield-Proxy
)

response = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[
        {"role": "user", "content": "Contact Sarah Connor at sarah@example.com"}
    ],
    stream=True
)

for chunk in response:
    print(chunk.choices[0].delta.content or "", end="")
```

---

## 📊 Performance & Memory Benchmarks

LLM-Shield-Proxy is engineered for sub-millisecond overhead and ultra-lightweight resource usage. Measured over 1,000 production streaming iterations:

| Metric | Average Latency | Median Latency | Footprint / Notes |
| :--- | :--- | :--- | :--- |
| **Tier 1 Regex Overhead** | `0.0294 ms` | `0.0291 ms` (`29.10 µs`) | Microsecond pattern scan |
| **Tier 2 NER Overhead** | `0.0033 ms` | `0.0032 ms` (`3.20 µs`) | Quantized local NER scan |
| **Total SSE Stream Overhead** | `0.0010 ms` | `0.0010 ms` (`0.97 µs`) | Added latency per SSE delta chunk |
| **Process RAM Footprint** | - | - | `24.55 MB` Resident Set Size |

To run the automated benchmark suite locally:

```bash
py tests/benchmark.py
```

---

## ⚠️ Known Limitations

Transparency is critical for security tooling. Please be aware of the following current limitations:
- **Text Only:** The proxy does not currently scan or redact text embedded inside base64 image payloads (e.g., OpenAI Vision models).
- **Supported Languages:** The Tier-2 ONNX NER model is currently optimized for English-language entities. 
- **Non-Standard Streaming:** Designed for standard Server-Sent Events (SSE). Custom or proprietary streaming protocols may bypass the sliding-window buffer.

---

## 🧪 Testing

Run the full automated test suite:

```bash
py -m pytest tests/
```

---

## 🚀 Enterprise Deployment & Operations

Designed for zero-friction adoption by DevOps, Site Reliability Engineers (SREs), and Network Administrators:

### 1. 🏥 Health Check Endpoints (Kubernetes & Swarm Probes)
Built-in liveness and readiness endpoints return `HTTP 200 OK` for Kubernetes, Docker Swarm, or AWS ECS health monitors:
```bash
curl http://localhost:8000/health
# Output: {"status":"ok","service":"llm-shield-proxy","version":"1.0.3"}

curl http://localhost:8000/livez
# Output: {"status":"ok","service":"llm-shield-proxy","version":"1.0.3"}
```

### 2. ⚙️ 12-Factor Environment Configuration
100% compliant with 12-factor app standards. All upstream target routing and API keys are injected via environment variables or a `.env` file without code modifications:
- `UPSTREAM_BASE_URL`: Base target URL (e.g. `https://api.openai.com` or internal `vLLM` server).
- `OPENAI_API_KEY`: Upstream API key passed to target providers.
- `REDIS_URL`: Optional Redis connection string for distributed multi-instance session caching.

### 3. 📈 Stateless & Horizontal Scaling
LLM-Shield-Proxy runs completely stateless by default. For high-volume enterprise deployments, instances scale horizontally behind edge proxies (NGINX, Traefik, AWS ALB):
```bash
docker-compose up -d --scale proxy=5
```
When configured with `REDIS_URL`, session vaults are shared across all proxy replicas, ensuring seamless session isolation across multi-instance clusters.

### 4. 🔒 Supply Chain Integrity & GPG Signature Verification
Every published release includes automated SHA-256 checksums (`checksums.txt`) and GPG detached signatures (`checksums.txt.asc`) signed by maintainer **Ninad Phalak**. You can verify checksums and cryptographic authenticity before deployment using:

```bash
# 1. Verify SHA-256 Checksums (Linux / macOS):
sha256sum -c checksums.txt

# On Windows (PowerShell):
Get-FileHash llm-shield-proxy-source-v1.0.3.zip -Algorithm SHA256

# 2. Verify Cryptographic GPG Signature:
gpg --verify checksums.txt.asc checksums.txt
```

---

## 🌍 Internationalization (i18n) & GDPR Roadmap

Currently, LLM-Shield-Proxy's Tier 1 Regex engine is optimized for North American PII (US SSNs, Phone Formats). To support global GDPR compliance, we are actively looking for contributors to help expand our regex payloads and Tier 2 ONNX models for:
- **European Formats:** UK NIN, EU Phone Numbers, IBANs.
- **APAC Data Structures:** India Aadhaar, APAC localized identifiers.
- **Multilingual NER ONNX Models:** Multilingual entity recognition models.

If you want to contribute to enterprise AI security, check out our [CONTRIBUTING.md](CONTRIBUTING.md) and claim a locale!

---

## 🗺️ Future Technical Roadmap (Performance & Scale)

We are committed to maintaining LLM-Shield-Proxy as the fastest zero-latency redaction engine for LLMs. Here are the core architectural optimizations planned for upcoming releases — contributions and PRs are warmly welcomed:

1. **ONNX Thread Tuning (Preventing CPU Contention)**
   - *Problem:* By default, ONNX Runtime attempts to use every available CPU core. In FastAPI, this competes with the event loop handling thousands of concurrent connections.
   - *The Fix:* Restrict ONNX by setting `sess_options.intra_op_num_threads = 1`. This forces ONNX execution onto a single thread, keeping CPU cores free for FastAPI's event loop to stream packets instantly.

2. **Persistent Connection Pooling (The TLS Trick)**
   - *Problem:* Opening a new TLS/SSL connection to OpenAI per request adds 50–100ms latency.
   - *The Fix:* Maintain a persistent `httpx.AsyncClient` HTTP/2 connection pool on server startup. The proxy opens pre-warmed secure tunnels, routing requests instantly with zero TLS setup overhead.

3. **Swap to `orjson` for Chunk Parsing**
   - *Problem:* In an SSE stream, standard Python `json.loads` parses hundreds of delta chunks per second.
   - *The Fix:* Swap built-in `json` for `orjson` (written in Rust). It parses streaming LLM chunks up to 10x faster, dropping proxy overhead to near zero.

4. **Cythonize the Sliding-Window Buffer**
   - *Problem:* The sliding-window buffer performs frequent string slicing and bracket matching.
   - *The Fix:* Use Cython or `mypyc` to compile `streaming.py` directly into a C-extension binary module. Retains Python readability while executing string operations at native C speed.

---

## 🏢 Using LLM-Shield-Proxy in Production?

I am actively working with enterprise security teams to map out advanced compliance features. If your startup or organization is using LLM-Shield-Proxy to unblock LLM streaming or pass SOC 2/HIPAA audits, I would love to hear from you.

Email the core maintainer at ninadphalak@gmail.com to share your feedback, request a feature, or feature your team as a case study.
