# Deploy & Security

Three deployment topologies, one security model. The proxy is designed to be safe by default — it binds localhost, forwards only to the single configured upstream, and never touches your secrets.

## Deployment topologies

Topology 1

#### Local dev sidecar

Run `distil proxy` on your laptop alongside your application. Default bind is `127.0.0.1:8788` — not reachable from the network. Zero infra overhead.

Topology 2

#### Container sidecar

Add the proxy as a sidecar container in the same Pod or Compose stack. Your app container calls `http://localhost:8788`; only the sidecar egresses to the upstream API.

Topology 3

#### Shared gateway

Run one proxy instance accessible to multiple services on a private network. Bind to an internal interface (`--host 0.0.0.0`), keep it behind a firewall or service mesh, never expose it to the public internet.

### Local dev sidecar

    # Terminal 1
    distil proxy --port 8788 --upstream https://api.anthropic.com

    # Terminal 2 — your application
    ANTHROPIC_BASE_URL=http://127.0.0.1:8788 python my_agent.py

### Container sidecar (Docker Compose)

    services:
      app:
        build: .
        environment:
          - ANTHROPIC_BASE_URL=http://distil-proxy:8788
        depends_on:
          - distil-proxy

      distil-proxy:
        image: python:3.12-slim
        command: >
          sh -c "pip install distil-llm &&
                 distil proxy --host 0.0.0.0 --port 8788
                   --upstream https://api.anthropic.com"
        ports: []  # not exposed externally

### Shared gateway (Kubernetes)

    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: distil-proxy
    spec:
      replicas: 2
      template:
        spec:
          containers:
          - name: distil
            image: python:3.12-slim
            command:
              - sh
              - -c
              - pip install distil-llm && distil proxy --host 0.0.0.0 --port 8788
            env:
              - name: DISTIL_UPSTREAM
                value: https://api.anthropic.com
    ---
    apiVersion: v1
    kind: Service
    metadata:
      name: distil-proxy
    spec:
      clusterIP: ...  # internal only — no LoadBalancer type
      ports:
      - port: 8788

## Security model

### Localhost-only by default

The proxy binds to `127.0.0.1` by default. It is not reachable from any other host unless you explicitly pass `--host 0.0.0.0`. In production topologies, always restrict network access to the proxy with a firewall rule or service mesh policy.

### Single configured upstream — no SSRF

The upstream URL is set once at startup (`--upstream`). The proxy does not read the destination from the incoming request, so a malicious request body cannot redirect traffic to an attacker-controlled server. All outbound connections go to exactly one destination.

No SSRF surface. The upstream is a constructor argument baked into the handler class at startup, not read from request headers or the request body. There is no mechanism for an incoming request to change where the proxy connects.

### No request body logging

The proxy overrides `log_message` to suppress all request logs. It reads the request body to compress it, then discards it — nothing is written to disk or stdout. API keys, message content, and tool results are never logged.

    # In distil/proxy.py — request logs are silenced by design
    def log_message(self, fmt: str, *args: object) -> None:
        pass

### Auth-mode gating — prompt-cache stability and subscription policy

The proxy compresses the `messages` array only — never the `system` prefix — so Anthropic prompt-cache prefix stability is preserved in all modes. Two flags give additional control:

- `--lossless-only` — policy mode: restricts the proxy to lossless strategies (no lossy output-shaping, no tool injection via `--expand`). The reversible Tier-1 digest still runs. Use for subscription or OAuth-gated deployments.
- `--verbatim` — skips the Tier-1 digest entirely, so the model sees message content byte-for-byte (Tier-0 only: JSON minification, run-length collapse). Use for interactive sessions or anywhere `distil_expand` recovery is unavailable. Lower savings.

    distil proxy --lossless-only --upstream https://api.anthropic.com
    distil proxy --verbatim     --upstream https://api.anthropic.com

### Hop-by-hop header stripping

The proxy strips connection-specific headers (`Connection`, `Transfer-Encoding`, `Keep-Alive`, etc.) before forwarding. This prevents header injection attacks and ensures clean HTTP/1.1 semantics at the upstream boundary.

## Data retention and zero-data retention (ZDR)

Distil itself retains no data. The proxy is stateless: it reads a request, compresses the `messages` array in memory, forwards it, and relays the response. Nothing is written to disk.

If your Anthropic contract includes Zero Data Retention (ZDR), the compressed payload forwarded by the proxy is covered by the same ZDR terms as a direct call — Distil does not introduce any additional retention surface.

RestoreStore is in-process only. The Tier-1 reversible digest stores original text in a Python dict for the duration of a single request. It is never persisted, never sent to the model, and is garbage-collected when the request handler exits.

## Managed gateway

Module: `distil/gateway.py` · Command: `distil gateway`

The managed gateway extends the basic proxy with per-tenant token and dollar accounting, a live auto-refreshing dashboard, and a JSON stats API. It is the right deployment target when multiple teams or applications share one distil instance and you want visibility into who is saving what.

### Starting the gateway

    # Default: binds 127.0.0.1:8789 and forwards to Anthropic
    $ distil gateway --port 8789 --upstream https://api.anthropic.com
    distil gateway listening on http://127.0.0.1:8789
      dashboard: http://127.0.0.1:8789/distil/dashboard
      → upstream: https://api.anthropic.com

    # Full flag set
    $ distil gateway \
        --host 0.0.0.0 \
        --port 8789 \
        --upstream https://api.anthropic.com \
        --pricing claude-opus-4-8 \
        --lossless-only

### Flags

Flag | Default | Description
--- | --- | ---
`--host` | `127.0.0.1` | Interface to bind on. Use `0.0.0.0` for network-accessible deployments (keep behind a firewall).
`--port` | `8789` | Port to listen on (distinct from the basic proxy default of 8788).
`--upstream` | `https://api.anthropic.com` | Real LLM API base URL. Set once at startup — cannot be overridden per-request.
`--pricing` | `claude-opus-4-8` | Model pricing key from the catalog; used for per-tenant dollar accounting in the dashboard.
`--lossless-only` | off | Policy mode: lossless strategies only — no lossy output-shaping, no tool injection (`--expand`). The reversible Tier-1 digest still runs. Use for subscription / OAuth sessions.
`--verbatim` | off | Skip the Tier-1 digest; Tier-0 only. Message content is byte-stable. Interactive-safe, lower savings.

### Per-tenant accounting

Every compressible request (`POST /v1/messages`, `/v1/chat/completions`, `/v1/responses`) is attributed to a tenant and its token counts are accumulated in-memory. Tenant identification priority:

1. `x-distil-tenant` header — explicit label set by the caller (e.g. `"team-search"`, `"team-checkout"`).
2. Anonymized key hash — if no explicit label, the gateway derives a stable 8-hex ID from `sha256(x-api-key or authorization)`, prefixed `anon-`. The raw key is never stored or logged — only the truncated hash.
3. `"default"` — used when no credential is present at all.

No raw key ever stored or logged. The `tenant_of()` function hashes the credential with SHA-256 and keeps only the first 8 hex characters. API keys cannot be reconstructed from the recorded tenant IDs.

### Management endpoints

JSON API

### `GET /distil/stats`

Returns a JSON snapshot of per-tenant and aggregate token/dollar savings. Suitable for polling from CI, alerting systems, or a cost dashboard.

Live dashboard

### `GET /distil/dashboard`

Self-contained dark HTML page that auto-refreshes every 5 seconds. Served inline — no CDN or external assets. Shows per-tenant requests, tokens saved, dollars saved, and compression rate.

    $ curl -s http://127.0.0.1:8789/distil/stats | python3 -m json.tool
    {
      "tenants": [
        {
          "tenant": "team-search",
          "requests": 142,
          "tokens_baseline": 184320,
          "tokens_compressed": 131904,
          "tokens_saved": 52416,
          "dollars_saved": 0.026208,
          "pct_saved": 28.44
        },
        {
          "tenant": "anon-3f9a1c2b",
          "requests": 31,
          "tokens_baseline": 40110,
          "tokens_compressed": 29110,
          "tokens_saved": 11000,
          "dollars_saved": 0.005500,
          "pct_saved": 27.42
        }
      ],
      "totals": {
        "requests": 173,
        "tokens_baseline": 224430,
        "tokens_compressed": 161014,
        "tokens_saved": 63416,
        "dollars_saved": 0.031708,
        "pct_saved": 28.26
      }
    }

All other paths pass through unchanged. The gateway adds management endpoints at `/distil/*` and compresses `POST /v1/messages`-family paths. Every other path and HTTP method is forwarded to the upstream transparently, with the same hop-by-hop header stripping and SSRF protection as the basic proxy.

## Async high-concurrency proxy

Module: `distil/aproxy.py` · Flag: `distil proxy --async` · Extra: `pip install 'distil-llm[async]'`

The threaded proxy (`distil/proxy.py`) handles the common case. Under high fan-out — thousands of concurrent streaming sessions — the GIL becomes a bottleneck. The async proxy replaces the threaded HTTP server with an aiohttp event loop and a single shared `ClientSession` with an unlimited connector, sustaining far more concurrent streams at the same compression behavior.

Same compression, async I/O. `distil/aproxy.py` calls the same `compress_messages()` path as the threaded proxy. Every Tier-0/1 transform, the hop-by-hop header stripping, the SSRF protection, and the `x-distil-compressed` / `x-distil-tokens-saved` response headers are identical. The only difference is the concurrency model.

### Starting the async proxy

    # Install the async extra first
    $ pip install 'distil-llm[async]'

    # Start the async proxy (same flags as the threaded proxy)
    $ distil proxy --async --port 8788 --upstream https://api.anthropic.com
    distil async proxy listening on http://127.0.0.1:8788
      → upstream: https://api.anthropic.com

All other `distil proxy` flags (`--host`, `--upstream`, `--lossless-only`, `--verbatim`, `--shape-output`) work identically with `--async`. Client configuration is unchanged — point your SDK at the same `base_url`.

### Architecture

`make_app()` in `distil/aproxy.py` builds an `aiohttp.web.Application` with a single shared `ClientSession` (created in `on_startup`, closed in `on_cleanup`) using `TCPConnector(limit=0)` — unlimited concurrent connections. Compressible paths (`/v1/messages`, `/v1/chat/completions`, `/v1/responses`) go through the compression handler; everything else is transparent passthrough. aiohttp is lazy-imported so the stdlib core of distil remains dependency-free when the async extra is not installed.

## Native Rust core

Crate: `rust/distil-core` · Module: `distil/native.py` · Build: `maturin develop --release`

The hot-path Tier-0 transforms and heuristic tokenizer are implemented as a PyO3 Rust crate. When the wheel is built and installed, `distil.native` transparently uses the fast native implementations and sets `BACKEND = "rust"`. When the crate is absent (source-only install, CI without Rust), `distil.native` falls back to the pure-Python implementations from `distil.compress.tier0` and `distil.tokenizer` with no import error — `BACKEND = "python"`. All callers get the same API either way.

### Functions

Python name | Description | Rust signature
--- | --- | ---
`minify_json(text)` | Parse + re-emit JSON without whitespace; returns `None` for non-JSON | `minify_json(&str) -> Option<String>`
`collapse_runs(text)` | RLE-encode consecutive identical lines | `collapse_runs(&str) -> String`
`count_tokens(text, subword_factor=1.33)` | Heuristic token count (`\w+|[^\w\s]` × factor) | `count_tokens(&str, f64) -> usize`

2.5 µs
collapse_runs — 30 KB input
20.2 µs
count_tokens — 30 KB input
26
Cargo unit tests passing
8
Python parity tests passing

Benchmarks measured on Apple M-series (aarch64-apple-darwin, `--release`), 100 samples each, on a ~30 KB synthetic log with repeated lines, prose, and JSON snippets. `collapse_runs` achieves ~12 GB/s throughput. `count_tokens` is dominated by Unicode character-class scanning.

### Build and install

    # One-time: install maturin
    $ uv tool install maturin

    # Build and install the Rust extension into the project virtualenv
    $ cd rust/distil-core
    $ maturin develop --release

    # Verify the Rust backend loaded
    $ python3 -c "import distil.native as n; print(n.BACKEND)"
    rust

Run the Rust unit tests with `cargo test` from `rust/distil-core` (26 passed; 0 failed). Run the Python parity tests with `uv run python -m pytest tests/test_native.py -v` (8 passed). Both test suites pass in pure-Python fallback mode too.

Pure-Python parity guaranteed. `distil/native.py` wraps both backends behind a single API. If the Rust wheel is not installed, every function falls back to the pure-Python equivalent — same semantics, no import error. The core always works; it is transparently accelerated when built with maturin.

## Threat model summary

Threat | Mitigation
--- | ---
Network exposure of the proxy | Binds `127.0.0.1` by default; explicit `--host` required to listen externally.
SSRF via request body | Upstream is set once at startup; cannot be overridden by request content.
Secret exfiltration via logs | Request logging is disabled; no body content is written anywhere.
Cache prefix corruption | The proxy never compresses the `system` prefix, so prompt-cache stability holds in all modes. `--verbatim` additionally makes message content byte-stable (Tier-0 only). `--lossless-only` is the policy mode (no tool injection, digest still runs).
Header injection | Hop-by-hop headers are stripped before forwarding in both directions.
Data retention beyond request | Stateless handler; RestoreStore is in-process, not persisted.
API key exposure in gateway logs | Gateway derives tenant ID via `sha256(key)[:8]`; raw key never stored, never logged, not reconstructable.