Deploy & Security

Three deployment topologies, one security model. The proxy is designed to be safe by default — it binds localhost, forwards only to the single configured upstream, and never touches your secrets.

Deployment topologies

Local dev sidecar

# Terminal 1
distil proxy --port 8788 --upstream https://api.anthropic.com

# Terminal 2 — your application
ANTHROPIC_BASE_URL=http://127.0.0.1:8788 python my_agent.py

Container sidecar (Docker Compose)

services:
  app:
    build: .
    environment:
      - ANTHROPIC_BASE_URL=http://distil-proxy:8788
    depends_on:
      - distil-proxy

  distil-proxy:
    image: python:3.12-slim
    command: >
      sh -c "pip install distil-llm &&
             distil proxy --host 0.0.0.0 --port 8788
               --upstream https://api.anthropic.com"
    ports: []  # not exposed externally

Shared gateway (Kubernetes)

apiVersion: apps/v1
kind: Deployment
metadata:
  name: distil-proxy
spec:
  replicas: 2
  template:
    spec:
      containers:
      - name: distil
        image: python:3.12-slim
        command:
          - sh
          - -c
          - pip install distil-llm && distil proxy --host 0.0.0.0 --port 8788
        env:
          - name: DISTIL_UPSTREAM
            value: https://api.anthropic.com
---
apiVersion: v1
kind: Service
metadata:
  name: distil-proxy
spec:
  clusterIP: ...  # internal only — no LoadBalancer type
  ports:
  - port: 8788

Security model

Localhost-only by default

The proxy binds to 127.0.0.1 by default. It is not reachable from any other host unless you explicitly pass --host 0.0.0.0. In production topologies, always restrict network access to the proxy with a firewall rule or service mesh policy.

Single configured upstream — no SSRF

The upstream URL is set once at startup (--upstream). The proxy does not read the destination from the incoming request, so a malicious request body cannot redirect traffic to an attacker-controlled server. All outbound connections go to exactly one destination.

No request body logging

The proxy overrides log_message to suppress all request logs. It reads the request body to compress it, then discards it — nothing is written to disk or stdout. API keys, message content, and tool results are never logged.

# In distil/proxy.py — request logs are silenced by design
def log_message(self, fmt: str, *args: object) -> None:
    pass

Auth-mode gating — prompt-cache stability and subscription policy

The proxy compresses the messages array only — never the system prefix — so Anthropic prompt-cache prefix stability is preserved in all modes. Two flags give additional control:

• --lossless-only — policy mode: restricts the proxy to lossless strategies (no lossy output-shaping, no tool injection via --expand). The reversible Tier-1 digest still runs. Use for subscription or OAuth-gated deployments.
• --verbatim — skips the Tier-1 digest entirely, so the model sees message content byte-for-byte (Tier-0 only: JSON minification, run-length collapse). Use for interactive sessions or anywhere distil_expand recovery is unavailable. Lower savings.

distil proxy --lossless-only --upstream https://api.anthropic.com
distil proxy --verbatim     --upstream https://api.anthropic.com

Hop-by-hop header stripping

The proxy strips connection-specific headers (Connection, Transfer-Encoding, Keep-Alive, etc.) before forwarding. This prevents header injection attacks and ensures clean HTTP/1.1 semantics at the upstream boundary.

Data retention and zero-data retention (ZDR)

Distil itself retains no data. The proxy is stateless: it reads a request, compresses the messages array in memory, forwards it, and relays the response. Nothing is written to disk.

If your Anthropic contract includes Zero Data Retention (ZDR), the compressed payload forwarded by the proxy is covered by the same ZDR terms as a direct call — Distil does not introduce any additional retention surface.

Managed gateway

Module: distil/gateway.py · Command: distil gateway

The managed gateway extends the basic proxy with per-tenant token and dollar accounting, a live auto-refreshing dashboard, and a JSON stats API. It is the right deployment target when multiple teams or applications share one distil instance and you want visibility into who is saving what.

Starting the gateway

copy# Default: binds 127.0.0.1:8789 and forwards to Anthropic
$ distil gateway --port 8789 --upstream https://api.anthropic.com
distil gateway listening on http://127.0.0.1:8789
  dashboard: http://127.0.0.1:8789/distil/dashboard
  → upstream: https://api.anthropic.com

# Full flag set
$ distil gateway \
    --host 0.0.0.0 \
    --port 8789 \
    --upstream https://api.anthropic.com \
    --pricing claude-opus-4-8 \
    --lossless-only

Flags

Per-tenant accounting

Every compressible request (POST /v1/messages, /v1/chat/completions, /v1/responses) is attributed to a tenant and its token counts are accumulated in-memory. Tenant identification priority:

1. x-distil-tenant header — explicit label set by the caller (e.g. "team-search", "team-checkout").
2. Anonymized key hash — if no explicit label, the gateway derives a stable 8-hex ID from sha256(x-api-key or authorization), prefixed anon-. The raw key is never stored or logged — only the truncated hash.
3. "default" — used when no credential is present at all.

Management endpoints

$ curl -s http://127.0.0.1:8789/distil/stats | python3 -m json.tool
{
  "tenants": [
    {
      "tenant": "team-search",
      "requests": 142,
      "tokens_baseline": 184320,
      "tokens_compressed": 131904,
      "tokens_saved": 52416,
      "dollars_saved": 0.026208,
      "pct_saved": 28.44
    },
    {
      "tenant": "anon-3f9a1c2b",
      "requests": 31,
      "tokens_baseline": 40110,
      "tokens_compressed": 29110,
      "tokens_saved": 11000,
      "dollars_saved": 0.005500,
      "pct_saved": 27.42
    }
  ],
  "totals": {
    "requests": 173,
    "tokens_baseline": 224430,
    "tokens_compressed": 161014,
    "tokens_saved": 63416,
    "dollars_saved": 0.031708,
    "pct_saved": 28.26
  }
}

Async high-concurrency proxy

Module: distil/aproxy.py · Flag: distil proxy --async · Extra: pip install 'distil-llm[async]'

The threaded proxy (distil/proxy.py) handles the common case. Under high fan-out — thousands of concurrent streaming sessions — the GIL becomes a bottleneck. The async proxy replaces the threaded HTTP server with an aiohttp event loop and a single shared ClientSession with an unlimited connector, sustaining far more concurrent streams at the same compression behavior.

Starting the async proxy

copy# Install the async extra first
$ pip install 'distil-llm[async]'

# Start the async proxy (same flags as the threaded proxy)
$ distil proxy --async --port 8788 --upstream https://api.anthropic.com
distil async proxy listening on http://127.0.0.1:8788
  → upstream: https://api.anthropic.com

All other distil proxy flags (--host, --upstream, --lossless-only, --verbatim, --shape-output) work identically with --async. Client configuration is unchanged — point your SDK at the same base_url.

Architecture

make_app() in distil/aproxy.py builds an aiohttp.web.Application with a single shared ClientSession (created in on_startup, closed in on_cleanup) using TCPConnector(limit=0) — unlimited concurrent connections. Compressible paths (/v1/messages, /v1/chat/completions, /v1/responses) go through the compression handler; everything else is transparent passthrough. aiohttp is lazy-imported so the stdlib core of distil remains dependency-free when the async extra is not installed.

Native Rust core

Crate: rust/distil-core · Module: distil/native.py · Build: maturin develop --release

The hot-path Tier-0 transforms and heuristic tokenizer are implemented as a PyO3 Rust crate. When the wheel is built and installed, distil.native transparently uses the fast native implementations and sets BACKEND = "rust". When the crate is absent (source-only install, CI without Rust), distil.native falls back to the pure-Python implementations from distil.compress.tier0 and distil.tokenizer with no import error — BACKEND = "python". All callers get the same API either way.

Functions

Benchmarks measured on Apple M-series (aarch64-apple-darwin, --release), 100 samples each, on a ~30 KB synthetic log with repeated lines, prose, and JSON snippets. collapse_runs achieves ~12 GB/s throughput. count_tokens is dominated by Unicode character-class scanning.

Build and install

copy# One-time: install maturin
$ uv tool install maturin

# Build and install the Rust extension into the project virtualenv
$ cd rust/distil-core
$ maturin develop --release

# Verify the Rust backend loaded
$ python3 -c "import distil.native as n; print(n.BACKEND)"
rust

Run the Rust unit tests with cargo test from rust/distil-core (26 passed; 0 failed). Run the Python parity tests with uv run python -m pytest tests/test_native.py -v (8 passed). Both test suites pass in pure-Python fallback mode too.

Threat model summary