LiteLLM × Distil

LiteLLM takes api_base. The same proxy also works behind the standalone LiteLLM Proxy — one YAML field per model.

Setup

Start the proxy against the provider this SDK talks to, then point the client at it. Nothing else in your code changes.

$ distil proxy --port 8788 --upstream https://api.openai.com &

import litellm

resp = litellm.completion(
    model="gpt-4o",
    api_base="http://127.0.0.1:8788/v1",   # ← the only change
    messages=[{"role": "user", "content": "..."}],
)

Running the standalone LiteLLM Proxy instead? Set each model's api_base to the distil proxy in your config YAML; every request routed through LiteLLM is then compressed transparently.

What actually happens

The proxy intercepts only the compressible paths — /v1/messages, /v1/chat/completions, /v1/responses and the Gemini generateContent routes. Everything else passes through untouched, and your API key travels in the request headers exactly as normal: the proxy never logs or stores it.

Large tool results are replaced by reversible digests carrying a content handle, and the original stays on your machine. The agent can pull any of it back mid-task through the distil_expand tool, so nothing is permanently discarded — which is what lets the compression be aggressive without being a gamble.

Check it before you trust it. distil simulate -m request.json replays one of your real requests through the pipeline with no model in the path and reports what would be compressed, what would be left byte-exact, and which rule protected it. See the CLI reference.

Verify it is actually routing

The most common failure is silent: the client never reaches the proxy and everything still works, just uncompressed. Two checks:

$ curl -s localhost:8788/distil/health
{"status":"ok"}

$ distil dashboard          # live savings; zero here means nothing is routing

Every response also carries x-distil-tokens-saved, so a request that went through the proxy is identifiable from its headers alone.

Full matrix of every supported SDK, including the in-process hooks that need no proxy: Integrations. Distil is a proxy, so anything that lets you set a base URL works — this page is a worked example, not the boundary of support.