LiteLLM × Distil

LiteLLM takes api_base. The same proxy also works behind the standalone LiteLLM Proxy — one YAML field per model.

Setup

Start the proxy against the provider this SDK talks to, then point the client at it. Nothing else in your code changes.

$ distil proxy --port 8788 --upstream https://api.openai.com &

import litellm

resp = litellm.completion(
    model="gpt-4o",
    api_base="http://127.0.0.1:8788/v1",   # ← the only change
    messages=[{"role": "user", "content": "..."}],
)

Running the standalone LiteLLM Proxy instead? Set each model's api_base to the distil proxy in your config YAML; every request routed through LiteLLM is then compressed transparently.

In-process: the LiteLLM Proxy hook

No sidecar needed on the standalone LiteLLM Proxy. Install the extra and register distil's async_pre_call_hook, which compresses each request just before LiteLLM routes it:

$ pip install 'distil-llm[litellm]'

# config.yaml
litellm_settings:
  callbacks: distil.integrations.litellm_hook.proxy_handler_instance

Or choose options in your own callback file: proxy_handler_instance = DistilCompressionHook(digest=False).

Lossless-only by default. Only the in-context-lossless Tier-0 transforms run. Set digest=True (or DISTIL_LITELLM_DIGEST=1 with the YAML form) to opt into Tier-1 digests. A LiteLLM hook cannot inject the distil_expand tool, so digest stubs sent from here have no in-conversation recovery; use the sidecar proxy above if you want digest mode.
Fail-open. Any error, unknown call type or unrecognized shape sends the original request unchanged.
Content-free. Message text is never logged; token counts go to the normal savings ledger (distil stats, distil dashboard), tagged verbatim or digest.
Coverage. completion / acompletion (OpenAI Chat shape, used for every provider) and anthropic_messages. Responses-API and embedding calls pass through. The hook is proxy-only: LiteLLM's SDK litellm.callbacks does not run async_pre_call_hook, so for SDK use see distil.integrations.litellm.completion.

What actually happens

The proxy intercepts only the compressible paths — /v1/messages, /v1/chat/completions, /v1/responses and the Gemini generateContent routes. Everything else passes through untouched, and your API key travels in the request headers exactly as normal: the proxy never logs or stores it.

Large tool results are replaced by reversible digests carrying a content handle, and the original stays on your machine. The agent can pull any of it back mid-task through the distil_expand tool, so nothing is permanently discarded — which is what lets the compression be aggressive without being a gamble.

Check it before you trust it. distil simulate -m request.json replays one of your real requests through the pipeline with no model in the path and reports what would be compressed, what would be left byte-exact, and which rule protected it. See the CLI reference.

Verify it is actually routing

The most common failure is silent: the client never reaches the proxy and everything still works, just uncompressed. Two checks:

$ curl -s localhost:8788/distil/health
{"status":"ok"}

$ distil dashboard          # live savings; zero here means nothing is routing

Every response also carries x-distil-tokens-saved, so a request that went through the proxy is identifiable from its headers alone.

Full matrix of every supported SDK, including the in-process hooks that need no proxy: Integrations. Distil is a proxy, so anything that lets you set a base URL works — this page is a worked example, not the boundary of support.