Integrations — one proxy, every SDK

Start distil proxy once and point any SDK's baseURL at it. No library changes, no monkey-patching — compression happens at the network layer.

How it works

The proxy (distil proxy, default http://127.0.0.1:8788) is a local HTTP server. It intercepts the three compressible paths across all major LLM APIs:

• /v1/messages — Anthropic Messages API
• /v1/chat/completions — OpenAI Chat Completions (also used by LiteLLM, LangChain, etc.)
• /v1/responses — OpenAI Responses API
• /v1beta/models/{model}:generateContent and :streamGenerateContent — Google Gemini REST API (also the /v1/… host variants)

All other paths and HTTP verbs pass through unchanged. Your API key travels in the request headers exactly as normal — the proxy never logs or stores it.

SDK integration matrix

Agents distil wrap routes for you

No configuration and no code change: distil wrap -- <agent> starts a proxy, points the agent at it, and restores everything on exit. Most read an environment variable; a few have no such contract and route through a config file wrap manages for that one session. This table and the next are generated from distil/targets.py — distil wrap --list prints the same thing in your terminal.

Agents it cannot reach — and what was checked

Each of these was read against its own primary documentation on the date shown. Where a base-URL setting exists, point it at the URL distil proxy prints (http://127.0.0.1:8788/v1) and verify with distil doctor; where it says — there is no such setting to point. Full reasoning: IDE-AGENTS.md.

Snippets

Anthropic Python SDK

import anthropic

client = anthropic.Anthropic(
    api_key="sk-ant-…",
    base_url="http://127.0.0.1:8788",
)
response = client.messages.create(
    model="claude-opus-4-5",
    max_tokens=1024,
    messages=[{"role": "user", "content": "Hello!"}],
)

OpenAI Python SDK

import openai

client = openai.OpenAI(
    api_key="sk-…",
    base_url="http://127.0.0.1:8788/v1",
)
response = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": "Hello!"}],
)

LiteLLM

import litellm

response = litellm.completion(
    model="claude-opus-4-5",
    api_base="http://127.0.0.1:8788",
    api_key="sk-ant-…",
    messages=[{"role": "user", "content": "Hello!"}],
)

Running the standalone LiteLLM Proxy instead?

Same mechanism, one YAML field: point each model's api_base at distil proxy, and every request routed through the LiteLLM Proxy is compressed transparently.

# Terminal 1 — distil sits in front of the real upstream
distil proxy --port 8788 --upstream https://api.anthropic.com

# config.yaml — Terminal 2
model_list:
  - model_name: claude-opus-4-5
    litellm_params:
      model: anthropic/claude-opus-4-5
      api_base: http://127.0.0.1:8788
      api_key: os.environ/ANTHROPIC_API_KEY

# Terminal 2 — start the LiteLLM Proxy against that config
litellm --config config.yaml

Cursor

Settings → enable Override OpenAI Base URL, set it to the proxy, and add your model. The key field is labelled "OpenAI API Key" but is sent to whatever endpoint you configure.

CrewAI

CrewAI's LLM takes base_url directly. Point it at the proxy and prefix the model with its provider.

$ pip install crewai
$ distil proxy --port 8788 --upstream https://api.openai.com &

import os
from crewai import Agent, LLM

llm = LLM(
    model="openai/gpt-4o",               # provider prefix is required
    base_url="http://127.0.0.1:8788/v1",  # ← the only change; end at /v1
    api_key=os.environ["OPENAI_API_KEY"],
)
agent = Agent(llm=llm, ...)

Agno

Agno's OpenAILike is the class it recommends for any OpenAI-compatible endpoint — it takes the same parameters as OpenAIChat and relaxes the OpenAI-only assumptions. Point base_url at the proxy and every model call the agent makes is compressed on the way out.

$ pip install agno openai
$ distil proxy --port 8788 --upstream https://api.openai.com &

import os
from agno.agent import Agent
from agno.models.openai.like import OpenAILike

agent = Agent(
    model=OpenAILike(
        id="gpt-4o",
        api_key=os.environ["OPENAI_API_KEY"],
        base_url="http://127.0.0.1:8788/v1",   # ← the only change
    )
)
agent.print_response("...")

Strands Agents

Strands' OpenAIModel forwards client_args straight to the underlying openai client, so base_url goes there. Nothing else in the agent changes.

$ pip install strands-agents
$ distil proxy --port 8788 --upstream https://api.openai.com &

import os
from strands import Agent
from strands.models.openai import OpenAIModel

model = OpenAIModel(
    client_args={
        "api_key": os.environ["OPENAI_API_KEY"],
        "base_url": "http://127.0.0.1:8788/v1",  # ← the only change
    },
    model_id="gpt-4o",
)
agent = Agent(model=model)

Microsoft AutoGen

AutoGen's OpenAIChatCompletionClient takes base_url directly, same as the raw OpenAI client it wraps.

$ pip install "autogen-agentchat" "autogen-ext[openai]"
$ distil proxy --port 8788 --upstream https://api.openai.com &

import os
from autogen_agentchat.agents import AssistantAgent
from autogen_ext.models.openai import OpenAIChatCompletionClient

client = OpenAIChatCompletionClient(
    model="gpt-4o",
    api_key=os.environ["OPENAI_API_KEY"],
    base_url="http://127.0.0.1:8788/v1",  # ← the only change
)
agent = AssistantAgent("assistant", model_client=client)

Prefer no sidecar at all? distil.integrations.autogen wraps a ChatCompletionClient or a FunctionTool callable in-process — see AutoGen × Distil for both.

LlamaIndex

LlamaIndex's OpenAI/Anthropic LLM classes take api_base directly, same as the raw client each one wraps.

$ pip install llama-index
$ distil proxy --port 8788 --upstream https://api.openai.com &

from llama_index.llms.openai import OpenAI

llm = OpenAI(model="gpt-5", api_base="http://127.0.0.1:8788/v1")  # ← the only change
query_engine = index.as_query_engine(llm=llm)

Prefer no sidecar at all? distil.integrations.llamaindex gives you a node postprocessor for retrieved context, an LLM wrapper, and a FunctionTool callable wrapper, all in-process — see LlamaIndex × Distil for all three.

Vercel AI SDK (TypeScript)

import { createAnthropic } from "@ai-sdk/anthropic";
import { generateText } from "ai";

const anthropic = createAnthropic({
  baseURL: "http://127.0.0.1:8788",
  apiKey: process.env.ANTHROPIC_API_KEY,
});

const { text } = await generateText({
  model: anthropic("claude-opus-4-5"),
  prompt: "Hello!",
});

LangChain.js (TypeScript)

import { ChatAnthropic } from "@langchain/anthropic";

const model = new ChatAnthropic({
  model: "claude-opus-4-5",
  apiKey: process.env.ANTHROPIC_API_KEY,
  anthropicApiUrl: "http://127.0.0.1:8788",
  // older versions: clientOptions: { baseURL: "http://127.0.0.1:8788" }
});

const response = await model.invoke([
  ["human", "Hello!"],
]);

Google Gemini REST

# Start proxy pointing at the Gemini API
distil proxy --port 8788 --upstream https://generativelanguage.googleapis.com

import google.generativeai as genai

genai.configure(
    transport="rest",
    client_options={"api_endpoint": "http://127.0.0.1:8788"},
)
model = genai.GenerativeModel("gemini-1.5-pro")
response = model.generate_content("Hello!")

See examples/python_gemini.py for a full runnable example including tool-use turns. The proxy transparently compresses text parts (Tier-0 lossless) and functionResponse payloads (Tier-1 reversible digest). functionCall, inlineData, fileData, model-authored text, and systemInstruction are always passed through unchanged.

Claude Code plugin

The first-class integration: a session-first savings status line plus slash commands, shipped as a marketplace-format plugin (plugins/distil). Wire it in one step with distil setup (or distil onboard), then route the agent through compression with distil wrap -- claude.

One pattern in every state: distil · <live> · total ▼<lifetime>. The live segment aggregates recent activity (last 15 min, all terminals), so it never flickers between sessions:

▼ = tokens saved · total = lifetime · de = decision-equivalence (shown only past the reporting floor, 50 A/B + 30 A/A shadow samples — a rate over a handful is noise; ✓ at 99% and above, ⚠ under 99%, ✗ under 95%). Sharing the line with git/cwd/model? Set DISTIL_STATUSLINE=minimal for a two-fact segment: distil ▼75.0K · 27.0M total.

MCP server

A zero-dependency Model Context Protocol server (stdlib only — no SDK) exposes distil's reversible compression to any MCP client (Claude Desktop, IDEs, agents) over stdio:

claude mcp add distil -- distil mcp   # Claude Code — one line, done

Every other client takes the same stdio config — Claude Desktop (~/Library/Application Support/Claude/claude_desktop_config.json), Cursor (.cursor/mcp.json), VS Code (.vscode/mcp.json):

{
  "mcpServers": {
    "distil": { "command": "distil", "args": ["mcp"] }
  }
}

No distil install? Run it from PyPI on demand — "command": "uvx", "args": ["--from", "distil-llm", "distil", "mcp"]. Restart the client after editing. To check the server without a client at all:

echo '{"jsonrpc":"2.0","id":1,"method":"tools/list"}' | distil mcp

Each tool carries MCP annotations (readOnlyHint, idempotentHint, openWorldHint: false), so a client knows distil_expand is a safe, repeatable, offline read instead of inferring it from prose.

This is the recall path, not the savings path. The MCP server does not compress your agent's traffic — distil wrap does that transparently, with no tool calls. What the MCP server adds is the other half: any agent, including one you never wrapped, can call distil_expand on a handle it sees in context and get the original back. Handles survive restarts and cross processes, and age out after DISTIL_RESTORE_TTL_DAYS (default 14).

In-process hooks (LiteLLM · LangChain · LangGraph · AutoGen · LlamaIndex)

Prefer not to run a sidecar? Compress the request in-process — the same reversible compression, no proxy. Every helper lazy-imports (or duck-types) its framework, so distil stays zero-dep.

# LiteLLM — drop-in for litellm.completion
from distil.integrations import litellm as distil_litellm
resp = distil_litellm.completion(model="claude-opus-4-8", messages=[...],
                                 distil_verbatim=True)  # optional, Tier-0 only

# LangChain — compress a message list before the model call (duck-typed)
from distil.integrations.langchain import compress_messages
msgs = compress_messages(state["messages"], verbatim=True)

# LangGraph — compress graph state right before the model node
from distil.integrations.langgraph import pre_model_hook
agent = create_react_agent(model, tools, pre_model_hook=pre_model_hook())

# AutoGen — compress a ChatCompletionClient's outgoing messages, or one tool's output
from distil.integrations.autogen import DistilModelClient, compressing_tool
client = DistilModelClient(OpenAIChatCompletionClient(model="gpt-4o"))
tool = FunctionTool(compressing_tool(get_weather), description="...")

# LlamaIndex — compress retrieved nodes, an LLM's outgoing calls, or one tool's output
from distil.integrations.llamaindex import DistilNodePostprocessor, DistilLLM
query_engine = index.as_query_engine(
    llm=DistilLLM(OpenAI(model="gpt-5")),
    node_postprocessors=[DistilNodePostprocessor()],
)

Tool/function messages and retrieved nodes get the reversible Tier-1 digest; human/system messages get Tier-0 lossless; the model's own words are never rewritten. compress() (LiteLLM), compress_messages() (LangChain), pre_model_hook() (LangGraph), DistilModelClient/compressing_tool() (AutoGen), and DistilLLM/DistilNodePostprocessor (LlamaIndex) are framework-free and unit-tested. The LangGraph hook returns only the updated message list, so every other state field is left intact; DistilModelClient delegates every attribute it doesn't wrap straight to the real client, and DistilLLM re-types the LLM you pass as a transparent subclass of its own class, so it still satisfies the isinstance checks LlamaIndex performs on an llm= argument.

LangChain / LangGraph as a package — langchain-distil

The two hooks above are also published as a standalone package, listed in LangChain’s own community middleware integrations. It is a thin wrapper over exactly those hooks — same certified compression path, nothing re-implemented — and it pulls distil-llm in as a dependency.

$ pip install langchain-distil

from langchain_distil import compress_messages, pre_model_hook, as_runnable

msgs  = compress_messages(msgs)                                   # a message list
graph = create_react_agent(model, tools, pre_model_hook=pre_model_hook())  # LangGraph state
chain = as_runnable() | llm                                       # or a chain step

as_runnable() imports langchain-core lazily, so importing the package never requires it. See LangChain × Distil or LangGraph × Distil for the full page.

ASGI middleware — when you host the endpoint

Every hook above sits in a client that calls out to a provider. If instead your own backend is the thing building the provider request — a FastAPI/Starlette/Litestar app that forwards to Anthropic/OpenAI/Gemini itself — wrap it once with DistilMiddleware, pure ASGI, no Starlette import:

from distil.integrations.asgi import DistilMiddleware

app = DistilMiddleware(app)  # wraps any ASGI 3 app; compresses matching POST bodies

It reuses the exact path detection and reversible compression distil proxy uses, so a handle minted here expands anywhere. See ASGI Middleware × Distil for the full page.

Observability headers

The proxy adds up to 9 response headers per compressed response. The first two appear on every compressed request; the rest are conditional:

The managed gateway (distil gateway) additionally adds x-distil-tenant: <id> for per-tenant accounting.

Node.js / TypeScript

Distil's universal path is the proxy — it's language-agnostic, so JS/TS stacks need no Distil-specific package. Start the proxy (or wrap your agent) and point your SDK's baseURL at it:

# start the proxy (needs Python 3.9+ — see Install)
distil proxy --port 8788 --upstream https://api.anthropic.com

// then in your Node/TS app — no code change beyond baseURL
const client = new Anthropic({ baseURL: "http://localhost:8788" });

Or wrap an existing CLI/agent in one shot: distil wrap -- <your-command> starts the proxy and injects ANTHROPIC_BASE_URL automatically.

Homebrew

brew tap dshakes/tap
brew install dshakes/tap/distil
distil proxy --port 8788