LangChain × Distil

Two routes. The middleware package compresses the message list in your own process — no proxy, no network hop. The proxy needs no code change at all beyond a base URL. Pick one; they are not meant to be stacked.

Middleware — langchain-distil

Listed in LangChain’s own community middleware integrations. If you arrived from there, this is the package.

$ pip install langchain-distil

from langchain_distil import compress_messages, pre_model_hook, as_runnable

# 1. compress a message list directly
msgs = compress_messages(msgs)

# 2. LangGraph: compress graph state before the model node
graph = create_react_agent(model, tools, pre_model_hook=pre_model_hook())

# 3. or drop it into a chain (langchain-core is imported lazily)
chain = as_runnable() | llm

Tool and function messages get the reversible Tier-1 digest, human and system messages are Tier-0 lossless, and assistant messages are never rewritten — a model’s own words are not distil’s to edit. Every digest is byte-exact recoverable through distil_expand. Pass verbatim=True for Tier-0 only, which is the right choice when no recovery tool is wired up.

It is a thin wrapper over distil.integrations.langchain and distil.integrations.langgraph, so it inherits the same certified compression path rather than re-implementing it. distil-llm comes along as a dependency — you do not install both by hand.

Setup — the proxy route

Start the proxy against the provider this SDK talks to, then point the client at it. Nothing else in your code changes.

$ distil proxy --port 8788 --upstream https://api.anthropic.com &

from langchain_anthropic import ChatAnthropic

llm = ChatAnthropic(
    model="claude-opus-4-8",
    base_url="http://127.0.0.1:8788",     # ← the only change
)

LangGraph users can compress graph state directly instead, with distil.integrations.langgraph.pre_model_hook() — no proxy required.

What actually happens

The proxy intercepts only the compressible paths — /v1/messages, /v1/chat/completions, /v1/responses and the Gemini generateContent routes. Everything else passes through untouched, and your API key travels in the request headers exactly as normal: the proxy never logs or stores it.

Large tool results are replaced by reversible digests carrying a content handle, and the original stays on your machine. The agent can pull any of it back mid-task through the distil_expand tool, so nothing is permanently discarded — which is what lets the compression be aggressive without being a gamble.

Verify it is actually routing

The most common failure is silent: the client never reaches the proxy and everything still works, just uncompressed. Two checks:

$ curl -s localhost:8788/distil/health
{"status":"ok"}

$ distil dashboard          # live savings; zero here means nothing is routing

Every response also carries x-distil-tokens-saved, so a request that went through the proxy is identifiable from its headers alone.

Full matrix of every supported SDK, including the in-process hooks that need no proxy: Integrations. Distil is a proxy, so anything that lets you set a base URL works — this page is a worked example, not the boundary of support.