Metadata-Version: 2.4
Name: hanzo-llm
Version: 1.82.2
Summary: Library to easily interface with LLM API providers
License: MIT
License-File: LICENSE
License-File: NOTICE
Author: Hanzo AI
Author-email: dev@hanzo.ai
Requires-Python: >=3.10,<4.0
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Provides-Extra: caching
Provides-Extra: extra-proxy
Provides-Extra: google
Provides-Extra: grpc
Provides-Extra: mlflow
Provides-Extra: proxy
Provides-Extra: semantic-router
Provides-Extra: utils
Requires-Dist: PyJWT (>=2.12.1,<3.0.0) ; extra == "proxy"
Requires-Dist: a2a-sdk (>=0.3.25,<0.4.0) ; (python_version >= "3.10") and (extra == "extra-proxy")
Requires-Dist: aiohttp (>=3.13.3)
Requires-Dist: apscheduler (>=3.11.2,<4.0.0) ; extra == "proxy"
Requires-Dist: azure-identity (>=1.25.3,<2.0.0) ; extra == "proxy" or extra == "extra-proxy"
Requires-Dist: azure-keyvault-secrets (>=4.10.0,<5.0.0) ; extra == "extra-proxy"
Requires-Dist: azure-storage-blob (>=12.28.0,<13.0.0) ; extra == "proxy"
Requires-Dist: backoff (>=2.2.1) ; extra == "proxy"
Requires-Dist: boto3 (>=1.42.73) ; extra == "proxy"
Requires-Dist: click (>=8.3.1)
Requires-Dist: cryptography (>=46.0.5) ; extra == "proxy"
Requires-Dist: diskcache (>=5.6.3,<6.0.0) ; extra == "caching"
Requires-Dist: fastapi (>=0.135.1) ; extra == "proxy"
Requires-Dist: fastapi-sso (>=0.21.0,<0.22.0) ; extra == "proxy"
Requires-Dist: fastuuid (>=0.14.0)
Requires-Dist: google-cloud-aiplatform (>=1.142.0) ; extra == "google"
Requires-Dist: google-cloud-iam (>=2.21.0,<3.0.0) ; extra == "extra-proxy"
Requires-Dist: google-cloud-kms (>=3.11.0,<4.0.0) ; extra == "extra-proxy"
Requires-Dist: grpcio (>=1.62.3,!=1.68.*,!=1.69.*,!=1.70.*,!=1.71.0,!=1.71.1,!=1.72.0,!=1.72.1,!=1.73.0) ; (python_version < "3.14") and (extra == "grpc")
Requires-Dist: grpcio (>=1.75.0) ; (python_version >= "3.14") and (extra == "grpc")
Requires-Dist: gunicorn (>=25.1.0,<26.0.0) ; extra == "proxy"
Requires-Dist: httpx (>=0.28.1)
Requires-Dist: importlib-metadata (>=6.8.0)
Requires-Dist: jinja2 (>=3.1.6,<4.0.0)
Requires-Dist: jsonschema (>=4.26.0,<5.0.0)
Requires-Dist: llm-proxy-extras (>=0.4.50) ; extra == "proxy"
Requires-Dist: mcp (>=1.26.0,<2.0.0) ; (python_version >= "3.10") and (extra == "proxy")
Requires-Dist: mlflow (>=3.10.1) ; (python_version >= "3.10") and (extra == "mlflow")
Requires-Dist: numpydoc ; extra == "utils"
Requires-Dist: openai (>=2.29.0)
Requires-Dist: orjson (>=3.11.7,<4.0.0) ; extra == "proxy"
Requires-Dist: polars (>=1.39.3,<2.0.0) ; (python_version >= "3.10") and (extra == "proxy")
Requires-Dist: prisma (==0.15.0) ; extra == "extra-proxy"
Requires-Dist: pydantic (>=2.12.0,<3.0.0)
Requires-Dist: pynacl (>=1.6.2,<2.0.0) ; extra == "proxy"
Requires-Dist: pyroscope-io (>=0.8,<0.9) ; (sys_platform != "win32") and (extra == "proxy")
Requires-Dist: python-dotenv (>=1.2.2)
Requires-Dist: python-multipart (>=0.0.22,<0.0.23) ; (python_version >= "3.10") and (extra == "proxy")
Requires-Dist: pyyaml (>=6.0.3,<7.0.0) ; extra == "proxy"
Requires-Dist: redisvl (>=0.16.0,<0.17.0) ; (python_version >= "3.9" and python_version < "3.14") and (extra == "extra-proxy")
Requires-Dist: resend (>=2.26.0) ; extra == "extra-proxy"
Requires-Dist: rich (>=14.3.3) ; extra == "proxy"
Requires-Dist: rq ; extra == "proxy"
Requires-Dist: semantic-router (>=0.1.12) ; (python_version >= "3.9" and python_version < "3.14") and (extra == "semantic-router")
Requires-Dist: soundfile (>=0.13.1,<0.14.0) ; extra == "proxy"
Requires-Dist: tiktoken (>=0.12.0)
Requires-Dist: tokenizers (>=0.22.2)
Requires-Dist: uvicorn (>=0.42.0,<1.0.0) ; extra == "proxy"
Requires-Dist: uvloop (>=0.22.1,<0.23.0) ; (sys_platform != "win32") and (extra == "proxy")
Requires-Dist: websockets (>=16.0,<17.0) ; extra == "proxy"
Project-URL: Documentation, https://docs.hanzo.ai
Project-URL: Homepage, https://hanzo.ai
Project-URL: Repository, https://github.com/hanzoai/llm
Project-URL: documentation, https://docs.hanzo.ai
Project-URL: homepage, https://hanzo.ai
Project-URL: repository, https://github.com/hanzoai/llm
Description-Content-Type: text/markdown

<p align="center"><img src=".github/hero.svg" alt="llm" width="880"></p>

<h1 align="center">Hanzo LLM</h1>
<p align="center">Call many LLM providers in the OpenAI format, from Python or through a proxy.</p>
<h4 align="center">
    <a href="https://pypi.org/project/hanzo-llm/" target="_blank">
        <img src="https://img.shields.io/pypi/v/hanzo-llm.svg" alt="PyPI Version">
    </a>
</h4>

<p align="center"><em>Forked from <a href="https://github.com/BerriAI/litellm">BerriAI/litellm</a> (MIT).</em></p>

## Install

Python 3.10 or newer.

```shell
pip install hanzo-llm             # the Python library
pip install 'hanzo-llm[proxy]'    # the library and the proxy server
```

It imports as `llm` and puts an `llm` command on the path. Python 3.9 gets
1.81.15, the last release that allowed it, and that release's `[proxy]` extra
cannot install.

## First call

`completion` takes a model name prefixed with its provider and answers in the
OpenAI response shape. This call goes to a local Ollama with `qwen3:0.6b`
pulled:

```python
from llm import completion

r = completion(
    model="ollama_chat/qwen3:0.6b",
    api_base="http://127.0.0.1:11434",
    messages=[{"role": "user", "content": "Say hello in five words."}],
)
print(r.choices[0].message.content)
```

The hosted Hanzo API takes the same call with `model="openai/<model id>"`,
`api_base="https://api.hanzo.ai/v1"` and a Hanzo API key as `api_key`. Model
ids are listed at `https://api.hanzo.ai/v1/models`.

## Proxy

The proxy serves the same format over HTTP. Name each model and where it runs
in a config file:

```yaml
model_list:
  - model_name: qwen3
    litellm_params:
      model: ollama_chat/qwen3:0.6b
      api_base: http://127.0.0.1:11434
```

Start it, then call it:

```shell
llm --config config.yaml --port 4000
```

```shell
curl -s http://127.0.0.1:4000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model": "qwen3", "messages": [{"role": "user", "content": "Say hello in five words."}]}'
```

This config sets no key, so the proxy answers any caller.

## Use Hanzo LLM for

<details>
<summary><b>Agents</b> - Invoke A2A Agents (Python SDK + AI Gateway)</summary>

[**Supported Providers**](https://docs.hanzo.ai/a2a#add-a2a-agents) - LangGraph, Vertex AI Agent Engine, Azure AI Foundry, Bedrock AgentCore, Pydantic AI

### Python SDK - A2A Protocol

```python
from llm.a2a_protocol import A2AClient
from a2a.types import SendMessageRequest, MessageSendParams
from uuid import uuid4

client = A2AClient(base_url="http://localhost:10001")

request = SendMessageRequest(
    id=str(uuid4()),
    params=MessageSendParams(
        message={
            "role": "user",
            "parts": [{"kind": "text", "text": "Hello!"}],
            "messageId": uuid4().hex,
        }
    )
)
response = await client.send_message(request)
```

### AI Gateway (Proxy Server)

**Step 1.** [Add your Agent to the AI Gateway](https://docs.hanzo.ai/a2a#adding-your-agent)

**Step 2.** Call Agent via A2A SDK

```python
from a2a.client import A2ACardResolver, A2AClient
from a2a.types import MessageSendParams, SendMessageRequest
from uuid import uuid4
import httpx

base_url = "http://localhost:4000/a2a/my-agent"  # Hanzo LLM gateway + agent name
headers = {"Authorization": "Bearer sk-1234"}    # Hanzo LLM Virtual Key

async with httpx.AsyncClient(headers=headers) as httpx_client:
    resolver = A2ACardResolver(httpx_client=httpx_client, base_url=base_url)
    agent_card = await resolver.get_agent_card()
    client = A2AClient(httpx_client=httpx_client, agent_card=agent_card)

    request = SendMessageRequest(
        id=str(uuid4()),
        params=MessageSendParams(
            message={
                "role": "user",
                "parts": [{"kind": "text", "text": "Hello!"}],
                "messageId": uuid4().hex,
            }
        )
    )
    response = await client.send_message(request)
```

[**Docs: A2A Agent Gateway**](https://docs.hanzo.ai/a2a)

</details>

<details>
<summary><b>MCP Tools</b> - Connect MCP servers to any LLM (Python SDK + AI Gateway)</summary>

### Python SDK - MCP Bridge

```python
from mcp import ClientSession, StdioServerParameters
from mcp.client.stdio import stdio_client
from llm import experimental_mcp_client
import llm

server_params = StdioServerParameters(command="python", args=["mcp_server.py"])

async with stdio_client(server_params) as (read, write):
    async with ClientSession(read, write) as session:
        await session.initialize()

        # Load MCP tools in OpenAI format
        tools = await experimental_mcp_client.load_mcp_tools(session=session, format="openai")

        # Use with any Hanzo LLM model
        response = await llm.acompletion(
            model="gpt-4o",
            messages=[{"role": "user", "content": "What's 3 + 5?"}],
            tools=tools
        )
```

### AI Gateway - MCP Gateway

**Step 1.** [Add your MCP Server to the AI Gateway](https://docs.hanzo.ai/mcp#adding-your-mcp)

**Step 2.** Call MCP tools via `/chat/completions`

```bash
curl -X POST 'http://0.0.0.0:4000/v1/chat/completions' \
  -H 'Authorization: Bearer sk-1234' \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "gpt-4o",
    "messages": [{"role": "user", "content": "Summarize the latest open PR"}],
    "tools": [{
      "type": "mcp",
      "server_url": "llm_proxy/mcp/github",
      "server_label": "github_mcp",
      "require_approval": "never"
    }]
  }'
```

### Use with Cursor IDE

```json
{
  "mcpServers": {
    "HanzoLLM": {
      "url": "http://localhost:4000/mcp/",
      "headers": {
        "x-llm-api-key": "Bearer sk-1234"
      }
    }
  }
}
```

[**Docs: MCP Gateway**](https://docs.hanzo.ai/mcp)

</details>

---

## Supported Providers ([Docs](https://docs.hanzo.ai/providers))

| Provider                                                                            | `/chat/completions` | `/messages` | `/responses` | `/embeddings` | `/image/generations` | `/audio/transcriptions` | `/audio/speech` | `/moderations` | `/batches` | `/rerank` |
|-------------------------------------------------------------------------------------|---------------------|-------------|--------------|---------------|----------------------|-------------------------|-----------------|----------------|-----------|-----------|
| [Abliteration (`abliteration`)](https://docs.hanzo.ai/providers/abliteration) | ✅ |  |  |  |  |  |  |  |  |  |
| [AI/ML API (`aiml`)](https://docs.hanzo.ai/providers/aiml) | ✅ | ✅ | ✅ | ✅ | ✅ |  |  |  |  |  |
| [AI21 (`ai21`)](https://docs.hanzo.ai/providers/ai21) | ✅ | ✅ | ✅ |  |  |  |  |  |  |  |
| [AI21 Chat (`ai21_chat`)](https://docs.hanzo.ai/providers/ai21) | ✅ | ✅ | ✅ |  |  |  |  |  |  |  |
| [Aleph Alpha](https://docs.hanzo.ai/providers/aleph_alpha) | ✅ | ✅ | ✅ |  |  |  |  |  |  |  |
| [Amazon Nova](https://docs.hanzo.ai/providers/amazon_nova) | ✅ | ✅ | ✅ |  |  |  |  |  |  |  |
| [Anthropic (`anthropic`)](https://docs.hanzo.ai/providers/anthropic) | ✅ | ✅ | ✅ |  |  |  |  |  | ✅ |  |
| [Anthropic Text (`anthropic_text`)](https://docs.hanzo.ai/providers/anthropic) | ✅ | ✅ | ✅ |  |  |  |  |  | ✅ |  |
| [Anyscale](https://docs.hanzo.ai/providers/anyscale) | ✅ | ✅ | ✅ |  |  |  |  |  |  |  |
| [AssemblyAI (`assemblyai`)](https://docs.hanzo.ai/pass_through/assembly_ai) | ✅ | ✅ | ✅ |  |  | ✅ |  |  |  |  |
| [Auto Router (`auto_router`)](https://docs.hanzo.ai/proxy/auto_routing) | ✅ | ✅ | ✅ |  |  |  |  |  |  |  |
| [AWS - Bedrock (`bedrock`)](https://docs.hanzo.ai/providers/bedrock) | ✅ | ✅ | ✅ | ✅ |  |  |  |  |  | ✅ |
| [AWS - Sagemaker (`sagemaker`)](https://docs.hanzo.ai/providers/aws_sagemaker) | ✅ | ✅ | ✅ | ✅ |  |  |  |  |  |  |
| [Azure (`azure`)](https://docs.hanzo.ai/providers/azure) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |  |
| [Azure AI (`azure_ai`)](https://docs.hanzo.ai/providers/azure_ai) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |  |
| [Azure Text (`azure_text`)](https://docs.hanzo.ai/providers/azure) | ✅ | ✅ | ✅ |  |  | ✅ | ✅ | ✅ | ✅ |  |
| [Baseten (`baseten`)](https://docs.hanzo.ai/providers/baseten) | ✅ | ✅ | ✅ |  |  |  |  |  |  |  |
| [Bytez (`bytez`)](https://docs.hanzo.ai/providers/bytez) | ✅ | ✅ | ✅ |  |  |  |  |  |  |  |
| [Cerebras (`cerebras`)](https://docs.hanzo.ai/providers/cerebras) | ✅ | ✅ | ✅ |  |  |  |  |  |  |  |
| [Clarifai (`clarifai`)](https://docs.hanzo.ai/providers/clarifai) | ✅ | ✅ | ✅ |  |  |  |  |  |  |  |
| [Cloudflare AI Workers (`cloudflare`)](https://docs.hanzo.ai/providers/cloudflare_workers) | ✅ | ✅ | ✅ |  |  |  |  |  |  |  |
| [Codestral (`codestral`)](https://docs.hanzo.ai/providers/codestral) | ✅ | ✅ | ✅ |  |  |  |  |  |  |  |
| [Cohere (`cohere`)](https://docs.hanzo.ai/providers/cohere) | ✅ | ✅ | ✅ | ✅ |  |  |  |  |  | ✅ |
| [Cohere Chat (`cohere_chat`)](https://docs.hanzo.ai/providers/cohere) | ✅ | ✅ | ✅ |  |  |  |  |  |  |  |
| [CometAPI (`cometapi`)](https://docs.hanzo.ai/providers/cometapi) | ✅ | ✅ | ✅ | ✅ |  |  |  |  |  |  |
| [CompactifAI (`compactifai`)](https://docs.hanzo.ai/providers/compactifai) | ✅ | ✅ | ✅ |  |  |  |  |  |  |  |
| [Custom (`custom`)](https://docs.hanzo.ai/providers/custom_llm_server) | ✅ | ✅ | ✅ |  |  |  |  |  |  |  |
| [Custom OpenAI (`custom_openai`)](https://docs.hanzo.ai/providers/openai_compatible) | ✅ | ✅ | ✅ |  |  | ✅ | ✅ | ✅ | ✅ |  |
| [Dashscope (`dashscope`)](https://docs.hanzo.ai/providers/dashscope) | ✅ | ✅ | ✅ |  |  |  |  |  |  |  |
| [Databricks (`databricks`)](https://docs.hanzo.ai/providers/databricks) | ✅ | ✅ | ✅ |  |  |  |  |  |  |  |
| [DataRobot (`datarobot`)](https://docs.hanzo.ai/providers/datarobot) | ✅ | ✅ | ✅ |  |  |  |  |  |  |  |
| [Deepgram (`deepgram`)](https://docs.hanzo.ai/providers/deepgram) | ✅ | ✅ | ✅ |  |  | ✅ |  |  |  |  |
| [DeepInfra (`deepinfra`)](https://docs.hanzo.ai/providers/deepinfra) | ✅ | ✅ | ✅ |  |  |  |  |  |  |  |
| [Deepseek (`deepseek`)](https://docs.hanzo.ai/providers/deepseek) | ✅ | ✅ | ✅ |  |  |  |  |  |  |  |
| [ElevenLabs (`elevenlabs`)](https://docs.hanzo.ai/providers/elevenlabs) | ✅ | ✅ | ✅ |  |  | ✅ | ✅ |  |  |  |
| [Empower (`empower`)](https://docs.hanzo.ai/providers/empower) | ✅ | ✅ | ✅ |  |  |  |  |  |  |  |
| [Fal AI (`fal_ai`)](https://docs.hanzo.ai/providers/fal_ai) | ✅ | ✅ | ✅ |  | ✅ |  |  |  |  |  |
| [Featherless AI (`featherless_ai`)](https://docs.hanzo.ai/providers/featherless_ai) | ✅ | ✅ | ✅ |  |  |  |  |  |  |  |
| [Fireworks AI (`fireworks_ai`)](https://docs.hanzo.ai/providers/fireworks_ai) | ✅ | ✅ | ✅ |  |  |  |  |  |  |  |
| [FriendliAI (`friendliai`)](https://docs.hanzo.ai/providers/friendliai) | ✅ | ✅ | ✅ |  |  |  |  |  |  |  |
| [Galadriel (`galadriel`)](https://docs.hanzo.ai/providers/galadriel) | ✅ | ✅ | ✅ |  |  |  |  |  |  |  |
| [GitHub Copilot (`github_copilot`)](https://docs.hanzo.ai/providers/github_copilot) | ✅ | ✅ | ✅ | ✅ |  |  |  |  |  |  |
| [GitHub Models (`github`)](https://docs.hanzo.ai/providers/github) | ✅ | ✅ | ✅ |  |  |  |  |  |  |  |
| [Google - PaLM](https://docs.hanzo.ai/providers/palm) | ✅ | ✅ | ✅ |  |  |  |  |  |  |  |
| [Google - Vertex AI (`vertex_ai`)](https://docs.hanzo.ai/providers/vertex) | ✅ | ✅ | ✅ | ✅ | ✅ |  |  |  |  |  |
| [Google AI Studio - Gemini (`gemini`)](https://docs.hanzo.ai/providers/gemini) | ✅ | ✅ | ✅ |  |  |  |  |  |  |  |
| [GradientAI (`gradient_ai`)](https://docs.hanzo.ai/providers/gradient_ai) | ✅ | ✅ | ✅ |  |  |  |  |  |  |  |
| [Groq AI (`groq`)](https://docs.hanzo.ai/providers/groq) | ✅ | ✅ | ✅ |  |  |  |  |  |  |  |
| [Heroku (`heroku`)](https://docs.hanzo.ai/providers/heroku) | ✅ | ✅ | ✅ |  |  |  |  |  |  |  |
| [Hosted VLLM (`hosted_vllm`)](https://docs.hanzo.ai/providers/vllm) | ✅ | ✅ | ✅ |  |  |  |  |  |  |  |
| [Huggingface (`huggingface`)](https://docs.hanzo.ai/providers/huggingface) | ✅ | ✅ | ✅ | ✅ |  |  |  |  |  | ✅ |
| [Hyperbolic (`hyperbolic`)](https://docs.hanzo.ai/providers/hyperbolic) | ✅ | ✅ | ✅ |  |  |  |  |  |  |  |
| [IBM - Watsonx.ai (`watsonx`)](https://docs.hanzo.ai/providers/watsonx) | ✅ | ✅ | ✅ | ✅ |  |  |  |  |  |  |
| [Infinity (`infinity`)](https://docs.hanzo.ai/providers/infinity) |  |  |  | ✅ |  |  |  |  |  |  |
| [Jina AI (`jina_ai`)](https://docs.hanzo.ai/providers/jina_ai) |  |  |  | ✅ |  |  |  |  |  |  |
| [Lambda AI (`lambda_ai`)](https://docs.hanzo.ai/providers/lambda_ai) | ✅ | ✅ | ✅ |  |  |  |  |  |  |  |
| [Lemonade (`lemonade`)](https://docs.hanzo.ai/providers/lemonade) | ✅ | ✅ | ✅ |  |  |  |  |  |  |  |
| [Hanzo LLM Gateway (`llm_proxy`)](https://docs.hanzo.ai/providers/llm_proxy) | ✅ | ✅ | ✅ | ✅ | ✅ |  |  |  |  |  |
| [Llamafile (`llamafile`)](https://docs.hanzo.ai/providers/llamafile) | ✅ | ✅ | ✅ |  |  |  |  |  |  |  |
| [LM Studio (`lm_studio`)](https://docs.hanzo.ai/providers/lm_studio) | ✅ | ✅ | ✅ |  |  |  |  |  |  |  |
| [Maritalk (`maritalk`)](https://docs.hanzo.ai/providers/maritalk) | ✅ | ✅ | ✅ |  |  |  |  |  |  |  |
| [Meta - Llama API (`meta_llama`)](https://docs.hanzo.ai/providers/meta_llama) | ✅ | ✅ | ✅ |  |  |  |  |  |  |  |
| [Mistral AI API (`mistral`)](https://docs.hanzo.ai/providers/mistral) | ✅ | ✅ | ✅ | ✅ |  |  |  |  |  |  |
| [Moonshot (`moonshot`)](https://docs.hanzo.ai/providers/moonshot) | ✅ | ✅ | ✅ |  |  |  |  |  |  |  |
| [Morph (`morph`)](https://docs.hanzo.ai/providers/morph) | ✅ | ✅ | ✅ |  |  |  |  |  |  |  |
| [Nebius AI Studio (`nebius`)](https://docs.hanzo.ai/providers/nebius) | ✅ | ✅ | ✅ | ✅ |  |  |  |  |  |  |
| [NLP Cloud (`nlp_cloud`)](https://docs.hanzo.ai/providers/nlp_cloud) | ✅ | ✅ | ✅ |  |  |  |  |  |  |  |
| [Novita AI (`novita`)](https://novita.ai/models/llm?utm_source=github_hanzo_llm&utm_medium=github_readme&utm_campaign=github_link) | ✅ | ✅ | ✅ |  |  |  |  |  |  |  |
| [Nscale (`nscale`)](https://docs.hanzo.ai/providers/nscale) | ✅ | ✅ | ✅ |  |  |  |  |  |  |  |
| [Nvidia NIM (`nvidia_nim`)](https://docs.hanzo.ai/providers/nvidia_nim) | ✅ | ✅ | ✅ |  |  |  |  |  |  |  |
| [OCI (`oci`)](https://docs.hanzo.ai/providers/oci) | ✅ | ✅ | ✅ |  |  |  |  |  |  |  |
| [Ollama (`ollama`)](https://docs.hanzo.ai/providers/ollama) | ✅ | ✅ | ✅ | ✅ |  |  |  |  |  |  |
| [Ollama Chat (`ollama_chat`)](https://docs.hanzo.ai/providers/ollama) | ✅ | ✅ | ✅ |  |  |  |  |  |  |  |
| [Oobabooga (`oobabooga`)](https://docs.hanzo.ai/providers/openai_compatible) | ✅ | ✅ | ✅ |  |  | ✅ | ✅ | ✅ | ✅ |  |
| [OpenAI (`openai`)](https://docs.hanzo.ai/providers/openai) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |  |
| [OpenAI-like (`openai_like`)](https://docs.hanzo.ai/providers/openai_compatible) |  |  |  | ✅ |  |  |  |  |  |  |
| [OpenRouter (`openrouter`)](https://docs.hanzo.ai/providers/openrouter) | ✅ | ✅ | ✅ |  |  |  |  |  |  |  |
| [OVHCloud AI Endpoints (`ovhcloud`)](https://docs.hanzo.ai/providers/ovhcloud) | ✅ | ✅ | ✅ |  |  |  |  |  |  |  |
| [Perplexity AI (`perplexity`)](https://docs.hanzo.ai/providers/perplexity) | ✅ | ✅ | ✅ |  |  |  |  |  |  |  |
| [Petals (`petals`)](https://docs.hanzo.ai/providers/petals) | ✅ | ✅ | ✅ |  |  |  |  |  |  |  |
| [Predibase (`predibase`)](https://docs.hanzo.ai/providers/predibase) | ✅ | ✅ | ✅ |  |  |  |  |  |  |  |
| [Recraft (`recraft`)](https://docs.hanzo.ai/providers/recraft) |  |  |  |  | ✅ |  |  |  |  |  |
| [Replicate (`replicate`)](https://docs.hanzo.ai/providers/replicate) | ✅ | ✅ | ✅ |  |  |  |  |  |  |  |
| [Sagemaker Chat (`sagemaker_chat`)](https://docs.hanzo.ai/providers/aws_sagemaker) | ✅ | ✅ | ✅ |  |  |  |  |  |  |  |
| [Sambanova (`sambanova`)](https://docs.hanzo.ai/providers/sambanova) | ✅ | ✅ | ✅ |  |  |  |  |  |  |  |
| [Snowflake (`snowflake`)](https://docs.hanzo.ai/providers/snowflake) | ✅ | ✅ | ✅ |  |  |  |  |  |  |  |
| [Text Completion Codestral (`text-completion-codestral`)](https://docs.hanzo.ai/providers/codestral) | ✅ | ✅ | ✅ |  |  |  |  |  |  |  |
| [Text Completion OpenAI (`text-completion-openai`)](https://docs.hanzo.ai/providers/text_completion_openai) | ✅ | ✅ | ✅ |  |  | ✅ | ✅ | ✅ | ✅ |  |
| [Together AI (`together_ai`)](https://docs.hanzo.ai/providers/togetherai) | ✅ | ✅ | ✅ |  |  |  |  |  |  |  |
| [Topaz (`topaz`)](https://docs.hanzo.ai/providers/topaz) | ✅ | ✅ | ✅ |  |  |  |  |  |  |  |
| [Triton (`triton`)](https://docs.hanzo.ai/providers/triton-inference-server) | ✅ | ✅ | ✅ |  |  |  |  |  |  |  |
| [V0 (`v0`)](https://docs.hanzo.ai/providers/v0) | ✅ | ✅ | ✅ |  |  |  |  |  |  |  |
| [Vercel AI Gateway (`vercel_ai_gateway`)](https://docs.hanzo.ai/providers/vercel_ai_gateway) | ✅ | ✅ | ✅ |  |  |  |  |  |  |  |
| [VLLM (`vllm`)](https://docs.hanzo.ai/providers/vllm) | ✅ | ✅ | ✅ |  |  |  |  |  |  |  |
| [Volcengine (`volcengine`)](https://docs.hanzo.ai/providers/volcano) | ✅ | ✅ | ✅ |  |  |  |  |  |  |  |
| [Voyage AI (`voyage`)](https://docs.hanzo.ai/providers/voyage) |  |  |  | ✅ |  |  |  |  |  |  |
| [WandB Inference (`wandb`)](https://docs.hanzo.ai/providers/wandb_inference) | ✅ | ✅ | ✅ |  |  |  |  |  |  |  |
| [Watsonx Text (`watsonx_text`)](https://docs.hanzo.ai/providers/watsonx) | ✅ | ✅ | ✅ |  |  |  |  |  |  |  |
| [xAI (`xai`)](https://docs.hanzo.ai/providers/xai) | ✅ | ✅ | ✅ |  |  |  |  |  |  |  |
| [Xinference (`xinference`)](https://docs.hanzo.ai/providers/xinference) |  |  |  | ✅ |  |  |  |  |  |  |

[**Read the Docs**](https://docs.hanzo.ai/)

## Run in developer mode

Start the database and Prometheus from the repo root; `.env.example` lists the
settings:

```shell
docker compose up db prometheus
```

Then run the proxy from a virtual environment:

```shell
python -m venv .venv && source .venv/bin/activate
pip install -e '.[proxy]'
pip install prisma
prisma generate
python llm/proxy/proxy_cli.py
```

# Contributing

We welcome contributions to Hanzo LLM! Whether you're fixing bugs, adding features, or improving documentation, we appreciate your help.

## Quick Start for Contributors

This requires poetry to be installed.

```bash
git clone https://github.com/hanzoai/llm.git
cd llm
make install-dev    # Install development dependencies
make format         # Format your code
make lint           # Run all linting checks
make test-unit      # Run unit tests
make format-check   # Check formatting only
```

For detailed contributing guidelines, see [CONTRIBUTING.md](CONTRIBUTING.md).

## Code Quality / Linting

Hanzo LLM follows the [Google Python Style Guide](https://google.github.io/styleguide/pyguide.html).

Our automated checks include:
- **Black** for code formatting
- **Ruff** for linting and code quality
- **MyPy** for type checking
- **Circular import detection**
- **Import safety checks**


All these checks must pass before your PR can be merged.


# Support

dev@hanzo.ai

# Contributors

<!-- ALL-CONTRIBUTORS-LIST:START - Do not remove or modify this section -->
<!-- prettier-ignore-start -->
<!-- markdownlint-disable -->

<!-- markdownlint-restore -->
<!-- prettier-ignore-end -->

<!-- ALL-CONTRIBUTORS-LIST:END -->

<a href="https://github.com/hanzoai/llm/graphs/contributors">
  <img src="https://contrib.rocks/image?repo=hanzoai/llm" />
</a>



