Metadata-Version: 2.5
Name: resilient-llm-gateway
Version: 0.1.0
Summary: High-throughput LLM Gateway and Circuit Breaker for distributed AI agents
Project-URL: Homepage, https://github.com/your-org/llm-gateway
Project-URL: Repository, https://github.com/your-org/llm-gateway
Project-URL: Issues, https://github.com/your-org/llm-gateway/issues
Author: LLM Gateway Contributors
License: MIT
Keywords: asyncio,circuit-breaker,gateway,llm,openai,rate-limiter
Classifier: Development Status :: 3 - Alpha
Classifier: Framework :: AsyncIO
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Software Development :: Libraries
Classifier: Typing :: Typed
Requires-Python: >=3.12
Requires-Dist: httpx<1.0.0,>=0.27.0
Requires-Dist: pydantic-settings[yaml]<3.0.0,>=2.3.0
Requires-Dist: pydantic<3.0.0,>=2.7.0
Requires-Dist: redis[hiredis]<6.0.0,>=5.0.0
Requires-Dist: structlog>=24.0.0
Provides-Extra: all
Requires-Dist: llm-gateway[dev,proxy,semantic-cache]; extra == 'all'
Provides-Extra: dev
Requires-Dist: anyio[trio]>=4.4.0; extra == 'dev'
Requires-Dist: fakeredis>=2.23.0; extra == 'dev'
Requires-Dist: mypy>=1.10.0; extra == 'dev'
Requires-Dist: pytest-asyncio>=0.23.0; extra == 'dev'
Requires-Dist: pytest-cov>=5.0.0; extra == 'dev'
Requires-Dist: pytest-docker>=3.1.0; extra == 'dev'
Requires-Dist: pytest>=8.2.0; extra == 'dev'
Requires-Dist: respx>=0.21.0; extra == 'dev'
Requires-Dist: ruff>=0.4.0; extra == 'dev'
Requires-Dist: types-pyyaml; extra == 'dev'
Provides-Extra: proxy
Requires-Dist: fastapi<1.0.0,>=0.111.0; extra == 'proxy'
Requires-Dist: prometheus-client>=0.20.0; extra == 'proxy'
Requires-Dist: uvicorn[standard]>=0.30.0; extra == 'proxy'
Provides-Extra: semantic-cache
Requires-Dist: fastembed>=0.3.0; extra == 'semantic-cache'
Requires-Dist: numpy>=1.26.0; extra == 'semantic-cache'
Description-Content-Type: text/markdown

# ⚡ LLM Gateway & Circuit Breaker

[![Python 3.12+](https://img.shields.io/badge/python-3.12%2B-blue.svg)](https://www.python.org/downloads/)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
[![Tests Passing](https://img.shields.io/badge/tests-209%20passed-brightgreen.svg)](tests/)
[![Coverage](https://img.shields.io/badge/coverage-93.5%25-success.svg)](tests/)
[![Throughput](https://img.shields.io/badge/throughput-1700%2B%20req%2Fs-blueviolet.svg)](tests/load/)

A high-throughput, distributed **LLM Gateway and Circuit Breaker** designed to protect AI agents, async workers, and microservices from upstream LLM provider outages, rate limits ($429$), and latency spikes.

---

## 🌟 Key Features

* **🛡️ Atomic Redis Circuit Breaker (FSM)**: Lock-free finite state machine (`CLOSED` → `OPEN` → `HALF_OPEN`) powered by single-roundtrip Redis Lua scripts with probe throttling and clock-skew protection.
* **⏱️ Dual-Budget Distributed Rate Limiter**: Atomic token bucket tracking both Requests Per Minute (**RPM**) and Tokens Per Minute (**TPM**) with $O(1)$ token estimation and all-or-nothing consumption.
* **🔀 Resilient Routing & Fallback Chains**: Multi-tier failover chains across primary and secondary providers with **safe pre-yield streaming failovers**.
* **⚡ Tiered Caching Layer**:
  * **Tier 1**: Sub-millisecond deterministic SHA-256 exact match cache in Redis.
  * **Tier 2**: Local semantic similarity cache using `fastembed` ONNX embeddings (no external API calls to check cache).
* **🔌 Unified Provider Adapters**: Native connection pooling and normalized OpenAI completion shapes for **OpenAI**, **Anthropic**, **Azure OpenAI**, **Ollama**, and **Mock/Offline**.
* **📊 Prometheus Metrics & Observability**: Real-time histograms and counters for request latencies, status codes, and provider health.
* **📦 Dual Distribution**: Use as an embedded **Python Library** or run as a standalone **FastAPI Docker Proxy**.

---

## 🏗️ System Architecture

```
                                  ┌───────────────────────────────┐
                                  │   Client Request (OpenAI SDK) │
                                  └───────────────┬───────────────┘
                                                  │
                                          [ Auth & Metrics ]
                                                  │
                                                  ▼
                                     ┌─────────────────────────┐
                                     │     Gateway Router      │
                                     └────────────┬────────────┘
                                                  │
                 ┌────────────────────────────────┴────────────────────────────────┐
                 │                                                                 │
                 ▼                                                                 ▼
      ┌────────────────────┐                                             ┌───────────────────┐
      │   Exact Match      │──(Cache Hit)──► [ Return Cached Response ]  │  Semantic Cache   │
      │   SHA-256 Cache    │                                             │  (fastembed ONNX) │
      └────────────────────┘                                             └───────────────────┘
                 │ (Cache Miss)
                 ▼
      ┌────────────────────┐
      │  Circuit Breaker   │──(Open / Tripped)──► [ Fast 503 Rejection / Cascade Fallback ]
      │  Atomic Lua Script │
      └──────────┬─────────┘
                 │ (Allowed)
                 ▼
      ┌────────────────────┐
      │ Rate Limiter Bucket│──(Exceeded)────────► [ 429 / Cascade Fallback ]
      │  (RPM + TPM Budget)│
      └──────────┬─────────┘
                 │ (Granted)
                 ▼
     ┌─────────────────────────────────────────────────────────────┐
     │                     Provider Dispatch                       │
     │   [Primary: OpenAI]  ──► (Fail) ──►  [Fallback: Anthropic]   │
     └─────────────────────────────────────────────────────────────┘
```

---

## 📦 Installation

### As a Python Library
```bash
# Core library
pip install llm-gateway

# With FastAPI standalone proxy dependencies
pip install "llm-gateway[proxy]"

# With local ONNX semantic cache dependencies
pip install "llm-gateway[semantic-cache]"

# With full development & test suite
pip install "llm-gateway[dev,proxy,semantic-cache]"
```

---

## 🚀 Quick Start

### 1. Using as a Python Library (Direct Asyncio)

```python
import asyncio
from llm_gateway import GatewayConfig, GatewayRouter
from llm_gateway.core.providers.base import ChatCompletionRequest, ChatMessage

async def main():
    # 1. Define configuration with fallback routes
    config = GatewayConfig(
        providers={
            "openai-main": {
                "type": "openai",
                "api_key": "sk-...",
                "default_model": "gpt-4o",
                "circuit_breaker": {"failure_threshold": 3, "recovery_timeout_s": 30},
                "rate_limit": {"rpm": 500, "tpm": 100000},
            },
            "anthropic-fallback": {
                "type": "anthropic",
                "api_key": "sk-ant-...",
                "default_model": "claude-3-5-sonnet-20241022",
            },
        },
        routes={
            "default": {
                "primary": "openai-main",
                "fallbacks": ["anthropic-fallback"],
            }
        },
        redis={"url": "redis://localhost:6379/0", "fail_open": True},
    )

    # 2. Instantiate router
    router = GatewayRouter(config)

    # 3. Dispatch chat completion
    req = ChatCompletionRequest(
        model="gpt-4o",
        messages=[ChatMessage(role="user", content="Explain quantum computing in 10 words.")],
    )
    response = await router.route(req, route_name="default")
    print(response.choices[0].message.content)

    await router.close()

if __name__ == "__main__":
    asyncio.run(main())
```

---

### 2. Running as a Standalone Docker Proxy

#### Step 1: Create your `gateway.yaml`
```yaml
providers:
  openai:
    type: openai
    api_key: "${OPENAI_API_KEY}"
    default_model: gpt-4o
    circuit_breaker:
      failure_threshold: 3
      recovery_timeout_s: 30
    rate_limit:
      rpm: 1000
      tpm: 200000

  anthropic:
    type: anthropic
    api_key: "${ANTHROPIC_API_KEY}"
    default_model: claude-3-5-sonnet-20241022

  mock-offline:
    type: mock
    default_model: mock-gpt-4o
    mock_response: "Hello from local mock LLM!"

routes:
  default:
    primary: openai
    fallbacks: [anthropic, mock-offline]

redis:
  url: "redis://localhost:6379/0"
  fail_open: true

auth:
  enabled: false
```

#### Step 2: Start the Stack with Docker Compose
```bash
docker compose up --build
```
* **LLM Gateway Endpoint**: `http://localhost:8000`
* **Prometheus Metrics**: `http://localhost:9090`

#### Step 3: Call with the OpenAI SDK
```python
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="any-string",  # or your Bearer token if auth.enabled is true
)

response = client.chat.completions.create(
    model="default",
    messages=[{"role": "user", "content": "Hello LLM Gateway!"}],
)

print(response.choices[0].message.content)
```

---

## 📡 API Endpoints

| Method | Endpoint | Description |
|---|---|---|
| `POST` | `/v1/chat/completions` | OpenAI-compatible chat completion (supports JSON & SSE streaming `stream=true`) |
| `GET` | `/v1/models` | List all available gateway routes and default models |
| `GET` | `/health/live` | Liveness probe (Kubernetes / Docker) |
| `GET` | `/health/ready` | Readiness probe (verifies provider connectivity) |
| `GET` | `/metrics` | Prometheus metrics exporter |

---

## ⚙️ Configuration Reference

| Parameter | Environment Variable | Default | Description |
|---|---|---|---|
| `redis.url` | `LLM_GATEWAY__REDIS__URL` / `REDIS_URL` | `redis://localhost:6379/0` | Redis connection URI |
| `redis.fail_open` | `LLM_GATEWAY__REDIS__FAIL_OPEN` | `true` | When true, gateway continues operating if Redis is unreachable |
| `auth.enabled` | `LLM_GATEWAY__AUTH__ENABLED` | `false` | Enable static Bearer token authentication |
| `auth.tokens` | `LLM_GATEWAY__AUTH__TOKENS` | `[]` | List of authorized Bearer tokens |
| `cache.exact_ttl_s` | `LLM_GATEWAY__CACHE__EXACT_TTL_S` | `3600` | Exact-match cache TTL in seconds |
| `cache.semantic_threshold` | `LLM_GATEWAY__CACHE__SEMANTIC_THRESHOLD` | `0.92` | Cosine similarity threshold for semantic cache hits |

---

## 🧪 Testing & Load Benchmarking

### Automated Test Suite
```bash
# Run all 209 unit, integration, and load tests
pytest tests/unit/ tests/integration/ tests/load/ --cov=llm_gateway
```

### Live Concurrency & Stress Testing
```bash
# Non-streaming stress test (1,000 requests, 50 concurrent workers)
python3 stress_test.py -n 1000 -c 50

# Streaming SSE stress test (200 requests, 20 concurrent workers)
python3 stress_test.py -n 200 -c 20 --stream
```

### Export OpenAPI & JSON Schemas
```bash
python3 scripts/export_schema.py --all ./schemas
```

---

## 📄 License

This project is licensed under the [MIT License](LICENSE).
