Metadata-Version: 2.4
Name: switchboard-llm
Version: 0.1.0
Summary: An OpenAI-compatible LLM router that saves cost without losing quality.
Project-URL: Homepage, https://github.com/archit0/switchboard
Project-URL: Repository, https://github.com/archit0/switchboard
Project-URL: Issues, https://github.com/archit0/switchboard/issues
Author: Archit Dwivedi
License-Expression: MIT
License-File: LICENSE
Keywords: anthropic,cost,frugalgpt,gateway,gemini,llm,mixture-of-agents,openai,router
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.11
Requires-Dist: fastapi>=0.110
Requires-Dist: httpx>=0.27
Requires-Dist: uvicorn[standard]>=0.29
Description-Content-Type: text/markdown

# switchboard

[![CI](https://github.com/archit0/switchboard/actions/workflows/ci.yml/badge.svg)](https://github.com/archit0/switchboard/actions/workflows/ci.yml)
[![PyPI](https://img.shields.io/pypi/v/switchboard-llm.svg)](https://pypi.org/project/switchboard-llm/)

An **OpenAI-compatible LLM router** that saves cost without losing quality. Point
any OpenAI client at it and it routes each request to the cheapest model that can
handle it — easy prompts to a small model, hard ones to a parallel
**Mixture-of-Agents** — trading a little latency for large savings while holding
(or beating) frontier-model quality on a representative workload.

```python
from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="anything")
client.chat.completions.create(model="router-cost", messages=[{"role": "user", "content": "..."}])
```

It works on top of **any OpenAI-compatible gateway that fronts multiple providers**
behind one key (e.g. a LiteLLM proxy) — so one client can reach OpenAI, Anthropic,
and Google models just by changing the `model` field. The router is a thin policy
on top of that.

---

## Install

```bash
pip install switchboard-llm        # or: uv add switchboard-llm
```

Configure your gateway (any OpenAI-compatible endpoint):

```bash
export OPENAI_API_KEY=...                  # your gateway key
export OPENAI_BASE_URL=https://.../v1      # your endpoint
```

## Use it

**As a server** (drop-in for any OpenAI client):

```bash
switchboard serve                          # http://localhost:8000/v1  (use --port to change)
```
```python
from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="anything")
r = client.chat.completions.create(model="router", messages=[{"role": "user", "content": "Hi"}])
print(r.model_extra["switchboard"])        # route, cost, savings telemetry
```

**As a library:**

```python
import asyncio
from switchboard import Engine

async def main():
    eng = Engine()
    rr = await eng.answer([{"role": "user", "content": "What is 17 * 23?"}], mode="cost")
    print(rr.content, f"${rr.cost:.6f}", f"{rr.savings_pct:.0f}% cheaper than Opus")
    await eng.aclose()

asyncio.run(main())
```

**From the CLI:**

```bash
switchboard ask "Prove sqrt(2) is irrational" --mode quality
switchboard models                         # probe which gateway models are actually live
```

---

## The honest thesis (read this first)

The goal is a router that is **cheaper than a frontier model (e.g. Opus) and
matches-or-beats it on benchmarks**. That is achievable — but only as a
**portfolio result over a realistic workload**, not a per-query miracle. The iron
law:

> On a *single hard query*, you cannot both beat the frontier model **and** be
> cheaper than it on that same query.

What you *can* do, and what this does:

| Traffic | What the router does | Outcome |
|---|---|---|
| **Easy queries** (most real traffic) | route to a cheap model | quality ties Opus, **5–50× cheaper** |
| **Hard queries** (the minority) | **Mixture-of-Agents**: several cheap/mid models answer in parallel, a synthesizer fuses them | quality can **match or exceed** a single Opus call, still **< Opus cost** |
| **Repeats** | exact-match cache | **free** |

Averaged over the workload, total spend is well below always-Opus and mean
accuracy is **equal-or-better**. Grounded in **RouteLLM**, **FrugalGPT** (cascade
with a judge), and **Mixture-of-Agents**.

---

## Modes

Pick the strategy via the `model` field:

| `model` | strategy |
|---|---|
| `router` / `router-balanced` | triage → single cheap (easy) / single mid (moderate) / Mixture-of-Agents (hard) |
| `router-cost` | **FrugalGPT cascade** — answer cheap, a judge scores it, escalate only if low |
| `router-quality` | bias one tier up — best quality while staying under Opus cost |

Any **real** model id (`claude-opus-4-8`, `gpt-5.5`, …) passes straight through, so
this also works as a plain multi-provider proxy.

## How it works

```
request ─► [cache] ─► [triage: how hard?] ─► [policy] ──► single cheap model      (easy)
                                                      └─► single mid model        (moderate)
                                                      └─► Mixture-of-Agents        (hard)
                                                            proposers ∥ ─► synthesizer
```

- **Triage** (`src/switchboard/classify.py`) — free heuristics (length, code/math
  markers, multi-step verbs) decide obvious cases; a tiny LLM classifier scores the
  ambiguous middle. Output: difficulty 1–5 → tier.
- **Policy / execution** (`src/switchboard/engine.py`) — `single`, `moa` (parallel
  proposers + synthesizer), or `cascade` (cheap → judge → escalate).
- **Cost accounting** — every response carries its internal cost, an estimate of
  what always-Opus would have cost, and the savings %, under a `switchboard` key.

---

## Results

On **GSM8K (50 items, exact numeric grading)**, baseline = always `claude-opus-4-8`:

| config | accuracy | total cost | vs Opus |
|---|---|---|---|
| always-Opus | 100.0% | $0.3674 | baseline |
| `router-cost` | **100.0%** | $0.0064 | **57× cheaper — Pareto win** |
| `router-quality` | 100.0% | $0.2781 | 1.3× cheaper |
| `router-balanced` | 92.0% | $0.0611 | 6× cheaper but lost accuracy |

Reproduce: `python -m bench.run_gsm8k --n 50 --seed 0`. Full write-up and honest
caveats in [`RESULTS.md`](RESULTS.md). (The verifier is what makes routing safe —
`router-balanced` has none and lost 8 points; `router-cost`'s judge is the fix.)

---

## Limitations & next steps

- **Pricing is a list-price proxy** (`src/switchboard/config.py`). Drop your real
  rate card into `pricing.json` (`{"model": [in_per_1M, out_per_1M]}`) to override.
- **Triage under-detects "deceptively simple" trap questions** — `router-cost`/
  `router-quality` compensate via the judge/MoA.
- **Streaming is simulated** (full answer computed, then chunked) — MoA can't
  token-stream; only the single-model path could truly stream.
- **Semantic cache** (embed prompt → nearest neighbour) is not yet wired.
- **The gateway's `/v1/models` list may be stale** — trust `switchboard models`.

## License

MIT — see [LICENSE](LICENSE).
