Metadata-Version: 2.5
Name: switchboard-router
Version: 0.1.0
Summary: Pick the cheapest LLM that can handle each prompt, with the open Switchboard router.
Project-URL: Homepage, https://huggingface.co/spaces/switchboardllm/switchboard
Project-URL: Model, https://huggingface.co/switchboardllm/switchboard-router-0.5b
Author: Switchboard
License-Expression: Apache-2.0
License-File: LICENSE
Keywords: cost,llm,llm-routing,ollama,openai,router
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Typing :: Typed
Requires-Python: >=3.9
Provides-Extra: local
Requires-Dist: torch>=2.1; extra == 'local'
Requires-Dist: transformers>=4.45; extra == 'local'
Description-Content-Type: text/markdown

# switchboard-router

Pick the cheapest LLM that can handle each prompt.

`switchboard-router` asks [Switchboard](https://huggingface.co/switchboardllm/switchboard-router-0.5b), a 0.5B open router model, which tier a prompt needs: `small`, `mid` or `frontier`. Easy prompts go to a cheap model, and you pay frontier prices only when the prompt needs it.

```
pip install switchboard-router
```

## Quickstart

You need one way to run the model. The easiest is [Ollama](https://ollama.com): install it and leave it running. If the model isn't in Ollama yet, the first call pulls it (994 MB) from Hugging Face, which takes a minute or two. To do that ahead of time:

```
ollama pull hf.co/switchboardllm/switchboard-router-0.5b-GGUF
```

```python
from switchboard_router import Router

router = Router()  # uses Ollama when it's running

router.route("Write a haiku about rain").tier
# 'small'


router.pick_model("Unscramble the following words: eglar, ngareo, kobo")
# 'anthropic/claude-opus-5.5'
```

From the terminal:

```
switchboard "Design a rate limiter for 10M requests per second across 5 regions"
{"tier": "mid", "domain": "general", "valid": true, "model": "anthropic/claude-sonnet-5.5"}
```

## Route, then call the model

`pick_model` returns a model id for the tier. By default these are Claude models on [Surplus](https://surplusllm.com), which serves every model behind one OpenAI-compatible key:

```python
from openai import OpenAI
from switchboard_router import Router

router = Router()
llm = OpenAI(base_url="https://api.surplusllm.com/api/v1", api_key="sk-surplus-...")

prompt = "Write a Python function that reverses a string"
reply = llm.chat.completions.create(
    model=router.pick_model(prompt),
    messages=[{"role": "user", "content": prompt}],
)
```

Use your own models per tier (give all three, or the rest fall back to the defaults with a warning):

```python
router = Router(models={"small": "my/small-model", "mid": "my/mid-model", "frontier": "my/frontier-model"})
```

## Backends

| Backend | Use it when | How |
|---|---|---|
| `ollama` | You run models locally | `Router(backend="ollama")` |
| `openai` | You serve the model with llama-server or vLLM, or call it on the Surplus API | `Router(backend="openai", base_url="http://localhost:8080/v1", backend_model="switchboard")` |
| `transformers` | You want it in-process | `pip install 'switchboard-router[local]'`, then `Router(backend="transformers")` |

`Router()` with no backend uses Ollama if it's running, then Transformers if it's installed, and otherwise tells you what to install.

## Failing safe

If the backend can't be reached, errors out, or the model's output isn't valid JSON, the decision is `frontier` with `valid=False` and the reason in `decision.error`. A failed routing call costs you money, never quality. With `fail_safe=False`, those cases raise instead. Empty or non-string prompts always raise `ValueError` or `TypeError`.

Requests time out after 15 seconds by default (`Router(timeout=5)` to change it), and redirects are not followed, so an API key is never sent to another host.

## Limits

- The router reads the first 2,000 characters of a prompt. A question at the very end of a long prompt may be missed.
- It reads the prompt it's given, so a prompt can try to talk it into a cheaper tier. Don't use it as a security boundary.

## How good is the routing?

Switchboard v0.1 on 10,000 held-out prompts: 64.5% cheaper than sending every prompt to Claude Opus 5.5, and 5.5% of prompts were sent to a cheaper tier than they needed. Method, limits and the full results are in the [model card](https://huggingface.co/switchboardllm/switchboard-router-0.5b). Try it in your browser: [Switchboard demo](https://huggingface.co/spaces/switchboardllm/switchboard).

## License

Apache-2.0.
