Metadata-Version: 2.5
Name: switchboard-router
Version: 0.4.0
Summary: Pick the cheapest LLM that can handle each prompt, with the open Switchboard router.
Project-URL: Homepage, https://huggingface.co/spaces/switchboardllm/switchboard
Project-URL: Model, https://huggingface.co/switchboardllm/switchboard-router-0.5b
Author: Switchboard
License-Expression: Apache-2.0
License-File: LICENSE
Keywords: cost,llm,llm-routing,ollama,openai,proxy,router,surplus
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Typing :: Typed
Requires-Python: >=3.9
Provides-Extra: local
Requires-Dist: torch>=2.1; extra == 'local'
Requires-Dist: transformers>=4.45; extra == 'local'
Description-Content-Type: text/markdown

# switchboard-router

Pick the cheapest LLM that can handle each prompt.

`switchboard-router` asks [Switchboard](https://huggingface.co/switchboardllm/switchboard-router-0.5b), a 0.5B open router model, which tier a prompt needs: `small`, `mid` or `frontier`. Easy prompts go to a cheap model, and you pay frontier prices only when the prompt needs it.

```
pip install switchboard-router
```

## Quickstart

You need one way to run the model. The easiest is [Ollama](https://ollama.com): install it and leave it running. If the model isn't in Ollama yet, the first call pulls it (994 MB) from Hugging Face, which takes a minute or two. To do that ahead of time:

```
ollama pull hf.co/switchboardllm/switchboard-router-0.5b-GGUF
```

```python
from switchboard_router import Router

router = Router()  # uses Ollama when it's running

router.route("Write a haiku about rain").tier
# 'small'


router.pick_model("Unscramble the following words: eglar, ngareo, kobo")
# 'anthropic/claude-opus-5.5'
```

From the terminal:

```
switchboard "Design a rate limiter for 10M requests per second across 5 regions"
{"tier": "mid", "domain": "general", "valid": true, "model": "anthropic/claude-sonnet-5.5"}
```

## Route, then call the model

`pick_model` returns a model id for the tier. By default these are Claude models on [Surplus](https://surplusllm.com), which serves every model behind one OpenAI-compatible key:

```python
from openai import OpenAI
from switchboard_router import Router

router = Router()
llm = OpenAI(base_url="https://api.surplusllm.com/api/v1", api_key="sk-surplus-...")

prompt = "Write a Python function that reverses a string"
reply = llm.chat.completions.create(
    model=router.pick_model(prompt),
    messages=[{"role": "user", "content": prompt}],
)
```

Use your own models per tier (give all three, or the rest fall back to the defaults with a warning):

```python
router = Router(models={"small": "my/small-model", "mid": "my/mid-model", "frontier": "my/frontier-model"})
```

## Put it in front of your app: `switchboard serve`

Run a local OpenAI-compatible proxy, point your app at it, and send `model="auto"`. No other code changes.

```
switchboard serve
```

```python
from openai import OpenAI

client = OpenAI(base_url="http://127.0.0.1:8787/v1", api_key="sk-surplus-...")  # your Surplus key
reply = client.chat.completions.create(
    model="auto",
    messages=[{"role": "user", "content": "Write a haiku about rain"}],
)
```

Each `auto` request is routed on its last user message, then forwarded to the [Surplus API](https://surplusllm.com) with your key. Streaming works. Any other model name passes straight through, so you can mix `auto` with fixed models. Every response says what happened in its headers: `x-switchboard-tier`, `x-switchboard-domain` and `x-switchboard-model`.

- **Cold starts:** launch models answer 503 while their GPU wakes up. Switchboard retries for up to 60 seconds (`--cold-start-wait`), then sends the request to the tier's default model instead and marks it with `x-switchboard-fallback: 1`.
- **Real costs:** with `--record`, every routed request is written to the savings ledger with what Surplus actually charged (`x-surplus-cost-usd`). `switchboard --stats` then reports real spend next to the estimate.
- **Keys:** your app's key is forwarded as-is. To keep the key on the server instead, use `--upstream-key-env SURPLUS_API_KEY`.
- **Safety:** it listens on 127.0.0.1 only. Binding to another address needs `--allow-remote`, because anyone who can reach the proxy could spend through it. Prompts are never logged or stored.
- **Other APIs:** `--upstream https://...` points it at any OpenAI-compatible API.
- **Token limits:** Claude 5.5 models on Surplus think before they answer, and the thinking counts toward `max_tokens`. With a small limit (say 120) the visible reply can come back empty, so leave room.

## Use it inside vLLM Semantic Router

[vLLM Semantic Router](https://github.com/vllm-project/semantic-router) can ask a remote scorer how hard a request is. `switchboard sr-backend` answers those calls, so Switchboard plugs into it with configuration only:

```
switchboard sr-backend --backend ollama --host 0.0.0.0 --allow-remote --access-key-env SB_KEY
```

It serves `POST /classify` on port 8090. Tiers map onto vLLM SR's verdicts: small is `easy`, mid is `medium`, frontier is `hard`.

- **Real confidence:** with Ollama, vLLM or Transformers, each verdict comes with the probability the model gave it, read from the tier token's logprobs. Other backends send 1.0 for the chosen verdict.
- **Contracts:** `label_distribution.v1` (default) returns a score for each of the three verdicts. `score.v1` (`--contract score.v1`) returns one difficulty number from 0 to 1.
- **Failures:** if Switchboard can't grade a request it answers 503, never a guessed verdict. vLLM SR then treats the signal as unknown.
- **Keys:** with `--access-key-env`, every call must carry that key as a bearer token. Set the same value as `access_key` in vLLM SR. Listening on anything but loopback (for vLLM SR in Docker) needs both `--allow-remote` and a key.
- **Other flags:** `--ollama-host` if Ollama runs elsewhere, `--timeout` for the router backend, and `--max-concurrent` (default 4) for how many requests the backend handles at once.

[`examples/vllm-sr/config.yaml`](examples/vllm-sr/config.yaml) is a complete config that sends easy requests to Nemotron Nano, medium ones to Nemotron Super and hard ones to Nemotron Ultra, all on the Surplus API. It passes `vllm-sr config validate`. It has also been tested end to end with vLLM SR 0.4.0 in Docker: each prompt reached the expected model, and the scorer added 70–85 ms per request.

- **Keep the model warm:** vLLM SR waits 5 seconds (`deadline_ms`) for a verdict. A cold Ollama model can take longer, and then the request counts as unknown; this config sends unknown requests to the strongest model. Start Ollama with `OLLAMA_KEEP_ALIVE=30m` to keep the model loaded.
- **Surplus and OpenRouter responses:** vLLM SR 0.4.0 rejects the extra fields these APIs add to every reply (`reasoning`, `reasoning_details`, `provider`, `native_finish_reason`) and returns a 502. Routing still works; the reply is what fails. Until vLLM SR accepts those fields, use providers that return plain OpenAI responses, or put a proxy in front that removes the fields.

## Send one kind of prompt to a specific model

The router also names a domain (`code`, `math`, `writing` or `general`). Overrides send one tier and domain to a model of your choice, and everything else keeps the tier default. Failed decisions ignore overrides and always use the frontier model.

```python
router = Router(overrides={"small/code": "launch/Nock-AI/nock-coder-1.5b@18859c0"})
```

Models launched on the Surplus launchpad are served at a fixed $0.03 per 1M tokens. To see which ones you can call right now:

```
switchboard --launchpad
```

Switchboard has not measured how well any launchpad model answers a given tier, so test one on your own prompts before you rely on it.

## See what routing saved you

Turn on the ledger and every `pick_model` call is written to a local file: the time, tier, domain and model. The prompt is never stored, and nothing is sent anywhere.

```python
router = Router(ledger=True)   # ~/.switchboard/ledger.jsonl, or set SWITCHBOARD_LEDGER
```

```
switchboard "Summarize this ticket" --record
switchboard --stats
```

`--stats` compares what the recorded decisions would cost with sending every one to the frontier model. It's an estimate: it prices output tokens only and assumes every answer is 500 tokens long (change it with `--tokens-per-answer`).

To try your own prompts without installing anything, paste them into the [savings calculator](https://switchboardllm-switchboard.static.hf.space/savings.html). It runs the router in your browser.

## Backends

| Backend | Use it when | How |
|---|---|---|
| `ollama` | You run models locally | `Router(backend="ollama")` |
| `openai` | You serve the model with llama-server or vLLM, or call it on the Surplus API | `Router(backend="openai", base_url="http://localhost:8080/v1", backend_model="switchboard")` |
| `transformers` | You want it in-process | `pip install 'switchboard-router[local]'`, then `Router(backend="transformers")` |

`Router()` with no backend uses Ollama if it's running, then Transformers if it's installed, and otherwise tells you what to install.

The default model is the 0.5B. The same code runs the larger sizes, which were tested on the same 10,000 prompts:

| Size | Runs on | `backend_model` |
|---|---|---|
| [0.5B](https://huggingface.co/switchboardllm/switchboard-router-0.5b) | any laptop CPU, or the browser | default |
| [4B](https://huggingface.co/switchboardllm/switchboard-router-4b) | Ollama on your own machine | `hf.co/switchboardllm/switchboard-router-4b-GGUF` |
| [32B](https://huggingface.co/switchboardllm/switchboard-router-32b) | vLLM on one 48 GB GPU | the name your vLLM server gives it |

The 32B sends the fewest hard prompts to a cheap model; the model cards have the numbers.

## Failing safe

If the backend can't be reached, errors out, or the model's output isn't valid JSON, the decision is `frontier` with `valid=False` and the reason in `decision.error`. A failed routing call costs you money, never quality. With `fail_safe=False`, those cases raise instead. Empty or non-string prompts always raise `ValueError` or `TypeError`.

Requests time out after 15 seconds by default (`Router(timeout=5)` to change it), and redirects are not followed, so an API key is never sent to another host.

## Limits

- The router reads the first 2,000 characters of a prompt. A question at the very end of a long prompt may be missed.
- It reads the prompt it's given, so a prompt can try to talk it into a cheaper tier. Don't use it as a security boundary.

## How good is the routing?

Switchboard v0.1 on 10,000 held-out prompts: 64.5% cheaper than sending every prompt to Claude Opus 5.5, and 5.5% of prompts were sent to a cheaper tier than they needed. Method, limits and the full results are in the [model card](https://huggingface.co/switchboardllm/switchboard-router-0.5b). Try it in your browser: [Switchboard demo](https://huggingface.co/spaces/switchboardllm/switchboard).

## License

Apache-2.0.
