Metadata-Version: 2.4
Name: litellm-routing-testbed
Version: 0.1.0
Summary: Local proof harness for LiteLLM context-window routing
Author-email: Dark Light <darklight@noreply.com>
Maintainer-email: Dark Light <darklight@noreply.com>
License-Expression: MIT
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: fastapi==0.140.13
Requires-Dist: httpx==0.28.1
Requires-Dist: litellm[proxy]==1.94.0
Requires-Dist: PyYAML==6.0.3
Requires-Dist: tiktoken==0.13.0
Requires-Dist: uvicorn==0.51.0
Provides-Extra: dev
Requires-Dist: build==1.5.0; extra == "dev"
Requires-Dist: mypy==2.3.0; extra == "dev"
Requires-Dist: pytest==9.1.1; extra == "dev"
Requires-Dist: ruff==0.16.0; extra == "dev"
Requires-Dist: types-PyYAML==6.0.12.20260724; extra == "dev"
Dynamic: license-file

# LiteLLM Routing Testbed

This package proves LiteLLM's context-window-aware routing locally on macOS
without starting real vLLM instances. Two FastAPI dummy backends expose the
OpenAI Chat Completions API, count every request with LiteLLM's own public
token counter, and record one JSON object per received request.

## What it proves

- Instance A accepts at most 65,000 input tokens.
- Instance B accepts at most 256,000 input tokens.
- LiteLLM pre-call checks remove deployments whose `max_input_tokens` is too
  small before choosing a backend.
- The native mode load-balances requests that fit both deployments.
- The deterministic mode uses LiteLLM deployment `order` to prefer A and uses
  B only after A is removed by the context check.
- A request with 256,001 input tokens is rejected by LiteLLM without reaching
  either backend.

## Requirements

- Apple Silicon macOS
- Python 3.11 or newer
- Free local ports 4000, 4011, and 4012 by default

All services run on loopback and require no external API, database, Redis,
container runtime, or model weights. The package forces LiteLLM to use its
bundled model-cost map, so startup does not fetch the remote cost-map copy.

## Install

```bash
cd litellm-routing-testbed
python3 -m venv .venv
.venv/bin/pip install -e .
```

For development and verification:

```bash
.venv/bin/pip install -e '.[dev]'
```

All direct dependencies are exactly pinned in `pyproject.toml`.

## Run the native proof

The shortest complete run starts all services, executes the required target
matrix plus 12 small-request probes, and always shuts down:

```bash
.venv/bin/routing-testbed all --mode native
```

Run lifecycle stages separately when inspecting live services:

```bash
.venv/bin/routing-testbed up --mode native
.venv/bin/routing-testbed status --mode native
.venv/bin/routing-testbed test --mode native
.venv/bin/routing-testbed down --mode native
```

## Run deterministic small-first routing

```bash
.venv/bin/routing-testbed all --mode deterministic --small-probes 0
```

This mode adds `order: 1` to A and `order: 2` to B. The context check still
runs first, so requests through 65k select A and larger valid requests select
B.

## Custom ports and target sizes

Every command accepts the same network and runtime options:

```bash
.venv/bin/routing-testbed all \
  --small-port 5011 \
  --large-port 5012 \
  --proxy-port 5000 \
  --targets 10000,50000,64000,70000,120000,200000,255000,256001
```

When commands are run separately, repeat the same options so they address the
same state file and endpoints. Use `--logs-dir` and `--run-dir` to relocate
runtime artifacts.

## Endpoints

The two dummies expose:

- `GET /health`
- `GET /v1/models`
- `POST /v1/chat/completions`

LiteLLM listens on `http://127.0.0.1:4000` by default. The public test model is
`router-model`.

## Evidence

Runtime evidence remains after `down`:

- `logs/native-instance_A_65k.jsonl`
- `logs/native-instance_B_256k.jsonl`
- `logs/deterministic-instance_A_65k.jsonl`
- `logs/deterministic-instance_B_256k.jsonl`
- `logs/instance_A_65k.log`
- `logs/instance_B_256k.log`
- `logs/litellm.log`
- `logs/test-results-native.json`
- `logs/test-results-deterministic.json`

Each backend JSONL row contains the UTC timestamp, instance, target port,
client-reported target tokens, independently counted input tokens,
`max_tokens`, model, and request ID. The assistant response content also
contains `handled_by` and `input_tokens`, which the client uses for its
pass/fail table. Mode-scoped filenames preserve native and deterministic
evidence when the two runs are executed consecutively.

## Cleanup

`all` calls `down` in a `finally` block. If a terminal is interrupted during a
separate run, execute:

```bash
.venv/bin/routing-testbed down
```

The process manager checks stored command signatures before signaling PIDs, so
a stale state file cannot cause it to terminate an unrelated reused PID.

See [ROUTING.md](ROUTING.md) for the verified LiteLLM fields and routing
semantics.
