Metadata-Version: 2.4
Name: cre-router
Version: 0.1.0
Summary: Cluster, Route, Escalate: a cascaded framework for cost-aware LLM serving
Project-URL: Homepage, https://github.com/ymoslem/CRE-Router
Project-URL: Repository, https://github.com/ymoslem/CRE-Router
Project-URL: Issues, https://github.com/ymoslem/CRE-Router/issues
Author: Yasmin Moslem
License-Expression: Apache-2.0
License-File: LICENSE
Keywords: cascades,litellm,llm,model-serving,quality-estimation,routing,vllm
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Science/Research
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.10
Requires-Dist: anyio>=4.0
Requires-Dist: numpy>=1.26
Requires-Dist: pyyaml>=6.0
Requires-Dist: scikit-learn>=1.4
Requires-Dist: sentence-transformers>=3.0
Provides-Extra: dev
Requires-Dist: fastapi>=0.115; extra == 'dev'
Requires-Dist: httpx>=0.27; extra == 'dev'
Requires-Dist: pytest>=8.0; extra == 'dev'
Provides-Extra: eval
Requires-Dist: vllm>=0.19; extra == 'eval'
Provides-Extra: full
Requires-Dist: accelerate>=1.0; extra == 'full'
Requires-Dist: datasets>=3.0; extra == 'full'
Requires-Dist: fastapi>=0.115; extra == 'full'
Requires-Dist: httpx>=0.27; extra == 'full'
Requires-Dist: litellm>=1.60; extra == 'full'
Requires-Dist: torch>=2.4; extra == 'full'
Requires-Dist: transformers>=4.57; extra == 'full'
Requires-Dist: uvicorn>=0.30; extra == 'full'
Requires-Dist: vllm>=0.19; extra == 'full'
Provides-Extra: qe
Requires-Dist: accelerate>=1.0; extra == 'qe'
Requires-Dist: datasets>=3.0; extra == 'qe'
Requires-Dist: torch>=2.4; extra == 'qe'
Requires-Dist: transformers>=4.57; extra == 'qe'
Provides-Extra: serve
Requires-Dist: fastapi>=0.115; extra == 'serve'
Requires-Dist: httpx>=0.27; extra == 'serve'
Requires-Dist: litellm>=1.60; extra == 'serve'
Requires-Dist: torch>=2.4; extra == 'serve'
Requires-Dist: transformers>=4.57; extra == 'serve'
Requires-Dist: uvicorn>=0.30; extra == 'serve'
Description-Content-Type: text/markdown

# CRE-Router

Implementation of the paper [Cluster, Route, Escalate: Cascaded Framework for Cost-Aware LLM Serving](https://arxiv.org/abs/2606.27457).

Production LLM serving trades accuracy against cost. CRE-Router routes each
query to the most cost-effective model in a pool, then escalates low-quality
outputs to a stronger model:

- **Stage 1 (clustering-based routing).** Queries are embedded and clustered
  offline; each cluster is routed to the model that minimizes a cost-adjusted
  error score (Error + lambda * Cost), with lambda tuned once to satisfy a
  latency (TPOT) budget.
- **Stage 2 (quality-estimation cascade).** A lightweight ModernBERT
  classifier inspects each efficient-model output and escalates low-quality
  answers up an ordered ladder of stronger models.

Both stages train only on task-correctness labels obtainable from standard
benchmark evaluation; no extra annotation is required.

## Install

```bash
pip install "cre-router[full]"
```

`full` installs the whole pipeline on one machine. Narrower installs are
available through the `serve`, `qe`, and `eval` extras; see the
[installation guide](https://github.com/ymoslem/CRE-Router#installation).

## Usage

The workflow is driven by the `cre` CLI, one stage per step:

`cre cluster` → `cre evaluate` → `cre fit` → `cre qe-train` → `cre serve`

Run `cre <stage> --help` for options. Runnable end-to-end quickstarts (a
no-GPU routing-table demo and a full serving walkthrough) are in the
[README](https://github.com/ymoslem/CRE-Router#readme).

## Links

- **Source and documentation:** https://github.com/ymoslem/CRE-Router
- **Paper:** https://arxiv.org/abs/2606.27457
- **Reproducing the paper:** https://github.com/ymoslem/CRE-Router/blob/main/REPRODUCE.md

Apache-2.0 licensed.
