Metadata-Version: 2.4
Name: llm-cascade
Version: 0.1.0
Summary: Free-first LLM routing for Python: local Ollama -> OpenRouter free models -> paid Anthropic API, with automatic fallback and a savings ledger.
Project-URL: Homepage, https://github.com/luandv92/llm-cascade
Project-URL: Repository, https://github.com/luandv92/llm-cascade
Project-URL: Issues, https://github.com/luandv92/llm-cascade/issues
Project-URL: Changelog, https://github.com/luandv92/llm-cascade/blob/main/CHANGELOG.md
Author: luandv92
License-Expression: MIT
License-File: LICENSE
Keywords: ai,anthropic,claude,cost-optimization,llm,ollama,openrouter,router
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Typing :: Typed
Requires-Python: >=3.10
Requires-Dist: tomli>=2.0.1; python_version < '3.11'
Provides-Extra: anthropic
Requires-Dist: anthropic>=0.40.0; extra == 'anthropic'
Provides-Extra: dev
Requires-Dist: pytest>=8.0; extra == 'dev'
Requires-Dist: ruff>=0.6; extra == 'dev'
Description-Content-Type: text/markdown

# llm-cascade

[![CI](https://github.com/luandv92/llm-cascade/actions/workflows/ci.yml/badge.svg)](https://github.com/luandv92/llm-cascade/actions/workflows/ci.yml)
[![PyPI](https://img.shields.io/pypi/v/llm-cascade.svg)](https://pypi.org/project/llm-cascade/)
[![License: MIT](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE)
[![Python versions](https://img.shields.io/pypi/pyversions/llm-cascade.svg)](https://pypi.org/project/llm-cascade/)

Free-first LLM routing for Python: local Ollama ($0) -> OpenRouter free models
($0) -> paid Anthropic API, with automatic fallback and a savings ledger.
Zero required dependencies.

## Why

Most teams route every prompt to the same frontier model regardless of how
hard the task actually is, and the API bill grows with usage instead of with
value delivered. llm-cascade flips that: it tries the cheapest tier that can
plausibly do the job first, and only escalates when that tier is unreachable
or genuinely too weak for the task. In production, that means classification,
extraction, and summarization run for $0 on a local model or a free cloud
model, while code review, planning, and long-form writing still get a
frontier model -- and if the paid API is ever down or out of credits, your
scheduled jobs degrade to a free tier instead of crashing.

## Install

```bash
pip install llm-cascade
```

The base package has **zero required third-party dependencies**. To use the
paid Anthropic tier, install the extra:

```bash
pip install "llm-cascade[anthropic]"
```

Without it, the local (Ollama) and free-cloud (OpenRouter) tiers still work
end to end -- you'll only see an error if a task actually routes to the paid
tier and the SDK isn't installed.

## Quickstart

```python
from llm_cascade import route

result = route("summarize", "Paste a long document here...")
print(result.text)        # the completion
print(result.tier)        # which tier served it: trivial/local/cheap/light/heavy
print(result.model)       # e.g. "local:ollama", "openrouter:qwen/...", "claude-sonnet-5"
print(result.cost_usd)    # 0.0 for local/free-cloud tiers
```

Or from the command line:

```bash
llm-cascade "Summarize this text: ..." --task summarize
llm-cascade --routes   # show the tier ladder and task -> tier table
llm-cascade --stats    # show cumulative cost and free-vs-paid call counts
```

## How routing works

Each call resolves a **task name** (or a tier name directly) to a **tier**,
then runs that tier's provider chain in order until one succeeds:

| Tier      | Chain                                            | Use for                                  |
|-----------|---------------------------------------------------|-------------------------------------------|
| `trivial` | local Ollama only ($0)                            | high-volume micro-tasks, never worth the network |
| `local`   | local Ollama ($0) -> OpenRouter free ($0) -> paid  | mechanical NLP: classify, extract, summarize, translate |
| `cheap`   | OpenRouter free ($0) -> paid                       | manual opt-in tier, invoke directly with `task="cheap"` |
| `light`   | paid (Sonnet); free-cloud only if `LLM_CASCADE_LIGHT_FREE=1` | customer-facing copy that needs reliable polish |
| `heavy`   | paid (Opus), adaptive thinking, high effort        | real reasoning: code, review, planning, long-form |

If the paid API call fails for *any* reason -- bad or revoked key, no
credits, a provider outage, a network blip -- llm-cascade degrades to the
free tiers instead of raising, so scheduled or automated callers don't break
outright. It only raises once every tier has been tried and failed.

The default task -> tier table:

```
classify, extract, summarize, translate, chat  -> local
rewrite                                        -> light
code, review, plan, creative                   -> heavy
```

Anything not in that table falls back to `default_task_tier` (`light` by
default). See `config/routes.example.toml` for the full, commented schema.

## Configuration

Configuration is resolved with this precedence (lowest to highest):

1. Built-in defaults.
2. An optional TOML file: `~/.llm_cascade/routes.toml`, or the path given in
   `LLM_CASCADE_CONFIG`.
3. Environment variables, for the handful of settings that make sense as env
   vars.

Copy [`config/routes.example.toml`](config/routes.example.toml) to
`~/.llm_cascade/routes.toml` to add your own task names, retune which tier
handles a task, or override model ids and pricing.

Environment variables:

| Variable | Purpose |
|---|---|
| `ANTHROPIC_API_KEY` | Enables the paid tier (`light`/`heavy`, and the final safety net). |
| `OPENROUTER_API_KEY` | Enables the free-cloud tier (OpenRouter `:free` models). |
| `LLM_CASCADE_CONFIG` | Path to a TOML config file, instead of `~/.llm_cascade/routes.toml`. |
| `LLM_CASCADE_LOG_PATH` | Path to the savings ledger, instead of `~/.llm_cascade/log.jsonl`. |
| `LLM_CASCADE_LIGHT_FREE` | Set to `1` to try free-cloud first on the `light` tier too. |
| `LLM_CASCADE_NO_FREE_CLOUD` | Set to any value to disable the free-cloud tier globally. |
| `LLM_CASCADE_HTTP_REFERER` / `LLM_CASCADE_APP_TITLE` | Sent as `HTTP-Referer` / `X-Title` on OpenRouter requests (optional; omitted if unset). |
| `LLM_CASCADE_OPENROUTER_MODEL` / `LLM_CASCADE_OPENROUTER_MODELS` | Override the free model (singular) or ordered list (comma-separated) tried on OpenRouter. |
| `LLM_CASCADE_OLLAMA_BASE_URL` | Ollama server URL (default `http://localhost:11434`). |

See [`.env.example`](.env.example) for a copy-pasteable starting point.

## Savings ledger

Every call -- free or paid -- appends one line to a JSONL ledger
(`~/.llm_cascade/log.jsonl` by default). `stats()` / `llm-cascade --stats`
summarize it:

```bash
$ llm-cascade --stats
{
  "paid_calls": 3,
  "free_calls": 41,
  "total_cost_usd": 0.0842,
  "by_model": {
    "local:ollama": 30,
    "openrouter:qwen/qwen3-next-80b-a3b-instruct:free": 11,
    "claude-sonnet-5": 3
  }
}
```

## Comparison

Unlike heavier multi-provider routers, llm-cascade ships with zero required
dependencies and a single opinionated cascade -- it's a routing policy you can
read in one file, not a platform.

## Contributing

Issues and pull requests are welcome at
[github.com/luandv92/llm-cascade](https://github.com/luandv92/llm-cascade).
Before opening a PR:

```bash
pip install -e ".[dev,anthropic]"
ruff check .
pytest tests/ -v
```

Tests never touch real networks -- Ollama, OpenRouter, and the Anthropic SDK
are all mocked (see `tests/conftest.py`).

## License

MIT -- see [LICENSE](LICENSE).
