Metadata-Version: 2.4
Name: oq-ai-router
Version: 0.1.0
Summary: Provider-agnostic LLM routing: free-first fallback chains, host-supplied policy, per-tenant keys. Transport by LiteLLM.
Author-email: Tushar Bansal <btushar@gmail.com>
License-Expression: AGPL-3.0-or-later
Project-URL: Homepage, https://github.com/orbitqube-tech/oq-ai-router
Keywords: llm,routing,fallback,litellm,free-tier,multi-provider
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: litellm>=1.93.0
Provides-Extra: dev
Requires-Dist: pytest>=8; extra == "dev"
Dynamic: license-file

# oq-ai-router

Provider-agnostic LLM routing for Python. Free-first fallback chains, a routing
policy supplied by *your* application, and per-tenant credentials — with the
actual transport delegated to [LiteLLM](https://github.com/BerriAI/litellm).

Extracted from a production application rather than designed in the abstract.

## What this is, and what it deliberately is not

The hard-won lesson behind this package: **most of an "AI layer" is already
solved, and the part that isn't is the part specific to your application.**

| Concern | Owner |
|---|---|
| The HTTP call, per-provider message shapes | LiteLLM |
| Fallback execution, retries, cooldowns | LiteLLM |
| Exception normalization across providers | LiteLLM |
| **Which models to try, in what order, for what kind of work** | **your app, via a `RoutingPolicy`** |
| **Free-tier-first ordering** | **your app** — no library knows which providers *you* have free access to |
| **Reading messy JSON out of free models** | this package |
| **What to do when there is no key at all** | your app |

So this is a thin policy layer, not a gateway and not a framework. It ships
**no routing table of its own** — a library that hardcodes one app's taxonomy is
not reusable, and adopting it would mean editing the library.

## Install

```bash
pip install oq-ai-router
```

## Use

Define your application's own categories and model preferences, register the
policy once at startup, then call:

```python
import oq_ai_router
from oq_ai_router import AiConfig, TablePolicy, complete_json

MY_POLICY = TablePolicy(
    # Free sources first, paid last. This ORDER is the free-first mechanism.
    provider_order=["omniroute", "openrouter", "groq", "anthropic"],
    models={
        "omniroute":  {"chat": ["auto/offline", "auto/cheap"]},
        "openrouter": {"chat": ["openrouter/free"]},
        "groq":       {"chat": ["llama-3.3-70b-versatile"]},
        "anthropic":  {"chat": ["claude-3-5-haiku-latest"]},
    },
)
oq_ai_router.set_default_policy(MY_POLICY)

result = complete_json(
    "chat",
    [{"role": "user", "content": "Reply as JSON: {\"ok\": true}"}],
    config=AiConfig(keys={"groq": "gsk-..."}),
)
result["data"]        # the parsed object
result["model_used"]  # which model actually answered
```

Categories are yours. One app's are `explain | suggest | audit | concierge`,
another's are `vision | json | prose`; both route through this package with no
edits to it. If table-driven ordering cannot express what you need (latency
budgets, freshness windows, live quota telemetry), implement the two-method
`RoutingPolicy` protocol directly.

### Per-tenant keys

`AiConfig` is injected by the host — the package never reads env, a database or
app settings on its own. Multi-tenant hosts pass a different `AiConfig` per
request; `env_config()` is the zero-config path for single-tenant use.

## Zero-credential operation

Providers may declare `requires_key=False` for a **local gateway that holds the
upstream credentials itself**. The bundled `omniroute` entry does this, so a
fresh self-hoster running [OmniRoute](https://github.com/diegosouzapw/OmniRoute)
gets a working AI chain with **no API keys at all**.

Verified 2026-07-24: a completion through `oq_ai_router` → LiteLLM Router →
a keyless OmniRoute returned successfully.

> **Do not** attempt to route a consumer AI *subscription* (Claude Pro/Max,
> ChatGPT Plus, Gemini One) through this or any proxy. Those OAuth flows are
> scoped to each vendor's own first-party clients; reusing them breaches the
> consumer terms and risks the account. Use metered API keys (every major
> provider has a free tier) or a genuinely free lane.

## Design notes worth knowing

- **A Router is cached per (chain, credentials).** LiteLLM's `Router` expects a
  static deployment list, but per-tenant keys resolve per request. Cache keys
  are hashed, so a credential cannot leak through a dict repr or a traceback.
- **Cooldowns are per Router**, therefore shared only between tenants on the
  same credentials — which is the correct boundary, since separate keys have
  separate quotas.
- **Per-provider timeouts.** A direct paid API should fail fast so the chain
  moves on; a local gateway draining free quota across upstreams legitimately
  takes longer. Measured, not guessed.
- **JSON validity is handled here, not by LiteLLM**, which has no concept of
  "returned 200, body would not parse". Free models emit think-tags, code
  fences and trailing commas; `parse_lenient` copes, and unparseable output
  falls through to the next model.

## Evals and cost guardrails

Model ids rotate and free models vary wildly, so "which model for which
category" is an empirical question. `oq_ai_router.evals` answers it by measurement:
run a corpus across every model in a chain and take **the cheapest model that
still passes** — accuracy, cost and latency scored together.

**Two corpora, both first-class.** A clean synthesized corpus measures the happy
path; the failures that actually matter arrive wearing real client data that can
never be committed. So:

| | Lives | Committed | Defines |
|---|---|---|---|
| **public** | `evals/corpus/public/` | yes | the contract — what we promise |
| **private** | outside the repo, via `OQ_AI_ROUTER_PRIVATE_CORPUS` | never | reality — what actually broke |

Both run through one schema and one scorer, and results are reported
**separately**: a green public score beside a red private score means the
contract holds but reality has moved, which is the most actionable signal here.
Private cases never expose their text or id in a report — only the failure kind.

**Closing the loop** is the point. When a private case fails, derive a
synthesized twin with `synthesize_case`, read it yourself, and commit it to the
public corpus. Over time the public corpus accumulates the *shape* of every real
failure without absorbing a byte of real data. Redaction handles the identifier
formats it knows (PAN, GSTIN, CIN, email, phone) and is a starting point for
human review, **never a security boundary**.

`oq_ai_router.budget` adds the spend ceilings: a per-request cap (pushed upstream
as `X-OmniRoute-Budget` where the gateway enforces it) and a per-tenant ledger
for the slow bleed no single request would trip. Cost is recorded even when a
cap is breached — money spent is spent, and dropping it would let violations
repeat indefinitely.

## Licence

**Dual licensed**: AGPL v3 or later for community use, plus a proprietary
commercial licence for anyone who needs to keep modifications private or embed
this in a closed-source product. See [LICENSING.md](LICENSING.md).

Note the Affero clause: running a *modified* version as a network service
obliges you to offer its source to that service's users.

A CLA must be in place before any outside contribution is accepted, because
dual licensing only works while the copyright is wholly owned. Until then this
repository does not accept contributions.
