Metadata-Version: 2.5
Name: llm-router-gate
Version: 1.0.0
Summary: A CLI and SDK for routing LLM calls and gating according to price.
License: MIT License
        
        Copyright (c) 2026 Adrian Ernesto Radillo
        
        Permission is hereby granted, free of charge, to any person obtaining a copy
        of this software and associated documentation files (the "Software"), to deal
        in the Software without restriction, including without limitation the rights
        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
        copies of the Software, and to permit persons to whom the Software is
        furnished to do so, subject to the following conditions:
        
        The above copyright notice and this permission notice shall be included in all
        copies or substantial portions of the Software.
        
        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
        SOFTWARE.
License-File: LICENSE
Requires-Python: >=3.14
Requires-Dist: anthropic[aiohttp]>=0.122.0
Requires-Dist: httpx>=0.28.1
Requires-Dist: python-dotenv>=1.2.3
Requires-Dist: rich>=15.0.0
Description-Content-Type: text/markdown

# llm-router-gate
## Installation
### Install globally to use as a CLI
```{bash}
uv tool install llm-router-gate
```
### Or install within your project to use as an SDK
```{bash}
uv add llm-router-gate
```

### Ephemeral call
You can also invoke the CLI from PyPI without any local installation:
```{bash}
uv x llm-router-gate --help
```
## What it is for

`llm-router-gate` is a **single-shot, single-turn LLM probe with the HTTP layer left
visible**. It sends exactly one user message to exactly one model, streams the
answer, and then prints everything a developer normally has to reach for a proxy
or `curl -v` to see:

- the outgoing request line, **all** headers, and the pretty-printed JSON body;
- the response status line and **all** response headers (rate-limit counters,
  `request-id`, `cf-ray`, `content-type: text/event-stream`, …);
- the raw SSE lines as they arrive (`RAW STREAM: data: {...}`);
- a reconstructed *final* response payload, with generated text truncated so the
  structure stays readable;
- a token/cost report, with the **provenance** of every number it prints.

It is deliberately **not** a chat client. There is no conversation history, no
system prompt, no tool use, no retries. Think of it as the LLM equivalent of
`curl -v` plus a calculator: use it to answer "what exactly goes over the wire,
what exactly comes back, and what did that cost?"

Good uses:

- comparing how three backends (Anthropic, llama.cpp, OpenRouter) frame the same
  prompt and the same concepts (reasoning, usage, finish reasons);
- checking that a local llama.cpp build honours a body field you just added;
- getting a hard cost number for a prompt before wiring it into a loop;
- debugging "the model returned nothing" — the raw stream dump usually shows the
  text arrived on a channel your client wasn't reading.

## Requirements

The project uses **uv**; run everything through it so the lockfile is respected:

```bash
uv run src/router.py "..." [options]
```

Credentials are read from the environment, with `.env` loaded automatically via
`python-dotenv` (`load_dotenv()` at import time):

| Provider     | Needs                                             |
| ------------ | ------------------------------------------------- |
| `anthropic`  | `ANTHROPIC_API_KEY` (read by the `anthropic` SDK) |
| `openrouter` | `OPENROUTER_API_KEY`                              |
| `local`      | nothing — a llama.cpp server on port 8090         |

Note that the request logger prints **every** header, including `x-api-key` and
`Authorization`. That is the point of the tool, but it means: do not paste raw
output into an issue, a PR, or a chat window without redacting those two lines
first. (The transcript above has them replaced with `XXXXX` by hand.)

## CLI reference
After installing with `uv tool install`, you may run the command directly from
any terminal window:
```
llm-router-gate PROMPT [-p {anthropic,local,openrouter}] [-m MODEL]
                         [-t MAX_TOKENS] [--local-url URL]
                         [--reasoning | --no-reasoning] [-y]
```

| Flag                             | Default                              | Meaning                                                                                                                                       |
| -------------------------------- | ------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------- |
| `PROMPT` (positional)            | —                                    | Sent verbatim as a single `{"role": "user"}` message. Quote it.                                                                                |
| `-p`, `--provider`               | `anthropic`                          | Which backend to hit. Selects the code path *and* the cost behaviour.                                                                          |
| `-m`, `--model`                  | per-provider                         | **Required** for `anthropic` (argparse errors out otherwise). Defaults to `ggml-org/SmolLM3-3B-GGUF:Q4_K_M` for `local`, `z-ai/glm-5.2:free` for `openrouter`. |
| `-t`, `--max-tokens`             | `1024`                               | Output cap. For Anthropic it is *also* the worst case in the pre-flight cost estimate.                                                          |
| `--local-url`                    | `http://localhost:8090/v1/chat/completions` | Point at a different llama.cpp / vLLM / LM Studio instance.                                                                             |
| `--reasoning` / `--no-reasoning` | *omitted* → provider default         | Tri-state thinking toggle. See below.                                                                                                          |
| `-y`, `--yes`                    | off                                  | Skip the `Proceed with API call? (y/N)` gate. Use in scripts; keep it off when experimenting against a paid model.                              |

### Examples

```bash
# Paid call, with a confirmation gate and a real cost report.
llm-router-gate "Explain entropy" -m claude-haiku-4-5 -t 500

# Local llama.cpp, defaults to SmolLM3-3B.
llm-router-gate "Explain entropy" -p local -t 200 -y

# Same, thinking forced off.
llm-router-gate "Say hello" -p local -t 100 -y --no-reasoning

# OpenRouter, thinking forced on.
llm-router-gate "Explain entropy" -p openrouter -m z-ai/glm-5.2:free --reasoning

# A different local server.
llm-router-gate "Hi" -p local --local-url http://localhost:1234/v1/chat/completions -y
```

## Two code paths, on purpose

`main()` dispatches to one of two coroutines. They are kept separate because the
two wire protocols disagree about almost everything except "it's a POST".

### `run_anthropic()` — the SDK path

Uses `anthropic.AsyncAnthropic` with an **injected** `httpx.AsyncClient` carrying
`event_hooks`. That injection is the whole trick: the SDK does the auth, framing,
SSE parsing and retry logic, while the hooks still let you see the raw traffic.
It's the pattern to copy whenever you need observability without reimplementing a
vendor SDK.

Sequence:

1. `GET /v1/models` — resolves the model's `display_name` and proves the key works.
2. `POST /v1/messages/count_tokens` — the **exact** input token count, including
   framing overhead. Falls back to the `~4 chars/token` heuristic if it fails.
3. Prints a *pessimistic* estimate: `exact_input × in_rate + max_tokens × out_rate`,
   i.e. the cost if the model uses its entire budget. Then the y/N gate.
4. Streams via `client.messages.stream()` / `stream.text_stream`.
5. `get_final_message()` → dumps the payload, then reports **actual** usage and
   cost from `usage.input_tokens` / `usage.output_tokens`.

Pricing comes from `FAMILY_PRICING_TIERS`, matched by substring on the lowercased
model id (`opus` / `sonnet` / `haiku`), defaulting to Sonnet rates. These are
**hardcoded base rates in $/1M tokens** and are the part of this script most
likely to be stale — they ignore long-context surcharges, batch discounts, and
cache read/write rates. Treat the cost figure as an order of magnitude, and
update `FAMILY_PRICING_TIERS` when Anthropic's price list moves.

Extended thinking is **not** wired on this path: `--reasoning` is silently
ignored for `-p anthropic` because no `thinking` block is ever sent.

### `run_openai_compatible()` — the hand-rolled path

One `httpx.AsyncClient.stream("POST", ...)` and a hand-written SSE reader, shared
by `local` and `openrouter`. Differences between the two are gated on
`is_local`:

|                        | `local` (llama.cpp)                              | `openrouter`                     |
| ---------------------- | ------------------------------------------------ | -------------------------------- |
| Auth                   | none                                             | `Authorization: Bearer $OPENROUTER_API_KEY` |
| Usage in stream        | not requested                                    | `stream_options.include_usage: true` |
| Reasoning toggle field | `chat_template_kwargs.enable_thinking`           | `reasoning: {"enabled": bool}`   |
| Token counts           | `timings.predicted_n` / `prompt_n` on last chunk | `usage.completion_tokens` / `prompt_tokens` |

The reader loop: skip anything not prefixed `data:`, break on `[DONE]`, `json.loads`
the rest, keep the last chunk as `final_chunk`, and latch `chunk["usage"]` whenever
it appears.

## The reasoning/answer two-channel model

This is the subtlety that motivated most of the current code, and the thing to
understand before extending it.

An OpenAI-compatible `delta` can carry generated text on **more than one field**,
and different servers name them differently:

```
delta.reasoning_content  ──→ reasoning accumulator   (llama.cpp)
delta.reasoning          ──→ reasoning accumulator   (OpenRouter)
delta.content            ──→ answer accumulator      (everyone)
choice.text              ──→ answer accumulator      (legacy completions-style)
```

They are accumulated into two separate lists and rendered differently:
**reasoning is printed dimmed, the answer is printed at normal brightness.** The
final payload reflects both — `message.content` is the answer, and a
`message.reasoning` key is added only if a reasoning trace was actually received.

Two consequences worth internalising:

1. A client that reads only `delta.content` will report an **empty response** from
   a thinking model, even though the server streamed hundreds of tokens. That is
   not a server bug and not an empty generation; it is a channel mismatch.
2. **Reasoning tokens are charged against `max_tokens`.** A thinking model given
   `-t 10` can burn the entire budget mid-thought and terminate with
   `finish_reason: "length"` before emitting a single answer token. If you get an
   empty answer from `-p local`, raise `-t` (try 100+) or pass `--no-reasoning`
   before suspecting anything else.

### `--reasoning` semantics

Implemented with `argparse.BooleanOptionalAction` and `default=None`, giving three
distinct states rather than the usual two:

| Invocation        | `args.reasoning` | Behaviour                                        |
| ----------------- | ---------------- | ------------------------------------------------ |
| omitted           | `None`           | **No field is sent.** The provider/model default stands. |
| `--reasoning`     | `True`           | Force thinking on.                               |
| `--no-reasoning`  | `False`          | Force thinking off.                              |

The tri-state matters: a plain `store_true` flag cannot express "don't express an
opinion", and silently sending `false` is not the same request as sending nothing.
The active state is echoed in the pre-request panel as
`Reasoning: provider default | on | off`.

For `local`, the flag becomes `chat_template_kwargs: {"enable_thinking": ...}`.
llama.cpp forwards `chat_template_kwargs` into the Jinja chat template, and
SmolLM3's template reads `enable_thinking`. This is therefore a
**template-level** knob: against a GGUF whose template does not reference that
variable, the field is accepted and quietly does nothing. If the toggle appears
to have no effect, inspect the model's chat template before blaming the client.

## Token reporting and its provenance

Non-Anthropic providers get no cost figure at all — only counts, and every panel
names its `Source:` so you never have to guess how solid a number is. Resolution
order in `run_openai_compatible()`:

1. **`provider usage block`** — `usage.prompt_tokens` / `usage.completion_tokens`
   from the stream. Authoritative. This is the OpenRouter path.
2. **`llama.cpp timings block`** — `final_chunk["timings"]["predicted_n"]` and
   `prompt_n`. Authoritative, and the normal local path.
3. **`heuristic ~4 chars/token`** — `len(text) // 4`. Last resort only.

Output tokens are labelled **"Output Tokens (generated, reasoning included)"**
because `predicted_n` counts everything the model produced, reasoning and answer
alike. Never derive this number from `len(full_content)`: with a thinking model
that undercounts massively — an answer-less run would report `1` output token
against a real `predicted_n` of 10.

The input count in the pre-request panel is *always* the crude heuristic for
these providers (there is no local `count_tokens` endpoint); the post-request
panel replaces it with a real number when the server supplied one.

## Extending it

- **Truncation.** `RESPONSE_TEXT_PREVIEW_CHARS = 50` controls how much generated
  text survives into the final-payload dump. `print_final_payload` deep-copies
  via `json.dumps`/`loads` before truncating, so the live objects are untouched;
  it handles both the OpenAI shape (`choices[0].message.content` / `.reasoning`)
  and the Anthropic shape (`content[]` blocks with a `text` key). Full text is
  still streamed to stdout — only the JSON echo is shortened.
- **The `RAW STREAM:` lines** are unconditional in the read loop. They are the
  most useful thing in the output when diagnosing a new server, and the most
  noisy otherwise. Gate them behind a `--raw` flag if that becomes annoying.
- **Error handling is intentionally flat.** Both paths catch broad `Exception`,
  print it red, and return; there are no retries and no non-zero exit on API
  failure. Fine for a probe, wrong for anything automated — add explicit exit
  codes before putting this in a pipeline.
- **A new OpenAI-compatible backend** usually needs only a URL, an auth header
  and possibly a new reasoning field name. Add it to the `is_local` branches
  rather than forking the reader; the dual-channel accumulator already covers
  every delta shape seen so far.
- **Wiring Anthropic extended thinking** means adding a `thinking={"type":
  "enabled", "budget_tokens": N}` argument in `run_anthropic()` and consuming
  `thinking` content blocks from the stream, so `--reasoning` stops being a no-op
  there. Currently unimplemented.
