Metadata-Version: 2.4
Name: relay-gateway
Version: 0.1.0
Summary: Smart AI Gateway — transparent LLM proxy with routing, savings, and quality evidence
Project-URL: Homepage, https://relay-gateway.fly.dev
Project-URL: Repository, https://github.com/pulkyeet/relay
Requires-Python: >=3.12
Requires-Dist: aiosqlite>=0.22.1
Requires-Dist: alembic>=1.14.0
Requires-Dist: fastapi>=0.115.0
Requires-Dist: httpx>=0.28.0
Requires-Dist: jinja2>=3.1.6
Requires-Dist: opentelemetry-api>=1.28.0
Requires-Dist: opentelemetry-exporter-otlp-proto-http>=1.28.0
Requires-Dist: opentelemetry-instrumentation-fastapi>=0.49b0
Requires-Dist: opentelemetry-instrumentation-httpx>=0.49b0
Requires-Dist: opentelemetry-sdk>=1.28.0
Requires-Dist: pydantic-settings>=2.5.0
Requires-Dist: pydantic>=2.9.0
Requires-Dist: pyyaml>=6.0.2
Requires-Dist: sqlalchemy[asyncio]>=2.0.0
Requires-Dist: uvicorn[standard]>=0.32.0
Provides-Extra: postgres
Requires-Dist: asyncpg>=0.30.0; extra == 'postgres'
Description-Content-Type: text/markdown

# Relay — Smart AI Gateway

A self-hosted gateway that sits between any LLM-using app (opencode, custom scripts,
your terminal) and OpenRouter. Routes each request to the cheapest model that can
handle it, logs the cost, and proves the savings against a stated baseline with a
published quality delta from a benchmark of real traffic.

---

## Quickstart (5 minutes)

```bash
# 1. Install + init
uv tool install git+https://github.com/pulkyeet/relay
relay init

# 2. Add your OpenRouter key
# Edit ~/.relay/relay.yaml, paste your key under openrouter.api_key (or set
# the OPENROUTER_API_KEY environment variable).

# 3. Start the server
relay run

# 4. Use it
open http://127.0.0.1:8000/health
# → {"status": "ok", "version": "0.1.0"}
```

> The distribution is named `relay-gateway` on PyPI (`relay` was taken); the
> commands it installs are `relay` and `relay-benchmark`. Once a release is
> published, `uv tool install relay-gateway` works too.

---

## opencode integration

Add a custom provider to `~/.config/opencode/opencode.jsonc`:

```jsonc
{
  "provider": {
    "relay": {
      "npm": "@ai-sdk/openai-compatible",
      "name": "Relay Gateway",
      "options": {
        "baseURL": "http://127.0.0.1:8000/v1"
      },
      "models": {
        "cheap":  { "name": "Cheap (alias → ling-2.6-flash)" },
        "smart":  { "name": "Smart (alias → deepseek-v4-pro)" },
        "baseline":  { "name": "Baseline (alias → deepseek-v4-flash)" }
      }
    }
  }
}
```

Then inside opencode:
1. Run `/connect`
2. Search for "relay" — it's the custom provider you just added
3. Paste your gateway key (the `key` from `relay.yaml`)

Now when you run `/models`, you see the Relay models. Choose "Cheap" for the
cheapest tier, "Smart" for quality-critical tasks, or "Baseline" to use the
counterfactual reference model directly.

> **Note:** Traffic through the Relay provider uses your OpenRouter API key
> (you pay per-token to OpenRouter). Your opencode-go subscription is a
> separate path — choose the "Relay" provider when you want routing + savings,
> or use the default opencode-go provider when you don't.

---

## CLI reference

| Command | What it does |
|---------|------------|
| `relay init` | Writes starter config to `~/.relay/`, generates a gateway key, creates the SQLite DB, prints the opencode snippet |
| `relay run` | Starts the gateway server on `127.0.0.1:8000`. Flags: `--host`, `--port`, `--config` |
| `relay migrate` | Runs Alembic migrations (Postgres only; SQLite auto-creates tables on startup) |
| `relay-benchmark sample` | Draws 20% of recent request_log traffic into `benchmark_case` rows (stratified by category) |
| `relay-benchmark run` | Executes each benchmark case through BOTH the routed path and the baseline model. **Costs money.** Use `--dry-run` first. |
| `relay-benchmark judge` | Calls an LLM judge (default: `gpt-4o`) to score routed vs baseline answers. Computes Cohen's κ against your hand-labels. Blocks if κ < 0.7. |
| `relay-benchmark annotate` | Walks you through each tuning-split case, shows both answers, prompts for a verdict (`r/b/t/s/q`). Writes to `~/.relay/judge_handlabels.yaml`. |
| `relay-benchmark ratios` | Derives per-category output-length calibration ratios from a judged run. Writes `calibration_ratio` rows. |
| `relay-benchmark status` | Shows the latest benchmark run's quality delta, κ, and methodology. |

---

## The savings story

### How it works

1. **Route smart**: the gateway checks each request against your tier rules and
   picks the cheapest model that can handle it. Every decision is logged with
   its reason (`X-Relay-Route-Reason` header).

2. **Measure honestly**: every request logs input/output tokens, actual cost
   (from OpenRouter prices), the chosen model, and the counterfactual cost if
   the baseline model had served it instead.

3. **Publish the number**: the dashboard shows cumulative money saved, savings
   this week (broken down by cheap-routing vs cache hits vs failovers), and a
   quality delta from the latest benchmark run — all backed by a stated
   methodology.

### Honest accounting

Relay never hides bad news. The formula `savings = counterfactual - actual`
produces:

- **Positive** → you saved money vs always using the baseline
- **Negative** → a failover or error cost more than the baseline would have.
  Shown in red on the dashboard, counted in the aggregate.
- **Zero** → cache hit with a known baseline → full save displayed.
- **NULL** → the baseline model isn't in the price cache, so we can't compute
  savings. Dashboard shows "—" (unknown), never "$0.00" — that would be a lie.

### Naive vs calibrated

- **Naive** (default): assumes the baseline would have produced the same number
  of output tokens as the routed model did. Labeled as an estimate.
- **Calibrated**: applies per-category output-length ratios learned from a
  benchmark run. The formula becomes `baseline_in_price × input_tokens +
  baseline_out_price × (output_tokens × ratio)`. Labeled as a measurement.

---

## The benchmark workflow

Phase 5 turned the savings number from an estimate into a measurement. Here's
the workflow to fire the benchmark and get the published quality delta:

```bash
# 1. Dogfood: run in passthrough mode for a few days to capture real traffic
#    In relay.yaml: mode: passthrough

# 2. Sample 20% of logged traffic
relay-benchmark sample --seed 0

# 3. Dual-path run (costs real money — preview with --dry-run)
relay-benchmark run --dry-run
relay-benchmark run --concurrency 4

# 4. Score with the LLM judge
relay-benchmark judge --run-id 1

# 5. Hand-label cases for κ calibration
relay-benchmark annotate --run-id 1

# 6. Re-run judge now that hand-labels exist
relay-benchmark judge --run-id 1

# 7. Compute calibration ratios
relay-benchmark ratios --run-id 1

# 8. Flip methodology in relay.yaml:
#    savings:
#      methodology: calibrated
#      calibration_run_id: 1
#    Restart: relay run
```

The dashboard hero now shows the real quality delta with the methodology tag,
and the digest one-liner includes it.

---

## Current model lineup

| Tier | Primary | Sibling (failover) |
|------|---------|-------------------|
| cheap | `inclusionai/ling-2.6-flash` | `nex-agi/nex-n2-mini` |
| smart | `deepseek/deepseek-v4-pro` | `xiaomi/mimo-v2.5-pro` |
| baseline | `deepseek/deepseek-v4-flash` | `xiaomi/mimo-v2.5` |

Baseline for the counterfactual is `models[0]` of the baseline tier:
`deepseek/deepseek-v4-flash`.

---

## Configuration reference

All config lives in `~/.relay/relay.yaml` (or wherever `RELAY_CONFIG` points).

| Key | Default | Description |
|-----|---------|-------------|
| `gateway_keys` | `[]` | List of `{name, key, spend_cap_usd}` pairs |
| `openrouter.api_key` | `""` | OpenRouter API key (or `OPENROUTER_API_KEY` env) |
| `openrouter.base_url` | `https://openrouter.ai/api/v1` | — |
| `database.url` | `sqlite+aiosqlite:///~/.relay/relay.db` | SQLite default; `postgresql+asyncpg://...` for Postgres |
| `mode` | `normal` | `passthrough` for dogfooding |
| `savings.baseline_model` | `""` | The model you'd always have used (budget baseline) |
| `savings.methodology` | `naive` | `calibrated` after a benchmark run |
| `savings.calibration_run_id` | `null` | `benchmark_run.id` to read ratios from |
| `savings.digest_webhook_url` | `""` | Optional POST-to endpoint for the weekly one-liner |
| `benchmark.sample_fraction` | `1.0` | Fraction of traffic drawn into the benchmark (1.0 = use all) |
| `benchmark.min_sample_size` | `5` | Floor on sample size at low traffic |
| `benchmark.judge_model` | `openai/gpt-4o` | Model used as the LLM judge |
| `benchmark.kappa_threshold` | `0.7` | Minimum Cohen's κ to publish a quality claim |
| `dashboard.admin_token` | `""` | Password-protect `/` and `/digest` with Bearer auth |
| `log_payloads` | `false` | Store request/response bodies (auto-forced in passthrough mode) |
| `prices` | `{}` | Static price overrides (keyed by model id, `{input, output}` in USD) |

---

## Architecture notes

- **Single-user, self-hosted, no network calls.** Everything runs on your local
  machine. The only outbound calls are to OpenRouter (and optionally the digest
  webhook).
- **OpenRouter only.** No Anthropic adapter, no opencode-go provider — just the
  OpenAI-compatible endpoint forwarded to OpenRouter.
- **SQLite by default (sidecar mode).** No Postgres needed. Tables auto-create on
  startup. For production with multiple users, the Postgres option exists.
- **No Docker, no cloud deploy.** `uvx relay init && relay run` is the only
  install path.
- **No learned classifier router** — the deterministic rules router ships and
  the benchmark gives you the evidence it works. A learned router trained on
  judge labels is the documented upgrade path (see: Maybe-Pile), but is not
  built.

---

## Maybe-Pile (documented, not built)

- **Learned router:** classifier trained on judge labels. The flywheel is
  `traffic → judge labels → better router`. Not built; documented as the
  architecture's upgrade path once traffic volume justifies it.
- **Shadow sampling:** continuously re-run a slice of production traffic against
  the baseline. The benchmark runner contains most of this code; wiring it to
  production traffic is a future step.
- **Semantic cache:** higher hit rate than exact-match, but hot-path cost and
  wrong-answer risk.
- **Anthropic-format endpoint** (`/v1/messages`): not needed for single-user
  opencode + OpenRouter usage; explicitly out of scope.

---

## Known limitations

- Single-user design. No multi-tenant, no rate limiting, no team dashboards.
- Gateway key auth only — no SSO, no OAuth, no API-key rotation.
- Dashboard auth is a shared secret (`dashboard.admin_token`), not per-user.
  Empty on localhost by default.
- The benchmark's dual-path run costs real money on OpenRouter (~$5–30 for a
  full 200-case run). The judge model call also costs money (~$2–5 for 200
  cases with `gpt-4o`).
- Streaming response bodies are reassembled from SSE deltas with a 64KB cap.
  Very long responses (10K+ tokens of output) get truncated with a `_truncated`
  flag.
