📚 costhelm · Help

This service routes LLM calls across three paid providers plus a local Ollama, with automatic failover, rate-limit tracking, persistent logging, and a manual override. The free tiers this gateway grew up on (GitHub Models, Groq, Cerebras, NVIDIA NIM) were retired in 2026-08 — GitHub Models answers 410 and the others' quotas kept breaking runs. Below: how to use it, what each provider offers, and how to get keys.

Using the gateway

Python (built-in client)

from client import LLM, ask

# one-shot
print(ask("Explain transformers in 3 lines"))

# explicit provider (shortcut keys: g, o, or, oa)
print(ask("hi", provider="or"))   # openrouter
print(ask("hi", provider="g"))    # gemini

# full client
llm = LLM()
r = llm.chat("hi", provider="gemini", max_tokens=128, temperature=0.5)
print(r["text"], r["provider"], r["latency_ms"])

# streaming
for chunk in llm.stream("count to 5"):
    print(chunk, end="", flush=True)

HTTP (OpenAI-style)

curl http://localhost:8111/v1/chat \
  -H "Content-Type: application/json" \
  -d '{"prompt": "hello", "provider": "g", "max_tokens": 128}'

Shortcut keys

The retired providers' shortcuts (gr, c, gh, n) still resolve, but land on an unregistered provider unless you resurrect one via .env.

Without provider, the gateway tries them in failover order (configurable via LLM_ORDER in .env). Explicit provider = no failover (errors surface immediately).

Provider overview

1. Gemini (Google AI Studio, paid tier)

• Best models: gemini-3.1-pro, gemini-3.1-flash, gemini-3.1-flash-lite ★ default, gemini-2.5-flash
• Context: 1M tokens in / 65K out
• Paid Tier 1 (Flash-Lite): thousands of RPM; the gateway paces itself at 150 RPM · 1M TPM · 10K RPD — a deliberate under-estimate, raised in costhelm/routing/core.py once the account's real tier is confirmed

Get a key:

1. Go to aistudio.google.com/app/apikey
2. Sign in with your Google account
3. Click Create API key → pick or create a project → copy the AIza… key
4. Attach the project to a billing account for paid-tier limits

2. OpenRouter one key, many models

• Paid credits unlock the full catalog (300+ models). Default: meta-llama/llama-3.3-70b-instruct ★ default ($0.13/$0.40 per Mtok at the 2026-08 catalog rate)
• Gotcha: OpenRouter's openai/gpt-oss-120b endpoint 400s on reasoning: "off" — "Reasoning is mandatory for this endpoint" — which is why the old Groq STANDARD-rung model did not survive the host move
• Paid accounts are credit-metered with no fixed published RPM; the gateway paces itself at 60 RPM · 500K TPM · 5K RPD as a deliberate under-estimate

Get a key:

1. Go to openrouter.ai
2. Sign in with Google or GitHub
3. Keys → Create Key → copy the sk-or-… key
4. Add credits — without them only :free models answer, under tight daily caps

3. OpenAI

• Default: gpt-5.6-terra ★ default ($2.00/$12.00 per Mtok). The bare alias gpt-5.6 routes to sol, the dearest of the family — name the variant you mean
• Context: 922K in / 128K out
• Never free at any tier; the gateway paces itself at 60 RPM · 200K TPM as a deliberate under-estimate

Get a key:

1. Go to platform.openai.com/api-keys
2. Create new secret key → copy the sk-proj-… key
3. Fund the account — usage tier (and real rate limits) grow with spend

4. Ollama (local)

• Runs on your machine — zero rate limits, zero cost, 0ms cooldown
• Default: gemma4:31b

Set up:

1. Install: ollama.com → download for your OS
2. Pull a model: ollama pull gemma4:31b (or llama3.2, phi4, etc.)
3. Verify: ollama list
4. The gateway auto-detects via OLLAMA_URL (default: http://localhost:11434) — but registers only when OLLAMA_MODEL is set

Retired free tiers

Groq, Cerebras, NVIDIA NIM and GitHub Models are still supported by the code but carry no keys and sit outside the default routing. Their old free-tier pacing rows remain in costhelm/routing/core.py; set a key and add the name back to LLM_ORDER to resurrect one. Note the Groq key also powered the Whisper speech-to-text provider, which is dark while the key is absent.

Summary table

RPM/RPD/TPM here are the gateway's self-imposed pacing from costhelm/routing/core.py — deliberate under-estimates of each paid account's real limits, because too low only throttles locally while too high buys 429s. Raise them once the accounts' real tiers are confirmed.

Configuration (.env)

The gateway reads from .env. All keys are optional — providers without keys are skipped.

GEMINI_API_KEY_1=...        # numbered slots become a gemini_1..gemini_N pool
GEMINI_MODEL=gemini-3.1-flash-lite

OPEN_ROUTER_API_KEY=...     # note the underscore: OPEN_ROUTER, not OPENROUTER
OPENROUTER_MODEL=meta-llama/llama-3.3-70b-instruct

OPENAI_API_KEY=...
OPENAI_MODEL=gpt-5.6-terra

OLLAMA_URL=http://localhost:11434
OLLAMA_MODEL=gemma4:31b     # ollama registers only when this is set

LLM_ORDER=ollama,gemini,openrouter,openai
COSTHELM_PORT=8111

openai goes last in LLM_ORDER on purpose: the order is a failover ring, so the dearest provider belongs at the end of it. The bare alias gpt-5.6 routes to sol, the most expensive of the family — name the variant you mean.

HTTP API reference

How failover works

1. Caller sends request (with or without provider).
2. Router estimates token usage from prompt length.
3. Walks providers in LLM_ORDER, skipping any that:
   • Are inside their per-provider cooldown window (a fixed per-provider seconds value in costhelm/routing/core.py, 1 s for the paid providers, 0 for local Ollama)
   • Have hit RPM, RPD, TPM, or daily token cap
   • Have a max_ctx smaller than the prompt
4. First eligible provider gets the call. On 5xx / timeout / 429 → moves to next.
5. If provider=... was explicitly set, no failover — error surfaces.
6. Every call (success or fail) is logged to gateway.db with provider, model, tokens, latency, status.