Metadata-Version: 2.4
Name: costhelm
Version: 0.1.0
Summary: Self-hosted cost-governance gateway for LLM APIs: per-model pricing, principal attribution, hard budget enforcement, cost/quality routing, semantic caching, and OTel GenAI observability.
Project-URL: Homepage, https://costhelm.com
Project-URL: Repository, https://github.com/gadekar-pravin/costhelm
Author-email: Pravin Gadekar <pbgadekar@gmail.com>
License: MIT
License-File: LICENSE
Keywords: budget,cost,gateway,gemini,llm,observability,openai,routing
Classifier: Development Status :: 3 - Alpha
Classifier: Framework :: FastAPI
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Requires-Python: >=3.11
Requires-Dist: fastapi>=0.110
Requires-Dist: httpx>=0.27
Requires-Dist: jsonschema>=4.21
Requires-Dist: opentelemetry-api>=1.27
Requires-Dist: opentelemetry-exporter-otlp-proto-http>=1.27
Requires-Dist: opentelemetry-sdk>=1.27
Requires-Dist: pydantic>=2.6
Requires-Dist: python-dotenv>=1.0
Requires-Dist: pyyaml>=6.0
Requires-Dist: uvicorn[standard]>=0.27
Provides-Extra: modal
Requires-Dist: modal>=1.5.1; extra == 'modal'
Description-Content-Type: text/markdown

# costhelm

A self-hosted **cost-governance gateway for LLM APIs**. Your agents talk to
one local HTTP endpoint; costhelm owns the provider keys and answers every
call with the numbers that actually matter: what it cost, who spent it, and
how much budget is left.

Most gateways treat cost as a report you read after the money is gone.
costhelm treats it as a **precondition**: every call is projected at
worst-case cost and admitted against a hard budget *before* any provider is
contacted. A refusal is an HTTP 402 with the arithmetic in the body — limit,
spend, projection, shortfall — and it costs $0, because nothing was sent.

```
call 4: ollama/phi4:latest — $0.017000 (total $0.047000) — 'Running budget checks before...'
call 5: REFUSED — session:budget-agent-demo would exceed its lifetime budget:
        spent $0.047000 of $0.050000, this call projects $0.016000
```

## Features

| | |
|---|---|
| **Hard budgets** | Per-principal ceilings (tenant/project/user/agent/session, glob-able) with minute/hour/day/month/lifetime periods. Pre-call admission on a worst-case projection; spend is always re-derived from the ledger, never a parallel counter. Runtime overrides via `POST /v1/budget` — clamp a runaway session without a deploy. |
| **Priced ledger** | One SQLite row per call, priced from a per-model `pricing.yaml` (glob patterns, cache-read/write and batch multipliers) and attributed to all five principal dimensions. CI fails if a configured default model would price at $0. |
| **Tier routing + cascade** | A small router LLM classifies each prompt (TINY/LARGE/HUGE); the caller's declared *role* clamps the tier with floors and ceilings; candidates are ordered by cost, quality, or a tradeoff dial. Low-confidence answers escalate one tier — confidence is scored from structural signals, never a judge LLM. |
| **Provider failover** | Eight providers behind one API (OpenAI, Gemini with a numbered key pool, OpenRouter, Ollama, Groq, Cerebras, NVIDIA, GitHub Models). RPM/RPD/TPM pacing, capability-aware picking (tools/vision/reasoning/structured), failure-class cooldowns — all configurable in `routing.yaml`. |
| **Semantic cache** | Embed → cosine match → skip the provider entirely. A hit bills $0 and records the tokens as *saved*; truncated and empty completions are refused into the cache. Off by default, one line to enable. |
| **Observability** | OTel spans with `gen_ai.*` attributes plus cost; two zero-build HTML dashboards; `/v1/refusals` captures the 402s that by design leave no span and no ledger row. Prompt/completion capture is off by default. |
| **Optional auth** | One env var (`COSTHELM_API_KEYS`) guards `/v1/*` with bearer or `X-API-Key`. Off by default for localhost. |

## Quickstart

```bash
uvx costhelm serve
```

(or from a clone: `uv sync && uv run costhelm serve`)

Add a provider key to `.env` (see [.env.example](.env.example)) — or none at
all if [Ollama](https://ollama.com) is running locally:

```bash
curl -s localhost:8111/v1/chat -H 'Content-Type: application/json' \
  -d '{"prompt": "hello", "max_tokens": 50}'
```

Every response carries `cost`, `budget`, `cache`, and `router_decision`
envelopes. Open `http://localhost:8111/` for the dashboard.

To see budget enforcement end-to-end, run the example agent — it arms a $0.05
ceiling, spends it, and prints the 402:

```bash
uv run python examples/budget_agent.py
```

## How a request flows

```
POST /v1/chat
  │
  1. build_context      normalise, estimate tokens, resolve principal, agent pin
  2. semantic cache     hit → return stored answer, bill $0, record tokens saved
  3. select_candidates  router-LLM tier classify → role clamp → cost/quality order
  4. dispatch           ┌ cascade loop ──────────────────────────────┐
                        │  for each candidate:                       │
                        │    budget admission (worst-case, pre-call) │──402
                        │    provider call (+1 retry, backoff)       │
                        │    structured-output validate/repair       │
                        │    confidence check → maybe escalate tier ─┘
                        │    meter → priced ledger row, OTel span
                        └ return text + cost/budget/cache envelopes
```

Each stage is a module in [`costhelm/pipeline/`](costhelm/pipeline/), and each
is inert unless configured: with the shipped config files, costhelm behaves
like a plain failover gateway.

## Configuration

Four YAML files, three-layer resolution — an env-var path override wins, then
`~/.costhelm/<name>.yaml`, then the packaged default. `costhelm paths` prints
what resolved where.

| File | Governs |
|---|---|
| [`pricing.yaml`](costhelm/economics/pricing.yaml) | $/Mtok per model, cache/batch multipliers, quality weights |
| [`budgets.yaml`](costhelm/economics/budgets.yaml) | Spend ceilings per principal (ships empty: nothing is refused) |
| [`routing.yaml`](costhelm/routing/routing.yaml) | Tiers, roles, ladders, escalation, provider limits, agent pins |
| [`cache.yaml`](costhelm/cache/cache.yaml) | Semantic cache (ships disabled) |

Secrets and switches live in `.env` — see [.env.example](.env.example). A
provider registers only if its key variable is set.

## Client

```python
from costhelm.client import CosthelmClient, BudgetExceeded

async with CosthelmClient() as gw:
    await gw.set_budget("session:run-42", 0.05, period="day")
    try:
        r = await gw.chat("hello", request={"session": "run-42", "auto_route": "worker"})
        print(r["text"], r["cost"]["total_usd"])
    except BudgetExceeded as e:
        print("refused:", e.envelope["shortfall_usd"])
```

The wire contract is plain HTTP JSON — any language works. The client holds
no provider credential; that boundary is the point.

## Deployment

`deploy/` has systemd/launchd/Windows-service templates, a
[Modal](https://modal.com) wrapper, and a docker-compose file for a local
Jaeger to receive the OTel spans. Docs: https://costhelm.com

## Origins

costhelm grew out of an agentic-AI course project (a gateway serving a
budget-aware agent runtime) and was extracted and restructured into a
standalone framework.

## License

[MIT](LICENSE)
