Metadata-Version: 2.4
Name: latenzy
Version: 0.2.0
Summary: Per-model LLM latency monitoring — Prometheus exporter and Grafana dashboards for Claude, OpenAI, and Gemini.
Project-URL: Homepage, https://github.com/amitpatole/latenzy
Project-URL: Documentation, https://amitpatole.github.io/latenzy/
Project-URL: Changelog, https://github.com/amitpatole/latenzy/blob/main/CHANGELOG.md
Author-email: Amit Patole <amit.patole@gmail.com>
License-Expression: AGPL-3.0-only
License-File: LICENSE
Classifier: Development Status :: 3 - Alpha
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: System :: Monitoring
Requires-Python: >=3.10
Requires-Dist: httpx>=0.27
Requires-Dist: prometheus-client>=0.20
Requires-Dist: pydantic>=2.7
Requires-Dist: pyyaml>=6.0
Provides-Extra: otel
Requires-Dist: opentelemetry-api>=1.20; extra == 'otel'
Requires-Dist: opentelemetry-exporter-otlp-proto-http>=1.20; extra == 'otel'
Requires-Dist: opentelemetry-sdk>=1.20; extra == 'otel'
Description-Content-Type: text/markdown

# latenzy

![latenzy — per-model LLM latency monitoring](brand/og-latenzy.png)

[![DOI](https://zenodo.org/badge/DOI/10.5281/zenodo.21796983.svg)](https://doi.org/10.5281/zenodo.21796983)
[![License: AGPL-3.0-only](https://img.shields.io/badge/License-AGPL--3.0--only-blue.svg)](LICENSE)
[![PyPI](https://img.shields.io/pypi/v/latenzy.svg)](https://pypi.org/project/latenzy/)
[![Docs](https://img.shields.io/badge/docs-latenzy-4d7ea6.svg)](https://amitpatole.github.io/latenzy/)

**Documentation: [amitpatole.github.io/latenzy](https://amitpatole.github.io/latenzy/)**

Per-model LLM latency monitoring for enterprises. latenzy is a synthetic prober and
Prometheus exporter that measures what the lab-level status pages can't: the latency
**your** account gets from **each model** — `claude-sonnet-4-6` vs `gpt-4o` vs
`gemini-2.0-flash`, not "Anthropic is up".

Latency is tenant-specific: it depends on your rate-limit tier, your region, and the
path you take to the model (direct API, Bedrock, Vertex). latenzy runs inside your
network on your keys and exports per-model metrics your existing Prometheus + Grafana
stack can alert on.

## What it measures

Every probe cycle, for each configured `(provider, model, endpoint, prompt_class)`:

| Metric | Meaning |
|---|---|
| `latenzy_ttft_seconds` | time to first streamed token (histogram) |
| `latenzy_request_duration_seconds` | total request duration (histogram) |
| `latenzy_output_tokens_per_second` | streaming throughput over the generation span (histogram) |
| `latenzy_probes_total{outcome=...}` | probe count by `ok` / `rate_limited` / `timeout` / `error` |
| `latenzy_last_success_timestamp_seconds` | staleness signal for alerting |

All metrics are labelled `source`, `provider`, `model`, `endpoint`, `prompt_class`.

Histograms are observed only for successful probes, so failures never skew latency
percentiles. Prompts are deterministic per `prompt_class` (small/medium/large) —
comparing models on unequal inputs is meaningless.

Every metric carries a `source` label: `synthetic` for the prober's canaries and
`live` for real application traffic (see below). One dashboard shows both.

## Live traffic (passive instrumentation)

The prober answers "is this model slow right now"; to also chart your *own* traffic's
p95, wrap real LLM calls with the framework-agnostic `LiveRecorder`. It emits the same
metric names under `source="live"`, so the same dashboards work — no separate pipeline.

```python
from latenzy import LiveRecorder, Metrics, classify_prompt, measure_stream

recorder = LiveRecorder(Metrics())  # shares your app's Prometheus registry

with recorder.observe(
    provider="openai", model="gpt-4o", prompt_class=classify_prompt(text=prompt)
) as obs:
    for chunk in measure_stream(client.stream(prompt), obs):  # marks first-token timing
        handle(chunk)
    obs.output_tokens = n_tokens
```

Label values are charset-validated (they may come from user input), so a host app can't
explode metric cardinality. A raised exception is recorded as an `error` outcome and
re-raised. A runnable, key-free walkthrough is in
[`examples/demo_live.py`](examples/demo_live.py):

```console
$ python examples/demo_live.py
recorded live call: model=gpt-4o prompt_class=small tokens=4
recorded live call: model=gpt-4o prompt_class=large tokens=9

--- /metrics (live source) ---
latenzy_ttft_seconds_count{...,prompt_class="small",...,source="live"} 1.0
latenzy_ttft_seconds_sum{...,prompt_class="small",...,source="live"} 0.0553...
latenzy_ttft_seconds_sum{...,prompt_class="large",...,source="live"} 0.3202...
latenzy_probes_total{...,outcome="ok",prompt_class="small",...,source="live"} 1.0
```

### OpenTelemetry

Emit to OpenTelemetry as well as (or instead of) Prometheus with the `otel` extra
(`pip install 'latenzy[otel]'`). The prober and `LiveRecorder` take any
`RecordSink`, so `FanoutSink(Metrics(), OTelBridge(meter))` sends to both;
instrument names follow the OTel GenAI conventions
(`gen_ai.client.operation.duration`, `gen_ai.client.token.usage`). For
`latenzy run`, enable it in config:

```yaml
otel:
  enabled: true
  endpoint: https://collector.internal:4318/v1/metrics   # omit → console
```

## Quick start

```bash
pip install latenzy

export ANTHROPIC_API_KEY=... OPENAI_API_KEY=... GEMINI_API_KEY=...
latenzy doctor -c latenzy.yaml   # validate config, check keys are present
latenzy once   -c latenzy.yaml   # one probe cycle, human-readable results
latenzy run    -c latenzy.yaml   # probe on an interval + serve /metrics
```

See [`latenzy.example.yaml`](latenzy.example.yaml) for the full configuration.
API keys are read from environment variables only — they have no place in config files.

## Grafana + Prometheus

- **Dashboard** — Grafana dashboard library ID
  [**25642**](https://grafana.com/grafana/dashboards/25642/) (Dashboards → Import →
  `25642`), or import [`dashboards/latenzy-model-comparison.json`](dashboards/latenzy-model-comparison.json):
  the model-comparison view (TTFT p95, total-latency p95, tokens/sec, failure and 429
  ratio, staleness) filterable by provider, model, endpoint, and prompt class. Import
  it into any Grafana; it prompts for your Prometheus datasource. The
  [`dashboards-share/`](dashboards-share/) copy is the same dashboard in Grafana's
  "export for sharing externally" format for grafana.com uploads (generated by
  `scripts/share_dashboard.py`; a test keeps the two in sync).
- **Recording rules** — [`prometheus/recording_rules.yml`](prometheus/recording_rules.yml):
  hourly and daily p50/p95/p99 series (`latenzy:ttft_seconds:p95_1h`, ...), so
  dashboards and alerts never recompute histogram quantiles.
- **Alert rules** — [`prometheus/alert_rules.yml`](prometheus/alert_rules.yml):
  probe staleness, TTFT SLO breach, rate-limit pressure, failure ratio.

## Standalone bundle (no existing Grafana needed)

```bash
cd deploy
mkdir -p secrets && openssl rand -hex 32 > secrets/latenzy_token
export ANTHROPIC_API_KEY=... OPENAI_API_KEY=... GEMINI_API_KEY=...
export GRAFANA_ADMIN_PASSWORD=...   # no default password ships with the bundle
docker compose up -d                # prober + Prometheus + Grafana, pre-provisioned
```

Grafana serves the comparison dashboard read-only at `http://localhost:3000`
(loopback-published only). The bundle mounts the same `dashboards/` and
`prometheus/` files from the repo, so the bundled and published copies cannot drift.

## Security posture

- Binds `127.0.0.1` by default. Binding a routable interface **refuses to start**
  unless `exporter.auth_token_env` is set; the token is checked in constant time.
- API keys are sent in request headers only and never appear in logs, URLs, metrics,
  or error output.
- Probe cost is bounded: `max_output_tokens` defaults to 16.

## Status

v0.1.0 — first real release: prober + exporter, Grafana dashboard (library ID
25642), recording/alert rules, standalone bundle, security-hardened through a
four-round red-team loop (see [`SECURITY.md`](SECURITY.md)). Now also: passive
live-traffic instrumentation (`LiveRecorder`, `source="live"`) and an
OpenTelemetry meter bridge (`latenzy[otel]`). Docs:
[amitpatole.github.io/latenzy](https://amitpatole.github.io/latenzy/).

License: AGPL-3.0-only. Dual licensing available for enterprises — contact the author.

— amitpatole
