Metadata-Version: 2.4
Name: llm-eval-exporter
Version: 0.2.0
Summary: A lightweight Prometheus exporter for LLM eval metrics: faithfulness, semantic drift, cost, and CI/CD regression gating.
Author-email: Harshitha <your-email@example.com>
License: MIT License
        
        Copyright (c) 2026 Harshitha
        
        Permission is hereby granted, free of charge, to any person obtaining a copy
        of this software and associated documentation files (the "Software"), to deal
        in the Software without restriction, including without limitation the rights
        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
        copies of the Software, and to permit persons to whom the Software is
        furnished to do so, subject to the following conditions:
        
        The above copyright notice and this permission notice shall be included in all
        copies or substantial portions of the Software.
        
        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
        SOFTWARE.
Project-URL: Homepage, https://github.com/your-username/llm-eval-exporter
Project-URL: Repository, https://github.com/your-username/llm-eval-exporter
Project-URL: Issues, https://github.com/your-username/llm-eval-exporter/issues
Keywords: llm,observability,prometheus,eval,mlops,monitoring,grafana,ci-cd
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Topic :: System :: Monitoring
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: prometheus-client>=0.20.0
Requires-Dist: numpy>=1.24.0
Requires-Dist: pyyaml>=6.0
Provides-Extra: embeddings
Requires-Dist: sentence-transformers>=2.2.0; extra == "embeddings"
Provides-Extra: anthropic
Requires-Dist: anthropic>=0.30.0; extra == "anthropic"
Provides-Extra: dev
Requires-Dist: pytest>=7.4.0; extra == "dev"
Dynamic: license-file

# llm-eval-exporter

A lightweight Prometheus exporter for LLM eval metrics — hallucination
risk (via LLM-as-judge faithfulness scoring), semantic drift across
model/prompt versions, latency, and token cost — designed to plug into
observability stacks teams already run, instead of requiring adoption
of a new platform.

## Why this exists

Most LLM observability tools (Langfuse, Braintrust, Arize Phoenix,
etc.) are full platforms: their own UI, their own storage, their own
workflow. That's the right call for teams building observability from
scratch. But plenty of teams already run Prometheus + Grafana for
every other service they operate, and would rather their LLM calls
show up as three more panels on an existing dashboard than as a
separate tool with a separate login.

`llm-eval-exporter` wraps your LLM calls, computes eval scores, and
exposes them as standard Prometheus metrics. It's a library, not a
platform — a few lines of Python, one `/metrics` endpoint, no new UI
to learn.

## What it tracks

| Metric | What it tells you |
|---|---|
| `llm_request_latency_seconds` | p50/p95/p99 latency per model/provider |
| `llm_request_total` | Request volume and error rate |
| `llm_tokens_total` | Prompt/completion token usage (cost proxy) |
| `llm_faithfulness_score` | LLM-judge score (0-1) for how well a response sticks to its provided context — a hallucination proxy |
| `llm_semantic_drift` | Cosine distance between a response and its recorded baseline for the same prompt — catches silent behavior changes across model upgrades or prompt edits |
| `llm_eval_failures_total` | Count of responses that failed a faithfulness or drift threshold |

## Quickstart

```bash
pip install -r requirements.txt
python examples/demo.py
# metrics now live at http://localhost:9464/metrics
```

To run against the real Anthropic API instead of mocks:

```bash
export ANTHROPIC_API_KEY=sk-ant-...
python examples/anthropic_example.py
```

This traces a real `claude-sonnet-5` call and judges its faithfulness
with a cheaper `claude-haiku-4-5` call — the standard cost-effective
pattern for LLM-as-judge in production (you don't need your most
expensive model just to check "did this answer stick to the
context").

Then, optionally, bring up Prometheus + Grafana pointed at it:

```bash
docker compose -f examples/docker-compose.yml up
# Grafana at http://localhost:3000 (anonymous admin access, demo only)
```

## Usage in your own code

```python
from llm_eval_exporter import LLMTracker, start_metrics_server

start_metrics_server(port=9464)  # call once at app startup

tracker = LLMTracker(
    judge_fn=my_judge_fn,   # any callable: str prompt -> str response
    embed_fn=my_embed_fn,   # optional: defaults to sentence-transformers locally
)

with tracker.track(
    model="gpt-4o",
    provider="openai",
    prompt_id="refund_policy_qa",  # stable id -> enables drift tracking
    context=retrieved_context,     # enables faithfulness scoring
) as call:
    response = your_llm_client.call(prompt)
    call.set_response(
        response.text,
        prompt_tokens=response.usage.prompt_tokens,
        completion_tokens=response.usage.completion_tokens,
    )
```

`judge_fn` and `embed_fn` are intentionally pluggable — bring your own
LLM client (Anthropic, OpenAI, a local model) rather than depending on
one vendor's SDK.

## Design notes

- **SQLite by default** for baseline embeddings and eval logs — a
  single-file, zero-ops dependency, matching this project's "drop it
  into an existing stack" philosophy. Swap for Postgres if it needs to
  scale past one instance.
- **Faithfulness scoring is a lightweight, single-call analogue of
  RAGAS-style faithfulness** — not a claim to replace a dedicated eval
  framework, just enough signal to catch regressions cheaply.
- **Drift detection needs a `prompt_id`** you assign to prompts you
  care about tracking over time; the first call for a given id just
  records a baseline, so drift shows up starting from the second call.

## Status

Early-stage side project — built to explore a gap in the current LLM
observability tooling landscape (see `docs/market-notes.md` for the
reasoning), not a production-ready alternative to the platforms above.
Contributions and issues welcome.

## Running tests

```bash
pip install -r requirements.txt
pytest tests/ -v
```
