Metadata-Version: 2.4
Name: evalio-ai
Version: 0.1.3
Summary: Python SDK for Evalio AI - LLM A/B testing platform
Author: Evalio AI
License-Expression: MIT
Project-URL: Homepage, https://github.com/Asicvfx/Evalio-AI
Project-URL: Repository, https://github.com/Asicvfx/Evalio-AI
Project-URL: Issues, https://github.com/Asicvfx/Evalio-AI/issues
Project-URL: Documentation, https://github.com/Asicvfx/Evalio-AI/blob/main/sdk/README.md
Keywords: llm,experimentation,ab-testing,openai,sdk,evalio
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Requires-Python: >=3.9
Description-Content-Type: text/markdown
Requires-Dist: httpx>=0.25
Requires-Dist: openai>=1.0
Provides-Extra: anthropic
Requires-Dist: anthropic>=0.25; extra == "anthropic"

# Evalio Python SDK

Python SDK for [Evalio AI](https://evalio.ai), an LLM experimentation platform for routing traffic between model or prompt variants, reporting metrics, and collecting feedback.

## Package name

Install package:

```bash
pip install evalio-ai
```

Import path stays:

```python
from evalio import EvalioClient
```

With Anthropic support:

```bash
pip install "evalio-ai[anthropic]"
```

## Quick Start

```python
from evalio import EvalioClient

client = EvalioClient(
    api_key="sk-evalio-your-key",
    base_url="https://evalio-ai-production.up.railway.app",
    provider_keys={"openai": "sk-your-openai-key"},
)

response = client.run(
    experiment_id=12,
    messages=[{"role": "user", "content": "How do I reset my password?"}],
    user_id="user-123",
    idempotency_key="support-reset-user-123-msg-001",
    provider_params={"temperature": 0.2},
)

print(response.content)
print(response.variant)
print(response.request_id)
print(response.cost_usd)
print(response.latency_ms)
```

## What the SDK does

1. Calls Evalio to get a routing decision
2. Calls the assigned provider/model
3. Reports completion metrics back to Evalio
4. Returns the provider response

Sticky assignment means the same `user_id` stays on the same variant after the first assignment.

## Per-call provider parameters

If your app needs JSON mode or other call-specific provider options, pass them at runtime:

```python
response = client.run(
    experiment_id=12,
    messages=[{"role": "user", "content": "Return valid JSON only"}],
    user_id="user-123",
    idempotency_key="json-user-123-msg-001",
    provider_params={
        "temperature": 0.2,
        "response_format": {"type": "json_object"},
    },
)
```

Runtime params are merged on top of the variant's saved `extra_params`.

## Feedback

```python
client.feedback(
    response.request_id,
    score=1,
    comment="Helpful answer",
    idempotency_key=f"feedback:{response.request_id}:thumbs-up",
)
```

## Retry safety

`EvalioClient.run(..., idempotency_key=...)` and `EvalioClient.feedback(..., idempotency_key=...)`
can be retried safely. The backend replays the original result instead of creating duplicates.

## Supported providers

- OpenAI
- Anthropic via optional extra
