Metadata-Version: 2.4
Name: testrelic-deepeval
Version: 0.1.0
Summary: TestRelic SDK for DeepEval - LLM evaluation results land in TestRelic instead of Confident AI.
Project-URL: Homepage, https://testrelic.ai
Project-URL: Documentation, https://docs.testrelic.ai/deepeval
Project-URL: Repository, https://github.com/testrelic-ai/testrelic-python-sdk
Author-email: TestRelic <engineering@testrelic.ai>
License: MIT
License-File: LICENSE
Keywords: deepeval,evaluation,llm,testing,testrelic
Classifier: Development Status :: 3 - Alpha
Classifier: Framework :: Pytest
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Requires-Python: >=3.9
Requires-Dist: httpx<1.0,>=0.27
Requires-Dist: platformdirs>=4.0
Requires-Dist: pydantic<3.0,>=2.6
Requires-Dist: tomli-w>=1.0
Requires-Dist: tomli>=2.0; python_version < '3.11'
Requires-Dist: typer<1.0,>=0.12
Provides-Extra: deepeval
Requires-Dist: deepeval<4.0,>=2.0; extra == 'deepeval'
Provides-Extra: dev
Requires-Dist: deepeval<4.0,>=2.0; extra == 'dev'
Requires-Dist: mypy>=1.10; extra == 'dev'
Requires-Dist: pytest-cov>=5.0; extra == 'dev'
Requires-Dist: pytest>=8.0; extra == 'dev'
Requires-Dist: respx>=0.21; extra == 'dev'
Requires-Dist: ruff>=0.6; extra == 'dev'
Provides-Extra: otel
Requires-Dist: opentelemetry-api>=1.25; extra == 'otel'
Requires-Dist: opentelemetry-sdk>=1.25; extra == 'otel'
Description-Content-Type: text/markdown

# testrelic-deepeval

The TestRelic Python SDK for [DeepEval](https://github.com/confident-ai/deepeval). Capture
LLM evaluation results from your existing DeepEval test suite and ship them to your
TestRelic workspace in one line of install.

```bash
pip install testrelic-deepeval
testrelic login
deepeval test run tests/
```

That's it. The pytest plugin captures DeepEval's in-memory `TestRun` at session finish
and uploads cases + metrics to TestRelic's `/api/v1/evals/*` endpoints.

## What this is (and isn't)

- **Is**: a pytest plugin + Python client for sending DeepEval results to TestRelic
- **Is**: a drop-in `testrelic.evaluate()` wrapper for programmatic eval runs
- **Is**: a CLI (`testrelic login`, `testrelic test`, `testrelic view`)
- **Is not**: a replacement for DeepEval (use DeepEval to author tests; we just receive
  the results)
- **Is not**: a Confident AI proxy — we use TestRelic's own API, not Confident's

## Install

```bash
pip install testrelic-deepeval                  # core
pip install "testrelic-deepeval[deepeval]"      # also pull deepeval if not already installed
pip install "testrelic-deepeval[otel]"          # OTel tracing (Phase 4, preview)
```

Supported Python versions: 3.9, 3.10, 3.11, 3.12.

## Authenticate

```bash
testrelic login --api-key tr_yourkey_here
```

Credentials are stored at `~/.testrelic/credentials.toml` with `0600` permissions on
POSIX. Or pass credentials via environment:

```bash
export TESTRELIC_API_KEY=tr_yourkey
export TESTRELIC_BASE_URL=https://platform.testrelic.ai/api/v1/evals  # only override if self-hosted
```

Precedence (highest to lowest): explicit `configure()` args → env vars → credentials
file → built-in defaults.

## Use with pytest

No code changes needed. After install + login:

```bash
deepeval test run tests/
# or, if you want the wrapper that double-checks credentials:
testrelic test tests/
```

Every test run uploads as a new eval run in TestRelic, annotated with the current
branch, commit, and CI run URL (auto-detected for GitHub Actions, GitLab, Jenkins,
CircleCI, and Buildkite).

## Use programmatically

```python
from deepeval.test_case import LLMTestCase
from deepeval.metrics import AnswerRelevancyMetric
from testrelic import evaluate

results = evaluate(
    test_cases=[LLMTestCase(input="Hi", actual_output="Hello")],
    metrics=[AnswerRelevancyMetric(threshold=0.7)],
)
# results is whatever deepeval.evaluate() returns; upload is automatic
```

## Datasets

```python
from testrelic import datasets

# Pull a dataset version into a deepeval EvaluationDataset
ds = datasets.pull("customer-support-qa", label="latest")

# Push goldens up to a new version
datasets.push(
    alias="customer-support-qa",
    goldens=[{"input": "...", "expected_output": "..."}],
    label="v2",
    description="Refreshed Q1 2026 examples",
)
```

## Migrating from Confident AI

See [docs/migration-from-confident-ai.md](docs/migration-from-confident-ai.md). The
TL;DR is:

1. `pip install testrelic-deepeval`
2. `testrelic login`
3. Remove `CONFIDENT_API_KEY` from CI
4. Run your existing tests unchanged

Or, in one shot: `testrelic migrate-from-confident`.

## Offline / flaky networks

Failed uploads land in `~/.testrelic/queue/`. Replay them with:

```bash
testrelic drain
```

The plugin never fails your test run because of an upload error — uploads run at
`pytest_sessionfinish(trylast=True)` and swallow exceptions with a warning log.

## CLI

| Command | Purpose |
|---|---|
| `testrelic login` | Save credentials |
| `testrelic logout` | Remove credentials |
| `testrelic test <path>` | Wrap `deepeval test run` with credential check |
| `testrelic view [run_id]` | Open latest (or specific) eval run in browser |
| `testrelic drain` | Replay queued offline uploads |
| `testrelic version` | Print SDK version |
| `testrelic migrate-from-confident` | Print migration steps |

## Development

```bash
git clone https://github.com/testrelic-ai/testrelic-python-sdk
cd testrelic-python-sdk
pip install -e ".[dev]"
ruff check src tests
mypy src/testrelic
pytest
```

## License

MIT. See [LICENSE](LICENSE).
