Metadata-Version: 2.4
Name: latitude-telemetry-prime-intellect
Version: 0.1.0
Summary: Export Prime Intellect Verifiers eval rollouts to Latitude as OTLP traces and scores
Project-URL: repository, https://github.com/latitude-dev/latitude-llm/tree/main/packages/telemetry/prime-intellect
Project-URL: homepage, https://github.com/latitude-dev/latitude-llm/tree/main/packages/telemetry/prime-intellect#readme
Project-URL: documentation, https://github.com/latitude-dev/latitude-llm/tree/main/packages/telemetry/prime-intellect#readme
Author-email: Latitude Data SL <hello@latitude.so>
Maintainer-email: Latitude Data SL <hello@latitude.so>
License-Expression: MIT
Keywords: evals,latitude,opentelemetry,otlp,prime-intellect,telemetry,verifiers
Requires-Python: <3.15,>=3.11
Requires-Dist: certifi>=2024.2.2
Description-Content-Type: text/markdown

# latitude-telemetry-prime-intellect

Export [Prime Intellect Verifiers](https://github.com/PrimeIntellect-ai/verifiers) eval
rollouts to [Latitude](https://latitude.so) as OTLP traces (and optional custom scores
from rewards/metrics).

This is the Prime Intellect counterpart to the other harness integrations
([`latitude-telemetry-hermes`](../hermes), Claude Code, Pi, OpenClaw). Verifiers has no
host plugin entry point for observability, so this package ships as a **library + CLI**
you call from your eval script or after a run.

## Install

```bash
pip install latitude-telemetry-prime-intellect
```

```bash
export LATITUDE_API_KEY=lat_xxx
export LATITUDE_PROJECT=my-project
```

## Usage

### After `run_eval` (recommended for scripts)

```python
from verifiers.v1.cli.eval.runner import run_eval
from latitude_telemetry_prime_intellect import export_episodes

episodes = await run_eval(env, config)
export_episodes(episodes)  # ships traces + reward scores, then flushes
```

### As `Env.run_slot` `on_complete`

```python
from latitude_telemetry_prime_intellect import make_on_complete

on_complete = make_on_complete(next=persist_episode)  # chain your append_episode
await env.run_slot(slot, ctx, semaphore, on_complete)
```

### Post-hoc from a results directory

After `uv run eval …` / `prime eval …`:

```bash
latitude-prime-intellect-export export ./outputs/<run-dir>
# or
python -m latitude_telemetry_prime_intellect export ./outputs/<run-dir>
```

Reads `traces.jsonl`, `episodes.jsonl`, or `results.jsonl` when present.

## Configuration

| Env | Default | Description |
|-----|---------|-------------|
| `LATITUDE_API_KEY` | — | API key (required) |
| `LATITUDE_PROJECT` / `LATITUDE_PROJECT_SLUG` | — | Project slug (required) |
| `LATITUDE_BASE_URL` | `https://ingest.latitude.so` | Ingest origin (plugin appends `/v1/traces`) |
| `LATITUDE_API_BASE_URL` | `https://api.latitude.so` | Public API origin for scores |
| `LATITUDE_EXPORT_SCORES` | `true` | POST rewards/metrics as custom scores |
| `LATITUDE_NO_CONTENT` | `false` | Structure/timing only |
| `LATITUDE_DEBUG` | `false` | Verbose logging |
| `LATITUDE_PRIME_INTELLECT_TELEMETRY_ENABLED` / `LATITUDE_TELEMETRY_ENABLED` | `true` | Master switch |

Telemetry stays off until both `LATITUDE_API_KEY` and a project are set. Export is
fail-open: a Latitude error never fails your eval.

## How it works

Each Verifiers `Trace` becomes one Latitude trace:

```text
interaction (rollout root; session = eval/episode id)
├── llm_request      (one per ModelCall: model, usage, finish reason, messages)
└── tool_call:<name> (tool_execution; one per tool result)
```

Rewards map to custom scores (`sourceId = verifiers.reward.<name>`) against the same
trace id (Verifiers' 32-hex `Trace.id`).

## Development

```bash
cd packages/telemetry/prime-intellect
uv sync --all-groups
uv run pytest tests/ -x
```
