Metadata-Version: 2.4
Name: dj-evals
Version: 0.5.0
Summary: Internal Django SSE eval runner
Requires-Python: >=3.11
Description-Content-Type: text/markdown
Requires-Dist: django>=5.0
Provides-Extra: test
Requires-Dist: pytest>=8.0; extra == "test"
Requires-Dist: pytest-asyncio>=0.23; extra == "test"
Provides-Extra: dev
Requires-Dist: ruff>=0.8; extra == "dev"

# dj-evals

Internal Django helper for running one eval function with multiple argument sets in parallel, so you can visually compare output, tool calls, usage, cost, and completion behavior as the runs happen.

## Django setup

Add `dj_evals` to `INSTALLED_APPS` and mount your eval view:

```python
INSTALLED_APPS = [
    "dj_evals",
]
```

```python
from django.urls import path

urlpatterns = [
    path("evals/run/", eval_run),
]
```

Protect the eval view with your app's normal authorization checks before using it in production.

## Usage

```python
from dj_evals import handle_eval_request

async def eval_run(request):
    return await handle_eval_request(
        request,
        allowed_paths={"myapp.evals.echo_eval"},
    )
```

Generate a URL for the view with one or more eval argument dictionaries:

```python
from dj_evals import generate_eval_url

url = "/evals/run/" + generate_eval_url(
    "myapp.evals.echo_eval",
    {"model": "gpt-4.1", "text": "hello"},
    {"model": "gemini-3-flash-preview", "text": "hello"},
)
```

The GET request renders the comparison page. The page starts one POST request per argument set. Each POST response is a direct SSE stream from the eval run, and the browser renders those events into the comparison panels.

## Eval function contract

The eval function must be importable by dotted path and whitelisted in `allowed_paths`. It can be a sync or async function that returns one event, or an async generator that yields events.

```python
def echo_eval(text="hello", model=""):
    return {"type": "response.output_text.delta", "delta": text}


async def async_eval(text="hello", model=""):
    return {"type": "response.output_text.delta", "delta": text}


async def streaming_eval(text="hello"):
    yield {"type": "response.output_text.delta", "delta": text}
    yield {"type": "response.completed", "response": {"cost": 0}}
```

Eval argument dictionaries are passed to the eval function as keyword arguments.

## Helper utilities

You can yield `EvalEvent` for custom progress messages in the output panel:

```python
from dj_evals import EvalEvent


async def streaming_eval():
    event: EvalEvent = {
        "type": "dj_evals.event",
        "message": "Fetched documents",
    }
    yield event
```

## Developers

Run a local demo server to try the library in a browser:

```bash
uv run --with daphne daphne -p 8003 examples.local_server:application
# or, if you have just installed:
just run
```

Then open <http://127.0.0.1:8003/>. The demo redirects to an eval URL that starts three argument sets side by side and streams their events over SSE.

Run checks from this repo:

```bash
uv run --with ruff ruff check .
uv run --with pytest --with pytest-asyncio pytest -q
# or, if you have just installed:
just test
```
