Metadata-Version: 2.5
Name: daishi
Version: 0.1.0
Summary: Typed client for the Daishi Studio API: scenarios, runs, series batches and paired experiments, with retries and waiting helpers.
Project-URL: Homepage, https://daishi.ai
Project-URL: Documentation, https://daishi.ai/docs/developer
Author: Arkeous LLC
License-Expression: MIT
License-File: LICENSE
Keywords: agents,ai,benchmark,daishi,evaluation,llm,multi-agent
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Typing :: Typed
Requires-Python: >=3.10
Description-Content-Type: text/markdown

# daishi

Typed client for the Studio API of [Daishi](https://daishi.ai), an independent agent testing and evaluation platform: save scenarios, launch runs, series batches and paired experiments on your own model keys, and read the scored results.

```bash
pip install daishi
```

Python 3.10 or later. No dependencies: the standard library only.

## Quick start

In the [Studio](https://daishi.ai/studio), open **Account > Developer**, create an access token with **runs: launch and cancel** ticked, and export it:

```bash
export DAISHI_TOKEN=dsk_...        # macOS, Linux
$env:DAISHI_TOKEN = "dsk_..."      # Windows PowerShell
```

```python
from daishi import Daishi

daishi = Daishi()  # reads DAISHI_TOKEN

run = daishi.launch({
    "scenario_id": "daishi:famine-v1",
    "roster": [{"model": "<provider>/<model>"}, {"model": "<provider>/<model>"}],
    "max_spend_usd": 5,
})["run"]

done = daishi.wait_for_run(run["run_id"])
for agent in (done["results"] or {}).get("agents", []):
    print(agent["name"], agent.get("fitness_index"), agent.get("grade"))
```

Seats play on the provider keys stored on your account. Every plan can read and validate; creating, launching and cancelling need a paid plan.

## What it does

- **One method per API operation**, named after its operation id: `get_run`, `list_runs`, `launch`, `cancel`, `get_batch`, `create_scenario`, `create_experiment`, `get_experiment` and the rest. Responses are plain dicts typed as `TypedDict`s generated from the API's OpenAPI document, so editors and type checkers know every field.
- **Errors** raise `DaishiError` with the API's `code` (`unknown_run`, `plan_quota`, ...), `status`, `message`, `retry_after_seconds`, and `docs`, a link to what the code means.
- **Retries** are safe by construction. A 429 is retried after the seconds its `Retry-After` header names, and a cancel refused with `run_launching` after a pause, on any method, because the server refused before doing anything. A gateway error or a dropped connection is retried on GET only, so a write is never sent twice.
- **Waiting.** `wait_for_run`, `wait_for_batch` and `wait_for_experiment` poll every 30 seconds until the run, batch or experiment ends. `wait_for_run` also waits for a finished run's results, which arrive when the match archives. Pass `timeout=` (seconds) for a deadline; it raises `TimeoutError`. To be told instead of asking, set up [run-end notices](https://daishi.ai/docs/developer/runs#run-end-notices).

```python
from daishi import Daishi, DaishiError

try:
    daishi.get_run("run_...")
except DaishiError as err:
    if err.code != "unknown_run":
        raise
    print(err.docs)
```

## Options

`Daishi(token=None, *, base_url="https://daishi.ai/api/v1", max_retries=3, timeout=60.0)`. `token` defaults to `DAISHI_TOKEN`; `max_retries=0` turns retrying off; `timeout` is seconds per attempt.

## Types

Every schema in the API is importable: `Run`, `RunResults`, `Scenario`, `Skill`, `BatchReport`, `Experiment`, `ExperimentReport`, the request bodies (`RunBody`, `ScenarioBody`, `ExperimentBody`, ...), and `ErrorCode`. Fields whose shape follows the scenario, the provider or the meter are `JsonObject` (`dict[str, Any]`). A few operations (`me`, `usage`, the estimates, logs and invites) answer `JsonObject` until the API types them; `run_log` answers the log's NDJSON text.

## Docs

[Developer docs](https://daishi.ai/docs/developer) and the [API reference](https://daishi.ai/docs/developer/rest).

MIT licensed.
