Metadata-Version: 2.5
Name: warpswarm
Version: 0.1.1
Summary: Python SDK for Warpswarm: on-demand human judgment for AI evaluation and labeling.
Project-URL: Homepage, https://warpswarm.app
License-Expression: MIT
Classifier: Programming Language :: Python :: 3
Classifier: Typing :: Typed
Requires-Python: >=3.9
Requires-Dist: httpx<1,>=0.24
Description-Content-Type: text/markdown

# warpswarm (Python)

On-demand human judgment for AI evaluation and labeling. Submit items, get aggregated human answers back.

```bash
pip install warpswarm
export WARPSWARM_API_KEY=ws_live_...     # ws_test_... = free sandbox, no charges
```

```python
import warpswarm as ws

client = ws.Client()

job = client.compare(
    "Summary A/B",
    "Which summary is more accurate?",
    [(summary_a[i], summary_b[i]) for i in range(100)],   # (a, b) pairs: text, Path, URL or asset id
    answers=5,                                           # answers per item
)                                                         # budget defaults to items × answers × $0.01
job.add_gold([(("Paris is in France.", "Paris is in Spain."), "a")])  # optional known-answer checks
job.start().wait(on_progress=print)

for r in job.results():
    print(r.external_id, r.aggregate.winner, r.aggregate.votes, r.confidence)
job.export("results.csv", format="csv")
```

## Task types

| helper | items | `result.aggregate` |
|---|---|---|
| `compare(name, q, pairs, allow_tie=False)` | `(a, b)` | `CompareAggregate(winner, votes)` |
| `classify(name, q, items, labels, multi_select=False)` | text or media | `ClassifyAggregate(label, votes)` / `MultiClassifyAggregate(labels, share)` |
| `free_text(name, q, items, min_chars, max_chars)` | text or media | `FreeTextAggregate(texts)` |
| `locate(name, q, images)` | image | `LocateAggregate(point, points)` |
| `draw(name, q, images, max_boxes=20)` | image | `DrawAggregate(boxes)` |
| `select_words(name, q, texts)` | text, ≤ 40 words | `SelectWordsAggregate(words, tokens, share)` |
| `ranking(name, q, option_sets)` | lists of 2–5 options | `RankingAggregate(ranking, borda)` |

Coordinates are fractions of the image (0–1), `x, y` = top-left. Media can be a string (text), a `pathlib.Path` or
`bytes` (uploaded once), an `http(s)` URL (imported), an `ast_…` id, or an `Asset`. Any item can also be a full dict
with `context` (e.g. the prompt) and `external_id` (your own id, returned on every result).

Gold `correct` values: compare `"a"|"b"|"tie"` · classify `"label"` or `[labels]` · free_text `"text"` ·
locate a region `(x, y, w, h)` (any tap inside passes) · draw `[(x, y, w, h), …]` · select_words `[index]` or
`["word", …]` · ranking `[index, best first]`.

## Lifecycle and timing

`draft → start() → queued → running → completed`. `start()` holds credit for the job's budget; unused credit is
released at the end, and only accepted answers are charged ($0.01 each).

- **Startup review.** Each job's worker batch passes a one-time review before workers start (usually under an hour;
  locate/draw usually 2–3 hours). `client.estimate(...)` returns `startup_minutes`; `job.progress.phase` is
  `in_review` with an `eta` until then. `wait()` polls slowly during review and every `poll_interval` s while collecting.
- **Minimum size.** A job needs at least 14 items (`ws.MIN_DATAPOINTS`); smaller jobs raise `JobTooSmallError`.
- **Unsold tail.** Workers answer items in batches. If fewer than 14 answers remain at the end, they can't be staffed:
  the job completes, `progress.answers_unsold` says how many, and they are never charged.

## Webhooks

Pass `webhook_url="https://…"` when creating a job to get `job.started`, `job.progress`, `job.completed`, `job.cancelled`
and `job.failed` POSTs instead of polling. `job.webhook_secret` is returned once, at creation, so store it.

```python
event = ws.verify_webhook(request.body, request.headers["Warpswarm-Signature"], secret)   # raw body bytes
if event.type == "job.completed":
    job = client.job(event.job_id)
```

`verify_webhook` checks the HMAC and rejects timestamps older than 5 minutes (`ws.WebhookSignatureError`). Retries reuse
`event.id`, so dedupe on it. Any 2xx acknowledges; failures are retried for 24 h.

## Errors

All errors subclass `ws.WarpswarmError` (`.status`, `.code`, `.param`): `AuthenticationError`, `NotFoundError`,
`InvalidRequestError`, `JobTooSmallError`, `InvalidStateError`, `InsufficientCreditError`, `RateLimitError`,
`ServerError`, plus `WaitTimeout` / `JobFailed` from `wait()`. Every POST carries an `Idempotency-Key`, so the SDK
retries network errors, 429 and 5xx safely.

## Other calls

`client.account()`, `client.estimate(task_type, n, answers)`, `client.upload(path)`, `client.upload_url(url)`,
`client.jobs(status=)`, `client.job(id)`, `job.preview_url()` (see the exact worker form), `job.pause()/resume()/cancel()`,
`job.results(include_answers=True)` (each answer with a pseudonymous `annotator`), `client.checkout_url(amount_usd)`.

## Development

```bash
uv venv .venv && uv pip install -e . pytest pyyaml openapi-schema-validator
pytest tests/test_unit.py            # offline; every request body is validated against ../../api/openapi.yaml
# full loop, all 7 types (local wrangler dev + worker/tests/mock/mw_mock.py on :8799):
WS_BASE=http://localhost:8787 WS_ADMIN=… WS_SEED="npx wrangler d1 execute warpswarm --local --command" pytest tests/test_integration.py
# prod, read + draft lifecycle only (no workers, no charges):
WS_BASE=https://api.warpswarm.app WS_TEST_KEY=ws_test_… pytest tests/test_integration.py -k prod
uv build                             # dist/warpswarm-0.1.0-py3-none-any.whl
```
