Metadata-Version: 2.4
Name: annotators
Version: 0.1.0
Summary: Python client for annotators.ai: human review for any model output
Author: annotators
License-Expression: MIT
Project-URL: Homepage, https://annotators.ai
Project-URL: Documentation, https://annotators.ai/docs
Keywords: annotation,labeling,human review,human-in-the-loop,llm,evaluation,webhooks
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Typing :: Typed
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: httpx>=0.27
Provides-Extra: dev
Requires-Dist: pytest>=8; extra == "dev"
Dynamic: license-file

# annotators

The Python client for [annotators.ai](https://annotators.ai): human review for any model output.
Send what your model returned to have it checked, or leave the output out and have the item labeled.
A machine check goes first, and people see only what it doubts.

```bash
pip install annotators
```

Python 3.10 or later. The full API reference is at [annotators.ai/docs](https://annotators.ai/docs).

## Get an API key

Sign up at [annotators.ai/console](https://annotators.ai/console/?signup), create a workspace, and
make a key under **API keys**. The client reads it from the environment:

```bash
export ANNOTATORS_API_KEY=ak_...
```

## Your first annotation

A task says what an item looks like and which fields to fill. Create one once:

```python
from annotators import Annotators

client = Annotators()  # reads ANNOTATORS_API_KEY

client.tasks.create(
    slug="ticket-triage",
    name="Support ticket triage",
    instructions="Pick the team that should handle the ticket.",
    fields=[
        {"name": "team", "kind": "choice", "options": ["billing", "bug", "account", "other"]},
        {"name": "urgent", "kind": "bool"},
    ],
)
```

Field kinds are `choice`, `multi_label`, `bool`, `score` (with `levels`), `extract` (a list of
strings) and `text`.

Then send an item, with what your model returned and how sure it was:

```python
a = client.annotate(
    "ticket-triage",
    input={"subject": "Charged twice", "body": "I was billed twice for October."},
    output={"team": "billing", "urgent": False},   # what your model returned
    source={"model": "my-model", "confidence": 0.62},
    metadata={"ticket_id": "T-1042"},              # comes back with the verdict
    idempotency_key="ticket-T-1042",               # retries don't create a second item
)
print(a["id"], a["status"])
```

At or above the task's `accept_threshold` (0.9 by default) the output is approved without anyone
looking. At 0.62 it goes to people, and you can review it yourself under **Review** in the console.
Wait for the verdict, or have it sent to a webhook (below):

```python
a = client.annotations.wait(a["id"])     # polls; fine for scripts
print(a["verdict"], a["final_output"])   # approved, corrected or rejected
```

`final_output` is the answer to use: your output if it was approved, the reviewer's if they
corrected it, and `None` if it was rejected.

## Who reviews

Left out, `workforce` is your own team (`private`): the members of your console workspace and the
reviewers you invite.

```python
r = client.reviewers.invite("sam@example.com", display_name="Sam")   # the token is shown once
client.reviewers.remove(r["id"])                                     # frees their seat
```

Invited reviewers and members who review in the console share your plan's reviewer seats. Past the
limit, `invite` fails with `AnnotatorsError` 402.

The annotators.org crowd (`workforce="open"` or `"vetted"`) is on a waitlist. Join it under
**Review** in the console. Until your workspace has crowd access, asking for the crowd fails with
`AnnotatorsError` 403.

## Webhooks

Set a webhook URL under **Settings** in the console, or pass `callback_url=` with a call. Each
completed annotation or survey is POSTed there, signed with your workspace's signing secret (also
under **Settings**), and retried with backoff until your endpoint answers 2xx. Verify the signature
against the raw request body:

```python
from flask import Flask, request
from annotators import WebhookVerificationError, verify

app = Flask(__name__)

@app.post("/annotators")
def annotators_webhook():
    try:
        event = verify(request.get_data(), request.headers.get("Annotators-Signature", ""),
                       WEBHOOK_SECRET)
    except WebhookVerificationError:
        return "", 400
    if event["type"] == "annotation.completed":
        a = event["data"]
        save_triage(a["metadata"]["ticket_id"], a["final_output"], a["verdict"])
    return "", 204
```

`verify` rejects a bad signature, and a timestamp more than five minutes off, then returns the
parsed event. Retried deliveries keep their event `id`, so you can skip ones you've seen.

## Machine checks

With a confidence in `source`, overall or per field (`"field_confidence": {"urgent": 0.4}`), your
number decides: at or above `accept_threshold` the output is approved, below it people review it.
Fields you give no confidence for go to people. Without a confidence, our decision model checks the
output. Pass `check="annotators-judge-1"` (with the call or in the task's `defaults`) to have the
decision model check it even when you send a confidence, or `check="none"` to skip the machine.

`mode` picks who looks: `"auto"` (the default; machine first, people only if doubtful), `"human"`
(always people), `"machine"` (never people) or `"sample"` (people review a `sample_rate` share).

## No model of your own

Leave `output` out and the item is labeled for you, from the task's examples and any you pass with
the call:

```python
a = client.annotate(
    "ticket-triage",
    input={"subject": "Can't log in", "body": "Password reset email never arrives."},
    examples=[{"input": past.input, "output": past.output}],
    mode="machine",            # never people: doubtful results come back as low_confidence
)
team, confidence = a["final_output"], a["agreement"]
```

## Surveys

Ask a number of people for their reaction to a message, headline or document (a task with
`kind="survey"`):

```python
s = client.surveys.create("message-reaction", input={"message": draft}, n_responses=50)
```

## Errors and settings

API errors raise `AnnotatorsError`, with the HTTP `status` and the API's `detail`.

```python
Annotators(api_key="ak_...", base_url="https://api.annotators.ai", timeout=30)
```

Both default to the environment: `ANNOTATORS_API_KEY` and `ANNOTATORS_BASE_URL`. The client is a
context manager (`with Annotators() as client:`) and closes its HTTP connection on exit.

## Links

- [annotators.ai](https://annotators.ai)
- [API reference](https://annotators.ai/docs)
- [Console](https://annotators.ai/console/)
- [MCP server](https://pypi.org/project/annotators-mcp/) for agents: `request_human_review` as a tool
