Metadata-Version: 2.4
Name: thetabench
Version: 0.2.0
Summary: Gymnasium-compatible SDK for ThetaBench web agent training & evaluation
Project-URL: Homepage, https://github.com/theta-rl-lab/thetabench
Project-URL: Documentation, https://github.com/theta-rl-lab/thetabench
Project-URL: Repository, https://github.com/theta-rl-lab/thetabench
Project-URL: Issues, https://github.com/theta-rl-lab/thetabench/issues
Author: Rahul Sulegaokar
License-Expression: MIT
License-File: LICENSE
Keywords: benchmark,evaluation,gymnasium,reinforcement-learning,web-agent
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.10
Requires-Dist: gymnasium>=1.0
Requires-Dist: httpx>=0.27
Requires-Dist: pydantic>=2.0
Provides-Extra: browser
Requires-Dist: playwright>=1.40; extra == 'browser'
Provides-Extra: dev
Requires-Dist: pytest-asyncio>=0.23; extra == 'dev'
Requires-Dist: pytest>=8; extra == 'dev'
Requires-Dist: respx>=0.21; extra == 'dev'
Description-Content-Type: text/markdown

# ThetaBench Python SDK

Gymnasium-compatible SDK for training and evaluating web agents on
ThetaBench environments.

## Installation

```bash
pip install thetabench

# With browser-mode support (Playwright)
pip install 'thetabench[browser]'
playwright install chromium
```

## Quick start

The SDK talks to a running ThetaBench server over HTTP. Point it at any URL
that exposes the standard `/api/health`, `/api/sim/*`, and `/api/rl` routes
— a local dev server, a Vercel deployment, or your own self-hosted instance.

### REST mode (fast — 1-5 ms / step, ideal for RL training)

```python
import thetabench

env = thetabench.make("http://localhost:3000", task_id="<task-id>")
obs, info = env.reset()

print(f"Task: {info['task_goal']}")
print(f"Max steps: {info['max_steps']}")

done = False
while not done:
    action = my_agent(obs, info)
    obs, reward, terminated, truncated, info = env.step(action)
    done = terminated or truncated

result = env.finish()
print(f"Score: {result['score']:.0%} | Steps: {result['steps']} | Reward: {result['total_reward']:.2f}")
env.close()
```

### Browser mode (realistic — Chromium + accessibility tree, ~100-500 ms / step)

```python
import thetabench

env = thetabench.make(
    "http://localhost:3000",
    task_id="<task-id>",
    mode="browser",
    headless=True,
)
obs, info = env.reset()

# obs includes screenshot, accessibility tree, URL, and structured state
print(f"URL: {obs['url']}")

obs, reward, terminated, truncated, info = env.step({"type": "navigate", "url": "/some/path"})
obs, reward, terminated, truncated, info = env.step({"type": "fill", "selector": "#price", "value": "34.99"})
obs, reward, terminated, truncated, info = env.step({"type": "click", "selector": "button:has-text('Save')"})

result = env.finish()
env.close()
```

### Curriculum training

```python
import thetabench

def my_agent(obs, info):
    # your decision logic here
    return {"action": "navigate", "url": "/some/path"}

runner = thetabench.CurriculumRunner(
    base_url="http://localhost:3000",
    mastery_threshold=0.8,
)

for stage_result in runner.run(my_agent):
    print(f"Stage {stage_result['stage']}: {stage_result['title']}")
    print(f"  Score: {stage_result['avg_score']:.0%}")
    print(f"  Mastery: {'Yes' if stage_result['mastery_achieved'] else 'No'}")

runner.close()
```

### Browse the task catalog

```python
from thetabench import ThetaBenchClient

client = ThetaBenchClient("http://localhost:3000")

# List all tasks on the connected server
tasks = client.list_tasks()
print(f"Total tasks: {tasks.total}")

# Filter by domain / difficulty / type / curriculum stage
filtered = client.list_tasks(domain="<domain>", difficulty="easy")
for t in filtered.tasks:
    print(f"  {t.id}: {t.title}")

# Walk the curriculum
curriculum = client.get_curriculum()
for stage in curriculum.stages:
    print(f"Stage {stage.stage}: {stage.title} ({len(stage.taskIds)} tasks)")

# Server self-identity (site, version, task counts)
health = client.fetch_health()
print(f"Connected to: {health['site']} (v{health['version']}, {health['tasks']} tasks)")

client.close()
```

## CLI

```bash
thetabench --url http://localhost:3000 info
thetabench --url http://localhost:3000 tasks --domain <domain> --difficulty easy
thetabench --url http://localhost:3000 run --task <task-id> --agent path/to/your_agent.py
thetabench --url http://localhost:3000 eval --agent path/to/your_agent.py --output results.json
```

Your agent file must export `agent_step(obs, info) -> action`. Optionally
`agent_response(obs, info) -> str | None` for retrieval / impossibility
tasks. Optionally a module-level `MODEL = "..."` for results metadata.

## API Reference

### `thetabench.make(base_url, task_id, *, mode='rest', **kwargs) -> ThetaBenchEnv`

Factory that constructs a `ThetaBenchEnv` pointed at a running server.

| Param | Type | Default | Description |
|---|---|---|---|
| `base_url` | str | (required) | Server URL — local dev server, Vercel deployment, or self-hosted. |
| `task_id` | str | (required) | Task identifier exposed by the server's `/api/sim/tasks`. |
| `mode` | `"rest" \| "browser"` | `"rest"` | Transport mode. |
| `**kwargs` | | | Forwarded to `ThetaBenchEnv` — `seed`, `config_overrides`, `headless`, `viewport`, `render_mode`. |

The server self-identifies its site at `/api/health`; the SDK does not
maintain a hardcoded site → URL map.

### `ThetaBenchEnv` (Gymnasium)

| Method | Returns | Description |
|---|---|---|
| `reset()` | `(obs, info)` | Start a new episode. |
| `step(action)` | `(obs, reward, terminated, truncated, info)` | Execute one action. |
| `evaluate()` | `dict` | Mid-episode score check (non-destructive). |
| `finish(agent_response=None)` | `dict` | End the episode and get the final score. |
| `close()` | `None` | Release HTTP + browser resources. |

### `CurriculumRunner`

| Method | Description |
|---|---|
| `run(agent_fn, agent_response_fn=None, start_stage=1)` | Generator yielding stage results, advancing on mastery. |
| `run_stage(stage, agent_fn, ...)` | Run all tasks in one stage. |
| `run_task(task_id, agent_fn, ...)` | Run a single task. |
| `get_curriculum()` | List of stages from the server. |

### `BatchRunner`

| Method | Description |
|---|---|
| `run_all(domain=None, difficulty=None, task_type=None)` | Run every (filtered) task; returns standardized JSON results with per-domain and per-stage breakdowns. The `meta.site` field is filled from the server's `/api/health` response. |
| `close()` | Close the underlying HTTP client. |

### `ThetaBenchClient`

Thin typed httpx wrapper over the standard endpoints. See the source for
the full method list — `start_episode`, `finish_episode`, `evaluate`,
`step`, `observe`, `reset_env`, `get_state`, `get_snapshot`, `get_episode`,
`list_tasks`, `get_task`, `get_curriculum`, `log_action`, `get_action_space`,
`fetch_health`.

## License

MIT — see [LICENSE](./LICENSE).
