Metadata-Version: 2.4
Name: testorim
Version: 0.1.0
Summary: Run Testorim AI browser tests from Python and pytest: plain-English tests, real browsers, verdicts with evidence.
Project-URL: Homepage, https://testorim.com
Project-URL: Documentation, https://docs.testorim.com
Author-email: Testorim <info@fulgic.com>
License-Expression: MIT
License-File: LICENSE
Keywords: ai-qa,browser-testing,ci,e2e,end-to-end,playwright,pytest,qa,testing,testorim
Classifier: Development Status :: 4 - Beta
Classifier: Framework :: Pytest
Classifier: Intended Audience :: Developers
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Software Development :: Quality Assurance
Classifier: Topic :: Software Development :: Testing
Classifier: Topic :: Software Development :: Testing :: Acceptance
Classifier: Typing :: Typed
Requires-Python: >=3.9
Description-Content-Type: text/markdown

# Testorim for Python

[Testorim](https://testorim.com) is an AI QA service: describe a web test in plain English, or pick a saved one, and Testorim runs it in a real browser and returns a verdict with the evidence. This package runs Testorim tests from Python code and from pytest.

## Install

```bash
pip install testorim
```

Python 3.9 or newer. The package has no dependencies.

## Quick start

Create an API key in Testorim under **Settings, API keys**, then:

```bash
export TESTORIM_API_KEY=tst_live_...
```

```python
from testorim import Testorim

client = Testorim()  # reads TESTORIM_API_KEY

run = client.run_test(
    "shop.example.com",  # a project's name, address or id
    "Add the Blue Top to the cart, open the cart and check that it shows 1 item.",
)
print(run.summary())
run.assert_passed()  # raises TestorimRunFailed unless the verdict is passed
```

`run_test` starts the run and waits for the verdict, polling every 3 seconds for up to 600 seconds. Name the buttons, fields and text as they appear on the page.

More of the client:

```python
client.projects()                                   # [Project(id, name, base_url), ...]
client.create_project("https://shop.example.com")   # returns the existing project if the address is already tested
client.tests("Shop")                                # the project's saved tests

# Replay a saved test: by name within a project, or by id
run = client.run_saved("Checkout", project="Shop")
run = client.run_saved("5f0c...")

# A negative test passes when the app refuses what the description tries
client.run_test("Shop", "Sign in with a wrong password and check that an error is shown.", expect_failure=True)

# Run against another address, for example a pull request's preview
client.run_test("Shop", "...", base_url="https://pr-42.preview.shop.example.com")

# Start without waiting, then wait, read or stop it
run = client.run_test("Shop", "...", wait=False)
run = client.wait_for(run.id, timeout=900, on_status=lambda r: print(r.status))
run = client.get_run(run.id)
client.cancel(run.id)
```

When the wait runs out, `run_test`, `run_saved` and `wait_for` return the run as it is (`run.done` is False) instead of raising. Pass `raise_on_timeout=True` to get `TestorimTimeout` instead.

The site has to be reachable from the internet: Testorim's browsers refuse `localhost` and private addresses. To test work in progress, run against a preview deployment or expose your dev server through a tunnel. Every run counts against your workspace's plan.

### The run

| Attribute | What |
|---|---|
| `id`, `url` | The run's id and its page in Testorim |
| `status` | `pending`, `running`, `completed`, `failed` or `cancelled` |
| `verdict` | `passed`, `failed` or `needs_review` once finished; `None` while it runs and for a cancelled run |
| `passed`, `failed`, `needs_review`, `cancelled`, `done` | True or False |
| `passed_count`, `failed_count`, `skipped_count`, `duration_seconds` | Step counts and how long the steps took |
| `report` | The written report, in Markdown |
| `steps`, `failed_steps` | The steps, and the failed ones: `number` (from 1), `action`, `target`, `error`, `blame`, `blame_text`, `unconfirmed` |
| `start_url`, `final_url` | Where the run started and ended |
| `video_url`, `trace_url`, `screenshot_url` | The recording, the Playwright trace and the final screenshot. These links expire an hour after they were read; `get_run` gives fresh ones |
| `summary()` | Readable lines: verdict, counts, why, each failed step and the run's page |
| `assert_passed()` | Returns the run if it passed, else raises `TestorimRunFailed` with the summary |

`summary()` leaves the evidence links out, because they open without signing in and the summary often lands in CI logs.

## pytest

Installing the package adds a pytest plugin:

```python
import pytest

@pytest.mark.testorim
def test_checkout(testorim):
    testorim.run_saved("Checkout", project="Shop").assert_passed()

@pytest.mark.testorim
def test_sign_up(testorim):
    testorim.run_test(
        "Shop",
        "Sign up with a new email address and check that the dashboard says Welcome.",
    ).assert_passed()
```

- **`testorim` fixture**: a client for the session. Without `TESTORIM_API_KEY` the test fails with a message saying how to create a key; set `TESTORIM_SKIP_WITHOUT_KEY=1` to skip it instead.
- **`@pytest.mark.testorim`**: select these tests with `pytest -m testorim`, or leave them out with `pytest -m "not testorim"`.
- **`--testorim-base-url URL`**: every run made through the fixture opens this address instead of the project's own. A `base_url` passed to a single run still wins.

A failed run fails the test with its summary:

```
E   testorim.errors.TestorimRunFailed: Verdict: FAILED (1 passed, 1 failed, 1 skipped, 41s)
E   Run: https://app.testorim.com/runs/...
E   Failed step 2: click "Create account": The page showed Something went wrong [the app did not behave as described]
```

### GitHub Actions: test each pull request's preview

```yaml
name: Browser tests
on: pull_request

jobs:
  testorim:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - run: pip install testorim pytest
      - name: Run the Testorim tests against the preview
        env:
          TESTORIM_API_KEY: ${{ secrets.TESTORIM_API_KEY }}
          TESTORIM_SKIP_WITHOUT_KEY: "1" # pull requests from forks get no secrets: skip there
        run: pytest -m testorim --testorim-base-url "https://pr-${{ github.event.number }}.preview.example.com"
```

Put your preview's real address in `--testorim-base-url`, and make sure the preview is deployed before this job runs. A saved test that types a saved password runs only on its project's own host or on one of the project's environments, so Testorim refuses to replay it on a preview at another host.

## Verdicts and blame

| Verdict | Meaning |
|---|---|
| `passed` | Every step passed and the report confirms the request was met |
| `failed` | A step failed. Each failed step says who was at fault (below) |
| `needs_review` | Every step passed, but the report could not confirm the request was met. `summary()` gives the reason |
| `None` | The run was cancelled, or has not finished |

Each failed step carries a `blame`:

| `blame` | Meaning |
|---|---|
| `app` | The app did not behave as described: a real finding |
| `test` | The test could not do what it described; check the wording against the page |
| `unsupported` | The check asked for is not supported |
| `internal` | A Testorim internal error, not your app |

`unconfirmed` is True when a step found text that differs from what was expected but cannot tell whether the page or the expected text is wrong.

`assert_passed()` raises for every verdict but `passed`, so a run that needs review stops CI for a person. To let it through, check `run.failed` yourself.

## Errors

Every exception derives from `TestorimError`.

| Exception | When | Attributes |
|---|---|---|
| `AuthenticationError` | 401: the API key is missing, wrong, revoked or expired | `status`, `body` |
| `RefusedError` | 402, 429 or 503: Testorim will not start the run now | `code`, `upgrade_url`, `retryable`, `retry_after` |
| `NotFoundError` | 404, or no project or saved test matches the name you gave | `code` (`other_workspace` when the key's owner has it in another workspace) |
| `ApiError` | Any other error status, such as 400, 403 (`read_only_role`, `api_key_scope`) or 500. The three above derive from it | `status`, `body`, `code` |
| `TestorimUnreachable` | The API could not be reached; the message names the host | |
| `TestorimRunFailed` | `assert_passed()` on a run that did not pass. Also an `AssertionError` | `run` |
| `TestorimTimeout` | A wait with `raise_on_timeout=True` ran out | `run` |
| `TestorimError` | No API key, or a name that matches more than one project or saved test | |

`RefusedError.code` is one of:

| `code` | Status | Meaning | `retryable` |
|---|---|---|---|
| `onboarding_exhausted` | 402 | The account has never subscribed. `upgrade_url` is the pricing page | No |
| `locked` | 402 | Runs are paused: the plan ended or a payment is overdue. `upgrade_url` is the pricing page | No |
| `plan_quota_exhausted` | 429 | The period's runs, browser minutes or AI allowance are used up | Yes, once the period renews |
| `concurrency_limit` | 429 | The plan's concurrent runs are all in use; no run was created | Yes, when a run finishes |
| `service_busy` | 503 | Testorim's own capacity, not your quota | Yes |

A 429 with no `code` is a rate limit; `retry_after` says how many seconds to wait. The client never retries a trigger by itself: a retried trigger is a second run.

## Configuration

| Setting | Default | What |
|---|---|---|
| `TESTORIM_API_KEY` or `Testorim(api_key=...)` | none | The API key, from Settings, API keys |
| `TESTORIM_API_URL` or `Testorim(api_url=...)` | `https://app.testorim.com` | Set it if your workspace lives on another host |
| `Testorim(timeout=...)` | `30` | Seconds to wait for each API request |
| `Testorim(base_url=...)` | none | The address every run made by this client opens instead of the project's own |
| `TESTORIM_SKIP_WITHOUT_KEY=1` | off | The pytest fixture skips instead of failing when no key is set |

## Links

- Documentation: <https://docs.testorim.com>
- Coding agents (Claude Code, Codex, Cursor, VS Code and others): the Testorim MCP server, `npx -y @testorim/cli mcp`, in [`@testorim/cli`](https://www.npmjs.com/package/@testorim/cli), which is also the command-line tool
