Metadata-Version: 2.4
Name: testery
Version: 1.18.0
Summary: Testery CLI
Home-page: https://github.com/testery/testery-cli
Author: Testery
Author-email: chris.harbert@testery.io
Classifier: Programming Language :: Python :: 3
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: Click
Requires-Dist: requests>=2.28.0
Requires-Dist: requests_toolbelt
Requires-Dist: python-dateutil
Requires-Dist: python-dotenv
Dynamic: author
Dynamic: author-email
Dynamic: classifier
Dynamic: description
Dynamic: description-content-type
Dynamic: home-page
Dynamic: license-file
Dynamic: requires-dist
Dynamic: summary

# Testery CLI

A Click-based Python CLI that wraps the [Testery](https://testery.io) REST API and
provides a **generic test harness** for running your tests locally or on Testery's
cloud testing grid. Kick off runs from CI/CD, manage environments and schedules,
upload build artifacts, and standardize how your tests are invoked and reported.

## Installation

Requires Python 3 and pip:

```bash
pip install testery
```

Upgrade with:

```bash
pip install testery --upgrade
```

This installs the `testery` console script on your PATH.

## Authentication

Most commands talk to the Testery API and need an API token. The `test` command
only needs a token when running `--remote`; local runs need no token.

A token is resolved in this order:

1. `--token <token>` passed on the command line
2. `--profile <name>` → the named profile in `~/.testery/credentials`
3. the `[default]` profile in `~/.testery/credentials`
4. the `TESTERY_API_TOKEN` environment variable

The credentials file is INI-style:

```ini
[default]
token = your-token-here

[dev]
token = a-different-token
```

### Log in and save a token

The easiest way to populate the credentials file is `login`, which opens the Testery
app in your browser. Navigate to **Settings > Integrations** to find or create an API
token, then paste it back into the prompt (a URL containing `?token=...` is also
accepted):

```bash
testery login                 # saves to [default]
testery login --profile dev   # saves to [dev]
```

### Verify a token

```bash
testery verify-token --token <yourTesteryApiToken>
```

> Add the hidden `--testery-dev` flag to any command to target the Testery dev API
> (`https://api.dev.testery.io`) instead of production.

---

## `testery test` — the generic test harness

`test` is a standardized wrapper around your test framework's runner. It runs your
tests **locally by default** (on Windows, Linux, or macOS) or **on Testery** with
`--remote`, using the same options either way. It autodetects the framework,
installs the runner if needed, controls the output format, and reports a normalized
pass/fail summary.

> `test` replaces and deprecates [`create-test-run`](#create-test-run-legacy).
> `create-test-run` continues to work unchanged for existing pipelines.

### Run tests locally

```bash
# Autodetect the framework in the current directory and run it
testery test

# Point at a project directory and pick the framework explicitly
testery test --working-dir ./web --framework playwright

# Filter by tags, choose how many workers, and request JUnit + JSON artifacts
testery test --framework playwright \
  --include-tags "@smoke" \
  --runners 4 \
  --output-format junit --output-format json
```

Supported frameworks today: **`playwright`** and **`cucumber-js`** (more to come).
With `--framework autodetect` (the default) the harness inspects `package.json` and
config files to choose the runner — a project with a `cucumber.js` config is run with
cucumber-js even when Playwright is also a dependency.

If the runner or its browsers aren't installed, the harness installs them
(`npm install` / `npx playwright install`). Pass `--no-install` to skip this.

### Output formats (`--output-format`)

Repeatable. Controls the result artifacts produced for local runs:

| Format   | Behavior |
| -------- | -------- |
| `native` | (default) Streams exactly what the underlying runner prints — identical to running e.g. `npx playwright test` yourself. |
| `junit`  | Writes a JUnit XML report to the output directory. |
| `json`   | Writes a JSON report to the output directory. |

Artifacts are written to `--output-dir` (default `testery-results/`). A normalized
pass/fail summary is always printed using `--output` (`pretty`, `json`, or `teamcity`).

```bash
testery test --output-format native --output-format junit --output-dir ./reports
```

### Passing framework-specific arguments

Two ways, designed so new runner flags never require a CLI change:

```bash
# 1. Everything after `--` is forwarded verbatim to the runner
testery test --framework playwright -- --grep @smoke --workers 4 --headed

# 2. Or as a single quoted string
testery test --framework playwright --framework-params "--grep @smoke --workers 4"
```

### Injecting variables

`--variable KEY=VALUE` (repeatable) is injected into the runner's process
environment for local runs (and sent as run variables for `--remote`):

```bash
testery test --variable TARGET_URL=https://staging.example.com --variable DEBUG=1
```

### Run tests on Testery (remote)

`--remote` submits a run to the Testery cloud. This path is identical to
`create-test-run` and requires `--token`, `--project`, and `--environment`:

```bash
testery test --remote \
  --token <yourTesteryApiToken> \
  --project <projectKey> \
  --environment <environmentKey> \
  --build-id <uniqueBuildId> \
  --wait-for-results --fail-on-failure
```

### Key options

| Option | Applies to | Description |
| ------ | ---------- | ----------- |
| `--local` / `--remote` | both | Run on this machine (default) or submit to Testery. |
| `--working-dir`, `--project-dir` | local | Project directory to run. Defaults to the current directory. |
| `--framework` | local | `playwright`, `cucumber-js`, or `autodetect` (default). |
| `--output-format`, `--output-formats` | local | `native` (default), `junit`, `json`. Repeatable. |
| `--output-dir` | local | Where junit/json artifacts are written. |
| `--framework-params` | local | Extra runner arguments as a single quoted string. |
| `--install` / `--no-install` | local | Auto-install the runner/browsers if missing (default: install). |
| `--include-tags` / `--exclude-tags` | both | Comma-separated tag filters. |
| `--test-filter-regex` | both | Filter tests by regular expression. Repeatable. |
| `--runner-count`, `--runners` | both | Parallel runners (remote) / workers (local). |
| `--playwright-project` | both | The Playwright project to run. |
| `--retry-failed-tests` | both | Retry failed tests once. |
| `--variable` | both | `KEY=VALUE` (use `secure:KEY=VALUE` to encrypt for remote). Repeatable. |
| `--output` | both | Summary format: `pretty` (default), `json`, `teamcity`. |
| `--fail-on-failure` | both | Exit non-zero if any test fails. |
| `--wait-for-results` | remote | Poll until the remote run completes. |

Remote-only options accepted for a uniform interface: `--git-ref`/`--commit`,
`--git-branch`/`--branch`, `--test-name`, `--status-name`, `--test-suite`,
`--latest-deploy`, `--copies`, `--build-id`, `--include-all-tags`,
`--parallelize-by-file`, `--parallelize-by-test`, `--timeout-minutes`,
`--test-timeout-seconds`, `--skip-vcs-updates`, `--deploy-id`,
`--apply-test-selection-rules`.

---

## Running tests (other commands)

### create-test-run (legacy)

Submits a test run to Testery. Superseded by `testery test --remote`, but kept for
backward compatibility.

```bash
testery create-test-run --token <token> --project <projectKey> \
  --build-id <buildId> --environment <environmentKey> --wait-for-results
```

`--fail-on-failure` returns exit code 1 if there are test failures. Output formats:
`teamcity`, `pretty`, `json`, `appveyor`, `octopus`.

### run-test-plan

Submits a saved Test Plan run.

```bash
testery run-test-plan --token <token> --test-plan-key <planKey> --environment-key <envKey>
```

### upload-test-run

Uploads results from a JUnit XML file as a Testery test run.

```bash
testery upload-test-run --token <token> --project-key <projectKey> \
  --environment-key <envKey> --path ./results.xml
```

---

## Monitoring and reporting

```bash
# Follow a single run until it finishes
testery monitor-test-run --token <token> --test-run-id <id> --fail-on-failure

# Watch all active runs for N minutes
testery monitor-test-runs --token <token> --duration 10

# List currently-active runs
testery list-active-test-runs --token <token> --output json

# List recent runs (optionally filtered)
testery list-test-runs --token <token> --limit 25 --filter <filter> --output json

# Write per-test results to a file (e.g. SonarQube format)
testery report-test-run --token <token> --test-run-id <id> --output sonarcube --outfile results.xml

# Cancel runs
testery cancel-test-run --token <token> --test-run-id <id>
testery cancel-test-plan-run --token <token> --test-plan-key <planKey> --test-plan-run-id <id>
```

---

## Environments

```bash
# Create
testery create-environment --token <token> --key <key> --name <name> \
  --variable "KEY1=FOO1" --variable "secure:KEY3=SECRET" --pipeline-stage <stage>

# Update (or create with --create-if-not-exists)
testery update-environment --token <token> --key <key> --name <name> --variable "KEY1=FOO1"

# Add variables from a .env file
testery add-env-vars-from-file --token <token> --environment-key <key> --env-file ./.env --overwrite

# Upload a file that tests can read from the working directory
testery upload-environment-file --token <token> --environment-key <key> \
  --file-name config.json --source-path ./config.json

# List / delete
testery list-environments --token <token> --pipeline-stage <stage> --show-archived
testery delete-environment --token <token> --key <key>
```

---

## Pipeline stages

```bash
# Update or create a pipeline stage with variables
testery update-pipeline-stage --token <token> --name <name> \
  --variable "KEY1=FOO1" --variable "secure:KEY2=SECRET" --create-if-not-exists

# Add variables from a .env file
testery add-stage-vars-from-file --token <token> --name <name> --env-file ./.env --overwrite
```

---

## Schedules

```bash
# Run on a cron interval
testery create-schedule --token <token> --schedule-name "Nightly" --schedule-type interval \
  --cron "0 2 * * *" --project-key <projectKey> --environment-key <envKey>

# Run when a project is deployed
testery create-schedule --token <token> --schedule-name "On Deploy" --schedule-type deploy \
  --project-key <projectKey> --environment-key <envKey> --deploy-project <projectKey>

# List / delete
testery list-schedules --token <token> --output json --show-archived
testery delete-schedule --token <token> --name "Nightly"
```

---

## Deploys

Notify Testery of a deploy (triggers any `ON_DEPLOY` schedules).

```bash
testery create-deploy --token <token> --project <projectKey> --environment <envKey> \
  --build-id <buildId> --commit <gitSha> --wait-for-results --fail-on-failure
```

---

## Build artifacts

```bash
# Upload a file or directory of artifacts tied to a build id
testery upload-build-artifacts --token <token> --project <projectKey> \
  --build-id <buildId> --path ./dist

# Attach a file to an existing test run
testery add-file --token <token> --test-run-id <id> --kind DotCover ./coverage.json
```

---

## Account

```bash
testery verify-token --token <token>          # check a token is valid
testery login                                 # save a token to ~/.testery/credentials
testery load-users --token <token> --user-file ./users.txt   # bulk-add users (one email per line)
```

---

## Development

```bash
pip install -r requirements.txt    # install dev + runtime deps
pip install --editable .           # local editable install (provides `testery` on PATH)
python setup.py bdist_wheel        # build a wheel
```

### Testing

```bash
pytest tests/test_unit.py          # unit tests (no network/token needed)
flake8                             # lint
```

- Unit tests mock `requests.*` and the local runner subprocess, and drive commands
  with Click's `CliRunner` — see `tests/test_unit.py`.
- Integration tests in `tests/test_integration.py` hit the real API and require
  `TESTERY_TOKEN` (loaded from `.env`). Set `USE_TESTERY_DEV=True` to target dev.

### Docker harness (testing the `test` command on Linux)

`tests/e2e/Dockerfile` builds an image that pip-installs the CLI and runs
`testery test` against cucumber-js and Playwright fixture projects, validating the
install and the local-execution path on Linux:

```bash
docker build -f tests/e2e/Dockerfile -t testery-harness-smoke .
docker run --rm testery-harness-smoke
# or, as a pytest (skipped when Docker is unavailable):
pytest tests/e2e/test_docker_harness.py
```
