Metadata-Version: 2.4
Name: regrun
Version: 0.1.0
Summary: YAML-driven regression test runner for HTTP APIs, MCP servers, and shell commands
License-Expression: MIT
License-File: LICENSE
Author: David Marsa
Author-email: david.marsa@neomanex.com
Requires-Python: >=3.11
Classifier: Development Status :: 4 - Beta
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Software Development :: Testing
Requires-Dist: click (>=8.1)
Requires-Dist: httpx (>=0.28)
Requires-Dist: jinja2 (>=3.1)
Requires-Dist: jsonpath-ng (>=1.6)
Requires-Dist: pydantic (>=2.7)
Requires-Dist: pydantic-settings (>=2.3)
Requires-Dist: pyyaml (>=6.0)
Requires-Dist: structlog (>=24.0)
Requires-Dist: websockets (>=13.0)
Project-URL: Homepage, https://github.com/daviunx/regrun
Project-URL: Issues, https://github.com/daviunx/regrun/issues
Project-URL: Repository, https://github.com/daviunx/regrun
Description-Content-Type: text/markdown

# regrun

YAML-driven regression test runner for HTTP APIs, MCP servers, and shell commands.

Define your test cases declaratively in YAML files, then run them against live services with a single command.

## Install

```bash
pip install regrun
```

## Quick Start

```bash
# Run all tests in a directory
regrun run tests/regression/

# Filter by layer (setup auto-included)
regrun run tests/regression/ --layer api

# Dry run — show test plan without executing
regrun run tests/regression/ --dry-run

# JSON output with verbose logging
regrun run tests/regression/ --output json --verbose
```

## CLI Reference

```
regrun run <test-dir> [OPTIONS]
```

| Option | Type | Default | Description |
|--------|------|---------|-------------|
| `--layer` | `setup\|api\|mcp\|chat` | all | Filter to one layer (setup auto-included unless `--skip-setup`) |
| `--group` | `1,2,3` | all | Comma-separated group IDs |
| `--priority` | `high\|medium\|low` | all | Filter groups by priority |
| `--dry-run` | flag | false | Print test plan, do not execute |
| `--output` | `text\|json` | `text` | Output format |
| `--verbose`, `-v` | flag | false | Log request/response bodies |
| `--fail-fast` | flag | false | Stop on first failure |
| `--skip-setup` | flag | false | Skip setup layer |

## YAML Format

Each `.yaml` file in the test directory defines a set of test groups:

```yaml
meta:
  product: my-api
  layer: api                # setup | api | mcp | chat
  runner: httpx             # httpx | fastmcp | bash | websocket
  endpoint: "http://localhost:8000"
  default_auth: prod

variables:
  RUN_ID: "{{timestamp}}"   # Built-in: unix timestamp + 4 hex chars

auth:
  prod:
    type: bearer
    token: "{{PROD_JWT}}"
    org_header: "my-org"

groups:
  - id: 1
    name: "Health Check"
    priority: high
    tests:
      - id: "T.1"
        name: "GET /health"
        method: GET
        path: "/health"
        auth: none
        assert:
          status: 200
```

### Runners

- **httpx** — HTTP API tests (`method`, `path`, `body`, `query_params`)
- **fastmcp** — MCP tool calls via fastmcp CLI (`tool`, `args`)
- **bash** — Shell commands (`commands` list with `cmd` and optional `capture`)
- **websocket** — Streaming WebSocket tests (`url`, `send`, `wait_for`)

### Assertion Operators

| Assertion | Scope | Example |
|-----------|-------|---------|
| `status` | HTTP status code | `status: 200` or `status: [200, 201]` |
| `is_error` | MCP error flag | `is_error: false` |
| `has_error` | WebSocket error | `has_error: false` |
| `last_exit_code` | Bash exit code | `last_exit_code: 0` |
| `contains` | Body substring | `contains: "success"` |

**JSONPath operators** (under `json_path:`):

`exists`, `not_empty`, `equals`, `contains`, `gt`, `gte`, `lt`, `lte`, `starts_with`, `matches`

### Variable Capture

```yaml
capture:
  TASK_ID: "$.id"        # JSONPath from response
  RAW: stdout            # Full stdout (bash only)
```

Captured variables propagate to all subsequent tests across files.

## Environment Variables

| Variable | Default | Description |
|----------|---------|-------------|
| `REGRUN_API_ENDPOINT` | — | Override `meta.endpoint` in all files |
| `REGRUN_MCP_ENDPOINT` | — | Override `meta.mcp_endpoint` in all files |
| `REGRUN_TIMEOUT` | `30` | Per-test HTTP timeout (seconds) |
| `REGRUN_MCP_TIMEOUT` | `60` | Per-test MCP call timeout (seconds) |
| `REGRUN_WS_TIMEOUT` | `30` | Per-test WebSocket timeout (seconds) |
| `REGRUN_VERBOSE` | `false` | Log request/response bodies |

## License

MIT

