Metadata-Version: 2.4
Name: regrun
Version: 0.10.0
Summary: YAML-driven regression test runner for HTTP APIs, MCP servers, and shell commands
License-Expression: MIT
License-File: LICENSE
Author: David Marsa
Author-email: david.marsa@neomanex.com
Requires-Python: >=3.11,<4.0
Classifier: Development Status :: 4 - Beta
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Software Development :: Testing
Requires-Dist: click (>=8.1)
Requires-Dist: fastmcp (>=3.4.2,<4.0.0)
Requires-Dist: httpx (>=0.28)
Requires-Dist: jinja2 (>=3.1)
Requires-Dist: jsonpath-ng (>=1.6)
Requires-Dist: pydantic (>=2.7)
Requires-Dist: pydantic-settings (>=2.3)
Requires-Dist: pyyaml (>=6.0)
Requires-Dist: structlog (>=24.0)
Requires-Dist: websockets (>=13.0)
Project-URL: Company, https://neomanex.com
Project-URL: Homepage, https://github.com/daviunx/regrun
Project-URL: Issues, https://github.com/daviunx/regrun/issues
Project-URL: Repository, https://github.com/daviunx/regrun
Description-Content-Type: text/markdown

# regrun

Deterministic YAML-driven regression test runner for APIs, MCP servers, and WebSocket streams.

[![PyPI](https://img.shields.io/pypi/v/regrun)](https://pypi.org/project/regrun/)
[![Python 3.11+](https://img.shields.io/badge/python-3.11%2B-blue)](https://www.python.org/)
[![License: MIT](https://img.shields.io/badge/license-MIT-green)](LICENSE)

---

## What is regrun?

regrun lets you define regression tests as YAML files and run them against live services, with no test framework required. You describe what to call, what to assert, and what to capture; regrun handles execution, variable interpolation, and reporting. It supports four runners: REST APIs (httpx), MCP tools (fastmcp CLI), shell commands (bash), and WebSocket streams (websocket). Tests share a variable store across files, so a JWT captured in setup is available to every subsequent test without any wiring.

---

## Installation

```bash
pip install regrun
```

Requires Python 3.11 or later.

The MCP runner requires `uvx` and the `fastmcp` CLI available on `PATH`:

```bash
pip install fastmcp
```

---

## Quick Start

Create two test files for a fictional `myapp` running at `http://localhost:8000`.

**`tests/regression/00_setup.yaml`** acquires a JWT:

```yaml
meta:
  product: myapp
  layer: setup
  runner: httpx
  endpoint: "http://localhost:8000"

variables:
  RUN_ID: "{{timestamp}}"
  TEST_EMAIL: "regtest-{{RUN_ID}}@example.com"
  TEST_PASSWORD: "TestPass123!"

groups:
  - id: 1
    name: "Auth"
    priority: high
    tests:
      - id: "S.1"
        name: "Login and capture JWT"
        method: POST
        path: "/api/v1/auth/login"
        auth: none
        org_header: false
        body:
          email: "{{TEST_EMAIL}}"
          password: "{{TEST_PASSWORD}}"
        assert:
          status: 200
          json_path:
            "$.access_token": { exists: true }
        capture:
          APP_JWT: "$.access_token"
```

**`tests/regression/01_api.yaml`** exercises the API with the captured token:

```yaml
meta:
  product: myapp
  layer: api
  runner: httpx
  endpoint: "http://localhost:8000"
  default_auth: prod

auth:
  prod:
    type: bearer
    token: "{{APP_JWT}}"

groups:
  - id: 1
    name: "Items"
    priority: high
    tests:
      - id: "A.1"
        name: "List items returns array"
        method: GET
        path: "/api/v1/items"
        assert:
          status: 200
          json_path:
            "$": { not_empty: true }
```

Run the tests:

```bash
regrun run tests/regression/
```

Expected output:

```
tests/regression/  •  2 tests

  [PASS]  S.1  Login and capture JWT         (142ms)
  [PASS]  A.1  List items returns array       (38ms)

  2 passed, 0 failed  •  180ms
```

---

## Failure Diagnostics & Run Artifacts

**One run tells you everything about a failure, by default.** No flag needed. When a test fails or errors, regrun renders a `Failures` section between the results table and the summary, so `Result: PASS|FAIL` stays the last line while a tail-clipped terminal still shows the diagnostics:

```
Failures (1)
============================================================

[Tools] A9.1  List tools
  request: GET http://api:8000/tools
  headers: {'Authorization': '[REDACTED]', 'Content-Type': 'application/json'}
  response status: 500
  response body: {"detail": "RecursionError: maximum recursion depth exceeded"}
  ✗ status: Status 500 != 200
      expected: 200
      actual:   500

------------------------------------------------------------
  Total: 1  Passed: 0  Failed: 1  ...
  Result: FAIL

Full report: /Users/you/.regrun/runs/myapp/20260711-161301/report.txt (json: report.json)
```

Each failed test's diagnostics carries the **request echo** (method/URL/headers/body for httpx, tool+args for MCP, the rendered command list for bash, url/send/wait_for for WebSocket), the **response** status + body, **every** failed assertion at full length (not truncated), and the `eventually` **attempt count**. Passing tests stay terse. `--output json` includes `diagnostics` as an additive field (omitted when null).

**Redaction:** request headers are redacted at capture time by canonical sensitive-field patterns (`authorization`, `*token*`, `*key*`, `cookie`, …), and any resolved auth-token value is scrubbed wherever it appears (including a token echoed back in a response body). Response bodies are truncated to 2000 chars (see `REGRUN_DIAG_BODY_LIMIT`) with a `…[truncated, N total chars]` annotation.

**Persistent artifacts:** *every* run (pass, fail, or `--fail-fast` abort) writes the full `report.txt` + `report.json` + `junit.xml` to `{REGRUN_RUNS_DIR or ~/.regrun/runs}/{product}/{YYYYMMDD-HHMMSS}/`, and stdout ends with the pointer line above. An AI agent (or a human) reads the file instead of re-running a multi-minute suite to see a truncated error. Timestamped dirs are never auto-pruned (plain text, negligible size).

**JUnit XML for GitLab Tests tab:** The `junit.xml` artifact follows the JUnit spec as consumed by GitLab: one `<testsuite>` per source YAML file, one `<testcase>` per test. Failed tests carry a `<failure>` element with the full diagnostics body (same redaction as report.txt), errored tests `<error>`, skipped tests `<skipped/>`. Bodies are capped at 16 KB. Wire it in your `.gitlab-ci.yml`:

```yaml
artifacts:
  when: always
  reports:
    junit: regrun-runs/**/junit.xml
  paths:
    - regrun-runs/
```

---

## How It Works (Execution Model)

**File ordering:** One order governs the whole engine: **layer rank first** (`setup`, `api`, `mcp`, `chat`; an unrecognised layer runs last), **then the file's stem in byte order**. Numeric prefixes (`00_`, `01_`, `02_`) enforce the intended order. The runner, the `--file` and `--rerun-failed` selectors, the shard planner and the linter's ordering rules all read that one definition, so a plan, a report and a lint finding can never disagree about which file comes first.

The stem is the filename without `.yaml`, and excluding the extension is deliberate: it is a constant that carries no ordering intent, and comparing it against real characters is what makes punctuated names surprising. Given two files in one layer:

| Order | Why |
|---|---|
| `00_setup` before `00_setup-extra` | A name that is a prefix of another runs first, whatever the longer one continues with. A shell's `ls` disagrees, because it compares `.yaml` against `-extra` |
| `00_setup-extra` before `00a_x` | `_` precedes `a` |
| `00.b` before all of them | `.` precedes `_` and every letter |
| `00A_x` before `00a_x` | Byte order, never case-folded: a suite's order must not depend on a locale |

Names built from digits, letters and underscores are unaffected by any of this, which is what the numeric-prefix convention is for.

> **Upgrading from 0.9.x can reorder files within a layer.** 0.9.x ordered by the full filename, extension included. The order moves for any pair whose stem is a byte-for-byte prefix of another stem that continues with a character sorting below `.` (a hyphen is the common one), so `00_setup` and `00_setup-extra` swap. Run `regrun run <dir> --dry-run` and read the file list before upgrading a pinned CI suite. Lint rule **E002** compares by the same key, so its verdict can flip for such a pair: a suite holding cleanup file `10_cleanup.yaml` next to mcp file `10_cleanup-extra.yaml` was not flagged before and is flagged now, correctly, because that mcp file does run after the cleanup file. Renaming either file clears it.

**Setup dependency:** When you pass `--layer api` or `--layer mcp`, the setup file is auto-included and runs before the target layer. When setup runs as a dependency, `--group` and `--priority` filters are not applied to it: it always runs in full so captured variables stay available. Filters apply to setup only when it is the explicit target (`--layer setup`). Skip setup entirely with `--skip-setup` when variables are already populated from a prior run segment.

**Selecting a setup file (`--file`):** The setup layer is ordered and single-homed: the first setup file owns the suite's `variables:` and `meta.env_file`, and any setup file after it may read them. So selecting a setup-layer file also runs **every setup file sorting before it** (plus their own `requires` closures). A setup file sorting *after* the selection is never pulled: nothing it produces can have existed when the selected file ran in a full suite, so needing it would be a suite defect rather than a dependency. Selecting the first setup file therefore runs that file alone, and `--skip-setup` still removes the whole layer, which makes a `--file` pattern that only matched setup files an error, never a silent zero-file run.

**Cleanup dependency (sweep-first):** A group flagged `cleanup: true` is the mirror of the setup layer on the teardown side. It is always retained under `--group` / `--priority` filters (so filtered iteration runs still sweep), and it still **executes** when `--fail-fast` aborts the run, in the failing file and every later file, while all other remaining tests are skipped. The run's exit code still reflects the original failure. Suppress cleanup groups with `--skip-cleanup` when iterating locally. Because within-run cleanup can never be guaranteed (a SIGKILL or crashed run defeats any teardown), the durable pattern is a *pattern-based, capture-independent* sweep at the **start** of the run (in `00_setup`) that deletes all prior-run artifacts. The run that needs a clean environment is the one that sweeps it. Only such capture-independent sweeps should be flagged `cleanup: true`.

**Declared file dependencies:** A file lists the files it consumes captured values from in `meta.requires:` (stems, no `.yaml`). The setup layer is the bootstrap contract every file already depends on and is never listed. Declaring a dependency buys three things: the file selectors pull the producer in automatically, sharding keeps a file and its closure together, and a failed producer reports its consumers as BLOCKED. Lint rule W012 finds the couplings a suite has not declared yet.

A `requires:` entry that names no file in the suite directory, names the file itself, or closes a cycle aborts the run before anything executes: there is no order that satisfies it, and dropping it silently would leave the run green with blocked-skip quietly disabled. A producer left out by your own narrowing (`--file`, `--layer`, `--shard`) is the one tolerated case, because a filtered run cannot judge a producer it never loaded.

**BLOCKED:** When a file fails, every file that requires it (directly or transitively) is not run, and its tests are reported `BLOCKED` naming the file that blocked them. A failed `layer: setup` file blocks every later file, declared dependency or not: setup is the bootstrap contract nobody declares, so a seed file reporting green after a dead auth bootstrap would send the reader to the wrong place. Blocked tests are a sub-kind of skipped: one broken producer yields one actionable failure instead of a cascade of red that all has the same cause. The exit code is driven by real failures and errors, so a run whose only red is a blocked consumer still points at the producer.

**Variable persistence:** File-level variables are merged once per file at parse time. A variable already set by an earlier file (for example `RUN_ID` defined in setup) is never overwritten by a later file's `variables` block. This ensures identifiers stay consistent across the entire run.

**Layer concept:** Tests are organised into four layers, processed in this order:

| Layer | Purpose |
|-------|---------|
| `setup` | Auth, seed data, environment configuration |
| `api` | REST API surface tests |
| `mcp` | MCP tool tests |
| `chat` | WebSocket and streaming tests |

---

## CLI Reference

```
regrun run TEST_DIR [OPTIONS]
```

`TEST_DIR` is a path to a directory containing YAML test files.

| Flag | Type | Default | Description |
|------|------|---------|-------------|
| `--layer` | `setup\|api\|mcp\|chat` | all | Filter to one layer (setup auto-included) |
| `--group` | `1,2,3` | all | Comma-separated group IDs |
| `--priority` | `high\|medium\|low` | all | Filter groups by priority |
| `--dry-run` | flag | false | Print test plan without executing |
| `--output` | `text\|json` | text | Output format |
| `--verbose`, `-v` | flag | false | Log full request/response bodies |
| `--fail-fast` | flag | false | Stop on first failure (cleanup groups still run) |
| `--skip-setup` | flag | false | Skip setup layer |
| `--skip-cleanup` | flag | false | Skip cleanup-flagged groups (use when iterating; leaks must be swept later) |
| `--skip-preflight` | flag | false | Skip `preflight:` dependency-health checks (deliberate local override) |
| `--skip-sweep` | flag | false | Skip the declared `sweep:` block (use when iterating; leaks must be swept later) |
| `--no-lock` | flag | false | Bypass the per-product run lock (allow a concurrent run for this product) |
| `--no-strict-vars` | flag | false | Render an unresolved `{{VAR}}` as a literal and warn, instead of failing the test |
| `--file` | stem or glob (repeatable) | all | Run only the matching files, plus the setup layer and the `requires` closure of each match (a setup-layer match pulls the setup files sorting before it) |
| `--rerun-failed` | flag | false | Run only the files that failed, errored or were blocked in the latest report for this product and target |
| `--shard` | `k/n` | none | Run shard `k` of `n`. **Each shard requires its own database and index prefix** |
| `--budget-seconds` | float | none | Fail the run when its wall time exceeds this many seconds |

**Time budgets** are off unless declared. A file declares its own ceiling with `meta.budget_seconds`; `--budget-seconds` covers the whole run. An overrun reds the run and the report names the file and the overrun, but it never reclassifies a test: the test passed, the budget did not.

**Sharding requires disjoint environments.** Each shard needs its own database and its own search-index prefix. Two shards against one stack overwrite each other's fixtures and both verdicts become meaningless. regrun cannot verify the precondition, so it is the caller's to honour. Shards are built from the dependency graph: a file and its `requires` closure always land in the same shard, the setup layer runs in every shard, and a `serial: true` file gets the last shard to itself. Balance comes from greedy longest-processing-time packing, by test count when no timings are supplied. Plans are deterministic, so `--dry-run --shard k/n` can be diffed before a matrix is wired.

**Narrowing does not narrow validation.** `--file`, `--rerun-failed` and `--shard` are applied *after* every discovered file has been parsed and validated, because selection reads what the files declare (the `requires` closure, the shard weights, the run order). So one schema-invalid file aborts every run of that directory, including a `--file` run that did not select it, and including `--dry-run`. That is intended: a suite holding a file the engine cannot load is not a suite a narrowed green can be trusted from, and the rest of the directory is the context the selection was computed in. (`--layer` and `--skip-setup` are the exception, because they narrow file *discovery* rather than the selection, so files they exclude are never read.)

To recover, lint the directory: `regrun lint <dir>` names the file and the exact key path as **E006**, for every offending file at once rather than one abort at a time. Fix the keys it reports, then re-run the narrowed command.

Examples:

```bash
# Smoke test only
regrun run tests/regression/ --priority high

# MCP layer only
regrun run tests/regression/ --layer mcp

# Specific groups as JSON
regrun run tests/regression/ --group 1,3 --output json

# Preview without running
regrun run tests/regression/ --dry-run

# Iterate on group 5 without running the sweep groups
regrun run tests/regression/ --group 5 --skip-cleanup

# One file, with setup and everything it requires
regrun run tests/regression/ --file 11_search_e2e

# Every file of one family (glob), repeatable
regrun run tests/regression/ --file "02_mcp_*" --file 05_flows

# One setup file, with the setup files that run before it
regrun run tests/regression/ --file 00g_seed_directory

# After a red run: re-run only what broke
regrun run tests/regression/ --rerun-failed

# Shard 2 of 3, against that shard's own stack
regrun run tests/regression/ --shard 2/3

# Diff the planned subsets before wiring a CI matrix
regrun run tests/regression/ --dry-run --shard 1/3
```

## `sql` Runner

A first-class runner for Postgres statements. It absorbs the psql half of the fleet's hand-rolled bash steps and resolves the docker-exec-vs-direct-psql dispatch **once**, in Python: no more copy-pasting the `command -v docker && docker info` guard into every suite. **No new DB driver is added:** the runner shells out to `psql` exactly as the bash steps did.

Declare a connection in `meta.sql_connection` and put SQL in a test's `sql:` field:

```yaml
meta:
  product: myapp
  layer: api
  runner: sql
  sql_connection:
    docker_container: "{{ env.get('MYAPP_COMPOSE_PROJECT', 'myapp') }}-db-1"
    docker_user: postgres
    database: "{{ env.get('MYAPP_DB', 'myapp_test') }}"
    fallback_dsn: "{{ env.get('MYAPP_DSN', 'postgres://postgres@localhost:5432/myapp_test') }}"

groups:
  - id: 5
    name: DB invariants
    tests:
      - id: SQL.1
        name: no orphaned rows
        sql: "SELECT to_jsonb(count(*)) FROM events WHERE org_id IS NULL;"
        assert:
          last_exit_code: 0
          contains: "0"
```

Dispatch: the runner probes `shutil.which("docker")` + `docker info` (cached per run). When docker is available it runs `docker exec -i {container} psql -U {user} -d {db}`; otherwise `psql {fallback_dsn}`. Every invocation carries `-v ON_ERROR_STOP=1 -q -t -A` and receives the statement on stdin. Stdout is parsed JSON-or-string exactly like the bash runner, so `contains` / `json_path` on `to_jsonb(...)` output transfer 1:1.

- **Connection values are Jinja-renderable strings**. Keep the product-prefixed env convention (`{{ env.get('MYAPP_DB', ...) }}`); there are no new `REGRUN_SQL_*` vars.
- **SQL only.** App-command steps (`docker compose exec … python -m …` seeders/reindexers) and OpenSearch curl steps stay `runner: bash`.
- **Adoption is pin-gated:** `runner: sql` **hard-fails to parse on a pre-0.8.0 binary** (Literal enforcement). A suite may adopt it only after its CI pin is ≥ 0.8.0.

## `preflight:` Dependency-Health Checks

A top-level `preflight:` block lists read-only probes that run **once, before any group**, and abort the whole run in seconds naming the failed dependency. This kills the degraded-backend grind regime (a slow/broken backend otherwise burns the full suite budget retrying).

```yaml
meta:
  product: myapp
  layer: api
  runner: httpx
  endpoint: http://myapp-api:8000

preflight:
  - name: api-reachable
    runner: httpx
    method: GET
    path: /health
    assert:
      status: 200
  - name: db-reachable
    runner: sql
    sql: "SELECT 1;"
    assert:
      last_exit_code: 0

groups:
  - id: 5
    name: API surface
    tests: [...]
```

- Checks carry a `Test`-shaped body on any runner, plus a `name` (used in the abort message) and a `timeout` (default `10.0`s).
- **Constraints (validation-enforced):** a check may **not** use `eventually:` or `capture:`. A health probe must not retry a degraded backend into looking healthy, nor feed run state.
- Checks are collected across **all** loaded files in file order. The first failure aborts: the CLI prints `PREFLIGHT FAILED: <name>` + diagnostics and exits non-zero having executed **zero** groups.
- `--skip-preflight` bypasses them (deliberate local override); `--dry-run` lists them.
- On a passing run the report header prints `preflight: N checks passed`.
- **Compat:** `preflight:` is **silently ignored by a pre-0.8.0 binary** (unknown key). A CI log missing the `preflight:` header line was run by an old pin. Lint **W006** and the header count make this detectable while pins lag.

## Per-Product Run Lock

Every run holds an exclusive `fcntl.flock` on `{REGRUN_RUNS_DIR|~/.regrun/runs}/{product}/.lock` for its duration, mechanically enforcing the sweep-first no-concurrency assumption. A second concurrent run for the same product **exits code 2** naming the product + lock path:

```
Another regression run for 'myapp' is in progress (lock: /…/.regrun/runs/myapp/.lock)
```

- flock **self-releases on process death** (incl. SIGKILL), so there is no stale-lock protocol.
- `--no-lock` bypasses the lock (allow a deliberate concurrent run).
- **Network filesystems:** flock semantics are unreliable over NFS/CIFS. `REGRUN_RUNS_DIR` is expected to be local (`~/.regrun` or the CI workspace).

### `regrun lint`

Static analysis of a suite: no network, no execution. Encodes the regression-testing discipline (assertion strength, budget floors, sweep hygiene, the `auth: none` trap) as mechanical checks so violations surface at commit time instead of months later as flakes. Exit code is `1` if any **error** rule fires (`0` otherwise); `--strict` elevates warnings to errors.

```
regrun lint TARGET [OPTIONS]
```

`TARGET` is a directory path or a product name from `regrun.yaml`.

| Flag | Type | Default | Description |
|------|------|---------|-------------|
| `--strict` | flag | false | Treat warnings as errors |
| `--budget-floor` | float | `75.0` | Minimum `eventually:` ceiling (seconds) before W003 fires |
| `--allow-positional` | glob (repeatable) | (none) | File glob(s) where positional array asserts (W002) are permitted |

| Rule | Sev | Meaning |
|------|-----|---------|
| E001 | error | Duplicate group id within a file |
| E002 | error | mcp-layer file (`runner: fastmcp` / `default_auth: mcp`) sorts after a `*cleanup*` file (the shared api_key is revoked by cleanup) |
| E003 | error | A test has `auth:` with a null value (the `auth: none` string-literal trap) |
| E004 | error | A test on an auth-consuming runner (`httpx`/`fastmcp`/`websocket`) references an auth profile absent from that file's own `auth:` block, via `auth:` or `meta.default_auth` |
| E005 | error | A `meta.requires:` entry no run can satisfy: an unknown stem, the file itself, a cycle, or a file that runs after its dependent |
| E006 | error | The file parses as YAML but does not validate against the schema: an undeclared key, a missing required key, or a wrong type. One finding per validation error, carrying the failing location and the message. Everything `run` refuses to load, `lint` reports |
| W001 | warn | MCP tool test asserts `is_error` with no `json_path` on the response |
| W002 | warn | `equals`/`contains` on a positional array path (`[0]`/`[*]`): rank-0 fragile. Suppress per-test with an inline `# lint: allow-positional` comment, or per-file with `--allow-positional` |
| W003 | warn | `eventually:` worst-case ceiling below the budget floor (default 75s) |
| W004 | warn | POST/create-shaped test whose body/args carry no `{{RUN_ID}}` and no run-scoped declared variable. An inline `{{timestamp}}` does not satisfy it: the builtin is recomputed per render, so the value is uncapturable and unsweepable. Negative tests, non-create verbs and bodyless POSTs are skipped; suppress an irreducible create with `# lint: allow-nocreate` |
| W005 | warn | Cleanup-flagged group references a variable captured in another group (capture-dependent sweep) |
| W006 | warn | The suite directory declares no `preflight:` block in any file: missing dependency-health probes (adoption nudge) |
| W007 | warn | The suite has neither a `sweep:` block nor a `cleanup: true` group sorting before its first create-shaped test (sweep-first is unenforced) |
| W008 | warn | A bash `cmd` carries a hardcoded `http(s)://` host or a `psql -d <name>` database literal not wrapped in `{{ env.get(...) }}` or `${VAR:-default}` |
| W009 | warn | A `json_path` condition whose only operator is `exists: true`, which a JSONPath match on `null` satisfies. Suppress per-test with `# lint: allow-exists` |
| W010 | warn | A `$.data.*` path in an mcp-layer file, where asserts and captures run on the post-normalize body |
| W011 | warn | Fixture-name prefixes created by create-shaped tests that appear in no sweep step or `cleanup: true` group (unswept fixture families) |
| W012 | warn | A file uses a variable another suite file captures without declaring `meta.requires:` for it. Cleared by declaring it, by producing the value locally, or by the producer being a setup file |

```bash
# Lint before committing suite changes
regrun lint tests/regression/

# CI gate: fail on any warning too
regrun lint tests/regression/ --strict
```

---

## YAML Schema Reference

### `meta` block (required)

```yaml
meta:
  product: myapp              # Used for reporting only; does not need to match any registered name
  layer: api                  # setup | api | mcp | chat
  runner: httpx               # httpx | fastmcp | bash | websocket
  endpoint: "http://localhost:8000"      # Base URL for httpx runner
  mcp_endpoint: "http://localhost:9000"  # MCP base URL; falls back to endpoint if omitted
  default_auth: prod          # Auth key applied to all tests without explicit auth:
  env_file: ".env.test"       # Path to .env file, relative to the test file's directory
  strict_vars: true           # Default true: an unresolved {{VAR}} fails the test
  requires: ["01_provider"]   # Files (stems) this one consumes captured values from
  serial: false               # True: asserts process-global state, never shares a shard
  budget_seconds: 60          # Optional wall-clock ceiling for this file
  health_path: "/health"      # Read by external orchestration only; the engine ignores it
  mcp_health_path: "/mcp/health"
```

The `product` field appears in report output. It does not need to match any external registry.

**Unknown `meta` keys are rejected at load.** A typo such as `require:` would otherwise be accepted and silently ignored, leaving the file with no declared dependency and nothing anywhere saying so.

### Unknown keys are rejected everywhere

Every block in this reference rejects keys it does not declare: the file itself, `meta`, an `auth` profile, `preflight` and `sweep` steps, a group, a test, `assert`, a bash command, `eventually`, `ws_config` and `meta.sql_connection`. A key a block does not declare is read by nothing, so accepting it would mean accepting a condition or a setting that never takes effect. (`variables:` and a `capture:` mapping are open by design: their keys are variable names you choose.)

Both entrypoints are loud about it. `regrun run` aborts before executing anything, naming the file, the location and the key:

```
Error: Failed to parse 01_items.yaml: 1 validation error for TestFile
groups.0.tests.0.commands.0.assert
  Extra inputs are not permitted [type=extra_forbidden, ...]
```

`regrun lint` reports the same thing as error **E006**, so the static gate cannot pass a file the engine would refuse:

```
01_items.yaml:A.1 E006 (error) schema violation at groups.0.tests.0.commands.0.assert: Extra inputs are not permitted
```

There is no opt-out. Read the reported location, then move the key to the block that declares it, or remove it.

### `variables` block

```yaml
variables:
  RUN_ID: "{{timestamp}}"               # Unix timestamp + 4 hex chars (unique per run)
  TODAY: "{{date}}"                     # YYYY-MM-DD
  REQUEST_ID: "{{uuid}}"               # UUID4
  API_TOKEN: "{{env.MY_SECRET_TOKEN}}"  # Environment variable passthrough
  BASE_EMAIL: "admin@myapp.io"          # Static value
```

Built-in variables:

| Variable | Description |
|----------|-------------|
| `{{timestamp}}` | Unix timestamp + 4 hex chars, unique per run, use as resource name suffix |
| `{{date}}` | Current date as `YYYY-MM-DD` |
| `{{uuid}}` | UUID4 |
| `{{env.VAR_NAME}}` | Reads `VAR_NAME` from the process environment |

Full Jinja2 template syntax is supported. The engine runs in `StrictUndefined` mode: an undefined variable logs a warning and returns the raw template string rather than raising an exception.

Variables set by earlier files are preserved. Downstream files skip re-initialization of keys that already exist in the store.

### `auth` block

```yaml
auth:
  prod:
    type: bearer                # bearer | api_key
    token: "{{APP_JWT}}"
    org_header: "myapp"         # Sets X-Org-Slug header; omit if not needed
  service_key:
    type: api_key
    token: "{{SERVICE_API_KEY}}"
```

### `groups` block

```yaml
groups:
  - id: 1
    name: "Auth Flow"
    priority: high          # high | medium | low  (default: medium)
    context: prod           # prod | fresh | both  (default: prod)
    tests:
      - ...
  - id: 2
    name: "CRUD Operations"
    priority: medium
    tests:
      - ...
  - id: 3
    name: "Environment Sweep"
    cleanup: true           # survives filters; runs even on --fail-fast abort (default: false)
    tests:
      - ...
```

`cleanup: true` marks a group as a teardown/sweep that must run even on filtered or aborted runs (see *Cleanup dependency* above). Reserve it for capture-independent, pattern-based sweeps only. `regrun lint` flags (W005) a cleanup group that depends on variables captured elsewhere.

### Test fields by runner

**httpx (REST API)**

```yaml
- id: "A.2"
  name: "Create item"
  method: POST
  path: "/api/v1/items"
  auth: prod                  # Named auth key, "none", or omit to use default_auth
  org_header: true            # false to suppress X-Org-Slug
  body:
    name: "Widget {{RUN_ID}}"
    price: 9.99
  query_params:
    expand: metadata
  assert:
    status: 201
    json_path:
      "$.id": { exists: true }
      "$.name": { starts_with: "Widget" }
  capture:
    ITEM_ID: "$.id"
```

**fastmcp (MCP tools)**

```yaml
- id: "M.1"
  name: "List items via MCP"
  tool: items_list
  args:
    status: "active"
    limit: 10
  auth: service_key
  assert:
    is_error: false
    json_path:
      "$[0].id": { exists: true }
      "$": { not_empty: true }
  capture:
    FIRST_ITEM_ID: "$[0].id"
```

**bash (shell commands)**

```yaml
- id: "S.2"
  name: "Seed test user"
  runner: bash
  commands:
    - cmd: |
        docker exec myapp-postgres psql -U postgres -d myapp \
          -c "INSERT INTO users (email) VALUES ('seed@example.com') ON CONFLICT DO NOTHING;"
      capture:
        RAW_OUTPUT: stdout
  assert:
    last_exit_code: 0
    contains: "INSERT"
```

Bash commands run from the directory where you invoke `regrun`, not from the test file location. Use absolute paths or `docker exec` rather than relative paths.

**websocket (streaming)**

```yaml
- id: "C.1"
  name: "Chat session produces response"
  url: "ws://localhost:8000/api/v1/ws/chat?session_id={{SESSION_ID}}"
  send:
    message: "What is the status of my account?"
    session_id: "{{SESSION_ID}}"
  wait_for: "agent_completed"      # Event type that terminates collection
  timeout: 60000                   # Milliseconds (overrides file-level timeout)
  ws_config:
    text_event: text_delta         # Override only if your server uses non-default field names
  assert:
    has_error: false
    json_path:
      "$.response_text": { not_empty: true }
      "$.event_count": { gt: 1 }
  capture:
    CHAT_RESPONSE: "$.response_text"
```

The runner connects, sends `send` as a JSON frame, collects events until `wait_for` is received, and returns an aggregated result dict:

| Field | Type | Description |
|-------|------|-------------|
| `response_text` | `str` | All `text_delta` fragments joined |
| `events` | `list[str]` | Ordered list of all event types received |
| `event_count` | `int` | Total number of events |
| `tool_calls` | `list[str]` | Tool names from `tool_call` events |
| `duration_ms` | `float` | Wall time from connect to termination event |
| `error` | `str\|null` | Error message if an `error` event was received or timeout occurred |

**`ws_config` options** (all have defaults; omit unless overriding):

| Field | Default | Description |
|-------|---------|-------------|
| `event_type_field` | `event_type` | Primary key used to read the event type from each frame |
| `event_type_fallback` | `type` | Fallback key if primary is absent |
| `text_event` | `text_delta` | Event type whose payload contributes to `response_text` |
| `text_field` | `data.delta` | Dot-path to the text content within a text event |
| `tool_call_event` | `tool_call` | Event type that signals a tool was called |
| `tool_name_field` | `data.tool_name` | Dot-path to the tool name within a tool call event |
| `error_event` | `error` | Event type that signals an error |
| `error_field` | `data.content` | Dot-path to the error message within an error event |

**Per-test runner override**, used in setup files that mix bash and httpx:

```yaml
# In a file with meta.runner: bash, a single test can use httpx instead:
- id: "P.1"
  runner: httpx               # Overrides the file-level meta.runner
  method: POST
  path: "/api/v1/auth/login"
  auth: none
  org_header: false
  body:
    email: "{{TEST_EMAIL}}"
    password: "{{TEST_PASSWORD}}"
  assert:
    status: 200
  capture:
    APP_JWT: "$.access_token"
```

Pure `api` or `mcp` files should not use per-test `runner:` overrides; the file's `meta.runner` applies uniformly.

---

## Assertion Vocabulary

### Top-level assertions

| Key | Values | Runner |
|-----|--------|--------|
| `status` | `200` or `[200, 201]` | httpx |
| `is_error` | `true\|false` | fastmcp |
| `has_error` | `true\|false` | websocket |
| `last_exit_code` | `0` | bash |
| `contains` | substring string | all runners |

### `json_path` operators

Each entry under `json_path:` maps a JSONPath expression to one **or more** operators. Every operator listed under a path is evaluated and reported separately, and the test passes only when all of them pass (AND):

| Operator | Example | Description |
|----------|---------|-------------|
| `exists` | `"$.id": { exists: true }` | Field presence check |
| `equals` | `"$.status": { equals: "active" }` | Exact match (string-coerced fallback) |
| `contains` | `"$.name": { contains: "Widget" }` | Substring |
| `gt` | `"$.total": { gt: 0 }` | Greater than |
| `gte` | `"$.count": { gte: 1 }` | Greater than or equal |
| `lt` | `"$.errors": { lt: 10 }` | Less than |
| `lte` | `"$.errors": { lte: 5 }` | Less than or equal |
| `starts_with` | `"$.key": { starts_with: "ntk_" }` | Prefix check |
| `matches` | `"$.slug": { matches: "^[a-z0-9-]+$" }` | Regex search |
| `not_empty` | `"$.items": { not_empty: true }` | Value is non-empty string, list, or dict |
| `not_contains` | `"$.results[*].id": { not_contains: "{{FORBIDDEN_ID}}" }` | Array exclusion: passes when no value matched by the path equals the expected value (all matches, string-coerced); empty/missing match set passes |
| `any_contains` | `"$.results[*].name": { any_contains: "{{RUN_ID}}" }` | Array presence: scans every value matched by the path and passes when at least one value's string form contains the substring, whatever its position. An empty/missing match set FAILS, the opposite of `not_contains` |

Note: numeric operators (`gt`, `gte`, `lt`, `lte`) are the correct names. `greater_than`, `less_than`, `>=`, and `<=` are not valid.

A path carrying several operators (`"$.slug": { not_empty: true, starts_with: "myapp-" }`) reports one result per operator, so the report names which one failed. An unrecognised key under a path fails the test rather than being ignored.

### Response normalization (fastmcp runner)

Assertions and captures run against the **normalized** tool response body, not the raw MCP payload. Know the shape before writing `json_path`:

| Raw shape | Normalized to | Assert against |
|-----------|---------------|-----------------|
| `{success, data: {...}}` (dict `data`) | `data` hoisted to top level; siblings (`hints`, `success`, `error_code`, ...) dropped | `$.<field>`, never `$.data.*` |
| Any other dict | Unchanged (stays flat) | `$.<field>` as returned |
| `is_error: true` with a plain-string body | Wrapped as `{is_error: true, _raw_text: "<text>"}` | `$._raw_text` with `contains` |

Captures follow the same rule: a `capture:` path written against the pre-normalize shape (e.g. `$.data.id` when `data` gets hoisted) silently captures nothing instead of erroring.

Source: `src/regrun/runners/mcp_response.py::_detect_and_normalize` (envelope hoist), `fastmcp_runner.py::_embed_is_error` (string-body wrap).

---

## Variable Capture

```yaml
capture:
  ITEM_ID: "$.id"                # JSONPath from JSON response
  OWNER_EMAIL: "$.owner.email"   # Nested path
  RAW_OUTPUT: stdout             # Full stdout (bash runner only)
```

Captured variables are stored in the shared `VariableStore` and are available to all subsequent tests in the run, including tests in later YAML files. This is how a JWT captured in `00_setup.yaml` is accessible in `01_api_surface.yaml` without any re-declaration.

**Collision avoidance:** suffix resource names with `{{RUN_ID}}` to prevent conflicts across runs:

```yaml
body:
  name: "Test item {{RUN_ID}}"
```

---

## Auth Patterns Guide

| Pattern | YAML | When to use |
|---------|------|-------------|
| File default | `meta.default_auth: prod` | All tests in file use the same auth |
| Per-test override | `auth: admin` | One test needs different credentials |
| No auth | `auth: none` | Login, register, org creation endpoints |
| Suppress org header | `org_header: false` | Bare-domain endpoints where `X-Org-Slug` causes 400 errors |

`auth: none` is a string literal, not YAML null. Always write `auth: none` explicitly. Writing `auth:` with no value parses as `null` and fails.

**Multi-file auth flow:** setup acquires credentials, downstream files consume them.

`00_setup.yaml`:
```yaml
meta:
  runner: httpx
  endpoint: "http://localhost:8000"
# No default_auth: login endpoint needs no auth

groups:
  - id: 1
    tests:
      - id: "S.1"
        name: "Login"
        method: POST
        path: "/api/v1/auth/login"
        auth: none
        org_header: false
        body:
          email: "{{TEST_EMAIL}}"
          password: "{{TEST_PASSWORD}}"
        assert:
          status: 200
        capture:
          APP_JWT: "$.access_token"
```

`01_api_surface.yaml`:
```yaml
meta:
  runner: httpx
  endpoint: "http://localhost:8000"
  default_auth: prod          # APP_JWT now available from setup

auth:
  prod:
    type: bearer
    token: "{{APP_JWT}}"      # Captured in 00_setup.yaml
    org_header: "myapp"
```

---

## Complete Example

A self-contained two-file example for a fictional `myapp` REST service.

**`tests/regression/00_setup.yaml`**

```yaml
meta:
  product: myapp
  layer: setup
  runner: bash
  endpoint: "http://localhost:8000"

variables:
  RUN_ID: "{{timestamp}}"
  TEST_EMAIL: "regtest-{{RUN_ID}}@example.com"
  TEST_PASSWORD: "TestPass123!"

groups:
  - id: 1
    name: "Seed"
    priority: high
    tests:
      - id: "S.1"
        name: "Verify database is ready"
        runner: bash
        commands:
          - cmd: "docker exec myapp-postgres pg_isready -U postgres"
            capture:
              RAW_OUTPUT: stdout
        assert:
          last_exit_code: 0
          contains: "accepting connections"

      - id: "S.2"
        name: "Login and capture JWT"
        runner: httpx
        method: POST
        path: "/api/v1/auth/login"
        auth: none
        org_header: false
        body:
          email: "{{TEST_EMAIL}}"
          password: "{{TEST_PASSWORD}}"
        assert:
          status: 200
          json_path:
            "$.access_token": { exists: true }
        capture:
          APP_JWT: "$.access_token"

      - id: "S.3"
        name: "Create API key"
        runner: httpx
        method: POST
        path: "/api/v1/api-keys"
        auth: session
        body:
          name: "regression-key-{{RUN_ID}}"
        assert:
          status: 201
          json_path:
            "$.key": { starts_with: "ak_" }
        capture:
          API_KEY: "$.key"

auth:
  session:
    type: bearer
    token: "{{APP_JWT}}"
```

**`tests/regression/01_api_surface.yaml`**

```yaml
meta:
  product: myapp
  layer: api
  runner: httpx
  endpoint: "http://localhost:8000"
  default_auth: prod

auth:
  prod:
    type: bearer
    token: "{{APP_JWT}}"
    org_header: "myapp"

groups:
  - id: 1
    name: "Items CRUD"
    priority: high
    tests:
      - id: "A.1"
        name: "List items"
        method: GET
        path: "/api/v1/items"
        assert:
          status: 200
          json_path:
            "$": { not_empty: true }

      - id: "A.2"
        name: "Create item"
        method: POST
        path: "/api/v1/items"
        body:
          name: "Regression item {{RUN_ID}}"
          price: 19.99
        assert:
          status: 201
          json_path:
            "$.id": { exists: true }
            "$.name": { contains: "Regression item" }
        capture:
          ITEM_ID: "$.id"

      - id: "A.3"
        name: "Get item by ID"
        method: GET
        path: "/api/v1/items/{{ITEM_ID}}"
        assert:
          status: 200
          json_path:
            "$.id": { equals: "{{ITEM_ID}}" }
            "$.price": { equals: "19.99" }

      - id: "A.4"
        name: "Delete item"
        method: DELETE
        path: "/api/v1/items/{{ITEM_ID}}"
        assert:
          status: 204
```

Run it:

```bash
regrun run tests/regression/
```

---

## Environment Variables

| Variable | Default | Description |
|----------|---------|-------------|
| `REGRUN_TIMEOUT` | `30` | Per-test HTTP timeout (seconds) |
| `REGRUN_MCP_TIMEOUT` | `60` | Per-test MCP call timeout (seconds) |
| `REGRUN_WS_TIMEOUT` | `30` | Per-test WebSocket timeout (seconds) |
| `REGRUN_VERBOSE` | `false` | Log full request/response bodies |
| `REGRUN_API_ENDPOINT` | (none) | Override `meta.endpoint` globally (for CI) |
| `REGRUN_MCP_ENDPOINT` | (none) | Override `meta.mcp_endpoint` globally (for CI) |
| `REGRUN_RUNS_DIR` | `~/.regrun/runs` | Base dir for persisted `report.txt` + `report.json` + `junit.xml` artifacts |
| `REGRUN_DIAG_BODY_LIMIT` | `2000` | Max response-body chars kept in failure diagnostics |

---

## CI Integration

In CI, services run as Docker containers with network aliases instead of `*.localhost` domains. Use the endpoint override variables to point regrun at the container aliases.

**GitLab CI:**

```yaml
regression:
  stage: test
  services:
    - name: myapp-api:latest
      alias: api
    - name: myapp-mcp:latest
      alias: mcp
  variables:
    REGRUN_API_ENDPOINT: "http://api:8000"
    REGRUN_MCP_ENDPOINT: "http://mcp:9000"
  script:
    - pip install regrun
    - regrun run tests/regression/
```

**GitHub Actions:**

```yaml
jobs:
  regression:
    runs-on: ubuntu-latest
    services:
      api:
        image: myapp-api:latest
        ports:
          - 8000:8000
    steps:
      - uses: actions/checkout@v4
      - run: pip install regrun
      - run: regrun run tests/regression/
        env:
          REGRUN_API_ENDPOINT: "http://localhost:8000"
```

The endpoint override applies to every test file in the run. YAML files keep their local `*.localhost` URLs for developer use; CI overrides them without any file changes.

---

## File Structure

**Recommended test directory layout:**

```
tests/regression/
  00_setup.yaml          # Setup: auth, seed data, environment checks
  01_api_surface.yaml    # REST API surface tests
  02_mcp_surface.yaml    # MCP tool tests
  03_chat_surface.yaml   # WebSocket / streaming tests
```

Numeric prefixes control the within-layer sort order (see **File ordering** above: layer rank, then the stem in byte order). The setup layer is always processed first whatever the names are, but `00_` makes the intent explicit and keeps directory listings readable.

---

## Development

Install dependencies and run the test suite:

```bash
poetry install
poetry run pytest
```

Tests live at `tests/integration/cli/` and cover CLI behaviour end-to-end.

---

## License

MIT

---

Maintained by [Neomanex](https://neomanex.com).

