Metadata-Version: 2.5
Name: qa-screens
Version: 0.2.0
Summary: MCP server for visual QA: capture pages, SSIM-diff them against reference screenshots, and let an AI refactor safely. Supports OAuth tokens, sessions and cookies for testing authenticated apps.
Project-URL: Homepage, https://github.com/astuanax/qa-screens
Project-URL: Issues, https://github.com/astuanax/qa-screens/issues
Author: Len Dierickx
License: MIT
License-File: LICENSE
Keywords: mcp,playwright,qa,screenshots,ssim,visual-regression
Requires-Python: >=3.10
Requires-Dist: httpx>=0.27
Requires-Dist: mcp<3,>=2.2
Requires-Dist: numpy>=1.26
Requires-Dist: opencv-python-headless>=4.8
Requires-Dist: pillow>=10
Requires-Dist: playwright>=1.45
Requires-Dist: scikit-image>=0.22
Provides-Extra: ai
Requires-Dist: google-genai>=1.0; extra == 'ai'
Provides-Extra: dev
Requires-Dist: pytest-asyncio>=0.23; extra == 'dev'
Requires-Dist: pytest>=8; extra == 'dev'
Description-Content-Type: text/markdown

# qa-screens

An MCP server for **visual QA during AI refactoring**. It captures pages with
Playwright, compares them with reference screenshots using SSIM, and returns
the score, the changed regions and a labelled *REFERENCE | LIVE* preview image,
so the agent can see what broke and fix it.

- **Design conformance**: compare the running site with golden or Figma screenshots.
- **Refactor safety**: capture *before* and *after* a change and prove it is pixel-neutral.
- **A/B**: compare two running servers, such as `main` and a worktree.
- **Apps behind a login**: OAuth (client credentials, password, refresh, authorization code + PKCE),
  bearer tokens, cookies, sessions, localStorage/sessionStorage and HTTP basic auth.
- **Self-reporting**: significant errors and crashes are filed as GitHub issues automatically,
  with secrets scrubbed. See [Error reporting](#error-reporting).

## Quick start

No configuration is needed for a first run:

```bash
claude mcp add qa-screens -- uvx qa-screens
```

Then ask the agent: *"Use qa-screens to make the current homepage the reference, then check it."*

1. The first call downloads Chromium (it starts in the background as soon as the server starts).
2. `update_reference(page="home", confirm=true)` screenshots `http://localhost:8080/home/`
   and saves it as `screenshots/home.png`. Append `-mobile` to the name for the mobile version.
3. `run_qa` compares the live site with the references from then on.

If your site runs somewhere else, the error message says so. Set `base_url` in
`.qa-screens.json` (see [Configure](#configure)) or pass `base_url` to the tool.

## Install

```bash
pip install qa-screens          # or: uv tool install qa-screens
```

Chromium is downloaded automatically on first use. To do it up front, run
`python -m playwright install chromium`.

### Claude Code

```bash
claude mcp add qa-screens -- uvx qa-screens
```

or in a project's `.mcp.json`:

```json
{
  "mcpServers": {
    "qa-screens": { "command": "uvx", "args": ["qa-screens"] }
  }
}
```

To run the latest `main` without PyPI, use `uvx --from git+https://github.com/astuanax/qa-screens qa-screens`.

Copy [`skills/qa-screenshots`](skills/qa-screenshots/SKILL.md) into `.claude/skills/`
so the agent runs QA on its own after every UI change.

### Other MCP clients

The server speaks MCP over stdio. The command is `qa-screens` (or `python -m qa_screens`),
and it runs in the project directory, or wherever `QA_SCREENS_ROOT` points.

## Configure

Put a `.qa-screens.json` in the project root. Every key is optional. A mistake in this
file never stops the tools: bad JSON, unknown keys (with a "did you mean" hint) and invalid
values fall back to the defaults, and every tool result lists them under `config_warnings`
(the CLI prints them as `warning:` lines).

```json
{
  "base_url": "http://localhost:8080",
  "references_dir": "qa/screenshots",
  "route_template": "/nl-be/{name}/",
  "routes": { "home": "/", "nl-be": "/nl-be/" },
  "threshold": 0.9,
  "mobile_device_scale_factor": 1,
  "mask_selectors": [".carousel", "[data-testid=clock]"],
  "default_profile": null
}
```

| key | default | meaning |
|---|---|---|
| `base_url` | `http://localhost:8080` | where the site runs (not started by qa-screens) |
| `references_dir` | `screenshots` | reference `.png`/`.jpg` files; the file name is the page name |
| `route_template` | `/{name}/` | page name → path; `{name}` excludes the `-mobile` suffix |
| `routes` | `{}` | per-page overrides (a path, or a full URL) |
| `threshold` | `0.90` | minimum SSIM to pass |
| `max_changed_ratio` | `0.02` | also fail when more than this share of pixels clearly changed colour (`1` disables) |
| `mobile_suffix` | `-mobile` | names ending in it are captured in a 390px mobile context |
| `desktop_viewport` / `mobile_viewport` | 1440×900 / 390×844 | |
| `mobile_device_scale_factor` | `2` | set `1` if mobile references are 390px wide |
| `mask_selectors` | `[]` | elements hidden before capture (dynamic content) |
| `wait_until`, `navigation_timeout_ms` | `networkidle`, `30000` | |
| `runtime_dir` | `.qa-screens/runtime` | captures, diffs, previews, reports (add it to `.gitignore`) |
| `default_profile` | `null` | auth profile used when a tool doesn't pass one |
| `ai_critique`, `gemini_model` | `false` | optional Gemini second opinion (`pip install qa-screens[ai]`, `GEMINI_API_KEY`) |

The environment variables `QA_SCREENS_BASE_URL`, `QA_SCREENS_REFERENCES_DIR`,
`QA_SCREENS_ROUTE_TEMPLATE`, `QA_SCREENS_THRESHOLD`, `QA_SCREENS_MAX_CHANGED_RATIO` and
`QA_SCREENS_PROFILE` override the file. `QA_SCREENS_FIGMA_API` changes the Figma API URL.

## Tools

| tool | purpose |
|---|---|
| `qa_config` | effective config, references and their URLs, auth profiles; start here |
| `run_qa` | QA all or some pages; returns a JSON summary plus previews of the worst failures |
| `qa_page` | QA one page (fast fix loop); `url` overrides the mapping |
| `capture` | screenshot any URL, full page or a single element (`clip_selector`) |
| `compare_images` | SSIM two image files, for example a Figma export and a capture |
| `capture_set` / `compare_sets` | before/after parity check for refactors |
| `ab_compare` | compare two running servers page by page |
| `update_reference` | make the page's current look its reference: first baseline or accepted redesign (needs `confirm=true`, keeps a backup, refuses error pages) |
| `sync_figma` | download Figma frames as references (`FIGMA_TOKEN`) |
| `auth_update_profile` | create or update an auth profile: token, OAuth, cookies, headers, storage |
| `auth_browser_login` | log in with a real browser (form, SSO or MFA) and save the session |
| `auth_oauth_login` | OAuth authorization code + PKCE in a browser window |
| `auth_import_storage_state` | import a Playwright `storageState` JSON |
| `auth_check` | check that a profile is logged in (status, redirect, screenshot) |
| `auth_profiles` / `auth_delete_profile` | list, inspect (never shows secrets) or delete profiles |
| `error_reports` | error-reporting status; `flush=true` posts pending reports |

Page results are `PASS`, `FAIL` or `ERROR`. A page fails when its SSIM is below `threshold`
**or** more than `max_changed_ratio` of it clearly changed colour. SSIM is a page-wide average,
so a recoloured header scores about 0.99 and would pass on SSIM alone.
`ERROR` means the comparison is meaningless:
an HTTP 4xx/5xx, a navigation failure or a missing reference. The agent should fix
the environment, not the CSS.

## Authentication for apps

A **profile** is a named session that is applied to every browser context that uses it:

```text
auth_update_profile(
  name="staging",
  origins=["https://app.staging.example.com"],
  oauth={"grant_type": "client_credentials",
         "token_url": "https://idp.example.com/oauth/token",
         "client_id": "qa-bot", "client_secret": "env:QA_CLIENT_SECRET",
         "audience": "https://api.example.com"},
  apply_token_as=["header", "local_storage:access_token"])

run_qa(profile="staging", base_url="https://app.staging.example.com")
```

- **Secrets**: any value can be `"env:VAR"`. It is resolved from the server's environment
  at use time, so it never passes through the AI conversation.
- **Scoping**: headers and tokens go only to the profile's `origins` (default: the
  base URL's origin), never to CDNs or third parties. Cookies follow normal browser rules.
- **Token placement** (`apply_token_as`): `header` (`Authorization: Bearer …`),
  `header:X-Api-Key`, `local_storage:<key>`, `session_storage:<key>`,
  `local_storage_json:<key>` (the whole token object), `cookie:<name>`.
- **Refresh**: expired tokens are refreshed through `refresh_token` or by re-running the
  client-credentials/password grant, including mid-session in the long-lived server.
- **Storage**: profiles live in `~/.local/state/qa-screens/profiles/` with mode `0600`,
  outside the project, so they are never committed. `QA_SCREENS_STATE_DIR` moves the location.

## Error reporting

qa-screens reports its own bugs, so they reach the maintainers without a manual bug report.

- **Significant errors** (unexpected exceptions in a tool, not user errors like a missing
  reference or bad credentials) are written to `~/.local/state/qa-screens/reports/pending/`
  and posted in the background as a GitHub issue on `astuanax/qa-screens`.
- **Crashes**: uncaught exceptions are logged to a crash file, and hard crashes (segfault,
  abort) are captured by `faulthandler`. On the **next start-up** the server scans for these
  files and posts them.
- A report stays pending until it has been posted, so reports made offline or without a token
  are sent later.
- **De-duplication**: each error has a stable fingerprint. A repeat within 24h is not
  re-posted, and an open issue with the same fingerprint gets a comment instead of a new
  issue. At most 10 posts per day.
- **Privacy**: tokens, JWTs, cookies, passwords, URL query strings and credentials, and the
  home directory path are redacted. Tool arguments that hold secrets are dropped entirely.

| env var | default | |
|---|---|---|
| `QA_SCREENS_ERROR_REPORTING` | `on` | `on`, `local` (write files, never post) or `off` |
| `QA_SCREENS_ISSUE_REPO` | `astuanax/qa-screens` | where issues go (point it at your fork) |
| `QA_SCREENS_GITHUB_TOKEN` | falls back to `GITHUB_TOKEN`, then `gh auth token` | needs `issues:write` |
| `QA_SCREENS_GITHUB_API` | `https://api.github.com` | API URL, for GitHub Enterprise |

Without a token nothing is posted; reports wait in `pending/`. `qa-screens reports --flush`
posts them by hand.

## CLI (CI-friendly)

```bash
qa-screens                       # MCP server over stdio (same as `qa-screens serve`)
qa-screens run                   # QA every reference; exit 1 on FAIL/ERROR
qa-screens run nl-be faq --viewport desktop --base-url http://localhost:8080 --json
qa-screens reports --flush       # post pending error reports
```

A report is written to `.qa-screens/runtime/reports/latest.json` on every run.

## How it works

1. The reference name maps to a URL (`route_template` / `routes`). Names ending in `-mobile`
   use a mobile context (390px, touch, mobile UA).
2. The page is loaded with the HTTP cache disabled (so the CSS you just edited is what gets
   measured), with animations, transitions and scrollbars disabled, and after
   `document.fonts.ready`. It is captured full-page.
3. SSIM is computed per colour channel. Grayscale SSIM, as the original tool used, scores a
   pure hue change at 0.9996. Captures above 12MP are decoded at reduced resolution, so long
   pages never run out of memory. Different sizes are handled by `align`: `resize` (legacy
   default), `crop` or `pad` (height changes count as differences).
4. Separately, pixels whose colour clearly changed are measured on lightly blurred images,
   so JPEG artefacts and antialiasing don't count. Their share of the page is `changed_ratio`.
5. Changed pixels are grouped into region boxes. A red heatmap and a cropped side-by-side
   preview are written, and the preview is returned to the agent as an image.

## Development

```bash
uv venv && uv pip install -e ".[dev]"
python -m playwright install chromium
pytest
```

The tests exercise the server the way an agent does: through an in-process MCP client,
against a fake web app (login form, session cookies, OAuth provider with PKCE, basic auth)
and fake GitHub and Figma APIs. Assertions are about what those services actually received.

Releases are published to PyPI by GitHub Actions when a `v*` tag is pushed (trusted publishing).

## Links

- [sus.bot](https://sus.bot)

## License

MIT
