# Four-Layer Test Architecture for VIP

This document describes how VIP structures its tests into four layers. The key rule is that each layer only communicates with the one directly below it -- a `.feature` file doesn't call httpx, and the httpx client has no idea what Gherkin is.

## The Four Layers

```
Layer 1: Test           →  what we're testing (the Gherkin scenario)
Layer 2: DSL            →  how we express it (step definitions + fixtures)
Layer 3: Driver Port    →  what we need from the system (protocols/interfaces)
Layer 4: Driver Adapter →  how we interact with the system (API calls, browser clicks)
```

```mermaid
graph TD
    subgraph "Layer 1: Test"
        F[".feature files<br/>(Gherkin scenarios)"]
    end

    subgraph "Layer 2: DSL"
        S["Step definitions<br/>(given / when / then)"]
        FX["Fixtures<br/>(conftest.py)"]
        S
        FX
    end

    subgraph "Layer 3: Driver Port"
        CC["ConnectClient"]
        WC["WorkbenchClient"]
        PC["PackageManagerClient"]
    end

    subgraph "Layer 4: Driver Adapter"
        API["httpx<br/>(API calls)"]
        UI["Playwright<br/>(browser)"]
    end

    F --> S
    S --> CC
    S --> WC
    S --> PC
    CC --> API
    CC --> UI
    WC --> API
    WC --> UI
    PC --> API
```

Skip a layer and you end up with feature files full of HTTP status codes, or step definitions that break every time an endpoint changes.

## Layer 1: The Test (`.feature` files)

Feature files are pure business scenarios. They contain no implementation details -- no URLs, no HTTP calls, no CSS selectors.

```gherkin
@connect
Feature: Connect authentication
  Scenario: User can log in via the web UI
    Given Connect is accessible at the configured URL
    When a user navigates to the Connect login page
    And enters valid credentials
    Then the user is successfully authenticated
    And the Connect dashboard is displayed
```

What's **not** here:
- No URLs or endpoints
- No HTTP status codes
- No CSS selectors or page structure
- No database queries
- No setup or teardown logic

The scenario describes what the user experiences. How the system delivers that is entirely somebody else's problem.

### Channel annotations

VIP supports running the same scenario through different channels using product marker tags (`@connect`, `@workbench`, `@package_manager`). The same business scenario can be verified through both the API and the UI when step definitions support both paths.

### Tagging rules

Every feature file **must** have a product marker tag (`@connect`, `@workbench`, or `@package_manager`). The tag controls auto-skip: when a product is not configured, all scenarios with its tag are skipped automatically. Forgetting the tag breaks this mechanism and causes confusing failures.

### Version gating

Use the `min_version` marker for features that only exist in certain product versions:

```python
@pytest.mark.min_version(product="connect", version="2024.09.0")
```

This skips the test when the deployed version is older than the specified minimum, so the same test suite works across multiple product releases. Versions are parsed and compared with `vip.version.ProductVersion`, which understands Posit's calendar versioning scheme (`YYYY.MM.patch` plus `-dev`/`-daily`/`-preview`/`+build` suffixes).

When the deployed or required version cannot be determined at all (unconfigured, or unparseable), the test is skipped and flagged with a distinct **"N/A (version)"** report status rather than run optimistically -- a version gap should be visible in the report, not hidden behind a possibly-spurious pass.

`min_version` can only skip a whole test. When a UI change requires switching *behavior or selectors* rather than skipping outright -- for example, a redesigned dialog that closes via Escape instead of a Cancel button -- use a versioned page-object class plus a `get_<page>(version)` factory in `src/vip_tests/workbench/pages/`. Subclasses hold only the delta from the version they change in (cumulative inheritance), and behavior differences that aren't expressible as a selector change use a strategy dict keyed by version threshold, mirroring `vip.idp`'s IdP strategy dict. See `src/vip_tests/workbench/pages/homepage.py` (`Homepage_2026_05`, `get_homepage`, `get_new_session_dialog_close_strategy`) for the pattern.

## Layer 2: The DSL (step definitions + fixtures)

Step definitions are where the fluent API lives. They translate Gherkin steps into actions using fixtures and driver ports.

### Given -- setting up the world

```python
@given("Connect is accessible at the configured URL")
def connect_accessible(connect_client):
    """Guard step -- ensures the product is available."""
    if connect_client is None:
        pytest.skip("Connect is not configured")
```

Given steps collect preconditions. They use **fixtures** (like `connect_client`, `vip_config`) to access configuration and verify the world is ready. When a precondition isn't met, they call `pytest.skip()` rather than fail -- this marks the test as skipped with a clear reason instead of producing a confusing assertion error.

### When -- performing the action

```python
@when("a user navigates to the Connect login page")
def navigate_to_login(page, connect_url):
    page.goto(f"{connect_url}/__login__")
```

When steps perform the action under test. They call the system through driver ports (API clients or Playwright pages). The `target_fixture` parameter passes results to subsequent steps.

### Then -- verifying the outcome

```python
@then("the user is successfully authenticated")
def user_authenticated(page, connect_url):
    assert "/__login__" not in page.url
```

Then steps verify the outcome. Importantly, they should verify **actual system state**, not just the action's return value. Make a separate call to fetch current state when possible.

### Fixtures as glue

Pytest fixtures (`conftest.py`) are the glue between layers. They provide:
- **Configuration**: `vip_config`, `connect_url`, `test_username`
- **Clients**: `connect_client`, `workbench_client`, `pm_client`
- **Browser state**: `page`, `browser_context_args`
- **Feature flags**: `email_enabled`, `monitoring_enabled`

## Layer 3: The Driver Port (protocols/interfaces)

Driver ports define **what** the DSL needs from the system without saying **how**. In VIP, these are the client interfaces in `src/vip/clients/`.

```python
# src/vip/clients/connect.py -- the port (interface)
class ConnectClient:
    """Client interface for interacting with the Connect API in tests."""

    def current_user(self) -> dict: ...
    def list_content(self) -> list[dict]: ...
    def deploy_content(self, bundle: bytes, name: str) -> dict: ...
```

Key rules for driver ports:
- **String identifiers and parameters** for maximum flexibility (allows invalid values in negative tests); non-string payloads like binary bundles are fine
- **Return dicts**, not custom model objects
- **No product SDK dependencies** -- use raw HTTP
- **Minimal surface** -- add methods only when tests need them

The port is the contract between the DSL and the system. It never changes when you switch between API and UI testing.

## Layer 4: The Driver Adapter (implementations)

This is where the same interface gets different implementations.

### API adapter (httpx)

```python
class ConnectClient:
    def __init__(self, base_url: str, *, api_key: str | None = None):
        headers = {"Authorization": f"Key {api_key}"} if api_key else {}
        self._client = httpx.Client(base_url=base_url, headers=headers)

    def current_user(self) -> dict:
        resp = self._client.get("/v1/user")
        resp.raise_for_status()
        return resp.json()
```

Straightforward HTTP calls that map requests and responses.

### UI adapter (Playwright)

```python
@when("a user navigates to the Connect login page")
def navigate_to_login(page, connect_url):
    page.goto(f"{connect_url}/__login__")

@when("enters valid credentials")
def enter_credentials(page, test_username, test_password):
    page.fill("[name='username']", test_username)
    page.fill("[name='password']", test_password)
    page.click("[data-automation='login-panel-submit']")
```

Same business action, completely different implementation. The UI adapter uses Playwright to open a browser, navigate pages, fill forms, and click buttons.

## VIP's Layer Mapping

| Layer | 4-Layer Concept | VIP Implementation |
|-------|----------------|-------------------|
| 1. Test | Scenario / specification | `.feature` files with `@product` tags |
| 2. DSL | Fluent API / step builders | `pytest_bdd` step definitions in `.py` files |
| 3. Driver Port | Interface / protocol | Client classes in `src/vip/clients/` |
| 4. Driver Adapter | API / UI implementation | httpx clients + Playwright page interactions |

## Why This Architecture Matters

VIP's layer structure is an application of Valentina Jemuović's [architecture behind acceptance tests](https://journal.optivem.com/p/the-architecture-behind-acceptance-tests) to how we test Posit Team deployments.

In practice, it means changes stay local:

- **Business rules change?** Update the `.feature` files. Step definitions and clients stay the same.
- **API endpoint changes?** Update the API client. Features, steps, and UI tests don't move.
- **UI redesign?** Update the Playwright steps. Features, API steps, and clients don't move.
- **New feature?** Add new `.feature` + `.py` pairs and extend clients. Everything existing stays untouched.

The `.feature` files only need to change when the requirements change.

## Tuning timeouts on slow hardware

VIP ships with generous defaults, but on small QA VMs some operations — launching
a Workbench session, deploying content to Connect — can take much longer than on
production hardware.  Set the `VIP_TIMEOUT_SCALE` environment variable to multiply
every operation timeout uniformly:

```bash
VIP_TIMEOUT_SCALE=3 vip verify --connect-url https://connect.example.com
```

This scales:
- All Playwright wait/expect timeouts (Workbench session start, IDE load, page load, …)
- IdP login-form and network-idle waits
- API polling deadlines (`wait_for_task`, `wait_for_system_check`)
- The default httpx client timeout (30 s)
- The pytest subprocess budget (`--test-timeout`)

Deliberately **not** scaled:
- The 5-minute MFA prompt window — a human wait, not a server operation
- UI busy-loop pacing sleeps (e.g. `wait_for_timeout(500)`) — scaling poll intervals
  only slows the loop without preventing any timeout
- Explicit values: callers that pass an explicit timeout value opt out of scaling

Values < 1.0 are valid and useful for speeding up CI smoke checks
(`VIP_TIMEOUT_SCALE=0.5`).

## Fast confidence checks with `--basic`

`vip verify --basic` runs only the core subset of tests, excluding detailed or
long-running checks tagged `@slow`. Today the `@slow` set covers the heavier
Workbench checks — IDE extension installation, job execution, Git operations,
and publishing to Connect — so a `--basic` run still exercises auth, IDE launch,
sessions, runtime versions, packages, data sources, and Chronicle.

`--basic` composes with `--categories`: `vip verify --categories workbench --basic`
runs the Workbench category minus its `@slow` scenarios. The mechanism is
product-agnostic — tag any feature `@slow` to exclude it from basic runs.

## Adding a New Test: Layer-by-Layer Checklist

1. **Layer 1 -- Feature file**: Write the Gherkin scenario with a `@product` tag. Focus on business intent, not implementation.

2. **Layer 2 -- Step definitions**: Implement `given`/`when`/`then` functions. Reuse existing steps where possible. Use `target_fixture` to pass state between steps.

3. **Layer 3 -- Driver port**: Check if the client already has the method you need. If not, add a method to the appropriate client in `src/vip/clients/`.

4. **Layer 4 -- Driver adapter**: Implement the client method using httpx (API) or add Playwright steps (UI). Keep each adapter focused on one way of talking to the system.

## Anti-Patterns to Avoid

- **Leaking implementation into features**: Don't put URLs, status codes, or selectors in `.feature` files.
- **Skipping the port layer**: Don't make HTTP calls directly in step definitions. Go through a client.
- **Fat step definitions**: If a step definition is more than ~10 lines, it's doing too much. Push logic down to the client layer.
- **Shared mutable state**: Use `target_fixture` to pass state between steps, not module-level globals.
- **Testing implementation, not behavior**: Scenarios should describe what the user experiences, not how the system works internally.

## Writing a Test Extension

VIP's extension mechanism lets you load custom test directories alongside the built-in suite:

```bash
vip verify --config vip.toml --extensions ./my-custom-tests
```

Or in `vip.toml`:

```toml
[general]
extension_dirs = ["./my-custom-tests"]
```

Use `vip scaffold` to generate a ready-to-run reference implementation:

```bash
vip scaffold --output ./my-custom-tests
```

This creates `examples/cross_product_validation/` — a full GxP validation example that verifies
R/Python runtime versions and package installability across Connect and Workbench. It follows the
same four-layer architecture as the built-in suite.

Two canonical examples ship with VIP:

- `examples/custom_tests/` — minimal HTTP health-check (simplest possible extension)
- `examples/cross_product_validation/` — cross-product runtime + package validation (GxP pattern)

**Key requirement for auto-skip to work in extensions:** apply `@pytest.mark.connect` and/or
`@pytest.mark.workbench` decorators directly on every `@scenario` function. With pytest-bdd,
scenario-level Gherkin tags (e.g. `@connect` on a `Scenario:` block) do become pytest markers and
participate in auto-deselect via `item.get_closest_marker()`. However, feature-level tags apply to
every scenario in the file — if you tag the whole `Feature:` with both `@connect` and `@workbench`,
every scenario requires both products, which causes incorrect deselection when only one product is
configured. The safe, portable pattern is to tag each `Scenario:` with only the product(s) it
actually uses, or to apply `@pytest.mark.connect`/`@pytest.mark.workbench` directly on the
`@scenario` decorator function in Python.
