Metadata-Version: 2.4
Name: pl-e2e-test
Version: 0.1.1
Summary: Fuzz PrairieLearn question lifecycles against the real Python element runtime
License-Expression: MIT
Project-URL: Homepage, https://github.com/PrairieLearn/pl-e2e-test
Project-URL: Issues, https://github.com/PrairieLearn/pl-e2e-test/issues
Project-URL: Repository, https://github.com/PrairieLearn/pl-e2e-test
Requires-Python: >=3.13
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: chevron==0.14.0
Requires-Dist: filelock>=3.20
Requires-Dist: jsonschema>=4.26
Requires-Dist: lxml>=6.1
Requires-Dist: numpy>=2.4
Requires-Dist: platformdirs>=4.5
Requires-Dist: prairielearn
Requires-Dist: pyright>=1.1.414
Requires-Dist: pytest>=9.1
Requires-Dist: ruff>=0.16
Dynamic: license-file

# pl-e2e-test

`pl-e2e-test` fuzzes PrairieLearn v3 questions through their real Python
generate, prepare, render, parse, grade, test, and file phases. It was extracted
from the `pl-oer-lv102` course so that courses can install one uv-managed tool
instead of copying the harness and its support scripts.

The package installs the reusable `prairielearn_e2e` Python API and these commands:

- `pl-e2e-test` runs the installed lifecycle pytest suite.
- `pl-e2e-test-diff` fuzzes questions selected from a Git diff.
- `pl-e2e-add-regression-seed` records a failing concrete variant seed.
- `pl-e2e-merge-reports` merges parallel shard results into terminal and JUnit reports.

## Requirements and installation

- Python 3.13 or newer
- Git 2.30 or newer

The harness depends on pytest, Pyright, Ruff, Chevron, JSON Schema, lxml, NumPy,
and PrairieLearn. PrairieLearn's Python package is currently consumed from its Git
repository, so the course project must tell uv where to resolve that dependency:

```toml
[dependency-groups]
dev = ["pl-e2e-test"]

[tool.uv.sources.prairielearn]
git = "https://github.com/PrairieLearn/PrairieLearn.git"
subdirectory = "apps/prairielearn/python"
```

Then lock and install the environment:

```sh
uv lock
uv sync
```

For local development of this package, use a path source in the consuming project:

```toml
[tool.uv.sources.pl-e2e-test]
path = "../pl-e2e-test"
editable = true
```

## Running lifecycle tests

Pass a question, a directory of questions, or an `info.json` file. Repeat
`--question-path` to select multiple roots:

```sh
uv run pl-e2e-test \
  --question-path questions \
  --seed-count 3
```

The installed suite exposes separate `generate`, `prepare`, `render`, and `grade`
pytest markers. Any remaining arguments are passed to pytest:

```sh
uv run pl-e2e-test \
  --question-path questions/chapter/example \
  --fuzz-seeds -m render -x -vv
```

`--fuzz-seeds` creates and reports a random 64-bit master seed. Reproduce the same
variant set with `--fuzz-seed N`. Without either option, deterministic seeds start at
zero. Question metadata with `singleVariant: true` always uses seed zero.

### CI sharding

Use `--shard INDEX/COUNT` to divide the selected question variants between parallel
jobs. The index is one-based. All selected lifecycle checks for one question and seed
stay together, and variants are assigned round-robin so shard sizes differ by at most
one. Pytest filters such as `-m` and `-k` are applied before sharding.

Add `--shard-report PATH` to write a mergeable result blob. Every shard must use the
same checkout and arguments. Sharded fuzz runs must also use one explicit shared
`--fuzz-seed`; automatic random fuzz seeds are rejected. After downloading all blobs,
merge them into a combined terminal summary and JUnit report:

```sh
uv run pl-e2e-merge-reports e2e-reports \
  --junitxml pl-e2e-results.xml
```

The merger rejects missing or duplicate shards, mismatched test collections, mixed
fuzz seeds, corrupt reports, and tests left unexecuted by interruption or `-x`.
It exits unsuccessfully when the aggregate suite failed or was incomplete.

For example, a course can use this GitHub Actions matrix:

```yaml
jobs:
  e2e:
    runs-on: ubuntu-latest
    strategy:
      fail-fast: false
      matrix:
        shard: [1, 2, 3, 4]
    steps:
      - uses: actions/checkout@v4
      - uses: astral-sh/setup-uv@v9
        with:
          enable-cache: true
      - run: uv python install && uv sync --locked
      - name: Run shard
        run: |
          uv run pl-e2e-test \
            --question-path questions \
            --seed-count 10 \
            --fuzz-seed "${{ github.run_id }}" \
            --shard "${{ matrix.shard }}/4" \
            --shard-report "e2e-reports/shard-${{ matrix.shard }}.pl-e2e-report.json"
      - name: Upload shard report
        if: ${{ !cancelled() }}
        uses: actions/upload-artifact@v4
        with:
          name: e2e-report-${{ matrix.shard }}
          path: e2e-reports/
          if-no-files-found: error
          retention-days: 1

  e2e-report:
    if: ${{ !cancelled() }}
    needs: e2e
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: astral-sh/setup-uv@v9
      - run: uv python install && uv sync --locked
      - uses: actions/download-artifact@v4
        with:
          pattern: e2e-report-*
          path: e2e-reports
          merge-multiple: true
      - name: Merge reports
        run: |
          uv run pl-e2e-merge-reports e2e-reports \
            --junitxml pl-e2e-results.xml
      - name: Upload merged JUnit report
        if: ${{ always() }}
        uses: actions/upload-artifact@v4
        with:
          name: pl-e2e-results
          path: pl-e2e-results.xml
          if-no-files-found: warn
```

The course must provide `.prairielearn/schemas/infoQuestion.json`. The harness checks
question metadata against that schema and uses the exact PrairieLearn revision
installed in the uv environment. It stores a bare, partial PrairieLearn repository in
the platform-standard user cache directory. The first use of a new Git source or
revision requires network access; later runs reuse that exact revision offline. The
uv lockfile, rather than the cache, decides which revision is tested.

To use an existing PrairieLearn worktree or bare repository instead, pass
`--prairielearn-path /path/to/PrairieLearn`. The repository must already contain the
installed revision. An explicit path is authoritative: the harness does not fetch into
it or fall back to the managed cache. In either mode, only the pinned revision's Python
element controllers are archived into the test run's temporary directory. Standard
cache environment settings such as `XDG_CACHE_HOME` can relocate the managed cache;
CI systems can persist that directory between runs if desired.

### Question-local configuration

Store saved concrete seeds and narrow phase skips in `.question-e2e.json` beside the
question's `info.json`:

```json
{
  "regression_seeds": [1843927501],
  "skip_methods": {
    "grade": "Temporarily blocked by an upstream issue"
  }
}
```

Saved regression seeds run before newly generated seeds. Skips require a nonempty
reason and may name only `generate`, `prepare`, `render`, or `grade`. Manual-grading
questions automatically skip only the grade check.

Record the concrete seed shown in a failing pytest case with:

```sh
uv run pl-e2e-add-regression-seed \
  questions/chapter/example 1843927501
```

The command atomically creates or updates `.question-e2e.json`, preserves other
settings, and sorts and deduplicates the seed list.

## Diff-based fuzzing

The diff runner tests changed question directories. A non-ignored changed file outside
`questions/` selects every question:

```sh
uv run pl-e2e-test-diff \
  --base origin/main \
  --head HEAD \
  --seed-count 10
```

Configure optional repository-relative ignore rules in the course's `pyproject.toml`:

```toml
[tool.pl-e2e-test]
diffignore = [
  ".github",
  "README.md",
  "*lock.yaml",
]
```

Rules use Git's native ignore syntax and are evaluated in order, so later rules may
negate earlier ones with `!`. A missing `pyproject.toml`, tool table, or `diffignore`
entry means that no paths are ignored. Additional arguments are forwarded to pytest.

To migrate from `.question-e2e.diffignore`, copy its non-comment rules into the
`diffignore` array. The former file and `--e2ediffignore` option are no longer used.

In pull-request CI, fetch the base commit before running the command and pass the
appropriate base and head SHAs. Set `--fuzz-seed` to replay a reported run.
`pl-e2e-test-diff` accepts the same `--shard` and `--shard-report` options. If a diff
selects no questions, it writes a successful empty shard report so the merge job can
still verify that every matrix job completed.

## Python API

The original API remains available:

```python
from prairielearn_e2e import PrairieLearnBackend, QuestionCase, QuestionHarness
```

`QuestionHarness.run_generate`, `run_prepare`, `run_render`, and `run_grade` execute
increasingly complete portions of the lifecycle. `run_variant` remains an alias for
the complete grade path.

## Development

This project uses uv 0.9 or newer:

```sh
uv sync
make test
make format
make smoke-dist
```

`make test` runs pytest, Pyright, Ruff linting, and Ruff's formatting check.
`make smoke-dist` builds both distribution formats, installs each without dependencies
into an isolated target, and imports it using the locked development dependencies.

## Publishing

There are three supported publishing paths. Choose one for a release; do not run the
local and GitHub paths for the same version because PyPI versions are immutable.

### GitHub-hosted trusted publishing

Configure the PyPI project to trust the `publish.yml` workflow in the GitHub `pypi`
environment. From a clean branch, run:

```sh
make publish-version VERSION=0.2.0
```

This runs the full checks, updates `pyproject.toml` and `uv.lock`, builds the
distributions, creates a release commit and annotated `v0.2.0` tag, and atomically
pushes the branch and tag. The tag starts the workflow on `ubuntu-latest`; the workflow
rechecks the tag/version match, tests, builds, generates attestations, and publishes
with PyPI trusted-publishing OIDC credentials. Use `REMOTE=name` for a different Git
remote.

If an unpushed commit already contains the requested version, use the guarded bypass:

```sh
make publish-version VERSION=0.2.0 ALLOW_PREBUMPED_VERSION=1
```

### Direct local publishing

To publish the version already present in `pyproject.toml` directly from this machine,
configure a uv-supported PyPI credential such as `UV_PUBLISH_TOKEN`, then run:

```sh
make publish-local
```

This runs all checks, builds and smoke-tests the current wheel and source archive, and
passes only those two version-matched files to `uv publish`. It does not create a Git
commit or tag.

### Local self-hosted GitHub runner

The same trusted-publishing workflow can run on a repository-scoped Linux ARM64 runner
inside Docker. The container has no host mounts and does not receive the Docker socket.
Authenticate `gh` with repository administration permission, then run:

```sh
make runner-start
make publish-self-hosted RELEASE_TAG=v0.2.0
```

The tag must already exist. This dispatches `publish.yml` from `main` with the
`self-hosted` runner choice. `make runner-ensure` starts Docker Desktop on macOS when
needed and repairs the runner before dispatch. Other lifecycle commands are
`runner-status`, `runner-stop`, `runner-remove`, `runner-logs`, and `clean-runner`.
