Metadata-Version: 2.4
Name: pl-e2e-test
Version: 0.1.4
Summary: Fuzz PrairieLearn question lifecycles against the real Python element runtime
License-Expression: MIT
Project-URL: Homepage, https://github.com/PrairieLearn/pl-e2e-test
Project-URL: Issues, https://github.com/PrairieLearn/pl-e2e-test/issues
Project-URL: Repository, https://github.com/PrairieLearn/pl-e2e-test
Requires-Python: >=3.13
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: chevron==0.14.0
Requires-Dist: filelock>=3.20
Requires-Dist: jsonschema>=4.26
Requires-Dist: lxml>=6.1
Requires-Dist: numpy>=2.4
Requires-Dist: platformdirs>=4.5
Requires-Dist: prairielearn
Requires-Dist: pyright>=1.1.414
Requires-Dist: pytest>=9.1
Requires-Dist: rich>=14
Requires-Dist: ruff>=0.16
Dynamic: license-file

# pl-e2e-test

`pl-e2e-test` fuzzes PrairieLearn v3 questions through their real Python
generate, prepare, render, parse, grade, test, and file phases. It was extracted
from the `pl-oer-lv102` course so that courses can install one uv-managed tool
instead of copying the harness and its support scripts.

The package installs the reusable `prairielearn_e2e` Python API and a single
`pl-e2e-test` command:

- `pl-e2e-test` (no command, or any pytest/harness options) runs the installed
  lifecycle pytest suite.
- `pl-e2e-test diff` fuzzes questions selected from a Git diff.
- `pl-e2e-test parallel` runs local shards and merges their results.
- `pl-e2e-test add-seed` records a failing concrete variant seed.
- `pl-e2e-test reports` merges parallel shard results into terminal and JUnit reports.
- `pl-e2e-test version` prints the installed version.
- `pl-e2e-test help [COMMAND]` prints top-level help, or one command's help.

## Requirements and installation

- Python 3.13 or newer
- Git 2.30 or newer

The harness depends on pytest, Pyright, Ruff, Chevron, JSON Schema, lxml, NumPy,
and PrairieLearn. PrairieLearn's Python package is currently consumed from its Git
repository, so the course project must tell uv where to resolve that dependency:

```toml
[dependency-groups]
dev = ["pl-e2e-test"]

[tool.uv.sources.prairielearn]
git = "https://github.com/PrairieLearn/PrairieLearn.git"
subdirectory = "apps/prairielearn/python"
```

Then lock and install the environment:

```sh
uv lock
uv sync
```

For local development of this package, use a path source in the consuming project:

```toml
[tool.uv.sources.pl-e2e-test]
path = "../pl-e2e-test"
editable = true
```

## Running lifecycle tests

Pass a question, a directory of questions, or an `info.json` file. Repeat
`--question-path` to select multiple roots:

```sh
uv run pl-e2e-test \
  --question-path questions \
  --seed-count 3
```

The installed suite exposes separate `generate`, `prepare`, `render`, and `grade`
pytest markers. Any remaining arguments are passed to pytest:

```sh
uv run pl-e2e-test \
  --question-path questions/chapter/example \
  --fuzz-seeds -m render -x -vv
```

`--fuzz-seeds` creates and reports a random 64-bit master seed. Reproduce the same
variant set with `--fuzz-seed N`. Without either option, deterministic seeds start at
zero. Question metadata with `singleVariant: true` always uses seed zero.

### Local parallel runs

Run multiple shards in one machine or CI job, with six concurrent processes by default:

```sh
uv run pl-e2e-test parallel --shard-count 6 run -- \
  --question-path questions --fuzz-seeds

uv run pl-e2e-test parallel --shard-count 6 --junitxml results.xml diff -- \
  --base main --fuzz-seed 42
```

Launcher options go before the required `run` or `diff` selector; arguments after
`--` are forwarded to that command. Normal runs remain deterministic. For
`--fuzz-seeds` and diff runs, the launcher generates one shared master seed unless
`--fuzz-seed` is supplied, and prints a command for replaying the run without sharding.

Each shard runs in a separate Python process using the current environment and
working directory. Terminals show Rich progress; redirected output prints completion
lines. All shards finish before reports are merged, even when a test fails. Captured
failure output and successful stderr are shown before the combined summary.
Interrupting the launcher stops and reaps its children.

Intermediate reports are temporary. Pass the launcher's `--junitxml` option to retain
merged JUnit output. Forwarded `--shard`, `--shard-report`, `--junitxml`, and
`--junit-xml` options are rejected because the launcher manages those outputs.
Choose the course with the working directory, or `course_root` in the Python API.

The same orchestration is available to Python callers:

```python
from prairielearn_e2e import run_parallel

status = run_parallel(
    ["--question-path", "questions", "--fuzz-seeds"],
    command="run",
    shard_count=6,
    course_root="/path/to/course",
    junitxml="results.xml",
)
```

`run_parallel` returns an integer exit status, including 130 on interruption.
Relative JUnit paths are resolved against the course root. With `junitxml=None`,
merged output is temporary. Invalid launcher settings raise `ValueError`.

Courses using `scripts/run_e2e_shards.py` can replace its invocation with
`pl-e2e-test parallel`, move `--fuzz-seed` into the forwarded arguments, and explicitly
pass `--fuzz-seeds` for full-suite runs to preserve the script's fuzzing behavior.
For example:

```make
test-e2e-sharded:
	uv run --active pl-e2e-test parallel --shard-count "$(E2E_SHARD_COUNT)" run -- \
		$(E2E_TEST_ARGS) $(E2E_PRAIRIELEARN_ARG) --fuzz-seeds $(E2E_FUZZ_SEED_ARG) $(PYTEST_ARGS)

test-e2e-fuzz-sharded:
	uv run --active pl-e2e-test parallel --shard-count "$(E2E_SHARD_COUNT)" diff -- \
		--base "$(E2E_DIFF_BASE)" --head "$(E2E_DIFF_HEAD)" \
		--seed-count "$(E2E_FUZZ_SEED_COUNT)" $(E2E_FUZZ_SEED_ARG) $(E2E_PRAIRIELEARN_ARG) $(PYTEST_ARGS)
```

### CI sharding

Use `--shard INDEX/COUNT` to divide the selected question variants between parallel
jobs. The index is one-based. All selected lifecycle checks for one question and seed
stay together, and variants are assigned round-robin so shard sizes differ by at most
one. Pytest filters such as `-m` and `-k` are applied before sharding.

Add `--shard-report PATH` to write a mergeable result blob. Every shard must use the
same checkout and arguments. Sharded fuzz runs must also use one explicit shared
`--fuzz-seed`; automatic random fuzz seeds are rejected. After downloading all blobs,
merge them into a combined terminal summary and JUnit report:

```sh
uv run pl-e2e-test reports e2e-reports \
  --junitxml pl-e2e-results.xml
```

The merger rejects missing or duplicate shards, mismatched test collections, mixed
fuzz seeds, corrupt reports, and tests left unexecuted by interruption or `-x`.
It exits unsuccessfully when the aggregate suite failed or was incomplete.

For example, a course can use this GitHub Actions matrix:

```yaml
jobs:
  e2e:
    runs-on: ubuntu-latest
    strategy:
      fail-fast: false
      matrix:
        shard: [1, 2, 3, 4]
    steps:
      - uses: actions/checkout@v4
      - uses: astral-sh/setup-uv@v9
        with:
          enable-cache: true
      - run: uv python install && uv sync --locked
      - name: Run shard
        run: |
          uv run pl-e2e-test \
            --question-path questions \
            --seed-count 10 \
            --fuzz-seed "${{ github.run_id }}" \
            --shard "${{ matrix.shard }}/4" \
            --shard-report "e2e-reports/shard-${{ matrix.shard }}.pl-e2e-report.json"
      - name: Upload shard report
        if: ${{ !cancelled() }}
        uses: actions/upload-artifact@v4
        with:
          name: e2e-report-${{ matrix.shard }}
          path: e2e-reports/
          if-no-files-found: error
          retention-days: 1

  e2e-report:
    if: ${{ !cancelled() }}
    needs: e2e
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: astral-sh/setup-uv@v9
      - run: uv python install && uv sync --locked
      - uses: actions/download-artifact@v4
        with:
          pattern: e2e-report-*
          path: e2e-reports
          merge-multiple: true
      - name: Merge reports
        run: |
          uv run pl-e2e-test reports e2e-reports \
            --junitxml pl-e2e-results.xml
      - name: Upload merged JUnit report
        if: ${{ always() }}
        uses: actions/upload-artifact@v4
        with:
          name: pl-e2e-results
          path: pl-e2e-results.xml
          if-no-files-found: warn
```

The course must provide `.prairielearn/schemas/infoQuestion.json`. The harness checks
question metadata against that schema and uses the exact PrairieLearn revision
installed in the uv environment. It stores a bare, partial PrairieLearn repository in
the platform-standard user cache directory. The first use of a new Git source or
revision requires network access; later runs reuse that exact revision offline. The
uv lockfile, rather than the cache, decides which revision is tested.

To use an existing PrairieLearn worktree or bare repository instead, pass
`--prairielearn-path /path/to/PrairieLearn`. The repository must already contain the
installed revision. An explicit path is authoritative: the harness does not fetch into
it or fall back to the managed cache. In either mode, only the pinned revision's Python
element controllers are archived into the test run's temporary directory. Standard
cache environment settings such as `XDG_CACHE_HOME` can relocate the managed cache;
CI systems can persist that directory between runs if desired.

### Harness configuration

Selector settings are read automatically from the course's `pyproject.toml`:

```toml
[tool.pl-e2e-test]
elementSolutionSelectors = ["pl-sketch"]
nonGradableSelectors = ['pl-sketch[read-only="true"]', 'pl-sketch[read-only="1"]']
```

These are the defaults. `elementSolutionSelectors` identifies elements that provide
solutions without a `correct_answers` entry. `nonGradableSelectors` excludes
display-only answers from grading checks. Each supplied list replaces its default;
omitted settings keep their defaults, and `[]` disables a list.

Use `--config path/to/settings.toml` to select another TOML file with the same
`[tool.pl-e2e-test]` table. This works with normal runs, `diff`, and arguments after
`parallel run` or `parallel diff`. Explicit paths are relative to the working directory.

### Question-local configuration

Store saved concrete seeds and narrow phase skips in `.question-e2e.json` beside the
question's `info.json`:

```json
{
  "regression_seeds": [1843927501],
  "skip_methods": {
    "grade": "Temporarily blocked by an upstream issue"
  }
}
```

Saved regression seeds run before newly generated seeds. Skips require a nonempty
reason and may name only `generate`, `prepare`, `render`, or `grade`. Manual-grading
questions automatically skip only the grade check.

Record the concrete seed shown in a failing pytest case with:

```sh
uv run pl-e2e-test add-seed \
  questions/chapter/example 1843927501
```

The command atomically creates or updates `.question-e2e.json`, preserves other
settings, and sorts and deduplicates the seed list.

## Diff-based fuzzing

The diff runner tests changed question directories. A non-ignored changed file outside
`questions/` selects every question:

```sh
uv run pl-e2e-test diff \
  --base origin/main \
  --head HEAD \
  --seed-count 10
```

Configure optional repository-relative ignore rules in the course's `pyproject.toml`:

```toml
[tool.pl-e2e-test]
diffignore = [
  ".github",
  "README.md",
  "*lock.yaml",
]
```

Rules use Git's native ignore syntax and are evaluated in order, so later rules may
negate earlier ones with `!`. A missing `pyproject.toml`, tool table, or `diffignore`
entry means that no paths are ignored. Additional arguments are forwarded to pytest.

To migrate from `.question-e2e.diffignore`, copy its non-comment rules into the
`diffignore` array. The former file and `--e2ediffignore` option are no longer used.

In pull-request CI, fetch the base commit before running the command and pass the
appropriate base and head SHAs. Set `--fuzz-seed` to replay a reported run.
`pl-e2e-test diff` accepts the same `--shard` and `--shard-report` options. If a diff
selects no questions, it writes a successful empty shard report so the merge job can
still verify that every matrix job completed.

## Python API

The original API remains available:

```python
from prairielearn_e2e import PrairieLearnBackend, QuestionCase, QuestionHarness
```

`QuestionHarness.run_generate`, `run_prepare`, `run_render`, and `run_grade` execute
increasingly complete portions of the lifecycle. `run_variant` remains an alias for
the complete grade path.

Pass `config=load_config(course_root=course_root)` to `QuestionHarness` to load the
course's TOML settings, or `config=load_config("settings.toml")` for an explicit file.
`load_config` and `HarnessConfig` are exported from `prairielearn_e2e`. Omitting
`config` uses the built-in defaults.

## Development

This project uses uv 0.9 or newer:

```sh
uv sync
make test
make format
make smoke-dist
```

`make test` runs pytest, Pyright, Ruff linting, and Ruff's formatting check.
`make smoke-dist` builds both distribution formats, installs each without dependencies
into an isolated target, and imports it using the locked development dependencies.

## Publishing

There are three supported publishing paths. Choose one for a release; do not run the
local and GitHub paths for the same version because PyPI versions are immutable.

### GitHub-hosted trusted publishing

Configure the PyPI project to trust the `publish.yml` workflow in the GitHub `pypi`
environment. From a clean branch, run:

```sh
make publish-version VERSION=0.2.0
```

This runs the full checks, updates `pyproject.toml` and `uv.lock`, builds the
distributions, creates a release commit and annotated `v0.2.0` tag, and atomically
pushes the branch and tag. The tag starts the workflow on `ubuntu-latest`; the workflow
rechecks the tag/version match, tests, builds, generates attestations, and publishes
with PyPI trusted-publishing OIDC credentials. Use `REMOTE=name` for a different Git
remote.

If an unpushed commit already contains the requested version, use the guarded bypass:

```sh
make publish-version VERSION=0.2.0 ALLOW_PREBUMPED_VERSION=1
```

### Direct local publishing

To publish the version already present in `pyproject.toml` directly from this machine,
configure a uv-supported PyPI credential such as `UV_PUBLISH_TOKEN`, then run:

```sh
make publish-local
```

This runs all checks, builds and smoke-tests the current wheel and source archive, and
passes only those two version-matched files to `uv publish`. It does not create a Git
commit or tag.

### Local self-hosted GitHub runner

The same trusted-publishing workflow can run on a repository-scoped Linux ARM64 runner
inside Docker. The container has no host mounts and does not receive the Docker socket.
Authenticate `gh` with repository administration permission, then run:

```sh
make runner-start
make publish-self-hosted RELEASE_TAG=v0.2.0
```

The tag must already exist. This dispatches `publish.yml` from `main` with the
`self-hosted` runner choice. `make runner-ensure` starts Docker Desktop on macOS when
needed and repairs the runner before dispatch. Other lifecycle commands are
`runner-status`, `runner-stop`, `runner-remove`, `runner-logs`, and `clean-runner`.
