Metadata-Version: 2.4
Name: pytest-prairielearn-grader
Version: 0.3.1
Summary: A pytest plugin for autograding Python code. Designed for use with the PrairieLearn platform.
Project-URL: repository, https://github.com/eliotwrobson/pl-python-autograder-v2
Author-email: Eliot Robson <eliot.robson24@gmail.com>
License: MIT License
        
        Copyright (c) 2025 Eliot W. Robson
        
        Permission is hereby granted, free of charge, to any person obtaining a copy
        of this software and associated documentation files (the "Software"), to deal
        in the Software without restriction, including without limitation the rights
        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
        copies of the Software, and to permit persons to whom the Software is
        furnished to do so, subject to the following conditions:
        
        The above copyright notice and this permission notice shall be included in all
        copies or substantial portions of the Software.
        
        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
        SOFTWARE.
License-File: LICENSE
Classifier: Development Status :: 3 - Alpha
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Requires-Python: >=3.11
Requires-Dist: dill>=0.4.0
Requires-Dist: prettytable>=3.8.0
Requires-Dist: pytest
Provides-Extra: notebook
Requires-Dist: nbformat>=5.0; extra == 'notebook'
Description-Content-Type: text/markdown

# pytest-prairielearn-grader

A pytest plugin for autograding Python code in [PrairieLearn](https://www.prairielearn.com/). Student code runs in isolated subprocess sandboxes with configurable security restrictions, timeouts, and detailed feedback.

## Installation

```bash
pip install pytest-prairielearn-grader
```

For notebook grading support:

```bash
pip install pytest-prairielearn-grader[notebook]
```

## Quick Start

See [quick_start.md](quick_start.md) for a full tutorial with PrairieLearn integration examples.

## Key Features

### Sandboxed Execution

Student code runs in a separate subprocess via Unix sockets, providing:
- Security isolation from the grading harness
- Import whitelist/blacklist enforcement
- Builtin function restrictions
- Timeout enforcement
- Privilege dropping (Unix)

### Student-Friendly Feedback Mode

Control how much information students see when their code fails. Use `@pytest.mark.output(level=...)` per test, or set `output_level` globally in your `ConfigObject`:

```python
from pytest_prairielearn_grader import ConfigObject

autograder_config = ConfigObject(
    output_level="friendly",  # Global default for all tests
)
```

**Output levels:**
| Level | Shows |
|-------|-------|
| `"none"` | Exception class name only (e.g., `AssertionError`) |
| `"message"` | Exception name + first line of message *(default)* |
| `"traceback"` | Full exception with traceback |
| `"friendly"` | Only the exception message text — no class name, no traceback |

Per-test markers override the global setting:

```python
@pytest.mark.output(level="friendly")
@pytest.mark.grading_data(name="Test add", points=5)
def test_add(sandbox: StudentFixture) -> None:
    assert_fn_equal(sandbox, "add", args=(2, 3), expected=5)
```

### Assertion Helpers

Student-friendly assertion functions that produce clean, readable failure messages:

```python
from pytest_prairielearn_grader.assertions import assert_equal, assert_fn_equal

@pytest.mark.output(level="friendly")
@pytest.mark.grading_data(name="Test calculation", points=5)
def test_calc(sandbox: StudentFixture) -> None:
    # Produces: "Checking: add(2, 3)\nExpected output: 5\nYour code output: 4\n..."
    assert_fn_equal(sandbox, "add", args=(2, 3), expected=5)
```

Available helpers:
- `assert_equal(actual, expected)` — compare any values
- `assert_approx_equal(actual, expected, rtol=1e-5, atol=1e-8)` — numeric comparison
- `assert_fn_equal(sandbox, func_name, args=..., expected=...)` — call + compare
- `assert_fn_approx_equal(sandbox, func_name, args=..., expected=...)` — call + approx compare
- `assert_true(condition)` / `assert_false(condition)` — boolean checks

### Ungradable Submission Detection

When student code has a `SyntaxError`, the submission is automatically marked as **ungradable** (`"gradable": false` in results). This means the student does not lose a grading attempt and sees a clear error message.

```python
# Opt out if you want syntax errors to score 0 instead:
autograder_config = ConfigObject(
    syntax_errors_ungradable=False,
)
```

The grader also detects test collection failures (grader-side errors) and marks those as ungradable with a message for course staff.

### ConfigObject

Type-safe, immutable configuration for all autograder settings:

```python
from pytest_prairielearn_grader import ConfigObject

autograder_config = ConfigObject(
    sandbox_timeout=2.0,
    import_whitelist=["numpy", "math"],
    builtin_whitelist=["len", "range", "sum"],
    output_level="friendly",
    syntax_errors_ungradable=True,
    starting_vars={"coefficient": 10},
    names_for_user=["coefficient"],
)
```

See [ConfigObject documentation](src/pytest_prairielearn_grader/config.py) for all options.

## Results JSON Format

The autograder produces `autograder_results.json` with the following structure:

```json
{
    "gradable": true,
    "score": 0.85,
    "tests": [
        {
            "test_id": "test_student.py::test_function[student_code]",
            "name": "Test Function",
            "max_points": 5,
            "points": 5.0,
            "points_frac": 1.0,
            "outcome": "passed",
            "message": ""
        }
    ]
}
```

When a submission is ungradable:

```json
{
    "gradable": false,
    "format_errors": ["SyntaxError: invalid syntax (line 3)"],
    "message": "Your code could not be parsed. Please fix the syntax errors and resubmit.",
    "tests": []
}
```

## Development

```bash
# Install dev dependencies
pip install -e ".[dev]"

# Run tests
uv run pytest tests -p no:prairielearn-grader

# Run specific scenario
uv run pytest tests/test_autograder_scenarios.py -k "test_friendly_feedback" -v

# Format
ruff format src/ tests/

# Lint
ruff check src/ tests/ --fix

# Type check
mypy src/
```

## Docker Images

- `eliotwrobson/grader-python-pytest:latest` — full image with numpy, pandas, matplotlib, sympy
- `eliotwrobson/grader-python-pytest:lite` — minimal image with core dependencies only

## Related Issues

- [PrairieLearn #11137 — Redesign of Python Autograder](https://github.com/PrairieLearn/PrairieLearn/issues/11137)
- [PrairieLearn #13739 — New Python autograder tracking issue](https://github.com/PrairieLearn/PrairieLearn/issues/13739)
- [PrairieLearn #12113 — Customizable external grader test case visibility](https://github.com/PrairieLearn/PrairieLearn/issues/12113)
- [PrairieLearn #9636 — Mark submission as ungradable](https://github.com/PrairieLearn/PrairieLearn/issues/9636)
- [PrairieLearn #14143 — OOM/SIGKILL handling](https://github.com/PrairieLearn/PrairieLearn/issues/14143)
