Testing Recipes

Recipes are real Python modules — test them like any other Python code. The scaffolder drops a tests/test_recipe.py stub when you run pretorin recipe new <id>; this page covers what to put in it.

Three Layers Worth Testing

A recipe has three layers and each rewards a different test style:

1. Pure helpers inside scripts/ — redaction, normalization, parsers. Plain unit tests with no fixtures. Fastest feedback loop; most coverage per line of test code.
2. run against a fake ctx — the script’s main entry point. Mock ctx.api_client so you don’t hit the network.
3. End-to-end through the recipe runner — load the recipe, call its script through the runner, assert the result. Slower but proves the manifest, the importlib-based dispatch, and the script all line up.

Unit-Testing Helpers

If you’ve factored out helpers into scripts/redact.py or scripts/normalize.py, import them directly:

# tests/test_helpers.py
from scripts.redact import redact_aws_keys

def test_redact_aws_keys_replaces_full_key() -> None:
    text = "AKIAIOSFODNN7EXAMPLE"
    redacted = redact_aws_keys(text)
    assert "AKIA" not in redacted
    assert "[REDACTED:AWS_KEY]" in redacted

The recipe runner adds the scripts/ directory to sys.path. In tests, make sure your pytest.ini or pyproject.toml does the same:

[tool.pytest.ini_options]
pythonpath = ["scripts"]

Testing run with a Fake ctx

The script’s run function takes a ctx argument typed as Any (loose intentionally — see Script contract). A MagicMock with AsyncMock for the I/O methods is enough:

# tests/test_run.py
from unittest.mock import AsyncMock, MagicMock
import pytest

from scripts.run_scan import run

@pytest.mark.asyncio
async def test_run_returns_summary_for_no_rules() -> None:
    ctx = MagicMock()
    ctx.system_id = "sys-1"
    ctx.api_client = MagicMock()
    ctx.api_client.get_test_manifest = AsyncMock(
        return_value={"applicable_stigs": []}
    )

    result = await run(ctx, stig_id="EMPTY_STIG")

    assert result["stig_id"] == "EMPTY_STIG"
    assert result["summary"]["total"] == 0

The four scanner recipe tests in pretorin-cli’s test suite use exactly this shape — patch get_test_manifest, call run, assert on the returned summary.

End-to-End Through the Runner

The strongest test exercises the full path: registry loads the manifest, runner imports the script, script runs against a fake API client. This is what tests/recipes/test_code_evidence_capture.py does for the code-evidence-capture recipe and it’s the regression-test pattern to copy.

Sketch:

import pytest
from pretorin.recipes import loader as loader_module
from pretorin.recipes.loader import clear_cache
from pretorin.recipes.registry import RecipeRegistry
from pretorin.recipes.runner import RecipeScriptContext, run_script
from unittest.mock import AsyncMock, MagicMock

@pytest.fixture(autouse=True)
def _isolate(tmp_path, monkeypatch):
    clear_cache()
    monkeypatch.setattr(loader_module, "_user_recipes_root", lambda: tmp_path / "u")
    monkeypatch.setattr(loader_module, "_project_recipes_root", lambda start=None: None)

@pytest.mark.asyncio
async def test_my_recipe_end_to_end() -> None:
    registry = RecipeRegistry()
    entry = registry.get("my-recipe")
    assert entry is not None

    api_client = MagicMock()
    api_client.create_evidence = AsyncMock(return_value={"id": "ev-1"})
    ctx = RecipeScriptContext(
        system_id="sys-1",
        framework_id="nist-800-53-r5",
        api_client=api_client,
        logger=MagicMock(),
        recipe_id="my-recipe",
        recipe_version="0.1.0",
        recipe_context_id=None,
    )

    result = await run_script(
        recipe=entry.active,
        script_name="capture",
        ctx=ctx,
        params={"control_id": "ac-2"},
    )

    assert result["evidence_id"] == "ev-1"

For a community recipe outside the pretorin source tree, point the loader at your recipe’s directory:

monkeypatch.setattr(loader_module, "_user_recipes_root", lambda: my_recipe_parent)

Run It Locally

Before wiring the recipe into an agent, exercise it directly:

pretorin recipe run my-recipe --param key=value --param limit=20

pretorin recipe run loads the recipe through the registry (or --path for a not-yet-registered directory), opens a recipe execution context, calls the script, prints the return value, and closes the context. It bypasses the MCP boundary, so:

• Use it for fast iteration on the script itself.
• Pure transformation recipes (return data, don’t write to the platform) work end-to-end.
• Recipes that do write through ctx.api_client need explicit audit_metadata on each call — the MCP boundary stamps automatically; this command does not. See Writer tools.

--no-context skips opening the execution context for recipes that don’t need it.

Validate as a Smoke Test

pretorin recipe validate <id> runs the manifest schema check, the script existence check, and the description-quality check. Add it to your CI as a shell-out smoke test:

- name: Validate recipes
  run: |
    pretorin recipe validate my-recipe
    pretorin recipe validate my-other-recipe

This catches “you renamed the script and forgot to update the manifest” faster than any pytest assertion will.

What Not to Test

• Don’t test the platform API. Your recipe is an adapter; testing what create_evidence does is pretorin’s job, not yours. Mock the client.
• Don’t test pydantic validation of the manifest. That’s already covered by pretorin’s loader tests. If your manifest is malformed, pretorin recipe validate will tell you.
• Don’t test redaction patterns. Use pretorin.evidence.redact’s helpers and trust them.