User Guide¶

This guide provides detailed instructions on how to use Plan-Lint to validate AI agent plans against security policies.

Installation¶

Install Plan-Lint using pip:

pip install plan-lint

For development installation:

git clone https://github.com/masonlogan-dev/plan-lint.git
cd plan-lint
pip install -e .

Basic Usage¶

Command Line Interface¶

Plan-Lint provides a simple CLI for validating plans:

plan-lint validate --plan path/to/plan.json

This will validate the plan against default policies and output any violations found.

Python API¶

You can also use Plan-Lint programmatically in your Python applications:

from planlint import validate_plan

plan = {
    "steps": [
        {
            "id": "step1",
            "tool": "execute_query",
            "parameters": {
                "query": "SELECT * FROM users"
            }
        }
    ]
}

result = validate_plan(plan)

if result.is_valid:
    print("Plan is valid!")
else:
    print(f"Found {len(result.violations)} violations:")
    for violation in result.violations:
        print(f"- {violation.rule}: {violation.message} (Severity: {violation.severity})")

Validating Plans¶

Plan Format¶

Plan-Lint expects plans in the following JSON format:

{
  "steps": [
    {
      "id": "step1",
      "tool": "tool_name",
      "parameters": {
        "param1": "value1",
        "param2": "value2"
      }
    },
    {
      "id": "step2",
      "tool": "another_tool",
      "parameters": {
        "param1": "value1"
      }
    }
  ]
}

Each step must have: - A unique id - A tool name - A parameters object containing the tool's parameters

CLI Options¶

The validate command provides several options:

plan-lint validate --plan path/to/plan.json [OPTIONS]

Options: - --policies: Path to custom policy files (can be specified multiple times) - --context: Path to a JSON file with additional context for policy evaluation - --output-format: Format of the output (json, yaml, or text, default: text) - --config: Path to a configuration file

Example:

plan-lint validate --plan plan.json --policies custom_policy.rego --context context.json --output-format json

Configuration File¶

You can provide a configuration file to customize Plan-Lint's behavior:

# plan-lint.yaml
policies:
  - path/to/policy1.rego
  - path/to/policy2.rego
policy_directories:
  - path/to/policy_dir
context_file: path/to/context.json
output_format: yaml

Then use it with:

plan-lint validate --plan plan.json --config plan-lint.yaml

Working with Policies¶

Default Policies¶

Plan-Lint comes with built-in policies that check for common security issues:

SQL Injection Detection: Identifies potential SQL injection vulnerabilities
Sensitive Data Exposure: Detects exposure of sensitive information like passwords and API keys
Excessive Transaction Amounts: Flags transactions with unusually high amounts
Unauthorized Admin Operations: Identifies operations that require administrative privileges
Dangerous Code Execution: Detects potentially harmful code execution patterns

To use only specific default policies:

plan-lint validate --plan plan.json --use-default-policies sql_injection,sensitive_data

Custom Policies¶

You can create custom policies to meet your specific security and compliance requirements. Policies are written in Rego, the policy language used by Open Policy Agent.

To use custom policies:

plan-lint validate --plan plan.json --policies custom_policy1.rego custom_policy2.rego

For details on writing custom policies, see the Policy Authoring Guide.

Policy Directories¶

You can also specify directories containing policy files:

plan-lint validate --plan plan.json --policy-directories path/to/policies

Providing Context¶

Policies may need additional context information to properly evaluate a plan. You can provide this context as a JSON file:

{
  "user_role": "admin",
  "environment": "production",
  "is_maintenance_window": true,
  "max_transaction_amount": 5000
}

Use it with:

plan-lint validate --plan plan.json --context context.json

or programmatically:

from planlint import validate_plan, load_context

plan = {...}  # Your plan
context = load_context("context.json")

result = validate_plan(plan, context=context)

Understanding Results¶

Validation Result Structure¶

The validation result includes:

Whether the plan is valid (is_valid)
A list of policy violations found
A summary of the validation

Each violation includes:

The rule identifier
A human-readable message explaining the issue
The severity level (low, medium, high, critical)
The category of the violation (security, privacy, etc.)
The ID of the step that caused the violation
Additional metadata specific to the violation

Output Formats¶

Plan-Lint supports multiple output formats:

Text (Default)¶

Validation Results:
✘ Plan validation failed with 1 violation

Violations:
- [HIGH] sql_injection_detection: Potential SQL injection detected in query (step: step1)
  SQL query: SELECT * FROM users WHERE username = 'admin' OR '1'='1'

JSON¶

{
  "is_valid": false,
  "violations": [
    {
      "rule": "sql_injection_detection",
      "message": "Potential SQL injection detected in query",
      "severity": "high",
      "category": "security",
      "step_id": "step1",
      "metadata": {
        "query": "SELECT * FROM users WHERE username = 'admin' OR '1'='1'"
      }
    }
  ],
  "summary": "Plan validation failed with 1 violation"
}

YAML¶

is_valid: false
violations:
  - rule: sql_injection_detection
    message: Potential SQL injection detected in query
    severity: high
    category: security
    step_id: step1
    metadata:
      query: SELECT * FROM users WHERE username = 'admin' OR '1'='1'
summary: Plan validation failed with 1 violation

Integration Examples¶

Integration with LangChain¶

from langchain.chains import LLMChain
from langchain.llms import OpenAI
from langchain.prompts import PromptTemplate
from planlint import validate_plan

llm = OpenAI(temperature=0)
prompt = PromptTemplate(
    input_variables=["task"],
    template="Generate a plan to {task}. Format as JSON with steps."
)

chain = LLMChain(llm=llm, prompt=prompt)

# Generate a plan
response = chain.run(task="transfer money from account A to account B")
plan = json.loads(response)

# Validate the plan
result = validate_plan(plan)

if not result.is_valid:
    print("The generated plan has security issues:")
    for violation in result.violations:
        print(f"- {violation.message}")
    # Handle violations (e.g., regenerate or modify the plan)
else:
    # Execute the validated plan
    execute_plan(plan)

Integration with FastAPI¶

from fastapi import FastAPI, HTTPException
from pydantic import BaseModel
from typing import List, Dict, Any
from planlint import validate_plan

app = FastAPI()

class Step(BaseModel):
    id: str
    tool: str
    parameters: Dict[str, Any]

class Plan(BaseModel):
    steps: List[Step]

@app.post("/validate-plan")
async def validate_plan_endpoint(plan: Plan):
    result = validate_plan(plan.dict())

    if not result.is_valid:
        return {
            "valid": False,
            "violations": [v.to_dict() for v in result.violations],
            "message": "Plan validation failed"
        }

    return {
        "valid": True,
        "message": "Plan validation successful"
    }

Testing Policies¶

Plan-Lint includes a test command to verify your custom policies:

plan-lint test --policies path/to/policy.rego

This will run any test cases defined in the policy file or associated test files.

For verbose output:

plan-lint test --policies path/to/policy.rego --verbose

Troubleshooting¶

Common Issues¶

Policy syntax errors:
Error: Error loading policy: rego_parse_error
Solution: Check your Rego syntax and ensure all brackets, braces, and parentheses are balanced.
Invalid plan format:
Error: Error: Plan must have a 'steps' field that is an array
Solution: Ensure your plan follows the expected format with a 'steps' array.
Missing context data:
Error: Error evaluating policy: undefined field context
Solution: Provide the required context information for your policies.
Policy not working as expected:
Solution: Use the --verbose flag to see detailed evaluation information, which can help identify why a policy isn't catching violations.

Getting Help¶

If you encounter issues not covered in this guide:

Check the GitHub repository for known issues
Open a new issue with details about your problem and steps to reproduce it

Advanced Configuration¶

Environment Variables¶

Plan-Lint supports the following environment variables:

PLANLINT_POLICY_DIRS: Colon-separated list of policy directories
PLANLINT_DEFAULT_POLICIES: Comma-separated list of default policies to use
PLANLINT_OUTPUT_FORMAT: Default output format (text, json, yaml)

Example:

export PLANLINT_POLICY_DIRS=/path/to/policies:/another/path
export PLANLINT_DEFAULT_POLICIES=sql_injection,excessive_amount
export PLANLINT_OUTPUT_FORMAT=json

plan-lint validate --plan plan.json

Ignoring Rules¶

You can ignore specific rules in your plan by adding metadata to steps:

{
  "steps": [
    {
      "id": "step1",
      "tool": "execute_query",
      "parameters": {
        "query": "SELECT * FROM users"
      },
      "metadata": {
        "planlint": {
          "ignore_rules": ["sql_injection_detection"]
        }
      }
    }
  ]
}

This tells Plan-Lint not to flag this step for the specified rules.