Metadata-Version: 2.4
Name: promptlint
Version: 0.1.0
Summary: A comprehensive prompt analyzer for LLM optimization, security scanning, and cost estimation
Home-page: https://github.com/fyunusa/promptlint
Author: PromptLint Contributors
Author-email: PromptLint Team <team@promptlint.dev>
License: MIT
Project-URL: Homepage, https://github.com/yourusername/promptlint
Project-URL: Bug Tracker, https://github.com/yourusername/promptlint/issues
Project-URL: Documentation, https://github.com/yourusername/promptlint#readme
Keywords: prompt,linting,analysis,optimization,security,cost,estimation
Classifier: Development Status :: 4 - Beta
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Natural Language :: English
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Topic :: Text Processing :: Linguistic
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: typer>=0.9.0
Requires-Dist: rich>=13.7.0
Requires-Dist: pydantic>=2.5.0
Requires-Dist: PyYAML>=6.0
Requires-Dist: tiktoken>=0.5.0
Provides-Extra: dev
Requires-Dist: pytest>=7.4.0; extra == "dev"
Requires-Dist: pytest-cov>=4.1.0; extra == "dev"
Requires-Dist: black>=23.12.0; extra == "dev"
Requires-Dist: ruff>=0.1.0; extra == "dev"
Requires-Dist: mypy>=1.7.0; extra == "dev"
Dynamic: author
Dynamic: home-page
Dynamic: license-file
Dynamic: requires-python

# PromptLint

> A developer tool for analyzing, scoring, and optimizing LLM prompts with built-in security scanning and cost estimation.

## Overview

PromptLint is a CLI tool that helps developers write better prompts by providing actionable feedback on clarity, cost efficiency, and security. It's designed as Phase 1 of a larger vision (PromptIR) but delivers immediate value as a standalone linting tool.

**Think of it as:** ESLint for prompts, with diff capabilities and security scanning.

---

## Quick Start

### Installation

```bash
# Install from source
poetry install

# Make CLI available
poetry run promptlint --help
```

### Basic Usage

```bash
# Score a prompt
poetry run promptlint score examples/good_prompt.txt

# Compare two prompt versions
poetry run promptlint diff examples/basic_prompt.txt examples/good_prompt.txt

# Security analysis
poetry run promptlint security examples/injection_prompt.txt

# Cost estimation across models
poetry run promptlint estimate examples/good_prompt.txt

# List supported models
poetry run promptlint models
```

---

## Features

### 1. Prompt Scoring

Scores prompts on three dimensions:

- **✨ Clarity (0-10)**: How well-structured and unambiguous the prompt is
  - Checks for clear instructions
  - Detects ambiguous phrases ("as needed", "maybe", "try to")
  - Validates output format specification
  - Identifies conflicting instructions

- **💰 Cost Efficiency (0-10)**: Token count and API cost estimates
  - Counts input tokens using tiktoken
  - Estimates output tokens based on complexity
  - Calculates cost for multiple models
  - Identifies expensive patterns

- **🛡️ Security (0-10)**: Detects prompt injection vulnerabilities
  - Finds injection patterns (e.g., "ignore all previous instructions")
  - Detects unvalidated variables
  - Checks for risky operations
  - Rates overall security risk

### 2. Prompt Diffing

Compare two prompt versions and see impact:

```bash
promptlint diff old_prompt.txt new_prompt.txt
```

Shows:
- Line-by-line changes
- Clarity, cost, and security deltas
- Cost impact across models
- Recommendation on whether the change improves the prompt

### 3. Security Analysis

Dedicated security scanning:

```bash
promptlint security prompt.txt
```

Detects:
- Prompt injection attempts
- Unguarded user input
- Risky patterns (code execution, information disclosure)
- Provides severity ratings and remediation suggestions

### 4. Cost Estimation

Estimate costs across models:

```bash
promptlint estimate my_prompt.txt
# Shows costs for GPT-4o, GPT-4o-mini, and Claude 3.5 Sonnet by default

# Or specify models:
promptlint estimate my_prompt.txt --models gpt-4o,gpt-4-turbo,claude-3.5-haiku
```

---

## Architecture

### Project Structure

```
promptlint/
├── promptlint/
│   ├── __init__.py
│   ├── cli.py                    # Main CLI interface (Typer)
│   │
│   ├── core/
│   │   ├── __init__.py
│   │   ├── models.py             # Pydantic data models
│   │   ├── parser.py             # Extract prompt structure
│   │   ├── analyzer.py           # Orchestrate analyzers
│   │   └── differ.py             # Compare prompts
│   │
│   ├── analyzers/
│   │   ├── __init__.py
│   │   ├── clarity.py            # Clarity scoring
│   │   ├── cost.py               # Cost analysis
│   │   └── security.py           # Security scanning
│   │
│   ├── reporters/
│   │   ├── __init__.py
│   │   ├── console.py            # Rich terminal output
│   │   ├── json_reporter.py      # JSON format
│   │   └── markdown.py           # Markdown format
│   │
│   └── utils/
│       ├── __init__.py
│       ├── tokenizer.py          # Token counting
│       └── pricing.py            # Model pricing data
│
├── tests/
│   ├── test_promptlint.py        # Comprehensive tests
│   └── fixtures/
│       ├── good_prompt.txt
│       ├── bad_prompt.txt
│       └── injection_prompt.txt
│
├── examples/
│   ├── basic_prompt.txt
│   ├── good_prompt.txt
│   ├── ambiguous_prompt.txt
│   └── injection_prompt.txt
│
├── pyproject.toml                # Poetry configuration
└── README.md                      # This file
```

### Data Models (Pydantic)

Core models form the foundation for prompt analysis:

- `ParsedPrompt`: Structured representation of a prompt
- `Instruction`: Individual instruction or directive
- `Variable`: Placeholder/variable in prompt
- `ScoreResult`: Complete analysis result
- `Issue`: Found problem in prompt
- `CostEstimate`: Cost prediction for a model
- `DiffReport`: Comparison between two prompts

---

## Module Descriptions

### Core Modules

#### `core/parser.py`

Extracts structure from raw prompt text:

```python
parsed = PromptParser.parse_file("my_prompt.txt")
# or
parsed = PromptParser.parse(raw_text)
```

Detects:
- Instructions (imperative statements)
- Variables ({name}, {{name}}, <name>, ${name})
- Conditionals (if/then logic)
- Output format requirements
- Examples
- Metadata

#### `core/analyzer.py`

Orchestrates all analyzers:

```python
result = Analyzer.analyze(parsed_prompt, models=['gpt-4o', 'gpt-4o-mini'])
```

Returns `ScoreResult` with:
- Scores for clarity, cost, security
- List of issues found
- Suggestions for improvement
- Cost estimates for each model

#### `core/differ.py`

Compares two prompts:

```python
report = Differ.diff_prompts("old.txt", "new.txt", models=['gpt-4o'])
# or
report = Differ.diff(parsed_old, parsed_new, models=['gpt-4o'])
```

### Analyzers

#### `analyzers/clarity.py`

Scores prompt clarity using heuristics:

- Clear structure detection
- Instruction counting
- Ambiguous phrase detection
- Output format validation
- Conflict detection

**Scoring:** 5.0 base + adjustments for various factors, clamped to 0-10

#### `analyzers/cost.py`

Analyzes token count and costs:

- Token counting using tiktoken (with fallback)
- Output token estimation based on complexity
- Cost calculation for multiple models
- Expensive pattern detection

**Scoring:** 10.0 base, reduced for high token counts

#### `analyzers/security.py`

Detects security vulnerabilities:

**High-risk patterns:**
- Prompt override attempts ("ignore all previous instructions")
- System prompt disclosure ("reveal your system prompt")
- Role confusion attacks
- Code execution risks

**Medium-risk patterns:**
- Unvalidated variables
- Debug mode references
- Potential information disclosure

**Scoring:** 10.0 base, reduced by severity of findings

### Reporters

#### `reporters/console.py`

Beautiful terminal output using Rich:

```python
reporter = ConsoleReporter()
reporter.print_score_report(result)
reporter.print_diff_report(diff_report)
reporter.print_security_report(result)
reporter.print_cost_estimate(result)
```

#### `reporters/json_reporter.py`

JSON format for machine parsing:

```python
json_str = JSONReporter.print_score_report(result)
```

#### `reporters/markdown.py`

Markdown format for documentation:

```python
markdown_str = MarkdownReporter.score_to_markdown(result)
```

### Utils

#### `utils/tokenizer.py`

Token counting:

```python
tokenizer = Tokenizer()
tokens = tokenizer.count_tokens(text, model='gpt-4o')
estimated_output = Tokenizer.estimate_output_tokens(prompt_text, complexity='normal')
```

Supports automatic detection with tiktoken, fallback to word-based estimation.

#### `utils/pricing.py`

Model pricing data:

```python
pricing = PricingData.get_pricing('gpt-4o')
cost = PricingData.calculate_cost('gpt-4o', input_tokens=100, output_tokens=200)
models = PricingData.get_supported_models()
```

---

## CLI Commands

### `promptlint score [PROMPT_FILE]`

Score a single prompt.

**Options:**
- `--model, -m`: Primary model (default: gpt-4o)
- `--format, -f`: Output format (console, json, markdown)
- `--output, -o`: Save to file

**Examples:**

```bash
promptlint score my_prompt.txt
promptlint score my_prompt.txt --model gpt-4o-mini --format json
promptlint score my_prompt.txt --format markdown --output report.md
```

### `promptlint diff [OLD] [NEW]`

Compare two prompts.

**Options:**
- `--format, -f`: Output format (console, json, markdown)
- `--output, -o`: Save to file
- `--models, -m`: Comma-separated models to compare

**Examples:**

```bash
promptlint diff old.txt new.txt
promptlint diff old.txt new.txt --models gpt-4o,claude-3.5-sonnet
promptlint diff old.txt new.txt --format json --output diff.json
```

### `promptlint security [PROMPT_FILE]`

Security analysis only.

**Options:**
- `--format, -f`: Output format (console, json, markdown)
- `--output, -o`: Save to file

**Examples:**

```bash
promptlint security risky_prompt.txt
promptlint security risky_prompt.txt --format markdown --output security_report.md
```

### `promptlint estimate [PROMPT_FILE]`

Cost estimation across models.

**Options:**
- `--models, -m`: Comma-separated models

**Examples:**

```bash
promptlint estimate my_prompt.txt
promptlint estimate my_prompt.txt --models gpt-4o,gpt-4o-mini,claude-3.5-sonnet
```

### `promptlint models`

List supported models and pricing.

---

## Testing

Run the comprehensive test suite:

```bash
# Using pytest
pytest tests/

# Or with poetry
poetry run pytest tests/

# With coverage
poetry run pytest tests/ --cov=promptlint
```

Test categories:

- **Parser tests**: Variable/instruction extraction, example detection
- **Analyzer tests**: Clarity, cost, and security scoring
- **Differ tests**: Prompt comparison and impact analysis
- **Integration tests**: File I/O and end-to-end workflows

---

## Example Usage

### Example 1: Analyze a Prompt

```bash
$ poetry run promptlint score examples/good_prompt.txt

📊 Prompt Analysis Report
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

┏━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━┓
┃ Metric        ┃ Score       ┃ Details            ┃
┡━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━┩
│ ✨ Clarity     │ 9.2/10 ✓    │ Excellent          │
│ 💰 Cost        │ 8.1/10 ✓    │ Reasonable         │
│ 🛡️ Security    │ 9.5/10 ✓    │ Low risk           │
└───────────────┴─────────────┴────────────────────┘

💡 Suggestions:
  • Adding examples would improve output quality

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Overall Score: 8.9/10
```

### Example 2: Compare Prompts

```bash
$ poetry run promptlint diff examples/basic_prompt.txt examples/good_prompt.txt

📊 Prompt Comparison
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Added: +8 | Removed: -1 | Modified: 0

┏━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━┓
┃ Metric   ┃ Delta    ┃ Change %  ┃
┡━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━┩
│ Clarity  │ ↑ +2.5   │ +25%      │
│ Security │ → +0.0   │ 0%        │
│ Cost     │ ↑ +0.02  │ +8%       │
└──────────┴──────────┴───────────┘

┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ 📋 Recommendation                                                              ┃
┣━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┫
┃ ✅ Clarity improved significantly                                             ┃
┃ ✅ This change improves the prompt overall                                    ┃
┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛
```

### Example 3: Security Analysis

```bash
$ poetry run promptlint security examples/injection_prompt.txt

🛡️ Security Analysis
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

❌ High Severity Issues:
  • HIGH RISK: Prompt override attempt (line 2)
    → Remove or rephrase this instruction to prevent prompt injection

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Overall Security Score: 3/10
```

---

## Configuration

Future support for `.promptlint.yaml`:

```yaml
default_model: gpt-4o
output_format: console

rules:
  clarity:
    min_score: 7.0
  cost:
    max_tokens: 2000
    warn_threshold: 1000
  security:
    min_score: 8.0

ignore_patterns:
  - "# promptlint-ignore"
```

---

## Technical Stack

- **Python 3.11+**
- **Typer**: CLI framework
- **Rich**: Terminal output formatting
- **Pydantic**: Data validation and models
- **tiktoken**: Token counting (OpenAI tokenizer)
- **PyYAML**: Configuration (optional, future)

### Dev Dependencies

- **pytest**: Testing
- **black**: Code formatting
- **ruff**: Linting
- **mypy**: Type checking

---

## Roadmap

### Phase 1 (Current): MVP ✅
- ✅ Prompt scoring (clarity, cost, security)
- ✅ Diff functionality
- ✅ CLI interface
- ✅ Multiple output formats
- ✅ Comprehensive tests
- ✅ Example prompts

### Phase 2 (Next): Enhancement
- LLM-based consistency testing
- Regression testing with snapshots
- VS Code extension
- Web UI for analysis

### Phase 3: PromptIR Evolution
- Graph visualization of prompt logic
- Optimization layer
- Execution simulation
- Multi-model comparison

### Phase 4: SaaS
- Team collaboration
- API access
- Enterprise features
- Prompt versioning

---

## Contributing

Contributions welcome! Areas for improvement:

- Enhanced heuristics for clarity scoring
- Support for more models and pricing data
- Plugin system for custom rules
- Performance optimizations
- Additional output formats

---

## License

MIT

---

## Notes

- **Model pricing** is current as of November 2025. Check `utils/pricing.py` for updates.
- **Token counting** uses tiktoken when available for accuracy. Falls back to word-based estimation.
- **Security patterns** are based on common OWASP prompt injection techniques.
- **Clarity heuristics** combine structural analysis with common best practices.

---

## Troubleshooting

### "tiktoken not found" warning

Token counting will use word-based fallback. Install tiktoken for accuracy:

```bash
poetry add tiktoken
```

### No cost estimates shown

Check that models are supported:

```bash
promptlint models
```

Add unsupported models to `utils/pricing.py`.

### Parser not finding expected instructions

The parser uses heuristics. Help by:
- Using imperative verbs (generate, create, analyze, etc.)
- Starting instructions on new lines
- Being explicit about the task

---

**Made with ❤️ for prompt engineers everywhere.**
