Metadata-Version: 2.4
Name: promptracer
Version: 0.1.0
Summary: The open-source prompt engineering toolkit. Test, compare, and evaluate prompts across any LLM.
Project-URL: Homepage, https://github.com/Unlucko/promptracer
Project-URL: Repository, https://github.com/Unlucko/promptracer
Project-URL: Issues, https://github.com/Unlucko/promptracer/issues
Author-email: Cesar Segura <cesarfsegurap09@gmail.com>
License-Expression: MIT
License-File: LICENSE
Keywords: ai,anthropic,gemini,llm,ollama,openai,prompt-engineering
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.10
Requires-Dist: httpx>=0.27
Requires-Dist: pyyaml>=6.0
Requires-Dist: rich>=13.0
Requires-Dist: typer>=0.12
Provides-Extra: all
Requires-Dist: anthropic>=0.39; extra == 'all'
Requires-Dist: google-genai>=1.0; extra == 'all'
Requires-Dist: openai>=1.0; extra == 'all'
Provides-Extra: anthropic
Requires-Dist: anthropic>=0.39; extra == 'anthropic'
Provides-Extra: dev
Requires-Dist: pytest-asyncio>=0.24; extra == 'dev'
Requires-Dist: pytest>=8.0; extra == 'dev'
Requires-Dist: ruff>=0.8; extra == 'dev'
Provides-Extra: google
Requires-Dist: google-genai>=1.0; extra == 'google'
Provides-Extra: openai
Requires-Dist: openai>=1.0; extra == 'openai'
Description-Content-Type: text/markdown

# PromptRacer

**The open-source prompt engineering toolkit.** Test, compare, and evaluate your prompts across any LLM — from your terminal or Python code.

> Free & open-source alternative to LangSmith, PromptLayer, and Humanloop.

[![PyPI](https://img.shields.io/pypi/v/promptracer)](https://pypi.org/project/promptracer/)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
[![Python 3.10+](https://img.shields.io/badge/python-3.10+-blue.svg)](https://www.python.org/downloads/)

---

## Why PromptRacer?

- **One `pip install`** — no web app, no database, no account needed
- **Multi-provider** — OpenAI, Anthropic, Google Gemini, Ollama (local models)
- **Compare models side-by-side** — latency, cost, and quality in one table
- **LLM-as-judge evaluation** — automated scoring with customizable criteria
- **Version your prompts** — save as YAML, track changes with git
- **Template variables** — reusable prompts with `{{placeholders}}`
- **CLI + Python API** — use it however you want

## Install

```bash
pip install promptracer

# With specific providers:
pip install promptracer[openai]          # OpenAI
pip install promptracer[anthropic]       # Anthropic (Claude)
pip install promptracer[google]          # Google Gemini
pip install promptracer[all]             # All providers
```

Ollama works out of the box — no extra install needed (just have Ollama running locally).

## Quick Start

### Python API

```python
from promptracer import Prompt, compare, evaluate

# Create a prompt with template variables
p = Prompt("Translate to {{lang}}: {{text}}")
p.set_vars(lang="English", text="Hola mundo")

# Run against a single model
result = p.run("openai/gpt-4o")
print(result.response)   # "Hello world"
print(result.latency)    # 0.82
print(result.cost)       # 0.0003

# Compare across models
results = compare(p, models=[
    "openai/gpt-4o",
    "anthropic/claude-sonnet-4-6",
    "gemini/gemini-2.5-flash",
    "ollama/llama3",
])
results.print_table()
# ┌──────────────────────────────┬──────────────┬─────────┬─────────────────┬─────────┐
# │ Model                        │ Response     │ Latency │ Tokens (in/out) │ Cost    │
# ├──────────────────────────────┼──────────────┼─────────┼─────────────────┼─────────┤
# │ openai/gpt-4o [fastest]      │ Hello world  │ 0.52s   │ 24/3            │ $0.0001 │
# │ anthropic/claude-sonnet-4-6  │ Hello world  │ 0.61s   │ 22/3            │ $0.0001 │
# │ gemini/gemini-2.5-flash      │ Hello world  │ 0.48s   │ 20/3            │ $0.0000 │
# │ ollama/llama3 [cheapest]     │ Hello world! │ 1.20s   │ 18/4            │ FREE    │
# └──────────────────────────────┴──────────────┴─────────┴─────────────────┴─────────┘

# Evaluate quality with LLM-as-judge
eval_result = evaluate(result, criteria="accuracy", judge="openai/gpt-4o-mini")
eval_result.print()
# ╭──────────── PromptRacer Eval ────────────╮
# │ Score: 9/10                            │
# │ Criteria: accuracy                     │
# │ Reasoning: Accurate translation...     │
# ╰────────────────────────────────────────╯
```

### Save & Load Prompts as YAML

```python
# Save — version control with git!
p.save("prompts/translate.yaml")

# Load
p2 = Prompt.load("prompts/translate.yaml")
```

```yaml
# prompts/translate.yaml
template: "Translate to {{lang}}: {{text}}"
name: translate
system: You are a professional translator
vars:
  lang: English
  text: Hola mundo
```

### Prompt Versioning

```python
p = Prompt("Translate: {{text}}")
p.update("Translate accurately: {{text}}")
p.update("Translate accurately and naturally: {{text}}")

# View history
p.history()  # [{'version': 1, ...}, {'version': 2, ...}, {'version': 3, ...}]

# Compare versions
old, new = p.diff(1, 3)
```

### CLI

```bash
# Set your API keys
export OPENAI_API_KEY=sk-...
export ANTHROPIC_API_KEY=sk-ant-...
export GEMINI_API_KEY=AI...

# Run a prompt
promptracer run "What is Python?" -m openai/gpt-4o

# Compare models
promptracer compare "Explain quantum computing" \
  -m "openai/gpt-4o,anthropic/claude-sonnet-4-6,ollama/llama3"

# With template variables
promptracer compare "Translate to {{lang}}: {{text}}" \
  -v lang=French -v text="Good morning" \
  -m "openai/gpt-4o,gemini/gemini-2.5-flash"

# Evaluate a response
promptracer eval "Write a haiku about coding" \
  -m openai/gpt-4o \
  -j openai/gpt-4o-mini \
  -c "creativity and adherence to haiku format"

# Create a prompt template
promptracer init my-prompt
```

### Async Support

```python
import asyncio
from promptracer import Prompt
from promptracer.compare import acompare

async def main():
    p = Prompt("Explain {{topic}} simply")
    p.set_vars(topic="quantum computing")

    # All models run concurrently
    results = await acompare(p, models=[
        "openai/gpt-4o",
        "anthropic/claude-sonnet-4-6",
        "ollama/llama3",
    ])
    results.print_table()

asyncio.run(main())
```

## Supported Models

| Provider | Prefix | Example | API Key |
|----------|--------|---------|---------|
| OpenAI | `openai/` | `openai/gpt-4o` | `OPENAI_API_KEY` |
| Anthropic | `anthropic/` | `anthropic/claude-sonnet-4-6` | `ANTHROPIC_API_KEY` |
| Google Gemini | `gemini/` or `google/` | `gemini/gemini-2.5-flash` | `GEMINI_API_KEY` |
| Ollama | `ollama/` | `ollama/llama3` | None (local) |

## Contributing

Contributions are welcome! Please open an issue or submit a PR.

```bash
# Dev setup
git clone https://github.com/Unlucko/promptracer.git
cd promptracer
python -m venv .venv && source .venv/bin/activate
pip install -e ".[dev]"

# Run tests
pytest

# Lint
ruff check src/
```

## License

MIT License — use it however you want.
