Metadata-Version: 2.4
Name: prompt-ninja
Version: 1.0.1
Summary: Version, validate, and regression-test LLM prompts as software artifacts
Project-URL: Homepage, https://github.com/haami-com/prompt-ninja
Project-URL: Repository, https://github.com/haami-com/prompt-ninja
Project-URL: Issues, https://github.com/haami-com/prompt-ninja/issues
License-Expression: MIT
License-File: LICENSE
Keywords: ai,llm,openrouter,prompt-engineering,prompt-testing
Classifier: Development Status :: 5 - Production/Stable
Classifier: Environment :: Console
Classifier: Framework :: Pydantic :: 2
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Software Development :: Testing
Requires-Python: >=3.12
Requires-Dist: click<9,>=8.1
Requires-Dist: httpx<1,>=0.27
Requires-Dist: openai<3.0.0,>=2.0.0
Requires-Dist: pydantic<3,>=2.9
Requires-Dist: python-dotenv<2,>=1.0
Requires-Dist: rich<15,>=13.7
Provides-Extra: server
Requires-Dist: fastapi<1,>=0.115; extra == 'server'
Requires-Dist: pypdf<6,>=5.1; extra == 'server'
Requires-Dist: python-docx<2,>=1.1; extra == 'server'
Requires-Dist: python-multipart<1,>=0.0.20; extra == 'server'
Requires-Dist: uvicorn[standard]<1,>=0.34; extra == 'server'
Description-Content-Type: text/markdown

# Prompt Ninja

Prompt Ninja turns prompts into versioned, validated, and regression-tested
software artifacts. A `*.prompt.toml` file keeps the instructions, model,
typed inputs, output contract, and semantic tests together in one file you can
commit alongside your code.

## Install

```bash
pip install prompt-ninja
export OPENROUTER_API_KEY=your-key
```

That installs the CLI and the Python API. The optional `server` extra adds the
FastAPI backend for the Board of Prompts UI:

```bash
pip install 'prompt-ninja[server]'
```

Generation, execution, semantic tests, updates, and AI-assisted repair use
OpenRouter. Plain loading, rendering, and validation do not require an API key.

## From a description to a tested prompt

### 1. Create the artifact

Describe the behavior you want. Prompt Ninja generates a versioned
`*.prompt.toml` file with model settings, typed inputs, an output contract, and
semantic tests:

```bash
mkdir -p prompts
prompt-ninja generate \
  --goal "Turn release notes into a concise customer update" \
  --output prompts/customer-update.prompt.toml
```

Open the generated file and review it like source code. Instructions live under
`[prompt]`, inputs under `[[variables]]`, and behavioral expectations under
`[[tests]]`.

Structured JSON generation also writes an importable companion Pydantic module
beside the TOML and sets `metadata.output` to that model's dotted path. Run the
command from the project root. Unknown consumers are represented by
`metadata.used_by = []`; Prompt Ninja does not invent a consumer path.
Its generated tests use model-valid object expectations for structured output
and natural-language semantic expectations for text output.

### 2. Add or change expectations

Add semantic tests directly to the artifact without prescribing exact output
wording:

```toml
[[tests]]
name = "avoids internal terminology"
variable.release_notes = "The API gateway migration begins July 30."
expected_output = "A customer-friendly update that preserves the date and does not use internal engineering terminology."
```

### 3. Validate or repair the artifact

```bash
prompt-ninja validate prompts/customer-update.prompt.toml
prompt-ninja validate prompts
```

Validation locally checks the TOML schema, variables, defaults, model settings,
output contract, and importable Pydantic paths. If validation fails, ask Prompt
Ninja's versioned repair prompt to produce a corrected artifact:

```bash
prompt-ninja validate prompts/customer-update.prompt.toml --fix
```

The replacement must validate before it is written, and the original is
preserved as `.bak`.

### 4. Run semantic regression tests

Run one prompt, one named expectation, or the complete prompt directory:

```bash
prompt-ninja test --prompt prompts/customer-update.prompt.toml --verbose
prompt-ninja test \
  --prompt prompts/customer-update.prompt.toml \
  --test-name "avoids internal terminology"
prompt-ninja test-prompts --prompts-dir prompts --plain
```

Tests use an LLM judge to evaluate meaning rather than exact wording. Commands
exit non-zero on failure, and `--plain` produces CI-friendly output. Failed
cases show the complete rationale plus actionable prompt or test-case changes.

### 5. Update the prompt from feedback

Ask Prompt Ninja to revise the prompt implementation. Tests and the pass
threshold are protected, and the candidate runs the complete contract before
promotion:

```bash
prompt-ninja update \
  prompts/customer-update.prompt.toml \
  "Preserve dates and avoid internal engineering terminology"
```

The command prints complete diagnostics. A passing candidate is written and the
previous version is preserved as `.bak`; a failing candidate leaves the original
artifact untouched.

### 6. Load it in your application

```python
from prompt_ninja import PromptCollection, PromptNinja

prompt = PromptNinja.from_file("prompts/customer-update.prompt.toml")
result = await prompt.run_openrouter(
    {"release_notes": "Search launches July 30 with CSV export."}
)

prompts = PromptCollection(dir="prompts")
customer_update = prompts.customer_update
```

## Prefer a UI?

The repository includes the **Board of Prompts**, a guided UI where multiple
LLMs enhance your brief, draft alternatives, judge them, and produce the final
tested `*.prompt.toml` artifact. You can attach up to five reference files and
choose the models used at each stage.

See the [UI setup instructions](https://github.com/haami-com/prompt-ninja#ui),
run `prompt-ninja COMMAND --help` for CLI options, or browse the
[complete documentation](https://github.com/haami-com/prompt-ninja/blob/main/docs/reference.md).
