Metadata-Version: 2.5
Name: promptmod
Version: 0.1.0
Summary: Modularize, version, lint and benchmark LLM prompts. Built for Open WebUI and regulated environments.
License: Apache-2.0
License-File: LICENSE
Requires-Python: >=3.10
Requires-Dist: jinja2>=3.1
Requires-Dist: pydantic>=2.0
Requires-Dist: pyyaml>=6.0
Provides-Extra: dev
Requires-Dist: pytest; extra == 'dev'
Provides-Extra: pdf
Requires-Dist: pypdf; extra == 'pdf'
Provides-Extra: tokens
Requires-Dist: tiktoken; extra == 'tokens'
Description-Content-Type: text/markdown

# promptmod

Modularize, version, lint and benchmark LLM prompts. Built for Open WebUI and regulated environments:
runs fully offline, no prompt content ever leaves your network.

```bash
pip install promptmod            # core (jinja2, pydantic, pyyaml)
pip install "promptmod[tokens]"  # exact token counts via tiktoken
```

## Workflow

```bash
# 1. Baseline: store the monolith unchanged as v1
promptmod import monolith.txt --name assistant

# 2. Modularize (offline heuristic split) and prove it is lossless
promptmod split monolith.txt --out prompts/assistant --name assistant
promptmod verify monolith.txt prompts/assistant

# 3. Inspect, lint, version
promptmod tokens prompts/assistant
promptmod lint prompts/assistant
promptmod commit prompts/assistant -m "modular split"      # -> v2
promptmod diff assistant 1 2

# 4. Benchmark baseline vs candidate on the same questions (any OpenAI-compatible endpoint)
promptmod bench assistant questions.yaml --baseline 1 --candidate 2 \
  --url http://localhost:3000/api --path /chat/completions --model gemma-4 --key-env OWUI_KEY \
  --promptfoo-dir pf/ --fail-on-gate

# 5. Release, save into Open WebUI, rollback
promptmod promote assistant 2
promptmod push assistant production --label production --url http://localhost:3000
promptmod rollback assistant
```

Questions file (`questions.yaml`): list of `{id, task, question, expect_contains, expect_not_contains, expect_citation, expect_refusal}`.

## Serving in Open WebUI
Import `src/promptmod/owui_filter.py` as a Filter Function (or bake `promptmod` into your Docker image).
It injects the `production` version as the system prompt, caches it, falls back to the last good version,
and stamps `prompt_version` in the request metadata. Set the `store_path` valve to the mounted store directory.

## Notes
- Store (`.promptmod/`) is plain JSON: commit to Git for review and audit; every promote/rollback is logged.
- Open WebUI's prompt API differs across releases; verify `push` against your version.
- Benchmark checks are rule-based (contains, citation regex, refusal). LLM-as-judge is on the roadmap.

## Prompt sets (one prompt per Open WebUI model/purpose, shared modules)
See `examples/docs/promptmod.yaml`. Each prompt lists the shared modules it uses; edit a module once and every prompt using it gets a new version.
```bash
promptmod set status                  # latest / production version, tokens, few-shot IDs per prompt
promptmod set commit --all -m "msg"
promptmod set bench  --all --url http://localhost:11434 --model gemma4   # per-prompt questions, gate
promptmod set apply  --all --candidate production                       # write model presets into Open WebUI
```

## Few-shots per document type
Examples live in `fewshots/<type>.yaml` with IDs and a status (`draft` -> `approved`). Only approved ones render, via a `"fewshot:<type>:<k>"` entry in a prompt's module list. The version records the IDs used, and the benchmark never tests on a prompt's own examples.
```bash
promptmod fewshot generate --type excel --n 8 --url http://localhost:11434 --model gemma4
promptmod fewshot list --type excel
promptmod fewshot approve --type excel --ids excel-001 excel-003     # admin review
```
**End users:** install `src/promptmod/owui_fewshot_action.py` as an Open WebUI *Action*. A button under each answer saves that Q&A as a *draft* example (document type guessed from the file, or asked). Drafts are never used until an admin approves them.
