Metadata-Version: 2.4
Name: promptolution
Version: 2.3.0
Summary: A framework for prompt optimization and a zoo of prompt optimization algorithms.
License-File: LICENSE
Author: Tom Zehle
Requires-Python: >=3.10,<3.13
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Provides-Extra: experiment
Provides-Extra: transformers
Provides-Extra: vllm
Requires-Dist: accelerate (>=0.20.0) ; extra == "transformers"
Requires-Dist: datasets (>=2.0.0) ; extra == "experiment"
Requires-Dist: fastparquet (>=2024.11.0)
Requires-Dist: hydra-core (>=1.3.0)
Requires-Dist: hydra-submitit-launcher (>=1.2.0) ; extra == "experiment"
Requires-Dist: numpy (>=1.26.0,<3.0.0)
Requires-Dist: openai (>=1.0.0)
Requires-Dist: pandas (>=2.2.2)
Requires-Dist: scikit-learn (>=1.5.2)
Requires-Dist: scipy (>=1.15)
Requires-Dist: tqdm (>=4.66.5)
Requires-Dist: transformers (>=4.48.0) ; extra == "transformers"
Requires-Dist: vllm (>=0.13.0) ; extra == "vllm"
Description-Content-Type: text/markdown

![Coverage](https://img.shields.io/badge/Coverage-95%25-brightgreen)
[![CI](https://github.com/gepromptet/promptolution/actions/workflows/ci.yml/badge.svg?branch=main)](https://github.com/gepromptet/promptolution/actions/workflows/ci.yml)
[![Docs](https://github.com/gepromptet/promptolution/actions/workflows/docs.yml/badge.svg?branch=main)](https://github.com/gepromptet/promptolution/actions/workflows/docs.yml)
[![PyPI version](https://img.shields.io/pypi/v/promptolution.svg)](https://pypi.org/project/promptolution/)
![Code Style](https://img.shields.io/badge/Code%20Style-black-black)
![Python Versions](https://img.shields.io/badge/Python%20Versions-%E2%89%A53.10-blue)
[![Getting Started](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/gepromptet/promptolution/blob/main/tutorials/getting_started.ipynb)

![promptolution](https://github.com/user-attachments/assets/84c050bd-61a1-4f2e-bc4e-874d9b4a69af)

<p align="center">
<img height="60" alt="lmu_logo" src="https://github.com/user-attachments/assets/5aecd0d6-fc2d-48b2-b395-d1877578a3c5" />
<img height="60" alt="mcml" src="https://github.com/user-attachments/assets/d9f3b18e-a5ec-4c3f-b449-e57cb977f483" />
<img height="60" alt="ellis_logo" src="https://github.com/user-attachments/assets/60654a27-0f8f-4624-a1d5-5122f2632bec" />
<img height="60" alt="uni_freiburg_color" src="https://github.com/user-attachments/assets/f5eabbd2-ae6a-497b-857b-71958ed77335" />
<img height="60" alt="tum_logo" src="https://github.com/user-attachments/assets/982ec2f0-ec14-4dc2-8d75-bfae09d4fa73" />
</p>

## 🚀 What is Promptolution?

**Promptolution** is a unified, modular framework for prompt optimization built for researchers and advanced practitioners who want full control over their experimental setup. Unlike end-to-end application frameworks with high abstraction, promptolution focuses exclusively on the optimization stage, providing a clean, transparent, and extensible API. It allows for simple prompt optimization for one task up to large-scale reproducible benchmark experiments.

<img width="808" height="356" alt="promptolution_framework" src="https://github.com/user-attachments/assets/e3d05493-30e3-4464-b0d6-1d3e3085f575" />

### Key Features

- Implementation of many current prompt optimizers out of the box.
- Unified LLM backend supporting API-based models, Local LLMs, and vLLM clusters.
- Built-in response caching to save costs and parallelized inference for speed.
- Detailed logging and token usage tracking for granular post-hoc analysis.

Have a look at our [Release Notes](https://gepromptet.github.io/promptolution/release-notes/) for the latest updates to promptolution.

## 📚 Scientific Publications Powered by Promptolution

- **CANTANTE: Optimizing Agentic Systems via Contrastive Credit Attribution** — Zehle, 2026. [arXiv](https://arxiv.org/abs/2605.13295)
- **MO-CAPO: Multi-Objective Cost-Aware Prompt Optimization** — Büssing et al., 2026. [arXiv](https://arxiv.org/abs/2605.18869)
- **promptolution: A Unified, Modular Framework for Prompt Optimization** — Zehle et al., 2026. [EACL 2026](https://aclanthology.org/2026.eacl-demo.21/)
- **Can Calibration of Positional Encodings Enhance Long Context Utilization?** — Zehle & Aßenmacher, 2026. [EACL 2026](https://aclanthology.org/2026.findings-eacl.120/)
- **Disambiguation-Centric Finetuning Makes Enterprise Tool-Calling LLMs More Realistic and Less Risky** — Hathidara et al., 2025. [arXiv](https://arxiv.org/abs/2507.03336)
- **CAPO: Cost-Aware Prompt Optimization** — Zehle et al., 2025. [AutoML 2025](https://proceedings.mlr.press/v293/zehle25a.html)

## 🔧 Installation and Quickstart

```
pip install promptolution
```

For local inference, add `[transformers]` (HuggingFace) or `[vllm]` (vLLM serving), or both.

Promptolution offers three ways to optimize prompts, from a one-line call to full manual control.

### Quickstart: `promptolution.optimize`

For a classification task, `optimize` builds the components for you. Simply provide your data, an LLM and its credentials, and a description of the task and start optimizing. `evaluate` then scores the resulting prompts on data the optimizer never saw.

```python
import pandas as pd
from promptolution import optimize, evaluate
from promptolution.utils import dev_test_split

# DataFrame with columns "x" (input) and "y" (label)
df = pd.read_csv("your_data.csv")
dev_df, test_df = dev_test_split(df, test_frac=0.2)

task_description = "Classify each sentence as subjective or objective."
prompts = optimize(dev_df, task_description, model_id="gpt-4o-mini", api_key="YOUR_API_KEY")
scores = evaluate(prompts, test_df, task_description, model_id="gpt-4o-mini", api_key="YOUR_API_KEY")
print(scores)  # DataFrame: prompt, score, best first
```

Optimization and evaluation are separate calls, so you can skip the evaluation entirely, score on
several datasets, or bring your own splits. If your data already ships with splits (as HuggingFace
datasets usually do), pass them directly instead of calling `dev_test_split`.

This covers classification only. For judge or reward tasks, or for full control over every
component, build them yourself (below).

### Full control: build the components

```python
import pandas as pd
from promptolution.llms import APILLM
from promptolution.tasks import ClassificationTask
from promptolution.predictors import MarkerBasedPredictor
from promptolution.optimizers import CAPO
from promptolution.utils import evaluate_prompts, dev_test_split

# DataFrame with columns "x" (input) and "y" (label)
df = pd.read_csv("your_data.csv")
dev_df, test_df = dev_test_split(df, test_frac=0.2)

llm = APILLM(
    model_id="gpt-4o-mini",
    api_url="https://api.openai.com/v1",
    api_key="YOUR_API_KEY",
)
task = ClassificationTask(
    dev_df, task_description="Classify each sentence as subjective or objective."
)
predictor = MarkerBasedPredictor(llm)
optimizer = CAPO(
    predictor=predictor,
    meta_llm=llm,
    task=task,
    initial_prompts=["Classify the text as objective or subjective.", ...],
)

best_prompts = optimizer.optimize(n_steps=10)
test_task = ClassificationTask(test_df, task_description=task.task_description)
scores = evaluate_prompts(best_prompts, test_task, predictor)
print(scores)  # DataFrame: prompt, score, evaluated on the held-out split, best first
```

The `initial_prompts` may be omitted, they are then generated from the `task_description`.

### Experiments: config-driven runs & grids (Hydra)

For reproducible research, config-driven runs and experiment grids (from YAML/CLI, locally or on
SLURM, with result files and restart), use the `promptolution-experiment` CLI: see the
[experiment guide](https://gepromptet.github.io/promptolution/guides/experiment/).

Full tutorial: [Getting Started notebook](https://github.com/gepromptet/promptolution/blob/main/tutorials/getting_started.ipynb) · [Docs](https://gepromptet.github.io/promptolution/)

## 🧠 Featured Optimizers

| **Name**      | **Paper**                                                        | **Init prompts** | **Exploration** | **Costs** | **Parallelizable** | **Few-shot** |
| ------------- | ---------------------------------------------------------------- | ---------------- | --------------- | --------- | ------------------ | ------------ |
| `CAPO`        | [Zehle et al., 2025](https://openreview.net/forum?id=UweaRrg9D0) | required         | 👍              | 💲        | ✅                 | ✅           |
| `EvoPromptDE` | [Guo et al., 2023](https://openreview.net/forum?id=ZG3RaNIsO8)   | required         | 👍              | 💲💲      | ✅                 | ❌           |
| `EvoPromptGA` | [Guo et al., 2023](https://openreview.net/forum?id=ZG3RaNIsO8)   | required         | 👍              | 💲💲      | ✅                 | ❌           |
| `OPRO`        | [Yang et al., 2023](https://openreview.net/forum?id=Bb4VGOWELI)  | optional         | 👎              | 💲💲      | ❌                 | ❌           |

## 🏗 Components

- **`Task`** – Manages the dataset, evaluation metrics, and subsampling.
- **`Predictor`** – Defines how to extract the answer from the model's response.
- **`LLM`** – A unified interface handling inference, token counting, and concurrency.
- **`Optimizer`** – The core component that implements the algorithms that refine prompts.

## 🤝 Contributing

Contributions are welcome! See [CONTRIBUTING.md](CONTRIBUTING.md) for the workflow, code quality guidelines, and how to run tests.

## 📄 Citation

If you use Promptolution in your research, please cite:

```bibtex
@inproceedings{zehle2026promptolution,
  title={promptolution: A unified, modular framework for prompt optimization},
  author={Zehle, Tom and Hei{\ss}, Timo and Schlager, Moritz and A{\ss}enmacher, Matthias and Feurer, Matthias},
  booktitle={Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 3: System Demonstrations)},
  pages={282--296},
  year={2026}
}
```

---

Developed by **Timo Heiß**, **Moritz Schlager**, **Tom Zehle**, and **Henri Oberpaur** (LMU Munich, MCML, ELLIS, TUM, Uni Freiburg).

