Metadata-Version: 2.4
Name: hemp-eval
Version: 0.1.0
Summary: Help Evaluate My Prompt: work out how to test an LLM prompt from the prompt and your data, run it, and compare prompt versions.
Keywords: llm,evaluation,prompt,eval,dspy,llm-as-a-judge,prompt-engineering
Author: NithilanVishvanath
Author-email: NithilanVishvanath <nithilanvishvanath@gmail.com>
License-Expression: MIT
License-File: LICENSE
Classifier: Development Status :: 3 - Alpha
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Operating System :: OS Independent
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Software Development :: Testing
Classifier: Typing :: Typed
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Requires-Dist: dspy>=3.4.1
Requires-Dist: litellm>=1.106.6
Requires-Dist: pydantic>=2.6
Requires-Dist: pyyaml>=6.0.3
Requires-Dist: rich>=13.7
Requires-Dist: typer>=0.12
Requires-Dist: pillow>=10 ; extra == 'images'
Requires-Python: >=3.11
Project-URL: Homepage, https://github.com/NithilanVishvanath/hemp
Project-URL: Repository, https://github.com/NithilanVishvanath/hemp
Project-URL: Issues, https://github.com/NithilanVishvanath/hemp/issues
Provides-Extra: images
Description-Content-Type: text/markdown

# HEMP: Help Evaluate My Prompt

You have a prompt and some example data. HEMP works out **how that prompt should be tested**, runs the test, and tells you whether a new version of the prompt is better or worse, down to the examples that got fixed or broke.

```
hemp plan  prompt.txt data.jsonl   # decide how to test the prompt (you review the plan)
hemp run   prompt.txt data.jsonl   # run it, score it, compare with simple baselines
hemp diff                          # after editing the prompt: what got better, what broke
```

- **It reads your data, not just your prompt.** An answer column with labels gets an exact-match check, JSON answers get field-by-field checks, free text gets meaning-based checks.
- **Yes/no questions for an LLM judge** cover what code can't check ("Is the answer polite?"). No 1–10 scores.
- **You can check the judge.** Label a few outputs yourself and HEMP reports how often the judge agrees with you, and tunes it to agree more.
- **Baselines give scores context.** "Always answer the most common label" and "no instructions at all" run alongside your prompt.
- **Local and private by default.** It runs on your machine with Ollama, or with any model LiteLLM supports (OpenAI, Gemini, Anthropic, ...). There's no account and no server.
- **Text and images.** A column of image paths is sent to the model as real images.

## Install

```
pip install hemp-eval
pip install "hemp-eval[images]"   # also scale down large images before sending them
```

The command is `hemp` and the Python package is `hemp`. Python 3.11+ is required.

You also need a model. Either:

- **Local (default):** install [Ollama](https://ollama.com), then `ollama pull qwen2.5:7b` and `ollama pull nomic-embed-text`. Keep Ollama running.
- **Hosted:** set `HEMP_MODEL` to a [LiteLLM model name](https://docs.litellm.ai/docs/providers) and its API key, e.g. `HEMP_MODEL=openai/gpt-4o-mini` with `OPENAI_API_KEY`, or `HEMP_MODEL=gemini/gemini-2.0-flash` with `GEMINI_API_KEY`.

Then check your setup with `hemp doctor`. It confirms each model answers, and says exactly what to fix if one doesn't. Use `--offline` to check without making any calls.

## Quick start

**1. Write your prompt** with `{column}` where each row's data goes, in `prompt.txt`:

```
Classify the sentiment of this product review as exactly one of: positive, negative, neutral.
Reply with the label only.

Review: {review}
```

**2. Collect examples** in `data.jsonl` (CSV and JSON work too), ideally with the right answer. 20–100 rows is a good start.

```
{"review": "Great product, arrived early.", "label": "positive"}
{"review": "Broke after two days.", "label": "negative"}
```

**3. Plan.** Run `hemp plan prompt.txt data.jsonl`. HEMP profiles the data, asks the planner model how to test the prompt, and writes `prompt.plan.yaml`:

```yaml
columns:
  input: [review]
  answer: label

checks:  # automatic, no LLM needed
  - check: allowed_values
    values: [negative, positive, neutral]
  - check: exact_match
    compare_to: label

questions:  # an LLM judge answers each; "yes" = pass
  - id: q1
    ask: Does the label fit the review's overall tone?

baselines:  # simple strategies your prompt should beat
  - majority_class
  - zero_shot
```

Read it, and delete or add checks and questions as you like. `hemp validate prompt.plan.yaml` checks your edits, and `hemp metrics` lists every available check.

**4. Run.** `hemp run prompt.txt data.jsonl` runs your prompt on every row exactly as written and scores it:

```
| Check             | Your prompt | majority_class (baseline) | zero_shot (baseline) |
| allowed_values    | 100% (8/8)  | 100% (8/8)                |   0% (0/8)           |
| exact_match_label | 100% (8/8)  |  38% (3/8)                |   0% (0/8)           |
```

Then it lists the examples your prompt got wrong. Results are kept in `.hemp/hemp.db`; `hemp show` lists past runs.

**5. Improve and compare.** Edit the prompt, run again, then `hemp diff`:

```
Changed: prompt   Same: plan, dataset, target model, judge model
| Check             | Run #1 | Run #2 | Change  | Fixed | Broke |
| exact_match_label | 75%    | 100%   | +25 pts |   3   |   0   |
Fixed (3): "It's ok"  before: positive  ->  now: neutral
```

Baselines are reused between runs when nothing they depend on changed, so each round only pays for your prompt.

## Images

A column of image paths (`.png`, `.jpg`, `.jpeg`, `.gif`, `.webp`, `.bmp`), image URLs, or `data:image/...` URIs is an image column. Use it in the prompt like any other column:

```
Classify this photo as cat, dog or bird. Reply with the label only.
{image}
```

```
image,label
photos/cat_01.jpg,cat
photos/dog_01.jpg,dog
```

- **What the model gets:** the image itself, at the spot where `{image}` is, not the path.
- **Paths:** relative paths are looked up next to the data file.
- **Size:** with `hemp-eval[images]`, images bigger than 1536 px are scaled down first. Set the limit with `HEMP_IMAGE_MAX_SIDE`; `0` turns it off.
- **Baselines and judge:** the zero_shot baseline sends just the image, and the judge sees the images too.
- **Model:** use one that can see images, such as a Gemini or GPT-4o model, or a local `ollama_chat/gemma3:4b`.

## Check and tune the judge

Judge scores are only useful if the judge answers the way you would. After a run with questions:

```
hemp calibrate prompt.plan.yaml
```

HEMP shows outputs one at a time, with the judge's answer hidden. You answer each question yourself: `y`, `n`, `s` to skip, `q` to stop. It then reports how often the judge agrees with you, as accuracy and Cohen's kappa (which corrects for lucky agreement), per question.

With 8 or more labels, HEMP also tunes the judge with [DSPy](https://dspy.ai): it picks worked examples from half your labels and tests on the other half. If the tuned judge agrees with you more, it's saved next to the plan and `hemp run` uses it from then on.

## Commands

| Command | What it does |
|---|---|
| `hemp doctor` | Check that your models are reachable, with a fix for each problem. |
| `hemp profile data.jsonl` | Column types and roles (input, answer, id) and value stats. No model calls. |
| `hemp plan prompt.txt data.jsonl` | Write an editable evaluation plan. `--no-llm` uses rules only. |
| `hemp validate plan.yaml` | Check an edited plan. |
| `hemp metrics` | List the checks a plan can use and their settings. |
| `hemp run prompt.txt data.jsonl` | Run and score. Useful options: `--target-model`, `--judge-model`, `--no-judge`, `--no-baselines`. |
| `hemp show [run]` | List runs, or show one with its failed examples. |
| `hemp diff [A] [B]` | Compare two runs (default: the latest two). `--fail-on-regression` for CI. |
| `hemp calibrate plan.yaml` | Label outputs, measure and tune the judge. |

## Models

| Used for | Default | Change with |
|---|---|---|
| Planning, running your prompt, judging | `ollama_chat/qwen2.5:7b` | `HEMP_MODEL` for all three, or `HEMP_PLANNER_MODEL`, `HEMP_TARGET_MODEL`, `HEMP_JUDGE_MODEL` |
| Comparing free-text answers by meaning | `ollama/nomic-embed-text` | `HEMP_EMBED_MODEL` |

Any [LiteLLM model name](https://docs.litellm.ai/docs/providers) works. A judge at least as strong as the model being tested gives better scores.

## Your own checks

Write a check in a Python file:

```python
# my_metrics.py
import hemp


@hemp.metric("polite")
def polite(ctx: hemp.MetricContext, params: hemp.MetricParams) -> hemp.MetricResult:
    """The output says please or thanks."""
    text = ctx.output.lower()
    return hemp.MetricResult.check("please" in text or "thank" in text)
```

Pass it with `--plugin my_metrics.py` to `hemp plan` or `hemp metrics`. The planner can choose it, and the plan remembers the file.

## Examples

The [examples folder](https://github.com/NithilanVishvanath/hemp/tree/main/examples) has ready-to-run projects:

| Example | What it shows |
|---|---|
| `sentiment` | Labels: exact match and baselines |
| `extraction` | JSON output checked field by field |
| `qa` | Free-text answers, a judge question, and a custom check |
| `colors` | Images: name the color of each swatch (needs a vision model) |

For example: `hemp plan examples/sentiment/prompt.txt examples/sentiment/data.jsonl`.

## Development

```
git clone https://github.com/NithilanVishvanath/hemp
cd hemp
uv sync
uv run pytest                   # unit tests, no models needed
uv run pytest -m integration    # tests that call a real model
uv run ruff check . && uv run ruff format .
uv run mypy
```

## License

MIT
