Metadata-Version: 2.4
Name: pymelvil
Version: 0.3.0
Summary: Optimized, versioned classifier prompts from labeled examples — GEPA prompt evolution with a classification-specific layer.
Project-URL: Homepage, https://github.com/harishsiravuri/melvil
Project-URL: Issues, https://github.com/harishsiravuri/melvil/issues
Author-email: Harish Varma Siravuri <harish.siravuri@gmail.com>
License: MIT License
        
        Copyright (c) 2026 Harish Varma Siravuri
        
        Permission is hereby granted, free of charge, to any person obtaining a copy
        of this software and associated documentation files (the "Software"), to deal
        in the Software without restriction, including without limitation the rights
        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
        copies of the Software, and to permit persons to whom the Software is
        furnished to do so, subject to the following conditions:
        
        The above copyright notice and this permission notice shall be included in all
        copies or substantial portions of the Software.
        
        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
        SOFTWARE.
License-File: LICENSE
Keywords: classification,dspy,gepa,llm,prompt-optimization
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.10
Requires-Dist: dspy>=3.0
Requires-Dist: gepa>=0.0.20
Requires-Dist: pyyaml>=6.0
Provides-Extra: dev
Requires-Dist: datasets>=3.0; extra == 'dev'
Requires-Dist: pytest>=8.0; extra == 'dev'
Requires-Dist: ruff>=0.6; extra == 'dev'
Provides-Extra: hf
Requires-Dist: datasets>=3.0; extra == 'hf'
Description-Content-Type: text/markdown

# melvil

**Labeled examples + a label taxonomy in → an optimized, versioned classifier prompt out.**

melvil wraps the [GEPA](https://github.com/gepa-ai/gepa) reflective prompt-evolution
engine with a classification-specific layer that generic prompt optimizers lack:
confusion-driven reflection, a per-label prompt codebook, and hard-example mining.
It is a pure Python library — the API is the product.

*Named for [Melvil Dewey](https://en.wikipedia.org/wiki/Melvil_Dewey), who gave
libraries a system for putting things in the right category.*

```python
import melvil as mv

examples = mv.load_csv("tickets.csv")                  # text,label columns
train, dev = mv.train_dev_split(examples, dev_size=100, seed=0)
spec = mv.TaskSpec.from_examples("ticket-intents", examples)
cfg = mv.Config(task_model="openai/gpt-4.1-mini",
                reflection_model="openai/gpt-4.1", budget="light")
artifact = mv.optimize(spec, train, dev, cfg)
print(artifact.render())                               # deployable prompt string
artifact.save("ticket_intents.v1.json")
```

## Install

```bash
pip install pymelvil             # the PyPI distribution is `pymelvil`; you `import melvil`
pip install 'pymelvil[hf]'       # + HuggingFace dataset loaders
```

From a checkout:

```bash
pip install -e '.[hf,dev]'       # + test/lint tooling
```

Model names are [LiteLLM](https://docs.litellm.ai) ids (`openai/...`,
`anthropic/...`, `openrouter/...`); set the matching API key env var
(`OPENAI_API_KEY`, `OPENROUTER_API_KEY`, ...).

## What the benchmarks say (read this before choosing features)

We benchmark honestly, including against ourselves — full protocol and numbers in
[benchmarks/RESULTS.md](benchmarks/RESULTS.md). The pre-registered confirmation pass
(8 public datasets, fresh seeds, light budget, gpt-4.1-mini) found:

- **Vanilla GEPA over melvil's rendered prompt is the strongest configuration**
  (mean test accuracy 0.781 vs 0.755 for the full classification layer and 0.762 for
  MIPROv2; seed prompt 0.703). At light budgets, prefer `features=mv.Features.none()`.
- The classification layer's per-component updates trade whole-prompt coverage for
  structure; at ~6–10 accepted proposals per run that trade loses, especially on hard
  small-taxonomy tasks. Whether it wins at medium/heavy budgets is an open question.
- Hard-example mining, behind its (strict, quarantined) accept gate, kept exemplars in
  0/24 confirmation runs — treat it as a safety-gated no-op at light budgets.

## What the classification layer does

Each feature is independently toggleable via `mv.Features` (**all off by default** — the default is the benchmark-strongest vanilla-GEPA configuration; enable the layer with `mv.Features.all()` or per-flag;
`Features.none()` is an explicit alias for the default):

1. **Per-label codebook** — the prompt is not a free-text blob but named
   components: a task instruction, one definition per label, and boundary
   rules. GEPA evolves them per-component.
2. **Confusion-driven reflection** — every full dev evaluation updates a
   confusion matrix; reflection rounds are pointed at the label components on
   both sides of the currently worst confused boundary, and the reflection LM
   is shown the top confused pairs with concrete misclassified examples.
3. **Hard-example mining** — dev examples that stay misclassified across the
   run become candidate few-shot exemplars, selected to cover the top confused
   boundaries, and kept only if they don't hurt dev accuracy. (Exemplars come
   from dev, so dev scores of exemplar-augmented artifacts are mildly
   optimistic — judge them on test.)

The rendered prompt always ends with a fixed, non-evolvable output-format
contract, so the optimizer can never break parseability.

## Everything else you get

- **`PromptArtifact`** — versioned JSON: components, models, budget and cost
  actually spent, dev scores + confusion matrix, the dev-score-vs-budget curve,
  config hash, lineage (`parent_id`). `artifact.diff(other)` gives a
  per-component diff with score deltas.
- **`evaluate(artifact, data, model=...)`** — accuracy, macro-F1, per-label
  P/R/F1, confusion, cost; pass a different `model` for a transfer evaluation.
  `report(...)` renders it as markdown.
- **Cost estimation before spending** — `estimate_optimize_cost(spec, train,
  dev, cfg)` is a dry-run upper bound; measured spend comes from the LM call
  history and lands in the artifact.
- **Run directories & resume** — every run writes `runs/<task>/<hash>-s<seed>/`
  (config, engine state, reflection traces, artifact);
  `optimize(..., resume=True)` continues an interrupted run.
- **Progress** — pass `on_round=lambda info: ...` for live dev score / spend
  after every full dev evaluation, or just read the default logging.
- **Offline testing** — `melvil.testing` ships fake LMs with a real
  optimization gradient; the whole test suite runs with no API keys.

## Worked example & docs

- [examples/quickstart.py](examples/quickstart.py) — the snippet above, runnable end-to-end on AG News.
- [examples/agnews_demo.ipynb](examples/agnews_demo.ipynb) — notebook walkthrough with a live progress callback.
- [docs/api.md](docs/api.md) — public API reference.
- [benchmarks/](benchmarks/) — the honest-benchmark harness (datasets × arms × seeds).

## Development

```bash
pip install -e '.[dev]'
pytest          # green with no API keys — fake-LM offline suite
ruff check .
```

License: MIT.
