Metadata-Version: 2.4
Name: qe-bench
Version: 0.2.1
Summary: Predict benchmark scores from a fraction of the queries
Project-URL: Homepage, https://quench.helivan.io
Project-URL: Documentation, https://quench.helivan.io/docs
Project-URL: Repository, https://github.com/helivan-research/quench
Project-URL: Changelog, https://github.com/helivan-research/quench/blob/main/CHANGELOG.md
Author-email: Helivan <hayden@helivan.io>
License: MIT
License-File: LICENSE
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.8
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.8
Requires-Dist: requests>=2.28.0
Provides-Extra: dev
Requires-Dist: mypy>=1.0; extra == 'dev'
Requires-Dist: pytest-cov>=4.0; extra == 'dev'
Requires-Dist: pytest>=7.0; extra == 'dev'
Requires-Dist: ruff>=0.1.0; extra == 'dev'
Requires-Dist: types-requests>=2.28.0; extra == 'dev'
Description-Content-Type: text/markdown

# Quench

Predict benchmark scores from a fraction of the queries.

Quench uses behavioral similarity to predict how a model will perform on an entire benchmark after evaluating only a small subset of queries. This dramatically reduces evaluation costs while maintaining accurate predictions.

**Web App:** https://quench.helivan.io

## What is a "Model"?

In Quench, a "model" is any configuration that produces responses to queries:

- **Different LLMs** (GPT-4, Claude, Llama, etc.)
- **Different prompts** (system prompts, few-shot examples, instructions)
- **Different temperatures** or sampling parameters
- **Different retrieval configurations** (RAG pipelines, vector stores)
- **Different fine-tunes** of the same base model
- **Any combination** of the above

If you want to compare how two configurations perform on a benchmark, each configuration is a "model" in Quench.

## Installation

```bash
pip install quench
```

## Quickstart

### 1. Create an Account & API Key

1. Sign up at [quench.helivan.io](https://quench.helivan.io)
2. Go to the user menu (top-right) and create an **API Key**
3. Copy the key (starts with `qk_`)

### 2. Authenticate

```python
import quench

# Option A: set in code
quench.api_key = "qk_your_key_here"

# Option B: set via environment variable
# export QUENCH_API_KEY="qk_your_key_here"
```

### 3. Browse Available Benchmarks

```python
# List public benchmarks
benchmarks = quench.Benchmark.list()
for b in benchmarks:
    print(f"  {b['name']:40s} category={b.get('metadata', {}).get('category', '?')}")

# Filter by category
safety_benchmarks = quench.Benchmark.list(category="safety")
```

### 4. Load a Benchmark

```python
benchmark = quench.Benchmark.load("jailbreak-safety-v1")

print(f"Models: {len(benchmark.models)}")
print(f"Queries: {len(benchmark.get_query_dictionary())}")
print(f"Scores: {benchmark.scores}")
```

### 5. Get Optimal Queries

Not all queries are equally informative. Quench identifies which queries best differentiate between models:

```python
optimal = benchmark.get_optimal_queries(budget=20)
print(f"Evaluate these {len(optimal['queries'])} queries:")
for q in optimal["queries"]:
    print(f"  {q['subtask']}/{q['query_id']}")
```

### 6. Predict

Run your model on the optimal queries, then predict:

```python
# Run your model on the optimal queries
responses = {}
for q in optimal["queries"]:
    st, qid = q["subtask"], q["query_id"]
    answer = my_model(q["question"])  # your model here
    responses.setdefault(st, {})[qid] = {
        "question": q["question"],
        "response": [answer],
    }

# Predict (~7s)
results = benchmark.predict({"my-model": responses})

print(f"Predicted score: {results['predicted_scores']['my-model']:.3f}")
print(f"Confidence interval: {results['confidence_interval']['my-model']}")
print(f"Most similar to: {results['similar_models']['my-model'][0]['model']}")
```

## Core Concepts

### Benchmarks

A benchmark is a collection of queries organized into subtasks, with responses from multiple models. Quench learns the behavioral patterns across these models to predict scores for new ones.

### Optimal Query Selection

Quench identifies which queries best differentiate between models using leave-one-out cross-validation, so you can evaluate the most valuable ones first.

```python
# Get the 15 most informative queries
optimal = benchmark.get_optimal_queries(budget=15)

# Or: how many queries do I need for 5% error?
budget = benchmark.estimate_query_budget(target_error=0.05)
print(f"Need ~{budget['estimated_queries']} queries for 5% prediction error")
```

### Fast Prediction

Quench keeps optimal query embeddings warm in memory. When you predict using the optimal query set, predictions complete in ~7 seconds with no setup needed:

```python
results = benchmark.predict({"my-model": responses})  # ~7s
```

For custom query sets outside the optimal set, you can explicitly stage:

```python
session = benchmark.stage(custom_query_ids)                 # Preloads embeddings
results = benchmark.predict(data, session=session)          # ~7s
```

### Prediction

Given partial responses (a model's answers to a subset of queries), Quench:
1. Embeds the new model's responses
2. Computes behavioral similarity to cached models via embedding distances
3. Applies classical MDS to get low-dimensional model representations
4. Trains Ridge regression on MDS coordinates to predict the overall benchmark score

## Example: Full Workflow

```python
import quench

quench.api_key = "qk_..."

# Load benchmark
benchmark = quench.Benchmark.load("jailbreak-safety-v1")
print(f"{len(benchmark.models)} models, {len(benchmark.get_query_dictionary())} queries")

# Get optimal queries
optimal = benchmark.get_optimal_queries(budget=20)

# Run your model on the optimal queries
responses = {}
for q in optimal["queries"]:
    st, qid = q["subtask"], q["query_id"]
    answer = my_model(q["question"])  # Your model here
    responses.setdefault(st, {})[qid] = {
        "question": q["question"],
        "response": [answer],
    }

# Predict (~7s)
results = benchmark.predict({"my-model": responses})
print(f"Predicted score: {results['predicted_scores']['my-model']:.1%}")
print(f"95% CI: {results['confidence_interval']['my-model']}")
print(f"Similar models: {[m['model'] for m in results['similar_models']['my-model']]}")
```

### Prediction Response

```python
{
    "predicted_scores": {"my-model": 0.87},
    "confidence_interval": {"my-model": [0.82, 0.92]},
    "similar_models": {
        "my-model": [
            {"model": "gpt4", "similarity": 0.95, "distance": 1.2},
        ]
    },
    "overlap_info": {
        "query_count": 20,
        "total_benchmark_queries": 500,
        "coverage": 0.04
    },
    "mds_coordinates": {"my-model": [0.1, -0.3], ...}
}
```

## API Reference

### Authentication

```python
import quench

# Simple (module-level)
quench.api_key = "qk_..."

# Advanced (thread-safe, multiple keys)
from quench import QuenchClient
client = QuenchClient(api_key="qk_...")
benchmark = client.benchmarks.load("my-benchmark")
```

### Benchmark Operations

```python
# Load a benchmark
benchmark = quench.Benchmark.load("benchmark_name")

# Create a new benchmark
benchmark = quench.Benchmark.create("new_name", data,
    embedding_model="openai/text-embedding-3-small",
    category="safety")

# Wait for processing (embeddings computed asynchronously)
benchmark.wait_until_ready(
    timeout=600,
    on_progress=lambda progress, pct: print(f"Processing: {progress} ({pct}%)")
)

# List public benchmarks
benchmarks = quench.Benchmark.list(category="math")

# List your benchmarks
mine = quench.Benchmark.list_mine()

# Delete a benchmark
benchmark.delete()
```

### Model Management

```python
# Add a model
benchmark.add_model(model_data)
benchmark.add_model(subtask_data, model_name="my_model")

# Remove a model
benchmark.remove_model("model_name")
```

### Prediction & Query Selection

```python
# Get optimal queries for a budget
optimal = benchmark.get_optimal_queries(budget=20)

# Predict scores (~7s with optimal queries)
results = benchmark.predict({"my-model": responses})

# For custom query sets, stage first
session = benchmark.stage(custom_query_ids)
results = benchmark.predict({"my-model": responses}, session=session)

# Estimate queries needed for target error
budget = benchmark.estimate_query_budget(target_error=0.05)
```

### Inspection

```python
benchmark.models                                # List of model names
benchmark.scores                                # {model: score} dict
benchmark.status                                # "processing", "ready", "failed"
benchmark.progress                              # Processing progress (when processing)
benchmark.summary()                             # Lightweight model summaries
benchmark.visualize()                           # Interactive MDS visualization HTML
benchmark.get_query_dictionary()                # {subtask/query_id: question}
benchmark.get_query_dictionary(subtask="math")  # {query_id: question}
benchmark.get_model_metadata("gpt4")            # Model metadata dict
```

### Utilities

```python
quench.embedding_providers()    # Available embedding models
quench.benchmark_categories()   # Available categories
```

## Data Format

### Benchmark/Model Data

```python
{
    "model_name": {
        "subtask_name": {
            "query_id": {
                "question": str,        # The prompt/question
                "response": [str],      # Model's response(s)
                "score": float,         # Optional: 0-1 score for this query
            }
        },
        "score": float  # Optional: overall score for this model
    }
}
```

## Feedback & Support

- **Issues:** [github.com/helivan-research/quench/issues](https://github.com/helivan-research/quench/issues)
- **Email:** info@helivan.io

## License

MIT

---

Built by [Helivan Research](https://helivan.io)
