Metadata-Version: 2.5
Name: contambench
Version: 0.1.0
Summary: Stressor suites, labeled contaminated workloads, and reference measurements for testing LLM serving
Project-URL: Homepage, https://github.com/armanchhetri/contambench
Project-URL: Source, https://github.com/armanchhetri/contambench
Author-email: Arman Chhetri <ac7240@rit.edu>, Yinxi Liu <yxlics@rit.edu>
License-Expression: Apache-2.0
License-File: LICENSE
Keywords: benchmark,dataset,llm,serving,stress testing,vllm
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: Software Development :: Testing
Classifier: Topic :: System :: Benchmark
Requires-Python: >=3.10
Requires-Dist: matplotlib>=3.7
Requires-Dist: numpy>=1.24
Requires-Dist: pandas>=2.0
Requires-Dist: pyarrow>=14
Requires-Dist: pyyaml>=6
Requires-Dist: requests>=2.31
Requires-Dist: scipy>=1.10
Provides-Extra: guidellm
Requires-Dist: guidellm==0.7.1; extra == 'guidellm'
Provides-Extra: test
Requires-Dist: jsonschema>=4; extra == 'test'
Requires-Dist: pytest>=8; extra == 'test'
Description-Content-Type: text/markdown

# contambench

`contambench` is the Python package for the ContamBench data record: stressor suites,
labeled contaminated workloads, and per-request reference measurements for testing
open-weight LLM serving stacks.

A *stressor* is a request whose suffix drives decoding to the output cap. *Workload
contamination* is the share `c` of stressors in a request stream; the term is unrelated
to training-data contamination. A small fraction of stressors can push co-located
benign requests past their latency objectives, and capacity tests that replay benign
prompt pools do not exercise this case.

The package fetches and verifies the record, writes labeled workloads at a chosen `c`,
runs YAML-configured campaigns against an endpoint you operate, and scores any run with
the SLO oracle used for the published results. It does not introduce a new
load-generation technique; it drives the bundled open-loop harness or GuideLLM.

- Data record: Zenodo, [doi:10.5281/zenodo.23482023](https://doi.org/10.5281/zenodo.23482023)
- Project page: to be added on publication
- Paper: *ContamBench: Stressor Suites and Contaminated Workload Traces for Testing LLM Serving*, ICSE 2027 Tool Demonstrations and Data Showcase (under review)

## Install

```bash
pip install contambench                 # Python 3.10+, CPU only, no torch
pip install "contambench[guidellm]"     # adds GuideLLM 0.7.1
```

## 1. Get the data

```bash
contam-bench data fetch                 # all three products, ~630 MB, into ~/.cache/contambench/0.1.0
contam-bench data fetch --subset d1     # one product only: d1, d2, or d3
contam-bench data fetch --from DIR      # copy from a local download of the record
contam-bench data status                # what is present, and where
```

Every file is checked against the record's `SHA256SUMS`. Set `CONTAMBENCH_DATA=DIR` to
use an extracted record somewhere else.

## 2. Reproduce the published tables

```bash
contam-bench report
```

`report` recomputes the stressor-suite table, the contamination sweep at the high
operating point of each deployment, and the GuideLLM reuse table from the request logs,
then compares the sweep with the published values. It runs on a CPU in a few seconds.

## 3. Work with the data, no GPU needed

```bash
# a labeled workload at c = 0.10 from the Llama-3.2-3B suite; writes workload.csv and workload.manifest.json
contam-bench export --suite llama-3.2-3b-instruct --c 0.10 --out workload.csv

# score any run in the D3 format: violation rate, benign-only rate, benign tail inflation, Stage B
contam-bench score ~/.cache/contambench/0.1.0/d3_execution/runs/aws_qwen7b/rq2_sweep --arrival-rate 0.17

# copy the example campaign configurations into ./examples
contam-bench examples

# an offline campaign with synthetic latencies, to see the workflow end to end
contam-bench run --config examples/simulate.yaml
contam-bench score contambench-results/raw/<run_id>
```

Suites: `llama-3.2-3b-instruct`, `qwen2.5-7b-instruct`, `llama-3.1-8b-instruct`,
`mistral-7b-instruct-v0.3`, `qwen2.5-math-7b`. `export --k` sets how many top prompts by
Stage-A success rate the stressors are drawn from (default 50); `--benign FILE` uses your
own benign pool with the columns `prompt_id, prompt, success_rate`.

## 4. Test your own deployment

These steps send traffic to a live OpenAI-compatible endpoint and need the GPU behind
it. Run them only against systems you operate or are authorized to test. Set
`OPENAI_API_KEY` if your endpoint requires a key; keys are never written to the run logs.

```bash
# optional: record vLLM scheduler state (queue depth, KV-cache use, preemptions) during the campaign
contam-bench telemetry --endpoint http://HOST:8000 --interval 1 --out contambench-results/vllm/metrics.jsonl &

# step 1: benign capacity calibration -> capacity and operating points at 60/80/90% of the plateau
contam-bench calibrate --config examples/calibrate.yaml --endpoint http://HOST:8000

# step 2: contamination sweep at the calibrated rates (edit arrival_rate in the YAML first)
contam-bench run --config examples/sweep.yaml --endpoint http://HOST:8000

# step 3: score it
contam-bench score contambench-results/raw/<run_id>
```

Report the Stage-B line of `score` together with any result. A suite that saturates in
isolation can still fail to reach the cap on your deployment; the Mistral-7B suite in
the record is an example.

### Example configurations (`contam-bench examples` copies them to `./examples`)

| File | Campaign |
|---|---|
| `simulate.yaml` | offline run with synthetic latencies |
| `calibrate.yaml` | benign arrival-rate sweep, 50 clients |
| `sweep.yaml` | c ∈ {0, 0.02, 0.05, 0.10, 0.15} at three arrival rates, three trials |
| `control_semantic_suffix.yaml` | a plain-language suffix in place of the optimized one |
| `output_caps.yaml` | the same workload under caps of 512, 1024, 2048, 4096 tokens |

The main keys are `attack_model` (resolves the D1 suite for that model in the fetched
record), `attack_file` (a file path, or `contambench:<path inside the record>`, for
example `contambench:d1_stressors/controls/random_suffix.json`), `workload_size`,
`contamination_ratio`, `num_clients`, `arrival_rate`, `arrival_model`,
`requests_per_client`, `num_trials`, `max_tokens`, `temperature`, `seed`, and
`auto_slo: true`, which takes the SLO budgets from the c = 0 trials of the same run.
Command-line options such as `--endpoint` override the file.

Each run writes `contambench-results/raw/<run_id>/` with `requests.parquet`,
`requests.csv`, and `metadata.json`, in the same format as the record's D3 runs.

## 5. Reuse a workload in GuideLLM

An exported CSV is a GuideLLM dataset file. Hold the client fixed, run once with the
benign pool and once with the contaminated export, and score both against the same
budgets:

```bash
contam-bench export --suite llama-3.2-3b-instruct --c 0.10 --out workload.csv
contam-bench guidellm -- run \
  --backend '{"kind": "openai_http", "target": "http://HOST:8000", "stream": true,
              "extras": {"body": {"temperature": 0.7, "max_tokens": 4096}}}' \
  --profile kind=poisson,rate=13.5 --constraint kind=max_requests,count=1000 \
  --data '{"kind": "csv_file", "path": "workload.csv"}' \
  --data-column-mapper '{"kind": "generative_column_mapper", "column_mappings": {"text_column": "prompt"}}' \
  --output kind=json,path=guidellm_c0p10.json
```

## The oracle

For each arrival rate, the budget of end-to-end latency, time to first token, and time
between tokens is the 99th percentile over the c = 0 requests of the same run. A request
violates when it exceeds any of the three. Rates are computed per trial and averaged,
with a Student-t 95% interval over trials.

## Not in this package

Stressor generation, LoopLLM suffix optimization against a new checkpoint, needs a GPU
workstation and is not included. A new suite needs Stage-A records from generation and
Stage-B records from a contamination sweep on the target deployment.

## Development

```bash
pip install -e ".[test]"
pytest                                   # CPU tests; set CONTAMBENCH_DATA to also run the data checks
```

## License

Code: Apache-2.0. The data record is CC BY-NC 4.0, because its benign prompts come from
Stanford Alpaca.
