Metadata-Version: 2.4
Name: genai-bench
Version: 0.0.4
Summary: A powerful benchmark tool designed for comprehensive token-level performance evaluation of large language model (LLM) serving systems.
Author-email: Chang Su <chang.s.su@oracle.com>
License: MIT
License-File: LICENSE
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Topic :: System :: Benchmark
Classifier: Typing :: Typed
Requires-Python: <3.13,>=3.10
Requires-Dist: click>=8.1.7
Requires-Dist: datasets>=3.1.0
Requires-Dist: gevent>=24.2.1
Requires-Dist: httpx>=0.24.0
Requires-Dist: huggingface-hub>=0.20.0
Requires-Dist: locust>=2.37.14
Requires-Dist: matplotlib>=3.9.2
Requires-Dist: numpy>=1.26.4
Requires-Dist: oci-openai>=0.1.0
Requires-Dist: oci>=2.163.0
Requires-Dist: openai>=1.0.0
Requires-Dist: openpyxl>=3.1.5
Requires-Dist: pandas>=2.2.2
Requires-Dist: pillow>=11.1.0
Requires-Dist: pydantic>=2.8.2
Requires-Dist: requests>=2.32.3
Requires-Dist: rich>=13.8.0
Requires-Dist: tenacity>=8.2.3
Requires-Dist: transformers>=4.44.2
Provides-Extra: aws
Requires-Dist: boto3>=1.34.0; extra == 'aws'
Requires-Dist: botocore>=1.34.0; extra == 'aws'
Provides-Extra: azure
Requires-Dist: azure-identity>=1.15.0; extra == 'azure'
Requires-Dist: azure-storage-blob>=12.19.0; extra == 'azure'
Provides-Extra: dev
Requires-Dist: black>=24.8.0; extra == 'dev'
Requires-Dist: isort>=5.13.2; extra == 'dev'
Requires-Dist: mypy>=1.11.1; extra == 'dev'
Requires-Dist: pre-commit>=3.8.0; extra == 'dev'
Requires-Dist: pytest-cov>=5.0.0; extra == 'dev'
Requires-Dist: pytest>=8.3.2; extra == 'dev'
Requires-Dist: ruff~=0.15.0; extra == 'dev'
Provides-Extra: docs
Requires-Dist: mike>=2.0.0; extra == 'docs'
Requires-Dist: mkdocs-material-extensions>=1.3.0; extra == 'docs'
Requires-Dist: mkdocs-material>=9.5.0; extra == 'docs'
Requires-Dist: mkdocs-minify-plugin>=0.8.0; extra == 'docs'
Requires-Dist: mkdocs>=1.5.3; extra == 'docs'
Requires-Dist: pymdown-extensions>=10.5; extra == 'docs'
Provides-Extra: gcp
Requires-Dist: google-auth>=2.25.0; extra == 'gcp'
Requires-Dist: google-cloud-storage>=2.13.0; extra == 'gcp'
Provides-Extra: multi-cloud
Requires-Dist: azure-identity>=1.15.0; extra == 'multi-cloud'
Requires-Dist: azure-storage-blob>=12.19.0; extra == 'multi-cloud'
Requires-Dist: boto3>=1.34.0; extra == 'multi-cloud'
Requires-Dist: botocore>=1.34.0; extra == 'multi-cloud'
Requires-Dist: google-auth>=2.25.0; extra == 'multi-cloud'
Requires-Dist: google-cloud-storage>=2.13.0; extra == 'multi-cloud'
Description-Content-Type: text/markdown

<!-- PROJECT LOGO -->
<br />
<div align="center">
  <a href="https://github.com/sgl-project/genai-bench">
    <img src="https://raw.githubusercontent.com/sgl-project/genai-bench/main/docs/assets/logo.png" alt="Logo" width="" height="150">
  </a>

<h3 align="center">
Unified, accurate, and beautiful LLM Benchmarking
</h3>

<div align="center">

[![PyPI version](https://img.shields.io/pypi/v/genai-bench)](https://pypi.org/project/genai-bench/)
[![Python versions](https://img.shields.io/python/required-version-toml?tomlFilePath=https%3A%2F%2Fraw.githubusercontent.com%2Fsgl-project%2Fgenai-bench%2Fmain%2Fpyproject.toml)](https://github.com/sgl-project/genai-bench)
[![Types - Mypy](https://img.shields.io/badge/types-mypy-blue)](https://github.com/sgl-project/genai-bench)
[![Coverage - coverage](https://img.shields.io/badge/coverage-93%25-brightgreen)](https://github.com/sgl-project/genai-bench)
[![License](https://img.shields.io/github/license/sgl-project/genai-bench)](https://github.com/sgl-project/genai-bench/blob/main/LICENSE)

</div>

<p align="center">
| <a href="https://docs.sglang.ai/genai-bench/user-guide/"><b>User Guide</b></a> | <a href="https://docs.sglang.ai/genai-bench/development/contributing/"><b>Contribution Guideline</b></a> |
</p>

</div>

<p align="center"><img src="https://raw.githubusercontent.com/sgl-project/genai-bench/main/docs/assets/ui_dashboard.png" alt="UI" width="1000" height=""></p>

## Introduction

Genai-bench is a powerful benchmark tool designed for comprehensive token-level performance evaluation of large language model (LLM) serving systems.

It provides detailed insights into model serving performance, offering both a user-friendly CLI and a live UI for real-time progress monitoring.

## Features

- 🛠️ **CLI Tool**: Validates user inputs and initiates benchmarks seamlessly.
- 📊 **Live UI Dashboard**: Displays current progress, logs, and real-time metrics.
- 📝 **Rich Logs**: Automatically flushed to both terminal and file upon experiment completion.
- 📈 **Experiment Analyzer**: Generates comprehensive Excel reports with pricing and raw metrics data, plus flexible plot configurations (default 2x4 grid) that visualize key performance metrics including throughput, latency (TTFT, E2E, TPOT), error rates, and RPS across different traffic scenarios and concurrency levels. Supports custom plot layouts and multi-line comparisons.

## Installation

**Quick Start**: Install with `pip install genai-bench`.
Alternatively, check [Installation Guide](https://docs.sglang.ai/genai-bench/getting-started/installation) for other options.

## How to use

### Quick Start

1. **Run a benchmark** against your model:
   ```bash
   # Text generation (chat completions)
   genai-bench benchmark --api-backend "your-backend" \
     --api-base "http://localhost:8080" \
     --api-key "your-api-key" \
     --api-model-name "your-model" \
     --task text-to-text \
     --max-time-per-run 5 \
     --max-requests-per-run 100

   # Image generation (OpenAI-compatible /v1/images/generations)
   genai-bench benchmark --api-backend openai \
     --api-base "http://localhost:8080" \
     --api-key "your-api-key" \
     --api-model-name "your-model" \
     --model-tokenizer "gpt2" \
     --task text-to-image \
     --traffic-scenario "I(1024,1024)" \
     --max-time-per-run 60 \
     --max-requests-per-run 10 \
     --dataset-path /path/to/image_prompts.txt
   ```

2. **Generate Excel reports** from your results:
   ```bash
   genai-bench excel --experiment-folder ./experiments/your_experiment \
     --excel-name results --metric-percentile mean
   ```

3. **Create visualizations**:
   ```bash
   genai-bench plot --experiments-folder ./experiments \
     --group-key traffic_scenario --preset 2x4_default
   ```

### Next Steps

If you're new to GenAI Bench, check out the [Getting Started](https://docs.sglang.ai/genai-bench/getting-started/) page.

For detailed instructions, advanced configuration options, and comprehensive examples, check out the [User Guide](https://docs.sglang.ai/genai-bench/user-guide/).

## Development

If you are interested in contributing to GenAI-Bench, you can use the [Development Guide](https://docs.sglang.ai/genai-bench/developer-guide/).