Metadata-Version: 2.4
Name: llm-bench-studio
Version: 0.1.1
Summary: Production-grade benchmarking platform for OpenAI-compatible LLM endpoints
License: MIT
Classifier: Programming Language :: Python :: 3
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Requires-Python: >=3.10
Description-Content-Type: text/markdown
Requires-Dist: fastapi>=0.115.0
Requires-Dist: uvicorn[standard]>=0.30.0
Requires-Dist: pydantic>=2.0.0
Requires-Dist: pydantic-settings>=2.0.0
Requires-Dist: httpx>=0.27.0
Requires-Dist: sqlalchemy[asyncio]>=2.0.0
Requires-Dist: aiosqlite>=0.20.0
Requires-Dist: openpyxl>=3.1.0
Requires-Dist: reportlab>=4.0.0
Requires-Dist: psutil>=6.0.0
Requires-Dist: python-multipart>=0.0.9
Requires-Dist: python-dotenv>=1.0.0

# LLM Benchmark Studio

Production-grade, local-first performance evaluation suite for OpenAI-compatible LLM endpoints. 

LLM Benchmark Studio allows you to easily evaluate, benchmark, and visualize the throughput, latency, and token-level performance of any OpenAI-compatible API endpoint (including **vLLM, LM Studio, Ollama, SGLang, TGI, OpenAI**, and local custom routers).

---

## Key Features

- **🔌 Connections Manager**: Configure and store endpoint connections, target models, temperature settings, and custom request headers with a built-in lightweight network connection and latency tester.
- **📝 Prompt Dataset Editor**: Import prompt datasets (CSV/JSON), customize individual prompts, and estimate token counts using client-side tokenizer heuristics.
- **⚡ Performance Benchmarking Engine**: Run concurrent evaluations using customized batch sizes and warmup requests. Calculate throughput (Tokens/sec), Time to First Token (TTFT), and Inter-Token Latency (ITL).
- **🎛️ Force Max Tokens (Ignore EOS)**: Optional switch to ignore instruct model End-of-Sequence (EOS) tokens, forcing models to generate exactly the requested output token length.
- **🔎 Server-Side Token Verification**: Requests use `"stream_options": {"include_usage": true}` to extract exact, server-reported input/output token counts for absolute timing accuracy.
- **📊 Parameter Sweep Matrix**: Benchmark multiple parameter combinations (concurrencies, token sizes) in sequence. Generates sweep combinations automatically and visualizes performance frontiers.
- **📈 Real-Time Charts & Dashboard**: Live progress tracking and interactive Apache ECharts displaying latency distributions and tokens/second throughput.
- **📄 Executive Reports**: Export comparative benchmark reports side-by-side as print-ready HTML, compiled PDF documents (using WeasyPrint with a structured ReportLab fallback), or detailed multi-sheet Excel workbooks.

---

## Installation

Install LLM Benchmark Studio using `pip`:

```bash
pip install llm-bench-studio
```

*Note: Requires Python 3.10 or higher.*

---

## Quick Start

Launch both the user interface and backend server using a single command:

```bash
llm-bench-studio start
```

Once running, open your browser and navigate to:
👉 **[http://localhost:8005](http://localhost:8005)**

### Custom Host and Port Configuration

To bind the server to a custom interface or port, pass options to the CLI:

```bash
llm-bench-studio start --host 0.0.0.0 --port 8080
```

---

## Local Development Setup

If you wish to run LLM Benchmark Studio from source or contribute changes:

1. **Clone the Repository**:
   ```bash
   git clone https://github.com/bezawadasiddinikhilesh/LLM_Bench_Studio.git
   cd LLM_Bench_Studio
   ```

2. **Backend Setup**:
   ```bash
   # Create a virtual environment
   python3 -m venv venv
   source venv/bin/activate
   
   # Install dependencies
   pip install -r backend/requirements.txt
   
   # Start the API server in reload mode
   python3 -m uvicorn backend.main:app --port 8005 --reload
   ```

3. **Frontend Setup**:
   ```bash
   # Install node dependencies
   npm install
   
   # Run Vite development server
   npm run dev
   ```

4. **Compiling a Package**:
   Compile Vite assets and build a wheel distribution package locally:
   ```bash
   ./build_package.sh
   ```

---

## License

This project is licensed under the MIT License.
