Metadata-Version: 2.1
Name: llmdep
Version: 0.2.0
Summary: Distributed LLM Evaluation Benchmark Server
Home-page: https://github.com/tjunlp-lab/DEP
Author: TJUNLP
License: MIT
Platform: UNKNOWN
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: tqdm
Requires-Dist: pandas
Requires-Dist: jsonlines
Requires-Dist: requests
Requires-Dist: fastapi
Requires-Dist: uvicorn
Requires-Dist: numpy
Requires-Dist: scikit-learn
Requires-Dist: evaluate
Requires-Dist: datasets
Requires-Dist: nltk
Requires-Dist: jieba
Requires-Dist: rouge-chinese
Requires-Dist: rouge-score
Requires-Dist: sacrebleu
Requires-Dist: absl-py
Requires-Dist: six
Requires-Dist: packaging

# llmdep - Distributed LLM Evaluation Benchmark Server

[![Python](https://img.shields.io/badge/python-3.9+-blue.svg)](https://www.python.org/)
[![License](https://img.shields.io/badge/license-MIT-green.svg)](LICENSE)

A distributed benchmark server for evaluating Large Language Models (LLMs) across multiple benchmarks.

## Features

- **Distributed Evaluation**: Support concurrent worker processes
- **Multiple Benchmarks**: ARC, ETHICS, MMLU-Pro, BiPaR, and more
- **Rate Limiting**: Token bucket based rate limiting for API calls
- **Batch Processing**: Process multiple model-benchmark pairs in batch
- **Resume Support**: Continue interrupted tasks with `--run_id`

## Installation

```bash
pip install llmdep
```

Or install from source:

```bash
pip install -e .
```

## Quick Start

### Configure Benchmark Workspace

Benchmarks should live outside the installed pip package. Configure a benchmark root once:

```bash
llmdep config set dataset_root /data/eval_datasets
```

The directory should contain benchmark folders directly:

```text
/data/eval_datasets/
├── BiPaR/
│   ├── dataset_card.json
│   ├── methods/
│   ├── data/
│   └── cache/
└── gsm8k/
    ├── dataset_card.json
    ├── methods/
    ├── data/
    └── cache/
```

Check the active benchmark root:

```bash
llmdep list benchmark
```

Configure a model root once as well:

```bash
llmdep config set model_root /data/llm_models
```

The directory should contain only user model folders directly. `BaseModel` is not placed here; it remains inside the installed `llmdep` package.

```text
/data/llm_models/
└── qwen-plus/
    ├── model.py
    └── model_card.json
```

Each `model.py` should expose a `Model` class with `call_with_prompt(...)`. Import the base class from the installed package:

```python
from llm.BaseModel import BaseModel

class Model(BaseModel):
    def call_with_prompt(self, input_data):
        ...
```

### Single Task

```bash
# Generate answers
llmdep gen_eval -b ARC -m qwen-plus -w 10

# Generate answers and evaluate
llmdep gen_eval -b ARC -m qwen-plus -w 10 --evaluate
```

### Batch Processing

Create a task configuration file `tasks.json`:

```json
[
    {"model": "qwen-plus", "benchmark": "ARC_parquet"},
    {"model": "deepseek", "benchmark": "ethics"}
]
```

Then run:

```bash
llmdep batch --config tasks.json -w 10
```

### Resume Interrupted Task

```bash
llmdep gen_eval --run_id 20260210-164048-322281
```

## Commands

### gen_eval

Generate model answers and optionally evaluate.

```bash
llmdep gen_eval [options]

Options:
  -b, --benchmark     Benchmark name (e.g., ARC, ethics)
  -m, --model         Model name (e.g., qwen-plus, deepseek)
  -w, --worker_nums   Number of concurrent workers (default: 2)
  --batch_size        Batch size (default: 1)
  --rate_limit        Token bucket fill rate (default: 5.0)
  --bucket_capacity   Token bucket capacity (default: 10.0)
  --limited_test      Quick test with first 10 questions, or pass N
  --evaluate         Run evaluation after generation
  --run_id          Resume from run_id
```

### evaluate

Run evaluation on existing results.

```bash
llmdep evaluate -b ARC -r 20260210-164048-322281_qwen-plus
```

### list

List available benchmarks or models.

```bash
llmdep list benchmark
llmdep list model
```

### dashboard

Start the local web control panel.

```bash
llmdep dashboard
llmdep dashboard --host 127.0.0.1 --port 8765
```

### batch

Batch process tasks from config file.

```bash
llmdep batch --config tasks.json
```

## Project Structure

```
llmdep/
├── llm/                  # Packaged BaseModel compatibility module
├── mep_client/           # Client for generating answers
│   ├── run/
│   └── API/
├── method_server/      # Benchmark implementations
│   ├── ARC_parquet/
│   ├── ethics/
│   ├── BiPaR/
│   └── ...
└── llmdep_cli.py       # CLI entry point
```

## Adding New Models

1. Configure `model_root`: `llmdep config set model_root /data/llm_models`
2. Create folder under `/data/llm_models/<your_model>/`
2. Implement `model.py` inheriting from `BaseModel`
3. Create `model_card.json`

## Adding New Benchmarks

1. Configure `dataset_root`: `llmdep config set dataset_root /data/eval_datasets`
1. Create folder under `/data/eval_datasets/<benchmark_name>/`
2. Implement `prepare_ques.py` - Prepare questions and golden answers
3. Implement `eval_script.py` - Run evaluation
4. Create `dataset_card.json`

## License

MIT License


