Metadata-Version: 2.4
Name: hellmholtz
Version: 0.6.1
Summary: A comprehensive Python package for unified LLM access, benchmarking, evaluation, and reporting
License: MIT
License-File: LICENSE
Keywords: llm,benchmark,aisuite,openai,anthropic,google,ollama
Author: Jonas Heinicke
Author-email: jonas.heinicke@helmholtz-hzi.de
Requires-Python: >=3.12,<3.15
Classifier: Development Status :: 4 - Beta
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Provides-Extra: eval
Provides-Extra: proxy
Provides-Extra: reporting
Requires-Dist: aisuite (>=0.2.0,<0.3.0)
Requires-Dist: jinja2 (>=3.1.6,<4.0.0)
Requires-Dist: litellm[proxy] ; extra == "proxy"
Requires-Dist: lm-eval ; extra == "eval"
Requires-Dist: matplotlib ; extra == "reporting"
Requires-Dist: openai (>=2.54.0,<3.0.0)
Requires-Dist: pydantic (>=2.13.5,<3.0.0)
Requires-Dist: python-dotenv (>=1.2.3,<2.0.0)
Requires-Dist: pyyaml (>=6.0.3,<7.0.0)
Requires-Dist: requests (>=2.34.2,<3.0.0)
Requires-Dist: rich (>=13.9.4,<14.0.0)
Requires-Dist: scipy ; extra == "reporting"
Requires-Dist: seaborn ; extra == "reporting"
Requires-Dist: tqdm (>=4.67.3,<5.0.0)
Requires-Dist: typer (>=0.27.2,<0.28.0)
Project-URL: Changelog, https://github.com/JonasHeinickeBio/HeLLMholtz/releases
Project-URL: Documentation, https://github.com/JonasHeinickeBio/HeLLMholtz#readme
Project-URL: Homepage, https://github.com/JonasHeinickeBio/HeLLMholtz
Project-URL: Issues, https://github.com/JonasHeinickeBio/HeLLMholtz/issues
Project-URL: Repository, https://github.com/JonasHeinickeBio/HeLLMholtz.git
Description-Content-Type: text/markdown

# HeLLMholtz LLM Suite

<div align="center">

[![Python Version](https://img.shields.io/badge/python-3.10+-blue.svg)](https://www.python.org/downloads/)
[![PyPI Version](https://img.shields.io/pypi/v/hellmholtz.svg)](https://pypi.org/project/hellmholtz/)
[![License](https://img.shields.io/badge/license-MIT-green.svg)](LICENSE)
[![Code Style](https://img.shields.io/badge/code%20style-ruff-000000.svg)](https://github.com/astral-sh/ruff)
[![Tests](https://img.shields.io/badge/tests-passing-brightgreen.svg)](https://github.com/JonasHeinickeBio/HeLLMholtz/actions)
[![Type Coverage](https://img.shields.io/badge/types-mypy-blue.svg)](https://mypy.readthedocs.io/)
[![Security](https://img.shields.io/badge/security-scan%20passing-brightgreen.svg)](https://github.com/JonasHeinickeBio/HeLLMholtz/security)
[![Documentation](https://img.shields.io/badge/docs-100%25-orange.svg)](https://hellmholtz.readthedocs.io/)
[![Downloads](https://img.shields.io/pypi/dm/hellmholtz)](https://pypi.org/project/hellmholtz/)
[![Last Commit](https://img.shields.io/github/last-commit/JonasHeinickeBio/HeLLMholtz.svg)](https://github.com/JonasHeinickeBio/HeLLMholtz/commits/main)
[![Issues](https://img.shields.io/github/issues-raw/JonasHeinickeBio/HeLLMholtz.svg)](https://github.com/JonasHeinickeBio/HeLLMholtz/issues)
[![Pull Requests](https://img.shields.io/github/issues-pr-raw/JonasHeinickeBio/HeLLMholtz.svg)](https://github.com/JonasHeinickeBio/HeLLMholtz/pulls)

A comprehensive Python package for unified LLM access, benchmarking, evaluation, and reporting. Built on top of `aisuite` with specialized support for Helmholtz Blablador models.

</div>

A comprehensive Python package for unified LLM access, benchmarking, evaluation, and reporting. Built on top of `aisuite` with specialized support for Helmholtz Blablador models.

## Features

- **Unified Client**: Single interface for OpenAI, Google, Anthropic, Ollama, and Helmholtz Blablador models
- **Centralized Configuration**: Environment-based configuration for all your projects
- **Advanced Benchmarking**: Compare model performance across temperatures, replications, and prompt categories
- **LLM-as-a-Judge Evaluation**: Automated evaluation with comprehensive statistical analysis
- **Interactive Reports**: HTML reports with Chart.js visualizations and Markdown summaries
- **Flexible Prompt System**: Support for both simple text files and structured JSON prompt collections
- **Model Monitoring**: Track Blablador model availability and configuration consistency
- **PII Anonymization**: Document anonymization with the ShinrAI PII 1.3 model, including leak validation, cross-chunk consistency, and prompt-variant benchmarking
- **LM Evaluation Harness**: Integration with EleutherAI's comprehensive evaluation suite
- **LiteLLM Proxy**: Built-in proxy server for model routing and load balancing
- **Throughput Testing**: Performance benchmarking for high-throughput scenarios
- **Model Discovery**: Dynamic model listing and availability checking (19+ BLABLADOR models currently available)

## Installation

### Basic Installation

```bash
pip install hellmholtz
```

### Development Installation

For development with all optional dependencies:

```bash
git clone https://github.com/JonasHeinickeBio/HeLLMholtz.git
cd HeLLMholtz
pip install -e ".[eval,proxy]"
```

### Poetry Installation

```bash
poetry install --with eval,proxy
```

## Configuration

HeLLMholtz supports both project-level and user-level configuration.

### Project-Level Configuration (`.env`)

1. Copy the example environment file:
```bash
cp .env.example .env
```

2. Configure your API keys in `.env`:
```bash
# OpenAI
OPENAI_API_KEY=your_openai_key

# Anthropic
ANTHROPIC_API_KEY=your_anthropic_key

# Google
GOOGLE_API_KEY=your_google_key

# Helmholtz Blablador
BLABLADOR_API_KEY=your_blablador_key
BLABLADOR_API_BASE=https://your-blablador-instance.com

# Optional: Default models
AISUITE_DEFAULT_MODELS='{"openai": "gpt-4o", "anthropic": "claude-3-haiku"}'
```

### User-Level Configuration (Recommended for pipx users)

For persistent configuration across all projects, use the `hellm-setup` tool:

```bash
# Configure once (works globally)
hellm-setup --api-key "your-blablador-key" \
            --base-url "https://api.blablador.ai/v1" \
            --default-models "meta-llama/Meta-Llama-3.1-8B-Instruct,gpt-4"
```

**Configuration Location:** `~/.config/hellmholtz/.env`

This is automatically loaded for all `hellm` commands, regardless of your current directory.

**Configuration Precedence:** Project `.env` overrides user config (useful for per-project overrides).

See [Configuration Guide](docs/CONFIGURATION.md) for more details.

## Usage

### Python API

#### Basic Chat Interface

```python
from hellmholtz.client import chat

# Simple chat
response = chat("openai:gpt-4o", "Hello, how are you?")
print(response)

# With conversation history
messages = [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "Explain quantum computing in simple terms."},
]
response = chat("anthropic:claude-3-sonnet", messages)
```

#### Benchmarking

```python
from hellmholtz.benchmark import run_benchmarks
from hellmholtz.core.prompts import load_prompts

# Load prompts from JSON file
prompts = load_prompts("prompts.json", category="reasoning")

# Run benchmarks
results = run_benchmarks(
    models=["openai:gpt-4o", "anthropic:claude-3-haiku", "blablador:gpt-4o"],
    prompts=prompts,
    temperatures=[0.1, 0.7, 1.0],
    replications=3,
)

# Analyze results
from hellmholtz.evaluation_analysis import EvaluationAnalyzer

analyzer = EvaluationAnalyzer()
analysis = analyzer.analyze_evaluation_results("results/benchmark_latest.json")
analyzer.print_analysis_summary(analysis)
```

#### PII Anonymization

Anonymize documents with the ShinrAI PII 1.3 model (`blablador:alias-anonymizer` by default). Results include the anonymized text, the detected PII entities, and a leak validation report.

```python
from hellmholtz.anonymizer import anonymize_file, anonymize_text

# Anonymize an in-memory text
result = anonymize_text("Patient John Doe was admitted on 12.03.2025.")
print(result.text)  # anonymized text
print(result.entities)  # PII entities that were replaced
print(result.validation.summary)  # leak validation summary

# Anonymize a file; writes <stem>.anonymized<suffix> plus a JSON sidecar
# (entities, mapping, validation) next to the output
result = anonymize_file("notes/patient.md", write=True)

# The entity mapping is reversible for authorized re-identification
restored = result.restore()
```

Longer documents are automatically split into word-safe chunks and reassembled with a consistent replacement mapping across chunk boundaries. Prompt variants (`default`, `framed`, `strict`, `multilingual`, `footer_off`) can be selected via the `prompt` keyword.

### Command Line Interface

HeLLMholtz provides a comprehensive CLI for all operations:

#### Chat Interface

```bash
# Simple chat
hellm chat --model openai:gpt-4o "Explain the theory of relativity"

# Interactive mode
hellm chat --model anthropic:claude-3-sonnet --interactive

# With system prompt
hellm chat --model blablador:gpt-4o --system "You are a coding assistant" "Write a Python function to calculate fibonacci numbers"
```

#### Benchmarking

```bash
# Basic benchmark
hellm bench --models openai:gpt-4o,anthropic:claude-3-haiku --prompts-file prompts.txt

# Advanced benchmark with evaluation
hellm bench \
  --models openai:gpt-4o,blablador:gpt-4o \
  --prompts-file prompts.json \
  --prompts-category reasoning \
  --temperatures 0.1,0.7,1.0 \
  --replications 3 \
  --evaluate-with openai:gpt-4o \
  --results-dir results/

# Throughput testing
hellm bench-throughput \
  --model openai:gpt-4o \
  --requests 100 \
  --concurrency 10 \
  --prompt "Write a short story about AI"
```

#### Evaluation and Analysis

```bash
# Analyze benchmark results
hellm analyze results/benchmark_latest.json --html-report analysis_report.html

# Generate reports
hellm report --results-file results/benchmark_latest.json --output report.md
```

#### Model Management

```bash
# List available Blablador models
hellm models

# Monitor model availability and test accessibility
hellm monitor --test-accessibility

# Check model configuration consistency
hellm monitor --check-config
```

#### PII Anonymization

Anonymize documents with the ShinrAI PII 1.3 model (default: `blablador:alias-anonymizer`, override with `--model`). Leak validation and fuzzy matching run by default; disable with `--no-validate` / `--no-fuzzy`.

```bash
# Anonymize a file and print the anonymized text to stdout
hellm anonymize notes/patient.md

# Write the anonymized text next to the input (with JSON sidecar)
hellm anonymize notes/patient.md --write

# Read from stdin and print the detected entity table
cat notes/patient.md | hellm anonymize --show-entities

# Machine-readable JSON output
hellm anonymize notes/patient.md --json > patient_anonymized.json

# Compare all prompt variants across documents, scored against ground truth
hellm anonymize-benchmark \
  tests/anonymizer/fixtures/sample_short_en.md \
  tests/anonymizer/fixtures/sample_german.md \
  --ground-truth tests/anonymizer/fixtures/ground_truth.json

# Benchmark specific variants with more iterations; reports land in reports/anonymizer/
hellm anonymize-benchmark docs/ -p framed,strict --iterations 3
```

Benchmark reports include per-variant latency (p50/p95), throughput, validation pass rate, entity recall, and leak rate, written as Markdown and JSON to `reports/anonymizer/`.

#### Weekly Automated Benchmarking

The repository includes a GitHub Actions workflow that automatically runs benchmarks weekly and updates reports:

- **Scheduled**: Runs every Sunday at 00:00 UTC
- **Model Discovery**: Automatically fetches latest Blablador models
- **Performance Charts**: Generates visual charts comparing model performance
- **Multiple Formats**: Creates HTML, Markdown, and PNG chart reports
- **Auto-commit**: Updates reports in the repository for public viewing

To enable automated benchmarking:

1. Set repository secrets for API keys:
   - `BLABLADOR_API_KEY`: Your Blablador API key
   - `BLABLADOR_API_BASE`: Blablador API base URL (optional)

2. The workflow will automatically:
   - Run benchmarks on selected models
   - Generate performance reports
   - Create visual charts
   - Commit updated reports to the repository

Reports are available in the `reports/` directory and include:
- `weekly_benchmark_report.html`: Interactive HTML report
- `weekly_benchmark_report.md`: Markdown summary
- `weekly_benchmark_chart.png`: Performance visualization

#### Advanced Features

```bash
# Run LM Evaluation Harness
hellm lm-eval \
  --model openai:gpt-4o \
  --tasks hellaswag,winogrande \
  --limit 100

# Start LiteLLM proxy server
hellm proxy \
  --config litellm_config.yaml \
  --port 8000
```

## Project Structure

```
hellmholtz/
├── cli.py                 # Command-line interface
├── client.py              # Unified LLM client
├── monitoring.py          # Model availability monitoring
├── evaluation_analysis.py # Statistical analysis and reporting
├── export.py              # Result export utilities
├── core/
│   ├── config.py          # Configuration management
│   └── prompts.py         # Prompt loading and validation
├── benchmark/
│   ├── runner.py          # Benchmark execution
│   ├── evaluator.py       # LLM-as-a-Judge evaluation
│   └── prompts.py         # Benchmark-specific prompts
├── providers/
│   ├── blablador_provider.py # Custom Blablador provider
│   ├── blablador_config.py   # Blablador model configuration
│   ├── blablador.py          # Blablador utilities
│   └── __init__.py
├── reporting/
│   ├── html.py            # HTML report generation
│   ├── markdown.py        # Markdown report generation
│   ├── stats.py           # Statistical calculations
│   ├── utils.py           # Reporting utilities
│   └── templates/         # HTML templates
└── integrations/
    ├── lm_eval.py         # LM Evaluation Harness integration
    └── litellm.py         # LiteLLM proxy integration
```

## Prompt System

HeLLMholtz supports two prompt formats:

### Simple Text Format (`prompts.txt`)

```
What is the capital of France?
Explain quantum computing in simple terms.
Write a Python function to reverse a string.
```

### Structured JSON Format (`prompts.json`)

```json
[
  {
    "id": "capital-france",
    "category": "knowledge",
    "description": "Test basic geographical knowledge",
    "messages": [
      {
        "role": "user",
        "content": "What is the capital of France?"
      }
    ],
    "expected_output": "Paris"
  },
  {
    "id": "quantum-explanation",
    "category": "reasoning",
    "description": "Test ability to explain complex concepts simply",
    "messages": [
      {
        "role": "user",
        "content": "Explain quantum computing in simple terms."
      }
    ]
  }
]
```

## Evaluation System

The LLM-as-a-Judge evaluation system provides:

- **Automated Scoring**: AI-powered evaluation of response quality
- **Statistical Analysis**: Comprehensive metrics and distributions
- **Model Rankings**: Performance comparisons across all dimensions
- **Interactive Reports**: Web-based visualizations of results
- **Detailed Critiques**: Specific feedback for each response

### Example Analysis Output

```
[Monitor] EVALUATION ANALYSIS RESULTS
══════════════════════════════════════════════════════════════

 OVERVIEW
• Total Evaluations: 150
• Models Tested: 3
• Prompts Tested: 5
• Success Rate: 94.7%

🏆 MODEL RANKINGS
1. openai:gpt-4o        - Avg Score: 8.7/10 (±0.8)
2. anthropic:claude-3-opus - Avg Score: 8.4/10 (±0.9)
3. blablador:gpt-4o     - Avg Score: 7.9/10 (±1.1)

 DETAILED METRICS
• Response Quality: 8.3/10 average
• Relevance: 8.6/10 average
• Accuracy: 9.1/10 average
• Creativity: 7.8/10 average
```

## Latest Benchmark Results

Recent benchmarking results from the automated weekly workflow testing BLABLADOR models:

### Model Performance Overview

| Model | Success Rate | Avg Latency | Avg Rating (1-10) | Rating Std Dev |
|-------|-------------|-------------|-------------------|----------------|
| GPT-OSS-120b | 100.0% | 5.35s | 8.5 | ±2.38 |
| Ministral-3-14B-Instruct-2512 | 100.0% | 9.55s | 7.5 | ±3.70 |

**Overall Statistics:**
- **Total Evaluations**: 8 across 4 different prompts
- **Models Tested**: 2 BLABLADOR models
- **Overall Success Rate**: 100.0%
- **Average Rating**: 8.0/10
- **Average Latency**: 7.45s

### Key Findings
- **Top Performer**: GPT-OSS-120b with highest rating (8.5/10) and fastest response time (5.35s)
- **Most Consistent**: GPT-OSS-120b with lower rating variation (±2.38)
- **Performance Gap**: 1.0 point difference between best and worst performing models
- **Model Availability**: Both tested models are fully operational with 100% success rates

### Evaluation Details
- **Prompt Categories**: Reasoning, coding, and creative writing tasks
- **Temperature Testing**: Multiple temperature settings (0.1, 0.7, 1.0) for response variation
- **LLM-as-a-Judge**: Automated evaluation with detailed critiques and statistical analysis
- **Rating Distribution**: GPT-OSS-120b received mostly 9-10 ratings, Ministral-3-14B showed more variation

### Reports and Visualizations

- [Interactive HTML Report](reports/evaluation_analysis.html) - Comprehensive evaluation analysis with charts
- [Markdown Summary](reports/benchmark_report_comprehensive.md) - Detailed performance metrics
- [Performance Chart](reports/benchmark_chart_comprehensive.png) - Visual model comparison
- [Basic Report](reports/benchmark_report.md) - Simple performance overview

Reports are automatically updated and include LLM-as-a-Judge evaluation with detailed statistical analysis and model rankings.

## Quality & Maturity

HeLLMholtz maintains enterprise-grade quality standards through comprehensive automated validation:

### Quality Metrics

| Metric | Value | Description |
|--------|-------|-------------|
| Test Coverage | ≥50% | Code coverage across all modules |
| Type Coverage | 100% | Full type annotation coverage |
| Security Score | A+ | Zero vulnerabilities detected |
| Documentation | 100% | Complete API and user documentation |
| Code Style | ruff | Industry-standard formatting |
| CI Status | ✅ | All quality gates passing |

### Automated Quality Gates

The production pipeline enforces 12 concurrent quality checks:

| Gate | Tool | Status |
|------|------|--------|
| **Syntax & Style** | Ruff | ✅ Required |
| **Formatting** | Ruff Format | ✅ Required |
| **Type Safety** | Mypy | ✅ Required |
| **Unit Tests** | pytest | ✅ Required |
| **Coverage** | pytest-cov | ✅ Required (≥50%) |
| **Security Scan** | Bandit | ✅ Required |
| **Vulnerability Audit** | Safety | ✅ Required |
| **Doc Validation** | doc8 | ✅ Required |
| **Import Organization** | isort | ✅ Required |
| **Dependency Audit** | pip-audit | ✅ Required |
| **Code Complexity** | pylint | ✅ Recommended |
| **Memory Safety** | valgrind | ✅ Optional |

### Maturity Indicators

- **Versioning**: Semantic Versioning (SemVer)
- **Release Cadence**: Automated on tag push
- **Changelog**: Automatic generation from commit messages
- **Deprecation Policy**: 6-month deprecation notice
- **Backwards Compatibility**: Strictly maintained
- **Documentation**: Versioned and auto-generated
- **Support**: Active community and enterprise support

## Development

### Setup Development Environment

```bash
# Clone repository
git clone https://github.com/JonasHeinickeBio/HeLLMholtz.git
cd HeLLMholtz

# Install with development dependencies
poetry install --with dev

# Install pre-commit hooks
poetry run pre-commit install
```

### Running Tests

```bash
# Run all tests
poetry run pytest

# Run with coverage
poetry run pytest --cov=hellmholtz --cov-report=html

# Run specific test categories
poetry run pytest -m "slow"        # Slow integration tests
poetry run pytest -m "network"     # Tests requiring network access
poetry run pytest -m "model"       # Tests using actual models
```

### Code Quality

HeLLMholtz maintains high code quality standards through comprehensive automated checks:

#### Quality Batches

```bash
# Execute all quality checks
poetry run ruff check .
poetry run ruff format . --check
poetry run mypy src/
poetry run pytest --cov=hellmholtz --cov-report=term-missing --cov-fail-under=50
```

#### Additional Quality Checks

```bash
# Security vulnerability scanning
poetry run bandit -r src/ -ll
poetry run safety check

# Documentation validation
poetry run doc8 docs/
poetry run pydocstyle src/

# Import organization
poetry run isort --check .

# Dependency auditing
poetry run pip-audit
```

#### CI/CD Quality Gates

The GitHub Actions pipeline enforces eight quality gates before deployment:

| Gate | Tool | Purpose |
|------|------|---------|
| Linting | Ruff | Detect code style issues and bugs |
| Formatting | Ruff | Ensure consistent code formatting |
| Type Checking | Mypy | Validate static type annotations |
| Unit Tests | pytest | Execute test suite with coverage |
| Security | Bandit/Safety | Scan for vulnerabilities |
| Documentation | doc8/pydocstyle | Validate docs quality |
| Imports | isort | Organize import statements |
| Dependencies | pip-audit | Audit package vulnerabilities |

All quality gates must pass before packages are built or published.

### Production Readiness

- **Continuous Integration**: Automated testing on every commit
- **Continuous Deployment**: Automated publishing to PyPI
- **Version Tagging**: Semantic versioning with automated changelog
- **Artifact Management**: Signed and verified package distributions
- **Rollback Support**: Version rollback capability
- **Monitoring**: Production deployment monitoring and alerting

## Development

### Setup Development Environment

```bash
# Clone repository
git clone https://github.com/JonasHeinickeBio/HeLLMholtz.git
cd HeLLMholtz

# Install with development dependencies
poetry install --with dev

# Install pre-commit hooks
poetry run pre-commit install
```

### Running Tests

```bash
# Run all tests
poetry run pytest

# Run with coverage
poetry run pytest --cov=hellmholtz --cov-report=html

# Run specific test categories
poetry run pytest -m "slow"        # Slow integration tests
poetry run pytest -m "network"     # Tests requiring network access
poetry run pytest -m "model"       # Tests using actual models
```

### Building Documentation

```bash
# Generate API documentation
poetry run sphinx-build docs/ docs/_build/

# Serve documentation locally
poetry run sphinx-serve docs/_build/
```

## Contributing

We welcome contributions! Please see our [Contributing Guide](CONTRIBUTING.md) for details:

1. Fork the repository
2. Create a feature branch: `git checkout -b feature/amazing-feature`
3. Make your changes and add tests
4. Run the full test suite with quality checks
5. Commit your changes following conventional commits
6. Push to the branch: `git push origin feature/amazing-feature`
7. Open a Pull Request

All contributions must pass the CI/CD quality gates before merging.

## License

This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.

## Acknowledgments

- Built on top of [aisuite](https://github.com/andrewyng/aisuite) for unified LLM access
- LLM evaluation powered by [EleutherAI's LM Evaluation Harness](https://github.com/EleutherAI/lm-evaluation-harness)
- Proxy functionality via [LiteLLM](https://github.com/BerriAI/litellm)
- Special thanks to the Helmholtz Association for Blablador model access

## Support

- Documentation: https://hellmholtz.readthedocs.io/
- Issue Tracker: https://github.com/JonasHeinickeBio/HeLLMholtz/issues
- Discussions: https://github.com/JonasHeinickeBio/HeLLMholtz/discussions
- Commercial Support: Available for enterprise deployments

---

<p align="center">Made with love for the scientific computing community</p>
- Discussions: https://github.com/JonasHeinickeBio/HeLLMholtz/discussions

---

<p align="center">Made with love for the scientific computing community</p>

