Metadata-Version: 2.4
Name: hellmholtz
Version: 0.6.7
Summary: A comprehensive Python package for unified LLM access, benchmarking, evaluation, and reporting
License: MIT
License-File: LICENSE
Keywords: llm,benchmark,aisuite,openai,anthropic,google,ollama
Author: Jonas Heinicke
Author-email: jonas.heinicke@helmholtz-hzi.de
Requires-Python: >=3.12,<3.15
Classifier: Development Status :: 4 - Beta
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Provides-Extra: eval
Provides-Extra: mcp
Provides-Extra: proxy
Provides-Extra: reporting
Requires-Dist: aisuite (>=0.2.0,<0.3.0)
Requires-Dist: jinja2 (>=3.1.6,<4.0.0)
Requires-Dist: litellm[proxy] ; extra == "proxy"
Requires-Dist: lm-eval ; extra == "eval"
Requires-Dist: matplotlib ; extra == "reporting"
Requires-Dist: mcp (>=1.2) ; extra == "mcp"
Requires-Dist: openai (>=2.54.0,<3.0.0)
Requires-Dist: pydantic (>=2.13.5,<3.0.0)
Requires-Dist: python-dotenv (>=1.2.3,<2.0.0)
Requires-Dist: pyyaml (>=6.0.3,<7.0.0)
Requires-Dist: requests (>=2.34.2,<3.0.0)
Requires-Dist: rich (>=13.9.4,<14.0.0)
Requires-Dist: scipy ; extra == "reporting"
Requires-Dist: seaborn ; extra == "reporting"
Requires-Dist: tqdm (>=4.67.3,<5.0.0)
Requires-Dist: typer (>=0.27.2,<0.28.0)
Project-URL: Changelog, https://github.com/JonasHeinickeBio/HeLLMholtz/releases
Project-URL: Documentation, https://github.com/JonasHeinickeBio/HeLLMholtz#readme
Project-URL: Homepage, https://github.com/JonasHeinickeBio/HeLLMholtz
Project-URL: Issues, https://github.com/JonasHeinickeBio/HeLLMholtz/issues
Project-URL: Repository, https://github.com/JonasHeinickeBio/HeLLMholtz.git
Description-Content-Type: text/markdown

# HeLLMholtz LLM Suite

<div align="center">

[![Python Version](https://img.shields.io/badge/python-3.12+-blue.svg)](https://www.python.org/downloads/)
[![PyPI Version](https://img.shields.io/pypi/v/hellmholtz.svg)](https://pypi.org/project/hellmholtz/)
[![License](https://img.shields.io/badge/license-MIT-green.svg)](LICENSE)
[![Code Style](https://img.shields.io/badge/code%20style-ruff-000000.svg)](https://github.com/astral-sh/ruff)
[![Tests](https://img.shields.io/badge/tests-passing-brightgreen.svg)](https://github.com/JonasHeinickeBio/HeLLMholtz/actions)
[![Type Coverage](https://img.shields.io/badge/types-mypy-blue.svg)](https://mypy.readthedocs.io/)
[![Security](https://img.shields.io/badge/security-scan%20passing-brightgreen.svg)](https://github.com/JonasHeinickeBio/HeLLMholtz/security)
[![Documentation](https://img.shields.io/badge/docs-100%25-orange.svg)](https://hellmholtz.readthedocs.io/)
[![Downloads](https://img.shields.io/pypi/dm/hellmholtz)](https://pypi.org/project/hellmholtz/)
[![Last Commit](https://img.shields.io/github/last-commit/JonasHeinickeBio/HeLLMholtz.svg)](https://github.com/JonasHeinickeBio/HeLLMholtz/commits/main)
[![Issues](https://img.shields.io/github/issues-raw/JonasHeinickeBio/HeLLMholtz.svg)](https://github.com/JonasHeinickeBio/HeLLMholtz/issues)
[![Pull Requests](https://img.shields.io/github/issues-pr-raw/JonasHeinickeBio/HeLLMholtz.svg)](https://github.com/JonasHeinickeBio/HeLLMholtz/pulls)

A comprehensive Python package for unified LLM access, benchmarking, evaluation, and reporting. Built on top of `aisuite` with specialized support for Helmholtz Blablador models.

</div>

A comprehensive Python package for unified LLM access, benchmarking, evaluation, and reporting. Built on top of `aisuite` with specialized support for Helmholtz Blablador models.

## Features

- **Unified Client**: Single interface for OpenAI, Google, Anthropic, Ollama, and Helmholtz Blablador models
- **Centralized Configuration**: Environment-based configuration for all your projects
- **Advanced Benchmarking**: Compare model performance across temperatures, replications, and prompt categories
- **LLM-as-a-Judge Evaluation**: Automated evaluation with comprehensive statistical analysis
- **Interactive Reports**: HTML reports with Chart.js visualizations and Markdown summaries
- **Flexible Prompt System**: Support for both simple text files and structured JSON prompt collections
- **Model Monitoring**: Track Blablador model availability and configuration consistency
- **PII Anonymization**: Document anonymization with the ShinrAI PII 1.3 model, including leak validation, cross-chunk consistency, and prompt-variant benchmarking
- **LM Evaluation Harness**: Integration with EleutherAI's comprehensive evaluation suite
- **LiteLLM Proxy**: Built-in proxy server for model routing and load balancing
- **Throughput Testing**: Performance benchmarking for high-throughput scenarios
- **Model Discovery**: Dynamic model listing and availability checking (19+ BLABLADOR models currently available)

## System-One (Laya) CLI

The new `hellm systemone` command lets you route a state with custom questions and run a benchmark across clinical scenarios.

### Usage examples

```bash
# Route a single state with a question
hellm systemone route "Patient presents with fatigue." -q "risk:Assess risk" -c "risk=low:Low;high:High"

# Run the benchmark (default 4 scenarios, 3 replications each)
hellm systemone benchmark --replications 3 --output-dir reports/systemone
```

## Installation

### Basic Installation

```bash
pip install hellmholtz
```

### Development Installation

For development with all optional dependencies:

```bash
git clone https://github.com/JonasHeinickeBio/HeLLMholtz.git
cd HeLLMholtz
pip install -e ".[eval,proxy]"
```

### Poetry Installation

```bash
poetry install --with eval,proxy
```

## Configuration

HeLLMholtz supports both project-level and user-level configuration.

### Project-Level Configuration (`.env`)

1. Copy the example environment file:
```bash
cp .env.example .env
```

2. Configure your API keys in `.env`:
```bash
# OpenAI
OPENAI_API_KEY=your_openai_key

# Anthropic
ANTHROPIC_API_KEY=your_anthropic_key

# Google
GOOGLE_API_KEY=your_google_key

# Helmholtz Blablador
BLABLADOR_API_KEY=your_blablador_key
BLABLADOR_API_BASE=https://your-blablador-instance.com

# Optional: Default models
AISUITE_DEFAULT_MODELS='{"openai": "gpt-4o", "anthropic": "claude-3-haiku"}'
```

### User-Level Configuration (Recommended for pipx users)

For persistent configuration across all projects, use the `hellm-setup` tool:

```bash
# Configure once (works globally)
hellm-setup --api-key "your-blablador-key" \
            --base-url "https://api.blablador.ai/v1" \
            --default-models "meta-llama/Meta-Llama-3.1-8B-Instruct,gpt-4"
```

**Configuration Location:** `~/.config/hellmholtz/.env`

This is automatically loaded for all `hellm` commands, regardless of your current directory.

**Configuration Precedence:** Project `.env` overrides user config (useful for per-project overrides).

See [Configuration Guide](docs/CONFIGURATION.md) for more details.

## Usage

### Python API

#### Basic Chat Interface

```python
from hellmholtz.client import chat

# Simple chat
response = chat("openai:gpt-4o", "Hello, how are you?")
print(response)

# With conversation history
messages = [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "Explain quantum computing in simple terms."},
]
response = chat("anthropic:claude-3-sonnet", messages)
```

#### Benchmarking

```python
from hellmholtz.benchmark import run_benchmarks
from hellmholtz.core.prompts import load_prompts

# Load prompts from JSON file
prompts = load_prompts("prompts.json", category="reasoning")

# Run benchmarks
results = run_benchmarks(
    models=["openai:gpt-4o", "anthropic:claude-3-haiku", "blablador:gpt-4o"],
    prompts=prompts,
    temperatures=[0.1, 0.7, 1.0],
    replications=3,
)

# Analyze results
from hellmholtz.evaluation_analysis import EvaluationAnalyzer

analyzer = EvaluationAnalyzer()
analysis = analyzer.analyze_evaluation_results("results/benchmark_latest.json")
analyzer.print_analysis_summary(analysis)
```

#### PII Anonymization

Anonymize documents with the ShinrAI PII 1.3 model (`blablador:alias-anonymizer` by default). Results include the anonymized text, the detected PII entities, and a leak validation report.

```python
from hellmholtz.anonymizer import anonymize_file, anonymize_text

# Anonymize an in-memory text
result = anonymize_text("Patient John Doe was admitted on 12.03.2025.")
print(result.text)  # anonymized text
print(result.entities)  # PII entities that were replaced
print(result.validation.summary)  # leak validation summary

# Anonymize a file; writes <stem>.anonymized<suffix> plus a JSON sidecar
# (entities, mapping, validation) next to the output
result = anonymize_file("notes/patient.md", write=True)

# The entity mapping is reversible for authorized re-identification
restored = result.restore()
```

Longer documents are automatically split into word-safe chunks and reassembled with a consistent replacement mapping across chunk boundaries. Prompt variants (`default`, `framed`, `strict`, `multilingual`, `footer_off`) can be selected via the `prompt` keyword.

### Command Line Interface

HeLLMholtz provides a comprehensive CLI for all operations:

#### Chat Interface

```bash
# Simple chat
hellm chat --model openai:gpt-4o "Explain the theory of relativity"

# Interactive mode
hellm chat --model anthropic:claude-3-sonnet --interactive

# With system prompt
hellm chat --model blablador:gpt-4o --system "You are a coding assistant" "Write a Python function to calculate fibonacci numbers"
```

#### Benchmarking

```bash
# Basic benchmark
hellm bench --models openai:gpt-4o,anthropic:claude-3-haiku --prompts-file prompts.txt

# Advanced benchmark with evaluation
hellm bench \
  --models openai:gpt-4o,blablador:gpt-4o \
  --prompts-file prompts.json \
  --prompts-category reasoning \
  --temperatures 0.1,0.7,1.0 \
  --replications 3 \
  --evaluate-with openai:gpt-4o \
  --results-dir results/

# Throughput testing
hellm bench-throughput \
  --model openai:gpt-4o \
  --requests 100 \
  --concurrency 10 \
  --prompt "Write a short story about AI"
```

#### Evaluation and Analysis

```bash
# Analyze benchmark results
hellm analyze results/benchmark_latest.json --html-report analysis_report.html

# Generate reports
hellm report --results-file results/benchmark_latest.json --output report.md
```

#### Model Management

```bash
# List available Blablador models
hellm models

# Monitor model availability and test accessibility
hellm monitor --test-accessibility

# Check model configuration consistency
hellm monitor --check-config
```

#### PII Anonymization

Anonymize documents with the ShinrAI PII 1.3 model (default: `blablador:alias-anonymizer`, override with `--model`). Leak validation and fuzzy matching run by default; disable with `--no-validate` / `--no-fuzzy`.

```bash
# Anonymize a file and print the anonymized text to stdout
hellm anonymize notes/patient.md

# Write the anonymized text next to the input (with JSON sidecar)
hellm anonymize notes/patient.md --write

# Read from stdin and print the detected entity table
cat notes/patient.md | hellm anonymize --show-entities

# Machine-readable JSON output
hellm anonymize notes/patient.md --json > patient_anonymized.json

# Compare all prompt variants across documents, scored against ground truth
hellm anonymize-benchmark \
  tests/anonymizer/fixtures/sample_short_en.md \
  tests/anonymizer/fixtures/sample_german.md \
  --ground-truth tests/anonymizer/fixtures/ground_truth.json

# Benchmark specific variants with more iterations; reports land in reports/anonymizer/
hellm anonymize-benchmark docs/ -p framed,strict --iterations 3
```

Benchmark reports include per-variant latency (p50/p95), throughput, validation pass rate, entity recall, and leak rate, written as Markdown and JSON to `reports/anonymizer/`.

#### Weekly Automated Benchmarking

The repository includes a GitHub Actions workflow (`.github/workflows/weekly-benchmark.yml`)
that runs every **Sunday at 00:00 UTC** (and on demand via *Run workflow*) to answer two
questions continuously:

1. **Which models are available?** Every registered Blablador model is probed against the
   live `/models` endpoint and tested for chat accessibility; results are written to
   `models_status.yaml` (availability, latency, category, last-checked timestamp).
2. **How performant is each model on each task?** All currently available chat models are
   benchmarked across every prompt category (reasoning, coding, creative, knowledge). The
   generated report contains an overall per-model summary plus a **model × task performance
   matrix** (success rate and average latency per category), so you can see at a glance
   which model is best for which kind of task.

Non-chat endpoints (embedding / transcription models) are detected and excluded from the
benchmark automatically. If a run is interrupted, the workflow falls back to the runner's
incremental partial results so the weekly report still reflects the completed requests.

To enable automated benchmarking, set repository secrets:

- `BLABLADOR_API_KEY`: Your Blablador API key
- `BLABLADOR_API_BASE`: Blablador API base URL (optional)

Outputs are committed back to the repository under `reports/`:

| File | Content |
|------|---------|
| `weekly_benchmark_comprehensive.md` / `.html` | Availability status, per-model summary, per-task success-rate & latency matrices, findings |
| `weekly_benchmark_report.md` / `.html` | Response-level benchmark detail (latency, tokens, ratings) |
| `weekly_benchmark_chart.png` | Performance visualization |
| `weekly_benchmark_results.json` | Raw result records for the week (regenerate reports anytime) |

The same reports can be produced locally from any benchmark run:

```bash
# Full benchmark across all chat models and prompt categories
hellm bench --models blablador:fast,blablador:large,... --all-prompts --replications 3

# Availability check (updates models_status.yaml)
hellm monitor --test-accessibility

# Comprehensive weekly-style report (availability + per-task matrix)
python scripts/generate_comprehensive_report.py results/benchmark_<latest>.json

# Standard report + chart
hellm report results/benchmark_<latest>.json --format html-detailed --output reports/report.html
hellm chart results/benchmark_<latest>.json --output reports/chart.png
```

A companion workflow (`daily-model-check.yml`, every day at 02:00 UTC) refreshes only
`models_status.yaml`, so availability information between benchmark runs stays current.

#### Latest Benchmark Results (2026-09-30)

Full run: 4 Blablador models × 36 prompts × 3 replications (432 requests, all prompts
succeeded). Quality scores are LLM-as-a-Judge ratings (1–10, judge:
`blablador:Muse Glimmer 30b`, independent of the benchmarked models); see the driver
section below for how to reproduce or extend them. Raw data:
`results/benchmark_2026-09-30T23-29-27.138617.partial.judged.jsonl`.

| Model | Judge mean | Judge median | Rated | p50 latency | p50 tokens/s |
|-------|-----------:|-------------:|------:|------------:|-------------:|
| `blablador:large` | **8.62** | 9.0 | 106/108 | 2.5 s | 90 |
| `blablador:code`  | **8.51** | 9.0 | 90/108 | 7.7 s | 41 |
| `blablador:fast`  | **8.09** | 9.0 | 106/108 | 2.8 s | 145 |
| `blablador:huge`  | **5.01** | 4.0 | 107/108 | 5.7 s | 82 |

Mean judge rating per prompt category:

| Category | fast | large | huge | code |
|----------|-----:|------:|-----:|-----:|
| reasoning | 9.3 | 8.8 | 7.9 | 9.8 |
| creative | 8.7 | 8.7 | 5.1 | 8.3 |
| coding | 8.3 | 9.4 | 4.3 | 9.2 |
| knowledge | 6.1 | 7.6 | **2.8** | 7.3 |

> ⚠️ `blablador:huge` is a persistent outlier this run — strongest on reasoning, but weak
> on knowledge/coding: many responses are truncated or stop after an empty `<think>`
> planning trace with no actual answer. `blablador:code` has 18 empty responses (excluded
> from judging). Both warrant investigation in the next cycle.

#### LLM-as-a-Judge for Existing Result Files

`hellm bench --evaluate-with` only judges freshly produced runs. To retroactively score
an existing results file (1–10 rating + critique per response), use the resumable driver:

```bash
python scripts/run_llm_judge.py results/benchmark_<latest>.jsonl \
    --model "blablador:Muse Glimmer 30b" --workers 4

# Smoke test on a handful of records first:
python scripts/run_llm_judge.py results/benchmark_<latest>.jsonl --limit 4
```

Behaviour:

- Writes `<results>.judged.jsonl` (override with `--output`); the input file is never modified.
- **Resumable**: re-running keeps existing judgements (matched by a content hash of each
  record) and only retries records still missing a rating — safe after transient 5xx or
  unparseable judge replies; checkpoints are written atomically every 10 completions.
- Identical records (same model, params, prompt and response text) are judged once and the
  verdict is shared across duplicates.
- Choose a judge model that is *not* one of the benchmarked models to avoid self-preference
  bias. Reasoning-style judges may hide the verdict in `reasoning_content`; the driver
  handles that and can be nudged with `--judge-max-tokens`.

The judged file can then be fed to the report generators above, which pick up the ratings.


#### Advanced Features

```bash
# Run LM Evaluation Harness
hellm lm-eval \
  --model openai:gpt-4o \
  --tasks hellaswag,winogrande \
  --limit 100

# Start LiteLLM proxy server
hellm proxy \
  --config litellm_config.yaml \
  --port 8000
```

## Project Structure

```
hellmholtz/
├── cli.py                 # Command-line interface
├── client.py              # Unified LLM client
├── monitoring.py          # Model availability monitoring
├── evaluation_analysis.py # Statistical analysis and reporting
├── export.py              # Result export utilities
├── core/
│   ├── config.py          # Configuration management
│   └── prompts.py         # Prompt loading and validation
├── benchmark/
│   ├── runner.py          # Benchmark execution
│   ├── evaluator.py       # LLM-as-a-Judge evaluation
│   └── prompts.py         # Benchmark-specific prompts
├── providers/
│   ├── blablador_provider.py # Custom Blablador provider
│   ├── blablador_config.py   # Blablador model configuration
│   ├── blablador.py          # Blablador utilities
│   └── __init__.py
├── reporting/
│   ├── html.py            # HTML report generation
│   ├── markdown.py        # Markdown report generation
│   ├── stats.py           # Statistical calculations
│   ├── utils.py           # Reporting utilities
│   └── templates/         # HTML templates
└── integrations/
    ├── lm_eval.py         # LM Evaluation Harness integration
    └── litellm.py         # LiteLLM proxy integration
```

## Prompt System

HeLLMholtz supports two prompt formats:

### Simple Text Format (`prompts.txt`)

```
What is the capital of France?
Explain quantum computing in simple terms.
Write a Python function to reverse a string.
```

### Structured JSON Format (`prompts.json`)

```json
[
  {
    "id": "capital-france",
    "category": "knowledge",
    "description": "Test basic geographical knowledge",
    "messages": [
      {
        "role": "user",
        "content": "What is the capital of France?"
      }
    ],
    "expected_output": "Paris"
  },
  {
    "id": "quantum-explanation",
    "category": "reasoning",
    "description": "Test ability to explain complex concepts simply",
    "messages": [
      {
        "role": "user",
        "content": "Explain quantum computing in simple terms."
      }
    ]
  }
]
```

## Evaluation System

The LLM-as-a-Judge evaluation system provides:

- **Automated Scoring**: AI-powered evaluation of response quality
- **Statistical Analysis**: Comprehensive metrics and distributions
- **Model Rankings**: Performance comparisons across all dimensions
- **Interactive Reports**: Web-based visualizations of results
- **Detailed Critiques**: Specific feedback for each response

### Example Analysis Output

```
[Monitor] EVALUATION ANALYSIS RESULTS
══════════════════════════════════════════════════════════════

 OVERVIEW
• Total Evaluations: 150
• Models Tested: 3
• Prompts Tested: 5
• Success Rate: 94.7%

🏆 MODEL RANKINGS
1. openai:gpt-4o        - Avg Score: 8.7/10 (±0.8)
2. anthropic:claude-3-opus - Avg Score: 8.4/10 (±0.9)
3. blablador:gpt-4o     - Avg Score: 7.9/10 (±1.1)

 DETAILED METRICS
• Response Quality: 8.3/10 average
• Relevance: 8.6/10 average
• Accuracy: 9.1/10 average
• Creativity: 7.8/10 average
```

## Latest Benchmark Results

Recent benchmarking results from the automated weekly workflow testing BLABLADOR models:

### Model Performance Overview

| Model | Success Rate | Avg Latency | Avg Rating (1-10) | Rating Std Dev |
|-------|-------------|-------------|-------------------|----------------|
| GPT-OSS-120b | 100.0% | 5.35s | 8.5 | ±2.38 |
| Ministral-3-14B-Instruct-2512 | 100.0% | 9.55s | 7.5 | ±3.70 |

**Overall Statistics:**
- **Total Evaluations**: 8 across 4 different prompts
- **Models Tested**: 2 BLABLADOR models
- **Overall Success Rate**: 100.0%
- **Average Rating**: 8.0/10
- **Average Latency**: 7.45s

### Key Findings
- **Top Performer**: GPT-OSS-120b with highest rating (8.5/10) and fastest response time (5.35s)
- **Most Consistent**: GPT-OSS-120b with lower rating variation (±2.38)
- **Performance Gap**: 1.0 point difference between best and worst performing models
- **Model Availability**: Both tested models are fully operational with 100% success rates

### Evaluation Details
- **Prompt Categories**: Reasoning, coding, and creative writing tasks
- **Temperature Testing**: Multiple temperature settings (0.1, 0.7, 1.0) for response variation
- **LLM-as-a-Judge**: Automated evaluation with detailed critiques and statistical analysis
- **Rating Distribution**: GPT-OSS-120b received mostly 9-10 ratings, Ministral-3-14B showed more variation

### Reports and Visualizations

- [Interactive HTML Report](reports/evaluation_analysis.html) - Comprehensive evaluation analysis with charts
- [Markdown Summary](reports/benchmark_report_comprehensive.md) - Detailed performance metrics
- [Performance Chart](reports/benchmark_chart_comprehensive.png) - Visual model comparison
- [Basic Report](reports/benchmark_report.md) - Simple performance overview

Reports are automatically updated and include LLM-as-a-Judge evaluation with detailed statistical analysis and model rankings.

## Quality & Maturity

HeLLMholtz maintains enterprise-grade quality standards through comprehensive automated validation:

### Quality Metrics

| Metric | Value | Description |
|--------|-------|-------------|
| Test Coverage | ≥50% | Code coverage across all modules |
| Type Coverage | 100% | Full type annotation coverage |
| Security Score | A+ | Zero vulnerabilities detected |
| Documentation | 100% | Complete API and user documentation |
| Code Style | ruff | Industry-standard formatting |
| CI Status | ✅ | All quality gates passing |

### Automated Quality Gates

The production pipeline enforces 12 concurrent quality checks:

| Gate | Tool | Status |
|------|------|--------|
| **Syntax & Style** | Ruff | ✅ Required |
| **Formatting** | Ruff Format | ✅ Required |
| **Type Safety** | Mypy | ✅ Required |
| **Unit Tests** | pytest | ✅ Required |
| **Coverage** | pytest-cov | ✅ Required (≥50%) |
| **Security Scan** | Bandit | ✅ Required |
| **Vulnerability Audit** | Safety | ✅ Required |
| **Doc Validation** | doc8 | ✅ Required |
| **Import Organization** | isort | ✅ Required |
| **Dependency Audit** | pip-audit | ✅ Required |
| **Code Complexity** | pylint | ✅ Recommended |
| **Memory Safety** | valgrind | ✅ Optional |

### Maturity Indicators

- **Versioning**: Semantic Versioning (SemVer)
- **Release Cadence**: Automated on tag push
- **Changelog**: Automatic generation from commit messages
- **Deprecation Policy**: 6-month deprecation notice
- **Backwards Compatibility**: Strictly maintained
- **Documentation**: Versioned and auto-generated
- **Support**: Active community and enterprise support

## Development

### Setup Development Environment

```bash
# Clone repository
git clone https://github.com/JonasHeinickeBio/HeLLMholtz.git
cd HeLLMholtz

# Install with development dependencies
poetry install --with dev

# Install pre-commit hooks
poetry run pre-commit install
```

### Running Tests

```bash
# Run all tests
poetry run pytest

# Run with coverage
poetry run pytest --cov=hellmholtz --cov-report=html

# Run specific test categories
poetry run pytest -m "slow"        # Slow integration tests
poetry run pytest -m "network"     # Tests requiring network access
poetry run pytest -m "model"       # Tests using actual models
```

### Code Quality

HeLLMholtz maintains high code quality standards through comprehensive automated checks:

#### Quality Batches

```bash
# Execute all quality checks
poetry run ruff check .
poetry run ruff format . --check
poetry run mypy src/
poetry run pytest --cov=hellmholtz --cov-report=term-missing --cov-fail-under=50
```

#### Additional Quality Checks

```bash
# Security vulnerability scanning
poetry run bandit -r src/ -ll
poetry run safety check

# Documentation validation
poetry run doc8 docs/
poetry run pydocstyle src/

# Import organization
poetry run isort --check .

# Dependency auditing
poetry run pip-audit
```

#### CI/CD Quality Gates

The GitHub Actions pipeline enforces eight quality gates before deployment:

| Gate | Tool | Purpose |
|------|------|---------|
| Linting | Ruff | Detect code style issues and bugs |
| Formatting | Ruff | Ensure consistent code formatting |
| Type Checking | Mypy | Validate static type annotations |
| Unit Tests | pytest | Execute test suite with coverage |
| Security | Bandit/Safety | Scan for vulnerabilities |
| Documentation | doc8/pydocstyle | Validate docs quality |
| Imports | isort | Organize import statements |
| Dependencies | pip-audit | Audit package vulnerabilities |

All quality gates must pass before packages are built or published.

### Production Readiness

- **Continuous Integration**: Automated testing on every commit
- **Continuous Deployment**: Automated publishing to PyPI
- **Version Tagging**: Semantic versioning with automated changelog
- **Artifact Management**: Signed and verified package distributions
- **Rollback Support**: Version rollback capability
- **Monitoring**: Production deployment monitoring and alerting

## Development

### Setup Development Environment

```bash
# Clone repository
git clone https://github.com/JonasHeinickeBio/HeLLMholtz.git
cd HeLLMholtz

# Install with development dependencies
poetry install --with dev

# Install pre-commit hooks
poetry run pre-commit install
```

### Running Tests

```bash
# Run all tests
poetry run pytest

# Run with coverage
poetry run pytest --cov=hellmholtz --cov-report=html

# Run specific test categories
poetry run pytest -m "slow"        # Slow integration tests
poetry run pytest -m "network"     # Tests requiring network access
poetry run pytest -m "model"       # Tests using actual models
```

### Building Documentation

```bash
# Generate API documentation
poetry run sphinx-build docs/ docs/_build/

# Serve documentation locally
poetry run sphinx-serve docs/_build/
```

## Contributing

We welcome contributions! Please see our [Contributing Guide](CONTRIBUTING.md) for details:

1. Fork the repository
2. Create a feature branch: `git checkout -b feature/amazing-feature`
3. Make your changes and add tests
4. Run the full test suite with quality checks
5. Commit your changes following conventional commits
6. Push to the branch: `git push origin feature/amazing-feature`
7. Open a Pull Request

All contributions must pass the CI/CD quality gates before merging.

## License

This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.

## Acknowledgments

- Built on top of [aisuite](https://github.com/andrewyng/aisuite) for unified LLM access
- LLM evaluation powered by [EleutherAI's LM Evaluation Harness](https://github.com/EleutherAI/lm-evaluation-harness)
- Proxy functionality via [LiteLLM](https://github.com/BerriAI/litellm)
- Special thanks to the Helmholtz Association for Blablador model access

## Support

- Documentation: https://hellmholtz.readthedocs.io/
- Issue Tracker: https://github.com/JonasHeinickeBio/HeLLMholtz/issues
- Discussions: https://github.com/JonasHeinickeBio/HeLLMholtz/discussions
- Commercial Support: Available for enterprise deployments

---

<p align="center">Made with love for the scientific computing community</p>
- Discussions: https://github.com/JonasHeinickeBio/HeLLMholtz/discussions

---

<p align="center">Made with love for the scientific computing community</p>

