Metadata-Version: 2.5
Name: astro-eval
Version: 0.1.1
Summary: A reusable evaluation framework for domain-specific language models.
Project-URL: Homepage, https://github.com/aashutoshkumarbhardwaj/AstroGPT
Project-URL: Repository, https://github.com/aashutoshkumarbhardwaj/AstroGPT
Project-URL: Issues, https://github.com/aashutoshkumarbhardwaj/AstroGPT/issues
Requires-Python: >=3.12
Requires-Dist: numpy>=2.5.2
Requires-Dist: tokenizers>=0.23.1
Requires-Dist: torch>=2.13.0
Description-Content-Type: text/markdown

<p align="center">
  <img src="https://raw.githubusercontent.com/aashutoshkumarbhardwaj/AstroGPT/master/src/astro_eval/logo.jpg" alt="Astro-Eval Logo" width="400"/>
</p>

# Astro-Eval

A reusable evaluation framework for language models.

Evaluate models with standardized benchmarks, generation metrics, factuality analysis, error analysis, experiment tracking, model comparison, and reproducible reports.

[![PyPI Version](https://img.shields.io/pypi/v/astro-eval.svg)](https://pypi.org/project/astro-eval/)
[![Python Version](https://img.shields.io/badge/python-3.12%20%7C%203.13-blue.svg)](https://pypi.org/project/astro-eval/)
[![License](https://img.shields.io/badge/license-MIT-green.svg)](https://github.com/aashutoshkumarbhardwaj/AstroGPT/blob/master/LICENSE)

```bash
pip install astro-eval
```

---

## Why Astro-Eval?

Evaluating an LLM should not mean maintaining a collection of one-off scripts. 

Astro-Eval provides a reusable evaluation layer that separates model execution from evaluation logic.

```text
                    Model
                      │
                      ▼
                 Astro-Eval
                      │
        ┌─────────────┼─────────────┐
        ▼             ▼             ▼
    Benchmarks     Metrics      Factuality
        │             │             │
        └─────────────┼─────────────┘
                      ▼
               Error Analysis
                      │
                      ▼
              Reports / Tracking
```

---

## Features

- **Model Adapters**: Decouple model loading and generation implementations from the evaluation loop.
- **Reusable Benchmarks**: Standardized schema for loading, validating, and iterating domain-specific prompts.
- **Generation Metrics**: Automatic characterization of output length and processing throughput (tokens/sec).
- **Diversity & Repetition**: Built-in distinct-N diversity metrics and repetition rate analysis.
- **Scientific/Domain Metrics**: Specialized metrics such as scientific term checking, prompt overlap, and composite quality scores.
- **Factuality Scoring**: Detailed factuality extraction, concept coverage, and reference overlap checks.
- **Error Analysis**: Automated classification of generation errors (PASS, PARTIAL, FAIL).
- **Model Comparison**: Multi-model comparison leaderboard generator with automated winner determinations.
- **Experiment Tracking**: Saves reproducible run records containing hyperparameters, configurations, and aggregate scores.
- **Markdown Reports**: Exports full evaluation reports detailing overview, generation, domain quality, and factuality.

---

## Quick Start

### 1. Installation
```bash
pip install astro-eval
```

### 2. Run an Evaluation
```bash
astro-eval run \
  --model astrogpt \
  --checkpoint checkpoints_50m/best.pt \
  --benchmark astronomy
```

### 3. List Available Benchmarks
```bash
astro-eval benchmarks list
```

### 4. List Registered Model Adapters
```bash
astro-eval adapters list
```

### 5. Generate a Markdown Report
```bash
astro-eval report \
  --input astro_eval_results/best_evaluation.json \
  --output report.md
```

---

## Python Developer API

Integrate evaluation directly into your development or training pipelines:

```python
from astro_eval import (
    Evaluator,
    load_benchmark,
    get_adapter,
)

# 1. Load benchmark prompts
benchmark = load_benchmark("astronomy")

# 2. Get and load model adapter
model = get_adapter("astrogpt")
model.load("checkpoints_50m/best.pt")

# 3. Initialize Evaluator
evaluator = Evaluator(
    model=model,
    benchmark=benchmark,
)

# 4. Run evaluation
results = evaluator.evaluate(
    max_new_tokens=150,
    temperature=0.9,
    top_k=100,
)

# 5. Output summary metrics
print(results.summary())
```

---

## Model Adapters

Astro-Eval separates model-specific code from evaluation logic. 

An adapter implements:
- `load(checkpoint_path)`: Loads and compiles the model weight parameters onto the device.
- `generate(prompt, max_new_tokens, temperature, top_k)`: Runs generation inference and returns structured text details.
- `evaluate_loss(x, y)`: Computes forward validation loss over tokens.
- `metadata()`: Returns dictionary of checkpoint iteration and config.

The framework currently includes:
- **AstroGPT**: Adapter implementation for the AstroGPT astronomical model.

The adapter architecture allows additional model implementations to be added without changing the evaluation engine.

---

## Architecture

```text
src/astro_eval/
├── adapters/
│   ├── base.py
│   ├── registry.py
│   └── astrogpt.py
│
├── benchmarks/
│   ├── astronomy.json
│   ├── astrophysics.json
│   ├── cosmology.json
│   ├── general_science.json
│   ├── physics.json
│   └── prompts.json
│
├── metric/
│   ├── builtin.py
│   ├── defaults.py
│   └── registry.py
│
├── schema/
│   └── result.py
│
├── evaluator.py
├── factuality.py
├── error_analysis.py
├── comparison.py
├── experiments.py
├── report.py
└── cli.py
```

---

## Example Evaluation

Astro-Eval was initially developed and validated using AstroGPT, an astronomy-focused language model.

Example evaluation metrics:

| Metric | Result |
|---|---:|
| Validation Loss | 3.3323 |
| Distinct-1 | 0.9179 |
| Distinct-2 | 0.9952 |
| Distinct-3 | 1.0000 |
| Repetition Rate | 0.0000 |
| Tokens/sec | 104.22 |

The framework also produces factuality and error-analysis outputs for benchmark-level evaluation.

---

## Design Philosophy

Astro-Eval is designed around four principles:
1. **Model-agnostic evaluation**: Run standard benchmarks across any LLM architecture.
2. **Reproducible experiments**: Track runs with unique IDs to easily compare iterations.
3. **Extensible metrics and adapters**: Easily register custom metrics or model APIs.
4. **Machine-readable results**: Save evaluation outputs in structured, versioned JSON schemas.

---

## Roadmap

- [x] Model adapter system
- [x] Benchmark system
- [x] Generation metrics
- [x] Factuality evaluation
- [x] Error analysis
- [x] Model comparison
- [x] Experiment tracking
- [x] Markdown reporting
- [x] Python API
- [x] CLI
- [ ] More model adapters
- [ ] More public benchmarks
- [ ] Statistical significance testing
- [ ] HTML evaluation dashboards

---

## Contributing

Contributions are welcome. You can contribute by:
- Adding a model adapter
- Adding a benchmark
- Adding a metric
- Improving evaluation methods
- Improving documentation
- Reporting bugs

Open an issue or submit a pull request on GitHub.
