Metadata-Version: 2.4
Name: crimson-score
Version: 0.2.0
Summary: CRIMSON: A Clinically-Grounded LLM-Based Metric for Generative Radiology Report Evaluation
Author: Mohammed Baharoon, Thibault Heintz, Siavash Raissi, Mahmoud Alabbad, Mona Alhammad, Hassan AlOmaish, Sung Eun Kim, Oishi Banerjee, Pranav Rajpurkar
License-Expression: MIT
Project-URL: Repository, https://github.com/rajpurkarlab/CRIMSON
Project-URL: Paper, https://arxiv.org/abs/2603.06183
Project-URL: Model (HuggingFace), https://huggingface.co/rajpurkarlab/medgemma-4b-it-crimson
Keywords: radiology,report evaluation,medical AI,NLP,LLM
Requires-Python: >=3.10
Description-Content-Type: text/markdown
Requires-Dist: openai>=2.26.0
Requires-Dist: transformers>=5.3.0
Requires-Dist: torch>=2.10.0
Requires-Dist: peft>=0.18.1
Requires-Dist: accelerate>=1.13.0
Requires-Dist: pandas>=3.0.1
Requires-Dist: numpy>=2.4.3
Requires-Dist: scipy>=1.17.1
Requires-Dist: tqdm>=4.67.3
Requires-Dist: pyyaml>=6.0.3
Requires-Dist: pydantic>=2.12.5
Requires-Dist: sentence-transformers>=5.2.3
Requires-Dist: scikit-learn>=1.8.0
Requires-Dist: openpyxl>=3.1.5
Requires-Dist: Pillow>=12.1.1
Provides-Extra: vllm
Requires-Dist: vllm>=0.18.0; extra == "vllm"
Provides-Extra: dev
Requires-Dist: pytest; extra == "dev"
Requires-Dist: twine; extra == "dev"

# CRIMSON: A Clinically-Grounded LLM-Based Metric for Generative Radiology Report Evaluation

[Paper](https://arxiv.org/abs/2603.06183) | [Model (HuggingFace)](https://huggingface.co/rajpurkarlab/medgemma-4b-it-crimson)

CRIMSON is a clinically grounded evaluation framework for chest X-ray report generation that assesses reports based on diagnostic correctness, contextual relevance, and patient safety. Unlike prior metrics, CRIMSON incorporates full clinical context, including patient age, indication, and guideline-based decision rules. CRIMSON evaluates only abnormal findings, excluding normal findings from scoring. The framework categorizes errors into a comprehensive taxonomy covering false findings, missing findings, and eight attribute-level errors (e.g., location, severity, measurement, and diagnostic overinterpretation). Each finding is assigned a clinical significance level (urgent, actionable non-urgent, non-actionable, or expected/benign), based on a guideline developed in collaboration with attending cardiothoracic radiologists, enabling severity-aware weighting that prioritizes clinically consequential mistakes over benign discrepancies. Findings are weighted as 1.0 (urgent), 0.5 (actionable non-urgent), 0.25 (non-actionable), or 0.0 (expected/benign), and attribute errors are weighted as 0.5 (significant) or 0.0 (negligible). After weighting, the framework produces a score in the range of (-1, 1], where 1 represents a perfect report, 0 indicates the report is no more informative than a normal template, and negative scores indicate more weighted errors than correct findings.

## Installation

Step 1: Create and activate a dedicated environment (Conda or Python `venv`):

```bash
# Option 1: Conda
conda create -n crimson python=3.12 -y
conda activate crimson

# Option 2: Python venv
python -m venv crimson
source crimson/bin/activate

python -m pip install --upgrade pip
```

Step 2: Install CRIMSON:

```bash
pip install crimson-score
```

Or install from source:

```bash
git clone https://github.com/rajpurkarlab/CRIMSON.git
cd CRIMSON
pip install -e .
```

## Usage

By default, CRIMSON uses the fine-tuned [MedGemmaCRIMSON model](https://huggingface.co/rajpurkarlab/medgemma-4b-it-crimson) via HuggingFace.

To use OpenAI GPT models instead, set your API key:

```bash
export OPENAI_API_KEY="your-openai-api-key"
```

### Scoring a report pair

```python
from CRIMSON import CRIMSONScore

# Default: uses the HuggingFace MedGemmaCRIMSON model
scorer = CRIMSONScore()
result = scorer.evaluate(
    reference_findings="Cardiomegaly. Small bilateral pleural effusions.",
    predicted_findings="Normal heart size. Small left pleural effusion.",
)

print(f"CRIMSON Score: {result['crimson_score']:.2f}")
print(f"False findings: {result['error_counts']['false_findings']}")
print(f"Missing findings: {result['error_counts']['missing_findings']}")
print(f"Attribute errors: {result['error_counts']['attribute_errors']}")
```

### Scoring with patient context

Providing patient context (age, indication) enables context-dependent clinical significance assignment — for example, missing aortic atherosclerosis in an 82-year-old with routine preoperative evaluation is expected/benign, but in a 25-year-old with chest pain it may be actionable.

```python
result = scorer.evaluate(
    reference_findings="Bibasilar atelectasis. Mild cardiomegaly. Aortic atherosclerosis with vascular calcification.",
    predicted_findings="Bibasilar atelectasis. Mild cardiomegaly.",
    patient_context={
        "Age": "82",
        "Indication": "Routine preoperative evaluation",
    },
)
```

### Batch evaluation

Score multiple report pairs in a single call. For HuggingFace and vLLM backends this runs batched GPU inference, which is significantly faster than calling `evaluate()` in a loop.

```python
results = scorer.evaluate_batch(
    reference_findings_list=[
        "Cardiomegaly. Small bilateral pleural effusions.",
        "No acute cardiopulmonary abnormality.",
        "Right lower lobe pneumonia.",
    ],
    predicted_findings_list=[
        "Normal heart size. Small left pleural effusion.",
        "No acute cardiopulmonary abnormality.",
        "Right lower lobe consolidation concerning for pneumonia.",
    ],
    patient_contexts=[
        {"Age": "72", "Indication": "Shortness of breath"},
        None,
        {"Age": "45", "Indication": "Cough and fever"},
    ],
)

for i, result in enumerate(results):
    print(f"Sample {i}: {result['crimson_score']:.2f}")
```

### Using vLLM (faster inference)

For large-scale evaluation, vLLM provides significantly faster inference via continuous batching:

```bash
pip install crimson-score[vllm]
```

```python
scorer = CRIMSONScore(api="vllm")
result = scorer.evaluate(reference_findings="...", predicted_findings="...")
```

### Using an OpenAI model (requires API key)

```python
scorer = CRIMSONScore(api="openai", model_name="gpt-5.2")
result = scorer.evaluate(reference_findings="...", predicted_findings="...")
```

## Reference

```bibtex
@article{baharoon2026crimson,
  title={CRIMSON: A Clinically-Grounded LLM-Based Metric for Generative Radiology Report Evaluation},
  author={Baharoon, Mohammed and Heintz, Thibault and Raissi, Siavash and Alabbad, Mahmoud and Alhammad, Mona and AlOmaish, Hassan and Kim, Sung Eun and Banerjee, Oishi and Rajpurkar, Pranav},
  journal={arXiv preprint arXiv:2603.06183},
  year={2026}
}
```
