Metadata-Version: 2.4
Name: pathvlm-litebench
Version: 0.12.0
Summary: A lightweight toolkit for benchmarking and visualizing vision-language models in computational pathology.
Author: PathVLM-LiteBench Contributors
License: MIT
Keywords: computational pathology,vision-language model,CLIP,retrieval,zero-shot classification,prompt sensitivity,medical AI
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Science/Research
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Scientific/Engineering :: Medical Science Apps.
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: numpy
Requires-Dist: pandas
Requires-Dist: pillow
Requires-Dist: matplotlib
Requires-Dist: scikit-learn
Requires-Dist: torch
Requires-Dist: transformers<5,>=4.30
Requires-Dist: tqdm
Provides-Extra: dev
Requires-Dist: pytest; extra == "dev"
Requires-Dist: jupyter; extra == "dev"
Dynamic: license-file

# PathVLM-LiteBench

[![CI](https://github.com/GuoYixuan0130/PathVLM-LiteBench/actions/workflows/ci.yml/badge.svg)](https://github.com/GuoYixuan0130/PathVLM-LiteBench/actions/workflows/ci.yml)
[![PyPI](https://img.shields.io/pypi/v/pathvlm-litebench.svg)](https://pypi.org/project/pathvlm-litebench/)

PathVLM-LiteBench is a lightweight, CPU-compatible and laptop-GPU accelerated toolkit for benchmarking and visualizing vision-language models in computational pathology under limited computing resources.

This project focuses on patch-level pathology image-text retrieval, zero-shot classification, prompt sensitivity analysis, embedding caching, and visualization reports using frozen CLIP/PLIP-style vision-language models, with CUDA acceleration when available.

<p align="center">
  <img src="docs/assets/retrieval_hero.png" alt="Zero-shot text-to-patch retrieval with PLIP: top-4 colorectal histology patches retrieved for the prompts 'adipose tissue', 'lymphocytes', and 'colorectal adenocarcinoma epithelium'." width="780">
</p>

<p align="center"><sub>
Zero-shot text&rarr;patch retrieval with a <strong>frozen</strong> PLIP encoder &mdash; no fine-tuning &mdash; reproducible with <code>pathvlm-litebench demo retrieval --model plip</code>.<br>
Patches: NCT-CRC-HE-100K / CRC-VAL-HE-7K by Kather, J. N. <em>et al.</em>, via
<a href="https://doi.org/10.5281/zenodo.1214456">Zenodo (DOI 10.5281/zenodo.1214456)</a>,
licensed under <a href="https://creativecommons.org/licenses/by/4.0/">CC BY 4.0</a>.
Individual patches are unmodified; they are only arranged into a retrieval grid.
</sub></p>

## TL;DR Quickstart

Install the toolkit and CLI straight from PyPI (CPU works out of the box; see the
PyTorch note for CUDA):

```bash
pip install pathvlm-litebench
pathvlm-litebench --help        # list all commands
```

This gives you the `pathvlm-litebench` command and the importable library. To run
the **bundled demos**, clone the repo instead — the example scripts ship with the
source, not the PyPI wheel:

```bash
# 1. Clone and install in editable mode
git clone https://github.com/GuoYixuan0130/PathVLM-LiteBench.git
cd PathVLM-LiteBench
python -m venv .venv && source .venv/bin/activate   # Windows: .venv\Scripts\activate
pip install -e .

# 2. Run a demo (downloads CLIP weights on first run)
pathvlm-litebench demo retrieval --model clip --device auto

# 3. List everything else you can do
pathvlm-litebench demo          # list runnable demos
```

The first demo uses generated RGB images as a smoke test, so it runs on a laptop
without a real dataset. Point it at your own patches with `--image_dir
path/to/your_patch_folder`. The rest of this README covers datasets, configs, and
reports in detail.

## Motivation

Recent computational pathology foundation models and vision-language models have shown strong potential in histopathology image understanding. However, reproducing large-scale pretraining is computationally expensive and often infeasible for students or small research groups.

PathVLM-LiteBench aims to provide a low-compute and reproducible toolkit for evaluating and comparing pathology vision-language models using frozen models and lightweight utilities.

Instead of training large models from scratch, this project focuses on:

- Frozen vision-language model inference
- Patch-level image-text retrieval
- Zero-shot classification
- Prompt sensitivity analysis
- Embedding caching
- Visualization and HTML reporting
- CPU-compatible smoke tests and laptop-friendly experimentation
- CUDA-accelerated inference on consumer-grade laptop GPUs when available

The project does not require A100 GPUs, multi-GPU training, or large-scale VLM pretraining.

## Current Stage

The current version supports a minimal but complete patch-level workflow:

- Loading patch images from a folder
- Encoding images with CLIP-style models
- Encoding text prompts with CLIP-style models
- Computing image-text similarity
- Retrieving top-k images for each prompt
- Saving top-k visualization grids
- Generating HTML retrieval reports
- Caching image embeddings
- Running zero-shot classification
- Running prompt sensitivity analysis
- Rendering patch-coordinate heatmaps from existing score artifacts
- Scoring coordinate-aware patch manifests against one text prompt
- Scoring coordinate-aware patch manifests against prompt sets
- Comparing saved prompt-scored patch-coordinate score artifacts

The current demos use simple RGB images for smoke testing. These demo images are not pathology images.

## Model Registry

PathVLM-LiteBench uses a lightweight model registry so that demos can accept either a short model key or a Hugging Face model name.

In CLI commands, `--model` can be either:

- a registered model key, such as `clip`, `plip`, or `conch`
- a full Hugging Face model name, such as `openai/clip-vit-base-patch32`

The model key is the short name users type in PathVLM-LiteBench commands. The resolved model name is the actual model identifier used by the loader. For example, both `clip` and `clip-vit-base-patch32` resolve to `openai/clip-vit-base-patch32`.

Currently supported model keys:

| Model key | Resolved model name | Status | Notes |
|---|---|---|---|
| `clip` | `openai/clip-vit-base-patch32` | Implemented | Default CLIP baseline |
| `clip-vit-base-patch32` | `openai/clip-vit-base-patch32` | Implemented | Alias for CLIP ViT-B/32 |
| `plip` | `vinid/plip` | Implemented | Pathology-specific CLIP-compatible PLIP wrapper |
| `conch` | `MahmoodLab/CONCH` | Optional | CONCH wrapper; requires gated Hugging Face access and the optional CONCH package |

You can run demos with a registered model key:

```bash
python examples/01_patch_text_retrieval_demo.py --model clip
```

or with a Hugging Face model name:

```bash
python examples/01_patch_text_retrieval_demo.py --model openai/clip-vit-base-patch32
```

PLIP is supported through a CLIP-compatible Hugging Face wrapper. CONCH is supported through the official `mahmoodlab/CONCH` package and requires gated Hugging Face access. If the optional package or authentication is missing, `--model conch` raises a clear setup error.

To use CONCH locally, first install its optional package and authenticate with Hugging Face:

```bash
pip install git+https://github.com/Mahmoodlab/CONCH.git
hf auth login
```

## Device Support

PathVLM-LiteBench supports explicit device selection for model-based demos:

```bash
python examples/01_patch_text_retrieval_demo.py --model clip --device auto
python examples/01_patch_text_retrieval_demo.py --model clip --device cuda
python examples/01_patch_text_retrieval_demo.py --model clip --device cpu
```

`auto` is the default mode. It uses CUDA if available and falls back to CPU otherwise.

This design keeps the toolkit compatible with CPU-only environments while allowing faster patch embedding on consumer-grade laptop GPUs.

The retrieval-metrics demo (`examples/04_retrieval_metrics_demo.py`) does not use a model and therefore does not need a `--device` argument.

## Prompt Templates

PathVLM-LiteBench includes a small built-in prompt template library for common pathology concepts such as `tumor`, `normal`, `necrosis`, `inflammation`, `stroma`, `lymphocyte`, and `gland`.

These templates are intended for lightweight experimentation with zero-shot classification and prompt sensitivity analysis.

Example:

```python
from pathvlm_litebench.prompts import get_prompt_variants, build_class_prompts

tumor_prompts = get_prompt_variants("tumor")
class_prompts = build_class_prompts(["tumor", "normal", "necrosis"])
```

The example above produces:

```python
tumor_prompts == [
    "a histopathology image of tumor tissue",
    "a pathology patch showing malignant tissue",
    "a microscopic image of cancerous tissue",
    "a H&E stained tissue patch with tumor region",
]

class_prompts == [
    "a histopathology image of tumor",
    "a histopathology image of normal",
    "a histopathology image of necrosis",
]
```

`get_prompt_variants("tumor")` returns several alternative text prompts for the same concept. This is useful when you want to test whether retrieval results change when the wording changes.

`build_class_prompts(["tumor", "normal", "necrosis"])` returns one default prompt per class. This is useful for zero-shot classification, where each patch image is compared against one text prompt for each candidate class.

The prompt library is not a clinical ontology. It is a lightweight research utility for reproducible prompt experiments.

Zero-shot with built-in class prompt generation:

```bash
python examples/02_zero_shot_classification_demo.py --model clip --device auto --class_names tumor normal necrosis --top_k 2
```

Here, `--class_names tumor normal necrosis` defines the candidate labels. Because no `--class_prompts` are provided, the demo automatically builds one prompt per class with the default template `a histopathology image of {class_name}`. `--top_k 2` prints the two highest-scoring class predictions per patch.

Prompt sensitivity with pathology prompt templates:

```bash
python examples/03_prompt_sensitivity_demo.py --model clip --device auto --use_pathology_prompts --concepts tumor normal necrosis --top_k 2
```

Here, `--use_pathology_prompts` tells the demo to use the built-in prompt variants instead of a single custom prompt list. `--concepts tumor normal necrosis` selects which built-in concept groups to test. For each concept, the demo retrieves the top-k patches for several prompt variants, then compares whether those variants return similar patches.

If no real pathology patch folder is passed, the demos still use generated RGB images as smoke tests. For meaningful CPath experiments, provide `--image_dir path/to/your_patch_folder`.

## Benchmark Configuration

PathVLM-LiteBench supports lightweight JSON configuration files for reproducible experiments.

Example retrieval config:

```text
configs/retrieval_demo_config.json
```

Run retrieval demo with config:

```bash
python examples/01_patch_text_retrieval_demo.py --config configs/retrieval_demo_config.json
```

JSON config can store model, device, image_dir, prompts, top_k, cache, visualization, and report settings. Command-line arguments can override config values.

Zero-shot classification also supports config-driven runs:

```bash
python examples/02_zero_shot_classification_demo.py --config configs/zero_shot_demo_config.json
```

Prompt sensitivity analysis also supports config-driven runs:

```bash
python examples/03_prompt_sensitivity_demo.py --config configs/prompt_sensitivity_demo_config.json
```

Example config:

```json
{
  "task": "prompt_sensitivity",
  "model": "clip",
  "device": "auto",
  "image_dir": "dataset/MHIST/images",
  "top_k": 5,
  "use_pathology_prompts": true,
  "concepts": ["tumor", "normal", "necrosis"],
  "save_report": true,
  "report_dir": "outputs/prompt_sensitivity_demo"
}
```

The `dataset/` folder is local and ignored by Git. The config file is a template and does not include data.

Manifest-specific retrieval evaluation options such as `--manifest`, `--image_root`, `--split`, `--label_prompts`, and `--recall_k` are currently passed via CLI.

Common option meanings:

| Option | Meaning |
|---|---|
| `--image_dir` | Folder of patch images for unlabeled runs. |
| `--manifest` | CSV file listing patch paths and optional labels/metadata. |
| `--image_root` | Root folder used to resolve relative `image_path` values in a manifest. |
| `--split` | Optional manifest subset to run, such as `test`. |
| `--prompts` | Text queries for retrieval. |
| `--label_prompts` | One target label per retrieval prompt, used to compute Recall@K. |
| `--class_names` | Candidate zero-shot class labels. |
| `--class_prompts` | One text prompt per class name. If omitted, default class prompts are generated. |
| `--top_k` | Number of top retrieval results or class predictions to show/save. |
| `--recall_k` | Retrieval metric cutoffs, such as Recall@1, Recall@5, and Recall@10. |
| `--use_pathology_prompts` | Use built-in prompt variants for prompt sensitivity. |
| `--concepts` | Built-in prompt concept groups to evaluate, such as `tumor normal necrosis`. |
| `--dry-run` | Validate and preview a planned run without model inference when supported. |
| `--max-images` | Limit the number of manifest records for a quick smoke run. |

## Patch Manifest Support

PathVLM-LiteBench can also read CSV manifests for real patch datasets.

Example manifest:

```text
image_path,label,split,case_id
patches/patch_001.png,tumor,train,case_001
patches/patch_002.png,normal,train,case_001
patches/patch_003.png,necrosis,test,case_002
```

Example usage:

```python
from pathvlm_litebench.data import load_patch_manifest, records_to_image_paths, get_unique_labels

records = load_patch_manifest("manifest.csv", image_root="path/to/dataset")
image_paths = records_to_image_paths(records)
labels = get_unique_labels(records)
```

The manifest loader does not load images or run models. It only prepares structured metadata for downstream retrieval, classification, and evaluation workflows.

## Manifest Conversion

PathVLM-LiteBench uses a standard patch manifest format:

```csv
image_path,label,split,case_id,slide_id
patch_001.png,tumor,test,case_001,slide_001
patch_002.png,normal,test,case_001,slide_001
```

For command-line demos, users are encouraged to convert dataset-specific annotation files into this standard format.

Example: convert MHIST annotations:

```bash
pathvlm-litebench convert-manifest \
  --preset mhist \
  --input dataset/MHIST/annotations.csv \
  --output dataset/MHIST/manifest.csv \
  --image_root dataset/MHIST/images \
  --require_exists
```

Generic conversion:

```bash
pathvlm-litebench convert-manifest \
  --input annotations.csv \
  --output manifest.csv \
  --path_column "Image Name" \
  --label_column "Majority Vote Label" \
  --split_column "Partition"
```

The local `dataset/` folder is intended for private local datasets and is ignored by Git.

## Manifest Sampling

For quick low-compute experiments, users can create a smaller balanced manifest:

```bash
pathvlm-litebench sample-manifest \
  --input dataset/MHIST/manifest.csv \
  --output dataset/MHIST/manifest_test_50_per_class.csv \
  --split test \
  --samples_per_label 50 \
  --seed 42
```

This does not copy images. It only creates a smaller CSV manifest.

Then run retrieval or zero-shot evaluation with:

```bash
python examples/01_patch_text_retrieval_demo.py \
  --manifest dataset/MHIST/manifest_test_50_per_class.csv \
  --image_root dataset/MHIST/images \
  --model clip \
  --device auto
```

This is useful for laptop-friendly testing and balanced class evaluation. `dataset/` is local and ignored by Git.

## Repository Structure

```text
PathVLM-LiteBench/
|-- pathvlm_litebench/
|   |-- config/
|   |-- data/
|   |-- evaluation/
|   |-- models/
|   |-- prompts/
|   |-- retrieval/
|   `-- visualization/
|-- examples/
|   |-- 01_quick_start.ipynb
|   |-- 01_patch_text_retrieval_demo.py
|   |-- 02_zero_shot_classification_demo.py
|   |-- 03_prompt_sensitivity_demo.py
|   |-- 04_retrieval_metrics_demo.py
|   `-- 05_patch_coordinate_heatmap_demo.py
|-- configs/
|-- docs/
|-- tests/
|-- README.md
`-- pyproject.toml
```

## Installation

Clone the repository:

```bash
git clone https://github.com/GuoYixuan0130/PathVLM-LiteBench.git
cd PathVLM-LiteBench
```

Create and activate a virtual environment:

```bash
python -m venv .venv

# Windows
.venv\Scripts\activate

# Linux / macOS
source .venv/bin/activate
```

Install the package in editable mode. This installs all dependencies and the
`pathvlm-litebench` command in one step:

```bash
pip install -e .
```

> **PyTorch note:** `pip install -e .` installs a default `torch` build that
> works on CPU. For CUDA acceleration on a laptop GPU, install the matching
> torch wheel for your CUDA version first, following the official selector at
> https://pytorch.org/get-started/locally/, then run `pip install -e .`.

If you only need the libraries without the CLI, `pip install -r requirements.txt`
also works.

## Command-Line Interface

Once installed, you can inspect the toolkit with:

```bash
pathvlm-litebench version
pathvlm-litebench models
pathvlm-litebench demos
pathvlm-litebench demo
pathvlm-litebench convert-manifest --help
pathvlm-litebench sample-manifest --help
pathvlm-litebench summarize-report --help
pathvlm-litebench compare-reports --help
pathvlm-litebench validate-config --help
pathvlm-litebench run-zero-shot-grid --help
pathvlm-litebench render-coordinate-heatmap --help
pathvlm-litebench score-coordinate-heatmap --help
pathvlm-litebench score-coordinate-heatmap-prompt-set --help
pathvlm-litebench compare-coordinate-heatmap-scores --help
```

The inspection and help commands above do not download models. Model-based evaluation or scoring commands load model weights only when those runs are executed.

## Data Preparation

To run PathVLM-LiteBench on real pathology patches, prepare a folder of patch-level images and pass it with `--image_dir`.

For the shortest MHIST-style workflow, see [docs/small_dataset_quickstart.md](docs/small_dataset_quickstart.md).
See [docs/data_preparation.md](docs/data_preparation.md) for details.
For a step-by-step guide on running the toolkit with real pathology patch folders, see [docs/real_patch_workflow.md](docs/real_patch_workflow.md).
For an end-to-end public patch benchmark recipe using conversion, sampling, zero-shot, prompt grids, and report comparison, see [docs/public_patch_benchmark_workflow.md](docs/public_patch_benchmark_workflow.md).
For patch-coordinate heatmaps from existing scores, see [docs/patch_coordinate_heatmap_workflow.md](docs/patch_coordinate_heatmap_workflow.md).
For single-prompt patch-coordinate heatmap scoring, see [docs/prompt_scored_coordinate_heatmap_workflow.md](docs/prompt_scored_coordinate_heatmap_workflow.md).
For prompt-set patch-coordinate heatmap scoring and comparison, see [docs/prompt_set_coordinate_heatmap_workflow.md](docs/prompt_set_coordinate_heatmap_workflow.md).

## Quick Start

### Demo 1: Patch-Text Retrieval

Run the minimal patch-text retrieval demo:

```bash
python examples/01_patch_text_retrieval_demo.py --model clip --device auto
```

If no image folder is provided, the script automatically creates a small demo folder with simple RGB images. These demo images are only used to verify that the pipeline works end-to-end.

### Demo 2: Zero-Shot Classification

Run the zero-shot classification demo:

```bash
python examples/02_zero_shot_classification_demo.py --model clip --device auto
```

Run zero-shot evaluation from a CSV manifest:

```bash
python examples/02_zero_shot_classification_demo.py \
  --manifest path/to/manifest.csv \
  --image_root path/to/dataset_root \
  --model clip \
  --device auto \
  --split test \
  --top_k 3
```

If labels are available in the manifest, the demo reports:
- accuracy
- balanced accuracy
- macro precision / recall / F1
- per-class precision / recall / F1
- confusion matrix

If `class_names` are not provided, they can be inferred from manifest labels. This is intended for lightweight patch-level evaluation, not clinical diagnosis.

Example MHIST-style manifest evaluation command:

```bash
python examples/02_zero_shot_classification_demo.py \
  --manifest dataset/MHIST/manifest.csv \
  --image_root dataset/MHIST/images \
  --model clip \
  --device auto \
  --split test \
  --class_names HP SSA \
  --class_prompts \
    "a histopathology image of hyperplastic polyp" \
    "a histopathology image of sessile serrated adenoma" \
  --top_k 2
```

The current CLIP baseline is not pathology-specific, so low performance is expected and should be interpreted as a baseline.

This demo classifies each patch by comparing its image embedding with class text prompt embeddings.

To save predictions and metrics:

```bash
python examples/02_zero_shot_classification_demo.py \
  --manifest dataset/MHIST/manifest.csv \
  --image_root dataset/MHIST/images \
  --model clip \
  --device auto \
  --split test \
  --class_names HP SSA \
  --class_prompts \
    "a histopathology image of hyperplastic polyp" \
    "a histopathology image of sessile serrated adenoma" \
  --top_k 2 \
  --save_report \
  --report_dir outputs/zero_shot_demo
```

This generates:

- `predictions.csv`
- `errors.csv`
- `metrics.json`

`predictions.csv` stores per-image predictions. `metrics.json` stores aggregate classification metrics when labels are available.

When labels are available, `--save_report` also generates:

- `errors.csv`: misclassified samples only
- prediction distribution summary in `metrics.json`
- true label distribution
- predicted label distribution
- error rate
- optional warning when predictions collapse into one class

This is useful for detecting class bias in general CLIP baselines.

To generate a compact Markdown summary from a saved zero-shot report:

```bash
pathvlm-litebench summarize-report \
  --task zero-shot \
  --report_dir outputs/zero_shot_demo
```

This writes `experiment_summary.md` in the report directory. The summary is generated from saved artifacts only; it does not run model inference or download model weights.

### Demo 3: Prompt Sensitivity Analysis

Run the prompt sensitivity demo:

```bash
python examples/03_prompt_sensitivity_demo.py --model clip --device auto
```

This demo evaluates whether different prompt variants for the same concept retrieve similar top-k image results.

To save prompt sensitivity results:

```bash
python examples/03_prompt_sensitivity_demo.py \
  --model clip \
  --device auto \
  --image_dir dataset/MHIST/images \
  --use_pathology_prompts \
  --concepts tumor normal necrosis \
  --top_k 5 \
  --save_report \
  --report_dir outputs/prompt_sensitivity_demo
```

This generates:

- `prompt_sensitivity_summary.csv`
- `prompt_sensitivity_details.csv`
- `prompt_sensitivity_metrics.json`

The summary CSV stores concept-level stability metrics. The details CSV stores prompt-level top-k retrieved indices and scores. The JSON stores full results and experiment metadata.

To generate a compact Markdown summary from a saved prompt sensitivity report:

```bash
pathvlm-litebench summarize-report \
  --task prompt-sensitivity \
  --report_dir outputs/prompt_sensitivity_demo
```

This writes `experiment_summary.md` in the report directory without rerunning model inference.

Prompt sensitivity analysis can also be configured with JSON:

```bash
python examples/03_prompt_sensitivity_demo.py --config configs/prompt_sensitivity_demo_config.json
```

Command-line arguments can override config values. For meaningful analysis, use real pathology patch images via `--image_dir` or `config.image_dir`.

## Compare Saved Reports

After generating multiple reports for the same task, you can create a compact Markdown comparison without rerunning inference:

```bash
pathvlm-litebench compare-reports \
  --task zero-shot \
  --report_dirs outputs/zero_shot_clip outputs/zero_shot_plip \
  --run_names clip plip \
  --output outputs/zero_shot_comparison.md
```

The same command supports `--task retrieval` and `--task prompt-sensitivity`. The comparison reads saved CSV/JSON artifacts only and writes a local Markdown file. Do not commit generated comparisons if they contain dataset-specific paths or metrics.

## Zero-Shot Prompt Grids

To run a small zero-shot prompt grid from JSON config:

```bash
pathvlm-litebench validate-config \
  configs/zero_shot_prompt_grid_mhist_sample.json
```

```bash
pathvlm-litebench run-zero-shot-grid \
  --config configs/zero_shot_prompt_grid_mhist_sample.json
```

Preview the expanded model/prompt runs without loading models:

```bash
pathvlm-litebench run-zero-shot-grid \
  --config configs/zero_shot_prompt_grid_mhist_sample.json \
  --dry-run
```

Each run writes a normal zero-shot report directory under the configured `output_root`. When `save_comparison` is enabled, the command also writes a zero-shot comparison Markdown file from saved artifacts. Generated reports stay under `outputs/` and are ignored by Git.

For config fields, output layout, and interpretation guidance, see [docs/prompt_grid_workflow.md](docs/prompt_grid_workflow.md).

## Example Commands

Patch-text retrieval with custom prompts:

```bash
python examples/01_patch_text_retrieval_demo.py --model clip --device auto --prompts "a red image" "a blue image" "a black image" --top_k 2
```

Patch-text retrieval on your own patch image folder:

```bash
python examples/01_patch_text_retrieval_demo.py --model clip --device auto --image_dir path/to/your/patches --prompts "tumor region" "normal tissue" --top_k 5
```

Manifest-based patch-text retrieval with Recall@K:

```bash
python examples/01_patch_text_retrieval_demo.py \
  --manifest path/to/manifest.csv \
  --image_root path/to/dataset_root \
  --model clip \
  --device auto \
  --split test \
  --prompts \
    "a histopathology image of tumor tissue" \
    "a histopathology image of normal tissue" \
    "a histopathology image showing necrosis" \
  --label_prompts tumor normal necrosis \
  --recall_k 1 5 10 \
  --top_k 5 \
  --save_html_report
```

`--label_prompts` maps each text prompt to a manifest label. If labels are available, the demo prints text-to-image Recall@K for lightweight patch-level retrieval benchmarking.

When `--label_prompts` is used with a labeled manifest, retrieval results in the terminal and HTML report also include:
- the retrieved patch label
- the target label for each text prompt
- whether each retrieved patch is a positive match
- Recall@K metrics in the terminal

To save structured retrieval outputs:

```bash
python examples/01_patch_text_retrieval_demo.py \
  --manifest dataset/MHIST/manifest.csv \
  --image_root dataset/MHIST/images \
  --model clip \
  --device auto \
  --split test \
  --prompts \
    "a histopathology image of hyperplastic polyp" \
    "a histopathology image of sessile serrated adenoma" \
  --label_prompts HP SSA \
  --recall_k 1 5 10 \
  --top_k 5 \
  --save_html_report \
  --save_report \
  --report_dir outputs/retrieval_demo
```

This generates:

- `retrieval_results.csv`
- `retrieval_metrics.json`
- optionally `retrieval_report.html`

`retrieval_results.csv` stores prompt-level top-k retrieval results. `retrieval_metrics.json` stores Recall@K metrics and experiment metadata when labels are available. Outputs are ignored by Git.

To generate a compact Markdown summary from a saved retrieval report:

```bash
pathvlm-litebench summarize-report \
  --task retrieval \
  --report_dir outputs/retrieval_demo
```

This writes `experiment_summary.md` in the report directory without rerunning model inference.

Save top-k visualization grids:

```bash
python examples/01_patch_text_retrieval_demo.py --model clip --device auto --save_visualization
```

Generate an HTML retrieval report:

```bash
python examples/01_patch_text_retrieval_demo.py --model clip --device auto --save_html_report
```

Use image embedding cache:

```bash
python examples/01_patch_text_retrieval_demo.py --model clip --device auto --use_cache
```

Combine cache, visualization, and HTML report:

```bash
python examples/01_patch_text_retrieval_demo.py --model clip --device auto --use_cache --save_visualization --save_html_report --top_k 3
```

Zero-shot classification with custom pathology-style class prompts:

```bash
python examples/02_zero_shot_classification_demo.py --model clip --device auto --class_names tumor normal necrosis --class_prompts "a histopathology image of tumor tissue" "a histopathology image of normal tissue" "a histopathology image showing necrosis" --top_k 2
```

Zero-shot classification with built-in class prompt generation:

```bash
python examples/02_zero_shot_classification_demo.py --model clip --device auto --class_names tumor normal necrosis --top_k 2
```

Prompt sensitivity with a different top-k value:

```bash
python examples/03_prompt_sensitivity_demo.py --model clip --device auto --top_k 2
```

Prompt sensitivity with pathology prompt templates:

```bash
python examples/03_prompt_sensitivity_demo.py --model clip --device auto --use_pathology_prompts --concepts tumor normal necrosis --top_k 2
```

## Example Output

The retrieval demo prints top-k results for each text prompt:

```text
========== Retrieval Results ==========

Prompt: a red image
  Top 1: index=0, score=0.2841, path=examples/demo_patches/patch_red.png
  Top 2: index=3, score=0.2317, path=examples/demo_patches/patch_green.png
```

The zero-shot demo prints predicted labels:

```text
========== Zero-Shot Classification Results ==========

Image: examples/demo_patches/patch_red.png
Predicted: red (confidence=0.3012)
Top predictions:
  Top 1: class=red, probability=0.3012, logit=0.2841
```

The prompt sensitivity demo prints concept-level stability metrics:

```text
========== Prompt Sensitivity Results ==========

Concept: red_like
Number of prompts: 3
Mean top-k overlap: 0.6667
Mean similarity std: 0.0312
```

Exact scores may vary depending on model version and runtime environment.

## Outputs

Generated outputs are saved under `outputs/` by default and are ignored by Git.

Typical outputs include:

```text
outputs/
|-- cache/
|   |-- image_embeddings.pt
|   `-- image_paths.json
`-- retrieval_demo/
    |-- retrieval_report.html
    `-- topk_prompt_*.png
```

The automatically generated demo images are saved under:

```text
examples/demo_patches/
```

This folder is also ignored by Git.

## Low-Compute Design

This project is designed for laptop-friendly experimentation.

Current design choices:

- Use frozen CLIP-style models instead of training large models
- Start from patch-level images instead of full whole-slide image processing
- Avoid heavy dependencies such as OpenSlide and TIAToolbox in the early stage
- Cache image embeddings to reduce repeated computation
- Keep demos simple, reproducible, and easy to inspect
- Provide terminal outputs, image grids, and HTML reports
- Run smoke tests on CPU-only environments
- Accelerate model inference with CUDA on consumer-grade laptop GPUs when available
- Use default `--device auto` mode to choose CUDA when available and otherwise fall back to CPU

## Current Limitations

- CLIP remains the default baseline model. PLIP is available as a pathology-specific CLIP-compatible model key.
- CONCH requires gated Hugging Face access and the optional `mahmoodlab/CONCH` package.
- The built-in demo images are not pathology images.
- Whole-slide image processing is not supported in the current version.
- No large-scale benchmark dataset is included.
- Prompt sensitivity analysis currently focuses on retrieval stability rather than clinical validity.

Future milestones may add more packaged patch-level benchmark examples, but the current release remains patch-level and local-first.

## Roadmap

- [x] Build basic project structure
- [x] Implement a CLIP model wrapper
- [x] Support patch image loading
- [x] Implement image-text retrieval
- [x] Add minimal command-line retrieval demo
- [x] Add top-k image visualization
- [x] Add embedding caching
- [x] Add HTML retrieval report
- [x] Add zero-shot classification utility
- [x] Add zero-shot classification demo
- [x] Add prompt sensitivity analysis utility
- [x] Add prompt sensitivity demo
- [x] Add lightweight model registry
- [x] Add manifest conversion and balanced sampling utilities
- [x] Add retrieval metrics such as Recall@K
- [x] Add classification metrics beyond accuracy
- [x] Add config-driven retrieval, zero-shot, and prompt sensitivity demos
- [x] Add pathology-specific PLIP wrapper
- [x] Add zero-shot, retrieval, and prompt sensitivity experiment summary Markdown utilities
- [x] Add multi-run comparison summaries for saved report artifacts
- [x] Implement optional CONCH wrapper
- [x] Add zero-shot prompt-grid runner
- [x] Document public patch benchmark workflow
- [x] Add benchmark config validation CLI
- [x] Add patch-coordinate heatmap rendering from existing score artifacts
- [x] Add single-prompt patch-coordinate heatmap scoring and config workflow
- [x] Add artifact-only score comparison for prompt-scored coordinate heatmaps
- [x] Add prompt-set patch-coordinate heatmap scoring workflow
- [x] Keep public documentation focused on user workflows

Related docs:

- Getting started: [small dataset quickstart](docs/small_dataset_quickstart.md), [real patch workflow](docs/real_patch_workflow.md), [data preparation](docs/data_preparation.md)
- Benchmark workflows: [public patch benchmark workflow](docs/public_patch_benchmark_workflow.md), [prompt-grid workflow](docs/prompt_grid_workflow.md)
- Coordinate heatmaps: [artifact-only heatmap workflow](docs/patch_coordinate_heatmap_workflow.md), [single-prompt scoring workflow](docs/prompt_scored_coordinate_heatmap_workflow.md), [prompt-set workflow](docs/prompt_set_coordinate_heatmap_workflow.md)
- Project and release: [project positioning](docs/project_positioning.md), [v0.11.0 release notes](docs/release_notes_v0.11.0.md)

## Research Positioning

PathVLM-LiteBench is intended as a lightweight research engineering project for computational pathology and medical vision-language model evaluation.

The project does not aim to reproduce large-scale foundation model pretraining. Instead, it focuses on reproducible low-compute evaluation workflows that can help students and small research groups analyze pathology vision-language models.

For a more detailed discussion of the research motivation and scope, see [docs/project_positioning.md](docs/project_positioning.md).

## Contributing

Contributions are welcome. Please see [CONTRIBUTING.md](CONTRIBUTING.md) for development principles, testing instructions, and contribution guidelines.

## License

This project is released for academic and research purposes.
