Metadata-Version: 2.4
Name: unicorn_eval
Version: 1.1.0
Summary: Evaluation and adaption method for the UNICORN Challenge
Home-page: https://github.com/DIAGNijmegen/unicorn_eval
Author: Joeran Bosma, Clément Grisi, Marina D`Amato, Luc Builtjes, Lena Philipp, Fennie van der Graaf, Judith Lefkes, Michelle Stegeman, Rianne Weber
Author-email: Clement.Grisi@radboudumc.nl
Project-URL: Bug Tracker, https://github.com/DIAGNijmegen/unicorn_eval/issues
Platform: unix
Platform: linux
Platform: osx
Platform: cygwin
Platform: win32
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: scikit-learn==1.1.1
Requires-Dist: numpy==1.26.1
Requires-Dist: pandas
Requires-Dist: psutil
Requires-Dist: torch==2.0.1
Requires-Dist: torchvision==0.15.2
Requires-Dist: scikit-survival
Requires-Dist: pycocoevalcap
Requires-Dist: transformers==4.28.1
Requires-Dist: bert_score
Requires-Dist: evalutils==0.3.1
Requires-Dist: openslide-python
Requires-Dist: pycm
Provides-Extra: testing
Requires-Dist: pytest>=6.0; extra == "testing"
Requires-Dist: pytest-cov>=2.0; extra == "testing"
Requires-Dist: mypy>=0.910; extra == "testing"
Requires-Dist: flake8>=3.9; extra == "testing"
Requires-Dist: tox>=3.24; extra == "testing"
Dynamic: author-email
Dynamic: description
Dynamic: description-content-type
Dynamic: home-page
Dynamic: license-file
Dynamic: project-url

# 🧪 UNICORN Evaluation Toolkit

Welcome to the official evaluation repository for the [UNICORN Challenge](https://unicorn.grand-challenge.org/) — a benchmark for foundation models in pathology, radiology, and medical language processing. This repository provides the official evaluation code and a library of **adaptors** used to turn frozen features into predictions in **vision tasks**.

[![PyPI version](https://img.shields.io/pypi/v/unicorn-eval)](https://pypi.org/project/unicorn-eval/)

## 🚀 Challenge Overview

The UNICORN Challenge evaluates how well foundation models generalize across multiple modalities with minimal task-specific supervision:

- 🧠 **Language** and **Vision-Language** tasks: your model directly outputs predictions.
- 👁️ **Vision** tasks: your model outputs features. These are then converted to predictions using **adaptors** — lightweight models like k-NN, linear classifiers, or shallow MLPs.

We provide a few built-in adaptors, but you're highly encouraged to propose your own!<br>
We maintain the full list of adaptors available on the [Supported Adaptors](src/unicorn_eval/adaptors/README.md) page.


## 🧩 Contributing a Custom Adaptor

Have a better idea for how to turn features into predictions?

You’re welcome to contribute a custom adaptor! Here's how:

1. Add your adaptor to `src/unicorn_eval/adaptors/`.
2. Inherit from one of the base adaptor classes in [`base.py`](src/unicorn_eval/adaptors/base.py).
3. Open a pull request with:
    - Your adaptor code
    - A short description
    - A **unique name** (we’ll include your **team name** in the adaptor name to ensure you receive credit).

✅ Once accepted, your adaptor becomes selectable at submission time — and your team gets full recognition when it’s used!

> 💡 Keep in mind: we **prioritize originality**. If your adaptor is too similar to an existing one, it may not be accepted — so submit early and make it your own!

## 📦 Adaptors vs. Algorithms: What's the Difference?

In **vision tasks**, submissions consist of:
- A **feature extractor** (your algorithm)
- An **adaptor** (used to turn features into predictions)

You can experiment with different adaptors **on top of the same algorithm** without using up your submission slots.<br>
Want to try a different adaptor? Send us a request by email, we’ll run the new adaptor strategy for you on top of the existing features. Requests should be submitted via email using the provided template (to be shared soon).

In **language** and **vision-language** tasks, the algorithm outputs predictions directly, so no adaptor is needed.

## Summary

| **Modality**         | **What You Submit**                        | **Are Adaptors Used?** | **Submission Limit Applies To** |
|-----------------------|--------------------------------------------|-------------------------|-----------------------------------|
| **Vision**            | Algorithm (feature extractor) + Adaptor   | ✅ Yes                  | Algorithm only                   |
| **Language**          | Algorithm (predictive)                    | ❌ No                   | Algorithm                        |
| **Vision-Language**   | Algorithm (predictive)                    | ❌ No                   | Algorithm                        |
