Metadata-Version: 2.4
Name: iaa-od
Version: 4.1.1
Summary: A Python package for analysing object detection datasets through Inter-Annotatator Agreement (IAA) metrics.
Project-URL: Homepage, https://gitlab.com/MrVideo/iaa.py
Author-email: Mario Merlo <mario.merlo@polimi.it>
License-Expression: MIT
License-File: LICENSE
Keywords: annotation quality,dataset analysis,inter-annotator agreement,object detection
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.10
Requires-Dist: matplotlib>=3.10.6
Requires-Dist: mplcursors>=0.7
Requires-Dist: numpy>=2.2.6
Requires-Dist: scikit-learn>=1.7.2
Requires-Dist: scipy>=1.15.3
Provides-Extra: dev
Requires-Dist: basedpyright>=1.32.1; extra == 'dev'
Requires-Dist: jupyter-lsp>=2.3.0; extra == 'dev'
Requires-Dist: jupyterlab-lsp>=5.2.0; extra == 'dev'
Requires-Dist: jupyterlab>=4.4.10; extra == 'dev'
Requires-Dist: jupytext>=1.18.1; extra == 'dev'
Description-Content-Type: text/markdown

# `iaa-od`: a comprehensive IAA analysis suite

This package provides a comprehensive Inter-Annotator Agreement analysis suite for Object Detection tasks.

## Installation

You can install the `iaa-od` package by cloning this repository and then running the following command on its top level:

```
pip install .
```

### Development installation

If you want to make modifications to the package, you may install it like so:

```
pip install -e .
```

If you wish to do some testing through Jupyter Notebooks, install the optional dependencies as well:

```
pip install -e ".[dev]"
```

## Computed metrics

Below is a list of currently implemented metrics.

### Krippendorff's $\alpha$

Krippendorff's $\alpha$ coefficient is defined as:

$$\alpha = 1 - \frac{D_o}{D_e}$$

and is computed in our package according to [this paper by Klaus Krippendorff](https://repository.upenn.edu/handle/20.500.14332/2089).

The *units* in each image are found through agglomerative clustering.

### Rater vitality

[Rater vitality](10.1109/SocialSens.2018.00018) measures the contribution of each annotator to the global agreement value. It is computed as:

$$\nu_a = \alpha - \alpha_a$$

where $\alpha_a$ is the value of Krippendorff's $\alpha$ computed excluding annotator $a$ from the dataset.

### Scale complexity

[Scale complexity](10.1109/SocialSens.2018.00018) measures the ambiguity found across categories of the same dataset by grouping them into user-defined macro-categories. It is computed as:

$$\text{SC} = \alpha_M - \alpha$$

where $\alpha_M$ is the value of Krippendorff's $\alpha$ computed on the macro-categories. If $\text{SC} > 0$, categories grouped in the same macro-categories are ambiguous.

### L-Score (Localisation Score)

L-Score measures the localisation agreement reached by the annotators by only considering the locations of their bounding boxes. It is computed as:

$$L_{\text{score}} = \frac{|A|}{2|A|-1} \left[ \sum_{i=2}^{|A|} \left(\frac{i-1}{|A|}\, p_i\right) - p_1 + 1\right]$$

L-Score $\in [0,1]$, where 1 indicates perfect localisation agreement.

### C-Score (Categorisation Score)

C-Score measures the categorisation agreement reached by annotators by analysing intersecting bounding boxes and comparing their labels.

Computation is not reported here for brevity.

C-Score $\in [0,1]$, where 1 indicates perfect categorisation agreement.

### Count and size granularity

Count and size granularity measure the behaviour of annotators when annotating large groups of object:

- an annotator who groups many instances into a single bounding box annotates "by group";
- an annotator who creates several bounding boxes for each instance annotates "by instance".

Computation of these metrics is not reported here for brevity.

Count and size granularities both rely on a benchmark annotator, found during computation, to whom all other annotators are compared to.

## Working example

`main.py` provides a working example of the package by performing all available analyses on a dataset with three annotators. More examples are also provided in the `experiments` directory: specifically, `experiments/images.py` shows how to utilise the `visualisation` module.

## Jupyter notebooks

The `notebooks` directory contains a number of Jupyter notebooks to play around with data.
