Metadata-Version: 2.4
Name: iaa-kit
Version: 0.1.2
Summary: Inter-annotator agreement metrics and bootstrap confidence intervals for review and labeling workflows.
Author: AuraOne
License-Expression: MIT
Project-URL: Homepage, https://github.com/auraoneai/iaa-kit
Project-URL: Documentation, https://github.com/auraoneai/iaa-kit#readme
Project-URL: Repository, https://github.com/auraoneai/iaa-kit.git
Project-URL: Issues, https://github.com/auraoneai/iaa-kit/issues
Project-URL: Changelog, https://github.com/auraoneai/iaa-kit/blob/main/CHANGELOG.md
Keywords: inter-annotator-agreement,krippendorff-alpha,cohen-kappa,fleiss-kappa,gwet,annotation-quality
Classifier: Development Status :: 3 - Alpha
Classifier: Environment :: Console
Classifier: Intended Audience :: Science/Research
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering
Classifier: Topic :: Software Development :: Quality Assurance
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: numpy>=1.23
Dynamic: license-file

# iaa-kit

Compute inter-annotator agreement for labeling, review, and rubric-calibration workflows.

`iaa-kit` is for annotation leads, evaluation engineers, and data scientists who need inspectable agreement metrics without a statistics service. It combines a pure-NumPy implementation of Krippendorff alpha, Cohen and weighted kappa, Fleiss kappa, Gwet AC1/AC2, prevalence and bias diagnostics, partial credit, missing-data handling, and deterministic bootstrap confidence intervals.

## Inspectable Output

The Python API returns the Krippendorff alpha estimate and seeded bootstrap interval as a tuple. The installed `iaa-kit analyze` CLI writes JSON with item and rater counts, scale type, Krippendorff alpha with a seeded 95% interval, Fleiss kappa, and two-rater Cohen/weighted kappa values when applicable. The Python API also exposes the Gwet, prevalence, bias, and partial-credit functions.

## Runtime Boundary

Metrics run entirely in the local Python process over caller-provided arrays or a local JSON rating matrix. NumPy is the only runtime dependency. There are no network requests, model calls, telemetry hooks, or uploaded annotations.

## Install

Python library and CLI:

```bash
python -m pip install iaa-kit==0.1.2
```

## Python API Quickstart

```bash
python - <<'PY'
from iaa_kit import krippendorff_alpha

alpha, low, high = krippendorff_alpha(
    [[1, 1, 2], [2, 2, 2]],
    scale="ordinal",
    n_boot=200,
    seed=7,
)
print({"alpha": alpha, "ci95": [low, high]})
PY
```

## CLI Quickstart

```bash
iaa-kit analyze \
  --input examples/ratings_matrix.json \
  --scale nominal \
  --bootstrap-samples 200 \
  --seed 7 \
  --out agreement.json
```

## Documentation

- Metric selection: [`docs/when-to-use-which-metric.md`](docs/when-to-use-which-metric.md)
- Ordinal versus nominal labels: [`docs/ordinal-vs-nominal.md`](docs/ordinal-vs-nominal.md)
- Bootstrap CI tutorial: [`docs/bootstrap-ci-tutorial.md`](docs/bootstrap-ci-tutorial.md)

## Release Status

Registry and artifact status verified July 13, 2026: version `0.1.2` is published on PyPI and tagged `v0.1.2` in the public repository. The wheel includes the Python API and the `iaa-kit analyze` entry point. The project is alpha software. No annotation-volume, adoption, or benchmark claim is made.

## Limits

Agreement statistics do not establish label validity or reviewer correctness. The package does not recruit reviewers, host annotation data, or include real annotation datasets; bundled inputs are synthetic.

## Next Action

Run the Python API or JSON CLI on a representative rating matrix with the correct scale, then document the metric and threshold rationale before using the result as a gate.
