Metadata-Version: 2.4
Name: checkmaite
Version: 0.7.0
Summary: CheckMAITE - an implementation of T&E tools using maite
Author: OpenTeams
License: Apache-2.0
License-File: LICENSE.md
Keywords: artificial intelligence,jatic,testing and evaluation
Classifier: Development Status :: 1 - Planning
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Requires-Python: <3.15,>=3.11
Requires-Dist: dataeval-plots==0.0.6
Requires-Dist: dataeval==1.0.6
Requires-Dist: datamaite[fmv,ic,od]==0.6.0
Requires-Dist: fsspec>=2023.1.0
Requires-Dist: ipython>=8.13
Requires-Dist: maite<0.10,>=0.9.4
Requires-Dist: matplotlib<3.10.8,>=3.7.1
Requires-Dist: modelmaite==0.2.0
Requires-Dist: motmetrics<1.5,>=1.4.0
Requires-Dist: nrtk[pybsm]==1.0.4
Requires-Dist: opencv-python-headless>=4.9.0.80
Requires-Dist: pandas>=2
Requires-Dist: pillow>=13.0.0
Requires-Dist: polars>=1.37.0
Requires-Dist: pydantic>=2.12.0
Requires-Dist: ray[serve]>=2.9.0
Requires-Dist: scikit-learn>=1.5.2
Requires-Dist: smqtk-core>=0.21.0
Requires-Dist: torch>=2.2.0
Requires-Dist: torchmetrics>=1.0.0
Requires-Dist: torchvision>=0.17.0
Requires-Dist: typing-extensions>=4.14.1
Requires-Dist: universal-pathlib>=0.3.8
Requires-Dist: xaitk-saliency[maite]>=0.16.0
Provides-Extra: aws
Requires-Dist: s3fs>=2023.1.0; extra == 'aws'
Provides-Extra: azure
Requires-Dist: adlfs>=2023.1.0; extra == 'azure'
Provides-Extra: cloud
Requires-Dist: adlfs>=2023.1.0; extra == 'cloud'
Requires-Dist: gcsfs>=2023.1.0; extra == 'cloud'
Requires-Dist: s3fs>=2023.1.0; extra == 'cloud'
Provides-Extra: gcs
Requires-Dist: gcsfs>=2023.1.0; extra == 'gcs'
Provides-Extra: onnx
Requires-Dist: modelmaite[onnx]==0.2.0; extra == 'onnx'
Provides-Extra: onnx-cuda
Requires-Dist: modelmaite[onnx-cuda]==0.2.0; extra == 'onnx-cuda'
Provides-Extra: reporting
Requires-Dist: markdown>=3.5; extra == 'reporting'
Requires-Dist: xhtml2pdf>=0.2.19; extra == 'reporting'
Provides-Extra: visdrone
Requires-Dist: modelmaite[visdrone]==0.2.0; extra == 'visdrone'
Description-Content-Type: text/markdown

# CheckMAITE

<!-- --8<-- [start:tagline] -->

CheckMAITE is an integration point for AI testing and evaluation tooling for
computer vision workflows.

CheckMAITE provides API and UI workflows that make model evaluation and dataset
analysis straightforward and reproducible. It integrates JATIC tools through
common MAITE-compatible interfaces so users can run model evaluation,
robustness testing, and dataset analysis workflows from a shared application
layer.

<!-- --8<-- [end:tagline] -->

CheckMAITE is based on the `maite` protocols (see [maite
documentation](https://mit-ll-ai-technology.github.io/maite/)).

## Description

**What is CheckMAITE?**
CheckMAITE is a Python API which makes testing and evaluation of models and
datasets straightforward and reproducible. It can be used to run a wide variety
of **Model Evaluation** and **Dataset Analysis** investigations for both
**Object Detection** and **Image Classification** computer vision problems.

It was built as an integration point for all [CDAO
JATIC](https://cdao.pages.jatic.net/public/) tools and has since expanded focus.
Our goal is to make T&E easier for analysts!

To learn more please visit our [published documentation](https://openteams-ai.github.io/checkmaite/).

## Supported environment

- **OS:** Linux x86_64 (CI-tested). macOS x86_64 and arm64 are supported but not
  CI-tested.
- **Python:** 3.11, 3.12, 3.13, and 3.14 with `uv` or pip. conda supports 3.11,
  3.12, and 3.13; the committed conda lock is for 3.13. `checkmaite-plugins` is
  `<3.12` on all paths.
- **GPU:** CPU is the supported baseline. CUDA is optional for PyTorch.
- **Hardware, architectures, and internet access:** see [System
  requirements](https://openteams-ai.github.io/checkmaite/reference/system_requirements.html).
- **Container:** a batch container is built from the repository `Dockerfile` for
  Linux AMD64, with `cpu` and NVIDIA `cuda` targets. See [Run CheckMAITE in a
  container](https://openteams-ai.github.io/checkmaite/get-started/container.html).

## Limitations and prerequisites

<!-- --8<-- [start:limitations] -->

CheckMAITE evaluates models and datasets you supply.

- Built-in loaders cover COCO, YOLO, and VisDrone object detection, and YOLO
  image classification.
- Models and datasets expose an `index2label` map as `dict[int, str]`. Default
  torchvision weights build it from the weights. Custom torchvision weights and
  ONNX models need an `index2label` in their JSON config (default key
  `index2label`, override with `index2label_key`), given as a list or a dict;
  both are normalized to `dict[int, str]`. `num_classes` defaults to
  `len(index2label)`. For object detection, set it when the model head has a
  different class count (e.g. a background slot). For classification it must
  equal `len(index2label)`.
- Where a model's and a dataset's `index2label` overlap, a shared index must
  name the same class and a shared class name must use the same index.
  CheckMAITE does not validate this; mismatches produce silently wrong per-class
  results.
- ONNX models need the `onnx` (or `onnx-cuda`) extra, and VisDrone models need
  the `visdrone` extra; neither is in the base install. ONNX wrappers accept only
  `uint8` images or finite float images in `[0, 1]` (cast to `float32`); other
  integer dtypes raise `TypeError`. Non-finite or out-of-`[0, 1]` float images
  raise `ValueError`.
- Object-detection boxes are `float32` `xyxy` arrays of shape `(N, 4)`. See
  [Conventions][limitations-conventions].
- This package provides the Python API and the batch container. The web UI is the
  separate CheckMAITE application (`checkmaite-frontend`). PDF export needs the
  `reporting` extra.
- CPU is the supported baseline. Machine, OS, and Python bounds are in the
  [supported environment][limitations-supported-environment].

<!-- --8<-- [end:limitations] -->

[limitations-conventions]: https://openteams-ai.github.io/checkmaite/reference/conventions.html
[limitations-supported-environment]: #supported-environment

## Installation

CheckMAITE is published on [PyPI](https://pypi.org/project/checkmaite/). Install
it with any of:

```bash
python -m pip install checkmaite   # pip
uv add checkmaite                  # uv project (or: uv pip install checkmaite)
poetry add checkmaite              # Poetry project
```

With conda, create an environment and install from PyPI (CheckMAITE is not on
conda-forge):

```bash
conda create -n checkmaite python=3.12 pip
conda activate checkmaite
python -m pip install checkmaite
```

On macOS, `brew install libomp` is required (`dataeval` loads LightGBM, which
needs `libomp.dylib`).

To install from source, optional extras (`reporting`, `onnx`, `cloud`, ...),
and the locked uv and conda environments, see the [Setup
Guide](https://openteams-ai.github.io/checkmaite/get-started/install_setup.html).
To run CheckMAITE as a batch container instead of installing the package, see
[Run CheckMAITE in a
container](https://openteams-ai.github.io/checkmaite/get-started/container.html).

## Usage

<!-- --8<-- [start:example] -->

This example builds a tiny image classification dataset, checks it for
duplicates with DataEval, and evaluates a two-class torchvision model on it with
MAITE. It runs on a CPU in under a minute and needs no downloads. The model has
untrained weights, so its accuracy is meaningless.

<!-- markdownlint-disable MD013 -- import lines cannot be wrapped -->

```python
import json
from pathlib import Path

# Import checkmaite before torch so it can configure PyTorch on macOS.
from checkmaite.core.image_classification import DataevalCleaning, MaiteEvaluation
from checkmaite.core.image_classification.dataset_loaders import load_yolo_classification_dataset
from checkmaite.core.image_classification.metrics import accuracy_multiclass_torch_metric_factory
from checkmaite.core.image_classification.models import TorchvisionICModel

import torch
import torchvision
from PIL import Image

# A tiny YOLO classification dataset: <root>/<split>/<class>/<image>.
# Each "dog" image is a copy of a "cat" image, so the dataset has duplicates.
root = Path("example_data")
for label in ["cat", "dog"]:
    (root / "test" / label).mkdir(parents=True, exist_ok=True)
    for i in range(4):
        Image.new("RGB", (64, 64), color=(60 * i, 0, 0)).save(root / "test" / label / f"{i}.jpg")
dataset = load_yolo_classification_dataset(root_dir=root, dataset_id="example", split="test")

# A two-class torchvision model (untrained weights) and its index2label config.
torch.save(torchvision.models.resnext50_32x4d(num_classes=2).state_dict(), root / "model.pt")
(root / "model.json").write_text(json.dumps({"index2label": {"0": "cat", "1": "dog"}}))
model = TorchvisionICModel(
    model_name="resnext50_32x4d", weights_path=root / "model.pt", config_path=root / "model.json"
)
metric = accuracy_multiclass_torch_metric_factory(num_classes=2)

# Check the dataset with DataEval, then evaluate the model with MAITE.
cleaning = DataevalCleaning().run(datasets=[dataset])
print(cleaning.outputs.duplicates)

evaluation = MaiteEvaluation().run(datasets=[dataset], models=[model], metrics=[metric])
print(evaluation.outputs.metrics)
```

<!-- markdownlint-enable MD013 -->

The output lists the duplicate images (`exact=[[0, 4], [1, 5], [2, 6], [3, 7]]`)
and the accuracy result. Every capability follows the same pattern: wrap your
model, dataset, and metric, then call `run` on a capability.

<!-- --8<-- [end:example] -->

To learn how to get started using `CheckMAITE`, please visit our [quick start guide](https://openteams-ai.github.io/checkmaite/get-started/index.html).

There are also [How-to
Guides](https://openteams-ai.github.io/checkmaite/how-to/index.html) &
[Tutorials](https://openteams-ai.github.io/checkmaite/tool-usage/index.html)
that will help get new users up to speed on the available functionality.

## Roadmap

Planned features and releases for the next six months are in
[ROADMAP.md](ROADMAP.md).

## Contributing

The `CheckMAITE` team welcomes contributions of all forms - questions,
documentation, and code contributions. Our contribution strategy is in
[CONTRIBUTING.md](CONTRIBUTING.md).

## Reporting bugs and requesting features

Report bugs and request features as issues, using the bug report or feature
request template:

- **JATIC users:** the [GitLab issue
  board](https://gitlab.jatic.net/jatic/orchestration-interoperability/checkmaite/-/issues)
  ([bug
  report](https://gitlab.jatic.net/jatic/orchestration-interoperability/checkmaite/-/issues/new?issuable_template=bug-report),
  [feature
  request](https://gitlab.jatic.net/jatic/orchestration-interoperability/checkmaite/-/issues/new?issuable_template=feature-request)).
- **Everyone else:** the [GitHub issue
  tracker](https://github.com/openteams-ai/checkmaite/issues) ([bug
  report](https://github.com/openteams-ai/checkmaite/issues/new?template=bug_report.yml),
  [feature
  request](https://github.com/openteams-ai/checkmaite/issues/new?template=feature_request.yml)).

[CONTRIBUTING.md](CONTRIBUTING.md) describes what to include and how issues are
managed.

## Security

Report security vulnerabilities privately, as described in
[SECURITY.md](SECURITY.md). Do not open an ordinary issue.

## Authors and acknowledgment

This project was created for [CDAO JATIC](https://cdao.pages.jatic.net/public/)
and is maintained by OpenTeams with collaborative community support.

### CDAO Funding Acknowledgment

<!-- --8<-- [start:acknowledgment] -->

This material is based upon work supported by the Chief Digital and Artificial
Intelligence Office under Contract No. W519TC-25-9-2041. The views and
conclusions contained herein are those of the author(s) and should not be
interpreted as necessarily representing the official policies or endorsements,
either expressed or implied, of the U.S. Government.

<!-- --8<-- [end:acknowledgment] -->
