Metadata-Version: 2.4
Name: checkmaite
Version: 0.5.0
Summary: CheckMAITE - an implementation of T&E tools using maite
Author: OpenTeams
License: Apache-2.0
License-File: LICENSE.md
Keywords: artificial intelligence,jatic,testing and evaluation
Classifier: Development Status :: 1 - Planning
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Requires-Python: <3.14,>=3.10
Requires-Dist: dataeval-plots==0.0.6
Requires-Dist: dataeval==1.0.6
Requires-Dist: datamaite[fmv,ic,od]==0.6.0
Requires-Dist: fsspec>=2023.1.0
Requires-Dist: httpx>=0.28.1
Requires-Dist: ipython>=8.13
Requires-Dist: maite<0.10,>=0.9.4
Requires-Dist: matplotlib<3.10.8,>=3.7.1
Requires-Dist: modelmaite==0.2.0
Requires-Dist: nrtk[pybsm]==1.0.4
Requires-Dist: numba>=0.60.0
Requires-Dist: opencv-python-headless>=4.9.0.80
Requires-Dist: pandas>=2
Requires-Dist: pillow>=11.0.0
Requires-Dist: polars>=1.37.0
Requires-Dist: pydantic>=2.12.0
Requires-Dist: ray[serve]>=2.9.0
Requires-Dist: scikit-learn>=1.5.2
Requires-Dist: setuptools<82.0.0,>=78.1.1
Requires-Dist: smqtk-core>=0.21.0
Requires-Dist: smqtk-detection>=0.22.0
Requires-Dist: torch>=2.2.0
Requires-Dist: torchmetrics>=1.0.0
Requires-Dist: torchvision>=0.17.0
Requires-Dist: transformers[timm]!=4.51.0,>=4.48.0
Requires-Dist: typing-extensions>=4.14.1
Requires-Dist: universal-pathlib>=0.3.8
Requires-Dist: xaitk-jatic==0.8.1
Provides-Extra: aws
Requires-Dist: s3fs>=2023.1.0; extra == 'aws'
Provides-Extra: azure
Requires-Dist: adlfs>=2023.1.0; extra == 'azure'
Provides-Extra: cloud
Requires-Dist: adlfs>=2023.1.0; extra == 'cloud'
Requires-Dist: gcsfs>=2023.1.0; extra == 'cloud'
Requires-Dist: s3fs>=2023.1.0; extra == 'cloud'
Provides-Extra: gcs
Requires-Dist: gcsfs>=2023.1.0; extra == 'gcs'
Provides-Extra: onnx
Requires-Dist: onnx>=1.14.0; extra == 'onnx'
Requires-Dist: onnxruntime<1.24.0,>=1.14.0; extra == 'onnx'
Provides-Extra: onnx-cuda
Requires-Dist: onnx>=1.14.0; extra == 'onnx-cuda'
Requires-Dist: onnxruntime-gpu<1.24.0,>=1.14.0; extra == 'onnx-cuda'
Provides-Extra: reporting
Requires-Dist: markdown>=3.5; extra == 'reporting'
Requires-Dist: xhtml2pdf>=0.2.16; extra == 'reporting'
Description-Content-Type: text/markdown

# CheckMAITE

`CheckMAITE` is an integration point for AI T&E tooling for computer vision that
is based on the `maite` protocols (see [maite
documentation](https://mit-ll-ai-technology.github.io/maite/)).

## Description

**What is CheckMAITE?**
CheckMAITE is a Python API which makes testing and evaluation of models and
datasets straightforward and reproducible. It can be used to run a wide variety
of **Model Evaluation** and **Dataset Analysis** investigations for both
**Object Detection** and **Image Classification** computer vision problems.

It was built as an integration point for all [CDAO
JATIC](https://cdao.pages.jatic.net/public/) tools and has since expanded focus.
Our goal is to make T&E easier for analysts!

To learn more please visit our [published documentation](https://openteams-ai.github.io/checkmaite/).

## Supported environment

- **OS:** Linux x86_64 (CI-tested). macOS x86_64 and arm64 are supported but not
  CI-tested.
- **Python:** 3.10, 3.11, 3.12, and 3.13 with `uv` or pip. conda supports 3.11,
  3.12, and 3.13; the committed conda lock is for 3.13. `checkmaite-plugins` is
  `<3.12` on all paths.
- **GPU:** CPU is the supported baseline. CUDA is optional for PyTorch.
- **Hardware, architectures, and internet access:** see [System
  requirements](https://openteams-ai.github.io/checkmaite/reference/system_requirements.html).
- **Container:** a batch container is built from the repository `Dockerfile` for
  Linux AMD64, with `cpu` and NVIDIA `cuda` targets. See [Run CheckMAITE in a
  container](https://openteams-ai.github.io/checkmaite/get-started/container.html).

## Limitations and prerequisites

<!-- --8<-- [start:limitations] -->

CheckMAITE evaluates models and datasets you supply.

- Built-in loaders cover COCO, YOLO, and VisDrone object detection, and YOLO
  image classification.
- Models and datasets expose an `index2label` map as `dict[int, str]`. Default
  torchvision weights build it from the weights. Custom torchvision weights and
  ONNX models need an `index2label` in their JSON config (default key
  `index2label`, override with `index2label_key`), given as a list or a dict;
  both are normalized to `dict[int, str]`. `num_classes` defaults to
  `len(index2label)`. For object detection, set it when the model head has a
  different class count (e.g. a background slot). For classification it must
  equal `len(index2label)`.
- Where a model's and a dataset's `index2label` overlap, a shared index must
  name the same class and a shared class name must use the same index.
  CheckMAITE does not validate this; mismatches produce silently wrong per-class
  results.
- ONNX models need the `onnx` (or `onnx-cuda`) extra. ONNX wrappers accept only
  `uint8` images or finite float images in `[0, 1]` (cast to `float32`); other
  integer dtypes raise `TypeError`. Non-finite or out-of-`[0, 1]` float images
  raise `ValueError`.
- Object-detection boxes are `float32` `xyxy` arrays of shape `(N, 4)`. See
  [Conventions][limitations-conventions].
- This package provides the Python API and the batch container. The web UI is the
  separate CheckMAITE application (`checkmaite-frontend`). PDF export needs the
  `reporting` extra.
- CPU is the supported baseline. Machine, OS, and Python bounds are in the
  [supported environment][limitations-supported-environment].

<!-- --8<-- [end:limitations] -->

[limitations-conventions]: https://openteams-ai.github.io/checkmaite/reference/conventions.html
[limitations-supported-environment]: #supported-environment

## Installation

For detailed installation instructions please refer to [Setup Guide](https://openteams-ai.github.io/checkmaite/get-started/install_setup.html).
To run CheckMAITE as a batch container instead of installing the package, see
[Run CheckMAITE in a
container](https://openteams-ai.github.io/checkmaite/get-started/container.html).

On macOS, `brew install libomp` is required (`dataeval` loads LightGBM, which
needs `libomp.dylib`).

## Usage

To learn how to get started using `CheckMAITE`, please visit our [quick start guide](https://openteams-ai.github.io/checkmaite/get-started/index.html).

There are also [How-to
Guides](https://openteams-ai.github.io/checkmaite/how-to/index.html) &
[Tutorials](https://openteams-ai.github.io/checkmaite/tool-usage/index.html)
that will help get new users up to speed on the available functionality.

## Contributing

The `CheckMAITE` team welcomes contributions of all forms - questions,
documentation, and code contributions. Our contribution strategy is in
[CONTRIBUTING.md](CONTRIBUTING.md).

## Reporting bugs and requesting features

Report bugs and request features as issues, using the bug report or feature
request template:

- **JATIC users:** the [GitLab issue
  board](https://gitlab.jatic.net/jatic/orchestration-interoperability/checkmaite/-/issues)
  ([bug
  report](https://gitlab.jatic.net/jatic/orchestration-interoperability/checkmaite/-/issues/new?issuable_template=bug-report),
  [feature
  request](https://gitlab.jatic.net/jatic/orchestration-interoperability/checkmaite/-/issues/new?issuable_template=feature-request)).
- **Everyone else:** the [GitHub issue
  tracker](https://github.com/openteams-ai/checkmaite/issues) ([bug
  report](https://github.com/openteams-ai/checkmaite/issues/new?template=bug_report.yml),
  [feature
  request](https://github.com/openteams-ai/checkmaite/issues/new?template=feature_request.yml)).

[CONTRIBUTING.md](CONTRIBUTING.md) describes what to include and how issues are
managed.

## Security

Report security vulnerabilities privately, as described in
[SECURITY.md](SECURITY.md). Do not open an ordinary issue.

## Authors and acknowledgment

This project was created for [CDAO JATIC](https://cdao.pages.jatic.net/public/)
and is maintained by OpenTeams with collaborative community support.

### CDAO Funding Acknowledgment

<!-- --8<-- [start:acknowledgment] -->

This material is based upon work supported by the Chief Digital and Artificial
Intelligence Office under Contract No. W519TC-25-9-2041. The views and
conclusions contained herein are those of the author(s) and should not be
interpreted as necessarily representing the official policies or endorsements,
either expressed or implied, of the U.S. Government.

<!-- --8<-- [end:acknowledgment] -->
