Metadata-Version: 2.5
Name: modalitycheck
Version: 0.1.0
Summary: Test multimodal data pipelines with reproducible fault injection and cross-modal invariants.
Project-URL: Homepage, https://github.com/modalitycheck/modalitycheck
Project-URL: Documentation, https://github.com/modalitycheck/modalitycheck/tree/main/docs
Project-URL: Repository, https://github.com/modalitycheck/modalitycheck
Project-URL: Issue Tracker, https://github.com/modalitycheck/modalitycheck/issues
Project-URL: Changelog, https://github.com/modalitycheck/modalitycheck/blob/main/CHANGELOG.md
Author: ModalityCheck contributors
License: MIT License
        
        Copyright (c) 2026 ModalityCheck contributors
        
        Permission is hereby granted, free of charge, to any person obtaining a copy
        of this software and associated documentation files (the "Software"), to deal
        in the Software without restriction, including without limitation the rights
        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
        copies of the Software, and to permit persons to whom the Software is
        furnished to do so, subject to the following conditions:
        
        The above copyright notice and this permission notice shall be included in all
        copies or substantial portions of the Software.
        
        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
        SOFTWARE.
License-File: LICENSE
Keywords: computer-vision,data-quality,machine-learning,multimodal,mutation-testing,testing
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Software Development :: Testing
Classifier: Typing :: Typed
Requires-Python: >=3.10
Provides-Extra: dev
Requires-Dist: build>=1.2; extra == 'dev'
Requires-Dist: mypy>=1.10; extra == 'dev'
Requires-Dist: pytest>=8.0; extra == 'dev'
Requires-Dist: ruff>=0.5; extra == 'dev'
Requires-Dist: twine>=5.1; extra == 'dev'
Description-Content-Type: text/markdown

# ModalityCheck

**Test the tests that protect your multimodal AI pipelines.**

[![Python 3.10+](https://img.shields.io/badge/python-3.10+-blue.svg)](https://www.python.org/downloads/)
[![Type Checked with mypy](https://img.shields.io/badge/mypy-strict-green.svg)](https://mypy.readthedocs.io/)
[![Linted with Ruff](https://img.shields.io/badge/code%20style-ruff-000000.svg)](https://github.com/astral-sh/ruff)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)

**ModalityCheck** is a lightweight, zero-dependency Python library for **mutation testing and fault injection** in multimodal AI/ML data pipelines (video, audio, text, sensor streams).

---

## Why Use ModalityCheck?

In multimodal AI systems (like Video-LLMs, Speech-Vision assistants, or robotics perception), data pipelines combine disparate media streams that easily fall out of sync:
- Hardware latency or Bluetooth lag causes audio to drift hundreds of milliseconds behind video.
- Sensor dropouts or network glitches cause a required modality (e.g. depth map, subtitles) to silently drop.
- Payloads become empty or truncated during batch ingestion.

When fed corrupted or misaligned data, **multimodal AI models fail silently**: attention mechanisms hallucinate, cross-attention weights degrade, and model accuracy drops without throwing an explicit crash.

Data engineers write validators to guard against these issues. But **how do you know your validators actually work?**

> **ModalityCheck tests your validators.** It acts like a "fire drill" for your pipeline by deliberately injecting controlled faults into your data and verifying that your validators catch every single defect.

---

## Where Should You Use This?

| Stage | Where It Fits | Benefit |
| :--- | :--- | :--- |
| **CI/CD Pipelines** | Run as part of your automated test suite alongside `pytest`. | Prevents regressions in your data validation rules before deploying new pipeline code. |
| **Data Ingestion Guards** | Place validators in streaming workers (Kafka, AWS SQS, GCP Pub/Sub). | Quarantines corrupted or desynchronized samples before they reach model inference. |
| **Dataset Curation & Pretraining** | Audit multimodal training datasets (WebVid, Ego4D, AudioSet, etc.). | Identifies edge cases and blind spots in dataset filtering scripts. |
| **Edge & Robotics Systems** | Test timestamp synchronization between cameras, LiDAR, and IMU sensors. | Guarantees multi-sensor alignment within sub-second tolerance thresholds. |

---

## How It Works: The Workflow

```text
       [ Healthy Multimodal Sample ]
           (Video + Audio + Text)
                     │
                     ▼
       ┌───────────────────────────┐
       │     ModalityCheck         │
       │     Fault Scenarios       │
       └─────────────┬─────────────┘
                     │ (Injects controlled mutations:
                     │  dropped audio, 400ms time drift)
                     ▼
       [ Mutated / Corrupted Sample ]
                     │
                     ▼
       ┌───────────────────────────┐
       │   Your Pipeline Guards    │
       │   (Required, Skew, etc.)  │
       └─────────────┬─────────────┘
                     │
         ┌───────────┴───────────┐
         ▼                       ▼
    [ CAUGHT? ]             [ MISSED? ]
         │                       │
         ▼                       ▼
  Report: 100% Score      Flags Pipeline Blind Spot!
  Safe to feed model      (Tighten validation rules)
```

---

## How Does It Look?

Running ModalityCheck produces a clear, structured mutation report:

```text
======================================================================
 REAL-WORLD DEMO: MODALITYCHECK MUTATION TESTING
======================================================================

1. Testing clean sample in production pipeline:
   Status: success
   Result: Model successfully analyzed clip_104.mp4 with audio_104.flac

2. Running ModalityCheck fault injection (testing the pipeline's guards):
   - Scenarios Tested: 3
   - Defects Caught:   3
   - Detection Rate:   100%

   Case #1: [CAUGHT]
     Scenario:            drop_modality
     Expected issue code: missing_modality
     Reported issues:     ["missing_modality: Required modality 'audio' is missing."]

   Case #2: [CAUGHT]
     Scenario:            shift_timestamp
     Expected issue code: timestamp_skew
     Reported issues:     ['timestamp_skew: Timestamp skew 0.420000s exceeds limit 0.150000s.']

   Case #3: [CAUGHT]
     Scenario:            shift_timestamp
     Expected issue code: timestamp_skew
     Reported issues:     ['timestamp_skew: Timestamp skew 0.580000s exceeds limit 0.150000s.']

3. Demonstrating that corrupted data is prevented from reaching the AI model:
   Status: REJECTED_BY_PIPELINE
   Errors: ['[timestamp_skew] Timestamp skew 0.420000s exceeds limit 0.150000s.']

   Protected AI model from processing out-of-sync audio!
======================================================================
```

---

## Quickstart Guide

### 1. Installation

Create a virtual environment and install the package:

```bash
python -m venv .venv

# On Linux/macOS:
source .venv/bin/activate
# On Windows:
# .venv\Scripts\activate

python -m pip install -e ".[dev]"
```

### 2. Basic Example

```python
from modalitycheck import (
    DropModality,
    ModalityData,
    MultimodalSample,
    RequiredModalities,
    ShiftTimestamp,
    TimestampSkew,
    evaluate_mutations,
)

# 1. Define your sample (lightweight payload references without heavy media loading)
sample = MultimodalSample(
    sample_id="clip-001",
    modalities={
        "video": ModalityData(value="s3://bucket/video.mp4", timestamp=1.00),
        "audio": ModalityData(value="s3://bucket/audio.wav", timestamp=1.05),
        "text": ModalityData(value="A person opens a door.", timestamp=1.02),
    },
)

# 2. Define your pipeline's validation rules
validators = [
    RequiredModalities({"video", "audio", "text"}),
    TimestampSkew(max_skew_seconds=0.10),  # Max 100ms drift allowed
]

# 3. Define the fault scenarios to test
scenarios = [
    DropModality(modality="audio"),
    ShiftTimestamp(modality="video", seconds=0.50),  # 500ms video lag
]

# 4. Evaluate whether your validators catch the faults
report = evaluate_mutations(
    samples=[sample],
    scenarios=scenarios,
    validators=validators,
)

print(f"Detection rate: {report.detection_rate:.0%}")
for case in report.cases:
    status = "CAUGHT" if case.detected else "MISSED"
    print(f"[{status}] Scenario '{case.scenario}' -> Expected: {case.expected_issue_code}")
```

---

## Writing Custom Validators and Scenarios

ModalityCheck is designed to be easily extensible using Python `Protocol` interfaces:

### Custom Validator Example
```python
from modalitycheck import Issue, MultimodalSample, ValidationResult

class NonEmptyPayloads:
    """Ensure no modality contains an empty or null payload."""

    def validate(self, sample: MultimodalSample) -> ValidationResult:
        issues = []
        for name, data in sample.modalities.items():
            if not data.value:
                issues.append(
                    Issue(
                        code="empty_payload",
                        message=f"Modality '{name}' is empty.",
                        sample_id=sample.sample_id,
                        modality=name,
                    )
                )
        return ValidationResult(tuple(issues))
```

### Custom Fault Scenario Example
```python
from dataclasses import dataclass
from modalitycheck import ModalityData, MultimodalSample

@dataclass(frozen=True)
class CorruptPayload:
    """Simulates payload corruption in a target modality."""

    modality: str
    name: str = "corrupt_payload"
    expected_issue_code: str = "empty_payload"

    def apply(self, sample: MultimodalSample) -> MultimodalSample:
        modalities = dict(sample.modalities)
        if self.modality in modalities:
            modalities[self.modality] = ModalityData(value="")  # wipe payload
        return sample.with_modalities(modalities)
```

---

## In-Depth Documentation

Explore the comprehensive guides in the [`docs/`](docs/) directory:

- 🏛️ **[Architecture & Internal Design](docs/ARCHITECTURE.md)**: Zero-copy memory model, structural subtyping (PEP 544), mutation testing theory, and immutability guarantees.
- 🔧 **[Troubleshooting & Common Pitfalls](docs/TROUBLESHOOTING.md)**: Debugging 0% detection rates, floating-point timestamp precision, Mypy protocols, and common gotchas.
- 📖 **[API Reference](docs/API_REFERENCE.md)**: Complete specification of all classes, methods, parameters, and return types.

---

## Running the Real-World Pipeline Demo

A complete end-to-end simulation showing an audio-visual AI model guarded by an ingestion pipeline is included in [`docs/examples/real_world_pipeline.py`](docs/examples/real_world_pipeline.py):

```bash
python docs/examples/real_world_pipeline.py
```

---

## Development & Publishing

Run all quality checks and verify the package build:

```bash
# Run unit tests
pytest

# Static type checking (Mypy strict mode)
mypy src tests docs/examples

# Code formatting and linting
ruff check .

# Build distribution wheels & source archive
python -m build

# Validate package metadata for PyPI
python -m twine check dist/*
```

---

## Roadmap

1. Add built-in validators for duplicate IDs, empty payloads, and monotonic timestamps.
2. Add property-based testing (Hypothesis) and mutation coverage reports.
3. Add optional lightweight adapters for popular dataset formats (HuggingFace `datasets`, WebDataset).
4. Provide standard CI/CD GitHub Action templates for pipeline validation.

---

## License

MIT. See [`LICENSE`](LICENSE).
