Metadata-Version: 2.4
Name: dqm-ml-job
Version: 2.0.0rc1
Summary: Python library designed provide pipelining tools dqm-ml library to compute data quality metrics for Machine Learning
Author-email: Safenai <support@safenai.io>
License-Expression: Apache-2.0
Project-URL: Homepage, https://irt-systemx.github.io/dqm-ml
Project-URL: Documentation, https://irt-systemx.github.io/dqm-ml
Project-URL: Repository, https://github.com/IRT-SystemX/dqm-ml
Keywords: ml,metrics,data
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Programming Language :: Python
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Typing :: Typed
Requires-Python: >=3.10
Description-Content-Type: text/markdown
Requires-Dist: dqm-ml-core>=2.0.0rc0
Requires-Dist: pyyaml>=6.0
Requires-Dist: tqdm>=4.65.0

# DQM-ML Job

Orchestration engine for DQM-ML V2. Handles data loading, processing, and output writing.

## Installation

```bash
pip install dqm-ml-job
```

> **Note:** `dqm-ml-job` handles data loading and orchestration. To compute metrics, you also need at least one of: `dqm-ml-core`, `dqm-ml-images`, or `dqm-ml-pytorch` (see Dependencies below).

## Quick Start

### Using Python

```python
from dqm_ml_job.cli import execute

# Execute a data quality job from a YAML config
execute(["-p", "config.yaml"])
```

### Using Python Module

```bash
python -m dqm_ml_job.cli -p config.yaml
```

Example `config.yaml`:

```yaml
dataloaders:
  my_data:
    type: parquet
    path: data/train.parquet

metrics_processor:
  completeness:
    type: completeness
    input_columns: [col_a, col_b]
```

## Dependencies

DQM-ML is modular — `dqm-ml-job` provides the orchestration, but you need additional packages to compute actual metrics:

```bash
# For Completeness and Representativeness
pip install dqm-ml-job dqm-ml-core

# For Visual Features
pip install dqm-ml-job dqm-ml-images

# For Domain Gap
pip install dqm-ml-job dqm-ml-pytorch

# All metrics
pip install dqm-ml-job dqm-ml-core dqm-ml-images dqm-ml-pytorch
```

## Key Components

### DatasetPipeline

The main orchestrator that:

- Loads the configuration
- Discovers plugins via entry points
- Executes the streaming loop
- Manages memory and I/O efficiency

### Protocols

| Protocol | Description |
|----------|-------------|
| **DataLoader** | Factory for creating data selections (e.g., Parquet, CSV loaders) |
| **DataSelection** | Represents a specific subset of data and provides an iterator over batches |
| **OutputWriter** | Persists computed features or metrics to disk |

## Built-in Loaders

| Loader | Description |
|--------|-------------|
| **parquet** | Optimized loading using PyArrow |
| **csv** | Flexible loading using Pandas |

## See Also

- [Configuration Guide](https://safenai.github.io/dqm-ml-workspace/docs/configuration/)
- [Architecture Documentation](https://safenai.github.io/dqm-ml-workspace/docs/dqm-ml-v2/)
