Metadata-Version: 2.4
Name: dqm-ml-pipeline
Version: 1.1.5
Summary: Python library designed provide pipelining tools dqm-ml library to compute data quality metrics for Machine Learning
Author-email: Safenai <support@safenai.io>
License-Expression: Apache-2.0
Project-URL: Homepage, https://irt-systemx.github.io/dqm-ml
Project-URL: Documentation, https://irt-systemx.github.io/dqm-ml
Project-URL: Repository, https://github.com/IRT-SystemX/dqm-ml
Keywords: ml,metrics,data
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Programming Language :: Python
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Typing :: Typed
Requires-Python: >=3.10
Description-Content-Type: text/markdown
Requires-Dist: dqm-ml-core>=0.0.1
Requires-Dist: pyyaml>=6.0
Requires-Dist: duckdb>=1.0.0

# DQM-ML Pipeline

This package provides the orchestration engine for DQM-ML V2. It handles the lifecycle of data processing, from loading to metric computation and output writing.

## Key Components

### `DatasetPipeline`

The main orchestrator that:

* Loads the configuration.
* Discovers plugins via entry points.
* Executes the streaming loop.
* Manages memory and I/O efficiency.

### Protocols

* `DataLoader`: A factory for creating data selections (e.g., Parquet, CSV loaders).
* `DataSelection`: Represents a specific subset of data and provides an iterator over batches.
* `OutputWriter`: Persists computed features or metrics to disk.

## Built-in Loaders

* `parquet`: Optimized loading using PyArrow.
* `csv`: Flexible loading using Pandas.
