mldebug

mldebug

A lightweight Python package for validating and comparing datasets in machine learning pipelines.

Provides tools to run validation checks on reference and current datasets and return reports of detected issues.

@dataclass(frozen=True, slots=True)
class Issue:

Represents a detected issue from a validation or monitoring check.

This is the atomic output of all checks and is intended to be consumed by downstream reporting, alerting, or debugging components.

Parameters
  • name (str): Identifier of the issue type (e.g., "ks_test", "missing_values").
  • metric (str): Name of the metric used to detect the issue (e.g., "distribution_shift_score", "missing_rate_increase").
  • severity (Severity): Importance level of the issue.
  • message (str): Human-readable explanation of the issue.
  • feature (str | None): Feature associated with the issue. None for global issues.
  • value (float | None): Observed metric value that triggered the issue (if applicable).
  • threshold (float | None): Threshold used for comparison. Interpretation depends on the metric.
Issue( name: str, metric: str, severity: Severity, message: str, feature: str | None = None, value: float | None = None, threshold: float | None = None)
name: str
metric: str
severity: Severity
message: str
feature: str | None
value: float | None
threshold: float | None
@dataclass(frozen=True, slots=True)
class Report:

Aggregated output of a full ML debugging run.

Parameters
  • issues (list[Issue]): Collection of detected issues.
Report(issues: list[Issue])
issues: list[Issue]
def summary(self) -> dict[str, typing.Any]:

Summarize issues by severity and total count.

def to_dict(self) -> dict[str, typing.Any]:

Serialize report for logging / APIs.

def score(self) -> dict[str, typing.Any]:

Return a dataset quality score.

The score represents data quality based only on feature-level issues. System-level issues (e.g. schema errors, invalid inputs) are not included in the score but are available in the report.

Returns
  • dict[str, Any]: Dictionary containing:

overall_score : float Dataset quality score in [0, 100]. Higher is better.

feature_scores : dict[str, float] Per-feature scores.

status : str pass / warning / fail.

system_issue_count : int Number of system-level issues.

def is_clean(self) -> bool:

Check if report contains no issues.

Returns
  • bool: True if the report contains no issues, False otherwise.
def has_critical(self) -> bool:

Check if any issue has critical severity.

Returns
  • bool: True if at least one issue has critical severity, False otherwise.
def highest_severity(self) -> Severity | None:

Return the highest severity level in the report.

Returns
  • Severity | None: The highest severity level (CRITICAL > WARNING > INFO), or None if no issues are present.
class Severity(enum.Enum):

Severity level of a detected issue.

INFO: Informational issue with no immediate impact.

WARNING: Potential problem that should be reviewed.

CRITICAL: Serious issue likely to affect model performance or reliability.

INFO = <Severity.INFO: 'info'>
WARNING = <Severity.WARNING: 'warning'>
CRITICAL = <Severity.CRITICAL: 'critical'>
class FeatureType(enum.Enum):

Supported feature types in mldebug.

Defines the canonical feature categories used across schema validation, normalization, and feature-level checks.

NUMERIC: Numeric features validated using numeric-based validation checks.

CATEGORICAL: Categorical features validated using category-based validation checks.

NUMERIC = <FeatureType.NUMERIC: 'numeric'>
CATEGORICAL = <FeatureType.CATEGORICAL: 'categorical'>
def validate( reference: Mapping[str, ArrayLike], current: Mapping[str, ArrayLike], schema: Mapping[str, FeatureType]) -> Report:

Run validation checks on reference and current datasets.

This is the main entrypoint of the library. It performs schema analysis (validation and mismatch detection) followed by feature-level checks based on the provided schema, and returns a structured report of issues.

Parameters
  • reference (Mapping[str, ArrayLike]): Reference dataset keyed by feature name (e.g. training data).
  • current (Mapping[str, ArrayLike]): Current dataset keyed by feature name (e.g. production data).
  • schema (Mapping[str, FeatureType]): Mapping of feature names to their expected types.
Returns
  • Report: Aggregated report containing all detected issues.
def infer_schema( dataset: Mapping[str, ArrayLike]) -> dict[str, FeatureType]:

Infer a feature schema from raw dataset values.

Parameters
  • dataset (Mapping[str, ArrayLike]): Mapping from feature names to array-like values.
Returns
  • dict[str, FeatureType]: Inferred schema mapping feature names to inferred feature types.
__version__ = '0.8.0'