Metadata-Version: 2.4
Name: ai-critic
Version: 5.0.0
Summary: Task-aware ML model auditing across performance, robustness, fairness, calibration, explainability, data quality and complexity.
License: MIT
Project-URL: Homepage, https://github.com/YOUR_USER/ai-critic
Project-URL: Documentation, https://github.com/YOUR_USER/ai-critic#readme
Project-URL: Bug Tracker, https://github.com/YOUR_USER/ai-critic/issues
Project-URL: Changelog, https://github.com/YOUR_USER/ai-critic/blob/main/CHANGELOG.md
Keywords: machine-learning,model-evaluation,fairness,mlops,ai-critic,calibration,robustness,explainability,scikit-learn,xgboost,lightgbm,responsible-ai
Classifier: Development Status :: 5 - Production/Stable
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Software Development :: Quality Assurance
Requires-Python: >=3.9
Description-Content-Type: text/markdown
Requires-Dist: numpy>=1.24
Requires-Dist: scikit-learn>=1.3
Requires-Dist: scipy>=1.11
Requires-Dist: pandas>=2.0
Provides-Extra: reports
Requires-Dist: matplotlib>=3.7; extra == "reports"
Requires-Dist: jinja2>=3.1; extra == "reports"
Provides-Extra: all
Requires-Dist: ai-critic[reports]; extra == "all"

<h1 align="center">ai-critic</h1> <p align="center">
<b>Audit machine-learning models by what they do—not just by their algorithm.</b>
</p> <p align="center">
  Task-aware evaluation across performance, robustness, explainability, calibration, data quality, fairness, and complexity.
</p> <p align="center">
  <a href="https://pypi.org/project/ai-critic/"><img src="https://img.shields.io/pypi/v/ai-critic.svg" alt="PyPI version"></a>
  <a href="https://pypi.org/project/ai-critic/"><img src="https://img.shields.io/pypi/pyversions/ai-critic.svg" alt="Python versions"></a>
  <img src="https://img.shields.io/badge/License-MIT-yellow.svg" alt="MIT License">
</p>

`ai-critic` is an extensible model-auditing library built around an `AuditPipeline`, a `BaseEvaluator` interface, and seven reusable evaluators. You declare the **task type**; the library selects task-specific evaluation strategies and reports both measurements and limitations.

## Get started

Install the library:

```bash
python -m pip install ai-critic
```

Audit a model with an explicit task type:

```python
import ai_critic
from ai_critic import EvaluationConfig

report = ai_critic.audit(
    model,
    (X_test, y_test ),
    task_type="classification",
    evaluation=EvaluationConfig(cv_folds=5, random_state=42),
)

print(report.overall_score)
print(report.overall_verdict.value)
print(report.dimensions["performance"].to_dict())
```

`model` should be a fitted, scikit-learn-compatible estimator (or another object supported by the selected strategy). `X_test` and `y_test` should be the evaluation data—not the data used to fit the model—unless your evaluation protocol intentionally says otherwise.

### Choose a task type

```python
"regression"
"classification"
"clustering"
"anomaly_detection"
"dimensionality_reduction"
"reinforcement_learning"
```

For **clustering**, **anomaly detection without labels**, and **dimensionality reduction**, a target is optional, but the task type is required:

```python
report = ai_critic.audit(
    clusterer,
    X,
    task_type="clustering",
)
```

The legacy supervised call `ai_critic.audit(model, X, y)` is still supported for compatibility. New code should pass `task_type` explicitly. An algorithm does not determine its task: a tree, linear model, SVM, or neural network may be used for different task types. Logistic regression is usually a classifier.

### Read the report

```python
# Full JSON-serializable report
payload = report.to_dict()

# Compact terminal report
from ai_critic.reporters import print_report
print_report(report)

# Optional self-contained HTML report
from ai_critic.reporters import save_html_report
save_html_report(report, "audit.html")
```

Each dimension contains a `score`, `status`, `verdict`, `summary`, `details`, and `suggestions`. A score is `None` when a dimension is not applicable or the available evidence is insufficient; check `status` to distinguish those cases.

## 30-second mental model

```
Declare task type + provide model and evaluation data
                         │
                         ▼
                 Select task strategy
                         │
                         ▼
       Run the seven existing dimension evaluators
                         │
             ┌───────────┴───────────┐
             ▼                       ▼
   Task-appropriate metrics   Explicit limitations/status
             └───────────┬───────────┘
                         ▼
       Report metrics, protocol, weights, and verdict
```

> **Interpretation:** Dimension scores are task-specific indicators—not universal measurements of model quality, and not necessarily comparable across task types. Read the metric names, protocols, score transformations, and limitations alongside any score.

## Task types and metric choices

The strategy is selected by **task type**, not by model family. Metrics are only reported when their interpretation is supported by the task and available data.

| Task type | Example performance evidence | Additional task-specific evidence |
| --- | --- | --- |
| `regression` | Cross-validated R², MAE, RMSE, MAPE | Perturbation impact on R²; permutation importance scored by R² |
| `classification` | Cross-validated macro F1, accuracy, precision, recall, MCC; ROC-AUC for binary classification | Out-of-fold probability calibration when `predict_proba` is available; group F1 comparisons |
| `clustering` | Adjusted Rand Index (ARI) and normalized mutual information when reference labels are supplied; otherwise silhouette | ARI stability under perturbations; feature-to-cluster separation diagnostics |
| `anomaly_detection` | Average precision / ROC-AUC when reference anomaly labels are supplied; F1 from predicted labels | Agreement of anomaly flags under perturbations; score sensitivity where supported |
| `dimensionality_reduction` | Reconstruction error when `inverse_transform` is available; otherwise trustworthiness | Pairwise-distance stability and component-loading diagnostics where available |
| `reinforcement_learning` | Episode returns; a normalized performance score only when a target or baseline is declared | Action stability under observation perturbations and action sensitivity |

### The seven dimensions

| Dimension | What it evaluates | Examples of task-specific choices |
| --- | --- | --- |
| **Performance** | Task outcome using validation data or a declared reference | F1-macro for classification; R² for regression; silhouette/ARI for clustering; episode return for RL |
| **Robustness** | Change under reproducible feature or observation perturbations | F1/R² degradation, cluster-label agreement, anomaly-label agreement, embedding geometry, or policy-action stability |
| **Explainability** | Feature contribution, separation, component loading, or policy sensitivity | Permutation importance for supervised prediction; cluster separation; anomaly score sensitivity; embedding loadings |
| **Calibration** | Agreement between predicted probabilities and observed event rates | Out-of-fold ECE/MCE/Brier-style evidence for classifiers; not applicable to tasks without a probabilistic event target |
| **Data quality** | Dataset-level issues such as missing values, duplicates, constants, outliers, and classification imbalance | Target-independent checks also work for unsupervised inputs |
| **Fairness** | Differences across explicitly supplied sensitive groups | Group F1/R², anomaly rates, cluster composition, representation geometry, or episode returns depending on task |
| **Complexity** | Structural heuristics and measured single-input latency when supported | Feature/sample ratio, estimator parameters, tree depth, and prediction/transform/action latency |

These are audit diagnostics, not guarantees of fairness, safety, causality, or production performance. In particular, a disparity check is only as meaningful as the sensitive attributes, data coverage, and metric provided.

## Inputs and evaluation configuration

### Supervised tabular data

Pass either the legacy three-argument form or a feature/target pair:

```python
report = ai_critic.audit(
    model,
    X_test,
    y_test,
    task_type="regression",
)

# Equivalent data-pair form
report = ai_critic.audit(
    model,
    (X_test, y_test),
    task_type="regression",
)
```

NumPy arrays and pandas DataFrames/Series are supported. DataFrame column names are carried into reports where a strategy can use them.

### Unsupervised tabular data

```python
cluster_report = ai_critic.audit(
    clusterer,
    X,
    task_type="clustering",
)

embedding_report = ai_critic.audit(
    reducer,
    X,
    task_type="dimensionality_reduction",
)
```

When you have reference labels for an unsupervised output, pass them as a pair. For example, cluster labels can then be compared with external ground truth using ARI/NMI rather than relying only on internal validity metrics:

```python
cluster_report = ai_critic.audit(
    clusterer,
    (X, reference_labels),
    task_type="clustering",
)
```

### Anomaly-detection labels

For supervised anomaly evaluation, `y` should identify the anomaly class. The default supports binary labels `0/1` and `-1/1` (with `-1` interpreted as the anomaly for the latter convention). For another positive label, declare it:

```python
config = EvaluationConfig(anomaly_label="fraud")
report = ai_critic.audit(
    detector,
    (X, labels),
    task_type="anomaly_detection",
    evaluation=config,
)
```

Without reference labels, the predicted anomaly rate is descriptive only. The performance dimension is `inconclusive`; it does not invent an accuracy or ranking score.

### Reinforcement-learning episodes

The general API accepts a mapping containing observations and episode rewards. A policy should expose `act`, `predict`, or be callable for action-based diagnostics.

```python
from ai_critic import EvaluationConfig

report = ai_critic.audit(
    policy,
    {
        "observations": observations,
        "episodes": [
            {"rewards": episode_1_rewards},
            {"rewards": episode_2_rewards},
        ],
    },
    task_type="reinforcement_learning",
    evaluation=EvaluationConfig(
        target_return=100.0,
        return_baseline=0.0,
        random_state=42,
    ),
)
```

Returns are environment-specific. Without `target_return` (or a declared baseline/target reference), the library reports observed return statistics but leaves the normalized performance score **inconclusive**. Supply observations aligned with the policy inputs; supply group attributes aligned with episodes if you want group-return diagnostics.

### EvaluationConfig reference

`EvaluationConfig` exposes evaluation protocol choices. Defaults are shown below.

| Field | Default | Purpose |
| --- | --- | --- |
| `cv_folds` | `5` | Number of folds where the task protocol uses cross-validation |
| `random_state` | `42` | Seed for splits and deterministic perturbations |
| `n_repeats` | `5` | Repeats for permutation-importance estimates |
| `n_bins` | `10` | Number of bins used for calibration diagnostics |
| `noise_levels` | `(0.01, 0.05, 0.10, 0.20)` | Relative noise magnitudes for robustness checks |
| `dropout_fraction` | `0.30` | Fraction of features dropped in applicable robustness checks |
| `outlier_fraction` | `0.05` | Fraction of rows perturbed in applicable checks |
| `sensitive_features` | `None` | Sensitive feature names or column indices for group diagnostics |
| `metrics` | `None` | Reserved metric-selection input; task strategies determine the valid metrics |
| `anomaly_label` | `None` | Positive anomaly label when it is not inferred from `0/1` or `-1/1` |
| `target_return` | `None` | Declared target return for normalized RL performance |
| `return_baseline` | `None` | Optional RL reference return used in the normalization |
| `group_column` | `None` | Optional group metadata for caller-defined protocols |
| `max_samples` | `None` | Optional sample-limit metadata for caller-defined protocols |

Example:

```python
config = EvaluationConfig(
    cv_folds=3,
    random_state=7,
    n_repeats=10,
    noise_levels=(0.02, 0.05, 0.10),
)
```

The report records the protocol and relevant score transformations in each dimension's `details`. Configuration values do not turn an unsupported metric into a valid one.

## Statuses, scores, and aggregation

Every dimension reports one of these statuses:

| Status | Meaning | Score |
| --- | --- | --- |
| `ok` | The dimension produced an interpretable measurement | Numeric value, normally in `[0, 1]` |
| `not_applicable` | The dimension has no valid interpretation for this task/protocol | `null` |
| `inconclusive` | The dimension could apply, but required data or capabilities are missing | `null` |

`not_applicable` and `inconclusive` dimensions are **excluded** from the overall weighted mean. They are not assigned a perfect score. Their reasons appear in the dimension result and in `report.aggregation["excluded_dimensions"]`.

The report's `aggregation` object states the aggregate method, included dimensions, excluded dimensions, weights, and score-normalization convention. The aggregate is therefore inspectable, but it is still a summary of task-specific normalized diagnostics—not a universal quality score.

```python
print(report.aggregation)

for name, dimension in report.dimensions.items():
    print(name, dimension.status, dimension.score, dimension.summary)
```

## Fairness and sensitive features

Fairness checks need an explicit group definition. Supply column names when `X` is a DataFrame or indices for array inputs:

```python
report = ai_critic.audit(
    model,
    X,
    y,
    task_type="classification",
    sensitive_features=["age_group", "region"],
)
```

Or include the same list in `EvaluationConfig(sensitive_features=[...])`.

The current input path converts tabular features to numeric arrays. Encode categorical group attributes before auditing, and verify that the resulting values preserve the intended group membership. If no group is declared or detected, fairness is reported as `inconclusive`, not as a passing score.

## Weights, parallel execution, and custom evaluators

### Override dimension weights

```python
report = ai_critic.audit(
    model,
    (X, y),
    task_type="classification",
    weights={"performance": 2.0, "fairness": 1.5},
)
```

Weights are shown in `report.aggregation`. Dimensions with no valid score are excluded regardless of weight. Do not compare aggregates built with different weights as if they were identical measures.

### Run evaluators in parallel

```python
report = ai_critic.audit(
    model,
    (X, y),
    task_type="classification",
    parallel=True,
)
```

Evaluators with declared dependencies still run after their prerequisites. Parallelism can be slower for small datasets or estimators that already use their own parallel workers.

### Add a custom evaluator

`AuditPipeline` and `BaseEvaluator` remain the extension points. A custom evaluator returns a `DimensionResult` and can use the shared `context` passed by the pipeline.

```python
from ai_critic.core.base import BaseEvaluator
from ai_critic.core.types import DimensionResult, Verdict

class MyEvaluator(BaseEvaluator):
    name = "my_check"
    weight = 1.0
    depends_on = ["performance"]

    def evaluate(self, model, X, y, problem_type, context=None):
        score = 0.9
        return DimensionResult(
            name=self.name,
            score=score,
            verdict=self._score_to_verdict(score),
            summary="Custom check completed.",
            details={"task_type": context.get("task_type").value},
        )

report = ai_critic.audit(
    model,
    (X, y),
    task_type="classification",
    evaluators=[MyEvaluator()],  # Supplying this list replaces the defaults.
)
```

To extend one dimension across task types, prefer adding or registering a task strategy and delegating from the existing evaluator rather than creating a parallel audit library.

## Reports and CI gates

### JSON-serializable report

```python
import json

with open("audit.json", "w", encoding="utf-8") as f:
    json.dump(report.to_dict(), f, indent=2)
```

A dimension with no meaningful score is serialized with `"score": null` and a status explaining why.

### HTML report

```python
from ai_critic.reporters import save_html_report

path = save_html_report(report, "audit.html")
print(f"Saved to {path}")
```

The reporter renders the current dimension score/status and embeds the report data. It does not add metrics that were not computed.

### Gate a pipeline or deployment check

```python
ai_critic.gate(report, min_score=0.75)
```

`gate()` raises `RuntimeError` when `overall_score` is below the supplied threshold. Choose a threshold with the task, included dimensions, weights, and declared limitations in mind. The gate does not independently validate that a report includes every dimension your release policy requires.

## Command-line interface

Audit a supervised CSV with a target column:

```bash
ai-critic score model.pkl data.csv \
  --target target \
  --task-type classification \
  --json
```

Audit an unlabeled tabular dataset:

```bash
ai-critic score clusterer.pkl data.csv \
  --task-type clustering \
  --report audit.html
```

Add sensitive feature columns and a gate threshold:

```bash
ai-critic score model.pkl data.csv \
  --target outcome \
  --task-type classification \
  --sensitive age_group,region \
  --report audit.html \
  --gate 0.75
```

The CLI currently reads tabular CSV data. For RL episode collections or other structured data, use the Python API.

## Scope and limitations

- **Evaluation data matter.** Use a representative held-out set or document why another protocol is appropriate. Cross-validation on supplied data does not prevent leakage that already occurred before the data reached the audit.

- **Preprocessing is part of the evaluated estimator.** Pass the fitted pipeline when transformations are needed. The audit does not reconstruct your training pipeline from raw model weights.

- **Task metrics have assumptions.** For example, R² can be negative; its displayed normalized score is clipped to `[0, 1]` and the raw metric is retained in `details`. Check the metric itself and its transformation.

- **Some diagnostics are heuristics.** Feature-concentration and complexity scores are descriptive diagnostics, not proof of causal importance, optimal model size, or fairness.

- **Fairness requires valid groups.** Incorrect encodings, small cohorts, omitted attributes, and unrepresentative data can make group comparisons misleading or inconclusive.

- **Unsupervised quality is not ground truth.** Internal metrics such as silhouette measure a chosen geometric property; they do not establish that clusters are useful for a real application.

- **RL returns depend on the environment.** A return has no universal good/bad scale. Declare a reference before interpreting a normalized score.

- **Model API support varies.** A strategy may require methods such as `predict`, `predict_proba`, `fit_predict`, `transform`, `inverse_transform`, `score_samples`, or `act`. Missing methods should produce an explicit inconclusive result where applicable.

- **An audit is not a certification.** The report is evidence to inform review; it is not a guarantee of safety, compliance, fairness, robustness, or deployment readiness.

## Compatibility

The primary integration is with scikit-learn-style estimators. The library also works with compatible estimator wrappers where their required methods, cloning behavior, and data types are supported. Check each task strategy's available metrics when using XGBoost, LightGBM, CatBoost, a custom estimator, a dimensionality reducer, an anomaly detector, or a policy object.

Python requirement: `>=3.9`.

## Development and tests

From the source checkout:

```bash
python -m pip install -e ".[reports]"
python -m pytest tests test -q
```

Run only the task-strategy contracts:

```bash
python -m pytest tests/test_task_strategies.py -q
```

Build local distribution artifacts:

```bash
python -m pip install build
python -m build
```

## Documentation map

| If you want to... | Start with |
| --- | --- |
| Run the basic audit | [Get started](#get-started) |
| Choose a task and understand its metrics | [Task types and metric choices](#task-types-and-metric-choices) |
| Configure evaluation protocols | [Inputs and evaluation configuration](#inputs-and-evaluation-configuration) |
| Interpret null scores and aggregate weights | [Statuses, scores, and aggregation](#statuses-scores-and-aggregation) |
| Audit group disparities | [Fairness and sensitive features](#fairness-and-sensitive-features) |
| Extend an evaluator | [Weights, parallel execution, and custom evaluators](#weights-parallel-execution-and-custom-evaluators) |
| Export or gate results | [Reports and CI gates](#reports-and-ci-gates) |
| Understand caveats | [Scope and limitations](#scope-and-limitations) |

## License

This project is distributed under the MIT License. See the repository's license file for details.
