Metadata-Version: 2.1
Name: xai-auditor
Version: 0.1.0
Summary: Quantitative Explainability Auditing and Model Risk Management for Machine Learning
Author: Explainability Auditor Team
License: Apache-2.0
Project-URL: Homepage, https://github.com/cleanpigg/xai-auditor
Project-URL: Documentation, https://github.com/cleanpigg/xai-auditor#readme
Project-URL: Bug Tracker, https://github.com/cleanpigg/xai-auditor/issues
Keywords: xai,explainable-ai,shap,lime,model-governance,model-risk-management,responsible-ai,auditing
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Science/Research
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.8
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Software Development :: Quality Assurance
Requires-Python: >=3.8
Description-Content-Type: text/markdown
Provides-Extra: all
Provides-Extra: dev
Requires-Dist: numpy >=1.20.0 ; extra == "all"
Requires-Dist: pandas >=1.3.0 ; extra == "all"
Requires-Dist: scikit-learn >=1.0.0 ; extra == "all"
Requires-Dist: scipy >=1.7.0 ; extra == "all"
Requires-Dist: shap >=0.40.0 ; extra == "all"
Requires-Dist: lime >=0.2.0 ; extra == "all"
Requires-Dist: pytest >=7.0.0 ; extra == "dev"
Requires-Dist: pytest-cov >=4.0.0 ; extra == "dev"
Requires-Dist: flake8 >=5.0.0 ; extra == "dev"
Requires-Dist: mypy >=1.0.0 ; extra == "dev"

# xai-auditor

Quantitative Explainability Auditing and Model Risk Management for Machine Learning.

`xai-auditor` is an independent, production-grade Python library designed to audit the reliability, stability, fidelity, consistency, manipulation resistance, distributional drift, and operational coverage of post-hoc Explainable AI (XAI) pipelines such as SHAP and LIME.

---

## Overview

In regulated and high-stakes machine learning applications, post-hoc explanations must be audited with the same mathematical rigor applied to model predictive accuracy. `xai-auditor` evaluates whether surrogate explainers produce faithful, robust, and dependable explanations across six core dimensions:

1. **Local Lipschitz Stability**: Attribution invariance and rank preservation under small feature perturbations.
2. **Attribution Fidelity**: Empirical causal validity measured via ROAR (Remove and Retain) and KAR (Keep and Retain) probability curves.
3. **Cross-Method Explainer Consistency**: Consensus between SHAP and LIME evaluated via Top-K Jaccard overlap, Spearman rank correlation, Sign Agreement, Cosine Similarity, and Rank-Biased Overlap (RBO).
4. **Adversarial Manipulation Resistance**: Robustness against explainer hyperparameter tuning (kernel widths, sampling budgets, background reference sets) measured by the Explanation Manipulation Index (EMI).
5. **Distributional Explanation Drift**: Temporal and cohort attribution shift evaluated via Population Stability Index (PSI) and Wasserstein Distance.
6. **Operational Coverage & Latency**: Service level agreement (SLA) verification evaluating explanation success rates and p50/p95/p99 execution latency.

---

## Installation

```bash
pip install xai-auditor
```

Optional dependencies for deep integration with NumPy, Pandas, Scikit-Learn, SHAP, and LIME:

```bash
pip install xai-auditor[all]
```

---

## Quickstart

```python
from xai_auditor import Auditor
from xai_auditor.models.builtins import BuiltinLogisticRegression

# 1. Prepare data and model
# X can be a list of lists, a numpy array, or a pandas DataFrame
X = [
    [0.2, 1.1, 0.4, 0.9],
    [0.8, 0.3, 0.1, 0.2],
    [0.5, 0.9, 0.7, 0.6],
    [0.1, 0.2, 0.8, 0.4],
    [0.9, 1.4, 0.3, 0.8],
    [0.3, 0.5, 0.2, 0.1],
]
y = [1, 0, 1, 0, 1, 0]
feature_names = ["credit_utilization", "annual_income", "debt_ratio", "payment_history"]

model = BuiltinLogisticRegression(weights=[0.8, -0.6, 1.2, -0.9], bias=0.1)

# 2. Run the audit
auditor = Auditor(
    model=model,
    X=X,
    y=y,
    feature_names=feature_names,
)
result = auditor.run()

# 3. Inspect scores, risk tier, and metrics
print(f"Explainability Trust Score: {result.trust_score}/100")
print(f"Risk Classification Tier:   {result.risk_tier}")
print(f"Regulatory Approval Status: {result.approval_status}")

# 4. Access individual dimension scores
for dim, dim_result in result.dimension_results.items():
    print(f" - {dim_result.name}: {dim_result.score}/100 (Weight: {int(dim_result.weight * 100)}%)")

# 5. Export results to JSON or HTML
json_str = result.to_json(indent=2)
result.save_html_report("audit_report.html")
```

---

## Auditing Individual Dimensions

Users can run individual audit dimensions independently without running the full suite:

```python
from xai_auditor import (
    StabilityAuditor,
    FidelityAuditor,
    ConsistencyAuditor,
    ManipulationAuditor,
    DriftAuditor,
    CoverageAuditor,
)

# Example: Run only cross-method consistency (SHAP vs LIME)
consistency_auditor = ConsistencyAuditor(top_k=3, rbo_p=0.9)
res = consistency_auditor.audit(model=model, X=X, feature_names=feature_names)

print(f"Consistency Score: {res.score}/100")
print(f"Top-K Jaccard:    {res.metrics['top_k_jaccard']}")
print(f"Sign Agreement:   {res.metrics['sign_agreement']}")
print(f"Rank-Biased RBO:  {res.metrics['rank_biased_overlap']}")
```

---

## Supported Models and Explainers

### Models
- Scikit-Learn classifiers implementing `predict_proba`
- XGBoost, LightGBM, CatBoost binary classifiers
- PyTorch / TensorFlow binary classification modules
- Built-in standalone reference models (XGBoost emulation, Random Forest, Logistic Regression, Multilayer Perceptron)
- Arbitrary Python prediction callables: `f(X) -> probabilities`

### Explainers
- **SHAP**: Permutation Shapley engine with efficiency constraint enforcement
- **LIME**: Kernel-weighted local linear ridge surrogate with exponential distance weighting

---

## Scoring and Risk Classification

The composite **Explainability Trust Score** (0-100) aggregates all six dimensions using established regulatory weights:

| Audit Dimension | Weight | Primary Testing Objective |
| :--- | :--- | :--- |
| **Attribution Fidelity (ROAR/KAR)** | 25% | Causal accuracy of feature attribution under ablation |
| **Local Lipschitz Stability** | 20% | Invariance under input feature perturbation noise |
| **Cross-Method Consistency** | 15% | Multi-explainer consensus (SHAP vs LIME) including RBO |
| **Manipulation Resistance** | 15% | Robustness against hyperparameter cherry-picking (EMI) |
| **Distributional Explanation Drift** | 15% | Temporal attribution stability (PSI & Wasserstein) |
| **Operational Coverage & Latency** | 10% | Generation success rate and tail SLA execution latency |

### Risk Tiers
- **Low Risk (85 - 100)**: Approved for unconditional production deployment.
- **Medium Risk (75 - 84)**: Approved with standard governance telemetry.
- **High Risk (60 - 74)**: Conditional approval with mandatory secondary review.
- **Critical Risk (0 - 59)**: Deployment rejected; hard re-audit mandated.

---

## Command-Line Interface (CLI)

`xai-auditor` provides a command-line tool `xai-audit` for batch auditing and continuous integration:

```bash
xai-audit --data data.csv --target is_default --model logistic_regression --output report.json --html report.html
```

---

## License

Apache License 2.0.

