Metadata-Version: 2.4
Name: valencity
Version: 0.1.0
Summary: A privacy & ML safety toolkit for production ML pipelines
Project-URL: Homepage, https://github.com/ihakawati/valencity
Project-URL: Documentation, https://github.com/ihakawati/valencity#readme
Project-URL: Repository, https://github.com/ihakawati/valencity
Project-URL: Issues, https://github.com/ihakawati/valencity/issues
Author-email: Abdallah El-Hakawati <abdallahelhakawati@gmail.com>
License-Expression: MIT
License-File: LICENSE
Keywords: data-leakage,data-validation,machine-learning,ml-safety,pii,privacy
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Security
Requires-Python: >=3.9
Requires-Dist: faker>=10.0.0
Requires-Dist: jinja2>=3.0.0
Requires-Dist: mkdocs-material>=9.0.0
Requires-Dist: mkdocs>=1.4.0
Requires-Dist: numpy>=1.21.0
Requires-Dist: pandas>=1.5.0
Requires-Dist: rich>=13.0.0
Requires-Dist: scikit-learn>=1.0.0
Requires-Dist: scipy>=1.7.0
Requires-Dist: typer>=0.9.0
Provides-Extra: all
Requires-Dist: mypy>=1.0.0; extra == 'all'
Requires-Dist: pytest-cov>=4.0.0; extra == 'all'
Requires-Dist: pytest>=7.0.0; extra == 'all'
Requires-Dist: ruff>=0.1.0; extra == 'all'
Requires-Dist: spacy>=3.0.0; extra == 'all'
Provides-Extra: dev
Requires-Dist: mypy>=1.0.0; extra == 'dev'
Requires-Dist: pytest-cov>=4.0.0; extra == 'dev'
Requires-Dist: pytest>=7.0.0; extra == 'dev'
Requires-Dist: ruff>=0.1.0; extra == 'dev'
Provides-Extra: nlp
Requires-Dist: spacy>=3.0.0; extra == 'nlp'
Description-Content-Type: text/markdown

<div align="center">
 
# 🛡️ valencity

**The ML Safety Fortress**  
*Privacy Engineering • Data Validation • Leakage Prevention*

[![PyPI version](https://img.shields.io/pypi/v/valencity.svg?color=blue)](https://pypi.org/project/valencity)
[![Python versions](https://img.shields.io/pypi/pyversions/valencity.svg)](https://pypi.org/project/valencity)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
[![Codecov](https://img.shields.io/codecov/c/github/ihakawati/valencity)](https://codecov.io/gh/ihakawati/valencity)
[![Downloads](https://static.pepy.tech/badge/valencity)](https://pepy.tech/project/valencity)

<p align="center">
  <a href="#features">Features</a> •
  <a href="#installation">Installation</a> •
  <a href="#quick-start">Quick Start</a> •
  <a href="#documentation">Documentation</a> •
  <a href="#contributing">Contributing</a>
</p>

</div>

---

## 🚀 Why valencity?

valencity is a comprehensive toolkit designed to make machine learning pipelines **safer**, **compliant**, and **reliable**. It bridges the gap between data engineering and privacy compliance, offering a "fortress" of protection for your data assets.

| Feature | valencity 🛡️ | Great Expectations | Pandera | Custom Scripts |
|---------|:----------:|:------------------:|:-------:|:--------------:|
| **PII Detection** | ✅ 50+ Patterns | ❌ | ❌ | ⚠️ Manual |
| **Masking/Anonymization** | ✅ K-Anon, DP | ❌ | ❌ | ❌ |
| **Leakage Prevention** | ✅ Full Suite | ❌ | ❌ | ❌ |
| **Drift Detection** | ✅ 4+ Methods | ✅ | ❌ | ⚠️ Manual |
| **Safe Cross-Validation** | ✅ Built-in | ❌ | ❌ | ❌ |
| **HTML Reports** | ✅ Beautiful | ✅ | ❌ | ❌ |

---

## ✨ Features

### 🕵️ PII Detection & Anonymization
- Detect **50+ types of PII** including IBAN, Passports, and API Keys.
- **Smart Masking**: Redact, hash, partial mask, or replace with fake data.
- **NLP Support**: Named Entity Recognition for names and addresses (via spaCy).

### ✅ Data Validation & Quality
- **Drift Detection**: Catch distribution shifts before they break your model.
- **Schema Validation**: Enforce strict data contracts on your DFs.
- **Quality Checks**: Nulls, duplicates, outliers, and cardinality.

### 🚫 Leakage Prevention
- **Safe Cross-Validation**: Automatic leakage checks during CV.
- **Leakage Detectors**: Find target leakage, train/test overlap, and temporal leakage.
- **Safe Splitters**: Time-series and group-aware train/test splitting.

---

## 📦 Installation

```bash
# Install core package
pip install valencity

# Install with NLP support (for names/addresses)
pip install valencity[nlp]
python -m spacy download en_core_web_sm
```

---

## ⚡ Quick Start

### 1. Protect Your Data (PII)

```python
import pandas as pd
from valencity.pii import PIIDetector, PIIMasker

# Scan for PII
df = pd.read_csv("users.csv")
detector = PIIDetector()
report = detector.scan_dataframe(df)

if report.has_pii:
    print(f"⚠️ Found PII in columns: {report.pii_columns}")

# Mask it
masker = PIIMasker(strategy="partial")  # j***@example.com
safe_df = masker.mask_dataframe(df)
```

### 2. Validate Quality & Drift

```python
from valencity.validation import DriftDetector, DataSchema

# 1. Define expectations
schema = DataSchema.from_dataframe(train_df)

# 2. Check for drift in production
detector = DriftDetector(method="ks_test")
detector.fit(reference_df=train_df)

drift_report = detector.detect(current_df=new_data)
if drift_report.has_drift:
    print(f"🚨 Data drift detected using {drift_report.method}!")
```

### 3. Prevent Model Leakage

```python
from valencity.leakage import SafeCrossValidator
from sklearn.linear_model import LogisticRegression

# Safe CV ensures preprocessing only sees training fold
# Prevents subtle data leakage bugs
cv = SafeCrossValidator(n_splits=5, preprocessor=StandardScaler())

scores = cv.cross_val_score(LogisticRegression(), X, y)
print(f"Realistic Accuracy: {scores.mean():.4f}")
```

---

## 🗺️ Roadmap

- [ ] **CLI Tool**: `valencity scan data.csv`
- [ ] **Privacy Module**: Differential Privacy & K-Anonymity
- [ ] **Synthetic Data**: Generate safe, realistic test data
- [ ] **Integrations**: MLflow, Airflow, and dbt support

---

## 🤝 Contributing

We welcome contributions! Please check out the [Issues](https://github.com/ihakawati/valencity/issues) page.

1. Fork the repo
2. Create your feature branch (`git checkout -b feature/amazing-feature`)
3. Commit your changes (`git commit -m 'Add amazing feature'`)
4. Push to the branch (`git push origin feature/amazing-feature`)
5. Open a Pull Request

---

<div align="center">

**Built with ❤️ for the ML Community**

</div>
