Metadata-Version: 2.4
Name: explainer-ai
Version: 0.4.2
Summary: ML model evaluation and monitoring framework with explainability, fairness analysis, and Superset dashboards
Author: Cognome Inc
License: Copyright (c) 2026 Cognome Inc. All rights reserved.
        
        This software and associated documentation files (the "Software") may not be
        copied, modified, merged, published, distributed, sublicensed, or sold without
        the prior written permission of Cognome Inc.
        
        The Software is provided "as is", without warranty of any kind, express or
        implied, including but not limited to the warranties of merchantability, fitness
        for a particular purpose, and noninfringement. In no event shall the authors or
        copyright holders be liable for any claim, damages, or other liability, whether
        in an action of contract, tort, or otherwise, arising from, out of, or in
        connection with the Software or the use or other dealings in the Software.
        
Project-URL: Homepage, https://github.com/Cognome-Inc/tapestry
Project-URL: Repository, https://github.com/Cognome-Inc/tapestry
Project-URL: Issues, https://github.com/Cognome-Inc/tapestry/issues
Keywords: ml,model-monitoring,evaluation,explainability,fairness,llm,superset
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Software Development :: Quality Assurance
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: alembic>=1.13.0
Requires-Dist: jupyter>=1.1.1
Requires-Dist: litellm>=1.67.0
Requires-Dist: notebook>=7.4.0
Requires-Dist: numpy>=2.0.2
Requires-Dist: pandas>=2.2.3
Requires-Dist: pandera>=0.24.0
Requires-Dist: psycopg2-binary>=2.9.10
Requires-Dist: pydantic>=2.11.3
Requires-Dist: python-dotenv>=1.1.0
Requires-Dist: rich>=13.0.0
Requires-Dist: scipy>=1.11.0
Requires-Dist: sqlalchemy>=2.0.40
Requires-Dist: typer>=0.12.0
Provides-Extra: dev
Requires-Dist: pytest>=8.4.0; extra == "dev"
Provides-Extra: dagster
Requires-Dist: dagster>=1.10.11; extra == "dagster"
Provides-Extra: traditional
Requires-Dist: xgboost[cpu]; extra == "traditional"
Requires-Dist: shap>=0.47.2; extra == "traditional"
Provides-Extra: observability
Requires-Dist: opentelemetry-api>=1.20.0; extra == "observability"
Requires-Dist: opentelemetry-sdk>=1.20.0; extra == "observability"
Requires-Dist: opentelemetry-exporter-otlp>=1.20.0; extra == "observability"
Provides-Extra: superset
Requires-Dist: requests>=2.31.0; extra == "superset"
Requires-Dist: tomli>=2.0.0; python_version < "3.11" and extra == "superset"
Provides-Extra: api
Requires-Dist: fastapi>=0.115.0; extra == "api"
Requires-Dist: uvicorn[standard]>=0.30.0; extra == "api"
Provides-Extra: full
Requires-Dist: dagster>=1.10.11; extra == "full"
Requires-Dist: xgboost[cpu]; extra == "full"
Requires-Dist: shap>=0.47.2; extra == "full"
Requires-Dist: opentelemetry-api>=1.20.0; extra == "full"
Requires-Dist: opentelemetry-sdk>=1.20.0; extra == "full"
Requires-Dist: opentelemetry-exporter-otlp>=1.20.0; extra == "full"
Requires-Dist: requests>=2.31.0; extra == "full"
Requires-Dist: tomli>=2.0.0; python_version < "3.11" and extra == "full"
Requires-Dist: fastapi>=0.115.0; extra == "full"
Requires-Dist: uvicorn[standard]>=0.30.0; extra == "full"
Dynamic: license-file

# Tapestry

An ML model evaluation and governance framework for production pipelines. Monitor traditional ML and generative AI systems with automated dashboarding, alerting, ground truth tracking, and a full model registry.

```
pip install explainer-ai
```

## What It Does

Tapestry sits between your ML pipeline and your stakeholders. Models push predictions through Tapestry, which runs evaluators, tracks performance over time, fires alerts when things degrade, and renders governance dashboards — all backed by PostgreSQL.

```
ML Pipeline ──> Tapestry Evaluators ──> Database ──> Dashboards / Alerts
                     │                                     │
                     ├── Confidence, drift, SHAP            ├── Control Tower
                     ├── Hallucination, PHI, bias           ├── Per-model dashboards
                     ├── Ground truth matching               └── Slack/email alerts
                     └── Demographic fairness
```

## Try It in 60 Seconds

No model needed — this generates synthetic data so you can see Tapestry work end-to-end.

```bash
# 1. Install and start PostgreSQL
pip install explainer-ai
tapestry init   # creates docker-compose.yml, then:
docker compose up -d
tapestry init --engine "postgresql://tapestry:tapestry@localhost:5433/tapestry"

# 2. Register a demo asset with evaluators
cat > demo_asset.json << 'EOF'
{
  "asset_key": "demo_classifier",
  "evaluator_type": "traditional",
  "identifier_columns": ["id"],
  "config": {
    "custom_evaluators": [
      {"name": "confidence-analysis", "arguments": {
        "predicted_class_column": "prediction",
        "confidence_score_column": "confidence",
        "model_id": "demo-v1", "threshold": 0.7
      }},
      {"name": "class-distribution", "arguments": {
        "predicted_class_column": "prediction",
        "confidence_score_column": "confidence",
        "model_id": "demo-v1"
      }}
    ]
  },
  "description": "Demo classifier to try out Tapestry"
}
EOF
tapestry asset register -f demo_asset.json

# 3. Generate some fake predictions and evaluate them
python -c "
import pandas as pd, numpy as np
np.random.seed(42)
df = pd.DataFrame({
    'id': range(100),
    'prediction': np.random.choice(['positive', 'negative'], 100),
    'confidence': np.round(np.random.beta(5, 2, 100), 3),
})
df.to_csv('demo_predictions.csv', index=False)
print(f'Generated {len(df)} predictions')
"
tapestry evaluate demo_classifier --file demo_predictions.csv

# 4. Check results
tapestry db status
```

You should see evaluation results written to the database. From here you can explore the REST API (`tapestry serve`), set up alerts, or build dashboards.

## Getting Started (with your own models)

### 1. Start PostgreSQL and initialize

The included Docker Compose starts both PostgreSQL and Apache Superset:

```bash
docker compose up -d
```

Then initialize Tapestry:

```bash
tapestry init --engine "postgresql://tapestry:tapestry@localhost:5433/tapestry"
```

This creates a fully-commented `.tapestry.toml` config file, initializes the database schema, and syncs evaluator definitions. Or just run `tapestry init` — it will offer to create the `docker-compose.yml` for you.

Superset is available at [http://localhost:8088](http://localhost:8088) (login: `admin` / `admin`).

If you have an existing PostgreSQL and Superset, pass your own connection strings instead.

### 2. Register your first model

The interactive wizard walks you through evaluator selection and optional governance metadata:

```bash
tapestry asset init
```

Or register from a JSON config file:

```bash
tapestry asset register -f my_model.json
```

### 3. Push data for evaluation

Via the REST API:

```bash
tapestry serve start

curl -X POST http://localhost:8084/api/v1/assets/my_model/ingest \
  -H "Content-Type: application/json" \
  -d '{"data": [{"record_id": "1", "pred": "A", "score": 0.95}]}'
```

Or evaluate a local file directly:

```bash
tapestry evaluate my_model --file predictions.csv
```

### 4. Track performance

```bash
tapestry asset list                          # see all registered models
tapestry ground-truth status my_model        # check ground truth coverage
tapestry alerts check                        # evaluate alert rules
tapestry dashboard create-control-tower      # build governance dashboard
```

## Features

### Evaluation

- **Traditional ML** — confidence analysis, SHAP explainability, data drift (KS test), class distribution, entropy, run-over-run comparison, model validation from verified outcomes
- **Generative AI** — LLM-based judges for hallucination, fairness/bias, PHI detection, factual accuracy, traceability, prompt injection, audit readiness
- **Demographic fairness** — per-group pass rates, statistical fairness tests, and CHAI-aligned ground truth metrics (TPR, FPR, AUC, Brier score per subgroup)
- **Evaluation dimensions** — each evaluator maps to a dimension (performance, explainability, drift, stability, fairness, faithfulness, safety, compliance) for organized reporting

### Ground Truth Monitoring

Predictions are logged automatically. Verified outcomes can arrive hours, days, or weeks later — Tapestry matches them and computes accuracy, F1, precision, recall, MAE, RMSE, and R² as ground truth becomes available.

```bash
tapestry ground-truth ingest my_model \
  --file reviewed_outcomes.csv \
  --ground-truth-column actual_label
```

### Model Registry & Governance

Track model inventory, ownership, lifecycle, and impact commitments:

```bash
tapestry asset completeness                  # governance coverage scores
tapestry asset show-impact my_model          # ROI tracking vs. targets
```

Each asset can include inventory metadata (AI type, risk tier, deployment environment), ownership (model owner, clinical sponsor, escalation contact), lifecycle dates, and impact commitments with baseline/target values.

### Alerting

Rule-based alerts on evaluation metrics with email, Slack, and webhook delivery:

```toml
# .tapestry.toml
[[alerts.rules]]
name = "low_pass_rate"
condition = "pass_rate_below"
threshold = 80.0
severity = "P1"
asset_key = "my_model"
channels = ["slack", "email"]
```

Available conditions: `pass_rate_below`, `drift_score_above`, `staleness_days_above`, `ground_truth_coverage_below`, `model_performance_below`, `failure_count_above`, `consecutive_failures`.

### Dashboards

Automated Apache Superset dashboards:

```bash
tapestry dashboard create my_model           # per-asset dashboard
tapestry dashboard create-control-tower      # fleet-wide governance
tapestry dashboard create-comparison         # multi-model comparison
tapestry dashboard create-generative         # cross-asset LLM evaluation
```

The **Control Tower** provides a single view of your entire AI portfolio: overall pass rates, model quality scores, compliance status, drift trends, staleness tracking, alert history, and impact/ROI tracking.

### REST API

Full API for programmatic access (interactive docs at `/docs`):

```bash
tapestry serve start                         # start in background

# Asset management
curl http://localhost:8084/api/v1/assets/
curl http://localhost:8084/api/v1/assets/my_model/health

# Data ingestion
curl -X POST http://localhost:8084/api/v1/assets/my_model/ingest \
  -H "Content-Type: application/json" \
  -d '{"data": [...]}'

# Ground truth
curl -X POST http://localhost:8084/api/v1/ground-truth \
  -H "Content-Type: application/json" \
  -d '{"asset_key": "my_model", "records": [...]}'

# Coverage & metrics
curl http://localhost:8084/api/v1/ground-truth/coverage?asset_key=my_model
curl http://localhost:8084/api/v1/metrics/summary?asset_key=my_model
```

## Installation

```bash
pip install explainer-ai                     # core
pip install "explainer-ai[api]"                # REST API (FastAPI + uvicorn)
pip install "explainer-ai[dagster]"            # Dagster asset checks
pip install "explainer-ai[superset]"           # Superset dashboard support
pip install "explainer-ai[full]"               # everything
```

### From source

```bash
git clone https://github.com/Cognome-Inc/tapestry.git
cd tapestry
uv sync --dev --extra full --group traditional
```

## Code Example

```python
import pandas as pd
from tapestry.config import TapestryConfig
from tapestry.generative.tapestry import GenerativeTapestry
from tapestry import TapestryDB

# Sample LLM outputs to evaluate
df = pd.DataFrame({
    "record_id": [1, 2, 3],
    "context": [
        "The patient is a 55-year-old male with hypertension.",
        "Lab results show A1c of 7.2%, poorly controlled diabetes.",
        "The patient reported no known drug allergies.",
    ],
    "raw_model_response": [
        "The patient is a 55-year-old male with hypertension.",       # faithful
        "Lab results show A1c of 5.0%, excellent control.",           # hallucinated
        "The patient has a severe allergy to penicillin.",            # hallucinated
    ],
})

# Configure evaluators
config = TapestryConfig(
    identifier=["record_id"],
    asset_key="clinical_qa",
    custom_evaluators=[
        {"name": "hallucination", "arguments": {"model": "gpt-4o-mini"}},
        {"name": "phi_detection", "arguments": {"model": "gpt-4o-mini"}},
    ],
)

# Run evaluation
evaluator = GenerativeTapestry(config=config, backend="litellm")
results = evaluator.run(df)

# Write to database
db = TapestryDB.from_config()
evaluator.write_evaluators_to_db(results, db)
```

For local models, use any litellm-supported provider: `{"model": "ollama/llama3", "api_base": "http://localhost:11434"}`.

## CLI Reference

| Command | Description |
|---------|-------------|
| **Setup** | |
| `tapestry init` | Guided first-time setup (config, DB, evaluators) |
| `tapestry doctor` | Check project health (DB connection, config, etc.) |
| **Assets** | |
| `tapestry asset init` | Interactive wizard to create an asset config |
| `tapestry asset register -f <file>` | Register assets from JSON file or directory |
| `tapestry asset list` | List all registered assets |
| `tapestry asset show <key>` | Show full config and governance details |
| `tapestry asset completeness` | Show governance completeness scores |
| `tapestry asset show-impact <key>` | Show impact commitments with ROI status |
| **Evaluation** | |
| `tapestry evaluate <asset> --file <csv>` | Run evaluators on a local CSV |
| `tapestry serve [start\|stop\|status\|logs]` | REST API server management |
| **Ground Truth** | |
| `tapestry ground-truth ingest <asset>` | Match ground truth to predictions |
| `tapestry ground-truth status <asset>` | Show coverage and match latency |
| **Alerts** | |
| `tapestry alerts list` | Show configured alert rules |
| `tapestry alerts check` | Evaluate all rules and fire notifications |
| `tapestry alerts history` | Show recent alert events |
| **Dashboards** | |
| `tapestry dashboard create <asset>` | Per-asset Superset dashboard |
| `tapestry dashboard create-control-tower` | Governance overview dashboard |
| `tapestry dashboard create-comparison` | Multi-model comparison dashboard |
| **Database** | |
| `tapestry db init` | Create schema and run migrations |
| `tapestry db status` | Show tables and row counts |
| `tapestry db reset --force` | Drop and recreate schema |
| `tapestry migrate upgrade head` | Run pending database migrations |

All commands accept `--schema`/`-s` and `--engine`/`-e` overrides, or read from `.tapestry.toml` / environment variables.

## Project Structure

```
src/tapestry/
    api/            # FastAPI REST API (routers, schemas, dependencies)
    alerts/         # Rule-based alert engine and delivery (email, Slack, webhook)
    cli/            # CLI commands (init, serve, asset, evaluate, ground-truth, etc.)
    core/           # Base classes, URI generation, demographics, retry logic
    db/             # SQLAlchemy models, migrations, schema management
    generative/     # LLM-based evaluators (litellm backend)
    models/         # Pydantic models for model registry / inventory
    monitoring/     # Ground truth metrics computation
    observability/  # OpenTelemetry integration
    superset/       # Dashboard builders, chart/view specs, Control Tower
    traditional/    # XGBoost, SHAP, drift, confidence evaluators
    config.py       # TapestryConfig, EvaluatorConfig, ScoreConfig
    registry.py     # Evaluator plugin registry with dimension mapping
```

## Development

```bash
git clone https://github.com/Cognome-Inc/tapestry.git
cd tapestry
uv sync --dev --extra full --group traditional

uv run pytest                               # run tests
uv run black --check src/ tests/            # format check
uv run ruff check src/ tests/               # lint
uv run mypy src/tapestry/                   # type check
```

## Documentation

Full docs are available in [`docs/`](docs/):

- [Installation](docs/getting-started/installation.md)
- [Quickstart](docs/getting-started/quickstart.md)
- [Core Concepts](docs/getting-started/concepts.md)
- [Configuration](docs/configuration/tapestry-config.md)
- [Asset Registration](docs/rest-api/asset-registration.md)
- [REST API](docs/rest-api/overview.md)
- [Generative Evaluators](docs/evaluators/generative/index.md)
- [Traditional Evaluators](docs/evaluators/traditional/index.md)
- [Evaluation Dimensions](docs/evaluators/dimensions.md)
- [Ground Truth Monitoring](docs/integrations/ground-truth.md)
- [Model Registry](docs/integrations/model-registry.md)
- [Impact Tracking & ROI](docs/integrations/impact-tracking.md)
- [Superset Dashboards](docs/integrations/superset.md)
- [Alerting](docs/integrations/alerting.md)
- [Dagster Integration](docs/integrations/dagster.md)
- [Database Schema](docs/integrations/database.md)

## License

See [LICENSE](LICENSE) for details.
