Metadata-Version: 2.4
Name: explainer-ai
Version: 0.4.1
Summary: ML model evaluation and monitoring framework with explainability, fairness analysis, and Superset dashboards
Author: Cognome Inc
License: Copyright (c) 2026 Cognome Inc. All rights reserved.
        
        This software and associated documentation files (the "Software") may not be
        copied, modified, merged, published, distributed, sublicensed, or sold without
        the prior written permission of Cognome Inc.
        
        The Software is provided "as is", without warranty of any kind, express or
        implied, including but not limited to the warranties of merchantability, fitness
        for a particular purpose, and noninfringement. In no event shall the authors or
        copyright holders be liable for any claim, damages, or other liability, whether
        in an action of contract, tort, or otherwise, arising from, out of, or in
        connection with the Software or the use or other dealings in the Software.
        
Project-URL: Homepage, https://github.com/Cognome-Inc/tapestry
Project-URL: Repository, https://github.com/Cognome-Inc/tapestry
Project-URL: Issues, https://github.com/Cognome-Inc/tapestry/issues
Keywords: ml,model-monitoring,evaluation,explainability,fairness,llm,superset
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Software Development :: Quality Assurance
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: alembic>=1.13.0
Requires-Dist: jupyter>=1.1.1
Requires-Dist: litellm>=1.67.0
Requires-Dist: notebook>=7.4.0
Requires-Dist: numpy>=2.0.2
Requires-Dist: pandas>=2.2.3
Requires-Dist: pandera>=0.24.0
Requires-Dist: psycopg2-binary>=2.9.10
Requires-Dist: pydantic>=2.11.3
Requires-Dist: python-dotenv>=1.1.0
Requires-Dist: rich>=13.0.0
Requires-Dist: scipy>=1.11.0
Requires-Dist: sqlalchemy>=2.0.40
Requires-Dist: typer>=0.12.0
Provides-Extra: dev
Requires-Dist: pytest>=8.4.0; extra == "dev"
Provides-Extra: dagster
Requires-Dist: dagster>=1.10.11; extra == "dagster"
Provides-Extra: traditional
Requires-Dist: xgboost[cpu]; extra == "traditional"
Requires-Dist: shap>=0.47.2; extra == "traditional"
Provides-Extra: observability
Requires-Dist: opentelemetry-api>=1.20.0; extra == "observability"
Requires-Dist: opentelemetry-sdk>=1.20.0; extra == "observability"
Requires-Dist: opentelemetry-exporter-otlp>=1.20.0; extra == "observability"
Provides-Extra: superset
Requires-Dist: requests>=2.31.0; extra == "superset"
Requires-Dist: tomli>=2.0.0; python_version < "3.11" and extra == "superset"
Provides-Extra: api
Requires-Dist: fastapi>=0.115.0; extra == "api"
Requires-Dist: uvicorn[standard]>=0.30.0; extra == "api"
Provides-Extra: full
Requires-Dist: dagster>=1.10.11; extra == "full"
Requires-Dist: xgboost[cpu]; extra == "full"
Requires-Dist: shap>=0.47.2; extra == "full"
Requires-Dist: opentelemetry-api>=1.20.0; extra == "full"
Requires-Dist: opentelemetry-sdk>=1.20.0; extra == "full"
Requires-Dist: opentelemetry-exporter-otlp>=1.20.0; extra == "full"
Requires-Dist: requests>=2.31.0; extra == "full"
Requires-Dist: tomli>=2.0.0; python_version < "3.11" and extra == "full"
Requires-Dist: fastapi>=0.115.0; extra == "full"
Requires-Dist: uvicorn[standard]>=0.30.0; extra == "full"
Dynamic: license-file

# Tapestry

An ML model evaluation and governance framework for production pipelines. Monitor traditional ML and generative AI systems with automated dashboarding, alerting, ground truth tracking, and a full model registry.

```
pip install explainer-ai
```

## What It Does

Tapestry sits between your ML pipeline and your stakeholders. Models push predictions through Tapestry, which runs evaluators, tracks performance over time, fires alerts when things degrade, and renders governance dashboards — all backed by PostgreSQL.

```
ML Pipeline ──> Tapestry Evaluators ──> Database ──> Dashboards / Alerts
                     │                                     │
                     ├── Confidence, drift, SHAP            ├── Control Tower
                     ├── Hallucination, PHI, bias           ├── Per-model dashboards
                     ├── Ground truth matching               └── Slack/email alerts
                     └── Demographic fairness
```

## Getting Started

### 1. Start PostgreSQL and initialize

If you don't have a PostgreSQL instance, use the included Docker Compose:

```bash
docker compose up -d
```

Then initialize Tapestry:

```bash
tapestry init --engine "postgresql://tapestry:tapestry@localhost:5432/tapestry"
```

This creates a fully-commented `.tapestry.toml` config file, initializes the database schema, and syncs evaluator definitions. If you have an existing PostgreSQL server, pass your own connection string instead.

### 2. Register your first model

The interactive wizard walks you through evaluator selection and optional governance metadata:

```bash
tapestry asset init
```

Or register from a JSON config file:

```bash
tapestry asset register -f my_model.json
```

### 3. Push data for evaluation

Via the REST API:

```bash
tapestry serve start

curl -X POST http://localhost:8084/api/v1/assets/my_model/ingest \
  -H "Content-Type: application/json" \
  -d '{"data": [{"record_id": "1", "pred": "A", "score": 0.95}]}'
```

Or evaluate a local file directly:

```bash
tapestry evaluate my_model --file predictions.csv
```

### 4. Track performance

```bash
tapestry asset list                          # see all registered models
tapestry ground-truth status my_model        # check ground truth coverage
tapestry alerts check                        # evaluate alert rules
tapestry dashboard create-control-tower      # build governance dashboard
```

## Features

### Evaluation

- **Traditional ML** — confidence analysis, SHAP explainability, data drift (KS test), class distribution, entropy, run-over-run comparison, model validation from verified outcomes
- **Generative AI** — LLM-based judges for hallucination, fairness/bias, PHI detection, factual accuracy, traceability, prompt injection, audit readiness
- **Demographic fairness** — per-group pass rates, statistical fairness tests, and CHAI-aligned ground truth metrics (TPR, FPR, AUC, Brier score per subgroup)
- **Evaluation dimensions** — each evaluator maps to a dimension (performance, explainability, drift, stability, fairness, faithfulness, safety, compliance) for organized reporting

### Ground Truth Monitoring

Predictions are logged automatically. Verified outcomes can arrive hours, days, or weeks later — Tapestry matches them and computes accuracy, F1, precision, recall, MAE, RMSE, and R² as ground truth becomes available.

```bash
tapestry ground-truth ingest my_model \
  --file reviewed_outcomes.csv \
  --ground-truth-column actual_label
```

### Model Registry & Governance

Track model inventory, ownership, lifecycle, and impact commitments:

```bash
tapestry asset completeness                  # governance coverage scores
tapestry asset show-impact my_model          # ROI tracking vs. targets
```

Each asset can include inventory metadata (AI type, risk tier, deployment environment), ownership (model owner, clinical sponsor, escalation contact), lifecycle dates, and impact commitments with baseline/target values.

### Alerting

Rule-based alerts on evaluation metrics with email, Slack, and webhook delivery:

```toml
# .tapestry.toml
[[alerts.rules]]
name = "low_pass_rate"
condition = "pass_rate_below"
threshold = 80.0
severity = "P1"
asset_key = "my_model"
channels = ["slack", "email"]
```

Available conditions: `pass_rate_below`, `drift_score_above`, `staleness_days_above`, `ground_truth_coverage_below`, `model_performance_below`, `failure_count_above`, `consecutive_failures`.

### Dashboards

Automated Apache Superset dashboards:

```bash
tapestry dashboard create my_model           # per-asset dashboard
tapestry dashboard create-control-tower      # fleet-wide governance
tapestry dashboard create-comparison         # multi-model comparison
tapestry dashboard create-generative         # cross-asset LLM evaluation
```

The **Control Tower** provides a single view of your entire AI portfolio: overall pass rates, model quality scores, compliance status, drift trends, staleness tracking, alert history, and impact/ROI tracking.

### REST API

Full API for programmatic access (interactive docs at `/docs`):

```bash
tapestry serve start                         # start in background

# Asset management
curl http://localhost:8084/api/v1/assets/
curl http://localhost:8084/api/v1/assets/my_model/health

# Data ingestion
curl -X POST http://localhost:8084/api/v1/assets/my_model/ingest \
  -H "Content-Type: application/json" \
  -d '{"data": [...]}'

# Ground truth
curl -X POST http://localhost:8084/api/v1/ground-truth \
  -H "Content-Type: application/json" \
  -d '{"asset_key": "my_model", "records": [...]}'

# Coverage & metrics
curl http://localhost:8084/api/v1/ground-truth/coverage?asset_key=my_model
curl http://localhost:8084/api/v1/metrics/summary?asset_key=my_model
```

## Installation

```bash
pip install explainer-ai                     # core
pip install explainer-ai[api]                # REST API (FastAPI + uvicorn)
pip install explainer-ai[dagster]            # Dagster asset checks
pip install explainer-ai[superset]           # Superset dashboard support
pip install explainer-ai[full]               # everything
```

### From source

```bash
git clone https://github.com/Cognome-Inc/tapestry.git
cd tapestry
uv sync --dev --extra full --group traditional
```

## Code Example

```python
import pandas as pd
from tapestry.config import TapestryConfig
from tapestry.generative.tapestry import GenerativeTapestry
from tapestry import TapestryDB

# Sample LLM outputs to evaluate
df = pd.DataFrame({
    "record_id": [1, 2, 3],
    "context": [
        "The patient is a 55-year-old male with hypertension.",
        "Lab results show A1c of 7.2%, poorly controlled diabetes.",
        "The patient reported no known drug allergies.",
    ],
    "raw_model_response": [
        "The patient is a 55-year-old male with hypertension.",       # faithful
        "Lab results show A1c of 5.0%, excellent control.",           # hallucinated
        "The patient has a severe allergy to penicillin.",            # hallucinated
    ],
})

# Configure evaluators
config = TapestryConfig(
    identifier=["record_id"],
    asset_key="clinical_qa",
    custom_evaluators=[
        {"name": "hallucination", "arguments": {"model": "gpt-4o-mini"}},
        {"name": "phi_detection", "arguments": {"model": "gpt-4o-mini"}},
    ],
)

# Run evaluation
evaluator = GenerativeTapestry(config=config, backend="litellm")
results = evaluator.run(df)

# Write to database
db = TapestryDB.from_config()
evaluator.write_evaluators_to_db(results, db)
```

For local models, use any litellm-supported provider: `{"model": "ollama/llama3", "api_base": "http://localhost:11434"}`.

## CLI Reference

| Command | Description |
|---------|-------------|
| **Setup** | |
| `tapestry init` | Guided first-time setup (config, DB, evaluators) |
| `tapestry doctor` | Check project health (DB connection, config, etc.) |
| **Assets** | |
| `tapestry asset init` | Interactive wizard to create an asset config |
| `tapestry asset register -f <file>` | Register assets from JSON file or directory |
| `tapestry asset list` | List all registered assets |
| `tapestry asset show <key>` | Show full config and governance details |
| `tapestry asset completeness` | Show governance completeness scores |
| `tapestry asset show-impact <key>` | Show impact commitments with ROI status |
| **Evaluation** | |
| `tapestry evaluate <asset> --file <csv>` | Run evaluators on a local CSV |
| `tapestry serve [start\|stop\|status\|logs]` | REST API server management |
| **Ground Truth** | |
| `tapestry ground-truth ingest <asset>` | Match ground truth to predictions |
| `tapestry ground-truth status <asset>` | Show coverage and match latency |
| **Alerts** | |
| `tapestry alerts list` | Show configured alert rules |
| `tapestry alerts check` | Evaluate all rules and fire notifications |
| `tapestry alerts history` | Show recent alert events |
| **Dashboards** | |
| `tapestry dashboard create <asset>` | Per-asset Superset dashboard |
| `tapestry dashboard create-control-tower` | Governance overview dashboard |
| `tapestry dashboard create-comparison` | Multi-model comparison dashboard |
| **Database** | |
| `tapestry db init` | Create schema and run migrations |
| `tapestry db status` | Show tables and row counts |
| `tapestry db reset --force` | Drop and recreate schema |
| `tapestry migrate upgrade head` | Run pending database migrations |

All commands accept `--schema`/`-s` and `--engine`/`-e` overrides, or read from `.tapestry.toml` / environment variables.

## Project Structure

```
src/tapestry/
    api/            # FastAPI REST API (routers, schemas, dependencies)
    alerts/         # Rule-based alert engine and delivery (email, Slack, webhook)
    cli/            # CLI commands (init, serve, asset, evaluate, ground-truth, etc.)
    core/           # Base classes, URI generation, demographics, retry logic
    db/             # SQLAlchemy models, migrations, schema management
    generative/     # LLM-based evaluators (litellm backend)
    models/         # Pydantic models for model registry / inventory
    monitoring/     # Ground truth metrics computation
    observability/  # OpenTelemetry integration
    superset/       # Dashboard builders, chart/view specs, Control Tower
    traditional/    # XGBoost, SHAP, drift, confidence evaluators
    config.py       # TapestryConfig, EvaluatorConfig, ScoreConfig
    registry.py     # Evaluator plugin registry with dimension mapping
```

## Development

```bash
git clone https://github.com/Cognome-Inc/tapestry.git
cd tapestry
uv sync --dev --extra full --group traditional

uv run pytest                               # run tests
uv run black --check src/ tests/            # format check
uv run ruff check src/ tests/               # lint
uv run mypy src/tapestry/                   # type check
```

## Documentation

Full docs are available in [`docs/`](docs/):

- [Installation](docs/getting-started/installation.md)
- [Quickstart](docs/getting-started/quickstart.md)
- [Core Concepts](docs/getting-started/concepts.md)
- [Configuration](docs/configuration/tapestry-config.md)
- [Asset Registration](docs/rest-api/asset-registration.md)
- [REST API](docs/rest-api/overview.md)
- [Generative Evaluators](docs/evaluators/generative/index.md)
- [Traditional Evaluators](docs/evaluators/traditional/index.md)
- [Evaluation Dimensions](docs/evaluators/dimensions.md)
- [Ground Truth Monitoring](docs/integrations/ground-truth.md)
- [Model Registry](docs/integrations/model-registry.md)
- [Impact Tracking & ROI](docs/integrations/impact-tracking.md)
- [Superset Dashboards](docs/integrations/superset.md)
- [Alerting](docs/integrations/alerting.md)
- [Dagster Integration](docs/integrations/dagster.md)
- [Database Schema](docs/integrations/database.md)

## License

See [LICENSE](LICENSE) for details.
