Metadata-Version: 2.4
Name: explainer-ai
Version: 0.3.1
Summary: ML model evaluation and monitoring framework with explainability, fairness analysis, and Superset dashboards
Author: Cognome Inc
License: Copyright (c) 2026 Cognome Inc. All rights reserved.
        
        This software and associated documentation files (the "Software") may not be
        copied, modified, merged, published, distributed, sublicensed, or sold without
        the prior written permission of Cognome Inc.
        
        The Software is provided "as is", without warranty of any kind, express or
        implied, including but not limited to the warranties of merchantability, fitness
        for a particular purpose, and noninfringement. In no event shall the authors or
        copyright holders be liable for any claim, damages, or other liability, whether
        in an action of contract, tort, or otherwise, arising from, out of, or in
        connection with the Software or the use or other dealings in the Software.
        
Project-URL: Homepage, https://github.com/Cognome-Inc/tapestry
Project-URL: Repository, https://github.com/Cognome-Inc/tapestry
Project-URL: Issues, https://github.com/Cognome-Inc/tapestry/issues
Keywords: ml,model-monitoring,evaluation,explainability,fairness,llm,superset
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Software Development :: Quality Assurance
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: alembic>=1.13.0
Requires-Dist: jupyter>=1.1.1
Requires-Dist: litellm>=1.67.0
Requires-Dist: notebook>=7.4.0
Requires-Dist: numpy>=2.0.2
Requires-Dist: pandas>=2.2.3
Requires-Dist: pandera>=0.24.0
Requires-Dist: psycopg2-binary>=2.9.10
Requires-Dist: pydantic>=2.11.3
Requires-Dist: python-dotenv>=1.1.0
Requires-Dist: rich>=13.0.0
Requires-Dist: scipy>=1.11.0
Requires-Dist: sqlalchemy>=2.0.40
Requires-Dist: typer>=0.12.0
Provides-Extra: dev
Requires-Dist: pytest>=8.4.0; extra == "dev"
Provides-Extra: dagster
Requires-Dist: dagster>=1.10.11; extra == "dagster"
Provides-Extra: traditional
Requires-Dist: xgboost[cpu]; extra == "traditional"
Requires-Dist: shap>=0.47.2; extra == "traditional"
Provides-Extra: observability
Requires-Dist: opentelemetry-api>=1.20.0; extra == "observability"
Requires-Dist: opentelemetry-sdk>=1.20.0; extra == "observability"
Requires-Dist: opentelemetry-exporter-otlp>=1.20.0; extra == "observability"
Provides-Extra: superset
Requires-Dist: requests>=2.31.0; extra == "superset"
Requires-Dist: tomli>=2.0.0; python_version < "3.11" and extra == "superset"
Provides-Extra: api
Requires-Dist: fastapi>=0.115.0; extra == "api"
Requires-Dist: uvicorn[standard]>=0.30.0; extra == "api"
Provides-Extra: full
Requires-Dist: dagster>=1.10.11; extra == "full"
Requires-Dist: xgboost[cpu]; extra == "full"
Requires-Dist: shap>=0.47.2; extra == "full"
Requires-Dist: opentelemetry-api>=1.20.0; extra == "full"
Requires-Dist: opentelemetry-sdk>=1.20.0; extra == "full"
Requires-Dist: opentelemetry-exporter-otlp>=1.20.0; extra == "full"
Requires-Dist: requests>=2.31.0; extra == "full"
Requires-Dist: tomli>=2.0.0; python_version < "3.11" and extra == "full"
Requires-Dist: fastapi>=0.115.0; extra == "full"
Requires-Dist: uvicorn[standard]>=0.30.0; extra == "full"
Dynamic: license-file

# Tapestry

An ML model evaluation and governance framework for production pipelines, supporting traditional ML and generative AI evaluation with automated dashboarding and alerting.

## Features

- **Traditional ML evaluation** — confidence analysis, SHAP explainability, data drift, class distribution, entropy
- **Generative AI evaluation** — LLM-based judges for hallucination, fairness, PHI detection, factual accuracy, traceability, and more
- **Demographic fairness** — per-group pass rates, chi-squared tests, and CHAI-aligned ground truth metrics across demographic subgroups
- **Automated Superset dashboards** — per-asset, cross-asset generative, multi-model comparison, and Control Tower governance dashboards
- **Alerting** — rule-based alerts on evaluation metrics with Slack/email notification
- **REST API** — FastAPI-based API for data ingestion, evaluation, metrics, and asset management
- **Dagster integration** — optional asset checks for pipeline monitoring (`pip install explainer-ai[dagster]`)
- **Database persistence** — PostgreSQL storage with migration support and URI-based record linking

## Installation

```bash
pip install explainer-ai

# With optional extras
pip install explainer-ai[dagster]       # Dagster asset checks
pip install explainer-ai[superset]      # Superset dashboard support
pip install explainer-ai[full]          # All optional extras
```

Or with uv:

```bash
uv pip install explainer-ai
uv pip install explainer-ai[dagster,superset]
```

### From source

```bash
git clone https://github.com/Cognome-Inc/tapestry.git
pip install ./tapestry
```

## Quick Start

### First-Time Setup

```bash
tapestry init --engine "postgresql://user:password@host:5432/database" --schema "tapestry_dev"
```

This creates a `.env` file, copies the example `.tapestry.toml` config, initializes the database schema, and syncs all built-in evaluator definitions.

You can also set up manually:

```bash
export TAPESTRY_ENGINE="postgresql://user:password@host:5432/database"
export TAPESTRY_SCHEMA="tapestry_dev"
tapestry db init
```

### Generative Evaluation

This example evaluates a small dataset for hallucinations using an OpenAI model.
Set your `OPENAI_API_KEY` environment variable, then copy this into a script and run it.

```python
import pandas as pd
from datetime import datetime
from tapestry.config import TapestryConfig
from tapestry.generative.tapestry import GenerativeTapestry
from tapestry import TapestryDB

# 1. Sample data — each row has a context and an LLM response to evaluate
df = pd.DataFrame({
    "record_id": [1, 2, 3],
    "context": [
        "The patient is a 55-year-old male with a history of hypertension.",
        "Lab results show hemoglobin A1c of 7.2%, indicating poorly controlled diabetes.",
        "The patient reported no known drug allergies.",
    ],
    "raw_model_response": [
        "The patient is a 55-year-old male with hypertension.",           # faithful
        "Lab results show A1c of 5.0%, indicating excellent control.",    # hallucinated
        "The patient has a severe allergy to penicillin.",                # hallucinated
    ],
})

# 2. Configure — pick which evaluators to run and which LLM judges them
config = TapestryConfig(
    identifier=["record_id"],
    asset_key="readme_example",
    custom_evaluators=[
        {"name": "hallucination", "arguments": {"model": "gpt-4o-mini"}},
    ],
    batch_size=10,
)

# 3. Run evaluation
evaluator = GenerativeTapestry(config=config, backend="litellm")
results = evaluator.run(df)
print(results[["record_id", "HallucinationDetection", "HallucinationDetection reasoning"]])

# 4. (Optional) Write results to the database
db = TapestryDB(engine_url="postgresql+psycopg2://user:pass@localhost:5432/mydb", db_schema="tapestry_dev")
evaluator.write_evaluators_to_db(results, db)
```

For other providers, change the `arguments` — e.g. `{"model": "ollama/llama3", "api_base": "http://localhost:11434"}` for local Ollama models.

### REST API

Start the API server and push data for evaluation:

```bash
# Start the server
tapestry serve

# Register an asset with its evaluator config (one-time)
curl -X POST http://localhost:8084/api/v1/assets/ \
  -H "Content-Type: application/json" \
  -d '{
    "asset_key": "my_model",
    "evaluator_type": "traditional",
    "identifier_columns": ["record_id"],
    "config": {
      "custom_evaluators": [
        {"name": "confidence-analysis", "arguments": {"predicted_class_column": "pred", "confidence_score_column": "score", "model_id": "v1", "threshold": 0.9}}
      ]
    }
  }'

# Push data for evaluation (repeatable — config is stored)
curl -X POST http://localhost:8084/api/v1/assets/my_model/ingest \
  -H "Content-Type: application/json" \
  -d '{"data": [{"record_id": "1", "pred": "A", "score": 0.95}]}'

# Query results
curl http://localhost:8084/api/v1/assets/my_model/health
```

Interactive API docs are available at [http://localhost:8084/docs](http://localhost:8084/docs).

### Dashboards

```bash
# Per-asset dashboard
tapestry dashboard create <asset_key>

# Cross-asset generative evaluation (with demographic fairness tab)
tapestry dashboard create-generative

# Multi-model comparison
tapestry dashboard create-comparison

# Governance Control Tower
tapestry dashboard create-control-tower
```

Requires a `.tapestry.toml` with Superset connection details. All commands support `--dry-run` to create SQL views only.

## CLI Reference

| Command | Description |
|---------|-------------|
| `tapestry init` | Guided first-time setup (`.env`, config, DB) |
| `tapestry db init` | Create schema and run migrations |
| `tapestry db status` | Show tables, row counts, migration status |
| `tapestry db sync-checks` | Sync evaluator definitions to the database |
| `tapestry db reset --force` | Drop and recreate schema |
| `tapestry asset register -f <file>` | Register assets from JSON config file or directory |
| `tapestry asset list` | List all registered assets |
| `tapestry asset show <key>` | Show full config for a registered asset |
| `tapestry asset delete <key>` | Remove an asset registration |
| `tapestry serve` | Start the REST API server |
| `tapestry dashboard create <asset>` | Per-asset Superset dashboard |
| `tapestry dashboard create-generative` | Cross-asset generative dashboard |
| `tapestry dashboard create-comparison` | Multi-model comparison dashboard |
| `tapestry dashboard create-control-tower` | Governance overview dashboard |
| `tapestry migrate upgrade head` | Run pending migrations |
| `tapestry migrate autogenerate -m "msg"` | Generate migration from model changes |

All commands accept `--schema`/`-s` and `--engine`/`-e` overrides.

## Project Structure

```
src/tapestry/
    api/            # FastAPI REST API (routers, schemas, dependencies)
    core/           # Base classes, URI generation, demographics
    traditional/    # XGBoost + SHAP evaluation
    generative/     # LLM-based evaluators (litellm)
    db/             # SQLAlchemy models, migrations
    superset/       # Dashboard builders, chart/view specs
    cli/            # CLI commands (serve, db, dashboard, alerts, migrate)
    config.py       # TapestryConfig, EvaluatorConfig, ScoreConfig
    registry.py     # Evaluator registry
```

## Development

For contributing to Tapestry itself, clone the repo and install from source:

```bash
git clone https://github.com/Cognome-Inc/tapestry.git
cd tapestry
uv sync --dev --extra full
```

```bash
uv run pytest                               # Run tests
uv run black .                              # Format code
uv run ruff check src/ tests/               # Lint
```

## Documentation

See [`docs/`](docs/) for full documentation:

- [Getting Started](docs/getting-started/quickstart.md)
- [REST API](docs/rest-api/overview.md)
- [Asset Registration](docs/rest-api/asset-registration.md)
- [Configuration](docs/configuration/tapestry-config.md)
- [Generative Evaluators](docs/evaluators/generative/index.md)
- [Traditional Evaluators](docs/evaluators/traditional/index.md)
- [Demographic Fairness](docs/evaluators/traditional/demographic-performance.md)
- [Superset Dashboards](docs/integrations/superset.md)
- [Database Schema](docs/integrations/database.md)
- [Alerting](docs/integrations/alerting.md)
- [Dagster Integration](docs/integrations/dagster.md)
