Metadata-Version: 2.5
Name: dataeval-plots
Version: 0.0.14
Summary: DataEval companion package for plotting utilities
Project-URL: Homepage, https://dataeval.ai/
Project-URL: Repository, https://github.com/aria-ml/dataeval/
Project-URL: Documentation, https://dataeval.readthedocs.io/
Author-email: Andrew Weng <andrew.weng@ariacoustics.com>, Ryan Wood <ryan.wood@ariacoustics.com>
Maintainer-email: ARiA <dataeval@ariacoustics.com>
License-Expression: MIT
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Scientific/Engineering
Requires-Python: >=3.11
Requires-Dist: matplotlib>=3.7.1
Requires-Dist: numpy>=1.24.2; python_version < '3.12'
Requires-Dist: numpy>=1.26.2; python_version >= '3.12'
Requires-Dist: polars>=1.0.0
Provides-Extra: all
Requires-Dist: altair>=5.0; extra == 'all'
Requires-Dist: plotly>=5.0; extra == 'all'
Requires-Dist: seaborn>=0.12; extra == 'all'
Provides-Extra: altair
Requires-Dist: altair>=5.0; extra == 'altair'
Provides-Extra: opencv
Requires-Dist: opencv-python-headless>=4.12.0.88; extra == 'opencv'
Provides-Extra: plotly
Requires-Dist: plotly>=5.0; extra == 'plotly'
Provides-Extra: projection
Requires-Dist: scikit-learn>=1.3; extra == 'projection'
Provides-Extra: projection-all
Requires-Dist: pacmap>=0.7; extra == 'projection-all'
Requires-Dist: phate>=1.0; extra == 'projection-all'
Requires-Dist: scikit-learn>=1.3; extra == 'projection-all'
Requires-Dist: umap-learn>=0.5; extra == 'projection-all'
Provides-Extra: seaborn
Requires-Dist: seaborn>=0.12; extra == 'seaborn'
Description-Content-Type: text/markdown

# DataEval Plots

Multi-backend plotting utilities for DataEval outputs.

## Installation

```bash
# Minimal - no plotting backend included
pip install dataeval-plots

# With matplotlib plotting (recommended)
pip install dataeval-plots[matplotlib]

# With multiple backends
pip install dataeval-plots[matplotlib,plotly]

# Everything
pip install dataeval-plots[all]
```

For development:

```bash
pip install -e dataeval-plots[all]
```

## Available Backends

| Backend | Status | Install With | Description |
|---------|--------|--------------|-------------|
| matplotlib | ✅ Default | `[matplotlib]` | Standard publication-quality plots |
| seaborn | ✅ Available | `[seaborn]` | Statistical data visualization |
| plotly | ✅ Available | `[plotly]` | Interactive web-based plots |
| altair | ✅ Available | `[altair]` | Declarative visualization grammar |

## Usage

### Option 1: Import from dataeval-plots directly

```python
from dataeval_plots import plot
from dataeval.metrics.bias import coverage

result = coverage(embeddings)
fig = plot(result, images=dataset, top_k=6)
fig.savefig("coverage.png")
```

### Option 2: Import from dataeval core (convenience)

```python
from dataeval import plotting
from dataeval.metrics.bias import coverage

result = coverage(embeddings)
fig = plotting.plot(result, images=dataset)
```

### Option 3: Set default backend

```python
from dataeval_plots import plot, set_default_backend

# Set seaborn as default
set_default_backend("seaborn")
fig = plot(result, images=dataset)  # Uses seaborn

# Override for a specific plot
fig = plot(result, backend="matplotlib", images=dataset)
```

## Features

- **Multi-backend architecture**: Support for matplotlib (default), seaborn, plotly, and altair
- **Optional dependencies**: Install only the backends you need
- **Clean separation**: Core dataeval has zero plotting dependencies
- **Protocol-based design**: Loose coupling via structural typing (`Plottable` protocol)
- **Extensible**: Easy to add new backends via `BasePlottingBackend` or custom outputs via `Plottable`
- **Lazy loading**: Backends are only imported when first used
- **Type safe**: Static type checking with mypy/pyright via `@runtime_checkable` protocols
- **DRY architecture**: Centralized routing logic in `BasePlottingBackend`

## Architecture

The package uses a **protocol-based architecture** for loose coupling between dataeval and dataeval-plots:

```
dataeval/                           # Core package
    outputs/
        _bias.py                    # CoverageOutput, BalanceOutput, DiversityOutput
        _stats.py                   # BaseStatsOutput
        _workflows.py               # SufficiencyOutput
        _drift.py                   # DriftMVDCOutput
    plotting.py                     # Convenience hook to dataeval-plots

dataeval-plots/                     # Separate plotting package
    src/dataeval_plots/
        __init__.py                 # Main plot() function
        _registry.py                # Backend registry with lazy loading
        protocols.py                # Protocol definitions (Plottable hierarchy)
        backends/
            _base.py                # BasePlottingBackend (abstract routing)
            _matplotlib.py          # MatplotlibBackend (default)
            _seaborn.py             # SeabornBackend
            _plotly.py              # PlotlyBackend
            _altair.py              # AltairBackend
```

### Protocol-Based Design

All DataEval output classes implement the `Plottable` protocol, which requires:

- `plot_type()`: Returns a string identifying the plot type (e.g., "balance")
- `meta()`: Returns execution metadata

This enables:

- **Loose coupling**: dataeval-plots doesn't import concrete classes from dataeval
- **Type safety**: Static and runtime type checking via `@runtime_checkable` protocols
- **Extensibility**: Anyone can create custom outputs implementing `Plottable`
- **Zero dependencies**: Core dataeval has no plotting dependencies

## Supported Output Types

| Output Type | Plot Type | Description | Source |
|-------------|-----------|-------------|--------|
| `BalanceOutput` | "balance" | Heatmap of class balance metrics | [dataeval/_bias.py](../dataeval/src/dataeval/outputs/_bias.py) |
| `DiversityOutput` | "diversity" | Visualization of diversity indices | [dataeval/_bias.py](../dataeval/src/dataeval/outputs/_bias.py) |
| `SufficiencyOutput` | "sufficiency" | Learning curves with extrapolation | [dataeval/_workflows.py](../dataeval/src/dataeval/outputs/_workflows.py) |
| `BaseStatsOutput` | "stats" | Statistical histograms and distributions | [dataeval/_stats.py](../dataeval/src/dataeval/outputs/_stats.py) |
| `DriftMVDCOutput` | "drift_mvdc" | Drift detection plots (MVDC analysis) | [dataeval/_drift.py](../dataeval/src/dataeval/outputs/_drift.py) |
| `FeasibilityOutput` | "feasibility" | Which accuracy requirements the evidence demonstrates or rules out | [dataeval/_feasibility.py](../dataeval/src/dataeval/performance/_feasibility.py) |

Each output type implements the `Plottable` protocol and can be plotted using any registered backend.

## Reading the Feasibility Plot

`plot()` draws the result of DataEval's experimental `Feasibility().evaluate(...)`. It answers one question per
quantity you asked about: for each accuracy or AP requirement, does the evidence demonstrate it, rule it out, or leave
it unknown?

```python
from dataeval.performance import Feasibility
from dataeval_plots import plot

result = Feasibility().evaluate(dataset, target={"accuracy": 0.95}, embeddings=embeddings)

fig = plot(result)  # one band per quantity
fig = plot(result, plot_evidence=True)  # one row per evidence source, in place of the bands
```

### What the plot shows

Each quantity gets its own band, with a "Required ..." axis (for example "Required accuracy" or "Required AP") and a
subtitle that says which models the band judges:

| Subtitle | Judged models |
|----------|---------------|
| Models may use the whole image | Models that may use the whole input |
| Models may use only the embeddings | Models restricted to the embeddings |
| Models give one answer per group | Models that give one answer per group |

The axis is split into three zones. Each zone classifies the requirements that fall inside it:

- **Demonstrated** runs from the axis minimum up to the lower confidence bound of the best tested model, which the
  label under the band names, for example "95.7% lower confidence bound". The zone reads "demonstrated".
- **Unknown** lies between the other two zones. The evidence neither demonstrates nor rules out these requirements.
- **Ruled out** runs from the upper confidence bound of the lowest applicable ceiling to the top of the axis. The zone
  reads "ruled out\*" wherever the label fits.

The `*` marks a footnote: "ruled out" holds only under the recorded assumptions. Wherever "ruled out\*" appears, the
caption under the same band ends with that footnote, "\* Ruled out under the recorded assumptions, at the stated
confidence."

Each of your requirements is marked on the axis, and the marker states the determination in words:

| Marker | Meaning |
|--------|---------|
| "95%: demonstrated ✓" | The requirement is at or below the lower confidence bound |
| "95%: insufficient evidence ?" | The requirement falls in the unknown zone, and the evidence cannot decide it |
| "95%: ruled out\* ✗" | The requirement is above the upper confidence bound of a ceiling, under the recorded assumptions (see the footnote) |
| "95%: not ruled out ?" | The requirement also falls in the unknown zone: it is not above any ceiling's upper confidence bound, and no tested model reaches it |

The caption under the band reports the observed score next to the lower confidence bound it supports, for example
"Your model scored 91.0% on 2,000 evaluation images; given sampling uncertainty, its lower confidence bound is
88.5%." The sentence about the observed score appears only when an observed score exists. The caption then states
that the bounds apply to new images from the same population, and that confidence holds jointly across the whole
evaluation (for example "Bounds apply to new images from the same population, at 95% joint confidence.").

Numbers read as DataEval prints them. Bounds are given to 0.1%, a lower bound rounded down and an upper bound rounded
up, so rounding never overstates what the evidence shows. Requirements and confidence keep their own decimals, such as
"95.5%" or "at 97.5% joint confidence".

When a ceiling bounds the band, the caption ends with the footnote "\* Ruled out under the recorded assumptions, at
the stated confidence." A group-based ceiling adds "(group-based ceiling; validity pending lead-scientist sign-off)".

A ceiling that applies only to narrower models than the band judges never draws a zone. It appears as a note instead,
for example "Group-based ceiling (72.1%) applies only to models that give one answer per group."

### Other outcomes

- **Ruled out.** A requirement above the lowest applicable ceiling's upper confidence bound is marked "ruled out\*
  ✗". The ruled-out zone covers everything above that bound, which the label under the band names, for example
  "93.0% upper confidence bound". The caption ends with the footnote "\* Ruled out under the recorded assumptions, at
  the stated confidence."
- **Evidence conflict.** When bounds from different sources contradict each other, the band draws no zones. A dashed
  warning outline reads "Evidence conflicts: check assumptions", both clashing bounds are marked, and the caption
  reads "Bounds disagree: check the assumptions, such as evaluation images entering training. Even when they hold,
  the chance that any bound fails is at most 5%." The 5% is the stated alpha, at 95% confidence.
- **Bounds at 0% or 100%.** A lower confidence bound of 0% demonstrates nothing and an upper confidence bound of 100%
  rules nothing out, so neither draws a zone.

### Showing the evidence

`plot_evidence=True` replaces each band with one row per evidence source. Each row shows the observed score as a dot
and the confidence bound as a tick, with the legend "● observed · ┃ lower confidence bound" (and "· ┃ upper confidence
bound" when a ceiling has one). The caption says what both mean: "Observed on the evaluation images; bounds apply to
new images from the same population, at 95% joint confidence." In a conflict, the conflict caption follows it.
Evidence that does not apply to the judged models is drawn faint, and its label names the models it applies to, for
example "Group-based ceiling (one answer per group)", so this view has no narrower-scope notes. A source that produced
no bound for the quantity is listed faint as "no bound available", and when no source has a bound the view says "No
bound is available yet." instead of showing empty axes.

## Extending the Package

### Creating Custom Outputs

You can create custom output classes that work with the plotting system by implementing the `Plottable` protocol:

```python
from dataclasses import dataclass
from numpy.typing import NDArray
from dataeval_plots.protocols import Plottable, ExecutionMetadata

@dataclass
class MyCustomOutput:
    """Custom output that reuses existing plot type."""
    uncovered_indices: NDArray

    def plot_type(self) -> str:
        return "balance"  # Reuse existing balance plotting

    def meta(self) -> ExecutionMetadata:
        return ExecutionMetadata.empty()

# Works seamlessly with existing backends
result = MyCustomOutput(uncovered_indices=my_data)
fig = plot(result, images=my_images)
```

### Creating Custom Backends

Extend `BasePlottingBackend` to create a new plotting backend:

```python
from dataeval_plots.backends._base import BasePlottingBackend
from dataeval_plots.protocols import PlottableBalance
from dataeval_plots import register_backend

class CustomBackend(BasePlottingBackend):
    """Custom plotting backend using your preferred library."""

    def _plot_balance(self, output: PlottableBalance, **kwargs):
        # Implement balance plotting
        return my_figure

    # Implement other _plot_* methods...

# Register and use
register_backend("custom", CustomBackend())
fig = plot(result, backend="custom")
```

The `BasePlottingBackend` class handles all routing logic automatically - you just implement the plot-type-specific methods (`_plot_coverage`, `_plot_balance`, etc.).
