Metadata-Version: 2.4
Name: vertebrae
Version: 0.1.0a1
Summary: Benchmark feature extractors and transfer-learning backbones using OverlapIndex.
License-Expression: MIT
License-File: LICENSE
Keywords: embeddings,feature-extraction,model-evaluation,transfer-learning,overlapindex,separatix
Author: Niklas Melton
Author-email: niklasmelton@gmail.com
Requires-Python: >=3.9,<3.15
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Science/Research
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Scientific/Engineering :: Information Analysis
Provides-Extra: audio
Provides-Extra: cloud
Provides-Extra: dask
Provides-Extra: distributed
Provides-Extra: gcs
Provides-Extra: graph
Provides-Extra: hf
Provides-Extra: jax
Provides-Extra: keras
Provides-Extra: mlp
Provides-Extra: onnx
Provides-Extra: openclip
Provides-Extra: ray
Provides-Extra: s3
Provides-Extra: tensorflow
Provides-Extra: tensorflow-hub
Provides-Extra: timeseries
Provides-Extra: timm
Provides-Extra: torch
Provides-Extra: torchvision
Provides-Extra: trees
Provides-Extra: video
Requires-Dist: av (>=12,<16) ; extra == "hf" or extra == "video"
Requires-Dist: boto3 (>=1.34,<2.0) ; extra == "s3" or extra == "cloud"
Requires-Dist: catboost (>=1.2) ; extra == "trees"
Requires-Dist: datasets (>=2.0,<3.0) ; extra == "hf"
Requires-Dist: dgl (>=2.0) ; extra == "graph"
Requires-Dist: distributed (>=2024.5,<2025.0) ; extra == "dask" or extra == "distributed"
Requires-Dist: flax (>=0.8) ; extra == "jax"
Requires-Dist: google-cloud-storage (>=2.16,<3.0) ; extra == "gcs" or extra == "cloud"
Requires-Dist: jax (>=0.4) ; extra == "jax"
Requires-Dist: joblib (>=1.3,<2.0)
Requires-Dist: keras (>=3.0) ; extra == "keras"
Requires-Dist: lightgbm (>=4.0) ; extra == "trees"
Requires-Dist: numpy (>=2.0,<3.0)
Requires-Dist: onnxruntime (>=1.18,<2.0) ; extra == "onnx"
Requires-Dist: open-clip-torch (>=2.24,<3.0) ; extra == "openclip"
Requires-Dist: overlapindex (>=0.1.3a3)
Requires-Dist: pandas (>=2.2,<3.0)
Requires-Dist: pillow (>=11.3,<12) ; python_version == "3.9"
Requires-Dist: pillow (>=12,<13) ; python_version >= "3.10" and python_version < "3.15"
Requires-Dist: psutil (>=5.9,<6.0)
Requires-Dist: pydantic (>=2.0,<3.0)
Requires-Dist: pytorchvideo (>=0.1.5,<0.2.0) ; extra == "hf" or extra == "video"
Requires-Dist: ray (>=2.30,<3.0) ; extra == "ray" or extra == "distributed"
Requires-Dist: scikit-learn (>=1.5,<2.0)
Requires-Dist: scipy (>=1.13,<2.0)
Requires-Dist: sentence-transformers (>=3.0,<4.0) ; extra == "hf"
Requires-Dist: separatix (>=0.1.0a4)
Requires-Dist: soundfile (>=0.13,<0.14) ; extra == "hf" or extra == "audio"
Requires-Dist: tensorflow (>=2.16) ; extra == "tensorflow" or extra == "tensorflow-hub"
Requires-Dist: tensorflow-hub (>=0.16,<0.17) ; extra == "tensorflow-hub"
Requires-Dist: timm (>=1.0,<2.0) ; extra == "timm"
Requires-Dist: torch (>=2.1) ; extra == "hf" or extra == "audio" or extra == "timeseries" or extra == "video" or extra == "torch" or extra == "timm" or extra == "torchvision" or extra == "openclip" or extra == "mlp" or extra == "graph"
Requires-Dist: torch-geometric (>=2.5) ; extra == "graph"
Requires-Dist: torchvision (>=0.16) ; extra == "torchvision"
Requires-Dist: transformers (>=4.0,<5.0) ; extra == "hf" or extra == "audio" or extra == "timeseries" or extra == "video"
Requires-Dist: xgboost (>=2.0) ; extra == "trees"
Project-URL: Documentation, https://vertebrae.readthedocs.io/
Project-URL: Homepage, https://github.com/NiklasMelton/vertebrae
Project-URL: Repository, https://github.com/NiklasMelton/vertebrae
Description-Content-Type: text/markdown

# Vertebrae

<table>
  <tr>
    <td width="100" valign="top">
      <a href="https://github.com/NiklasMelton/vertebrae">
        <img
          src="https://github.com/NiklasMelton/vertebrae/blob/develop/img/vertebrae_logo.png?raw=true"
          alt="Vertebrae logo"
          width="140"
        />
      </a>
    </td>
    <td valign="top">

`vertebrae` is a Python package for evaluating feature extractors and
transfer-learning backbones on labeled datasets. It supports dense and sparse
precomputed embeddings, scikit-learn pipelines, custom callable extractors, ONNX
models, local PyTorch and Keras modules, segmentation token workflows, optional
embedding compression, and optional model families spanning Hugging Face,
sentence-transformers, timm, torchvision, OpenCLIP/SigLIP, TensorFlow Hub,
JAX/Flax, tree ensembles, graph models, and hosted embedding APIs. It can also
evaluate labeled embedding units such as document regions, tokens, frames,
keypoints, depth cells, and latent slots emitted directly by a model.

The package uses the `overlapindex` library as its primary separation metric and
adds a `separatix` complexity diagnostic to reports when an evaluated embedding
clears a configurable overlap-quality threshold. The full evaluation flow wraps
those diagnostics with practical dataset handling, named target and hierarchy
views, caching, memory-aware subsampling, stability analysis, artifact-backed
execution, custom embedding metrics, and report generation.

</tr>
</table>



## Installation

```bash
pip install vertebrae
```

For local development:

```bash
poetry install --with dev
```

Optional Hugging Face and sentence-transformers support:

```bash
pip install "vertebrae[hf]"
```

Optional Hugging Face audio support only:

```bash
pip install "vertebrae[audio]"
```

Optional Hugging Face time-series support only:

```bash
pip install "vertebrae[timeseries]"
```

Optional Hugging Face video support only:

```bash
pip install "vertebrae[video]"
```

Optional local PyTorch model support:

```bash
pip install "vertebrae[torch]"
pip install "vertebrae[timm]"
pip install "vertebrae[torchvision]"
pip install "vertebrae[openclip]"
```

Optional local Keras model support:

```bash
pip install "vertebrae[keras]"
pip install "vertebrae[tensorflow]"
pip install "vertebrae[tensorflow-hub]"
```

Optional ONNX Runtime support:

```bash
pip install "vertebrae[onnx]"
```

Optional JAX/Flax, tree ensemble, and graph model support:

```bash
pip install "vertebrae[jax]"
pip install "vertebrae[trees]"
pip install "vertebrae[graph]"
```

Optional distributed execution backends:

```bash
pip install "vertebrae[ray]"
pip install "vertebrae[dask]"
pip install "vertebrae[distributed]"
```

Optional cloud artifact stores:

```bash
pip install "vertebrae[s3]"
pip install "vertebrae[gcs]"
pip install "vertebrae[cloud]"
```

## Quick Start

### Precomputed embeddings

```python
from vertebrae import BenchmarkDataset, Evaluator, DatasetIdentity
from vertebrae.extractors import PrecomputedExtractor

dataset = BenchmarkDataset.from_embeddings(embeddings=Z, labels=y, identity=DatasetIdentity.declared("example-dataset", "1"))
extractor = PrecomputedExtractor(name="baseline_embeddings")

result = Evaluator(dataset=dataset, extractor=extractor).run()

print(result.to_dataframe())
result.save_json("result.json")
result.save_markdown("report.md")
```

Every root dataset requires an explicit `DatasetIdentity`. A declared identity is the
recommended production choice; change its revision whenever the dataset content,
ordering, targets, groups, or annotations change. Manifest identity hashes only a
caller-provided source manifest. Full content hashing and ephemeral UUID identity are
available only through the explicit `DatasetIdentity.from_content()` and
`DatasetIdentity.ephemeral()` constructors. Path-based datasets are never scanned or
hashed automatically. See [Dataset identity](docs/datasets.md#dataset-identity).

Sparse matrices are supported as embedding inputs as well.

By default, vertebrae also runs a Separatix complexity diagnostic when the
evaluated embedding reaches `overlap_macro >= 0.80`. That extra diagnostic does
not affect ranking. It adds report guidance about what kind of downstream
classifier complexity the labeled geometry appears to need.

Multi-label classification datasets are supported through the same constructors.
Use per-sample label sequences or a binary indicator matrix with `label_names`:

```python
from vertebrae import DatasetIdentity
dataset = BenchmarkDataset.from_embeddings(
    embeddings=Z,
    labels=[
        ("outdoor", "vehicle"),
        ("outdoor", "vehicle"),
        ("indoor",),
        ("indoor",),
        ("outdoor", "animal"),
        ("animal",),
    ],
    identity=DatasetIdentity.declared("example-dataset", "1"),
)

result = Evaluator(dataset=dataset, extractor=PrecomputedExtractor()).run()
```

OverlapIndex receives a dense multi-label indicator target internally, and
Separatix runs with `target_mode="multilabel"`.

Regression targets are supported when explicitly requested so numeric class
identifiers are not accidentally interpreted as continuous targets:

```python
from vertebrae import DatasetIdentity
dataset = BenchmarkDataset.from_arrays(
    X=samples,
    y=targets,
    modality="tabular",
    target_type="regression",
    target_names=["quality_score"],
    identity=DatasetIdentity.declared("example-dataset", "1"),
)

result = Evaluator(dataset=dataset, extractor=extractor).run()
```

Regression scoring uses `ContinuousOverlapIndex` through vertebrae's internal
scoring adapter and appears in reports as continuous overlap diagnostics.

### Multiple target views

When one embedding should be compared against several aligned targets, register
named target views on the dataset and enable them in `Benchmark`. Views can be
classification, multi-label, or regression targets; each is reported as a
separate result variant.

```python
from vertebrae import Benchmark, BenchmarkDataset, TargetView, TargetViewConfig, DatasetIdentity

dataset = BenchmarkDataset.from_embeddings(embeddings=Z, labels=leaf_labels, identity=DatasetIdentity.declared("example-dataset", "1"))
dataset = dataset.with_target_views(
    [
        TargetView(name="coarse", targets=coarse_labels),
        TargetView(name="quality", targets=quality_scores, target_type="regression"),
    ]
)

result = Benchmark(
    dataset,
    [extractor],
    target_view_config=TargetViewConfig(enabled=True, views=("coarse", "quality")),
).run()
```

For taxonomies represented as label paths, use `with_label_hierarchy(...)` and
`LabelViewConfig` instead. `output_views` and `output_levels` can route named
extractor outputs to the target or hierarchy view that they should evaluate.

Classification labels keep exact typed semantic identity in the dataset and label
catalog. For example, integer `1` and string `"1"` are separate classes even though
their ordinary string forms match. Metric adapters receive marked semantic-key strings
in both local and artifact-backed runs, while results retain the original typed values,
stable internal keys, and disambiguated display text in the catalog. Hierarchy paths
are encoded structurally rather than by joining values with a delimiter.

### Optional embedding compression

```python
from vertebrae import BenchmarkDataset, EmbeddingCompressionConfig, Evaluator, DatasetIdentity
from vertebrae.extractors import PrecomputedExtractor

dataset = BenchmarkDataset.from_embeddings(embeddings=Z, labels=y, identity=DatasetIdentity.declared("example-dataset", "1"))
extractor = PrecomputedExtractor(name="baseline_embeddings")

compression = EmbeddingCompressionConfig(
    enabled=True,
    method="prefix_truncate",
    n_components=256,
    assume_matryoshka=True,
)

result = Evaluator(
    dataset=dataset,
    extractor=extractor,
    compression_config=compression,
).run()
```

Supported compression methods include `pca`, `incremental_pca`,
`truncated_svd`, random projections, `prefix_truncate`, and `quantize`.

### Scikit-learn pipelines

```python
from sklearn.decomposition import TruncatedSVD
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import Normalizer

from vertebrae import BenchmarkDataset, Evaluator, DatasetIdentity
from vertebrae.extractors import SklearnExtractor

pipeline = Pipeline(
    [
        ("tfidf", TfidfVectorizer(ngram_range=(1, 2), min_df=2)),
        ("svd", TruncatedSVD(n_components=128, random_state=42)),
        ("norm", Normalizer()),
    ]
)

dataset = BenchmarkDataset.from_arrays(texts, labels, modality="text", identity=DatasetIdentity.declared("example-dataset", "1"))
extractor = SklearnExtractor(
    name="tfidf_svd",
    pipeline=pipeline,
    # This manually versioned identity opts the fitted live pipeline into reuse.
    cache_identity="tfidf-svd-v1",
)

result = Evaluator(dataset=dataset, extractor=extractor).run()
```

### Local PyTorch models

```python
import numpy as np
import torch

from vertebrae import BenchmarkDataset, Evaluator, DatasetIdentity
from vertebrae.extractors import TorchExtractor

model = torch.load("/path/to/local_model.pt", map_location="cpu")
model.eval()


def collate_fn(batch):
    return torch.as_tensor(np.asarray(batch), dtype=torch.float32)


def output_fn(raw_output):
    return raw_output if isinstance(raw_output, torch.Tensor) else raw_output["embeddings"]


dataset = BenchmarkDataset.from_arrays(features, labels, modality="tabular", identity=DatasetIdentity.declared("example-dataset", "1"))
extractor = TorchExtractor(
    name="local_torch",
    model=model,
    collate_fn=collate_fn,
    output_fn=output_fn,
    device="cpu",
    checkpoint_paths=["/path/to/local_model.pt"],
    cache_identity="local-torch-model-v1",
    recipe_data={"checkpoint": "/path/to/local_model.pt"},
)

result = Evaluator(dataset=dataset, extractor=extractor).run()
```

### Local Keras models

```python
import numpy as np

from vertebrae import BenchmarkDataset, Evaluator, DatasetIdentity
from vertebrae.extractors import KerasExtractor


def collate_fn(batch):
    return np.asarray(batch, dtype=np.float32)


dataset = BenchmarkDataset.from_arrays(features, labels, modality="tabular", identity=DatasetIdentity.declared("example-dataset", "1"))
extractor = KerasExtractor(
    name="local_keras",
    model=model,
    collate_fn=collate_fn,
    call_method="call",
    checkpoint_paths=["/path/to/model.keras"],
    cache_identity="local-keras-model-v1",
    recipe_data={"checkpoint": "/path/to/model.keras"},
)

result = Evaluator(dataset=dataset, extractor=extractor).run()
```

`checkpoint_paths` contributes content-digested provenance and profiling evidence; a
path copied only into `recipe_data` is descriptive metadata. Because these adapters
receive already-loaded live model objects, the path cannot prove that the in-memory
weights match the file, so reusable caching still requires a maintained
`cache_identity`. Torch extraction defaults to evaluation plus inference mode and
restores the model's prior training state afterward. Notebook-local, nested, or
otherwise nonportable adapter functions likewise require an explicit identity.

### Hugging Face audio backbones

```python
from vertebrae import BenchmarkDataset, CacheConfig, Evaluator, DatasetIdentity
from vertebrae.extractors import HFAudioExtractor

dataset = BenchmarkDataset.from_audio_arrays(
    audio=waveforms,
    labels=labels,
    sampling_rate=16_000,
    identity=DatasetIdentity.declared("example-dataset", "1"),
)
extractor = HFAudioExtractor(
    name="wav2vec2_base",
    model_id="facebook/wav2vec2-base",
    pooling="mean",
)

# This introductory model ID is intentionally unpinned, so do not reuse its output.
result = Evaluator(
    dataset=dataset,
    extractor=extractor,
    cache_config=CacheConfig(enabled=False),
).run()
```

### Hugging Face multi-modal models

```python
from vertebrae import BenchmarkDataset, CacheConfig, Evaluator, DatasetIdentity
from vertebrae.extractors import HFMultimodalExtractor

dataset = BenchmarkDataset.from_multimodal(
    inputs={"image": images, "caption": captions},
    labels=labels,
    modalities={"image": "image", "caption": "text"},
    identity=DatasetIdentity.declared("example-dataset", "1"),
)

extractor = HFMultimodalExtractor(
    name="clip_like",
    model_id="openai/clip-vit-base-patch32",
    input_modalities={"image": "image", "caption": "text"},
    outputs=[
        {"name": "image_branch", "source": "image", "model_output": "image_embeds"},
        {"name": "text_branch", "source": "text", "model_output": "text_embeds"},
        {"name": "fused", "source": "fused", "model_output": "pooler_output"},
    ],
)

# This introductory model ID is intentionally unpinned, so do not reuse its output.
result = Evaluator(
    dataset=dataset,
    extractor=extractor,
    cache_config=CacheConfig(enabled=False),
).run()
```

### Hugging Face time-series backbones

```python
from vertebrae import BenchmarkDataset, CacheConfig, Evaluator, DatasetIdentity
from vertebrae.extractors import HFTimeSeriesExtractor

dataset = BenchmarkDataset.from_time_series(
    series=series,
    labels=labels,
    identity=DatasetIdentity.declared("example-dataset", "1"),
)
extractor = HFTimeSeriesExtractor(
    name="patchtst",
    model_id="some-local-or-hf-timeseries-model",
    pooling="mean",
)

# This placeholder may resolve to mutable remote state.
result = Evaluator(
    dataset=dataset,
    extractor=extractor,
    cache_config=CacheConfig(enabled=False),
).run()
```

### Hugging Face video backbones

```python
from vertebrae import BenchmarkDataset, CacheConfig, Evaluator, DatasetIdentity
from vertebrae.extractors import HFVideoExtractor

dataset = BenchmarkDataset.from_video_arrays(
    frames=clips,
    labels=labels,
    frame_rate=24.0,
    identity=DatasetIdentity.declared("example-dataset", "1"),
)
extractor = HFVideoExtractor(
    name="videomae_base",
    model_id="MCG-NJU/videomae-base",
    pooling="mean",
    num_frames=16,
)

# This introductory model ID is intentionally unpinned, so do not reuse its output.
result = Evaluator(
    dataset=dataset,
    extractor=extractor,
    cache_config=CacheConfig(enabled=False),
).run()
```

### Multi-extractor comparison

```python
from vertebrae import Benchmark

benchmark = Benchmark(dataset)
benchmark.add_extractor(tfidf_extractor)
benchmark.add_extractor(sentence_transformer_extractor)
benchmark.add_extractor(custom_extractor)

result = benchmark.run()
print(result.to_dataframe())
```

You can also benchmark multiple embedding outputs from the same backbone without
duplicating extractor classes. This is useful for comparing intermediate layers,
pooling strategies, or multi-head outputs from one model. When the dataset has
hierarchy metadata from `with_label_hierarchy(...)`, outputs can be routed to
different hierarchy levels:

```python
from vertebrae import Benchmark, CacheConfig, LabelViewConfig
from vertebrae.extractors import HFVisionExtractor

benchmark = Benchmark(
    dataset,
    label_view_config=LabelViewConfig(
        output_levels={
            "mid_cls": "family",
            "final_cls": "leaf",
        },
    ),
    # The example model IDs below are intentionally unpinned.
    cache_config=CacheConfig(enabled=False),
)
benchmark.add_extractor(
    HFVisionExtractor(
        name="mnist_vit",
        model_id="farleyknight-org-username/vit-base-mnist",
        outputs=[
            {"name": "final_cls", "pooling": "cls"},
            {"name": "mid_cls", "pooling": "cls", "hidden_layer": 6},
        ],
        image_mode="rgb",
        batch_size=8,
    )
)

result = benchmark.run()
print(result.to_dataframe()[["extractor", "label_view", "overlap_macro"]])
```

Each configured output is scored as its own result variant, so this run produces
rows such as `mnist_vit:mid_cls[level=family]` and
`mnist_vit:final_cls[level=leaf]`. See `examples/hf_vision_mnist.py` for a fuller
example that compares multi-output Hugging Face vision embeddings alongside a
classical scikit-learn image baseline.
For a more realistic image workflow, `examples/caltech101_vision_foundation_models.py`
downloads a laptop-sized Caltech-101 subset with a few related category pairs,
compares DINOv2 with a tiny supervised ViT baseline, and can include gated DINOv3
embeddings when `VERTABRAE_INCLUDE_DINOV3=1` is set.

### Retrieval and matching

`RetrievalBenchmark` evaluates frozen query embeddings against an explicit gallery
and graded relevance judgments. It is an exact, training-free ranking protocol and
is separate from ordinary labeled OverlapIndex benchmarking.

```python
from vertebrae import RetrievalBenchmark, RetrievalConfig, RetrievalDataset, DatasetIdentity
from vertebrae.extractors import PrecomputedExtractor

dataset = RetrievalDataset.from_embeddings(
    query_embeddings,
    gallery_embeddings,
    relevance=[("query-1", "document-9", 2.0)],
    query_ids=["query-1"],
    gallery_ids=["document-9"],
    identity=DatasetIdentity.declared("example-dataset", "1"),
)

result = RetrievalBenchmark(
    dataset,
    [PrecomputedExtractor("candidate")],
    retrieval_config=RetrievalConfig(primary_metric="ndcg@10"),
).run()
```

Relevance can be a NumPy/scipy query-by-gallery matrix or sparse
`(query_id, gallery_id, grade)` records. Nested Python lists are records; construct
nested-list matrices explicitly with `RetrievalDataset.from_relevance_matrix(...)`.
Every query must retain an eligible positive after exclusions. Reports include NDCG, precision, recall,
hit rate, MRR, mAP, and similarity diagnostics. See
[the retrieval guide](docs/retrieval.md) for branch-aware extractors,
bidirectional scoring, exclusions, compression, and artifact workflows.

### Fixed-prompt zero-shot alignment

For contrastive extractors with independently encodable sample and text branches,
`ZeroShotBenchmark` evaluates whether fixed class prompts address the dataset in the
model's shared embedding space. It trains no head and requires explicit prompts. The
zero-shot rank and ordinary sample-embedding overlap are reported side by side; they
are not combined into a universal backbone score. See `docs/zero_shot.md` and the
network-free `examples/zero_shot_callable.py` workflow.

Use `ZeroShotCandidate(extractor, sample_branch, text_branch)` when compared models
use different branch names. OpenCLIP keeps its image-only ordinary default while its
native `text_branch` remains available for zero-shot. Callable adapters cache only
when their functions have portable paths or an explicit `cache_identity`.

Retrieval and zero-shot compression keep both sides in one representation space.
Learned retrieval transforms are fitted on the gallery and applied to queries;
learned zero-shot transforms are fitted on samples and applied to prompt embeddings.
The CLI exposes these paired workflows through `compress-retrieval` and
`compress-zero-shot`.

### Custom embedding metrics

Every benchmark always records the built-in overlap metric. You can score the
same full embedding batch with additional metrics and choose one as the ranking
criterion. A custom metric returns a finite aggregate `score` and may include
JSON-safe diagnostics, warnings, and metadata.

For protocol parity, categorical labels and groups passed to custom metrics are
marked semantic-key strings locally and on distributed workers; regression targets
remain numeric. Read original typed provenance and display values from
`target_metadata["label_catalog"]` when needed.

```python
from vertebrae import Benchmark, CallableMetric

def domain_margin(embeddings, labels, *, groups=None, seed=None):
    return {"score": 0.87, "diagnostics": {"rule": "domain margin"}}

result = Benchmark(
    dataset,
    [extractor],
    metrics=[CallableMetric("domain_margin", domain_margin)],
    primary_metric="domain_margin",
).run()
```

The overlap result remains available in every `ExtractorResult` and continues
to drive stability and Separatix. For artifact or CLI workflows, use an
importable callable path such as `my_project.metrics:domain_margin`; see
[the scoring guide](https://github.com/NiklasMelton/vertebrae/blob/develop/docs/scoring.md)
and [the distributed-readiness guide](https://github.com/NiklasMelton/vertebrae/blob/develop/docs/distributed_readiness.md).

### Dense segmentation tokens

Dense segmentation evaluation scores spatial feature cells after they are aligned
to semantic mask labels. It measures representation organization for retained
tokens; it is not an IoU, mask-accuracy, or boundary-quality metric.

```python
from vertebrae import (
    Benchmark,
    CallableSpatialExtractor,
    SegmentationConfig,
    SegmentationDataset,
    SpatialLayout,
    SpatialOutputSpec,
)

dataset = SegmentationDataset.from_arrays(
    images=images,
    semantic_masks=masks,
    identity=DatasetIdentity.declared("example-dataset", "1"),
)

extractor = CallableSpatialExtractor(
    "encoder",
    transform_fn=extract_spatial_features,
    output_specs=[
        SpatialOutputSpec(
            name="stage_4",
            layout=SpatialLayout(grid_height=14, grid_width=14),
        )
    ],
)

result = Benchmark(
    dataset=dataset,
    extractors=[extractor],
    segmentation_config=SegmentationConfig(max_tokens_per_class=10_000),
).run()
```

See `docs/segmentation.md` for background handling, ambiguity filtering,
instance caps, grouped diagnostics, and precomputed segmentation embeddings.

### Structured units from native model outputs

Structured extractors flatten one declared per-parent unit matrix into a grouped
embedding dataset, preserving unit provenance and parent groups. This supports
representation diagnostics for regions, tokens, frames, keypoints, depth cells,
and latent slots. It does not substitute task-native metrics such as mAP, IoU,
WER/CER, OKS, depth error, or reconstruction quality.

Unit annotations may use single-label, multi-label, or one-/multi-target
regression targets, provided every parent uses the same resolved schema. Local
unit IDs may repeat across parents; materialization generates parent-aware
global IDs and retains the supplied value as `local_unit_id` in provenance.
Dense and scipy sparse per-parent matrices are supported without premature
densification.

```python
from vertebrae import Benchmark, BenchmarkDataset, UnitAnnotation, DatasetIdentity
from vertebrae.extractors import CallableStructuredExtractor, StructuredOutputSpec

dataset = BenchmarkDataset.from_arrays(X=pages, y=document_labels, modality="image", identity=DatasetIdentity.declared("example-dataset", "1"))
dataset = dataset.with_unit_annotations(
    [
        UnitAnnotation(labels=["heading", "body"]),
        UnitAnnotation(labels=["heading", "body"]),
    ],
    unit_type="document_region",
)

extractor = CallableStructuredExtractor(
    name="layout_encoder",
    transform_fn=extract_region_embeddings,  # one 2D matrix per page
    output_specs=[StructuredOutputSpec(name="regions", unit_type="document_region")],
)
result = Benchmark(dataset, [extractor]).run()
```

When the model emits unmatched rows (for example special tokens or sampled
frames), supply an explicit alignment rule such as
`drop_special_rows(leading=1)` or `select_frame_rows(...)`. Typed adapters are
also available for detection/layout, sequence labeling, keypoints, depth, and
latent slots. See
[the dataset guide](https://github.com/NiklasMelton/vertebrae/blob/develop/docs/datasets.md)
and [the extractor guide](https://github.com/NiklasMelton/vertebrae/blob/develop/docs/feature_extractors.md),
alongside the runnable
`examples/structured_*.py` workflows.

## Supported Workflows

`vertebrae` is designed for:

- precomputed dense or sparse embeddings,
- NumPy arrays and pandas DataFrames,
- single-label classification, multi-label classification, and explicit regression targets,
- hierarchy label views and named target views for scoring the same embeddings against
  different targets,
- graph-node, graph-edge, entity, pair, triplet-derived, and generic labeled-unit
  embedding diagnostics,
- scikit-learn transformers and pipelines,
- custom Python callable extractors,
- dense segmentation token materialization from spatial feature maps,
- structured unit materialization from native token, frame, region, keypoint, depth,
  or latent-slot outputs, with explicit alignment when rows do not already match,
- Hugging Face text backbones through `HFTextExtractor`,
- Hugging Face vision backbones through `HFVisionExtractor`,
- Hugging Face audio backbones through `HFAudioExtractor`,
- Hugging Face image-text and other structured multi-modal backbones through
  `HFMultimodalExtractor`,
- Hugging Face video backbones through `HFVideoExtractor`,
- Hugging Face time-series backbones through `HFTimeSeriesExtractor`,
- sentence-transformers through `SentenceTransformerExtractor`,
- timm, torchvision, OpenCLIP, SigLIP, TensorFlow Hub, JAX/Flax, tree-leaf,
  graph, and hosted embedding API extractors,
- local PyTorch modules through `TorchExtractor`,
- local Keras modules through `KerasExtractor`,
- local ONNX Runtime sessions through `ONNXExtractor`,
- single-output and multi-output extractor evaluation,
- single-extractor evaluation,
- multi-extractor comparisons,
- JSON and Markdown reports,
- repeated-run stability analysis,
- optional Separatix complexity diagnostics in local and artifact-backed reports,
- custom full-batch embedding metrics with a selectable primary ranking metric,
- exact, training-free query--gallery retrieval and matching with explicit graded relevance,
- fixed-prompt, training-free zero-shot semantic alignment for compatible text-aligned models,
- optional embedding compression and compressed-variant comparisons,
- local embedding caching and reproducible artifacts,
- artifact-backed distributed embedding and scoring through the `vertebrae` CLI,
- artifact-backed `Benchmark.run()` dispatch through explicit local, Ray, and Dask
  backends,
- local paths, `s3://...`, and `gs://...` artifact stores.

Distributed CLI commands include `vertebrae plan`, `vertebrae fit-extractor`, `vertebrae embed-shard`,
`vertebrae merge-embeddings`, `vertebrae write-labels`, `vertebrae write-groups`,
`vertebrae materialize-segmentation`, `vertebrae materialize-structured`,
`vertebrae compress`, `vertebrae score`,
`vertebrae plan-retrieval`, `vertebrae embed-retrieval-shard`,
`vertebrae merge-retrieval-embeddings`, `vertebrae write-retrieval-relevance`,
`vertebrae compress-retrieval`, `vertebrae score-retrieval`,
`vertebrae plan-zero-shot`, `vertebrae embed-zero-shot-shard`,
`vertebrae merge-zero-shot-embeddings`, `vertebrae write-zero-shot-protocol`,
`vertebrae compress-zero-shot`, `vertebrae score-zero-shot`,
`vertebrae zero-shot-from-artifacts`,
`vertebrae diagnose-complexity`, `vertebrae score-repeats`,
`vertebrae collect-scores`, `vertebrae benchmark-from-artifacts`,
`vertebrae slurm-array`, `vertebrae slurm-score-array`, and
`vertebrae run-embedding-shards`.

All CLI pickle inputs, including fitted-extractor bundles, are trusted-input-only.
Artifact-backed embedding workers are transform-only: the driver or
`fit-extractor` fits once on the complete selected dataset before shard dispatch.

Reusable artifacts use cache identity schema v2: hashes cover complete typed values.
Array manifest v2 commits immutable digest-named array data, while composite artifact
manifest v2 commits an array plus metadata or labels plus metadata as one coherent
publication. Readers validate component sizes and SHA-256 digests as well as array
shape, dtype, storage format, and sparse `nnz`.

Callable/live-model extractors that cannot prove a stable identity still evaluate but
bypass cache reuse and record `cache_status="bypassed_unsafe_identity"`. Reuse requires
an explicit `cache_identity`, a model identifier that is itself a content-digested
local path, or a remote revision pinned to a full immutable commit hash. Merely
declaring a checkpoint beside an already-loaded live model does not prove that the
object contains those weights. Importable callable identities include referenced
modules, helper callables, and exact global configuration; optional backend versions
also participate in extractor recipes. Raw-cache
eligibility propagates to compression and every derived artifact: disabling raw
caching or using an unsafe identity cannot produce a reusable derived cache. Legacy
cache identity and array/composite manifest schemas are intentionally not read;
standalone JSON and label store APIs remain current.

Streaming-safe extractors are transformed in independent batches. Every batch must
return the same unique output names, row counts, feature widths, dtypes, dense/sparse
form, recipes, metadata contracts, and parent coverage. Extractors declaring
`streaming_safe=False` receive one full transform call instead.

For Ray or Dask cluster runs, the configured `cache_dir` can be either a shared local
path or a cloud object-store URI such as `s3://team-bucket/vertebrae/run-001` or
`gs://team-bucket/vertebrae/run-001`. Workers need credentials for the selected store.

## Reports and Results

Resource profiling can be enabled when deployment cost is part of representation
selection:

```python
from vertebrae import ResourceProfilingConfig

result = Benchmark(
    dataset,
    extractors=[small_backbone, large_backbone],
    resource_profiling_config=ResourceProfilingConfig(enabled=True),
).run()

print(result.to_dataframe())
print([item.name for item in result.quality_cohort()])
```

The profile observes the benchmark's real local extraction calls. It reports
first-call and warm latency, observed throughput, host memory, supported device/model
footprints, and logical raw/compressed embedding storage. It does not change quality
ranking. A cache hit remains a cache hit and reports inference as unmeasured; use
`CacheConfig(force_recompute=True)` when a fresh inference measurement is required.

Native model/device adapters cover local Torch, Keras, ONNX, Hugging Face,
sentence-transformers, timm, torchvision, OpenCLIP/SigLIP, graph-model, and TensorFlow
Hub and JAX/Flax extractors. Device peaks are reported only for a successfully reset
single-device profiling window; cache hits, multi-device Torch, and JAX allocator
limits remain explicitly unavailable. Checkpoint sizes come only from explicit paths;
model identifiers and external caches are never scanned implicitly. Reports
distinguish complete, partial, unavailable, and CPU-not-applicable evidence.

Retrieval and zero-shot benchmarks accept the same embedding and profiling configs and
report independent endpoint profiles. Artifact-backed workers persist local observations;
merged results use a distinct distributed profile with worker-first latency and aggregate
compute throughput rather than presenting worker data as one local run. Embedding storage
reports both logical bytes and the actual persisted array-object size when available.

Latency values are meaningful only with their recorded batch size, device, precision,
and synchronization context. They are workload observations, not hardware-normalized
scores or controlled load tests.

Each benchmark run returns structured results that include:

- dataset summary,
- extractor summary and recipe metadata,
- overlap scores plus per-class, per-label, or per-target diagnostics,
- Separatix recommendation, confidence, and report details when the overlap gate passes,
- label-view, target-view, segmentation, structured-unit, grouping, and target-type
  metadata when present,
- every configured metric result and the selected primary ranking metric,
- compression metadata and compressed dimensions,
- stability summaries,
- warnings and recommendations,
- reproducibility metadata.

Only results whose selected metric declares `aggregate_valid=True` participate in
rankings, quality cohorts, top recommendations, or resource comparisons. When no
valid aggregate remains, reports say that ranking is unavailable instead of promoting
an invalid result.

Results can be rendered directly to Markdown or JSON:

```python
result.save_json("result.json")
result.save_markdown("report.md")
```

You can also convert rankings into a DataFrame with `result.to_dataframe()`.

Compression-aware results include the compression method and compressed
dimension, and quantized runs preserve calibration metadata in the structured
result payload.

Separatix is stored as the default classifier-complexity report field. JSON
output preserves the full Separatix report. Markdown and DataFrame views
surface the main recommendation, confidence, and compact explanation fields
that are usually the most actionable.

Separatix is also the source of probe-style summary fields, so vertebrae does not
fit a second probe system alongside the complexity diagnostic.

The key component of the report is the performance and comparison table. The
following is the current output generated by
`examples/sklearn_wine_pipeline.py`.

| rank | extractor | primary_metric | primary_score | overlap_macro | stability_interval | weakest_class | best_probe | probe_metric | probe_score | embedding_dim | recommendation |
| --- | --- | --- | ---: | ---: | --- | --- | --- | --- | ---: | ---: | --- |
| 1 | wine_standard_scaler_all_features | overlap | 0.9232 | 0.9232 | 0.9058-0.9279 | class_1 | linear | balanced_accuracy | 0.9812 | 13 | strong_candidate |
| 2 | wine_standard_scaler_pca_6 | overlap | 0.9179 | 0.9179 | 0.9051-0.9368 | class_1 | kernel_approx | balanced_accuracy | 0.9767 | 6 | strong_candidate |
| 3 | wine_minmax_pca_2 | overlap | 0.9079 | 0.9079 | 0.9366-0.9455 | class_1 | smooth_poly | balanced_accuracy | 0.9756 | 2 | strong_candidate |
| 4 | wine_quantile_pca_1 | overlap | 0.4554 | 0.4554 | 0.3455-0.4554 | class_2 |  |  |  | 1 | poor_frozen_representation_weak_class_attention |

The full generated table also includes `target_view`, `label_view`,
`overlap_score`, `overlap_weighted`, compression fields, and Separatix fields.
The compact table above keeps the README readable while preserving the values used
for the ranking.

By default, extractors are ranked by overlap. When a custom `primary_metric` is
configured, they are ranked by that metric instead; the overlap columns remain
available for representation diagnostics and Separatix gating.

The easiest way to interpret the report is:

- Start with `primary_metric` and `primary_score`. By default these are overlap;
  with a custom metric they identify the configured ranking signal.
- Inspect `overlap_macro` and per-class overlap scores as the standard vertebrae
  representation diagnostic, even when another metric ranks the candidates.
- Use the vertebrae `recommendation` field as a quick summary of representation quality under the benchmark protocol.
- Use `separatix_recommendation` and `separatix_confidence` to understand what kind of downstream classifier complexity the labeled embedding seems to imply once the representation is already reasonably separated.
- When Separatix columns are blank, the diagnostic was disabled, did not clear the
  overlap gate, or did not return that field. Check the per-extractor skip reason
  before interpreting the blank.
- Treat `best_probe`, `probe_metric`, and `probe_score` as Separatix-derived
  quick-check columns. They are blank when Separatix is disabled, skipped, or does
  not report a baseline probe score. The metric is target-aware rather than always
  being accuracy.

Problem classes change the interpretation and detail columns, not the overall report
workflow:

| problem class | target metadata | detailed overlap output | typical probe metric |
| --- | --- | --- | --- |
| single-label classification | class names and counts | per-class and pairwise scores | balanced accuracy |
| multi-label classification | label names, cardinality, and density | per-label scores | macro/micro F1 or sample Jaccard |
| regression | target names and target statistics | per-target continuous overlap | R² or a regression-specific metric |
| hierarchy or named target views | active view and available views | one ranked result per evaluated view | follows the projected target type |
| dense segmentation | source-image groups and token provenance | per-class token overlap | classification metric when applicable |

Two other network-free examples exercise the same report schema with different
target types:

| problem class | generated by | extractor | overlap_macro | weakest target | recommendation |
| --- | --- | --- | ---: | --- | --- |
| multi-label classification | `examples/multilabel_precomputed_embeddings.py` | toy_multilabel_embeddings | 0.0000 | outdoor | poor_frozen_representation_weak_class_attention |
| structured regression | `examples/structured_depth.py` | depth_samples:depth_cells | 0.9732 |  | continuous_structure_above_null |

The toy multi-label result is intentionally not presented as a strong representation:
its labels cross the three embedding clusters, and the report correctly flags the
weak result. The structured-depth result evaluates continuous target structure; it
is not an RMSE or depth-estimation benchmark.

See [results and reports](docs/results_and_reports.md) for the complete schema and
the additional metadata retained for multi-output, structured, relational, and
zero-shot workflows.

In the per-extractor Markdown section, Separatix also adds:

- a plain-language recommendation text,
- a decision path showing the main diagnostic branches,
- normalized summary scores such as signal, overlap, linearity, nonlinearity, and reliability,
- warnings and skipped diagnostics when part of the complexity audit did not run.

As a rule of thumb, a strong overlap score plus `linear_likely_sufficient`
usually points to an embedding that should work well with simple downstream
classifiers, while a strong overlap score plus
`smooth_nonlinear_recommended` or `kernel_or_local_recommended` suggests the
embedding is promising but may benefit from a more flexible decision boundary.

## Optional Extractors

Optional integrations are available through extras such as `torch`, `keras`,
`tensorflow`, `onnx`, `hf`, `timm`, `torchvision`, `openclip`,
`tensorflow-hub`, `jax`, `trees`, and `graph`:

- `TorchExtractor`
- `KerasExtractor`
- `ONNXExtractor`
- `SentenceTransformerExtractor`
- `HFTextExtractor`
- `HFVisionExtractor`
- `HFAudioExtractor`
- `HFTimeSeriesExtractor`
- `HFVideoExtractor`
- `HFMultimodalExtractor`
- `TimmVisionExtractor`
- `TorchvisionVisionExtractor`
- `OpenCLIPExtractor`
- `SigLIPExtractor`
- `TFHubExtractor`
- `JAXFlaxExtractor`
- `TreeLeafEmbeddingExtractor`
- `GraphModelExtractor`
- `HostedEmbeddingExtractor`
- `CallableSpatialExtractor`
- `PrecomputedSpatialExtractor`
- `CallableStructuredExtractor`
- `PrecomputedStructuredExtractor`

These workflows rely on optional dependencies and lazy imports, so the core package stays lightweight.
See `examples/onnx_extractor.py` for a local ONNX export workflow.
See `examples/hf_vision_mnist.py` for a laptop-friendly comparison that runs
MNIST handwritten digit data through final and mid-layer Hugging Face vision
embeddings, and
`examples/caltech101_vision_foundation_models.py` for a single-label Caltech-101
workflow with automatic local data reuse/downloads and a less trivial default
class slice. See
`examples/sklearn_wine_pipeline.py` for a network-free real-data scikit-learn
pipeline comparison.
See `docs/feature_extractors.md` for the full extractor matrix and install
mapping, `docs/segmentation.md` for dense spatial workflows, and
`docs/compression.md` for compression options and guidance.

## Command Line Interface

`vertebrae` includes a CLI for deterministic embedding shard planning and artifact
merging in local or batch-style workflows. Distributed orchestration commands accept
`--backend local|ray|dask`, with `--ray-address` and `--dask-address` available for
cluster connections. Cloud artifact stores use the same `--cache-dir` flag, plus
provider options such as `--s3-endpoint-url`, `--s3-profile`, `--s3-region`, and
`--gcs-project`. The CLI can also derive compressed embedding artifacts with
`vertebrae compress`, materialize structured unit artifacts with
`vertebrae materialize-structured`, and evaluate importable custom metrics through
repeatable `vertebrae score --metric module:callable` options. Run
`vertebrae --help` to see the available commands.

## Notes

- The package targets Python `>=3.9,<3.15`.
- The public API is centered on `BenchmarkDataset`, `EmbeddingUnitDataset`,
  `SegmentationDataset`, `RetrievalDataset`, `ZeroShotDataset`, `Evaluator`,
  `Benchmark`, `RetrievalBenchmark`, `ZeroShotBenchmark`, structured and spatial
  adapters, extractor wrappers, metric adapters, config dataclasses, and structured
  result objects.

## License

MIT

