Metadata-Version: 2.4
Name: tpips
Version: 0.1.2
Summary: TPIPS: Text-Prompted Image Perceptual Similarity
Project-URL: Source, https://github.com/adobe-research/TPIPS
Project-URL: Homepage, https://peterwang512.github.io/TPIPS/
Project-URL: Documentation, https://github.com/adobe-research/TPIPS/blob/main/DOCUMENTATION.md
Project-URL: Paper, https://arxiv.org/abs/2607.18237
Project-URL: Issues, https://github.com/adobe-research/TPIPS/issues
License-Expression: LicenseRef-Adobe-Research-License
License-File: LICENSE
Requires-Python: >=3.10
Requires-Dist: huggingface-hub>=0.26
Requires-Dist: numpy>=1.26
Requires-Dist: peft>=0.19
Requires-Dist: pillow>=10
Requires-Dist: torch>=2.7
Requires-Dist: torchvision>=0.22
Requires-Dist: transformers>=5.0
Description-Content-Type: text/markdown

# TPIPS API reference

TPIPS is a text-conditioned perceptual image metric. It compares images with
respect to a text factor/aspect such as `"lighting"`, `"pose"`, or `"overall"`.

**On this page**

- [`tpips.load_model`](#tpipsload_model)
- [`model.similarity`](#modelsimilarity)
- [`model.distance`](#modeldistance)
- [`model.embed`](#modelembed)

TPIPS requires Python 3.10 or later. Install matching PyTorch and torchvision
builds from the [official PyTorch installer](https://pytorch.org/get-started/locally/),
then install the package:

```bash
pip install tpips
```

For datasets, evaluation, training, and the interactive demo, see the
[repository README](https://github.com/adobe-research/TPIPS/blob/main/README.md).

## `tpips.load_model`

```python
tpips.load_model(
    model_type,
    model_path=None,
    device="cuda",
    *,
    cache_dir=None,
)
```

Loads a released TPIPS model from Hugging Face or a compatible local checkpoint.
The returned model is placed on `device` and set to evaluation mode.

**Parameters**

- **model_type** (`str`) – Selects the TPIPS metric architecture.

  | Value | Supported methods | Description |
  | --- | --- | --- |
  | `"embedding"` | `similarity`, `distance`, `embed` | Encodes each text-image input independently. This is the recommended default and the only variant with reusable image embeddings. |
  | `"early_fusion"` | `similarity`, `distance` | Jointly encodes each aligned image pair. Its exchange-equivariant attention design makes scores invariant to image order, apart from numerical noise. |
  | `"activation_dist"` | `distance` | Computes a learned distance from text-conditioned features at several backbone layers. It does not define cosine similarity or independent image embeddings. |

- **model_path** (`str` or `os.PathLike`, optional) – Hugging Face repository
  ID or local checkpoint directory. If omitted, loads the official checkpoint
  for `model_type`.
- **device** (`str`, optional) – Torch destination such as `"cuda"`,
  `"cuda:1"`, or `"cpu"`. Default: `"cuda"`.
- **cache_dir** (`str` or `os.PathLike`, optional) – Hugging Face cache
  override for the TPIPS checkpoint, Qwen backbone, and processor. By default,
  Hugging Face uses its standard cache and environment variables such as
  `HF_HOME`.

**Returns**

A device-resident `EmbeddingModel`, `EarlyFusionModel`, or
`ActivationDistModel` in evaluation mode.

**Raises**

- **ValueError** – If `model_type` is unsupported; a local path names a file
  rather than a directory; or the checkpoint configuration is invalid or does
  not match the requested model type.
- **FileNotFoundError** – If a checkpoint directory does not contain
  `config.json` or `model.pt`.
- **RuntimeError** – If a Hub download, checkpoint read, model construction,
  state-dict load, or device transfer fails.
- **TypeError** – If a path or device argument has an unsupported type.

**Checkpoint configuration**

A compatible checkpoint directory contains `config.json` and `model.pt`.
The configuration declares the metric architecture and backbone family:

```json
{
  "model": {
    "type": "embedding",
    "backbone_family": "qwen3_vl_embed",
    "name": "Qwen/Qwen3-VL-Embedding-8B"
  }
}
```

`model.type` must match `model_type`. `model.backbone_family` currently must
be `qwen3_vl_embed`, and `model.name` identifies the concrete Hugging Face
backbone. Official TPIPS releases already contain the correct configuration.

The default checkpoints are:

- [Embedding](https://huggingface.co/sywang/TPIPS-Embed-Qwen3VL-8B)
- [Early fusion](https://huggingface.co/sywang/TPIPS-EarlyFusion-Qwen3VL-8B)
- [Activation distance](https://huggingface.co/sywang/TPIPS-ActDiff-Qwen3VL-8B)

**Notes**

The first call downloads both the TPIPS checkpoint and its Qwen backbone.
Subsequent calls reuse the Hugging Face cache. A CUDA GPU is recommended.
FlashAttention is optional because TPIPS falls back to PyTorch SDPA.

Evaluation mode disables dropout but does not disable autograd. Use
`torch.inference_mode()` for ordinary inference, as shown below.

Direct construction is available from the metric packages:

```python
from tpips.embedding import EmbeddingQwen3VLEmbedModel
from tpips.early_fusion import EarlyFusionQwen3VLEmbedModel
from tpips.activation_dist import ActivationDistQwen3VLEmbedModel
```

Constructing these classes initializes a Qwen backbone and processor but does
not load a released TPIPS `model.pt`. Direct construction is intended for
architecture development or manually loading a compatible state dict. Normal
inference code should use `tpips.load_model`.

**Examples**

```python
import torch
import tpips
from PIL import Image

model = tpips.load_model("embedding", device="cuda")

image_a = Image.open("a.jpg").convert("RGB")
image_b = Image.open("b.jpg").convert("RGB")

with torch.inference_mode():
    similarity = model.similarity(image_a, image_b, factor="lighting")
    distance = model.distance(image_a, image_b, factor="lighting")

print(similarity.item())  # higher means more similar
print(distance.item())    # lower means more similar
```

Load another Hub repository, a local checkpoint, or a dedicated cache:

```python
hub_model = tpips.load_model(
    "embedding",
    "organization/custom-tpips",
    device="cuda:1",
    cache_dir="/data/huggingface-cache",
)

local_model = tpips.load_model(
    "early_fusion",
    "/models/my-tpips",
    device="cpu",
)
```

## `model.similarity`

```python
model.similarity(
    img_a,
    img_b,
    factor="overall",
    *,
    value_range=(0.0, 1.0),
    channel_last=False,
)
```

Computes text-conditioned cosine similarity for aligned image pairs. Higher
values mean that the images are more similar with respect to `factor`.

This method is implemented by the `embedding` and `early_fusion` variants.
`activation_dist.similarity()` raises `NotImplementedError`; use
`distance()` for that model.

**Parameters**

- **img_a** – First image or explicit image batch.
- **img_b** – Second image or explicit image batch. Images are paired
  element-by-element with `img_a`.
- **factor** (`str`, optional) – One text criterion used for the entire call,
  such as `"lighting"`, `"subject color"`, or `"composition"`. Use
  `"overall"` or `None` for general perceptual similarity. Default:
  `"overall"`.
- **value_range** (two-element tuple, optional) – Inclusive finite bounds for
  every numeric image. Numeric pixels are linearly mapped from this range to
  `[0, 1]`. Use `(0, 255)` for byte-valued input, including `uint8`
  arrays. PIL images ignore this argument. Default: `(0.0, 1.0)`.
- **channel_last** (`bool`, optional) – Numeric image layout. `False` means
  CHW or BCHW; `True` means HWC or BHWC. PIL images ignore this argument.
  Default: `False`.

**Image inputs**

| Input | Interpretation | `channel_last=False` | `channel_last=True` |
| --- | --- | --- | --- |
| `PIL.Image.Image` | One image | Any PIL mode | Same; converted to RGB |
| 3-dimensional NumPy array | One image | `(3, H, W)` | `(H, W, 3)` |
| 3-dimensional Torch tensor | One image | `(3, H, W)` | `(H, W, 3)` |
| 4-dimensional NumPy array | Explicit batch | `(B, 3, H, W)` | `(B, H, W, 3)` |
| 4-dimensional Torch tensor | Explicit batch | `(B, 3, H, W)` | `(B, H, W, 3)` |
| Python `list` | Explicit batch | PIL or 3-dimensional numeric items | Layout applies to numeric items |

A list or 4-dimensional value is an explicit batch even when it contains one
image. A Python list must be non-empty and may mix PIL images, 3-dimensional
NumPy arrays, and 3-dimensional Torch tensors. It cannot contain a nested
4-dimensional batch. One `value_range` and one `channel_last` setting apply
to every numeric image in the call.

Numeric images must have exactly three channels, non-zero height and width, and
finite pixels inside `value_range`. NumPy integer and floating dtypes are
supported. Torch tensors must be real, non-boolean, non-complex, and
non-quantized.

`img_a` and `img_b` must contain the same number of images. TPIPS pairs
`img_a[i]` only with `img_b[i]`; it does not broadcast inputs or produce an
all-pairs matrix. A singular image may be paired with an explicit one-image
batch, in which case the output remains an explicit batch.

**Returns**

A device-resident Torch tensor:

- Shape `()` when both inputs are singular.
- Shape `(B,)` when either input is an explicit batch, including `(1,)` for
  a one-image batch.

Embedding and early-fusion similarity are cosine similarities. The
early-fusion attention design makes the result invariant to swapping the input
images, apart from numerical noise.

**Raises**

- **NotImplementedError** – If called on `activation_dist`.
- **TypeError** – If an image type, numeric dtype, or `value_range` bound has
  an unsupported type.
- **ValueError** – If an input is empty; dimensions, channels, or
  `value_range` are invalid; pixels are non-finite or outside the declared
  range; a list contains a 4-dimensional batch; or pair counts differ.
- **RuntimeError** – If model preprocessing or inference fails.

**Autograd**

TPIPS does not enter `torch.no_grad()` or `torch.inference_mode()`.
Floating Torch inputs retain their autograd connection through normalization,
Qwen preprocessing, the backbone, and the TPIPS head. PIL and NumPy inputs
cannot carry a PyTorch input-gradient graph.

To compute image gradients without accumulating model-parameter gradients:

```python
model.requires_grad_(False)
image_a = image_a.detach().requires_grad_(True)
image_b = image_b.detach().requires_grad_(True)

score = model.similarity(image_a, image_b, factor="overall")
score.backward()

gradient_a = image_a.grad
gradient_b = image_b.grad
```

Backpropagation retains the large backbone's activations and uses substantially
more memory than inference.

**Examples**

Compare HWC `uint8` images:

```python
import numpy as np
from PIL import Image

image_a = np.asarray(Image.open("a.jpg").convert("RGB"))
image_b = np.asarray(Image.open("b.jpg").convert("RGB"))

score = model.similarity(
    image_a,
    image_b,
    factor="lighting",
    channel_last=True,
    value_range=(0, 255),
)
```

Compare an aligned batch:

```python
import torch
from PIL import Image

left = [Image.open("a0.jpg"), Image.open("a1.jpg")]
right = [Image.open("b0.jpg"), Image.open("b1.jpg")]

with torch.inference_mode():
    scores = model.similarity(left, right, factor="lighting")

assert scores.shape == (2,)
```

## `model.distance`

```python
model.distance(
    img_a,
    img_b,
    factor="overall",
    *,
    value_range=(0.0, 1.0),
    channel_last=False,
)
```

Computes text-conditioned distance for aligned image pairs. Lower values mean
that the images are more similar with respect to `factor`.

This method is implemented by all three model variants.

**Parameters**

- **img_a**, **img_b** – Images or aligned image batches in the formats
  documented by `similarity()`.
- **factor** (`str`, optional) – Text criterion used for the entire call.
  Default: `"overall"`.
- **value_range** (two-element tuple, optional) – Inclusive numeric pixel range.
  Default: `(0.0, 1.0)`.
- **channel_last** (`bool`, optional) – Whether numeric inputs use HWC or BHWC
  rather than CHW or BCHW. Default: `False`.

Input conversion, validation, aligned batching, output shapes, and autograd
behavior are identical to `similarity()`.

**Returns**

A device-resident Torch tensor with shape `()` for singular inputs or `(B,)`
for an explicit batch.

For `embedding` and `early_fusion`, distance is exactly
`1 - similarity`. For `activation_dist`, it is a learned non-negative
feature distance and is not a cosine distance. Scores from different variants
should not be compared numerically.

**Raises**

- **TypeError**, **ValueError**, **RuntimeError** – Under the same input and
  execution conditions documented by `similarity()`.

**Notes**

Activation distance requires aligned patch grids. When the two images in a pair
have different spatial sizes, the model resizes the larger image to the smaller
exact height and width with antialiased bicubic Torch interpolation and emits a
`UserWarning`.

**Examples**

```python
import torch
import tpips

model = tpips.load_model("activation_dist", device="cuda")

with torch.inference_mode():
    distances = model.distance(batch_a, batch_b, factor="composition")
```

## `model.embed`

```python
model.embed(
    images,
    factor="overall",
    normalized=False,
    *,
    value_range=(0.0, 1.0),
    channel_last=False,
)
```

Encodes images into reusable text-conditioned vectors. This method is available
only on the `embedding` model. `early_fusion.embed()` and
`activation_dist.embed()` raise `NotImplementedError` because those
architectures do not define independent image embeddings.

**Parameters**

- **images** – One image or an explicit image batch in the formats documented
  by `similarity()`.
- **factor** (`str`, optional) – Text criterion used for every image. Default:
  `"overall"`.
- **normalized** (`bool`, optional) – If `True`, returns unit-L2 vectors. If
  `False`, returns raw vectors. Default: `False`.
- **value_range** (two-element tuple, optional) – Inclusive numeric pixel range.
  Default: `(0.0, 1.0)`.
- **channel_last** (`bool`, optional) – Whether numeric inputs use HWC or BHWC
  rather than CHW or BCHW. Default: `False`.

Image conversion, validation, explicit-batch semantics, and autograd behavior
are the same as for `similarity()`.

**Returns**

A device-resident float32 Torch tensor:

- Shape `(D,)` for a singular image.
- Shape `(B, D)` for an explicit batch, including `(1, D)` for a one-image
  batch.

Rows preserve input order. The factor is part of the embedding input, so only
compare or index vectors generated with the same factor.

**Raises**

- **NotImplementedError** – If called on `early_fusion` or
  `activation_dist`.
- **TypeError**, **ValueError**, **RuntimeError** – Under the same input and
  execution conditions documented by `similarity()`.

**Examples**

Reuse normalized embeddings to compare one query against a gallery:

```python
import torch
import tpips

model = tpips.load_model("embedding", device="cuda")

with torch.inference_mode():
    query = model.embed(query_image, factor="style", normalized=True)
    gallery = model.embed(gallery_images, factor="style", normalized=True)
    similarities = gallery @ query

assert similarities.shape == (len(gallery_images),)
```

# Citation

If you find TPIPS useful for your research, please cite:

```bibtex
@article{wang2026tpips,
      title={The Many Senses of Visual Similarity: A Text-Prompted Image Perceptual Metric},
      author={Wang, Sheng-Yu and Nitzan, Yotam and Hertzmann, Aaron and Zhu, Jun-Yan and Shechtman, Eli and Efros, Alexei A. and Zhang, Richard},
      journal={arXiv preprint arXiv:2607.18237},
      year={2026}
}
```

# License

TPIPS is provided under the [Adobe Research License](https://github.com/adobe-research/TPIPS/blob/main/LICENSE) for noncommercial research use. See the license for the complete terms.
