Metadata-Version: 2.5
Name: canvit-pytorch
Version: 0.2.1
Summary: CanViT (Canvas Vision Transformer): model, pretraining, task specialization and evaluation
Project-URL: Repository, https://github.com/m2b3/CanViT
Project-URL: Homepage, https://m2b3.github.io/CanViT/
Project-URL: Paper, https://arxiv.org/abs/2603.22570
Project-URL: Checkpoints, https://huggingface.co/canvit
Author-email: Yohaï-Eliel Berreby <me@yberreby.com>, Sabrina Du <sabrina.du@mail.mcgill.ca>
License-Expression: MIT
License-File: LICENSE
Keywords: active-vision,canvit,foundation-model,pytorch,vision-transformer
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Science/Research
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.12
Requires-Dist: canvit-core<0.3.0,>=0.2.1
Requires-Dist: huggingface-hub>=1.3.2
Requires-Dist: numpy<2.4.0,>=2.2.0
Requires-Dist: safetensors>=0.7.0
Requires-Dist: torch>=2.14.0; sys_platform == 'darwin'
Requires-Dist: torch>=2.9.0
Requires-Dist: torchvision>=0.22.0
Provides-Extra: demo
Requires-Dist: matplotlib>=3.10.8; extra == 'demo'
Requires-Dist: pillow>=10.0.0; extra == 'demo'
Requires-Dist: tyro>=1.0.3; extra == 'demo'
Provides-Extra: evaluate
Requires-Dist: pandas>=2.0.0; extra == 'evaluate'
Requires-Dist: pillow>=10.0.0; extra == 'evaluate'
Requires-Dist: pyarrow>=20.0.0; extra == 'evaluate'
Requires-Dist: tqdm>=4.60.0; extra == 'evaluate'
Requires-Dist: transformers>=4.50.0; extra == 'evaluate'
Requires-Dist: tyro>=1.0.0; extra == 'evaluate'
Provides-Extra: live
Requires-Dist: onnx>=1.23.0; extra == 'live'
Requires-Dist: onnxruntime>=1.30.0; extra == 'live'
Requires-Dist: onnxscript>=0.7.2; extra == 'live'
Requires-Dist: pillow>=10.0.0; extra == 'live'
Requires-Dist: playwright>=1.63.0; extra == 'live'
Requires-Dist: tyro>=1.0.0; extra == 'live'
Provides-Extra: pretrain
Requires-Dist: comet-ml; extra == 'pretrain'
Requires-Dist: matplotlib; extra == 'pretrain'
Requires-Dist: pyarrow; extra == 'pretrain'
Requires-Dist: torch>=2.10; extra == 'pretrain'
Requires-Dist: transformers>=4.50.0; extra == 'pretrain'
Requires-Dist: tyro; extra == 'pretrain'
Requires-Dist: xxhash; extra == 'pretrain'
Provides-Extra: specialize
Requires-Dist: comet-ml>=3.30.0; extra == 'specialize'
Requires-Dist: dinov3; extra == 'specialize'
Requires-Dist: matplotlib>=3.8.0; extra == 'specialize'
Requires-Dist: pillow>=10.0.0; extra == 'specialize'
Requires-Dist: tqdm>=4.60.0; extra == 'specialize'
Requires-Dist: transformers>=4.50.0; extra == 'specialize'
Requires-Dist: tyro>=1.0.0; extra == 'specialize'
Provides-Extra: viz
Requires-Dist: pillow>=10.0.0; extra == 'viz'
Requires-Dist: tyro>=1.0.0; extra == 'viz'
Description-Content-Type: text/markdown

# canvit-pytorch

The PyTorch implementation of CanViT (Canvas Vision Transformer). It
contains the model, checkpoint loading, preprocessing, policies, pretraining,
task specialization, and evaluation code. The paper is
[CanViT: Toward Active-Vision Foundation Models](https://arxiv.org/abs/2603.22570) (NeurIPS 2026).

## Install

```bash
uv add canvit-pytorch
```

## Load a Hugging Face checkpoint

This example loads a PyTorch checkpoint, processes a user-provided image, and
returns the recurrent state after one glimpse. Replace the image path with an
RGB image available to your environment.

```python
import torch
from PIL import Image

from canvit_pytorch import CanViTForPretraining, Viewpoint, sample_at_viewpoint
from canvit_pytorch.preprocess import preprocess

scene = preprocess(512)(Image.open("path/to/image.jpg").convert("RGB")).unsqueeze(0)
model = CanViTForPretraining.from_pretrained(
    "canvit/canvitb16-add-vpe-pretrain-g128px-s512px-in21k-dv3b16-2026-02-02"
).eval()
viewpoint = Viewpoint.full_scene(batch_size=1, device=scene.device)
state = model.init_state(batch_size=1, canvas_grid_size=32)
glimpse = sample_at_viewpoint(spatial=scene, viewpoint=viewpoint, glimpse_size_px=128)

with torch.inference_mode():
    output = model(glimpse=glimpse, state=state, viewpoint=viewpoint)
print(output.state.recurrent_cls.shape)  # [1, 1, 768]
```

The example uses the [ImageNet-21k pretrained CanViT-B checkpoint](https://huggingface.co/canvit/canvitb16-add-vpe-pretrain-g128px-s512px-in21k-dv3b16-2026-02-02).
The [fine-tuned ImageNet-1k classifier](https://huggingface.co/canvit/canvitb16-add-vpe-finetune-g128px-s512px-in1k-2026-04-06)
and separate task probes are also available from the [PyTorch checkpoints](https://huggingface.co/canvit).
The root [README](https://github.com/m2b3/CanViT) documents classification,
segmentation, native backends, and the training and evaluation entry points.
PyTorch inputs use NCHW: scenes and feature maps are
`[B, C, H, W]`, while model glimpses are `[B, 3, h, w]`. Viewpoint centers use
`(row, col)` scene coordinates in `[-1, 1]`; each scale is the crop's
half-side.

MIT license. See [LICENSE](https://github.com/m2b3/CanViT/blob/main/LICENSE).
