Metadata-Version: 2.4
Name: visin-fusion
Version: 0.7.0
Summary: Camera and LiDAR fusion models and training tools
Author: Toomas Tahves
License-Expression: MIT
Project-URL: Homepage, https://github.com/visin-platform/visin-fusion
Project-URL: Documentation, https://visin-platform.github.io/visin-fusion/
Project-URL: Changelog, https://github.com/visin-platform/visin-fusion/blob/main/CHANGELOG.md
Project-URL: Issues, https://github.com/visin-platform/visin-fusion/issues
Keywords: semantic segmentation,sensor fusion,lidar,autonomous driving,pytorch
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Science/Research
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Scientific/Engineering :: Image Recognition
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: torch>=2.2
Requires-Dist: torchvision>=0.17
Requires-Dist: timm>=1.0
Requires-Dist: einops>=0.7
Requires-Dist: numpy
Requires-Dist: scipy
Requires-Dist: pydantic>=2
Requires-Dist: tqdm
Requires-Dist: tensorboard
Requires-Dist: Pillow>=9.1
Requires-Dist: opencv-python
Requires-Dist: pandas
Requires-Dist: psutil
Requires-Dist: nvidia-ml-py
Requires-Dist: GPUtil
Requires-Dist: thop
Requires-Dist: fvcore
Provides-Extra: train
Provides-Extra: visin
Requires-Dist: visin; extra == "visin"
Requires-Dist: python-dotenv; extra == "visin"
Requires-Dist: requests; extra == "visin"
Provides-Extra: onnx
Requires-Dist: onnx>=1.16; extra == "onnx"
Requires-Dist: onnxruntime>=1.18; extra == "onnx"
Provides-Extra: hf
Requires-Dist: huggingface_hub>=0.24; extra == "hf"
Provides-Extra: dev
Requires-Dist: pytest; extra == "dev"
Requires-Dist: pytest-cov; extra == "dev"
Requires-Dist: coverage>=7.10; extra == "dev"
Requires-Dist: ruff>=0.6; extra == "dev"
Requires-Dist: mypy>=1.10; extra == "dev"
Requires-Dist: pre-commit>=3.5; extra == "dev"
Requires-Dist: build>=1.0; extra == "dev"
Requires-Dist: twine>=6.1; extra == "dev"
Dynamic: license-file

# Visin Fusion

A Python library for semantic segmentation from camera images and projected LiDAR. Five research
model families share one call: `model(rgb, lidar)` returns class logits shaped
`[batch, classes, height, width]`.

## Install

```bash
python -m pip install visin-fusion
```

From a source checkout instead: `git clone https://github.com/visin-platform/visin-fusion.git`, then `python -m pip install -e .` in it.

The install provides the models and the training and evaluation pipeline. Add `.[visin]` for Visin
reporting and datasets, or `.[dev]` for tests. See the
[installation and API guide](docs/library.md) for details.

## Use a model

<!-- doctest -->
```python
import torch
from visin_fusion.models import CLFTv2

model = CLFTv2(num_classes=4, mode="cross_fusion", pretrained=False).eval()
rgb = torch.randn(1, 3, 256, 256)
lidar = torch.randn(1, 3, 256, 256)  # camera-aligned LiDAR projection
with torch.inference_mode():
    logits = model(rgb, lidar)  # [1, 4, 256, 256]
```

| Model | Main difference |
| --- | --- |
| [CLFT](docs/architecture/CLFT.md) | Global ViT features fused during decoding |
| [CLFTv2](docs/architecture/CLFTv2.md) | Hierarchical Swin fusion pyramid |
| [MaskFormerFusion](docs/architecture/MaskFormer.md) | Class-labelled mask queries |
| [Mask2FormerFusion](docs/architecture/Mask2Former.md) | Multi-scale masked-attention queries |
| [DeepLabV3+](docs/architecture/DeepLabV3Plus.md) | Two convolutional branches with late fusion |

Each model page links its research papers. See [model comparison](docs/models.md) for the fusion points
and [Python API](docs/reference/python-api.md) for modes, training and checkpoints. The LiDAR input is
a three-channel projected image, not a raw point cloud.

## Optional pipeline

For ZOD, Waymo, and Iseauto datasets, follow [Download and train](docs/download-and-train.md):
install, download, and run the pipeline with CLI commands.

Train, test, visualize and benchmark CLFTv2 on the included sample dataset (the [getting started](docs/getting-started.md) page also shows how to use your own data):

```bash
visin-fusion quickstart
```

The [pipeline guide](docs/running.md) covers configs, Docker, SLURM and individual stages. Add
`.[visin]` and a Visin pipeline key to report runs; see the [Visin integration guide](docs/visin.md). To rank checkpoints
on a written-down protocol, see [ranking on a suite](docs/evaluation.md).

## Tests

```bash
python -m pip install -e '.[visin,dev]'
pytest tests/unit
python tools/coverage.py  # unit + end-to-end; requires >90% package coverage
```

See [tests/README.md](tests/README.md) for focused end-to-end commands. The full documentation is at
[visin-platform.github.io/visin-fusion](https://visin-platform.github.io/visin-fusion/).

## License

[MIT](LICENSE).
