Metadata-Version: 2.4
Name: rf-cutie
Version: 1.0.1
Summary: Cutie: a video object segmentation framework (Roboflow-maintained fork for reliable packaging and PyPI releases).
Author-email: Rex Cheng <hkchengrex@gmail.com>
Maintainer-email: Roboflow <help@roboflow.com>
License-Expression: MIT
Project-URL: Homepage, https://github.com/roboflow/rf-Cutie
Project-URL: Source, https://github.com/roboflow/rf-Cutie
Project-URL: Repository, https://github.com/roboflow/rf-Cutie.git
Project-URL: Issues, https://github.com/roboflow/rf-Cutie/issues
Project-URL: Bug Tracker, https://github.com/roboflow/rf-Cutie/issues
Project-URL: Changelog, https://github.com/roboflow/rf-Cutie/releases
Project-URL: Upstream Project, https://github.com/hkchengrex/Cutie
Project-URL: Paper, https://arxiv.org/abs/2310.12982
Keywords: video-object-segmentation,computer-vision,deep-learning,pytorch,segmentation,cutie
Classifier: Development Status :: 5 - Production/Stable
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Operating System :: OS Independent
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Scientific/Engineering :: Image Recognition
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: torch
Requires-Dist: numpy>=1.21
Requires-Dist: einops>=0.6
Requires-Dist: omegaconf>=2.3
Provides-Extra: inference
Requires-Dist: hydra-core>=1.3.2; extra == "inference"
Requires-Dist: Pillow>=9.5; extra == "inference"
Requires-Dist: torchvision; extra == "inference"
Requires-Dist: requests; extra == "inference"
Requires-Dist: tqdm>=4.66.1; extra == "inference"
Provides-Extra: evaluation
Requires-Dist: hydra-core>=1.3.2; extra == "evaluation"
Requires-Dist: Pillow>=9.5; extra == "evaluation"
Requires-Dist: torchvision; extra == "evaluation"
Requires-Dist: tqdm>=4.66.1; extra == "evaluation"
Requires-Dist: pycocotools>=2.0.7; extra == "evaluation"
Requires-Dist: hickle>=5.0; extra == "evaluation"
Provides-Extra: train
Requires-Dist: gitpython>=3.1; extra == "train"
Requires-Dist: thin-plate-spline>=1.2.2; extra == "train"
Requires-Dist: tensorboard>=2.11; extra == "train"
Requires-Dist: Pillow>=9.5; extra == "train"
Requires-Dist: opencv-python>=4.8; extra == "train"
Requires-Dist: hydra-core>=1.3.2; extra == "train"
Requires-Dist: torchvision; extra == "train"
Provides-Extra: gui
Requires-Dist: Pillow>=9.5; extra == "gui"
Requires-Dist: opencv-python>=4.8; extra == "gui"
Requires-Dist: scipy>=1.7; extra == "gui"
Requires-Dist: tqdm>=4.66.1; extra == "gui"
Requires-Dist: hydra-core>=1.3.2; extra == "gui"
Requires-Dist: PySide6>=6.2.0; extra == "gui"
Requires-Dist: av>=0.5.2; extra == "gui"
Requires-Dist: requests; extra == "gui"
Requires-Dist: torchvision; extra == "gui"
Provides-Extra: video
Requires-Dist: Pillow>=9.5; extra == "video"
Requires-Dist: opencv-python>=4.8; extra == "video"
Requires-Dist: hydra-core>=1.3.2; extra == "video"
Requires-Dist: pycocotools>=2.0.7; extra == "video"
Requires-Dist: tqdm>=4.66.1; extra == "video"
Provides-Extra: data
Requires-Dist: Pillow>=9.5; extra == "data"
Requires-Dist: pycocotools>=2.0.7; extra == "data"
Requires-Dist: tqdm>=4.66.1; extra == "data"
Requires-Dist: hickle>=5.0; extra == "data"
Dynamic: license-file

# [Putting the Object Back into Video Object Segmentation](https://hkchengrex.github.io/Cutie)

> **Fork notice:** This fork packages and maintains Cutie for reliable installation and PyPI releases, while preserving the upstream project's research attribution and documentation.

[Ho Kei Cheng](https://hkchengrex.github.io/), [Seoung Wug Oh](https://sites.google.com/view/seoungwugoh/), [Brian Price](https://www.brianpricephd.com/), [Joon-Young Lee](https://joonyoung-cv.github.io/), [Alexander Schwing](https://www.alexander-schwing.de/)

University of Illinois Urbana-Champaign and Adobe

CVPR 2024, Highlight

[[arXiV]](https://arxiv.org/abs/2310.12982) [[PDF]](https://arxiv.org/pdf/2310.12982.pdf) [[Project Page]](https://hkchengrex.github.io/Cutie/) [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1yo43XTbjxuWA7XgCUO9qxAi7wBI6HzvP?usp=sharing)

## Highlight

Cutie is a video object segmentation framework -- a follow-up work of [XMem](https://github.com/hkchengrex/XMem) with better consistency, robustness, and speed.
This repository contains code for standard video object segmentation and a GUI tool for interactive video segmentation.
The GUI tool additionally contains the "permanent memory" (from [XMem++](https://github.com/max810/XMem2)) option for better controllability.

![overview](https://imgur.com/k84c965.jpg)

## Demo Video

https://github.com/hkchengrex/Cutie/assets/7107196/83a8abd5-369e-41a9-bb91-d9cc1289af70

Source: https://raw.githubusercontent.com/hkchengrex/Cutie/main/docs/sources.txt

## Installation

Tested on Ubuntu only.

**Prerequisite:**

- Python 3.10+
- A PyTorch build for your hardware; image profiles also require matching torchvision

**Clone our repository:**

```bash
git clone https://github.com/hkchengrex/Cutie.git
```

**Install the model core with pip:**

```bash
cd Cutie
pip install -e .
```

Install feature-specific dependencies only when needed:

| Profile | Command | Use case |
| --- | --- | --- |
| Inference | `pip install -e '.[inference]'` | Scripting examples and default model download. |
| Evaluation | `pip install -e '.[evaluation]'` | `cutie/eval_vos.py`, BURST, and multi-scale score outputs. |
| Training | `pip install -e '.[train]'` | Distributed training. |
| GUI | `pip install -e '.[gui]'` | Interactive desktop tool from a source checkout. |
| Video | `pip install -e '.[video]'` | `scripts/process_video.py`. |
| Data | `pip install -e '.[data]'` | Dataset conversion and multi-scale utility scripts. |

The core package contains only the model dependencies. Choose a PyTorch and
torchvision build compatible with your CPU, CUDA, or MPS environment before
installing a profile that requires image transforms.

**Download the pretrained models:**

```python
python cutie/utils/download_models.py
```

## Quick Start

### Scripting Demo

This is probably the best starting point if you want to use Cutie in your project. Hopefully, the script is self-explanatory (additional comments in `scripting_demo.py`). If not, feel free to open an issue. For more advanced usage, like adding or removing objects, see `scripting_demo_add_del_objects.py`.

```python
@torch.inference_mode()
@torch.cuda.amp.autocast()
def main():

    cutie = get_default_model()
    processor = InferenceCore(cutie, cfg=cutie.cfg)
    # the processor matches the shorter edge of the input to this size
    # you might want to experiment with different sizes, -1 keeps the original size
    processor.max_internal_size = 480

    image_path = './examples/images/bike'
    images = sorted(os.listdir(image_path))  # ordering is important
    mask = Image.open('./examples/masks/bike/00000.png')
    palette = mask.getpalette()
    objects = np.unique(np.array(mask))
    objects = objects[objects != 0].tolist()  # background "0" does not count as an object
    mask = torch.from_numpy(np.array(mask)).cuda()

    for ti, image_name in enumerate(images):
        image = Image.open(os.path.join(image_path, image_name))
        image = to_tensor(image).cuda().float()

        if ti == 0:
            output_prob = processor.step(image, mask, objects=objects)
        else:
            output_prob = processor.step(image)

        # convert output probabilities to an object mask
        mask = processor.output_prob_to_mask(output_prob)

        # visualize prediction
        mask = Image.fromarray(mask.cpu().numpy().astype(np.uint8))
        mask.putpalette(palette)
        mask.show()  # or use mask.save(...) to save it somewhere


main()
```

### Interactive Demo

Start the interactive demo with:

```bash
python interactive_demo.py --video ./examples/example.mp4 --num_objects 1
```

[See more instructions here](docs/INTERACTIVE.md).
If you are running this on a remote server, X11 forwarding is possible. Start by using `ssh -X`. Additional configurations might be needed but Google would be more helpful than me.

![demo](https://i.imgur.com/nqlYqTq.jpg)

(For single video evaluation, see the unofficial script `scripts/process_video.py` from https://github.com/hkchengrex/Cutie/pull/16)

## Training and Evaluation

1. [Running Cutie on video object segmentation data.](docs/EVALUATION.md)
2. [Training Cutie.](docs/TRAINING.md)

## Citation

```bibtex
@inproceedings{cheng2023putting,
  title={Putting the Object Back into Video Object Segmentation},
  author={Cheng, Ho Kei and Oh, Seoung Wug and Price, Brian and Lee, Joon-Young and Schwing, Alexander},
  booktitle={arXiv},
  year={2023}
}
```

## References

- The GUI tools uses [RITM](https://github.com/SamsungLabs/ritm_interactive_segmentation) for interactive image segmentation. This repository also contains a redistribution of their code in `gui/ritm`. That part of code follows RITM's license.

- For automatic video segmentation/integration with external detectors, see [DEVA](https://github.com/hkchengrex/Tracking-Anything-with-DEVA).

- The interactive demo is developed upon [IVS](https://github.com/seoungwugoh/ivs-demo), [MiVOS](https://github.com/hkchengrex/MiVOS), and [XMem](https://github.com/hkchengrex/XMem).

- We used [ProPainter](https://github.com/sczhou/ProPainter) in our video inpainting demo.

- Thanks to [RTIM](https://github.com/SamsungLabs/ritm_interactive_segmentation) and [XMem++](https://github.com/max810/XMem2) for making this possible.
