Metadata-Version: 2.5
Name: telekinesis-trackers
Version: 0.1.1
Summary: Deployment visual tracking runtimes
Author-email: Telekinesis <support@telekinesis.ai>
Requires-Python: >=3.11
Requires-Dist: numpy<3,>=1.26
Requires-Dist: onnxruntime<2,>=1.20
Requires-Dist: pillow<13,>=10
Requires-Dist: requests<3,>=2.31
Provides-Extra: cpu
Provides-Extra: dev
Requires-Dist: build<2,>=1.2; extra == 'dev'
Requires-Dist: onnx<2,>=1.17; extra == 'dev'
Requires-Dist: pyright<2,>=1.1.400; extra == 'dev'
Requires-Dist: pytest<10,>=8; extra == 'dev'
Requires-Dist: ruff<1,>=0.11; extra == 'dev'
Provides-Extra: examples
Requires-Dist: loguru<1,>=0.7; extra == 'examples'
Requires-Dist: opencv-python<5,>=4.8; extra == 'examples'
Provides-Extra: gpu
Requires-Dist: cupy-cuda12x[ctk]<15,>=14; extra == 'gpu'
Requires-Dist: onnxruntime-gpu[cuda,cudnn]<1.27,>=1.21; extra == 'gpu'
Provides-Extra: mcbyte
Requires-Dist: supervision<1,>=0.26.1; extra == 'mcbyte'
Requires-Dist: trackers[mask]==2.6.0; extra == 'mcbyte'
Description-Content-Type: text/markdown

<div align="center">
  <p>
    <a href="https://telekinesis.ai">
      <img width="100%" src="https://assets.telekinesis.ai/Telekinesis+Banner.png" alt="Telekinesis">
    </a>
  </p>

![Python](https://img.shields.io/badge/python-3.11%2B-blue)
![Runtime](https://img.shields.io/badge/runtime-ONNX%20Runtime-blue)

</div>

<p align="center">
  <a href="https://github.com/telekinesis-ai">GitHub</a>
  &nbsp;&bull;&nbsp;
  <a href="https://www.linkedin.com/company/telekinesis-ai/">LinkedIn</a>
  &nbsp;&bull;&nbsp;
  <a href="https://x.com/telekinesis_ai">X</a>
  &nbsp;&bull;&nbsp;
  <a href="https://discord.gg/7NnQ3bQHqm">Discord</a>
</p>

# Telekinesis Trackers

Telekinesis Trackers provides visual tracking runtimes for robotics and computer
vision applications within the Telekinesis ecosystem. It runs compatible ONNX
model bundles on CPU, with NVIDIA GPU execution available for CUTIE.

It includes:

- Multi-object mask propagation with CUTIE
- Point tracking with TAPIR
- Click-assisted mask initialization with RITM

## Supported Trackers

| Runtime | Task | Execution |
| --- | --- | --- |
| `CutieTracker` | Multi-object mask propagation | CPU / CUDA |
| `MaskInitializer` | RITM click-assisted masks | CPU only |
| `TapirTracker` | Offline or causal point tracking | CPU only |

Installing the GPU extra does not enable GPU execution for the CPU-only adapters.
The table describes implemented runtime paths, not tracking-quality or performance
benchmarks. CUDA hardware tests are opt-in; see [Tests](DEVELOPMENT.md#tests).

Contributor setup, model exporting, S3 publishing, and tests are documented in
[DEVELOPMENT.md](DEVELOPMENT.md).

## Release Model

Telekinesis Trackers is in active development (pre-1.0). APIs and model bundle
contracts may evolve between releases. See [CHANGELOG.md](CHANGELOG.md) for
release notes and [DEVELOPMENT.md](DEVELOPMENT.md#workflow) for the development
and release workflow.

## Installation

Python 3.11 or newer is required. Create an isolated environment before
installing the package. For example, with Conda:

```sh
conda create -n telekinesis-trackers python=3.11
conda activate telekinesis-trackers
```

Install the package from PyPI:

```sh
pip install telekinesis-trackers
```

The base package includes CPU ONNX Runtime, NumPy, Pillow, and requests, so it
can run inference without an extra. `[cpu]` remains a compatibility alias.

### Switch to NVIDIA GPU

After installing the package, replace CPU ONNX Runtime with the GPU runtime and
install its CUDA libraries and CuPy:

```sh
pip uninstall -y onnxruntime onnxruntime-gpu
pip install "onnxruntime-gpu[cuda,cudnn]>=1.21,<1.27" "cupy-cuda12x[ctk]>=14,<15"
```

Uninstalling both runtime distributions first also repairs environments where
both were installed, since they share the same Python module directory.
`pip install "telekinesis-trackers[gpu]"` alone installs both runtime packages;
if you use that extra, run the replacement commands above afterward.

Because the package declares CPU ONNX Runtime as a dependency, `pip check` will
report `onnxruntime` as missing after this switch. Pip does not recognize the GPU
distribution as its replacement. Upgrading or reinstalling `telekinesis-trackers`
may restore the CPU dependency; repeat the replacement commands afterward.

The distribution is named `telekinesis-trackers`; import it in Python as
`telekinesis.trackers`. For installation from the GitLab package registry, see
the [package installation guide](DEVELOPMENT.md#usage).

PyTorch, Transformers, CUTIE, and TAPIR are not runtime dependencies.

GPU execution needs a CUDA 12-compatible NVIDIA driver. The commands above install
the user-space CUDA libraries, and the ONNX Runtime upper bound keeps it on CUDA 12.
See the [ORT requirements](https://onnxruntime.ai/docs/execution-providers/CUDA-ExecutionProvider.html)
and [CuPy installation guide](https://docs.cupy.dev/en/stable/install.html).

## Verify Installation

Check that the package imports successfully (this does not load models or test
ONNX Runtime/CUDA initialization):

```python
from telekinesis import trackers

print("Available CUTIE tracker:", trackers.CutieTracker)
```

## Usage

### Automatic Model Loading

CUTIE, RITM, and TAPIR download their pretrained bundles on first use:

```python
from telekinesis.trackers import CutieTracker, MaskInitializer

cutie = CutieTracker()
ritm = MaskInitializer()
```

Models are fetched from `https://assets.telekinesis.ai/trackers/`, validated,
and extracted under:

```text
~/.cache/telekinesis/trackers/cutie/480x864-dynamic
~/.cache/telekinesis/trackers/ritm/480x864-20
~/.cache/telekinesis/trackers/tapir-online/causal-2
~/.cache/telekinesis/trackers/tapir-offline/offline-2-2frames
```

The cached bundle is reused without another network request. The asset origin,
cache location, progress display, and network settings are fixed. Pass
`model_dir` only to use a local bundle instead.

### CUTIE Mask Tracking

Frames are uint8 RGB arrays shaped `(H,W,3)`. Seed labels are integer arrays
shaped `(H,W)` where zero is background and values 1–255 are object IDs.

```python
from telekinesis.trackers import CutieTracker

tracker = CutieTracker()
tracker.seed(seed_labels, first_rgb)
labels, alive_fraction = tracker.execute(next_rgb)
```

Use `CutieTracker(device="cuda")` (or `"cuda:N"`) to select a GPU explicitly,
and `device="cpu"` to force CPU execution. The default `"auto"` selects CUDA
when ONNX Runtime advertises it, otherwise CPU. A selected CUDA backend that
cannot initialize raises an error; it does not silently run the tracker on CPU.
`tracker.device` reports the selected device.

On CUDA, graph features and recurrent state stay on the GPU through I/O binding;
CuPy performs memory attention and output resizing there. Input validation and
frame preprocessing remain on CPU; returned labels are NumPy arrays. Existing
float32 bundles work without re-exporting. Unsupported graph operators may still
use ONNX Runtime's CPU provider.

The pretrained graph has 480x864 padded dimensions and a dynamic object axis.
Frames can be reduced with `max_internal_size`; objects are selected from the IDs
present in the seed mask. Use `correct(image_rgb, labels)` to replace the current
mask and memory state, and `reset_session()` to clear the tracker.

### RITM Mask Initialization

`MaskInitializer` creates one object's mask from positive and negative `(x, y)`
clicks:

```python
from telekinesis.trackers import MaskInitializer

initializer = MaskInitializer()
probability = initializer.predict_proba(
    image_rgb,
    positive_points=[[320, 200]],
)
mask = initializer.predict(
    image_rgb,
    positive_points=[[320, 200]],
    negative_points=[[20, 20]],
    previous_mask=probability,
)
```

Images and previous-mask probabilities are resized internally, and results are
returned at the original image size. Calls are stateless; pass the preceding
probability map when refining the same object. The pretrained bundle accepts up
to 20 positive and 20 negative clicks.

### TAPIR Point Tracking

TAPIR runs on CPU. Online (causal) mode processes frames sequentially and downloads
the default bundle on first use:

```python
from telekinesis.trackers import TapirTracker

tracker = TapirTracker(mode="online")  # Also the default for TapirTracker()
tracks = tracker.seed(first_rgb, query_points_xy)
tracks = tracker.step(next_rgb)
```

Offline mode processes a complete clip together:

```python
tracker = TapirTracker(mode="offline")
tracks = tracker.track(video_rgb, query_points_xy)
```

Both default bundles require exactly **two query points** and use 256x256
processing. The default offline bundle requires exactly **two frames**; the
online bundle accepts successive frames through `step()`. Switching mode does
not make these dimensions dynamic.

The bundles are cached under `~/.cache/telekinesis/trackers/tapir-online/causal-2`
and `~/.cache/telekinesis/trackers/tapir-offline/offline-2-2frames`. Cached bundles
are reused. Pass `model_dir` to use a custom bundle; its mode is inferred unless
you explicitly specify one. An explicit mode must match the bundle. `"causal"`
is accepted as an alias for `"online"`; `tracker.mode` retains the manifest value
`"causal"` or `"offline"`.

## Resources

- [Development guide](DEVELOPMENT.md): contributor setup, model exports, tests, and CI/CD
- [Changelog](CHANGELOG.md): release notes
- [Telekinesis Examples](https://github.com/telekinesis-ai/telekinesis-examples): examples across the Telekinesis ecosystem
- [Telekinesis Documentation](https://docs.telekinesis.ai): SDK documentation

## Support

For issues and questions:

- Open an issue in this repository with the tracker name, runtime extra, and steps to reproduce.
- Contact the Telekinesis development team at [support@telekinesis.ai](mailto:support@telekinesis.ai) or on [Discord](https://discord.com/invite/7NnQ3bQHqm).
