Metadata-Version: 2.5
Name: nat-tokenizer
Version: 0.1.0
Summary: NAT action tokenizer: robot action chunks to time-ordered discrete tokens and back
Author: Ahad Jawaid
License-Expression: MIT
License-File: LICENSE
Requires-Python: >=3.9
Requires-Dist: huggingface-hub>=0.20
Requires-Dist: numpy>=1.21
Requires-Dist: safetensors>=0.4
Requires-Dist: torch>=2
Provides-Extra: test
Requires-Dist: pytest>=7; extra == 'test'
Description-Content-Type: text/markdown

# NAT action tokenizer

Turns robot action chunks into discrete tokens for a vision-language-action model, and tokens back into actions.
Every 32-step chunk of up to 7 action columns becomes **14 tokens**, in time order, so a model can stream actions
before it has generated the whole chunk.

## Quick start

```bash
pip install nat-tokenizer
```

or from source: `pip install "git+https://github.com/ahadjawaid/nat.git"`.

The weights and config download from the Hugging Face repository
[ahad-j/nat-action-tokenizer](https://huggingface.co/ahad-j/nat-action-tokenizer) on first use.

```python
import numpy as np
from nat_tokenizer import NATActionTokenizer

tokenizer = NATActionTokenizer.from_pretrained()   # downloads ahad-j/nat-action-tokenizer

# 1. Fit the normalization once, on your dataset's actions (any shape ending in action_dim).
tokenizer.fit(dataset_actions)

# 2. Tokenize 32-step chunks: (batch, 32, action_dim) or one chunk (32, action_dim).
tokens = tokenizer(action_chunks)            # list (one per chunk) of lists of ints in [0, 25600)

# 3. Decode back to actions in your original units.
actions = tokenizer.decode(tokens, action_dim=action_chunks.shape[-1])   # (batch, 32, action_dim)

# 4. Save the fitted tokenizer (keeps your min and max) and load it later the same way.
tokenizer.save_pretrained("my-nat-tokenizer")
tokenizer = NATActionTokenizer.from_pretrained("my-nat-tokenizer")
```

## Normalization (on by default)

The tokenizer was trained on actions in [-1, 1]. With `normalize=True` (the default) every column is mapped from
its min and max to [-1, 1] before encoding (values outside are clipped) and decoded actions are mapped back, so you
work in your own units throughout. Set the min and max with `fit(dataset_actions)`, or pass them directly:

```python
tokenizer = NATActionTokenizer.from_pretrained(action_min=[...], action_max=[...])
```

If your actions are already in [-1, 1], pass `normalize=False` to both calls:
`tokenizer(chunks, normalize=False)` and `tokenizer.decode(tokens, action_dim=d, normalize=False)`.
A column whose min equals its max is encoded as 0 and decoded back to its constant value.

## Streaming (causal decoding)

Within a group the 14 tokens are time ordered: the first 4 decode the chunk's first 8 steps, the first 7 its first 16,
the first 11 its first 24, and all 14 the whole chunk. A prefix decodes exactly the same first steps as the full chunk,
so a model can execute actions while it is still generating:

```python
for steps in (8, 16, 24, 32):
    prefix = [chunk[:tokenizer.prefix_token_count(steps)] for chunk in tokens]
    first_steps = tokenizer.decode(prefix, action_dim=7)      # (batch, steps, 7)
```

Streaming works for actions of up to 7 columns (one group).

## Action dimensions

Columns are split into consecutive groups of up to 7 (8 columns = 7 + 1, 14 = 7 + 7, 21 = 7 + 7 + 7). Each group gets
its own 14 tokens and a chunk's tokens are the groups' tokens one after another: `tokenizer.tokens_per_chunk(action_dim)`
gives the count (14 per group). Chunks must be exactly 32 steps long.

## GPU and CUDA graphs

```python
tokenizer = tokenizer.to("cuda")                      # eager on the GPU
tokenizer = tokenizer.to("cuda", cuda_graphs=True)    # captures the encoder and decoder once per batch shape
```

Inputs and outputs stay NumPy. GPU results match the CPU up to floating point (rarely a token on a quantization
boundary differs).

## API

| call | does |
|---|---|
| `NATActionTokenizer.from_pretrained(repo_or_folder, action_min=None, action_max=None)` | load (optionally with your min and max) |
| `tokenizer.fit(actions)` | per-column min and max of your actions (returns the tokenizer) |
| `tokenizer(chunks, normalize=True)` | `(batch, 32, d)` or `(32, d)` actions to a list of token lists |
| `tokenizer.decode(tokens, action_dim, normalize=True)` | token lists (whole chunks, or one group's prefix) to `(batch, steps, d)` actions |
| `tokenizer.prefix_token_count(steps)` | tokens that decode the first `steps` (8, 16, 24 or 32) |
| `tokenizer.tokens_per_chunk(action_dim)` | 14 per group of 7 columns |
| `tokenizer.to(device, cuda_graphs=False)` | run on a device, optionally with CUDA graphs |
| `tokenizer.save_pretrained(directory)` | save, with the fitted min and max |
| `tokenizer.vocab_size` | 25,600 (4 scales x 6,400 codes) |

## Model

4 latents per chunk, 4 FSQ scales of 6,400 codes (one coarse token per two latents, three finer tokens per latent),
trained on a large corpus of robot and human action data (every window once, batch 4,096). The repository holds only
the weights (`model.safetensors`) and a JSON config; the network is plain PyTorch in the `nat-tokenizer` package.

## Reconstruction on robot benchmarks, and speed

Random 32-step chunks of each benchmark's demonstrations, each normalized by its own min and max (MetaWorld's actions clipped to the [-1, 1] its simulator executes). Errors are mean absolute differences between the input and its reconstruction. Speed is in milliseconds per call on an RTX 4090 (shared with another job while measured) and 4 CPU threads, mean and SD over 20 calls.

| benchmark | chunks x shape | MAE in [-1, 1] units | MAE | max error |
|---|---|---|---|---|
| LIBERO-10 | 1000 x (32, 7) | 0.0304 | 0.0210 | 1.9762 |
| MetaWorld mt4 (clipped) | 1000 x (32, 4) | 0.0651 | 0.0651 | 1.7578 |
| RoboMimic mt4 | 1000 x (32, 7) | 0.0466 | 0.0404 | 1.5712 |
| RoboCasa mt4 | 1000 x (32, 12) | 0.0130 | 0.0107 | 0.9970 |

| device | batch of 32 x 7 chunks | encode ms | decode ms |
|---|---|---|---|
| cpu, eager | 1 | 11.57 ± 0.39 | 29.86 ± 4.06 |
| cpu, eager | 64 | 322.06 ± 1.76 | 263.23 ± 6.67 |
| cuda, eager | 1 | 6.54 ± 1.67 | 11.07 ± 1.18 |
| cuda, eager | 64 | 7.86 ± 2.41 | 12.67 ± 1.80 |
| cuda, CUDA graphs | 1 | 0.71 ± 0.57 | 2.44 ± 0.39 |
| cuda, CUDA graphs | 64 | 1.05 ± 0.35 | 5.18 ± 2.09 |

## Development

```bash
pip install -e ".[test]"
pytest
```

## License

MIT, see [LICENSE](LICENSE).
