Metadata-Version: 2.4
Name: neural_tilde
Version: 0.0.1
Summary: Exporting PyTorch neural audio synthesis models in ExecuTorch for neural_tilde.
Author-email: Jasper Shuoyang Zheng <jasper.zheng.u@gmail.com>
License-Expression: CC-BY-NC-4.0
Project-URL: Homepage, https://github.com/jasper-zheng/neural_tilde
Project-URL: Repository, https://github.com/jasper-zheng/neural_tilde
Keywords: executorch,pytorch,neural audio,max msp,audio synthesis,rave
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Operating System :: OS Independent
Classifier: Topic :: Multimedia :: Sound/Audio
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: torch>=2.12.0
Provides-Extra: coreml
Requires-Dist: coremltools>=9.0; extra == "coreml"
Provides-Extra: dev
Requires-Dist: build; extra == "dev"
Requires-Dist: twine; extra == "dev"
Dynamic: license-file

# `neural`: neural audio synthesis in Max/MSP

**Max externals for running PyTorch neural synthesis models in realtime/offline.**   
Models are exported from PyTorch with [ExecuTorch](https://pytorch.org/executorch/) and run inside Max with hardware acceleration on CPU / GPU / ANE (Apple Neural Engine).



This package has two families of Max objects:

**Live models**: streaming realtime DSP models (e.g. neural vocoder, neural codecs...):

| object | use it for |
| --- | --- |
| `neural.live~` | a single model instance on one audio stream |
| `mc.neural.live~` | the same model broadcast across the channels of an `mc` signal |
| `mcs.neural.live~` | a fixed batch of streams processed in one pass (takes a batch-size argument) |

**Generative models** — one-shot offline generators (e.g. latent-diffusion text-to-audio, audio-to-audio):

| object | use it for |
| --- | --- |
| `neural.gen~` | run a generative model and write the output into a `buffer~` |

**Utility**:

| object | use it for |
| --- | --- |
| `neural.tokenizer` | turn a text prompt into token ids and attention mask |
| `neural.gaussianize` | map uniform `jit.noise` to Gaussian noise |

**Supported models:**  
 - [Live] RAVE (Realtime Audio Variational autoEncoder) 
 - [Gen] Stable Audio 3 (text-to-audio, audio-to-audio)
 - [Gen] CodiCodec-Flow

**Note:** Currently only available for MaxMSP on Apple Silicon, macOS. For Windows/CUDA, it's on the todo list. 

**Acknowledgement:** `neural.live~` and its `mc`, `mcs` variants reused an extensive amount of code from [`nn~`](https://github.com/acids-ircam/nn_tilde), migrated from TorchScript/libtorch to more modern ExecuTorch. `nn~` is the work by Antoine Caillon & Axel Chemla--Romeu-Santos (acids-ircam), licensed **CC BY-NC 4.0**.


## How It Works

 - **Export**: Export a PyTorch neural audio model using [ExecuTorch](https://docs.pytorch.org/executorch/main/getting-started.html), with a target hardware backend (see below). This results in a `.pte` (the runnable
ExecuTorch program + weights) file.  
 - **Config**: Create a `.json` sidecar that details inlet/outlet, ratio, attributes, condition shapes, etc. 
 - **Load**: Hand `neural.live~` / `neural.gen~` the `.pte` and `.json` files in Max.

### Supported back-ends (chosen when exporting the model):

- **[Core ML](https://developer.apple.com/documentation/coreml)** — Leverage Apple Neural Engine (ANE) for hardware acceleration. Needs **macOS 15+** at runtime.
- **[MLX](https://github.com/ml-explore/mlx)** — Apple-Silicon GPU; Good for one-shot / buffer processing.
- **[XNNPACK](https://github.com/google/XNNPACK)** — CPU inference with optimized kernels.
- **portable** — plain, unoptimized C++ kernels, maximum compatibility
- ~~CUDA~~ - Windows CUDA in progress


## Install the externals

Download the pre-released externals. To install:

1. Unzip and copy the `neural_tilde` folder into your Max **Packages** folder, e.g.
   `~/Documents/Max 9/Packages/`.
2. Restart Max. The streaming objects (`neural.live~`, `mc.neural.live~`, `mcs.neural.live~`) and the
   generative objects (`neural.gen~`, `neural.tokenizer`) are now available.

If macOS quarantines the externals (objects fail to instantiate), clear the quarantine flag or ad-hoc re-sign them. In Terminal, type:

```bash
cd ~/Documents/Max\ 9/Packages/neural_tilde
xattr -dr com.apple.quarantine externals/*.mxo
# or:  codesign --force --deep -s - externals/*.mxo
```


## Use a pretrained model

**Make sure to add your model's path in Max:** The `.pte` and its `.json` should have **the same** filename. Put them together in a folder. In Max, open "Options/File Preferences", click "Add Path" on the bottom left corner, add your model's folder. In Max, create an external using:

```
[neural.live~ mymodel forward]
```

### APIs:

- **Arguments:** `neural.live~ <model.pte> <method> [buffer size]`. The method defaults to
  `forward`; an autoencoder might expose `encode` / `decode` instead (one object per
  method). The `.pte` extension is optional.
- **Inlets / outlets** are created from the sidecar labels — one signal inlet per input
  channel, one signal outlet per output channel.
- **Turn computation on:** the `enable_model` attribute defaults to **off**, so the
  object outputs silence until you send it:

  ```
  enable_model 1
  ```

- **Buffer size** is adopted from the model's exported size (fixed-shape programs), so
  it overrides the optional box argument.

Useful messages / attributes:

| message / attribute | effect |
| --- | --- |
| `enable_model 0/1` | disable / enable inference (default `0`) |
| `reload <path>` | load a different model at runtime |
| `get_methods` | print the methods available in the loaded `.pte` |
| `@<name> <v>` / `<name> <v>` | the model's settable scalar controls, one **Max attribute per sidecar `attributes` entry** (e.g. `@temperature 0.8`), settable as box-args, in the inspector, or by message; clamped to the model's range and fed to the model every block. The sidecar `description` is the inspector label. |
| `get <name>` / `get_attributes` | print one control's value / list them all with values |

Use `mc.neural.live~` to run one model across every channel of an `mc.` patch cord, or
`mcs.neural.live~` to process a fixed batch of streams in a single forward pass.


## Generative models (Stable Audio 3)

Some models aren't streaming DSP at all — they're **one-shot generators**: hand them a text
prompt and they produce a fixed clip in a single (multi-second) pass. **Stable Audio 3 (SA3)**
is a latent-diffusion text→audio model of this kind. Two objects handle it:

- **`neural.gen~`** — a generic latent-diffusion *runner*. It loads the model `.pte`, takes its
  conditioning on **one inlet per condition input** (token ids, attention mask, and any
  negative-prompt pair), samples the model's noise itself, runs the generation **on a worker
  thread** (the audio thread never blocks), writes the result into a named **`buffer~`**, and bangs
  a **`done`** outlet.
- **`neural.tokenizer`** — turns a `prompt` into `input_ids` + `attention_mask`, emitting each as a
  single-key dictionary on its **two outlets**. Keeping it separate means `neural.gen~` stays
  generic (a model with a different tokenizer, or none, just wires a different source into its
  inlets), and a second tokenizer can feed a negative prompt.

### The model bundle

Put these four files in one folder and add it to Max's **Options → File Preferences** (the `.pte`
is large and user-supplied, ~2.9 GB for SA3, so it is not shipped with the package):

| file | what it is |
| --- | --- |
| `full_pipeline_T32.pte` | the ExecuTorch program + weights |
| `full_pipeline_T32.json` | model sidecar — typed inputs/output (no tokenizer settings) |
| `full_pipeline_T32.tokenizer.json` | the HuggingFace fast tokenizer |
| `full_pipeline_T32.tokenizer.config.json` | tokenizer settings `neural.tokenizer` reads |

The SA3 export session emits all four (see its `export/emit_sidecars.py`). The protocol is in
[`EXECUTORCH_PROTOCOL.md`](EXECUTORCH_PROTOCOL.md) §7.

### Patch it up

```
[prompt lofi house loop]              <- message to the tokenizer
       |
[neural.tokenizer full_pipeline_T32.tokenizer.config.json]
       |        |     <- outlet 0 = {input_ids}, outlet 1 = {attention_mask}
       |        |
[neural.gen~ full_pipeline_T32 @seconds_total 3 @seed 0]
       |               (inlet 0 = input_ids, inlet 1 = attention_mask)
   (done bang)        ... writes into [buffer~ gen 132300 2]
```

1. Create `[buffer~ gen 132300 2]` (stereo; SA3 outputs ~2.97 s at 44.1 kHz). Send `neural.gen~`
   the message `set gen` to target it.
2. Send the prompt to **`neural.tokenizer`** (`prompt lofi house loop`); wire its two outlets into
   the matching `neural.gen~` condition inlets (each is a single-key dict matched **by name**, so
   the order you wire them doesn't matter). `neural.gen~` **caches** them.
3. Send **`generate`** (or `bang`) to `neural.gen~`. After a few seconds the `done` outlet bangs,
   the buffer is filled, and you can play it with `groove~ gen` / `play~`.

You can pass `neural.tokenizer` the `*.tokenizer.config.json` (it self-configures), or the bare
`*.tokenizer.json` (it auto-looks for the matching `*.tokenizer.config.json` beside it, and warns
if none is found and it falls back to defaults).

### APIs

**`neural.gen~`** — `neural.gen~ <model.pte> [method] [buffer]`:

| message / attribute | effect |
| --- | --- |
| *(condition inlets)* | one inlet per `condition`-role input, in sidecar order (e.g. `input_ids`, `attention_mask`, and for a CFG model `neg_input_ids`, `neg_attention_mask`). Each accepts a **`dictionary`** (matched by key name — typically from `neural.tokenizer`) or a bare **`list`** (assigned to that inlet's condition by position). Cached; does not auto-run. The leftmost inlet also takes the control messages below. A condition left unsupplied is zero-filled at generate (a zero mask ⇒ unconditional) with a warning. |
| `generate` / `bang` | start a generation (ignored if one is already running) |
| `set <buffer>` | target `buffer~` for the output |
| `@<name> <v>` / `<name> <v>` | the model's scalar controls, one **attribute per sidecar `attribute`-role input** (e.g. `@seconds_total 3`), settable as box-args, in the inspector, or by message; clamped to the model's range. Each attribute's sidecar `description` is shown as its inspector label. |
| `init <buffer>` / `init` | **audio-to-audio**: point an `audio_init`-role model at a source `buffer~` to vary from (the host resamples / channel-maps / crops it to the model's geometry); `init` with no arg clears it (generate from silence) |
| `get_attributes` / `get <name>` | list the model's attributes + values / print one |
| `seed <n>` | RNG seed for the generation noise (reproducible) |
| `reload <path>` | load a different generative model |
| `get_methods` | print the methods available in the loaded `.pte` |

> **Audio-to-audio.** A sibling SA3 model bundle (`audio2audio_T32.*`) adds an `init_audio` input:
> send `init <buffer~>` to vary an existing clip, steered by the prompt. Its strength is the
> `@init_noise_level` attribute (`0`–`1`; `1` ignores the init and reduces to text-to-audio). The
> patch is otherwise identical to text-to-audio above.

**`neural.tokenizer`** — `neural.tokenizer <tokenizer.json | *.config.json>`:

| message / attribute | effect |
| --- | --- |
| `prompt <text…>` | tokenize the text and output two single-key dictionaries — outlet 0 = `{ids_key}`, outlet 1 = `{mask_key}` |
| `reload <path>` | load a different tokenizer / config |
| `@max_length 256` | pad/truncate length |
| `@pad_token <pad>` `@padding_side right` | padding token and side |
| `@ids_key input_ids` `@mask_key attention_mask` | keys for the two single-key output dictionaries (= the model's condition input names) |

A config file sets these attributes for you; a live `@attr`/message still overrides.

> **Negative prompt (classifier-free guidance).** A CFG model bundle (`*_cfg_T32.*`) adds
> `neg_input_ids` + `neg_attention_mask` condition inputs and a `@cfg_scale` attribute (1 = no
> guidance; ~7 = stronger prompt adherence). Wire a **second** `neural.tokenizer` configured with
> `@ids_key neg_input_ids @mask_key neg_attention_mask` into those inlets; an empty negative prompt
> = unconditional. Leave it unwired and `neural.gen~` zero-fills the negatives (also unconditional).

> **Notes.** The ~2.9 GB weights load lazily on the **first** `generate`, so the first run has
> extra warm-up latency; later runs are fast. Generation runs off the audio thread, so DSP keeps
> playing throughout. A `buffer~` carries Max's project sample-rate label while the data is 44.1
> kHz — if your project runs at a different rate, the data is correct but plays back at the wrong
> pitch (resample, or run the project at 44.1 kHz).


## Export your own neural synthesis model

Models are produced with the helper in [`python_tools/`](python_tools/). It provides the **`neural_tilde`** Python module. Install it and the export back-ends by:

```bash
pip install -e .  
# Install the `neural_tilde` module, cached_conv, executorch, coremltools
```

### Streaming models for `neural.live~`

See example codes below:
 - Subclass `neural_tilde.LiveModule`, 
 - Build your network (for a click-free streaming **convolutional net**, one option is [`cached_conv`](https://github.com/acids-ircam/cached_conv); any stateful mechanism works),
 - Use `register_method(...)` to describe each method's channels / ratios /
labels, 
 - Optionally give a method extra inputs — register them first with
   `register_attribute` / `register_noise` / `register_condition`, then list them in order via
   `register_method(..., inputs=[...])`; the `forward` takes them after the audio tensor
   (`forward(self, x, noise, temperature)`). These reuse the same role mechanism as `neural.gen~`,
 - Use `export_to_pte(..., delegate="coreml")` to export, which will result in the model weights `.pte` and the sidecar `.json`

```python
import cached_conv as cc
from neural_tilde import LiveModule

cc.use_cached_conv(True)              # MUST be set before building the model

class MyModel(LiveModule):
    def __init__(self):
        super().__init__()
        self.net = cc.CachedSequential(
            cc.Conv1d(1, 16, 3, padding=cc.get_padding(3)),
            cc.Conv1d(16, 1, 3, padding=cc.get_padding(3)),
        )
    def forward(self, x):             # [batch, 1, time] -> [batch, 1, time]
        return self.net(x)

model = MyModel()
model.register_method("forward", in_channels=1, in_ratio=1,
                      out_channels=1, out_ratio=1)
model.export_to_pte("mymodel", delegate="coreml", buffer_size=4096)
# -> mymodel.pte + mymodel.json  (cached_conv state persists across blocks on Core ML)
```

<!-- **Why this streams:** `cached_conv` replaces zero-padding with *cached* padding (it
keeps each conv's previous-block tail in a buffer). On the Core ML delegate those cache
buffers are taken over as native Core ML **state** that persists across blocks, so
`neural.live~` runs the model click-free in real time. (MLX does not persist this state, that's why we perfer Core ML instead of MLX) -->

**Example:** A complete exemple that builds and exports an untrained conv-net is in
[`examples/export_example.py`](examples/export_example.py):

```bash
python examples/export_example.py        
# Results in tiny_stream.pte + tiny_stream.json
```

> A model may carry internal mutable state that persists across blocks (what makes a `cached_conv`
> net stream click-free); the host loads one instance per object and reuses it, so nothing special
> is needed in the sidecar. The full producer/consumer protocol is documented in
> [`EXECUTORCH_PROTOCOL.md`](EXECUTORCH_PROTOCOL.md).

If export complains about `flatc`, prepend `<executorch>/cmake-out/third-party/flatc_ep/bin` to your `PATH`.

### Generative models for `neural.gen~`

Offline generators (e.g. text→audio) use [`neural_tilde.GenModule`](python_tools/gen_module.py)
instead of `LiveModule`. You wrap an existing `nn.Module`, declare each input **by name** with its
**role** — `condition` (tokens/masks), scalar `attribute` controls, seeded `noise`, an optional
init `buffer` — then `register_method(...)` one or more generation paths, each listing the inputs
its method consumes (in `forward` order) plus the audio output it produces. Just like `LiveModule`,
a model can expose **several methods with different inputs** (e.g. `prompt2audio` and `audio2audio`);
the host selects one by name (`[neural.gen~ mymodel.pte audio2audio]`). The `seed` and the tokenizer
bundle are per-model. It writes a `kind:"gen"` sidecar `.json` and, if you register a tokenizer, the
`*.tokenizer.json` + `*.tokenizer.config.json` bundle that `neural.tokenizer` loads.

```python
from neural_tilde import GenModule

# methods resolve to my_pipeline.<name>(*inputs), inputs in the registered order:
#   my_pipeline.prompt2audio(input_ids, attention_mask, seconds_total, x, noises) -> [1, 2, length]
#   my_pipeline.audio2audio(init_audio, seconds_total, x, noises)                 -> [1, 2, length]
gm = GenModule(my_pipeline)
gm.register_condition("input_ids", [1, 256], "int64")
gm.register_condition("attention_mask", [1, 256], "int32")
gm.register_attribute("seconds_total", 3.0, 0.0, 384.0,
                      "Total length in seconds to condition the generation on.")
gm.register_noise("x", [1, 256, 32])
gm.register_noise("noises", [7, 1, 256, 32])
gm.register_buffer_input("init_audio", channels=2, length=131072, sample_rate=44100)
gm.register_method("prompt2audio",
                   inputs=["input_ids", "attention_mask", "seconds_total", "x", "noises"],
                   out_channels=2, out_length=131072, out_sample_rate=44100)
gm.register_method("audio2audio",
                   inputs=["init_audio", "seconds_total", "x", "noises"],
                   out_channels=2, out_length=131072, out_sample_rate=44100)
gm.register_tokenizer("tokenizer.json", max_length=256)   # shared by both exporters
gm.set_seed(0)
gm.export_to_pte("mymodel", delegate="mlx", decompose_conv=True)
# -> mymodel.pte + mymodel.json (+ mymodel.tokenizer.json + .tokenizer.config.json)
```

> `register_tokenizer(...)` lives on the shared exporter base (backed by the `neural_tilde.Tokenizer`
> class), so a `LiveModule` that takes a text prompt — feeding the tokens in as `condition` inputs —
> can ship the same standalone tokenizer bundle. The bundle is independent of the model kind.

**Example:** a tiny stand-in generator (tokens + noise + duration → stereo audio) is in
[`examples/export_gen_example.py`](examples/export_gen_example.py):

```bash
python examples/export_gen_example.py        # -> tiny_gen.pte + tiny_gen.json
```

This is the same protocol the Stable Audio 3 export pipeline produces; the full input-role / sidecar
schema is §7 of [`EXECUTORCH_PROTOCOL.md`](EXECUTORCH_PROTOCOL.md).

## Build the externals from source

Please refer to [Build.md](Build.md)

## Credits

C++ externals and the Python tools derive from `nn~` by **Antoine Caillon** and
**Axel Chemla--Romeu-Santos** (ACIDS-IRCAM). ExecuTorch migration and the `neural`
fork: this repository. License: **CC BY-NC 4.0**.
