Metadata-Version: 2.4
Name: qnnx-backend
Version: 0.2.0
Summary: A PyTorch torch.compile backend for Qualcomm QNN.
Author: Thomas Le Gall
License-Expression: MIT
Project-URL: Repository, https://gitlab.com/legall.thomas.ll/qnnx-backend
Project-URL: ORCID, https://orcid.org/0009-0007-4387-0435
Requires-Python: <3.14,>=3.13
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: torch<2.15,>=2.13
Requires-Dist: onnxruntime-qnn<2.6,>=2.4
Requires-Dist: onnx
Requires-Dist: onnxscript
Provides-Extra: dev
Requires-Dist: numpy; extra == "dev"
Requires-Dist: pytest; extra == "dev"
Requires-Dist: pytest-cov; extra == "dev"
Requires-Dist: hatch; extra == "dev"
Requires-Dist: ruff; extra == "dev"
Requires-Dist: pyright; extra == "dev"
Dynamic: license-file

# qnnx-backend

`qnnx-backend` is an experimental PyTorch `torch.compile` backend for Qualcomm QNN.

It enables PyTorch models to run through ONNX Runtime's QNN Execution Provider on supported Qualcomm NPU and GPU devices.

> [!WARNING]
> `qnnx-backend` is currently experimental and intended for inference only.

## Features

- PyTorch integration through `torch.compile(..., backend="qnn")`
- Qualcomm NPU/HTP acceleration
- Qualcomm GPU acceleration
- Persistent QNN context caching across Python process restarts
- Automatic NPU → GPU → CPU fallback
- Explicit device selection
- Operator compatibility validation
- Static model parameter and buffer freezing
- QNN availability detection

## Requirements

- Windows ARM64
- Qualcomm Snapdragon platform with QNN support
- Python 3.13


> **Note:** Python 3.14 is not currently supported because compatible stable PyTorch wheels for Windows ARM64 are not yet available. Support can be added once PyTorch provides the required ARM64 wheels.
### Tested configuration

The following versions are covered by the compatibility test matrix:

- Python 3.13
- PyTorch 2.13 and 2.14
- ONNX Runtime QNN 2.4 and 2.5
- Snapdragon X Elite

## Installation

`qnnx-backend` currently supports **Python 3.13 ARM64** on Windows ARM64.

> **Important:** make sure you are using a native ARM64 Python installation.  
> A Python x64 installation will not use the required Windows ARM64 wheels.

Python.org provides dedicated **Windows ARM64 installers** for supported Python versions.

You can verify your Python architecture with:

```bash
python -c "import platform; print(platform.machine())"
```

The output should be:

```text
ARM64
```

Then install PyTorch:

```bash
python -m pip install torch==2.14.0+cpu --index-url https://download.pytorch.org/whl/cpu
```

Finally, install `qnnx-backend`:

```bash
python -m pip install qnnx-backend
```

## Usage

### Basic usage

`qnnx_backend` integrates with PyTorch through `torch.compile`.

```python
import torch
import qnnx_backend

model = MyModel().eval()
x = torch.randn(1, 32)

compiled_model = torch.compile(
    model,
    backend="qnn",
)

with torch.inference_mode():
    output = compiled_model(x)
```

By default, `qnnx_backend` automatically selects an available QNN accelerator.

### Checking QNN availability

You can check whether at least one QNN accelerator is available:

```python
qnnx_backend.is_available()
```

You can also check a specific device:

```python
qnnx_backend.is_available("npu")
qnnx_backend.is_available("gpu")
```

### Device selection

The QNN backend supports the following device options:

- `"auto"` — automatically selects an available accelerator
- `"npu"` — uses the Qualcomm NPU/HTP backend
- `"gpu"` — uses the Qualcomm GPU backend
- `"cpu"` — uses the original PyTorch graph without QNN acceleration

The default is `"auto"`.

```python
compiled_model = torch.compile(
    model,
    backend="qnn",
    options={
        "device": "npu",
    },
)
```

### Backend options

| Option | Values | Default | Description |
| --- | --- | --- | --- |
| `device` | `auto`, `npu`, `gpu`, `cpu` | `auto` | Selects the execution backend. |
| `strict` | `True`, `False` | `False` | Controls whether compilation failures can fall back. |
| `fp16` | `True`, `False` | `False` | Enables HTP FP16 precision for NPU execution. |
| `cache` | `True`, `False` | `False` | Enables persistent QNN context caching across process restarts. |
| `cache_dir` | path | platform default | Overrides the persistent context cache directory. |

### Automatic fallback

With the default configuration, `qnnx_backend` falls back when QNN compilation is not possible.

In `"auto"` mode, devices are attempted in the following order:

```text
NPU -> GPU -> CPU
```

For example:

```python
compiled_model = torch.compile(
    model,
    backend="qnn",
)
```

If the graph cannot be compiled for the NPU, the GPU is attempted. If neither QNN accelerator can execute the graph, PyTorch CPU execution is used.

### Strict mode

Use `strict=True` when execution on the selected QNN device is required:

```python
compiled_model = torch.compile(
    model,
    backend="qnn",
    options={
        "device": "npu",
        "strict": True,
    },
)
```

In strict mode, a QNN compilation or session creation failure is propagated instead of falling back to CPU.

This is useful when testing whether a graph is actually supported by a specific QNN backend.

### NPU FP16 execution

For NPU execution, FP16 precision can be enabled with:

```python
compiled_model = torch.compile(
    model,
    backend="qnn",
    options={
        "device": "npu",
        "fp16": True,
    },
)
```

This option controls the QNN HTP FP16 precision setting.

### Persistent context cache

QNN graph compilation can introduce a significant startup cost.

Persistent context caching allows `qnnx_backend` to reuse a previously compiled QNN context across Python process restarts:

```python
compiled_model = torch.compile(
    model,
    backend="qnn",
    options={
        "device": "npu",
        "cache": True,
    },
)
```

On the first execution, the QNN context is compiled and stored on disk. Subsequent executions of the same model and configuration can reuse the cached context instead of compiling it again.

Persistent context caching is supported for both QNN NPU and GPU execution.

A custom cache directory can be specified with:

```python
compiled_model = torch.compile(
    model,
    backend="qnn",
    options={
        "device": "npu",
        "cache": True,
        "cache_dir": "./qnn-cache",
    },
)
```

The cache key includes the exported model and relevant QNN compilation configuration, so incompatible contexts are stored separately.
## Inference

`qnnx_backend` currently targets inference workloads.

Models should be switched to evaluation mode before compilation:

```python
model.eval()
```

Inference should normally be performed using:

```python
with torch.inference_mode():
    output = compiled_model(x)
```

Training and backward execution are not currently supported.

## Model parameters and buffers

Model parameters and buffers are captured when the QNN graph is compiled.

For example:

```python
compiled_model = torch.compile(
    model.eval(),
    backend="qnn",
)

compiled_model(x)
```

After compilation, modifying parameters or buffers of the original model does not update the existing QNN session.

Recompile the model after changing its parameters:

```python
compiled_model = torch.compile(
    model.eval(),
    backend="qnn",
)
```

## Static shapes

`qnnx_backend` currently targets static input shapes.

Compile the model using the input shapes that will be used during inference.

Dynamic-shape support is not currently provided.

## Operator support

`qnnx_backend` validates exported ONNX operators against the currently supported QNN backends.

Operator support may differ between NPU/HTP and GPU.

See the [operator compatibility list](docs/operators.md) for the current support matrix.

## Benchmarks

Benchmarks were performed on Windows ARM64 with a Snapdragon X Elite.

The results below show steady-state inference latency after warm-up.

### Small models

Steady-state inference latency after warm-up. Values are median latency in milliseconds (lower is better).

| Model | Batch | PyTorch CPU | QNN GPU | QNN NPU |
| --- | ---: | ---: | ---: | ---: |
| Small MLP | 16 | 0.042 ms | 0.201 ms | 0.203 ms |
| Small CNN | 16 | 0.299 ms | 0.953 ms | **0.212 ms** |
| Small Transformer | 16 | 0.750 ms | 1.459 ms | **0.306 ms** |

### Medium models

Steady-state inference latency after warm-up. Values are median latency in milliseconds (lower is better).

| Model | Batch | PyTorch CPU | QNN GPU | QNN NPU |
| --- | ---: | ---: | ---: | ---: |
| Medium MLP | 16 | 2.523 ms | 0.505 ms | **0.333 ms** |
| Medium CNN | 16 | 134.117 ms | 11.292 ms | **2.143 ms** |
| Medium Transformer | 16 | 47.343 ms | 9.176 ms | **2.036 ms** |

### Persistent context cache

Persistent context caching also reduces time to first inference across Python process restarts:

| Device | Cold startup | Warm startup | Reduction |
| --- | ---: | ---: | ---: |
| NPU | 3.792 s | 2.352 s | 38.0% |
| GPU | 3.886 s | 2.010 s | 48.3% |

Each warm measurement was performed in a fresh Python process using a context generated during the corresponding cold run.

See [detailed benchmarks](docs/benchmarks.md) for batch-size scaling, first-call latency, median, p95, standard deviation, and benchmark methodology.

## Author

Developed by **Thomas Le Gall** — [ORCID](https://orcid.org/0009-0007-4387-0435) · [GitLab](https://gitlab.com/legall.thomas.ll).

## License

This project is licensed under the MIT License. See the `LICENSE` file for details.

Copyright © 2026 Thomas Le Gall.
