Metadata-Version: 2.4
Name: qnnx-backend
Version: 0.1.0
Summary: A PyTorch torch.compile backend for Qualcomm QNN.
Author: Thomas Le Gall
License-Expression: MIT
Project-URL: Repository, https://gitlab.com/legall.thomas.ll/qnnx-backend
Project-URL: ORCID, https://orcid.org/0009-0007-4387-0435
Requires-Python: <3.14,>=3.13
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: torch<2.14,>=2.13
Requires-Dist: onnxruntime-qnn>=2.4
Requires-Dist: onnx
Requires-Dist: onnxscript
Provides-Extra: dev
Requires-Dist: numpy; extra == "dev"
Requires-Dist: pytest; extra == "dev"
Requires-Dist: pytest-cov; extra == "dev"
Dynamic: license-file

# qnnx-backend

`qnnx-backend` is an experimental PyTorch `torch.compile` backend for Qualcomm QNN.

It enables PyTorch models to run through ONNX Runtime's QNN Execution Provider on supported Qualcomm NPU and GPU devices.

> [!WARNING]
> `qnnx-backend` is currently experimental and intended for inference only.

## Features

- PyTorch integration through `torch.compile(..., backend="qnn")`
- Qualcomm NPU/HTP acceleration
- Qualcomm GPU acceleration
- Automatic NPU → GPU → CPU fallback
- Explicit device selection
- Operator compatibility validation
- Static model parameter and buffer freezing
- QNN availability detection

## Requirements

- Windows ARM64
- Qualcomm Snapdragon platform with QNN support
- Python 3.13

### Tested configuration

- PyTorch 2.13
- ONNX Runtime QNN 2.4
- Snapdragon X Elite

## Installation

`qnnx-backend` currently requires **Python 3.13 ARM64** on Windows ARM64.

> **Important:** make sure you are using a native ARM64 Python installation.  
> A Python x64 installation will not use the required Windows ARM64 wheels.

Python.org provides a dedicated **Windows ARM64 installer** for Python 3.13.

You can verify your Python architecture with:

```bash
python -c "import platform; print(platform.machine())"
```

The output should be:

```text
ARM64
```

Then install PyTorch:

```bash
python -m pip install torch==2.13.0+cpu --index-url https://download.pytorch.org/whl/cpu
```

Finally, install `qnnx_backend`:

```bash
python -m pip install qnnx-backend
```

## Usage

### Basic usage

`qnnx_backend` integrates with PyTorch through `torch.compile`.

```python
import torch
import qnnx_backend

model = MyModel().eval()
x = torch.randn(1, 32)

compiled_model = torch.compile(
    model,
    backend="qnn",
)

with torch.inference_mode():
    output = compiled_model(x)
```

By default, `qnnx_backend` automatically selects an available QNN accelerator.

### Checking QNN availability

You can check whether at least one QNN accelerator is available:

```python
qnnx_backend.is_available()
```

You can also check a specific device:

```python
qnnx_backend.is_available("npu")
qnnx_backend.is_available("gpu")
```

### Device selection

The QNN backend supports the following device options:

- `"auto"` — automatically selects an available accelerator
- `"npu"` — uses the Qualcomm NPU/HTP backend
- `"gpu"` — uses the Qualcomm GPU backend
- `"cpu"` — uses the original PyTorch graph without QNN acceleration

The default is `"auto"`.

```python
compiled_model = torch.compile(
    model,
    backend="qnn",
    options={
        "device": "npu",
    },
)
```

### Backend options

| Option | Values | Default | Description |
| --- | --- | --- | --- |
| `device` | `auto`, `npu`, `gpu`, `cpu` | `auto` | Selects the execution backend. |
| `strict` | `True`, `False` | `False` | Controls whether compilation failures can fall back. |
| `fp16` | `True`, `False` | `False` | Enables HTP FP16 precision for NPU execution. |

### Automatic fallback

With the default configuration, `qnnx_backend` falls back when QNN compilation is not possible.

In `"auto"` mode, devices are attempted in the following order:

```text
NPU -> GPU -> CPU
```

For example:

```python
compiled_model = torch.compile(
    model,
    backend="qnn",
)
```

If the graph cannot be compiled for the NPU, the GPU is attempted. If neither QNN accelerator can execute the graph, PyTorch CPU execution is used.

### Strict mode

Use `strict=True` when execution on the selected QNN device is required:

```python
compiled_model = torch.compile(
    model,
    backend="qnn",
    options={
        "device": "npu",
        "strict": True,
    },
)
```

In strict mode, a QNN compilation or session creation failure is propagated instead of falling back to CPU.

This is useful when testing whether a graph is actually supported by a specific QNN backend.

### NPU FP16 execution

For NPU execution, FP16 precision can be enabled with:

```python
compiled_model = torch.compile(
    model,
    backend="qnn",
    options={
        "device": "npu",
        "fp16": True,
    },
)
```

This option controls the QNN HTP FP16 precision setting.

## Inference

`qnnx_backend` currently targets inference workloads.

Models should be switched to evaluation mode before compilation:

```python
model.eval()
```

Inference should normally be performed using:

```python
with torch.inference_mode():
    output = compiled_model(x)
```

Training and backward execution are not currently supported.

## Model parameters and buffers

Model parameters and buffers are captured when the QNN graph is compiled.

For example:

```python
compiled_model = torch.compile(
    model.eval(),
    backend="qnn",
)

compiled_model(x)
```

After compilation, modifying parameters or buffers of the original model does not update the existing QNN session.

Recompile the model after changing its parameters:

```python
compiled_model = torch.compile(
    model.eval(),
    backend="qnn",
)
```

## Static shapes

Version 0.1.0 currently targets static input shapes.

Compile the model using the input shapes that will be used during inference.

Dynamic-shape support is not currently provided.

## Operator support

`qnnx_backend` validates exported ONNX operators against the currently supported QNN backends.

Operator support may differ between NPU/HTP and GPU.

See the [operator compatibility list](docs/operators.md) for the current support matrix.

## Benchmarks

Benchmarks were performed on Windows ARM64 with a Snapdragon X Elite.

The results below show steady-state inference latency after warm-up.

### Small models

Steady-state inference latency after warm-up. Values are median latency in milliseconds (lower is better).

| Model | Batch | PyTorch CPU | QNN GPU | QNN NPU |
| --- | ---: | ---: | ---: | ---: |
| Small MLP | 16 | 0.042 ms | 0.201 ms | 0.203 ms |
| Small CNN | 16 | 0.299 ms | 0.953 ms | **0.212 ms** |
| Small Transformer | 16 | 0.750 ms | 1.459 ms | **0.306 ms** |

### Medium models

Steady-state inference latency after warm-up. Values are median latency in milliseconds (lower is better).

| Model | Batch | PyTorch CPU | QNN GPU | QNN NPU |
| --- | ---: | ---: | ---: | ---: |
| Medium MLP | 16 | 2.523 ms | 0.505 ms | **0.333 ms** |
| Medium CNN | 16 | 134.117 ms | 11.292 ms | **2.143 ms** |
| Medium Transformer | 16 | 47.343 ms | 9.176 ms | **2.036 ms** |

See [detailed benchmarks](docs/benchmarks.md) for batch-size scaling, first-call latency, median, p95, standard deviation, and benchmark methodology.

## Author

Developed by **Thomas Le Gall** — [ORCID](https://orcid.org/0009-0007-4387-0435) · [GitLab](https://gitlab.com/legall.thomas.ll).

## License

This project is licensed under the MIT License. See the `LICENSE` file for details.

Copyright © 2026 Thomas Le Gall.
