Metadata-Version: 2.4
Name: onnx-genai
Version: 0.1.0.dev3
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Rust
Classifier: License :: OSI Approved :: MIT License
Requires-Dist: onnxruntime>=1.27,<2
Requires-Dist: onnxruntime-ep-mlx>=0.27 ; platform_system == 'Darwin'
Requires-Dist: onnxruntime-gpu[cuda,cudnn]>=1.27,<2 ; (platform_system == 'Linux' and extra == 'cuda') or (platform_system == 'Windows' and extra == 'cuda')
Provides-Extra: cuda
Summary: ONNX Runtime-backed text generation engine (same API as nxrt.genai)
Author: Justin Chu
License: MIT
Requires-Python: >=3.11
Description-Content-Type: text/markdown; charset=UTF-8; variant=GFM
Project-URL: repository, https://github.com/justinchuby/onnx-genai

# onnx-genai

ONNX Runtime-backed text generation for Python.

`onnx-genai` exposes a small, high-level generation API implemented on
[ONNX Runtime](https://onnxruntime.ai/). It is the ONNX Runtime-compatible
counterpart of `nxrt.genai`: both packages present the **same** `Engine` /
`GenerateResult` API, but `onnx_genai` runs on ONNX Runtime while `nxrt.genai`
runs on the native nxrt runtime.

```python
import onnx_genai

engine = onnx_genai.Engine.from_dir("path/to/model_dir")

result = engine.generate("Hello, world!", max_tokens=64, temperature=0.8)
print(result.text)

# Streaming
def on_token(text, token_id, finish_reason):
    print(text, end="", flush=True)

engine.generate_stream("Tell me a story", on_token, max_tokens=128)
```

The model directory must contain the ONNX graph(s), `tokenizer.json`, and model
metadata (`inference_metadata.yaml` or `genai_config.json`).

## Wheels

Distributed as stable-ABI (`abi3`) wheels tagged `cp310-abi3`, so a single wheel
per platform loads on CPython 3.10 and newer. ONNX Runtime is **not** bundled:
`onnx_genai` loads `libonnxruntime` from the installed `onnxruntime` (CPU) or
`onnxruntime-gpu` (CUDA) wheel at import time, so it uses whichever execution
providers you installed.

## Related packages

- **`nxrt`** — the low-level nxrt runtime (inference sessions, eager ops) plus
  `nxrt.genai`, the same generation API backed by the native runtime.
- **`onnx-genai-server`** — the `onnx-genai` command-line tool, including the
  OpenAI-compatible server (`onnx-genai serve`).

See <https://github.com/justinchuby/onnx-genai>.

