Metadata-Version: 2.4
Name: haru-speech-vllm
Version: 0.1.1
Summary: Serve Haru with vLLM-Omni
Author: higurashi
License-Expression: Apache-2.0
Project-URL: Homepage, https://higurashi.ai
Project-URL: Repository, https://github.com/higurashi-ai/Haru
Project-URL: Issues, https://github.com/higurashi-ai/Haru/issues
Project-URL: Model, https://huggingface.co/higurashi-ai/Haru-4B
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: haru-speech>=0.1.1
Requires-Dist: vllm>=0.32
Requires-Dist: vllm-omni>=0.32
Dynamic: license-file

# haru_vllm

<!-- This README is also the project description on PyPI, where relative links do not resolve. Link to other files in the repository with absolute GitHub URLs. -->

Serve Haru with [vLLM-Omni](https://github.com/vllm-project/vllm-omni): many
concurrent full-duplex conversations on one GPU, over the realtime WebSocket
API.

This is a plugin. Installing it registers the Haru model, pipeline and duplex
policy with vLLM-Omni; nothing in vLLM-Omni is patched.

## Install and run

```bash
pip install vllm==0.32.0 vllm-omni==0.32.0
pip install -e ../haru -e .             # from this directory

vllm serve higurashi-ai/Haru-4B --omni --deploy-config "$(python -m haru_vllm.deploy)"
```

The server listens on `ws://localhost:8000/v1/realtime?duplex=1`. To use it
from the web demo:

```bash
python -m haru.server --backend vllm --vllm-url http://localhost:8000
```

`python -m haru_vllm.deploy` prints the path of the bundled deploy
configuration ([`deploy/haru.yaml`](https://github.com/higurashi-ai/Haru/blob/main/haru_vllm/haru_vllm/deploy/haru.yaml)). Copy and edit
it to change the number of concurrent conversations (`duplex_session.max_sessions`
and `max_num_seqs`) or the cache size.

Tested with vLLM 0.32.0 and vLLM-Omni 0.32.0, the releases on PyPI. It also runs
with vLLM 0.33.0 and the vLLM-Omni main branch at commit `04e03a4`. The plugin
uses model-runner hooks of vLLM-Omni that may change between versions.

## Protocol

The endpoint speaks vLLM-Omni's realtime duplex protocol (OpenAI Realtime
events). One conversation:

1. `session.update` opens the session. `instructions` is the character card,
   `ref_audio` a `data:audio/...;base64,` URI of the voice. Audio input is
   `pcm_f32le` at 24 kHz.
2. Append one frame (3200 samples) with `input_audio_buffer.append`. This first
   append starts the conversation: the prompt runs and the model's frame 0
   comes back. Its audio is not used; send silence.
3. From then on, every appended frame `k` is answered by model frame `k + 1`:
   one `response.output_audio.delta` (PCM16, 24 kHz, 3200 samples) and, when
   the frame has text, a `response.output_audio_transcript.delta`.

Options in `session.extra_body`, fixed when the session is created:

| Key | Default | Meaning |
| --- | --- | --- |
| `guidance_scale` | 1.5 | Audio guidance, 1 to 2 |
| `top_k`, `top_p` | 25, 1.0 | Text sampling (with the session's `temperature`, default 0.7) |
| `eop_bias` | 0 | Added to the logit of the "start speaking" token |
| `speak_first` | false | The model opens the conversation |
| `dry` | true | Repetition penalty on the text stream |
| `seed` | random | Seed of the audio sampling |

## How it is built

One autoregressive stage. Each conversation is a single resumable engine
request whose prompt grows by one position per frame; the language model runs
on vLLM's engine (paged attention cache, Gated DeltaNet state, CUDA graphs).
The audio parts run in model hooks of the same step: the user frame is encoded
before the forward pass, and the MeanFlow head and codec decoder run right
after sampling, so a frame's audio leaves in the step that produced it. The
codec runs as Triton kernels replayed from CUDA graphs when `triton` is
installed (PyTorch's Linux CUDA wheels install it), and in plain PyTorch
otherwise. Set `HARU_CODEC_BACKEND=torch` (or `triton`) in the environment of
`vllm serve` to override that choice.
