Metadata-Version: 2.4
Name: omnivoice-triton-server
Version: 0.1.1
Summary: OpenAI-compatible OmniVoice server with batched multi-GPU Triton inferers
Requires-Python: >=3.12
Description-Content-Type: text/markdown
Requires-Dist: fastapi
Requires-Dist: uvicorn[standard]
Requires-Dist: pydantic>=2
Requires-Dist: pydantic-settings>=2
Requires-Dist: python-multipart
Requires-Dist: httpx
Requires-Dist: numpy
Requires-Dist: regex
Requires-Dist: jieba
Requires-Dist: torch
Requires-Dist: torchaudio
Requires-Dist: soundfile
Requires-Dist: triton
Requires-Dist: transformers
Requires-Dist: accelerate
Requires-Dist: pydub
Requires-Dist: concurrent-log-handler

# OmniVoice Triton Server

OpenAI-compatible OmniVoice TTS server built for batched GPU inference. The
server runs FastAPI workers for request handling and one or more independent GPU
inferer processes for model execution.

中文说明: [docs/README.zh-CN.md](docs/README.zh-CN.md)

## Origin

This repository combines three code lines into one deployable service:

- `omnivoice-server`: API server, request routing, socket IPC, batching, metrics,
  deployment scripts, and tests.
- `omnivoice-triton`: Triton/hybrid inference backend pieces and CUDA/Triton
  acceleration code.
- `k2-fsa/OmniVoice`: selected OmniVoice model/runtime code under
  `src/modeling`.

The code is kept in a single tree because scheduling, chunking, graph capture,
and model invocation need to be tuned together. This is not a clean upstream
mirror; it is a server-oriented integration.

## Main Optimizations

- Multi-GPU serving with `--gpu-inferer N`. Each GPU inferer is a separate
  process with its own model weights on one CUDA device.
- FastAPI worker count defaults to the effective GPU inferer count when
  `--fastapi-workers` is not specified.
- Worker-side preprocessing: validation, semantic text chunking, duration
  splitting, clone reference audio ingestion, response formatting, and SSE
  framing run outside the inferer.
- Async local TCP socket IPC between workers and inferers. Clone reference audio
  is sent through the socket; the server does not create temporary prompt audio
  files for normal requests.
- Shared-memory metrics snapshots, so `/metrics` does not block generation.
- Clone audio prompt LRU cache controlled by
  `--max-clone-audio-prompt-cache`.
- `chunk_mode` request control:
  - `concurrent`: default. Clone chunks share the same clone prompt. Auto/design
    generate chunk 0 first, then use that result as continuity prompt for the
    remaining chunks, which can run concurrently.
  - `sequential`: each chunk uses the previous generated chunk as continuity
    prompt.
  - `none`: still chunks text, but estimates a larger chunk size from the model
    context limit to avoid unnecessary splitting.
- Mixed-language semantic chunking with CJK/Thai/Hangul character counting,
  non-CJK token counting, punctuation-aware recursive splitting, and balanced
  packing near the configured word limit.
- Fixed-shape CUDA Graph prewarming with compact batch/width buckets, memory
  headroom checks, graph hit/miss metrics, and graph-aware microbatch splitting
  for oversized batches.

CPU inferer code was removed. Scale this server with GPU inferer processes.

## Requirements

- Python 3.12 or newer.
- PyTorch, Triton, Transformers, FastAPI, and the packages in
  `requirements.txt`.
- NVIDIA GPU for inference. The default runner mode is `hybrid`.
- OmniVoice model files available either from a Hugging Face model id or a local
  path passed with `--model-id` / `OMNIVOICE_MODEL_ID`.

## Quick Start

```bash
python -m venv .venv
. .venv/bin/activate
pip install omnivoice-triton-server

export CUDA_VISIBLE_DEVICES=0
omnivoice-triton-server start
```

Two-GPU example:

```bash
CUDA_VISIBLE_DEVICES=0,1 \
omnivoice-triton-server start \
  --port 9194 \
  --model-id /path/to/OmniVoice \
  --gpu-inferer 2 \
  --max-batch-size 16 \
  --max-batch-latency 250 \
  --cuda-stream-count 2 \
  --runner-mode hybrid \
  --num-step 32
```

Installing the package adds an `omnivoice-triton-server` Python console command.
The module entrypoint is also available:

```bash
python -m omnivoice-triton-server start --port 9194
```

Stop a foreground/background process by port or pid file:

```bash
omnivoice-triton-server stop --port 9194
omnivoice-triton-server stop --pid-file logs/20260520-212301/server.pid --no-port
```

Stop a systemd deployment:

```bash
omnivoice-triton-server stop --systemd --service-name omnivoice-server
```

Install or update a systemd deployment from the installed CLI:

```bash
omnivoice-triton-server install-service \
  --cuda-visible-devices 0,1 \
  --python "$(command -v python)" \
  --service-name omnivoice-server \
  --working-dir "$PWD" \
  -- \
  --port 9194 \
  --model-id /path/to/OmniVoice \
  --gpu-inferer 2 \
  --max-batch-size 16
```

`CUDA_VISIBLE_DEVICES` is a deployment choice. The benchmark below used
`CUDA_VISIBLE_DEVICES=6,7` on one 8-GPU test server because those two devices
were selected for that run; use the device ids that are correct on your machine.
`scripts/start_server.sh` is only a POSIX shell convenience wrapper around the
same module entrypoint.

The repository also includes a shell wrapper for source-tree deployments:

```bash
scripts/install_systemd_service.sh \
  --cuda-visible-devices 0,1 \
  --python /path/to/python \
  --service-name omnivoice-server \
  -- \
  --port 9194 \
  --model-id /path/to/OmniVoice \
  --gpu-inferer 2 \
  --max-batch-size 16
```

Arguments after `--` are passed directly to `omnivoice-triton-server start`.
The script writes `/etc/omnivoice/<service>.sh` and
`/etc/systemd/system/<service>.service`, then reloads, enables, and restarts
the unit unless `--no-enable` or `--no-start` is used.

## Important Arguments

- `--model-id`: local model path or Hugging Face model id.
- `--gpu-inferer`: number of GPU inferer processes to launch. The launcher
  clamps this to the number of visible CUDA devices.
- `--fastapi-workers`: API worker count. Defaults to effective GPU inferer
  count when omitted.
- `--max-batch-size`, `--max-batch-latency`: scheduler batching controls.
- `--cuda-stream-count`: backend worker streams per inferer.
- `--cuda-graph-min-width`, `--cuda-graph-max-width`: graph width controls.
- `--cuda-graph-auto-width-tokens-per-word`,
  `--cuda-graph-auto-max-width`: context-limit estimation for `chunk_mode=none`.
- `--num-step`: global generation step count. Default: `32`.
- `--max-clone-audio-prompt-cache`: clone prompt LRU size. Default: `32`.
- `--max-continuity-audio-tokens`,
  `--max-continuity-text-words`: chunk continuity prompt limits.
- `--text-chunk-words` and `--text-chunk-*`: chunking and packing controls.
- `--log-dir`, `--log-run-id`, `--log-file`, `--pid-file`: runtime log layout.

All settings can also be set with `OMNIVOICE_*` environment variables.
Python defaults live in `src/config.py`; shell scripts do not define service
defaults.

## API

Speech endpoint:

```bash
curl -X POST http://127.0.0.1:9194/v1/audio/speech \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "tts-1",
    "input": "Hello from OmniVoice.",
    "voice": "auto",
    "response_format": "wav",
    "speed": 1.0,
    "chunk_mode": "concurrent"
  }' \
  --output speech.wav
```

Voice design endpoint:

```bash
curl -X POST http://127.0.0.1:9194/v1/audio/design \
  -F 'text=Hello from a designed voice.' \
  -F 'instruct=female, young adult, moderate pitch' \
  -F 'chunk_mode=concurrent' \
  -F 'response_format=wav' \
  --output design.wav
```

Voice clone endpoint:

```bash
curl -X POST http://127.0.0.1:9194/v1/audio/clone \
  -F 'text=Hello from a cloned voice.' \
  -F 'ref_audio=@ref.wav;type=audio/wav' \
  -F 'ref_text=Text spoken in the reference audio.' \
  -F 'chunk_mode=concurrent' \
  -F 'response_format=wav' \
  --output clone.wav
```

Supported response formats are `wav` and raw `pcm`.

## Benchmark

These numbers are for capacity planning on one local test host. They are not a
hardware-independent promise.

Test configuration:

- Hardware used by this service: 2 x NVIDIA GeForce RTX 3080, 20 GiB each.
- Host inventory: 8 visible RTX 3080 GPUs; this run used
  `CUDA_VISIBLE_DEVICES=6,7`.
- Launch: `--gpu-inferer 2 --fastapi-workers 2 --runner-mode hybrid --dtype fp16
  --max-batch-size 16 --max-batch-latency 250 --cuda-stream-count 2
  --num-step 32`.
- Load generator: 1000 requests scheduled at 100 req/s. Latency therefore
  includes queueing after the offered load exceeds service capacity.
- Audio quality smoke: auto, design, and clone outputs were checked by ASR on
  both short and long texts.

### Throughput And Latency

| Workload | Wall time | Completed req/s | Generated audio | Audio realtime | RTF | Mean latency | p50 | p95 | p99 |
| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
| Short speech/design | 61.553 s | 16.246 | 786.100 s | 12.771x | 0.0783 | 27.2485 s | 26.6065 s | 50.8381 s | 54.6932 s |
| Mixed short/medium/long speech/design/clone | 247.159 s | 4.046 | 2,648.752 s | 10.717x | 0.0933 | 120.7916 s | 119.3613 s | 228.5990 s | 236.4849 s |

`Audio realtime` is generated audio duration divided by wall time. `RTF` is
wall time divided by generated audio duration.

### Scheduler Efficiency

| Workload | Client requests | Backend tasks | Tasks/request | Backend batches | Tasks/backend batch | Backend task/s |
| --- | ---: | ---: | ---: | ---: | ---: | ---: |
| Short speech/design | 1,000 | 1,000 | 1.000 | 68 | 14.706 | 16.246 |
| Mixed short/medium/long speech/design/clone | 1,000 | 1,733 | 1.733 | 67 | 25.866 | 7.011 |

The mixed run has more backend tasks than client requests because long inputs are
split into chunks. The useful batching signal is `Tasks/backend batch`: higher
means the scheduler kept the GPU inferers fed with larger model batches.

### Mixed Workload Breakdown

| Kind | Requests | Mean latency | p95 | Max |
| --- | ---: | ---: | ---: | ---: |
| speech | 900 | 122.8774 s | 228.6261 s | 236.6119 s |
| design | 50 | 129.3640 s | 229.6973 s | 230.9176 s |
| clone | 50 | 74.6742 s | 143.6043 s | 145.6617 s |

### CUDA Graph Behavior

Final mixed run graph state, aggregated across two GPU inferers:

- Graph entries per inferer: 14.
- Captured shapes per inferer:
  `(2,8,128)`, `(2,8,160)`, `(2,8,256)`, `(2,8,512)`,
  `(8,8,64)`, `(8,8,128)`, `(8,8,160)`, `(8,8,256)`,
  `(16,8,128)`, `(16,8,160)`, `(16,8,256)`,
  `(32,8,64)`, `(32,8,128)`, `(32,8,160)`.
- During the mixed 1000-request run: 11,520 graph hits and 32 graph misses
  after subtracting the pre-run counters.
- Max backend batch seen by each inferer: 49 and 46 tasks.

The graph miss count is the number to watch when changing chunking, graph width
buckets, or `--max-batch-size`; sustained misses usually mean requests are
falling outside the prewarmed shape plan.

## Development Checks

```bash
python -m py_compile \
  src/app.py src/audio.py src/chunking.py src/config.py src/infer_client.py \
  src/inferer.py src/launcher.py src/protocol.py

PYTHONPATH=src python tests/test_chunking.py
python tests/test_api.py
python tests/load_1000_rps100.py --total 1000 --rate 100 --concurrency-limit 512
python tests/load_mixed_1000.py --total 1000 --rate 100 --ref-audio /path/to/ref.wav
```

Runtime artifacts, logs, generated audio, model weights, and local environment
files are ignored by `.gitignore`.

See `docs/DEPLOYMENT.md` for operational details.
