Metadata-Version: 2.4
Name: tokenspeed-smg-grpc-servicer
Version: 0.12.0.post20260922
Summary: SMG gRPC servicer implementations for LLM inference engines (vLLM, MLX, TokenSpeed, SGLang)
Author-email: Chang Su <mckvtl@gmail.com>, Simo Lin <linsimo.mark@gmail.com>
License-Expression: Apache-2.0
Project-URL: Homepage, https://github.com/smg-project/smg
Project-URL: Repository, https://github.com/smg-project/smg
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Operating System :: OS Independent
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: tokenspeed-smg-grpc-proto>=0.4.20
Requires-Dist: grpcio>=1.81.1
Requires-Dist: grpcio-reflection>=1.81.1
Requires-Dist: grpcio-health-checking>=1.81.1
Provides-Extra: vllm
Requires-Dist: vllm>=0.19.0; extra == "vllm"
Requires-Dist: pyzmq>=25.0.0; extra == "vllm"
Requires-Dist: msgspec>=0.18.0; extra == "vllm"
Provides-Extra: vllm-redis
Requires-Dist: redis>=5.0; extra == "vllm-redis"
Requires-Dist: msgspec>=0.18.0; extra == "vllm-redis"
Provides-Extra: sglang
Requires-Dist: sglang>=0.5.20; extra == "sglang"
Provides-Extra: mlx
Requires-Dist: tokenspeed-smg-grpc-proto>=0.4.7; extra == "mlx"
Requires-Dist: mlx>=0.22.0; extra == "mlx"
Requires-Dist: mlx-lm>=0.22.0; extra == "mlx"
Requires-Dist: transformers<5.13; extra == "mlx"
Provides-Extra: tokenspeed
Requires-Dist: pyzmq>=25.0.0; extra == "tokenspeed"
Requires-Dist: msgspec>=0.18.0; extra == "tokenspeed"
Dynamic: license-file

# smg-grpc-servicer

gRPC servicer implementations for LLM inference engines. Supports vLLM, MLX, TokenSpeed, and SGLang.

## Installation

For vLLM:

```bash
pip install smg-grpc-servicer[vllm]
```

For MLX:

```bash
pip install smg-grpc-servicer[mlx]
```

For TokenSpeed, install the TokenSpeed runtime first, then install the servicer bridge:

```bash
pip install smg-grpc-servicer
```

For SGLang:

```bash
pip install smg-grpc-servicer[sglang]
```

## Usage

### vLLM

```bash
vllm serve meta-llama/Llama-2-7b-hf --grpc
```

#### Worker-side multimodal processing (media refs)

By default the smg router fetches and preprocesses images itself and sends
pixel tensors. A vLLM gRPC worker can instead accept media references (URLs)
and run vLLM's own multimodal processor:

```bash
SMG_VLLM_MM_PROCESSOR=inprocess vllm serve Qwen/Qwen3-VL-8B-Instruct --grpc \
    --allowed-media-domains example.com
```

The worker then advertises `mm_processor=inprocess` and `mm_media_ref_schemes`
through `GetServerInfo`; a router with media-reference support forwards
`media_refs` only to workers that advertise, and a router without it ignores the
labels and keeps sending preprocessed tensors. vLLM's `--allowed-media-domains`,
`--allowed-local-media-path`, `--media-io-kwargs`, `--limit-mm-per-prompt` and
`VLLM_*_FETCH_TIMEOUT` govern fetching on the worker; without
`--allowed-media-domains` the worker fetches from any host the router forwards.
Related knobs: `SMG_VLLM_MM_MAX_INFLIGHT` (default 64) bounds concurrent media
jobs; `SMG_VLLM_MM_MAX_ITEMS` (default 16) caps references per request;
`SMG_VLLM_MM_MAX_ITEM_BYTES` (default 32 MiB) caps inline `data:` payloads.

On the router side, `SMG_MM_PROCESSING` selects `auto` (default: forward when
the model's spec opts in and every registered worker of the model advertises
`mm_processor`), `router` (always preprocess) or `worker` (strict: 400 when a
request cannot be forwarded); the outcome is counted in
`smg_mm_processing_total{model,mode,reason}`. It is read from the router's
environment only and has no router-config equivalent. Any other value stops the
router at startup instead of quietly reverting to `auto`. On the worker path the
router never expands placeholders, so routing decisions that weigh the prompt's token
count (cache-aware policies, load estimates) see one token per media item where
the worker will schedule the full placeholder run. The `E2E_MM_PROCESSING=worker`
e2e lanes run the multimodal suites in this mode, and
`crates/multimodal/scripts/check_worker_anchor_parity.py` checks that a spec's
anchor is the token vLLM expands.

To move fetching and processing out of the vLLM process, run the GPU-free
sidecar next to a private Redis and point the worker at it
(`pip install smg-grpc-servicer[vllm,vllm-redis]`):

```bash
python -m smg_grpc_servicer.vllm.mm_sidecar --model Qwen/Qwen3-VL-8B-Instruct \
    --redis-url redis://127.0.0.1:6379/0 --allowed-media-domains example.com
SMG_VLLM_MM_PROCESSOR=redis SMG_VLLM_MM_REDIS_URL=redis://127.0.0.1:6379/0 \
    vllm serve Qwen/Qwen3-VL-8B-Instruct --grpc
```

The sidecar and the worker must agree on model, vLLM version, dtype, video
backend, media/processor kwargs and `--limit-mm-per-prompt` (pass the flag to
both processes; the sidecar's limit is the one that applies, the limit is
resolved per modality before hashing so equivalent spellings match, and the key
namespace is derived from all of these): the worker advertises
`mm_processor=redis` only while a sidecar with a matching fingerprint keeps its
`hello` key alive, and rejects results that disagree. Jobs and results travel over Redis lists under
`smg:mm:v1:{namespace}`; results carry full tensors keyed by a per-attempt job
id and expire after 120 s. Knobs: `SMG_VLLM_MM_SIDECAR_TIMEOUT_MS` (30000),
`SMG_VLLM_MM_SIDECAR_MAX_QUEUE` (256, fail fast when the queue is deeper),
`SMG_VLLM_MM_SIDECAR_NAMESPACE` (override the derived namespace).

### MLX

```bash
python -m smg_grpc_servicer.mlx --model meta-llama/Llama-2-7b-hf --host 0.0.0.0 --port 50051
```

### TokenSpeed

```bash
python -m smg_grpc_servicer.tokenspeed --model meta-llama/Llama-2-7b-hf --host 0.0.0.0 --port 50051
```

### SGLang

```bash
sglang serve --model-path meta-llama/Llama-2-7b-hf --grpc-mode
```

## Architecture

```
smg-grpc-servicer[vllm]    ──optional dep──>  vllm       (lazy import)
smg-grpc-servicer[mlx]     ──optional dep──>  mlx-lm     (lazy import)
smg-grpc-servicer          ──external runtime──>  tokenspeed (lazy import)
smg-grpc-servicer[sglang]  ──optional dep──>  sglang     (lazy import)
smg-grpc-servicer          ──depends on────>  smg-grpc-proto  (hard dependency)
vllm                       ──optional──────>  smg-grpc-servicer (via vllm serve --grpc)
sglang                     ──optional──────>  smg-grpc-servicer (via --grpc-mode)
```

Backend dependencies are isolated via extras or runtime installs to avoid conflicts between vLLM, MLX, TokenSpeed, and SGLang.

## Development

See [DEVELOPMENT.md](DEVELOPMENT.md) for local development setup, CI, and release workflows.
