Metadata-Version: 2.1
Name: llm-inference-engine
Version: 0.1.6
Summary: LLM inferencing utilities for multiple projects.
License: MIT
Author: Enshuo (David) Hsu
Requires-Python: >=3.11,<4.0
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Description-Content-Type: text/markdown

![Python Version](https://img.shields.io/pypi/pyversions/llm-inference-engine)
![PyPI](https://img.shields.io/pypi/v/llm-inference-engine)

# llm-inference-engine

A unified Python interface for LLM inference across multiple backends. Separates **model configuration** (how a model behaves) from **inference engines** (where it runs), so you can swap backends without changing application code.

## Table of Contents

- [Installation](#installation)
- [Quick Start](#quick-start)
- [LLM Configs](#llm-configs)
  - [BasicLLMConfig](#basicllmconfig)
  - [ReasoningLLMConfig](#reasoningllmconfig)
  - [Qwen3LLMConfig](#qwen3llmconfig)
  - [OpenAIReasoningLLMConfig](#openaireasoningllmconfig)
  - [Passing extra_body for Hybrid Thinking Models](#passing-extra_body-for-hybrid-thinking-models)
- [Inference Engines](#inference-engines)
  - [OpenAIInferenceEngine](#openaiinferenceengine)
  - [AzureOpenAIInferenceEngine](#azureopenaiinferenceengine)
  - [VLLMInferenceEngine](#vllminferenceengine)
  - [SGLangInferenceEngine](#sglanginferenceengine)
  - [OpenRouterInferenceEngine](#openrouterinferenceengine)
  - [OllamaInferenceEngine](#ollamainferenceengine)
  - [HuggingFaceHubInferenceEngine](#huggingfacehubinferenceengine)
  - [LiteLLMInferenceEngine](#litellminferenceengine)
- [Chat Methods](#chat-methods)
- [End-to-End Examples](#end-to-end-examples)
- [Message Logging](#message-logging)
- [Config Cheat Sheet](#config-cheat-sheet)

## Installation

```bash
pip install llm-inference-engine
```

Then install the client library for your backend:

| Backend | Install |
|---------|---------|
| OpenAI / Azure OpenAI | `pip install openai` |
| vLLM (OpenAI-compatible) | `pip install openai` |
| SGLang (OpenAI-compatible) | `pip install openai` |
| OpenRouter | `pip install openai` |
| Ollama | `pip install ollama` |
| Hugging Face Hub | `pip install huggingface-hub` |
| LiteLLM | `pip install litellm` |

## Quick Start

```python
from llm_inference_engine import BasicLLMConfig, OpenAIInferenceEngine

config = BasicLLMConfig(max_new_tokens=1024, temperature=0.5)
engine = OpenAIInferenceEngine(model="gpt-4.1-mini", config=config)

response = engine.chat([{"role": "user", "content": "Hello!"}])
print(response["response"])
```

## LLM Configs

Configs control **model behavior**: token limits, temperature, message preprocessing, and response postprocessing. Pick the config that matches your model type.

### BasicLLMConfig

For standard non-reasoning models (GPT-4.1, Llama, Qwen3-instruct, etc.).

```python
from llm_inference_engine import BasicLLMConfig

config = BasicLLMConfig(max_new_tokens=2048, temperature=0.7)
```

Response format: `{"response": "...", "tool_calls": [...]}`

### ReasoningLLMConfig

For always-on reasoning models that wrap thinking in `<think>...</think>` tags (DeepSeek-R1, Qwen3.5-thinking, GPT-OSS, Gemma 4, etc.). Automatically parses thinking and response content.

```python
from llm_inference_engine import ReasoningLLMConfig

config = ReasoningLLMConfig(max_new_tokens=4096, temperature=0.6)
```

Response format: `{"reasoning": "...", "response": "...", "tool_calls": [...]}`

Custom thinking tokens (if the model uses different delimiters):

```python
config = ReasoningLLMConfig(thinking_token_start="<reasoning>", thinking_token_end="</reasoning>")
```

### Qwen3LLMConfig

For **Qwen3 hybrid thinking** models. Appends `/think` or `/no_think` tokens to system and user messages to toggle thinking mode. This is the token-based approach that works across all serving engines.

```python
from llm_inference_engine import Qwen3LLMConfig

# Enable thinking
config = Qwen3LLMConfig(thinking_mode=True, max_new_tokens=4096)

# Disable thinking for faster responses
config = Qwen3LLMConfig(thinking_mode=False, max_new_tokens=2048)
```

Response format: `{"reasoning": "...", "response": "...", "tool_calls": [...]}` (reasoning is empty when thinking is disabled)

### OpenAIReasoningLLMConfig

For OpenAI o-series models (o4-mini etc.). Handles reasoning effort, removes unsupported temperature parameter, and concatenates system prompts into user prompts (required by these models).

```python
from llm_inference_engine import OpenAIReasoningLLMConfig

config = OpenAIReasoningLLMConfig(reasoning_effort="low")   # "low", "medium", or "high"
```

Response format: `{"reasoning": "...", "response": "...", "tool_calls": [...]}`

### Passing `extra_body` for Hybrid Thinking Models

Some models (Gemma 4, Qwen3.5, etc.) use **server-side thinking control** via the `extra_body` parameter in the OpenAI chat completion API. Since `extra_body` format depends on the serving engine (vLLM, SGLang, etc.) rather than the model itself, this is handled through kwargs on any config class rather than a dedicated config.

All config classes accept arbitrary kwargs that get passed through to the API call:

**Gemma 4 with thinking enabled (vLLM / SGLang):**

```python
from llm_inference_engine import ReasoningLLMConfig

config = ReasoningLLMConfig(
    max_new_tokens=4096,
    temperature=1.0,
    extra_body={
        "chat_template_kwargs": {"enable_thinking": True}
    }
)
```

**Gemma 4 with thinking disabled:**

```python
from llm_inference_engine import BasicLLMConfig

config = BasicLLMConfig(
    max_new_tokens=2048,
    extra_body={
        "chat_template_kwargs": {"enable_thinking": False}
    }
)
```

**Qwen3.5 with thinking enabled (vLLM):**

```python
from llm_inference_engine import ReasoningLLMConfig

config = ReasoningLLMConfig(
    max_new_tokens=4096,
    extra_body={
        "chat_template_kwargs": {"enable_thinking": True}
    }
)
```

**Qwen3.5 with thinking disabled (vLLM):**

```python
from llm_inference_engine import BasicLLMConfig

config = BasicLLMConfig(
    max_new_tokens=2048,
    extra_body={
        "chat_template_kwargs": {"enable_thinking": False}
    }
)
```

> **Tip:** Use `ReasoningLLMConfig` when thinking is enabled (it parses `<think>` tags in the response). Use `BasicLLMConfig` when thinking is disabled.

## Inference Engines

Engines define **where** the model runs. Each engine maps config parameters to the backend's expected format (e.g., `max_new_tokens` becomes `max_completion_tokens` for OpenAI, `num_predict` for Ollama).

### OpenAIInferenceEngine

```python
from llm_inference_engine import OpenAIInferenceEngine

engine = OpenAIInferenceEngine(model="gpt-4.1-mini", config=config)
```

Set `OPENAI_API_KEY` environment variable, or pass `api_key=` directly.

### AzureOpenAIInferenceEngine

```python
from llm_inference_engine import AzureOpenAIInferenceEngine

engine = AzureOpenAIInferenceEngine(
    model="gpt-4.1-mini",
    api_version="2025-03-01-preview",
    azure_endpoint="https://your-resource.openai.azure.com/",
    api_key="your-api-key"
)
```

Set `AZURE_OPENAI_API_KEY` environment variable, or pass `api_key=` directly.

Set `AZURE_OPENAI_ENDPOINT` environment variable, or pass `azure_endpoint=` directly.

### VLLMInferenceEngine

For models served via [vLLM](https://docs.vllm.ai/en/latest/serving/openai_compatible_server.html).

```python
from llm_inference_engine import VLLMInferenceEngine

engine = VLLMInferenceEngine(
    model="Qwen/Qwen3.5-35B-A3B",
    api_key="",                                  # optional for local servers
    base_url="http://localhost:8000/v1"           # default
)
```

### SGLangInferenceEngine

For models served via [SGLang](https://docs.sglang.ai/basic_usage/openai_api.html).

```python
from llm_inference_engine import SGLangInferenceEngine

engine = SGLangInferenceEngine(
    model="google/gemma-4-26B-A4B-it",
    api_key="",                                   # optional for local servers
    base_url="http://localhost:30000/v1"           # default
)
```

### OpenRouterInferenceEngine

```python
from llm_inference_engine import OpenRouterInferenceEngine

engine = OpenRouterInferenceEngine(model="openai/gpt-oss-120b")
```

Set `OPENROUTER_API_KEY` environment variable, or pass `api_key=` directly.

#### Reasoning field key compatibility

OpenAI-compatible servers have exposed the model's reasoning/thinking text under
different JSON keys across versions (e.g. vLLM/SGLang historically used
`reasoning_content`, while OpenRouter and newer builds use `reasoning`). By
default each engine probes the known keys in order (`reasoning_content`,
`reasoning`, `reason`) and uses the first one present, so reasoning is picked up
regardless of server version. If your server exposes it under a non-standard key,
override the probe order per instance:

```python
engine = VLLMInferenceEngine(
    model="Qwen/Qwen3.5-35B-A3B",
    reasoning_keys=["my_custom_reasoning_field", "reasoning_content"],
)
```

This applies to `OpenAIInferenceEngine`, `VLLMInferenceEngine`,
`SGLangInferenceEngine`, and `OpenRouterInferenceEngine`.

### OllamaInferenceEngine

```python
from llm_inference_engine import OllamaInferenceEngine

engine = OllamaInferenceEngine(
    model_name="qwen3:27b",
    num_ctx=4096,                                 # context length
    keep_alive=300                                 # seconds to hold model in memory
)
```

### HuggingFaceHubInferenceEngine

```python
from llm_inference_engine import HuggingFaceHubInferenceEngine

engine = HuggingFaceHubInferenceEngine(model="meta-llama/Llama-4-Scout-17B-16E-Instruct")
```

### LiteLLMInferenceEngine

```python
from llm_inference_engine import LiteLLMInferenceEngine

engine = LiteLLMInferenceEngine(
    model="openai/gpt-4.1-mini",
    api_key="your-api-key"
)
```

## Chat Methods

All engines support four chat methods:

| Method | Sync/Async | Returns |
|--------|-----------|---------|
| `chat()` | Sync | `dict` |
| `chat_stream()` | Sync | `Generator[dict]` |
| `chat_async()` | Async | `dict` |
| `chat_async_stream()` | Async | `AsyncGenerator[dict]` |

### Sync

```python
# Non-streaming
response = engine.chat(messages)
print(response["response"])

# With verbose output (prints to terminal in real-time)
response = engine.chat(messages, verbose=True)

# Streaming
for chunk in engine.chat_stream(messages):
    print(chunk["data"], end="", flush=True)  # chunk["type"] is "reasoning" or "response"
```

### Async

```python
import asyncio

async def main():
    # Non-streaming
    response = await engine.chat_async(messages)
    print(response["response"])

    # Streaming
    stream = await engine.chat_async_stream(messages)
    async for chunk in stream:
        print(chunk["data"], end="", flush=True)

asyncio.run(main())
```

### Streaming Chunk Format

Streaming methods yield dicts with `type` and `data` keys:

```python
{"type": "reasoning", "data": "Let me think..."}    # thinking content
{"type": "response", "data": "The answer is 42."}    # response content
{"type": "tool_calls", "data": [{"name": "...", "arguments": "..."}]}  # tool calls
```

## End-to-End Examples

### GPT-4.1 via Azure OpenAI

```python
from llm_inference_engine import BasicLLMConfig, AzureOpenAIInferenceEngine

config = BasicLLMConfig(max_new_tokens=1024, temperature=0.5)
engine = AzureOpenAIInferenceEngine(
    model="gpt-4.1-mini",
    api_version="2025-03-01-preview",
    azure_endpoint="https://your-resource.openai.azure.com/"
)

response = engine.chat([{"role": "user", "content": "Summarize this document."}])
print(response["response"])
```

### GPT-OSS Reasoning via OpenRouter

```python
from llm_inference_engine import OpenAIReasoningLLMConfig, OpenRouterInferenceEngine

config = ReasoningLLMConfig(reasoning_effort="medium")
engine = OpenRouterInferenceEngine(model="openai/gpt-oss-120b", config=config)

response = engine.chat([
    {"role": "system", "content": "You are a math tutor."},
    {"role": "user", "content": "Prove that sqrt(2) is irrational."}
])
print(response["reasoning"])  # chain-of-thought
print(response["response"])   # final answer
```

### GPT-5.4 via OpenAI

```python
from llm_inference_engine import ReasoningLLMConfig, OpenAIInferenceEngine

config = ReasoningLLMConfig(max_new_tokens=4096, temperature=0.7)
engine = OpenAIInferenceEngine(model="gpt-5.4-mini", config=config)

response = engine.chat([{"role": "user", "content": "Write a Python function to merge two sorted lists."}])
print(response["response"])
```

### Qwen3.5 with Thinking via vLLM

```python
from llm_inference_engine import ReasoningLLMConfig, VLLMInferenceEngine

config = ReasoningLLMConfig(
    max_new_tokens=4096,
    temperature=0.6,
    extra_body={"chat_template_kwargs": {"enable_thinking": True}}
)
engine = VLLMInferenceEngine(model="Qwen/Qwen3.5-35B-A3B", base_url="http://localhost:8000/v1")

response = engine.chat([{"role": "user", "content": "What causes tides?"}])
print(response["reasoning"])
print(response["response"])
```

### Gemma 4 VLM with Thinking + High Resolution for OCR (vLLM)

For vision tasks with Gemma 4, use `mm_processor_kwargs` in `extra_body` to control the vision token budget per image. Supported values: 70, 140, 280 (default), 560, or 1120 tokens. Higher values preserve more detail for OCR (https://huggingface.co/google/gemma-4-E4B).

> **Note:** `max_soft_tokens` can also be set at server launch via `--mm-processor-kwargs '{"max_soft_tokens": 1120}'`. The `extra_body` approach below overrides per request.

```python
import base64
from llm_inference_engine import ReasoningLLMConfig, VLLMInferenceEngine

config = ReasoningLLMConfig(
    max_new_tokens=4096,
    temperature=1.0,
    top_p=0.95,
    extra_body={
        "chat_template_kwargs": {"enable_thinking": False}, # OCR doesn't require thinking
        "mm_processor_kwargs": {"max_soft_tokens": 1120} # max detail for OCR 
        "top_k": 64
    }
)
engine = VLLMInferenceEngine(model="google/gemma-4-27b-it", base_url="http://localhost:8000/v1")

with open("document.png", "rb") as f:
    b64_image = base64.b64encode(f.read()).decode()

messages = [
    {
        "role": "user",
        "content": [
            {"type": "image_url", "image_url": {"url": f"data:image/png;base64,{b64_image}"}},
            {"type": "text", "text": "Extract all text from this document."}
        ]
    }
]

response = engine.chat(messages)
print(response["reasoning"])  # Gemma's thinking process
print(response["response"])   # extracted text
```

### Gemma 4 VLM via SGLang

```python
from llm_inference_engine import ReasoningLLMConfig, SGLangInferenceEngine

config = ReasoningLLMConfig(
    max_new_tokens=4096,
    temperature=1.0,
    extra_body={
        "chat_template_kwargs": {"enable_thinking": True},
        "mm_processor_kwargs": {"max_soft_tokens": 1120}
    }
)
engine = SGLangInferenceEngine(model="google/gemma-4-27b-it", base_url="http://localhost:30000/v1")
```

### Async Batch Processing with Rate Limiting

```python
import asyncio
from llm_inference_engine import BasicLLMConfig, VLLMInferenceEngine

config = BasicLLMConfig(max_new_tokens=512)
engine = VLLMInferenceEngine(
    model="Qwen/Qwen3.5-32B",
    max_concurrent_requests=10,
    max_requests_per_minute=60,
    config=config
)

questions = ["What is Python?", "What is Rust?", "What is Go?"]

async def main():
    tasks = [
        engine.chat_async([{"role": "user", "content": q}])
        for q in questions
    ]
    responses = await asyncio.gather(*tasks)
    for q, r in zip(questions, responses):
        print(f"Q: {q}\nA: {r['response']}\n")

asyncio.run(main())
```

## Message Logging

Use `MessagesLogger` to capture full conversation history (prompts + responses).

```python
from llm_inference_engine import MessagesLogger

logger = MessagesLogger(store_images=False)  # replaces image URLs with "[image]"

response = engine.chat(messages, messages_logger=logger)

# Retrieve logged conversations
for conversation in logger.get_messages_log():
    for msg in conversation:
        print(f"{msg['role']}: {msg['content'][:100]}")
```

## Config Cheat Sheet

| Model | Config | Key Settings |
|-------|--------|-------------|
| GPT-4.1, GPT-5.4 | `BasicLLMConfig` | `max_new_tokens`, `temperature` |
| o4-mini, GPT-OSS | `OpenAIReasoningLLMConfig` | `reasoning_effort` |
| Qwen3 (hybrid) | `Qwen3LLMConfig` | `thinking_mode` |
| Qwen3.5 (hybrid, vLLM) | `ReasoningLLMConfig` | `extra_body={"chat_template_kwargs": {"enable_thinking": True}}` |
| Gemma 4 (hybrid) | `ReasoningLLMConfig` | `extra_body={"chat_template_kwargs": {"enable_thinking": True}}` |
| DeepSeek-R1 | `ReasoningLLMConfig` | defaults |
| Llama, Mistral | `BasicLLMConfig` | `max_new_tokens`, `temperature` |

