Metadata-Version: 2.4
Name: multi-temp-lm
Version: 0.1.0
Summary: Intelligent temperature routing for LLM queries using local embeddings
Author-email: Your Name <your.email@example.com>
License-Expression: MIT
Project-URL: Homepage, https://github.com/yourusername/multi-temp-lm
Project-URL: Repository, https://github.com/yourusername/multi-temp-lm
Project-URL: Issues, https://github.com/yourusername/multi-temp-lm/issues
Keywords: llm,temperature,routing,embeddings,litellm,openai
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: sentence-transformers>=2.2.0
Requires-Dist: numpy>=1.21.0
Requires-Dist: onnxruntime>=1.15.0
Requires-Dist: optimum[onnxruntime]>=1.12.0
Provides-Extra: dev
Requires-Dist: pytest>=7.0; extra == "dev"
Requires-Dist: pytest-asyncio>=0.21.0; extra == "dev"
Requires-Dist: pytest-cov>=4.0; extra == "dev"
Requires-Dist: black>=23.0; extra == "dev"
Requires-Dist: mypy>=1.0; extra == "dev"
Requires-Dist: ruff>=0.1.0; extra == "dev"
Dynamic: license-file

# multi-temp-lm

[![CI](https://github.com/yourusername/multi-temp-lm/actions/workflows/ci.yml/badge.svg)](https://github.com/yourusername/multi-temp-lm/actions/workflows/ci.yml)
[![PyPI](https://img.shields.io/pypi/v/multi-temp-lm)](https://pypi.org/project/multi-temp-lm/)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)

**Intelligent temperature routing for LLM queries using local embeddings.**

`multi-temp-lm` provides a callable `MultiTempRouter` object that automatically classifies query intent and selects the optimal `temperature` value for any LLM SDK. Zero configuration, zero API calls, and universal SDK compatibility.

## Quick Start

```python
from multi_temp_lm import MultiTempRouter
import litellm

router = MultiTempRouter()

response = litellm.completion(
    model="gpt-4",
    messages=[{"role": "user", "content": "Write a poem about stars"}],
    temperature=router,  # Auto-selects optimal temperature
)
```

## Installation

```bash
pip install multi-temp-lm
```

**Dependencies:**
- `sentence-transformers` (for embeddings; model downloads on first use ~22MB with ONNX)
- `numpy`
- `onnxruntime` (optional, for quantized inference)

## Features

- **Intent-aware temperature routing** — Automatically classifies query type and selects optimal temperature
- **Local embeddings** — Uses `sentence-transformers` with ONNX quantization (~22MB), no API calls
- **Universal SDK compatibility** — Works with LiteLLM, OpenAI, Anthropic, Google, Cohere, Ollama, and more
- **Async support** — Non-blocking with `asyncio.to_thread()`
- **LRU cache** — Repeated queries return instantly
- **Customizable** — Override temperatures, add categories, or plug in your own classifier
- **Type annotated** — Ships with `py.typed` marker for full mypy support

## How It Works

`multi-temp-lm` uses a small, local sentence-transformer model (`all-MiniLM-L6-v2`) to classify the intent of every user query. Based on the detected category, it returns a temperature value optimized for that task:

- **Factual / analytical** queries → low temperature (deterministic, precise)
- **Creative / coding** queries → moderate temperature (balanced coherence and diversity)
- **Open-ended chat** → default temperature

The router runs entirely on your machine. It does not send any data to external APIs for classification.

## Default Temperatures

| Category | Temperature | Rationale |
|---|---|---|
| `factual` | 0.2 | High determinism for facts and recall |
| `analytical` | 0.2 | Low variance for structured analysis |
| `creative` | 0.9 | High diversity for poems, stories, and brainstorming |
| `coding` | 0.3 | Focused output for code generation and debugging |
| `reasoning` | 0.4 | Step-by-step logic with controlled creativity |
| `summarization` | 0.3 | Consistent, concise summaries |
| `conversational` | 0.7 | Natural, balanced dialogue |
| `default` | 0.7 | Fallback for unrecognized queries |

Override any of these with the `temperature_map` constructor argument.

## Usage Examples

### LiteLLM (sync)

```python
import litellm
from multi_temp_lm import MultiTempRouter

router = MultiTempRouter()

response = litellm.completion(
    model="gpt-4",
    messages=[{"role": "user", "content": "Explain quantum mechanics."}],
    temperature=router,
)
```

### LiteLLM (async)

```python
response = await litellm.acompletion(
    model="gpt-4",
    messages=messages,
    temperature=await router.aroute(messages),
)
```

### OpenAI SDK (sync)

```python
from openai import OpenAI
from multi_temp_lm import MultiTempRouter

client = OpenAI()
router = MultiTempRouter()

response = client.chat.completions.create(
    model="gpt-4",
    messages=messages,
    temperature=router,
)
```

### OpenAI SDK (async)

```python
from openai import AsyncOpenAI

client = AsyncOpenAI()

response = await client.chat.completions.create(
    model="gpt-4",
    messages=messages,
    temperature=await router.aroute(messages),
)
```

### Anthropic

The Anthropic SDK does not accept callable temperature values. Use `router.for_messages(messages)` for explicit pre-computation:

```python
import anthropic
from multi_temp_lm import MultiTempRouter

client = anthropic.Anthropic()
router = MultiTempRouter()

response = client.messages.create(
    model="claude-3-5-sonnet-20240620",
    max_tokens=1024,
    messages=messages,
    temperature=router.for_messages(messages),
)
```

### Google Vertex AI

```python
import vertexai
from vertexai.generative_models import GenerativeModel
from multi_temp_lm import MultiTempRouter

router = MultiTempRouter()
model = GenerativeModel("gemini-1.5-flash")

response = model.generate_content(
    messages,
    generation_config={"temperature": router.for_messages(messages)},
)
```

### Cohere

```python
import cohere
from multi_temp_lm import MultiTempRouter

client = cohere.Client()
router = MultiTempRouter()

response = client.chat(
    model="command-r",
    message="Explain quantum mechanics.",
    temperature=router.for_messages(messages),
)
```

### Ollama

```python
import ollama
from multi_temp_lm import MultiTempRouter

router = MultiTempRouter()

response = ollama.chat(
    model="llama3",
    messages=messages,
    options={"temperature": router.for_messages(messages)},
)
```

### Generic `for_messages()` Pattern

If your SDK does not support callable temperature, use the explicit pre-computation pattern. This works with **any** SDK that accepts a numeric `temperature`:

```python
from multi_temp_lm import MultiTempRouter

router = MultiTempRouter()

temperature = router.for_messages(messages)
# Pass `temperature` to any SDK call
```

## API Reference

### `MultiTempRouter.__init__`

```python
MultiTempRouter(
    model_name: str = "all-MiniLM-L6-v2",
    temperature_map: Optional[Dict[str, float]] = None,
    references: Optional[List[ReferenceExample]] = None,
    classifier: Optional[Callable[[str], str]] = None,
    default_temperature: float = 0.7,
    override_temperature: Optional[float] = None,
    debug: bool = False,
    cache_size: int = 128,
)
```

| Parameter | Type | Default | Description |
|---|---|---|---|
| `model_name` | `str` | `"all-MiniLM-L6-v2"` | Sentence-transformers model to use for embeddings |
| `temperature_map` | `Dict[str, float]` | `None` | Override default category-to-temperature mapping |
| `references` | `List[ReferenceExample]` | `None` | Custom labeled reference examples for classification |
| `classifier` | `Callable[[str], str]` | `None` | Custom classifier callable replacing embeddings logic |
| `default_temperature` | `float` | `0.7` | Fallback temperature for unknown categories and `__float__` |
| `override_temperature` | `float` | `None` | If set, bypasses classification and always returns this value |
| `debug` | `bool` | `False` | If `True`, stores debug info on each call |
| `cache_size` | `int` | `128` | Maximum size of the LRU query cache |

### `__call__(messages)`

```python
router(messages: List[Message]) -> float
```

Classifies the query in *messages* and returns the optimal temperature. This is the primary entry point for SDKs that accept callable temperature (e.g., LiteLLM, OpenAI).

### `for_messages(messages)`

```python
router.for_messages(messages: List[Message]) -> float
```

Explicit pre-computation. Returns a plain `float` that any SDK will accept. This is the universal fallback for SDKs that do not support callable temperature.

### `aroute(messages)`

```python
await router.aroute(messages: List[Message]) -> float
```

Async route using `asyncio.to_thread()`. Does not block the event loop under concurrent load. Use this for async SDK patterns.

### `__float__()`

```python
float(router) -> float
```

Returns `default_temperature`. Used as a last-resort fallback when an SDK only accepts numeric values and does not support callables.

### `classify(query_text)`

```python
router.classify(query_text: str) -> str
```

Classifies raw query text into an intent category (e.g., `"factual"`, `"creative"`, `"coding"`). Uses the embedding-based classifier if available, otherwise falls back to a keyword heuristic.

### `get_debug_info()`

```python
router.get_debug_info() -> Optional[DebugInfo]
```

Returns a `DebugInfo` dataclass with the last classification result if `debug=True` was passed to the constructor. Returns `None` otherwise.

`DebugInfo` fields:
- `query_text: str` — The extracted user query
- `detected_category: str` — The classified intent category
- `category_scores: Dict[str, float]` — Average cosine similarity per category
- `chosen_temperature: float` — The temperature that was returned
- `override_active: bool` — Whether an override was active

### `set_override(temperature)`

```python
router.set_override(temperature: Optional[float]) -> None
```

Sets or clears the temperature override. When set, the router bypasses classification and always returns the given temperature. Pass `None` to clear.

## Advanced Customization

### Override Temperature Map

```python
from multi_temp_lm import MultiTempRouter

router = MultiTempRouter(
    temperature_map={
        "factual": 0.1,
        "creative": 1.0,
        "coding": 0.2,
    }
)
```

### Add Custom Categories and References

```python
from multi_temp_lm import MultiTempRouter, ReferenceExample

router = MultiTempRouter(
    references=[
        ReferenceExample("What is the capital of France?", "factual"),
        ReferenceExample("Write a haiku about rain.", "creative"),
        ReferenceExample("Refactor this Java class.", "coding"),
        ReferenceExample("My custom domain query.", "custom_category"),
    ],
    temperature_map={
        "custom_category": 0.5,
    },
)
```

### Use a Custom Classifier

```python
def my_classifier(query: str) -> str:
    if "def " in query or "class " in query:
        return "coding"
    return "default"

router = MultiTempRouter(classifier=my_classifier)
```

When a custom classifier is provided, the embedding-based classifier is completely bypassed.

### Adjust Cache Size

```python
router = MultiTempRouter(cache_size=256)
```

Set `cache_size=0` to disable caching entirely.

### Override Mode

Force a fixed temperature regardless of classification:

```python
router = MultiTempRouter()
router.set_override(0.5)

# Always returns 0.5 until override is cleared
router.set_override(None)
```

### Debug Mode

Enable debug mode to inspect classification decisions:

```python
router = MultiTempRouter(debug=True)

temperature = router.for_messages([{"role": "user", "content": "Write a poem about the ocean."}])

debug = router.get_debug_info()
print(debug.detected_category)   # "creative"
print(debug.chosen_temperature)  # 0.9
print(debug.category_scores)     # {"factual": 0.12, "creative": 0.89, ...}
```

## Performance

Classification latency depends on hardware and the embedding backend:

- **ONNX runtime** (default): typically 5–30 ms per query on a modern CPU
- **PyTorch fallback**: typically 20–80 ms per query on a modern CPU
- **Cache hit**: sub-millisecond for repeated queries

The first call downloads and caches the `all-MiniLM-L6-v2` model (~22 MB with ONNX). Subsequent calls use the cached model.

## Architecture Overview

```
User Code
    |
    | temperature=router
    v
Any LLM SDK (LiteLLM / OpenAI / Anthropic / Google / Cohere / Ollama / ...)
    |
    | router(messages)
    v
MultiTempRouter
    |
    +-- MessageExtractor → query_text
    |
    +-- QueryCache (LRU) → cache hit? return float
    |
    +-- EmbeddingClassifier → embedding + cosine similarity
    |       +-- ReferenceStore (pre-computed reference vectors)
    |
    +-- TemperatureMapper → category → float
    |
    v
float temperature
```

1. **Message Extractor** pulls the user query text from the message list.
2. **Query Cache** checks for an exact string match; if found, returns instantly.
3. **Embedding Classifier** computes a local embedding for the query and compares it against pre-computed reference embeddings via cosine similarity.
4. **Temperature Mapper** maps the best-matching category to a temperature value.
5. The result is cached and returned to the SDK.

All classification is performed locally. No external API calls are made for routing.

## Contributing

Contributions are welcome! Please open an issue or pull request on GitHub.

1. Fork the repository
2. Create a feature branch (`git checkout -b feature/my-feature`)
3. Install dev dependencies: `pip install -e ".[dev]"`
4. Run tests: `pytest`
5. Run linting: `ruff check src tests` and `mypy src`
6. Submit a pull request

## License

This project is licensed under the MIT License. See [LICENSE](LICENSE) for details.
