Metadata-Version: 2.5
Name: llm-queue
Version: 0.1.0b1
Summary: LLM calls through a queue on Valkey / Redis: RPM and TPM budgets per account and region, balancing, fallback, health and status.
Project-URL: Homepage, https://github.com/techiewonk/llm-queue
Project-URL: Documentation, https://techiewonk.github.io/llm-queue/
Project-URL: Source, https://github.com/techiewonk/llm-queue
Project-URL: Issues, https://github.com/techiewonk/llm-queue/issues
Project-URL: Changelog, https://github.com/techiewonk/llm-queue/blob/main/CHANGELOG.md
Author: The llm-queue authors
License-Expression: MIT
License-File: LICENSE
License-File: NOTICE
Keywords: bedrock,langchain,langgraph,llm,openai,queue,rate-limit,redis,valkey
Classifier: Development Status :: 4 - Beta
Classifier: Framework :: AsyncIO
Classifier: Intended Audience :: Developers
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Topic :: System :: Distributed Computing
Classifier: Typing :: Typed
Requires-Python: >=3.11
Requires-Dist: aiobotocore<4,>=2.23
Requires-Dist: hiredis<4,>=3
Requires-Dist: orjson<4,>=3.10
Requires-Dist: pydantic<3,>=2.10
Requires-Dist: redis<9,>=5.0.1
Requires-Dist: structlog<27,>=24.1
Requires-Dist: uuid-utils<1,>=0.19
Provides-Extra: http
Requires-Dist: fastapi<1,>=0.115.3; extra == 'http'
Provides-Extra: langchain-aws
Requires-Dist: langchain-aws<2,>=0.2.24; extra == 'langchain-aws'
Requires-Dist: langchain-core<2,>=0.3.68; extra == 'langchain-aws'
Provides-Extra: langchain-openai
Requires-Dist: langchain-core<2,>=0.3.68; extra == 'langchain-openai'
Requires-Dist: langchain-openai<2,>=0.3.26; extra == 'langchain-openai'
Provides-Extra: metrics
Requires-Dist: prometheus-client<1,>=0.20; extra == 'metrics'
Provides-Extra: openai
Requires-Dist: httpx<1,>=0.27; extra == 'openai'
Requires-Dist: openai<4,>=1.40; extra == 'openai'
Description-Content-Type: text/markdown

# llm-queue

LLM calls through a queue on Valkey or Redis, under RPM / TPM budgets per **deployment**
(model x account x region), balanced across regions and accounts, with fallback regions,
fallback models, health checks and status. Works with Amazon Bedrock and OpenAI-compatible
servers (vLLM), and with LangChain. The queue, the budgets and the results live in one store, so
admission (deployment, RPM, TPM, concurrency slot) is one atomic step, and 1 pod behaves the
same as 50.

## Install

```
pip install "llm-queue[langchain-aws]"            # Bedrock + LangChain
pip install "llm-queue[openai,langchain-openai]"  # OpenAI-compatible
pip install "llm-queue[http]"                     # status and health routes for FastAPI
```

## Example

```python
from llm_queue import LLMQueue, CallFailed, request_context

llmq = LLMQueue.from_env()               # LLMQ_STORE_URL, LLMQ_REGISTRY, ...
await llmq.start()                       # publish / load the registry, run the workers
try:
    with request_context(priority=True, user_id=uid):
        reply = await llmq.converse("gpt120b", {"messages": [...]}, max_wait_secs=5)
    print(reply.response, reply.meta.deployment)
except CallFailed as exc:                # exc.kind: rate_limited, model_unavailable, ...
    ...
await llmq.stop()
```

## Documentation

Full documentation: **https://techiewonk.github.io/llm-queue/**

- [Getting started](https://techiewonk.github.io/llm-queue/tutorials/getting-started/): install, local stack, a registry, the first call
- [How-to guides](https://techiewonk.github.io/llm-queue/how-to/configure/): configure, tune, call, operate, share models across services, test
- [Explanation](https://techiewonk.github.io/llm-queue/explanation/architecture/): architecture, the life of a call, admission, failures and recovery
- [Reference](https://techiewonk.github.io/llm-queue/reference/configuration/): settings, error kinds, CLI and HTTP, the Python API

Project: [CHANGELOG](https://github.com/techiewonk/llm-queue/blob/main/CHANGELOG.md),
[CONTRIBUTING](https://github.com/techiewonk/llm-queue/blob/main/CONTRIBUTING.md),
[STABILITY](https://github.com/techiewonk/llm-queue/blob/main/STABILITY.md),
[SECURITY](https://github.com/techiewonk/llm-queue/blob/main/SECURITY.md).
