Metadata-Version: 2.4
Name: llmbroker
Version: 1.1.0
Project-URL: Homepage, https://andgineer.github.io/llmbroker/
Project-URL: Documentation, https://andgineer.github.io/llmbroker/
Author-email: Andrey Sorokin <andrey@sorokin.engineer>
License: Copyright (c) 2026 Andrey Sorokin (andrey@sorokin.engineer)
        
        Permission is hereby granted, free of charge, to any person obtaining a copy of
        this software and associated documentation files (the “Software”), to deal in
        the Software without restriction, including without limitation the rights to use,
        copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the
        Software, and to permit persons to whom the Software is furnished to do so,
        subject to the following conditions:
        
        The above copyright notice and this permission notice shall be included in all
        copies or substantial portions of the Software.
        
        THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND,
        EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES
        OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
        NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT
        HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
        WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
        FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
        OTHER DEALINGS IN THE SOFTWARE.
License-File: LICENSE
Keywords: one,two
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Requires-Python: >=3.11
Requires-Dist: httpx>=0.28.1
Provides-Extra: aws
Requires-Dist: aioboto3>=13.0; extra == 'aws'
Provides-Extra: mongodb
Requires-Dist: motor>=3.6.0; extra == 'mongodb'
Provides-Extra: postgres
Requires-Dist: asyncpg>=0.30.0; extra == 'postgres'
Provides-Extra: redis
Requires-Dist: redis>=5.0; extra == 'redis'
Provides-Extra: sqlite
Requires-Dist: aiosqlite>=0.22.1; extra == 'sqlite'
Provides-Extra: vault
Requires-Dist: hvac>=2.0; extra == 'vault'
Description-Content-Type: text/markdown

[![Build Status](https://github.com/andgineer/llmbroker/workflows/CI/badge.svg)](https://github.com/andgineer/llmbroker/actions)
[![Coverage](https://raw.githubusercontent.com/andgineer/llmbroker/python-coverage-comment-action-data/badge.svg)](https://htmlpreview.github.io/?https://github.com/andgineer/llmbroker/blob/python-coverage-comment-action-data/htmlcov/index.html)
# llmbroker

Turn a crowd of free, rate-limited LLMs into one reliable model — no premium subscription, no single point of failure.

Many free LLMs are unreliable and mediocre on their own. llmbroker pools them and routes
across the pool, turning quantity into dependable, good-enough quality — without paying
for a premium model.

No LangChain, no heavy deps.

```python
llms = llmbroker.Broker("llms.toml")
print(llms.ask("Explain Python decorators in one sentence").text)
```

Put your LLMs in `llms.toml` — or grab a preset:

```bash
llmbroker preset freetier > llms.toml
```

The only setup is the API keys. Ask llmbroker which ones your pool needs — each with a note on
where to get it, ordered easiest-and-most-valuable first:

```bash
llmbroker env llms.toml > .env
```

Running with only some of the keys is the normal, intended mode — add the keys that are easy
and worth it, and the pool assembles itself from whatever is present. A missing key just means
that one model stays inactive; it is not an error.

The same `Broker` scales straight to a server or cluster — add Redis or Postgres to share cooldown state across instances, the calling code stays the same.

**Why llmbroker:**

- **Automatic failover** — when one LLM is rate-limited or down, the next one in
  the pool answers instead; you get a reply, not an error, as long as any LLM is up.
- **Chat, tools & agents** — one-shot `ask`, multi-turn `chat`, and function/tool
  calling for agentic workflows.
- **Async-first** — `llmbroker.AsyncBroker` built on asyncio for FastAPI, agents, and async workers;
  `llmbroker.Broker` wraps the same engine in a blocking API for plain scripts.
- **Pluggable backends** — swap registry, state store, and telemetry independently: DB, Redis etc.
- **API keys configurable storage** — environment variables, DB, AWS, Vault, your own store.
- **Multi-user mode** -- optionally separate secrets and LLMs list for each application user.
- **Self-regulating pool** — rate a reply's quality and llmbroker learns, per kind of
  task, which models are actually worth routing to; a consistently weak model quietly
  drops to the back of the queue instead of being nagged about by a human.

# Documentation

[llmbroker](https://andgineer.github.io/llmbroker/)

# Developers

Do not forget to run `. ./activate.sh`.

For work it need [uv](https://github.com/astral-sh/uv) installed.

Use [pre-commit](https://pre-commit.com/#install) hooks for code quality:

    pre-commit install

## Allure test report

* [Allure report](https://andgineer.github.io/llmbroker/builds/tests/)

# Scripts
Install [invoke](https://docs.pyinvoke.org/en/stable/) preferably with [uv tool](https://docs.astral.sh/uv/):

    uv tool install invoke

For a list of available scripts run:

    invoke --list

For more information about a script run:

    invoke <script> --help

The bundled `freetier` preset drifts as providers change their free tiers; refresh
it with `invoke catalog-refresh`, which prints the maintenance runbook
(`presets/freetier-refresh-prompt.md`).

## Coverage report
* [Codecov](https://app.codecov.io/gh/andgineer/llmbroker/tree/main/src%2Fllmbroker)
* [Coveralls](https://coveralls.io/github/andgineer/llmbroker)

> Created with cookiecutter using [template](https://github.com/andgineer/cookiecutter-python-package)
