Metadata-Version: 2.4
Name: omni-local-llm
Version: 0.1.0
Summary: A unified Python interface and API server for running local Large Language Models.
Project-URL: Homepage, https://github.com/matteoespo/omni-local-llm
Project-URL: Repository, https://github.com/matteoespo/omni-local-llm
Project-URL: Issues, https://github.com/matteoespo/omni-local-llm/issues
Author-email: matteoespo <matteoespo@users.noreply.github.com>
License: MIT
License-File: LICENSE
Keywords: ai,chatbot,inference,llama-cpp,llm,local-llm,machine-learning,ollama,openai
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Typing :: Typed
Requires-Python: >=3.12
Requires-Dist: fastapi>=0.111.0
Requires-Dist: huggingface-hub>=1.15.0
Requires-Dist: llama-cpp-python>=0.3.23
Requires-Dist: ollama>=0.6.2
Requires-Dist: uvicorn>=0.30.1
Description-Content-Type: text/markdown

<p align="center">
  <img src="https://img.shields.io/badge/🧠-Omni--Local--LLM-blueviolet?style=for-the-badge&logoColor=white" alt="Omni-Local-LLM" height="50"/>
</p>

<h3 align="center">One interface. Every local LLM.</h3>

<p align="center">
  <em>A unified Python SDK and OpenAI-compatible API server for local large language models.</em>
</p>

<p align="center">
  <a href="https://github.com/matteoespo/omni-local-llm/actions/workflows/ci.yml"><img src="https://github.com/matteoespo/omni-local-llm/actions/workflows/ci.yml/badge.svg" alt="CI"></a>
  <a href="https://github.com/matteoespo/omni-local-llm/actions/workflows/codeql.yml"><img src="https://github.com/matteoespo/omni-local-llm/actions/workflows/codeql.yml/badge.svg" alt="CodeQL"></a>
  <a href="https://pypi.org/project/omni-local-llm/"><img src="https://img.shields.io/pypi/v/omni-local-llm?color=blue&label=PyPI" alt="PyPI"></a>
  <a href="https://pypi.org/project/omni-local-llm/"><img src="https://img.shields.io/pypi/pyversions/omni-local-llm" alt="Python"></a>
  <a href="https://github.com/matteoespo/omni-local-llm/blob/main/LICENSE"><img src="https://img.shields.io/github/license/matteoespo/omni-local-llm?color=green" alt="License"></a>
  <a href="https://github.com/matteoespo/omni-local-llm/stargazers"><img src="https://img.shields.io/github/stars/matteoespo/omni-local-llm?style=social" alt="Stars"></a>
</p>

<p align="center">
  <a href="#-quick-start">Quick Start</a> •
  <a href="#-features">Features</a> •
  <a href="#-architecture">Architecture</a> •
  <a href="#-installation">Installation</a> •
  <a href="#-usage">Usage</a> •
  <a href="#-api-server">API Server</a> •
  <a href="#-contributing">Contributing</a>
</p>

---

## Features

| Feature | Description |
|---|---|
| **Multi-Backend** | Swap between [Ollama](https://ollama.com/) and [llama.cpp](https://github.com/ggerganov/llama.cpp) with a single parameter change |
| **Chat Sessions** | Built-in conversation memory and history management |
| **Streaming** | Real-time token-by-token streaming (sync & async) |
| **Tool Calling** | Native function/tool calling support across backends |
| **JSON Mode** | Enforce structured JSON output from any supported model |
| **OpenAI-Compatible API** | Drop-in FastAPI server compatible with the OpenAI SDK |
| **Async-First** | Full `async`/`await` support for high-concurrency workloads |
| **Auto Model Pull** | Automatically downloads models from Ollama or Hugging Face Hub |

---

## Quick Start

```bash
# Install
pip install omni-local-llm

# Or with uv (recommended)
uv pip install omni-local-llm
```

```python
from omnillm import LocalLLMManager

manager = LocalLLMManager()
response = manager.chat(
    backend="ollama",
    model="llama3",
    messages=[{"role": "user", "content": "Hello!"}]
)
print(response)
```

**That's it.** Switch to llama.cpp by changing one word:

```python
response = manager.chat(
    backend="llama.cpp",
    model="unsloth/llama-3-8b-Instruct-GGUF",
    messages=[{"role": "user", "content": "Hello!"}],
    filename="llama-3-8b-Instruct-Q4_K_M.gguf"
)
```

---

## Architecture

```
┌─────────────────────────────────────────────────────────┐
│                      Your Application                   │
├─────────────┬─────────────────────┬─────────────────────┤
│   Python    │   Interactive CLI   │  OpenAI-Compatible  │
│     SDK     │  (python -m omnillm)│   FastAPI Server    │
├─────────────┴─────────────────────┴─────────────────────┤
│                    ChatSession                          │
│              (History · Streaming · Tools)              │
├─────────────────────────────────────────────────────────┤
│                  LocalLLMManager                        │
│              (Backend Registry & Router)                │
├────────────────────────┬────────────────────────────────┤
│    OllamaAdapter       │       LlamaCPPAdapter          │
│  (ollama Python SDK)   │  (llama-cpp-python + HF Hub)  │
└────────────────────────┴────────────────────────────────┘
```

### Module Map

| Module | Purpose |
|---|---|
| [`omnillm/core/base.py`](omnillm/core/base.py) | `LLMBackend` — abstract base class for all adapters |
| [`omnillm/core/manager.py`](omnillm/core/manager.py) | `LocalLLMManager` — backend registry & request router |
| [`omnillm/core/session.py`](omnillm/core/session.py) | `ChatSession` — conversation state, streaming, tool calls |
| [`omnillm/adapters/ollama_adapter.py`](omnillm/adapters/ollama_adapter.py) | Ollama integration via official Python client |
| [`omnillm/adapters/llamacpp_adapter.py`](omnillm/adapters/llamacpp_adapter.py) | llama.cpp integration + automatic GGUF model caching |
| [`omnillm/server.py`](omnillm/server.py) | OpenAI-compatible FastAPI HTTP server |
| [`omnillm/__main__.py`](omnillm/__main__.py) | Interactive CLI chat interface |

---

## Installation

### Prerequisites

- **Python 3.12+**
- **[uv](https://github.com/astral-sh/uv)** (recommended) or pip
- At least one backend:
  - [Ollama](https://ollama.com/) installed and running, **or**
  - A GGUF model on [Hugging Face Hub](https://huggingface.co/) (auto-downloaded)

### Install from source

```bash
git clone https://github.com/matteoespo/omni-local-llm.git
cd omni-local-llm
uv pip install -e .
```

### Install from PyPI

```bash
pip install omni-local-llm
# or
uv pip install omni-local-llm
```

---

## Usage

### Python SDK

#### Basic Chat

```python
from omnillm import LocalLLMManager

manager = LocalLLMManager()

response = manager.chat(
    backend="ollama",
    model="llama3",
    messages=[{"role": "user", "content": "Hello!"}]
)
print(response)
```

#### Chat Session (with Memory)

```python
session = manager.create_session(
    backend="ollama",
    model="llama3",
    system_prompt="You are a helpful assistant."
)

response1 = session.send("Hello, I am Bob.")
response2 = session.send("What is my name?")  # Remembers "Bob"
```

#### Async & Streaming

```python
import asyncio
from omnillm import LocalLLMManager

async def main():
    manager = LocalLLMManager()
    session = manager.create_session(backend="ollama", model="llama3")

    stream = await session.asend("Tell me a short story.", stream=True)
    async for chunk in stream:
        print(chunk, end="", flush=True)

asyncio.run(main())
```

#### Tool Calling

```python
tools = [{
    "type": "function",
    "function": {
        "name": "get_weather",
        "description": "Get current weather for a city",
        "parameters": {
            "type": "object",
            "properties": {
                "city": {"type": "string", "description": "City name"}
            },
            "required": ["city"]
        }
    }
}]

response = manager.chat(
    backend="ollama",
    model="llama3",
    messages=[{"role": "user", "content": "What's the weather in Tokyo?"}],
    tools=tools
)
# response["tool_calls"] contains the function call
```

#### JSON Mode

```python
response = manager.chat(
    backend="ollama",
    model="llama3",
    messages=[{"role": "user", "content": "List 3 colors as JSON"}],
    json_mode=True
)
# Returns valid JSON string
```

---

### Interactive CLI

Start an interactive chat session directly in your terminal:

```bash
# Using Ollama
python -m omnillm --backend ollama --model llama3

# Using llama.cpp (auto-downloads from Hugging Face)
python -m omnillm --backend llama.cpp \
    --model unsloth/llama-3-8b-Instruct-GGUF \
    --filename llama-3-8b-Instruct-Q4_K_M.gguf

# With a system prompt
python -m omnillm --backend ollama --model llama3 \
    --system "You are a pirate. Respond only in pirate speak."
```

---

### API Server

Launch an OpenAI-compatible local API server:

```bash
python -m omnillm.server
# Server runs on http://localhost:8000
```

#### Use with the OpenAI SDK

```python
import openai

client = openai.OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="not-needed"
)

response = client.chat.completions.create(
    model="ollama/llama3",          # prefix with backend name
    messages=[{"role": "user", "content": "Explain relativity in one sentence."}]
)
print(response.choices[0].message.content)
```

#### Use with curl

```bash
curl -X POST http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "ollama/llama3",
    "messages": [{"role": "user", "content": "Hello!"}]
  }'
```

---

## Running Tests

```bash
# With uv (recommended)
uv run pytest

# Or activate the venv first
source .venv/bin/activate
pytest

# With coverage
uv run pytest --cov=omnillm --cov-report=term-missing
```

---

## Contributing

Contributions are welcome! Please see the [Contributing Guide](CONTRIBUTING.md) for details on:

- Setting up your development environment
- Adding new LLM backend adapters
- Submitting pull requests

---

## License

This project is licensed under the **MIT License** — see the [LICENSE](LICENSE) file for details.

---

## Roadmap

- [ ] vLLM backend adapter
- [ ] KoboldCpp backend adapter
- [ ] Token usage tracking and reporting
- [ ] Vision/multimodal support
- [ ] Model hot-swapping in sessions
- [ ] Prompt template management
- [ ] Docker image for the API server

---

<p align="center">
  Made with ❤️ by <a href="https://github.com/matteoespo">matteoespo</a>
</p>
