Metadata-Version: 2.4
Name: tinfer-ai
Version: 0.1.0
Summary: Tinfer — Tiny Inference Engine. Run LLMs locally with GPU acceleration.
Author-email: Sunil <sunil@tinfer.dev>
License: MIT
Project-URL: Homepage, https://github.com/sunil/tinfer
Keywords: llm,inference,ai,gpu,cuda,gguf,local
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: Microsoft :: Windows
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.8
Description-Content-Type: text/markdown
Requires-Dist: requests>=2.28.0

# Tinfer — Tiny Inference Engine

Run LLMs locally with GPU acceleration. No dependencies, no cloud, no API keys.

## Install

```bash
pip install tinfer
```

## Quick Start

### CLI Chat
```bash
tinfer -m model.gguf -p "Hello, what is AI?"
```

### Server + WebUI
```bash
tinfer-server -m model.gguf --port 8080
# Open http://localhost:8080 for the chat interface
```

### Python API
```python
from tinfer import Server, chat

# Start server programmatically
with Server("model.gguf", port=8080, n_gpu_layers=-1) as s:
    response = chat("What is artificial intelligence?")
    print(response)
```

### HTTP API (OpenAI-compatible)
```bash
curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"messages": [{"role": "user", "content": "Hello"}], "max_tokens": 100}'
```

## Features

- **CLI** — Interactive chat and text completion
- **Server** — HTTP server with built-in WebUI
- **API** — OpenAI-compatible `/v1/chat/completions` endpoint
- **GPU** — CUDA acceleration out of the box
- **Quantized** — Run GGUF models (Q4, Q5, Q8) efficiently
- **MoE** — Mixture of Experts model support
- **Vision** — Multimodal model support

## Python Client

```python
from tinfer import chat, complete, models, set_server

# Point to a running server
set_server("http://localhost:8080")

# Simple chat
print(chat("Explain quantum computing in 3 sentences"))

# Chat with system prompt
print(chat([
    {"role": "system", "content": "You are a pirate."},
    {"role": "user", "content": "What's the weather?"}
]))

# List models
for m in models():
    print(m["id"])
```
