Metadata-Version: 2.4
Name: open-infer
Version: 0.1.0
Summary: A developer-friendly, extensible open-source runtime for local LLM inference.
Keywords: llm,inference,transformers,local-inference,machine-learning
Author: Avinash Mahadev Yerolkar
Author-email: Avinash Mahadev Yerolkar <avinashmyerolkar@gmail.com>
License-Expression: Apache-2.0
License-File: LICENSE
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Operating System :: OS Independent
Requires-Dist: torch>=2.13.0
Requires-Dist: transformers>=5.14.1
Requires-Dist: fastapi>=0.115 ; extra == 'api'
Requires-Dist: uvicorn>=0.32 ; extra == 'api'
Requires-Dist: streamlit>=1.40 ; extra == 'ui'
Requires-Dist: requests>=2.32 ; extra == 'ui'
Requires-Python: >=3.10
Project-URL: Homepage, https://github.com/avinashmyerolkar/open-infer
Project-URL: Repository, https://github.com/avinashmyerolkar/open-infer
Project-URL: Issues, https://github.com/avinashmyerolkar/open-infer/issues
Provides-Extra: api
Provides-Extra: ui
Description-Content-Type: text/markdown

# OpenInfer

A developer-friendly, extensible open-source runtime for local LLM inference.

OpenInfer sits between low-level libraries like Hugging Face Transformers (where you configure everything by hand) and heavyweight production inference engines like vLLM or TensorRT-LLM (powerful but complex). It gives you a simple API for local inference while handling model loading, device detection, and configuration internally.

> Status: early / alpha (`v0.1`). Package name: `open-infer` (PyPI) / `open_infer` (import).

## Installation

```bash
pip install open-infer                 # library only
pip install open-infer[api]            # + REST API server
pip install open-infer[ui]             # + browser playground UI
pip install open-infer[api,ui]         # everything
```

## Usage

### As a library

```python
from open_infer import InferenceEngine

engine = InferenceEngine("Qwen/Qwen2.5-1.5B-Instruct")
response = engine.generate("Explain KV Cache.")
```

Device (`cuda` / `mps` / `cpu`) is auto-detected. Models load in float16 on GPU/MPS, float32 on CPU.

### As a REST API

```bash
open-infer-api
```

Starts a server at `http://127.0.0.1:8000` (`OPEN_INFER_API_HOST` / `OPEN_INFER_API_PORT` to override). The model loaded at startup comes from `OPEN_INFER_MODEL` (default: `Qwen/Qwen2.5-1.5B-Instruct`).

| Endpoint | Description |
|---|---|
| `GET /health` | Liveness check |
| `GET /model` | Currently loaded model |
| `POST /model` | Switch model — `{"model": "<huggingface-id>"}` |
| `POST /generate` | Generate text — `{"prompt": "...", "max_new_tokens": 200}` |

Interactive docs at `http://127.0.0.1:8000/docs`.

### As the Playground UI

```bash
open-infer-api    # in one terminal — the UI talks to the API, it doesn't replace it
open-infer-ui      # in another terminal
```

A browser-based chat UI (`OPEN_INFER_API_URL` to point it at a different API). Pick a model from the sidebar's curated presets (all ≤4GB in float16) or paste a custom Hugging Face model ID.

## Features

- Automatic hardware detection (CUDA / Apple Silicon MPS / CPU)
- One-line model loading via `InferenceEngine`
- Repetition-safe generation (`repetition_penalty`, `no_repeat_ngram_size`)
- REST API with live model switching, no restart needed
- Browser playground UI with curated, memory-safe model presets

## Roadmap

OpenInfer is early — current focus is a stable core (`InferenceEngine`, REST API, UI). Planned next: streaming responses, KV cache, continuous batching, quantization, and additional backends (GGUF, ONNX, MLX) beyond the current Transformers backend.

## License

Apache 2.0 — see [LICENSE](LICENSE).
