Metadata-Version: 2.4
Name: npuserver
Version: 1.1.1
Summary: A high-performance Flask server tailored for running Large Language Models (LLMs) locally on Intel NPUs using OpenVINO GenAI.
License: MIT
Keywords: intel-npu,openvino,openvino-genai,llm,flask,openai-compatible,local-inference,ai
Author: Durga Sai
Author-email: dsainvg.20.12@gmail.com
Requires-Python: >=3.12
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Requires-Dist: flask (>=3.0.0)
Requires-Dist: huggingface-hub (>=0.20.0)
Requires-Dist: openvino-genai (>=2024.5.0)
Project-URL: Bug Tracker, https://github.com/dsainvg/npuserver/issues
Project-URL: Homepage, https://github.com/dsainvg/npuserver
Project-URL: Repository, https://github.com/dsainvg/npuserver
Description-Content-Type: text/markdown

# NPU Server (`npuserver`)

A high-performance Python library and Flask backend tailored specifically for running Large Language Models (LLMs) locally on **Intel NPUs** using OpenVINO GenAI.

This server provides an OpenAI-compatible API for seamless integration with existing tools, robust NPU memory management, and dynamic on-the-fly hardware compilation of Hugging Face models into optimized NPU blobs.

## Core Features

- 🚀 **OpenAI-Compatible API**: Seamlessly integrate with any existing LLM tooling (like LangChain, AutoGen, or custom frontends) using the standard `/v1/chat/completions` endpoint. Fully supports real-time Server-Sent Event (SSE) streaming.
- 🧠 **Strict Memory Management**: The Intel NPU has limited, highly specialized memory. This server gives you complete explicit control over it. Load and unload models programmatically while aggressively garbage-collecting to prevent NPU memory leaks. 
- ⚡ **On-The-Fly Compilation**: If a downloaded Hugging Face model hasn't been compiled for the NPU, the server intelligently intercepts the load request and dynamically compiles an optimized OpenVINO `.blob` before serving it.
- 🚫 **No Background Downloads**: To prevent runaway bandwidth usage and unexpected latency, the server strictly enforces that models must be downloaded locally *before* it attempts to load or compile them.
- 🌐 **Static Model Registry**: Ships with a fully decoupled, static HTML dashboard (`serve/index.html`) designed for easy hosting on GitHub Pages to cleanly display your available models without exposing backend connections.

---

## 📦 Installation

Ensure you have Python installed and your Intel NPU drivers configured properly on Windows.

```bash
# 1. Clone the repository
git clone https://github.com/durgasai299792458/npuserver.git
cd npuserver

# 2. Setup a virtual environment
python -m venv venv
venv\Scripts\activate

# 3. Install the package locally
pip install -e .
```

*Required Core Dependencies:* `openvino-genai`, `flask`, `huggingface-hub`

---

## 🛠️ Usage Guide

### 1. Starting the Server

The server runs on Flask. You can spin it up programmatically using the library:

```python
from npuserver import run_server

# Starts the NPU backend on port 8080
run_server(port=8080)
```

### 2. Using the Python Client Library

You can remotely control the server's NPU memory directly from your Python scripts using the built-in client functions, without needing to write raw HTTP requests manually.

```python
import npuserver

# 1. Fetch a list of all available models
models = npuserver.get_models_status(api_base_url="http://localhost:8080")
for m in models:
    print(f"{m['name']} is {m['status']}")

# 2. Explicitly load a model into NPU memory
print("Loading model...")
npuserver.load_model("durgasai299792458/Qwen3-4B-OpenVINO-INT4-npu-i")

# 3. Completely wipe the NPU memory and free resources
npuserver.unload_model()
```

### 3. Downloading Models Manually

Because the server strictly refuses to download gigabytes of data in the background, you must ensure the model exists in the cache first. You can download compatible models (like `Qwen2.5-3B-OpenVINO-INT4-npu` or `gemma-4-E2B-OpenVINO-INT4`) directly using Hugging Face utilities, or the server will read them if they are cached natively.

---

## 📡 HTTP API Reference

If you are building your own frontend or using standard REST clients (like `curl` or Postman), use these endpoints:

### 🧠 Model Memory Management

#### `POST /load`
Loads a specific model into NPU memory. If another model is currently active, it is safely ejected and garbage collected first. If the requested model is downloaded but uncompiled, it halts to compile it first.
```bash
curl -X POST http://localhost:8080/load \
     -H "Content-Type: application/json" \
     -d '{"model": "durgasai299792458/Qwen3-4B-OpenVINO-INT4-npu-i"}'
```

#### `POST /unload`
Explicitly unloads the currently active model and triggers Python garbage collection to flush the NPU VRAM immediately.
```bash
curl -X POST http://localhost:8080/unload
```

#### `GET /models`
Returns a consolidated JSON array of all models (active, compiled, and available remotely on your GitHub Pages registry).

#### `GET /health`
Returns the server's status (e.g., whether it is idle or currently processing a generation task).

### 💬 Inference

#### `POST /v1/chat/completions`
A standard OpenAI chat endpoint. 
**Strict Execution Note:** You MUST call `/load` before sending a chat completion request. The server will reject the request with a `400 Bad Request` if the NPU memory is empty.

```bash
curl -X POST http://localhost:8080/v1/chat/completions \
     -H "Content-Type: application/json" \
     -d '{
       "messages": [
         {"role": "system", "content": "You are a helpful AI assistant running on an Intel NPU."},
         {"role": "user", "content": "Write a short poem about microprocessors."}
       ],
       "max_tokens": 2048,
       "temperature": 0.7,
       "stream": true
     }'
```

---

## 📂 Cache & File Structure

By default, `npuserver` neatly organizes heavy model files in your user cache directory so your code repository stays clean and lightweight.

On Windows, models are stored at: `C:\Users\<username>\.cache\npuserver\`

- `...\hf\`: Stores the raw weights and safetensors downloaded directly from the Hugging Face hub.
- `...\compiled\`: Stores the optimized `.blob` files successfully compiled by OpenVINO GenAI.

**Troubleshooting Tip:** If a hardware compilation ever fails, gets interrupted, or becomes corrupted, simply navigate to the `compiled\` directory, delete the specific model's folder, and the server will safely attempt to re-compile it from scratch on your next `/load` request!

