Metadata-Version: 2.1
Name: siliconscavenger
Version: 0.7.1
Summary: Heterogeneous inference scheduler for CPU + Intel iGPU + NVIDIA dGPU - Ollama-compatible API, CLI, and Python SDK
Keywords: llm,inference,scheduler,ollama,llama,gpu,heterogeneous,cpu,igpu,dgpu,cuda,openvino,vulkan,local-ai,fastapi
Author: Jai Ansh Singh Bindra
Author-Email: Anubhav Choudhery <anubhavchoudhery@gmail.com>
Maintainer-Email: JBAC EdTech <anubhavchoudhery@gmail.com>
License: MIT License
         
         Copyright (c) 2026 JBAC EdTech — Anubhav Choudhery & Jai Ansh Singh Bindra
         
         Permission is hereby granted, free of charge, to any person obtaining a copy
         of this software and associated documentation files (the "Software"), to deal
         in the Software without restriction, including without limitation the rights
         to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
         copies of the Software, and to permit persons to whom the Software is
         furnished to do so, subject to the following conditions:
         
         The above copyright notice and this permission notice shall be included in all
         copies or substantial portions of the Software.
         
         THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
         IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
         FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
         AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
         LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
         OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
         SOFTWARE.
         
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: Microsoft :: Windows
Classifier: Operating System :: POSIX :: Linux
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: C++
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: System :: Hardware
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Project-URL: Documentation, https://pypi.org/project/siliconscavenger/
Project-URL: Changelog, https://pypi.org/project/siliconscavenger/#history
Requires-Python: >=3.9
Provides-Extra: server
Requires-Dist: fastapi>=0.110; extra == "server"
Requires-Dist: uvicorn[standard]>=0.29; extra == "server"
Requires-Dist: numpy>=1.24; extra == "server"
Requires-Dist: sse-starlette>=1.6; extra == "server"
Requires-Dist: httpx>=0.27; extra == "server"
Requires-Dist: python-multipart>=0.0.9; extra == "server"
Provides-Extra: audio
Requires-Dist: optimum[openvino]>=1.19; extra == "audio"
Requires-Dist: soundfile>=0.12; extra == "audio"
Requires-Dist: scipy>=1.11; extra == "audio"
Provides-Extra: dev
Requires-Dist: pytest>=7; extra == "dev"
Requires-Dist: ruff>=0.4; extra == "dev"
Requires-Dist: numpy>=1.24; extra == "dev"
Requires-Dist: httpx>=0.27; extra == "dev"
Requires-Dist: fastapi>=0.110; extra == "dev"
Requires-Dist: uvicorn[standard]>=0.29; extra == "dev"
Requires-Dist: sse-starlette>=1.6; extra == "dev"
Requires-Dist: pytest-asyncio>=0.23; extra == "dev"
Requires-Dist: soundfile>=0.12; extra == "dev"
Requires-Dist: scipy>=1.11; extra == "dev"
Requires-Dist: matplotlib>=3.8; extra == "dev"
Requires-Dist: python-multipart>=0.0.9; extra == "dev"
Provides-Extra: benchmarks
Requires-Dist: httpx>=0.27; extra == "benchmarks"
Requires-Dist: matplotlib>=3.8; extra == "benchmarks"
Provides-Extra: all
Requires-Dist: siliconscavenger[audio,benchmarks,server]; extra == "all"
Description-Content-Type: text/markdown

# SiliconScavenger

[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)
[![Python](https://img.shields.io/badge/python-3.9%2B-blue.svg)](https://pypi.org/project/siliconscavenger/)
[![PyPI](https://img.shields.io/pypi/v/siliconscavenger.svg)](https://pypi.org/project/siliconscavenger/)

**A heterogeneous inference scheduler for CPU + Intel iGPU + NVIDIA dGPU — Ollama-compatible REST API, Python SDK, and CLI.**

Developed by **JBAC EdTech** (Anubhav Choudhery & Jai Ansh Singh Bindra).

---

## What it does

SiliconScavenger routes LLM inference requests across all three compute devices on your machine — CPU, Intel integrated GPU, and NVIDIA dGPU — **concurrently**, with:

- A **lock-free C++ scheduler** (Python never sits in the per-token hot path)
- An **Ollama-compatible REST API** (`/api/generate`, `/api/chat`, `/api/tags`, `/api/embeddings`, …)
- An **OpenAI-compatible REST API** (`/v1/chat/completions`, `/v1/embeddings`, …)
- **Tool calling** and **JSON structured output** on both API surfaces
- **Session affinity** — multi-turn chats pin to the same device for KV-cache reuse
- A **live web dashboard** showing per-device utilisation, queue depth, and session count
- A **CLI** mirroring Ollama's UX (`scavenger serve`, `scavenger pull`, `scavenger create`, …)

### Measured performance (live run, Aug 2026)

Against an unmodified Ollama 0.30.10 install, same machine, same model (`qwen2.5:3b` Q4_K_M), 24-request concurrent trace:

| System | TTFT p50 | TTFT p95 |
|---|---|---|
| **SiliconScavenger CPU+iGPU** | **2 538 ms** | 46 388 ms |
| Ollama stock (CPU) | 63 047 ms | 89 219 ms |
| Ollama iGPU-enabled | 71 203 ms | 88 156 ms |

**24.8× faster p50 TTFT** vs Ollama stock. iGPU was at 95–98 % utilisation throughout the Scavenger run — GPU-bound, not CPU-bound.

3-device simulation (CPU + iGPU + NVIDIA RTX 3060): **+94.6 % p95 TTFT improvement** when dGPU absorbs 56 % of requests.

---

## Hardware requirements

| Device | Backend | Required |
|---|---|---|
| x86 CPU | OpenVINO CPU | Always (minimum) |
| Intel iGPU (12th Gen Core+) | OpenVINO GPU + llama.cpp Vulkan | Recommended |
| NVIDIA dGPU (Ampere/Ada) | llama.cpp CUDA (sm_86+) | Optional |

Minimum tested: Windows 11, Python 3.12, Intel Core i7-12650H.  
Full dGPU support validated on NVIDIA RTX 3060 Laptop GPU (CUDA 12.6, driver 581.95).

---

## Installation

### Quickstart — server mode

```bash
pip install "siliconscavenger[server]"
scavenger serve
# Open http://127.0.0.1:11434/dashboard/
```

### With audio/ASR support

```bash
pip install "siliconscavenger[server,audio]"
```

### All extras

```bash
pip install "siliconscavenger[all]"
```

> **Note on the C++ extension**: `pip install` triggers a CMake build of the native
> `_scavenger_core` pybind11 extension. You need:
> - Visual Studio 2022 Build Tools (Windows) **or** GCC 12+ / Clang 15+ (Linux)
> - CMake ≥ 3.26
> - For iGPU: OpenVINO 2024.x runtime
> - For dGPU: CUDA Toolkit 12.x + nvcc in PATH

---

## CLI reference

All commands are available as `scavenger <subcommand>`:

```
Usage: scavenger {serve,run,pull,list,version,create,import-ollama}
```

### `scavenger serve` — start the inference server

```bash
# Mock echo backend (no model needed, useful for API testing):
scavenger serve

# Real inference with a GGUF model:
scavenger serve --model path/to/qwen2.5-3b-instruct-q4_k_m.gguf

# Custom host/port:
scavenger serve --host 0.0.0.0 --port 8080

# Pre-flight device check (smoke-tests all registered slots, exits 1 if any fail):
scavenger serve --check-devices

# All options:
scavenger serve --help
```

### `scavenger list` — list registered models

```bash
scavenger list
# NAME                SIZE    MODIFIED
# qwen2.5:3b          1840.5M 2026-08-02T20:46:09Z
# my-coder:v1         1840.5M 2026-08-02T20:46:38Z
```

### `scavenger pull` — register a local GGUF model

```bash
scavenger pull /path/to/model.gguf
scavenger pull /path/to/model.gguf --name mymodel:v1
```

### `scavenger create` — create a model from a Modelfile

```bash
# Modelfile syntax is a subset of Ollama's:
cat > Modelfile <<'EOF'
FROM qwen2.5:3b
SYSTEM You are a helpful coding assistant.
PARAMETER temperature 0.5
PARAMETER top_p 0.9
EOF

scavenger create my-coder:v1 -f Modelfile
```

### `scavenger import-ollama` — import a model Ollama already downloaded

```bash
# Reads directly from ~/.ollama/models/blobs — no re-download:
scavenger import-ollama qwen2.5:3b
scavenger import-ollama qwen3:8b
```

### `scavenger version`

```bash
scavenger version
# 0.7.0
```

---

## REST API

The server starts on `http://127.0.0.1:11434` by default (same port as Ollama).

### Ollama-compatible endpoints

#### `POST /api/generate` — single-turn completion

```bash
curl http://localhost:11434/api/generate \
  -d '{"model":"qwen2.5:3b","prompt":"Why is the sky blue?","stream":false}'
```

```json
{
  "model": "qwen2.5:3b",
  "response": "The sky appears blue because...",
  "done": true,
  "eval_count": 120,
  "routed_device": "igpu"
}
```

**Streaming** (`"stream": true` or omitted) sends newline-delimited JSON chunks, matching Ollama's wire format exactly.

**JSON mode**: add `"format": "json"` to enforce structured output.

#### `POST /api/chat` — multi-turn chat with session affinity

```bash
curl http://localhost:11434/api/chat \
  -d '{
    "model": "qwen2.5:3b",
    "stream": false,
    "messages": [{"role":"user","content":"Hello! What can you do?"}]
  }'
```

```json
{
  "message": {"role": "assistant", "content": "I can help with..."},
  "done": true,
  "session_id": "a1b2c3d4...",
  "routed_device": "igpu"
}
```

**Session pinning**: pass `"session_id"` from the previous response to pin subsequent turns to the same device:

```bash
curl http://localhost:11434/api/chat \
  -d '{
    "model": "qwen2.5:3b",
    "stream": false,
    "session_id": "a1b2c3d4...",
    "messages": [{"role":"user","content":"Tell me more."}]
  }'
```

**Tool calling**:

```bash
curl http://localhost:11434/api/chat \
  -d '{
    "model": "qwen2.5:3b",
    "stream": false,
    "messages": [{"role":"user","content":"What time is it in Tokyo?"}],
    "tools": [{
      "type": "function",
      "function": {
        "name": "get_time",
        "description": "Get current time for a city",
        "parameters": {
          "type": "object",
          "properties": {"city": {"type": "string"}},
          "required": ["city"]
        }
      }
    }]
  }'
```

#### `POST /api/embeddings`

```bash
curl http://localhost:11434/api/embeddings \
  -d '{"model":"qwen2.5:3b","prompt":"The quick brown fox"}'
```

#### `GET /api/tags` — list models (Ollama-compatible)

```bash
curl http://localhost:11434/api/tags
```

#### `GET /api/ps` — running models + active sessions

```bash
curl http://localhost:11434/api/ps
```

#### `POST /api/create` — create model from Modelfile (streaming)

```bash
curl http://localhost:11434/api/create \
  -d '{"name":"my-model:v1","modelfile":"FROM qwen2.5:3b\nSYSTEM You are helpful."}'
```

#### `POST /api/pull`, `DELETE /api/delete`, `POST /api/show`, `POST /api/copy`

Ollama-compatible model management endpoints.

---

### OpenAI-compatible endpoints

Drop-in replacement for the OpenAI Python client:

```python
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:11434/v1",
    api_key="not-needed",
)

response = client.chat.completions.create(
    model="qwen2.5:3b",
    messages=[{"role": "user", "content": "Explain gradient descent simply."}],
)
print(response.choices[0].message.content)
```

#### `POST /v1/chat/completions`

```python
# Tool calling
response = client.chat.completions.create(
    model="qwen2.5:3b",
    messages=[{"role": "user", "content": "Get the weather in London"}],
    tools=[{
        "type": "function",
        "function": {
            "name": "get_weather",
            "description": "Get weather for a location",
            "parameters": {
                "type": "object",
                "properties": {"location": {"type": "string"}},
                "required": ["location"],
            },
        }
    }],
)

# JSON structured output
response = client.chat.completions.create(
    model="qwen2.5:3b",
    messages=[{"role": "user", "content": "List 5 colors as JSON"}],
    response_format={"type": "json_object"},
)
```

#### `POST /v1/embeddings`

```python
response = client.embeddings.create(
    model="qwen2.5:3b",
    input="The quick brown fox",
)
print(response.data[0].embedding[:5])
```

#### `GET /v1/models`

```bash
curl http://localhost:11434/v1/models
```

---

### Monitoring & admin endpoints

#### `GET /health` — overall health

```bash
curl http://localhost:11434/health
# {"status": "ok", "scheduler": "running", "mock": false}
```

#### `GET /health/devices` — per-device status

```bash
curl http://localhost:11434/health/devices
# {
#   "devices": {
#     "cpu":  {"degraded": false, "queue_depth": 0},
#     "igpu": {"degraded": false, "queue_depth": 2},
#     "dgpu": {"degraded": false, "queue_depth": 0}
#   }
# }
```

#### `GET /telemetry` — live utilisation

```bash
curl http://localhost:11434/telemetry
# {
#   "cpu":  {"util_pct": 42.0, "vram_used_mb": null},
#   "igpu": {"util_pct": 87.0, "vram_used_mb": 1240},
#   "dgpu": {"util_pct": 65.0, "vram_used_mb": 3072}
# }
```

#### `POST /admin/recover-device/{device}` — clear degraded flag

```bash
curl -X POST http://localhost:11434/admin/recover-device/igpu
```

---

### Audio / ASR (requires `[audio]` extra)

```bash
pip install "siliconscavenger[server,audio]"
export SCAVENGER_WHISPER_MODEL_PATH=/path/to/whisper-base

# OpenAI-compatible multipart upload:
curl http://localhost:11434/v1/audio/transcriptions \
  -F file=@recording.wav \
  -F model=whisper-1

# Native JSON endpoint:
curl http://localhost:11434/api/audio \
  -d "{\"audio\": \"$(base64 -w0 recording.wav)\", \"language\": \"en\"}"
```

---

## Python SDK

The `siliconscavenger` package exposes the C++ scheduler directly:

```python
import siliconscavenger as ss

print(ss.version())          # "0.7.0"
print(ss.has_real_backends()) # True if built with real GGUF backends

# Create a scheduler (real inference requires --model path or env var)
scheduler = ss.Scheduler()

# Enqueue a task (CPU inference token array or embedding)
import numpy as np
task_id = scheduler.enqueue_task(np.array([1.0, 2.0, 3.0], dtype=np.float32))

# Retrieve result (blocks until done)
result = scheduler.wait_for_result(task_id)

# Live telemetry
tel = scheduler.get_telemetry()
print(tel["cpu"]["util_pct"])   # CPU utilisation %
print(tel["igpu"]["util_pct"])  # iGPU utilisation %
print(tel["dgpu"]["util_pct"])  # dGPU utilisation % (when CUDA build)
```

### Session management (via the API)

```python
import httpx

BASE = "http://localhost:11434"

# Start a session
r = httpx.post(f"{BASE}/api/chat", json={
    "model": "qwen2.5:3b",
    "stream": False,
    "messages": [{"role": "user", "content": "Hello"}],
})
session_id = r.json()["session_id"]
device = r.json()["routed_device"]   # "cpu" | "igpu" | "dgpu"

# Continue (pinned to same device for KV-cache reuse)
r2 = httpx.post(f"{BASE}/api/chat", json={
    "model": "qwen2.5:3b",
    "stream": False,
    "session_id": session_id,
    "messages": [{"role": "user", "content": "Tell me more"}],
})
```

---

## Dashboard

Open `http://127.0.0.1:11434/dashboard/` after `scavenger serve`:

- **Device cards**: CPU / iGPU / dGPU with live utilisation gauges and queue depth
- **3-lane timeline**: per-device request history
- **Session strip**: active session count
- Links to `/api/ps` and `/health/devices` for raw data

---

## Environment variables

| Variable | Default | Effect |
|---|---|---|
| `SCAVENGER_LLAMA_MODEL_PATH` | — | Path to GGUF model; activates real inference |
| `SCAVENGER_PREFER_DGPU` | `0` | Set to `1` to bias first-turn routing to dGPU |
| `SCAVENGER_WHISPER_MODEL_PATH` | — | Path to OpenVINO Whisper model for ASR |
| `SCAVENGER_KV_STATE_DIR` | `~/.scavenger/kv` | Directory for KV session state files |

---

## Architecture

```
┌────────────────────────────────────────────────┐
│              Client Layer                       │
│  CLI  │  Ollama REST  │  OpenAI REST  │  UI     │
└────────────────┬───────────────────────────────┘
                 │  Python (FastAPI / asyncio)
        ┌────────▼─────────┐
        │  pybind11 bridge  │
        └────────┬──────────┘
                 │  C++17 native
        ┌────────▼──────────────────┐
        │  Scavenger Core            │
        │  lock-free task queue      │
        │  completion-time scoring   │
        │  3-device hw poller        │
        └────┬─────────┬────────┬───┘
             │         │        │
    ┌─────────▼─┐  ┌──▼──────┐  ┌▼────────────┐
    │ OpenVINO  │  │ Vulkan  │  │ CUDA/llama  │
    │ CPU+iGPU  │  │ iGPU    │  │ dGPU sm_86  │
    └───────────┘  └─────────┘  └─────────────┘
```

Python is never in the per-token hot path. The native C++ core owns the scheduler, queues, and memory manager.

---

## License

[MIT](LICENSE) — Copyright (c) 2026 JBAC EdTech (Anubhav Choudhery & Jai Ansh Singh Bindra)
