Metadata-Version: 2.1
Name: moe-l2
Version: 0.3.0
Summary: MoE inference L2 hot-cache scheduler — run large MoE models on consumer GPUs
License: Proprietary
Project-URL: Homepage, https://github.com/yalun753/yalund-moe-l2
Project-URL: Source, https://github.com/yalun753/yalund-moe-l2
Keywords: moe,mixture-of-experts,llm,inference,caching,ollama,llama.cpp
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: License :: Other/Proprietary License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: System :: Hardware :: Hardware Drivers
Requires-Python: >=3.10
Description-Content-Type: text/markdown
Requires-Dist: httpx>=0.27
Requires-Dist: gguf>=0.10
Provides-Extra: predictor
Requires-Dist: sentence-transformers>=3.0; extra == "predictor"

# moe-l2

**Run large MoE models on consumer GPUs.** A transparent proxy that predicts which experts your prompt needs, preloads them into a shared-memory LRU cache, so you can run 16 GB+ models on 8 GB GPUs.

## How it works

MoE models have many "experts" but only activate a few per token. moe-l2 predicts your prompt's domain (codegen, math, chinese_tech, etc.) and preloads the relevant experts into an mmap'd LRU cache before they're needed.

```
user → moe-l2 proxy (localhost:11435)
    ├── predict domain
    ├── preload domain experts → /dev/shm/moe_l2/
    └── forward to ollama (localhost:11434)
```

## Quick start

```bash
pip install moe-l2                   # keyword-only predictor (zero extra deps)
pip install moe-l2[predictor]        # hybrid: keyword + semantic embedding
moe-l2 start --model model.gguf --l2-size 4GB
```

## Usage

### 1. L2 proxy (recommended)

Start the transparent proxy — sits between your client and ollama:

```bash
moe-l2 start --model /models/DeepSeek-V2-Lite.Q4_K_M.gguf --l2-size 4GB
```

All default ollama tools work through it (curl, open-webui, langchain):

```bash
# streaming
curl http://localhost:11435/api/chat -d '{
  "model":"qwen3:4b",
  "messages":[{"role":"user","content":"write a Python script"}],
  "stream":true
}'

# blocking
curl http://localhost:11435/api/chat -d '{
  "model":"qwen3:4b",
  "messages":[{"role":"user","content":"hello"}],
  "stream":false
}'
```

### 2. Monitor cache stats

```bash
moe-l2 stats --port 11435
```

Example output:
```
moe-l2 cache stats
  requests:     47
  hits:         42     (89.4%)
  misses:        5
  slots_used:  32/48  (66.7%)
  memory:     456 MB  (68.3% of 668 MB)
```

### 3. Use as a library

```python
from moe_l2 import predict, predict_hybrid, domain_to_expert_ids
from moe_l2.cache import L2Cache

# Predict domain (zero-dependency mode)
domain = predict("print hello world")  # → "codegen"

# Hybrid mode (falls back to semantic embedding)
from moe_l2 import enable_semantic
enable_semantic()
domain = predict_hybrid("deploy nginx on ubuntu")  # → "chinese_tech"

# Setup L2 cache
cache = L2Cache(
    model_path="/models/model.gguf",
    slots_per_layer=48  # auto-calculated from --l2-size
)
expert_map = load_mapping()
cache.preload_domain("codegen", expert_map)

# Stats
cache.stats()  # → {"hits": ..., "misses": ..., ...}
```

## Supported domains

| Domain | Examples |
|--------|---------|
| `codegen` | Python, JS, bash, API design |
| `debug` | error logs, stack traces, crash analysis |
| `math` | algebra, calculus, equations |
| `logic` | reasoning, puzzles, proofs |
| `general_qa` | general knowledge, facts, explanations |
| `chinese_tech` | 中文技术内容, NAS, 部署, 教程 |
| `creative_write` | storytelling, poetry, marketing |
| `translate` | translation between languages |

## Architecture

```
┌────────────────────────────────────────────────┐
│         Ollama Client (user-facing)             │
│    curl / open-webui / langchain / any tool     │
└──────────┬─────────────────────────────────────┘
           │ POST to :11435
┌──────────▼─────────────────────────────────────┐
│            moe-l2 proxy                         │
│  ┌─────────────────────────────────────────┐    │
│  │ Domain Predictor                        │    │
│  │  ┌──────────┐   ┌───────────────────┐   │    │
│  │  │ Keywords │ → │ Semantic (opt)    │   │    │
│  │  │ ~210词    │   │ all-MiniLM-L6-v2 │   │    │
│  │  └──────────┘   └───────────────────┘   │    │
│  │         ↓ domain                          │    │
│  │  ┌──────────────────────────────────┐    │    │
│  │  │ L2 Cache (LRU + mmap /dev/shm/) │    │    │
│  │  │  preload domain experts async   │    │    │
│  │  └──────────────────────────────────┘    │    │
│  └─────────────────────────────────────────┘    │
│           │ POST to :11434 (transparent)        │
└──────────┬─────────────────────────────────────┘
           │
┌──────────▼─────────────────────────────────────┐
│           Ollama / llama.cpp                    │
│    MoE inference with hot-cached experts        │
└────────────────────────────────────────────────┘
```

## CLI reference

| Command | Description |
|---------|-------------|
| `moe-l2 start --model <path> --l2-size <size>` | Start proxy + cache |
| `moe-l2 start --model <path> --gpu` | Start with GPU-accelerated llama-server |
| `moe-l2 stats --port <port>` | Show live cache stats |
| `moe-l2 download-bins [--release TAG]` | Download pre-built GPU binaries from GitHub |
| `moe-l2 stop --port <port>` | Stop proxy |

Options:
- `--model auto`: scan `/opt/data/models/*.gguf`
- `--l2-size 4GB` / `--l2-size 512MB`: target cache size
- `--port 11435` (default)
- `--gpu`: enable GPU mode (requires CUDA + NVIDIA GPU)

> **GPU binaries**: The repo does not track 500MB+ .so files. When you `pip install moe-l2`, binaries are included. For git-clone users, run `moe-l2 download-bins` to fetch them from GitHub Release.

## Project status

**Phase 2** — core components complete (2026-07-30):
- ✅ Domain predictor (keyword + optional semantic)
- ✅ L2 cache (mmap LRU, thread-safe, async preload)
- ✅ GGUF weight reader (direct memmap from .gguf)
- ✅ Transparent proxy (HTTP/SSE forwarding, predict+preload)
- ✅ CLI (start/stats with auto model detection, --gpu support)
- ✅ GPU end-to-end pipeline verified (DS-V2-Lite on RTX 4090 24GB, ~1.6 GiB VRAM)
- ✅ Bundled GPU binaries (A3-patched llama-server + CUDA libs, 532 MB)
- ✅ PyPI v0.2.0 → v0.3.0 release
- 🔲 llama.cpp C++ integration (direct mmap from L2 cache) — Phase 3
- 🔲 GPU LRU expert cache (keep hot experts in VRAM, reduce PCIe transfers)

## License

**All Rights Reserved.** This software is proprietary and confidential.
No part may be reproduced, distributed, or transmitted without prior written permission.
Copyright (c) 2026 yalun753.
