Metadata-Version: 2.4
Name: leastgen
Version: 0.1.0
Summary: LeastGen: High-performance local inference optimizer and context governance gateway for AI developer agents.
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: pyyaml>=6.0
Dynamic: license-file

# ⚡ LeastGen

**Zero-cost caching proxy for LLM agent traffic.** Sits between your AI coding agent and the API — intercepts repetitive prompt patterns, serves them from a local cache. After warm-up, ~70%+ of requests resolve at **$0 cost and ~0ms latency**.

```
Agent ──► LeastGen (:8766) ──► OpenRouter / API
                │
        ┌───────┴────────┐
        │  Seen ≥3×?      │
        │  YES ── cache   │
        │  NO  ── forward │
        └────────────────┘
```

---

## Quick Start

```bash
curl -sLS https://leastgen.com/install.sh | bash
```

This installs the package, creates a default config, and starts the proxy as a systemd user service on port `8766`.

**What you get:**
- A transparent proxy on `localhost:8766`
- Dashboard at `http://localhost:8766/dashboard`
- Metrics at `http://localhost:8766/metrics`
- Persistent SQLite cache at `~/.leastgen/data/`

### Prerequisites

- **Python 3.10+** — the proxy runs entirely in stdlib (no heavy ML deps)
- **An API key** — set `OPENROUTER_API_KEY` env var or put it in `~/.leastgen/config.yaml`
- **Ollama** (optional) — for local pipeline inference when the cache misses. Install from [ollama.com](https://ollama.com)
- **Hermes Agent** (optional) — LeastGen integrates natively via `model.base_url`

---

## Manual Installation

### 1. Install the package

```bash
git clone https://github.com/KhalidAlnujaidi/leastgen.git
cd leastgen
pip install -e .
```

### 2. Configure

Edit `~/.leastgen/config.yaml`:

```yaml
port: 8766
ttl_days: 7                          # Evict cache entries after 7 days of inactivity
learn_threshold: 3                   # Cache after N occurrences of the same pattern
upstream_url: "https://openrouter.ai/api/v1"
upstream_model: "deepseek/deepseek-chat"
local_planner_model: "vibethinker-3b"   # Ollama model for planning step
local_executor_model: "qwen3:4b"        # Ollama model for code generation step
api_key: "sk-or-..."                    # Optional — or set OPENROUTER_API_KEY
```

### 3. Set your API key

LeastGen looks for an API key in this order:
1. `api_key` field in `~/.leastgen/config.yaml`
2. `OPENROUTER_API_KEY` environment variable
3. `~/.config/free-claude-code/.env`

```bash
export OPENROUTER_API_KEY="sk-or-..."
```

### 4. Start the proxy

```bash
leastgen start
```

Or directly:

```bash
python3 -m leastgen.cli start --port 8766
```

### 5. Verify it's running

```bash
curl -s http://localhost:8766/metrics | python3 -m json.tool
```

Expected output:

```json
{
  "total": 0,
  "hits": 0,
  "hit_rate_pct": 0.0,
  "templates": 0,
  "uptime_seconds": 2,
  "cost_saved": 0.0,
  "cost_passed": 0.0
}
```

---

## Connecting Your Agent

### Hermes Agent

```bash
hermes config set model.base_url http://localhost:8766/v1
```

All LLM requests from Hermes now route through LeastGen.

### Any OpenAI-compatible client

Point your `base_url` to `http://localhost:8766/v1`. Everything OpenAI-compatible works — streaming, tool calls, reasoning tokens.

---

## GPU & VRAM Guidance

LeastGen's **caching layer needs zero GPU** — it's a lightweight Python proxy using SQLite. You can run it on any machine, even a Raspberry Pi.

The **local inference pipeline** (optional — the VibeThinker-3B planner + Qwen3:4b executor) does need VRAM if you want fully offline fallback. Here's what you need for each configuration:

### Configuration A: Caching Only (0 GB VRAM — any machine)

**No GPU required.** The proxy just intercepts, caches, and forwards to the API when it misses. Run this on a laptop, a $5 VPS, or alongside your agent.

```
Agent ──► LeastGen ──► cache hit (instant, $0)
                └────► cache miss ──► API ($)
```

Set in `~/.leastgen/config.yaml`:
```yaml
# No local models — pure caching proxy
# Every miss goes to the upstream API
```

### Configuration B: Lightweight Local Fallback (~4 GB VRAM)

Run smaller Ollama models for offline inference when cache misses. Works on most consumer GPUs (GTX 1060 6GB, RTX 2060, RTX 3050, RTX 3060, etc.).

```bash
# Install lightweight models via Ollama
ollama pull qwen2.5-coder:1.5b     # ~1.1 GB
ollama pull qwen2.5-coder:0.5b     # ~0.5 GB (ultra-light, for planning)
```

```yaml
# ~/.leastgen/config.yaml
local_planner_model: "qwen2.5-coder:0.5b"
local_executor_model: "qwen2.5-coder:1.5b"
```

**VRAM estimate:** ~2–3 GB — fits on any GPU with 4 GB or more.

### Configuration C: Standard Pipeline (~6 GB VRAM)

The default configuration with VibeThinker-3B + Qwen3:4b. Requires ~6 GB VRAM. Runs on RTX 2060 Super (8 GB), RTX 3060 (12 GB), RTX 4060, M-series Macs with 16 GB unified memory, etc.

```bash
ollama pull vibethinker-3b     # ~2 GB
ollama pull qwen3:4b           # ~4 GB
```

### Configuration D: High-Performance Pipeline (~10–12 GB VRAM)

Larger models for better code generation quality. Needs RTX 3090/4090, A4000+, or similar.

```bash
ollama pull deepseek-r1:8b        # R1 reasoning for planning
ollama pull qwen2.5-coder:7b      # Strong code executor
```

```yaml
local_planner_model: "deepseek-r1:8b"
local_executor_model: "qwen2.5-coder:7b"
```

### Quick VRAM Reference

| Config | Models | VRAM | GPU Examples |
|--------|--------|------|-------------|
| **A** — Cache only | None | **0 GB** | Any machine |
| **B** — Lightweight | Qwen 0.5B + 1.5B | **~2–3 GB** | GTX 1060, 3050, 2060 |
| **C** — Standard | VibeThinker-3B + Qwen3:4b | **~6 GB** | RTX 2060S, 3060, 4060 |
| **D** — High perf | R1:8b + Qwen2.5-Coder:7b | **~10–12 GB** | RTX 3090, 4090, A4000 |

### Tokens/Second Benchmarks (RTX A4500)

| Model | Tokens/sec | vs. Cloud |
|-------|-----------|-----------|
| VibeThinker-3B (Q4_K_M) | **201 tok/s** | ~2-4× faster |
| Qwen3:4b | **161 tok/s** | ~2-3× faster |
| Claude 5 Fable (cloud API) | ~50–100 tok/s | +200–500ms network latency |

### Fair-Comparison Note

Local proxy models may generate **~3× more tokens** than a frontier model to complete the same task — the reduced capability means more iterations to converge on a correct answer. However, these extra tokens are **generated entirely free** on local GPU at the speeds above. The "tokens saved" figures on the site are not a 1:1 apples-to-apples comparison with cloud tokens — they represent tokens that would have cost money if sent to the API. The local pipeline may do more total generation, but every token is instant and costs $0.00.

### CPU-Only Mode

Ollama models run on CPU by default if no GPU is detected. Performance is slower (seconds instead of milliseconds) but works on any machine. For CPU-only:

```bash
# Ollama automatically falls back to CPU if no NVIDIA GPU
ollama pull qwen2.5-coder:1.5b     # ~1.1 GB RAM on CPU
```

Note: on CPU, responses take 5–30 seconds instead of <1 second on GPU. The caching layer is still instant — only the fallback path is slower.

---

## Dashboard & Monitoring

### Live Dashboard

Open `http://localhost:8766/dashboard` in your browser.

Shows in real time:
- **Cache Hit Rate** — % of requests served locally
- **Tokens Saved / Passed** — cumulative counts
- **Cost Saved / Cost Incurred** — estimated $
- **Templates** — distinct patterns learned
- **Hit rate chart** — live time-series
- **Recent Activity** — last 20 cache/learn/forward events

### Metrics Endpoint

```bash
curl -s http://localhost:8766/metrics | python3 -m json.tool
```

Returns: `total`, `hits`, `template_hits`, `hit_rate_pct`, `templates`, `cached_responses`, `learned_patterns`, `uptime_seconds`, `prompt_tokens_saved`, `completion_tokens_saved`, `tokens_saved`, `tokens_passed`, `cost_saved`, `cost_passed`, `recent`

### Run a Savings Audit

```bash
leastgen audit
```

Scans your Hermes session history and estimates how much LeastGen would save on your actual traffic patterns.

---

## Systemd Service (Auto-Start)

The install script sets this up automatically. To do it manually:

```bash
# Create the service file
mkdir -p ~/.config/systemd/user
cat > ~/.config/systemd/user/leastgen.service << 'EOF'
[Unit]
Description=LeastGen Local Inference Optimizer Caching Gateway
After=network.target

[Service]
Type=simple
ExecStart=%h/.local/bin/leastgen start
Restart=always
RestartSec=5

[Install]
WantedBy=default.target
EOF

# Enable and start
systemctl --user daemon-reload
systemctl --user enable leastgen.service
systemctl --user start leastgen.service

# Check status
systemctl --user status leastgen
```

---

## Enterprise Value Projection

Based on **Anthropic Claude 5 Fable pricing** ($10/1M input, $50/1M output) at 20M prompt tokens per developer per day with a ~70:30 prompt-to-completion ratio:

| Team Size | Annual Tokens Saved | Annual API Cost Avoided |
|-----------|-------------------|----------------------|
| 1 Developer | 7.1 Billion | **$155,000** |
| 10 Developers | 71.5 Billion | **$1,575,000** |
| 50 Developers | 357.5 Billion | **$7,875,000** |
| 100 Developers | 715 Billion | **$15,750,000** |

(See [leastgen.com](https://www.leastgen.com) for the live projection calculator and benchmarks.)

---

## Architecture

```
┌──────────────────────────────────────────────────────┐
│                    Your Agent                         │
│  base_url = http://localhost:8766/v1                  │
└────────────────────────┬─────────────────────────────┘
                         │
                         ▼
┌──────────────────────────────────────────────────────┐
│              LeastGen (port 8766)                      │
│                                                        │
│  Incoming request ──► template_normalize()             │
│                         │                              │
│                    ┌────┴────┐                         │
│                    │         │                         │
│               exact hit  template hit                  │
│               (hash)    (pattern)                      │
│                    │         │                         │
│               serve from   serve from                  │
│               cache        cache                       │
│                    │         │                         │
│                    └────┬────┘                         │
│                         │                              │
│               match?    │                              │
│                    ┌────┴────┐                         │
│                    │  seen   │                         │
│                    │  ≥ 3×?  │                         │
│                    │  YES ──► learn & cache            │
│                    │  NO  ──► forward to API           │
│                    └─────────┘                         │
└────────────────────────────────────────────────────────┘
                         │
                         ▼
┌──────────────────────────────────────────────────────┐
│              Upstream API (OpenRouter, etc.)           │
│  https://openrouter.ai/api/v1/chat/completions         │
└──────────────────────────────────────────────────────┘
```

---

## Troubleshooting

### Proxy won't start
```bash
# Check if port is in use
ss -tlnp | grep 8766
# Kill any hanging process
pkill -9 -f leastgen
```

### "No API key" error
```bash
export OPENROUTER_API_KEY="sk-or-..."
# Or add to config:
#   api_key: "sk-or-..."  # in ~/.leastgen/config.yaml
```

### Dashboard shows 0% hit rate
Normal on a fresh start — the gate needs ≥3 occurrences of a pattern before caching. Work normally and the hit rate climbs as patterns repeat.

### High cache miss rate
Template normalization only uses the last user message's first ~10 significant words. Long, unique prompts may not match previously cached templates. The exact hash cache still catches identical repeats.

### Reset all state (start fresh)
```bash
systemctl --user stop leastgen
rm -rf ~/.leastgen/data/cache.db
systemctl --user start leastgen
```

---

## Links

- **Website:** [https://www.leastgen.com](https://www.leastgen.com)
- **GitHub:** [https://github.com/KhalidAlnujaidi/leastgen](https://github.com/KhalidAlnujaidi/leastgen)
- **Author:** Khalid Alnujaidi — khalidnujaidi@gmail.com
