Metadata-Version: 2.4
Name: vaultlm
Version: 1.1.0
Summary: Local AI inference for privacy-first edge deployment — Bonsai-8B with TurboAgent KV compression
Author: VaultLM
License: Apache-2.0
Keywords: llm,inference,edge-ai,privacy,bonsai,qwen,gguf
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.10
Description-Content-Type: text/markdown
Requires-Dist: requests>=2.31.0
Requires-Dist: openai>=1.30.0
Provides-Extra: gpu
Requires-Dist: nvidia-ml-py>=12.0.0; extra == "gpu"
Requires-Dist: psutil>=5.9.0; extra == "gpu"
Provides-Extra: kv
Requires-Dist: turboagent-ai[llama,native,server]==1.1.0; extra == "kv"
Requires-Dist: nvidia-ml-py>=12.0.0; extra == "kv"
Requires-Dist: psutil>=5.9.0; extra == "kv"
Provides-Extra: all
Requires-Dist: turboagent-ai[llama,native,server]==1.1.0; extra == "all"
Requires-Dist: nvidia-ml-py>=12.0.0; extra == "all"
Requires-Dist: psutil>=5.9.0; extra == "all"
Requires-Dist: huggingface-hub>=0.23.0; extra == "all"

# VaultLM

Local AI inference for privacy-first edge deployment.

Runs **Bonsai-8B** (Qwen3-8B-Q1_0) on any NVIDIA GPU (RTX 30xx/40xx/50xx) with
optional **TurboAgent KV compression** (v1.1) for 5× memory efficiency at long contexts.

All inference is local — no data leaves your machine.

---

## Requirements

- Python 3.10+
- NVIDIA GPU: RTX 30xx (SM 86), RTX 40xx (SM 89), or RTX 50xx (SM 120)
- CUDA Toolkit (installed automatically by `vaultlm setup`)
- ~13 GB disk space (model + binaries)
- Windows 10/11 or Linux

---

## Install

```bash
pip install vaultlm[all]
```

The PyPI wheel contains only the Python code and CLI. The model, PrismML binaries,
and the custom llama-cpp-python build are downloaded and compiled on first run:

```bash
vaultlm setup
```

`setup` will:
1. Detect your GPU (SM version)
2. Install CUDA Toolkit if missing (`winget` on Windows, `apt` on Linux)
3. Clone llama-cpp-python and vendor the PrismML fork into it
4. Build llama-cpp-python with CUDA + correct SM architecture
5. Download the PrismML CUDA binary and DLLs
6. Download Bonsai-8B-Q1_0.gguf from Hugging Face

All files land in `~/.vaultlm/`.

---

## Quickstart

```bash
# One-time setup
vaultlm setup

# Start server (TurboAgent KV compression enabled automatically if ABI check passes)
vaultlm serve

# Run smoke tests
vaultlm test

# Run specific suite
vaultlm test --suite coherence
vaultlm test --suite niah
vaultlm test --suite kv        # TurboAgent KV compression validation
vaultlm test --suite stress    # 60-request marathon benchmark
```

---

## Optional: GPU monitoring extras

```bash
pip install vaultlm[gpu]
vaultlm test --suite gpu
```

---

## v1.0 vs v1.1

| Feature | v1.0 | v1.1 |
|---|---|---|
| Backend | subprocess llama-server | TurboAgent (+ subprocess fallback) |
| KV compression | None | turbo3 (~5.28×) |
| Context @ 12 GB VRAM | ~32k | ~65k+ |
| GPU support | RTX 30xx | RTX 30xx / 40xx / 50xx |

v1.1 falls back silently to v1.0 behaviour if the PrismML ABI build is not present
(i.e. if `vaultlm setup` has not been run, or was run with `--skip-build`).

---

## CPU / Vulkan

```bash
vaultlm serve --cpu    # Vulkan backend, no CUDA required
vaultlm test --suite throughput -- --path cpu
```

---

## Results directory

Stress test JSON results are saved to `~/.vaultlm/results/`.

---

## License

Apache 2.0
