Metadata-Version: 2.4
Name: baremetalrt
Version: 0.23.5
Summary: BareMetalRT — CLI for the Windows-native local AI agent runtime
Author-email: "Bare Metal AI, Inc." <brian@baremetalrt.ai>
License: Copyright (c) 2025-2026 BareMetalRT. All rights reserved.
        
        This software and associated documentation files (the "Software") are the
        proprietary property of BareMetalRT. The Software is licensed, not sold.
        
        Permission is hereby granted to use the Software solely for personal,
        non-commercial evaluation purposes, subject to the following conditions:
        
        1. You may not copy, modify, merge, publish, distribute, sublicense, or sell
           copies of the Software.
        
        2. You may not use the Software for commercial purposes without a separate
           commercial license agreement from BareMetalRT.
        
        3. You may not reverse engineer, decompile, or disassemble the Software,
           except to the extent that such activity is expressly permitted by
           applicable law.
        
        4. You may not remove or alter any proprietary notices, labels, or marks
           on the Software.
        
        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
        SOFTWARE.
        
        ---
        
        Note: The TensorRT-LLM submodule (engine/tensorrt-llm/) is licensed separately
        under NVIDIA's Apache License 2.0. See engine/tensorrt-llm/LICENSE for details.
        
Project-URL: Homepage, https://baremetalrt.ai
Project-URL: Repository, https://github.com/baremetalrt/baremetalrt
Project-URL: Documentation, https://baremetalrt.ai/docs
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
License-File: LICENSE_COMPLIANCE_MATRIX.md
License-File: NOTICE
Requires-Dist: httpx>=0.24.0
Dynamic: license-file

# BareMetalRT

[![PyPI](https://img.shields.io/pypi/v/baremetalrt)](https://pypi.org/project/baremetalrt/)
[![Release](https://img.shields.io/github/v/release/baremetalrt/dist?include_prereleases)](https://github.com/baremetalrt/dist/releases/latest)
[![License](https://img.shields.io/badge/license-proprietary-blue)](LICENSE)

**The Windows-native local AI agent runtime for NVIDIA RTX PCs.**

> **Public beta.** Free to use and actively developed. Core inference, chat, and the agent loop are solid; some features (MCP connectors, the full model catalog, multi-GPU) are still being hardened across hardware. Expect rough edges, and please [report issues](https://github.com/baremetalrt/dist/issues) — that feedback directly shapes what ships next.

Run reliable tool-calling agents — with voice, vision, RAG, memory, and a coding agent — entirely on your own GPU. No cloud, no per-token bill, no data leaving the machine. BareMetalRT uses NVIDIA's own TensorRT-LLM CUDA kernels (the engine behind cloud inference APIs) natively on Windows, and can pool multiple consumer GPUs — even mismatched cards across machines — into a single tensor-parallel runtime over standard networking.

**[Download Installer](https://github.com/baremetalrt/dist/releases/latest)** | **[Live Demo](https://baremetalrt.ai/demo)** | **[Documentation](https://baremetalrt.ai/docs)** | **[PyPI](https://pypi.org/project/baremetalrt/)** | **[Technical Paper](paper/main.pdf)**

## How It Works

1. **Install** the BareMetalRT daemon on each Windows machine with an NVIDIA GPU
2. **Connect** your GPU to your account at [baremetalrt.ai](https://baremetalrt.ai)
3. **Run inference** — the system automatically shards models across your available GPUs

```
┌──────────────────────────────────┐
│  baremetalrt.ai                  │
│  Auth, routing, OpenAI-compat API│
└──────────────┬───────────────────┘
               │ WebSocket
    ┌──────────┼──────────┐
    │          │          │
┌───▼───┐ ┌───▼───┐ ┌───▼───┐
│Node A │ │Node B │ │Node C │
│3090   │ │4060   │ │3060   │
│24GB   │ │8GB    │ │12GB   │
│Daemon │ │Daemon │ │Daemon │
└───┬───┘ └───┬───┘ └───┬───┘
    └── TCP AllReduce ──┘
```

## Why BareMetalRT

| | Cloud (OpenAI, etc.) | Local (Ollama, LM Studio) | Distributed (Petals, Exo) | **BareMetalRT** |
|---|---|---|---|---|
| **Kernels** | Optimized (proprietary) | Generic CUDA | Generic CUDA | **TensorRT-LLM (1,500+ optimized .cu)** |
| **Multi-GPU** | NVLink (datacenter only) | Single GPU only | Pipeline parallelism | **Tensor parallelism over TCP** |
| **Heterogeneous GPUs** | No | N/A | No | **Yes — different VRAM, different SMs** |
| **Windows native** | N/A | Yes | Partial | **Yes** |
| **Cost** | Per-token pricing | Free (your hardware) | Free (your hardware) | **Free (your hardware)** |
| **Privacy** | Data leaves your machine | Fully local | Weights distributed | **Weights distributed, execution local** |

**The key difference:** every other consumer GPU project uses pipeline parallelism, which leaves GPUs idle 50% of the time. BareMetalRT is the first to achieve tensor parallelism across heterogeneous consumer GPUs — both GPUs compute on every layer, every token.

## Benchmarks

### Single-GPU throughput — the everyday path

Most work runs on one card. Measured on an RTX 4070 SUPER (12 GB), int4 (W4A16-AWQ), streaming decode:

| Model | Throughput |
|---|---|
| Llama 3.2 3B | 130 tok/s |
| DeepSeek-R1 Distill 7B / Llama 8B | 75 tok/s |
| Qwen 2.5 Coder 7B | 75 tok/s |
| Llama 3.1 8B | 74 tok/s |
| Qwen 2.5 7B | 73 tok/s |

Every one of these streams far faster than a person reads, on a single consumer GPU.

### Heterogeneous tensor parallelism — the hard result

BareMetalRT can also split one model across **mismatched** GPUs — even across machines — over standard networking. Tested with Mistral 7B (14 GB FP16) across an RTX 4070 SUPER (12 GB) and an RTX 4060 Laptop (8 GB), **a model too large for either card alone**:

| Configuration | Throughput | Latency | Notes |
|---|---|---|---|
| **TP=2 — Mistral 7B** | **12.5 tok/s** | **80 ms/tok** | KV cache + overlapped AllReduce |
| TP=2 — Ethernet (1 ms ping) | 3.6 tok/s | 276 ms/tok | TinyLlama 1.1B |
| TP=2 — WiFi (316 ms ping) | 3.6 tok/s | 277 ms/tok | TinyLlama 1.1B |

> **Key finding:** a 300× improvement in network speed (WiFi → Ethernet) yielded *zero* throughput change. GPU-synchronization overhead — not network latency — is the dominant bottleneck. The correctness result — identical output from mismatched GPUs over a commodity network — is what matters.

## System Requirements

- Windows 10/11 (64-bit)
- NVIDIA GPU (RTX 2000+; 8 GB VRAM or more recommended)
- An up-to-date NVIDIA driver

The CUDA runtime and TensorRT-LLM are bundled with the installer — no separate CUDA Toolkit or TensorRT installation is required.

## Quick Start

### 1. Install

Download the code-signed MSI from [GitHub Releases](https://github.com/baremetalrt/dist/releases/latest) and run it. It bundles the CUDA runtime and TensorRT-LLM — no separate installs — and registers the daemon to start on boot. Everything is served locally at `http://localhost:8080`.

### 2. Load a model and chat

Open the app at [baremetalrt.ai/app](https://baremetalrt.ai/app) — it auto-detects the daemon on your machine — or drive it from the terminal with the `bmrt` CLI:

```bash
pip install baremetalrt
bmrt status                  # GPU + daemon state
bmrt models                  # catalog + download status
bmrt pull qwen3-4b-int4      # download a catalog model
bmrt run qwen3-4b-int4       # load it and chat in the terminal
```

The CLI is a thin wrapper over the local daemon's REST API, defaulting to `http://localhost:8080` (override with `--url`).

### 3. Use the OpenAI-compatible API

Local chat is free — use the in-app chat or the `bmrt run` CLI above. The programmatic **OpenAI/Anthropic-compatible `/v1` API is a paid feature** (Starter plan or higher); on the free tier, `/v1` returns `402`. The built-in chat and CLI are never gated.

With a Starter+ plan on the node, the local endpoint is served at `http://localhost:8080/v1`:

```bash
curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "qwen3-4b-int4", "messages": [{"role": "user", "content": "Hello!"}]}'
```

To reach your GPU **remotely** through the cloud relay, sign in at [baremetalrt.ai/app](https://baremetalrt.ai/app), mint a `bmrt_` key in Account Settings, and point any OpenAI- or Anthropic-compatible client (Python `openai` SDK, Continue, Cursor, `curl`) at `https://baremetalrt.ai/v1`:

```bash
curl https://baremetalrt.ai/v1/chat/completions \
  -H "Authorization: Bearer bmrt_your_api_key" \
  -H "Content-Type: application/json" \
  -d '{"model": "qwen3-4b-int4", "messages": [{"role": "user", "content": "Hello!"}]}'
```

## Technical Details

- **FP32 precision correctness** — custom CUDA kernel performs AllReduce on-GPU in FP32, achieving the theoretical floor of IEEE 754 arithmetic. 2,500x more accurate than FP16. Identical to NCCL on NVLink.
- **Asymmetric-tolerant transport** — GPUs with different VRAM and compute capabilities participate in the same AllReduce without barrier stalls. The slower GPU sets the pace; the faster GPU waits on a non-blocking receive.
- **TensorRT plugin integration** — custom `IPluginV2DynamicExt` plugins intercept every AllReduce/AllGather call at execution time, replacing NCCL with our TCP transport without modifying TRT-LLM's model definitions.
- **Overlapped AllReduce** — TCP recv runs in a background thread during GPU sync wait. When sync_wait > recv_time, the network transfer adds zero time to the critical path.
- **Double-buffered pinned memory** — four page-locked host buffers alternate between consecutive AllReduce calls, preventing data races between in-flight transfers.
- **TensorRT-LLM on Windows** — full native port of NVIDIA's inference engine (Conan profiles, FMHA kernels, nanobind bindings, MSVC/CUDA interop). No WSL, no Docker.

See [Architecture](docs/ARCHITECTURE.md) for the full system design, or read the [technical paper](paper/main.pdf).

## What's in This Repo

This is the **public product repo** — the server, web UI, installer, and documentation.

```
baremetalrt/
├── server/        # FastAPI server (auth, chat relay, node management)
├── web/           # Product web app (chat UI, account, downloads)
├── site/          # Landing page and demo
├── installer/     # Windows installer (WiX/Burn MSI, code-signed)
├── cli/           # bmrt CLI (pip install baremetalrt)
├── docs/          # Documentation
│   ├── ARCHITECTURE.md
│   ├── API.md
│   ├── QUICKSTART.md
│   └── MISSION.md
└── paper/         # Technical paper (arXiv-ready LaTeX)
```

The inference engine, transport layer, and daemon are in a separate private repository.

## API

BareMetalRT exposes an OpenAI-compatible API. See [API docs](https://baremetalrt.ai/docs) or the [API reference](docs/API.md).

## Current Status

**v0.21.19** — [Changelog](CHANGELOG.md)

- Single-GPU and cross-machine TP=2 multi-GPU inference on Windows — native, no WSL, no Docker
- Native serving with clean VRAM management, real batching/concurrency, and worker self-heal
- 24 ready-to-run models in the shipping catalog — Qwen 3, Llama 3.x, Mistral, DeepSeek-R1 distills, Phi, Gemma, plus a Qwen2-VL vision model — int4 (W4A16-AWQ) and FP16, from 0.6B to 8B on a single consumer card
- Reliable tool-calling / agent loop, an MCP host, skills, workflows, and routines
- Built-in voice (STT + TTS; runs on 8 GB cards via CPU-mode Whisper)
- Vision (attach images), local RAG with cited answers, and cross-session agent memory
- Local coding agent with a VS Code-class editor and whole-repo map
- Presence — a photoreal voice avatar (preview)
- Egress Mode — seal a node fully air-gapped, with offline Ed25519 licensing
- Web chat UI with streaming; OpenAI- and Anthropic-compatible APIs
- Code-signed MSI installer with automatic GPU claiming
- User accounts: email + password, Google/GitHub OAuth, 2FA, enterprise SSO (OIDC/SAML) + SCIM
- Chat history encrypted on-device — never stored on our servers

## Roadmap

- **TP=4+** — scale beyond two GPUs using ring AllReduce (transport implemented, untested beyond TP=2)
- **Mixture-of-Experts** — replace dense AllReduce with sparse expert routing. 2 of 8 experts per token = 75% of the mesh available for concurrent serving. See [Architecture § Future](docs/ARCHITECTURE.md#future-mixture-of-experts).
- **Distributed KV cache** — page KV entries to peer GPU VRAM across the mesh, enabling 128K+ context on consumer hardware
- **Asymmetric weight splitting** — proportional column assignment based on per-GPU VRAM

## Security

Found a vulnerability? See [SECURITY.md](SECURITY.md) for our responsible disclosure policy.

## License

BareMetalRT is proprietary software. See [LICENSE](LICENSE) for terms.
