Metadata-Version: 2.2
Name: gpumesh
Version: 0.9.0
Summary: Borrow your friends' GPUs: a terminal-based distributed compute mesh in pure Python
License: MIT
Project-URL: Homepage, https://github.com/Samurai007AK/gpumesh
Project-URL: Documentation, https://github.com/Samurai007AK/gpumesh#readme
Project-URL: Repository, https://github.com/Samurai007AK/gpumesh
Project-URL: Issues, https://github.com/Samurai007AK/gpumesh/issues
Keywords: gpu,distributed,compute,mesh,ml,pytorch,cuda
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: System :: Distributed Computing
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: cloudpickle
Provides-Extra: gpu
Requires-Dist: torch; extra == "gpu"
Provides-Extra: tunnel
Requires-Dist: pyngrok; extra == "tunnel"
Provides-Extra: sysinfo
Requires-Dist: psutil; extra == "sysinfo"
Provides-Extra: notebook
Requires-Dist: pandas; extra == "notebook"
Provides-Extra: ui
Requires-Dist: rich; extra == "ui"
Requires-Dist: questionary; extra == "ui"
Provides-Extra: dev
Requires-Dist: pytest; extra == "dev"
Requires-Dist: build; extra == "dev"
Requires-Dist: twine; extra == "dev"
Provides-Extra: all
Requires-Dist: gpumesh[gpu,notebook,sysinfo,tunnel,ui]; extra == "all"

# gpumesh

> **Borrow your friends' GPUs.** A distributed compute mesh that lets you share GPU power across machines on your network — with a single decorator, one CLI command, or a Python API.

[![PyPI version](https://img.shields.io/pypi/v/gpumesh.svg)](https://pypi.org/project/gpumesh/)
[![Python](https://img.shields.io/pypi/pyversions/gpumesh.svg)](https://pypi.org/project/gpumesh/)
[![License](https://img.shields.io/pypi/l/gpumesh.svg)](https://github.com/Samurai007AK/gpumesh/blob/main/LICENSE)
[![Tests](https://img.shields.io/badge/tests-449%20passed-brightgreen)](https://github.com/Samurai007AK/gpumesh)

---

## What is gpumesh?

gpumesh turns multiple machines into a single, unified GPU compute pool. You start a **coordinator** on one machine, join **workers** from other machines (laptops, desktops, servers — anything with Python), and then run code across all of them as if they were one device.

```
 Machine A (coordinator)       Machine B (worker)       Machine C (worker)
 ┌─────────────────────┐      ┌──────────────────┐     ┌──────────────────┐
 │  RTX 4090           │      │  RTX 3080        │     │  T4              │
 │  Score: 120.5       │◄────►│  Score: 85.2     │◄───►│  Score: 12.0     │
 │                     │      │                  │     │                  │
 │  @accelerate(mesh)  │      │  receives task   │     │  receives task   │
 │  def train():       │      │  runs train()    │     │  runs train()    │
 └─────────────────────┘      └──────────────────┘     └──────────────────┘
          │
          ▼
    Results collected automatically
```

**Use cases:**
- Hyperparameter search across multiple GPUs
- Data preprocessing sharded across machines
- Model training on a pool of consumer GPUs
- Any embarrassingly parallel workload

---

## Features

### Transparent Acceleration

```python
@accelerate(mesh)
def train(lr, epochs):
    return {"accuracy": 0.95}

# Single call → best local device
result = train(lr=0.01, epochs=100)

# Batch call → spread across ALL mesh devices
results = train.map([
    {"lr": 0.01, "epochs": 100},
    {"lr": 0.05, "epochs": 200},
])
```

| Scenario | What happens |
|----------|-------------|
| Single call `func(x)` | Runs on best **local** device (CPU/GPU) |
| Batch call `func.map([...])` | Spreads across **all** mesh devices |
| Mesh unreachable | Falls back to **local** execution silently |
| `GPUMESH_LOCAL=1` | Forces local-only (no mesh) |
| `GPUMESH_VERBOSE=1` | Prints which device handled each task |

### Hardware Selection

Target specific GPU types:

```python
@accelerate(mesh, gpu="A100")
def train(model):
    return model.cuda().forward(x)
```

### Resource Specs

Declare what your task needs:

```python
@accelerate(mesh, cores=8, memory="16GB", timeout=300)
def heavy_computation(data):
    return processed
```

### Auto Device Placement

PyTorch models are automatically placed on the best device:

```python
@accelerate(mesh)
def train_model(model, data):
    # model is automatically moved to the best GPU
    return model(data)
```

### Fault Tolerance

- Dead workers are detected and tasks are re-queued
- Straggler workers are deprioritized
- Graceful fallback to local execution if mesh is unavailable
- Crash diagnostics on worker failures

### Smart Scheduling

- **Benchmark scoring** — each worker gets a 0-100 performance score
- **Memory-aware** — tasks routed to workers with enough VRAM
- **Straggler deprioritization** — slow workers get fewer tasks
- **TTL expiry** — stale workers are automatically pruned

### One-Line Setup

```bash
gpumesh setup    # Detects hardware, guides you through coordinator/worker
gpumesh quickjoin  # One-click: detect GPU and join mesh
```

---

## Installation

### Basic install

```bash
pip install gpumesh
```

### With optional extras

```bash
pip install gpumesh[gpu]       # GPU detection + CUDA benchmarks (requires torch)
pip install gpumesh[tunnel]    # ngrok for public URLs
pip install gpumesh[sysinfo]   # System info (psutil)
pip install gpumesh[notebook]  # DataFrame support (pandas)
pip install gpumesh[ui]        # Beautiful setup wizard (rich + questionary)
pip install gpumesh[all]       # Everything above
```

### Requirements

- **Python 3.9+**
- **cloudpickle** (automatically installed)
- Optional: **PyTorch** for GPU detection and CUDA benchmarks

---

## Quick Start

### Step 1: Start a coordinator (one machine)

```bash
gpumesh setup
```

The wizard will:
1. Detect your hardware (CPU cores, GPU model, VRAM)
2. Ask if you want to be a **coordinator** or **worker**
3. Generate a token and show connection info
4. Display a live radar of connected workers

Or start directly:

```bash
gpumesh serve --port 8000 --token mysecret
```

### Step 2: Join a worker (another machine)

```bash
gpumesh setup
```

Choose **Worker** and enter the coordinator's URL and token. Or:

```bash
gpumesh join http://coordinator-ip:8000 --token mysecret
```

Or use quickjoin (auto-detects GPU):

```bash
gpumesh quickjoin http://coordinator-ip:8000 --token mysecret
```

### Step 3: Use your mesh

**Option A: The `@accelerate` decorator (recommended)**

```python
from gpumesh import GPUMesh, accelerate

mesh = GPUMesh("http://coordinator:8000", token="mysecret")

@accelerate(mesh)
def train(lr, epochs):
    # Your code here — runs on all connected GPUs automatically
    return {"accuracy": 0.95}

# Single call → best local device
result = train(lr=0.01, epochs=100)

# Batch call → spread across all mesh devices
results = train.map([
    {"lr": 0.01, "epochs": 100},
    {"lr": 0.05, "epochs": 200},
])
```

**Option B: The Python API**

```python
from gpumesh import GPUMesh

mesh = GPUMesh("http://coordinator:8000", token="mysecret")

# List connected workers
workers = mesh.workers()
print(workers)
# [{'id': 'w1', 'device': 'cuda', 'device_name': 'RTX 3080', 'score': 85.0}]

# Distribute a function across all workers
results = mesh.distribute(
    function=train_model,
    params=[
        {"lr": 0.01, "epochs": 100},
        {"lr": 0.05, "epochs": 200},
    ],
)

# Convert to DataFrame (optional)
df = mesh.results_to_dataframe(results)
```

**Option C: CLI job submission**

```bash
# Submit a Python script with payloads
gpumesh submit train.py --payloads payloads.json --wait

# Check status
gpumesh status JOB_ID

# Cancel if needed
gpumesh cancel JOB_ID
```

---

## CLI Commands

### Server & Connection

| Command | Description |
|---------|-------------|
| `gpumesh setup` | Interactive setup wizard — detects hardware, guides coordinator/worker choice |
| `gpumesh serve [--port 8000] [--token SECRET]` | Start coordinator server |
| `gpumesh join URL [--token SECRET]` | Join mesh as a worker |
| `gpumesh quickjoin [URL] --token TOKEN` | One-click: detect GPU and join mesh |
| `gpumesh worker --token TOKEN` | Start a worker that broadcasts and waits to be claimed |
| `gpumesh radar` | Scan for nearby gpumesh devices on the network |
| `gpumesh show-connection` | Show saved URL and token (for sharing) |
| `gpumesh disconnect` | Clear saved connection |

### Job Management

| Command | Description |
|---------|-------------|
| `gpumesh submit SCRIPT --payloads FILE [--wait]` | Submit a Python script as a job |
| `gpumesh status JOB_ID` | Check job progress and results |
| `gpumesh cancel JOB_ID` | Cancel a running job |
| `gpumesh kill [--force]` | Kill all gpumesh tasks |

### Monitoring

| Command | Description |
|---------|-------------|
| `gpumesh workers` | List connected workers with status |
| `gpumesh devices` | Show all GPUs as one unified pool |

### Useful Flags

| Flag | Description |
|------|-------------|
| `--port PORT` | Server port (default: 8000) |
| `--token SECRET` | Authentication token |
| `--url URL` | Coordinator URL (or set `GPUMESH_URL` env var) |
| `--token SECRET` | Auth token (or set `GPUMESH_TOKEN` env var) |
| `--tailscale` | Use Tailscale for encrypted network access |
| `--no-discovery` | Disable UDP auto-discovery |
| `--timeout SECONDS` | Per-task timeout (default: 240) |
| `--wait` | Wait for job completion after submit |

---

## Python API

### GPUMesh Client

```python
from gpumesh import GPUMesh

mesh = GPUMesh("http://coordinator:8000", token="mysecret")
```

### Listing Workers

```python
workers = mesh.workers()
# [
#     {'id': 'w1', 'device': 'cuda', 'device_name': 'RTX 3080', 'hostname': 'laptop-a', 'score': 85.0, 'alive': True},
#     {'id': 'w2', 'device': 'cuda', 'device_name': 'T4', 'hostname': 'server-b', 'score': 12.0, 'alive': True},
# ]
```

### Distributing Functions

```python
def train_model(lr, epochs):
    # Your training code
    return {"accuracy": 0.95, "lr": lr}

results = mesh.distribute(
    function=train_model,
    params=[
        {"lr": 0.01, "epochs": 100},
        {"lr": 0.05, "epochs": 200},
        {"lr": 0.1, "epochs": 300},
    ],
    timeout=600,  # optional, default 300s
)
# [{'accuracy': 0.95, 'lr': 0.01}, {'accuracy': 0.95, 'lr': 0.05}, ...]
```

### Device Access

```python
# List all devices (local + remote)
devices = mesh.devices()
# [{'index': 0, 'hostname': 'laptop-a', 'device': 'cuda', 'device_name': 'RTX 3080', 'score': 85.2, 'status': 'alive'}]

# Count GPUs
count = mesh.device_count()  # 2

# Total compute score
total = mesh.total_score()  # 205.7

# Auto-pick best device
best = mesh.auto_device()
```

### Job Management

```python
# Submit a raw job
job_id = mesh.submit(name="preprocess", script="process.py", payloads=[{"file": "data.csv"}])

# Check status
status = mesh.status(job_id)
# {'finished': False, 'counts': {'pending': 0, 'running': 2, 'done': 1}, 'tasks': [...]}

# Convert results to DataFrame
df = mesh.results_to_dataframe(results)
```

### Starting Workers from Python

```python
# Start a coordinator (non-blocking, runs in background thread)
GPUMesh.start_coordinator(port=8000, token="mysecret")

# Join as a worker (non-blocking)
info = GPUMesh.add_worker("http://coordinator:8000", token="mysecret")
```

---

## @accelerate Decorator

The `@accelerate` decorator makes your mesh resources transparent to your code.

### Pattern 1: Basic Usage

```python
from gpumesh import GPUMesh, accelerate

mesh = GPUMesh("http://coordinator:8000", token="mysecret")

@accelerate(mesh)
def preprocess(chunk_id, data_path):
    import pandas as pd
    df = pd.read_parquet(data_path)
    return {"chunk": chunk_id, "rows": len(df)}

# Single call → runs locally on best device
result = preprocess(chunk_id=0, data_path="data.parquet")

# Batch call → spreads across ALL mesh devices
results = preprocess.map([
    {"chunk_id": 0, "data_path": "part0.parquet"},
    {"chunk_id": 1, "data_path": "part1.parquet"},
])
```

### Pattern 2: Hardware Selection

```python
@accelerate(mesh, gpu="A100")
def train(model):
    return model.cuda().forward(x)
```

### Pattern 3: Resource Specs

```python
@accelerate(mesh, cores=8, memory="16GB", timeout=300)
def heavy_computation(data):
    return processed
```

### Pattern 4: Global Install (Import Hook)

```python
from gpumesh import GPUMesh, accelerate

mesh = GPUMesh("http://coordinator:8000", token="mysecret")
accelerate.install(mesh)  # Set global mesh

# Now all @accelerate functions auto-use the mesh
@accelerate  # No parentheses needed
def train(lr, epochs):
    return {"accuracy": 0.95}
```

### Pattern 5: Binding to a Device

```python
@accelerate(mesh)
def predict(x):
    return model(x)

# Bind to a specific device
gpu_predict = predict.to("cuda")
result = gpu_predict(x)
```

### Pattern 6: Mesh Fallback

```python
# If the mesh is unreachable, @accelerate falls back to local execution
@accelerate(mesh)
def train(lr, epochs):
    return {"accuracy": 0.95}

# This works even if the coordinator is down — runs locally
result = train(lr=0.01, epochs=100)
```

---

## Network Options

| Method | Setup | Best For | Encrypted |
|--------|-------|----------|-----------|
| **LAN** | None | Same Wi-Fi, fastest | No |
| **Tailscale** | Install Tailscale | Remote teams, encrypted | Yes |
| **ngrok** | `pip install gpumesh[tunnel]` | Public access, demos | Yes |

### LAN (Default)

No setup required. Workers discover the coordinator automatically on the same network via UDP broadcast.

```bash
# Coordinator
gpumesh serve --port 8000

# Worker (on another machine)
gpumesh join http://192.168.1.10:8000 --token mysecret
```

### Tailscale

Encrypted tunnel across the internet. Both machines need Tailscale installed.

```bash
# Coordinator
gpumesh serve --port 8000 --tailscale

# Worker
gpumesh join http://tailscale-ip:8000 --token mysecret
```

### ngrok

Public URL for demos and testing.

```bash
# Coordinator
gpumesh serve --port 8000 --public
# Prints: ngrok tunnel → https://abc123.ngrok.io

# Worker
gpumesh join https://abc123.ngrok.io --token mysecret
```

---

## How It Works

### Architecture

```
┌─────────────────────────────────────────────────────────────┐
│                      Coordinator                             │
│                                                              │
│  ┌──────────┐  ┌──────────┐  ┌──────────┐  ┌──────────┐    │
│  │ Job Queue │  │ Task DB  │  │ Workers  │  │ Events   │    │
│  │          │  │ (SQLite) │  │ Registry │  │ Log      │    │
│  └────┬─────┘  └──────────┘  └────┬─────┘  └──────────┘    │
│       │                           │                          │
│       └───────────┬───────────────┘                          │
│                   │                                          │
│          HTTP API (port 8000)                                │
└───────────────────┼──────────────────────────────────────────┘
                    │
        ┌───────────┼───────────┐
        │           │           │
   ┌────▼────┐ ┌────▼────┐ ┌────▼────┐
   │ Worker  │ │ Worker  │ │ Worker  │
   │ RTX4090 │ │ RTX3080 │ │ T4      │
   │ Score:  │ │ Score:  │ │ Score:  │
   │ 120.5   │ │ 85.2    │ │ 12.0    │
   └─────────┘ └─────────┘ └─────────┘
```

### Job Flow

1. **Submit** — Client sends a job (Python script + payloads) to the coordinator
2. **Queue** — Coordinator splits payloads into tasks and stores them in SQLite
3. **Claim** — Workers pull tasks based on their benchmark score and available memory
4. **Execute** — Each task runs in an isolated subprocess on the worker
5. **Report** — Workers report results back to the coordinator
6. **Collect** — Client polls for results or waits for completion

### Benchmark Scoring

Each worker runs a benchmark on join and gets a 0-100 score:

| Score Range | Typical GPU | Use Case |
|-------------|-------------|----------|
| 80-100 | RTX 4090, A100 | Heavy training, large models |
| 50-80 | RTX 3080, RTX 3090 | Medium training, inference |
| 20-50 | RTX 3060, T4 | Light tasks, preprocessing |
| 0-20 | CPU only | Very light tasks |

---

## Security

### Token Authentication

All communication is authenticated with a shared token:

```bash
# Coordinator generates a token
gpumesh serve --token mysecret

# Workers must provide the same token
gpumesh join http://coordinator:8000 --token mysecret
```

### Security Model

| Feature | Status |
|---------|--------|
| Token authentication | All API requests |
| Rate limiting | 5 failed attempts, then blocked |
| Process isolation | Tasks run in separate subprocesses |
| File permissions | Tokens stored with restricted permissions (0o600) |

### Important

> **Workers execute code from the coordinator.** Only share your URL and token with people you trust. gpumesh is designed for **trusted networks** (home labs, team clusters).

- **Code execution**: Function tasks (`@accelerate`) execute in the worker's process
- **Plaintext HTTP**: All communication uses HTTP. Use Tailscale for encrypted tunnels
- **No sandbox**: Tasks have full access to the worker machine
- **Token secret**: Keep it secret. Rotate if compromised

---

## Troubleshooting

| Problem | Fix |
|---------|-----|
| `command not found: gpumesh` | Use `python -m gpumesh` instead, or check your PATH |
| `401 bad token` | Ensure coordinator and worker use the same token |
| `coordinator unreachable` | Check firewall rules and that coordinator is running |
| `task timed out` | Increase `--timeout 600` or split into smaller tasks |
| `No connection could be made` | Windows: run `gpumesh serve` as Administrator for firewall rules |
| Worker not showing on coordinator | Check both machines are on the same network; try `gpumesh radar` |
| `ModuleNotFoundError: torch` | Install GPU extras: `pip install gpumesh[gpu]` |
| Worker crashes on submit | Check worker logs; increase `--timeout` for long tasks |
| UDP broadcast not working | Use direct `gpumesh join URL` instead of auto-discovery |

### Enable Verbose Logging

```bash
GPUMESH_VERBOSE=1 gpumesh serve
GPUMESH_VERBOSE=1 gpumesh join http://coordinator:8000 --token mysecret
```

### Force Local-Only Mode

```bash
GPUMESH_LOCAL=1 python my_script.py
```

---

## Limitations

- **Python only** — Tasks must be Python functions or scripts
- **No GPU memory sharing** — Each task gets its own process
- **No model sharding** — Each task runs on one machine at a time
- **Single coordinator** — Single point of failure (use Tailscale for reliability)
- **No built-in encryption** — Use Tailscale for encrypted tunnels

---

## Development

### Contributing

```bash
git clone https://github.com/Samurai007AK/gpumesh.git
cd gpumesh
pip install -e ".[dev]"
pytest
```

### Running Tests

```bash
pytest                    # Run all tests
pytest tests/test_api.py  # Run specific test file
pytest -v                 # Verbose output
```

### Building

```bash
python -m build
twine check dist/*
```

---

## License

MIT License. See [LICENSE](LICENSE) for details.
