Metadata-Version: 2.4
Name: distributed-llm
Version: 0.4.1
Summary: Distributed LLM inference — pool GPUs across multiple devices to run models no single machine can handle
Author: Distributed LLM Contributors
License: Apache-2.0
Project-URL: Homepage, https://github.com/distributed-llm/distributed-llm
Project-URL: Documentation, https://github.com/distributed-llm/distributed-llm#readme
Project-URL: Repository, https://github.com/distributed-llm/distributed-llm
Project-URL: Issues, https://github.com/distributed-llm/distributed-llm/issues
Keywords: llm,inference,distributed,pipeline-parallelism,gpu-sharing,peer-to-peer,open-source
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: System :: Distributed Computing
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: fastapi>=0.109.0
Requires-Dist: uvicorn>=0.27.0
Requires-Dist: pydantic>=2.5.0
Requires-Dist: pydantic-settings>=2.0.0
Requires-Dist: httpx>=0.27
Requires-Dist: pyyaml>=6.0
Requires-Dist: loguru>=0.7.0
Requires-Dist: pyjwt>=2.0.0
Requires-Dist: numpy>=1.26.0
Requires-Dist: psutil>=5.9.0
Provides-Extra: observability
Requires-Dist: prometheus-client>=0.20; extra == "observability"
Requires-Dist: opentelemetry-api>=1.21.0; extra == "observability"
Requires-Dist: opentelemetry-sdk>=1.21.0; extra == "observability"
Requires-Dist: opentelemetry-exporter-prometheus>=0.42b0; extra == "observability"
Provides-Extra: self-hosted
Requires-Dist: torch>=2.5.0; extra == "self-hosted"
Requires-Dist: transformers>=4.48.0; extra == "self-hosted"
Requires-Dist: accelerate>=1.0.0; extra == "self-hosted"
Requires-Dist: safetensors>=0.4.0; extra == "self-hosted"
Requires-Dist: huggingface-hub>=0.20.0; extra == "self-hosted"
Requires-Dist: sentencepiece>=0.1.99; extra == "self-hosted"
Requires-Dist: nvidia-ml-py>=12.0; extra == "self-hosted"
Provides-Extra: ray
Requires-Dist: ray[default]>=2.30.0; extra == "ray"
Provides-Extra: backends
Requires-Dist: vllm<0.9.0,>=0.7.0; extra == "backends"
Requires-Dist: sglang<0.5.0,>=0.4.0; extra == "backends"
Requires-Dist: llama-cpp-python>=0.3.0; extra == "backends"
Provides-Extra: dev
Requires-Dist: pytest>=7.0; extra == "dev"
Requires-Dist: pytest-asyncio>=0.23; extra == "dev"
Requires-Dist: hypothesis>=6.90; extra == "dev"
Requires-Dist: pytest-benchmark>=4.0; extra == "dev"
Requires-Dist: black>=24.0; extra == "dev"
Requires-Dist: ruff>=0.1.0; extra == "dev"
Requires-Dist: mypy>=1.8; extra == "dev"
Requires-Dist: opentelemetry-test-utils>=0.42b0; extra == "dev"
Requires-Dist: bandit[toml]>=1.7; extra == "dev"
Requires-Dist: detect-secrets>=1.5; extra == "dev"
Provides-Extra: sdk
Requires-Dist: httpx>=0.27.0; extra == "sdk"
Provides-Extra: cli
Requires-Dist: typer[all]>=0.9.0; extra == "cli"
Requires-Dist: rich>=13.7.0; extra == "cli"
Requires-Dist: prompt_toolkit>=3.0.0; extra == "cli"
Provides-Extra: ui
Requires-Dist: jinja2>=3.1.0; extra == "ui"
Requires-Dist: python-multipart>=0.0.6; extra == "ui"
Requires-Dist: qrcode[pil]>=7.4; extra == "ui"
Provides-Extra: examples
Requires-Dist: langchain-openai>=0.1.0; extra == "examples"
Requires-Dist: langchain-core>=0.1.0; extra == "examples"
Requires-Dist: llama-index-llms-openai>=0.1.0; extra == "examples"
Requires-Dist: llama-index-core>=0.10.0; extra == "examples"
Requires-Dist: crewai>=0.30.0; extra == "examples"
Provides-Extra: security
Requires-Dist: cryptography>=41.0; extra == "security"
Requires-Dist: pynacl>=1.5.0; extra == "security"
Provides-Extra: moderation
Requires-Dist: transformers>=4.36.0; extra == "moderation"
Requires-Dist: onnxruntime>=1.16.0; extra == "moderation"
Provides-Extra: mlflow
Requires-Dist: mlflow>=2.0.0; extra == "mlflow"
Provides-Extra: monitoring
Requires-Dist: opentelemetry-exporter-otlp-proto-grpc>=1.21.0; extra == "monitoring"
Requires-Dist: opentelemetry-proto>=1.21.0; extra == "monitoring"
Requires-Dist: nvidia-ml-py>=12.0; extra == "monitoring"
Requires-Dist: psutil>=5.9.0; extra == "monitoring"
Provides-Extra: model-extras
Requires-Dist: bitsandbytes>=0.41.0; extra == "model-extras"
Requires-Dist: peft>=0.14.0; extra == "model-extras"
Provides-Extra: wan
Requires-Dist: aioquic>=1.0.0; extra == "wan"
Provides-Extra: wandb
Requires-Dist: wandb>=0.18; extra == "wandb"
Requires-Dist: nvidia-ml-py>=12.0; extra == "wandb"
Provides-Extra: spark
Requires-Dist: pyspark>=3.4.0; extra == "spark"
Provides-Extra: spark-mlflow
Requires-Dist: distributed-llm[mlflow,spark]; extra == "spark-mlflow"
Provides-Extra: openai-agents
Requires-Dist: openai-agents>=0.1.0; extra == "openai-agents"
Requires-Dist: openai>=1.0.0; extra == "openai-agents"
Provides-Extra: agno
Requires-Dist: agno>=1.0.0; extra == "agno"
Provides-Extra: autogpt
Requires-Dist: autogpt>=0.4.0; extra == "autogpt"
Provides-Extra: vector-db-pinecone
Requires-Dist: pinecone>=3.0.0; extra == "vector-db-pinecone"
Provides-Extra: vector-db-qdrant
Requires-Dist: qdrant-client>=1.9.0; extra == "vector-db-qdrant"
Provides-Extra: vector-db-weaviate
Requires-Dist: weaviate-client>=4.0.0; extra == "vector-db-weaviate"
Provides-Extra: vector-db-milvus
Requires-Dist: pymilvus>=2.3.0; extra == "vector-db-milvus"
Provides-Extra: vector-db-all
Requires-Dist: distributed-llm[vector-db-milvus,vector-db-pinecone,vector-db-qdrant,vector-db-weaviate]; extra == "vector-db-all"
Provides-Extra: spot
Requires-Dist: httpx>=0.27; extra == "spot"
Provides-Extra: all
Requires-Dist: distributed-llm[agno,autogpt,backends,cli,examples,mlflow,observability,openai-agents,ray,sdk,security,self-hosted,spark,spot,ui,wan]; extra == "all"
Requires-Dist: distributed-llm[model-extras]; extra == "all"
Provides-Extra: all-dev
Requires-Dist: distributed-llm[all,dev]; extra == "all-dev"
Requires-Dist: locust>=2.20; extra == "all-dev"
Requires-Dist: safety>=3.0; extra == "all-dev"
Dynamic: license-file

# DistLLM — Distributed Inference Across All Your Devices

**Pool GPUs from every device you own to run models no single machine can handle.**

[![License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](LICENSE)
[![Python 3.10+](https://img.shields.io/badge/python-3.10+-blue.svg)](https://www.python.org/downloads/)

You have a gaming PC with an RTX 4090. Your laptop has an RTX 4060. Your friend has a desktop with an RTX 3080. None of you can run Llama 3.1 70B alone. **Together, you can.**

DistLLM splits large language models across all your devices using pipeline parallelism. Each device runs a fraction of the model layers. Automatic discovery. Auto-partitioning. Works over LAN, WiFi, or internet.

```
Your Laptop (RTX 4060) ────┐
                            │
Your Gaming PC (RTX 4090) ──┼──► DistLLM Cluster ──► Run 70B models
                            │
Friend's PC (RTX 3080) ────┘
```

## Why DistLLM?

| Problem | Solution |
|---------|----------|
| One GPU can't run today's best models | Split across all your devices |
| Cloud inference costs thousands/month | Use the GPUs you already own |
| Data privacy concerns with cloud APIs | Your data stays on your devices |
| Slow single-device inference | Pipeline parallelism = faster generation |
| Setting up distributed systems is hard | One command to start, one to join |

## Quick Start

```bash
pip install distllm

# On your main machine — start a cluster
distllm cluster start --model meta-llama/Llama-3.2-7B

# On every other machine — join the cluster
distllm cluster join
```

## How It Works

DistLLM uses **pipeline parallelism** — the model is split across devices by layers:

```
Device 1 (Laptop): Layers 0-5 ──→ Device 2 (Desktop): Layers 6-11 ──→ Device 3 (Friend's PC): Layers 12-17

Each device runs ~6 layers → fits in 6-8GB VRAM
Combined pool → runs models up to 70B parameters
```

Key capabilities:
- **Auto-discovery**: devices find each other on the same network automatically
- **Auto-partitioning**: automatically assigns layers based on each device's GPU
- **Node recovery**: if a device disconnects, remaining nodes take over
- **Straggler detection**: slow nodes are detected and worked around
- **WAN optimization**: token accumulation for low-latency cross-internet inference
- **Privacy-first**: keep sensitive layers on your own devices

## Installation

```bash
# Core package
pip install distllm

# With vLLM backend (recommended for NVIDIA GPUs)
pip install "distllm[vllm]"

# With llama.cpp backend (CPU, AMD, Apple Silicon)
pip install "distllm[llamacpp]"

# Development
pip install -e ".[dev]"
```

## Key Features

- **Pipeline parallelism** — split any HuggingFace model across N devices
- **Auto-discovery** — mDNS/zeroconf device finding on LAN
- **6 backends** — vLLM, llama.cpp, TensorRT-LLM, ExLlamaV2, ONNX, PyTorch
- **Auth plugin** — JWT authentication + RBAC role-based access control
- **Health watchdog** — Continuous node health monitoring with auto circuit-breaking
- **Semantic caching** — Deduplicate repeated prompts with embedding similarity
- **Token streaming** — `generate_stream()` for real-time token-by-token responses
- **Config validation** — Cross-field validation catches invalid combinations at load time
- **Circuit breaker** — Graduated backpressure for load shedding
- **Auto-partitioning** — hardware-aware DP solver for optimal layer assignment
- **Node recovery** — checkpoint-based recovery when nodes disconnect
- **Straggler detection** — statistical outlier detection for slow nodes
- **Dynamic rebalancing** — redistribute layers when nodes join/leave
- **P2P KV cache gossip** — CRDT-based cache sharing between nodes
- **Wide-area support** — token accumulation for internet-scale inference
- **Quantization** — 4-bit/8-bit to fit larger models on consumer GPUs
- **OpenAI-compatible API** — use any OpenAI client to send requests
- **Full observability** — Prometheus metrics, OTel tracing, structured logging
- **Interactive chat** — `distllm chat` for CLI-based interaction
- **Auth plugin** — JWT authentication with RBAC role-based access control
- **Health watchdog** — continuous node health monitoring with automatic failover
- **Semantic cache** — caching plugin with deduplication for repeated prompts
- **Token streaming** — `generate_stream()` SDK method for token-by-token responses
- **Health endpoints** — `/healthz` (liveness) and `/readyz` (readiness) for Kubernetes probes

## CLI Commands

```bash
# Start a coordinator node
distllm-coordinator --model meta-llama/Llama-3.2-1B --local --chat

# Start distributed coordinator
distllm-coordinator --model meta-llama/Llama-3.2-7B \
  --nodes laptop:50051:0:5 desktop:50052:6:11 friend:50053:12:17

# Start a worker node
distllm-node --node-id laptop --model meta-llama/Llama-3.2-7B \
  --start-layer 0 --end-layer 5 --total-layers 18 \
  --coordinator-host 192.168.1.100 --coordinator-port 50050

# Start the REST API server
distllm-api --model meta-llama/Llama-3.2-1B --local

# System diagnostics (check Python, CUDA, GPU, network, ports)
distllm doctor

# Quantize a model for smaller footprint
distllm tune quantize --model meta-llama/Llama-3.2-7B --bits 4

# Batch inference over a dataset
distllm tune batch --input prompts.jsonl --output results.jsonl

# Warm the semantic cache from prior prompts
distllm tune cache --preload cache_seed.jsonl
```

## Documentation

- [Architecture](docs/architecture.md) — Pipeline parallelism, node topology, KV cache management
- [Deployment](DEPLOYMENT.md) — Local, Docker, and multi-machine deployment
- [API Reference](docs/api.md) — OpenAI-compatible API docs

## Project Structure

```
src/distllm/
├── core/              # Coordinator, request pipeline, batch scheduler
├── dist/              # Distributed inference engine
│   ├── pipeline.py    # PipelineOrchestrator — multi-node execution
│   ├── worker.py      # WorkerNode — per-device model subset
│   ├── recovery.py    # NodeRecoveryManager — failure handling
│   ├── straggler.py   # StragglerDetector — slow node detection
│   ├── rebalancer.py  # Dynamic pipeline rebalancing
│   ├── wide_area.py   # WAN-optimized inference
│   ├── parallel.py    # Hybrid parallelism auto-selector
│   ├── p2p/           # P2P gossip protocol & discovery
│   └── partition/     # Hardware-aware auto-partitioner
├── api/               # OpenAI-compatible REST API
├── models/            # Model partitioning & loading
├── cli/               # CLI tool
├── sdk/               # Python client
├── observability/     # Metrics, tracing, logging
└── dashboard/         # Web dashboard
```

## Supported Model Architectures

GPT-2, GPT-Neo, Llama 2/3, Mistral, Mixtral, Qwen2.5, Phi, DeepSeek, StableLM, Pythia, Baichuan, ChatGLM, InternLM, and more via HuggingFace AutoModel.

## Roadmap

- **Current**: Pipeline parallelism across LAN devices, manual node configuration
- **Q3 2026**: Auto-discovery (mDNS), auto-partitioning, node recovery, GUI dashboard
- **Q4 2026**: NAT traversal (cross-internet), P2P model distribution, GPU reputation system
- **2027**: Federated clusters, speculative parallelism, privacy-preserving split, incentive system

## Contributing

See [CONTRIBUTING.md](CONTRIBUTING.md).

## License

Apache 2.0. See [LICENSE](LICENSE).
