Metadata-Version: 2.4
Name: distensor
Version: 0.1.1
Summary: Decentralized distributed training with explicit 4D parallelism control
Project-URL: Homepage, https://github.com/distensor/distensor
Project-URL: Documentation, https://distensor.dev/docs
Project-URL: Repository, https://github.com/distensor/distensor
Project-URL: Issues, https://github.com/distensor/distensor/issues
Author: disTensor Team
License-Expression: Apache-2.0
License-File: LICENSE
Keywords: deep-learning,distributed,machine-learning,parallelism,pytorch,training
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Typing :: Typed
Requires-Python: >=3.10
Requires-Dist: grpcio>=1.60
Requires-Dist: numpy>=1.24
Requires-Dist: protobuf>=4.25
Requires-Dist: safetensors>=0.4
Requires-Dist: torch>=2.0
Provides-Extra: cloud
Requires-Dist: alembic>=1.13; extra == 'cloud'
Requires-Dist: asyncpg>=0.29; extra == 'cloud'
Requires-Dist: authlib>=1.3; extra == 'cloud'
Requires-Dist: fastapi>=0.109; extra == 'cloud'
Requires-Dist: httpx>=0.27; extra == 'cloud'
Requires-Dist: opentelemetry-api>=1.22; extra == 'cloud'
Requires-Dist: opentelemetry-exporter-otlp>=1.22; extra == 'cloud'
Requires-Dist: opentelemetry-instrumentation-fastapi>=0.43b0; extra == 'cloud'
Requires-Dist: opentelemetry-instrumentation-grpc>=0.43b0; extra == 'cloud'
Requires-Dist: opentelemetry-sdk>=1.22; extra == 'cloud'
Requires-Dist: prometheus-client>=0.19; extra == 'cloud'
Requires-Dist: psycopg2-binary>=2.9; extra == 'cloud'
Requires-Dist: pydantic>=2.0; extra == 'cloud'
Requires-Dist: python-multipart>=0.0.6; extra == 'cloud'
Requires-Dist: redis>=5.0; extra == 'cloud'
Requires-Dist: sqlalchemy>=2.0; extra == 'cloud'
Requires-Dist: uvicorn>=0.27; extra == 'cloud'
Requires-Dist: websockets>=12.0; extra == 'cloud'
Provides-Extra: dev
Requires-Dist: absl-py>=2.0; extra == 'dev'
Requires-Dist: commitizen>=4.1; extra == 'dev'
Requires-Dist: docker>=7.0; extra == 'dev'
Requires-Dist: grpcio-tools>=1.60; extra == 'dev'
Requires-Dist: mypy-protobuf>=3.0; extra == 'dev'
Requires-Dist: mypy>=1.8; extra == 'dev'
Requires-Dist: myst-parser>=2.0; extra == 'dev'
Requires-Dist: pre-commit>=4.0; extra == 'dev'
Requires-Dist: pydata-sphinx-theme>=0.15; extra == 'dev'
Requires-Dist: pytest>=8.0; extra == 'dev'
Requires-Dist: ruff>=0.8; extra == 'dev'
Requires-Dist: sphinx-design>=0.5; extra == 'dev'
Requires-Dist: sphinxcontrib-mermaid>=0.9; extra == 'dev'
Provides-Extra: trainer
Provides-Extra: worker
Requires-Dist: psutil>=5.9; extra == 'worker'
Provides-Extra: worker-cuda
Requires-Dist: psutil>=5.9; extra == 'worker-cuda'
Provides-Extra: worker-mlx
Requires-Dist: mlx>=0.16; (platform_machine == 'arm64' and sys_platform == 'darwin') and extra == 'worker-mlx'
Requires-Dist: psutil>=5.9; extra == 'worker-mlx'
Description-Content-Type: text/markdown

# disTensor

[![Python 3.10+](https://img.shields.io/badge/python-3.10%2B-blue.svg)](https://www.python.org/downloads/)
[![License](https://img.shields.io/badge/license-Apache%202.0-green.svg)](LICENSE)
[![Version](https://img.shields.io/badge/version-0.1.0-orange.svg)](https://github.com/distensor/distensor)

Decentralized distributed training with explicit 4D parallelism control.

## What is disTensor?

disTensor is a distributed training platform that lets you train large models across heterogeneous consumer compute. You define parallelism strategy explicitly — data, tensor, pipeline, and context parallel dimensions — and disTensor handles orchestration, synchronization, and fault tolerance while preserving familiar PyTorch semantics.

The system uses a driver-mediated architecture: your process owns the training loop and data pipeline, the hub coordinates topology, and workers execute computation and sync directly via P2P. The control plane (gRPC) handles orchestration; the data plane (P2P) handles tensor transfers on the critical path.

## Installation

```bash
# Trainer SDK
pip install distensor

# Compute provider — macOS Apple Silicon
DISTENSOR_API_KEY=dt_p_... curl -fsSL https://api.distensor.com/install/macos.sh | bash

# Development (editable, from source)
make dev
```

## Quick Start

```python
import distensor as ds

# Local mode — no hub, no GPUs, simulated mesh
ds.init(mode="local")

mesh = ds.query_mesh()
print(f"Available workers: {mesh.available_workers}")

# Cloud mode — requires API key
ds.reset()
ds.init(mode="cloud", api_key="dt_u_...")
```

## Architecture

disTensor uses a hub-and-spoke model where the hub coordinates topology but stays off the critical training path. Workers sync directly via P2P for low-latency tensor transfers. The cloud API layer adds persistence, auth, billing, and a web console on top.

| Component | Package | Description |
|-----------|---------|-------------|
| **Hub** | `distensor.hub` | Central control plane — node/job/topology registries, gRPC services (mesh, job, training, checkpoint, sync) |
| **Cloud API** | `distensor.cloud` | Production HTTP layer — FastAPI, PostgreSQL, Redis, Prometheus. Auth, billing, worker registry, telemetry |
| **Node** | `distensor.node` | Compute worker daemon — model shard execution, P2P data plane, CUDA/MPS/CPU backends |
| **Trainer** | `distensor.training` | Orchestration — `Trainer.fit()`, pipeline scheduling (GPipe, 1F1B), shard loading, execution plans |
| **Sync** | `distensor.sync` | 4D parallel coordination — TP AllReduce, PP transfers, DP gradient sync, worker topology |
| **Optimizer** | `distensor.optim` | Distributed optimizer — ZeRO-3, gradient sync with bucketing, LR schedulers, state dict support |
| **Console** | `ui/` | React web UI for trainers, providers, and admins |
| **Node App** | `distributions/` | Provider surface — `distensor-server` supervisor + `distensor` TUI dashboard |

See [Architecture.md](Architecture.md) for diagrams, execution flows, and design principles. Full documentation at [docs.distensor.com](https://docs.distensor.com).

## Parallelism

| Mode | Abbreviation | Strategy |
|------|:---:|----------|
| Data Parallel | DP | Replicate model across workers, split data batches |
| Tensor Parallel | TP | Shard individual layers across workers |
| Pipeline Parallel | PP | Split model into sequential stages across workers |
| Context Parallel | CP | Split sequence dimension across workers (coming soon) |

## Development

```bash
make dev                  # Full bootstrap (uv sync + proto generation)
make test                 # Run full test suite
make lint                 # Ruff linting
make format               # Auto-format

make cloud-up             # Start Postgres, Redis, MinIO, Prometheus
make cloud-api            # Run HTTP API server (:8000)
make cloud-hub            # Run gRPC hub server (:50051)

make console-dev          # Vite dev server (:5173, proxies /api to :8000)

make sandbox-up           # Docker sandbox (coordinator + 8 workers)
make sandbox-cloud-up     # Full cloud stack in Docker

make help                 # All available targets
```

## Links

- [Documentation](https://docs.distensor.com)
- [Architecture](Architecture.md)
- [Contributing](CONTRIBUTING.md)
- [License](LICENSE) — Apache 2.0
