Metadata-Version: 2.4
Name: sixtytwo-cli
Version: 0.3.10
Summary: Sixtytwo CLI: `sixtytwo rent` reserves reliability-backed GPUs; `sixtytwo` qualifies, monitors, and NCCL-benchmarks your own GPU clusters, with Slurm/SkyPilot integration.
Author: Sixtytwo, Inc.
License-Expression: LicenseRef-Sixtytwo-Commercial
Keywords: gpu,nccl,skypilot,slurm,cluster,benchmarking
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Operating System :: POSIX :: Linux
Classifier: Operating System :: MacOS
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: System :: Distributed Computing
Classifier: Topic :: System :: Monitoring
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: rich<15,>=14.0
Requires-Dist: fastapi<1,>=0.115
Provides-Extra: server
Requires-Dist: fastapi<1,>=0.115; extra == "server"
Requires-Dist: uvicorn<1,>=0.34; extra == "server"
Requires-Dist: geoip2<6,>=4.8; extra == "server"
Requires-Dist: cryptography<46,>=43; extra == "server"
Provides-Extra: sentry
Requires-Dist: sentry-sdk[fastapi]<3,>=2.18; extra == "sentry"
Provides-Extra: serverless
Requires-Dist: runpod<2,>=1.7; extra == "serverless"
Requires-Dist: modal<2,>=1.0; extra == "serverless"
Provides-Extra: gpu
Requires-Dist: nvidia-ml-py3>=7.352.0; extra == "gpu"
Requires-Dist: torch>=2.11; extra == "gpu"
Provides-Extra: skypilot
Requires-Dist: skypilot<0.12,>=0.9.0; extra == "skypilot"
Provides-Extra: skypilot-aws
Requires-Dist: skypilot[aws]<0.12,>=0.9.0; extra == "skypilot-aws"
Provides-Extra: skypilot-gcp
Requires-Dist: skypilot[gcp]<0.12,>=0.9.0; extra == "skypilot-gcp"
Provides-Extra: skypilot-lambda
Requires-Dist: skypilot[lambda]<0.12,>=0.9.0; extra == "skypilot-lambda"
Provides-Extra: skypilot-runpod
Requires-Dist: skypilot[runpod]<0.12,>=0.9.0; extra == "skypilot-runpod"
Dynamic: license-file

# sixtytwo

`sixtytwo-cli` is the command-line client for [sixtytwo](https://sixtytwo.ai),
built for the terminal and for coding agents. It covers two jobs:

- **Rent reliability-backed GPUs.** `sixtytwo rent` reserves GPUs that sixtytwo
  has already qualified, watches them while you run, and refunds the unused time
  if hardware goes bad.
- **Qualify and monitor your own clusters.** `sixtytwo` runs hardware checks,
  NCCL benchmarks, straggler detection, and live fault monitoring on GPUs you
  already have, and keeps a per-node trust score from every test, fault, and
  recovery. NVIDIA and AMD Instinct (MI300X family) are both supported: AMD
  nodes are checked through rocm-smi and amd-smi, validated against real
  MI300X hardware.

`sixtytwo rent` stays light (no torch), so it runs anywhere. The qualification
and monitoring commands pull in the GPU stack only when you ask for it.

## Install

```bash
pip install sixtytwo-cli
```

Requirements: Python 3.11 or newer; macOS (Apple silicon), Linux x86_64/aarch64, or Windows.

Optional extras:

```bash
pip install 'sixtytwo-cli[gpu]'        # nvidia-ml-py + torch for local GPU checks
pip install 'sixtytwo-cli[skypilot]'   # SkyPilot cloud adapter (see below)
pip install 'sixtytwo-cli[server]'     # local dashboard + collector daemon
```

## Rent GPUs

```bash
sixtytwo rent catalog                  # list SKUs and prices
sixtytwo rent H100_SXM -n 8 -H 4       # reserve 8x H100 for 4 hours, wait for READY
sixtytwo rent ls                       # list my reservations
sixtytwo rent status <reservation-id>
sixtytwo rent ssh <reservation-id>     # open a shell on a READY reservation
```

Every node is validated before you get it and monitored while you use it. Add
`--region us|eu|asia` to pin a geography; billing follows actual usage.
Prefer the browser? The live storefront is at [sixtytwo.ai](https://sixtytwo.ai).

## Qualify and monitor your own GPUs

These commands read a `sixtytwo.yaml`, so run `init` once first. It detects this
machine (and your scheduler, if any) and writes the config into the current
directory; everything below loads it.

```bash
sixtytwo init                          # writes ./sixtytwo.yaml (run this first)
sixtytwo doctor                        # validate this box: nvidia-smi or rocm-smi, dcgmi, topology
sixtytwo doctor --all-nodes --json     # validate every node in the cluster
sixtytwo test --quick --all            # fast per-node qualification
sixtytwo test --full gpu-01,gpu-02     # full suite, including NCCL
sixtytwo launch --pre-check python train.py   # qualify, then run with live monitoring
sixtytwo nodes                         # trust scores and fault history
```

`doctor --all-nodes` reports each node's GPU inventory, not just the local
host's. Results land in a local trust registry, so a node's history (tests,
faults, recoveries) follows it over time.

Straggler detection runs at both stages. `test --full` flags a GPU that is
slower than its node peers (an identical kernel timed per GPU) or below its
model's reference baseline, before you commit a run to it. During a run,
`launch` watches per-rank step times and flags a rank that falls behind the
others, so a GPU that degrades into a straggler mid-job is caught too.

### Slurm clusters

Run `init` on the login node (or inside an allocation). It detects Slurm, fills
the node list from `sinfo`/`scontrol`, and sets `remote.mode: slurm`, so `test`
and `doctor --all-nodes` dispatch each node's checks through `srun`. Confirm the
GPU partition and per-node GPU count in `sixtytwo.yaml` (the per-node count
can't be detected from a GPU-less login node):

```yaml
cluster:
  slurm:
    partition: gpu        # your GPU partition
    gpus_per_node: 8
```

On a large or multi-partition cluster `init` lists every node it can see, so
trim `cluster.nodes` to the GPU nodes you actually want to qualify before
running `doctor --all-nodes` or `test --all`.

### SSH clusters (no scheduler)

Any set of boxes you can ssh into is a cluster. Set `remote.mode: ssh` and
list the nodes; entries accept `user@host:port`, so cloud pods with one
public port per node (RunPod, Vast) work directly:

```yaml
cluster:
  nodes:
    - root@203.0.113.7:45663
    - root@203.0.113.9:11542
remote:
  mode: ssh
  ssh_key_path: ~/.ssh/id_ed25519
```

`doctor --all-nodes` and `test --all --parallel 8` then drive every node over
ssh. Validated end to end against live multi-node rigs.

## SkyPilot

There are two ways to combine sixtytwo with
[SkyPilot](https://skypilot.readthedocs.io):

```bash
# 1. Qualify a cluster you launched on any cloud SkyPilot supports
sixtytwo skypilot qualify my-cluster --mode quick

# 2. Provision sixtytwo's own GPUs through SkyPilot with `cloud: sixtytwo`
pip install 'sixtytwo-cli[skypilot]'
sixtytwo skypilot install              # register sixtytwo as a SkyPilot cloud
sky show-gpus --cloud sixtytwo
```

`sixtytwo skypilot install` is opt-in. It wires the adapter into the active
virtualenv so `sky launch` can reserve reliability-backed GPUs with
`cloud: sixtytwo`. Run `sixtytwo skypilot --help` for the full set of
subcommands.

## Topology and fabric health

The full test suite maps how a node is actually wired and checks the fabric
it sits on:

- **GPU topology**: NVLink/XGMI link matrices (nvidia-smi topo, rocm-smi
  showtopo). PCIe-only or mixed-link nodes are flagged before they slow a
  collective.
- **InfiniBand fabric**: `ibnetdiscover` maps which leaf switch each node
  hangs off (one fabric-wide sweep per cluster, reused across nodes), and
  port counters are classified so congestion reads as congestion instead of
  a cable fault.
- **GPU to NIC affinity**: an NCCL probe across all GPUs verifies
  rail-optimized placement.
- **Diagnosis with opt-in repairs**: failures map to a remediation plan
  (restart fabric manager, reset wedged GPUs, fix clock sync). Everything is
  off by default and dry-run first; fixes only execute when
  `recovery.in_place_fixes.enabled` is set, stop at the first verified
  repair, and never escalate to a disruptive action without a fresh
  diagnosis.

## Hosted dashboards

Stream telemetry to [sixtytwo.ai](https://sixtytwo.ai) and watch the fleet
from a browser:

```bash
sixtytwo connect --source ssh --name my-cluster   # register (token from the account page)
sixtytwo monitor                                  # stream per-GPU telemetry
sixtytwo install-agent                            # or install as a systemd service
```

The cluster page shows live per-node tiles, a node x GPU utilization heat
map, and a 24h history. On Slurm clusters the agent also attributes GPU
hours to users and accounts, so the chargeback report (with CSV export)
fills itself in.

## Metrics for Grafana

`sixtytwo metrics serve` exposes a Prometheus `/metrics` endpoint backed by the
local trust registry: trust scores, fault counters, recovery downtime, and
per-check status. `sixtytwo metrics export-grafana` prints a curated dashboard
you can import as is.

```bash
sixtytwo metrics serve --host 0.0.0.0 --port 9620
sixtytwo metrics export-grafana --output sixtytwo-overview.json
```

## License

Commercial. The bundled `LICENSE` governs use. Learn more at
[sixtytwo.ai](https://sixtytwo.ai).
