Metadata-Version: 2.4
Name: svtop
Version: 0.2.0
Summary: A pragmatic TUI for monitoring multi-server NVIDIA GPU utilization
Author: Astrorix
License: MIT
Requires-Python: >=3.10
Requires-Dist: paramiko>=3.4.0
Requires-Dist: textual>=0.58.0
Description-Content-Type: text/markdown

# svtop

`svtop` monitors NVIDIA GPU usage across multiple SSH servers. It provides an
interactive TUI for humans and stable, plain-text or JSON commands for scripts
and agents.

The project keeps its state in `~/.svtop`, runs no daemon, and does not require
a central server.

## Features

- Concurrent multi-server GPU polling over SSH
- Real-time TUI with aggregate server rows and expandable per-GPU details
- Agent-readable inventory via `svtop list --format json`
- Portable, versioned configuration export and import
- Verified local-machine-to-server SSH key setup and status checks
- OpenSSH config fragment generation without modifying `~/.ssh/config`
- Dual-pane local/server transfer UI using `rsync` or an OSS cache
- Per-server proxy jump chains
- Password and private-key authentication
- Local configuration and connectivity diagnostics

`svtop` intentionally does **not** configure server-to-server keys or a
full-mesh trust network. SSH setup is limited to the machine running `svtop`
connecting to configured servers.

## Install

```bash
pip install svtop
```

For local development:

```bash
pip install -e .
```

Check the installed version and available commands:

```bash
svtop --version
svtop --help
```

## Monitor and configure

Launch the monitor:

```bash
svtop
svtop run --interval 2.5 --timeout 15
```

Manage server profiles interactively:

```bash
svtop config
```

Configuration is stored in `~/.svtop/servers.json` and
`~/.svtop/settings.json`. Server selectors accepted by non-interactive
commands are an exact server name or ID. Use an ID when names are ambiguous.

## List servers and GPUs

Probe all configured servers and print a fixed-column table:

```bash
svtop list
svtop -l
```

Print the stable JSON contract intended for agents and scripts:

```bash
svtop list --format json
```

The document uses `kind: "svtop.inventory"` and `schema_version: 1`. It contains
server IDs, endpoints, authentication metadata, probe status, GPU metrics, and
a summary. Passwords, private-key contents, and private-key paths are never
included. Error messages are sanitized before output. An unavailable GPU metric
is represented as JSON `null` and as `-` in the table instead of dropping the
GPU row.

Useful inventory options:

```bash
# Do not connect; return configured endpoints only.
svtop list --offline --format json

# Select one or more profiles in the requested order.
svtop list --server train-a --server 0f3a... --format json

# Make partial probe failure visible to automation with exit code 3.
svtop list --strict --timeout 20 --jobs 4 --format json
```

Per-server `status` values are `online`, `offline`, `fingerprint_mismatch`,
`unknown_host_key`, `auth_error`, `unreachable`, or `probe_error`. A valid
probe from a host with no NVIDIA GPUs is `online` with `gpus: []`. Without
`--strict`, reachable and failed servers are reported together and the command
exits successfully.

## Export and import configuration

Create a private portable export:

```bash
svtop export svtop-config.json
svtop -e svtop-config.json
```

Exports use `kind: "svtop.config"` and `schema_version: 1`, are written
atomically with mode `0600`, and do not overwrite an existing file unless
`--force` is supplied. Symlink destinations are rejected.

Credential migration rules:

- A profile originally added with a password is exported with that retained
  password even if `svtop` later switched its active login to a key. This lets
  the destination machine bootstrap its own key.
- If such a profile no longer has its saved password, export fails instead of
  silently creating a broken migration file. `--allow-incomplete` records the
  missing requirement explicitly.
- A profile originally added with a private key exports only a portable key
  hint and an `external_private_key` requirement. Private-key contents are
  never copied into the export.
- `--redact-secrets` removes saved passwords for a shareable inventory-style
  export. A redacted export cannot restore those credentials by itself.
- Secret-bearing stdout export is blocked unless `--unsafe-stdout` is given.

Optionally verify retained password credentials before writing:

```bash
svtop export svtop-config.json --verify-auth
```

Inspect an import without changing local state:

```bash
svtop import svtop-config.json --dry-run --format json
```

Apply it with an explicit conflict policy:

```bash
svtop import svtop-config.json --conflict fail
svtop import svtop-config.json --conflict skip
svtop import svtop-config.json --conflict replace
svtop import svtop-config.json --conflict rename
```

To replace the complete local inventory, use both confirmation flags:

```bash
svtop import svtop-config.json --replace --yes
```

An applied import writes the server and portable settings together and creates
private timestamped backups of existing files. Invalid schemas, duplicate IDs,
duplicate endpoints, and malformed profiles are rejected before any write.

Treat a non-redacted export like a password file. Transfer it over a secure
channel and remove it when migration is complete.

## Local-to-server SSH access

Open a configured server by name or ID:

```bash
svtop ssh train-a
svtop ssh open train-a
```

Install this machine's public key on selected servers, verify a fresh key-only
connection, and update only profiles that passed verification:

```bash
svtop ssh setup train-a train-b
svtop ssh setup --all --jobs 8
```

By default, `svtop` reuses a suitable local key. If none exists it creates a
dedicated Ed25519 key under `~/.svtop/keys`. Use `--key PATH` to select another
private key. A generated key is explicitly reported as passwordless together
with its path and SHA-256 fingerprint. Password prompts and unknown-host
confirmations are collected before concurrent work begins; use
`--non-interactive` for unattended runs with already saved credentials.

Verify key-only access without installing anything:

```bash
svtop ssh status
svtop ssh status train-a --format json
```

Generate an OpenSSH fragment for review or manual inclusion:

```bash
svtop ssh config
```

This command only prints text. It never edits `~/.ssh/config`.

Interactive `ssh open` and `ssh setup` show the unknown host fingerprint and
ask before recording it. Non-interactive checks reject unknown keys by default;
`--accept-new` trusts only previously unseen keys. Paramiko and OpenSSH both
read `~/.svtop/known_hosts` and the normal user known-hosts file, while newly
accepted svtop keys are written to the managed file. Changed keys still fail.
Structured commands report a changed key as `fingerprint_mismatch`, and
interactive `ssh open` prints the same classification before returning the SSH
process exit code. Avoid `--accept-new` when you cannot independently verify
the endpoint.

## Diagnostics

Check local tools, profiles, and key files:

```bash
svtop doctor
svtop doctor --format json
```

Also test key-only connectivity:

```bash
svtop doctor --network --server train-a
```

Diagnostic JSON uses `kind: "svtop.ssh-doctor"` and `schema_version: 1`.

## Exit codes

Non-interactive commands use these codes:

| Code | Meaning |
| ---: | --- |
| `0` | Completed successfully |
| `2` | Command-line syntax or option error reported by the argument parser |
| `3` | Partial failure or warning, including `list --strict` probe failures |
| `4` | Invalid input, configuration, selection, or unsafe operation refused |
| `5` | Complete SSH setup/status failure or diagnostic failure |

An SSH subprocess may return its own non-zero exit code from `svtop ssh open`.
JSON result documents for SSH setup, status, and doctor also include an
`exit_code` field.

## Transfer mode

Open the dual-pane transfer UI:

```bash
svtop transfer
svtop transfer --parallel 4
svtop transfer --relay oss
svtop transfer --relay rsync
```

Quick keys:

- Top bar `Left` / `Right`: choose `Local` or a configured server
- Pane path bar: paste a path and press `Enter`
- `Tab`: switch active pane
- `1` / `2`: cycle left/right endpoint
- `d`: toggle transfer direction
- `Enter` or double-click: open a directory
- `Space`: mark or unmark an item
- `F5`: transfer the current or selected items
- `Backspace`: go to the parent directory
- `r`: refresh the active pane

The `oss` relay uses `ossutil cp` through the configured OSS cache. The `rsync`
relay uses direct transfer for local/server operations and local staging for
server/server operations. Transfer workspace state is saved in
`~/.svtop/settings.json`.

## Remote requirements

- SSH access to each server
- `nvidia-smi` in the remote `PATH` for GPU monitoring
- `rsync` locally and on relevant servers for rsync transfer mode
- `ossutil` wherever OSS transfer mode is used
- A local OpenSSH client when proxy jump chains or shell access are used

Proxy jump chains are configured per profile as comma-separated
`user@host[:port]` entries, for example `ops@jump-a,core@jump-b:2222`.

## Security

Password authentication stores the password in plaintext in
`~/.svtop/servers.json` with mode `0600`. Prefer key authentication for normal
operation, restrict access to the account running `svtop`, and protect any
non-redacted export as a secret.

`svtop` never exports private-key contents, never automatically modifies the
user's OpenSSH config, and never installs keys between managed servers.
