Metadata-Version: 2.4
Name: mrg-iot
Version: 1.3.5
Summary: IoT Testbed Experiment Automation CLI for MergeTB / Sphere Testbed
Author: Sphere / NEU IoT Testbed contributors
License: MIT
Project-URL: Homepage, https://gitlab.com/sphere-neu/mrg-iot
Project-URL: Repository, https://gitlab.com/sphere-neu/mrg-iot
Project-URL: Issues, https://gitlab.com/sphere-neu/mrg-iot/-/issues
Keywords: iot,testbed,mergetb,sphere,automation,cli
Classifier: Development Status :: 4 - Beta
Classifier: Environment :: Console
Classifier: Intended Audience :: Science/Research
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: MacOS
Classifier: Operating System :: POSIX :: Linux
Classifier: Operating System :: Microsoft :: Windows
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: System :: Distributed Computing
Classifier: Topic :: System :: Networking
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: bcrypt>=5.0.0
Requires-Dist: betterproto==2.0.0b7
Requires-Dist: certifi
Requires-Dist: cffi>=2.0.0
Requires-Dist: cryptography>=46.0.3
Requires-Dist: grpclib>=0.4.8
Requires-Dist: h2>=4.3.0
Requires-Dist: hpack>=4.1.0
Requires-Dist: hyperframe>=6.1.0
Requires-Dist: mergetbapi==1.3.41
Requires-Dist: multidict>=6.7.0
Requires-Dist: packaging>=25.0
Requires-Dist: paramiko>=4.0.0
Requires-Dist: pycparser>=2.23
Requires-Dist: PyNaCl>=1.6.0
Requires-Dist: python-dateutil>=2.9.0
Requires-Dist: six>=1.17.0
Requires-Dist: typing_extensions>=4.15.0
Provides-Extra: dev
Requires-Dist: astroid==4.0.1; extra == "dev"
Requires-Dist: black==25.9.0; extra == "dev"
Requires-Dist: dill==0.4.0; extra == "dev"
Requires-Dist: invoke==2.2.1; extra == "dev"
Requires-Dist: isort==7.0.0; extra == "dev"
Requires-Dist: mccabe==0.7.0; extra == "dev"
Requires-Dist: mypy; extra == "dev"
Requires-Dist: mypy_extensions==1.1.0; extra == "dev"
Requires-Dist: pathspec==0.12.1; extra == "dev"
Requires-Dist: platformdirs==4.5.0; extra == "dev"
Requires-Dist: pylint==4.0.2; extra == "dev"
Requires-Dist: pytokens==0.2.0; extra == "dev"
Requires-Dist: tomlkit==0.13.3; extra == "dev"
Requires-Dist: types-paramiko; extra == "dev"
Dynamic: license-file

# mrg-iot

> IoT testbed experiment automation for the [Sphere / Merge Testbed](https://sphere-testbed.net) platform.

[![PyPI version](https://img.shields.io/pypi/v/mrg-iot.svg)](https://pypi.org/project/mrg-iot/)
[![Python versions](https://img.shields.io/pypi/pyversions/mrg-iot.svg)](https://pypi.org/project/mrg-iot/)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)
[![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)

`mrg-iot` is a command-line tool that drives the **entire lifecycle** of an IoT
experiment on the Merge Testbed — so you don't have to stitch together a dozen
`mergetb` and `ssh` calls by hand. From a single command it will:

1. **Create** an experiment and generate its network model for the devices you select.
2. **Realize** the model (allocate physical hardware).
3. **Materialize** the realization (bring the allocation online).
4. **Create an XDC** (eXperiment Development Container) and **attach** it to the materialization.
5. **SSH** into the XDC and open **SSH tunnels** for RTSP video streams (`:8554`), file downloads (`:9000`), and the experiment's `ExperimentControl` gRPC channel (`:17000`).
6. Give you an **interactive control loop** (or a persistent background daemon) to drive devices, watch camera feeds in VLC, and pull experiment artifacts.
7. **Clean up** every resource it created — even on partial failure.

---

## Table of contents

- [Requirements](#requirements)
- [Installation](#installation)
- [Quick start](#quick-start)
- [Authentication](#authentication)
- [Two ways to run](#two-ways-to-run)
  - [One-shot mode: `run`](#one-shot-mode-run)
  - [Persistent daemon mode: `connect`](#persistent-daemon-mode-connect)
- [The `spiot_ctl` control language](#the-spiot_ctl-control-language)
- [Command reference](#command-reference)
- [Input validation rules](#input-validation-rules)
- [Configuration & on-disk state](#configuration--on-disk-state)
- [Architecture](#architecture)
- [Troubleshooting](#troubleshooting)
- [Development](#development)
- [Release process](#release-process)
- [License](#license)

---

## Requirements

| Requirement | Notes |
|-------------|-------|
| **Python 3.9+** | 3.9, 3.10, 3.11, and 3.12 are supported and CI-tested. |
| **A Merge Testbed account** | With access to at least one IoT project (`neuiot`, `sphereiotpostlude`, or `iotdev`). |
| **Network access** | To the gRPC portal (`grpc.sphere-testbed.net:443`) and the SSH jump host (`jump.sphere-testbed.net:2022`). |
| **[VLC](https://www.videolan.org/vlc/)** | **Only** needed for `show video` / RTSP playback. `pip` cannot install it — install it from your OS package manager. The `vlc` binary must be on your `PATH`. |

---

## Installation

The supported, production install path is [`pipx`](https://pipx.pypa.io/) — it
puts `mrg-iot` on your `PATH` in its own isolated environment:

```bash
pipx install mrg-iot
```

Or, into a regular virtual environment:

```bash
python3 -m venv .venv && source .venv/bin/activate
pip install mrg-iot
```

Verify the install:

```bash
mrg-iot --version
mrg-iot --help
```

### Installing VLC (for video)

| Platform | Command |
|----------|---------|
| macOS | `brew install --cask vlc` |
| Debian/Ubuntu | `sudo apt-get install vlc` |
| Fedora | `sudo dnf install vlc` |
| Windows | [Download from videolan.org](https://www.videolan.org/vlc/) |

---

## Quick start

```bash
# 1. Authenticate once — credentials are cached under ~/.mrg-iot/session.json
mrg-iot login

# 2. Spin up an experiment end-to-end, interactively
mrg-iot run

# ...or fully scripted, no prompts:
mrg-iot run --non-interactive \
    --project neuiot \
    --devices s-echodot-1 s-googlenest-1 \
    --exp-name myexp \
    --exp-desc "Smart-speaker capture run" \
    --realization realiot \
    --duration 1w \
    --xdc myxdc
```

Inside the interactive control loop you can issue [`spiot_ctl` commands](#the-spiot_ctl-control-language):

```
spiot_ctl > exp devices
spiot_ctl > dev s-echodot-1 click_button
spiot_ctl > exit
```

---

## Authentication

```bash
mrg-iot login     # prompts for username + password, stores a bearer token
mrg-iot logout    # clears the stored session
```

Credentials are written to `~/.mrg-iot/session.json` with mode `0600`. Every
authenticated command transparently re-validates the stored token and
re-prompts only if it has expired — you rarely need to run `login` more than
once.

---

## Three ways to run

`mrg-iot` offers three execution models. `run` and `connect` drive the full
workflow from your laptop (SSH + tunnels); `ctl` is the lightweight in-XDC
client. Pick based on where you are and how long you need the session to stay
alive.

### One-shot mode: `run`

`mrg-iot run` owns the SSH session, the `spiot_ctl` channel, and the tunnels
**for the lifetime of the process**. When you type `exit` (or send
`Ctrl-C` / `SIGTERM`), it tears everything down. This is the simplest mode and
the default if you type `mrg-iot` with no subcommand.

```bash
mrg-iot run                       # interactive, prompts for everything
mrg-iot run --simplified          # interactive but skips optional prompts
mrg-iot run --non-interactive ... # fully scripted (see Quick start)
```

Useful `run` flags:

| Flag | Purpose |
|------|---------|
| `--non-interactive`, `-n` | No prompts; all required args must be supplied. |
| `--simplified`, `-s` | Interactive, but uses sensible defaults for optional steps. |
| `--commands-file`, `-cf <path>` | Run a newline-delimited file of `spiot_ctl` commands non-interactively. |
| `--download-files`, `-df` | Download the experiment archive on exit. |
| `--delete-xdc`, `-dx` | Delete the XDC during cleanup (default: keep). |
| `--delete-exp`, `-de` | Delete the experiment during cleanup (default: keep). |
| `--network`, `-net <name>` | Model network name (default: `mrg-iot-net`). |

> **Tip:** By default `run` *keeps* the XDC and experiment on exit so you can
> reconnect later with `connect`. Pass `--delete-xdc`/`--delete-exp` to do a
> full teardown.

### Persistent daemon mode: `connect`

For long-running experiments you don't want tied to a single terminal,
`mrg-iot connect` spawns a **detached background daemon** that holds the SSH
session and tunnels open. Your shell returns immediately, and you drive the
daemon from any terminal with `send`, `show`, `download`, and `disconnect`.

```bash
# Resolve + connect (interactive pickers if flags are omitted)
mrg-iot connect
mrg-iot connect --project neuiot --experiment myexp --realization realiot --xdc myxdc

# Drive the running experiment from anywhere
mrg-iot send "exp devices"
mrg-iot send "dev s-echodot-1 click_button"
mrg-iot send "exp cred s-echodot-1" "exp sleep 10"   # multiple commands, in order

# Open VLC viewers for every camera on the XDC
mrg-iot show video

# Pull the experiment traffic archive
mrg-iot download traffic                       # -> ~/Downloads/<enclave>.tar.gz
mrg-iot download traffic --output /tmp/run.tar.gz

# Shut the daemon down
mrg-iot disconnect
```

**How it works:** `connect` validates the enclave, then launches
`python -m mrg_iot.services.daemon.server` detached via `setsid`. The daemon
binds a Unix socket at `~/.mrg-iot/connection.sock` and advertises readiness in
`~/.mrg-iot/connection.json`. Client commands exchange **line-delimited JSON**
over that socket:

```
request:   {"cmd": "<name>", "args": <any JSON>}
response:  {"out": "<text>"} | {"err": "<text>"} ... {"done": <exit_code>}
```

Access to `spiot_ctl` is serialized with a lock, so concurrent `send` calls
never interleave their output. Daemon logs land in `~/.mrg-iot/daemon.log`.

### In-XDC mode: `ctl`

When you're already **inside an XDC** (e.g. SSH'd in yourself), you don't need
the SSH/tunnel machinery — the experiment's `ExperimentControl` server is
reachable directly over WireGuard. `mrg-iot ctl` connects to it and gives you
the same interactive `spiot_ctl >` prompt, replacing the standalone `spiot_ctl`
script. Install `mrg-iot` in the XDC and run:

```bash
mrg-iot ctl              # connect to the experiment (default host 192.168.254.1)
mrg-iot ctl 10.0.0.5     # override the ExperimentControl host
```

Each line you type is sent as one `RunCommand` gRPC call; type `exit` (or
`Ctrl-C` / `Ctrl-D`) to quit. Prompt history is saved to `~/.spiot_history`.

---

## The `spiot_ctl` control language

Whether you're in the interactive `run` loop or using `mrg-iot send`, you speak
to devices through `spiot_ctl`. The built-in `help` command lists everything;
the most common verbs:

| Command | Description |
|---------|-------------|
| `help` | Show all available commands. |
| `exp devices` | List the devices in the experiment. |
| `exp cred <device>` | Get a device's credentials (e.g. `exp cred s-echodot-1`). |
| `exp sleep <seconds>` | Pause for N seconds. |
| `exp read <filepath>` | Read a command file on the XDC and run every line in it. |
| `dev <device> help` / `dev <device> commands` | Show the commands a device supports. |
| `dev <device> <command> help` | Detailed help for one device command. |
| `dev <device> [-s <id>] <command> [args]` | Execute a device command (optionally storing a job id). |
| `query state <device> <command> <id>` | Check the state of a running command. |
| `query get_result <device> <command> <id>` | Fetch the result of a finished command. |
| `query wait_result <device> <command> <id>` | Block until a command finishes, then return its result. |
| `exit` | Tear down `spiot_ctl`, optionally download files, and clean up. |

The `device-commands/` directory in this repo contains a per-device reference
of supported commands (one `*-cmds.txt` file per device, e.g.
`s-echodot-1-cmds.txt`).

---

## Command reference

`mrg-iot` exposes resource-CRUD subcommands so you can manage individual
testbed objects without running the full flow. Add `--help` to any subcommand
for its exact flags, and `--debug`/`-d` to any command for verbose logging.

### Experiments

```bash
mrg-iot exp list [--project p]                              # list (all projects if omitted)
mrg-iot exp info   --project p --name n                     # details + realizations
mrg-iot exp create --project p --name n [--description d] \
        [--network net] [--realization r] [--duration 1w]   # create + realize + materialize
mrg-iot exp delete --project p --name n
```

### Realizations (alias: `rz`)

```bash
mrg-iot realization list   --project p --experiment e
mrg-iot realization info   --project p --experiment e --name r
mrg-iot realization status --project p --experiment e --name r
mrg-iot realization create --project p --experiment e --devices d1 d2 \
        [--name r] [--network net] [--duration 1w]          # build model, compile, push, realize
mrg-iot realization relinquish --project p --experiment e --name r
```

### Materializations (alias: `mtz`)

```bash
mrg-iot materialization list          --project p --experiment e
mrg-iot materialization info          --project p --experiment e --realization r
mrg-iot materialization status        --project p --experiment e --realization r
mrg-iot materialization create        --project p --experiment e --realization r
mrg-iot materialization dematerialize --project p --experiment e --realization r
```

### XDCs

```bash
mrg-iot xdc list [--project p]
mrg-iot xdc info   --project p --name n
mrg-iot xdc create --project p [--name n] [--duration 1w] [--no-wait]
mrg-iot xdc create --project p --experiment e --realization r        # create + attach
mrg-iot xdc attach --project p --name n --experiment e --realization r
mrg-iot xdc detach --project p --name n --experiment e --realization r
mrg-iot xdc delete --project p --name n
```

### Connection / daemon

```bash
mrg-iot connect    [--project p --experiment e --realization r --xdc x]
mrg-iot send       "<cmd>" ["<cmd>" ...]
mrg-iot show video
mrg-iot download   traffic [--output <path>]
mrg-iot disconnect
```

### In-XDC client

```bash
mrg-iot ctl [host]      # interactive spiot_ctl prompt, run from inside the XDC
```

### Typical cleanup sequence

```bash
mrg-iot xdc detach --project neuiot --name myxdc --experiment myexp --realization realiot
mrg-iot xdc delete --project neuiot --name myxdc
mrg-iot materialization dematerialize --project neuiot --experiment myexp --realization realiot
mrg-iot realization relinquish --project neuiot --experiment myexp --name realiot
mrg-iot exp delete --project neuiot --name myexp
```

---

## Input validation rules

Inputs are validated up front so bad arguments fail fast with a clear
`[mrg-iot] Invalid <label>: …` message (exit code `2`) rather than deep inside a
portal call:

- **Names** (`exp-name`, `realization`, `xdc`, `network`): must match
  `^[a-z][a-z0-9]*$` — start with a lowercase letter, lowercase letters and
  digits only, max **32** chars.
- **Description**: letters, digits, spaces, commas, periods, hyphens; max
  **256** chars.
- **Duration**: minimum **4 days**. Accepts forms like `1w`, `4d`, `1w2d3h`,
  `1 week`. Anything under 4 days is rejected.

---

## Configuration & on-disk state

Everything user-scoped lives under `~/.mrg-iot/`:

| File | Purpose |
|------|---------|
| `session.json` | `{"username", "token"}` — your cached login (mode `0600`). |
| `connection.json` | Active daemon descriptor (project, experiment, realization, xdc, pid, socket, started_at). Stale entries are auto-cleared when the pid is dead. |
| `connection.sock` | The daemon's Unix socket (mode `0600`). |
| `daemon.log` | Daemon stdout/stderr after daemonization. |

`--debug` additionally writes verbose logs to `debug.log` in the current
directory (and to stderr).

### Endpoints & known projects

- **gRPC portal:** `grpc.sphere-testbed.net:443` (via `mergetbapi==1.3.41`)
- **SSH jump host:** `jump.sphere-testbed.net:2022`
- **Tunneled ports:** `8554` (RTSP), `9000` (file download)
- **Known projects:** `neuiot`, `sphereiotpostlude`, `iotdev`

---

## Architecture

The codebase follows an **MVC + services** layout under the `mrg_iot/` package.
Controllers translate argparse args into service calls; services hold business
logic (portal gRPC client, SSH/tunnel/daemon plumbing); models hold data
structures and on-disk state; views own all user-facing output.

```
mrg_iot/
├── cli.py                  # argparse tree + main() dispatcher
├── constants.py            # pure data: paths, endpoints, model template, help text
├── controllers/            # argparse args -> service calls (@require_login)
│   ├── experiment.py  realization.py  materialization.py
│   ├── xdc.py  run.py  connection.py  _base.py
├── models/                 # dataclasses + on-disk state
│   ├── enclave.py  validators.py  session.py  connection.py  memory.py
├── views/                  # all user-facing output
│   ├── output.py  formatters.py  prompts.py  log_manager.py
├── services/               # business logic
│   ├── auth.py
│   ├── experiment_workflow.py   # the `run` state machine + spiot session lifecycle
│   ├── reconnect.py             # interactive pickers + readiness checks for `connect`
│   ├── ssh.py                   # XDC SSH connect + spiot_ctl bootstrap + framed reader
│   ├── tunnel.py                # Paramiko port forwarding (RTSP 8554, download 9000)
│   ├── api/                     # gRPC client (mergetbapi), retry wrapper, error handling
│   └── daemon/                  # detached background daemon (server/client/protocol)
└── utils/                  # duration parsing, file helpers
```

**The experiment workflow** (`services/experiment_workflow.py`):

```
Login/Session → Select Project → Select Devices → Create Experiment →
Build+Compile+Push Model → Realize → Materialize → Create/Attach XDC →
SSH to XDC + start spiot_ctl + open RTSP/download tunnels → Interactive loop → Cleanup
```

State is captured in an `ExpState` dataclass so cleanup can tear down exactly
what was created, even on partial failure. Both `run` and the `connect` daemon
share the same `start_spiot_session` code path.

---

## Troubleshooting

**SSL / TLS error talking to the portal (common on macOS):**

```bash
export SSL_CERT_FILE="$(python -m certifi)"
export REQUESTS_CA_BUNDLE="$SSL_CERT_FILE"
```

**`show video` does nothing / errors:** Make sure the `vlc` binary is installed
and on your `PATH` (`vlc --version`). `pip`/`pipx` cannot install VLC.

**`No active connection`:** The daemon isn't running, or its pid died. Run
`mrg-iot connect` again — stale `connection.json` state is cleared
automatically.

**A command hangs or times out:** Re-run with `--debug` and inspect `debug.log`
(and `~/.mrg-iot/daemon.log` for daemon mode) for the underlying gRPC/SSH error.

**Resources left behind after a crash:** Use the
[cleanup sequence](#typical-cleanup-sequence) to remove the XDC,
materialization, realization, and experiment individually.

---

## Development

```bash
git clone https://gitlab.com/sphere-neu/mrg-iot.git
cd mrg-iot
python3 -m venv .venv && source .venv/bin/activate
pip install -e ".[dev]"
```

Tests, type-checking, linting, formatting:

```bash
pytest tests/ -v                              # asyncio_mode=auto; no extra flags
pytest tests/test_daemon_protocol.py -v       # a single file
mypy mrg_iot
pylint mrg_iot
black mrg_iot tests && isort mrg_iot tests
```

> The `dev` extras (black, pylint, isort, etc.) require Python **3.10+**. On a
> 3.9 environment, install runtime deps only (`pip install -e .`).

---

## Release process

CI (`.gitlab-ci.yml`) runs `pytest` on every push, builds an sdist + wheel on
success, and publishes to PyPI on `vX.Y.Z` tags via OIDC trusted publishing.

**The git tag must equal `v$(cat mrg_version)`** — the publish job aborts
otherwise. To cut a release:

```bash
# bump the version
echo "1.3.2" > mrg_version
git commit -am "increased version to 1.3.2"
git tag v1.3.2
git push && git push --tags
```

---

## License

[MIT](LICENSE) © Sphere / NEU IoT Testbed contributors
