Metadata-Version: 2.4
Name: xrpld-lab
Version: 6.0.0
Summary: Build xrpld networks and standalone ledgers
License-Expression: ISC
License-File: LICENSE
Author: Denis Angell
Author-email: dangell@transia.co
Requires-Python: >=3.10,<4.0
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Requires-Dist: pyyaml (>=6.0.1,<7.0.0)
Requires-Dist: requests (>=2.31.0,<3.0.0)
Requires-Dist: xrpl-py (>=5.2.0,<6.0.0)
Project-URL: Repository, https://github.com/XRPLF/xrpld-lab
Description-Content-Type: text/markdown

# xrpld-lab

Build xrpld networks and standalone ledgers for testing and development.

## What it does

xrpld-lab creates local XRPLD networks using pre-built binaries or Docker images. Instead of building rippled from source inside Docker (slow), you build natively on your machine and deploy to a multi-node network in seconds.

Four deployment modes:

- **Standalone** -- single-node Docker ledger for quick testing
- **Network** -- multi-validator Docker cluster with explorer and VL
- **Local** -- multi-node native processes (no Docker for nodes, fastest iteration)
- **Ansible** -- generate ansible playbooks for remote server deployment

## Install

```bash
pip install xrpld-lab
```

Network and local clusters run every key and validator list operation through the [xrpld-publisher](https://github.com/XRPLF/xrpld-publisher) CLI, which runs rippled's `validator-keys` tool. xrpld-lab looks for each binary at its environment variable (`$XRPLD_PUBLISHER_BIN`, `$VALIDATOR_KEYS_BIN`), then beside the xrpld binary given by `--binary_path`, then on `PATH`. Build `validator-keys` with rippled (`-Dvalidator_keys=ON`) and xrpld-publisher with `cargo build --release`.

## Keystore

Each cluster's identity lives under `<workspace>/<cluster>/keystore/`, written by `xrpld-publisher`:

- `vl/key.json`, `vl/token.txt`, `vl/manifest.txt` -- the publisher's master key, its ed25519 signing token and manifest (`xrpld-publisher init`). The master key, as hex, is every node's `[validator_list_keys]` entry. With `--publishers N` the further publishers are `vl2` ... `vlN`, each trusted by every node.
- `vnodeN/key.json`, `vnodeN/token.txt`, `vnodeN/manifest.txt`, `vnodeN/attestation.txt` -- each validator's master key, its `[validator_token]`, its manifest and the domain attestation for `xrpl.vnodeN.transia.co` (`xrpld-publisher create-validator`).

Every deploy has each publisher prepare, sign and verify a list of every validator (`keystore/<publisher>/unsigned.json`, sequence = signing time in unix seconds, expiration 30 days later) and serves it as `vl/<publisher>.json`. Docker and local clusters serve `vl/` from the `vl` nginx container, which mounts the directory, so a list written there is served at once. A local cluster publishes that container on host port `80 + --port_offset`. A `vl/key.json` with no `key_type` field, from the older publisher signers, is refused; run `xrpld-publisher migrate-keys` on it or redeploy with `--genesis True`.

## Quick start

```bash
# Standalone XRPL ledger
xrpld-lab up:standalone

# 3-validator, 1-peer XRPL network from the rippleci/xrpld release image
xrpld-lab create:network --num_validators 3 --num_peers 1 --genesis True
xrpld-lab up --name 3.3.0-cluster

# XRPL network from a custom GitHub branch (local binary)
xrpld-lab create:network \
  --protocol xrpl \
  --build_server "https://github.com/XRPLF/rippled/tree/xrplf-smart-contracts" \
  --build_version <commit_hash> \
  --num_validators 3 --num_peers 1

# Local network (native processes, no Docker for nodes)
xrpld-lab create:network --protocol xrpl --local

# Ansible deployment to remote servers
xrpld-lab create:ansible \
  --protocol xrpl \
  --build_version 3.3.0 \
  --num_validators 6 --num_peers 2 \
  --vips 10.0.0.1 10.0.0.2 10.0.0.3 10.0.0.4 10.0.0.5 10.0.0.6 \
  --pips 10.0.0.7 10.0.0.8 \
  --genesis True --quorum 3
```

## Commands

### `up:standalone` -- Create and start a standalone ledger

```bash
xrpld-lab up:standalone [OPTIONS]
```

| Option | Default | Description |
|---|---|---|
| `--protocol` | `xrpl` | Protocol |
| `--version` | `3.3.0` | Build version |
| `--log_level` | `trace` | `warning`, `debug`, `trace` |
| `--network_id` | `1` | Network identifier |
| `--nodedb_type` | `NuDB` | `NuDB` (persistent) or `Memory` (fast) |
| `--node_size` | `huge` | `[node_size]` tier: `tiny`, `small`, `medium`, `large` or `huge` |
| `--ipfs` | `false` | Include IPFS server |
| `--server` | auto | Build server URL |
| `--public_key` | default | Validator list public key |
| `--tx_queue` | `xrpld` | `[transaction_queue]` preset: `xrpld` (xrpld's own defaults, fee escalation on) or `flat` (100000 txns per ledger at base fee, for perf runs) |
| `--config_overrides` | none | Path to YAML/JSON config overrides |

### `create:network` -- Create a multi-node network

```bash
xrpld-lab create:network [OPTIONS]
```

| Option | Default | Description |
|---|---|---|
| `--protocol` | `xrpl` | Protocol |
| `--num_validators` | `3` | Number of validator nodes |
| `--num_peers` | `1` | Number of peer nodes |
| `--build_version` | `3.3.0` | Build version or commit hash |
| `--build_server` | auto | Build server URL or GitHub branch URL |
| `--network_id` | `1025` | Network identifier |
| `--log_level` | `trace` | `warning`, `debug`, `trace` |
| `--genesis` | auto | `True`: fresh chain; `False`: preserve; unset: fresh when the workspace has no keystore for the cluster, preserve when it has one |
| `--quorum` | `n-1` | Consensus quorum |
| `--nodedb_type` | `NuDB` | `NuDB` (persistent) or `Memory` (fast) |
| `--node_size` | `huge` | `[node_size]` tier: `tiny`, `small`, `medium`, `large` or `huge` |
| `--local` | off | Run as native processes (no Docker for nodes) |
| `--binary_name` | `xrpld` | Binary name for local networks |
| `--binary_path` | `./xrpld` | Path to pre-built binary (GitHub URL mode) |
| `--tx_queue` | `xrpld` | `[transaction_queue]` preset: `xrpld` (xrpld's own defaults, fee escalation on) or `flat` (100000 txns per ledger at base fee, for perf runs) |
| `--config_overrides` | none | Path to YAML/JSON config overrides |
| `--ansible` | off | Also generate ansible deployment files |
| `--vips` | none | Validator IPs (for ansible) |
| `--pips` | none | Peer IPs (for ansible) |
| `--drill_user` | none | Confined drill account on every node (for ansible, see `create:ansible`) |
| `--hub` | none | `pnode<N>` every validator and the other peers reach the network through: validators and the other peers fix only the hub, the hub fixes them all, and every node gets `[peer_private]` 1. Default: validators mesh |

**Build server modes:**

- **XRPL Docker**: `--build_server` defaults to `rippleci`. Uses Docker image `rippleci/xrpld:<version>`.
- **XRPL GitHub**: Pass `--build_server "https://github.com/OWNER/repo/tree/branch"` with `--build_version <commit_hash>`. Copies local binary, resolves features from GitHub at that commit.

### `create:ansible` -- Create network with ansible deployment

Generates everything `create:network` does, plus a complete ansible directory for deploying to remote servers.
The node configs carry the remote network's `[validator_token]` and `[ips_fixed]`, so `up`,
`node:restart`, `update:node`, `fault:flag-split` and the cluster's `start.sh` refuse to start
an ansible cluster on the local host.

```bash
xrpld-lab create:ansible \
  --protocol xrpl \
  --vips 10.0.0.1 10.0.0.2 10.0.0.3 \
  --pips 10.0.0.4 \
  [OPTIONS]
```

| Option | Default | Description |
|---|---|---|
| `--vips` | required | Validator IP addresses |
| `--pips` | required | Peer IP addresses |
| `--ssh_port` | `22` | SSH port for ansible |
| `--ssh_user` | `ubuntu` | SSH user |
| `--ssh_key` | `~/.ssh/id_rsa` | SSH private key path |
| `--ansible_config` | none | YAML file with full ansible config (services, etc.) |
| `--admin_ips` | none | Remote addresses granted admin on every node's admin RPC/WS ports; those ports are then published on all host interfaces (default: loopback only) |
| `--drill_user` | none | Name of a confined account provisioned on every node for drill participants |

Plus all `create:network` options (protocol, build_server, etc.)

**Ansible config file** for complex deployments with services:

```yaml
ssh_port: 22
ssh_user: ubuntu
ssh_key_path: ~/.ssh/id_rsa
vips:
  - 10.0.0.1
  - 10.0.0.2
  - 10.0.0.3
pips:
  - 10.0.0.4
admin_ips:
  - 203.0.113.7
services:
  - ip: 10.0.0.4
    name: infra
    nginx:
      domain: example.com
      ssl_org: MyOrg
    redis: {}
    faucet:
      ws_url: ws://10.0.0.4:6018
      network_id: "1025"
      seed: sEdxxxxxxxxx
    status:
      port: 8687
```

`hosts.txt` lists every validator and peer under the `nodes` group, which the node
playbooks (`deps.yml`, `main.yml`, `clean.yml`, `alloy.yml`, `status.yml`, `drill.yml`) target; each
services host is its own group, targeted only by its service playbooks. A services host
whose IP is not in `vips` or `pips` is therefore never touched by a node playbook; one that
is also a node sits in both groups (the example host `10.0.0.4` is the peer, which
`status:` and `stream:` require). `vl:` requires `nginx:` on the same host: its vhost is
`vl.<domain>` served by that nginx, and the builder raises `ValueError` otherwise.

Services bind only where their clients are: redis is published on `127.0.0.1:6379` (the
host's status sampler checks it; the debugstream container reaches it by container name
over the docker network), and the websocketd log stream binds the default docker bridge
address (`docker0`), which the debugstream container reaches as `host.docker.internal`
and which is not routable from off-host. The `debug.<domain>` nginx vhost is the public
way in.

**Drill account** (`--drill_user <name>`, or `drill_user:` in the ansible config).
`drill.yml` re-runs on every deploy and gives each node a login account of that name with
no supplementary groups, ssh keys read from the root-owned
`/etc/ssh/authorized_keys.d/<name>` (created empty; the operator adds the keys), no ssh
forwarding, no cron or at, and every process it starts recorded by auditd. Its two verbs,
`drill-status server_info|peers|consensus_info|validators` and `drill-restart`, run as
root through sudo with their output logged. The play also installs the root-only
`/usr/local/sbin/drill-ops`, which the account cannot run: it takes one of `status`,
`restart`, `stop`, `start` (the docker action on the node's container), `stall <ms>` or
`unstall` (xrpld's `node_stall` admin RPC on the node's loopback admin port, with
`{"duration_ms": <ms>}` or `{"clear": true}`), exits 1 on anything else, and accepts
`<ms>` only as 1 to 9 digits with no leading zero. A drill engine's ssh key reaches it
through a forced command such as
`set -f; exec sudo -n /usr/local/sbin/drill-ops $SSH_ORIGINAL_COMMAND`. ufw rejects the account on the node's admin
RPC and WS ports on every interface, IPv4 and IPv6; the play installs ufw and sets its
incoming policy to allow before enabling it, so inbound filtering stays with the
provider firewall.

**Status service** (`status:` on the services host, which must be one of the nodes). Every
node gets `node_metrics.py` as the `xrpld-status` systemd unit: it samples `/proc`, the
admin RPC `server_info` and the XDGM datagram into a SQLite ring buffer and serves a
per-node dashboard plus `/api/latest`, `/api/series`, `/api/events`, `/api/health` and
`/api/ledger?index=N` (the hash the node holds for ledger N). Each
node's `[datagram_monitor]` is pointed at its own address and the sampler's `xdgm_port`
(default 9999) unless `--datagram_monitor` names another sink. The services host's sampler
also aggregates every node (`/api/network`, `/api/network/health`: 200 only when every
validator is `proposing`, every peer is `full`, and there is a validator and every validator
returns one ledger hash for the lowest validated index any node reports, 503 otherwise) and serves the network
dashboard at `/network`; its bare-domain nginx vhost proxies `/status/` to that route, `/status/api/`
to the aggregation and `/status/nodes/<name>/` to each node's dashboard. Samplers on the
other nodes bind `0.0.0.0:<port>`, admitted by ufw from the services host only. With
`allow_from`, ufw on every node also admits the port from those addresses, and the
services host's sampler binds `0.0.0.0:<port>` instead of loopback, so a gateway off the
cluster can reach `/network` and each node's dashboard directly. Operators write `/opt/xrpld-status/network.json` on the
services host after a deploy (`last_deploy`, `branches`, `faucet`, `vl`, `amendments`);
the network page renders it. `status.yml` re-runs on every deploy; the nginx vhost is a
`run_once` stage, so an existing deployment picks up the new locations with `--force` or
by deleting `.done_<host>_nginx`.

| `status` key | Default | Description |
|---|---|---|
| `port` | `8687` | Sampler HTTP port on every node |
| `xdgm_port` | `9999` | UDP port for xrpld's XDGM datagram |
| `interval` | `10` | Sampling interval in seconds |
| `process` | `xrpld` | Process name looked up in `/proc` for RSS |
| `disk_path` | `/var/lib/xrpld/db` | Filesystem measured for disk usage |
| `network_name` | nginx domain | Heading on the network dashboard |
| `allow_from` | `[]` | Addresses ufw admits to every node's sampler port beside the services host |
| `retain_raw_hours` / `retain_5m_days` / `retain_1h_days` | `48` / `30` / `365` | Ring buffer retention |

### `deploy:ansible` -- Run ansible deployment

```bash
xrpld-lab deploy:ansible --name <cluster> [--workspace <dir>]
```

Runs the generated `run.sh` in the cluster's ansible directory.

| Option | Default | Description |
|---|---|---|
| `--name` | required | Cluster name, as `<name>` or `<name>-cluster` |
| `--workspace` | `./workspace` | Workspace root holding the cluster directory |

### Operational commands

Every command below addresses a cluster directory under `--workspace` (default
`./workspace`) by `--name`, given either as `<name>` or as the full directory name
`<name>-cluster`; both resolve to the same directory. Commands that reach a node over
its admin or public RPC take `--port_offset`, the value the cluster was created with.

```bash
xrpld-lab up --name <cluster>                          # Start a network
xrpld-lab down --name <cluster>                        # Stop a network
xrpld-lab remove --name <cluster>                      # Stop with --remove and delete the directory
xrpld-lab update:node --name <cluster> --node_id <N> --node_type validator|peer \
  --build_version <version> (--build_server <url> | --image <image>) [--keep_history]
xrpld-lab vote:amendment --name <cluster> --amendment_name <name> [--node_id <N>] [--veto]
xrpld-lab fault:flag-split --name <cluster> --amendment_name <name> --node_id <N> [--node_id <N> ...] \
  --build_version <version> (--build_server <url> | --image <image>)
xrpld-lab node:stall --name <cluster> --node_id <N> --node_type validator|peer \
  [--duration_ms <ms> | --clear]
xrpld-lab node:restart --name <cluster> <node_dir> [--genesis]   # e.g. vnode2
xrpld-lab logs:local [--name <cluster>] [--node <node_dir>]
xrpld-lab health --vips <ip>... [--timeout <s>] [--interval <s>]
```

| Option | Default | Description |
|---|---|---|
| `--name` | required | Cluster name, as `<name>` or `<name>-cluster`; optional on `logs:local`, which reads the current directory without it |
| `--workspace` | `./workspace` | Workspace root holding the cluster directory (every command above except `health`) |
| `--port_offset` | `0` | The `--port_offset` the cluster was created with (`node:stall`, `vote:amendment`, `fault:flag-split`, `health`) |
| `--node_id` | required | Node index (`update:node`, `node:stall`; one validator instead of all on `vote:amendment`; repeated on `fault:flag-split`: the validators to downgrade) |
| `--node_type` | required | `validator` or `peer` (`update:node`, `node:stall`) |
| `--duration_ms` | `30000` | `node:stall`: stall length in ms; `--clear` lifts a stall early |
| `--genesis` | off | `node:restart`: reload the genesis ledger instead of resuming from the node's database |
| `--keep_history` | off | `update:node`: keep the node's ledgers instead of deleting `lib/` to re-sync it |
| `--veto` | off | `vote:amendment`: veto the amendment, so each validator withdraws support at the next flag ledger |
| `--vips` | required | `health`: validator external IPs, in node order |
| `--timeout` / `--interval` | `300` / `10` | `health`: overall deadline and seconds between polls |

`fault:flag-split` downgrades the named validators, with their ledgers kept, to a build without
the amendment and has the rest vote for it. Before any downgrade it reads the first voter's
`validators` and `feature` admin RPCs, and refuses unless the voters that validator trusts, less
those on its negative UNL, are more than 80% of the cluster validators it trusts, the downgraded
ones included, and the amendment is supported by its build and not yet enabled. The amendment
enables at a flag ledger once that majority has held for the cluster's `amendment_majority_time`,
and only then are the downgraded validators amendment-blocked.

### Validator list publishers

`create:network --publishers N` gives the cluster N publishers and `--vl_threshold T` writes `[validator_list_threshold]` (the number of lists a validator must be on to be trusted; xrpld's default is 1 below three publishers). More than one publisher needs the lab to serve the lists, so it is refused with `--vl_site` or ansible. Nodes read the lists only when the cluster is created with `--bootstrap_vl`, which writes the static `[validators]` beside them.

The `vl:*` commands act on a running cluster. Each key and list operation runs xrpld-publisher; `--binary_path` names an xrpld binary whose directory holds it.

```bash
xrpld-lab vl:create  --name <cluster> --publisher vl2         # a new publisher key, trusted by no node yet
xrpld-lab vl:publish --name <cluster> [--publisher vl] [--exclude vnode4 vnode5] \
  [--add_absent N] [--expiration_days 30] [--effective_in <s>] [--refresh_interval <min>]
xrpld-lab vl:rotate  --name <cluster> [--publisher vl]        # new signing key; vl:publish serves it
xrpld-lab vl:revoke  --name <cluster> [--publisher vl]        # serve the master key's revocation
xrpld-lab vl:trust   --name <cluster> --node vnode3 --publishers vl vl2 [--threshold 2]
xrpld-lab vl:status  --name <cluster> [--port_offset N]       # one JSON line per node
```

- `vl:publish` lists every running validator except `--exclude`, plus `--add_absent` keys that no node runs (`keystore/absentN`). Its sequence is the time in unix seconds, raised above the served list's, so a node never discards it as stale. `--refresh_interval` sets how often a node refetches the list (minutes, 1 to 1440; xrpld's default is 5).
- `vl:trust` rewrites one native node's `validators.txt` to the named publishers' sites and keys and the threshold, with no static `[validators]`, as an operator's file is, then restarts the node from its database. A node reads `[validator_list_keys]` only at startup.
- `vl:status` prints each node's state, validated ledger and hash, quorum, threshold, static key count, trusted validators and every publisher's list availability and sequence, from the admin `validators` and `server_info` RPCs.

Run publisher changes on a local cluster (`create:network --local`): its nodes start without `--quorum`, so xrpld computes quorum from the lists it trusts. Docker network nodes start with `--quorum`, which overrides that calculation.

### Local network peering

xrpld's peerfinder makes one outbound connection per remote address, so native nodes that all dial `127.0.0.1` connect only to the first fixed peer that answers and form a star around it; restarting that node partitions the rest. A local cluster therefore dials validator N at `127.0.0.<N+1>`. Linux routes all of `127.0.0.0/8`; macOS needs one loopback alias per validator, until the next reboot:

```bash
for n in 2 3 4 5 6; do sudo ifconfig lo0 alias 127.0.0.$n up; done   # five validators
```

Without the aliases `create:network --local` prints this warning and every node dials `127.0.0.1`.

## Config overrides

Override any xrpld config value with a YAML file:

```yaml
# overrides.yaml
node_size: medium
transaction_queue:
  ledgers_in_queue: 50
  maximum_txn_in_ledger: 5000
```

```bash
xrpld-lab up:standalone --config_overrides overrides.yaml
```

The `--tx_queue` preset is written first. Overrides apply on top of the generated config, section by section: a mapping merges into the section's keys, a list or a scalar replaces the section, and a section the generated config lacks is appended.

## Architecture

```
xrpld_lab/
  models.py           # Dataclasses: Protocol, NodeConfig, LabConfig, PortSet, etc.
  protocol.py         # ProtocolSpec: protocol-specific defaults in one place
  config_builder.py   # XrpldCfgBuilder + ValidatorsTxtBuilder
  config.py           # INI parsing, YAML overrides, 3-layer merge, ansible config loader
  amendments.py       # C++ feature macro parsing + genesis updates
  source_resolver.py  # GitHub fetch, binary download, feature/config resolution
  node_factory.py     # Creates NodeConfig for validator/peer/standalone
  publisher.py        # Publisher: runs xrpld-publisher for keys, tokens, signed lists
  unl.py              # vl:* commands: publish, rotate, revoke, trust, status
  compose_builder.py  # docker-compose.yml as structured dict
  script_builder.py   # Dockerfiles + start/stop shell scripts
  workflows.py        # LabRunner: orchestrates standalone/network/local/ansible
  cli.py              # Thin CLI: argparse -> LabConfig -> LabRunner.run()
  ansible_builder.py  # Ansible playbooks, inventory, host vars, services
  services/status/    # node_metrics.py sampler + node and network dashboards
  operations.py       # Node updates, amendment enabling, log viewing
  workspace.py        # Path resolution
  utils.py            # File I/O, colors, subprocess helpers
```

## Development

```bash
git clone https://github.com/XRPLF/xrpld-lab
cd xrpld-lab
poetry install
poetry run pytest tests
# also run the real xrpld-publisher and validator-keys end to end
XRPLD_PUBLISHER_BIN=/path/to/xrpld-publisher VALIDATOR_KEYS_BIN=/path/to/validator-keys \
  poetry run pytest tests/integration
```

## Current versions

- XRPL: `3.3.0`

## License

ISC. See [LICENSE](LICENSE).

