Metadata-Version: 2.4
Name: cas-toolbox
Version: 2026.45.2
Summary: Cluster Automation Scripts Toolbox
Author-email: Yufei Pan <pan@zopyr.us>
License: GPL-3.0-or-later
Project-URL: Homepage, https://github.com/yufei-pan/cas-toolbox
Project-URL: hpcp, https://github.com/yufei-pan/hpcp
Project-URL: multiCMD, https://github.com/yufei-pan/multiCMD
Project-URL: multiSSH3, https://github.com/yufei-pan/multiSSH3
Project-URL: simple-iotest, https://github.com/yufei-pan/simple-iotest
Project-URL: statbtrfs, https://github.com/yufei-pan/statbtrfs
Project-URL: Tee_Logger, https://github.com/yufei-pan/Tee_Logger
Project-URL: TSVZ, https://github.com/tsvz-org/TSVZ
Project-URL: iotest, https://github.com/yufei-pan/simple-iotest
Project-URL: firewatcher, https://github.com/yufei-pan/firewatcher
Classifier: License :: OSI Approved :: GNU General Public License v3 or later (GPLv3+)
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.6
Classifier: Operating System :: POSIX :: Linux
Requires-Python: >=3.6
Description-Content-Type: text/markdown
Requires-Dist: hpcp==9.80
Requires-Dist: multicmd==1.56
Requires-Dist: multissh3==6.39
Requires-Dist: simple-iotest==3.70.0
Requires-Dist: statbtrfs==0.38
Requires-Dist: tee-logger==6.44
Requires-Dist: tsvz==5.0
Requires-Dist: statblk==1.58
Requires-Dist: firewatcher==1.65

# cas-toolbox

[![PyPI version](https://img.shields.io/pypi/v/cas-toolbox.svg)](https://pypi.org/project/cas-toolbox/)
[![Python versions](https://img.shields.io/pypi/pyversions/cas-toolbox.svg)](https://pypi.org/project/cas-toolbox/)
[![License](https://img.shields.io/pypi/l/cas-toolbox.svg)](https://www.gnu.org/licenses/gpl-3.0.html)

**Cluster Automation Scripts Toolbox** — one stop shop for tool scripts used in
cluster automation operations in high performance computing.

`cas-toolbox` is a **meta-package**: it ships no code of its own. It pins exact
releases of the nine tools below and installs all of their console commands in a
single step, so a fresh host gets the whole toolbox with one `pip install`.

Every bundled tool is a **single self-contained `.py` file** — copy one onto a
host and it runs, no packaging required. (`multiSSH3`'s optional shell
autocomplete is the one exception: it needs `mssh_complete.py` next to
`multiSSH3.py`.)

## Project metadata

| | |
|---|---|
| **Current version** | `2026.45.2` |
| **Released** | 2026-11-03 |
| **Python** | >= 3.6 |
| **Platform** | Linux (POSIX); macOS for `multiSSH3`, `simple-iotest`, `statblk`, `tee-logger`, `TSVZ` |
| **License** | GPL-3.0-or-later |
| **Author** | Yufei Pan (pan@zopyr.us) |
| **Homepage** | [github.com/yufei-pan/cas-toolbox](https://github.com/yufei-pan/cas-toolbox) |
| **PyPI** | [pypi.org/project/cas-toolbox](https://pypi.org/project/cas-toolbox/) |
| **Build backend** | setuptools (`pyproject.toml`) |
| **Bundled tools** | 9 |

## Versioning and release cadence

Versions are `YYYY.WW.hotfix` — calendar year, ISO week number, and a hotfix
counter. For example `2026.40.1` is the first release cut in week 40 of 2026.

The bundle is deliberately **semi-stable**:

- **Mondays** are the normal re-cut. Last week's accumulated upstream changes are
  rolled into one release: the `==` pins of everything that moved are updated and
  the version bumps to the new `YYYY.WW`.
- **Mid-week** releases are for **emergency fixes only**. Only the affected tool's
  pin changes and the `hotfix` segment increments (`2026.39.1` → `2026.39.2`),
  leaving the week number alone.
- Releasing an individual upstream tool is independent and can happen any day.
  Its pin bump normally rides along in the next Monday roll-up.

Upstream tools use their own `x.y` / `x.y.z` schemes; the pins in this package
record exactly which release is in the bundle.

## Included tools

Versions listed are the ones **pinned by the current release** of `cas-toolbox`.

| Tool | Module | PyPI package | Pinned | Commands | What it does |
|---|---|---|---|---|---|
| [hpcp](https://github.com/yufei-pan/hpcp) | `hpcp.py` | [`hpcp`](https://pypi.org/project/hpcp/) | 9.80 | `hpcp` | Highly-parallel `cp` / imaging for HPC storage |
| [multiCMD](https://github.com/yufei-pan/multiCMD) | `multiCMD.py` | [`multicmd`](https://pypi.org/project/multiCMD/) | 1.56 | `mcmd`, `multicmd`, `multiCMD` | Run many subprocesses in parallel (base library) |
| [multiSSH3](https://github.com/yufei-pan/multiSSH3) | `multiSSH3.py` | [`multissh3`](https://pypi.org/project/multissh3/) | 6.39 | `mssh`, `mssh3`, `multissh`, `multissh3`, `multiSSH3`, `mssh-complete` | Parallel SSH across hosts, curses TUI |
| [simple-iotest](https://github.com/yufei-pan/simple-iotest) | `iotest.py` | [`simple-iotest`](https://pypi.org/project/simple-iotest/) | 3.70.0 | `iotest`, `simple-iotest` | Dead-simple filesystem I/O benchmark |
| [statbtrfs](https://github.com/yufei-pan/statbtrfs) | `statbtrfs.py` | [`statbtrfs`](https://pypi.org/project/statbtrfs/) | 0.38 | `statbtrfs` | Btrfs status / scrub orchestration |
| [Tee_Logger](https://github.com/yufei-pan/Tee_Logger) | `src/Tee_Logger.py` | [`tee-logger`](https://pypi.org/project/tee-logger/) | 6.44 | *(library only)* | Logger that tees to dated files + stdout |
| [TSVZ](https://github.com/tsvz-org/TSVZ) | `TSVZ.py` | [`tsvz`](https://pypi.org/project/tsvz/) | 5.0 | `tsvz`, `TSVZ` | Key-value store / in-memory DB backed by a TSV file |
| [statblk](https://github.com/yufei-pan/statblk) | `statblk.py` | [`statblk`](https://pypi.org/project/statblk/) | 1.58 | `statblk` | Block-device / partition info table |
| [firewatcher](https://github.com/yufei-pan/firewatcher) | `firewatcher.py` | [`firewatcher`](https://pypi.org/project/firewatcher/) | 1.65 | `firewatcher` | Journal/log incident capture with off-box context |

Internal dependency graph worth knowing when a pin moves:

- `multiCMD` is the foundation — `hpcp`, `statbtrfs` and `statblk` all depend on it.
- `simple-iotest` depends on `tee-logger`.

## Recent releases

| Version | Date | What changed |
|---|---|---|
| `2026.45.2` | 2026-11-03 | Mid-week cut for a feature release, so hosts can move to firewatcher 1.65 now. **`firewatcher` 1.64 → 1.65:** new `[notify] events_url` and `events_token` (or `FIREWATCHER_EVENTS_TOKEN`): every finished capture is POSTed as JSON to that service, with an excerpt of the capture (at most 16384 characters once JSON-encoded) and the token as `Authorization: Bearer`. **With `events_url` set, firewatcher's own LLM analysis is off whatever `[llm] enabled` says** (and `[llm] base_url` and `model` are not checked): the service judges, and the event's `push` field is `always_notify`. A 2xx answer means the service has the capture, and nothing goes to ntfy. When the POST fails, only `always_notify` captures are pushed to ntfy, and only with `[notify] enabled`, so with `always_notify` blank (the default) none are; the others stay in the capture folder and the journal. Without `events_url` nothing changes there. **Behavior change for URLs:** an LLM URL (with `[llm] enabled`), an ntfy URL (with `[notify] enabled`) or an events URL whose server part is not a plain host name or a bracketed IPv6 address with an optional `:port` (0–65535) now stops firewatcher at startup, as do some URLs carrying a `user:password` (firewatcher's README has the exact rules); 1.64 checked only the scheme, spaces and control characters. A `user:password` in a URL is stripped before the request, except for a password that starts with digits and a `/` (or with `/`, `?` or `#`): such a URL reads as a host, port and path as written, and the request goes to that host (firewatcher's docs/limitations.md). URLs are shown only as their server. The 2026.45.1 note on `--requires-mounts-for` is corrected: on systemd 229 and older (EL7), a path with `%`, `"` or `\` is still read as a different path. A running firewatcher service keeps its old version until it is restarted. |
| `2026.45.1` | 2026-11-02 | **`tsvz` 4.9 → 5.0 (major).** The store formats are unchanged (a compressed `.tsv` is now rewritten in members of at most 1 MiB, which 3.39 reads). **Breaking:** a key, value, header, defaults line or marker the file's encoding cannot hold (UTF-8 for `.tsvz`, `encoding=` for a `.tsv`), such as the lone surrogate `os.fsdecode` gives for a non-UTF-8 file name, raises `TSVZEncodeError` (a `ValueError`) and writes nothing, where 4.9 stored `?` (its `tsvz` stored U+FFFD for a non-UTF-8 argument, and its `.tsvz` file functions (`appendTabularFile`, `clearTabularFile`, `scrubTabularFile`) raised `UnicodeEncodeError`); to store `?`, pass `errors='replace'` to the writing functions or to `TSVZed`, `TSVZedLite` and `TSVZClient`. `TSVZed` and `TSVZedLite` refuse such a row when it is set, and `tsvz` exits 1 with the message. A rewrite that meets such text (a row changed in place) leaves the file as it is: `mapToFile()` and `hardMapToFile()` raise, and the rewrites a `TSVZed` makes itself report it once; with a `rewrite_interval` they are tried again after it (at least a second), the wait doubling at each refusal in a row up to a minute (or `rewrite_interval`, when longer), and a rewrite at `close()` does not wait it out; at the default `rewrite_interval=0` one is tried again on its own, on the same wait, only while the file is to be made again (it was removed, another process changed it, or a write to it failed), and a compressed `.tsv` another writer left damaged is made again at the first append after that wait; otherwise only one asked for is (`mapToFile()`, `hardMapToFile()`, `rewrite(force=True)`, `rewrite_on_exit`). Under a non-UTF-8 `encoding=`, a key read as U+FFFD cannot be deleted or updated. A `TSVZed` writes a `.tsv`-family file from its own thread only, so a Ctrl-C raises in the caller but no longer cuts the write (in 4.9 a Ctrl-C while `TSVZed()` opened a compressed store could leave it with no readable row); at exit a write in progress is finished, and a second Ctrl-C abandons it. A closed `TSVZed` or `TSVZedLite` is freed (4.9 kept every one until exit). Another process's append to a `.tsv` file is read from where the store left off, so an order set by `move_to_end` and `#` keys survive it, and any reload changes memory in place, so another thread never finds a live key missing. A `TSVZed`'s whole rewrite writes over the file and then cuts it, so a reader never meets it emptied (`scrubTabularFile`, `clearTabularFile`, `tsvz scrub` and `tsvz clear` of a `.tsv` still empty it first, as in 4.9). A file another process removed, or a compressed `.tsv` another writer left damaged, is made again from memory at the store's next write. An append to an existing empty `.tsv` writes the header first, and reading one gives an empty store; `TSVZed` on a compressed `.tsv` in a codec this Python lacks (zstd before 3.14, or bz2 or xz on a Python built without them) raises `ImportError` instead of reading the compressed bytes as text (TSVZ's README, L8 to L14). `.tsvz` stores see another process's append made right after their load, a commit or a clear (4.9 missed it until the next change) and raise a strict header mismatch at the first read. `tsvz` runs every operation through the engine `tsvz serve` uses, with the same output; a server of a `.tsv` (or other tab- or NUL-delimited) file is now found by `tsvz` commands, `TSVZClient` and `tsvz stop` (4.9's clients rejected its pointer file). On macOS, and wherever TSVZ locks with `lockf`, a read in another thread of the process can still drop a write's lock, as in 4.9 (deferred; TSVZ's docs/limitations.md lists it with 5.0's other limitations). Long-running programs that import TSVZ, and running `tsvz serve` handlers, keep 4.9 until they are restarted; 5.0's `tsvz stop` cannot stop a 4.9 handler of a `.tsv` (or other tab- or NUL-delimited) file (it rejects that handler's pointer file), so send that one SIGTERM (its pid is on the `pid` line of its pointer file, `STORE.serve`). **`multissh3` 6.38 → 6.39** (`mssh-complete` 0.11 → 0.12), a simplification pass. **Behavior change:** ssh, scp, rsync and ssh-copy-id always get mssh's `ConnectTimeout`, `ServerAliveInterval` and `ServerAliveCountMax` (`DEFAULT_SSH_*`: 30, 15 and 4), after `-ea`'s options, so an `-o` in `-ea` still wins but the same options in your ssh config (`~/.ssh/config`, `/etc/ssh/ssh_config`, `-ea '-F FILE'`) no longer do, where 2026.43.1 let your ssh config win: set `DEFAULT_SSH_CONNECT_TIMEOUT`, `DEFAULT_SSH_SERVER_ALIVE_INTERVAL` or `DEFAULT_SSH_SERVER_ALIVE_COUNT_MAX` in mssh's config file instead, or 0 to leave that option to the ssh config (an interval of 0 leaves the count to it too). An `-ea '-o ServerAliveInterval=60'` now also gets `ServerAliveCountMax=4`: a silent host fails after 240 s, not 180 s. A host dropped before login (sshd's `MaxStartups`) is tried again whatever address ssh's lines name, so OpenSSH 8.0 (RHEL 8) and its scp are retried too; a remote command that prints nothing and whose own ssh to another host is turned away the same way is run again (up to 3 times). A Ctrl-C (or a `-t` shorter than ssh's 30 s connect timeout) during `--copy_id -p` while ssh is still connecting can leave an empty `~/.ssh/ssh-copy-id.*` per host (`rmdir` them). Output lines end only at `\n` (a form feed or another Unicode line break no longer splits a long line), and in output headers, merged groups and `Skipping hosts:`, hosts that do not compact into one range are listed in natural order (`db2,db10`), not sorted as text. A message mssh cannot write to stderr is dropped (6.38 wrote it to stdout, so `mssh ... 2>&1 \| true` under `PYTHONUNBUFFERED=1` exited 1 without running anything). Output before a run into a closed pipe (`mssh -h \| true`, a usage error, the prompts) ends mssh by SIGPIPE: a shell still reports 141, and a program that starts mssh with Python's subprocess now sees -13 (6.38: 141). `--store_config_file`, its `.bak` and `--add_control_master_config` write through one atomic writer: root writing a user's config keeps its owner, and with the default answer `b` the new config keeps the old one's group where it can. Another user's config that you may write only through an ACL (or group or world write) is no longer written when you answer `o` or it is reached through a symlink: mssh prints `Error: Unable to write to the config file: ...` and the config on stdout (6.38 wrote such a file in place when it was 0600 or tighter). Under `-11` with `--error_only`, `--script` or `-Q`, the `Skipping unavailable host` line is no longer printed. A `__multiSSH3_UNAVAILABLE_HOSTS` line in an env file is no longer read. Autocomplete: the completion index is format 3, so the first TAB after the upgrade starts a rebuild (until it is ready, TAB offers options, the remote command's own completion and your shell history; `mssh --completion install` builds it at once). bash completes a remote command as bash-completion completes `sudo CMD`, and a completion spec's `-X`, `-P` and `-S` now apply only to the words compgen makes. In the Ctrl-R menu's history tab, rows written by 6.26 and older go in single-quoted, so the local shell expands no `!`, `$` or backtick in them. zsh no longer adds local file names beside a remote command's words where python-argcomplete is installed. Run `mssh --completion install` again as each account to get the new glue, which the bash and zsh changes above need (older glue keeps working). Quoting the local file names a remote command's completion offers, planned for 6.39 in the 2026.44.1 notes, is deferred to 6.40. **For scripts:** `mssh-complete eval` moved to `python3 tests/replay.py` in a source checkout (it now says so and exits 2); the index rebuild and the host prober lock with `flock`, and their `--lock` is accepted and ignored. **`hpcp` 9.79 → 9.80** (requires `multicmd` 1.56; without it, or with an older one, hpcp uses its built-in copy, the full 1.56): every `<TOOL>_PATH` that names no executable file is now named once at the start of every run (`Warning: CP_PATH=... is not an executable file; using cp from PATH.`), where 9.79 named one only when hpcp started that tool itself, so a bad `CP_PATH` (cp runs in the workers) was never named; a variable other software sets for its own use prints the line too, harmlessly. A removal (plain `-rm`, a move, `-rme`, a forced patch apply: `-siff` with `-fa`) holds one directory descriptor per level of the tree, and hpcp raises its soft open-files limit to the hard one at start, which the tools it runs inherit. A tree deeper than `ulimit -Hn` (524288 on systemd hosts, 4096 on EL7) stops with "Too many open files" errors and leaves its deep part; on macOS the soft limit (256) stays, so trees deeper than about 250 levels stop there, where 9.79 removed them. A forced apply now removes deeper trees than 9.79's, which stopped at the soft limit. `-dd` settles udev (at most 10 s) after its sgdisk runs and before every lsblk partition listing: in 9.79 a listing in the moment udev re-added the partition nodes could miss a partition or its type, so mkfs was skipped and the partition copied with dd (with `-ddr`, exit 234 or a filesystem not grown) or the destination failed (exit 180), and a disk patch (`-dd -diff`, `-siff`) could refuse with a spurious `Disk layout mismatch` on a partition's fs_type. The mke2fs feature probe asks once for all names and, only if that is rejected, once per name, 8 at a time (one unknown name of 31: 32 runs, where 9.79 halved the list in 11 runs, 9 at best). **`tee-logger` 6.43 → 6.44:** compressed logs (as `iotest` writes them) are written as one whole gzip member (xz or bz2 stream, zstd frame) per flush, so a gzip or zstd log stays readable after a run is killed and a restarted run, or a second process with the same `programName`, appends to the same day's file (6.43: unreadable past that point). The cost: gzip and zstd logs with frequent `WARNING`+ records, or `flush_interval=0`, are larger (one record in 100 at `WARNING`+: 2–5% larger; one in 10: 40–45%; `flush_interval=0`: three times the size); xz and bz2 logs are unchanged. A compressed batch that cannot be written (a full disk) is dropped and reported each time, and a partial write is cut back so later records stay readable. **Breaking for library callers:** `teeLogger.GZipFileHandler`, `BZ2FileHandler`, `XZFileHandler` and `ZSTDFileHandler` are gone (`BinFileHandler` stays), and `binary_mode=False` no longer affects compressed logs. Two 6.43 changes are taken back: `flush_interval` is taken with `float()` (`'30'` is 30 s, and NaN acts as `inf`: a write only on `WARNING` and above, once 1 MiB of records is waiting, and at close; `None` still raises `TypeError`), and console lines go through `print()` again, still one write per line, so a replaced `builtins.print` sees them. Compressed records waiting at a `fork()` are written before it (Python 3.7+), a child no longer writes its parent's waiting records again, and a multiprocessing worker that ends normally writes its last records (6.43 duplicated such records, corrupted gzip and zstd logs, and lost a worker's last ones); a child that leaves with `os._exit` or is killed, the workers a Pool's `terminate()` ends included, still loses its unwritten records. The `tee*` methods log to the file before they print, so `prog \| head -1` no longer loses the record from the file. A `logging.FileHandler` you attach to the logger named `programName` is no longer closed when another `teeLogger` of that name is made, and a day-folder named like a date that is not one (`2020-13-45`) is skipped with a message instead of aged by its mtime. **`statblk` 1.56 → 1.58** (1.57 was never pinned; both require `multicmd` 1.56): on Linux, a disk whose earlier smartctl is still stuck (older than `--timeout`, from an earlier refresh, run, cron job or another tool) shows `TIMEOUT` at once and gets no new smartctl, so `statblk -w` and repeated runs leave at most one stuck smartctl per disk instead of one per refresh (a few for a disk hidden by `-M` or a `-D` filter), and the refresh no longer waits out `--timeout` for that disk; macOS and `--timeout 0` are unchanged. The check sees only processes in statblk's own `/proc`: with `hidepid` as a non-root user (`--sudo`), or in a fresh container, it cannot see an earlier stuck smartctl, so run statblk as root on the host, or in hidepid's `gid=` group. Without `lsblk`, the `lsblk not found` note goes to stderr after the table (once per run, and under every table while `-w` redraws a terminal), only when there are rows and a column lsblk feeds (FSTYPE, UUID, MODEL, SERIAL), so a healthy `--check` stays silent; it now shows with `--sudo` too. A reader that closes the pipe early (`statblk \| head`) ends the run with 141, not 1, which `--check` uses for a problem. `--debug` no longer prints the "smartctl not found" hint. Library callers: `get_drives_info` no longer prints the lsblk note, and `cache_decorator`, `clear_caches` and `debug_msg` are gone. **`statbtrfs` 0.37 → 0.38** (requires `multicmd` 1.56): an unreadable filesystem prints only the note under the table (`Could not fetch complete info for <mounts>. Skipping...`); the three raw lines before it are gone. A reader that exits early (`statbtrfs \| head`) ends the run quietly with 141 instead of a BrokenPipeError traceback and exit 1 ("problems found"), and a Ctrl-C outside `-s` and `-w` (while btrfs hangs on a dying device) exits 130 quietly instead of a traceback. `-s` and `-w` clear the screen with `ESC[H ESC[J`, so a long run no longer pushes a table into the scrollback of terminals (VTE-based ones) every `-i` seconds. Library callers: `find_btrfs_mounts()` takes no argument and `execute_command(s)` lose `wait=`. **`firewatcher` 1.63 → 1.64:** `--install-service` and `--print-unit` write each `--requires-mounts-for` path the way systemd reads it back: made absolute and normalized as text (`.`, `..`, `//`, a trailing `/`; symlinks are not followed, as systemd follows none), in double quotes when it holds whitespace, a quote or a backslash, and with `%` doubled; a path with a newline or carriage return, or one that is not UTF-8, is refused before anything is installed. 1.63 wrote the path as given, so systemd split one with a space into two, expanded a `%` in it, and ignored a relative one or one with `..`, silently dropping the dependency. A unit already installed keeps its old line until `--install-service` writes it again. |
| `2026.44.2` | 2026-10-27 | Mid-week hotfix. **`firewatcher` 1.60 → 1.63** (1.61 and 1.62 were merged on GitHub as PRs #1 and #2 but never published; the firewatcher 1.61 on PyPI is a superseded build, use 1.63). **The LLM now sees the matched event:** a capture larger than `[llm] max_bytes` is sent as its header, the lines before and after the match and the end of the capture, where 1.60 sent the first 2 KB and the last ~46 KB and cut the event out of most large captures. Prompts fence log data as untrusted, `--always-notify` patterns are pushed whatever the model says, and the ntfy topic URL is redacted from output and errors. New optional `[llm] notes` / `--llm-notes`: a short note about the host, sent to the model as trusted context with each capture. **Behaviour changes:** a pattern set with no usable pattern now stops the daemon instead of watching for nothing, and settings are range-checked at startup, so a config 1.60 accepted can stop the service (check `journalctl -u firewatcher` after the restart); journalctl runs with `-o short-iso`, so journal lines in captures carry ISO timestamps; the journal is read only where journald runs or its files exist, otherwise `/var/log/syslog` or `/var/log/messages`. Captures keep going when the output folder fails (`--fallback-folder`), SIGTERM finishes the open capture, `--capture-time-max` caps how long repeated matches extend one, `--min-free-percent` keeps captures from filling a disk, and `--buffer-max-mb` bounds the memory for the lines kept before a match. After a restart the journal resumes from a saved position (`--position-file`), up to `--resume-max` (default an hour) back, so journal lines logged while it was stopped are matched once. SIGHUP reloads the patterns; `systemctl reload` needs the unit's `ExecReload=`, which units written by `--install-service` from 1.62 on have. Units written by `--install-service` now escape `%` and `$`, quote `'` and a bare `;`, and write newlines as C escapes in ExecStart arguments; a unit already installed keeps its old ExecStart until it is written again. A running firewatcher service keeps its old version until it is restarted. |
| `2026.44.1` | 2026-10-26 | **`hpcp` 9.78 → 9.79** (requires `multicmd` 1.55): **Behavior change:** every `<TOOL>_PATH` override is now the program hpcp starts, bare command names included (`CP_PATH=gcp`), also for the tools it runs through multiCMD (before, only mkfs and losetup used them), so a stray `MOUNT_PATH` or `CP_PATH` in the environment that names an executable file now runs as that tool; one that names no executable file is ignored, and the tool on PATH runs. "Running command:" lines show the tool's full path. The tools whose output hpcp parses (xfs_info, dumpe2fs, blkid, sgdisk, lsblk, df and others) run in the C locale, so on a translated system `-dd` no longer silently stops mirroring xfs and ext geometry. The mke2fs feature probe ignores mke2fs.conf (a stanza naming a feature this e2fsprogs lacks made hpcp drop every feature it meant to keep), asks once for all names and halves a rejected list: one mke2fs run when every name is known and about 9 for one unknown name of 31, where 9.78 ran one per name (31); with many unknown names (e2fsprogs 1.42.9) it takes more, 45 against 31. A multiCMD timeout is recognized wherever `Timeout!` sits in stderr (a tool's reply to SIGINT after it made it a return-code error). A tool that exits 124 or 137 with nothing on stderr no longer crashes hpcp. The 2026.43.1 note on `/hpcp-ext-feature-probe` is corrected: e2fsprogs 1.45.x–1.46.2 (EL8, Debian 11, Ubuntu 20.04) created it; 1.46.5 (Ubuntu 22.04) and 1.47 did not, and EL9 is untested. **`tee-logger` 6.42 → 6.43:** **Breaking:** where a compressed log file uses it, `flush_interval` must be a number: `None` or `'30'` raise `TypeError` and NaN raises `ValueError` when the logger is made (6.42 printed a logging error for every record below WARNING for the first two, and kept a CPU core busy for NaN). `float('inf')` no longer prints a traceback from the flusher thread. Each console line is one `write()`, so lines from several threads no longer interleave and a replaced `builtins.print` no longer changes them. A failed background flush is reported once instead of stopping the flusher for good (with xz and bz2 that stream's records are still lost); records logged during the midnight switch are no longer lost; a failed close of the old log, or a stderr that cannot be written, no longer makes a log call raise; a signal handler that logs while its own thread rolls the log over no longer deadlocks. It now ships the GPL-3.0 LICENSE in its sdist and wheel. **`multicmd` 1.54 → 1.56**, with **`statblk` 1.55 → 1.56** and **`statbtrfs` 0.36 → 0.37** embedding 1.55: `quiet=True` now also hides the lines about a command that could not be run (`Command path not found`, 127; `Command cannot be executed`, 126; -1), whose reasons stay in `task.stderr`, and the unused `__running_threads_lock` and `_live_procs_lock` are gone. A Ctrl-C, or a SIGTERM handler's `sys.exit()`, stops the commands a call waits for, skips the queued ones and records them as stopped (1.54's `join_threads()` left them running); at exit, a command still queued on multiCMD's threads fails (-1, "not started: the interpreter is exiting") instead of starting and outliving the program; a command not seen to exit 1 s after SIGKILL is no longer tracked, and its pid is named on stderr ("not seen to exit 1 s after SIGKILL"). 1.56 does this with simpler interrupt handling, about 700 lines lighter; its rare corners (for example, two Ctrl-Cs microseconds apart, a Ctrl-C inside `Popen` as a serial run starts a command) are listed in multiCMD's docs/limitations.md. New per-call environment: `env=` and `extra_env=` on `run_command`, `run_commands` and `AsyncExecutor`, and `mcmd -e NAME=VALUE` / `-e NAME`. A command that times out while it redraws a `\r` progress line keeps its `Timeout!` line. Without lsblk, statblk says so on stderr (`lsblk not found (it comes with util-linux): …`) and still prints the table; without btrfs-progs, statbtrfs says `btrfs not found on PATH: …` and exits 2 (not when each btrfs command has to run through sudo). **`multissh3` 6.37 → 6.38** (`mssh-complete` 0.10 → 0.11), autocomplete only; run `mssh --completion install` again as each account to get the new glue (older glue keeps working, without the newer features). bash completes a remote command's words from its own completion spec as bash would (`-C` completers such as aws, terraform and vault, `-W` word lists, `-G` globs, actions, `-X`, `-P`/`-S` and the `-o` options), where 6.37 ran only a `-F` function and offered local file names for the rest. Under readline, with a spec that has `-o nospace` (git, scp, rsync), mssh's words from history keep their trailing space and git-style words that end in a space no longer get a second one. zsh and fish pass the shell's locale to the engine; where `mssh-complete` is not on PATH the engine starts from cached bytecode (a bare query: 57 ms → 16 ms on Python 3.13); with `completion-ignore-case` on, `mssh WEB:a` groups only names that can be offered. Local file names offered for a remote command (by its spec's actions or `-G`, or when bash has no spec for it) are not yet quoted by mssh (planned for 6.39). |
| `2026.43.1` | 2026-10-21 | **`multissh3` 6.31 → 6.37.** **Breaking:** mssh no longer takes an abbreviation of one of its options (`--user` for `--username`, `-c` for `-ci`); spell them in full. Hosts skipped as recently unreachable now fail the run (hosts excluded with `-X` do not), and targets that name no hosts are an error. The word after `-ea` and `-ia` is their value also when it starts with `-` (`-ea -v`); a bare ssh or ipmitool option that needs its own value there (`-ea -p 2222`, quotes forgotten), or an mssh password option, still stops mssh. **Safety, and it needs a reinstall:** in a shell with `nocasematch` on (or after a completion file turned it on), bash's completion read `complete -f cmd` as `-F`, and TAB ran the remote command locally (6.31–6.36). The shell glue is a copy installed per user account, so for every account on every host where `mssh --completion install` was run, **run it again as that account**: until then the old glue, and the bug, stay. With bash's `completion-ignore-case` on, completion now matches the word in any case, as the shell's locale lowers letters; bash's completion gives the shell its `nocasematch` back after a Ctrl-C, keeps its variables from the shell's own completion functions, and works with any `IFS`. ssh can no longer prompt behind the curses view or show a typed password; `ConnectTimeout` 30 s and keepalives are the default (your ssh config wins), so one hung host no longer stops a run. File copies: `-ea`'s ssh options no longer reach rsync as files (a private key passed with `-i` was copied to the hosts). Passwords reach sshpass and ipmitool through their environment, not their command line; a `-p` password is not echoed, and the `> mssh` line and the history say `-p -` (`MSSH_PASSWORD` gives one; `--store_config_file` still writes it to the config file). `--ipmi` tries admin/admin only when no other login is configured. `root@dbs` with an alias whose members carry their own user no longer logs in as `root@postgres@db1`. The TUI no longer polls (1000 waiting hosts: 0.13 cores instead of 1.47), and output lines that hosts print alike are kept once (200 hosts × 20,000 identical lines with `-P`: 3.0 GB → 0.17 GB). A bare `mssh` with `DEFAULT_HOSTS` null no longer crashes (it means all hosts), and with `$XDG_RUNTIME_DIR` set, mssh reads its unreachable-host state where it writes it. **`hpcp` 9.73 → 9.78:** `hpcp -rm /` is refused (9.73 hung on it). Plain `-rm` and `-rme` remove by directory descriptor, so a directory swapped for a symlink in a tree others can write to no longer leads them outside it. Legacy `-fl -rme` with a bare relative destination (`data`, `.`) no longer builds its extras under `/` (9.73 removed `/data/extra`). `-rme` leaves a filesystem mounted below the destination alone with its files, as `rsync -x` does: bind mounts, btrfs subvolumes, and automounts and NFSv4 submounts mounted on entering a directory. Trees deeper than Python's recursion limit are listed, copied and removed. On Python 3.6, `-rm -pfl -F`, non-UTF-8 file names and `-rme`'s automount checks work. `-dd` no longer leaves `/hpcp-ext-feature-probe` in `/`: e2fsprogs 1.45.x–1.46.2 (EL8, Debian 11, Ubuntu 20.04) created it when 9.61–9.77 probed mke2fs as root; 1.46.5 (Ubuntu 22.04) and 1.47 did not, and EL9 is untested. It is an empty root-owned file, safe to delete by hand. The probe reads mke2fs in the C locale: a translated mke2fs made every feature count as known. **`tsvz` 4.7 → 4.9:** a read is redone when another tool rewrites a part in place under it (a scrub), and a read still changing after three tries is incomplete: `get` and `read` exit 1 and the handler refuses `pop` and `setdefault`. Reads start at the last snapshot part (`#_snapshot_#`, spec §19.10). `scrub` refuses a store declaring a spec version newer than it implements and writes nothing; so does `clear` on a single-part store (on a multi-part one it still appends tombstones). A marker value ending in a newline (`0<LF>`) no longer crashes the read. **`simple-iotest` 3.69.1 → 3.70.0:** new `--json` summary, `-bs/--block_size`, `--verify` (a CRC-32 trailer checked on every read), `meta` mode (create, stat and unlink of empty files), `--start_at` for synchronized multi-host starts, and an opt-in per-operation timeline (`-tl`). Fixed: lost results on long `-d` paths, the report printed twice, and misleading cache-drop and sync figures; invalid sizes exit 2. **`multicmd` 1.53 → 1.54**, with **`statblk` 1.54 → 1.55** and **`statbtrfs` 0.35 → 0.36** embedding it: the terminal-size fallback reads the columns, not the rows, so `statblk \| less` no longer squeezes the table to the terminal's height. These three, **`firewatcher` 1.59 → 1.60**, `multissh3`, `tsvz` and `simple-iotest` now ship the GPL-3.0 LICENSE in their sdist and wheel (`tee-logger` 6.42 does not yet). |
| `2026.42.2` | 2026-10-16 | **Hotfix: `hpcp` 9.72 → 9.73.** `-rm` with a destination (a move) no longer deletes sources it did not copy: it is skipped after any copy error (`Remove skipped`, 212), and otherwise removes only the files and links the copy listed, then empty directories, so files written during the move, FIFOs, sockets, device nodes and the files of a `-do` copy stay; a move onto itself (`hpcp -rm /data/x -d /data/`) is refused, and `-rm -di NEW.img` creates the image first (9.72 removed the sources and made no image). The move's removal goes by directory descriptors without following symlinks, so a source or directory swapped for a symlink during the copy is not removed from, and it needs about half of 9.72's metadata operations. `-rme -di` no longer empties the image it just filled; `-rme` keeps the copy of an excluded path (absolute or not, `-e 'cache/'` included) and a filesystem's `lost+found`; a `-d` glob no longer yields `/`. One unreadable file or directory no longer stops a copy (`Listing error`, 209; the rest is copied), and `-diff` refuses a partial listing instead of writing delete rows. Link tracking is opt-in (`-lt`; it had been inactive since May 2025). `-dd` mounts source partitions read-only unless they are mounted live, and xfs clones mount with `nouuid`. Pool workers get the run's settings (`-co`, `-ncd`, `-hs`, `-ctl` on macOS, Windows and Python 3.14+). |
| `2026.42.1` | 2026-10-13 | **`tsvz` 3.39 → 4.7 (major):** `.tsvz` / `.csvz` / `.nsvz` / `.psvz` files now follow tsvz-spec-v1, so a file 3.39 wrote under one of those names reads differently — rename it to `.tsv` (see TSVZ's *Breaking changes from 3.39*); `.tsv` / `.csv` get bug fixes only. 4.7 passes the whole spec conformance suite: carriage returns and zero bytes in values are written as `<CR>` and `<NUL>` (older versions read those as text, so stop older `tsvz serve` handlers before upgrading; a value 4.2/4.3 wrote with a raw zero byte now reads as damage), `parts` names codecs, and a part in a codec this Python lacks (zstd before 3.14) is refused. Damaged compressed members and zero-filled lines are recovered, skipped or stop the read as the store's `#_on_corruption_#` says (recover by default); appends never read the store; a key ending in whitespace can be read and deleted by name, so `tsvz keys s \| tsvz delete s -` deletes every key; tables line up CJK text. `multissh3` 6.25 → 6.31: config and env files are only trusted when only you (or root) can change them and are no longer read from the current directory (a host name defined only there now says so), stdin is closed when no TUI runs, faster output reading and host lookups, `~/.ssh/config` aliases and IPv6-only hosts, one Ctrl-C cancels the run, the summary table lines up CJK and colored output, and piped output (`mssh ... \| less`) is sized by the terminal's width instead of its height. New opt-in shell autocomplete for bash, zsh and fish (`mssh --completion install`, new `mssh-complete` command): TAB completion of hosts, options and commands from history and every host source, grey suggestions, a Ctrl-R picker, and warnings before Enter about unknown or unreachable hosts and conflicting options, from a background prober that only uses DNS, `ssh -G` (which runs the ssh config's `Match exec` commands) and ping. `multicmd` 1.49 → 1.53: real exit status, `-t` no longer kills a command after 60 s by default (0 = never is the new default; pass `-t 60` to keep the old behavior), commands from files or stdin, tables line up CJK text. `statblk` 1.42 → 1.54: `--check` health mode with exit codes, spun-down drives are not woken for SMART, CJK names and labels line up in the table. `statbtrfs` 0.30 → 0.35: `-w/--watch`, btrfs-progs 6.x missing devices, CJK paths line up in the table. `simple-iotest` 3.68.0 → 3.69.1: failed I/O ops are counted and make the exit code 1 (was always 0), files of 2 GiB and up are written and read in full, a run that won't fit on disk is refused before anything is written, new `-u/--unique_files` (dedup-proof test data) and `-cl/--cleanup`, `-addr` no longer needs `curl`, and read/index runs need half or none of the data memory. `tee-logger` 6.40 → 6.42: hang, crash-data-loss and log-cleanup fixes; the `Log file:` banner moves from stdout to stderr, compressed logs (as `iotest` writes them) use zstd instead of xz on Python 3.14+, and an unwritable log directory falls back to `$TMPDIR/teeLogger-<uid>/`. `hpcp` 9.64 → 9.72 (embeds multiCMD 1.52): disk-level patch images (`-dd -diff` makes and `-siff` applies a patch of whole GPT disks: changed files per filesystem, raw areas, identity values), a repair step and exact-case names on apply, exclusive opens and a disk lock while writing, labels / GPT names / exFAT serials copied exactly, `-e` matched inside each mounted filesystem, automounted ESPs and btrfs subvolume roots handled, forced removals that cannot leave the target, and a fix for `-diff /` with a blank line in its `-x` file (or `-e /`), which wrote a patch deleting every file: such patterns are now refused. `firewatcher` 1.58 → 1.59 (less CPU and memory per line). |
| `2026.40.1` | 2026-09-28 | `simple-iotest` 3.67.0 → 3.68.0: IOPS (call/sync) and op counts in every mode's report, including index; fixed worker count for repeated role keys (e.g. `write s`), which over-reported size and call bandwidth. |
| `2026.39.2` | 2026-09-23 | Mid-week hotfix. Restored `requires-python >= 3.6` (2026.39.1 had raised it to 3.8 for TSVZ 4.0 while still pinning `tsvz` 3.39, so pip on 3.6 silently stayed on 2026.38.1). `hpcp` 9.62 → 9.64, `multissh3` 6.24 → 6.25, `firewatcher` 1.57 → 1.58. |

Full history is in the [commit log](https://github.com/yufei-pan/cas-toolbox/commits/master).

## Requirements

- Python >= 3.6

Nothing outside the standard library is required; the optional libraries below
only add features or speed.

## Optional Python libraries

Heavier libraries are imported defensively — every tool runs without them, just
with reduced functionality or performance.

| Library | Used by | Effect when present |
|---|---|---|
| `curses` | `multiSSH3` | Interactive TUI instead of the plain text output |
| `numpy` | `iotest` | Much faster random-number generation in the I/O benchmark |
| `xxhash` | `hpcp` | Faster hashing; falls back to `hashlib.blake2b` |
| `resource` | `multiSSH3`, `TSVZ` | Resource-usage reporting |
| `argcomplete` | `TSVZ`, `statblk`, `statbtrfs` | Shell tab completion (`pip install '<tool>[completion]'`) |

`xxhash` is declared by `hpcp` but imported defensively, so `hpcp` still works
without it.

## Installation

Use pip:

```bash
pip install cas-toolbox
```

Use pipx:

```bash
pipx install cas-toolbox
```

Use uv (as an isolated tool, with `numpy` for the `iotest` benchmark):

```bash
uv tool install --with numpy cas-toolbox
```

Use uv to add it as a dependency:

```bash
uv add cas-toolbox
```

> **Note on `numpy`:** it is only used by `iotest`'s random-number generator and
> is not needed anywhere else in the toolbox.

To install one tool on its own, install its PyPI package directly instead of this
meta-package.

## Commands provided

| Command | Tool |
|---|---|
| `hpcp` | hpcp |
| `mcmd` / `multicmd` / `multiCMD` | multiCMD |
| `mssh` / `mssh3` / `multissh` / `multissh3` / `multiSSH3` | multiSSH3 |
| `mssh-complete` | multiSSH3 (autocomplete engine behind `mssh --completion`) |
| `iotest` / `simple-iotest` | simple-iotest |
| `statbtrfs` | statbtrfs |
| `tsvz` / `TSVZ` | TSVZ |
| `statblk` | statblk |
| `firewatcher` | firewatcher |

Every command supports `--help` / `-h` and `-V` / `--version`, except
`mssh-complete`, the engine the shell glue runs (run it bare for its usage).

## Verifying an install

```bash
hpcp -V && mcmd -V && mssh3 -V && iotest -V && statbtrfs -V \
  && TSVZ -V && statblk -V && firewatcher -V
```

`cas-toolbox` itself exposes no command — it is a dependency bundle only.

From a checkout that has the nine upstream repositories side by side,
`getCASVersions.sh` prints every tool's version at once:

```bash
./getCASVersions.sh
```

## Development

This repository holds only the meta-package (`pyproject.toml`, `uv.lock`,
`README.md`, `getCASVersions.sh`). The upstream sources live in their own
repositories — see the links below.

```bash
uv sync --frozen     # resolve the locked dependency set
uv build             # build wheel + sdist into dist/
```

Pins are exact (`==`) on purpose: the bundle is meant to be reproducible.

## Author

- Yufei Pan (pan@zopyr.us)

## License

GPL-3.0-or-later

## Links

- Project home: [github.com/yufei-pan/cas-toolbox](https://github.com/yufei-pan/cas-toolbox)
- [hpcp](https://github.com/yufei-pan/hpcp)
- [multiCMD](https://github.com/yufei-pan/multiCMD)
- [multiSSH3](https://github.com/yufei-pan/multiSSH3)
- [simple-iotest](https://github.com/yufei-pan/simple-iotest)
- [statbtrfs](https://github.com/yufei-pan/statbtrfs)
- [Tee_Logger](https://github.com/yufei-pan/Tee_Logger)
- [TSVZ](https://github.com/tsvz-org/TSVZ)
- [statblk](https://github.com/yufei-pan/statblk)
- [firewatcher](https://github.com/yufei-pan/firewatcher)
