Metadata-Version: 2.4
Name: ibm-aiu-smi
Version: 1.3.0
Summary: A tool to collect performance metrics from IBM Spyre during an execution of AI workload
Author: Torch-Spyre Authors
License-Expression: Apache-2.0
Keywords: IBM Spyre,monitoring,performance,device
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: System Administrators
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: System :: Monitoring
Requires-Python: >=3.12
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: spyremetrics>=0.5.0
Requires-Dist: psutil>=5.9.6
Provides-Extra: build
Requires-Dist: build; extra == "build"
Requires-Dist: setuptools<80.0,>=70.1.0; extra == "build"
Requires-Dist: wheel; extra == "build"
Provides-Extra: test
Requires-Dist: pytest~=9.0.0; extra == "test"
Requires-Dist: pydantic; extra == "test"
Requires-Dist: pytest-xdist; extra == "test"
Requires-Dist: transformers~=4.57.1; extra == "test"
Provides-Extra: lint
Requires-Dist: mypy~=1.13.0; extra == "lint"
Requires-Dist: pre-commit~=4.5.0; extra == "lint"
Requires-Dist: types-PyYAML~=6.0.12.20250915; extra == "lint"
Requires-Dist: uv~=0.10.0; extra == "lint"
Dynamic: license-file

# aiu-smi

`aiu-smi` is a tool to collect performance metrics during an execution of AI workload and
print them or save them to a file for further analyses, like `nvidia-smi dmon`.
It uses AIU's H/W performance counter to capture the performance metrics, as well as
software counters for full-stack performance analysis.
This version uses new API to access metric files, served by `spyremetrics` repo.

## How to install

### Prerequisite: Install `spyremetrics` package

`spyremetrics` package is a Python library to access metric files created by the Spyre runtime software.
We are preparing to publish this package from PyPI.

Meanwhile, please clone `spyre-metrics-api` repo and install it as:

```bash
git clone git@github.ibm.com:ibm-aiu/spyre-metrics-api.git
cd spyre-metrics-api
pip install --editable .
```

### Install `aiu-smi` package

We are also preparing to make `aiu-smi` package available from PyPI.

Meanwhile, install it from GHE repo:

```bash
git clone git@github.ibm.com:ibm-aiu/aiu-smi.git
cd aiu-smi
pip install --editable .
```

### PF-mode support: Install `aiu-monitor` RPM package

**NOTE:** This step is only needed to use `aiu-smi` in PF mode.

In PF mode, AIU runtime software creates performance metric files in the old format.
`spyremetrics` library uses a shared library `aiu-monitor.so` to read data from metric files
in the old format. Use `aiu-monitor` RPM package to install the shared library.

```bash
$ rpm -i ibm-aiu-monitor-1.3.0-1.x86_64.rpm
$ pip3 install aiu-smi
```

> **TIP**
> Installing an RPM package usually needs `root` access.
> To install by a general user, use following options of `rpm` command:
> ```bash
> $ rpm -i --nodeps --dbpath=$HOME/.local/lib/rpm --relocate /usr/lib=$HOME/.local/lib \
>     --relocate /opt/ibm/spyre/aiu-monitor=$HOME/.local --badreloc ibm-aiu-monitor-1.3.0-1.x86_64.rpm
> $ ls $HOME/.local/lib/
> libaiumonitor.cpython-312-x86_64-linux-gnu.so  rpm/
>
> # Specify --dbpath option to access the user-installed package info
> $ rpm -q --dbpath=$HOME/.local/lib/rpm ibm-aiu-monitor
> ibm-aiu-monitor-1.3.0-1.x86_64
> ```

## How to use `aiu-smi`

### Overview of operations

As shown in the following figure, some environment variables need to be set in each of two terminals: one for running workload and the other for running `aiu-smi`:

![aiu-smi overview](docs/aiusmi_overview_pub2.png)

### Note on VF mode
In VF mode, performance counters are configured when PFs are initialized (typically when the system starts up)
and Spyre workload users cannot change the settings, unlike in PF mode.
Since the older default configurations of performance counters are not compatible with what `aiu-smi` expects, the output of `aiu-smi`
will be incorrect, unless the default system config is updated. Specifically, with the old configs, `rdmem`, `wrmem`, and `rxpci`
become always 0, and `txpci` may show smaller values than expected. `busy%` may be inaccurate because it is derived from
`rdmem`, `wrmem`, `rxpci`, and `txpci`.
Sample output showing the difference between old and updated configs is [here](docs/vfmode_output_comparison_pub1.png).

For OpenShift environment, Spyre operator version 1.3 or later will support the new default that is compatible
with `aiu-smi`.

### Setting up environment variables
#### Mandatory settings
1. **aiu-smi terminal**

   There is no mandatory environment variable to run `aiu-smi`, but the optional setting below is recommended.

2. **Workload terminal** – tell Spyre driver to create metric files

   **For PF mode**, if installed from the wheel package:  
   ```bash
   cp <cloned-aiu-smi>/etc/senlib_config_aiusmi.json $HOME/.senlib.json
   ```
   If you already have `$HOME/.senlib.json`, merge the settings for `aiu-smi` into the existing config:
   ```bash
   mv $HOME/.senlib.json $HOME/.senlib.json.orig
   jq -s add $HOME/.senlib.json.orig <cloned-aiu-smi>/etc/senlib_config_aiusmi.json > $HOME/.senlib.json
   ```
   **For VF mode**,  
   ```bash
   export AIUPTI_ENABLE_METRICS=true
   ```

  The default metric file format in VF mode as of June 16, 2026 is the **old** format.
  To create a metric file in the new format, export following environment variable, too:  
  ```
  export AIUPTI_USE_NEW_FORMAT=true
  ```
  This setting will not be needed when the default is changed to the new format.

#### Optional settings to enable `actmem` and `peakmem` metrics (for torch-spyre only)

**Workload terminal** – Enable memory tracking in Flex:
```bash
export FLEX_ENABLE_MEMORY_TRACKING=1
```
<details>
<summary>Note on memory tracking files</summary>
The memory values are stored in `/tmp/memtrack/memory_{device_ID}.log` file by default.
This can be changed by setting the `MEMORY_TRACKING_FOLDER` env variable to the same value, in *both* the workload and aiu-smi terminals.
If `MEMORY_TRACKING_FOLDER` is set to `xyz`, the file which the code looks for would be `xyz/memory_{device_ID}.log`,
where the device_ID is filled by the code using the LOCAL_RANK for each available device.
</details>

### Run AIU workload with enabling performance counters
`senlib` loads config files during its initialization phase.
In PF mode, you can tell the path of your senlib config file to `senlib` by copying `configs/senlib_config_aiusmi.json` as
`~/.senlib.json`.

In VF mode, `flex` loads `libaiupti.so` and starts the metrics sampler thread by setting an environment variable
`AIUPTI_ENABLE_METRICS=true`. `libaiupti` creates a metrics file `/tmp/metrics.<busid>`, in the same manner as PF mode.

Until the new format becomes the default, `export AIUPTI_USE_NEW_FORMAT=true` is also needed.

<details>
<summary>Note on modifying `senlib_config_aiusmi.json` file</summary>
Current version of `aiu-smi` can collect metrics from multiple AIU devices.  For multi-aiu runs, make sure the senlib's
metric file path contains a magic string `%BUSID` to avoid overwriting the same metric file by multiple processes.
The magic string will be replaced by the actual PCIe bus ID when senlib creates a metric file.

```json
{
  "METRICS": {
    "general": {
      "enable": true,
      "path": "/tmp/metrics.%BUSID",
    }
  }
}
```
</details>

### Run `aiu-smi` to print performance summary
**Simultaneously** run [`aiu-smi`](aiu-smi/aiu-smi)
in **another terminal**.
Note that this command needs to run in the same AIU container as the AIU workload.

In the case of distributed workload launched by `torchrun`, the Spyre driver layer allocates an AIU device for each of child processes based on
the hardware configuration specified by the OpenShift AIU operator. This configuration is shown in
`.GENERAL.sen_busid[]` of `/etc/aiu/senlib_config.json` and copied to environment variables `PCIDEVICE_IBM_COM_AIU_PF`
and/or `AIU_WORLD_RANK_0`, `AIU_WORLD_RANK_1`, etc.

`aiu-smi` shows a list of metric files for each device with `ID`, and then periodically show metric data.
Each line shows the metrics of a single device (for the `ID` at the first column).
For example, if the container has two devices. `aiu-smi` prints two lines every second.

```
$ echo $PCIDEVICE_IBM_COM_AIU_PF
0000:13:00.0,0000:12:00.0
$ env | grep AIU_WORLD_RANK_
AIU_WORLD_RANK_0=0000:13:00.0
AIU_WORLD_RANK_1=0000:12:00.0
$ /tmp/aiu-perf-toolkit/bin/aiu-smi  # or $HOME/.local/bin/aiu-smi
#MetricFiles
# 0 /tmp/metrics.0000:13:00.0
# 1 /tmp/metrics.0000:12:00.0
#ID Date      Time     hostcpu hostmem    pwr  gtemp   busy    rdmem    wrmem    rxpci    txpci   rdrdma   wrrdma   pt_act
#   YYYYMMDD  HH:MM:SS       %       %      W      C      %     GB/s     GB/s     GB/s     GB/s     GB/s     GB/s        %
  0 20250617  21:14:01   276.0     4.9      -      -      -        -        -        -        -        -        -        -
  1 20250617  21:14:01   276.0     4.9      -      -      -        -        -        -        -        -        -        -
  0 20250617  21:14:02  1245.3     4.9      -      -      -        -        -        -        -        -        -        -
  1 20250617  21:14:02  1245.3     4.9      -      -      -        -        -        -        -        -        -        -
  ...
  0 20250617  21:14:06    34.1     5.0      -      -      -        -        -        -        -        -        -        -
  1 20250617  21:14:06    34.1     5.0    0.2   49.4      0    0.000    0.000    0.000    0.000    0.000    0.000     0.00
  0 20250617  21:14:07  1068.8     5.0   20.8   50.0      1    0.001    0.000    0.001    0.000    0.000    0.000     0.00
  1 20250617  21:14:07  1068.8     5.0   21.9   49.1      0    0.000    0.000    0.000    0.000    0.000    0.000     0.00
  0 20250617  21:14:08  1216.7     5.0   21.6   50.0      0    0.000    0.000    0.000    0.000    0.000    0.000     0.00
  1 20250617  21:14:08  1216.7     5.0   21.8   49.4      0    0.000    0.000    0.000    0.000    0.000    0.000     0.00
  0 20250617  21:14:09   464.4     5.0   23.2   50.6     38    7.631    0.035    0.032    0.004    0.007    0.000     0.00
  1 20250617  21:14:09   464.4     5.0   23.2   49.1     39    7.633    0.037    0.032    0.004    0.007    0.000     0.00
  0 20250617  21:14:10   295.1     5.0   26.7   50.6    100   22.767    0.103    0.003    0.013    0.021    0.000     0.00
  1 20250617  21:14:10   295.1     5.0   26.3   49.4    100   22.753    0.111    0.003    0.013    0.021    0.000     0.00
  0 20250617  21:14:11   326.6     5.0   24.5   50.0     84   18.431    0.084    0.002    0.011    0.017    0.000     0.00
  1 20250617  21:14:11   326.6     5.0   24.7   49.1     84   18.436    0.089    0.002    0.011    0.017    0.000     0.00
  0 20250617  21:14:12   345.6     4.9   22.1   50.0      0    0.000    0.000    0.000    0.000    0.000    0.000     0.00
  1 20250617  21:14:12   345.6     4.9   21.8   49.4      0    0.000    0.000    0.000    0.000    0.000    0.000     0.00
```


#### Description of metrics
|Metric group| Metric name  | Description |
|:-:|:------------:|:------------|
|   |hostcpu       |CPU usage in the host (100% = 1 core, not container-aware) |
|   |hostmem       |Memory usage in the host (not container-aware) |
|   |pwr           |Card power consumption |
|   |gtemp         |Maximum temperature of nine sensors |
|   |busy          |Time-based card utilization (number of 10ms internal sampling intervals that perform memory and/or data transfer activities) |
| D |rdmem, <br /> wrmem  |Average bandwidth to read/write from device memory to RaPid cores and vice versa |
| D |rxpci, <br /> txpci  |Average bandwidth to read/write from/to host PCIe |
| D |rdrdma, <br /> wrrdma|Average bandwidth to read/write data via AIU-to-AIU RDMA |
| U |peakmem, <br /> actmem|(torch-spyre only) Peak and actual device memory usage (in GiB) allocated by `flex` for the workload. </ul>|
| P |pt_active     |Estimated PT-array utilization calculated from power consumption, based on empirical measurement of their correlation|

### Notes on metric values
- Until senlib (device driver) initializes AIU, AIU-related metrics (i.e., metrics except hostcpu and hostmem) are shown as `-`.
- After finishing an AIU workload, the same metric values are printed repeatedly for a few seconds
  because metric files are not updated any more, and then
  `-` are shown again for AIU-related metrics when `aiu-smi` detects metric files became stale.

### Command line options
`aiu-smi --help` shows usage:
```bash
$ /tmp/aiu-perf-toolkit/bin/aiu-smi --help

Usage: aiu-smi [dmon] [-s] [-i <device-ids>] [-g <metric-list>] [-d <delay>] [-f <output-file>]
                      [--idle-power <watts>] [--llm-type <model-type>] [-v] [<metric-files>...]

options:
  -s | --csv:           Output csv format
  -i | --id:            Comma-separated device IDs to monitor (e.g., 0,1,2) [default=all]
  -g | --metric-groups: Select metric groups to print [default=DP]:
                          D=device Data rate, U=actual and peak device memory Usage (torch-spyre only),
                          P=Pt_act, A=All metrics.
                        Always print dev. ID, timestamp, host cpu/mem, power, temperature, and busy.
  -d | --delay:         Console summary interval in seconds [default=1sec]. Accept floating-point value.
                        Minimum is 0.1 sec on s390x console or VF mode, otherwise 0.01 sec.
  -f | --filename:      Log to a specified file, rather than to stdout
  --idle-power          Idle power baseline (W) for pt_act estimation [default=20.0]
  --llm-type            LLM model type for pt_act estimation: encoder|decoder [default=decoder]
  -v | --version:       Show version
  -h | -? | --help:     Show help message
```
**Note:** Currently, metric group 'U' implies 'D' because of implementation issue.
**Note:** `aiu-smi` also takes options from an environment variable `AIUSMI_OPTS`.
Command line options supersedes the options specified by the environment variable.

### Monitoring specific devices

Monitor specific devices using `-i` or `--id`:

```bash
$ aiu-smi -i 0,2              # Devices 0 and 2
$ aiu-smi --id=0,1 -d 2       # With 2-second interval
```

**Format:** Comma-separated IDs (e.g., `0,1,2`). Supports `-i 0,1` or `--id=0,1`.

**Example:**
```bash
$ aiu-smi -i 0,2
INFO: Monitoring 2 device(s): 0 2
#ID Date      Time     ...
  0 20250617  21:14:01 ...
  2 20250617  21:14:01 ...
```

IDs are zero-indexed from `AIU_WORLD_RANK_*` environment variables (or `PCIDEVICE_IBM_COM_AIU_PF` in OpenShift environments). Out-of-range IDs skipped with warning.

**Note:**
`aiu-smi` **removes an existing memory-mapped files** if there is no process that opens the files
because the existing file should be created by a workload ran in the past and
it should not be needed any more.
Be careful not to run `aiu-smi` after finishing the workload if you still need it,
otherwise `aiu-smi` removes the memory-mapped file.

## Additional Tools

### PT Active Modeling Tools

[This toolset](pt_active_model/) processes FLEX job trace JSON files and builds predictive models for PT Active based on Power consumption. The workflow supports splitting a multi-job trace into individual job files for separate analysis, and automatically analyzes idle power characteristics.

See [../pt_active_model/README.md](pt_active_model/README.md) for usage details.

### LLM Memory Calculator

For LLM inference workloads, the [LLM Memory Calculator](llm_calc/) provides theoretical calculations to help validate aiu-smi measurements:
- Expected memory footprint of model weights and KV Cache
- Theoretical bandwidth requirements (validates `rdmem`/`wrmem`)

See [../llm_calc/README.md](llm_calc/README.md) for usage details.
