Metadata-Version: 2.4
Name: spyremetrics
Version: 0.5.0
Summary: Python library for reading Spyre metric files
Author: Torch-Spyre Authors
License-Expression: Apache-2.0
Keywords: IBM Spyre,performance,metrics
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: System Administrators
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: System :: Monitoring
Classifier: Typing :: Typed
Requires-Python: >=3.12
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: numpy
Requires-Dist: psutil>=5.9.6
Provides-Extra: build
Requires-Dist: build; extra == "build"
Requires-Dist: setuptools<80.0,>=70.1.0; extra == "build"
Requires-Dist: wheel; extra == "build"
Provides-Extra: test
Requires-Dist: pytest~=9.0.0; extra == "test"
Requires-Dist: pydantic; extra == "test"
Requires-Dist: pytest-xdist; extra == "test"
Requires-Dist: transformers~=4.57.1; extra == "test"
Provides-Extra: lint
Requires-Dist: mypy~=1.13.0; extra == "lint"
Requires-Dist: pre-commit~=4.5.0; extra == "lint"
Requires-Dist: types-PyYAML~=6.0.12.20250915; extra == "lint"
Requires-Dist: uv~=0.10.0; extra == "lint"
Dynamic: license-file

# IBM Spyre Metrics API

## Overview

This repository provides API to access Spyre performance metric files.
It supports both the old and new format.

**`MetricFile`** class corresponds to a metric file. It provides a static method
`open_metrics()` to open metric files, with expanding `%BUSID` keyword.
An iterator `read_metrics()` reads performance metrics from the file and returns
a pair of **`MetricDataType`** object and the value read from the file.
Use `set_filters()` to limit the types of metrics read from a file if needed.
It receives a list of metric names.
The default behavior without using `set_filters()` is to return all metrics stored
in a metric file.

**`MetricDataType`** is a frozen data class object to show metadata of
each metric data, such as its name and ID, data type, unit, and scaling.
The ID is the value stored in metric files to specify the data type of
each metric data. `MetricDataType` objects are singleton in a process.

A typical usage looks like:

```python
from spyremetrics import MetricFile

# Expand "%BUSID" keyword and open metric files. Return a list of MetricFile objects.
metric_file_list = MetricFile.open_metrics(metric_file_path)
for mf in metric_file_list:
    mf.set_filters(['pwr', 'tempr', 'rdmem', 'wrmem', 'avgmem', 'peakmem'])
    for metric, val in mf.read_metrics():
        match metric.name:
            case 'pwr':     pwr     = val
            case 'tempr':   tempr   = val
            case 'rdmem':   rdmem   = val / 1024 / 1024  # convert to GiB
            case 'wrmem':   wrmem   = val / 1024 / 1024  # convert to GiB
            case 'avgmem':  avgmem  = val / 1024 / 1024  # convert to GiB
            case 'peakmem': peakmem = val / 1024 / 1024  # convert to GiB
```

## Installation

### Prerequisite

- Python >= 3.12
- numpy
- psutil >= 5.9.6

### Install from package

```bash
pip3 install spyremetrics
```

### Editable installation

```bash
# From the repository root
pip install -e .
```

### Build and install wheel package

```bash
# Build a wheel package
$ python3 -m build
$ ls -1 dist
spyremetrics-0.5.0-py3-none-any.whl
spyremetrics-0.5.0.tar.gz

# Install the wheel package
$ pip3 install dist/spyremetrics-0.5.0-py3-none-any.whl

...
Successfully installed spyremetrics-0.5.0
$ pip3 list | grep spyremetrics
spyremetrics            0.5.0
```

## Usage

### Example Script

Run the example script to see the package in action:

```bash
# Need a copy of metric file in new format
python3 example.py tests/test_metric.bin
```

The sample metric file was generated by
[gen_test_metrics.sh](scripts/gen_test_metrics.sh):

```bash
scripts/gen_test_metrics.sh > tests/test_metric.bin
```

### Iterating Over Metric Data Words

A typical usage to retrieve all available metrics:

```python
from spyremetrics import metric_file

# Expand "%BUSID" keyword and open metric files. Return a list of MetricFile objects.
metric_file_list = MetricFile.open_metrics(metric_file_path)
for mf in metric_file_list:
    for metric, val in mf.read_metrics():
        match metric.name:
            case 'pwr':     pwr     = val
            case 'tempr':   tempr   = val
            case 'rdmem':   rdmem   = val / 1024 / 1024  # convert to GiB
            case 'wrmem':   wrmem   = val / 1024 / 1024  # convert to GiB
            case 'avgmem':  avgmem  = val / 1024 / 1024  # convert to GiB
            case 'peakmem': peakmem = val / 1024 / 1024  # convert to GiB
```

`set_filters()` restricts metric types to be loaded:

```python
from spyremetrics import metric_file

metric_file_list = MetricFile.open_metrics(metric_file_path)
for mf in metric_file_list:
    mf.set_filters(['pwr', 'avgmem'])
    for metric, val in mf.read_metrics():
        match metric.name:
            case 'pwr':     pwr     = val
            case 'avgmem':  avgmem  = val / 1024 / 1024  # convert to GiB
```

## Development

### API classes

- Class: `MetricFile`
  - Represent a single metric file. The main class of this API.
    Use `open_metrics()` static method to open metric files,
    and `read_metrics()` to iterate through the data in the file.
    `set_filters()` selects metrics to be read from a metric file.
- Class: `MetricSection`
  - Represent a single section in a metric file. `MetricFile.sections()` iterates
    through the sections in a metric file and returns `MetricSection` object.
- Class: `MetricDataType`
  - Specify the data type and how to summarize the data.
    Its ID is the value stored in metric files. Each instance is singleton.
- Class: `SectionType`
  - Specify the section type. Its ID is the value stored in metric files.
    Each instance is singleton.
- Class: `ValueType`
  - Specify the data type and unit of a metric data.
    This ID is only referred from `MetricDataType` and never shown in metric files.
    Each instance is singleton.
- Class: `SummarizerType`
  - Specify how to summarize data if multiple words are stored in a metric file.
    This ID is only referred from MetricDataType and never shown in metric files.
    Each instance is singleton.

### Definition of Metric type, Section type, etc. in `section_types.json`

Supported metric types, section types, etc. are defined in a JSON file:
[`section_types.json`](spyremetrics/section_types.json).
This is the master definition of the data supported by Spyre metric files.

This JSON file has five types of definitions:

- Section type: `version`
  - The version of this definition. Note that adding/deleting metric and
    section types will affect the layout of a metric file.
    The version must be updated in such case.
- Section type: `section_type`
  - List of supported section types.
- Section type: `metric_type`
  - List of supported metric types.
- Section type: `value_type`
  - List of definitions on the data types and units of values.
- Section type: `summarizer_type`
  - List of definitions how to summarize data if multiple data words are stored
    for a single metric type.

### Converting `section_types.json` to `generated_section_types.py`

For reducing the time to start up, we provide generated source code in Python
that are converted form the configuration JSON file
[`spyremetrics/section_types.json`](spyremetrics/section_types.json)
as
[`spyremetrics/generated_section_types.py`](spyremetrics/generated_section_types.py)
in the GitHub repo.

When the JSON file is modified, the converted file needs to be re-generated by using
[`convert_section_types.py`](scripts/convert_section_types.py):

```bash
# From the repository root
python3 scripts/convert_section_types.py --output-dir spyremetrics spyremetrics/section_types.json
```

### Metric File Format

Brief description is in the beginning of
[`spyremetrics/metric_file.py`](spyremetrics/metric_file.py).

### Type Annotations

All functions include type annotations for better IDE support and type checking.

## License

Apache 2.0
