Metadata-Version: 2.4
Name: mprof
Version: 0.1.1
Summary: Within-kernel timeline profiler for CUDA: view, summary, diff over .mprof traces
License: MIT
Requires-Python: >=3.8
Description-Content-Type: text/markdown; charset=UTF-8; variant=GFM
Project-URL: Repository, https://github.com/Infatoshi/mprof

# mprof

A within-kernel timeline profiler for CUDA: it opens the opaque kernel box that
`nsys` leaves shut, on a time axis. One Rust binary, no Python runtime.

## Install

```bash
uv tool install git+https://github.com/Infatoshi/mprof     # from anywhere
uv tool install --force .                                  # from a local checkout
```

Installing builds the Rust binary, so a Rust toolchain is required on the
install machine. (`uv tool install mprof` by bare name is reserved for a future
PyPI release; don't rely on it -- it resolves whatever is on PyPI, not this
project. If you hand-delete `~/.local/bin/mprof`, reinstall with `--force`, not
a bare reinstall, or uv treats it as a no-op.)

That installs the `mprof` CLI (a single Rust binary). The analysis commands
(`view` / `summary` / `diff`) run anywhere on a captured trace -- this is the
local-view half of a remote-record / local-view workflow. Capture (`record` /
`setup`) needs a CUDA GPU box; see [Capture on a GPU box](#capture-on-a-gpu-box).

From a checkout you can instead `cargo build --release` and run
`./target/release/mprof`.

## Use

```bash
mprof record -- ./your_cuda_program     # capture: CUPTI PC sampling + auto SASS attach
mprof summary trace.mprof               # kernel triage: what hurts, what doesn't
mprof view trace.mprof                  # diagnosis-first GUI; pick kernel/view in-app
mprof diff base.mprof cand.mprof        # per-kernel timing delta (the optimize loop)
mprof doctor                            # check capture prerequisites on this machine
```

`record` defaults to CUPTI PC sampling and auto-attaches SASS to the sidecar.
`summary` prints the diagnosis as XML by default (for an LLM agent); add
`--human` for a colored panel or `--json` for tooling. `view` opens to the
Summary tab; the other views (Hotspots / Flame / Raster / Mega / Density) are
in-app tabs. `record nvbit|source ...` are advanced opt-ins.

## Rust Viewer

Open an included sample trace (more in `samples/`):

```bash
mprof samples/cutlass_sm120_pc_gemm.mprof   # PC-sampled GEMM
mprof samples/gemm.mprof                     # source-instrumented timeline
```

`samples/` holds a few small example traces: `gemm.mprof` (source-instrumented
block timeline), `gemm_cupti.mprof` (CUPTI Activity with v2 launch tags), and
`cutlass_sm120_pc_gemm.mprof` + its `.pc.json` sidecar (Blackwell GEMM with PC
sampling -- the one to try `Summary`/`Hotspots` on).

The viewer opens to the **Summary** diagnosis for the dominant kernel and lets
you switch kernel and view in-app -- there are no view-selection launch flags:

```bash
mprof trace.mprof                 # opens to the Summary tab
mprof trace.mprof --kernel gelu   # preselect a kernel by name
```

Top-bar tabs (when a kernel is selected): `Summary | Hotspots | Flame | Raster
| Mega | Density`, plus `Trace` to return to the outer per-stream/SM timeline.

- **Summary** -- the diagnosis: verdict + per-dimension state (ok/minor/issue)
  + hot instructions. Same content as `mprof summary`.
- **Hotspots** -- instructions ranked by PC-sample share, no fabricated time axis.
- **Flame / Raster** -- sampled SASS in instruction-category / stall-family lanes.
- **Mega / Density** -- projected SM-lane views (see the note below).

Interaction (mouse-first):

- Mouse wheel zooms time around the pointer.
- Click-drag the background pans; the horizontal/vertical scrollbars also pan
  (click a track to jump to that spot).
- `Shift` + wheel changes row height; `Ctrl`/`Command` + wheel does per-mode
  vertical (row/lane) zoom.
- Double-click a kernel in the `Trace` timeline to open it.

`Mega` uses the visual grammar from the MEGAPROFILER layout:
GPU-tinted SM bands, separator / loader / consumer / consumer / storer rows,
bottom time axis, hover metadata, and progressive detail as you zoom in. With
current CUPTI PC-sampling data the SM lanes are projected from sampled PCs,
not measured SM residency. Treat it as a readable PC/SASS flame view, not an
exact per-SM execution trace.

Use the `Density` tab in a selected-kernel view to see activity intensity as a
per-SM heatmap. MKPROF files use true time-local activity ranges; CUPTI PC
captures use projected PC-sample intensity because current PC sidecars do not
carry per-sample timestamps.

The Rust viewer also opens `MKPROF1.2` files directly:

```bash
mprof example_gpu0.mkprof
```

For `.mkprof` inputs, `Mega` renders the real dense activity matrix from the
file: GPU/SM lanes, logical instruction types, loader/consumer/storer timing,
controller/event markers, and the parameter names carried in the JSON schema.

Open the synthetic ActivityRecord hierarchy demo:

```bash
mprof --demo-activity
```

This demo shows the intended SM-lane activity raster, hover schema, and
zoom-revealed instruction slices. In the demo top bar, `Rows` switches between
SM, block, and warp lanes; `Depth` switches between automatic zoom-based
detail, phase-only bars, and instruction bars. It is not captured kernel data.

`record` defaults to CUPTI PC sampling and **auto-attaches SASS** to the
sidecar (it disassembles the program binary), so a single command gives you
the instruction-level data:

```bash
mprof record -- ./your_cuda_program     # writes trace.mprof + trace.mprof.pc.json
mprof view trace.mprof
```

Continuous PC sampling is the default lower-perturbation mode and maps samples
back to events by kernel function. SASS attach is best-effort: it is silently
skipped when the kernel lives in a shared library (e.g. cuBLAS/cuDNN) rather
than the program binary -- for that case attach manually with the advanced
`mprof sass <trace> <library.so>`.

PC sampling gives sampled PC offsets and stall reasons -- the basis for the
`Summary` and `Hotspots` views. NVBit remains the heavier microscope for
dynamic warp/lane traces, active masks, branch paths, and memory addresses.

Double-click a kernel in the `Trace` timeline to open it; the top-bar tabs
(`Summary | Hotspots | Flame | Raster | Mega | Density`) switch the view of the
selected kernel.

In the normal stream timeline, selecting a PC-sampled kernel and zooming until
the bar is wide enough draws an inline PC/SASS instruction inset inside that
kernel bar. This keeps the outer CUDA stream timeline visible while exposing
the inner sampled instruction structure for the selected launch.

Build and capture the reproducible transformer-forward example:

```bash
make -C producers/cuda/examples transformer_forward
mprof record -o transformer_forward.mprof -- producers/cuda/examples/transformer_forward
mprof summary transformer_forward.mprof
mprof view transformer_forward.mprof
```

`record` auto-attaches SASS (writing `transformer_forward.mprof.pc.json`), so
no separate step is needed.

## Diagnose and compare

`summary` and `diff` are built into the viewer binary -- no Python runtime:

```bash
mprof summary samples/cutlass_sm120_pc_gemm.mprof --human   # triage one kernel
mprof diff baseline.mprof candidate.mprof                   # per-kernel timing delta
cargo nextest run                                                 # tests
```

`diff` joins two traces on kernel name and reports the change in average
per-launch duration and total GPU time -- the optimize loop is `record` ->
edit kernel -> `record` -> `diff`.

## Capture on a GPU box

Capture needs a CUDA GPU box: the CUPTI producer is native code built against
the local toolkit, and PC sampling needs the perf-counter permission. From a
checkout:

```bash
mprof setup                              # builds the CUPTI producer against local CUDA
mprof doctor                             # checks toolkit / producer / perf-counter permission
mprof record -- ./your_cuda_program      # capture once the above are green
```

`mprof doctor` prints the exact `sudo` + reboot step if PC sampling is
restricted to admin users (`NVreg_RestrictProfilingToAdminUsers`). For a manual
build instead of `setup`:

```bash
make -C producers/cupti
make -C producers/cuda/examples
mprof record -o trace.mprof -- producers/cuda/examples/demo
```

NVBit support is source-scaffolded, but the NVBit release bundle is not
vendored in this repo. Install NVBit separately before building that producer.

## Remote / headless GPU boxes

`mprof summary` is headless, so on a remote GPU server you usually don't move
anything -- SSH in and diagnose in place (this is also the agent path):

```bash
ssh gpu-box 'mprof summary trace.mprof'          # XML diagnosis, no GUI needed
ssh gpu-box 'mprof summary trace.mprof --human'  # human panel
```

To open the GUI on another machine, copy the trace with your normal tooling
(the `.mprof` plus its `.pc.json` sidecar) and view it locally:

```bash
scp gpu-box:trace.mprof gpu-box:trace.mprof.pc.json .
mprof view trace.mprof
```

