Metadata-Version: 2.4
Name: nsys-ai
Version: 0.3.0
Summary: AI-powered analysis for NVIDIA Nsight Systems profiles — web viewer, timeline, kernel navigator, NVTX hierarchy
Author: GindaChen
License: MIT
Project-URL: Homepage, https://github.com/GindaChen/nsys-ai
Project-URL: Repository, https://github.com/GindaChen/nsys-ai
Project-URL: Issues, https://github.com/GindaChen/nsys-ai/issues
Project-URL: Changelog, https://github.com/GindaChen/nsys-ai/blob/main/CHANGELOG.md
Project-URL: Documentation, https://github.com/GindaChen/nsys-ai/blob/main/docs/user-guide.md
Classifier: Development Status :: 3 - Alpha
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: Software Development :: Debuggers
Classifier: Topic :: System :: Monitoring
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: duckdb>=1.0.0
Requires-Dist: pyarrow>=14.0.0
Requires-Dist: rich>=13.0.0
Requires-Dist: textual>=8.0.0
Provides-Extra: agent
Requires-Dist: anthropic>=0.20.0; extra == "agent"
Requires-Dist: litellm>=1.0.0; extra == "agent"
Provides-Extra: mcp
Requires-Dist: mcp<2,>=1.0.0; extra == "mcp"
Provides-Extra: ai
Requires-Dist: anthropic>=0.20.0; extra == "ai"
Requires-Dist: litellm>=1.0.0; extra == "ai"
Provides-Extra: tui
Requires-Dist: textual>=8.0.0; extra == "tui"
Provides-Extra: chat
Requires-Dist: litellm>=1.0.0; extra == "chat"
Requires-Dist: nsys-ai[tui]; extra == "chat"
Provides-Extra: cutracer
Requires-Dist: cutracer>=0.2.0; extra == "cutracer"
Provides-Extra: dev
Requires-Dist: pytest<9,>=8.2; extra == "dev"
Requires-Dist: pytest-asyncio<1,>=0.25; extra == "dev"
Requires-Dist: pytest-cov>=5.0; extra == "dev"
Requires-Dist: pytest-textual-snapshot>=1.1.0; extra == "dev"
Requires-Dist: coverage>=7.0; extra == "dev"
Requires-Dist: ruff>=0.4.0; extra == "dev"
Provides-Extra: all
Requires-Dist: nsys-ai[agent,chat,cutracer,mcp,tui]; extra == "all"
Dynamic: license-file

<div align="center">

# nsys-ai

**AI-powered analysis for NVIDIA Nsight Systems profiles**

Navigate GPU kernel timelines, diff two runs, and diagnose performance
bottlenecks with an evidence-first agent — from your browser or terminal.

> **Mission:** Build an agent that understands GPU performance from first
> principles — one that can identify pipeline bubbles, calculate MFU, assess
> arithmetic intensity, and diagnose the root causes that cost millions of GPU
> hours, turning months of expert debugging into minutes.

[![CI](https://github.com/GindaChen/nsys-ai/actions/workflows/ci.yml/badge.svg)](https://github.com/GindaChen/nsys-ai/actions)
[![PyPI](https://img.shields.io/pypi/v/nsys-ai)](https://pypi.org/project/nsys-ai/)
[![Python 3.10+](https://img.shields.io/badge/python-3.10+-3776AB?logo=python&logoColor=white)](https://www.python.org/downloads/)
[![License: MIT](https://img.shields.io/badge/license-MIT-green.svg)](https://github.com/GindaChen/nsys-ai/blob/main/LICENSE)

</div>

---

nsys-ai reads `.nsys-rep`, `.parquetdir`, or `.sqlite` exports from
[NVIDIA Nsight Systems](https://developer.nvidia.com/nsight-systems) and turns
them into something you can navigate and reason about: a web timeline, terminal
viewers, a before/after diff that reports whether a change actually helped, and
a set of deterministic analysis skills an LLM agent can drive. `.nsys-rep` files
use a Parquet cache by default; `.sqlite` remains the compatibility path.

## Installation

```bash
pip install nsys-ai
```

No CUDA and no Nsight install are required to analyze a profile. Python 3.10+
only. (Capturing a new `.nsys-rep`, or converting one, needs the `nsys` CLI on
your machine; analyzing an existing `.sqlite` does not.)

## Quick start

### 1. Capture a profile

For ML training, capture a few representative iterations rather than the whole
run — it keeps the profile small and the profiler overhead low. Mark the region
with the CUDA profiler API and trace CUDA plus NVTX:

```python
import torch

for step in range(warmup):
    train_step()
torch.cuda.synchronize()
torch.cuda.cudart().cudaProfilerStart()
for step in range(3):            # profile these iterations
    train_step()
torch.cuda.synchronize()
torch.cuda.cudart().cudaProfilerStop()
```

```bash
nsys profile --capture-range=cudaProfilerApi --trace=cuda,nvtx \
  -o my_training python train.py
# -> my_training.nsys-rep
```

`--trace=cuda` is what every skill relies on (GPU kernels, memory copies, CUDA
API). `nvtx` adds the annotation hierarchy that drives the iteration, region,
and layer views. To use the iteration tools (`iters`, `diff --iteration`),
annotate each step with a consistent NVTX marker — see
[Focused Profiling](https://github.com/GindaChen/nsys-ai/blob/main/docs/08-focused-profiling.md) and
[NVTX Annotations](https://github.com/GindaChen/nsys-ai/blob/main/docs/03-nvtx-annotations.md).

No workload handy? Download an example profile:

```bash
cd examples/example-20-megatron-distca && python download_data.py
# -> output/megatron_distca.nsys-rep
```

### 2. Open it

```bash
# Default: open the web timeline in your browser
nsys-ai my_training.nsys-rep

# Metadata and GPU info
nsys-ai info my_training.nsys-rep

# GPU kernel summary
nsys-ai summary my_training.nsys-rep --gpu 0
```

Prefer the terminal? The TUIs work the same way:

```bash
nsys-ai timeline my_training.nsys-rep --gpu 0   # Perfetto-style horizontal timeline
nsys-ai tui my_training.nsys-rep --gpu 0        # NVTX tree browser
```

### 3. Compare two runs

```bash
nsys-ai diff before.sqlite after.sqlite
```

### 4. Keep the investigation in one session

The 0.3.0 command surface is built around a session directory: findings,
proposals, run specifications, diffs, and decisions can move between CLI, Web,
TUI, and MCP without reconstructing state from terminal output.

```bash
SESSION=/tmp/nsys-ai/run-001

nsys-ai doctor run-before/profile.sqlite --format json
nsys-ai diagnose run-before/profile.sqlite --session "$SESSION"
nsys-ai ask --session "$SESSION" "what is the main bottleneck?"
nsys-ai diff run-before/profile.sqlite run-after/profile.sqlite \
  --no-ai --session "$SESSION"
nsys-ai review --session "$SESSION"
```

For a complete diagnose → propose → re-profile → diff → decision walkthrough,
including the `RunSpec` required by `propose`, see the
[user guide](https://github.com/GindaChen/nsys-ai/blob/main/docs/user-guide.md).

## Web timeline

A browser-based multi-GPU viewer with progressive rendering — no `--trim`
required. This is the default view when you run `nsys-ai <profile>`.

```bash
nsys-ai my_training.nsys-rep                       # opens in your browser
nsys-ai timeline-web my_training.nsys-rep --gpu 0 1 2 3
```

- Multi-GPU stacked view with color-coded separators
- Progressive rendering — pre-builds the NVTX tree at startup, then serves tiles
  in about a millisecond each
- NVTX hierarchy bars (L0-L5) per GPU
- AI chat sidebar (press `a`) and kernel search (press `/`)

| Input | Action |
|:-----:|--------|
| Swipe / `h` `l` / arrows | Pan through time |
| Swipe up-down / `j` `k` | Select stream |
| Pinch / `Shift+scroll` / `+` `-` | Zoom |
| `f` or `0` | Fit full time range |
| `Tab` | Next kernel |
| `/` | Search kernels |
| `n` | Toggle NVTX |
| `a` | AI chat |
| `?` | Help overlay |

## Timeline TUI

A Perfetto-style horizontal viewer with per-stream kernels, NVTX hierarchy
bars, and a time-cursor navigation model.

| Key | Action |
|:---:|--------|
| arrows | Pan time / select stream |
| `Shift+arrows` | Page pan (quarter viewport) |
| `Tab` | Snap to next kernel |
| `+` `-` | Zoom |
| `/` | Filter kernels by name |
| `m` | Minimum-duration threshold |
| `d` | Toggle demangled names |
| `B` | Save bookmark (with kernel + NVTX context) |
| `C` | Config panel (stream rows, tick density, NVTX depth) |
| `h` | Full help overlay |

## Profile diff

Comparing two profiles is the point of nsys-ai: it reports not just what changed
but whether the change is a likely regression or improvement.

```bash
# Terminal report
nsys-ai diff before.sqlite after.sqlite

# Interactive side-by-side web comparison
nsys-ai diff-web before.sqlite after.sqlite

# A specific device or time window
nsys-ai diff before.sqlite after.sqlite --gpu 0 --trim 39 42

# Compare one aligned iteration
nsys-ai diff before.sqlite after.sqlite --iteration 0

# Markdown (for a PR or issue) or JSON (for scripting)
nsys-ai diff before.sqlite after.sqlite --format markdown -o diff.md
nsys-ai diff before.sqlite after.sqlite --format json

# Gate CI: exit non-zero when the verdict is a likely regression, or when the
# two profiles could not be compared at all (for example one side recorded no
# GPU kernel activity because the profiling step failed)
nsys-ai diff before.sqlite after.sqlite --exit-on-regression

# Same gate with a custom regression threshold (default 5%)
nsys-ai diff before.sqlite after.sqlite --gate 3.0
```

The report covers top regressions and improvements, new and removed kernels,
NVTX region deltas, compute/NCCL overlap and idle changes, and a step-time
category rollup (compute / communication / idle). With `--format json` it adds a
top-level `verdict`, a `comparability_confidence` score, and a stable
content-derived `profile_id` per side. With no `--gpu`, the diff aggregates
across every device.

| Flag | Default | Description |
|------|---------|-------------|
| `--gpu N` | all GPUs | Restrict to one device |
| `--trim START END` | full span | Compare only this window (seconds) |
| `--iteration N` | — | Compare one aligned iteration (needs an NVTX marker) |
| `--format` | `terminal` | `terminal` \| `markdown` \| `json` |
| `--limit N` | 15 | Top regressions/improvements to show |
| `--sort` | `delta` | `delta` \| `percent` \| `total` |
| `--exit-on-regression` | — | Exit 1 when the verdict is `regression_likely`, or `inconclusive` because the profiles could not be compared |
| `--gate PCT` | 5.0 | Regression threshold (%) for the verdict; implies `--exit-on-regression` |

## Baselines

A diff needs something to compare against. Passing a raw file path is fragile in
CI: the path drifts between jobs and the file may not survive to the next run.
The `baseline` command keeps a local store of named snapshots so a stable name,
not a path, resolves the comparison.

```bash
# Tag a known-good run under a name (copies the resolved .sqlite into the store)
nsys-ai baseline tag main run.sqlite --reason "green main @ abc123"

# List and inspect what has been tagged
nsys-ai baseline list
nsys-ai baseline show main

# Compare a candidate against a tagged baseline by name
nsys-ai diff --against baseline:main candidate.sqlite
```

`tag` resolves the profile (including a `.nsys-rep` sidecar), copies the
self-contained `.sqlite` into the store, and records a deterministic `meta.json`
(content-derived `profile_id`, source path, reason, tagger, timestamp). The
snapshot stays valid even if the original file moves.

The store lives in `.nsys-ai-baselines/` under the current directory. Set
`NSYS_AI_BASELINE_ROOT` to point tag and resolve at a shared location so a job
that tags and a later job that diffs find the same store regardless of CWD:

```bash
export NSYS_AI_BASELINE_ROOT="$CI_CACHE/nsys-baselines"
nsys-ai baseline tag main run.sqlite --reason "green main" \
  && nsys-ai diff --against baseline:main candidate.sqlite
```

The `baseline:<name>` reference is accepted anywhere a baseline profile path is,
via `--against` or as the `before` positional.

## Commands

| Command | Description |
|---------|-------------|
| `open` | Quick-open a profile in the web UI or TUI |
| `timeline-web` | Web multi-GPU timeline (progressive rendering) |
| `timeline` | Timeline TUI |
| `tui` | NVTX tree TUI |
| `web` | Web viewer server |
| `info` | Profile metadata and GPU hardware |
| `doctor` | Check environment, ingest, cache, and profile health |
| `profile` | Capture a workload and write a reproducible RunSpec |
| `warm` | Build the Parquet cache and NVTX kernel map up front |
| `summary` | Top kernels and stream breakdown |
| `analyze` | Full auto-report (`--format json` emits evidence findings) |
| `overlap` | Compute / NCCL overlap analysis |
| `nccl` | NCCL collective breakdown |
| `iters` | Auto-detect training iterations |
| `tree` / `markdown` | NVTX hierarchy as text / markdown |
| `search` | Search kernels and NVTX by name |
| `report` | Generate a performance report |
| `diff` | Before/after profile comparison |
| `diff-web` | Side-by-side comparison web viewer |
| `baseline` | Manage named baseline snapshots (`tag`, `list`, `show`) |
| `diagnose` | Run the default evidence pack and publish findings |
| `propose` | Turn one finding into a verifiable proposal |
| `review` | Compare a pair or resume a session decision path |
| `optimize` | Run diagnose → propose → re-profile → diff as one session |
| `chat` | AI chat TUI for a profile |
| `ask` | One-shot AI question about a profile |
| `agent` | Agent auto-analysis (`analyze`, `ask`) |
| `skill` | List and run analysis skills |
| `evidence` | Build evidence findings for the timeline overlay |
| `root-cause` | Browse and submit root-cause patterns |
| `cutracer` | Instruction-level drill-down (`check`, `install`, `plan`, `run`, `analyze`) |
| `export` / `export-csv` / `export-json` | Perfetto JSON, flat CSV, flat JSON |
| `viewer` / `timeline-html` | Interactive HTML report / timeline |

Run `nsys-ai <command> --help` for flags.

## Analysis cache

The first command run against a profile builds a `<profile>.nsys-cache` directory
next to it: the tables analysis needs, exported to Parquet and queried through
DuckDB. Later commands reuse it and open in well under a second. The cache is
rebuilt automatically when the profile changes; deleting the directory is safe.

Measured on the reference captures (12 cores, 15 GB RAM), running eight skills:
`top_kernels`, `gpu_idle_gaps`, `overlap_breakdown`, `memory_transfers`,
`kernel_launch_overhead`, `stream_concurrency`, `tensor_core_usage`,
`nvtx_layer_breakdown`. Reproduce with
`python scripts/bench_cache.py <profile> --basket auto-policy`.

| Profile | Build | First NVTX query | Cache size | Eight skills: direct query → cached query |
|---------|-------|------------------|------------|-------------------------------------------|
| 93 MB | 1.9 s | +3.7 s | 17 MB | 5.8 s → 0.6 s |
| 235 MB | 2.6 s | +3.4 s | 22 MB | 11.2 s → 3.7 s |
| 924 MB | 8.4 s | +11.2 s | 84 MB | 33.2 s → 8.5 s |
| 3.7 GB | 27.4 s | +49.7 s | 277 MB | 83.4 s → 16.3 s |

The build exports Parquet; the kernel-to-NVTX map is built separately by the
first query that needs it, which is the "first NVTX query" column. Both are paid
once per profile.

On this workload the first run is never slower end to end than querying the
export directly, and lighter on memory at all four sizes, so the build is the
default — "this workload" being the eight skills above; a one-shot is a
different trade, see below. The
middle two sizes are a clear win on time (23% and 26%); at 93 MB and 3.7 GB the
time difference is 2-5%, inside run-to-run variance, and what those sizes gain
is memory — 7.1 GB against 10.2 GB on the largest. Every command after the
first is the warm row, which is where the cache pays for itself outright.

Two cases where it is not what you want:

- **One command against a very large profile, and no follow-up.** Set
  `NSYS_AI_CACHE_MODE=direct` to query the SQLite export in place — instant
  start, slower queries, and no cache written. `nsys-ai skill run` also takes
  `--no-cache` for the same effect. A light one-shot workload is also the case
  where the build costs more memory than it saves, so prefer direct if you are
  tight on RAM.
- **A read-only or full disk.** No setting needed: nsys-ai checks before it
  builds, says why it declined, and queries the export directly.

`NSYS_AI_CACHE_MODE=parquet` forces the build in the other direction. Both
values only decide whether a cache gets *built*: an existing valid cache is
still used, so delete the directory if you want the export read in place.

## Skills

Skills are self-contained analysis units that run without an LLM. The packaged
registry covers kernels, memory, NCCL/communicators, NVTX, MFU, idle,
root-cause, profile health, and more. Run `skill list` for the live catalog;
the count is intentionally not a compatibility contract.

```bash
nsys-ai skill list                                 # full catalog
nsys-ai skill run top_kernels profile.sqlite
nsys-ai skill run nccl_breakdown profile.sqlite
nsys-ai skill run profile_health_manifest profile.sqlite --format json
```

A few common ones:

| Skill | What it does |
|-------|-------------|
| `top_kernels` | Heaviest GPU kernels by total time |
| `gpu_idle_gaps` | Pipeline bubbles between kernels |
| `memory_transfers` | H2D / D2H / D2D transfer breakdown |
| `nccl_breakdown` | NCCL collective summary by type |
| `nccl_communicator_analysis` | Per-communicator NCCL topology and efficiency |
| `overlap_breakdown` | Compute / communication overlap |
| `kernel_launch_overhead` | CPU-to-GPU dispatch latency |
| `region_mfu` | Model FLOPs utilization for an NVTX region |
| `profile_health_manifest` | One-shot health summary (run this first) |

Skills are extensible — add one by dropping a Python file that exports a `SKILL`
constant. See [`skill list`](https://github.com/GindaChen/nsys-ai/blob/main/docs/agent_skills/commands/skill.md) for the full
catalog.

## AI analysis (optional)

The agent is a CUDA performance expert that runs the skills and cites the
evidence — kernel names, durations, timestamps — behind each diagnosis rather
than guessing. Targeted `ask` answers use a fixed evidence-first shape:
summary, primary diagnosis, cited evidence, confidence, recommended action,
and a final runnable verification command.

```bash
nsys-ai agent analyze profile.sqlite
nsys-ai agent ask profile.sqlite "why are there bubbles in the pipeline?"
nsys-ai ask profile.sqlite "is NCCL overlapping with compute?"
nsys-ai chat profile.sqlite                        # interactive chat TUI
```

The AI features need a provider API key. Set one of:

```bash
export ANTHROPIC_API_KEY=...      # or
export OPENAI_API_KEY=...         # or
export GEMINI_API_KEY=...
export NSYS_AI_MODEL=...          # optional: pick a specific model
```

Install the dependencies with the `agent` extra:

```bash
pip install 'nsys-ai[agent]'
```

With a key, targeted `ask` uses the model to select deep-dive skills and synthesize
the Summary; the remaining evidence-first sections and verification command are
built deterministically from skill output. If no key is set, the agent returns the
same answer shape with a deterministic Summary.

## Claude Code plugin

nsys-ai ships as a [Claude Code](https://claude.com/claude-code) plugin: the
`/nsys-ai` slash command turns a profile into a root cause, a proposed fix, and
an annotated timeline. See
[docs/claude-plugin-quickstart.md](https://github.com/GindaChen/nsys-ai/blob/main/docs/claude-plugin-quickstart.md) to install
and [docs/claude-plugin.md](https://github.com/GindaChen/nsys-ai/blob/main/docs/claude-plugin.md) for the full reference.

## Documentation

Start with the **[User guide](https://github.com/GindaChen/nsys-ai/blob/main/docs/user-guide.md)** — one workload from capture to a recorded
decision, on the command line. To drive the same workflow in a browser instead, see
[Guided loop setup](https://github.com/GindaChen/nsys-ai/blob/main/docs/guided-loop-setup.md).

Useful entry points for the next question:

- upgrading from 0.2.3 → [Migrating to 0.3.0](https://github.com/GindaChen/nsys-ai/blob/main/docs/user/migrating-to-0.3.0.md)
- something is not working → [Troubleshooting](https://github.com/GindaChen/nsys-ai/blob/main/docs/user/troubleshooting.md)
- checking an input before analysis → [Profile inputs](https://github.com/GindaChen/nsys-ai/blob/main/docs/user/profile-inputs.md)
- choosing a browser surface → [Choosing a Web viewer](https://github.com/GindaChen/nsys-ai/blob/main/docs/user/viewers.md)
- running a focused, no-LLM analysis → [Analysis skills](https://github.com/GindaChen/nsys-ai/blob/main/docs/user/skills.md)

The complete, maintained documentation index is
[docs/README.md](https://github.com/GindaChen/nsys-ai/blob/main/docs/README.md).

The rest of the `docs/` directory mirrors the relevant NVIDIA Nsight Systems reference
(capture, schema, NVTX, CUDA/NCCL trace) plus nsys-ai project guides:

| Guide | Topic |
|-------|-------|
| [User guide](https://github.com/GindaChen/nsys-ai/blob/main/docs/user-guide.md) | Capture, diagnose, propose, diff, decide — end to end |
| [doctor](https://github.com/GindaChen/nsys-ai/blob/main/docs/doctor.md) | Environment and profile health checks |
| [NVIDIA nsys CLI](https://github.com/GindaChen/nsys-ai/blob/main/docs/01-cli-reference.md) | The upstream `nsys` profiler CLI (capture-time) |
| [SQLite schema](https://github.com/GindaChen/nsys-ai/blob/main/docs/02-sqlite-schema.md) | Nsight export tables and queries |
| [NVTX annotations](https://github.com/GindaChen/nsys-ai/blob/main/docs/03-nvtx-annotations.md) | Annotating your code (and iteration markers) |
| [CUDA trace](https://github.com/GindaChen/nsys-ai/blob/main/docs/04-cuda-trace.md) | GPU kernel and memory tracing |
| [NCCL tracing](https://github.com/GindaChen/nsys-ai/blob/main/docs/05-nccl-tracing.md) | Multi-GPU collective analysis |
| [Python / PyTorch](https://github.com/GindaChen/nsys-ai/blob/main/docs/06-python-pytorch.md) | Profiling PyTorch workloads |
| [Containers](https://github.com/GindaChen/nsys-ai/blob/main/docs/07-container-profiling.md) | Profiling inside Docker / Slurm |
| [Focused profiling](https://github.com/GindaChen/nsys-ai/blob/main/docs/08-focused-profiling.md) | Capturing representative iterations |
| [CUTracer](https://github.com/GindaChen/nsys-ai/blob/main/docs/cutracer-instruction-analysis.md) | Instruction-level drill-down for top kernels |

The [`docs/sqlite-explorer/`](https://github.com/GindaChen/nsys-ai/tree/main/docs/sqlite-explorer/) directory holds an
interactive HTML explorer for the Nsight SQLite schema — open
`docs/sqlite-explorer/index.html` in a browser.

## Install tiers

```bash
pip install nsys-ai              # core: CLI, TUIs, skills, web/diff viewers
pip install 'nsys-ai[agent]'     # + LLM-backed agent (anthropic + litellm)
pip install 'nsys-ai[chat]'      # + chat TUI
pip install 'nsys-ai[mcp]'       # + stdio MCP transport (`nsys-ai-mcp`)
pip install 'nsys-ai[cutracer]'  # + CUTracer instruction-level workflow
pip install 'nsys-ai[all]'       # everything
```

The `ai` extra is kept as an alias of `agent` for backward compatibility.

## Development

```bash
git clone https://github.com/GindaChen/nsys-ai.git
cd nsys-ai
pip install -e '.[dev]'
pytest tests/ -v
```

See [CONTRIBUTING.md](https://github.com/GindaChen/nsys-ai/blob/main/CONTRIBUTING.md)
for the full contributor workflow, test layers, skill walkthrough, fixture
policy, and pull-request checklist.

**Guided optimization loop** (diagnose → propose → re-profile → diff → accept): see the
[User guide](https://github.com/GindaChen/nsys-ai/blob/main/docs/user-guide.md) for the CLI path and
[docs/guided-loop-setup.md](https://github.com/GindaChen/nsys-ai/blob/main/docs/guided-loop-setup.md) for the browser path.

---

## License

MIT — see [LICENSE](https://github.com/GindaChen/nsys-ai/blob/main/LICENSE).

<div align="center">
<sub>Built for GPU performance engineers.</sub>
</div>
