Metadata-Version: 2.5
Name: heapspy-mcp
Version: 0.1.1
Summary: MCP server that exposes heapspy heap snapshots and analysis to coding agents
License-Expression: MIT
Requires-Python: >=3.10
Requires-Dist: mcp<3,>=2.3
Requires-Dist: psutil>=5.9
Provides-Extra: pyspy
Requires-Dist: py-spy>=0.4; extra == 'pyspy'
Description-Content-Type: text/markdown

# heapspy-mcp

An MCP (Model Context Protocol) server that gives coding agents access to
[heapspy](../../README.md). It runs the `heapspy` binary, parses its `--json`
output (schema version 1, see [`docs/json-schema.md`](../../docs/json-schema.md))
and returns short, ranked summaries. The raw object graph is never sent to the
model.

The intended loop for an agent:

1. `list_python_processes` to find the PID.
2. `capture_snapshot` (after the user agrees) to get a snapshot handle.
3. `heap_summary` to see the top retainers.
4. Pick a suspect, then `retaining_path` to see the chain of references keeping it alive.
5. `map_to_source` to turn a name from that path into `file:line`.
6. Read the code and find where the holder grows without being cleared.

heapspy shows what is retained now. Where it was allocated is known only when
an allocation profile (`.hpya`) was recorded from the process before the
snapshot; see [Where was this memory allocated?](#where-was-this-memory-allocated).
Memory held by C extensions, and leaks made only of untracked objects (bare
ints, short strings), are not visible in the object graph. `heap_summary`
reports how much of RSS falls outside the graph.

The server also has process diagnostics for any process, not only Python: find
or scan processes by measured metrics, then measure one process's CPU,
scheduler behaviour and environment (cgroup, host) over a window, sample its
threads for memory stalls (page-fault I/O, reclaim, `memory.high` throttling),
and weigh competing explanations for its CPU use, slowness or stalls. See
[Process diagnostics](#process-diagnostics).

The same server also has CPU tools backed by [py-spy](https://github.com/benfred/py-spy):
where a process spends its time, and whether its threads are running, blocked
in a syscall, waiting for the GIL, or idle. See [CPU tools](#cpu-tools-py-spy).

## Install

Requires Python 3.10 or later and the `heapspy` binary.

From this repository:

```sh
cd python/heapspy-mcp
uv sync                # creates .venv with heapspy-mcp and its dependencies
uv run heapspy-mcp --help
```

Once published, `pip install heapspy-mcp` (and `pip install heapspy` for the
binary).

The CPU tools need py-spy (0.4 or later). It is an optional extra:

```sh
uv sync --extra pyspy                 # from this repository
pip install 'heapspy-mcp[pyspy]'      # once published
```

Without py-spy the server starts as usual and the CPU tools return
`py_spy_not_found`.

The server is built on the official `mcp` SDK, version 2 (`MCPServer`, which
replaced `FastMCP` in that release).

## Running

```sh
heapspy-mcp                                   # stdio (default)
heapspy-mcp --transport streamable-http --host 127.0.0.1 --port 8000
```

The HTTP endpoint is served at `/mcp`. It has no authentication; keep it on
localhost.

### Configuration

Flags take precedence over environment variables.

| Flag | Environment variable | Default | Meaning |
|---|---|---|---|
| `--heapspy-bin PATH` | `HEAPSPY_BIN` | `heapspy` on `PATH` | heapspy executable. A configured path that is not executable stops the server at startup. If nothing is configured and `heapspy` is not on `PATH`, the server starts and the analysis tools return `binary_not_found`. |
| `--allow PATTERN` | `HEAPSPY_MCP_ALLOW` | empty | Processes that `capture_snapshot` may dump. Repeatable; comma-separated lists are accepted; flag and variable entries are merged. Empty means live capture is disabled. |
| `--max-str N` | `HEAPSPY_MCP_MAX_STR` | 1024 | Per-string cap, in characters, used when a `capture_contents=true` call gives no `max_str` (1 to 1048576). |
| `--max-rss SIZE` | `HEAPSPY_MCP_MAX_RSS` | `8G` | Refuse live capture of processes with a larger RSS. Binary suffixes: `512M`, `8G`, `8GiB`, or plain bytes. |
| `--snapshot-dir DIR` | `HEAPSPY_MCP_SNAPSHOT_DIR` | `~/.cache/heapspy-mcp/snapshots` on Linux, `~/Library/Caches/heapspy-mcp/snapshots` on macOS, `%LOCALAPPDATA%\heapspy-mcp\Cache\snapshots` on Windows | Where captured snapshots are written. Created with mode 0700. |
| `--timeout SECONDS` | `HEAPSPY_MCP_TIMEOUT` | 120 | Timeout for `analyze`, `query` and `diff`. |
| `--capture-timeout SECONDS` | `HEAPSPY_MCP_CAPTURE_TIMEOUT` | 900 | Timeout for `dump`. |
| `--max-window SECONDS` | `HEAPSPY_MCP_MAX_WINDOW` | 10 | Cap on every measurement window of the process diagnostics. |
| `--no-host-context` | `HEAPSPY_MCP_NO_HOST_CONTEXT=1` | off | Leave host-wide load, steal and PSI out of the process tools' `environment` block. |
| `--proc-root DIR` | `HEAPSPY_MCP_PROC_ROOT` | `/proc` | procfs to read. |
| `--cgroup-root DIR` | `HEAPSPY_MCP_CGROUP_ROOT` | from `/proc/self/mountinfo` | A cgroup v2 mount to read instead, for example the host's cgroupfs mounted into a container. |
| `--max-stall-rate HZ` | `HEAPSPY_MCP_MAX_STALL_RATE` | 100 | Cap on the per-thread sampling rate of `watch_process_stalls`, and on the in-window sampling rates of `process_cpu`, `cpu_colocation` and `group_cpu` (1 to 1000). |
| `--max-cmdline-chars N` | `HEAPSPY_MCP_MAX_CMDLINE_CHARS` | 4096 | Cap on the `cmdline_max_chars` a call may ask for (16 to 131072). |
| `--allow-load-probes` | `HEAPSPY_MCP_ALLOW_LOAD_PROBES=1` | off | Enable `cache_sensitivity_probe`, which runs a benchmark that adds CPU load. |

| `--py-spy-bin PATH` | `PY_SPY_BIN` | `py-spy` on `PATH`, else next to the server's Python | py-spy executable for the CPU tools. A configured path that is not executable stops the server at startup. If none is found the CPU tools return `py_spy_not_found`. |
| `--cpu-max-duration SECONDS` | `HEAPSPY_MCP_CPU_MAX_DURATION` | 30 | Longest profile a CPU tool records (1 to 3600). Longer `duration_s` values are clamped, with a note in the result. |
| `--transport` | | `stdio` | `stdio` or `streamable-http`. |
| `--host`, `--port` | | `127.0.0.1`, `8000` | Bind address for `streamable-http`. |

Allowlist entries:

- `pid:1234` or `1234` matches that PID exactly.
- Anything else is a case-sensitive glob (`fnmatch`) tried against the process
  name, the executable's file name, and the whole command line joined by
  spaces. `gunicorn*`, `python3.12`, and `*manage.py runserver*` are all valid.

## Safety model

`capture_snapshot` is the only tool that touches a process. Taking a live dump
pauses the target and writes a memory image to disk, and it needs ptrace or
debug privileges. It is gated as follows, in this order:

1. `confirm` must be `true`. The tool description tells the agent to ask the
   user before setting it.
2. Exactly one of `pid`, `core_path` or `minidump_path`.
3. Contents are the agent's call. Without `capture_contents`, snapshots hold
   types, sizes and references only. With it, they also hold the text of
   `str` and `bytes` objects (cut at `max_str`) and real dict keys, which can
   include credentials and user data. Every such capture is logged.
4. For a `pid`: the allowlist must be non-empty and the process must match an
   entry, and its RSS must be at or below `--max-rss`. The server's own PID is
   refused.

Ingesting an existing core file or minidump needs only `confirm`, since no
process is touched.

Every capture is logged to stderr with the handle, target, contents setting and
output path, before and after the dump. A capture that records contents logs an
extra line saying so.

The CPU tools attach to live processes too; their gates are described in
[Safety of the CPU tools](#safety-of-the-cpu-tools).

The agent cannot change any of these settings: they are fixed when the server
starts. A refusal says which gate failed and what the user would have to
change.

The other tools only read snapshot files (or, for `list_python_processes`,
process metadata through `psutil` and `/proc/<pid>/maps`). `map_to_source` only
reads files inside the given repository; see below. The process diagnostics
are read-only too; their access rules are in
[Safety and access](#safety-and-access).

Tool output is bounded: every `top`, `limit`, `max_paths` and `max_targets`
argument is capped on the server (50 rows, 10 paths, 10 targets, 200
processes), retaining paths longer than 24 hops are elided in the middle, and
edge and root labels are clipped to 160 characters.

## Client configuration

Claude Code, `.mcp.json` in the project root:

```json
{
  "mcpServers": {
    "heapspy": {
      "command": "heapspy-mcp",
      "args": ["--allow", "gunicorn*", "--max-rss", "4G"],
      "env": {
        "HEAPSPY_BIN": "/usr/local/bin/heapspy"
      }
    }
  }
}
```

To run it from a checkout of this repository instead:

```json
{
  "mcpServers": {
    "heapspy": {
      "command": "uv",
      "args": ["run", "--directory", "python/heapspy-mcp", "heapspy-mcp"]
    }
  }
}
```

Leave out `--allow` to analyze existing snapshots and core files only.

## Tools

All tools return a JSON object (MCP structured content). Byte counts are
integers with a `*_human` companion such as `"412.6 MiB"`. Failures are tool
errors whose text is a JSON document:

```json
{"error": {"code": "not_found", "message": "no object at address 0x7f3a2c41b0d8 in app.hpy",
           "exit_status": 1, "hint": "..."}}
```

`code` is one of:

- a heapspy error code: `unimplemented`, `invalid_argument`, `not_found`,
  `invalid_snapshot`, `invalid_allocations`, `io`, `permission_denied`,
  `internal`, and for dumps `no_such_process`, `not_python`,
  `unsupported_python`, `invalid_image`, `truncated_image` and `unsupported`.
  A code this server does not know is reported as `internal`, with heapspy's
  code in `cli_code`;
- one added by the wrapper: `binary_not_found`, `timeout`, `usage_error` for
  exit status 2, `bad_output` for output that is not JSON, `schema_mismatch`,
  `unexpected_kind`, `cli_failed` for a failure with no error document, and
  `allocations_unsupported` for a heapspy that has no `--allocations`, or
  takes it but does not join (note `allocations_not_joined`);
- `capture_refused`, with a `reason`.

Most codes come with a `hint`. For captures: `permission_denied` names the
privilege (`CAP_SYS_PTRACE` or `kernel.yama.ptrace_scope` on Linux,
`SeDebugPrivilege` on Windows), `not_python` suggests checking that the PID is
the Python process and not a wrapper around it, `unsupported_python` lists the
versions heapspy reads (3.12 to 3.14 release builds), and `invalid_image` and
`truncated_image` describe what a core or minidump must contain.

A `snapshot` argument is a handle returned by `capture_snapshot`
(`hs-1a2b3c4d`, a file in the snapshot directory) or a path to a `.hpy` file.

### `list_python_processes(name_filter?, limit=50, measure_s=0)`

Python processes sorted by RSS: `pid`, `name`, `exe`, `cmdline` (redacted, see
[Command-line redaction](#command-line-redaction), and truncated to 200
characters), `rss_bytes`, `python_version`, `version_source`
(`exe_name` or `libpython_mapping`), `username`, `is_this_server`, and
`capture_allowed` (whether the allowlist matches). The version comes from the
executable name (`python3.12`), or on Linux from a mapped
`libpython3.12.so`, and on Windows from a mapped `python312.dll`.

On Linux each row also has `threads` and `threads_by_state`
(`running_or_runnable`, `sleeping`, `disk_wait`, `other`, from
`/proc/<pid>/task/*/stat`, at most 256 threads per process). With
`measure_s` above 0 (capped at 2), `cpu_percent_on_cpu` is measured over one
shared window for all listed processes (100 = one core busy) and
`cpu_percent_window_s` gives the window. The default is 0 so the listing stays
instant; then `cpu_percent_on_cpu` is `null` and
`cpu_percent_unavailable_reason` says why.

### `capture_snapshot(confirm, pid? | core_path? | minidump_path?, capture_contents=false, max_str?)`

Runs `heapspy dump --json` into the snapshot directory. Returns `snapshot`
(the handle), `path`, `target` (pid, name, RSS, the allowlist entry that
matched), `python_version`, `contents_captured`, `stats` (objects, edges,
types, roots, file size, `truncated`, `missing_bytes` of the image,
`read_errors`, and `pause_ms`, how long a live target was stopped; `null` for
cores and minidumps), the dump's `notes`, and `report` (`json`, or
`stderr_line` for an older heapspy, see
[Compatibility with older heapspy binaries](#compatibility-with-older-heapspy-binaries)).
A truncated snapshot also gets a `warning`: its numbers are lower bounds.
`max_str` (1 to 1048576) only applies with `capture_contents`; without it the
server default (`--max-str`, 1024) is used. The result's `max_str` is the cap
that was applied. An agent that finds the prefixes too short to tell what the
data is can capture again with a larger `max_str`.

heapspy writes into a private staging directory
(`<snapshot dir>/.<handle>.partial/`) that is removed on every path, so a dump
that fails or is killed by the capture timeout leaves nothing behind. On a
failed dump the error has heapspy's code and hint (see above).

### `heap_summary(snapshot, top=10, allocations?)`

Runs `heapspy analyze --json --top N`. Returns `top_retainers` (rank, address,
type, retained bytes and percentage, and a `path` with the rendered root path,
`root_kind`, `direct_holder` and `source_hints`), `type_histogram`, `totals`,
`memory` (RSS, graph bytes, bytes outside the graph) and `notes`.

With `allocations` (a path to a `.hpya` profile), it passes
`--allocations` and adds:

- `allocated_at` on each top retainer: the linked objects and bytes in what it
  retains (`exact_linked_bytes` is the exact part), and at most 3 allocating
  stacks, each with `allocated_in` (`file:line` of the innermost frame), at
  most 6 frames (innermost first), `top_stack_rank` and `source_hints`;
- `allocations`: `recording` (pid, Python version, start, duration,
  mechanisms, sampling parameters, overhead, samples, estimated bytes),
  `coverage`, the first 5 (or `top`) `top_stacks`, the `allocations_*` notes,
  and `caveats`; plus a `warning` when the profile does not match the
  snapshot and nothing was linked.

`source_hints` are `module.qualname` names derived from the path that
`map_to_source` can resolve: `sys.modules['app.cache'].__dict__['CACHE']`
gives `app.cache.CACHE`, and an `app.session.Session` instance followed by
`.payload` gives `app.session.Session.payload`.

Source locations recorded in the snapshot (from a heapspy whose reports have
them; see "Source locations" in `docs/json-schema.md`) are passed through:

- `type_location` and `type_location_reason` on each top retainer and
  histogram row: where the object's type is defined. A location is `location`
  (`file:line`, `file:~line` for a class, which is located near its first
  method; the class statement is at or before that line), `file`, `line`,
  `exact` and `via`. The reason says why it is `null`, e.g.
  `type_not_in_snapshot` for C types such as `builtins.dict`.
- `source_locations` on each path: at most 5 locations, the target's type
  first (`of: "type"`), then functions, classes, code objects and modules on
  the path from the target back to the root (`of: "hop"`, with the hop's
  `type`, `address` and `edge`).

`file` is the path the target process recorded, which can be outside the
repository or on another machine. To read the code, pass the type's name to
`map_to_source` with the snapshot.

With a heapspy without these fields, the summaries are as before: no
`type_location` and no `source_locations` keys.

### `retaining_path(snapshot, address? | type?, max_paths=3, max_targets=3)`

With `address`, runs `heapspy query --retaining-path`. With `type`, runs
`heapspy query --type` and then traces the `max_targets` largest instances.
Each target has `reachable`, `type_location` and up to `max_paths` paths,
each with `rendered`, `hops` (root first; a hop with a recorded location has
`source_loc`), `direct_holder`, `source_hints` and `source_locations`.

### `type_detail(snapshot, type, top=10, allocations?)`

Runs `heapspy query --type`. Returns `matched_types`, `type_locations` (the
`type_location` of each matched type), `instances`, shallow and
retained totals, `top_instances` with their paths (and `source_loc` for an
instance that is itself a function, class or module), `holders` (referrer type
and generalised edge, with a count) and `notes`. With `allocations`, each instance
gets `allocated_at` as in `heap_summary` (`top_stack_rank` is always `null`
here), and the result gets `allocations` with the notes and caveats.

### `diff_snapshots(before, after, top=10)`

Runs `heapspy diff`. Returns `identity_approximate` (always true) and
`identity_note`, `totals`, `by_type` and `by_site` growth rows, and `notes`.
Objects are matched by type and retaining site, not by address.

### `map_to_source(qualname, repo_path, snapshot?)`

Resolves a dotted name to candidates with `file`, `line`, `end_line`,
`location` (`file:line`), `kind` (`module`, `class`, `function`, `assignment`,
`instance_attribute`, `import`, `type_alias`) and an `excerpt` of at most 8
lines.

- The longest module prefix that exists as a file wins: `app.cache.Session`
  looks for `app/cache/Session.py`, then `app/cache.py` with the name
  `Session`.
- Modules are looked for in the repository root, in `src/`, `lib/` and
  `python/`, and in project directories up to two levels down that contain the
  top-level package (`services/billing/[src/]billing/`).
- `__main__.X` is heuristic, since `__main__` is whichever script was
  started. Every `.py` file in the repository (at most 500, with the skip and
  symlink rules below) that binds `X` at top level is a candidate, also a
  plain script with no main guard. Candidates are ranked: a file named
  `__main__.py`, then one with an `if __name__ == "__main__"` guard, then one
  starting with `#!`, then a definition over an import, then shallower paths.
  Each candidate lists its `main_evidence`, and the result has
  `heuristic: true` and `main_resolution` (a note, files scanned, whether the
  scan limit was reached). Compare with the process's command line from
  `list_python_processes`.
- With `snapshot` (a handle or `.hpy` path), the server first runs
  `heapspy query --locate NAME`. For a type, function or module the snapshot
  has a location for, the recorded file is mapped to a file inside
  `repo_path`: the path itself when it is absolute and inside the repository,
  then the path relative to the repository root, then a unique path-suffix
  match (`/srv/build/app/cache.py` finds `app/cache.py` when exactly one file
  ends that way). The result then has `heuristic: false` and `via:
  "snapshot"`, and each candidate has `matched_by` (`exact_path`,
  `repo_relative`, `suffix`) and `recorded` (the file, line, `exact` and `via`
  from the snapshot). The line is that of the definition found around the
  recorded line, so a class (recorded at its first method) is shown from its
  `class` statement. If the file no longer defines the name there, the
  candidate has `source_mismatch: true` and an excerpt at the recorded line.
  This settles `__main__` names, which are otherwise heuristic.
- The repository rules apply to recorded files as well: a recorded path
  outside `repo_path`, or one that leaves it through a symlink, is never
  opened.
- When the snapshot has no usable location, the name is resolved from the
  repository's files as below, and the result has `snapshot_lookup`:
  `status` (`not_in_snapshot`, `not_in_repo` with the `recorded` locations
  and why each was not used, or `no_location` with the snapshot's
  `reasons`) and a `note`. With a heapspy without `--locate`, the result is
  the same as without `snapshot`.
- Names are found as classes, functions, methods, nested classes,
  module-level and class-level assignments (including those inside `if` and
  `try` blocks), imports, and `self.X = ...` assignments in methods.
  `<locals>` segments (`app.make.<locals>.Handler`) are followed into
  function bodies.
- When nothing matches, the result has `partial_match` (the file and the part
  of the name that did not resolve) or a `note`.

The qualname must be dotted Python identifiers; paths are rejected.
`repo_path` must be an existing directory and may not contain `..`. Every file
is opened only if its real path, after following symlinks, is inside the
repository.

### `allocation_sites(profile, snapshot?, top=10)`

The top allocating Python stacks of a `.hpya` profile, by estimated bytes.
Each stack has `rank`, `estimated_bytes`, `allocated_in`, `frames` (at most 6,
innermost first, each with `function`, `file`, `line`, `location` and
`approximate`), `frames_total` and `source_hints`.

- With `snapshot` (handle or `.hpy` path), it runs
  `heapspy analyze SNAP --allocations PROFILE --json --top N`. Each stack also
  has `samples`, `events`, `linked_bytes` (bytes of objects still alive in the
  snapshot that link to the stack's samples) and `exact_linked_bytes`. The
  result has `recording` (mechanisms, sampling parameters, overhead and so
  on), `coverage`, the `allocations_*` notes, `caveats`, and a `warning` when
  nothing was linked (another pid, another CPython version, or a snapshot
  taken before recording started).
- Without `snapshot`, it runs
  `heapspy flamegraph --allocations PROFILE --folded --min-bytes=0`, the only
  profile-only output heapspy has, and adds up the folded stacks itself. Each
  stack then has `estimated_bytes` and `estimated_pct` only: `samples`,
  `events`, the linked fields, `recording` and `coverage` are not available,
  and frames have `approximate: null`. The result says so in `note`.

`source_hints` lists up to 3 frames from the innermost: `location`, `file`,
`line`, `function` and `qualname_guess`, a `module.qualname` for
`map_to_source` when `co_filename` is a relative path of identifiers
(`app/session.py` and `Session.__init__` give `app.session.Session.__init__`).
For an absolute `co_filename` there is no guess; read the file at the line.

### Where was this memory allocated?

1. Record a profile from the process, then take a snapshot of it (see
   heapspy's `docs/allocation-sampling.md`). The profile must come from the
   same pid and CPython version, and recording must start before the
   snapshot.
2. `allocation_sites(profile, snapshot)`: which stacks allocated the memory
   that is still alive (`linked_bytes`), next to how much each allocated while
   recording (`estimated_bytes`).
3. `heap_summary(snapshot, allocations=profile)`: for each top retainer, the
   stacks that allocated what it retains. `type_detail(..., allocations=...)`
   does the same per instance.
4. Read the file at `allocated_in`, or pass `qualname_guess` to
   `map_to_source`.

Limits, also in each result's `caveats`:

- Allocation data needs a `.hpya` file recorded with `heapspy record` (this
  server has no tool that records). Without a profile, heapspy says what
  keeps an object alive, not where it was allocated.
- Samples are taken only for arena, pool and large-object events during the
  recording window. Objects allocated before recording started, and blocks
  reused from freed memory, have no stack.
- Links through arena and pool samples are approximate: they name the most
  recent growth event that created the memory, not necessarily the call that
  allocated the object. Links through large-object samples are exact
  (`exact_linked_bytes`).
- `estimated_bytes` estimates what was allocated while recording;
  `linked_bytes` is what is still alive in the snapshot.

### Compatibility with older heapspy binaries

- `dump --json`: a heapspy without it rejects the flag while parsing
  arguments, before it touches the target, with clap's
  `unexpected argument '--json'` message. The server then runs the same dump
  without `--json`, parses the `wrote OUT: N objects, ...` line on stderr, and
  remembers this for the rest of its run. `report` is then `stderr_line`, and
  `types`, `read_errors`, `missing_bytes`, `pause_ms`, `python_version` and
  `notes` are `null` or empty. Failures from such a binary have no error
  document and are reported as `cli_failed` with heapspy's stderr. A newer
  heapspy prints `--json` in the `Usage:` line of every dump usage error, so
  that line alone is not taken as a sign of an older binary. The same
  fallback applies when a dump exits 0 without a dump document on stdout.
- `--allocations` and `flamegraph`: a heapspy without them gives
  `allocations_unsupported`. So does one that takes `--allocations` but
  reports the note `allocations_not_joined` (null `allocations` and
  `allocated_at`): it checks the profile, so a corrupt profile is still
  `invalid_allocations`, but it has no allocation data to return.
- Fields added within schema version 1 (such as `allocations` and
  `allocated_at`) are read as `null` when missing.

### Resource and prompt

- Resource `heapspy://workflow`: the workflow and its limits as text.
- Prompt `investigate_memory(target?)`: the same text, as a starting prompt.

## Process diagnostics

Linux only. These tools read `/proc`, cgroup files and, for `hw_counters`,
`perf_event_open(2)`. Every tool is keyed on a process. Host and cgroup
figures appear only as the `environment` of that process: what it is subject
to. The resource `heapspy://process-workflow` and the prompt
`investigate_process_cpu(target?)` describe the loop:

1. Not sure which process? `scan_processes` or `find_processes`.
2. `process_cpu(pid)` for one process and its environment.
3. Process hangs or is intermittently slow? `watch_process_stalls(pid)`, then
   `diagnose_process(pid)`.
4. Many processes on few CPUs, or sibling workers that behave differently?
   `cpu_colocation` for who shares the target's CPUs, and `group_cpu` to
   measure a set of processes at once and compare services.
5. `hw_counters(pid)` for IPC and cache misses, where a PMU exists; without
   one, its `software_fallback` and the opt-in `cache_sensitivity_probe`.
6. `diagnose_process(pid)` to weigh explanations against one window of numbers.
7. `snapshot_process_metrics` or `snapshot_group_metrics` before and after a
   change, then `compare_process_snapshots`; with three or more settings,
   `fit_per_slice_cost`.
8. For Python processes, the heap tools above.

Conventions:

- Field names carry their unit: `_seconds`, `_per_s`, `_pct`, `_ratio`,
  `_bytes`, `_cores`.
- Rates are measured over `window_s` (default 2 s, capped by `--max-window`).
  Every result has a `measurement` block with the window actually measured.
- A signal that cannot be measured is `null`, with the reason in an
  `unavailable` (or `*_unavailable_reason`) entry. It is never reported as 0.
- Lists are capped: 200 rows for `find_processes`, 20 per metric and 20 per
  condition for `scan_processes`, 20 threads per top list, 64 CPUs, 9 cgroup
  levels. Sampling reads at most 1024 threads per process in `process_cpu`,
  64 per process and 20000 in total in a scan, 128 threads in `hw_counters`,
  and 64 threads in `watch_process_stalls` (20 thread rows, 10 intervals per
  process, 3 per thread, 8 frames per example stack). `cpu_colocation` lists at
  most 8192 host tasks, tracks at most 512 that ran, and takes at most 64
  target pids; `group_cpu` measures at most 256 members, 64 threads each, in at
  most 32 groups. The result says when it truncated.
- In-window samplers keep to a budget: 5000 thread samples per second for
  `process_cpu` and `group_cpu`, 20000 task samples per second for
  `cpu_colocation`. With more threads the rate is lowered and the result
  says so (`rate_note`).

### Process search

`find_processes(sort_by?, filters..., limit=20, window_s=1)` lists every process
the server can see, filtered, and sorted by one metric:

| Metric | Source |
|---|---|
| `cpu_percent_on_cpu` | `utime + stime` delta from `/proc/<pid>/stat` (100 = one core) |
| `rss_bytes`, `rss_growth_bytes_per_s` | `/proc/<pid>/stat` RSS pages |
| `threads` | `/proc/<pid>/stat` |
| `voluntary_switches_per_s`, `nonvoluntary_switches_per_s` | `/proc/<pid>/task/*/status`, summed over threads |
| `runqueue_wait_ratio` | `/proc/<pid>/task/*/schedstat`: wait time / on-CPU time |
| `iowait_seconds` | `delayacct_blkio_ticks`, needs delay accounting |
| `open_fds`, `fd_limit_used_pct` | `/proc/<pid>/fd`, soft `Max open files` in `/proc/<pid>/limits` |
| `read_bytes_per_s`, `write_bytes_per_s` | `/proc/<pid>/io` (storage-layer bytes) |
| `uptime_s` | start time against `/proc/uptime` |
| `cfs_throttled_ratio` | `nr_throttled / nr_periods` of the process's cgroup in the window |
| `major_faults_per_s` | `majflt` delta from `/proc/<pid>/stat` |

Filters, all optional: `name` (glob on the name or the executable's file
name), `cmdline` (substring), `cmdline_regex`, `user` (name or uid), `pids`,
`parent_pid` with `include_descendants`, `cgroup` (substring of the path),
`container_id` (prefix), `pod_uid`, `state` (`/proc` state letters, e.g. `D`),
`is_python`, `include_kernel_threads`. `window_s=0` gives an instant listing
with the rate metrics `null`. `cmdline_max_chars` (default 200, capped by
`--max-cmdline-chars`) sets how much of each redacted command line is
returned; the start is kept. `scan_processes` takes it too. Each row has `pid`, `ppid`, `comm`, `exe`,
redacted `cmdline`, `user`, `state`, `cgroup`, `kubernetes` (pod UID, QoS
class, container id) and `metrics`.

`scan_processes(window_s=2, limit_per_metric=5, fd_threshold_pct=80,
major_fault_threshold_per_s=100)` makes one pass over every process and returns
`outliers_by_metric` (the top N for each metric above) and `conditions`, each
with its definition and raw values:

- `disk_wait_whole_window`: threads in state `D` in all 5 samples of the window.
- `zombies_by_parent`: processes in state `Z`, grouped by parent.
- `fd_usage_above_threshold`: open fds above `fd_threshold_pct` of the soft `RLIMIT_NOFILE`.
- `rss_grew_every_sample`: RSS strictly increased between each of the 5 samples.
- `cgroups_throttled_in_window`: cgroups of scanned processes whose `nr_throttled` rose.
- `major_faults_whole_window`: major faults at or above
  `major_fault_threshold_per_s` in each of the 4 intervals between the samples.
- `cgroup_memory_pressure_full`: cgroup v2 directories of scanned processes
  whose `memory.pressure` `full` total rose in the window, with `full` and
  `some` as a share of the window. The root cgroup's row covers the whole host
  and is listed last. On a host without a v2 hierarchy this is empty with a
  note (cgroup v1 `memory.pressure_level` is an eventfd notification and cannot
  be read).
- `disk_wait_in_memory_management`: threads in `D` at the end of the window
  whose kernel stack (as root) or `wchan` is in a memory-management category
  (`page_fault_io`, `direct_reclaim`, `memcg_high_throttle`, `dirty_throttle`,
  `compaction`, `oom`). Other `D` threads are listed as `d_state_candidates`
  for `watch_process_stalls`, including a page or folio wait seen only through
  `wchan`, which may be a fault or a `read(2)`. At most 64 threads are inspected.
- `frozen_or_stopped`: cgroups frozen by the freezer (v2 `cgroup.events`
  `frozen`, v1 `freezer.state` `FROZEN` or `FREEZING`) and processes in state
  `T` or `t`. These are held on purpose; a v1-frozen task sits in `D` in
  `__refrigerator` and is not listed as a memory wait.
- `oom_kill_counters`: memory cgroups whose `oom_kill` counter is above 0 (v2
  `memory.events`, v1 `memory.oom_control`).

Both read a bounded set of files per process at the start and end of the
window, plus `/proc/<pid>/stat` (RSS and the major-fault count; and thread
states for processes that had threads in `D`) at three points inside it, and
once per distinct cgroup `cpu.stat`, `memory.events` (or v1
`memory.oom_control` and `memory.failcnt`), `memory.pressure` and the freezer
state. Processes that exit mid-scan or cannot be read are skipped and counted
in `skipped`; they never cause an error. Each result has a `cost` block with
the time spent reading `/proc` and cgroup files, the number of distinct
cgroups read, the number of `D` threads inspected, and the list of files read.
Measured on a 4-core VM with about 1000 processes (1000 idle `sleep`
processes added), outside the window's sleeps: 0.29 to 0.32 s of CPU and
wall time per call before the memory conditions were added, 0.30 to 0.31 s
after, with 6 distinct cgroups read.

### `process_cpu(pid, window_s=2, top_threads=5, wait_sample_rate_hz=50)`

Samples the process and its threads at the start and end of the window.
Inside it, thread states and wait sites are sampled at `wait_sample_rate_hz`
(capped by `--max-stall-rate` and the thread-sample budget), and the host's
runnable count four times. `measurement.wait_site_sampling` gives the rate
used, the sample count and late samples. Four samples in a window were too
few: in the CPU-contention blind test (end of this file), a worker asleep
about 28% of the time showed no sleeping samples.

- `cpu`: `cpu_seconds_user`, `cpu_seconds_system`, `cpu_cores_used`,
  `cpu_percent_on_cpu`, `user_share`.
- `iowait_seconds`: block I/O delay from delay accounting. `null` with a reason
  when `kernel.task_delayacct=0`.
- `context_switches`: `voluntary_switches_per_s`, `nonvoluntary_switches_per_s`
  (summed over threads, from `/proc/<pid>/task/*/status`), `voluntary_share`,
  `switches_per_cpu_second`.
- `scheduler`: `on_cpu_seconds`, `runqueue_wait_seconds` and
  `runqueue_wait_ratio` (time runnable but waiting for a CPU, over time on CPU,
  from `/proc/<pid>/task/*/schedstat`), `runqueue_wait_seconds_per_s` (mean
  number of the process's threads waiting), `on_cpu_seconds_per_s`,
  `timeslices_per_s` (schedstat field 3, `pcount`), `cpu_migrations_per_s`
  (`se.nr_migrations`, needs `CONFIG_SCHED_DEBUG`), `avg_timeslice_us`
  (on-CPU time / timeslices; on-CPU time / switches, and said so, where
  there is no timeslice count), `sched_slice_us` (`se.slice` from
  `/proc/<tid>/sched` on 6.6+; otherwise debugfs `base_slice_ns`, the sysctl,
  or the EEVDF default for the running kernel, marked as an estimate) and
  `avg_timeslice_to_sched_slice_ratio`.
- `page_faults`: major and minor faults per second.
- `threads`: count, `threads_by_state`, `state_share_over_window`, started and
  exited in the window.
- `wait_sites`: where sleeping threads were blocked in the kernel
  (`/proc/<tid>/wchan` and `/proc/<tid>/syscall`), grouped as `futex`,
  `poll_epoll_select`, `timer_sleep`, `block_io`, `pipe_socket`, `other`, with
  the most common raw wait channels. `futex` covers mutexes, condition
  variables and the GIL.
- `top_threads`: by CPU seconds, non-voluntary and voluntary switches, and
  run-queue wait.
- `environment.cgroup`: the cgroup from `/proc/<pid>/cgroup`, its version, the
  Kubernetes pod UID, QoS class and container id parsed from the path, its own
  `cpu.stat` deltas (usage, user, system, `nr_periods`, `nr_throttled`,
  `throttled_seconds`, `cfs_throttled_ratio`) and limit, the same for each
  ancestor that has a quota (`ancestors_with_quota`, since a pod-level limit
  throttles its containers), `tightest_limit_cores`,
  `max_cfs_throttled_ratio`, PSI from `cpu.pressure`, `memory.pressure` and
  `io.pressure` (`avg10_pct`, `avg60_pct`, and `window_pct` from the `total`
  delta), and `cpuset_cpus_effective`.
- `environment.host` (unless `--no-host-context`): `cores_online`,
  `cpu_time_pct` (user, nice, system, idle, iowait, irq, softirq, steal),
  `context_switches_per_s`, `procs_running_mean` and `procs_running_per_core`
  (sampled in the window), load average, PSI for cpu, memory and io, and
  run-queue wait per CPU from `/proc/schedstat` (versions 15 to 17; other
  versions or a kernel without `CONFIG_SCHEDSTATS` give `null` and a reason).

On a cgroup v2 host everything comes from the unified hierarchy. On a hybrid
host where the `cpu` controller is still on cgroup v1, CPU accounting and
throttling come from the v1 `cpu` and `cpuacct` hierarchies (`cpu.stat`
`throttled_time`, `cpu.cfs_quota_us`, `cpuacct.usage*`) and pressure from the
v2 directory; `notes` says so. Kubernetes pod names are not looked up; the UID
is returned.

### `hw_counters(pid, window_s=2)`

Opens counters with `perf_event_open` through ctypes (no `perf` binary) on each
thread of the process, with `inherit` so threads started in the window are
counted when they exit: cycles, instructions, last-level cache read accesses
and misses (`PERF_TYPE_HW_CACHE`, falling back to the generic cache events),
context switches, CPU migrations and task clock. Returns `ipc`, `llc_mpki`
(misses per 1000 instructions), `llc_miss_ratio`, the software rates, and per
counter `raw_value`, `time_enabled_ns`, `time_running_ns`, `running_fraction`
and `multiplexed`. When the kernel multiplexes counters, `value` is
`raw_value * time_enabled / time_running`.

When a hardware event cannot be opened, its value is `null` and the reason is
exact: `unavailable: no PMU exposed (common in VMs)` with the event source
devices that do exist, `EACCES`/`EPERM` with `perf_event_paranoid` and the
capability needed (`CAP_PERFMON` on 5.8+, `CAP_SYS_ADMIN` before), `ENOENT`
for an event this PMU lacks, and so on. Software events are still reported.

When IPC or LLC misses are `null`, `software_fallback` lists what can be
measured instead over the same window: the perf software events
(`context_switches_per_s`, `context_switches_per_cpu_second`,
`cpu_migrations_per_s`), and from `/proc/<tid>/schedstat` `on_cpu_seconds_per_s`,
`timeslices_per_s`, `avg_timeslice_us`, `switches_per_cpu_second`,
`voluntary_share` and `runqueue_wait_ratio`. `see_also` points to
`cpu_colocation`, `fit_per_slice_cost` and `cache_sensitivity_probe`, and says
whether the probe is enabled on this server.

If `/sys/fs/resctrl` is mounted and the pid is in a resctrl group, `resctrl`
reports that group's L3 occupancy and cumulative memory-bandwidth bytes. These
cover every task in the group. The tool never creates a group. Per-process
monitoring groups are a possible future addition.

### `snapshot_process_metrics(pid, window_s=2, work_rate?, label?)` and `compare_process_snapshots(before, after)`

A snapshot measures one window (as `process_cpu`, plus hardware counters) and
keeps the key numbers in server memory under a token (`pm-1a2b3c4d`; the newest
64 are kept). `work_rate` is a number the caller supplies from its own
metrics, such as requests per second during the window. With it the snapshot
records `cpu_seconds_per_unit_work`.

The comparison gives each metric `before`, `after`, `change` and
`after_to_before_ratio`. When both snapshots have a `work_rate`, `work` gives
`work_rate_ratio`, `cpu_cores_ratio` and `cpu_per_unit_work_ratio`. A value
above 1 means CPU grew faster than work between the windows: the pattern to
look for when a feedback loop is suspected.

Process snapshots also keep `on_cpu_seconds_per_s`, `timeslices_per_s`,
`avg_timeslice_us` and `sched_slice_us`. `compare_process_snapshots` also
takes two group snapshots (below) and then compares per group key.

### `cpu_colocation(pid | pids | name | cmdline | ..., window_s=2, rate_hz=50, colocated_per_cpu=10)`

Which tasks shared CPUs with a target over the window, and how often they
woke. It answers what `runqueue_contention` alone does not: who preempts whom
and why per-unit CPU changed.

At the start it reads `stat`, `status` and `schedstat` of every task on the
host. Every task that runs in the window (its on-CPU time advanced, checked
every 0.5 s, or it was runnable) is then sampled at `rate_hz` for its state
and the CPU it last ran on (`/proc/<pid>/task/<tid>/stat` field 39; for a
runnable task, the run queue it waits on). At the end it reads `stat`,
`status`, `schedstat` and `sched` (for `se.slice`) of those tasks. The
server's own process is left out.

- `target_tasks`: per target thread, `home_cpu` (where most samples were),
  samples per CPU and runnable samples per CPU, `cpus_allowed`
  (`Cpus_allowed_list`), on-CPU and run-queue wait per second, voluntary and
  non-voluntary switches per second, `timeslices_per_s`, `avg_timeslice_us`
  with its source, `sched_slice_us`, `avg_sleep_us` and `short_sleeper`.
- `cpus`: for each CPU the targets ran on or may run on: target tasks and other
  busy tasks with that home CPU, `mean_runnable_tasks`, on-CPU and run-queue
  wait per second, voluntary and non-voluntary switches per second,
  `wakeups_per_s`, `colocated_wakeups_per_s` (non-target tasks only),
  `colocated_short_sleeper_wakeup_share`, the targets' non-voluntary switches
  there, `target_preemptions_per_colocated_wakeup`, and the co-located tasks
  (by their on-CPU time there) with the same per-task fields. A task's rates
  are spread over CPUs in proportion to its samples on each.
- `placement`: target and other busy tasks per CPU, mean runnable tasks per
  CPU, and the imbalance (`runnable_max_minus_min`, `runnable_max_to_min_ratio`,
  `runnable_cv`, `busy_tasks_max_minus_min`).
- `wakeup_preemption_attribution`: labelled `inference`. A task's voluntary
  switches are taken as its wakeups: each voluntary switch is a sleep, and
  each sleep ends in a wakeup. Per target thread, its non-voluntary switches
  (also per second on CPU) next to the co-located wakeups on its home CPU; the
  totals on the target CPUs; the share from short sleepers; the Pearson
  correlation across target threads between non-voluntary switches per second
  on CPU and co-located wakeups (per second on CPU, because a wakeup can only
  preempt the task that is running); and the top short sleepers. A short
  sleeper wakes at least 100 times a second and sleeps under 1 ms per wakeup
  on average, where sleep is the window minus on-CPU time minus run-queue wait
  (this needs no `CONFIG_SCHEDSTATS`). No trace is taken of which wakeup
  preempted which task.
- `cost`: tasks listed and tracked, rescans, the sampler's CPU seconds, and the
  files read.

The selector takes the `find_processes` filters: `pid`, `pids`, `name`,
`cmdline`, `cmdline_regex`, `parent_pid`, `cgroup`, `pod_uid`,
`container_id`. All threads of the matching processes are targets.

### `group_cpu(selector..., group_by="none", group_regex?, window_s=2, rate_hz=20, work_rate?, work_rates?, include_members=true)`

Measures a set of processes over one window, as `process_cpu` measures one.
A single-pid view hid that sibling workers of one service behaved very
differently; this one shows them side by side.

- `member_rows`: per member `cpu_cores_used`, `on_cpu_seconds_per_s`, user and
  system CPU per second, voluntary and non-voluntary switches per second,
  `voluntary_share`, `switches_per_cpu_second`, run-queue wait per second and
  ratio, `timeslices_per_s`, `avg_timeslice_us`, `cpu_migrations_per_s`, major
  and minor faults per second, `threads`, `runnable_share` (share of in-window
  thread samples in state R), `home_cpu`, samples per CPU and `cpus_allowed`.
- `groups`: per group key, for every metric above: `sum` (for rates that add
  up; `null` for ratios), `mean`, `p50`, `p90`, `min`, `max`, `spread` (max
  minus min), `max_to_min_ratio` and `cv`; for ratios also `pooled`, recomputed
  from the members' summed counts (for example `avg_timeslice_us` = total
  on-CPU time / total timeslices). `placement` gives members per home CPU and
  mean runnable member threads per CPU.
- `comparison`: with two or more groups, each group's per-member mean or
  pooled ratio side by side.
- `work`: with `work_rate` (one group) or `work_rates` (group key to units per
  second): `cpu_seconds_per_unit_work` (from schedstat on-CPU time, or clock
  ticks without schedstat), `timeslices_per_unit_work` and
  `units_of_work_per_timeslice`.

`group_by`: `none`, `comm`, `exe`, `parent`, `cgroup`, `pod`, `container`,
`cmdline_arg:N` (the Nth argument of the redacted command line, 0 being the
program), or `cmdline_regex` with `group_regex` (its first capture group,
searched in the redacted command line). The selector takes the
`find_processes` filters. The server's own process is left out.

### `snapshot_group_metrics(selector..., group_by, work_rate?, work_rates?, label?)` and `fit_per_slice_cost(tokens, group?, work_rates?)`

`snapshot_group_metrics` measures as `group_cpu` does and keeps each group's
numbers under a token (`pg-...`, in the same store as process snapshots).
Compare two with `compare_process_snapshots`.

`fit_per_slice_cost` takes three or more snapshots of one process or one group
taken at different settings (CPU count, co-location, slice length) and fits

    cpu_seconds_per_unit_work = a + b * timeslices_per_unit_work

which is the same as `a + b / units_of_work_per_timeslice`. `b`, reported as
`fixed_cost_per_timeslice_us`, is CPU each timeslice costs on top of the work
in it, for example refilling caches after other tasks ran; `a`,
`cost_per_unit_work_without_slicing_us`, is the cost with infinitely long
slices. The result has standard errors, R squared, residuals per point and
the largest residual as a share, and each point with its work rate, CPU per
unit of work, average timeslice and timeslices per second. With fewer than
three usable points it warns; with all points at the same timeslices per unit
of work, or a negative slope, it says the model does not describe them.
`work_rates` (token to units per second) supplies the rate for snapshots whose
rate was known only after the window.

### `cache_sensitivity_probe(confirm, mode="synthetic", pid?, cpu?, sizes_bytes?, phase_s=1, neighbour_bytes?)`

For hosts without hardware counters. Off unless the server runs with
`--allow-load-probes`, and every call needs `confirm=true`; without it the
result is `confirmation_required` with the plan (CPUs, sizes, longest
duration). It is the one process-diagnostics tool that is not read-only.

It runs a small benchmark in separate processes and never attaches to, reads
or signals the target. Per working-set size it walks a random linked list of
64-byte nodes (one dependent load per cache line) pinned to one CPU, first
alone and then shared:

- `mode="synthetic"`: two neighbours pinned to the same CPU, one walking a
  large buffer (default 4 x L2, at least 8 MiB) that evicts the probe's
  working set from the per-core caches, and one that sleeps about 50 us
  between short bursts, so its wakeups preempt the probe.
- `mode="colocated"`: the shared phase runs on the CPU where `pid` ran most
  (the pid must be on the allowlist), against the target's co-located tasks;
  the alone phase runs on another CPU.

Throughput is measured per second of the probe's own CPU time, so the share
of the CPU it gets does not count. Per size: `ns_per_access` alone and shared,
`slowdown_ratio`, the probe's timeslices per second and average timeslice
(from its own `/proc/thread-self/schedstat`), and `extra_cpu_us_per_timeslice`,
an estimate of the refill per slice to compare with
`fixed_cost_per_timeslice_us`. Sizes default to the host's caches from
`/sys/devices/system/cpu/cpu0/cache/`: L1d / 3 (at most 16 KiB), L2 / 2,
L2 x 3/4, L2 x 4, and L3 x 2 when that is at most 64 MiB.

It measures how sensitive a synthetic workload of a given size is to cache
sharing on this host. It does not measure the target's cache misses. It adds
up to one CPU of load on the chosen CPU for at most 2 x sizes x `phase_s`
seconds (at most 6 sizes, `phase_s` at most 3). The benchmark is C, built
once per server process with the host's C compiler into a temporary directory;
without a compiler it falls back to the same walk in Python, whose interpreter
overhead per access hides much of the difference. Every child process is
killed on every exit path. Refusals are `diagnostics_refused` with reason
`load_probes_disabled`, `confirmation_required` or `invalid_argument`; a
benchmark that fails returns `probe_failed`.

### `watch_process_stalls(pid, window_s=5, rate_hz=20)`

For a process that hangs or is intermittently slow. A process that keeps
taking page faults, or waits in reclaim, swap-in or `memory.high` throttling,
keeps leaving its hot path for the kernel; it looks hung or slow while its own
code is fine. This tool samples every thread at a fixed rate (`rate_hz`,
capped by `--max-stall-rate`, default 100; `window_s` capped by
`--max-window`) and reads, per sample:

- the state letter, `minflt`, `majflt` and `utime + stime` from
  `/proc/<pid>/task/<tid>/stat`;
- `/proc/<tid>/wchan`;
- the kernel stack from `/proc/<pid>/task/<tid>/stack` (root only; otherwise
  `measurement.stack_source` is `wchan` and `unavailable.kernel_stacks` says
  why).

Each sample gets one wait category (below). Per thread the result has the
share of samples in each state and category, major and minor faults per
second, CPU seconds, and its longest no-progress intervals. A no-progress
interval is a run of consecutive samples in which the thread's
`utime + stime` (in clock ticks) did not advance and the thread was not
voluntarily idle (`futex`, `poll`, `nanosleep`, `ipc_wait`). Each interval has
its duration, start offset, dominant category, counts by category and state,
the major faults taken in it, an example kernel stack (8 frames) and `wchan`,
and whether it began before or was still open at the end of the window.

Per process (`process`): fault totals and rates, CPU split and
`system_share`, `category_share` over all thread samples and
`category_share_of_active` (excluding voluntary idle),
`memory_stall_share_of_active` (share of non-idle samples in
`page_fault_io`, `direct_reclaim`, `memcg_high_throttle` or `compaction`),
`no_progress_seconds_by_category`, `memory_stall_no_progress_seconds_per_s`,
the 10 longest intervals across threads, the frames that matched each
category, unclassified stacks (10, innermost frames kept), and `memory`:
`VmRSS`, `VmHWM`, `RssAnon`, `RssFile`, `VmSwap` from `status` and `Rss`,
`Pss`, `Swap`, `SwapPss` from `smaps_rollup`.

`process.delay_accounting`: when delay accounting is on, the per-second
deltas of `thrashing_delay`, `swapin_delay`, `freepages_delay` (direct
reclaim), `blkio_delay`, `compact_delay` and `cpu_delay`, summed over the
process's threads, from the taskstats generic-netlink interface (needs
`CAP_NET_ADMIN`). When `kernel.task_delayacct=0` they are `null` and the reason
says how to enable it: `sysctl kernel.task_delayacct=1` or the `delayacct`
boot option. The server never changes the sysctl and has no option to.

`environment.cgroup_memory`: the memory cgroup. On cgroup v2: `memory.current`
against `memory.max` and `memory.high` (`usage_to_max_ratio`,
`usage_to_high_ratio`, and each ancestor with a limit), `memory.events`
deltas (`low`, `high`, `max`, `oom`, `oom_kill`, `oom_group_kill`),
`memory.stat` deltas and rates (`workingset_refault_anon` and `_file`,
`pgmajfault`, `pgfault`, `pgscan`, `pgsteal`, `pgscan_direct`,
`pgscan_kswapd`), `refaults_per_s`, and `memory.pressure` `some` and `full`
with `avg10_pct`, `avg60_pct` and `window_pct`. On a hybrid host where the
memory controller is on cgroup v1: `memory.usage_in_bytes` against the
tighter of `memory.limit_in_bytes` and `hierarchical_memory_limit`,
`memory.failcnt` as the `max` event count, `oom_kill` and `under_oom` from
`memory.oom_control`, the hierarchical (`total_`) `memory.stat` counters, and
`memory.pressure` from the process's v2 directory (PSI follows the v2
hierarchy). Values v1 does not have (`memory.high`, the `oom` event, `pgscan`,
`pgsteal`) are `null` with a reason; `memory.pressure_level` is an eventfd
notification and cannot be read. `environment.host`: `/proc/pressure/memory`
over the window, `/proc/vmstat` deltas and rates (`pgmajfault`,
`workingset_refault_*`, `allocstall_*` summed as `allocstall_total`,
`pgscan_direct`, `pgscan_kswapd`, `pgsteal_direct`, `pswpin`, `pswpout`,
`compact_stall`, `oom_kill`), and `meminfo` (`MemAvailable`, `SwapTotal`,
`SwapFree`), with a note when no swap is configured.

#### Wait categories

`CATEGORIES` in `src/heapspy_mcp/system/wait_categories.py` is the one table
that maps kernel frames to categories. It is checked in order and the first
category with a matching frame wins, so memory-management waits are found
even when the innermost frame is a generic sleep. Names are matched without
offsets and compiler suffixes (`.isra.0`), exactly or by prefix.

| Category | Meaning | Frames (examples) |
|---|---|---|
| `stopped_or_frozen` | stopped by a signal or ptrace (state `T`/`t`), or frozen by the cgroup freezer | `__refrigerator`, `do_freezer_trap`, `do_signal_stop`, `ptrace_stop`; `get_signal` as the innermost frame (v2 freeze) |
| `oom` | inside the OOM killer, or waiting for an OOM kill in its cgroup | `out_of_memory`, `mem_cgroup_out_of_memory`, `mem_cgroup_oom_synchronize` |
| `memcg_high_throttle` | over `memory.high`: reclaiming and then sleeping on return to user space | `mem_cgroup_handle_over_high` |
| `compaction` | compacting memory for a high-order allocation, or kcompactd | `compact_zone`, `try_to_compact_pages`, `__alloc_pages_direct_compact`, `kcompactd` |
| `direct_reclaim` | an allocation or a cgroup charge reclaims memory itself, or is throttled while others reclaim | `try_to_free_pages`, `try_to_free_mem_cgroup_pages`, `shrink_*`, `__alloc_pages_slowpath`, `try_charge_memcg`, `reclaim_throttle` |
| `dirty_throttle` | a writer paused until writeback catches up | `balance_dirty_pages` |
| `page_fault_io` | a page fault waits for the page to be read in: a file page (often a refault) or a swap-in | `do_swap_page`, `swap_readpage`; `folio_wait_bit_common`, `__folio_lock`, `wait_on_page_bit`, `io_schedule` when a fault frame (`filemap_fault`, `handle_mm_fault`, `exc_page_fault`, ...) is on the stack |
| `futex` | a mutex, condition variable, semaphore or the Python GIL | `futex_*`, `do_futex` |
| `poll` | poll, select, epoll or io_uring waits | `do_epoll_wait`, `ep_poll`, `do_select`, `do_sys_poll` |
| `nanosleep` | a timed sleep | `hrtimer_nanosleep`, `do_nanosleep` |
| `ipc_wait` | input from a socket, pipe or terminal, a child to exit, or a signal | `unix_stream_read_generic`, `anon_pipe_read`, `do_wait`, `sk_wait_data` |
| `block_io` | block I/O outside a page fault: `read(2)` of an uncached page, fsync, writeback, journal | `submit_bio_wait`, `jbd2_*`, `blk_mq_*`, page and folio waits without a fault frame |
| `running` | state `R`, or an empty stack with `wchan` 0 (it woke up between the reads); no wait site is known for a task on a CPU | |
| `other` | not in the table; the innermost frames are kept in `unclassified_stacks` | e.g. a fault waiting on `mmap_lock` (`rwsem_down_read_slowpath`) |
| `unavailable` | neither the stack nor `wchan` could be read | |

`futex`, `poll`, `nanosleep` and `ipc_wait` are voluntary idle and do not
count towards no-progress intervals. Without stacks (not root), a page or
folio wait seen through `wchan` counts as `page_fault_io` when the thread's
major-fault count rose since its previous sample, and as `block_io`
otherwise.

#### Frozen vs thrashing

A process frozen by the cgroup freezer, or stopped by `SIGSTOP` or a
debugger, also makes no progress, and a v1-frozen task is in state `D`. The
result reports this separately in `frozen_or_stopped`: the freezer state of
its cgroups (v2 `cgroup.events` `frozen` and `cgroup.freeze`, v1
`freezer.state`, `self_freezing`, `parent_freezing`), the share of samples in
state `T`/`t` and in `stopped_or_frozen`, and an `assessment`:
`frozen_by_cgroup_freezer`, `stopped_by_signal_or_ptrace`,
`held_in_freezer_or_signal_stop`, `partly_stopped_or_frozen` or
`not_frozen_or_stopped`. The no-progress intervals of such a process have the
dominant category `stopped_or_frozen`, not a memory one.

### `diagnose_process(pid, window_s=2, focus="all")` and `diagnose_process_cpu(pid, window_s=2)`

Measures the process, its environment and its hardware counters in one
window and, side by side in the same window, samples its threads as
`watch_process_stalls` does at 10 Hz and its CPUs as `cpu_colocation` does at
20 Hz. It then checks one fact and scores
these explanations for its CPU use, CPU wait, slowness or stalls:

| id | What it describes |
|---|---|
| `runqueue_contention` | runnable threads waiting for a CPU |
| `short_timeslices` | timeslices far shorter than the scheduler slice, ending in preemption, with frequent wakeups of co-located tasks |
| `cache_or_memory_bound_contention` | low IPC with many LLC misses |
| `cfs_throttling` | the cgroup or an ancestor hits its quota |
| `lock_contention` | frequent blocking and waking, mostly in futex |
| `spin_waiting_or_compute` | on CPU, few switches, normal IPC; counters cannot separate the two |
| `gil_convoy` | several Python threads, CPU near one core, frequent hand-offs |
| `steal` | the hypervisor runs other guests on this VM's CPUs |
| `memory_thrashing` | threads keep waiting for memory: refaults, swap-in, reclaim, `memory.high` throttling, compaction |
| `io_pressure` | I/O stalls (PSI), iowait, threads in block I/O outside page faults |

`memory_thrashing` replaces the earlier `memory_pressure`, which read only
memory PSI and the major-fault rate. It reads those (major faults at
100/s or more, PSI `full` at 1% or more, or `some` at 5% or more) plus
workingset refaults (100 pages/s or more, from the cgroup's `memory.stat`, or
the host's `/proc/vmstat` without one), the share of non-idle thread samples
in `page_fault_io`, `direct_reclaim`, `memcg_high_throttle` or `compaction`
(0.5 or more scores high, 0.1 some), and no-progress time in those waits
(0.2 s/s or more). Evidence against is the absence of each: under 5 major
faults/s, under 10 refaults/s, PSI `some` under 0.5%, and under 1% of
non-idle samples in a memory wait. Keeping both hypotheses would have counted
the same PSI and fault evidence twice. With per-thread categories,
`io_pressure`'s disk-wait share counts only `block_io` samples, since `D`
samples in page faults are memory waits. Refault and `memory.stat` counters
cover every process in the cgroup; the wait share is the process's own.

A thread that is runnable but not running is starved, not idle. When
`runqueue_wait_ratio` is at least 0.2 or the thread was in state R in at least
90% of samples (`runnable_share`), `spin_waiting_or_compute` judges busy and
idle by `cpu_demand_cores` (on-CPU time plus run-queue wait, per second)
instead of `cpu_cores_used`, and `runqueue_contention` gets evidence for from
the runnable share. When `voluntary_share` is under 0.5, most switches are
preemptions, so a high `switches_per_cpu_second` no longer counts against
compute or for lock contention. In the blind test, a starved worker using
0.151 cores with a run-queue wait ratio of 5.6 had compute marked
`not_indicated` as mostly idle; it now counts 1.0 cores of demand.

`short_timeslices` reads `avg_timeslice_to_sched_slice_ratio` (under 0.1
scores 3, under 0.25 scores 2, 0.5 or more is against), `voluntary_share`
(under 0.5 for: slices end in preemption; 0.7 or more against: the threads
block on their own), `colocated_wakeups_per_s` (1000 or more) and
`colocated_short_sleeper_wakeup_share` (0.5 or more). Its next measurements
are `cpu_colocation`, moving the co-located short sleepers to a separate
cpuset and comparing CPU per unit of work, comparing CPU per unit of work at
different slice lengths with `fit_per_slice_cost`, and, without a PMU,
`cache_sensitivity_probe`.

`fact_checks` comes first. `deliberately_frozen` is true when the process's
cgroup is frozen or at least 90% of thread samples were stopped or frozen.
Then the result says that the explanations are moot until the process is
thawed or continued, every hypothesis carries `moot_because`, and
`memory_thrashing` has evidence against.

`focus` (`all`, `cpu`, `memory`, `io`) limits which explanations are listed;
all are measured. `diagnose_process_cpu` is the same tool with
`focus="all"`, kept under its earlier name. Besides the hypotheses the result
has `signals`, `memory_stalls` (the stall numbers that were scored, the 3
longest no-progress intervals, `frozen_or_stopped`), `colocation` (the
co-location numbers that were scored, with the top 3 co-located tasks per
CPU), and
`environment.cgroup_memory` and `environment.host_memory` as in
`watch_process_stalls`.

Each hypothesis has `evidence_for` and `evidence_against` (signal, raw value,
threshold name and value, unit, weight), `unknown` (signals that could not be
measured, with the reason), a `score` (sum of weights for minus against), an
`assessment` (`consistent_with_measurements` for a score of 3 or more,
`partly_consistent` for 1 or 2, `not_indicated` for 0 or less with evidence,
`insufficient_signals` when nothing was measured) and `next_measurements`
that would separate it from the others, such as measuring IPC at two
densities, pinning with the static CPU manager, reducing threads per process
or removing the CPU limit on one instance. The ranking says how well each
explanation fits the numbers of one window. It does not establish a cause, and
several can hold at once.

All thresholds are in one table, `THRESHOLDS` in
`src/heapspy_mcp/system/diagnose.py`, and the result repeats it under
`thresholds`.

### Signals and what they separate

| Signal | Field | Separates |
|---|---|---|
| Voluntary vs non-voluntary switches | `voluntary_share`, `*_switches_per_s` | blocking (locks, I/O, GIL) vs preemption (CPU contention, quota) |
| Run-queue wait | `runqueue_wait_ratio` | waiting for a CPU vs running |
| Runnable share and CPU demand | `runnable_share`, `cpu_demand_cores` | a starved thread vs an idle one |
| Average timeslice vs scheduler slice | `avg_timeslice_us`, `sched_slice_us` | slices cut short by preemption vs full slices |
| Co-located wakeups | `colocated_wakeups_per_s`, `colocated_short_sleeper_wakeup_share` | preemption by short sleepers on the same CPU vs by other CPU-bound tasks |
| Placement | `placement` in `cpu_colocation` and `group_cpu` | uneven spread of tasks over CPUs vs an even one |
| Per-slice cost | `fixed_cost_per_timeslice_us` | CPU per unit of work that rises with slicing vs a fixed cost per unit |
| Host runnable per core | `procs_running_per_core` | host-wide CPU contention vs a quota on an otherwise idle host |
| PSI cpu, memory, io | `psi`, `pressure` | CPU vs memory vs I/O stalls, host and cgroup |
| CFS throttling | `cfs_throttled_ratio`, `throttled_seconds_per_s` | quota limits vs host contention |
| Steal | `steal_pct` | hypervisor contention vs guest-side causes |
| IPC, LLC MPKI | `ipc`, `llc_mpki` | memory or shared-cache bound vs compute or spin |
| Wait sites | `wait_sites.share_by_site` | futex vs poll vs block I/O |
| CPU per unit of work | `cpu_seconds_per_unit_work` | efficiency change vs load change |
| Major faults | `major_faults_per_s` | pages read from storage on access vs a working set in memory |
| Workingset refaults | `refaults_per_s`, `stat_per_s.workingset_refault_*` | a working set that does not fit (evicted pages needed again) vs first-time reads |
| Memory wait share | `memory_stall_share_of_active`, `category_share` | threads held in page-fault I/O, reclaim, `memory.high` throttling or compaction vs running, locks or block I/O |
| No-progress intervals | `longest_no_progress_intervals`, `no_progress_seconds_by_category` | a thread that stops for seconds in one wait vs one that keeps running |
| Memory PSI `full` | `pressure.full.window_pct` | every task in the cgroup stalled on memory at once vs some tasks |
| `memory.events` `high`, `max` | `events_in_window` | throttling at `memory.high` or reclaim at `memory.max` vs no limit pressure |
| System share of CPU | `system_share` | time in the kernel (reclaim, faults) vs user code |
| Freezer and `T` state | `frozen_or_stopped`, `fact_checks` | a process held on purpose vs one that is stalled |

### Safety and access

All of these tools are read-only except `cache_sensitivity_probe`, which
starts benchmark processes and is off by default.

- `find_processes`, `scan_processes`, `cpu_colocation` and `group_cpu` read the
  same per-task `/proc` files that `ps` and `top` show (`stat`, `status`,
  `schedstat`, `sched`, `cgroup`), and are open without the allowlist, so an
  agent can find the process to look at. They do not read wait channels,
  syscalls or stacks. Command lines are redacted.
- `cache_sensitivity_probe` needs `--allow-load-probes` on the server and
  `confirm=true` on the call; with `mode="colocated"` the pid must also be on
  the allowlist. It never touches the target.
- `process_cpu`, `hw_counters`, `snapshot_process_metrics`,
  `watch_process_stalls`, `diagnose_process` and `diagnose_process_cpu` read
  one process in depth (its threads' wait channels, kernel stacks and
  syscalls, its taskstats, and performance counters attached to it) and
  follow the same `--allow` list as `capture_snapshot`. An unlisted pid is
  refused with `diagnostics_refused` and a reason (`allowlist_empty`,
  `not_allowlisted`, `no_such_process`).
- The scan's memory conditions read the kernel stack (or `wchan`) only of
  threads in `D`, at most 64, and report only their wait category and three
  innermost frames.
- No tool changes kernel settings. Delay accounting is reported as `null`
  with how to enable it, never switched on.
- Host-wide context (load, steal, host PSI) is included by default because it
  is what the process is subject to and it describes no other process. On a
  shared host where that aggregate should not reach an agent, start the server
  with `--no-host-context`. The process's own cgroup is always reported.

### Command-line redaction

Command lines returned by `find_processes`, `scan_processes` and
`list_python_processes`, `cpu_colocation` and `group_cpu` pass through a
best-effort filter (`src/heapspy_mcp/system/redact.py`) before truncation,
so a cut never leaves part of a secret. Truncation keeps the start (the
program and the leading arguments); `cmdline_max_chars` sets the length per
call, up to `--max-cmdline-chars`. `group_by="cmdline_arg:N"` and
`cmdline_regex` read the redacted command line too. It replaces with
`<redacted>`: values of options whose name looks secret (`--password=x`,
`--token x`, `--api-key`, `--client-secret`), `NAME=value` pairs with a secret
name (`*_KEY`, `*_TOKEN`, `*_SECRET`, `*PASSWORD*`), `Authorization:` and
similar header values, the password (or a lone token) in `scheme://user:pass@`
URLs, and secret-named query parameters. It does not make a command line safe
to publish.

### What works where

| Placement | Works | Does not |
|---|---|---|
| Local machine | Everything for your own processes; root (or `CAP_SYS_PTRACE`) for other users' threads' wait channels, syscalls and `/proc/<pid>/io` | `hw_counters` on other users' processes without `CAP_PERFMON` |
| Pod without host access | Processes in the pod's own PID namespace, the pod's cgroup, host `/proc/stat` and PSI (not namespaced) | Other pods' processes; `perf_event_open` without `PERFMON` |
| Sidecar with `shareProcessNamespace: true` | The other containers of the pod, with `SYS_PTRACE` | Other pods |
| Node agent (DaemonSet, `hostPID: true`) | Every process on the node, its cgroup, pod UID and container id; add `--cgroup-root` with the host cgroupfs mounted if the container's cgroup namespace hides it | Pod names (only UIDs) |
| VM without a PMU | Everything except hardware counters, which are `null` with `no PMU exposed (common in VMs)`; `cache_or_memory_bound_contention` is then `insufficient_signals`. `hw_counters` returns `software_fallback`, and `cache_sensitivity_probe` (opt-in) measures this host's cache sensitivity with a synthetic workload | IPC, LLC misses |

Kernel features that change what is available, each reported as a reason when
missing: delay accounting (`kernel.task_delayacct`) for iowait and the
stall delays, `CONFIG_TASKSTATS` and `CAP_NET_ADMIN` for the taskstats
delays, `CONFIG_SCHEDSTATS` for `/proc/schedstat`, `CONFIG_SCHED_DEBUG` for
`se.nr_migrations` and `se.slice` (6.6+ for `se.slice`), PSI (`psi=1`) for pressure files, root
(`CAP_SYS_ADMIN`) and `CONFIG_STACKTRACE` for `/proc/<tid>/stack`, and a v2
hierarchy (unified or hybrid) for cgroup memory pressure and the v2 freezer.

See [`deploy/kubernetes/daemonset.yaml`](deploy/kubernetes/daemonset.yaml) for
a node-agent example.

## CPU tools (py-spy)

py-spy reads a running CPython process's stacks from outside, the way heapspy
reads its heap. The CPU tools answer:

| Tool | Question |
|---|---|
| `cpu_stacks(pid)` | What is every thread doing right now? |
| `cpu_top(pid, duration_s=5)` | Where are the running threads, and how much CPU did the process really use? |
| `cpu_profile(pid, duration_s)` | Where does the time go over a longer window, and is the GIL contended? |
| `cpu_flamegraph(pid, duration_s)` | Write a flame graph file for the user to open. |

Use the heap tools when memory grows or RSS is high. Use the CPU tools when a
process is slow, hot, hung, or a request takes too long. If both, start with
`cpu_top`: it is cheap, and it shows whether the time goes into allocation or
garbage collection, which would point back at the heap.

### What the numbers mean

A **sample** is one thread at one sampling tick. At 100 Hz for 10 s, a process
with 4 Python threads gives up to 4000 samples when idle threads are counted.

Every percentage in the profile tools is `pct_of_samples`: a share of the
samples in the stated sampling mode. It is not a share of machine CPU, nor of
one core. "40% of on-CPU samples in `parse()`" means `parse()` was on the
stack in 40% of the times a thread was seen running. Each result has
`sampling.mode` and `sampling.description`:

- `on_cpu` (default): only threads py-spy flags as active.
- `all` (`include_idle=true`): every Python thread at every tick, including
  sleeping, blocked and GIL-waiting threads. Use it to see where threads wait.
- `gil` / `gil_on_cpu` (`gil_only=true`): only the thread holding the GIL.

For real CPU time use `cpu_top`: `cpu_seconds_user` and `cpu_seconds_system`
are the kernel's accounting for the target process over the profile window
(psutil `cpu_times` deltas), with `wall_seconds` and `cpu_pct_of_one_core`.
`iowait_seconds` is the time the process spent waiting for block I/O,
from Linux delay accounting. Delay accounting is off by default on recent
kernels (`kernel.task_delayacct=0`); then `iowait_seconds` is `null` and
`iowait_note` says how to turn it on (`sysctl kernel.task_delayacct=1`, or boot
with `delayacct`). It is never reported as 0 when it was not measured.

### Idle, GIL and OS thread state

Three different signals, from three sources:

- **py-spy's active/idle flag.** On Linux py-spy calls a thread active when its
  OS thread is running, or when it holds the GIL, and idle otherwise; it also
  treats a few known wait functions (`threading` waits, `selectors.select`) as
  idle. A thread that waits for the GIL sleeps on a condition variable, so py-spy
  counts it as **idle**. py-spy alone cannot tell "waiting for the GIL" from
  "sleeping".
- **GIL ownership.** `owns_gil` in `cpu_stacks`, and the `--gil` pass in the
  profile tools. Only one thread holds the GIL at a time (on a GIL build).
- **OS thread state**, read by the server from `/proc/<pid>/task/<tid>/stat`
  (Linux only): `running/runnable` (R), `sleeping (blocked in a syscall)` (S),
  `disk wait (uninterruptible)` (D), `stopped` (T), `zombie` (Z). For a sleeping
  thread `cpu_stacks` adds `kernel_wait` (from `wchan`, for example
  `hrtimer_nanosleep`, `unix_stream_read_generic`, `futex_do_wait`) and
  `syscall` (from `/proc/.../syscall`, for example `clock_nanosleep`,
  `recvfrom`, `futex`, `fsync`), when the kernel exposes them.

`cpu_profile` and `cpu_top` combine these:

- `split`: the profile is recorded by three py-spy passes over the same window,
  run side by side with `--nonblocking`: all threads (`--idle`), active threads
  only, and the GIL holder (`--gil --idle`). `split` gives `on_cpu`, `idle`,
  `gil_held` and `on_cpu_without_gil`, each with `samples` and `pct_of_samples`
  of the all-threads samples. The passes sample independently, so the parts are
  estimates and need not add up exactly. `on_cpu_without_gil` is time running C
  code that released the GIL (I/O, compression, numpy), or spinning before
  taking it.
- `os_thread_states`: the server samples `/proc/<pid>/task/*/stat` at 20 Hz
  during the profile and reports `running`, `sleeping_futex` (asleep in
  `futex()`: waiting for the GIL, a lock, a queue or a condition),
  `sleeping_other` (sleep, socket read, poll), `disk_wait` and `stopped`, as
  percentages of thread-state samples. This covers all threads of the process,
  including threads without Python frames.
- `gil`: `held_pct_of_ticks` (in how many ticks some thread held the GIL),
  `threads_that_held_gil`, `futex_waiters_avg`, and `contention`: `likely` when
  the GIL was held in at least 80% of ticks, at least two threads took turns
  holding it, and on average at least half a thread was asleep on a futex.
  Futex waits include ordinary lock and queue waits, so the explanation says to
  confirm with `cpu_stacks` that the waiting threads are in plain Python code.
- `threads`: per thread, the samples in each pass and its OS state shares.

With `native=true` py-spy needs blocking reads, which cannot run side by side,
so `split` and `gil` are `null` and a note says why.

### `cpu_stacks(pid, include_idle=true, native=false, nonblocking=true, max_threads=50, max_frames=20)`

Runs `py-spy dump --json`. Per thread: `thread_id` (OS thread id),
`python_thread_id`, `name`, `active`, `owns_gil`, `os_state` (`code`, `label`,
and for a sleeping thread `kernel_wait` and `syscall`), and `frames`, innermost
first, each with `function`, `location` (`file:line` as py-spy shortens it),
`file`, `line`, `module` and `native`. `module` is derived from the shortened
path (`app/cache.py` gives `app.cache`) and is a guess for scripts run by path.
`summary` counts threads, active and idle threads, the GIL holder and OS states.
At most 200 threads and 100 frames per thread are returned.

`nonblocking=true` (the default) passes `--nonblocking`: py-spy reads the
target's memory while it runs. The target is never paused, but a stack can be
torn or wrong if the thread moved while it was read. `nonblocking=false` pauses
the process for the few milliseconds of the read and gives a consistent
snapshot; use it when a stack looks impossible. `native=true` also unwinds
C, C++ and Cython frames and always pauses the process (py-spy cannot read
native stacks without pausing). The OS state is read right after py-spy's
sample, not atomically with it, so a thread can show `active: false` with state
`R` when it woke up in between.

### `cpu_profile(pid, duration_s=10, rate_hz=100, include_idle=false, gil_only=false, subprocesses=false, native=false, top=15, top_stacks=5, save_folded=false)`

Runs `py-spy record --format raw --threads --nonblocking` into a private
temporary directory, parses the folded stacks in Python, and returns:

- `sampling` (mode, description, flags, number of py-spy passes) and
  `samples` (`total` in this mode, `ticks_estimate`, `py_spy_errors`).
- `top_self`: functions ranked by self samples (the function was the innermost
  frame). `top_total`: ranked by total samples (the function was anywhere on the
  stack; recursion counts once). Each row has `function`, `module`, `location`
  (the most sampled line), `samples` and `pct_of_samples`.
- `top_stacks`: whole stacks, innermost first, with the threads that ran them.
- `split`, `os_thread_states`, `gil` and `threads`, described above.
- `folded_path`: with `save_folded=true`, the folded file is kept in the
  snapshot directory (`cpu-xxxxxxxx.folded`, mode 0700 directory) for
  speedscope, `flamegraph.pl` or `inferno-flamegraph`. Otherwise `null`. With
  `subprocesses=true` the file's process frames contain command lines; the
  file stays on the local disk and its contents are never returned.

`duration_s` is a whole number, clamped to 1 .. `--cpu-max-duration` (default
30). `rate_hz` is clamped to 1 .. 1000. Lists are capped at 50 rows and stacks
at 30 frames.

### `cpu_top(pid, duration_s=5, top=10)`

An on-CPU profile like `cpu_profile`, summarised as `top_functions` (by self
samples), next to `wall_seconds`, `cpu_seconds_user`, `cpu_seconds_system`,
`iowait_seconds` and `cpu_pct_of_one_core` for the same window, plus `split`,
`gil` and `os_thread_states`. Compare the two: `pct_of_samples` says where the
running threads were; CPU seconds say how much running there was. A process
whose CPU seconds are near zero is waiting, not computing.

### `cpu_flamegraph(pid, duration_s=10, format="speedscope", include_idle=false, rate_hz=100)`

Runs `py-spy record --format speedscope` (or `flamegraph` for an SVG) into the
snapshot directory and returns `path`, size and sample count. The file's
contents are never returned through MCP: give the path to the user, who opens it
in https://www.speedscope.app or a browser.

### Errors

Errors use the same JSON shape as the heap tools. Codes: `attach_refused` (with
`reason`: `attach_disabled`, `not_allowlisted`, `invalid_argument`,
`no_such_process`, `process_inaccessible`), `py_spy_not_found`,
`permission_denied`, `no_such_process`, `not_python` (no CPython interpreter
found in the process), `timeout`, `bad_output`, `invalid_argument` and
`py_spy_failed`.

### Permissions

py-spy reads another process's memory, which needs the same rights as a
debugger. On Linux, any one of:

- run the server as the target's user with `kernel.yama.ptrace_scope` at 0
  (`sudo sysctl kernel.yama.ptrace_scope=0`); the Ubuntu default of 1 only lets
  a process trace its own children;
- give py-spy `CAP_SYS_PTRACE`: `sudo setcap cap_sys_ptrace+ep "$(which py-spy)"`;
- run the server as root.

In Docker, start the target's container with `--cap-add SYS_PTRACE` (and run
py-spy inside it, or share the PID namespace). In Kubernetes add `SYS_PTRACE` to
`securityContext.capabilities.add`. On macOS py-spy needs root. A refusal comes
back as `permission_denied` with these steps in `hint`; only the user can apply
them.

### Safety of the CPU tools

- **Allowlist.** The CPU tools use the same allowlist as `capture_snapshot`
  (`--allow` / `HEAPSPY_MCP_ALLOW`). It is empty by default, so the CPU tools are
  disabled until the user configures it. A pid that does not match is refused
  with `not_allowlisted`; the server's own pid is refused even with `--allow '*'`.
  With `subprocesses=true`, children of an allowlisted process are sampled too.
- **No `confirm` flag.** `capture_snapshot` needs `confirm=true` because it
  pauses the target and writes a memory image to disk. The CPU tools only read
  stacks and, by default, use `--nonblocking`, so the target is never paused;
  the allowlist is the user's consent to that. `nonblocking=false` and
  `native=true` pause the target for each sample (milliseconds) and are opt-in
  per call.
- **Logging.** Every attach is logged to stderr before py-spy runs: tool, pid,
  process name, the allowlist entry that matched, and the sampling settings;
  profiles also log when they finish.
- **No local variables.** py-spy's `--locals` prints the values of local
  variables. The server never passes it and drops any `locals` field from
  py-spy's output: values can hold credentials, tokens and user data, the same
  reason heapspy does not record string contents by default.
- **Timeouts and cleanup.** `py-spy dump` has a 30 s timeout; `py-spy record`
  gets `duration_s` plus 30 s. On a timeout or failure every py-spy process
  still running is killed and its output file deleted, and the temporary
  directory is always removed.
- **Bounded output.** Thread, frame, row and stack counts are capped as listed
  above; flame graphs are returned as paths only.

### CPU resource and prompt

- Resource `heapspy://workflow/cpu`: when to use CPU tools rather than heap
  tools, the CPU loop, and what the numbers mean.
- Prompt `investigate_cpu(target?)`: the same text, as a starting prompt.
- The server instructions sent at initialisation include the same text after
  the heap workflow.

## Development

```sh
cd python/heapspy-mcp
uv run pytest
uv run ruff check
uv run ruff format --check
```

The tests use `tests/fake_heapspy.py` as the `heapspy` binary. It prints the
documents in `docs/examples/` adjusted to its arguments, and can simulate
error documents, a wrong schema version, timeouts, output that is not JSON,
usage errors and oversized reports. The dump report and the analyze report
with allocations come from heapspy branches not merged here yet; copies are in
`tests/fixtures/heapspy/`, whose README names the source commits. The fake also
simulates a heapspy without `dump --json`, `--allocations` and `flamegraph`,
each dump error code, a profile that does not match the snapshot, and a dump
killed while writing.

#### Contract tests against a real heapspy

`tests/test_contract_real.py` runs a real heapspy binary. It is off by
default:

```sh
HEAPSPY_MCP_REAL_BIN=/path/to/heapspy uv run pytest tests/test_contract_real.py
```

It reads `--help` to see whether the binary has `dump --json` and
`--allocations`, and checks the matching behaviour (or the fallback). It
captures a running `/usr/bin/python3.12` (`HEAPSPY_MCP_REAL_PYTHON` overrides
it) and checks the capture, `heap_summary`, `type_detail` and `map_to_source`
on a `__main__` class of a script without a main guard; dumps `sleep`
(`not_python`), a missing pid, a garbage core and a missing core; runs the
allocation tools against a synthetic snapshot and profile in
`tests/fixtures/heapspy/` (the README there has the program that writes them);
and checks that the real dump and allocation reports have every field of the
copied examples. Live captures need ptrace rights and are skipped without
them. Child processes are killed on every path.

No heapspy branch has both features yet, so both were used. Results on the
development VM (Linux 6.18, CPython 3.12.3, run as root):

| binary | passed | skipped |
|---|---|---|
| `feature/dump-json-errors` at 2ae7ec2 (dump --json, no --allocations) | 8 | 1 (allocation report fields) |
| `feature/alloc-join` at b04cc2f (--allocations, no dump --json) | 8 | 1 (dump report fields) |

With the first, captures were parsed from the dump report, `sleep` gave
`not_python`, and the allocation tools gave `allocations_unsupported`. With
the second, captures fell back to the stderr line, dump failures were
`cli_failed`, and the allocation tools returned the numbers the synthetic
files were built with (203 linked objects, 3 of them through exact links).
The `feature/dump-json-errors` head (40c85ed) differs from 2ae7ec2 only in
`tests/e2e/`.

The process diagnostics have parser tests against fixture files from 5.x and
6.x kernels (`tests/fixtures/system/`) and fake cgroupfs trees, mocked
`perf_event_open` paths (success, no PMU, `EACCES`, multiplexing), and live
tests on Linux that start real processes: a CPU-bound loop, more spinning
processes than cores, a lock ping-pong, a process in a cgroup with a 0.2-core
quota (root only; skipped otherwise), and, for the search tools, a CPU burner,
a process with growing RSS, one holding thousands of fds, a parent of zombies
and a lock ping-pong. `uv run pytest -s tests/test_system_live.py` prints the
numbers each scenario measured.

The memory-stall tools have classifier tests against kernel stacks
(`tests/fixtures/system/kernel_stacks.txt`: stacks captured on 6.18, and call
chains of 4.4 to 6.6 for waits the development VM cannot produce, such as
swap-in and `memory.high` throttling), parser tests for v2 and v1
`memory.*`, `vmstat`, `smaps_rollup` and freezer files, fake `/proc` trees for
the sampler, and live tests in `tests/test_stalls_live.py` (root only). They
create their own cgroups and a 256 MiB temporary file, and remove both on
every path. `uv run pytest -s tests/test_stalls_live.py` prints the numbers.

The contention tools have parser tests for `/proc/<pid>/task/<tid>/stat`
field 39, schedstat `pcount`, `/proc/<pid>/sched` from a 6.18 kernel (with
`se.slice`) and a 5.15-style one (without), CPU lists and a fake sysfs cache
tree (`tests/test_contention_parsers.py`); the co-location math on a fake
`/proc` (`tests/test_colocation.py`); the group aggregation
(`tests/test_groups.py`); the fit on exact and degenerate points
(`tests/test_slice_fit.py`); the starvation gating and `short_timeslices`,
including a regression test on the signals of the starved worker from the
blind test (`tests/test_diagnose_contention.py`); and the tools through the
server (`tests/test_contention_tools.py`). The live re-test of the blind
test is described at the end of this file.

The thrash in those tests is file-backed, since the development VM has no
swap: a process maps the file and reads one byte of each page in a fixed
random order inside a memory cgroup limited to 64 MiB. Three things were
needed to make eviction happen. The file's page cache is dropped first with
`POSIX_FADV_DONTNEED`, or its pages stay charged to the cgroup that wrote
them and the scanner never faults. The order is random, because a sequential
scan is served by readahead: most faults then find the page already in cache
(minor faults), and the major-fault count stays low even while the cgroup
refaults gigabytes. And to see threads stop for seconds, a cgroup read limit
on the block device (16 MiB/s) stands in for a slow disk; the VM's disk is
served from the host's cache at more than 1 GB/s, so without it each refault
finishes in microseconds and the thread stays on a CPU, mostly in the
kernel. Results on the development VM (Linux 6.18, 4 cores, cgroup v1 memory
controller with v2 pressure, 5 s windows):

| Scenario | Major faults/s | Refaults/s (cgroup) | Wait categories | Longest no-progress | `diagnose_process` |
|---|---|---|---|---|---|
| (a1) 64 MiB limit | 8700 to 9900 | 250000 | `page_fault_io` 0.50 to 0.62, `running` the rest; `system_share` 0.98; cgroup memory PSI `full` 10 to 17% | none (CPU kept advancing, in the kernel) | `memory_thrashing` first, score 9 |
| (a2) 64 MiB limit and 16 MiB/s reads | 138 | 500 | `page_fault_io` 1.0, state `D` | 1.6 to 1.9 s, `page_fault_io`, stack `folio_wait_bit_common < filemap_fault < __do_fault < ...`; 0.93 to 0.97 s/s in memory waits | `memory_thrashing` first, score 7; `io_pressure` 1 |
| (b) same scan, no limit, file cached | 0 | 0 | `running` 1.0 | none | `memory_thrashing` not indicated (-5); `spin_waiting_or_compute` first |
| (c) CPU-bound loop | 0 | 0 | `running` 1.0 | none | `memory_thrashing` not indicated (-5) |
| (d1) `SIGSTOP` | 0 | 0 | `stopped_or_frozen` 1.0 | the whole window, `stopped_or_frozen` | `deliberately_frozen` true, all moot, `memory_thrashing` -8 |
| (d2) v1 freezer `FROZEN`, v2 `cgroup.freeze=1` | 0 | 0 | `stopped_or_frozen` 1.0 (v1: state `D` in `__refrigerator`) | the whole window, `stopped_or_frozen` | `deliberately_frozen` true, `frozen_by_cgroup_freezer` |

With (a1) running, `scan_processes` put it first on `major_faults_per_s`
(about 9900/s), listed it under `major_faults_whole_window` (lowest interval
9000/s) and its cgroup under `cgroup_memory_pressure_full` (`full` 11 to
12% of the window); `find_processes(sort_by="major_faults_per_s")` put it
first. With (a2), `disk_wait_in_memory_management` listed its thread as
`page_fault_io` from its stack. A frozen cgroup and a stopped process
appeared under `frozen_or_stopped`, and the v1-frozen task in `D` was not
listed as a memory wait.

The CPU tool tests use `tests/fake_pyspy.py` as `py-spy` in the same way. The
integration tests in `tests/test_cpu_integration.py` run the real py-spy against
real Python processes (`tests/cpu_victims.py`: a CPU-bound loop, threads blocked
in `time.sleep` and a socket read, four threads contending for the GIL, and an
`os.fsync` loop). They are off by default:

```sh
uv sync --extra pyspy
HEAPSPY_MCP_PYSPY_TESTS=1 uv run pytest -m pyspy_integration
```

They need ptrace rights (see Permissions) and use `/usr/bin/python3.12`,
`/usr/local/bin/python3.13` and `python3.14` where present; set
`HEAPSPY_MCP_TEST_PYTHONS` (separated by `:`) to choose others.

Results on the development VM (Linux 6.18, py-spy 0.4.3, CPython 3.12.3,
3.13.18 and 3.14.6, 3 s windows, two runs), which matched within noise across
the three versions:

- CPU-bound loop: `burn_cpu` had 100% of on-CPU samples; on-CPU was 100% of
  all samples; `cpu_pct_of_one_core` 99 to 100.
- Threads in `time.sleep` and `socket.recv`: both idle, state S, with
  `kernel_wait` `hrtimer_nanosleep` / `unix_stream_read_generic` and syscall
  `clock_nanosleep` / `recvfrom`; 0.00 CPU seconds and 0 on-CPU samples in 3 s.
- Four threads contending for the GIL: GIL held in 98 to 100% of ticks by 4
  threads in turn, 3.6 to 4.0 threads asleep on a futex on average (the three
  waiting workers and the main thread in `Event.wait`), `contention: likely`;
  on-CPU 20 to 27% and idle 73 to 80% of all samples; `cpu_pct_of_one_core`
  93 to 100, not 400.
- `os.fsync` loop: these numbers are noisy and depend on the VM's disk. The
  thread was in disk wait (D) in 37 to 75% of state samples and running in the
  rest; py-spy flagged 31 to 68% of samples on-CPU, roughly tracking the
  running share; `cpu_seconds_system` was 0.2 to 0.75 of about 3 s wall. Delay
  accounting is off on this VM, so `iowait_seconds` was `null`. In some runs
  `cpu_stacks` showed the thread idle with state R, because it woke up between
  py-spy's read and the `/proc` read.

### Live re-test of the CPU-contention blind test

`tests/test_contention_live.py` (root only, needs a C compiler and 4 CPUs)
re-runs the scenario of a blind test in which an agent, given only "CPU went
to 100% after capacity was halved, and context switches were high", found the
mechanism partly with scripts of its own. It builds `tests/contention_worker.c`
into a temporary directory and starts two services: `api`, 8 workers each
walking a 1.5 MiB working set at 6500 requests/s offered, and `edge`, 4
workers with a 16 KiB working set that sleep on short timers at 20000
requests/s offered. `cores(k)` pins every worker to CPUs 0 to k-1. Each check
takes medians over repeats. `uv run pytest -s tests/test_contention_live.py`
prints the numbers. On the development VM (Linux 6.18, 4 vCPUs, L2 2 MiB per
core, `se.slice` 2.1 ms, shared with other work), over five runs:

| Check | Result |
|---|---|
| `group_cpu` at 2 cores, api | `voluntary_share` 0.002 to 0.039; pooled `avg_timeslice_us` 170 to 327; `runnable_share` 1.0; non-voluntary switches 580 to 1150/s per worker, max/min across workers 1.1 to 33 |
| `group_cpu` at 2 cores, edge | `voluntary_share` 0.998 to 1.0; `avg_timeslice_us` 22 to 28; 3900 to 4600 voluntary switches/s per worker; `runnable_share` 0.67 to 0.73 |
| `cpu_colocation` at 2 cores, api as targets | placement 4+4 api workers in most runs, 5+3 in one; edge split 2+2, 3+1 or 4+0; co-located wakeups 4000 to 17000/s on CPUs with edge workers, 9/s on the CPU without; short-sleeper share 0.998; `target_preemptions_per_colocated_wakeup` 0.26 to 1.01; the top short sleepers are the edge workers |
| `fit_per_slice_cost` over 1, 2 and 4 cores (two orders per run, four runs) | `fixed_cost_per_timeslice_us` 39 to 79 (run medians 70, 71, 43, 56); per request without slicing 25 to 44 us; R squared 0.63 to 0.996 |
| `diagnose_process` on a starved api worker at 2 cores | `cpu_cores_used` 0.17 to 0.29 with `runqueue_wait_ratio` 2.5 to 4.8 and `runnable_share` 1.0; `runqueue_contention` 7, `short_timeslices` 6 (average slice 146 to 168 us of 2100, 5200 to 14000 co-located wakeups/s, short-sleeper share 0.997 to 0.998), `spin_waiting_or_compute` partly consistent with no idle evidence against |
| `cache_sensitivity_probe`, synthetic, CPU 3 | 16 KiB: slowdown 1.09 to 1.12, 11 to 14 us extra per slice; 1.5 MiB: slowdown 1.6 to 5.2, 50 to 111 us extra per slice, at 130 to 139 us slices |

The blind test measured about 70 us of refill per api slice. The fit and the
probe give the same order of magnitude; runs with other load on the VM give
the lower fitted costs and the lower R squared.
