Metadata-Version: 2.4
Name: llm-inspector
Version: 0.6.0
Summary: The htop for LLM inference. Measured. Not guessed.
Project-URL: Homepage, https://github.com/helasaoudi/llm-inspector
Project-URL: Repository, https://github.com/helasaoudi/llm-inspector
Project-URL: Bug Tracker, https://github.com/helasaoudi/llm-inspector/issues
Author-email: Hela Saoudi <helasaoudi024@gmail.com>
License: MIT
License-File: LICENSE
Keywords: cli,gpu,inference,llm,monitoring,observability,ollama,vllm
Classifier: Development Status :: 3 - Alpha
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: System :: Monitoring
Requires-Python: >=3.11
Requires-Dist: nvidia-ml-py>=12.535
Requires-Dist: psutil>=5.9
Requires-Dist: pydantic>=2.7
Requires-Dist: python-dateutil>=2.9
Requires-Dist: requests>=2.31
Requires-Dist: rich>=13.7
Requires-Dist: typer>=0.12
Provides-Extra: dev
Requires-Dist: mypy>=1.10; extra == 'dev'
Requires-Dist: pytest-cov>=5.0; extra == 'dev'
Requires-Dist: pytest-mock>=3.14; extra == 'dev'
Requires-Dist: pytest>=8.2; extra == 'dev'
Requires-Dist: ruff>=0.4; extra == 'dev'
Provides-Extra: torch
Requires-Dist: torch>=2.2; extra == 'torch'
Description-Content-Type: text/markdown

<div align="center">

# LLM Inspector

**The `htop` for LLM inference.**

*Measured. Not guessed.*

[![Python](https://img.shields.io/badge/python-3.11%2B-blue?logo=python&logoColor=white)](https://python.org)
[![License: MIT](https://img.shields.io/badge/license-MIT-green)](LICENSE)
[![Platforms](https://img.shields.io/badge/platform-macOS%20%7C%20Linux%20%7C%20GPU%20servers-lightgrey)](#)

**Inspect → Understand → Optimize**

</div>

---

LLM Inspector inspects live inference processes and shows exactly how GPU memory is being used, what model is running, how the runtime is configured, and where every reported value comes from.

Unlike traditional monitoring tools, it doesn't stop at inspection. It also analyzes the running workload and projects optimization opportunities—starting with quantization—to help you understand how different strategies would impact GPU memory **before** making any changes.

---

## Example

<div align="center">

![LLM Inspector demo](https://raw.githubusercontent.com/helasaoudi/llm-inspector/main/docs/assets/llm-inspector-demo.gif)

</div>

---

## Why?

Existing tools tell you that your GPU is using 18 GB.

LLM Inspector tells you **why**:

```text
Weights       7 GB
KV Cache      8 GB
Workspace     1 GB
Other         2 GB
```

Then it tells you what would happen if you optimized:

```text
FP8 would save ~3 GB of weights.
AWQ would save ~5 GB of weights.
Weight quantization won't fix an 8 GB KV Cache bottleneck.
```

That is the difference between a GPU monitor and an inference advisor.

---

## Embedded goes deeper

Most tools stop here:

```text
GPU
└── Process A
    └── 17.3 GB
```

LLM Inspector goes one level deeper:

```text
GPU
└── Process A
    ├── Weights
    ├── KV Cache
    ├── Workspace
    ├── Activations
    └── Optimization Analysis (Projected)
```

That bridge—**system observability + model internals**, with an `htop`-style CLI—is the innovation. Deep metrics come from optional embedded `attach()` inside the inference process (see the install guide).

---

## Measured vs Projected

| Section | Kind |
|---------|------|
| Process, Hardware, Model, Memory, Runtime | **Measured** from live sources (or `Unavailable` with a reason) |
| Optimization Analysis | **Projected** from measured inputs — never mutates the model |

```bash
llminspect inspect <pid> --verbose   # show provenance for every field
```

---

## Install

Works on any NVIDIA GPU machine — laptop, workstation, cloud VM, bare-metal server, or DGX. Docker is optional.

```bash
pip install llm-inspector
```

For embedded deep metrics (Weights / KV / Activations) in the same Python env as the model:

```bash
pip install "llm-inspector[torch]"
```

From source (contributors):

```bash
git clone https://github.com/helasaoudi/llm-inspector
cd llm-inspector
python -m venv .venv && source .venv/bin/activate
pip install -e ".[torch]"
```

---

## Try it

```bash
llminspect ps
llminspect inspect <pid>
llminspect inspect <pid> --verbose
```

With Ollama on the host (no Docker):

```bash
ollama run llama3
llminspect inspect $(pgrep -f "ollama serve") --verbose
```

---

## Supports

Ollama · vLLM · HuggingFace Transformers · FastAPI · custom PyTorch  
macOS · Linux · any NVIDIA GPU server (including DGX)

---

## Documentation

| Guide | When to read it |
|-------|-----------------|
| **[Install & Integration](docs/INSTALL_GUIDE.md)** | Host + Docker install, `attach()`, vLLM plugin, troubleshooting |
| [DGX notes](docs/DGX_GUIDE.md) | Extra detail from a DGX Spark inference-service setup |

---

## License

MIT
