Metadata-Version: 2.4
Name: mantissa-interpret
Version: 0.1.0
Summary: CNN interpretability on the mantissa C engine: occlusion, saliency and Grad-CAM
Author: Tekin Ertekin
License: MIT
Project-URL: Homepage, https://github.com/tekinertekin/mantissa-interpret
Project-URL: Base, https://github.com/tekinertekin/mantissa-cnn
Project-URL: Engine, https://github.com/tekinertekin/mantissa
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: numpy>=1.20
Requires-Dist: mantissa-cnn>=0.2.2
Provides-Extra: viz
Requires-Dist: matplotlib>=3.5; extra == "viz"
Provides-Extra: bench
Requires-Dist: matplotlib>=3.5; extra == "bench"
Provides-Extra: dev
Requires-Dist: pytest>=7; extra == "dev"
Dynamic: license-file

# mantissa-interpret

![License](https://img.shields.io/badge/license-MIT-blue.svg)
![Python](https://img.shields.io/badge/python-3.9%2B-3776AB.svg)
[![Base](https://img.shields.io/badge/base-mantissa--cnn-4B8BBE.svg)](https://github.com/tekinertekin/mantissa-cnn)
[![Engine](https://img.shields.io/badge/engine-mantissa-00599C.svg)](https://github.com/tekinertekin/mantissa)

**Seeing what a CNN looks at.** A trained classifier gives you a label; it does
not tell you *why*. `mantissa-interpret` answers that with three classic
attribution methods that turn a prediction into a **heatmap over the input** —
bright where the pixels mattered for the class, dark where they did not.

Everything runs on a fitted [`mantissa_cnn.Sequential`](https://github.com/tekinertekin/mantissa-cnn)
model, using the model's **own forward and backward passes through the mantissa
C engine** — the same engine it was trained on. There is no PyTorch/TensorFlow
dependency: the heatmaps come out of the same low-precision C kernels that
produced the prediction.

## The mantissa family

Part of the **mantissa** family: a low-precision engine written in C, with
small Python packages built on top. Each package sits under the one it depends
on — ⭐ marks where you are, and every other name links to its repo.

- [mantissa](https://github.com/tekinertekin/mantissa) — low-precision neural-network engine in C (the core)
  - [mantissa-perceptron](https://github.com/tekinertekin/mantissa-perceptron) — perceptron & ADALINE, the linear classics
  - [mantissa-nn](https://github.com/tekinertekin/mantissa-nn) — shared neural-net primitives (layers, engine binding)
    - [mantissa-cnn](https://github.com/tekinertekin/mantissa-cnn) — convolutional networks for images
      - [mantissa-auto-encoder](https://github.com/tekinertekin/mantissa-auto-encoder) — autoencoders for denoising & super-resolution
      - ⭐ **mantissa-interpret** — CNN interpretability (occlusion, saliency, Grad-CAM) *(you are here)*
      - [mantissa-embed](https://github.com/tekinertekin/mantissa-embed) — CNN metric learning (image embeddings for similarity & retrieval)
    - [mantissa-mlp](https://github.com/tekinertekin/mantissa-mlp) — multilayer perceptrons, fully-connected nets


## New to interpretability?

A CNN classifier turns an image into a label — but the label alone hides *why*.
**Interpretability** (here, *attribution*) recovers the missing reason: it marks
**which parts of the input the model actually used** for a given class, as a
heatmap laid over the image. That matters because a model can be right for the
wrong reason — keying on a watermark, a background, or a dataset artifact
instead of the object — and a bare accuracy number will never tell you. A
heatmap will.

The most direct way to see this is to **hide part of the image and watch the
prediction**. If covering a region makes the class probability collapse, that
region carried the evidence; if nothing changes, it did not:

![covering the important strokes of an 8 drops P(8) from 1.00 to 0.00; covering empty space leaves it at 1.00](https://raw.githubusercontent.com/tekinertekin/mantissa-interpret/main/assets/how_occlusion_works.png)

That single experiment *is* the first method. The three methods here are three
answers to "what did the model use?", trading detail for cost and clarity:

- **Occlusion** — exactly the picture above, done everywhere: slide a patch
  over the image and record how much each position, when hidden, drops the
  class probability. No math beyond running the model forward; the price is one
  forward pass per patch, and the resolution is patch-sized.
- **Saliency** — instead of hiding pixels, ask calculus which pixels *matter*:
  the gradient of the class score with respect to each input pixel. A large
  gradient means "nudging this pixel would move the score a lot." It is
  per-pixel sharp but noisy, and needs a single backward pass.
- **Grad-CAM** — work inside the network. At a convolutional layer each channel
  is a learned feature detector over a coarse grid; weight every channel by how
  much raising its activation would raise the target score (its gradient),
  add them up, and keep the positive part. The result is a smooth,
  **class-discriminative** region — ask about "3" vs "7" on the same image and
  the map moves — for one backward pass.

Occlusion and saliency answer *which pixels*; Grad-CAM answers *which region,
for this class*. Together they are the standard first tools for debugging a
model that is right for the wrong reason.

## Install

```sh
pip install mantissa-interpret
```

Pulls in `mantissa-cnn` (and transitively `mantissa-nn` + the `mantissa-core`
engine). For plotting the heatmaps, `pip install mantissa-interpret[viz]`.

## The three methods

| method | cost | granularity | class-discriminative | needs |
|---|---|---|---|---|
| `occlusion_map` | forward only, O(patches) | coarse (patch) | yes | — |
| `saliency_map` | one backward pass | per-pixel (noisy) | weakly | input gradient |
| `grad_cam` | one backward pass | per-region (smooth) | **yes** | activations + their gradient at a conv layer |

Each takes a fitted model and a single `(C, H, W)` image and returns a 2-D
heatmap normalized to `[0, 1]`, aligned to the input.

### Occlusion — `occlusion_map`

Slide a `patch`×`patch` window (filled with `fill`) across the image; at each
position, hide that window and re-run the model. If the target class
probability drops a lot, those pixels were important. The heatmap is the
per-pixel average drop (overlapping windows blend), normalized to `[0, 1]`.

```python
from mantissa_interpret import occlusion_map
heat = occlusion_map(net, image, target_class=7, patch=7, stride=3)
```

Only forward passes — model-agnostic and dead simple, but coarse (patch-sized)
and its cost scales with the number of windows. All occluded copies are run in
a single batched forward pass.

### Saliency — `saliency_map`

The first-order sensitivity of the class score to each pixel is just its
gradient. One forward pass primes the layers, we seed the gradient of the
target *logit* (a one-hot vector), and backpropagate to the input — the same
`backward` the model trains with, carried one step past the first layer's
weights down to the pixels. The map is `|gradient|` reduced over channels.

```python
from mantissa_interpret import saliency_map
heat = saliency_map(net, image, target_class=7)
```

Per-pixel and cheap (one backward pass), but high-frequency and noisy, and only
weakly class-discriminative — good for "which strokes", not "which region".

### Grad-CAM — `grad_cam`

At a convolutional layer, each channel is a learned feature detector over a
coarse grid. Grad-CAM weights every channel by the average gradient of the
target logit w.r.t. that channel's activation (how much "more of this feature"
raises the class score), sums the channels with those weights, keeps the
positive part, and upsamples to the image. We capture the layer's activation on
the forward pass and backpropagate the logit down to that same layer for the
gradient — then combine.

```python
from mantissa_interpret import grad_cam
heat = grad_cam(net, image, target_class=7)      # last Conv2D by default
```

Coarse (conv-grid resolution) but smooth and genuinely **class-discriminative**
— asking about different classes on the same image yields different maps.
`target_layer` selects which conv layer to read (default: the last one).

## Results

A LeNet-5 trained on an 8k-image MNIST subset with the mantissa C engine
(3 epochs, **96% test accuracy**), then explained on held-out digits. Reproduce
with `python examples/heatmaps_demo.py`.

![occlusion, saliency and Grad-CAM heatmaps on MNIST digits](https://raw.githubusercontent.com/tekinertekin/mantissa-interpret/main/assets/heatmaps.png)

Reading across a row, the three methods agree on the digit but differ exactly
as their math predicts:

- **occlusion** — coarse, patch-sized blobs on the strokes whose removal costs
  the class the most;
- **saliency** — the finest detail, tracing individual strokes, but visibly
  noisy (the raw per-pixel gradient);
- **Grad-CAM** — a smooth region over the class-defining strokes (the 7's
  diagonal, the 2's base, the 9's loop), with the least clutter.

Grad-CAM is also **class-discriminative** — ask it about a different target
class on the *same* image and the map moves:

![Grad-CAM for two different target classes on the same 7](https://raw.githubusercontent.com/tekinertekin/mantissa-interpret/main/assets/gradcam_classes.png)

All of this — feature maps, gradients, the optimization-free attribution — comes
out of the same low-precision C kernels the model was trained on.

## License

MIT — Tekin Ertekin.
