---
title: "Visualising Missingness"
source: The Turing Way
source_url: https://github.com/the-turing-way/the-turing-way
licence: CC-BY-4.0
domain: data-and-analysis
subdomain: turing-way
date_added: 2026-04-25
---

(pd-missing-data-visualising-missingness)=
# Visualising Missingness

Visualisation techniques can be incredibly helpful when trying to understand your data and any patterns of missingness. Different programming languages offer unique tools and packages to help with this. We will introduce *a few* visualisation tools for [python](#pd-missing-data-visualising-missingness-python) and for [R](#pd-missing-data-visualising-missingness-r). Lastly, we will use some of these tools to show simulated visualisations of the fictional dataset with all three types of data missingness, as introduced in {ref}`pd-missing-data-structures`. If you would like to try the below visualisations, please recreate the  example dataset (introduced in {ref}`pd-missing-data-structures`) in a file, which here we call "FictionalDataset.csv". 

```{note}
For simplicity we will be using all functions with their default parameters. However, all the functions presented have many input variables to allow for further customisation and tailoring. 
```

(pd-missing-data-visualising-missingness-plottypes)= 
## Plot Types Covered and Hyperlinks

1. Nullity Matrix ({ref}`Python<pd-missing-data-visualising-missingness-python1>`, {ref}`R <pd-missing-data-visualising-missingness-r1>`)
2. Nullity by column ({ref}`Python<pd-missing-data-visualising-missingness-python2>`, {ref}`R<pd-missing-data-visualising-missingness-r2>`)
3. Nullity Correlation ({ref}`Python<pd-missing-data-visualising-missingness-python3>`, {ref}`R<pd-missing-data-visualising-missingness-r3>`)
4. Beyond Pairwise Nullity Correlation ({ref}`Python<pd-missing-data-visualising-missingness-python4>`, {ref}`R<pd-missing-data-visualising-missingness-r4>`)


(pd-missing-data-visualising-missingness-python)=
## Python

The [missingno python package](https://github.com/ResidentMario/missingno) is a great tool for visualising missing data.
The main functions and their usage on the fictional dataset introduced in the previous subchapter are introduced below, where our dataset is in a dataframe format and called "df_fictional_dataset". 

```python
# Importing packages
import pandas as pd
import missingno as msno

# loading in dataset
df_fictional_dataset = pd.read_csv("FictionalDataset.csv", header = 0, index_col = 0)
```

(pd-missing-data-visualising-missingness-python1)=
### 1. Nullity Matrix 

```python
msno.matrix(df_fictional_dataset)
```

```{figure} ../../../figures/msno_matrix.png
---
height: 400px
name: msno-matrix
alt: >
  A nullity matrix plot generated using the `missingno` Python package, visualising missing data in our fictional dataset.
  Each column represents a variable (Age, Diastolic Blood Pressure, Systolic Blood Pressure, Blood Test Result, Motor Score and Cognitive Score), and each row corresponds to a data entry.
  Present values are shown as solid dark grey blocks, while missing values are shown as white gaps.
  On the right side, a vertical sparkline summarises the number of non-missing values in each row, allowing a quick visual comparison of how complete each row is.
---
Nullity matrix of our fictional dataset, produced via the missingno python package. 
```

Data entries with missing data are indicated by white, while all complete entries are shaded a dark grey. At the right of the plot there is a sparkline which summarises data completeness by row. In this instance, the 3rd row has the most missing data (only 2 complete entries), while the 4th and 5th rows are the most complete (no missing entries).


(pd-missing-data-visualising-missingness-python2)=
### 2. Nullity by Column

```python
msno.bar(df_fictional_dataset)
```

```{figure} ../../../figures/msno_bar.png
---
height: 400px
name: msno-bar
alt: >
  Bar plot showing the number of non-missing (complete) values per variable in a fictional dataset, created using the `missingno` Python package.
  Each bar represents a column in the dataset: Age, Diastolic Blood Pressure, Systolic Blood Pressure, Blood Test Result, Motor Score and Cognitive Score.
  The height of each bar corresponds to the proportion of complete entries (left y-axis), while absolute counts are labelled at the top of each bar (for example, Age and Motor Score both have all 8 values present). 
---
Barplot of the nullity per column of our fictional dataset, produced via the missingno python package. 
```

This is a simplification of the first visualisation method; a simple bar plot showing the number of complete values per variable.
The absolute number of values are shown on the top, while the left y-axis indicates the percentage of values that are not missing. 

(pd-missing-data-visualising-missingness-python3)=
### 3. Nullity Correlation

```python
msno.heatmap(df_fictional_dataset)
```

```{figure} ../../../figures/msno_heatmap.png
---
height: 400px
name: msno-heatmap
alt: >
  Heatmap showing pairwise correlations in missingness between variables in a fictional dataset, created using the missingno Python package. 
  Cells are colour-coded from dark blue (perfect positive correlation) to red (perfect negative correlation), with white representing no correlation.
  Systolic and Diastolic Blood Pressure have a perfect correlation of 1, indicating they are always missing together.
  Cognitive Score shows a moderate positive correlation (0.5) with both blood pressure variables, meaning it tends to be missing when they are.
  Blood Test Result shows weak negative correlations (−0.1) with the other variables.
  A vertical colour bar on the right maps the correlation values from −1 to 1.
---
Pairwise nullity correlation heatmap of our fictional dataset, produced via the missingno python package.
```
 Visualising what data is missing may not be enough to draw any inferences and understand the mechanism behind the missingness. A nullity correlation plot is one way to look into this. In this instance, we see that systolic blood pressure is always missing when diastolic blood pressure is also missing. Additionally, the cognitive score has a 50% chance of being missing if the blood pressure readings are missing. As the motor score and age don't have any missing values these are excluded. 


(pd-missing-data-visualising-missingness-python4)=
### 4. Beyond Pairwise Nullity Correlations, via a hierarchical clustering algorithm  

```python
msno.dendrogram(df_fictional_dataset)
```

```{figure} ../../../figures/msno_dendrogram.png
---
height: 400px
name: msno-dendrogram
alt: >
  Dendrogram plot showing hierarchical clustering of missingness patterns across variables in a fictional dataset, created using the missingno Python package.
  The horizontal axis lists variables: Motor Score, Age, Blood Test Result, Systolic Blood Pressure, Diastolic Blood Pressure, and Cognitive Score.
  Vertical branches reflect how similarly variables tend to be missing.
  Motor Score and Age are joined at a distance of zero, indicating identical missingness patterns.
  Diastolic and Systolic Blood Pressure are also tightly clustered, while Cognitive Score branches from them at a greater distance (~1.3), suggesting partial but not perfect correlation.
  Blood Test Result is more weakly linked, joining at a higher level.
---
Nullity correlation dendrogram of our fictional dataset, produced via the missingno python package.
```

In the dendrogram, the missingness of variables that are linked at a distance of zero, are fully correlated with each other.
Thus, we see the Motor Score and Age being connected at zero, and separately the two blood pressure measures being connected.
Variables that branch close to zero, predict each other's missingness well, but not perfectly.
For instance, cognitive score branches from the blood pressure variables at around 1.30.
This is because 2 out of 3 missing cognitive score values, also have missing blood pressure measurements. 

The strength of the above visualisations are a lot more evident in a much larger dataset.
You are encouraged to try them out on your own data to see for yourself! 

Additionally, although these visualisations are useful, in our case, a visualisation looking at the correlation between the values and the missingness of variables would give us more information. 


(pd-missing-data-visualising-missingness-r)=
## R

There are many readily available functions for visualising missingness in R, including many that look further into the mechanism of missingness.  A couple of libraries have useful functions, namely: [ggplot](https://ggplot2.tidyverse.org/), [visdat](https://cran.r-project.org/web/packages/visdat/readme/README.html), and [naniar](https://naniar.njtierney.com/). Similarly to the missigno package in Python, there are functions to visualise a nullity matrix or by column. For brevity, these will just be listed without the inclusion of figures. 

```{note}
Take care that the missing values are recognised in the given programming language and that each column is of the right data type (for instance, numeric and not strings). 
```

```R
fictional_dataset <- read.csv(file = "FictionalDatasetMiss.csv")
```

(pd-missing-data-visualising-missingness-r1)=
### 1. Nullity Matrix

```R
library(visdat)
vis_dat(fictional_dataset)
vis_miss(fictional_dataset)
```

(pd-missing-data-visualising-missingness-r2)=
### 2. Nullity by Column

```R
library(ggplot)
gg_miss_var(fictional_dataset)
```

An interesting extension of the gg_miss_var function can help in pointing out the number of instances a given combination of variables all have missing data:

```R
gg_miss_upset(fictional_dataset)
```

```{figure} ../../../figures/gg-miss-upset.png
---
height: 500px
name: gg-miss-upset
alt: >
  UpSet plot visualising intersections of missing data across variables in a fictional dataset, generated using the `gg_miss_upset` function in R.
  On the left, horizontal bars show the total number of missing values (set size) per variable.
  On the right, vertical bars indicate the number of observations with shared missingness across combinations of variables (intersection size).
  The largest intersection shows 2 observations missing both Diastolic and Systolic Blood Pressure.
  Other bars show individual missingness in variables such as Cognitive Score or Blood Test Result.
  Beneath the bars, a dot matrix identifies which variables are involved in each intersection.
---
A bar plot showing the number of intersections of missing values between different variables. The total number of missing values in a given variable are shown by the set size bar plot in the bottom left corner of the figure. A table under the main bar plot shows the intersecting variables for a given bar. 

```
(pd-missing-data-visualising-missingness-r3)=
### 3. Nullity Correlation

```R
gg_miss_fct(x = fictional_dataset, fct = Blood.Test.Result)
```

This function works slightly differently from the one introduced in the Python subsection.
It shows the number of missing values per variable and per given value in a specified categorical variable.
In this instance, the plot shows the number of missing values for every possible blood test result value (negative, positive, or missing).
This could be helpful to visualise if missingness is associated with the value of a specific categorical variable (there is no relationship in our example dataset).
For instance, men are less likely to fill out a depression survey; this would be immediately evident if population survey data was visualised with this function against gender. 

(pd-missing-data-visualising-missingness-r4)=
### 4. Beyond Pairwise Nullity Correlations


```R
library(naniar)

ggplot(fictional_dataset, 
       aes(x = Motor.Score, 
           y = Diastolic.Blood.Pressure)) + 
  geom_miss_point()
```

```{figure} ../../../figures/geom-miss-point.png
---
height: 400px
name: geom-miss-point
alt: >
  Scatterplot showing Diastolic Blood Pressure (y-axis) against Motor Score (x-axis), created using `geom_miss_point()` from the `naniar` R package.
  Each point represents an individual in the dataset.
  Blue points indicate complete (non-missing) values, while red points represent observations with missing Diastolic Blood Pressure.
  Missing values are plotted at a position 10% lower than the minimum observed value for that variable, allowing them to be visualised without being removed.
  The plot shows that individuals with higher Motor Scores are more likely to have missing blood pressure readings.
---
A scatterplot of diastolic blood pressure against motor score. Any missing values are plotted at 10% less than the smallest value of a given feature and are colored in red.
```

In {ref}`pd-missing-data-structures-mar`, we mentioned that some study participants were unable to attend the blood pressure clinic due to being older and frailer. This is partially reflected in the figure, as most individuals with high Motor Scores, are missing their blood pressure readings. 

However, an relationship observed does not necessarily equate to causation. Take the next figure as an example: 

```R
ggplot(fictional_dataset, 
       aes(x = Age, 
           y = Cognitive.Score)) + 
  geom_miss_point()
```

```{figure} ../../../figures/geom-miss-point-age.png
---
height: 400px
name: geom-miss-point-age
alt: >
  Scatterplot displaying Cognitive Score (y-axis) against Age (x-axis), created using `geom_miss_point()` from the `naniar` R package.
  Each point represents an observation in the dataset.
  Blue points indicate complete data, while red points mark missing values for the Cognitive Score, positioned at 10% below the minimum observed score to make them visible.
  The plot shows that individuals over age 80 are more likely to have missing cognitive scores (although this pattern results from how the fictional dataset was constructed).
---
A scatterplot of cognitive score against age. Any missing values are plotted at 10% less than the smallest value of a given feature and are colored in red. 
```

Here, anyone above the age of 80 does not have a cognitive score, so it may seem that being unable to complete the cognitive test is associated with being older.
However, when we defined our fictional dataset, we also defined the reason for any missing cognitive scores.
Specifically, those individuals had significant cognitive decline and had to withdraw from the study.
Therefore, the correlation observed in the plot is just by chance (although it does theoretically make sense)! 

(pd-missing-data-visualising-missingness-summary)=
## Summary

In this sub-chapter, we have explored different tools to visualise missing data.
Some tools simplify viewing any patterns, while others can even give insight into the mechanisms behind any missing data.
This is by no means an exhaustive list of visualisation methods and functions available in these programming languages.
Instead, this serves as an introduction to some useful tools and can be used as a starting point.
The table below summarizes the functions introduced: 

### Summary Table 

| Description                            | Python Function | R Function                   | R package               |
|----------------------------------------|-------------------|---------------------------------|-------------------------|
| Nullity matrix                         | {ref}`matrix<pd-missing-data-visualising-missingness-python1>`            | {ref}`vis_dat & vis_miss<pd-missing-data-visualising-missingness-r1>`                | visdat                  |
| Nullity by column                      | {ref}`bar<pd-missing-data-visualising-missingness-python2>`                |  {ref}`gg_miss_var & gg_miss_upset<pd-missing-data-visualising-missingness-r2>`     | ggplot2                 |
| Nullity Correlation                    | {ref}`heatmap<pd-missing-data-visualising-missingness-python3>`            | {ref}`gg_miss_fct<pd-missing-data-visualising-missingness-r2>`                    | ggplot2                 |
| Beyond Pairwise Nullity Correlations    | {ref}`dendrogram<pd-missing-data-visualising-missingness-python4>`         | {ref}`ggplot with geom_miss_point<pd-missing-data-visualising-missingness-r1>`      | ggplot2 & naniar        |
