--- title: Lesson 3 - Multi-Label Classification keywords: fastai sidebar: home_sidebar nb_path: "nbs/course2020/vision/03_Multi_Label.ipynb" ---
{% raw %}
{% endraw %}

Lesson Video:

{% raw %}
{% endraw %}

For this particular one, we want vision and data

{% raw %} 
from fastai.vision.all import *
{% endraw %}

Below you will find the exact imports for everything we use today

{% raw %} 
import pandas as pd

import torch
from torch import nn

from fastcore.meta import use_kwargs_dict

from fastai.callback.fp16 import to_fp16
from fastai.callback.progress import ProgressCallback
from fastai.callback.schedule import lr_find, fit_one_cycle

from fastai.data.block import MultiCategoryBlock, DataBlock
from fastai.data.external import untar_data, URLs
from fastai.data.transforms import RandomSplitter, ColReader

from fastai.metrics import accuracy_multi, BaseLoss

from fastai.vision.augment import aug_transforms
from fastai.vision.data import ImageBlock
from fastai.vision.learner import cnn_learner

from torchvision.models import resnet34
{% endraw %}

For this multi-label problem, we will use the Planet dataset, where it's a collection of satellite images with multiple labels describing the scene. I'll go through and explain a few different ways to make this dataset, highlighting some of the flexibility the new DataBlock API can do.

First, let's grab our data

{% raw %} 
planet_source = untar_data(URLs.PLANET_SAMPLE)
df = pd.read_csv(planet_source/'labels.csv')
{% endraw %}

Now let's look at how it's stored. Our DataFrame is formatted so our images filename is the first column, and the labels in the second

{% raw %} 
df.head()
image_name tags
0 train_21983 partly_cloudy primary
1 train_9516 clear cultivation primary water
2 train_12664 haze primary
3 train_36960 clear primary
4 train_5302 haze primary road
{% endraw %}

Method 1 (DataBlock)

{% raw %} 
batch_tfms = aug_transforms(flip_vert=True, max_lighting=0.1, max_zoom=1.05, max_warp=0.)
{% endraw %} {% raw %} 
planet = DataBlock(blocks=(ImageBlock, MultiCategoryBlock),
                   get_x=ColReader(0, pref=f'{planet_source}/train/', suff='.jpg'),
                   splitter=RandomSplitter(),
                   get_y=ColReader(1, label_delim=' '),
                   batch_tfms = batch_tfms)
{% endraw %}
  • Multi-label so we want a MultiCategoryBlock
  • get_x and get_y define how we expect to grab our data
  • ColReader works with Pandas DataFrames

Now we can dataloaders by passing in our source folder

{% raw %} 
dls = planet.dataloaders(df)
{% endraw %} {% raw %} 
dls.show_batch(max_n=9, figsize=(12,9))
{% endraw %}

Method 2: Lambda's

This next version will instead use lambda functions to grab our image names, which get's rid of the ColReader, if you're more familiar with these

{% raw %} 
blocks = (ImageBlock, MultiCategoryBlock)
{% endraw %}

First let's try our get_x. Our lambda function needs to return a Path() to our particular image. This can be done by including f'{x[0]}.jpg'

But what is that even doing? Let's take a look

{% raw %} 
get_x = lambda x:planet_source/'train'/f'{x[0]}.jpg'
{% endraw %}

If we pass in one row of our DataFrame, we should expect to see the entire path laid out in front of us!

{% raw %} 
val = df.values[0]; val

array(['train_21983', 'partly_cloudy primary'], dtype=object)
{% endraw %} {% raw %} 
get_x(df.values[0])

Path('/root/.fastai/data/planet_sample/train/train_21983.jpg')
{% endraw %}

Which it does! A nice, simple, and clean way to grab our paths. Let's see how our y getter will look like

{% raw %} 
get_y = lambda x:x[1].split(' ')
{% endraw %}

Looks fairly close to the previous version, if you pay attention. Remember that our x is the DataFrame's values, so if we grab position 1 from earlier, we can see that it's our labels!

Let's make our full PipeLine now that we're sure everything will work

{% raw %} 
planet = DataBlock(blocks=blocks,
                   get_x=get_x,
                   splitter=RandomSplitter(),
                   get_y=get_y,
                   batch_tfms=batch_tfms)
{% endraw %} {% raw %} 
dls = planet.dataloaders(df)
dls.show_batch(max_n=9, figsize=(12,9))
{% endraw %}

Method 3: Custom get_items Functions

That previous one worked fine, but shouldn't I be able to do a one-liner? Since it's all right there instead of defining our get_x and get_y? There IS! We can create our own function, where we should expect to return both an x and a y value. Let's make one

{% raw %} 
def _planet_items(x): return (
    f'{planet_source}/train/'+x.image_name+'.jpg', x.tags.str.split())
{% endraw %}

Our DataBlock now looks like so:

{% raw %} 
planet = DataBlock.from_columns(blocks=(ImageBlock, MultiCategoryBlock),
                   get_items = _planet_items, 
                   splitter=RandomSplitter(),
                   batch_tfms=batch_tfms)
{% endraw %}

That's all our DataBlock needs if we can plan accordingly. Looks pretty clean!

{% raw %} 
dls = planet.dataloaders(df)
dls.show_batch(max_n=9, figsize=(12,9))
{% endraw %}

Training a Model

We'll use a resnet34 for this task

{% raw %} 
from torchvision.models import resnet34
{% endraw %}

We'll use accuracy_multi for multi-classificaiton

{% raw %} 
from fastai.metrics import accuracy_multi
{% endraw %} {% raw %} 
learn = cnn_learner(dls, resnet34, pretrained=True, metrics=[accuracy_multi])
{% endraw %}

Model results are on a scale with a threshold instead of exact, allows for "it's not any" due to a sigmoid activation in the loss function

{% raw %} 
class BCEWithLogitsLossFlat(BaseLoss):
    "Same as `nn.BCEWithLogitsLoss`, but flattens input and target."
    @use_kwargs_dict(keep=True, weight=None, reduction='mean', pos_weight=None)
    def __init__(self, *args, axis=-1, floatify=True, thresh=0.5, **kwargs):
        if kwargs.get('pos_weight', None) is not None and kwargs.get('flatten', None) is True:
            raise ValueError("`flatten` must be False when using `pos_weight` to avoid a RuntimeError due to shape mismatch")
        if kwargs.get('pos_weight', None) is not None: kwargs['flatten'] = False
        super().__init__(nn.BCEWithLogitsLoss, *args, axis=axis, floatify=floatify, is_2d=False, **kwargs)
        self.thresh = thresh

    def decodes(self, x):    return x>self.thresh
    def activation(self, x): return torch.sigmoid(x)
{% endraw %} {% raw %} 
learn.loss_func = BCEWithLogitsLossFlat()
{% endraw %}

We'll find a good learning rate

{% raw %} 
learn.lr_find()
{% endraw %}

And train! (we'll also used mixed precision too!)

{% raw %} 
lr = 1e-2
learn = learn.to_fp16()
{% endraw %} {% raw %} 
learn.fit_one_cycle(5, slice(lr))
epoch train_loss valid_loss accuracy_multi time
0 0.761724 0.320393 0.894412 00:14
1 0.366634 0.170998 0.940882 00:11
2 0.256233 0.149577 0.949118 00:11
3 0.199033 0.137296 0.949706 00:11
4 0.170974 0.136933 0.949706 00:11
{% endraw %}

Great! Let's save and unfreeze to train the rest of our model!

{% raw %} 
learn.save('stage-1')
{% endraw %} {% raw %} 
learn.unfreeze()
learn.lr_find()
{% endraw %}

We can now go through and use a learning rate around 1e-5 or so, and then have an ending learning rate five times smaller than our starting

{% raw %} 
learn.fit_one_cycle(5, slice(1e-5, lr/5))
epoch train_loss valid_loss accuracy_multi time
0 0.147181 0.146752 0.947353 00:15
1 0.153571 0.138148 0.953530 00:15
2 0.147831 0.129908 0.953530 00:15
3 0.127491 0.124736 0.956765 00:15
4 0.114531 0.126795 0.954118 00:15
{% endraw %} {% raw %} 
learn.show_results(figsize=(15,15))
{% endraw %}