# Custom deep learning layers support {#tutorial_dnn_custom_layers}

@tableofcontents

@prev_tutorial{tutorial_dnn_javascript}
@next_tutorial{tutorial_dnn_OCR}

|    |    |
| -: | :- |
| Original author | Dmitry Kurtaev |
| Compatibility | OpenCV >= 3.4.1 |

## Introduction
Deep learning is a fast-growing area. New approaches to building neural networks
usually introduce new types of layers. These could be modifications of existing
ones or implementation of outstanding research ideas.

OpenCV allows importing and running networks from different deep learning frameworks.
There is a number of the most popular layers. However, you can face a problem that
your network cannot be imported using OpenCV because some layers of your network
can be not implemented in the deep learning engine of OpenCV.

The first solution is to create a feature request at https://github.com/opencv/opencv/issues
mentioning details such as a source of a model and a type of new layer.
The new layer could be implemented if the OpenCV community shares this need.

The second way is to define a **custom layer** so that OpenCV's deep learning engine
will know how to use it. This tutorial is dedicated to show you a process of deep
learning model's import customization.

## Define a custom layer in C++
Deep learning layer is a building block of network's pipeline.
It has connections to **input blobs** and produces results to **output blobs**.
There are trained **weights** and **hyper-parameters**.
Layers' names, types, weights and hyper-parameters are stored in files are
generated by native frameworks during training. If OpenCV encounters unknown
layer type it throws an exception while trying to read a model:

```
Unspecified error: Can't create layer "layer_name" of type "MyType" in function getLayerInstance
```

To import the model correctly you have to derive a class from cv::dnn::Layer with
the following methods:

@snippet dnn/custom_layers.hpp A custom layer interface

And register it before the import:

@snippet dnn/custom_layers.hpp Register a custom layer

@note `MyType` is a type of unimplemented layer from the thrown exception.

Let's see what all the methods do:

- Constructor

@snippet dnn/custom_layers.hpp MyLayer::MyLayer

Retrieves hyper-parameters from cv::dnn::LayerParams. If your layer has trainable
weights they will be already stored in the Layer's member cv::dnn::Layer::blobs.

- A static method `create`

@snippet dnn/custom_layers.hpp MyLayer::create

This method should create an instance of you layer and return cv::Ptr with it.

- Output blobs' shape computation

@snippet dnn/custom_layers.hpp MyLayer::getMemoryShapes

Returns layer's output shapes depending on input shapes. You may request an extra
memory using `internals`.

- Run a layer

@snippet dnn/custom_layers.hpp MyLayer::forward

Implement a layer's logic here. Compute outputs for given inputs.

@note OpenCV manages memory allocated for layers. In the most cases the same memory
can be reused between layers. So your `forward` implementation should not rely on that
the second invocation of `forward` will have the same data at `outputs` and `internals`.

- Optional `finalize` method

@snippet dnn/custom_layers.hpp MyLayer::finalize

The chain of methods is the following: OpenCV deep learning engine calls `create`
method once, then it calls `getMemoryShapes` for every created layer, then you
can make some preparations depend on known input dimensions at cv::dnn::Layer::finalize.
After network was initialized only `forward` method is called for every network's input.

@note Varying input blobs' sizes such height, width or batch size make OpenCV
reallocate all the internal memory. That leads to efficiency gaps. Try to initialize
and deploy models using a fixed batch size and image's dimensions.

## Example: custom layer from TensorFlow
This is an example of how to import a network with [tf.image.resize_bilinear](https://www.tensorflow.org/versions/master/api_docs/python/tf/image/resize_bilinear)
operation. This is also a resize but with an implementation different from OpenCV's built-in resize.

Let's create a single layer network:
~~~~~~~~~~~~~{.py}
inp = tf.placeholder(tf.float32, [2, 3, 4, 5], 'input')
resized = tf.image.resize_bilinear(inp, size=[9, 8], name='resize_bilinear')
~~~~~~~~~~~~~
OpenCV sees that TensorFlow's graph in the following way:

```
node {
  name: "input"
  op: "Placeholder"
  attr {
    key: "dtype"
    value {
      type: DT_FLOAT
    }
  }
}
node {
  name: "resize_bilinear/size"
  op: "Const"
  attr {
    key: "dtype"
    value {
      type: DT_INT32
    }
  }
  attr {
    key: "value"
    value {
      tensor {
        dtype: DT_INT32
        tensor_shape {
          dim {
            size: 2
          }
        }
        tensor_content: "\t\000\000\000\010\000\000\000"
      }
    }
  }
}
node {
  name: "resize_bilinear"
  op: "ResizeBilinear"
  input: "input:0"
  input: "resize_bilinear/size"
  attr {
    key: "T"
    value {
      type: DT_FLOAT
    }
  }
  attr {
    key: "align_corners"
    value {
      b: false
    }
  }
}
library {
}
```
Custom layers import from TensorFlow is designed to put all layer's `attr` into
cv::dnn::LayerParams but input `Const` blobs into cv::dnn::Layer::blobs.
In our case resize's output shape will be stored in layer's `blobs[0]`.

@snippet dnn/custom_layers.hpp ResizeBilinearLayer

Next we register a layer and try to import the model.

@snippet dnn/custom_layers.hpp Register ResizeBilinearLayer

## Example: custom layer from ONNX
ONNX groups operators into **domains**. The standard operators live in the
default domain `ai.onnx`; vendors and exporters often place their own ops in a
named domain such as `my.namespace`. When OpenCV imports an ONNX node, it looks
the op up in cv::dnn::LayerFactory by:

- the op_type alone, for nodes in the default `ai.onnx` domain (or no domain), and
- `"<domain>.<op_type>"`, for nodes in any non-default domain.

Node attributes are passed through to the layer constructor as
cv::dnn::LayerParams entries with the same names. Consider an op `MyCustomOp`
with attributes `scale` and `bias` that computes `y = scale * x + bias`. The
implementation can look like:

@snippet dnn/custom_layer_onnx.cpp CustomScaleBiasLayer

To import a model that uses this op, register the layer **before** calling
cv::dnn::readNetFromONNX. Use cv::dnn::LayerFactory::registerLayer for runtime
registration (and cv::dnn::LayerFactory::unregisterLayer when done) — pick the
right key for the domain of the op as described above:

@snippet dnn/custom_layer_onnx.cpp Register CustomScaleBiasLayer

A complete runnable example is available at
[samples/dnn/custom_layer_onnx.cpp](https://github.com/opencv/opencv/tree/5.x/samples/dnn/custom_layer_onnx.cpp).
Tiny ONNX models exercising both the default-domain and custom-domain registration
paths can be generated with
[generate_custom_layer_models.py](https://github.com/opencv/opencv_extra/tree/5.x/testdata/dnn/onnx/generate_custom_layer_models.py)
in the opencv_extra repository.

## Define a custom layer in Python
The following example shows how to customize OpenCV's layers in Python.

Let's consider the [Holistically-Nested Edge Detection](https://arxiv.org/abs/1504.06375)
model. Its `Crop` layers receive two input blobs and crop the first one to match the
spatial dimensions of the second. OpenCV's built-in `Crop` layer trims from the
top-left corner, whereas this model expects cropping from the center, so using the
built-in behaviour directly would produce shifted results with filled borders.

Next we're going to replace OpenCV's `Crop` layer that makes top-left cropping by
a centric one.

- Create a class with `getMemoryShapes` and `forward` methods

@snippet dnn/custom_layer.py CropLayer

@note Both methods should return lists.

- Register a new layer.

@snippet dnn/custom_layer.py Register

That's it! We have replaced an implemented OpenCV's layer to a custom one.
You may find a full script in the [source code](https://github.com/opencv/opencv/tree/5.x/samples/dnn/edge_detection.py).

<table border="0">
<tr>
<td>![](js_tutorials/js_assets/lena.jpg)</td>
<td>![](images/lena_hed.jpg)</td>
</tr>
</table>
