Using Depthwise Separable Convolutions in Tensorflow

Looking at all of the very large convolutional neural networks such as ResNets, VGGs, and the like, it begs the question on how we can make all of these networks smaller with less parameters while still maintaining the same level of accuracy or even improving generalization of the model using a smaller amount of parameters. One approach is depthwise separable convolutions, also known by separable convolutions in TensorFlow and Pytorch (not to be confused with spatially separable convolutions which are also referred to as separable convolutions). Depthwise separable convolutions were introduced by Sifre in “Rigid-motion scattering for image classification” and has been adopted by popular model architectures such as MobileNet and a similar version in Xception. It splits the channel and spatial convolutions that are usually combined together in normal convolutional layers

In this tutorial, we’ll be looking at what depthwise separable convolutions are and how we can use them to speed up our convolutional neural network image models.

After completing this tutorial, you will learn:

• What is a depthwise, pointwise, and depthwise separable convolution
• How to implement depthwise separable convolutions in Tensorflow
• Using them as part of our computer vision models

Let’s get started!

Overview

This tutorial is split into 3 parts:

• What is a depthwise separable convolution
• Why are they useful
• Using depthwise separable convolutions in computer vision model

What is a Depthwise Separable Convolution

Before diving into depthwise and depthwise separable convolutions, it might be helpful to have a quick recap on convolutions. Convolutions in image processing is a process of applying a kernel over volume, where we do a weighted sum of the pixels with the weights as the values of the kernels. Visually as follows:

Now, let’s introduce a depthwise convolution. A depthwise convolution is basically a convolution along only one spatial dimension of the image. Visually, this is what a single depthwise convolutional filter would look like and do:

The key difference between a normal convolutional layer and a depthwise convolution is that the depthwise convolution applies the convolution along only one spatial dimension (i.e. channel) while a normal convolution is applied across all spatial dimensions/channels at each step.

If we look at what an entire depthwise layer does on all RGB channels,

Notice that since we are applying one convolutional filter for each output channel, the number of output channels is equal to the number of input channels. After applying this depthwise convolutional layer, we then apply a pointwise convolutional layer.

Simply put a pointwise convolutional layer is a regular convolutional layer with a 1x1 kernel (hence looking at a single point across all the channels). Visually, it looks like this:

Why are Depthwise Separable Convolutions Useful?

Now, you might be wondering, what’s the use of doing two operations with the depthwise separable convolutions? Given that the title of this article is to speed up computer vision models, how does doing two operations instead of one help to speed things up?

To answer that question, let’s look at the number of parameters in the model (there would be some additional overhead associated with doing two convolutions instead of one though). Let’s say we wanted to apply 64 convolutional filters to our RGB image to have 64 channels in our output. Number of parameters in normal convolutional layer (including bias term) is $ 3 times 3 times 3 times 64 + 64 = 1792$. On the other hand, using a depthwise separable convolutional layer would only have $(3 times 3 times 1 times 3 + 3) + (1 times 1 times 3 times 64 + 64) = 30 + 256 = 286$  parameters, which is a significant reduction, with depthwise separable convolutions having less than 6 times the parameters of the normal convolution.

This can help to reduce the number of computations and parameters, which reduces training/inference time and can help to regularize our model respectively.

Let’s see this in action. For our inputs, let’s use the CIFAR10 image dataset of 32x32x3 images,

import tensorflow.keras as keras
from keras.datasets import mnist

# load dataset
(trainX, trainY), (testX, testY) = keras.datasets.cifar10.load_data()

Then, we implement a depthwise separable convolution layer. There’s an implementation in Tensorflow but we’ll go into that in the final example.

class DepthwiseSeparableConv2D(keras.layers.Layer):
  def __init__(self, filters, kernel_size, padding, activation):
    super(DepthwiseSeparableConv2D, self).__init__()
    self.depthwise = DepthwiseConv2D(kernel_size = kernel_size, padding = padding, activation = activation)
    self.pointwise = Conv2D(filters = filters, kernel_size = (1, 1), activation = activation)

  def call(self, input_tensor):
    x = self.depthwise(input_tensor)
    return self.pointwise(x)

Constructing a model with using a depthwise separable convolutional layer and looking at the number of parameters,

visible = Input(shape=(32, 32, 3))
depthwise_separable = DepthwiseSeparableConv2D(filters=64, kernel_size=(3,3), padding="valid", activation="relu")(visible)
depthwise_model = Model(inputs=visible, outputs=depthwise_separable)
depthwise_model.summary()

which gives the output

_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 input_15 (InputLayer)       [(None, 32, 32, 3)]       0         
                                                                 
 depthwise_separable_conv2d_  (None, 30, 30, 64)       286       
 11 (DepthwiseSeparableConv2                                     
 D)                                                              

=================================================================
Total params: 286
Trainable params: 286
Non-trainable params: 0
_________________________________________________________________

which we can compare with a similar model using a regular 2D convolutional layer,

normal = Conv2D(filters=64, kernel_size=(3,3), padding=”valid”, activation=”relu”)(visible)

which gives the output

_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 input (InputLayer)       [(None, 32, 32, 3)]       0         
                                                                 
 conv2d (Conv2D)          (None, 30, 30, 64)        1792      
                                                                 
=================================================================
Total params: 1,792
Trainable params: 1,792
Non-trainable params: 0
_________________________________________________________________

That corroborates with our initial calculations on the number of parameters done earlier and shows the reduction in number of parameters that can be achieved by using depthwise separable convolutions.

More specifically, let’s look at the number and size of kernels in a normal convolutional layer and a depthwise separable one. When looking at a regular 2D convolutional layer with $c$ channels as inputs, $w times h$ kernel spatial resolution, and $n$ channels as output, we would need to have $(n, w, h, c)$ parameters, that is $n$ filters, with each filter having a kernel size of $(w, h, c)$. However, this is different for a similar depthwise separable convolution even with the same number of input channels, kernel spatial resolution, and output channels. First, there’s the depthwise convolution which involves $c$ filters, each with a kernel size of $(w, h, 1)$ which outputs $c$ channels since it acts on each filter. This depthwise convolutional layer has $(c, w, h, 1)$ parameters (plus some bias units). Then comes the pointwise convolution which takes in the $c$ channels from the depthwise layer, and outputs $n$ channels, so we have $n$ filters each with a kernel size of $(1, 1, n)$. This pointwise convolutional layer has $(n, 1, 1, n)$ parameters (plus some bias units).

You might be thinking right now, but why do they work?

One way of thinking about it, from the Xception paper by Chollet is that depthwise separable convolutions have the assumption that we can separately map cross-channel and spatial correlations. Given this, there will be bunch of redundant weights in the convolutional layer which we can reduce by separating the convolution into two convolutions of the depthwise and pointwise component. One way of thinking about it for those familiar with linear algebra is how we are able to decompose a matrix into outer product of two vectors when the column vectors in the matrix are multiples of each other.

Using Depthwise Separable Convolutions in Computer Vision Models

Now that we’ve seen the reduction in parameters that we can achieve by using a depthwise separable convolution over a normal convolutional filter, let’s see how we can use it in practice with Tensorflow’s SeparableConv2D filter.

For this example, we will be using the CIFAR-10 image dataset used in the above example, while for the model we will be using a model built off VGG blocks. The potential of depthwise separable convolutions is in deeper models where the regularization effect is more beneficial to the model and the reduction in parameters is more obvious as opposed to a lighter weight model such as LeNet-5.

Creating our model using VGG blocks using normal convolutional layers,

from keras.models import Model
from keras.layers import Input, Conv2D, MaxPooling2D, Dense, Flatten, SeparableConv2D
import tensorflow as tf

# function for creating a vgg block
def vgg_block(layer_in, n_filters, n_conv):
	# add convolutional layers
	for _ in range(n_conv):
		layer_in = Conv2D(filters = n_filters, kernel_size = (3,3), padding='same', activation="relu")(layer_in)
	# add max pooling layer
	layer_in = MaxPooling2D((2,2), strides=(2,2))(layer_in)
	return layer_in

visible = Input(shape=(32, 32, 3))
layer = vgg_block(visible, 64, 2)
layer = vgg_block(layer, 128, 2)
layer = vgg_block(layer, 256, 2)
layer = Flatten()(layer)
layer = Dense(units=10, activation="softmax")(layer)

# create model
model = Model(inputs=visible, outputs=layer)

# summarize model
model.summary()

model.compile(optimizer="adam", loss=tf.keras.losses.SparseCategoricalCrossentropy(), metrics="acc")

history = model.fit(x=trainX, y=trainY, batch_size=128, epochs=10, validation_data=(testX, testY))

Then we look at the results of this 6-layer convolutional neural network with normal convolutional layers,

_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 input_1 (InputLayer)        [(None, 32, 32, 3)]       0         
                                                                 
 conv2d (Conv2D)             (None, 32, 32, 64)        1792                                                                       

 conv2d_1 (Conv2D)           (None, 32, 32, 64)        36928     
                                                                 
 max_pooling2d (MaxPooling2D  (None, 16, 16, 64)       0         
 )                                                               
                                                                 
 conv2d_2 (Conv2D)           (None, 16, 16, 128)       73856     
                                                                 
 conv2d_3 (Conv2D)           (None, 16, 16, 128)       147584                                                                     

 max_pooling2d_1 (MaxPooling  (None, 8, 8, 128)        0         
 2D)                                                             
                                                                 
 conv2d_4 (Conv2D)           (None, 8, 8, 256)         295168    
                                                                 
 conv2d_5 (Conv2D)           (None, 8, 8, 256)         590080    
                                                                 
 max_pooling2d_2 (MaxPooling  (None, 4, 4, 256)        0         
 2D)                                                             
                                                                 
 flatten (Flatten)           (None, 4096)              0         
                                                                 
 dense (Dense)               (None, 10)                40970     
                                                                 
=================================================================
Total params: 1,186,378
Trainable params: 1,186,378
Non-trainable params: 0
_________________________________________________________________
Epoch 1/10
391/391 [==============================] - 11s 27ms/step - loss: 1.7468 - acc: 0.4496 - val_loss: 1.3347 - val_acc: 0.5297
Epoch 2/10
391/391 [==============================] - 10s 26ms/step - loss: 1.0224 - acc: 0.6399 - val_loss: 0.9457 - val_acc: 0.6717
Epoch 3/10
391/391 [==============================] - 10s 26ms/step - loss: 0.7846 - acc: 0.7282 - val_loss: 0.8566 - val_acc: 0.7109
Epoch 4/10
391/391 [==============================] - 10s 26ms/step - loss: 0.6394 - acc: 0.7784 - val_loss: 0.8289 - val_acc: 0.7235
Epoch 5/10
391/391 [==============================] - 10s 26ms/step - loss: 0.5385 - acc: 0.8118 - val_loss: 0.7445 - val_acc: 0.7516
Epoch 6/10
391/391 [==============================] - 11s 27ms/step - loss: 0.4441 - acc: 0.8461 - val_loss: 0.7927 - val_acc: 0.7501
Epoch 7/10
391/391 [==============================] - 11s 27ms/step - loss: 0.3786 - acc: 0.8672 - val_loss: 0.8279 - val_acc: 0.7455
Epoch 8/10
391/391 [==============================] - 10s 26ms/step - loss: 0.3261 - acc: 0.8855 - val_loss: 0.8886 - val_acc: 0.7560
Epoch 9/10
391/391 [==============================] - 10s 27ms/step - loss: 0.2747 - acc: 0.9044 - val_loss: 1.0134 - val_acc: 0.7387
Epoch 10/10
391/391 [==============================] - 10s 26ms/step - loss: 0.2519 - acc: 0.9126 - val_loss: 0.9571 - val_acc: 0.7484

Let’s try out the same architecture but replace the normal convolutional layers with Keras’ SeparableConv2D layers instead:

# depthwise separable VGG block
def vgg_depthwise_block(layer_in, n_filters, n_conv):
	# add convolutional layers
	for _ in range(n_conv):
		layer_in = SeparableConv2D(filters = n_filters, kernel_size = (3,3), padding='same', activation='relu')(layer_in)
	# add max pooling layer
	layer_in = MaxPooling2D((2,2), strides=(2,2))(layer_in)
	return layer_in

visible = Input(shape=(32, 32, 3))
layer = vgg_depthwise_block(visible, 64, 2)
layer = vgg_depthwise_block(layer, 128, 2)
layer = vgg_depthwise_block(layer, 256, 2)
layer = Flatten()(layer)
layer = Dense(units=10, activation="softmax")(layer)
# create model
model = Model(inputs=visible, outputs=layer)

# summarize model
model.summary()

model.compile(optimizer="adam", loss=tf.keras.losses.SparseCategoricalCrossentropy(), metrics="acc")

history_dsconv = model.fit(x=trainX, y=trainY, batch_size=128, epochs=10, validation_data=(testX, testY))

Running the above code gives us the result:

_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 input_1 (InputLayer)        [(None, 32, 32, 3)]       0         
                                                                 
 separable_conv2d (Separab  (None, 32, 32, 64)       283       
leConv2D)                                                       
                                                                 
 separable_conv2d_2 (Separab  (None, 32, 32, 64)       4736      
 leConv2D)                                                       
                                                                 
 max_pooling2d (MaxPoolin  (None, 16, 16, 64)       0         
 g2D)                                                            
                                                                 
 separable_conv2d_3 (Separab  (None, 16, 16, 128)      8896      
 leConv2D)                                                       
                                                                 
 separable_conv2d_4 (Separab  (None, 16, 16, 128)      17664     
 leConv2D)                                                       
                                                                 
 max_pooling2d_2 (MaxPoolin  (None, 8, 8, 128)        0         
 g2D)                                                            
                                                                 
 separable_conv2d_5 (Separa  (None, 8, 8, 256)        34176     
 bleConv2D)                                                      
                                                                 
 separable_conv2d_6 (Separa  (None, 8, 8, 256)        68096     
 bleConv2D)                                                      
                                                                 
 max_pooling2d_3 (MaxPoolin  (None, 4, 4, 256)        0         
 g2D)                                                            
                                                                 
 flatten (Flatten)         (None, 4096)              0         
                                                                 
 dense (Dense)             (None, 10)                40970     
                                                                 
=================================================================
Total params: 174,821
Trainable params: 174,821
Non-trainable params: 0
_________________________________________________________________
Epoch 1/10
391/391 [==============================] - 10s 22ms/step - loss: 1.7578 - acc: 0.3534 - val_loss: 1.4138 - val_acc: 0.4918
Epoch 2/10
391/391 [==============================] - 8s 21ms/step - loss: 1.2712 - acc: 0.5452 - val_loss: 1.1618 - val_acc: 0.5861
Epoch 3/10
391/391 [==============================] - 8s 22ms/step - loss: 1.0560 - acc: 0.6286 - val_loss: 0.9950 - val_acc: 0.6501
Epoch 4/10
391/391 [==============================] - 8s 21ms/step - loss: 0.9175 - acc: 0.6800 - val_loss: 0.9327 - val_acc: 0.6721
Epoch 5/10
391/391 [==============================] - 9s 22ms/step - loss: 0.7939 - acc: 0.7227 - val_loss: 0.8348 - val_acc: 0.7056
Epoch 6/10
391/391 [==============================] - 8s 22ms/step - loss: 0.7120 - acc: 0.7515 - val_loss: 0.8228 - val_acc: 0.7153
Epoch 7/10
391/391 [==============================] - 8s 21ms/step - loss: 0.6346 - acc: 0.7772 - val_loss: 0.7444 - val_acc: 0.7415
Epoch 8/10
391/391 [==============================] - 8s 21ms/step - loss: 0.5534 - acc: 0.8061 - val_loss: 0.7417 - val_acc: 0.7537
Epoch 9/10
391/391 [==============================] - 8s 21ms/step - loss: 0.4865 - acc: 0.8301 - val_loss: 0.7348 - val_acc: 0.7582
Epoch 10/10
391/391 [==============================] - 8s 21ms/step - loss: 0.4321 - acc: 0.8485 - val_loss: 0.7968 - val_acc: 0.7458

Notice that there are significantly less parameters in the depthwise separable convolution version (~200k vs ~1.2m parameters), along with a slightly lower train time per epoch. Depthwise separable convolutions is more likely to work better on deeper models that might face an overfitting problem and on layers with larger kernels since there is a greater decrease in parameters and computations that would offset the additional computational cost of doing two convolutions instead of one. Next, we plot the train and validation and accuracy of the two models, to see differences in the training performance of the models:

 

The highest validation accuracy is similar for both models, but the depthwise separable convolution appears to have less overfitting to the train set, which might help it generalize better to new data.

Combining all the code together for the depthwise separable convolutions version of the model,

import tensorflow.keras as keras
from keras.datasets import mnist

# load dataset
(trainX, trainY), (testX, testY) = keras.datasets.cifar10.load_data()
# depthwise separable VGG block
def vgg_depthwise_block(layer_in, n_filters, n_conv):
	# add convolutional layers
	for _ in range(n_conv):
		layer_in = SeparableConv2D(filters = n_filters, kernel_size = (3,3), padding='same',activation='relu')(layer_in)
	# add max pooling layer
	layer_in = MaxPooling2D((2,2), strides=(2,2))(layer_in)
	return layer_in

visible = Input(shape=(32, 32, 3))
layer = vgg_depthwise_block(visible, 64, 2)
layer = vgg_depthwise_block(layer, 128, 2)
layer = vgg_depthwise_block(layer, 256, 2)
layer = Flatten()(layer)
layer = Dense(units=10, activation="softmax")(layer)
# create model
model = Model(inputs=visible, outputs=layer)

# summarize model
model.summary()

model.compile(optimizer="adam", loss=tf.keras.losses.SparseCategoricalCrossentropy(), metrics="acc")

history_dsconv = model.fit(x=trainX, y=trainY, batch_size=128, epochs=10, validation_data=(testX, testY))

Further Reading

This section provides more resources on the topic if you are looking to go deeper.

Papers:

• Rigid-Motion Scattering For Image Classification (depthwise separable convolutions)
• MobileNet
• Xception

APIs:

• Depthwise Separable Convolutions in Tensorflow (SeparableConv2D)

Summary

In this post, you’ve seen what are depthwise, pointwise, and depthwise separable convolutions. You’ve also seen how using depthwise separable convolutions allows us to get competitive results while using a significantly smaller number of parameters.

Specifically, you’ve learnt:

• What is a depthwise, pointwise, and depthwise separable convolution
• How to implement depthwise separable convolutions in Tensorflow
• Using them as part of our computer vision models

 

The post Using Depthwise Separable Convolutions in Tensorflow appeared first on Machine Learning Mastery.