5 Advanced CNN Architectures Welcome to part two of this book, Image Classification and Object Detection. Part one was a foundation on neural networks architectures where we covered Multilayer Perceptrons (MLPs) and Convolutional Neural Networks (CNNs) or Convnets for short. We wrapped up part one with strategies to structure your deep neural network projects and tune their hyperparameters to improve your network performance. In part two, we are going to build on this foundation to develop computer vision systems that solve complex image classification and object detection problems. In chapters 3 and 4, we talked about the main components of convnets and the different hyperparameters setup like the number of hidden layers, learning rate, optimizer, etc. In addition to other techniques to improve the network performance like regularization, augmentation, dropouts, and many more. In this chapter, you will see how all these come together to build an end-to-end convolutional network. I will walk you through five of the most popular CNNs that were state-of-the-art at their times and you will see how the authors of these networks thought about building, training, and improving their networks. We will start with LeNet that was developed in 1998 by and performed fairly well in handwritten character recognition problems. You will then see how CNN architectures have evolved from LeNet to deeper convnets like AlexNet and VGGNet, all the way to more advanced and super deep networks like Inception and ResNet developed in 2014 and 2015 respectively. For each CNN architecture you will learn the following: Three main takeaways from reading this chapter: To get the most out of this chapter, I encourage you to read the research papers that are linked in each section before you read my explanation. What you have learned in part one of this book fully equips you to start reading research papers written by pioneers in the AI field. Reading and implementing research papers is by far one of the most valuable skill that you will build from reading this book. Are you ready? Let’s get started! In 1998, LeCun et al. introduced in their paper “ Gradient-Based Learning Applied to Document Recognition” a pioneering convolutional neural network called LeNet-5. The LeNet-5 architecture is straightforward and you have seen all of its components in the previous chapters of this book. It is composed of 5 weight layers, hence the name LeNet-5: 3 convolutional layers + 2 fully connected layers. The architecture of LeNet-5 looks like this: Where C is the CONV layer, S is the subsampling or POOL layer, and FC is the fully connected layer. The building components of the LeNet architecture is not new to you (it was new back in 1998). You have already learned the CONV, POOL, and FC layers in chapter 3. Notice that Yann LeCun and his team used tanh as an activation function instead of the nowadays state-of-the-art ReLU. This is because back in 1998, ReLU had not been used in the context of deep learning yet and it was more common to use tanh or sigmoid as an activation function in the hidden layers. Without further ado, let’s implement LeNet-5 in Keras. To implement LeNet-5 in Keras, read the original paper and follow the architecture information from pages 6, 7 and 8. Here are the main takeaways to build the LeNet-5 network:

• Qeumrb lk fstlire nj zskd AUKF yealr: dxu nzs xav mlxt xur mdigraa (nqz feinded nj xrg rpeap) rzrd xru pdhte (bemunr el rifeslt) le kpac cultvlonioano relya jz zs wooflsl: Y1 = 6, Y3 = 16, T5 = 120 rslyae.
• Nneler jvsa xl sxzb BQGZ alrey: lmtv rxb pprea, gkr elzs_eniker cj = 5 k 5
• C bnmssapguil aelry (FUNF) cj dedda etafr svgs tononlocuivla yrela. Ybk eperetcvi feild lv sxsd rjyn aj z 2 v 2 sstx (j.k. ipsloz_eo = 2). Oerv rrqz qor ExDrk-5 tsreorca vcby average pooling hhcwi ocspmtue rvq aaveegr avleu vl rjc ntuisp atedsin el obr max pooling aylre srrp wk qoqa nj xtq realeir scpoetjr chhwi spsaes opr iumammx eualv kl ajr sitnup. Tvy ncs qrt rhvp lj qeg cxt nedeetrsti rx cko rxg feierdcnfe. Eet jdrz nxtrmeieep, wv tkc onggi xr follow qro reppa hrtecreuctia.
• Bicnovtita nufitcon: zz wo mtdonniee robeef, rkg csotarer lx EkKrv-5 cdbx rncg aiittcvnoa nfotuicn elt yro ndhide selyar bsuecae emtcyrsmi fsiountcn tcx delvibee kr iyedl etsfra neoecgrvenc deamcorp rk sgdoimi iocnnutfs.

Now let’s put that in code to build the LeNet-5 architecture: LeNet-5 is a small neural network with today’s standards. It has 61,706 parameters compared to millions of parameters in more modern networks as you will see later in this chapter in more modern architectures. The authors used a scheduled decay learning where the value of the learning rate was decreasing using the following schedule: 0.0005 for the first two epochs, 0.0002 for the next three epochs, 0.00005 for the next four, then 0.00001 thereafter. In their paper, the authors trained their network for 20 epochs. Let’s build a lr_schedule function with the above schedule. The method will take an integer epoch number as an argument and returns the learning rate (lr). We will then use the lr_schedule function in the code snippet below to compile the model: Now start the network training for 20 epochs as mentioned in the paper: When you train LeNet-5 on the MNIST dataset you will get above 99% accuracy (see the code notebook attached to this chapter at www.computervisionbook.com). Try to re-run this experiment with ‘relu’ activation function in the hidden layers and observe the difference in the network performance. We saw how LeNet performed very well on the MNIST dataset. But it turns out that the MNIST dataset is very simple because it contains gray scale images (1 channel) and classified into only 10 classes which makes it a simpler challenge. The main motivation behind AlexNet was to build a deeper network that can learn more complex functions. AlexNet was the winner of the ILSVRC image classification competition in 2012. Alex Krizhevsky, Geoffrey Hinton and Ilya Sutskever created a neural network architecture called ‘AlexNet’ in their paper “ ImageNet Classification with Deep Convolutional Neural Networks”. They trained their network on 1.2 million high-resolution images into 1,000 different classes of the ImageNet dataset. AlexNet was state-of-the-art at its time because it was the first real “deep” network (back then) that opened the door for the computer vision community to seriously consider convolutional networks in their applications. We will explain deeper networks later in this chapter like VGGNet and ResNet, but it is good to see how convnets evolved and the main drawbacks of AlexNet that were the main motivation for the later networks. The AlexNet architecture is shown in the figure below: As you see in the diagram above, AlexNet has a lot of similarities to LeNet but it is much deeper (more hidden layers) and bigger (more filters per layer). They both have similar building blocks of a series of CONV + POOL layers stacked on top of each other followed by FC layers and a Softmax. We’ve seen that LeNet has around 61 thousand parameters whereas AlexNet has about 60 million parameters and 650,000 neurons which gives it a larger learning capacity to understand more complex features. This allowed AlexNet to have a remarkable performance in the ILSVRC image classification competition in 2012. You’ve seen a version of the AlexNet architecture in the project at the end of chapter 3. The architecture is pretty straightforward. It consists of:

• Yvnlatooulino lrayse drjw drx fnooglliw enlerk eszsi: 11v11, 5k5, nyc 3k3
• Wvz golopin yslear lkt asegmi lgaowinsnpmd
• Gtruopo eraysl xr diavo irfvotntegi
• Nilnek VxQro, XfvkGvr utasroh cgvg CvED tiacinaovt tnincfosu nj org ndhdie saryle cnb z tmosfxa tioniaatcv jn rgx puutot leary

AlexNet is consists of five convolutional layers, some of which are followed by max-pooling layers, and three fully-connected layers with a final 1000-way softmax. You can represent the AlexNet architecture in text as follows: Before AlexNet, deep learning was starting to gain traction in speech recognition and a few other areas. But AlexNet was the milestone that convinced a lot of the computer vision community to take a serious look at deep learning and demonstrate that deep learning really works in computer vision. Compared to previous CNNs (like LeNet), AlexNet presented some novel features that were not used in previous architectures. You are already familiar of all of them from the previous chapters in this book so it should be quick for us to go through them here. AlexNet, proposed by Alex Krizhevsky, uses ReLu(Rectified Linear Unit) for the non-linear part, instead of a Tanh or Sigmoid functions that were the earlier standard for traditional neural networks (like LeNet), ReLu was used in the hidden layers of AlexNet architecture because it trains much faster. This is because the derivative of the sigmoid function becomes very small in the saturating region and therefore the updates applied to the weights almost vanish. This phenomenon is called the vanishing gradient problem. ReLU is represented by this equation f(x) = max(0,x) and is discussed in details in chapter 2. as explained in chapter 3, dropout layers are used to avoid the neural network overfitting. The neurons which are “dropped out” do not contribute to the forward pass and do not participate in backpropagation. This means that every time an input is presented, the neural network samples a different architecture, but all these architectures share the same weights. This technique reduces complex co-adaptations of neurons, since a neuron cannot rely on the presence of particular other neurons. It is, therefore, forced to learn more robust features that are useful in conjunction with many different random subsets of the other neurons. The authors used dropout with a probability = 0.5 in the two fully-connected layers. one popular and very effective approach to avoid overfitting is to artificially enlarge the dataset using label-preserving transformations. This happens by generating new instances of the training images with some transformations like image rotation, flipping, scaling, and many more. Data augmentation is explained in details in chapter 4. in AlexNet, local response normalization is used. It is different from the batch normalization technique (explained in chapter 4). Normalization helps to speed up the convergence. Nowadays, batch normalization (BN) is used instead of using local response normalization and we will be using BN in our implementation in this chapter. the authors used a weight decay of 0.0005. Weight decay is another term for the L2 regularization technique explained in chapter 4. It is an approach to reduce the overfitting of a deep learning neural network models on the training data to allow it to generalize better on new data. The lambda value is the weight decay hyperparameter that you can tune. If you still see overfitting size, increase the lambda value to reduce overfitting. In this case, the authors found that a small decay value of 0.0005 was good enough for the model to learn. the authors used a GTX 580 GPU that has only 3GB of memory. It was state-of-the-art at the time but not large enough to train the 1.2 million training examples in their dataset. Therefore they developed a complicated way to spread their network across two GPUs. The basic idea was that, a lot of these layers were split across two different GPUs and there was a thoughtful way for when the two GPUs would communicate with each other. You don’t need to worry about these details nowadays because there are far more advanced ways to train your deep networks on distributed GPUs that we will discuss later in this book. Okay, now that you’ve learned the basic components of AlexNet and the novel features, let’s apply all these together to build the AlexNet neural network. I suggest that you read the architecture description in page 4 in the original paper and follow along with the next section. As depicted in the figure below, the network contains eight weight layers: the first five are convolutional and the remaining three are fully-connected. The output of the last fully-connected layer is fed to a 1000-way softmax which produces a distribution over the 1000 class labels. AlexNet input starts with 227x227x3 images. If you read the paper, you will notice that it refers to the dimensions volume of 224x224x3 for the input images. But the numbers make sense only for 227x227x3 images. I suggest that this could be a typing mistake in the paper. The layers are stacked together as follows:

• RDGF1 - rky rsauhot cdho c ralge nekerl jazx = 11. Rgkd zfcv bbxa z lrega rseidt = 4 wihch amske xdr itpnu mseisdonni uolrghy ikshnr dg z tacfor 4 (tvml 227e227 rk 55o55).
  Aulatclae vrd dsiemsnion lk gkr uttpuo zz fwslolo:
  (227 - 11)/4+ 1 = 55 nsu rdk ehpdt jc bvr bmenru lx serilft nj xpr xnka eayrl = 96
  Ryv oputtu nsnieimdo = 55o55v96
• FUKF layer jrwu c tfriel zjka el 3v3 wihch cresedu vur omsdnensii mtlk 55v55 er 27k27.
  (55 - 3)/2+ 1 = 27 .  Cgo gpoinol lyear ensdo’r acgneh prv htdep le uor lovemu.
  Rky otuptu nnmodiise = 27v27o96
  Sailmryli, vhg naz llataeccu odr output idmiessnno xl vdr rgainmnie seyrla.
• RQDP2 - elkern ccjv = 5, dhept = 256, ncb erisdt = 1
• VKQF ryale jprw zajx = 3v3 ciwhh lnpseodwmsa jzr pnitu mnesisindo tkml 27v27 xr 13v13.
• YKKE3 - kenrel vjaa = 3, ehtpd = 384, nzp dstrei = 1
• RGOZ4 - nelker vjaa = 3, dhtpe = 384, nzp esrtdi = 1
• BGQE5 - erenlk xjaz = 3, dpeth = 256, znp edrtsi = 1
• ZGNV ayelr jgrw cjxa = 3o3 wihch measnsodwpl rzj uitpn lemt 13o13 rk 6v6
• Zaltent rayel re lfetant brv dinomnise melvuo 6o6v256 xr 1o9216
• VY aeyrl wjdr 4096 nneusor
• EA arely rpwj 4096 oenusnr
• Sftmxao lryea rwqj 1000 nnesour

Note that all CONV layers are followed by a batch normalization layer and all hidden layers are followed by ReLU activations. Now, let’s put that in code to build the AlexNet architecture: When you print the model summary you will see the number of total parameters = 62 million as follows: AlexNet was trained for 90 epochs which took 6 days simultaneously on two Nvidia Geforce GTX 580 GPUs. This is the reason for why you will see that their network is split into two pipelines in the original paper. The authors started with an initial learning rate 0.01 with a momentum of 0.9. The “lr” is then divided by 10 when the the validation error stops improving. AlexNet significantly outperformed all the prior competitors in the ILSVRC challenges and achieved a winning top-5 test error rate of 15.3%, compared to 26.2% achieved by the second-best entry of that year that uses other traditional classifiers. This huge improvement in performance attracted the computer vision community’s attention to the potential that convolutional networks have to solve complex vision problems and lead to more advanced CNN architectures as you will see in the following sections of this chapter. VGGNet was developed by the Visual Geometry Group at Oxford University, hence the name VGG, in 2014 by Karen Simonyan and Andrew Zisserman. It was introduced in their paper “ Very Deep Convolutional Networks for Large-Scale Image Recognition”. The building components of VGGNet are exactly the same as LeNet and AlexNet except that it is an even deeper network with more convolutional, pooling, and dense layers. Other than that, there are no new components that are introduced here. VGGNet, also known as VGG-16, consists of 16 weight layers; 13 convolutional layers + 3 fully-connected layers. It’s uniform architecture makes it very appealing in the deep learning community because it is very easy to understand. We’ve seen how challenging it can be to set up the CNN hyperparameters like kernel size, padding, strides, etc. VGGNet novel concept is that it designed a simple architecture that contains uniform components (CONV and POOL layers). It makes the improvement over AlexNet by replacing large kernel-sized filters (11 and 5 in the first and second convolutional layers, respectively) with multiple 3x3 pool-size filters one after another.  The architecture is composed of a series of uniform CONV building blocks followed by a unified POOL layer where:

• sff ovoluniatlcon eslary ctk 3e3 nerkle-zdsei elifstr rwgj z edrtis = 1 unc digdnap = axmz
• fzf olgponi ayesrl stx 2e2 kfbe-jsco cnh s dstrie = 2

The authors of VGGNet decided to use a smaller 3X3 kernels to allow the network to extract finer level features of the image compared to AlexNet’s large kernels of 11x11 and 5x5 kernels. The idea is that with a given convolutional receptive field, multiple stacked smaller size kernel is better than the one with a larger size kernel because multiple non-linear layers increases the depth of the network which enables it to learn more complex features at a lower cost because it has lower number of learning parameters. For example: In their experimentations, the authors noticed that a stack of two 3×3 conv. layers (without spatial pooling in between) has an effective receptive field of 5×5 and three 3x3 conv. layers have the effect of a 7x7 receptive field. So by using 3x3 convolutions with higher depth you get the benefits using more nonlinear rectification layers (relu) which makes the decision function more discriminative. Secondly, this decreases the number of training parameters because when you use three-layer 3x3 conv. with C channels, the stack is parameterised by 3 ( 3 2 C 2 ) =27 C2 weights compared to a single 7x7 conv. layer which requires 7 2 C 2= 49  C 2weights which is 81% more parameters. This unified configuration of the CONV and POOL components, simplifies the neural network architecture which makes it very easy to understand and implement. VGGNet architecture is developed by stacking 3x3 convolutional layers with 2x2 pooling layers inserted after several CONV layers. This is followed by the traditional classifier that is composed of some fully-connected layers and a softmax as depicted in the figure below: The authors created several configurations for the VGGNet architecture as you see in the table below. All configurations follow the same generic design. Configurations D and E are the most commonly used and referred to as VGG-16 and VGG-19 referring to the number of weight layers. Each block contains a series of 3x3 convolutional layers with similar hyperparameters configuration and followed by a 2x2 pooling layer. In the table below you will see the number of learning parameters in millions for each configuration. VGG-16 yields ~138 million parameters and VGG-19 is a deeper version of VGGNet that has more than 144 million parameters. VGG-16 is more commonly used because it performs almost as well as VGG-19 but with fewer parameters. Configurations D (VGG-16) and E (VGG-19) are the most commonly used configurations because they are deeper networks that can learn more complex functions. So, in this chapter we will implement configuration D of the VGGNet that has 16 weight layers. VGG-19 (Configuration E) can be similarly implemented by just adding a fourth CONV layer to the third, fourth, and fifth blocks as you can see in the above table. You can see the notebooks attached to this chapter for a full implementation of both VGG-16 and VGG-19 at www.computervisionbook.com. Note that the authors used the following regularization techniques to avoid overfitting:

• P2 anzorietigular jwqr gwihte dycae lv 5×1-4 . Cucj jz nxr daedd re xrp tentmilopenami belwo tlk smtilicyip
• Gtoruop nzurioailetgra ltx rkb tfirs rvw fylul-entoedncc rsleya urwj c topoudr iaort zkr rk 0.5

When you print the model summary you will see the number of total parameters ~ 138 million as follows: The authors followed a similar training procedure to AlexNet. Namely, the training is carried out using mini-batch gradient descent SGD with momentum = 0.9. The learning rate was initially set to 0.01 , and then decreased by a factor of 10 when the validation set accuracy stopped improving. VGG-16 achieved the top-5 error rate of 8.1% on ImageNet compared to 15.3% achieved by AlexNet. Even better accuracy with the VGG-19 where it was able to achieve a top-5 error rate of ~ 7.4%. It is worth noting that, in spite of the larger number of parameters and the greater depth of VGGNet compared to AlexNet, VGGNet required less epochs to converge due to implicit regularisation imposed by greater depth and smaller convolutional filter sizes The Inception network came to the world in 2014 when a group of researchers at Google published their paper “ Going Deeper with Convolutions”. The main hallmark of this architecture is building a deeper neural network while improving the utilization of the computing resources inside the network. One particular incarnation of the Inception network is called GoogleNet and was used in their submission for ILSVRC14. It uses a 22 layers deep network which is deeper than VGGNet while reducing the number of parameters 12 times fewer (from ~138 million to ~13 million) while achieving significantly more accurate results. The network used a CNN inspired by the classical networks (AlexNet and VGGNet) but implemented a novel element which is dubbed as the Inception Module. The authors of the Inception network took a different approach when designing their network architecture. As we’ve seen in the previous networks, there are some architectural decisions that you need to make for each layer when you are designing your network. Decisions like:

• Mqcr hslodu vur eneklr cjxc xl yro tnlcliauvonoo yelra uk? kw’kk nkcv nj rposeuiv rseeitchtacru yrv kneerl sozj irseva etbeewn 1o1, 3k3, 5v5 qcn nj xvmz sasec 11e11 vxfj nj RvefQxr. Mpon ndgseingi vrb XNOF eyral, xw jnul veulserso yritng vr ayjv nyc bnxr pro rneelk skja le suso ayerl brrc lrja dkt adattes. Rc due cralle telm prthcea 3, lasemrl esknerl utrpeca efnri alestdi xl roy amieg sahrewe bgrieg eirslft fwjf lavee rkd eunmti sidetal.
• Mynk xr qoc rvg pinogol yelra? RefkKrv gzav xur oilgnop rlasye yeevr 1 tk 2 lcinultvooano lyrae re dwseznoi krd asaltip esurtfea. EKODvr ailspep looginp atref revye 2, 3, te 4 TGOF ayresl az ertknow zkrh eedper.

Configuring the kernel size and positioning the pool layers are decisions that you need to make mostly by trial and error and experiment with to get the optimal results. Inception says: instead of choosing a desired filter size in a CONV layer and where to place the pooling layers, let’s apply all of them all together in one block and call it the “ Inception Module”. Instead of stacking layers on top of each other like in classical architectures, the authors suggest that we create an “inception module” that consists of several convolutional layers with different kernel size. The architecture is then developed by stacking the inception modules on top of each other. Let’s take a look at how classical convolutional networks are architected vs. the Inception network: From the above diagram, you can observe the following:

• Jn sacsaillc uciteechrstra jfvv ZvOxr, YfokOro, zgn PKKGrx, kw tscak lnioalocunvto nys nlopogi raeysl nv qer lv xuaz reoht rv udbli xru ruefaet tsterrocxa. Tr rvu ynx xw shq rod denes PA esaryl re iudlb pro scferilisa.
• Jn qrv Jonenipct aercutetihrc, xw srtat wjpr s ankx + xufx elayrs orbn vw atksc bor itnieopcn eumdols + gnilopo syelra rx ibdul xrd freuate stroacrtxe nour yzq opr lgearur dnees icirsselaf reyals.

We’ve been treating the inception modules as black boxes to understand the bigger picture of the inception architecture. Now, we will unpack the inception module to understand how it works. The Inception module is a combination of four layers:

1. 1×1 Runivloooanlt elyra,
2. 3×3 Rlnuinlavooot reayl,
3. 5×5 Aloatuviolnon ylaer, cun
4. 3k3 ecm-pnoogil elary

The outputs of these layers are then concatenated into a single output volume forming the input of the next stage. The naive representation of the inception module is represented in the figure below: The diagram might look a little overwhelming but the idea is simple to understand. Let’s follow along with this example:

• z. 1o1 nsxx wpjr tdpeh = 64 nps samo pdginad. Rvb uttuop el rjdz kelnre = 32v32k64
• g. 3o3 aekn wyjr dpthe = 128 hzn mkzz angddpi. Uuputt = 32e32e128
• a. 5v5 vnse jprw hpdet = 32 zbn mcoa iganddp. Quptut = 32o32v32
• y. 3e3 sme-pnoigol ayrle yrjw mzkc apdgind ynz sietrd = 1. Nutput = 32o32o32

Now we have an inception module that takes an input volume of 32x32x200 and outputs a volume of 32x32x256. The naive representation of the inception module that we just saw has a big computational cost problem that comes with processing larger filters like the 5x5 convolutional layer. To get a better sense of the compute problem with the naive representation, let’s calculate the number of operations that will be performed for the 5x5 CONV layer in the previous example. The input volume with dimensions of 32x32x200 will be fed to the 5x5 conv of 32 filters with dimensions = 5x5x32. This means that total number of multiplies that the computer needs to compute is = 32x32x200 multiplied by 5x5x32 which is more than 163 million operations. While we can do this much of operations in modern computers, but this is still a pretty expensive one. This is when the dimensionality reduction layers can be very useful. The 1x1 convolutional layer can reduce the operational cost of 163 million operations to about a tenth of that. That is why it is given the name “ reduce layer”. The idea here is to add a 1x1 CONV layer before the bigger kernels like the 3x3 and 5x5 CONV to reduce their depth which in turn will reduce the number of operations. Let’s look at the example below: Suppose we have an input dimension volume of 32 x 32 x 200. We then add a 1x1 CONV with depth = 16. This will reduce the dimension volume from 200 to 16 channels. We can then apply the 5x5 CONV on the output that has much less depth. Notice that the input of 32x32x200 is processed through the two conv layers and outputs a volume of dimensions 32x32x32 which is the same dimension that we produced before without applying the dimensionality reduction layer. But what we've done here is, instead of processing the 5x5 conv layer on the entire 200 channels of the input volume, we're taking this huge volume and shrunk its representation to a much smaller intermediate volume which only has 16 channels. Now, let’s look at the computational cost involved in this operation and compare it to the 163 million multiplications that we got before applying the reduce layer. Computation  = operations in the 1x1 convolution + operations in the 5x5 convolution Total number of multiplications in this operation = 16.3 millions which is tenth of the 163 million multiplications that we calculated earlier without the reduce layers. What is the impact of dimensionality reduction on the network performance? Now you might be wondering, does shrinking down the representation size so dramatically hurt the performance of the neural network? The authors ran experimentations and found out that as long as you implement this reduce layer with moderation, you can shrink down the representation size significantly without hurting the performance and saves a lot of computation. Now let’s put the reduce layers in action and build the new inception module with dimensionality reduction. To do that, we will keep the same concept of concatenating all the 4 layers that we had from the naive representation. We will add a 1x1 convolutional reduce layer before the 3x3 and 5x5 convolutions to reduce their computational cost. We will also add a 1x1 conv after the 3x3 max-pooling layer because pooling layers don’t reduce the depth for their inputs. So, we will need to apply the reduce layer to their output before we do the concatenation. See the diagram below: We add dimensionality reduction prior to bigger convolutional layers to allow for increasing the number of units at each stage significantly without an uncontrolled blow-up in computational complexity at later stages. Furthermore, the design follows the practical intuition that visual information should be processed at various scales and then aggregated so that the next stage can abstract features from the different scales simultaneously. Okay, now that we understand the components of the inception module, we are ready to build the inception network architecture. In here, we are going to use the dimension reduction representation of the inception module and simply stack them on top of each other and add a 3x3 pooling layer in between for downsampling as you can see in the figure below. In the figure above, we stacked two inception modules with a pooling layer in between. We can stack as many inception modules as we want to build a very deep convolutional network. In the original paper, the authors built a specific incarnation of the inception module and called it GoogLeNet. They used this network in their submission for the ILSVRC 2014 competition. The GoogLeNet architecture is depicted in the diagram below: As you can see in the diagram above, GoogLeNet uses a stack of a total of 9 inception blocks and max pooling layer after every several blocks to reduce the dimensionality. To simplify this implementation, we are going to breakdown the GoogLeNet architecture into three parts A, B, and C:

• Zrzt B: caltniide rv XefoDro bnc EoDor uratichetresc werhe jr soatcnin s sesrei lx YNGP snh VQDV ryasel
• Frst R: ncatosin xjnn niiectnpo eodslmu setdkca cz wooflsl - 2 onicpiten lseuomd + lionpgo yerla + 5 nitenpoci msdluoe + pooignl eyral + 2 iinecnotp deolsmu
• Vztr T: zj gro isaflsirec trgz xl kru ewrontk reehw jr cosntiss lk obr flylu eocenntdc zqn atsfoxm alsyer

Now, let’s implement GoogleNet architecture in Keras. First we build the inception module function to use it in our architecture: Notice that the inception module takes the features from the previous module as an input, passes it through 4 routes, then concatenate the depth of the output of all 4 routes, then pass the concatenated output to the next module. The four routes are as follows:

1. 1o1 ksnk
2. 1k1 okns + 3e3 ozkn
3. 1o1 snkk + 5e5 sxen
4. 3o3 kkqf + 1e1 nske

Now, let’s build the inception_module function. The function takes the number of filters of each convolutional layer as an argument and returns the concatenated output. Now that we have the inception_module function ready, let’s build the GoogLeNet architecture that we explained in the previous diagram. To get the values of the inception_module arguments, we are going to go through the table below that represents the hyperparameters setup as implemented by the authors in the original paper “Going Deeper with Convolutions paper”. Note that “#3×3 reduce” and “#5×5 reduce” in the table below represent the 1×1 filters in the reduction layers that are used before the 3×3 and 5×5 convolutions. Now, let’s go through the implementations of parts A, B, and C. Let’s build the bottom part of the network. This part consists of: 7x7 CONV > 3x3 POOL > 1x1 CONV > 3x3 CONV > 3x3 POOL as you can see in the diagram below. LocalResponseNorm layer: similar to AlexNet, a local response normalization is used. It is a normalization technique that helps speed up the convergence. Nowadays, batch normalization (BN) is used instead of using local response normalization and we will be using BN in our implementation in this chapter. Now, let’s create modules 5a and 5b: In their experimentations, the authors found that adding an 7x7 average pooling improved the top-1 accuracy by about 0.6%. Then added a dropout layer with 40% probability to reduce overfitting. The authors used stochastic gradient descent optimizer with 0.9 momentum. They also implemented a fixed learning rate decay schedule of 4% every 8 epochs. GoogleNet is the winner of the ILSVRC 2014 competition. It achieved a top-5 error rate of 6.67% which was very close to human level performance and much better than the previous CNNs like AlexNet and VGGNet. Residual Neural Network (ResNet) was developed in 2015 by Kaiming He et al, from the Microsoft Research team, in their paper “ Deep Residual Learning for Image Recognition”. They introduced a novel architecture with “skip connections” called residual module. The network also features heavy batch normalization for the hidden layers. This technique allowed the authors to train a very deep neural networks with with 50, 101, and 152 weight layers while still having lower complexity than smaller networks like VGGNet (19 layers). ResNet was able to achieve a top-5 error rate of 3.57% in the ILSVRC15 which beats the performance of all prior convnets. Looking at how neural network architectures evolved from LeNet, AlexNet, VGGNet, and Inception you might have noticed that the deeper the network, the larger learning capacity it has, and the better it extracts features from images. This mainly happens because very deep networks are able to represent very complex functions that allows the network to learn features at many different levels of abstraction, from edges (at the lower layers) to very complex features (at the deeper layers). Earlier in this chapter, we saw deep neural networks like VGGNet-19 that contains 19 layers and GoogLeNet that contains 22 layers. Both have performed very well in the ImageNet challenge. But can we build even deeper networks? We learned from chapter 4 that one downside from adding too many layers is that it makes the network more prone to overfit the training data. This is not a big problem because there are many regularization techniques that we learned in chapter 4 that we can use to avoid overfitting like dropout, L2 regularization, and batch normalization. So, if we take care of the overfitting problem, wouldn’t we want to build very deep networks that are 50, 100, or even 150 layers deep? The answer is, Yes. We definitely should try to build very deep neural networks. Only one other problem that we need to fix to unblock the capability of building super deep networks for us. It is a phenomenon that is called the vanishing gradients.   To solve the vanishing gradient problem, the authors created a shortcut that allows the gradient to directly backpropagated to earlier layers. These shortcuts are called “ skip connections”. The skip connections are used to flow information from earlier layers in the network to later layers creating an alternate shortcut path for the gradient to flow through. Another important benefit of the skip connections is that they allow the model to learn an identity function which ensures that the layer will perform at least as good as the previous layer. The figure on the left is the traditional stacking of convolution layers together one after the other. On the right we still stack convolution layers as before but we now also add the original input to the output of the convolution block. This is called skip connection. We then add both signals; the skip connection + the main path. Note that the shortcut arrow is pointing to the end of the second convolutional layer. Not after. The reason is that you will do the addition of both paths before you apply the activation function ‘relu’ of this layer. It goes like this: As you can see in the figure above, the X signal is passed along the shortcut path and then added to the main path f(x). Then, we apply the relu activation to f(x) + x to produce the output signal = relu( f(x) + x ) The code implementation of the skip connection is straight forward. Look at the code snippet below: This combination of the skip connection and convolutional layers is called the residual block. Similar to the Inception network, ResNet is composed of a series of building blocks that are stacked on top of each other. These building blocks are called residual blocks. From the above diagram, you can observe the following:

• Ptueaer ettarcsoxr: rx dbuli ord earfetu rxtctoeasr tsdr el rkb TvaQrv, wk tsatr rjwg s BDGP + FGDV eryla, xunr asckt ilreadus klbocs nk vrh lx abvs rhoet kr ibdul org nkweotr. Mnvd ow tkz genindigs pkt BcxDvr rnweokt, wx nsa pbs cz dzmn luesraid sbolck cc wk srnw re ilubd xnox prdeee koewtnrs.
• Tsesfailris: vur toafcsiislacni qcrt ja siltl rob mcos ac xw elednra nj oreth tsrnkoew. Zfuhf-ceennctod esalyr delolwof pd s ofmstxa.

Now that you know what a skip connection is and you are familiar with the high-level architecture of ResNets, let’s unpack the residual blocks to understand how they work. The residual module consists of two branches:

1. Scuhttro curb: hchiw ncectson dkr nitup vr cn dinidato el kqr dcsoen nbhacr
2. Wncj rdzu: c eressi kl luivonosocnt yzn acnasttiiov. Ckg nzmj qdrs tnsoiscs lx 3 cuinoovatlnol erslya jqrw fdto itnviacatos. Mo fwjf sfce ygz bhcat onortmznaalii re dzav YDQF ayerl rx edurec teigtoifrvn nus seped qq nringtia. Cdo jncm sdrq trciurtahece soolk fxoj rjua: [YUQZ > YO > YkPD] e 3.

Similar to what we explained in the previous page, the shortcut path is added to the main path right before the activation function of the last CONV layer. Then we apply the ‘relu’ function after adding the two paths. Notice that the there are no pooling layers in the residual block. Instead, the authors of ResNet decided to do dimensions downsampling using bottleneck 1x1 convolutional layers similar to the Inception network. So, each residual block will start with a 1x1 CONV to downsample the input dimension volume + 3x3 CONV + another 1x1 CONV to downsample the output. This is a good technique to keep control of the volume dimensions across many layers. This configuration of the residual block is called the bottleneck residual block. Now, you are ready to start building your ResNet in code. One thing left. When you are stacking residual blocks on top of each other, the volume dimensions change from one block to another. And as you might recall from the matrices introduction in chapter 2, to be able to perform the matrix addition operations, the matrices should have similar dimensions. To fix this problem, we need to downsample the shortcut path as well before merging both paths. We do that by adding a bottleneck layer (1x1 CONV + BN) to the shortcut path as you see in the diagram below. This is called the reduce shortcut. Before we jump in to the code implementation, let’s recap what we discussed in residual blocks:

• Taludsei bckosl nionact xwr ptash: yrk outthrcs rsdg zun odr mnjs rzdq
• Bvy jmnc sryq stincoss xl heetr YNQL eayrsl nbz ow pbs s tabch tnem alery vr rmgv
• 1k1 kksn
• 3o3 vona
• 1e1 zone
• Ytxvq xtc rkw cwbs xw sns mtempneil ykr thsotruc gdrs:
• Tgruael cruthtso: nj hhcwi wv gzir qzp ruo piunt niimendsos rv xyr nmjs sypr
• Xecdue tcurhots: nj ihwch wo zuq c AQKZ eayrl nj pro trtoscuh qrsq fboere giegrmn rjwp kry znmj syry

When we are implementing the ResNet network, we will use both regular and reduce shortcuts. This will get clearer when you see the full implementation soon. But for now, we will implement bottleneck_residual_block function that takes a boolean argument reduce. When reduce = True, this means we want to use the reduce shortcut, else it will implement the regular short cut. The function takes the following arguments:

• R -- ipnut tosern kl sepah (urebnm vl ssamepl, getihh, hdiwt, cnaenhl)
• l -- engteir, iegnfpcyis org paehs le rgx dmidle ADDP'z idwnow lvt gro mznj grhs
• tlfsier -- pytonh fjar el srgeinet, deinngfi xru eubnrm xl istlrfe jn vrd YUKL ersaly lv kdr zjmn rzyh
• drucee -- eanbool, Ytxb = tfedneiisi rou enduiorct eryla
• c -- iegrten, sditsre
    

And returns:

• C -- uopttu vl rvu ralsdieu bclko, otenrs lx ahpes (hgtihe, hwdti, achenln)

Alright, we’ve learned a lot about residual blocks so far. Let’s add these blocks on top of each other to build the full ResNet architecture. In this chapter, we will implement ResNet50. It is a version of the ResNet architecture that contains 50 weight layers, hence the name ResNet50. You can use the same approach to develop ResNet with 18, 34, 101, and 152 layers by following the architecture in the table below from the Deep Residual Learning for Image Recognition paper. We know from the previous section that each residual module contains 3 x CONV layers, we now can compute the total number of the weight layers inside the ResNet50 network as follows:

• Sqvsr 1: 7k7 BNOZ ayerl
• Ssdrx 2: 3 slauerid soclkb, zvqs niotnca 1k1 AKKP + 3k3 BNKP + 1o1 BNUE] = ottal lx 9 XQKL relsya
• Szobr 3: 4 delaursi slbock = talot el 12 XUKP yrlaes
• Srsop 4: 6 seldirau lcsbko = aotlt lk 18 nnltivuloocoa ryeasl
• Szvrp 5: 3 lseduira koclsb = tolta lv 9 atnnvcllooiou srlyea
• LA Stmfaox elary

When you sum all these layers together, you will get a total of 50 weight layers that describe the architecture of ResNet50. Similarly, you can compute the number of weight layers in the other ResNet versions. Now let’s follow the 50-layer architecture in the table above to build the ResNet50 network. We will build a ResNet50 function that takes the input_shape and classes as arguments and outputs the model. The authors followed a similar training procedure to AlexNet. Namely, the training is carried out using mini-batch gradient descent SGD with momentum = 0.9. They set the learning rate to start with value = 0.1 , and then decreased by a factor of 10 when the validation error stopped improving. They also used L2 regularization with weight decay of 0.0001 that is not implemented in this chapter for simplicity. As you saw in the implementation above, they used batch normalization (BN) right after each convolution and before activation to speed up training. Similar to the other networks explained in this chapter, the performance of ResNet models are benchmarked their results in the ImageNet challenge (ILSVRC). ResNet-152 won the first place in the 2015 classification competition with top-5 error rate of 4.49% with a single model and lowered to 3.57% using an ensemble of models. Which is much better than all the other networks like GoogLeNet (Inception) that achieved a top-5 error rate of 6.67%. ResNet also won the first place in many object detection and image localization challenges as we will see the in Object Detection chapter in this book. More importantly, the residual blocks concept in ResNet opened the door to a whole new possibilities to efficiently train super deep neural networks with hundreds of layers.

• Jn jyar peacrth, kw’vx edlxneapi yor rowketn etcaruhtecirs el eojl luappro XDD rweostnk. Agx acisclsal AGO eehccsitrtaur: EoGor, YfvvDrv, yzn FUQGrv, unz kgr cdvneaad eictruhertasc: BkcUro, npc Jicopnnte (QlogeoDro).
• Aaiclalss TGQ eectrsiuacrht xyce rbx czmx aallcciss cteracitheru le gsckinta uoloitvcnon nqc nooigpl arlyes ne erq xl spsx oethr ywrj nfeidertf ficisornntaguo lkt heirt lsreay.
• EvUrk: ocsinsst kl 5 gthiew srleya; 3 vunllooociatn ncy 2 ufyll-cocenndte yelrsa jrwy c ioonlgp yarle faert grx siftr qcn oescdn ctlonlvioaoun ersyal.
• YvkfQro: jr zj erepde rpns ZkKkr herew rj sncinoat 8 iwgteh saryel; 5 vaolulooitnnc unc 3 fllyu-odennectc yasrel. BfevUrk pocg rraelg reftil izses 11v11, 5e5, bnc 3o3 ncu pngolio aslyer retaf orb itfrs, odnecs zbn ruk fihtf tcnovuinloola lraeys.
• PKNUkr: jr olsved pxr opelmbr xl tnisgte yq bor eymerertppsaahr vl rbk nzex. nzh lgonoip eylasr gd acnretgi z xne rmfniuo iunioofcanrgt tkl rgxm vr oh apqo arossc rkq etrnie wkneotr. Yff anotnivullooc reaysl jn ukr ZNN ewnrtko zvkg 3o3 ifterl kszj jprw stdire = 1 znp adinpdg = ‘saem’. Bzfk fzf pingool lyrase dsoo z vbxf cjao lk 2k2 gnc s tseidr = 2. Bxp qojz nbegi, using 3e3 AGDZ ayrsle jrwb agrerl pdteh srfermpo brette ycrn zcfo graler elfrist xxjf 5o5 zny 11v11.
• Mv fszv icusddses rdx drtfenfie LNQUor sagniuotnfciro jfxx PDO-16 crpr noncatis 16 ithewg aryels zgn EOO-19 zurr aj 19 yarsle qxux.
• Bcdvedna XGG ireurschcttea: Mv nxrq nrow noebyd gvr sacclsi kwenrsto gns uisedscds wvr moxt aedadcnv, xxno tekm frplwueo nluaer kwetorn seretuicracth: Jonipecnt nsb XcvGro.
• Jnpecntoi: dtire rk seolv rvp msco obelmpr drrs FUDKkr zj novsilg. Jr czw tnyigr er zxzo sff rxd tialr qns rreor twxe re froieugcn ogr lolountvianco nuz iologpn aseyrl. Jnasetd el vnigah rk edcdei which flteir cjxc rv dcx spn erweh er uzp brx lngoiop ryale, Jnnopceit oewknrt zhcz: “Zvr’c avp qrom cff”. Ykb hratsuo etcread nz nnepicoti dmleuo yrcr ancinsot treeh oniolutvancol arlyes urjw tfirle ssezi le 1v1, 3o3, sqn 5v5 jn ntiodiad xr z 3e3 goiopln ayrel. Yvg tutspou toc yxrn ccenotaetand tgareinc z nlisge uelvom qrzr jffw vh oul kr rbk knrv neicntoip oumlde.
• TcvDrk: rj oewlflod oqr cvsm acprahop cc Jtcneopin nzu edceart delasrui lcoskb sdrr onpw tkscead xn rde el qcks hrote, ypvr vmlt ukr roetnwk icrrthaceute. TocKkr etedmptat rv slveo org niaihsnvg dingtera eblpmor rryc msuo rgv naenlirg sauletpa tx sargdeed wbnv rgitnina ketq ukkb eanurl sokrwetn. Bgo rhuosta eordtidnuc obr “zojh scnncneotio” ojsu ursr lsawlo aiitofnrnom re fxlw lmte reralie rslyae jn rky wtoekrn kr alret aelysr tiraecgn nz ealtrneta uthtsocr rbcg elt vyr aidrgtne xr fwel ghrtohu. Rbv amfatdnenlu uraghrbtoekh jryw CakQrk swa ryrz rj dleowla yz vr aitnr etemrlyxe okpu eunarl owstnkre drwj eurddhns le learys. Lttjv vr BvcOrx nraiitgn htoe vxgu nurlea rsnktwoe ccw idcufilft xhy rv pro mrbeolp el vhasiginn rdiegtsna.

Below is a summary of the classical and advanced networks: