1. Natalie Lang Tomer Malach
ResNet
Natalie Lang Tomer Malach
Deep Residual Learning for Image Recognition
Kaiming He Xiangyu Zhang Shaoqing Ren Jian Sun
Microsoft Research
Introduction
Work ResNet, by Microsoft, 2015
Made a revolution in the performance of the NN
Specifically, won 1st place in 5 major competition in CV field
Amazing!
Deep Learning and Its Applications to Signal and Image Processing and Analysis
Dr. Tammy Riklin Raviv 
Spring 2019

2. Motivation Deep learning continuously changing the world around us
Its Applications are everywhere:
Image Recognition - classification, detection
Healthcare - breast or skin-cancer diagnostics
Finance - predict the stock market
Predicting earthquakes - vital in saving a life
Lets start with some motivation to the field of NN in general
DL completely changing the world around us
Classify images and detect objects in them
Asist in detecting cancer
And so on and so on
In recent years neural networks became so good that they even outperform humans (!) in several task, for example in the task of image classification

3. Benchmarks Assess the relative performance of the nets
Checking them all on the same Datasets
CIFAR-10
60,000 images
32 x 32
10 labels
Classification
MNIST
60,000 images
28 x 28
10 labels
Classification
MNIST- relatively simple and small database comparing to others
CIFAR10-similarly. Data base for training a network to identify images of cat dog ship etc…

4. Grading The Networks Evaluating networks performance
The Top-5 error rate is the percentage of test
examples for which the correct class was not in the
top 5 scores predicted classes.
The Top-1 error rate is the percentage of test
examples for which the correct class was not the top
score class
So have benchmarks for evaluate the network performance
But how do we to give it a grade in practice?
For each image The network gives predictions…
Having database and grades
the competition begins…

5. ImageNet ~1,200,000 images 1000 labels classification
shallow
8 layers
19 layers
22 layers
152 layers
Top-5 Error Rate [%]
ImageNet
~1,200,000 images
1000 labels
classification
Large scale
A lot of labels
The net should give 1 label out of 1000 labels!
Indeed very hard task even for human observer
Winners results along the past years. Top 5 error scores
2012 – CNN comes in
2015 – ResNet
Comparing humans
CNN dates way back…

6. LeNet-5 [LeCun et al., 1998] First use of convolutional layers
Handwritten character recognition
Components: conv., pooling and FC layers
More then 20 years ago…

7. Overview 2012 First CNN-based winner shallow 8 layers 19 layers
Lets return to 2012
First CNN model based that won the challenge
Stated the big bang of Deep Neural networks

8. AlexNet [Krizhevsky et al., 2012]
Deeper – more layers
Bigger – more filters
First to use ReLU activation function
Basically similar to LeNet
same components :Conv, pool, FC
The main difference that AlexNet Just have more layers and much more filters

9. Overview 2014 Deeper Network shallow 8 layers 19 layers 22 layers
The next essential improvement

10. VGGNet [Simonyan and Zisserman, 2014]
AlexNet
VGG16
VGG19
Depth is a critical component.
Performs only 3x3 convolutions.
2 Versions: VGG16 or VGG19.
The deeper the better
More conv layers with smaller filters
Better performance using deeper net like VGG 19 than AlexNet with 8
We conclude that deeper is better
Quiet
Is that so? We’ll see about that…

11. Up to Now Motivation Benchmarks LeNet ILSVRC AlexNet VGG Lets recap
We saw a motivation for NN
We talk about benchmarks for evaluating their performance
We saw the NN not invented so recently but more than 20 years ago
Image net challenge for image recognition
Alex and VGG that was the winners of the recent years
Now Tomer Malach will continue

12. Problems In Deep Networks: Overfitting?
What happens when we continue stacking deeper layers on a “plain” convolutional neural network?
56 layer model performs worse on both training and test error.
Conclusion:
The deeper model performs worse, but it’s not caused by overfitting!

13. Vanishing Gradient x + x
Let’s demonstrate the problem with short computational graph example:
𝐹=𝑊2∙(𝑋∙𝑊1+𝑍) 𝑋
0.2
𝜕𝐹 𝜕𝑋 = 𝜕𝐹 𝜕𝐵 ∙ 𝜕𝐵 𝜕𝐴 ∙ 𝜕𝐴 𝜕𝑋 =𝑊2∙1∙𝑊2
𝜕𝐹 𝜕𝐴 = 𝜕𝐹 𝜕𝐵 ∙ 𝜕𝐵 𝜕𝐴 =𝑊2∙1=0.3
=0.12 x
𝐴=𝑋∙𝑊1=0.08
0.4 𝑤1 𝑤2 𝑊1 N1
𝑋(𝑖𝑛𝑝𝑢𝑡)
𝜕𝐹 𝜕𝐵 =𝑊2=0.3 N2
𝐹(𝑜𝑢𝑡𝑝𝑢𝑡) +
𝜕𝐹 𝜕𝑊1 = 𝜕𝐹 𝜕𝐵 ∙ 𝜕𝐵 𝜕𝐴 ∙ 𝜕𝐴 𝜕𝑊1 =𝑊2∙1∙𝑋
=𝟎.𝟎𝟔
𝐵=𝐴+𝑍=0.18
0.1
𝑍(𝑏𝑖𝑎𝑠) 𝑍 x
𝜕𝐹 𝜕𝑍 = 𝜕𝐹 𝜕𝐵 ∙ 𝜕𝐵 𝜕𝑍 =𝑊2∙1=0.3 𝐹
=𝐵∙𝑊2=0.054
0.3
𝜕𝐹 𝜕𝐹 =1 𝑊2
𝑊1=𝑊1− 𝜕𝐹 𝜕𝑊1 ∙𝑙𝑟, 𝑙𝑟= 10 −4
𝜕𝐹 𝜕𝑊2 =𝐵=0.18
𝑾𝟏=𝟎.𝟑𝟗𝟗𝟗𝟗𝟒≅𝟎.𝟒

14. The Hypothesis
The problem is an optimization problem, deeper models are harder to optimize!
The deeper model should be able to perform at least as well as the shallower model.
A solution by construction is copying the learned layers from the shallower model and setting additional layers to identity mapping.
Deeper has to be at least Better!
Shallow Network
Identity
Identity
Identity
input
output

15. Overview 2015
shallow
8 layers
19 layers
22 layers
152 layers

16. ResNet [He et al., 2015]
Use network layers to fit a residual mapping instead of directly trying to fit a desired underlying mapping.
relu
H(x)
F(x) + x +
F(x) X
identity
relu
relu X
“Plain” layers X
Residual block

17. ResNet - Architecture Stack residual blocks.
relu
Every residual block has two 3x3 conv layers. X
identity .. .
Periodically, double # of filters and downsample spatially using stride 2 (/2 in each dimension).
relu
Additional conv layer at the beginning.
No FC layers at the end (only FC 1000 to output classes).
Total depths of 34, 50, 101, or 152 layers for ImageNet. X
Residual block

18. ResNet - Training
Batch Normalization after every CONV layer (To avoid vanishing gradient).
Xavier / initialization from He et al (To avoid vanishing gradient).
SGD + Momentum (0.9).
Learning rate: 0.1, divided by 10 when validation error plateaus.
Mini-batch size 256.
Validation error
Weight decay of 1e-5.
No dropout used.

19. ResNet - Results
Able to train very deep networks without degrading (152 layers on ImageNet, 1202 on Cifar).
Deeper networks now achieve lowing training error as expected.
Swept 1st place in all ILSVRC and COCO 2015 competitions.

20. Comparing ResNet to others
ILSVRC 2015 classification winner (3.57% top 5 error)
better than “human performance”! [Russakovsky 2014]
Fei-Fei Li & Justin Johnson & Serena Yeung
May 1, 2018

21. Comparing ResNet to other networks performances
Figures copyright Alfredo Canziani, Adam Paszke, Eugenio Culurciello, 2017.
Fei-Fei Li & Justin Johnson & Serena Yeung
May 1, 2018

22. To Summarize Problems in Deep Networks ResNet
Overfitting
Vanishing gradient
ResNet
Architecture
Training
Results
Comparing ResNet to others

23. Thank You!