Deep Residual Learning for Image Recognition

Slides:

Advertisements

Similar presentations

A brief review of non-neural-network approaches to deep learning

Advertisements

Neural networks Introduction Fitting neural networks

ImageNet Classification with Deep Convolutional Neural Networks

Large-Scale Object Recognition with Weak Supervision

Spatial Pyramid Pooling in Deep Convolutional

Deep Convolutional Nets

Cascade Region Regression for Robust Object Detection

Spatial Localization and Detection

Neural networks (2) Reminder Avoiding overfitting Deep neural network Brief summary of supervised learning methods.

Deep Learning Overview Sources: workshop-tutorial-final.pdf

Spatial Pyramid Pooling in Deep Convolutional Networks for Visual Recognition arXiv: v4 [cs.CV(CVPR)] 23 Apr 2015 Kaiming He, Xiangyu Zhang, Shaoqing.

Lecture 3a Analysis of training of NN

Convolutional Neural Networks at Constrained Time Cost (CVPR 2015) Authors : Kaiming He, Jian Sun (MSR) Presenter : Hyunjun Ju 1.

Convolutional Neural Networks

Automatic Lung Cancer Diagnosis from CT Scans (Week 3)

When deep learning meets object detection: Introduction to two technologies: SSD and YOLO Wenchi Ma.

Deep Residual Learning for Image Recognition

Deep Residual Networks

Faster R-CNN – Concepts

CS 6501: 3D Reconstruction and Understanding Convolutional Neural Networks Connelly Barnes.

Object Detection based on Segment Masks

Compact Bilinear Pooling

Data Mining, Neural Network and Genetic Programming

Computer Science and Engineering, Seoul National University

Convolutional Neural Fabrics by Shreyas Saxena, Jakob Verbeek

The Problem: Classification

CSCI 5922 Neural Networks and Deep Learning: Convolutional Nets For Image And Speech Processing Mike Mozer Department of Computer Science and Institute.

Regularizing Face Verification Nets To Discrete-Valued Pain Regression

Ajita Rattani and Reza Derakhshani,

Inception and Residual Architecture in Deep Convolutional Networks

Neural Networks 2 CS446 Machine Learning.

Training Techniques for Deep Neural Networks

Multiple Wavelet Coefficients Fusion in Deep Residual Networks for Fault Diagnosis

Efficient Deep Model for Monocular Road Segmentation

Convolutional Networks

CS6890 Deep Learning Weizhen Cai

Machine Learning: The Connectionist

Deep Residual Learning for Image Recognition

Dynamic Routing Using Inter Capsule Routing Protocol Between Capsules

ECE 599/692 – Deep Learning Lecture 6 – CNN: The Variants

Layer-wise Performance Bottleneck Analysis of Deep Neural Networks

Introduction to Neural Networks

Image Classification.

SBNet: Sparse Blocks Network for Fast Inference

Towards Understanding the Invertibility of Convolutional Neural Networks Anna C. Gilbert1, Yi Zhang1, Kibok Lee1, Yuting Zhang1, Honglak Lee1,2 1University.

Incremental Training of Deep Convolutional Neural Networks

ALL YOU NEED IS A GOOD INIT

SAS Deep Learning Object Detection, Keypoint Detection

Very Deep Convolutional Networks for Large-Scale Image Recognition

8-3 RRAM Based Convolutional Neural Networks for High Accuracy Pattern Recognition and Online Learning Tasks Z. Dong, Z. Zhou, Z.F. Li, C. Liu, Y.N. Jiang,

A Proposal Defense On Deep Residual Network For Face Recognition Presented By SAGAR MISHRA MECE

Neural Networks Geoff Hulten.

Lecture: Deep Convolutional Neural Networks

Outline Background Motivation Proposed Model Experimental Results

Machine Learning – Neural Networks David Fenyő

RCNN, Fast-RCNN, Faster-RCNN

Deep Learning Some slides are from Prof. Andrew Ng of Stanford.

Inception-v4, Inception-ResNet and the Impact of

Deep Residual Learning for Automatic Seizure Detection

Heterogeneous convolutional neural networks for visual recognition

Convolutional Neural Network

CSCI 5922 Neural Networks and Deep Learning: Convolutional Nets For Image And Speech Processing Mike Mozer Department of Computer Science and Institute.

Course Recap and What’s Next?

Human-object interaction

低比特卷积神经网络的量化研究介绍主讲人：朱锋.

Natalie Lang Tomer Malach

CS295: Modern Systems: Application Case Study Neural Network Accelerator Sang-Woo Jun Spring 2019 Many slides adapted from Hyoukjun Kwon‘s Gatech “Designing.

VERY DEEP CONVOLUTIONAL NETWORKS FOR LARGE-SCALE IMAGE RECOGNITION

End-to-End Facial Alignment and Recognition

Xilai Li, Tianfu Wu, and Xi Song CVPR 2019 Presented by Dingquan Li

Presentation transcript:

Deep Residual Learning for Image Recognition Author : Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun (accepted to CVPR 2016) Presenter : Hyeongseok Son

The deeper, the better The deeper network can cover more complex problems Receptive field size ↑ Non-linearity ↑ However, training the deeper network is more difficult because of vanishing/exploding gradients problem Kaiming He, Xiangyu Zhang, Shaoqing Ren, & Jian Sun. “Deep Residual Learning for Image Recognition”. arXiv 2015.

Deep Neural Network Escape from few layers ReLU for solving gradient vanishing problem Dropout … Kaiming He, Xiangyu Zhang, Shaoqing Ren, & Jian Sun. “Deep Residual Learning for Image Recognition”. arXiv 2015.

Deep Neural Network Escape from 10 layers Normalized initialization Intermediate normalization layers Kaiming He, Xiangyu Zhang, Shaoqing Ren, & Jian Sun. “Deep Residual Learning for Image Recognition”. arXiv 2015.

Deep Neural Network Escape from 100 layers Residual network Kaiming He, Xiangyu Zhang, Shaoqing Ren, & Jian Sun. “Deep Residual Learning for Image Recognition”. arXiv 2015.

Kaiming He Dehazing using dark channel prior Guided image filter Computer vision using machine learning… Kaiming He, Xiangyu Zhang, Shaoqing Ren, & Jian Sun. “Deep Residual Learning for Image Recognition”. arXiv 2015.

Plain Network Plain nets: stacking 3x3 conv layers 56-layer net has higher training error and test error than 20-layers net Kaiming He, Xiangyu Zhang, Shaoqing Ren, & Jian Sun. “Deep Residual Learning for Image Recognition”. arXiv 2015.

Plain Network “Overly deep” plain nets have higher training error A general phenomenon, observed in many datasets Kaiming He, Xiangyu Zhang, Shaoqing Ren, & Jian Sun. “Deep Residual Learning for Image Recognition”. arXiv 2015.

Residual Network Naïve solution If extra layers are an identity mapping, then a training errors does not increase Kaiming He, Xiangyu Zhang, Shaoqing Ren, & Jian Sun. “Deep Residual Learning for Image Recognition”. arXiv 2015.

Residual Network Deeper networks also maintain the tendency of results Features in same level will be almost same An amount of changes is fixed Adding layers makes smaller differences Optimal mappings are closer to an identity Kaiming He, Xiangyu Zhang, Shaoqing Ren, & Jian Sun. “Deep Residual Learning for Image Recognition”. arXiv 2015.

Residual Network Plain block Difficult to make identity mapping because of multiple non-linear layers Kaiming He, Xiangyu Zhang, Shaoqing Ren, & Jian Sun. “Deep Residual Learning for Image Recognition”. arXiv 2015.

Residual Network Residual block If identity were optimal, easy to set weights as 0 If optimal mapping is closer to identity, easier to find small fluctuations -> Appropriate for treating perturbation as keeping a base information Kaiming He, Xiangyu Zhang, Shaoqing Ren, & Jian Sun. “Deep Residual Learning for Image Recognition”. arXiv 2015.

can treat perturbation Residual Network Difference between an original image and a changed image Preserving base information Some Network residual can treat perturbation

Residual Network Deeper ResNets have lower training error Kaiming He, Xiangyu Zhang, Shaoqing Ren, & Jian Sun. “Deep Residual Learning for Image Recognition”. arXiv 2015.

Residual Network Residual block Very simple Parameter-free A naïve residual block “bottleneck” residual block (for ResNet-50/101/152) Kaiming He, Xiangyu Zhang, Shaoqing Ren, & Jian Sun. “Deep Residual Learning for Image Recognition”. arXiv 2015.

Residual Network Shortcuts connections Identity shortcuts Projection shortcuts Kaiming He, Xiangyu Zhang, Shaoqing Ren, & Jian Sun. “Deep Residual Learning for Image Recognition”. arXiv 2015.

Network Design Basic design (VGG-style) Other remarks All 3x3 conv (almost) Spatial size/2 => #filters x2 Batch normalization Simple design, just deep Other remarks No max pooling (almost) No hidden fc No dropout Kaiming He, Xiangyu Zhang, Shaoqing Ren, & Jian Sun. “Deep Residual Learning for Image Recognition”. arXiv 2015.

Network Design ResNet-152 Use bottlenecks ResNet-152(11.3 billion FLOPs) has lower complexity than VGG- 16/19 nets (15.3/19.6 billion FLOPs) Kaiming He, Xiangyu Zhang, Shaoqing Ren, & Jian Sun. “Deep Residual Learning for Image Recognition”. arXiv 2015.

Results Deep Resnets can be trained without difficulties Deeper ResNets have lower training error, and also lower test error Kaiming He, Xiangyu Zhang, Shaoqing Ren, & Jian Sun. “Deep Residual Learning for Image Recognition”. arXiv 2015.

Results Deep Resnets can be trained without difficulties Deeper ResNets have lower training error, and also lower test error Kaiming He, Xiangyu Zhang, Shaoqing Ren, & Jian Sun. “Deep Residual Learning for Image Recognition”. arXiv 2015.

Results 1st places in all five main tracks in “ILSVRC & COCO 2015 Competitions” ImageNet Classification ImageNet Detection ImageNet Localization COCO Detection COCO Segmentation Kaiming He, Xiangyu Zhang, Shaoqing Ren, & Jian Sun. “Deep Residual Learning for Image Recognition”. arXiv 2015.

Quantitative Results ImageNet Classification Kaiming He, Xiangyu Zhang, Shaoqing Ren, & Jian Sun. “Deep Residual Learning for Image Recognition”. arXiv 2015.

Result Performances increase absolutely Based on ResNet-101 Existing techniques can use residual networks or features from it Kaiming He, Xiangyu Zhang, Shaoqing Ren, & Jian Sun. “Deep Residual Learning for Image Recognition”. arXiv 2015.

Qualitative Result Object detection Faster R-CNN + ResNet Kaiming He, Xiangyu Zhang, Shaoqing Ren, & Jian Sun. “Deep Residual Learning for Image Recognition”. arXiv 2015. Jifeng Dai, Kaiming He, & Jian Sun. “Instance-aware Semantic Segmentation via Multi-task Network Cascades”. arXiv 2015.

Qualitative Results Instance Segmentation Kaiming He, Xiangyu Zhang, Shaoqing Ren, & Jian Sun. “Deep Residual Learning for Image Recognition”. arXiv 2015.

Exploring over 1000 layers Test 1202 layers Training is finished Training error is similar Testing error is high because of over fitting Kaiming He, Xiangyu Zhang, Shaoqing Ren, & Jian Sun. “Deep Residual Learning for Image Recognition”. arXiv 2015.

Conclusion Deeper networks are better expectably ResNet is very simple We should use it from now on

Reference Kaiming He, Xiangyu Zhang, Shaoqing Ren, & Jian Sun. “Deep Residual Learning for Image Recognition”. arXiv 2015. Slides of Deep Residual Learning @ ILSVRC & COCO 2015 competitions

Thank You