1. Lecture 4a: Imagenet: Classification with Localization

2. Agenda ILSVRC competition Classification with Localization
Overfeat: integrated classification, localization, and detection
R-CNN (Regions with CNN)
SPP-net (Spatial Pyramid Pooling)
Fast R-CNN

3. Imagenet Database
Imagenet data base: 22 categories, ~14 mln labeled images ( ~700 images/class )

4. ILSVRC Classification over 1000 categories: Classification
Classification over 1000 categories:
1.2 million training images
50,000 validation images
150,000 testing images
Classification
Assign to each image label 5 guesses
Classification & Localization
5 guesses: label + bounding box
Detection:
any number of objects in image (including zero)
False positives are penalized

5. ILSVRC: Classification
top-5 labels

6. ILSVRC: Classification & Localization
top-5 labels + bounding box

7. ILSVRC 2014 vs PASCAL 2012 PASCAL 2012 ILSVRC 2013 ILSVRC 2014
# classes 20
200
Training
# images
5,717
395,909
456,567
# objects
13,609
345,854
478,807
Validation
5,823
20,121
13,841
55,502
testing
10,991
40,152

8. ILSVRC: Classification
These are two dogs from two distinct classes

9. ILSVRC: Classification
Groundtruth: ????

10. ILSVRC: Classification
Groundtruth: coffee mug

11. ILSVRC: Classification
Groundtruth: coffee mug
Top-5:
table lamp
lamp shade
printer
projector
desktop computer

12. AlexNet (2012) AlexNet – winner 2012 with 85% top-5 accuracy
8 layers (5 conv. + 3 fully connected layers+droput + soft-max)
650K neurons , 60 Mln weights
was trained on two GTX-580 with 3 GB memory.
training took 6 days

13. AlexNet : Training SGD parameters batch = 128 examples,
momentum = 0.9,
weight decay =
Weight initialization: a 0-mean Gaussian with std dev= 0.01.
Learning rate:
The learning rate was initialized at 0.01 and adjusted manually throughout training: divide the learning rate by 10 when the validation error rate stopped improving
The same for all layers,
Dropout for fully connected layer
90 epochs through whole image dataset.

14. GoogleNet (2014) Winner of 2014 with 93.5% top-5 accuracy
22 layers (2 conv. +9 inception+ linear with dropout+ softmax)
5 mln parameters
Trained on Google DistBelief cluster
~ 1 week on multi-GPU system (?)

15. Overfeat: Classification with localization

16. Overfeat: integrated classification, localization & detection
Overfeat (NYU) - a convolutional network to simultaneously classify, locate and detect objects.
Key ideas:
multiple scales
apply a ConvNet at multiple locations in the image in a sliding window
train the system to produce for each window
a distribution over categories
a prediction of the location and size of the bounding box with object relative to the viewing window
accumulate results for each categories at each location and scale

17. Overfeat: topology summary
First 5 layers are similar to Alexnet: conv. layer with ReLU and max pooling, but with the following differences:
no contrast normalization
pooling regions are non-overlapping
smaller stride to improve accuracy
Input 3x [221x221], Output of last convoluitonal layer 1024 x [5x5]
Feature Extraction: 3 x [231x231]  1024 x [5x5], down-sampling: 36:1

18. Overfeat: classification
Let’s takes image, and apply window [231x231].
Output of last conv. layer 1024 features x [5x5].
FC layers + log-loss will give scores for each class
Feature extractor has multiple poolings and strides with total scale ratio 36:1, so if we slide the input window with step 36, the feature window will slide with step 1.
231x231
5x5
Input window [231x231]
1024 Features:
x [5x5]
1000 class scores

19. Overfeat: classification
2 adjacent windows share many computations. Let’s get features for all windows locations simultaneously
Feature Extraction:
We compute first 5 layers for whole image ( correspond to 12:1 “subsampling” )
Classifier:
The classifier has a fixed-size 5x5 input which is applied to the layer 5. We will shift the classifier’s input window by 1 pixel through pooling layers without subsampling.
In the end we have [MxN] x C scores, where M, N are sliding windows index, and C – number of classes.
Input
Layer 5
Before pooling
After pool 3x3
Classifier map
245x245
17x17
[3x3] x [5x5]
[3x3] x C
281x 317
20x23
[6x9] x [5x5]
[6x9] x C

20. Overfeat: data augmentation and scaling
to locate objects in different sizes we rescale image to 6 scales (The typical ratio from one scale to another is about ~1.4 )
horizontal flipping.
Final post-processing:
for each class we took local spatial max for different locations and scales
take top-5 classes

21. Overfeat: Boosting
Boosting: train 7 different models with different init weights, and select the best result

22. Overfeat: Training parameters
Data augmentation:
Each image is down-sampled so that the smallest dimension is 256 pixels.
Then extract 5 random crops (and their horizontal flips) of size 221x221 pixels
Training:
weight initialization: randomly with (µ, σ) = (0, 1 × )
SGD with momentum =0.6
weight decay =1×10-5
learning rate = 0.05 , is decreased by ½ after 30, 50, 60, 70, 80 epochs
dropout

23. Overfeat: Localization
Starting from classification-trained network
Fix the feature extraction layers (1-5) and replace the classifier layers by a regression network
Regression net takes as input the features from layer 5
2 fully-connected hidden layers of size 4096 and 1024 channels
output layer 4 units for each class: with the coordinates for the bounding box edges.
Train regression net
the same set of scales as in multi-scale classification
ℓ2 loss between the predicted and true bounding box for each example
compare the prediction of the regressor at each spatial location with the ground-truth bounding box, shifted into the frame of reference

24. Overfeat: Regression Net Topology
Input 3x231x231
conv: 11×11 stride 4×4; ReLU; maxpool: 2×2 stride 2×2; output: 96x24x24
conv: 5×5 stride 1×1; ReLU; maxpool: 2×2 stride 2×2; output: 256x12x12
conv: 3×3 stride 1×1 0-padded; ReLU; output: 512x12x12
conv: 3×3 stride 1×1 0-padded; ReLU; output: 1024x12x12
conv: 3×3 stride 1×1 0-padded; ReLU; maxpool: 2×2 stride 2×2; output: 1024x6x6
full; ReLU; output : 4096x1x1
full; output: 1024x1x1
Regressor: output: 200 x 4

25. Overfeat: localization pipeline
The raw classifier/detector outputs a class confidence for each window location (multi- scale)

26. Overfeat: localization pipeline
The regression predicts the location of the object with respect to each window (multi-scale)

27. Overfeat: localization pipeline
3. Box merging for top5 classes

28. Overfeat: Localization pipeline
Choose top-5 classes Cs by taking the maximum detection class outputs across spatial locations for each scale s ∈ 1… 6
Initial set of bounding boxes: B ←Us Bs, where Bs the set of bounding boxes predicted by the regressor net for each class in Cs, across all spatial locations at scale s.
Repeat merging of boxes from B until done:
(b1, b2) = argmin b1!= b2∈B match_score (b1, b2) , where match_score = the sum of the distance between centers of the two bounding boxes and the intersection area of the boxes.
If (match_score(b1, b2) < t), then set B ← B\ {b1, b2} ∪ box_merge(b1, b2): box_merge computes the average of the bounding boxes’ coordinates.

29. R-CNN: Regions with CNN

30. R-CNN: Regions with CNN features
Regions with CNN detection approach:
generates ~2000 category-independent regions for the input image,
extracts a fixed-length feature vector from each region using a CNN,
classifies each region with category-specific linear SVM
R-CNN outperforms OverFeat, with a mAP = 31.4% vs 24.3%.
R. Girshick et al , “Rich feature hierarchies…”

31. R-CNN pipeline
Region detection  2000 regions , see
Feature extraction with Imagenet:
Region croped and scaled to [227 x 227]
5 conv.layers + 2FC  4096 features
SVM for 200 classes
Greedy non-maximum region suppression for each class:
rejects a region if it has an big overlap with a region which has > score

32. R-CNN Training
The key idea is to train feature extraction CNN on a large auxiliary dataset (ILSVRC-classification 1000 classes), followed by domain specific fine-tuning on a small dataset (200 classes):
Pre-training: Train Imagenet for classification
Replace last layer with FC layer to N+1 outputs:
N classes + 1 “background”
if ground truth class and gt box has IoU with region > ½  positive, otherwise  background
Regular SGD based training
batch from 128 images = 32 positive + 96 background
Random weights initialization

33. R-CNN: ILSVRC 2013 detection performance

34. R-CNN drawbacks Training is expensive in time:
Training is a multi-stage pipeline.
Fine-tune a ConvNet for detection using cross-entropy loss.
Train linear SVMs on ConvNet features computed on warped object proposals
Learn bounding-box regressors
Training is expensive in storage:
For SVM and regressor training, features are extracted from each warped object proposal in each image and written to disk – hundreds of GB
Test-time detection is slow.

35. R-CNN speed and R-CNN detection is very slow: 12.5 sec/image
time per 1 frame analysis

36. CNN with Spatial Pyramid pooling

37. SPP-net = Spatial Pyramid Pooling + CNN
Classical conv. net takes a fixed-size (e.g ) input image:
Need cropping or warping to transform original image to square shape
This constraint is related to Fully-Connected layer ONLY
Idea: let’s use Spatial Pooling Pyramid to transform any-shape image to ‘fixed-length” feature vector.
Kaiming He et al, “Spatial Pyramid Pooling in Deep Convolutional Networks …“

38. SPP-Net Soft Max Inner Product BACKWARD FORWARD ReLUP Inner Product
SPP(5x5+7x7+13x13)
Pooling [2x2, stride 2]
Convolutional layer [5x5]
Pooling [2x2, stride 2]
Convolutional layer [5x5]
Data Layer
Data Layer [28x28]

39. Spatial Pyramid Pooling
Example. SPP layer :
Input feature maps has size [13 x13]
3 pooling layers, which have fixed outputs size: [4x4], [2x2], and [1x1]
Size of pooling windows and stride for each pooling layer depends on input feature map size.

40. SPP-net training Network - based on Overfeat Training:
Data augmentation: horizontal flipping and Color altering
Dropout with 2 last FC layers
Init learning rate =0.01; divide by 10 when error plateau

41. SPP-net: Detection pipeline
find 2000 region-candidates ( like R-CNN )
extract the feature maps from the entire image
SPP maps each window of the features, corresponding to region-candidate, to a fixed-length representation
SVM using 2 FC layers

42. SPP-net vs R-CNN
Detection is ~100x faster than R-CNN, but training is very slow:
Multi-stage training (fine-tuning of last layers, SVM and regressors)
Still needs a lot of disk space to save features vectors

43. Fast R-CNN

44. Fast-RCNN
Fast R-CNN is similar to SPP-net, but it trains both classification and regression networks together using a multi-task loss in a single training stage:
Unlike SPP-net, all network layers can be updated during fine-tuning
The multi-task loss simplifies learning and improves detection accuracy
Currently the best and the fastest detection algorithm
3x faster training,
Detection only 0.3s ,
Best performance
R. Girshick, Fast R-CNN: code:

45. Fast-RCNN: architecture
Start from CNN trained for ImageNet classification
Add Region-of-Interest (RoI) pooling layer after last convolutional layer
Replace FC layers by two ‘sibling” nets:
Sof-max: estimates probability over (K+1 background) classes
Regression layer which predicts box (x,y,w,h) for each K classes

46. Fast-RCNN: architecture
Each RoI is pooled into fixed size vector
2 FC layers  RoI feature vector
Multi-task loss: softmax classifier + box regressor/ class

47. Fast-RCNN: RoI pooling layer
Input:
feature maps from the last convoluional layer [C x H x W]
List of regions of interest . Each RoI is a tuple (n; x; y; h;w) where n - image index (or scale), (r; c) region top-left location and (h;w) - height and width
Output: for each region
Max-pooled feature maps [C x H’ x W’] (H’ ≤ H, d W’ ≤ W).
The pooling area (“bin” ) size ~ [h/H’, w/W’] ( SPP with one level)
RoI pooling layer back-propagation:

48. Fast-RCNN: Multi-task Loss
For each RoI 2 outputs:
Sof-max: estimates probability over (K+1 background) classes
Regression layer: predicts box b=(x,y,w,h) for each class
Multi-task loss: L(p, k*, b, b*) = Lcls (p, k*) + λ*δ(k*) Lloc (b,b*)
k* is true label: δ(k* ) =1 and 0 for all other class
Lcls is regular log-loss: Lcls (p, k*) = -log (pk*)
Lloc is smooth L1 loss: Lloc (b,b*) = [ x-x’ ] + [ y-y’ ] + [ h-h’ ] +[ w-w’ ], where [t] is smooth L1 norm: [t] = min ( 0.5*|t| , 0.5*t2 )

49. Fast-RCNN: One pass training
Training is combines together fine-tuning of convolutional layers, training of of classifier and box-regressors:
Each batch consists of 2 images x 64 RoI per image:
16 “true” RoI ( IoU > ½ with ground truth box)
48 ‘background” RoI ( IoU < ½) , labeled as k=0
Data augmentation:
each image is horizontally flipped with prob= ½
Option: multi-scale image pyramid
SGD:
Momentum =0.9, weight decay=0.0005
weight initialization for FC layers – gaussian (σ = 0.01, 0.001)
initial global lr =0.001, after 30,000 iterations lr =0.0001

50. Fast R-CNN: detection time optimization
For classification FC layers usually take small amount of time comparing to convolutional layers. For detection FC will become heavy compute since we have to do it for each RoI.
We can compress a single fully-connected layer W using SVD-like decomposition of W = U*∑*VT into two fully-connected layers without a non-linearity between them:
First FC corresponds to (U*∑) and second to VT

51. BACKUP

52. ILSVRC: Localization

53. Overfeat: classification

54. Overfeat: ”fast” net topology
Input 3x231x231
convo: 11×11 stride 4×4; ReLU; maxpool: 2×2 stride 2×2; output: 96x24x24
convo: 5×5 stride 1×1; ReLU; maxpool: 2×2 stride 2×2; output: 256x12x12
convo: 3×3 stride 1×1 0-padded; ReLU; output: 512x12x12
convo: 3×3 stride 1×1 0-padded; ReLU; output: 1024x12x12
convo: 3×3 stride 1×1 0-padded; ReLU; maxpool: 2×2 stride 2×2; output: 1024x6x6
convo: 6×6 stride 1×1; ReLU; output: 3072x1x1
full; ReLU; output : 4096x1x1
full; output: 1000x1x1
softmax; output: 1000x1x1

55. Single-class Regression vs Per- Class Regression
Using a different top layer for each class in the regressor network for each class (Per-Class Regressor (PCR) surprisingly did not outperform using only a single network shared among all classes (44.1% vs. 31.3%).

56. Overfeat: Detection
The detection task differ from localization in that there can be any number of object in each image (including zero), and that false positives are penalized by the mean average precision (mAP) measure The main difference with the localization task, is the necessity to predict a background class when no object is present. Traditionally, negative examples are initially taken at random for training. Then the most offending negative errors are added to the training set in bootstrapping passes.

57. R-CNN: PASCAL VOC performance
2012 SIFT, HOG,…

58. R-CNN: PASCAL VOC performance
2014: Regions with CNN

59. R-CNN CODE
Requires Matlab!

60. CNN regression
Szegedy et all ( Google) 2010, “Deep Neural Networks for Object Detection”
start with Alexnet,
replace last soft-max layer with regression layer which generates an binary mask “d x d” : 1 if pixel is inside box, 0- otherwise;
train net by minimizing L2 error vs ground truth mask m:

61. CNN regression
Multi-scale

62. CNN regression Issues: Issue1:
Overlapping masks for multiple touching objects
Localization accuracy
Recognition of small objects
Issue1:
To deal with multiple touching objects, we generate not one but several masks, each representing either the full object or part of it.
we use one network to predict the object box mask and four additional networks to predict four halves of the box: bottom, top, left and right halves