Data Mining, Neural Network and Genetic Programming COMP422 Week 7 Convolutional Neural Network Yi Mei and Mengjie Zhang yi.mei@ecs.vuw.ac.nz mengjie.zhang@ecs.vuw.ac.nz

Outline Why CNN? Automatic Feature Extraction Feature Maps Number of Parameters in CNN CNN for Handwritten Digit Recognition

Traditional ANN What is the meaning of the hidden layers/nodes?

Convolutional Neural Network Traditional fully connected architecture is not efficient in most cases Too many weights, most redundant Does not employ domain knowledge Require huge amount of training data Hard to interpret Can we improve the architecture by employing domain knowledge? In image processing, what domain knowledge can we use?

Image Classification Input image: 32 *32 pixels Classification: face or non-face? Build a neural network for it How do you design the input layer?

Automatic Feature Extraction Directly use the raw pixel values so that no need to manually design high-level features Use neural network to automatically extract high-level features It is done by convolution Convolution mask What is the mask for calculating the mean for this region?

Automatic Feature Extraction Possible feature example The mean pixel value in the 3*3 area centered at pixel (8, 5), i.e at row 8 and column 5 The weighted sum pixel value in the 3*3 area centered at pixel (r, c) 4 5 6 7 1/9 1/9 1/9 x = 8 1/9 1/9 1/9 1/9 1/9 1/9 9 Input pixels Weight matrix c r

Automatic Feature Extraction Parameters to set Size of the matrix (convolution mask) The weight values The region (centered location) How to set them? Whether there is a dark area around the left eye region? 3*3 weight matrix, 1/9 for all weights, left eye region Whether there is a horizontal edge around the left eyebrow region?

Automatic Feature Extraction Applying a convolution mask (weight matrix) to a region generates a feature value The mean value of the left-eye region is dark -> there is a “left eye” Apply a convolution mask to the entire image generates a feature map

Automatic Feature Extraction A feature map can be seen as an enhanced (processed) image by applying some convolution mask to the original image What parameters to set for generating a feature map?

Number of Parameters in CNN Any parameter missing here? c r

How many parameters are there for this feature map? 64 hidden nodes 16 * 16 256 input nodes Receptive field 5 * 5 weight matrix

How many unique parameters are there for this feature map? 64 hidden nodes 16 * 16 256 input nodes Receptive field 5 * 5 weight matrix

Number of Parameters in CNN Calculate the size of feature map (output) from Input image size 𝑟𝑜 𝑤 𝑖𝑛 ×𝑐𝑜 𝑙 𝑖𝑛 Filter (Weight matrix) size 𝑟𝑜 𝑤 𝑓𝑖𝑙𝑡𝑒𝑟 ×𝑐𝑜 𝑙 𝑓𝑖𝑙𝑡𝑒𝑟 Number of pixels for each shift 𝑟𝑜 𝑤 𝑠ℎ𝑖𝑓𝑡 ×𝑐𝑜 𝑙 𝑠ℎ𝑖𝑓𝑡 Example: 16*16 input, 5*5 filter, 2*2 pixels per shift, what is the size of the feature map?

Multiple Feature Maps Different weight matrices and biases to extract different feature maps How many parameters in total? How many unique parameters in total? 8 * 8 feature map 64 hidden nodes 16 * 16 256 input nodes Feature map 1 … Feature map 12

CNN for Handwritten Digit Recognition Proposed by LeCun et al. in 1989

… 4 * 4 16 hidden nodes 5 * 5 8 * 8 wm1 64 hidden nodes 16 * 16 256 input nodes Feature map 1 5 * 5 wm8 … Feature map 12

… 4 * 4 16 hidden nodes 5 * 5 8 * 8 wm1 64 hidden nodes 16 * 16 256 input nodes Feature map 1 5 * 5 wm8 … Feature map 12

… … 4 * 4 16 hidden nodes 8 * 8 64 hidden nodes 16 * 16 256 input nodes Feature map 1 … … Feature map 12

CNN for Handwritten Digit Recognition

Subsampling Instead of aggregating the input values with weighted sum, we simply pick the maximal value Usually 2 x 2 mask Compress the image while keeping the key features c r

Design of CNN Number of hidden layers Type of each layer Input, output, fully connected, feature map, subsampling, … Configuration of fully connected layer Number of nodes Configuration of feature map layer Number of feature maps (weight matrices) Weight matrix size (3x3, 5x5, …) Shift size (every one pixel, every two pixels, …) Connection Configuration of subsampling layer Pool size (2x2, …)

Complete CNN Example

Weight Training in CNN BP algorithm Speed up techniques Feedforward: each node forward to its successors (not all nodes in the next layer) Back error propagation: each node back to its predecessors (not all the nodes in the previous layer) Speed up techniques Momentum Fan-in factor …

Hinton Diagram Visualise the weight matrix Size of square  value’s magnitude Color (black/white)  value’s sign (positive/negative)

Summary CNN for image processing Use raw pixels and automated feature extraction Domain knowledge about neighbourhood of pixels: shared weights Number of parameters in total vs Number of unique parameters