Feature Detection and Emotion Recognition Chris Matthews Advisor: Prof. Cotter

Motivation #1: Attempt to Answer a Long-Existing Question Used to definitively characterize what expressions the Mona Lisa is displaying (F.Y.I., she is 83% happy, 9% disgusted, 6% fearful and 2% angry, according to BBC News) Used to definitively characterize what expressions the Mona Lisa is displaying (F.Y.I., she is 83% happy, 9% disgusted, 6% fearful and 2% angry, according to BBC News)

Motivation #2: Create “Life-Like” Robots Create convincing artificial intelligence. Create convincing artificial intelligence.

Motivation #3: Enhance Society! Currently being used to teach autistic children to pick out facial subtleties and their corresponding emotions Currently being used to teach autistic children to pick out facial subtleties and their corresponding emotions

Methodology FEATURE DETECTION Isolate and crop particular areas of the face Isolate and crop particular areas of the face EMOTION RECOGNITION Training Train neural networks for each area Train neural networks for each area Combine the resultants from each and come out with a definitive result Combine the resultants from each and come out with a definitive result Alter variables of the networks by trial-and-error until the desired results are achieved Alter variables of the networks by trial-and-error until the desired results are achievedTesting Input new photos into the trained network and check results Input new photos into the trained network and check results

Feature Detection: SUSAN filtering for Edge Detection Because no derivatives are implemented in SUSAN, the algorithm excels in noisy images Because no derivatives are implemented in SUSAN, the algorithm excels in noisy images

Massive Problem: Boolean images don’t necessarily make Computer Vision problems easier! Mouth not fully enclosed Mouth not fully enclosed Only the pupil of the left eye is enclosed Only the pupil of the left eye is enclosed Even if everything was perfectly encapsulated, how would one make sense of the detected objects? Even if everything was perfectly encapsulated, how would one make sense of the detected objects?

Lesson Learned: Complete Automation is difficult!  New methodology for isolating parts of the face – manual labor.  Draw matrices over the approximate area of interest  Apply filters to detect the actual object of interest  Crop again based on those findings

Example: The Uncentered Eye The neural network will perform poorly if there is variance in either the x or y directions, from photo to photo The neural network will perform poorly if there is variance in either the x or y directions, from photo to photo

Voila!

On to the Emotion Training… Once the areas have been defined and scaled, they can be used as inputs to neural networks Once the areas have been defined and scaled, they can be used as inputs to neural networks

Introduction to Neural Networks: The Perceptron

Perceptron Implementation Initialize weight matrix and bias array to small, random values. Initialize weight matrix and bias array to small, random values. Feed an image through the network Feed an image through the network Calculate the error Calculate the error Readjust the weight matrix and bias array based on the error Readjust the weight matrix and bias array based on the error Iteratively train the network using a dictionary of photos. Iteratively train the network using a dictionary of photos.

Yet another problem! Each neuron has one weight value for each pixel Each neuron has one weight value for each pixel Weight matrix is too large to train! Weight matrix is too large to train!

Solution: PCA Principle Component Analysis generates a set of eigenvectors. Principle Component Analysis generates a set of eigenvectors. Each picture can be reconstructed using a weighted sum of these eigenvectors. Each picture can be reconstructed using a weighted sum of these eigenvectors.

Final Architecture Use a set of adaptive backpropagation networks, training on PCA coefficients. Use a set of adaptive backpropagation networks, training on PCA coefficients. Use majority rules to determine the emotion. Use majority rules to determine the emotion.

Results Training with 60 photos yielded 100% accuracy mapping to only two targets: happy and sad Training with 60 photos yielded 100% accuracy mapping to only two targets: happy and sad Training with 112 photos yielded 60% accuracy mapping to four targets: angry, fearful, happy, and sad. Training with 112 photos yielded 60% accuracy mapping to four targets: angry, fearful, happy, and sad.

Future Work Find larger and more diverse image dictionaries Find larger and more diverse image dictionaries Improve Feature Detection Improve Feature Detection Read Psychological Journals and apply their findings into the algorithms Read Psychological Journals and apply their findings into the algorithms

Questions?

A gross simplification of how SUSAN works Smallest Univalue Segment Assimilating Nucleus Smallest Univalue Segment Assimilating Nucleus Edge if n = (½)*pi*r^2 Edge if n = (½)*pi*r^2 Corner if n << (½)*pi*r^2 Corner if n << (½)*pi*r^2