+ Sridhar Godavarthy July 01, 2010 Defense of a Masters Thesis Computer Science and Engineering University of South Florida

Microexpression Spotting in Video Using Optical Strain Sridhar Godavarthy Examining Committee Dmitry B. Goldgof, Ph.D. – Major Professor Sudeep Sarkar, Ph.D. Rangachar Kasturi, Ph.D. July 01, 2010 Defense of a Masters Thesis Computer Science and Engineering University of South Florida

+ Minutes of the presentation Microexpressions - “micro” expressions. Goal: Detect “interesting” sequences containing μE. Approach: optical flow + strain thresholding. Result: True positive detection as high as 80%. Good performance on real time videos. Conclusion: Novel system. Scope for improvement. Need more datasets.

+ Introduction

+ Expressions Social emotion conveyance Non verbal Voluntary or involuntary 6 primary expressions 5 Anger Disgust Fear Happiness Sadness Surprise

+ Microexpressions – What? Subtle movements of the human body Observable Insufficient to convey emotion Masking an expression 1/25 th to 1/5 th of a second Almost impossible to suppress 07/01/2010 6

+ Why? Lie Detection Pain detection for autistic and anaesthetized patients Social signal processing( boredom/ concentration detection) Psychological counseling. 07/01/2010 7

+ State of the art 07/01/ Microexpression Research Psychology Vision IntentFACS classification Optical Flow Gabor Filters ANNs Rule Based

+ Objective Design a preprocessing system that Spots microexpressions. Handles small translational and rotational motion Improves performance of existing systems Greater weight to true positives. 07/01/2010 9

+ Some Fundamentals Optic Flow : Vector representation of temporal changes Strain: Relative deformation of material (skin) Haar Classifier / Viola-Jones Face detector Cascade of weak classifiers OpenCV implementation Uses Haar rectangular features

+ Brief overview of algorithm ( Main Idea) Skin deforms during an expression. Deformation peaks at peak of expression Duration of increased strain corresponds to microexpression 07/01/ ~22 frames ~5 frames Peak DetectionThresholding Frames  Strain Magnitude  Macro Expression Micro Expression

+ Algorithm

+ System Flow Split Frames Face Detection & Alignment Optical Flow Strain MapSplit into regions Strain patterns and Thresholding

+ Face detection Viola-Jones face detector -OpenCV implementation

+ Face Alignment: Rotation 07/01/

+ System Flow Split Frames Face Detection & Alignment Optical Flow Strain MapSplit into regions Strain patterns and Thresholding

+ Optical flow 07/01/ Black and Anandan Dense MJ Black’s Matlab imlementation of OF

+ System Flow Split Frames Face Detection & Alignment Optical Flow Strain MapSplit into regions Strain patterns and Thresholding

+ Facial strain 07/01/

+ System Flow Split Frames Face Detection & Alignment Optical Flow Strain MapSplit into regions Strain patterns and Thresholding

+ Region Splitting forehead(fh) right cheek (rc) left cheek (lc) right mouth(rm) left mouth(lm) right eye(re) below mouth(bm) AUs not covered Blink Close eyes Neck tightening Nostril flare Automated. M anual intervention if classifier fails

+ Datasets & Results

+ Datasets USF: IRBCanal9: EULAFound Videos: Fair use act USF (100) Canal9 (24) Found (4)

+ Threshold Determination Sl. No. Threshold as percentage of peak strain % True positives% False positives /01/

+ Thresholded Strain Maps – Sample 1 / 3 07/01/

+ Thresholded Strain Maps – Sample 2 / 3 07/01/

+ Thresholded Strain Maps – Sample 3 / 3 07/01/  False Positive ( Indicative only)

+ Results: Microexpression Spotting Dataset Name Number of sequences Number of Microexpressions % True positive %False Positive USF enacted USF questioning Canal Found videos Total /01/

+ Microexpression Spotting 07/01/

+ Negative Test Case – Rejects Expressions 07/01/

+ Concluding Remarks

+ Contributions and Conclusions Automated thresholding Automated alignment ( Partial ) Region wise detection Up to 80% true detection Microexpressions with expressions are detected.

+ Constraints Constant illumination Neutral face Some expressions may be falsely detected Talking

+ Future Work Dataset Collection Real time questioning videos Fully automated face alignment By matching optical flow vectors Automatic identification of neutral face Automatic portioning of faces Anthropomorphic landmark identification 07/01/

+ Related Publications Shreve, M., Godavarthy, S., Manohar, V., Goldgof, D., Sarkar, S., "Towards macro- and micro-expression spotting in video using strain patterns," Workshop on Applications of Computer Vision, 2009 pp:1-6 07/01/

+ Other Publications by author Candamo, J., Kasturi, R., Goldgof, D., Godavarthy, S., "Detecting Wires in Cluttered Urban Scenes Using a Gaussian Model, " to appear in Proceedings of International Conference on Pattern Recognition(ICPR 2010), Turkey, 2010 Godavarthy, S., Roomi, M. Md., “Adaptive Contrast Based Unsharp Masking,” in Proceedings of the National Workshop on Computer Vision, Graphics and Image Processing, Feb 2002 Godavarthy, S., Pandian, A., Roomi, M. Md., “Histogram Equalization by Measure of Enhancement,” in Proceedings of the National Workshop on Computer Vision, Graphics and Image Processing, Feb 2002 Godavarthy, S., Shankar, A., Roomi, M. Md., “Adaptive Watermarking-a FFT Approach”, Proceedings of International Conference on Advances in Telecommunication and Information Technology "Asia - Pacific Telecom 2000" (14th, 15th December 2000), Vellore 07/01/

+ Dr. Ekman on A-Rod

+ THANK YOU 38

+ Index Presentation: Minutes System Flow Thresholding Sample Strain Maps Results Negative Test Case Conclusion and Future Work Additional Slides Evolutionary psychology Detailed Flow Chart Optical Flow Elasticity and Strain FACS OF Vs OS Dataset Details

+ Additional Slides

+ Study of everything we discussed until now The child of ONE man - Paul Ekman. Over thirty years of research One of the world’s leading experts on lying. About 2 dozen books and innumerable articles Developed FACS Scientific Advisor to “Lie to Me” Co creator of Microexpression Training Tool (METTx) Evolutionary Psychology

+ Flow Chart 07/01/

+ Add: OF Motion Estimation: Optical Flow Method  Reflects the changes in the image due to motion  Computation is based on the following assumptions:  observed brightness of any object point is constant over time  nearby points in the image plane move in a similar manner  Minimization problem: (brightness const.) (smoothness const.)  Robust estimation framework (Black and Anandan, 1996)  Recast the least squared formulations with a different error-norm function instead of quadratic  Coarse-to-fine strategy  Construct a pyramid of spatially filtered and sub-sampled images  Compute flow values at lowest resolution and project to next level in the pyramid 43 10/29/2009

+ Def: Optical Flow is the apparent motion of brightness patterns in the image Ideally, same as the motion field Have to be careful: apparent motion can be caused by lighting changes without any actual motion Optical Flow Key assumptions Brightness constancy: projection of the same point looks the same in every frame Small motion: points do not move very far Spatial coherence: points move like their neighbors

Elasticity Different materials have different elasticity Elasticity can be modeled Known Calculate

+ What is Facial Strain? Strain on soft tissue when expressions are made. Anatomical method Uses a pair of frames to measure deformation Facial Strain

+ Finite Difference Method Compute spatial derivatives from discrete points. Forward Difference Method Central Difference Method Richardson extrapolation Strain Measurement

+ Thresholding Threshold Strain Maps to segment out μE 07/01/

+ The Facial Action Coding System (FACS) Coding of human expressions Observational and Anatomical 32 Action Units and 14 Action Descriptors Encode any possible [facial] expression. Also used for facial expression simulations

+ NameMethodType Conte nt AdvantagesDisadvantages FAST OTS Early methodOnly negative emotions FACS OAC  All muscles  Allows for discovery - MAX OTS Faster performance Only pre defined configurations. EMG ObtAC Muscular activity invisible to naked eye Interference from nearby muscles EMFACS OTS Faster performance. Only certain emotion expressions. O - Observational Obt - Obtrusive A – Anatomical T – Theoretical S – Selective C – Comprehensive

+ FACS examples 07/01/

+ Why Optical Strain? 07/01/

+ Datasets Dataset Name No. of Sequence s Approximat e Duration per sequence(s) Microexp ressions per sequenc e TotalResolution USF – feigned SD / HD USF – questioning 46514HD Canal9 dataset HD Found videos Very Low 07/01/

+ Index Presentation: Minutes System Flow Thresholding Sample Strain Maps Results Negative Test Case Conclusion and Future Work Additional Slides Evolutionary psychology Detailed Flow Chart Optical Flow Elasticity and Strain FACS OF Vs OS Dataset Details