Facial Expression Recognition By: Stephanie Tsai Nazia Hashmi Michelle Aleong

What is Facial Expression Recognition? Facial Expression Recognition has been defined as the biometric identification by scanning a person’s face and matching it against a library of faces Process by which the brain and mind understand and interpret the human face

Why Facial Expression? Behavioral assessment of emotion and paralinguistic displays Facial nerve disorders Computer systems that understand human behavior Speech recognition. Security systems. Lie detection. Video compression in telecommunications. Emotion for animation.

FACS Facial Action Coding System Most widely used method for measuring and describing facial behaviors Explains how to categorize facial behaviors based on the muscles that produce them Goal is to create a reliable means for skilled human scorers to determine the category in which to fit each facial behavior.

Once upon a time… Developed by Paul Ekman (UCSF) & William Friesen of Langley Porter Neuropsychiatric Institute in San Francisco in 1978 Current computer programs being developed at the University of Pittsburgh and Carnegie Mellon University, the other by a team at the Salk Institute in La Jolla, California

Action Units Diagram Action UnitDescriptionFacial MuscleExample Image 1Inner Brow Raiser Frontalis, pars medialis 2Outer Brow Raiser Frontalis, pars lateralis 3Brow Lowerer Corrugator supercilii 5Upper Lid Raiser Levator palpebrae superioris 10Upper Lip Raiser Levator labii superioris 25Lips Part Depressor labii inferioris AU Descriptio n Facial muscl e Example image 1 Inner Brow Raiser Frontalis, pars medialis 2 Outer Brow Raise r Frontalis, pars later alis 4 Brow Lowe rer Corrugat or super cilii, Depr essor super cilii 5 Upper Lid Raise r Levator palpe brae super ioris 6 Cheek Raise r Orbicula ris oculi, pars orbit alis 7 Lid Tight ener Orbicula ris oculi, pars palpe brali s 9 Nose Wrin kler Levator labii super ioris alaqu ae nasi 10 Upper Lip Raise r Levator labii super ioris 11 Nasolabi al Deep ener Zygomati cus mino r 12 Lip Corn er Pulle r Zygomati cus majo r 13 Cheek Puffe r Levator angul i oris (a.k.a. Cani nus) 14 Dimpler Buccinat or 15 Lip Corn er Depr essor Depresso r angul i oris ( a.k.a. Trian gular is) 16 Lower Lip Depr essor Depresso r labii inferi oris 17 Chin Raise r Mentalis 18 Lip Puck erer Incisivii labii super ioris and Incisi vii labii inferi oris 20 Lip stretc her Risorius w/ platy sma 22 Lip Funn eler Orbicula ris oris 23 Lip Tight ener Orbicula ris oris 24 Lip Press or Orbicula ris oris 25 Lips part* * Depresso r labii inferi oris or relax ation of Ment alis, or Orbi cular is oris 26 Jaw Drop Masseter, relax ed Temp orali s and inter nal Ptery goid 27 Mouth Stret ch Pterygoi ds, Diga stric 28 Lip Suck Orbicula ris oris 41 Lid droo p** Relaxati on of Levat or palpe brae super ioris 42 Slit Orbicularis oculi 43 Eyes Close d Relaxati on of Levat or palpe brae super ioris; Orbi cular is oculi, pars palpe brali s 44 Squint Orbicula ris oculi, pars palpe brali s 45 Blink Relaxati on of Levat or palpe brae super ioris; Orbi cular is oculi, pars palpe brali s 46 Wink Relaxati on of Levat or palpe brae super ioris; Orbi cular is oculi, pars palpe brali s 51 Head turn left 52 Head turn right 53 Head up 54 Head down 55 Head tilt left 56 Head tilt right 57 Head forw ard 58 Head back 61 Eyes turn left 62 Eyes turn right 63 Eyes up 64 Eyes down

How it Works Action Units (AUs) are the measurement units of FACS 44 AUs FAC coder “dissects” the expression and decomposes it into the specific AUs that produce the movement

Scoring The scores consist of the list of AUs that produce it Descriptive only AU

Problems with FACS Human-observer based methods for measuring facial expression are labor intensive, qualitative, and difficult to standardize. Less than 100% inter observer reliability

Superman (aka. the computer) to the rescue!! Goal is to make feasible more rigorous, quantitative measurement of facial expression in diverse applications Computers can recognize specific action units Unbiased based on person's gender, race or age.

Automated Face Analysis Training data on group of more than 200 people of different racial and ethnic backgrounds.  “The hardest and most time-consuming part of all this work is collecting a database of images that is diverse enough and big enough to train the computer," says Sejnowski. 3-generation system developed at CMU

Generation I

Generation 2

Generation 3

Current Research Competitions to explore the different methods to analyze the expressions from the same set of videos Research unit ongoing at CMU Department of Computer Science

Are you ready to have a computer know what you’re feeling?