Lip Feature Extraction Using Red Exclusion Trent W. Lewis and David M.W. Powers Flinders University of SA VIP2000

01/12/2000Lip Feature Extraction Using Red Exlucsion 2 Overview Context Lip Feature Extraction –Related Work (greyscale, horizontal edges, red and hue colour spaces) –Red Exclusion AVSR: Results and Issues Summary

01/12/2000Lip Feature Extraction Using Red Exlucsion 3 Context Audio Speech Recognition (ASR) Psycholinguistic Research Audio Visual Speech Recognition (AVSR)

01/12/2000Lip Feature Extraction Using Red Exlucsion 4 Context - ASR Up to 99% word accuracy However, –limited context –limited vocabulary –trained on individual –close microphone, cannot handle noise

01/12/2000Lip Feature Extraction Using Red Exlucsion 5 Context - Psycholinguistic McGurk Effect –A[ba] + V[ga]  [da] Viseme –visual phonemes –form complementary sets Demonstrates vision can assist the perception of speech  AVSR

01/12/2000Lip Feature Extraction Using Red Exlucsion 6 Context - AVSR Acoustic Features Visual Features –width –height –oral cavity Integration –Early –Late

01/12/2000Lip Feature Extraction Using Red Exlucsion 7 Lip Feature Extraction Pixel-BasedModel

01/12/2000Lip Feature Extraction Using Red Exlucsion 8 Lip Feature Extraction Pixel-BasedModel –raw pixels or minimal processing –retain linguistically relevant data –large amounts of data,  time –shift and lighting variant –normalisation and PCA

01/12/2000Lip Feature Extraction Using Red Exlucsion 9 Lip Feature Extraction Pixel-BasedModel –reduced input to set of hand-crafted features –width, height, average intensity, etc. –less features,  time –model fitting,  time –lose linguistically relevant features

01/12/2000Lip Feature Extraction Using Red Exlucsion 10 Lip Feature Extraction Pixel-BasedModel –feature extraction –Steps preprocess to enhance contrast locate mouth edges identify corners, height, and other key features train recogntion engine Our Approach

01/12/2000Lip Feature Extraction Using Red Exlucsion 11 Lip Feature Extraction Database 123

01/12/2000Lip Feature Extraction Using Red Exlucsion 12 Lip Feature Extraction Preprocessing Techniques –Grey-scale –Horizontal Edges –Red Analysis –Hue, Saturation, and Value (HSV) –Red Exclusion

01/12/2000Lip Feature Extraction Using Red Exlucsion 13 Lip Feature Extraction Grey-scale –vertical position of mouth minimum row sum –threshold minimum row average of min and max of row –search for above threshold pixels

01/12/2000Lip Feature Extraction Using Red Exlucsion 14 Lip Feature Extraction Grey-scale

01/12/2000Lip Feature Extraction Using Red Exlucsion 15 Lip Feature Extraction Grey-scale

01/12/2000Lip Feature Extraction Using Red Exlucsion 16 Lip Feature Extraction Horizontal Edges –high horizontal edge content –3x3, DY Prewitt operator

01/12/2000Lip Feature Extraction Using Red Exlucsion 17 Lip Feature Extraction Horizontal Edges “Found” CornersBinary Image

01/12/2000Lip Feature Extraction Using Red Exlucsion 18 Lip Feature Extraction Red Analysis –overcome bearded subjects –used for face location

01/12/2000Lip Feature Extraction Using Red Exlucsion 19 Lip Feature Extraction Red Analysis

01/12/2000Lip Feature Extraction Using Red Exlucsion 20 Lip Feature Extraction HSV –disentangles illumination from colour –  Illumination >  Hue

01/12/2000Lip Feature Extraction Using Red Exlucsion 21 Lip Feature Extraction HSV

01/12/2000Lip Feature Extraction Using Red Exlucsion 22 Lip Feature Extraction Red Exclusion –needed extraction method for AVSR –similar to Red Analysis face predominantly red variations occur in the blue and green colours

01/12/2000Lip Feature Extraction Using Red Exlucsion 23 Lip Feature Extraction Red Exclusion

01/12/2000Lip Feature Extraction Using Red Exlucsion 24 Lip Feature Extraction Corners found using Red Exclusion

01/12/2000Lip Feature Extraction Using Red Exlucsion 25 Lip Feature Extraction Comparison

01/12/2000Lip Feature Extraction Using Red Exlucsion 26 AVSR: Results and Issues Application for red exclusion Used in finding lip features –Width –Height –Key pixels

01/12/2000Lip Feature Extraction Using Red Exlucsion 27 AVSR: Results and Issues Visual Speech Recognition Static (%)Dynamic (%) Voicing Viseme Phoneme

01/12/2000Lip Feature Extraction Using Red Exlucsion 28 AVSR: Results and Issues AVSR - Integration Early Static Early Dynamic Late Voice/Vis Late Error Voice/Vis Phoneme

01/12/2000Lip Feature Extraction Using Red Exlucsion 29 Summary Vision can help ASR –AVSR Needed good extraction technique –Red Exclusion AVSR is difficult when both signals degraded

01/12/2000Lip Feature Extraction Using Red Exlucsion 30 Questions? Trent W. Lewis BSc (Cognitive Science) Flinders University