1. Lip Feature Extraction Using Red Exclusion Trent W. Lewis and David M.W. Powers Flinders University of SA VIP2000

2. 01/12/2000Lip Feature Extraction Using Red Exlucsion 2 Overview Context Lip Feature Extraction –Related Work (greyscale, horizontal edges, red and hue colour spaces) –Red Exclusion AVSR: Results and Issues Summary

3. 01/12/2000Lip Feature Extraction Using Red Exlucsion 3 Context Audio Speech Recognition (ASR) Psycholinguistic Research Audio Visual Speech Recognition (AVSR)

4. 01/12/2000Lip Feature Extraction Using Red Exlucsion 4 Context - ASR Up to 99% word accuracy However, –limited context –limited vocabulary –trained on individual –close microphone, cannot handle noise

5. 01/12/2000Lip Feature Extraction Using Red Exlucsion 5 Context - Psycholinguistic McGurk Effect –A[ba] + V[ga]  [da] Viseme –visual phonemes –form complementary sets Demonstrates vision can assist the perception of speech  AVSR

6. 01/12/2000Lip Feature Extraction Using Red Exlucsion 6 Context - AVSR Acoustic Features Visual Features –width –height –oral cavity Integration –Early –Late

7. 01/12/2000Lip Feature Extraction Using Red Exlucsion 7 Lip Feature Extraction Pixel-BasedModel

8. 01/12/2000Lip Feature Extraction Using Red Exlucsion 8 Lip Feature Extraction Pixel-BasedModel –raw pixels or minimal processing –retain linguistically relevant data –large amounts of data,  time –shift and lighting variant –normalisation and PCA

9. 01/12/2000Lip Feature Extraction Using Red Exlucsion 9 Lip Feature Extraction Pixel-BasedModel –reduced input to set of hand-crafted features –width, height, average intensity, etc. –less features,  time –model fitting,  time –lose linguistically relevant features

10. 01/12/2000Lip Feature Extraction Using Red Exlucsion 10 Lip Feature Extraction Pixel-BasedModel –feature extraction –Steps preprocess to enhance contrast locate mouth edges identify corners, height, and other key features train recogntion engine Our Approach

11. 01/12/2000Lip Feature Extraction Using Red Exlucsion 11 Lip Feature Extraction Database 123

12. 01/12/2000Lip Feature Extraction Using Red Exlucsion 12 Lip Feature Extraction Preprocessing Techniques –Grey-scale –Horizontal Edges –Red Analysis –Hue, Saturation, and Value (HSV) –Red Exclusion

13. 01/12/2000Lip Feature Extraction Using Red Exlucsion 13 Lip Feature Extraction Grey-scale –vertical position of mouth minimum row sum –threshold minimum row average of min and max of row –search for above threshold pixels

14. 01/12/2000Lip Feature Extraction Using Red Exlucsion 14 Lip Feature Extraction Grey-scale

15. 01/12/2000Lip Feature Extraction Using Red Exlucsion 15 Lip Feature Extraction Grey-scale

16. 01/12/2000Lip Feature Extraction Using Red Exlucsion 16 Lip Feature Extraction Horizontal Edges –high horizontal edge content –3x3, DY Prewitt operator

17. 01/12/2000Lip Feature Extraction Using Red Exlucsion 17 Lip Feature Extraction Horizontal Edges “Found” CornersBinary Image

18. 01/12/2000Lip Feature Extraction Using Red Exlucsion 18 Lip Feature Extraction Red Analysis –overcome bearded subjects –used for face location

19. 01/12/2000Lip Feature Extraction Using Red Exlucsion 19 Lip Feature Extraction Red Analysis

20. 01/12/2000Lip Feature Extraction Using Red Exlucsion 20 Lip Feature Extraction HSV –disentangles illumination from colour –  Illumination >  Hue

21. 01/12/2000Lip Feature Extraction Using Red Exlucsion 21 Lip Feature Extraction HSV

22. 01/12/2000Lip Feature Extraction Using Red Exlucsion 22 Lip Feature Extraction Red Exclusion –needed extraction method for AVSR –similar to Red Analysis face predominantly red variations occur in the blue and green colours

23. 01/12/2000Lip Feature Extraction Using Red Exlucsion 23 Lip Feature Extraction Red Exclusion

24. 01/12/2000Lip Feature Extraction Using Red Exlucsion 24 Lip Feature Extraction Corners found using Red Exclusion

25. 01/12/2000Lip Feature Extraction Using Red Exlucsion 25 Lip Feature Extraction Comparison

26. 01/12/2000Lip Feature Extraction Using Red Exlucsion 26 AVSR: Results and Issues Application for red exclusion Used in finding lip features –Width –Height –Key pixels

27. 01/12/2000Lip Feature Extraction Using Red Exlucsion 27 AVSR: Results and Issues Visual Speech Recognition Static (%)Dynamic (%) Voicing32.230.8 Viseme54.751.3 Phoneme14.713.6

28. 01/12/2000Lip Feature Extraction Using Red Exlucsion 28 AVSR: Results and Issues AVSR - Integration Early Static Early Dynamic Late Voice/Vis Late Error Voice/Vis Phoneme 20.118.029.019.5

29. 01/12/2000Lip Feature Extraction Using Red Exlucsion 29 Summary Vision can help ASR –AVSR Needed good extraction technique –Red Exclusion AVSR is difficult when both signals degraded

30. 01/12/2000Lip Feature Extraction Using Red Exlucsion 30 Questions? Trent W. Lewis BSc (Cognitive Science) Flinders University lewi0146@ist.flinders.edu.au