1. Matthias Wimmer, Bernd Radig, Michael Beetz Chair for Image Understanding Computer Science TU München, Germany matthias.wimmer@in.tum.de A Person and Context Specific Approach for Skin Colour Classification

2. 2005-12-18 2/9 Technische Universität München Matthias Wimmer Motivation motivation our approach results outlook see: A.E. Broadhurst, S. Baker: Setting Low-Level Vision Parameters, CMU-RI-TR-04-20, Robotics Institute, Carnegie Mellon University, 2004. Our scenario: Adaptive Skin Colour Classification Classifier adapts to person and context simple way: promote parameters: high-level to low-level vision module mathematically transform parameters

3. 2005-12-18 3/9 Technische Universität München Matthias Wimmer Motivation motivation our approach results outlook

4. 2005-12-18 4/9 Technische Universität München Matthias Wimmer Observations Skin colour depends on image conditions: illumination: light source, light colour, shadow, shading,… camera: type, settings,… visible person: ethnic group, tan,… Skin colour occupies a large area within colour space Skin colour varies greatly between images. Skin colour varies slightly within an image. motivation our approach results outlook green red skin colour pixels (red) and other pixels (blue), static skin colour clusters (white), adaptive skin colour clusters (yellow) green red image 1image 2

5. 2005-12-18 5/9 Technische Universität München Matthias Wimmer Our Approach Offline step: learn a mask that extracts skin colour pixels specific for the face detector motivation our approach results outlook Online steps: Step 1: detect the image specific skin colour using the face detector using the skin colour mask Step 2: calculate the input parameters Step 3: adapt the skin colour classifier

6. 2005-12-18 6/9 Technische Universität München Matthias Wimmer Learn the Calculation Rules Gather many training images Manually annotate images with ground truth Learn calculation rules via machine learning techniques e.g. linear regression, neural networks, model trees, … motivation our approach results outlook specify these (ground truth) learn those

7. 2005-12-18 7/9 Technische Universität München Matthias Wimmer Results good robustness for coloured persons exact shape outline detection of facial parts: eyes, lips, brows,… correctly detected pixels: fixed parameters:90.4%74.8%40.2% adaptive parameters:97.5%87.5%97.0% improvement: 0.08 0.17 1.41 motivation our approach results outlook adaptive parameters fixed parameters original image

8. 2005-12-18 8/9 Technische Universität München Matthias Wimmer Outlook We will create further adaptive colour classifiers lip teeth eyes brows, hair … Preliminary results for lip colour classifier: motivation our approach results outlook original fixed adaptive

9. 2005-12-18 9/9 Technische Universität München Matthias Wimmer Thank you!

10. 2005-12-18 10/9 Technische Universität München Matthias Wimmer Motivation Skin colour detection supports… face model fitting mimic recognition person identification gaze estimation fatigue detection (e.g. vehicle) hand tracking gesture recognition action recognition supervising work challenge our approach results outlook

11. 2005-12-18 11/9 Technische Universität München Matthias Wimmer Challenge Skin colour depends on image conditions: illumination: light source, light colour, shadow, shading,… camera: type, settings,… visible person: ethnic group, tan,… Skin colour occupies a large area within colour space challenge our approach results outlook

12. 2005-12-18 12/9 Technische Universität München Matthias Wimmer Challenge (2): non-skin colour pixels Skin colour pixels have to be separated from non- skin colour pixels. Areas of skin colour and non-skin colour overlap. Colour can not make a distinctive separation. challenge our approach results outlook

13. 2005-12-18 13/9 Technische Universität München Matthias Wimmer Our approach Offline step: learn the skin colour mask specific for the face detector Online steps: Step 1: detect the image specific skin colour model using the face detector using the skin colour mask Step 2: adapt a skin colour classifier Step 3: calculate the skin colour image challenge our approach results outlook

14. 2005-12-18 14/9 Technische Universität München Matthias Wimmer Offline: Learn the skin colour mask face image database with labeled skin colour pixels skin colour mask: array with 24 x 24 cells Computational steps: detect the face in every image every cell is assigned the relative number of labeled skin colour pixels at its position apply threshold 1. 2. 3. challenge our approach results outlook

15. 2005-12-18 15/9 Technische Universität München Matthias Wimmer Step 1: Detect the image specific skin colour model detect the face extract the skin colour pixels normalized RGB colour space: base= R + G + B r= R / base g= G / base skin colour model: mean values: μ r, μ g, μ base standard deviations: σ r, σ g, σ base challenge our approach results outlook

16. 2005-12-18 16/9 Technische Universität München Matthias Wimmer Step 2: Adapt a skin colour classifier non-adaptive skin colour classifier: skin :=0.35 r 0.5 0.2 g 0.7 200 base 740 adaptive skin colour classifier: skin :=low r r high r low g g high g low base base high base learn the bounds via the skin colour model mean value and standard deviation low r := μ r – 2σ r high r := μ r + 2σ r... linear function: low r := aμ r + bμ g + cμ base + dσ r + eσ g + fσ base + g... challenge our approach results outlook

17. 2005-12-18 17/9 Technische Universität München Matthias Wimmer Related work Feedback of information from high level vision components to low level vision components challenge our approach results outlook

18. 2005-12-18 18/9 Technische Universität München Matthias Wimmer Conclusion Challenge: much variation within skin colour illumination, camera, visible person skin colour occupies a large area within colour space We propose a way to reduce those variations exploit an image specific skin colour model adapt a skin colour classifier to that skin colour model We proved our approach using a simple but real-time capable skin colour classifier comparison: non-adaptive adaptive challenge our approach results outlook

19. 2005-12-18 19/9 Technische Universität München Matthias Wimmer Ongoing research Learn skin colour mask for other face detectors Specialize more powerful skin colour classifiers Recognize other feature images/colour images lip colour image tooth colour image eye colour image hair colour image eye brow colour image example: lip colour detection challenge our approach results outlook

20. 2005-12-18 20/9 Technische Universität München Matthias Wimmer Adaptive skin colour classifier non adaptive skin colour classifier: skin := 0.35 r 0.5 0.2 g 0.7 200 base 740 adaptive skin colour classifier: skin := low r r high r low g g high g low base base high base learn the bounds out of the skin colour model