Matthias Wimmer, Bernd Radig, Michael Beetz Chair for Image Understanding Computer Science Technische Universität München Adaptive Skin Color Classification

/10 Technische Universität München Matthias Wimmer Motivation Skin color detection supports…  face model fitting  mimic recognition  person identification  gaze estimation  fatigue detection (e.g. vehicle)  hand tracking  gesture recognition  action recognition  supervising work challenge our approach results outlook

/10 Technische Universität München Matthias Wimmer Challenge  Skin color depends on image conditions:  illumination: light source, light color, shadow, shading,…  camera: type, settings,…  visible person: ethnic group, tan,…  Skin color occupies a large area within color space challenge our approach results outlook

/10 Technische Universität München Matthias Wimmer Observations  Skin color varies greatly between images.  Skin color varies slightly within an image. Basic idea  learn image specific skin color characteristics  parameterize a skin color classifier accordingly skin color of image1 skin color of image2 challenge our approach results outlook

/10 Technische Universität München Matthias Wimmer Our approach Offline step:  learn the skin color mask  specific for any face detector Online steps:  Step 1: detect the image specific skin color model  using the face detector  using the skin color mask  Step 2: adapt a skin color classifier  Step 3: calculate the skin color image challenge our approach results outlook

/10 Technische Universität München Matthias Wimmer Skin color model & skin color classifier  detect the face  extract the skin color pixels  skin color model:  mean values: μ r, μ g, μ base  standard deviations: σ r, σ g, σ base  adaptive skin color classifier: skin :=low r ≤ r ≤ high r  low g ≤ g ≤ high g  low base ≤ base ≤ high base  learn the bounds low r := μ r – 2σ r high r := μ r + 2σ r... challenge our approach results outlook

/10 Technische Universität München Matthias Wimmer Results  better results for  colored persons  exact shape outline  detection of facial parts: eyes, lips, brows,…  correctly detected pixels:  non-adaptive approach:90.4%74.8%40.2%  adaptive approach:97.5%87.5%97.0%  improvement:7.9%17.0%141.3% challenge our approach results outlook

/10 Technische Universität München Matthias Wimmer Conclusion  Challenge: much variation within skin color  illumination, camera, visible person  skin color occupies a large area within color space  We propose a way to reduce those variations  exploit an image specific skin color model  adapt a skin color classifier to that skin color model  We proved our approach  using a simple but real-time capable skin color classifier  comparison: non-adaptive ↔ adaptive challenge our approach results outlook

/10 Technische Universität München Matthias Wimmer Ongoing research  Learn skin color mask for other face detectors  Specialize more powerful skin color classifiers  Recognize other feature images/color images  lip color image  tooth color image  eye color image  hair color image  eye brow color image example: lip color detection challenge our approach results outlook

/10 Technische Universität München Matthias Wimmer Thank you !

/10 Technische Universität München Matthias Wimmer Overview  Motivation / Challenge  Our approach  extracting skin color pixels  adapt skin color classifier  Results  Conclusion  Outlook

/10 Technische Universität München Matthias Wimmer Challenge (2): non-skin color pixels  Skin color pixels have to be separated from non- skin color pixels.  Areas of skin color and non-skin color overlap.  Color can not make a distinctive separation. challenge our approach results outlook

/10 Technische Universität München Matthias Wimmer Basic idea  learn image specific skin color characteristics  parameterize classifier accordingly with those characteristics

/10 Technische Universität München Matthias Wimmer Offline: Learn the skin color mask  face image database with labeled skin color pixels  skin color mask: array with 24 x 24 cells Computational steps:  detect the face in every image  every cell is assigned the relative number of labeled skin color pixels at its position  apply threshold challenge our approach results outlook

/10 Technische Universität München Matthias Wimmer Step 1: Detect the image specific skin color model  detect the face  extract the skin color pixels  normalized RGB color space: base= R + G + B r= R / base g= G / base  skin color model:  mean values: μ r, μ g, μ base  standard deviations: σ r, σ g, σ base challenge our approach results outlook

/10 Technische Universität München Matthias Wimmer Step 2: Adapt a skin color classifier  non-adaptive skin color classifier: skin :=0.35 ≤ r ≤ 0.5  0.2 ≤ g ≤ 0.7  200 ≤ base ≤ 740  adaptive skin color classifier: skin :=low r ≤ r ≤ high r  low g ≤ g ≤ high g  low base ≤ base ≤ high base  learn the bounds via the skin color model  mean value and standard deviation low r := μ r – 2σ r high r := μ r + 2σ r...  linear function: low r := aμ r + bμ g + cμ base + dσ r + eσ g + fσ base + g... challenge our approach results outlook

/10 Technische Universität München Matthias Wimmer Adaptive skin color classifier  non adaptive skin color classifier: skin := 0.35 ≤ r ≤ 0.5  0.2 ≤ g ≤ 0.7  200 ≤ base ≤ 740  adaptive skin color classifier: skin := low r ≤ r ≤ high r  low g ≤ g ≤ high g  low base ≤ base ≤ high base  learn the bounds out of the skin color model

/10 Technische Universität München Matthias Wimmer Related work  Feedback of information from high level vision components to low level vision components challenge our approach results outlook