Proceedings of the British Machine Vision Conference (BMVC), 2010 Qi Wang, Xilin Chen, Wen Gao Skin Color Weighted Disparity Competition for Hand Segmentation from Stereo Camera

Outline  Introduction  The Proposed Method  Mark hand pixels  Skin Color Weighted Disparity Competition  Experimental Results  Conclusion 2

Introduction 3  Human computer interaction (HCI)  Robustness  Accuracy  Efficiency  Hand segmentation and hand gesture  Clutter background  Lighting variation  Flexibility or concave character of the hand

Hand Segmentation 4 Monocular camera  Skin color detection [3,7,14]  Background subtraction [11,16]  Active contour [21]  Model-based method [5,17,18,19]  Interactive segmentation [4,10,15] Stereo camera [8,12]  Skin color weighted disparity competition

Reference 5 [3] A. Argyros and M. Lourakis. Real-time tracking of multiple skin-colored objects with a possibly moving camera. In European Conference on Computer Vision, [7] W.Y. Chang, C.S. Chen, and Y.P. Hung. Appearance-guided particle filtering for articulated hand tracking. In IEEE Computer Society Conference on Computer Vision and Pattern Recognition, [8] Q. Delamarre and O. Faugeras. 3d articulated models and multiview tracking with physical forces. Computer Vision and Image Understanding, [11] J. Letessier and F. Berard. Visual tracking of bare fingers for interactive surfaces. 17th Annual ACM symposium on User Interface Software and Technology, ACM. [12] C. Manders, F. Farbiz, J. H. Chong, K. Y. Tang, G. G. Chua, M. H. Loke, and M. L.Yuan. Robust hand tracking using a skin tone and depth joint probability model. In IEEE International Conference on Automatic Face & Gesture Recognition,2008. [14] Z. Mo, J.P. Lewis, and U. Neumann. Smartcanvas: a gesture-driven intelligent drawing desk system. 10th International Conference on Intelligent User Interfaces,ACM, [16] M. Shahzad and L. Joe. Visual touchpad: a two-handed gestural input device. In Proceedings of the 6th international conference on Multimodal interfaces, ACM. [21] L.G. Zhang, X. Chen, C.Wang, andW. Gao. Robust automatic tracking of skin-colored objects with level set based occlusion handling. In Proceedings of The 8th Int’l Gesture Workshop, GW’09, Bielefeld, Germany, [17] B. Stenger, P. R. S. Mendonca, and R. Cipolla. Model-based 3d tracking of an articulated hand. In IEEE Computer Society Conference on Computer Vision and Pattern Recognition, [18] B. Stenger, A. Thayananthan, P. H. S. Torr, and R. Cipolla. Model-based hand tracking using a hierarchical bayesian filter. IEEE Transactions on Pattern Analysis and Machine Intelligence, [19] Y. Wu, J. Lin, and T. S. Huang. Analyzing and capturing articulated hand motion in image sequences. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2005.

The Proposed Method 6

Marking the Confident Hand Pixels 7 1 Computing disparity map 2 Detecting face 3 Finding human body regions 4 Marking the confident hand pixels 5 Camshift algorithm

Marking the Confident Hand Pixels 8  Computing disparity map  Gray images  Block matching  Zero : occlusion  Detecting face  Finding human body regions  Marking the confident hand pixels  Camshift algorithm

Marking the Confident Hand Pixels 9  Computing disparity map  Detecting face [20]  Locally Assembled Binary (LAB) Haar feature  Average disparity as threshold  Foreground(FG) map and background(BG) map  Finding human body regions  Marking the confident hand pixels  Camshift algorithm [20] S. Yan, S. Shan, X. Chen, and W. Gao. Locally assembled binary (lab) feature with feature-centric cascade for fast and accurate face detection. In IEEE Conference on Computer Vision and Pattern Recognition, pages 1–7, 2008.

Locally Assembled Binary (LAB) Haar feature 10  Haar feature (OpenCV)  An Integral Image for rapid feature detection  * *wikipedi

Locally Assembled Binary (LAB) Haar feature 11

Marking the Confident Hand Pixels 12  Computing disparity map  Detecting face [20]  Locally Assembled Binary (LAB) Haar feature  Average disparity as threshold  Foreground(FG) map and background(BG) map  Finding human body regions  Marking the confident hand pixels  Camshift algorithm [20] S. Yan, S. Shan, X. Chen, and W. Gao. Locally assembled binary (lab) feature with feature-centric cascade for fast and accurate face detection. In IEEE Conference on Computer Vision and Pattern Recognition, pages 1–7, 2008.

Marking the Confident Hand Pixels 13  Computing disparity map  Detecting face  Finding human body regions  Union zero pixels in disparity map and FG map  Fill the wholes (occlusion)  Marking the confident hand pixels  Color close to the detected face (Hue-Saturation Histogram)  Disparity fall in local maxima of the FG map  Camshift algorithm

Marking the Confident Hand Pixels 14  Computing disparity map  Detecting face  Finding human body regions  Marking the confident hand pixels  Camshift algorithm  Histogram

Segmenting the Hand with Skin Color Weighted Disparity Competition 15 Support reasons  The hand disparity changes slightly.  The hand disparity is different from background disparity.  The hand is an enough simple object from color aspect.

Segmenting the Hand with Skin Color Weighted Disparity Competition 16 1 The skin color probability 2 The two competing disparity limits 3 Skin color weighted disparity competition

Segmenting the Hand with Skin Color Weighted Disparity Competition 17  The skin color probability.  HSV color space  Gaussian model   : the mean HSV vector of all marked hand pixels  : the standard deviation of the H, S or V HSV

Segmenting the Hand with Skin Color Weighted Disparity Competition 18  The two competing disparity limits   h: hand ; b: background  : the maximal bias (typically set to 2 or 1 respectively)  : : the mean disparity of all marked confident hand pixels  : the disparity of background (Watershed Algorithm)

Segmenting the Hand with Skin Color Weighted Disparity Competition 19  Skin color weighted disparity competition  Disparity verification  Skin probability  N: neighborhood size (typically set to 2)  ● N

Segmenting the Hand with Skin Color Weighted Disparity Competition  : Validate a hand pixel by competition  : less value means more satisfying  Ts : threshold for distinguishing the skin color from the non-skin color  a : a small tuning constant 20

Segmenting the Hand with Skin Color Weighted Disparity Competition 21 DC means disparity competition DLOH means the disparity limit of the hand DLOB means the disparity limit of the background

Experimental Results  Implemented in C++ by using OpenCV and the Triclops Stereo Vision SDK.  Video device: the Bumblebee 2 stereo camera  The test sequence contains 225 frames with the resolution of 1024x384 (512x384) 22

Experimental Results 23  Parameter  Skin color probability  Skin color weighted disparity competition

Experimental Results 24 Triclops Stereo Vision SDK

Experimental Results 25

Experimental Results 26 [4] Y. Y. Boykov and M. P. Jolly. Interactive graph cuts for optimal boundary & region segmentation of objects in n-d images. In IEEE International Conference on Computer Vision, volume 1, pages 105–112, [12] C. Manders, F. Farbiz, J. H. Chong, K. Y. Tang, G. G. Chua, M. H. Loke, and M. L. Yuan. Robust hand tracking using a skin tone and depth joint probability model. In IEEE International Conference on Automatic Face & Gesture Recognition, pages 1–6, 2008.

Experimental Results 27  Quantitative comparison:  A: the segmentation result  B: the ground truth from PhotoShop MethodAccuracy The proposed method83.8% Manders et al.’s method63.9% GraphCut59.2%

Experimental Results 28  Device : a Lenovo ThinkPad T400 with 2.5 GHz Dual Intel CPU and 2GB RAM  Computing time(no capture time)  The Frame Rate: 5 fps  ROIs tracking (Camshift algorithm)  Disparity competition is necessary for the successive frames ONLY. >>19.2 fps

Conclusion 29  This paper propose the skin color weighted disparity competition method for hand segmentation in clutter environment from stereo camera.  Limitation  hand-hand occlusion or hand-face occlusion  Future works  More accurate segmentation  Reconstruct the 3D appearance