1. Vision Based hand tracking for Interaction The 7th International Conference on Applications and Principles of Information Science (APIS2008) Dept. of Visual Contents, Dongseo University 2008. 01. 29 Seung Il Han, Jeon Yeong Mi, Jung Hye Kwon, Hwang-Kyu Yang, Byung Gook Lee hs12@dit.dongseo.ac.kr

2. Outline 2016-06-07APIS. 2008 - Vision Based hand tracking for Interaction, Han Seung Il, Dongseo Univ., hs12@dit.dongseo.ac.kr2 1. Purpose 2. System overview 3. Flow chart 4. Segmentation of a hand region 5. Matching stereo data 6. Conclusion 7. Future work

3. 2016-06-07APIS. 2008 - Vision Based hand tracking for Interaction, Han Seung Il, Dongseo Univ., hs12@dit.dongseo.ac.kr3 1. Introduction to hand tracking for interaction

4. 1.1 Purpose 2016-06-074APIS. 2008 - Vision Based hand tracking for Interaction, Han Seung Il, Dongseo Univ., hs12@dit.dongseo.ac.kr  Any user can interact anytime without any condition of space.  More comfortable to use digital-device on public place.

5. 1.2 Optical vs. computational reconstruction 2016-06-075APIS. 2008 - Vision Based hand tracking for Interaction, Han Seung Il, Dongseo Univ., hs12@dit.dongseo.ac.kr  Hardware  Bumblebee stereo camera  Display device  PC  Software  Image processing  Stereo matching

6. 2016-06-07APIS. 2008 - Vision Based hand tracking for Interaction, Han Seung Il, Dongseo Univ., hs12@dit.dongseo.ac.kr6 2. System overview

7. 2.1 System architecture 2016-06-07APIS. 2008 - Vision Based hand tracking for Interaction, Han Seung Il, Dongseo Univ., hs12@dit.dongseo.ac.kr7

8. 2.2 Flow chart 2016-06-07APIS. 2008 - Vision Based hand tracking for Interaction, Han Seung Il, Dongseo Univ., hs12@dit.dongseo.ac.kr8

9. 2.3 The segmentation of a hand region 2016-06-07APIS. 2008 - Vision Based hand tracking for Interaction, Han Seung Il, Dongseo Univ., hs12@dit.dongseo.ac.kr9 I f = ½ 255 ; j I i ¡ I b j > T h res h o ld 0 ; o t h erw i se [Background] [Captured] [Foreground] [Subtraction]

10. 2.3 The segmentation of a hand region 2016-06-07APIS. 2008 - Vision Based hand tracking for Interaction, Han Seung Il, Dongseo Univ., hs12@dit.dongseo.ac.kr10  Detection algorithm  Peer and Solina’s 2003 [1]  Hsu and Mottaleb Jain’s Central 2002 [2]  Hsu and Mottaleb Jain’s Ellipse 2002 [2]  M. Jones and J. Rehg’s Color Histogram [3, 4] [3] M. Jones, J. Rehg. “Statistical color models with application to skin”, In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Vol.1, pp.274-280, 2002. [1] Kovac, J. Peer, P. Solina, F. “Human Skin Color Clustering for Face Detection”, The IEEE Region, vol.2, pp.144-148, 2003. [2] R.L.Hsu, M.Abdel-Mottaleb, A.K.Jain, “Face Detection in Colour Images”, IEEE transactions, on Pattern Analysis and Machine Intelligence, vol.24, no.5, pp 696-706, 2002. [4] Shahzad Malik, “Real-time Hand Tracking and Finger Tracking for Interaction”, CSC2503F Project Report, December 18, 2003.

11. 2.3 The segmentation of a hand region 2016-06-07APIS. 2008 - Vision Based hand tracking for Interaction, Han Seung Il, Dongseo Univ., hs12@dit.dongseo.ac.kr11 OR * Uniform daylight illumination R > 95 ; G > 40 ; B > 20 an d max f R ; G ; B g ¡ m i n f R ; G ; B g > 15 an d j R ¡ G j > 15 an d R > G an d R > B * Flashlight or light daylight R > 220 ; G > 210 ; B > 170 an d j R ¡ G j · 15 an d R > B an d G > B  Peer and Solina’s 2003

12. 2.3 The segmentation of a hand region  Hsu and Mottaleb Jain’s Central 2002 2016-06-07APIS. 2008 - Vision Based hand tracking for Interaction, Han Seung Il, Dongseo Univ., hs12@dit.dongseo.ac.kr12 W C i ( Y ) = 8 > < > : W LC i + ( Y ¡ Y m i n )( W C i ¡ W LC i ) K l = Y m i n ; Y < K l W HC i + ( Y max ¡ Y )( W C i ¡ W HC i ) Y max ¡ K h ; K h < Y W C i ;e l se ¹ C b ( Y ) = 8 > < > : 108 + 10 ( K l ¡ Y ) K l ¡ Y m i n ; Y < K l 108 + 10 ( Y ¡ K h ) Y max ¡ K h ; K h < Y 108 ;e l se C r = 0 : 5 R ¡ 0 : 418 G ¡ 0 : 081 B C b = ¡ 0 : 168 R ¡ 0 : 331 G + 0 : 5 B RGB t o Y = 0 : 299 R + 0 : 587 G + 0 : 144 B YC b C r ¹ C r ( Y ) = 8 > < > : 154 ¡ 10 ( K l ¡ Y ) K l ¡ Y m i n ; Y < K l 154 + 22 ( Y ¡ K h ) Y max ¡ K h ; K h < Y 108 ;e l se

13. 2.3 The segmentation of a hand region  Hsu and Mottaleb Jain’s Central 2002 ¹ C b ( Y ) ¡ ® W C b ( Y ) < C b < ¹ C b ( Y ) + ® W C b ( Y ) ; ¹ C r ( Y ) ¡ ® W C r ( Y ) < C r < ¹ C r ( Y ) + ® W C r ( Y ) 2016-06-07APIS. 2008 - Vision Based hand tracking for Interaction, Han Seung Il, Dongseo Univ., hs12@dit.dongseo.ac.kr13 : a) : b) a)b)

14. 2.3 The segmentation of a hand region  Hsu and Mottaleb Jain’s Ellipse 2002 · x y ¸ = · cos µ s i n µ ¡ s i n µ cos µ ¸· C b ¡ c x C r ¡ c y ¸ [2] R.L.Hsu, M.Abdel-Mottaleb, A.K.Jain, “Face Detection in Colour Images”, IEEE transactions, on Pattern Analysis and Machine Intelligence, vol.24, no.5, pp.696-706, 2002. e cx = 1 : 6 ; e cy = 2 : 41 ; a = 25 : 39 ; b = 14 : 03 c x = 109 : 38 ; c y = 152 : 02 ; µ = 2 : 53 ra d i an 2016-06-07APIS. 2008 - Vision Based hand tracking for Interaction, Han Seung Il, Dongseo Univ., hs12@dit.dongseo.ac.kr14 ( x ¡ e cx ) 2 a 2 + ( y ¡ e cy ) 2 b 2 · 1

15. 2.3 The segmentation of a hand region  M. Jones and J. Rehg’s color histogram 2016-06-07APIS. 2008 - Vision Based hand tracking for Interaction, Han Seung Il, Dongseo Univ., hs12@dit.dongseo.ac.kr15 P ( s k i n ) + P ( ¡ s k i n ) = 1 P ( s k i n ) = T s T s + T n : The skin histogram : The non-skin histogram : The pixel count in bin rgb of : The total counts contained in n [ rg b ] H s T n H n s [ rg b ] T s H s H s H n H n P ( rg b j ¡ s k i n ) = n [ rg b ] T n P ( rg b j s k i n ) = s [ rg b ] T s P ( s k i n j rg b ) = P ( rg b j s k i n ) P ( s k i n ) P ( rg b j s k i n ) P ( s k i n ) + P ( rg b j ¡ s k i n ) P ( ¡ s k i n ) P ( s k i n j rg b ) ¸ ¾ s

16. 2.3 The segmentation of a hand region 2016-06-07APIS. 2008 - Vision Based hand tracking for Interaction, Han Seung Il, Dongseo Univ., hs12@dit.dongseo.ac.kr16 [Color image of background subtraction]  Technique of region extraction [Blob analysis of binary image] [Binary image of background subtraction] [Binary morphological opening]

17. 2.4 Detection of fingers 0 1 2 3 4 56 7 2016-06-07APIS. 2008 - Vision Based hand tracking for Interaction, Han Seung Il, Dongseo Univ., hs12@dit.dongseo.ac.kr17 [5] Image Processing Program by Visual C++, Published by HANBIT Media, pp.672-679, Vol. 10, No. 1779, 2007.  Contour tracing with clockwise algorithm

18. 2.4 Detection of fingers [Result of binary morphological opening][Contour tracing result of feature extraction] 2016-06-07APIS. 2008 - Vision Based hand tracking for Interaction, Han Seung Il, Dongseo Univ., hs12@dit.dongseo.ac.kr18  Result of contour tracing

19. 2.4 Detection of fingers  Detection of peak point  At each pixel in a hand contour  Compute k-curvature  Angle between the two vectors and  Peak point and valley point  Peak point  Valley point µ k < µ m i n µ k > µ max µ C i ( j ) 2016-06-07APIS. 2008 - Vision Based hand tracking for Interaction, Han Seung Il, Dongseo Univ., hs12@dit.dongseo.ac.kr19 j i [ C i ( j ) ; C i ( j ¡ k )][ C i ( j ) ; C i ( j + k )] w h ere ; k i scons t an t µ k C i ( j ¡ k ) C i ( j + k ) [4] Shahzad Malik, “Real-time Hand Tracking and Finger Tracking for Interaction”, CSC2503F Project Report, December 18, 2003. w h ere ; µ m i n = 90 ± ; µ max = 200 ±

20. 2.4 Detection of fingers  Detection of peak point 2016-06-07APIS. 2008 - Vision Based hand tracking for Interaction, Han Seung Il, Dongseo Univ., hs12@dit.dongseo.ac.kr20 [Peak point detection][Valley point detection]

21. 2.4 Detection of fingers  Detection of midpoint  Define as:  Midpoints represent the axis of the finger.  is a point that is k points to the left along a hand contour  is a point to the right  Let is the index of a finger tip cnorm ( x ) = 8 < : x + N :x < 0 x ¡ N : ( x + 1 ) > N x:o t h erw i se : w h ere N represen tt h enum b ero f po i n t sa l ong t h econ t ourper i me t er : 2016-06-07APIS. 2008 - Vision Based hand tracking for Interaction, Han Seung Il, Dongseo Univ., hs12@dit.dongseo.ac.kr21 cnorm ( x ) P ( cnorm ( i + k )) P ( cnorm ( i ¡ k )) i f t [4] Shahzad Malik, “Real-time Hand Tracking and Finger Tracking for Interaction”, CSC2503F Project Report, December 18, 2003.

22. 2.4 Detection of fingers  Detection of midpoint  A midpoint  Compute for Q ( k ) Q ( k ) = P ( cnorm ( i f t + k )) + P ( cnorm ( i f t ¡ k )) 2 Q ( k ) k m i n < k < k max 2016-06-07APIS. 2008 - Vision Based hand tracking for Interaction, Han Seung Il, Dongseo Univ., hs12@dit.dongseo.ac.kr22

23. 2.5 Matching stereo data 2016-06-07APIS. 2008 - Vision Based hand tracking for Interaction, Han Seung Il, Dongseo Univ., hs12@dit.dongseo.ac.kr23  Calculation of depth value xy P y x r B D f [6] Tomase Poggio, “Introductory Techniques for 3-D Computer Vision”, Massachussetts Institute of Technology, pp. 139-145, 1998. P x = x l + x r 2 ; x r l x l x : Distance using centimeter : Resolution ratio, : CCD Sensor size R C P y = y l + y r 2 = y ; P z = f B j x l ¡ x r j D D = P z RC

24. 2.5 Matching stereo data  Result of depth value  Real distance ≒ 66 Cm  Calculated depth value = 67.5 Cm 2016-06-07APIS. 2008 - Vision Based hand tracking for Interaction, Han Seung Il, Dongseo Univ., hs12@dit.dongseo.ac.kr24

25. 2016-06-07APIS. 2008 - Vision Based hand tracking for Interaction, Han Seung Il, Dongseo Univ., hs12@dit.dongseo.ac.kr25 3. Conclusion

26. 3.1 Conclusion  The hardware costs very low.  Hand tracking using an enhanced algorithm  Interaction with system using user’s hands  Free of conditions of time & space 2016-06-07APIS. 2008 - Vision Based hand tracking for Interaction, Han Seung Il, Dongseo Univ., hs12@dit.dongseo.ac.kr26

27. 3.2 Future Work  To perform a robust segmentation with ambient environment while detection the skin color.  The determination of peak-point count and decision of location.  Gesture recognition performance  To enhance the accuracy and speed of the system.  Interaction using hands. 2016-06-07APIS. 2008 - Vision Based hand tracking for Interaction, Han Seung Il, Dongseo Univ., hs12@dit.dongseo.ac.kr27

28. References 1. Kovac, J. Peer, P. Solina, F. “Human Skin Color Clustering for Face Detection”, The IEEE Region, vol.2, pp.144-148, 2003. 2. R.L.Hsu, M.Abdel-Mottaleb, A.K.Jain, “Face Detection in Colour Images”, IEEE Transactions on Pattern Analysis and Machine Intelligence, vol.24, no.5, pp.696-706, 2002. 3. M. Jones, J. Rehg. “Statistical color models with application to skin”, In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Vol.1, pp.274-280, 2002. 4. Shahzad Malik, “Real-time Hand Tracking and Finger Tracking for Interaction”, CSC2503F Project Report, December 18, 2003. 5. “Image Processing Program by Visual C++”, Published by HANBIT Media, pp.672-679, Vol. 10, No.1779, 2007. 6. Tomase Poggio, “Introductory Techniques for 3-D Computer Vision”, Massachussetts Institute of Technology, pp.139-145, 1998. 7. George V. Paul, Glenn J. Beach, Charlesn J.Cohen, “A Real-time Object Tracking System using a Color Camera”, pp.137-142, 2001. 8. J. Segen, S. Kumar. “3D hand pose estimation using a single camera”, In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Vol.1, pp.479-485, 1999. 2016-06-07APIS. 2008 - Vision Based hand tracking for Interaction, Han Seung Il, Dongseo Univ., hs12@dit.dongseo.ac.kr28

29. Thank You 2016-06-07APIS. 2008 - Vision Based hand tracking for Interaction, Han Seung Il, Dongseo Univ., hs12@dit.dongseo.ac.kr29