A New Fingertip Detection and Tracking Algorithm and Its Application on Writing-in-the-air System The th International Congress on Image and Signal Processing (CISP) Kunpeng Li and Xin Zhang School of Electronic and Information Engineering, South China University of Technology Speaker: Yi-Ting Chen

Outline Introduction Flowchart Proposed Method –Hand Mode Estimation –Detection and Tracking by the dual mode Experimental Result Conclusions 2

Introduction Natural Human Computer Interaction (NHCI) is an important and vibrant research topic for decades. NHCI always brings effective communication with the computer and great convenience to our daily life. Application: finger-painting, virtual mouse, gesture recognition, sign-language, writing system, etc. [1][2] present a writing-in-the-air (WIA) system. 3

Reference [1] Z. Ye, X. Zhang, L. Jin, Z. Feng, and S. Xu, “Finger-writing-in-the air system using kinect sensor,” in Proceedings of IEEE International Conference on Multimedia and Expo (ICME), [2] L. J. X Zhang, Z Ye, “A new writing experience: Finger writing in the air using a kinect sensor.” MultiMedia, IEEE, vol. 20, pp. 85–93,

Related Work The algorithm failed with the fast writing situation and environment with challenging lighting background. 5 [1] Z. Ye, X. Zhang, L. Jin, Z. Feng, and S. Xu, “Finger-writing-in-the air system using kinect sensor,” in Proceedings of IEEE International Conference on Multimedia and Expo (ICME), 2013.

Main Contributions We propose a new tracking-detection based robust and accurate fingertip position estimation algorithm. 6

Flowchart 7

8

Hand Mode Estimation We propose to use the projected 2D distance between the arm point and palm point as an additional feature. 9

Hand Mode Estimation 10

Hand Mode Estimation 11 Side mode Frontal mode

Flowchart 12

Detection-based Side Mode Fingertip Estimation Combine the skin and depth model’s result, the algorithm called choose-to-trust algorithm (CTTA). (1) Determine the segmentation quality. (2) Choose one segmentation model to trust for the final segmentation. (3) Estimate the fingertip again. 13

Detection-based Side Mode Fingertip Estimation 14

Detection-based Side Mode Fingertip Estimation (3) Estimate the fingertip again. –We use the decided model to add hand region which is omitted in the DSB-MM. 15

Flowchart 16

The Oriented Gradient Feature Using simplified oriented gradient (OG) feature to describe the fingertip. –The fingertip is not always the point with minimum depth. 17

The Oriented Gradient Feature 18

Tracking Feature 19 [14] R. E. Kalman, “A new approach to linear filtering and prediction problems,” Journal of Basic Engineering, vol. 82, pp. 35–45, 1960.

Multi-objective Optimization Strategy 20

Multi-objective Optimization Strategy 21

Finger-Writing Character Recognition Using mean filter to remove the noise caused by wrong fingertip detection. Reducing feature by LDA (Linear Discriminant Analysis) Recognized by MQDF classifier. It can recognize 6,763 frequent Chinese characters, 26 English letters (upper case and lower case) and 10 digits. 22

Experimental Results The testing system: –PC with Intel Core i CPU, 3.10GHz –4GB RAM –Only one Kinect with 20fps Three experiments are designed here including: –Experiment on hand mode detection –fingertip estimation –character recognition 23

Hand Mode detection We manually marked out hand mode of 1636 frames and regarded them as ground truth. 24

Fingertip Detection and Tracking Data Set: –SCUT-WIA-I : 3207 frames –SCUT-WIA-II : 3293 frames Writing fast, ambient environment changing Fingertip positions are manually labeled. We calculate the Euclidean distance between labeled and estimated fingertip position as error distance. 25

Fingertip Detection and Tracking 26 (Pixels)

Fingertip Detection and Tracking 27

Character Recognition Conducted on 375 videos totally 44,522 frames. Successfully recognize 6,763 frequent Chinese, all English character (lower and upper cases) and digits. 28

Conclusion In general, our fingertip estimation maintain real-time properties and improve the recognition accuracy. The hand-mode detection can achieves 97.63% in precision. For the side-mode, our CTTA is more robust. For the frontal-mode, our tracking feature and OG feature solves the detection problem. The final character recognition rate reaches 100% in the first five candidates for all types of characters. 29

Thanks for your listening! 30