Online Kinect Handwritten Digit Recognition Based on Dynamic Time Warping and Support Vector Machine Journal of Information & Computational Science, 2015 Chengzhang Qua, Dengyi Zhang Computer School, Wuhan University Jing Tian Department of Computer Science and Engineering, University of South Carolina Speaker: Yi-Ting Chen

Outline Introduction Flowchart Proposed Method –Preprocessing and Feature Extraction –Recognition Method Experiment Conclusions 2

Introduction Handwriting in-space owns more advantages in natural interaction and has become a hot topic in the past decades. 3D cameras that can capture depth data besides traditional color data also gain a rapid growth in recent years, particularly those with Kinect. 3

Introduction The Kinect uses an infrared projector and a special microchip to track the movement of objects and individuals in three dimensions. It records RGB, depth, and skeleton information. We use the recorded depth and skeleton frames to describe the motion of the user. 4

Main Contribution In this paper, a Kinect based interaction recognition system was developed and tested on a handwriting digit dataset. There are mainly two steps in current system –The first is acquiring data during interaction. –The second is data processing and recognition. 5

Related Work The finger detecting and tracking algorithm –Based on depth image: Ye et al.[21] Feng et al.[22] Feature processing and recognition. –Feng et al.[22] 90% on English characters and 80% on Chinese –Lang et al.[20] recognition rate of 97% –Huang et al.[19] 94.6% on digit recognition 6

Reference [19] F. A. Huang, C. Y. Su, T. T. Chu, Kinect-based mid-air handwritten digit recognition using multiple segments and scaled coding [C], 2013 International Symposium on Intelligent Signal Processing and Communications Systems (ISPACS), IEEE, 2013, [20] S. Lang, M. Block, R. Rojas, Sign language recognition using Kinect [C], Artificial Intelligence and Soft Computing, Springer, Berlin Heidelberg, 2012, [21] Z. Ye, X. Zhang, L. Jin et al., Finger-writing-in-the-air system using Kinect sensor [C], 2013 IEEE International Conference on Multimedia and Expo Workshops (ICMEW), IEEE, 2013, 1-4 [22] Z. Feng, S. Xu, X. Zhang et al., Real-time ¯ngertip tracking and detection using Kinect depth sensor for a new writing-in-the air system [C], Proceedings of the 4th International Conference on Internet Multimedia Computing and Service, ACM, 2012,

Flowchart 8

9

Fingertip Tracking 10 V(t)

Fingertip Tracking We confirm the area based on enforcing the temporal continuity of the fingertip position. (a) should be in a small area centred by hand skeleton joint (b) should vary in a small range between two consecutive frames. 11

Fingertip Tracking Algorithm 12

Connect the Points 13 When connecting the consecutive points to form a writing trajectory, we may obtain a highly jagged and noisy 3D trajectory.

Normalization and Smoothing 1. Move the 3D trajectory to the original point(0, 0, 0). 2. Normalize each position Follows a normal Gaussian distribution N(0; 1) over all the 3D points in all recorded data. 3. Employ a standard Kalman filter algorithm to calculate a smoothed trajectory. 14

Feature Extraction We extract 6 types of features from each point of 3D handwritten trajectory. 15

6 Types of Features 16

Flowchart 17

Dynamic Time Warping 18 [23] E. Keogh, C. A. Ratanamahatana, Exact indexing of dynamic time warping [J], Knowledge and Information Systems, 7(3), 2005, M M M N N N

Classic Dynamic Time Warping 19 [23] E. Keogh, C. A. Ratanamahatana, Exact indexing of dynamic time warping [J], Knowledge and Information Systems, 7(3), 2005, Q l … 1 1 k … I. Boundary condition: II. Monotonicity condition: III. Step size condition:

Calculating between features Calculating the distance between two 3D handwritten features: 20

Modified Dynamic Time Warping 21 Q l … 1 1 k … I. Boundary condition: II. Monotonicity condition: III. Step size condition:

Support Vector Machine 22

Support Vector Machine 23

Experiment Data set: 0~9 3D handwriting digits collected by 10 college students. total of 6104 handwritten digit samples 24

Confused Matrix Training number: 20 Testing number: 590 for each class The mean accuracy is 99.1% 25

Precision Recall Curve Nearly 100% decision value for most of recall value. 26

Confused Digits 27

Accuracy curves for other methods For training numbers

Experiment Compare: Huang et al.[19] perform 94.6% on digit recognition –100 samples for training –230 samples for testing 29 [19] F. A. Huang, C. Y. Su, T. T. Chu, Kinect-based mid-air handwritten digit recognition using multiple segments and scaled coding [C], 2013 ISPACS, IEEE, 2013,

Conclusion A human computer interaction system based on Kinect handwritten was proposed. 3D fingertip positions could be tracked in real time with our method. Recognition framework for 3D handwritten interaction could be built based on DTW and SVM. According to the evaluation results, the proposed method offer high recognition accuracy 30

Thanks for your listening! 31