Mean Euclidean Distance Error (mm) Hand Pose Learning: Combining Deep Learning and Hierarchical Refinement for 3D Hand Pose Estimation Min-Yu Wu; Ya-Hui Tang; Pai-Wei Ting; Li-Chen Fu Department of Computer Science and Information Engineering National Taiwan University Abstract We propose a hybrid method of training a deep learning model and hierarchical refinement for hand pose estimation in a 3D space using depth images. Methods Our system is decomposed into two parts. The first part is composed of conventional joint-position-loss layer and our proposed skeleton- difference layer. The second part is an additional loss layer that considers physical constraints of a hand pose, keeping the joint relations. Our approach achieves 8.45 mm, and our result outperforms the previous works. Designing a skeleton-difference (SD) layer that can allow a convolutional neural network (CNN) training process to effectively learn the shape as well as physical constraints of a hand. Employing a refinement method that is capable of hierarchically regressing a hand pose with an energy function. To represent the spatial relationship better, we transform the bones into vectors, and the loss is then defined as : 𝛹 𝑆𝐷L = 𝐿𝑜𝑠𝑠 𝑎 + 𝐿𝑜𝑠𝑠 𝑏 Fig. 2 Each finger represents one part and has three joints Fig. 1 The defined 16 Joints in ICVL dataset [4] Fig. 3 Angular loss 𝐿𝑜𝑠𝑠 𝑎 Fig. 4 Bone length 𝐿𝑜𝑠𝑠 𝑏 Discussion The best performance of SD loss function is 0.5. But when the weight is increased to 1, the performance starts to degrade. The reason is that SD loss is the compensation for ED loss. The former simply can’t steal the latter’s thunder. Fig. 5 The flowchart of our system Fig. 6 The proposed training approach Results Conclusions Table 1(updated) : SDNet: ZF-Net[3] with skeleton-difference loss function We proposed a novel architecture for hand pose estimation that combines the skeleton-difference network (SDNet) and a hierarchical refinement method. 𝛹 𝑇𝑜𝑡𝑎𝑙 = 𝜔 𝐷 𝛹 𝐷 + 𝜔 𝑆𝐷 𝛹 𝑆𝐷 Network Settings (weight) Mean Euclidean Distance Error (mm) Baseline (ZF-Net) 12.6990 SDNet ( 𝜔 𝑆𝐷 =0.125) 8.8209 SDNet ( 𝜔 𝑆𝐷 =0.25) 8.7291 SDNet ( 𝜔 𝑆𝐷 =0.5) 8.4520 SDNet ( 𝜔 𝑆𝐷 =1.0) 8.5104 Fig. 7 The fraction of success compared with [1,2] Contact References X. Sun, Y. Wei, S. Liang, X. Tang, and J. Sun, "Cascaded hand pose regression," in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2015, pp. 824-832. D. Tang, H. Jin Chang, A. Tejani, and T.-K. Kim, "Latent regression forest: Structured estimation of 3d articulated hand posture," in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2014, pp. 3786-3793. M. D. Zeiler and R. Fergus, "Visualizing and understanding convolutional networks," in European conference on computer vision, 2014, pp. 818-833. D. Tang, T.-H. Yu, and T.-K. Kim, "Real-time articulated hand pose estimation using semi-supervised transductive regression forests," in Proceedings of the IEEE international conference on computer vision, 2013, pp. 3224-3231 Ya-Hui Tang National Taiwan University Email: d05922027@csie.ntu.edu.tw Website: http://robotlab.csie.ntu.edu.tw/vision.html Phone: +886937928005