1. Mean Euclidean Distance Error (mm)
Hand Pose Learning: Combining Deep Learning and Hierarchical Refinement for
3D Hand Pose Estimation
Min-Yu Wu; Ya-Hui Tang; Pai-Wei Ting; Li-Chen Fu
Department of Computer Science and Information Engineering
National Taiwan University
Abstract
We propose a hybrid method of training a deep learning model and hierarchical refinement for hand pose estimation in a 3D space using depth images.
Methods
Our system is decomposed into two parts. The first part is composed of conventional joint-position-loss layer and our proposed skeleton- difference layer. The second part is an additional loss layer that considers physical constraints of a hand pose, keeping the joint relations. Our approach achieves 8.45 mm, and our result outperforms the previous works.
Designing a skeleton-difference (SD) layer that can allow a convolutional neural network (CNN) training process to effectively
learn the shape as well as physical constraints of a hand.
Employing a refinement method that is capable of hierarchically regressing a hand pose with an energy function.
To represent the spatial relationship better, we transform the bones into vectors, and the loss is then defined as :
𝛹 𝑆𝐷L = 𝐿𝑜𝑠𝑠 𝑎 + 𝐿𝑜𝑠𝑠 𝑏
Fig. 2 Each finger represents one part and has three joints
Fig. 1 The defined 16
Joints in ICVL dataset [4]
Fig. 3 Angular loss 𝐿𝑜𝑠𝑠 𝑎
Fig. 4 Bone length 𝐿𝑜𝑠𝑠 𝑏
Discussion
The best performance of SD loss function is 0.5. But when the weight is increased to 1, the performance starts to degrade.
The reason is that SD loss is the compensation for ED loss. The former simply can’t steal the latter’s thunder.
Fig. 5 The flowchart of our system
Fig. 6 The proposed training approach
Results
Conclusions
Table 1(updated) : SDNet: ZF-Net[3] with
skeleton-difference loss function
We proposed a novel architecture for hand pose estimation that combines the skeleton-difference network (SDNet) and a hierarchical refinement method.
𝛹 𝑇𝑜𝑡𝑎𝑙 = 𝜔 𝐷 𝛹 𝐷 + 𝜔 𝑆𝐷 𝛹 𝑆𝐷
Network Settings
(weight)
Mean Euclidean Distance Error (mm)
Baseline (ZF-Net)
SDNet ( 𝜔 𝑆𝐷 =0.125)
8.8209
SDNet ( 𝜔 𝑆𝐷 =0.25)
8.7291
SDNet ( 𝜔 𝑆𝐷 =0.5)
8.4520
SDNet ( 𝜔 𝑆𝐷 =1.0)
8.5104
Fig. 7 The fraction of success
compared with [1,2]
Contact
References
X. Sun, Y. Wei, S. Liang, X. Tang, and J. Sun, "Cascaded hand pose regression," in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2015, pp
D. Tang, H. Jin Chang, A. Tejani, and T.-K. Kim, "Latent regression forest: Structured estimation of 3d articulated hand posture," in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2014, pp
M. D. Zeiler and R. Fergus, "Visualizing and understanding convolutional networks," in European conference on computer vision, 2014, pp
D. Tang, T.-H. Yu, and T.-K. Kim, "Real-time articulated hand pose estimation using semi-supervised transductive regression forests," in Proceedings of the IEEE international conference on computer vision, 2013, pp
Ya-Hui Tang
National Taiwan University
Website:
Phone: