1. Xin Zhang, Zhichao Ye, Lianwen Jin, Ziyong Feng, and Shaojie Xu
A New Writing Experience : Finger Writing in the Air Using a Kinect Sensor
Xin Zhang, Zhichao Ye, Lianwen Jin,
Ziyong Feng, and Shaojie Xu
FINGER-WRITING-IN-THE-AIR SYSTEM USING KINECT SENSOR
Zhichao Ye, Xin Zhang, Lianwen Jin, Ziyong Feng, Shaojie Xu
IEEE International Conference on
Multimedia and Expo Workshops (ICMEW), 2013
MultiMedia, IEEE, 2013

2. Outline Introduction Related Work Proposed Method Experimental Results
Conclusion

3. Introduction

4. Introduction So far most of writing systems still rely on:
Keyboard
Touch screen
…(Extra devices)
Essential goal of HCI:
Making interaction between user and computer more natural
Anaglyph 紅藍眼鏡
Polarization 偏光眼鏡 –
在一般LCD TV前面貼上一層微相位差膜(Micro-retarder)，利用光的偏振方向 來將左眼與右眼的影像分離
Shutter 快門眼鏡 - 更新頻率120Hz以上播放左、右眼視角畫面，藉由快速切換左右眼資訊，使得左右眼分別看到正確的左眼與右眼畫面，經過視覺暫留與大腦融合後，即可呈現出具
深度感的立體影像。
此技術所顯示的3D畫面解析度不會下降，且立體效果非常的好。
然而由於要提升3D影像品質，因此左眼與右眼觀看螢幕的時間非常短，使得整體亮度會下降許多，亦是目前研發上需克服的重點。

5. Introduction In this paper:
Propose a finger-writing-in-the-air system (based on Kinect):
Using depth, color and motion information
Real-time
User-friendly and unconstrained
Anaglyph 紅藍眼鏡
Polarization 偏光眼鏡 –
在一般LCD TV前面貼上一層微相位差膜(Micro-retarder)，利用光的偏振方向 來將左眼與右眼的影像分離
Shutter 快門眼鏡 - 更新頻率120Hz以上播放左、右眼視角畫面，藉由快速切換左右眼資訊，使得左右眼分別看到正確的左眼與右眼畫面，經過視覺暫留與大腦融合後，即可呈現出具
深度感的立體影像。
此技術所顯示的3D畫面解析度不會下降，且立體效果非常的好。
然而由於要提升3D影像品質，因此左眼與右眼觀看螢幕的時間非常短，使得整體亮度會下降許多，亦是目前研發上需克服的重點。

6. Related Work

7. Related work Hand Segmentation Skin color: Depth: Motion: X X X
Gaussian (mixture) model[2]
Illumination and hand-face overlapping
Depth:
noise
Motion:
Motion Cue[3]
The hand should be the most distinct moving object. X X X

8. Related work Fingertip Detection Curvature[6] Template matching[1]
[1] L. Jin, D. Yang, L. Zhen, and J. Huang. A novel vision based finger-writing character recognition system. Journal of JCSC, 16(3):421–436, 2007.
[2] S. L. Phung, A. Bouzerdoum, and D. Chai. Skin segmentation using color pixel classification: Analysis and comparison. IEEE Trans. on PAMI, 27:148–154, 2005.
[3] Jonathan Alon, Vassilis Athitsos, Quan Yuan and Stan Sclaroff. A Unified Framework for Gesture Recognition and Spatiotemporal Gesture Segmentation. IEEE Trans. on PAMI, 31:1685–1699, 2009.
[6] D. Lee and S. Lee. Vision-based finger action recognition by angle detection and contour analysis. Journal of ETRI, 33(3):415–422, 2011.
Fingertip Detection
Curvature[6]
Template matching[1]
Geodesic distance

9. Fingertip Identification
Related work
[10] Ziyong Feng, Shaojie Xu, Xin Zhang, Lianwen Jin, Zhichao Ye and WeixinYang. Real-time Fingertip Tracking and Detection using Kinect Depth Sensor for a New Writing-in the Air System. In Proc. of IEEE ICIMCS, 2012.
Writing-in-the-air system [10]:
K-means
Hand Segmentation
Data Conversion
Region Clustering
Fingertip Identification
Arm point
Fingertip

10. Proposed Method

11. Flow Chart
Hand Segmentation Fingertip Detection

12. Hand Segmentation
DSB-MM segmentation algorithm

13. Hand Segmentation Depth Model Solve the issues: Hand D: lighting
hand-face overlapping
moving background
Hand D:
R(n) : hand region at frame n
ω : : growth factor ↑ ↑

14. Hand Segmentation
Depth Model
A static hand
A moving hand

15. Reduce the storage size
Hand Segmentation
Skin Model
YCbCr color space
Quantify Y Component into three regions:
Bright 170<𝑌≤255
Normal 84<𝑌≤169
Dark 0<𝑌≤84
Gaussian classifier[2]:
Reduce the storage size
𝑚 𝑠 𝑖 : mean vector of the i-th skin class
𝐶 𝑠 𝑖 : covariance of the i-th skin class
𝑚 𝑛𝑠 𝑖 : mean vector of the i-th non-skin
𝐶 𝑛𝑠 𝑖 :covariance of the i-th non-skin class
skin
Non-skin
(Squared Mahalanobis distance)

16. Hand Segmentation Skin Model Color Image Depth Model Skin Model
Depth + Skin

17. Hand Segmentation Background Model Codebook background model[8]
[8] K. Kim, T. H. Chalidabhongse, D. Harwood, and L. Davis. Real time foreground-background segmentation using code book model. Real-Time Imaging, 11:172–185, 2005.
Background Model
Codebook background model[8]

18. Hand Segmentation Background Model Codebook background model[8]
Color image A
Foreground result A
Color image B
Foreground result B

19. Hand Segmentation
DSB-MM segmentation algorithm

20. Hand Segmentation DSB-MM segmentation algorithm
Each model should have different reliabilities.
Adaptive voting system
A pixel is kept as hand pixel by

21. Hand Segmentation Artificial Neural Network (ANN)
(1) All the models contribute to the final result.
(2) None of them is absolutely reliable.
“1 0 0”, “0 1 0” or “0 0 1” representing 1/3, 1/2 or 2/3
Training:
resilient back propagation algorithm (RPROP)

22. Hand Segmentation
Origin
Depth
Skin
Background
Mixture

23. Flow Chart
Hand Segmentation Fingertip Detection

24. Fingertip Detection Side-mode & Frontal-mode -- (Red) : Side-mode
ㄧ(Blue) : Frontal-mode

25. Fingertip Detection Side-mode
Fingertip : the farthest point from the arm point
Palm point:
Ellipse fitting technique (center point)
Arm point:
The center of the increased region

26. Fingertip Detection Side-mode The farthest distance to the arm point:
Side-Mode Criterion:

27. Fingertip Detection Frontal-mode
Fingertip : the point with the smallest depth value

28. Experimental Results

29. Experimental Results 375 videos(44522 frames) Intel Core i5-2400 CPU
3.10 GHz and 4 Gbytes of RAM
20 frames per second(fps)
375 videos(44522 frames)
Recognition of the classifier:
6763 frequently used Chinese character
26 English letters (upper case & lower case)
10 digits

30. Experimental Results Finger-writing character recognition
Linking all detected fingertip positions + mean filter
Modified quadratic discriminant function (MQDF) character classifier[9]
[9] T. Long and L. Jin. Building Compact MQDF Classifier for Large Character Set Recognition by Subspace Distribution Sharing. Pattern Recognition, 41(9): , 2008.

31. Experimental Results
Error distance (Fingertip detection):

32. Experimental Results

33. Experimental Results

34. Conclusion

35. Conclusion Propose a real-time finger-writing-in-the-air system
Hand Segmentation:
Depth + Skin + Motion
Adaptive depth threshold of hand region
Fingertip Detection:
Side-mode
Frontal-mode   