1. NBKeyboard: An Arm-based Word-gesture keyboard
Haoran Zhang
Xiaozhong Zhang
Keren Ye
Welcome

2. Contents Related Work Design Space Exploration Implementation Details
User Study
Conclusion

3. Related Work Distance Freehand Pointing
Design space of freehand pointing and clicking interaction
Three approaches for gestural pointing and two for clicking
Difference:
They have not applied the pointing and clicking techniques to any specific applications. Rather than study only the techniques, we put the techniques into a specific context of text input on a specific Kinect platform.
Vogel, Daniel, and Ravin Balakrishnan. "Distant freehand pointing and clicking on very large, high resolution displays."

4. Related Work Vulture A word-gesture keyboard for mid-air operation
Use both elbow and wrist movement to control the mouse
Difference:
Our method differs from theirs in that we use the SmartWatch posture detection instead of the pinch detection to determine the state change of the cursor.
Markussen, Anders, Mikkel Rønne Jakobsen, and Kasper Hornbæk. "Vulture: a mid-air word-gesture keyboard."

5. Related Work Two-handed Approach Use right hand for pointing
Use left hand to simulate the pressing and releasing state of computer mouse
Difference
A burden for the end user in that large body motion makes user tired. Instead, we propose to use an additional SmartWatch. Shaking wrist is enough in the case of using SmartWatch thus we think it may release the burden of large body motion and lead to better performance.
Xiaoyu Ge, and Longhao Li. "Improving the Kinect-based mouse simulation with both multi-model and two-handed gesture-based approaches."

6. Design Space Exploration
Using Only One Hand
Eliminates the problem of incorporating extra devices
Suffers the problem of accurately recognizing small body motion
Not efficient since holding is required

7. Design Space Exploration
Using Both Hands
Uses large body motion to simulate click in order to get higher clicking accuracy
Eliminates the motion of “push and hold” thus more efficient
Burden of the user, not suitable for Virtual Keyboard
Especially not suitable for virtual keyboard because shorthand writing of virtual keyboard needs frequent mouse dragging which can be decomposed into clicking, moving, and releasing
Xiaoyu Ge, and Longhao Li. "Improving the Kinect-based mouse simulation with both multi-model and two-handed gesture-based approaches."

8. Design Space Exploration
Facilitate the Two-handed Approach using SmartWatch
Intuition: use wrist rather than body movements
Plan A - Motion Sensing
detect the dynamic motion
Plan B - Posture Sensing
detect the static posture
Further reduce the motion of the wrist
Shorthand writing of virtual keyboard needs frequent mouse dragging
Using an additional SmartWatch is more efficient in the task of text input. Body gestures should be applied frequently if users use the Kinect alone, yet shaking wrist may be enough in the case of using SmartWatch. Therefore, we believe that the movement should be much less when we use SmartWatch, which result in better performance.

9. Implementation Details
Prototype: NBKeyboard
Five Components

10. Implementation Details
Kinect Component
Captures body motion
Translates body motion into 3D mouse trace
SmartWatch Component
Distinguishes between the horizontal and vertical position
Generates click or release messages
Controller
Drives the User Interface to generate feedback
Invokes algorithm core to analyze user's input
User Interface
Provides feedback to the end user
Algorithm Core
Analyzes the sequence of keys pressed and translates them into words

11. Implementation Details
Shorthand Writing on NBKeyboard
From keys’ sequence to regular pattern
Different from HW1, we give up the optimization of inflection point
E.g., Keys’ sequence “thgfre” is mapped to pattern “t+h*g*f*r*e+”
From regular pattern to candidates
Natural language info - word-frequency dict
Why there is no inflection point?
We give up the optimization of inflection point here since the mouse trace translated from body motion contains more noises.

12. Study Setup Normal PC display Kinect with 30 FPS tracking frequency
Participants stood 1.5 m from the display
Window containing the keyboard was 50 x 30 cm
A dictionary of 17,807 words with word frequency

13. Study Setup
Left: keyboard screen, the Kinect sensor and a screen to show tracking info.
Right: a user typing text using his right hand and the mobile device in his left hand

14. Study Design Within-subjects design Six participants
Half use baseline first, half use our approach first
Two sessions of text entry each participant
Four transcription sentences each session

15. Study Procedure Familiarize with the devices Practice typing
Transcribe sentences using one approach
Short break
Transcribe sentences using the other approach

16. Study Variables Device Sentence WPM: Word Per Minute
Error rate: Minimum Word Distance (Minimum String Distance on a per-word level)

17. Study Results Two-Factor ANOVA with replication
Our method is significantly faster than the baseline with regard to WPM, F(1,15) = 4.95, p < .052.
Error rate for baseline and our approach was not significantly different, F(1,15) = 1.24, p = .27.

18. Study Discussion Click state detection Arm movement interference
Baseline detects lifted and lowered arm as click state
Our approach detects wrist twisting
Arm movement interference
Lifting and lowering arm may cause right arm to co-move
Twisting wrist doesn’t cause right arm co-movement

19. Demo Video

20. Conclusion A novel virtual keyboard interface
Within-object user study with 6 participants
Our approach was significantly faster than the baseline
Summary of some reasons for the typing speed increase

21. Thanks