NBKeyboard: An Arm-based Word-gesture keyboard Haoran Zhang Xiaozhong Zhang Keren Ye Welcome

Contents Related Work Design Space Exploration Implementation Details User Study Conclusion

Related Work Distance Freehand Pointing Design space of freehand pointing and clicking interaction Three approaches for gestural pointing and two for clicking Difference: They have not applied the pointing and clicking techniques to any specific applications. Rather than study only the techniques, we put the techniques into a specific context of text input on a specific Kinect platform. Vogel, Daniel, and Ravin Balakrishnan. "Distant freehand pointing and clicking on very large, high resolution displays."

Related Work Vulture A word-gesture keyboard for mid-air operation Use both elbow and wrist movement to control the mouse Difference: Our method differs from theirs in that we use the SmartWatch posture detection instead of the pinch detection to determine the state change of the cursor. Markussen, Anders, Mikkel Rønne Jakobsen, and Kasper Hornbæk. "Vulture: a mid-air word-gesture keyboard."

Related Work Two-handed Approach Use right hand for pointing Use left hand to simulate the pressing and releasing state of computer mouse Difference A burden for the end user in that large body motion makes user tired. Instead, we propose to use an additional SmartWatch. Shaking wrist is enough in the case of using SmartWatch thus we think it may release the burden of large body motion and lead to better performance. Xiaoyu Ge, and Longhao Li. "Improving the Kinect-based mouse simulation with both multi-model and two-handed gesture-based approaches."

Design Space Exploration Using Only One Hand Eliminates the problem of incorporating extra devices Suffers the problem of accurately recognizing small body motion Not efficient since holding is required

Design Space Exploration Using Both Hands Uses large body motion to simulate click in order to get higher clicking accuracy Eliminates the motion of “push and hold” thus more efficient Burden of the user, not suitable for Virtual Keyboard Especially not suitable for virtual keyboard because shorthand writing of virtual keyboard needs frequent mouse dragging which can be decomposed into clicking, moving, and releasing Xiaoyu Ge, and Longhao Li. "Improving the Kinect-based mouse simulation with both multi-model and two-handed gesture-based approaches."

Design Space Exploration Facilitate the Two-handed Approach using SmartWatch Intuition: use wrist rather than body movements Plan A - Motion Sensing detect the dynamic motion Plan B - Posture Sensing detect the static posture Further reduce the motion of the wrist Shorthand writing of virtual keyboard needs frequent mouse dragging Using an additional SmartWatch is more efficient in the task of text input. Body gestures should be applied frequently if users use the Kinect alone, yet shaking wrist may be enough in the case of using SmartWatch. Therefore, we believe that the movement should be much less when we use SmartWatch, which result in better performance.

Implementation Details Prototype: NBKeyboard Five Components

Implementation Details Kinect Component Captures body motion Translates body motion into 3D mouse trace SmartWatch Component Distinguishes between the horizontal and vertical position Generates click or release messages Controller Drives the User Interface to generate feedback Invokes algorithm core to analyze user's input User Interface Provides feedback to the end user Algorithm Core Analyzes the sequence of keys pressed and translates them into words

Implementation Details Shorthand Writing on NBKeyboard From keys’ sequence to regular pattern Different from HW1, we give up the optimization of inflection point E.g., Keys’ sequence “thgfre” is mapped to pattern “t+h*g*f*r*e+” From regular pattern to candidates Natural language info - word-frequency dict Why there is no inflection point? We give up the optimization of inflection point here since the mouse trace translated from body motion contains more noises.

Study Setup Normal PC display Kinect with 30 FPS tracking frequency Participants stood 1.5 m from the display Window containing the keyboard was 50 x 30 cm A dictionary of 17,807 words with word frequency

Study Setup Left: keyboard screen, the Kinect sensor and a screen to show tracking info. Right: a user typing text using his right hand and the mobile device in his left hand

Study Design Within-subjects design Six participants Half use baseline first, half use our approach first Two sessions of text entry each participant Four transcription sentences each session

Study Procedure Familiarize with the devices Practice typing Transcribe sentences using one approach Short break Transcribe sentences using the other approach

Study Variables Device Sentence WPM: Word Per Minute Error rate: Minimum Word Distance (Minimum String Distance on a per-word level)

Study Results Two-Factor ANOVA with replication Our method is significantly faster than the baseline with regard to WPM, F(1,15) = 4.95, p < .052. Error rate for baseline and our approach was not significantly different, F(1,15) = 1.24, p = .27.

Study Discussion Click state detection Arm movement interference Baseline detects lifted and lowered arm as click state Our approach detects wrist twisting Arm movement interference Lifting and lowering arm may cause right arm to co-move Twisting wrist doesn’t cause right arm co-movement

Demo Video

Conclusion A novel virtual keyboard interface Within-object user study with 6 participants Our approach was significantly faster than the baseline Summary of some reasons for the typing speed increase

Thanks