CS2610 - Simulating Mouse Pointing Using Kinect & Smartwatch (Wrist Pointing) Nannan Wen & Zhenjiang Fan

Project Hypothesis: Mouse pointing has its own limitations, for example, it needs a surface to support its movement, and the mouse itself has only just one functionality(it is a waste of money and energy). It is very inconvenient to carry around with and take a pointing action when you are giving a speech while standing. So we want to exploit rich and natural gestures brought by SmartWatch to simulate the pointing operation on the big screen.

Problem: -- Accuracy -- Acquisition speed -- Pointing and selection speed -- Comfortable use

Our goal: --- Reduce cursor movement time. --- Improve pointing accuracy on the big screen.

Solution proposal Cursor movement via Kinect sensor. Simulating mouse pointing via Android SmartWatch. Exploiting orientation data collected from Android SmartWatch. To implement mouse pointing on the big screen, we need to control cursor movement and simulate pointing action.

SmartWatch pointing design space 0 to 90 90 to 180 -90 to -180 0 to -90 0 to 90 90 to 180 -90 to -180 0 to -90 Mouse left up state Mouse left down state

Communication among devices(Data flow) Google Service Socket

Implementation We developed an Android Wearable application that installed on the SmartWatch to collect orientation, acceleration, and location info of the SmartWatch. Then the data will be transferred to an application on the Android phone, which is also responsible for sending the data to a PC application. The PC application, which is implemented with Kinect sensor service, is responsible for the movement of the cursor and simulating pointing operation.

Experiment ---5 participants ---Between subject design(randomize order) ---Independent variable: Kinect, Kinect + smartwatch ---Dependent variable: MT, Error Rate ---Experiment environment: --- Tool: Fitts Law Study Application

Results Movement Time(MT) comparison

Results Error rate comparison

Results Pair-wised T-test -- P = 0.00012 -- lesser than 0.005

Feedbacks --Cursor movement is not steady --Arm got tired easily --Delay --Not precise

Video demo

Things we learned & future work Now we know that we can implement the hypothesis without Kinect. Optimizations can be made during the data transmission process, so that reaction speed of the pointing operation will be improved. Kinect sensor detection service can be optimized too, in doing that, the cursor movement can be much more faster, smooth, and accurate.