1. CS2610 - Simulating Mouse Pointing Using Kinect & Smartwatch
(Wrist Pointing)
Nannan Wen & Zhenjiang Fan

2. Project Hypothesis:
Mouse pointing has its own limitations, for example, it needs a surface to support its movement, and the mouse itself has only just one functionality(it is a waste of money and energy).
It is very inconvenient to carry around with and take a pointing action when you are giving a speech while standing.
So we want to exploit rich and natural gestures brought by SmartWatch to simulate the pointing operation on the big screen.

3. Problem: -- Accuracy -- Acquisition speed
-- Pointing and selection speed
-- Comfortable use

4. Our goal: --- Reduce cursor movement time.
--- Improve pointing accuracy on the big screen.

5. Solution proposal Cursor movement via Kinect sensor.
Simulating mouse pointing via Android SmartWatch.
Exploiting orientation data collected from Android SmartWatch.
To implement mouse pointing on the big screen, we need to control cursor movement and simulate pointing action.

6. SmartWatch pointing design space
0 to 90
90 to 180
-90 to -180
0 to -90
0 to 90
90 to 180
-90 to -180
0 to -90
Mouse left up state
Mouse left down state

7. Communication among devices(Data flow)
Google Service
Socket

8. Implementation
We developed an Android Wearable application that installed on the SmartWatch to collect orientation, acceleration, and location info of the SmartWatch.
Then the data will be transferred to an application on the Android phone, which is also responsible for sending the data to a PC application.
The PC application, which is implemented with Kinect sensor service, is responsible for the movement of the cursor and simulating pointing operation.

9. Experiment ---5 participants
---Between subject design(randomize order)
---Independent variable: Kinect, Kinect + smartwatch
---Dependent variable: MT, Error Rate
---Experiment environment:
--- Tool: Fitts Law Study Application

10. Results
Movement Time(MT) comparison

11. Results
Error rate comparison

12. Results
Pair-wised T-test
-- P =
-- lesser than 0.005

13. Feedbacks --Cursor movement is not steady --Arm got tired easily
--Delay
--Not precise

14. Video demo

15. Things we learned & future work
Now we know that we can implement the hypothesis without Kinect.
Optimizations can be made during the data transmission process, so that reaction speed of the pointing operation will be improved.
Kinect sensor detection service can be optimized too, in doing that, the cursor movement can be much more faster, smooth, and accurate.