1. Final Honours Presentation Principal Investigator: João Lourenço Supervisor: Dr Hannah Thinyane

2. Presentation Outline  Introduction  Design and Implementation  User Study  Future Work  Conclusion 2

3. Introduction  Problem Statement  Research Motivation and Goals 3

4. Introduction  Problem Statement  Research Motivation and Goals 4

5. Problem Statement  Migration from 2D output to 3D output 3D TV Surround sound speakers  Suggests a need for migration from 2D input to 3D input Microsoft Kinect (previously Project Natal) Wii3D 5

6. Introduction  Problem Statement  Research Motivation and Goals 6

7. Research Motivation and Goals  Intuitive 3D Interaction Techniques are needed  Develop 3D Interaction using Nintendo Wii Remote  Compare Proposed interaction technique with existing techniques 7

8. Design and Implementation  Hardware  Software OS and Frameworks Bluetooth Libraries Point Tracker Gestures and Gesture Recognition User Interface 8

9. Design and Implementation  Hardware  Software OS and Frameworks Bluetooth Libraries Point Tracker Gestures and Gesture Recognition User Interface 9

10. Hardware  2 Nintendo Wii Remotes  Personal Computer with Bluetooth Capabilities  2 WiiGloves 10

11. Nintendo Wii Remotes  Layout Maximizing Volume 11

12. Nintendo Wii Remotes  Layout Maximizing Efficiency 12

13. WiiGlove  Circuit  2 Configurations 2 IRs per hand 4 IRs on one hand 13

14. Design and Implementation  Hardware  Software OS and Frameworks Bluetooth Libraries Point Tracker Gestures and Gesture Recognition User Interface 14

15. OS and Frameworks  Windows 7  Microsoft C#.NET 4.0 15

16. Design and Implementation  Hardware  Software OS and Frameworks Bluetooth Libraries Point Tracker Gestures and Gesture Recognition User Interface 16

17. Bluetooth  Microsoft Windows built-in Bluetooth Stack 17

18. Design and Implementation  Hardware  Software OS and Frameworks Bluetooth Libraries Point Tracker Gestures and Gesture Recognition User Interface 18

19. Libraries  WiimoteLib 1.7.0.0 – Wiimote Connection  Accord.NET 2.1.1.0 – Statistics  Math.NET Iridium 2008.8.16.470 – Linear algebra  32feet.NET 2.2.0.0 – Bluetooth Connection 19

20. Design and Implementation  Hardware  Software OS and Frameworks Bluetooth Libraries Point Tracker Gestures and Gesture Recognition User Interface 20

21. Point Tracker  Point tracking and probabilistic estimation were required to ensure that the system would handle multiple inputs (multitouch) and continue to function even in the event of the loss of points. A loss of points can occur for several reasons: An infra-red source can leave the field of view of the Wii Remote An infra-red source can move behind another infra-red source Two infra-red sources can come together and merge 21

22. Point Tracker  Several Components Stereoscopic Triangulation Solver Distance Minimizer Predictive Interpolator 22

23. Stereoscopic Triangulation  Achieved by solving for the point at the near intersection of two rays  Camera intrinsic properties needed Viewport width Viewport height Horizontal FOV Vertical FOV 23

24. Stereoscopic Triangulation 24

25. Stereoscopic Triangulation  Using the fields of view, a vector from the camera position to the detected point was obtained. These vectors can be thought of as rays from the infra-red source to the camera. σ the horizontal field of view ψ be the vertical field of view x be the horizontal component of the two dimensional coordinate y be the vertical component of the two dimensional coordinate 25

26. Stereoscopic Triangulation 26

27. Stereoscopic Triangulation  With the two rays from the two Wii Remote Cameras, the intersection of the two rays can be calculated using linear algebra. 27

28. Stereoscopic Triangulation 28

29. Stereoscopic Triangulation  Physical System – rays would never perfectly intersect  Need to take an estimate Estimate distance along first vector Estimate distance along second vector Average the two points and make note of the distance between them 29

30. 30

31. 31

32. Distance Minimization  The points that are reported by the two Wii Remotes are not necessarily sorted in the same order 32

33. Distance Minimization  Pairwise permutation of points in matrix form  Minimize distance error per row 33

34. Predictive Interpolation  Try to predict where a point should be – expected points Instantaneous velocity Average velocity over a period  Minimize the errors between the observed and expected points 34

35. Predictive Interpolation 35

36. Predictive Interpolation  The selected point per row is used as the point that the Wii3D System reports 36

37. Design and Implementation  Hardware  Software OS and Frameworks Bluetooth Libraries Point Tracker Gestures and Gesture Recognition User Interface 37

38. Gestures and Gesture Recognition  Supported gestures  Recognition methods 38

39. Click Gesture  A finger going down and forward, and then up and backward 39

40. Pan/Scroll Gesture  Two fingers that are close together moving in similar directions 40

41. Zoom Gesture  Two fingers moving in opposite directions with the distance between them increasing 41

42. Rotate Gesture  Two fingers moving in opposite directions with the distance between them constant 42

43. Circle Gesture  Hidden Markov Model Gesture 43

44. Multitouch Gestures  Number of points used for each gesture has been linked to the intensity of the gesture 44

45. Recognition  Movement – polling  Simple gestures – Finite State Automata  Complex Gestures – Hidden Markov Models 45

46. Design and Implementation  Hardware  Software OS and Frameworks Bluetooth Libraries Point Tracker Gestures and Gesture Recognition User Interface 46

47. User Interface  States: 47

48. User Interface  Upon start-up, the Wii3D System attempts to connect to the Wii Remotes 48

49. User Interface  Two possible errors – not 2 Wiimotes connected or no Wiimotes found 49

50. User Interface  Search for devices 50

51. User Interface  Found devices 51

52. User Interface  Allow the devices to install 52

53. User Interface  Connection success 53

54. User Interface  Start polling 54

55. User Interface  Currently polling 55

56. User Interface  Events received 56

57. User Study  Goals of the Experiment  Design and Methodology Tasks  Results 57

58. User Study  Goals of the Experiment  Design and Methodology Tasks  Results 58

59. Goals of the Experiment  To investigate whether the Wii3D System is a useful system  To investigate whether the Wii3D System could be used as an alternative to the mouse and keyboard 59

60. User Study  Goals of the Experiment  Design and Methodology Tasks  Results 60

61. Design and Methodology  20 participants  15 minutes per user  User consent required  No individual data stored, only collated results  Supervised by researcher 61

62. User Study  Goals of the Experiment  Design and Methodology Tasks  Results 62

63. Tasks  Introductory Questions  Tasks  Overview Questions 63

64. Introductory Questions  How long have you been using a mouse and keyboard for? 8 years  How often do you use a mouse and keyboard? 3 hours/day 64

65. Task 1 – Pointer Movement 65  Move pointers over targets

66. Task 2 - Clicking  Move over targets  Click gesture 66

67. Task 3 – Panning/Scrolling  Pan/Scroll and Image 67

68. Task 4 – Zooming  Zoom into and out of an image 68

69. Task 5 - Rotating  Rotate an image about the Z and X axes 69

70. Task 6 – Complex Gesture  Hidden Markov Models were used  Circle gesture 70

71. Overview Questions  Would you use the Wii3D System as an alternative to the mouse and keyboard? Why or why not?  If no, do you believe that the Wii3D System would be useful to someone else? Why?  Would you add/remove/change anything in the Wii3D System? Why?  You have undoubtedly had more experience using a mouse and keyboard than this type of system. What tasks would you find Wii3D more suitable for than a mouse and keyboard? 71

72. User Study  Goals of the Experiment  Design and Methodology Tasks  Results 72

73. Results  Preliminary Observations  Principal Results  Participant Subjective Feedback  Other Findings 73

74. Preliminary Observations  Confusion between rotation and zoom gestures  People are more used to mouse and keyboard  Hidden Markov Models not performing efficiently  Hidden Markov Models only successfully recognizing a circle gesture 50% of the time 74

75. Principal Results  None of the users would use Wii3D as a replacement for a mouse and Keyboard: Lack of typing support Gestures are not always recognized Accuracy of the Wii3D System pointers was not as good as a mouse 75

76. Principal Results  The majority of the participants stated that they thought that the Wii3D System would be useful 76

77. Movement  Mouse better for a single pointer  Wii3D better when there are more pointers 77

78. Clicking  Mouse better for single pointer clicks  Wii3D slightly better for multiple pointer clicks 78

79. Panning/Scrolling  Panning and scrolling is preferred with a mouse  Average and above ratings for Wii3D 79

80. Zooming  Zooming is preferred with the mouse  Best ratings out of all the gestures for Wii3D 80

81. Rotating  Preferred, again, is the mouse  Many users made zoom gestures instead 81

82. Complex Gesture  Circle gestures were recognized correctly 50% of the time using Hidden Markov Models  This result is reflected in the ten participants who rated this task as “Very Unintuitive” 82

83. Participant Subjective Feedback  Users thought that the Wii3D System would be useful for gaming and other three dimensional interactions  The users want typing functionality 83

84. Other Findings  The longer people have used computers, the more time they spend on them daily 84

85. Future Work  Improvements  Additions 85

86. Future Work  Improvements  Additions 86

87. Improvements  Multiple Wii Remotes  Acceleration in predictive interpolation  Dynamic Hidden Markov Models  Gaussian Mixture Models in HMM (continuous) 87

88. Future Work  Improvements  Additions 88

89. Additions  Gestures Double Click Flick Scaling an Object Typing  Artificial Neural Networks for Recognition 89

90. Conclusion  There is a use for the Wii3D System  The Wii3D System would not work as a replacement for the traditional mouse and keyboard 90

91. Questions/Comments 91