1. Davide Spano lucio.davide.spano@isti.cnr.it lucio.davide.spano@isti.cnr.it spano@di.unipi.it CNR-ISTI, HIIS Laboratory, Via G. Moruzzi 1 56124 Pisa, Italy

2. Gestures 16 May 2012Davide Spano Gesture Interfaces

3. Remotes (1)  Nintendo Wiimote, 2006  Buttons  Three-axis linear accelerometer  IR camera  Three axis gyroscope extension (Wii Motion Plus)  Game market opened to casual gamers 16 May 2012Davide Spano Gesture Interfaces

4. Remotes (2)  Playstation Move, 2010  Buttons  Spherical RGB light emitter plus camera  Three-axis accelerometer and gyroscope  Compass 16 May 2012Davide Spano Gesture Interfaces

5. Floor Devices (1)  Dance pads  Konami Dance Dance Revolution (1998)  Huge directional pads  Buttons 16 May 2012Davide Spano Gesture Interfaces

6. Floor Devices (2)  Wii Balance Board  Four pressure sensors  Snowboard games  Fitness games 16 May 2012Davide Spano Gesture Interfaces

7. Motion Tracking (1)  CamSpace, 2009  Object recognition  No haptic feedback 16 May 2012Davide Spano Gesture Interfaces

8. Motion Tracking (2)  Microsoft Kinect, 2009  RGB camera  Depth sensor  Microphone array  Still no haptic feedback 16 May 2012Davide Spano Gesture Interfaces

9. Multitouch  Two or more touch-sensed position on the screen  Patents already on 1984  iPhone 2007  Support in Mac OS X, Windows 7, Ubuntu … 16 May 2012Davide Spano Gesture Interfaces

10. Gestures Overview  Gesture types:  Static  Dynamic  Spatio-temporal  Conceptual:  Gestures are continuous  Gestures exploit time and space as arguments  Gestures enable single user parallel interaction  Need for a suitable programming paradigm 16 May 2012Davide Spano Gesture Interfaces

11. The Midas Touch 16 May 2012Davide Spano Gesture Interfaces TrackingEngaged Out Of range Tracking Not Tracking Mouse Touches, Skeleton Tracking Accelerometers, Gyroscope

12. Current Programming Model  Do gestures really fit this pattern? 16 May 2012Davide Spano Gesture Interfaces

13. Multitouch Frameworks 16 May 2012Davide Spano Gesture Interfaces Gesture Events Touch Events EndMoveStartRotatePinchSlide

14. Example: Pinch for zoom (1) 16 May 2012Davide Spano Gesture Interfaces

15. Example: Pinch for zoom (2) 16 May 2012Davide Spano Gesture Interfaces Start Move End

16. Full body tracking 16 May 2012Davide Spano Gesture Interfaces

17. Microsoft Kinect SDK 16 May 2012Davide Spano Gesture Interfaces

18. Primesense NITE Controls  State of the Art toolkit for full body gesture interaction  Reusable controls (recognizers) for NUI  Control list  Push detector  Swipe detector  Steady detector  Wave detector  Circle detector  Selectable slider (1D/2D)  All based on the observer pattern 16 May 2012Davide Spano Gesture Interfaces

19. Circle Control Sample 16 May 2012Davide Spano Gesture Interfaces

20. One Event is not enough! 16 May 2012Davide Spano Gesture Interfaces Pan Move Start Handler End 1 1 Gesture Touch ID Samples Handler 1

21. Gesture Definition Meta Model  Declarative  Compositional  Ground Terms (basic building blocks)  Composition Operators  Behaviour Association  Entire Gesture  Any other sub-component  Abstract Gesture Recognition Support  Formally defined through Petri Nets 16 May 2012Davide Spano Gesture Interfaces

22. Break… 16 May 2012Davide Spano Gesture Interfaces http://www.foxtrot.com/

23. Gesture Support Abstraction 16 May 2012Davide Spano Gesture Interfaces

24. Ground Terms and Composition Operators  Low-level device events as ground terms  Optional conditions on support state  Composition operators 16 May 2012Davide Spano Gesture Interfaces

25. Multitouch Gesture Support 16 May 2012Davide Spano Gesture Interfaces

26. Multitouch Gesture Description 16 May 2012Davide Spano Gesture Interfaces

27. Proton (1) 16 May 2012Davide Spano Gesture Interfaces

28. Proton (2) 16 May 2012Davide Spano Gesture Interfaces

29. Full Body Gesture Platform  Skeleton tracked as a set of joints  2 user full tracking + 4 only body centre  Features:  Joint (user id, joint id, position, orientation)  Current time 16 May 2012Davide Spano Gesture Interfaces

30. Full Body Example (1) 16 May 2012Davide Spano Gesture Interfaces Front Not Front Right hand Both hands

31. Full Body Example 16 May 2012Davide Spano Gesture Interfaces

32. A different programming model  Describe declaratively the gesture  Create new ones using the building blocks  Reuse existing ones through composition  Code the event handlers  Attach them to the events at the desired granularity  It is called only when it is needed (no further checks)  Temporal description and interface behaviour are completely separated  Complex Gestures can be stored as classes  They can still expose their low level events 16 May 2012Davide Spano Gesture Interfaces

33. Library Support 16 May 2012Davide Spano Gesture Interfaces

34. Multitouch with Full Body Experiment 16 May 2012Davide Spano Gesture Interfaces

35. Proof of Concept Applications 16 May 2012Davide Spano Gesture Interfaces

36. Demo Videos  Parallel Interaction with iPhone videovideo  “Multitouch” with Kinect videovideo  Same Gesture Definition! 16 May 2012Davide Spano Gesture Interfaces

37. Conclusion 16 May 2012Davide Spano Gesture Interfaces http://www.mojocomic.com/

38. References  Spano, L.D. A Model Based Approach for gesture interfaces (EICS 2011) pdfpdf  Kin K., Hartmann B., DeRose T., Agrawala M. Proton: Multitouch Gestures as Regular Expressions pdfpdf  Kinect for Windows Kinect for Windows  OpenNI OpenNI  Primesense NITE Primesense NITE 16 May 2012Davide Spano Gesture Interfaces