Davide Spano CNR-ISTI, HIIS Laboratory, Via G. Moruzzi Pisa, Italy

Gestures 16 May 2012Davide Spano Gesture Interfaces

Remotes (1)  Nintendo Wiimote, 2006  Buttons  Three-axis linear accelerometer  IR camera  Three axis gyroscope extension (Wii Motion Plus)  Game market opened to casual gamers 16 May 2012Davide Spano Gesture Interfaces

Remotes (2)  Playstation Move, 2010  Buttons  Spherical RGB light emitter plus camera  Three-axis accelerometer and gyroscope  Compass 16 May 2012Davide Spano Gesture Interfaces

Floor Devices (1)  Dance pads  Konami Dance Dance Revolution (1998)  Huge directional pads  Buttons 16 May 2012Davide Spano Gesture Interfaces

Floor Devices (2)  Wii Balance Board  Four pressure sensors  Snowboard games  Fitness games 16 May 2012Davide Spano Gesture Interfaces

Motion Tracking (1)  CamSpace, 2009  Object recognition  No haptic feedback 16 May 2012Davide Spano Gesture Interfaces

Motion Tracking (2)  Microsoft Kinect, 2009  RGB camera  Depth sensor  Microphone array  Still no haptic feedback 16 May 2012Davide Spano Gesture Interfaces

Multitouch  Two or more touch-sensed position on the screen  Patents already on 1984  iPhone 2007  Support in Mac OS X, Windows 7, Ubuntu … 16 May 2012Davide Spano Gesture Interfaces

Gestures Overview  Gesture types:  Static  Dynamic  Spatio-temporal  Conceptual:  Gestures are continuous  Gestures exploit time and space as arguments  Gestures enable single user parallel interaction  Need for a suitable programming paradigm 16 May 2012Davide Spano Gesture Interfaces

The Midas Touch 16 May 2012Davide Spano Gesture Interfaces TrackingEngaged Out Of range Tracking Not Tracking Mouse Touches, Skeleton Tracking Accelerometers, Gyroscope

Current Programming Model  Do gestures really fit this pattern? 16 May 2012Davide Spano Gesture Interfaces

Multitouch Frameworks 16 May 2012Davide Spano Gesture Interfaces Gesture Events Touch Events EndMoveStartRotatePinchSlide

Example: Pinch for zoom (1) 16 May 2012Davide Spano Gesture Interfaces

Example: Pinch for zoom (2) 16 May 2012Davide Spano Gesture Interfaces Start Move End

Full body tracking 16 May 2012Davide Spano Gesture Interfaces

Microsoft Kinect SDK 16 May 2012Davide Spano Gesture Interfaces

Primesense NITE Controls  State of the Art toolkit for full body gesture interaction  Reusable controls (recognizers) for NUI  Control list  Push detector  Swipe detector  Steady detector  Wave detector  Circle detector  Selectable slider (1D/2D)  All based on the observer pattern 16 May 2012Davide Spano Gesture Interfaces

Circle Control Sample 16 May 2012Davide Spano Gesture Interfaces

One Event is not enough! 16 May 2012Davide Spano Gesture Interfaces Pan Move Start Handler End 1 1 Gesture Touch ID Samples Handler 1

Gesture Definition Meta Model  Declarative  Compositional  Ground Terms (basic building blocks)  Composition Operators  Behaviour Association  Entire Gesture  Any other sub-component  Abstract Gesture Recognition Support  Formally defined through Petri Nets 16 May 2012Davide Spano Gesture Interfaces

Break… 16 May 2012Davide Spano Gesture Interfaces

Gesture Support Abstraction 16 May 2012Davide Spano Gesture Interfaces

Ground Terms and Composition Operators  Low-level device events as ground terms  Optional conditions on support state  Composition operators 16 May 2012Davide Spano Gesture Interfaces

Multitouch Gesture Support 16 May 2012Davide Spano Gesture Interfaces

Multitouch Gesture Description 16 May 2012Davide Spano Gesture Interfaces

Proton (1) 16 May 2012Davide Spano Gesture Interfaces

Proton (2) 16 May 2012Davide Spano Gesture Interfaces

Full Body Gesture Platform  Skeleton tracked as a set of joints  2 user full tracking + 4 only body centre  Features:  Joint (user id, joint id, position, orientation)  Current time 16 May 2012Davide Spano Gesture Interfaces

Full Body Example (1) 16 May 2012Davide Spano Gesture Interfaces Front Not Front Right hand Both hands

Full Body Example 16 May 2012Davide Spano Gesture Interfaces

A different programming model  Describe declaratively the gesture  Create new ones using the building blocks  Reuse existing ones through composition  Code the event handlers  Attach them to the events at the desired granularity  It is called only when it is needed (no further checks)  Temporal description and interface behaviour are completely separated  Complex Gestures can be stored as classes  They can still expose their low level events 16 May 2012Davide Spano Gesture Interfaces

Library Support 16 May 2012Davide Spano Gesture Interfaces

Multitouch with Full Body Experiment 16 May 2012Davide Spano Gesture Interfaces

Proof of Concept Applications 16 May 2012Davide Spano Gesture Interfaces

Demo Videos  Parallel Interaction with iPhone videovideo  “Multitouch” with Kinect videovideo  Same Gesture Definition! 16 May 2012Davide Spano Gesture Interfaces

Conclusion 16 May 2012Davide Spano Gesture Interfaces

References  Spano, L.D. A Model Based Approach for gesture interfaces (EICS 2011) pdfpdf  Kin K., Hartmann B., DeRose T., Agrawala M. Proton: Multitouch Gestures as Regular Expressions pdfpdf  Kinect for Windows Kinect for Windows  OpenNI OpenNI  Primesense NITE Primesense NITE 16 May 2012Davide Spano Gesture Interfaces