1. Information Sciences and Systems Kam Lai Janusz Konrad Prakash Ishwar GestureMouse: A Flexible Gesture-based User Interface http://vip.bu.edu/ {laikamho, jkonrad, pi}@bu.edu

2. Introduction  What? Novel hand gesture recognition system using Kinect that is:  Simple: only 3 tuning parameters  Accurate: > 95% recognition for different users, > 99% for same user!  Robust: to user height, body-shape, clothing  One-shot: just one example needed!  Customizable: user can select any gesture on-the-fly  Real-time: instantaneous training and recognition  What for?  Interactive information displays, gaming, authentication, music, dance, surgery, athletics, … GestureMouse 1 Lai, Konrad, Ishwar

3. How? Big Picture: GestureMouse 2 Lai, Konrad, Ishwar Nearest- Neighbor Classifier Query Gesture Label Gesture Dictionary: Log-covariances of skeleton-based hand gesture features... Gesture-1 Gesture-2... Gesture-N

4. GestureMouse 3 Lai, Konrad, Ishwar NN Classification in Feature-Covariance Space Label Gesture Dictionary: Log-covariances of skeleton-based hand gesture features... Gesture-1 Gesture-2... Gesture-N Query Gesture Parameter #1

5. GestureMouse 4 Lai, Konrad, Ishwar NN Classification in Feature-Covariance Space Gesture Dead time Gesture time  Only three parameters need to be set: - Log-covariance distance threshold θ (previous slide) - Hand movement threshold (beginning of gesture) - Gesture length (1 – 5 seconds)

6. Demo-1: Pre-defined Dictionary  Control a public information display  Open, browse, zoom-in/out, documents, images, … GestureMouse 5 Lai, Konrad, Ishwar

7. Demo-2: User-defined Dictionary  Users can train the system with long hand gestures (up to 5 seconds)  Users can adjust the log-covariance distance threshold θ to allow/prevent gestures of others from being recognized  Works robustly for complex gestures  Can be used for:  Controlling presentations, displays, …  Customizing PC gaming controller  Developing new gesture-based mimicry and memory games  Physical training, therapy, dance practice, education, …  Revocable biometric authentication GestureMouse 6 Lai, Konrad, Ishwar