1. The Challenges and Potential of End- User Gesture Customization Uran Oh 1 and Leah Findlater 2 1 Department of Computer Science 2 College of Information Studies University of Maryland, College Park  uranoh@cs.umd.edu | leahkf@umd.edu

2. Touchscreen gestures are widely used… Who designs these gestures? Design experts. Apple’s touchpad gestures

3. 1)Tools for supporting designers (developers) to create gestures with ease Previous Research: A figure from Gesture Coder MAGIC: [Ashbrook et al. 2010] Proton++: [Kin et al. 2012] Gesture Coder: [Lü et al. 2012]

4. 2) Methods for creating a gesture set that are intuitive and guessable by a wide range of users [Wobbrock et al. 2009], [Kray et al. 2010], [Ruiz et al. 2011] Previous Research: A figure from [Wobbrock et al. 2009]

5. (2) Methods for creating a gesture set that are intuitive and guessable by a wide range of users [Wobbrock et al. 2009], [Kray et al. 2010], [Ruiz et al. 2011] Previous research: A figure from [Wobbrock et al. 2009] Our focus: Supporting end-users Personal gestures for a single user

6. (2) Methods for creating a gesture set that are intuitive and guessable by a wide range of users [Wobbrock et al. 2009], [Kray et al. 2010], [Ruiz et al. 2011] Previous research: A figure from [Wobbrock et al. 2009] Our focus: Supporting end-users Personal gestures for a single user Why?

7. Memorability Efficiency Accessibility Potential Advantages of Self-defined Gestures…

8. [Nacenta et al. 2013] Memorability  Self-defined gestures improve memorability over predefined gestures

9. [Ouyang et al. 2012] Efficiency  Gestural shortcuts can be used as an efficient mean of accessing information

10. Accessibility [Anthony et al. 2013]  Customized gestures may improve accessibility for people with physical disabilities

11. Our Goal: To investigate the feasibility of end-user gesture creation 

12. Our Goal: To investigate the feasibility of end-user gesture creation  How do typical users create gestures? What are the challenges therein? How can we support the process?

13. v Task 1Task 2 Task 3Task 2 Open-Ended Gesture Creation Action-Specific Gesture Creation Saliency of Gesture Features Study With Three Tasks

14. Controlled lab study - 20 participants (age from 20 to 35, M=29.3) - Prior experience with touchscreen devices - Single one-hour session with 3 tasks - Think-aloud protocol Study Method  Apparatus - Samsung Galaxy Tab 2 (10.1’’ running Android 4.0.4)

15. v Task 1 Task 2 Task 3Task 2 Open-Ended Gesture Creation Action-Specific Gesture Creation Saliency of Gesture Features Are users able to create new gestures easily? If not, what are the barriers?

16.  Task 1: Open-ended Gesture Creation “Create as many gestures as possible”

17.  Task 1: Open-ended Gesture Creation For any purpose Any number of strokes, fingers, hands As long as they are: easy to draw, easy to remember, distinguishable

18.  Task 1: Open-ended Gesture Creation “Create as many gestures as possible”

19. 12.2 gestures created on average (SD = 8.1, range of 5 to 36) Gestures Created p3 Total number of gestures and the number of arbitrary gestures are correlated (Pearson’s r=.47, p=.037) Task 1: Open-ended Gesture Creation p5

20. 12.2 gestures created on average (SD = 8.1, range of 5 to 36) Gestures Created p3 p5 Total number of gestures and the number of arbitrary gestures are correlated (Pearson’s r=.47, p=.037) Task 1: Open-ended Gesture Creation p5

21. Tendency to focus on the familiar “I just thought of gestures my tablet PC had.” (P1) “These gestures are all I use, I cannot be more creative” (P8) Difficulties Creating Gestures  Opaque nature of gesture recognizer “Can I use all fingers?” (P2) Task 1: Open-ended Gesture Creation

22. v Task 1 Task 2 Task 3Task 2 Open-Ended Gesture Creation Action-Specific Gesture Creation Saliency of Gesture Features Users felt difficulty in creating new gestures Better understanding of recognizer is needed

23. 23 Task 1 Task 2 Task 3 Task 2 Open-Ended Gesture Creation Action-Specific Gesture Creation Saliency of Gesture Features What is a “good gesture” to end-users? How is it different from recognizer’s perspective?

24. Task 2: Action-Specific Gesture Creation Brainstorm gestures per action 12 Specific Actions Zoom-in Zoom-out Rotate Copy Cut Paste Select-single Select-multiple Previous Next Call-Mom Launch a web-browser

25. 12 Specific Actions Zoom-in Zoom-out Rotate Copy Cut Paste Select-single Select-multiple Previous Next Call-Mom Launch a web-browser Task 2: Action-Specific Gesture Creation Brainstorm gestures per action

26. Task 2: Action-Specific Gesture Creation Compose custom set of gestures, one per action Brainstorm gestures per action

27. Task 2: Action-Specific Gesture Creation Compose custom set of gestures, one per action Brainstorm gestures per action

28. Task 2: Action-Specific Gesture Creation Compose custom set of gestures, one per action Brainstorm gestures per action

29. Task 2: Action-Specific Gesture Creation Brainstorm gestures per action Compose custom set of gestures, one per action Create training examples (4 per selected gesture)

30. Task 2: Action-Specific Gesture Creation Brainstorm gestures per action Compose custom set of gestures, one per action Create training examples (4 per selected gesture) Rate satisfaction with the custom gesture set

31. Brainstorm gestures per action Compose custom set of gestures, one per action Create training examples (4 per selected gesture) Rate satisfaction with the custom gesture set Test recognition accuracy with $N recognizer Initial example Training examples Task 2: Action-Specific Gesture Creation

32. Reasons for selecting a gesture for custom set Others reasons: Generally preferred, fast, consistent, easy to remember, etc. Task 2: Action-Specific Gesture Creation

33. Need for improvement Participants gave up the opportunity to edit their gesture set to make improvements Task 2: Action-Specific Gesture Creation Only two participants were fully satisfied ( M=5.3, SD = 1.1 where 1=negative, 7=positive) Inability to improve gesture sets

34. Low Recognition Potential of the Custom Sets $N recognizer (default setting) with 5-fold cross validation  76–88% accuracy depending on amount of training Task 2: Action-Specific Gesture Creation

35. 35 Task 2 Task 3 Task 2 Action-Specific Gesture Creation Saliency of Gesture Features Customized set can be improved for both user’s and recognizer’s perspective Task 1 Open-Ended Gesture Creation

36. 36 Task 2 Task 3 Task 2 Action-Specific Gesture Creation Saliency of Gesture Features What features do users rely on to distinguish between gestures? Task 1 Open-Ended Gesture Creation

37. Gesture Features Judged Orientation Very slowVery fastslowfastmoderate Scale Aspect Ratio Speed Task 3: Saliency of Gesture Features Curviness Pattern Repetition 6 features from Rubine’s recognizer [Rubine. 1991]

38. Gesture Features Judged Orientation Scale Aspect Ratio Task 3: Saliency of Gesture Features Curviness Pattern Repetition Finger Count Stroke Count Stroke Order 3 touchscreen features 6 features from Rubine’s recognizer [Rubine. 1991] Very slowVery fastslowfastmoderate Speed

39. Orientation Scale Aspect Ratio Task 3: Saliency of Gesture Features Curviness Pattern Repetition Finger Count Stroke Count Stroke Order 3 touchscreen features 6 features from Rubine’s recognizer [Rubine. 1991] “Rank the distinguishability of 9 features” Very slowVery fastslowfastmoderate Speed

40. Objective features are more distinguishable Features that can be consistently interpreted/manipulated are considered distinguishable “Even if the same person is performing the gesture, it might not have the same speed and size” (P7) More distinctive Very fast Speed Scale Pattern Repetition Aspect Ratio Curviness Orientation Stroke Order Stroke count Finger count Task 3: Saliency of Gesture Features

41. Objective features are more distinguishable Features that can be consistently interpreted/manipulated are considered distinguishable “Even if the same person is performing the gesture, it might not have the same speed and size” (P7) More distinctive Very fast Speed Scale Pattern Repetition Aspect Ratio Curviness Orientation Stroke Order Stroke count Finger count Task 3: Saliency of Gesture Features

42. Objective features are more distinguishable Features that can be consistently interpreted/manipulated are considered distinguishable “Even if the same person is performing the gesture, it might not have the same speed and size” (P7) More distinctive Very fast Speed Scale Pattern Repetition Aspect Ratio Curviness Orientation Stroke Order Stroke count Finger count Task 3: Saliency of Gesture Features

43. 43 Task 2 Task 3 Task 2 Action-Specific Gesture Creation Saliency of Gesture Features Task 1 Open-Ended Gesture Creation Number of fingers/strokes, stroke order are distinguishable than speed or size

44. Summary Creating new gestures is hard for end-users Tendency to focus on the familiar Opaque nature of gesture recognizer  Objective features are more distinguishable Finger/stroke count, stroke order are more distinguishable than speed and scale Quality of gesture sets can be improved Users are not fully satisfied with their gesture sets Low recognition potential

45. Memorability Efficiency Accessibility  Potential Benefits of Allowing End-User Customization Take-away Message Systematic Support is Needed for End-User Customization

46. Future Work Mixed-initiative support for customization Feedback EditsTrain System Gesture set User

47. 47 The Challenges and Potential of End- User Gesture Customization Uran Oh 1 and Leah Findlater 2 1 Department of Computer Science 2 College of Information Studies University of Maryland, College Park  uranoh@cs.umd.edu | leahkf@umd.edu