1. Dialog Design - Gesture & Pen Interfaces This material has been developed by Georgia Tech HCI faculty, and continues to evolve. Contributors include Gregory Abowd, Jim Foley, Elizabeth Mynatt, Jeff Pierce, Colin Potts, Chris Shaw, John Stasko, and Bruce Walker. Comments directed to foley@cc.gatech.edu are encouraged. Permission is granted to use with acknowledgement for non-profit purposes. Last revision: November 2010.foley@cc.gatech.edu

2. Dialog Styles 1. Command languages 2. WIMP - Window, Icon, Menu, Pointer 3. Direct manipulation 4. Speech/natural language 5. Gesture & pen UI Design - Georgia Tech 2

3. Many Types of Gesture-based UIs Purpose  Text input vs. selecting objects and actions on objects Gestures constrained to 2D surface vs. not constrained (i.e., 3D) Single vs. multiple points  One finger vs. multi-touch Sensed via something in/on user’s hand vs. not  Stylus on tablet vs. finger on touchpad  Data glove vs. video for hand gestures Size of sensing area  iPhone – iPad – tablet computer – table-top – wall - room Type/scale of user motor control  Finger movements vs. hand movements vs. body movements Work area used by one user vs. more than one UI Design - Georgia Tech 3

4. Agenda Look at several points in the gesture design space  PDAs such as iPhone, iPad, Palm Pilot Text input with some more general uses Scroll, resize, rotate Stylus or fingers (for multi-touch)  Multi-touch, shared work space  3d Gestures, video recognition UI Design - Georgia Tech 4

5. PDAs – the New Swiss Army Knife UI Design - Georgia Tech 5

6. What is a Personal Digital Assistant? A multi-function, portable device supporting activities such as  Communications by voice, text, video  Listen, read, watch – audio, news, books, video  Web browsing  Calendaring  Gaming  Social networking  Navigating from point A to point B  Managing personal information of all sorts  Etc etc etc  Aka iPhone, iPad – a small but powerful computer with pen or gesture input UI Design - Georgia Tech 6

7. Personal Digital Asst. (PDA) UI Design - Georgia Tech 7 Palm VII HP Jornada Handspring Visor Circa 2000 Palm IIIc Apple Newton (1993) Been around for a longggg time

8. Contemporary PDAs UI Design - Georgia Tech 8 iPhone Acer Android Tablet Motorola NokiaBlackBerry iPad

9. Interacting with a PDA Text input methods  Soft keyboard  Free-form ink – no recognition  Stylized characters to facilitate recognition  Superimposed on keyboard  Block printing recognition  Script recognition Stylus or finger UI Design - Georgia Tech 9

10. Soft Keyboards Pros/cons of soft keyboards vs. hard keyboards? Finger vs. stylus soft keyboards? UI Design - Georgia Tech 10

11. Free-form Ink Ink is the data, take as is Human is responsible for understanding and interpretation Like a sketch pad UI Design - Georgia Tech 11

12. Examples Digital Ink - CMU  Video, CHI ‘98  View it at www. …….. Flatland - Xerox PARC  Video, CHI ‘99  View it at www. …. UI Design - Georgia Tech 12

13. Gesture Recognition - Graffiti Stylized characters to facilitate recognition Graffiti - Unistroke alphabet on Palm PDA  Unistroke = one stroke UI Design - Georgia Tech 13

14. Gesture on Keyboard - Cirrin Developed by Jen Mankoff (GT Ph.D -> Berkeley -> CMU) Word-level unistroke technique UIST ‘98 paper Use stylus to go from one letter to the next -> UI Design - Georgia Tech 14

15. Gesture on Keyboard - Quikwriting Developed by Ken Perlin UIST ‘98 paper UI Design - Georgia Tech 15

16. Quikwriting UI Design - Georgia Tech 16 pl http://mrl.nyu.edu/projects/quikwriting/ e Said to be as fast as graffiti, but have to learn more

17. Gesture Recognition - Shapewriter Aka Shark Live demo on iPad UI Design - Georgia Tech 17

18. Gesture Recognition – ShapeWriter/Shark Video UI Design - Georgia Tech 18

19. Recognizing Block Printing and Script Recognizing letters and numbers and special symbols Lots of commercial systems English, kanji, etc. Not perfect, but people aren’t either!  People - 96% handprinted single characters  Computer - >97% is really good OCR (Optical Character Recognition) UI Design - Georgia Tech 19

20. Recognition Issues Off-line vs. On-line  Off-line: After all writing is done, speed not an issue, only quality. Work with either a bit map or vector sequence  On-line: Must respond in real-time - but have richer set of features - acceleration, velocity, pressure Use best-guess pattern matching, including digram, trigram probabilities and word lists to remove ambiguity and to correct errors  1 I l  On-line: show choices or best guess UI Design - Georgia Tech 20

21. More Issues Boxed vs. Free-Form input  Sometimes encounter boxes on forms Printed vs. Cursive  Cursive is much more difficult to impossible Letters vs. Words  Cursive is easier to do in words vs individual letters, as words create more context Similar to typo correction on phones and tablets UI Design - Georgia Tech 21

22. Recognizing Printed Characters UI Design - Georgia Tech 22

23. Beyond Text: Gestures – single point UI Design - Georgia Tech 23 -Might mean delete -Insert -Paragraph Define a series of (hopefully) simple drawing gestures that mean different commands in a system

24. Brown’s Line-o-grammer UI Design - Georgia Tech 24

25. Beyond Text: Gestures – Multiple Points UI Design - Georgia Tech 25 Mac Trackpad: up to four points of input For power users

26. Pen Use Modes Inputting text vs. selecting vs. action  On iPhone/iPad Enter text mode by selecting a text entry box Single touch => select Single touch + depress => move action Multi touch => other actions Other ways to switch modes?  Mode icon(s) – for instance to draw lines vs. print text  Others with which you are familiar?? UI Design - Georgia Tech 26

27. Creative Uses of Multi-touch Surfaces Liquid Text – Craig Tashman  On my Mac LiquidText | Multitouch Document Manipulation.flv Jeff Han  http://www.ted.com/talks/jeff_han_demos_hi s_breakthrough_touchscreen.html http://www.ted.com/talks/jeff_han_demos_hi s_breakthrough_touchscreen.html BrailleTouch UI Design - Georgia Tech 27

28. Wall-size Single Touch => Single User UI Design - Georgia Tech 28 Complete with pie menu

29. Multi-touch => Multi-User Multi-touch permits multi-users  Two users can simultaneously operate on two areas or display or on two objects But can’t determine which user is doing what UI Design - Georgia Tech 29

30. Wall-size, Multi-Touch => Multi- User UI Design - Georgia Tech 30

31. Table-size, Multi-Touch: MS Surface UI Design - Georgia Tech 31 Simultaneous user multi- touch No user ID No pressure Modest price

32. Microsoft Surface ‪ UI Design - Georgia Tech 32

33. Table-size, Multi-Touch: Mitsubishi Diamond Touch Simultaneous user multi- touch Pressure!! User ID!! Expensive!!!! UI Design - Georgia Tech 33

34. Diamond Touch Video On my Mac  Diamond Touch CeBit 2011 – YouTube.flv UI Design - Georgia Tech 34

35. 3D Gestures, Video Recognition Imaginary On my Mac  ‪ interactive ui in minority report ‬ ‏.flv UI Design - Georgia Tech 35

36. 3D Gestures, Video Recognition This one is real On my Mac  ‪ Hands Free 3D_ Second Life Object Editing Demo ‬ ‏.flv UI Design - Georgia Tech 36

37. A Different Application Signature verification But not with a mouse :) UI Design - Georgia Tech 37

38. Pen-less Gestures As in iPhone & iPad – 2D Minority Report  http://www.youtube.com/watch?v=NwVBzx0LMNQ&feature=related http://www.youtube.com/watch?v=NwVBzx0LMNQ&feature=related Toshiba  http://www.youtube.com/watch?v=MDUN01U--jE&feature=fvsr http://www.youtube.com/watch?v=MDUN01U--jE&feature=fvsr Multi-touch  http://www.youtube.com/watch?v=ok1n0lwY1ZA&feature=related http://www.youtube.com/watch?v=ok1n0lwY1ZA&feature=related Put that there – gesture & speech  http://www.youtube.com/watch?v=RyBEUyEtxQo http://www.youtube.com/watch?v=RyBEUyEtxQo UI Design - Georgia Tech 38

39. The End UI Design - Georgia Tech 39