Dialog Design - Gesture & Pen Interfaces This material has been developed by Georgia Tech HCI faculty, and continues to evolve. Contributors include Gregory Abowd, Jim Foley, Elizabeth Mynatt, Jeff Pierce, Colin Potts, Chris Shaw, John Stasko, and Bruce Walker. Comments directed to are encouraged. Permission is granted to use with acknowledgement for non-profit purposes. Last revision: November

Dialog Styles 1. Command languages 2. WIMP - Window, Icon, Menu, Pointer 3. Direct manipulation 4. Speech/natural language 5. Gesture & pen UI Design - Georgia Tech 2

Many Types of Gesture-based UIs Purpose  Text input vs. selecting objects and actions on objects Gestures constrained to 2D surface vs. not constrained (i.e., 3D) Single vs. multiple points  One finger vs. multi-touch Sensed via something in/on user’s hand vs. not  Stylus on tablet vs. finger on touchpad  Data glove vs. video for hand gestures Size of sensing area  iPhone – iPad – tablet computer – table-top – wall - room Type/scale of user motor control  Finger movements vs. hand movements vs. body movements Work area used by one user vs. more than one UI Design - Georgia Tech 3

Agenda Look at several points in the gesture design space  PDAs such as iPhone, iPad, Palm Pilot Text input with some more general uses Scroll, resize, rotate Stylus or fingers (for multi-touch)  Multi-touch, shared work space  3d Gestures, video recognition UI Design - Georgia Tech 4

PDAs – the New Swiss Army Knife UI Design - Georgia Tech 5

What is a Personal Digital Assistant? A multi-function, portable device supporting activities such as  Communications by voice, text, video  Listen, read, watch – audio, news, books, video  Web browsing  Calendaring  Gaming  Social networking  Navigating from point A to point B  Managing personal information of all sorts  Etc etc etc  Aka iPhone, iPad – a small but powerful computer with pen or gesture input UI Design - Georgia Tech 6

Personal Digital Asst. (PDA) UI Design - Georgia Tech 7 Palm VII HP Jornada Handspring Visor Circa 2000 Palm IIIc Apple Newton (1993) Been around for a longggg time

Contemporary PDAs UI Design - Georgia Tech 8 iPhone Acer Android Tablet Motorola NokiaBlackBerry iPad

Interacting with a PDA Text input methods  Soft keyboard  Free-form ink – no recognition  Stylized characters to facilitate recognition  Superimposed on keyboard  Block printing recognition  Script recognition Stylus or finger UI Design - Georgia Tech 9

Soft Keyboards Pros/cons of soft keyboards vs. hard keyboards? Finger vs. stylus soft keyboards? UI Design - Georgia Tech 10

Free-form Ink Ink is the data, take as is Human is responsible for understanding and interpretation Like a sketch pad UI Design - Georgia Tech 11

Examples Digital Ink - CMU  Video, CHI ‘98  View it at www. …….. Flatland - Xerox PARC  Video, CHI ‘99  View it at www. …. UI Design - Georgia Tech 12

Gesture Recognition - Graffiti Stylized characters to facilitate recognition Graffiti - Unistroke alphabet on Palm PDA  Unistroke = one stroke UI Design - Georgia Tech 13

Gesture on Keyboard - Cirrin Developed by Jen Mankoff (GT Ph.D -> Berkeley -> CMU) Word-level unistroke technique UIST ‘98 paper Use stylus to go from one letter to the next -> UI Design - Georgia Tech 14

Gesture on Keyboard - Quikwriting Developed by Ken Perlin UIST ‘98 paper UI Design - Georgia Tech 15

Quikwriting UI Design - Georgia Tech 16 pl e Said to be as fast as graffiti, but have to learn more

Gesture Recognition - Shapewriter Aka Shark Live demo on iPad UI Design - Georgia Tech 17

Gesture Recognition – ShapeWriter/Shark Video UI Design - Georgia Tech 18

Recognizing Block Printing and Script Recognizing letters and numbers and special symbols Lots of commercial systems English, kanji, etc. Not perfect, but people aren’t either!  People - 96% handprinted single characters  Computer - >97% is really good OCR (Optical Character Recognition) UI Design - Georgia Tech 19

Recognition Issues Off-line vs. On-line  Off-line: After all writing is done, speed not an issue, only quality. Work with either a bit map or vector sequence  On-line: Must respond in real-time - but have richer set of features - acceleration, velocity, pressure Use best-guess pattern matching, including digram, trigram probabilities and word lists to remove ambiguity and to correct errors  1 I l  On-line: show choices or best guess UI Design - Georgia Tech 20

More Issues Boxed vs. Free-Form input  Sometimes encounter boxes on forms Printed vs. Cursive  Cursive is much more difficult to impossible Letters vs. Words  Cursive is easier to do in words vs individual letters, as words create more context Similar to typo correction on phones and tablets UI Design - Georgia Tech 21

Recognizing Printed Characters UI Design - Georgia Tech 22

Beyond Text: Gestures – single point UI Design - Georgia Tech 23 -Might mean delete -Insert -Paragraph Define a series of (hopefully) simple drawing gestures that mean different commands in a system

Brown’s Line-o-grammer UI Design - Georgia Tech 24

Beyond Text: Gestures – Multiple Points UI Design - Georgia Tech 25 Mac Trackpad: up to four points of input For power users

Pen Use Modes Inputting text vs. selecting vs. action  On iPhone/iPad Enter text mode by selecting a text entry box Single touch => select Single touch + depress => move action Multi touch => other actions Other ways to switch modes?  Mode icon(s) – for instance to draw lines vs. print text  Others with which you are familiar?? UI Design - Georgia Tech 26

Creative Uses of Multi-touch Surfaces Liquid Text – Craig Tashman  On my Mac LiquidText | Multitouch Document Manipulation.flv Jeff Han  s_breakthrough_touchscreen.html s_breakthrough_touchscreen.html BrailleTouch UI Design - Georgia Tech 27

Wall-size Single Touch => Single User UI Design - Georgia Tech 28 Complete with pie menu

Multi-touch => Multi-User Multi-touch permits multi-users  Two users can simultaneously operate on two areas or display or on two objects But can’t determine which user is doing what UI Design - Georgia Tech 29

Wall-size, Multi-Touch => Multi- User UI Design - Georgia Tech 30

Table-size, Multi-Touch: MS Surface UI Design - Georgia Tech 31 Simultaneous user multi- touch No user ID No pressure Modest price

Microsoft Surface ‪ UI Design - Georgia Tech 32

Table-size, Multi-Touch: Mitsubishi Diamond Touch Simultaneous user multi- touch Pressure!! User ID!! Expensive!!!! UI Design - Georgia Tech 33

Diamond Touch Video On my Mac  Diamond Touch CeBit 2011 – YouTube.flv UI Design - Georgia Tech 34

3D Gestures, Video Recognition Imaginary On my Mac  ‪ interactive ui in minority report ‬ ‏.flv UI Design - Georgia Tech 35

3D Gestures, Video Recognition This one is real On my Mac  ‪ Hands Free 3D_ Second Life Object Editing Demo ‬ ‏.flv UI Design - Georgia Tech 36

A Different Application Signature verification But not with a mouse :) UI Design - Georgia Tech 37

Pen-less Gestures As in iPhone & iPad – 2D Minority Report  Toshiba  Multi-touch  Put that there – gesture & speech  UI Design - Georgia Tech 38

The End UI Design - Georgia Tech 39