1. Dialog Design - Gesture & Pen Interfaces, Mobile Devices IAT 334 1 This material has been developed by Georgia Tech HCI faculty, and continues to evolve. Contributors include Gregory Abowd, Jim Foley, Elizabeth Mynatt, Jeff Pierce, Colin Potts, Chris Shaw, John Stasko, Bruce Walker, and Melody Moore Jackson. Comments directed to foley@cc.gatech.edu are encouraged. Permission is granted to use with acknowledgement for non-profit purposes. Last revision: October 2007. foley@cc.gatech.edu

2. Dialog Styles 1. Command languages 2. WIMP - Window, Icon, Menu, Pointer 3. Direct manipulation 4. Speech/natural language 5. Gesture & pen IAT 334 2

3. Agenda PDA overview Pen input styles IAT 334 3

4. How to use a PDA IAT 334 4

5. Personal Digital Asst. (PDA) IAT 334 5 Palm Treo Apple Newton (1993) Apple iPhone Blackberry Curve

6. PDAs Now ubiquitous Small displays Often touch and pen interfaces Small thumb-based keyboards Recent Improvements Wi-Fi, GPS, more memory, better CPU, better OS, BlueTooth IAT 334 6

7. Is it a PDA? Phone? GPS? Camera? Computer? Line between devices is blurred today Apple iPhone – phone, MP3 player, PDA, camera Palm Treo 700w – phone, Windows computer, PDA, camera Nokia N82 – Phone, GPS, 2 cameras, robot brain IAT 334 7

8. Cally/Callo IAT 3348

9. No Shredder… IAT 334 9

10. Input Options Pen / Stylus is dominant form Main techniques Free-form ink Soft keyboard Numeric keyboard => text Stroke recognition - strokes not in the shape of characters Hand printing / writing recognition Sometimes have or can connect keyboard IAT 334 10

11. Free-form Ink Ink is the data, take as is Human is responsible for understanding and interpretation Like a sketch pad IAT 334 11

12. Soft Keyboards Common on PDAs and mobile devices IAT 334 12

13. Soft Keyboard Presents a small diagram of keyboard You click on buttons/keys with pen or finger QWERTY vs. alphabetical Tradeoffs? Alternatives? 13 IAT 334 Apple iPhone soft keyboard

14. Numeric Keypad -T9 Tegic Communications developed You press out letters of your word, it matches the most likely word, then gives optional choices Faster than multiple presses per key Used in mobile phones www.tegic.com/t9 IAT 334 14

15. Stroke Recognition - Graffiti Graffiti - Unistroke alphabet on Palm PDA What are your experiences with Graffiti? Graffiti demo: http://www.youtube.com/watch?v=R-hbjG2hzuk IAT 334 15

16. Stroke Recognition - Cirrin Developed by Jen Mankoff (GT -> Berkeley CS Faculty) Word-level unistroke technique UIST ‘98 paper Use stylus to go from one letter to the next -> Nokia N8 does similar with QWERTY layout IAT 334 16

17. QuikWriting Break the gesture into octant components: Start pen in center, – drag in one of 8 directions – drag along edge – drag to center http://mrl.nyu.edu/~perlin/demos/quikwriting2_1.html IAT 33417

18. Hand Printing & Hand Writing Recognition Recognizing letters and numbers and special symbols Lots of commercial systems English, kanji, etc. Not perfect, but people aren’t either! People - 96% handprinted single characters Computer - >97% is really good OCR (Optical Character Recognition) IAT 334 18

19. Recognition Issues Off-line vs. On-line Off-line: After all writing is done, speed not an issue, only quality. Work with either a bit map or vector sequence On-line: Must respond in real-time - but have richer set of features - acceleration, velocity, pressure Use best-guess pattern matching, including digram, trigram probabilities and word lists to remove ambiguity 1 I l IAT 334 19

20. More Issues Boxed vs. Free-Form input Sometimes encounter boxes on forms Printed vs. Cursive Cursive is much more difficult to impossible Letters vs. Words Cursive is easier to do in words vs individual letters, as words create more context IAT 334 20

21. Pen Gesture Commands IAT 334 21 -Might mean delete -Insert -Paragraph Define a series of (hopefully) simple drawing gestures that mean different commands in a system

22. Pen Use Modes Often, want a mix of free-form drawing and special commands How does user switch modes? Mode icon on screen Button on pen Button on device IAT 334 22

23. Error Correction Having to correct errors can slow input tremendously Strategies Erase and try again When uncertain system shows list of best guesses... IAT 334 23

24. A Different Application Signature verification But not with a mouse :) IAT 334 24

25. Multi-touch interfaces Apple iPhone IAT 334 25 Capacitive touchscreen: http://www.apple.com/iphone/features/index.html#touch Gestures: flick, tap, pinch, un-pinch http://www.apple.com/iphone/features/index.html#map

26. Pen Videos Pick-and-Drop by Rekimoto http://www.youtube.com/watch?v=rFw9aMubL-Y http://www.youtube.com/watch?v=rFw9aMubL-Y I-Love-Sketch by Seok-Hyung Bae http://www.youtube.com/watch?v=5Hd2clwURlQ http://www.youtube.com/watch?v=5Hd2clwURlQ Jabberstamp by Hayes Raffle http://www.youtube.com/watch?v=NIe-XDHcsOE http://www.youtube.com/watch?v=NIe-XDHcsOE ShapeWriter iPhone App http://www.youtube.com/watch?v=TOg91yfvZpo http://www.youtube.com/watch?v=TOg91yfvZpo Marginalia : The Hybrid Textbook http://www.youtube.com/watch?v=zwYdmsdqqTk http://www.youtube.com/watch?v=zwYdmsdqqTk IAT 33426