1. People, Pens, and Computers François Guimbretière HCIL, University of Maryland francois@cs.umd.edu Ken Hinckley MSR kenh@microsoft.com

2. New devices, old tasks Middle picture from Sellen et al.

3. People, Pens, and Tablet PC The New Yorker Illustration from Ken Hinckley presentation at Stanford

4. Typical setting for today’s interface Fixed stable environment, with a keyboard, Indirect interaction, High precision pointing system

5. Typical Tablet PC use Portable, unstable environment, without a keyboard Direct interaction, Low precision aiming

6. Pen based interface for Tablet PC Document area How to create a more fluid interface? Interface framework How to make it more pen-friendly?

7. Pen based interface for Tablet PC Interface framework How to make it more pen-friendly?

8. Empirical foundations Use of strokes to cross target is more pen friendly Crossing is as efficient as point-and-click [Accot & Zhai, 2002] The basic interactor How expressive is it? CrossY: crossing-based GUI [Apitz & Guimbretière 04]

9. CrossY video

10. Previous Work Theoretical basis Steering Law, Trajectory-Based Tasks [Accot & Zhai 97-02] Limited scope examples Toggle Map [Baudish 98] Lotus Notes: multiple e-mail selection Conceptual design Visual Instruments: [Winograd & Guimbretière 98] Overloading Gedrics: [Geißler 95]

11. Command composition From stroke-by-stroke interaction Borders are used to validate/cancel

12. Command composition From stroke-by-stroke interaction Borders are used to validate/cancel To multi-command stroke

13. Scrolling Line by line area Page by page area Absolute area

14. CrossY scrollbar Overloading simplify interactions Shorter distances to issue commands Not as much precision necessary

15. CrossY scrollbar Overloading simplify interactions Shorter distances to issue commands Not as much precision necessary Extending stroke for repeat No need to wait for a timeout

16. Cursor control Cross to jump to an absolute position

17. Cursor control Cross to jump to an absolute position

18. Cursor control Cross to jump to an absolute position Near drag for coarse adjustment

19. Cursor control Cross to jump to an absolute position Near drag for coarse adjustment

20. Cursor control Cross to jump to an absolute position Near drag for coarse adjustment Far drag for fine adjustment Similar to FineSlider [Masui 95] But one single stroke

21. Use of directionality Continuous find and replace

22. Use of directionality Continuous find and replace Reverse direction for undo

23. What we learned Very well received by users HCIL open house UIST Space requirements Similar to point-and-click Trade-off with command combination due to sloppiness Overloading vs. easy discovery Consistency helps with adoption Known in Windowing systems

24. Pen based interface for Tablet PC Document area How to create a more fluid interface?

25. Scriboli and Stitching Ken’s presentation

26. My desk

27. Affordances of paper documents [Sellen 01] Easy to navigate Two-handed interactions and tactile feedback Reading across more than one document at once Easy to annotate Directly on the document or on a nearby pad Well accepted during meetings Socially accepted conventions Very difficult to modify Printed documents are created and edited as digital documents Expensive to distribute and archive

28. Bridging the gap: previous work Digital emulation FreeStyle system [Wang 89] MATE [Hardock 93] XLibris [Schilit 98], [Golovchinsky 02] Tight coupling DigitalDesk [Wellner 93], Ariel [Mackay 95] A-Book [Mackay 02] PaperLink [Arai 97] Intelligent Paper [Dymetman 98] Paper as input device Xax [Johnson 93] Anoto Paper PDA [Heiner 99], [Avrahami 01]

29. Cohabitation [Guimbretière 03]

30. Stroke capture From Anoto documentation Requirement Stroke coordinates on the page Page ID Large address space Possible technologies Anoto DataGlyphs [Hecht 94] MEMO pen [Nabeshima 95]

31. System architecture

32. My Desk with PADD

33. PapierCraft: command system for PADD [Liao, Guimbretiere and Hinckley 05] PADD document Anoto pen PADD notepad

34. PapierCraft: using PADD as proxies CopyPaste SynchronizationOn your Tablet PC

35. PapierCraft Gesture/Ink Use the command button Scopes Command selection Marking menu Writing down an unambiguous prefix of a command Batch mode processing

36. Early feedback and future work Small scale evaluation Commands with written words were very popular Higher recognition rate needed for strokes only feedback Contextual information might provide enough feedback during pastes Future work More reliable recognition engine Pen-based feedback Haptic feedback, LED Real time processing : streaming stroke

37. Conclusions Four pen-based interfaces CrossY explores crossing-based interfaces Scriboli explores high performance direct manipulation interfaces Stitching explores multi-devices interactions PapierCraft uses PADD as document proxies One goal Bring the ease of use of pen and paper to digital interfaces Future Seamless integration of both media

38. Acknowledgments Collaborators Jim Hollan Students Georg Apitz, Nicholas Chen (CrossY/ScribolY) Kevin Conroy, Dave Levin, Chunyuan Liao (PADD/ProofRite) Sponsors Microsoft HP Anoto, Logitech, Maxell Colleagues and friends Corinna Löckenhoff Samrat (Bobby) Bhattacharjee Ben Bederson