1. The Pebbles Project General Overview: Using Hand-Held Computers and PCs Together Human Computer Interaction Institute School of Computer Science Carnegie Mellon University Brad A. Myers Carnegie Mellon University bam@cs.cmu.edu http://www.cs.cmu.edu/~pebbles Brad A. Myers Carnegie Mellon University bam@cs.cmu.edu http://www.cs.cmu.edu/~pebbles SUNY Technology Conference 2002

2. 2 Brad Myers Multiple Devices l Most of our time spent in places where there is embedded technology l Offices, meeting rooms, classrooms, homes l People have handheld devices l Palm, PocketPC, cell phone l Often multiple devices will be available: l Mobile phone and PDA and a PC l Multiple people’s PDAs l “Smart Classrooms” l PDAs in a “Smart Room” l Most of our time spent in places where there is embedded technology l Offices, meeting rooms, classrooms, homes l People have handheld devices l Palm, PocketPC, cell phone l Often multiple devices will be available: l Mobile phone and PDA and a PC l Multiple people’s PDAs l “Smart Classrooms” l PDAs in a “Smart Room”

3. SUNY Technology Conference 2002 3 Brad Myers Handhelds will be communicating l 802.11 l Wi-Fi l BlueTooth l Cell-phone network l 3G l (Infrared) l 802.11 l Wi-Fi l BlueTooth l Cell-phone network l 3G l (Infrared)

4. SUNY Technology Conference 2002 4 Brad Myers Premises of our Research “With the coming wireless technologies, connecting the PCs and PDAs together will no longer be an occasional event for synchronization. Instead, the devices will frequently be in close, interactive communication.” — Brad Myers, “Using Hand-Held Devices and PCs Together,” Comm. ACM, Vol. 44, No. 11. Nov., 2001. pp. 34 - 41.

5. SUNY Technology Conference 2002 5 Brad Myers Research Agenda How can multiple devices be used effectively together, at the same time? How can the user interface and functionality be spread across multiple devices? How can multiple devices be used effectively together, at the same time? How can the user interface and functionality be spread across multiple devices?

6. SUNY Technology Conference 2002 6 Brad Myers Research Agenda How can multiple devices be used effectively together, at the same time? How can the user interface and functionality be spread across multiple devices? How can multiple devices be used effectively together, at the same time? How can the user interface and functionality be spread across multiple devices? “Multi-Machine User Interfaces”

7. SUNY Technology Conference 2002 7 Brad Myers Example: Power Point Control l Use PC to give the presentation l Use hand-held to control the PC l Two-way communication l Hand-held shows picture of slide, notes, list of titles, timer, etc. l Easy to switch to demo l Easy to click on movies and sounds l Use PC to give the presentation l Use hand-held to control the PC l Two-way communication l Hand-held shows picture of slide, notes, list of titles, timer, etc. l Easy to switch to demo l Easy to click on movies and sounds

8. SUNY Technology Conference 2002 8 Brad Myers Pebbles is: PEBBLESPEBBLES PEBBLESPEBBLES DAs for ntry of oth ytes and ocations from xternalources. DAs for ntry of oth ytes and ocations from xternalources. http://www.pebbles.hcii.cmu.edu/

9. SUNY Technology Conference 2002 9 Brad Myers Handhelds in an Office l Augment desktop applications l Use multiple devices at the same time l Augment desktop applications l Use multiple devices at the same time

10. SUNY Technology Conference 2002 10 Brad Myers Handhelds in Meetings l Attendees use handhelds to interact and annotate presentation l Augment collaboration l Take notes l Attendees use handhelds to interact and annotate presentation l Augment collaboration l Take notes

11. SUNY Technology Conference 2002 11 Brad Myers Handhelds in Classrooms l Data projectors for instructor’s slides l Students could have computers for: l Notetaking linked to instructor’s slides l In-class testing l Running simulations l Data projectors for instructor’s slides l Students could have computers for: l Notetaking linked to instructor’s slides l In-class testing l Running simulations

12. SUNY Technology Conference 2002 12 Brad Myers Handhelds in Homes l Interact with embedded computation l “Smart homes” l Not just speech and vision as interfaces l Interact with appliances, lights, etc. l “Personal Universal Controller” (PUC) l Interact with embedded computation l “Smart homes” l Not just speech and vision as interfaces l Interact with appliances, lights, etc. l “Personal Universal Controller” (PUC)

13. Office Use (For Individuals) l How can handhelds augment desktop applications?

14. SUNY Technology Conference 2002 14 Brad Myers Applications for Individuals l Extra input and output devices have been shown to be useful l But can be expensive and hard to configure l People have PDAs and are attached to PC l For example, cradles for recharging l Customizable, extensible l Extend desktop applications l Extra input and output devices have been shown to be useful l But can be expensive and hard to configure l People have PDAs and are attached to PC l For example, cradles for recharging l Customizable, extensible l Extend desktop applications

15. SUNY Technology Conference 2002 15 Brad Myers Scrolling with the PDA l For scrolling using the non-dominant hand l Studies showed parallel and efficient uses of both hands together l Generates Windows scrolling events l For scrolling using the non-dominant hand l Studies showed parallel and efficient uses of both hands together l Generates Windows scrolling events

16. SUNY Technology Conference 2002 16 Brad Myers Results of Study of Scrolling with PDA l Scrolling with buttons on PDA was fastest l PDA scrollers similar to mouse speed l Using 2 hands is effective! l Scrolling with buttons on PDA was fastest l PDA scrollers similar to mouse speed l Using 2 hands is effective! Winner!

17. SUNY Technology Conference 2002 17 Brad Myers Remote Clipboard l Transfer information between PDA and PC l Connects their clipboards together l Transfer content or reference l Works with all applications l Also between multiple computers l Transfer information between PDA and PC l Connects their clipboards together l Transfer content or reference l Works with all applications l Also between multiple computers

18. SUNY Technology Conference 2002 18 Brad Myers Shortcutter l User-created panels of controls l Create custom interfaces and extensions to PC applications l And then take them with you l Direct manipulation for edit, then set properties l Palm or PocketPC l User-created panels of controls l Create custom interfaces and extensions to PC applications l And then take them with you l Direct manipulation for edit, then set properties l Palm or PocketPC

19. SUNY Technology Conference 2002 19 Brad Myers Shortcutter Widgets l Buttons l Sliders l Knobs l Mouse pad l Graffiti Pad (Palm) l Gesture panel l Buttons l Sliders l Knobs l Mouse pad l Graffiti Pad (Palm) l Gesture panel

20. SUNY Technology Conference 2002 20 Brad Myers Shortcutter Actions l Send any keyboard key, mouse button, scrolling action or string to PC l Open a file or URL l Run an application l Invoke any PC menu or button l Windows message l Recorded l Switch to a different Shortcutter panel l Control the Mouse l Send any keyboard key, mouse button, scrolling action or string to PC l Open a file or URL l Run an application l Invoke any PC menu or button l Windows message l Recorded l Switch to a different Shortcutter panel l Control the Mouse

21. SUNY Technology Conference 2002 21 Brad Myers Shortcutter Actions, cont. l Control external devices through PC’s serial port l Directly (e.g., projectors) l X-10 for electrical devices l Macro l Can be multi-application l Application-specific l Same button, different messages l Useful for application sets: browsers, compilers l Control external devices through PC’s serial port l Directly (e.g., projectors) l X-10 for electrical devices l Macro l Can be multi-application l Application-specific l Same button, different messages l Useful for application sets: browsers, compilers

22. SUNY Technology Conference 2002 22 Brad Myers More Scenarios of Use l Lean-back mail reading l Controlling WinAmp l … and many others l Lean-back mail reading l Controlling WinAmp l … and many others

23. Use in Meetings l Enhance group’s collaboration and control

24. SUNY Technology Conference 2002 24 Brad Myers Original Application: Remote Commander l Allow PDAs to control a PC l Can be used with any application l Uses the standard (single) cursor l Don’t have to jump up and grab mouse l Perform all mouse and keyboard functions l Use PDA like touchpad l Graffiti or our own pop-up keyboard l Allow PDAs to control a PC l Can be used with any application l Uses the standard (single) cursor l Don’t have to jump up and grab mouse l Perform all mouse and keyboard functions l Use PDA like touchpad l Graffiti or our own pop-up keyboard

25. SUNY Technology Conference 2002 25 Brad Myers PocketPC version l Get PC’s screen onto PocketPC l Full view, or one-to-one zooming l Scroll with iPaq’s buttons l Get PC’s screen onto PocketPC l Full view, or one-to-one zooming l Scroll with iPaq’s buttons

26. SUNY Technology Conference 2002 26 Brad Myers Scribble l Multiple people draw on top of whatever on PC screen, not just PowerPoint l Each user has own cursor and color l Save by PrintScreen l Erase by refresh l Multiple people draw on top of whatever on PC screen, not just PowerPoint l Each user has own cursor and color l Save by PrintScreen l Erase by refresh

27. SUNY Technology Conference 2002 27 Brad Myers MultiCursor l For special applications that are aware of multiple inputs l E.g.: Shared Whiteboard l Single Display Groupware l Multiple people, one display l A number of interesting issues: l Palettes, widgets l Section handles l Undo l For special applications that are aware of multiple inputs l E.g.: Shared Whiteboard l Single Display Groupware l Multiple people, one display l A number of interesting issues: l Palettes, widgets l Section handles l Undo

28. SUNY Technology Conference 2002 28 Brad Myers Chat l Communicate to another PDA user through the PC l PC serves as a conduit l For side notes and messages l For example, in negotiation meetings l Send to all or to a specific person l Communicate to another PDA user through the PC l PC serves as a conduit l For side notes and messages l For example, in negotiation meetings l Send to all or to a specific person

29. SUNY Technology Conference 2002 29 Brad Myers “Semantic Snarfing” l Interacting at a distance l Grab contents to handheld l Picture l Menus l Text l “Re-visualization” l “Private Drill-down of public information l Interacting at a distance l Grab contents to handheld l Picture l Menus l Text l “Re-visualization” l “Private Drill-down of public information

30. SUNY Technology Conference 2002 30 Brad Myers Study of Laser Pointers l Studied properties of laser pointer interaction techniques l Hand-wiggle +/-8 pixels l Delay until moving average stable +/-1.5 sec l Delay until target acquired +/-1.5 sec l Different devices and poses do not help much l Studied properties of laser pointer interaction techniques l Hand-wiggle +/-8 pixels l Delay until moving average stable +/-1.5 sec l Delay until target acquired +/-1.5 sec l Different devices and poses do not help much

31. Home Use l Personal Universal Controller

32. SUNY Technology Conference 2002 32 Brad Myers Personal Universal Controller l Two-way communication l Appliances describe their functions l Handheld PUC creates interface based on descriptions l Handheld PUC controls the appliance l Appliance sends back status as feedback l Two-way communication l Appliances describe their functions l Handheld PUC creates interface based on descriptions l Handheld PUC controls the appliance l Appliance sends back status as feedback Specifications Control Feedback of Status

33. SUNY Technology Conference 2002 33 Brad Myers Initial Experiments l Use real PocketPC l Pretend that controls devices l Use real PocketPC l Pretend that controls devices

34. SUNY Technology Conference 2002 34 Brad Myers Initial Study results l For both appliances, users of actual interfaces: l Took about twice as long l Made at least twice as many mistakes as users of the handheld interfaces l Needed external help five times more often l For both appliances, users of actual interfaces: l Took about twice as long l Made at least twice as many mistakes as users of the handheld interfaces l Needed external help five times more often

35. SUNY Technology Conference 2002 35 Brad Myers Current Steps l XML specification language and protocol for describing appliances l Create panels automatically from the specification l Finding real appliances which we can control l XML specification language and protocol for describing appliances l Create panels automatically from the specification l Finding real appliances which we can control Specification Language <spec xmlns="puc.xsd" xmlns:xsi="http://www.w3.org/2001/XMLSc hema-instance" xmlns:xsd="http://www.w3.org/2001/XMLS chema" name="Audiophase 5 CD Stereo"> Off On Stereo Power Power Powr Pwr 10

36. SUNY Technology Conference 2002 36 Brad Myers Any PC app. PebblesPC Brad’s Palm Pilot Rob’s Pocket PC Ben’s Windows CE RemoteCmd MultiCursor On the PC PebblesDraw Various PDA apps Slideshow Commander PowerPoint OLE Automation Various dlls General Architecture Windows event stream Direct connection or sockets Serial, IR or sockets

37. SUNY Technology Conference 2002 37 Brad Myers Protocols l Can use Pebbles protocols to develop your own application l Libraries for Palm, Windows CE, PC l Independence from communication medium l Use by: l PalmAmp from IronCreek Software l Intel research l others… l Windows messages or sockets l Can use Pebbles protocols to develop your own application l Libraries for Palm, Windows CE, PC l Independence from communication medium l Use by: l PalmAmp from IronCreek Software l Intel research l others… l Windows messages or sockets

38. SUNY Technology Conference 2002 38 Brad Myers Downloads l Most of this software is available for free downloading l Also, about 20 published papers l First release in Feb, 1998 l Recently released version 5 (Feb, 2002) l Downloaded over 30,000 times l About 200 times a week l Most of this software is available for free downloading l Also, about 20 published papers l First release in Feb, 1998 l Recently released version 5 (Feb, 2002) l Downloaded over 30,000 times l About 200 times a week http://www.pebbles.hcii.cmu.edu/

39. SUNY Technology Conference 2002 39 Brad Myers More in Next Session! l Next session: 11:20 - 12:05 l Details of use in classrooms and for the handicapped: l SlideShow Commander! l Concept Tests l Handhelds as Assistive Technology l Next session: 11:20 - 12:05 l Details of use in classrooms and for the handicapped: l SlideShow Commander! l Concept Tests l Handhelds as Assistive Technology

40. SUNY Technology Conference 2002 40 Brad Myers Conclusions l Handhelds are becoming ubiquitous l Connecting technologies will improve l Important to study how can be used when connected to computers and each other l “Multi-Machine User Interfaces” will be increasingly important l People will want to use the most convenient device for their information and control l Handhelds are becoming ubiquitous l Connecting technologies will improve l Important to study how can be used when connected to computers and each other l “Multi-Machine User Interfaces” will be increasingly important l People will want to use the most convenient device for their information and control

41. SUNY Technology Conference 2002 41 Brad Myers Thanks to Our Sponsors! Supported by grants from: DARPA Microsoft Pittsburgh Digital Greenhouse NSF General Motors And equipment grants from: Hewlett Packard Lucent Technologies Palm Computing Symbol Technologies Supported by grants from: DARPA Microsoft Pittsburgh Digital Greenhouse NSF General Motors And equipment grants from: Hewlett Packard Lucent Technologies Palm Computing Symbol Technologies IBM SMART Technologies, Inc. Synergy Solutions, Inc. Handango

42. The Pebbles Project General Overview: Using Hand-Held Computers and PCs Together Human Computer Interaction Institute School of Computer Science Carnegie Mellon University Brad A. Myers Carnegie Mellon University bam@cs.cmu.edu http://www.cs.cmu.edu/~pebbles Brad A. Myers Carnegie Mellon University bam@cs.cmu.edu http://www.cs.cmu.edu/~pebbles SUNY Technology Conference 2002