The Pebbles Project General Overview: Using Hand-Held Computers and PCs Together Human Computer Interaction Institute School of Computer Science Carnegie Mellon University Brad A. Myers Carnegie Mellon University Brad A. Myers Carnegie Mellon University SUNY Technology Conference 2002

2 Brad Myers Multiple Devices l Most of our time spent in places where there is embedded technology l Offices, meeting rooms, classrooms, homes l People have handheld devices l Palm, PocketPC, cell phone l Often multiple devices will be available: l Mobile phone and PDA and a PC l Multiple people’s PDAs l “Smart Classrooms” l PDAs in a “Smart Room” l Most of our time spent in places where there is embedded technology l Offices, meeting rooms, classrooms, homes l People have handheld devices l Palm, PocketPC, cell phone l Often multiple devices will be available: l Mobile phone and PDA and a PC l Multiple people’s PDAs l “Smart Classrooms” l PDAs in a “Smart Room”

SUNY Technology Conference Brad Myers Handhelds will be communicating l l Wi-Fi l BlueTooth l Cell-phone network l 3G l (Infrared) l l Wi-Fi l BlueTooth l Cell-phone network l 3G l (Infrared)

SUNY Technology Conference Brad Myers Premises of our Research “With the coming wireless technologies, connecting the PCs and PDAs together will no longer be an occasional event for synchronization. Instead, the devices will frequently be in close, interactive communication.” — Brad Myers, “Using Hand-Held Devices and PCs Together,” Comm. ACM, Vol. 44, No. 11. Nov., pp

SUNY Technology Conference Brad Myers Research Agenda How can multiple devices be used effectively together, at the same time? How can the user interface and functionality be spread across multiple devices? How can multiple devices be used effectively together, at the same time? How can the user interface and functionality be spread across multiple devices?

SUNY Technology Conference Brad Myers Research Agenda How can multiple devices be used effectively together, at the same time? How can the user interface and functionality be spread across multiple devices? How can multiple devices be used effectively together, at the same time? How can the user interface and functionality be spread across multiple devices? “Multi-Machine User Interfaces”

SUNY Technology Conference Brad Myers Example: Power Point Control l Use PC to give the presentation l Use hand-held to control the PC l Two-way communication l Hand-held shows picture of slide, notes, list of titles, timer, etc. l Easy to switch to demo l Easy to click on movies and sounds l Use PC to give the presentation l Use hand-held to control the PC l Two-way communication l Hand-held shows picture of slide, notes, list of titles, timer, etc. l Easy to switch to demo l Easy to click on movies and sounds

SUNY Technology Conference Brad Myers Pebbles is: PEBBLESPEBBLES PEBBLESPEBBLES DAs for ntry of oth ytes and ocations from xternalources. DAs for ntry of oth ytes and ocations from xternalources.

SUNY Technology Conference Brad Myers Handhelds in an Office l Augment desktop applications l Use multiple devices at the same time l Augment desktop applications l Use multiple devices at the same time

SUNY Technology Conference Brad Myers Handhelds in Meetings l Attendees use handhelds to interact and annotate presentation l Augment collaboration l Take notes l Attendees use handhelds to interact and annotate presentation l Augment collaboration l Take notes

SUNY Technology Conference Brad Myers Handhelds in Classrooms l Data projectors for instructor’s slides l Students could have computers for: l Notetaking linked to instructor’s slides l In-class testing l Running simulations l Data projectors for instructor’s slides l Students could have computers for: l Notetaking linked to instructor’s slides l In-class testing l Running simulations

SUNY Technology Conference Brad Myers Handhelds in Homes l Interact with embedded computation l “Smart homes” l Not just speech and vision as interfaces l Interact with appliances, lights, etc. l “Personal Universal Controller” (PUC) l Interact with embedded computation l “Smart homes” l Not just speech and vision as interfaces l Interact with appliances, lights, etc. l “Personal Universal Controller” (PUC)

Office Use (For Individuals) l How can handhelds augment desktop applications?

SUNY Technology Conference Brad Myers Applications for Individuals l Extra input and output devices have been shown to be useful l But can be expensive and hard to configure l People have PDAs and are attached to PC l For example, cradles for recharging l Customizable, extensible l Extend desktop applications l Extra input and output devices have been shown to be useful l But can be expensive and hard to configure l People have PDAs and are attached to PC l For example, cradles for recharging l Customizable, extensible l Extend desktop applications

SUNY Technology Conference Brad Myers Scrolling with the PDA l For scrolling using the non-dominant hand l Studies showed parallel and efficient uses of both hands together l Generates Windows scrolling events l For scrolling using the non-dominant hand l Studies showed parallel and efficient uses of both hands together l Generates Windows scrolling events

SUNY Technology Conference Brad Myers Results of Study of Scrolling with PDA l Scrolling with buttons on PDA was fastest l PDA scrollers similar to mouse speed l Using 2 hands is effective! l Scrolling with buttons on PDA was fastest l PDA scrollers similar to mouse speed l Using 2 hands is effective! Winner!

SUNY Technology Conference Brad Myers Remote Clipboard l Transfer information between PDA and PC l Connects their clipboards together l Transfer content or reference l Works with all applications l Also between multiple computers l Transfer information between PDA and PC l Connects their clipboards together l Transfer content or reference l Works with all applications l Also between multiple computers

SUNY Technology Conference Brad Myers Shortcutter l User-created panels of controls l Create custom interfaces and extensions to PC applications l And then take them with you l Direct manipulation for edit, then set properties l Palm or PocketPC l User-created panels of controls l Create custom interfaces and extensions to PC applications l And then take them with you l Direct manipulation for edit, then set properties l Palm or PocketPC

SUNY Technology Conference Brad Myers Shortcutter Widgets l Buttons l Sliders l Knobs l Mouse pad l Graffiti Pad (Palm) l Gesture panel l Buttons l Sliders l Knobs l Mouse pad l Graffiti Pad (Palm) l Gesture panel

SUNY Technology Conference Brad Myers Shortcutter Actions l Send any keyboard key, mouse button, scrolling action or string to PC l Open a file or URL l Run an application l Invoke any PC menu or button l Windows message l Recorded l Switch to a different Shortcutter panel l Control the Mouse l Send any keyboard key, mouse button, scrolling action or string to PC l Open a file or URL l Run an application l Invoke any PC menu or button l Windows message l Recorded l Switch to a different Shortcutter panel l Control the Mouse

SUNY Technology Conference Brad Myers Shortcutter Actions, cont. l Control external devices through PC’s serial port l Directly (e.g., projectors) l X-10 for electrical devices l Macro l Can be multi-application l Application-specific l Same button, different messages l Useful for application sets: browsers, compilers l Control external devices through PC’s serial port l Directly (e.g., projectors) l X-10 for electrical devices l Macro l Can be multi-application l Application-specific l Same button, different messages l Useful for application sets: browsers, compilers

SUNY Technology Conference Brad Myers More Scenarios of Use l Lean-back mail reading l Controlling WinAmp l … and many others l Lean-back mail reading l Controlling WinAmp l … and many others

Use in Meetings l Enhance group’s collaboration and control

SUNY Technology Conference Brad Myers Original Application: Remote Commander l Allow PDAs to control a PC l Can be used with any application l Uses the standard (single) cursor l Don’t have to jump up and grab mouse l Perform all mouse and keyboard functions l Use PDA like touchpad l Graffiti or our own pop-up keyboard l Allow PDAs to control a PC l Can be used with any application l Uses the standard (single) cursor l Don’t have to jump up and grab mouse l Perform all mouse and keyboard functions l Use PDA like touchpad l Graffiti or our own pop-up keyboard

SUNY Technology Conference Brad Myers PocketPC version l Get PC’s screen onto PocketPC l Full view, or one-to-one zooming l Scroll with iPaq’s buttons l Get PC’s screen onto PocketPC l Full view, or one-to-one zooming l Scroll with iPaq’s buttons

SUNY Technology Conference Brad Myers Scribble l Multiple people draw on top of whatever on PC screen, not just PowerPoint l Each user has own cursor and color l Save by PrintScreen l Erase by refresh l Multiple people draw on top of whatever on PC screen, not just PowerPoint l Each user has own cursor and color l Save by PrintScreen l Erase by refresh

SUNY Technology Conference Brad Myers MultiCursor l For special applications that are aware of multiple inputs l E.g.: Shared Whiteboard l Single Display Groupware l Multiple people, one display l A number of interesting issues: l Palettes, widgets l Section handles l Undo l For special applications that are aware of multiple inputs l E.g.: Shared Whiteboard l Single Display Groupware l Multiple people, one display l A number of interesting issues: l Palettes, widgets l Section handles l Undo

SUNY Technology Conference Brad Myers Chat l Communicate to another PDA user through the PC l PC serves as a conduit l For side notes and messages l For example, in negotiation meetings l Send to all or to a specific person l Communicate to another PDA user through the PC l PC serves as a conduit l For side notes and messages l For example, in negotiation meetings l Send to all or to a specific person

SUNY Technology Conference Brad Myers “Semantic Snarfing” l Interacting at a distance l Grab contents to handheld l Picture l Menus l Text l “Re-visualization” l “Private Drill-down of public information l Interacting at a distance l Grab contents to handheld l Picture l Menus l Text l “Re-visualization” l “Private Drill-down of public information

SUNY Technology Conference Brad Myers Study of Laser Pointers l Studied properties of laser pointer interaction techniques l Hand-wiggle +/-8 pixels l Delay until moving average stable +/-1.5 sec l Delay until target acquired +/-1.5 sec l Different devices and poses do not help much l Studied properties of laser pointer interaction techniques l Hand-wiggle +/-8 pixels l Delay until moving average stable +/-1.5 sec l Delay until target acquired +/-1.5 sec l Different devices and poses do not help much

Home Use l Personal Universal Controller

SUNY Technology Conference Brad Myers Personal Universal Controller l Two-way communication l Appliances describe their functions l Handheld PUC creates interface based on descriptions l Handheld PUC controls the appliance l Appliance sends back status as feedback l Two-way communication l Appliances describe their functions l Handheld PUC creates interface based on descriptions l Handheld PUC controls the appliance l Appliance sends back status as feedback Specifications Control Feedback of Status

SUNY Technology Conference Brad Myers Initial Experiments l Use real PocketPC l Pretend that controls devices l Use real PocketPC l Pretend that controls devices

SUNY Technology Conference Brad Myers Initial Study results l For both appliances, users of actual interfaces: l Took about twice as long l Made at least twice as many mistakes as users of the handheld interfaces l Needed external help five times more often l For both appliances, users of actual interfaces: l Took about twice as long l Made at least twice as many mistakes as users of the handheld interfaces l Needed external help five times more often

SUNY Technology Conference Brad Myers Current Steps l XML specification language and protocol for describing appliances l Create panels automatically from the specification l Finding real appliances which we can control l XML specification language and protocol for describing appliances l Create panels automatically from the specification l Finding real appliances which we can control Specification Language <spec xmlns="puc.xsd" xmlns:xsi=" hema-instance" xmlns:xsd=" chema" name="Audiophase 5 CD Stereo"> Off On Stereo Power Power Powr Pwr 10

SUNY Technology Conference Brad Myers Any PC app. PebblesPC Brad’s Palm Pilot Rob’s Pocket PC Ben’s Windows CE RemoteCmd MultiCursor On the PC PebblesDraw Various PDA apps Slideshow Commander PowerPoint OLE Automation Various dlls General Architecture Windows event stream Direct connection or sockets Serial, IR or sockets

SUNY Technology Conference Brad Myers Protocols l Can use Pebbles protocols to develop your own application l Libraries for Palm, Windows CE, PC l Independence from communication medium l Use by: l PalmAmp from IronCreek Software l Intel research l others… l Windows messages or sockets l Can use Pebbles protocols to develop your own application l Libraries for Palm, Windows CE, PC l Independence from communication medium l Use by: l PalmAmp from IronCreek Software l Intel research l others… l Windows messages or sockets

SUNY Technology Conference Brad Myers Downloads l Most of this software is available for free downloading l Also, about 20 published papers l First release in Feb, 1998 l Recently released version 5 (Feb, 2002) l Downloaded over 30,000 times l About 200 times a week l Most of this software is available for free downloading l Also, about 20 published papers l First release in Feb, 1998 l Recently released version 5 (Feb, 2002) l Downloaded over 30,000 times l About 200 times a week

SUNY Technology Conference Brad Myers More in Next Session! l Next session: 11: :05 l Details of use in classrooms and for the handicapped: l SlideShow Commander! l Concept Tests l Handhelds as Assistive Technology l Next session: 11: :05 l Details of use in classrooms and for the handicapped: l SlideShow Commander! l Concept Tests l Handhelds as Assistive Technology

SUNY Technology Conference Brad Myers Conclusions l Handhelds are becoming ubiquitous l Connecting technologies will improve l Important to study how can be used when connected to computers and each other l “Multi-Machine User Interfaces” will be increasingly important l People will want to use the most convenient device for their information and control l Handhelds are becoming ubiquitous l Connecting technologies will improve l Important to study how can be used when connected to computers and each other l “Multi-Machine User Interfaces” will be increasingly important l People will want to use the most convenient device for their information and control

SUNY Technology Conference Brad Myers Thanks to Our Sponsors! Supported by grants from: DARPA Microsoft Pittsburgh Digital Greenhouse NSF General Motors And equipment grants from: Hewlett Packard Lucent Technologies Palm Computing Symbol Technologies Supported by grants from: DARPA Microsoft Pittsburgh Digital Greenhouse NSF General Motors And equipment grants from: Hewlett Packard Lucent Technologies Palm Computing Symbol Technologies IBM SMART Technologies, Inc. Synergy Solutions, Inc. Handango

The Pebbles Project General Overview: Using Hand-Held Computers and PCs Together Human Computer Interaction Institute School of Computer Science Carnegie Mellon University Brad A. Myers Carnegie Mellon University Brad A. Myers Carnegie Mellon University SUNY Technology Conference 2002