Using Hand-Held Computers and PCs Together: The Pebbles Project Human Computer Interaction Institute School of Computer Science Carnegie Mellon University Brad A. Myers Brad A. Myers

Brad Myers 2 Multiple Devices l Most of our time spent in places where there is embedded technology l Offices, meeting rooms, classrooms, homes l Often multiple devices will be available: l Mobile phone and PDA and a PC l Multiple people’s PDAs l PDAs in a “Smart Room” l Most of our time spent in places where there is embedded technology l Offices, meeting rooms, classrooms, homes l Often multiple devices will be available: l Mobile phone and PDA and a PC l Multiple people’s PDAs l PDAs in a “Smart Room”

Brad Myers 3 Handhelds will be communicating l l BlueTooth l Cell-phone network l (Infrared) l l BlueTooth l Cell-phone network l (Infrared)

Brad Myers 4 Premises of our Research “With the coming wireless technologies, connecting the PCs and PDAs together will no longer be an occasional event for synchronization. Instead, the devices will frequently be in close, interactive communication.” — Brad Myers, “Using Hand-Held Devices and PCs Together,” Comm. ACM, Vol. 44, No. 11. Nov., pp

Brad Myers 5 Research Agenda How can multiple devices be used effectively together, at the same time? How can the user interface and functionality be spread across multiple devices? How can multiple devices be used effectively together, at the same time? How can the user interface and functionality be spread across multiple devices?

Brad Myers 6 Research Agenda How can multiple devices be used effectively together, at the same time? How can the user interface and functionality be spread across multiple devices? How can multiple devices be used effectively together, at the same time? How can the user interface and functionality be spread across multiple devices? “Multi-Machine User Interfaces”

Brad Myers 7 Example: Power Point Control l Use PC to give the presentation l Use hand-held to control the PC l Two-way communication l Hand-held shows picture of slide, notes, list of titles, timer, etc. l Use PC to give the presentation l Use hand-held to control the PC l Two-way communication l Hand-held shows picture of slide, notes, list of titles, timer, etc.

Brad Myers 8 Pebbles is: PEBBLESPEBBLES PEBBLESPEBBLES DAs for ntry of oth ytes and ocations from xternalources. DAs for ntry of oth ytes and ocations from xternalources.

Brad Myers 9 Handhelds in an Office l Augment desktop applications l Use multiple devices at the same time l Augment desktop applications l Use multiple devices at the same time

Brad Myers 10 Handhelds in Meetings l Attendees use handhelds to interact and annotate presentation l Augment collaboration l Take notes l Attendees use handhelds to interact and annotate presentation l Augment collaboration l Take notes

Brad Myers 11 Handhelds in Classrooms l Data projectors for instructor’s slides l Students could have computers for: l Notetaking linked to instructor’s slides l In-class testing l Running simulations l Data projectors for instructor’s slides l Students could have computers for: l Notetaking linked to instructor’s slides l In-class testing l Running simulations

Brad Myers 12 Handhelds in Homes l Interact with embedded computation l “Smart homes” l Not just speech and vision as interfaces l Interact with appliances, lights, etc. l “Personal Universal Controller” (PUC) l Interact with embedded computation l “Smart homes” l Not just speech and vision as interfaces l Interact with appliances, lights, etc. l “Personal Universal Controller” (PUC)

Office Use (For Individuals) l How can handhelds augment desktop applications?

Brad Myers 14 Applications for Individuals l Extra input and output devices have been shown to be useful l But can be expensive and hard to configure l People have PDAs and are attached to PC l For example, cradles for recharging l Customizable, extensible l Extend desktop applications l Extra input and output devices have been shown to be useful l But can be expensive and hard to configure l People have PDAs and are attached to PC l For example, cradles for recharging l Customizable, extensible l Extend desktop applications

Brad Myers 15 Scrolling with the PDA l For scrolling using the non-dominant hand l Studies showed parallel and efficient uses of both hands together l Generates Windows scrolling events l For scrolling using the non-dominant hand l Studies showed parallel and efficient uses of both hands together l Generates Windows scrolling events

Brad Myers 16 Results of Study of Scrolling with PDA l Scrolling with buttons on PDA was fastest l PDA scrollers similar to mouse speed l Using 2 hands is effective! l Scrolling with buttons on PDA was fastest l PDA scrollers similar to mouse speed l Using 2 hands is effective! Winner!

Brad Myers 17 Remote Clipboard l Transfer information between PDA and PC l Connects their clipboards together l Transfer content or reference l Works with all applications l Also between multiple computers l Transfer information between PDA and PC l Connects their clipboards together l Transfer content or reference l Works with all applications l Also between multiple computers

Brad Myers 18 Shortcutter l User-created panels of controls l Create custom interfaces and extensions to PC applications l And then take them with you l Direct manipulation for edit, then set properties l Palm or PocketPC l User-created panels of controls l Create custom interfaces and extensions to PC applications l And then take them with you l Direct manipulation for edit, then set properties l Palm or PocketPC

Brad Myers 19 Shortcutter Widgets l Buttons l Sliders l Knobs l Mouse pad l Graffiti Pad (Palm) l Gesture panel l Buttons l Sliders l Knobs l Mouse pad l Graffiti Pad (Palm) l Gesture panel

Brad Myers 20 Shortcutter Actions l Send any keyboard key, mouse button, scrolling action or string to PC l Open a file or URL l Run an application l Invoke any PC menu or button l Windows message l Recorded l Switch to a different Shortcutter panel l Control the Mouse l Send any keyboard key, mouse button, scrolling action or string to PC l Open a file or URL l Run an application l Invoke any PC menu or button l Windows message l Recorded l Switch to a different Shortcutter panel l Control the Mouse

Brad Myers 21 Shortcutter Actions, cont. l Control external devices through PC’s serial port l Directly (e.g., projectors) l X-10 for electrical devices l Macro l Can be multi-application l Application-specific l Same button, different messages l Useful for application sets: browsers, compilers l Control external devices through PC’s serial port l Directly (e.g., projectors) l X-10 for electrical devices l Macro l Can be multi-application l Application-specific l Same button, different messages l Useful for application sets: browsers, compilers

Brad Myers 22 More Scenarios of Use l Lean-back mail reading l Controlling WinAmp l … and many others l Lean-back mail reading l Controlling WinAmp l … and many others

Brad Myers 23 Study of Individual Use l Time to tap on button depended on size l Few errors l People often didn’t look at PDA l Time to tap on button depended on size l Few errors l People often didn’t look at PDA

Brad Myers 24 Study of Individual Use l Moving hands to both PDA and mouse only about 15% slower than just moving to the mouse 1H Keyboard->Mouse728 1H Keyboard->PDA744 1H Mouse->Keyboard701 1H PDA->Keyboard639 Keyboard -> Mouse&PDA % Mouse&PDA -> Keyboard % % slower msec

Use in Meetings l Enhance group’s collaboration and control

Brad Myers 26 Original Application: Remote Commander l Allow PDAs to control a PC l Can be used with any application l Uses the standard (single) cursor l Don’t have to jump up and grab mouse l Perform all mouse and keyboard functions l Use PDA like touchpad l Graffiti or our own pop-up keyboard l Allow PDAs to control a PC l Can be used with any application l Uses the standard (single) cursor l Don’t have to jump up and grab mouse l Perform all mouse and keyboard functions l Use PDA like touchpad l Graffiti or our own pop-up keyboard

Brad Myers 27 PocketPC version l Get PC’s screen onto PocketPC l Full view, or one-to-one zooming l Scroll with iPaq’s buttons l Get PC’s screen onto PocketPC l Full view, or one-to-one zooming l Scroll with iPaq’s buttons

Brad Myers 28 Scribble l Multiple people draw on top of whatever on PC screen, not just PowerPoint l Each user has own cursor and color l Save by PrintScreen l Erase by refresh l Multiple people draw on top of whatever on PC screen, not just PowerPoint l Each user has own cursor and color l Save by PrintScreen l Erase by refresh

Brad Myers 29 SlideShow Commander l For PowerPoint l PC Only: Windows 95, 98, NT, 2000, ME, XP l Use PC to give the presentation l Full features of PowerPoint l Use hand-held as “remote control” for PC l Can wander away from the keyboard l Two-way communication l View, Navigate, Highlight, Time l For PowerPoint l PC Only: Windows 95, 98, NT, 2000, ME, XP l Use PC to give the presentation l Full features of PowerPoint l Use hand-held as “remote control” for PC l Can wander away from the keyboard l Two-way communication l View, Navigate, Highlight, Time

Brad Myers 30 View on Hand-Held: slide l See thumb-nail of current slide l Black and white or color l See thumb-nail of current slide l Black and white or color

Brad Myers 31 View on Hand-Held: notes l See the notes of the current slide

Brad Myers 32 View on Hand-Held: titles l See the list of titles

Brad Myers 33 View on handheld: Time l Multi-function timer l View large l And in corner l Multi-function timer l View large l And in corner

Brad Myers 34 Control Demonstrations on Palm l Control PC applications and external devices l Uses Shortcutter l Easy demo and resume show l Control PC applications and external devices l Uses Shortcutter l Easy demo and resume show

Brad Myers 35 Control Demonstrations on PocketPC l Task list of PC applications l Tap brings one to front l PowerPoint continues behind – easy to resume l Task list of PC applications l Tap brings one to front l PowerPoint continues behind – easy to resume

Brad Myers 36 Commercialized l SlideShow Commander commercialized by: l Available at Office Depot, CompUSA, etc. l Palm and PocketPC l SlideShow Commander commercialized by: l Available at Office Depot, CompUSA, etc. l Palm and PocketPC Synergy Solutions, Inc. Synergy Solutions, Inc.

Brad Myers 37 Studies of Presentations l Summer study of 2 HCI Master’s students l Contextual Inquiry of 9 presentations l Audience: 10 to hundreds l 7 used PowerPoint, 4 used NetMeeting l Found 220 “breakdowns” l Most were minor problems l Averaged 8.7 lost minutes per talk (14.5%) l Designed SlideShow Commander to eliminate some of these problems l Summer study of 2 HCI Master’s students l Contextual Inquiry of 9 presentations l Audience: 10 to hundreds l 7 used PowerPoint, 4 used NetMeeting l Found 220 “breakdowns” l Most were minor problems l Averaged 8.7 lost minutes per talk (14.5%) l Designed SlideShow Commander to eliminate some of these problems

Brad Myers 38 MultiCursor l For special applications that are aware of multiple inputs l E.g.: Shared Whiteboard l Single Display Groupware l Multiple people, one display l A number of interesting issues: l Palettes, widgets l Section handles l Undo l For special applications that are aware of multiple inputs l E.g.: Shared Whiteboard l Single Display Groupware l Multiple people, one display l A number of interesting issues: l Palettes, widgets l Section handles l Undo

Brad Myers 39 Chat l Communicate to another PDA user through the PC l PC serves as a conduit l For side notes and messages l For example, in negotiation meetings l Send to all or to a specific person l Communicate to another PDA user through the PC l PC serves as a conduit l For side notes and messages l For example, in negotiation meetings l Send to all or to a specific person

Brad Myers 40 Command Post of the Future l Large DARPA funded project l Make commanders more effective l Large DARPA funded project l Make commanders more effective

Brad Myers 41 Study of Laser Pointers l Studied properties of laser pointer interaction techniques l Hand-wiggle  8 pixels l Delay until moving average stable  1.5 sec l Delay until target acquired  1.5 sec l Different devices and poses do not help much l Studied properties of laser pointer interaction techniques l Hand-wiggle  8 pixels l Delay until moving average stable  1.5 sec l Delay until target acquired  1.5 sec l Different devices and poses do not help much

Brad Myers 42 Tap Speed Study l Tapping directly on SmartBoard is fastest l Semantic Snarfing next l But high error rate l Then regular mouse l Laser pointing slowest l Used separate physical button on handheld l Tapping directly on SmartBoard is fastest l Semantic Snarfing next l But high error rate l Then regular mouse l Laser pointing slowest l Used separate physical button on handheld

Brad Myers 43 “Semantic Snarfing” l Interacting at a distance l Grab contents to handheld l Picture l Menus l Text l Re-visualization l “Magic Lenses” l Interacting at a distance l Grab contents to handheld l Picture l Menus l Text l Re-visualization l “Magic Lenses”

Classroom Use l Using handheld computers in classrooms l For testing l Improve large lectures l Provide immediate feedback to instructor l Using handheld computers in classrooms l For testing l Improve large lectures l Provide immediate feedback to instructor

Brad Myers 45 Hardware l HP donated 110 Jornada 680 and computers l Windows CE l Lucent donated Wavelan wireless cards l HP donated 110 Jornada 680 and computers l Windows CE l Lucent donated Wavelan wireless cards

Brad Myers 46 Context l Collaborator: Prof. David Yaron of Chemistry l Chemistry : Modern Chemistry II l Spring, 2000 and Spring, 2001 l About 90 students l Mostly freshmen l Loaned one Jornada and Wavelan card to each student for the whole semester l So would get used to using it l Offset technical difficulties with benefits to student l Collaborator: Prof. David Yaron of Chemistry l Chemistry : Modern Chemistry II l Spring, 2000 and Spring, 2001 l About 90 students l Mostly freshmen l Loaned one Jornada and Wavelan card to each student for the whole semester l So would get used to using it l Offset technical difficulties with benefits to student

Brad Myers 47 Wireless Andrew l CMU has almost complete coverage of b support l Wavelan l Donations from Lucent, etc. l Funding from state of Pennsylvania, etc. l Can use computers wirelessly in all classrooms and offices l Most lawns, dorm rooms, etc. l Windows, Mac, Linux, Windows CE support l CMU has almost complete coverage of b support l Wavelan l Donations from Lucent, etc. l Funding from state of Pennsylvania, etc. l Can use computers wirelessly in all classrooms and offices l Most lawns, dorm rooms, etc. l Windows, Mac, Linux, Windows CE support

Brad Myers 48 Pebbles Tests l Create through html pages l Arbitrary html in questions and answers l Pictures, formatting, links l Use FrontPage, etc. to author l Embed tags to show question and answers l Create through html pages l Arbitrary html in questions and answers l Pictures, formatting, links l Use FrontPage, etc. to author l Embed tags to show question and answers

Brad Myers 49 Concept Test l Students allowed to answer multiple times l Server stores all answers l Prof. Yaron displayed questions and answers on the board l Evidence that significantly contributes to learning — Mazur, E. (1997). Peer Instruction: A User's Manual, Prentice Hall. l Students allowed to answer multiple times l Server stores all answers l Prof. Yaron displayed questions and answers on the board l Evidence that significantly contributes to learning — Mazur, E. (1997). Peer Instruction: A User's Manual, Prentice Hall.

Brad Myers 50 Instructor’s View l In class, instructor sees a chart of answers l Optionally refreshes every 3 seconds l May be projected for whole class to see l Top displays instructions for students l In class, instructor sees a chart of answers l Optionally refreshes every 3 seconds l May be projected for whole class to see l Top displays instructions for students

Brad Myers 51 Results l Unfortunately, no data (yet) about effects on learning l Students thought concept tests were a valuable part of the class l Students significantly preferred using handhelds for concept tests. l Unfortunately, no data (yet) about effects on learning l Students thought concept tests were a valuable part of the class l Students significantly preferred using handhelds for concept tests. Raise HandsHold up cardsHandhelds 25%4%63%

Home Use l Personal Universal Controller l Help for people with muscular disabilities l Personal Universal Controller l Help for people with muscular disabilities

Brad Myers 53 Personal Universal Controller l Two-way communication l Appliances describe their functions l Handheld PUC creates interface based on descriptions l Handheld PUC controls the appliance l Appliance sends back status as feedback l Two-way communication l Appliances describe their functions l Handheld PUC creates interface based on descriptions l Handheld PUC controls the appliance l Appliance sends back status as feedback Specifications Control Feedback of Status

Brad Myers 54 Initial Experiments l Use real PocketPC l Pretend that controls devices l Use real PocketPC l Pretend that controls devices

Brad Myers 55 PocketPC results l For both appliances, users of actual interfaces: l Took about twice as long l Made at least twice as many mistakes as users of the handheld interfaces l Needed external help five times more often l For both appliances, users of actual interfaces: l Took about twice as long l Made at least twice as many mistakes as users of the handheld interfaces l Needed external help five times more often

Brad Myers 56 Current Steps l XML specification language and protocol for describing appliances l Create panels automatically from the specification l Find real appliances which we can control l XML specification language and protocol for describing appliances l Create panels automatically from the specification l Find real appliances which we can control Specification Language <spec xmlns="puc.xsd" xmlns:xsi=" hema-instance" xmlns:xsd=" chema" name="Audiophase 5 CD Stereo"> Off On Stereo Power Power Powr Pwr 10

Brad Myers 57 Current Work: Handhelds for People with Muscular Disabilities l Using handhelds as interface to PCs l People with Muscular Dystrophy have fine-motor control but lose gross motor control l Difficulties with mouse and keyboard, but stylus OK l Handhelds as interface to other devices l Control room lights, telephone, wheelchair, etc. l Using handhelds as interface to PCs l People with Muscular Dystrophy have fine-motor control but lose gross motor control l Difficulties with mouse and keyboard, but stylus OK l Handhelds as interface to other devices l Control room lights, telephone, wheelchair, etc.

Brad Myers 58 Remote Commander Adaptations l Faster Acceleration l More flexible tapping l Turn off key repeat l Multiple Keyboards l Faster Acceleration l More flexible tapping l Turn off key repeat l Multiple Keyboards

Brad Myers 59 Any PC app. PebblesPC Brad’s Palm Pilot Rob’s Pocket PC Ben’s Windows CE RemoteCmd MultiCursor On the PC PebblesDraw Various PDA apps Slideshow Commander PowerPoint OLE Automation Various dlls General Architecture Windows event stream Direct connection or sockets Serial, IR or sockets

Brad Myers 60 Protocols l Can use Pebbles protocols to develop your own application l Libraries for Palm, Windows CE, PC l Independence from communication medium l Use by: l PalmAmp from IronCreek Software l Intel research l others… l Windows messages or sockets l Can use Pebbles protocols to develop your own application l Libraries for Palm, Windows CE, PC l Independence from communication medium l Use by: l PalmAmp from IronCreek Software l Intel research l others… l Windows messages or sockets

Brad Myers 61 Downloads l Most of this software is available for free downloading l First release in Feb, 1998 l Just released version 5 (Feb, 2002) l Downloaded over 30,000 times l About 200 times a week l Most of this software is available for free downloading l First release in Feb, 1998 l Just released version 5 (Feb, 2002) l Downloaded over 30,000 times l About 200 times a week

Brad Myers 62 Conclusions l Handhelds are becoming ubiquitous l Connecting technologies will improve l Important to study how can be used when connected to computers and each other l “Multi-Machine User Interfaces” will be increasingly important l People will want to use the most convenient device for their information and control l Handhelds are becoming ubiquitous l Connecting technologies will improve l Important to study how can be used when connected to computers and each other l “Multi-Machine User Interfaces” will be increasingly important l People will want to use the most convenient device for their information and control

Brad Myers 63 Thanks to Our Sponsors! Supported by grants from: DARPA Microsoft Pittsburgh Digital Greenhouse NSF And equipment grants from: Hewlett Packard Lucent Technologies Palm Computing Symbol Technologies Supported by grants from: DARPA Microsoft Pittsburgh Digital Greenhouse NSF And equipment grants from: Hewlett Packard Lucent Technologies Palm Computing Symbol Technologies IBM SMART Technologies, Inc. Synergy Solutions, Inc. Handango

Using Hand-Held Computers and PCs Together: The Pebbles Project Human Computer Interaction Institute School of Computer Science Carnegie Mellon University Brad A. Myers Brad A. Myers