Using Hand-Held Computers and PCs Together The Pebbles Project Brad A. Myers Human-Computer Interaction Institute School of Computer Science Carnegie Mellon University Brad A. Myers Human-Computer Interaction Institute School of Computer Science Carnegie Mellon University

Brad Myers2 Background l Many people have PDAs l Windows CE l 3Com PalmPilots / IBM Workpads l Predictions are for increased ubiquity l Price and distribution may equal GameBoys

Brad Myers3 Premises of our Research l Often multiple devices will be available: l PDA and a PC l Multiple people’s PDAs “With the coming wireless technologies, connecting the PCs and PDAs together will no longer be an occasional event for synchronization. Instead, the devices will frequently be in close, interactive communication.”

Brad Myers4 Research Agenda How can multiple devices be used effectively together, at the same time?

Brad Myers5 Example: Power Point Control l Use PC to give the presentation l Use hand-held to control the PC l Two-way communication l Hand-held shows picture of slide, notes, list of titles, etc.

Brad Myers6 Pebbles is: PEBBLES DAs for ntry of oth ytes and ocations from xternalources.

Brad Myers7 Meetings and Presentations l Computers rarely used by attendees l Previous Computer-Supported Cooperative Work (CSCW) approaches used significant hardware l Special rooms of installed computers l Instead, we will get leverage from people already having PDAs

Brad Myers8 Local Meetings l Targeting local meetings l Everyone is in same room l Often have a PC projected onto a wall l Examples: presentations, design reviews, brainstorming, briefings l Remote participants would use other CSCW software l Also, informal meetings l E.g. Two people in an office sharing a PC

Brad Myers9 At the Desktop l Newer PDAs are rechargeable l Should be kept in cradles while at the desktop l Investigate uses as an extra customizable input-output device l Extend desktop applications

Brad Myers10 Considerations l Use PDA in an appropriate way l Not as a replacement l Exploit key advantages of each technology l Must be natural, non-intrusive l PDA’s limitations l Small screen, (relatively) slow processor l Focus is usually on the PC’s screen l PDA’s screen is often secondary l Many PDAs at the same time

Brad Myers11 Connecting PDAs to the PC l Connect using serial cables l Supplied cradle or optional cable l Stays attached nicely l Connects to PC’s serial port l Working on wireless l Interference using InfraRed (IR) l Radio l Wireless LAN: WaveLan, Proxim l Wireless PAN: BlueTooth, HomeRF l Demonstrated for HandSpring Visor l Note: Not cell-phone WAN (Palm VII)

Brad Myers12 Connecting More than One l Easy with Wireless l Laptops: l Quatech QSP-100 PCMCIA l Can connect up to 8 PDAs to a laptop using both slots l About $500 each (4 ports) l Wires: Desktop PCs: l Can use various serial cards l About $400 for 8 ports

Brad Myers13 Original Application: Remote Commander l Allow PDA to control a PC l Can be used with any existing application l Uses the standard (single) cursor l Any application l Don’t have to jump up and grab mouse l No “floor control” l Evidence that floor control mechanisms interfere with smooth functioning of meeting

Brad Myers14 Current Study: Floor Control l Test multiple user floor control using a shared puzzle program l Various floor control options: l Each user has own cursor l Shared cursor l Free-for-all l Explicit grab l Explicit release l Moderator decides l Implicit grab when available

Brad Myers15 l Move stylus to move mouse l Works like a touchpad l Buttons using: l Tap to click l Drag button l Top button for left mouse button l Lower button for right l On palm-size Windows CE, use “Action” button PDA Side

Brad Myers16 Keyboard l Graffiti or Jot strokes sent as characters l Our keyboard includes special keys l Tap in “abc” corner on PalmPilot l On palm CE, add 3 rows above jot or keyboard area l On CE CLIO, can handwrite

Brad Myers17 Slide Show Commander l For PowerPoint’97 or 2000 l PalmPilot and Windows CE l Space = forward, backspace = back l Or hardware buttons

Brad Myers18 Slide Show on Windows CE l See snapshot and the notes together l Color or “black-and-white” l Scroll wheel for forward, backward

Brad Myers19 Studies of Presentations l Summer study of 2 HCI Master’s students l Contextual Inquiry of 9 presentations l Audience: 10 to hundreds l 7 used PowerPoint, 4 used NetMeeting l Found 220 “breakdowns” l Most were minor problems l 34 kinds l Averaged 8.7 lost minutes per talk (14.5%) l Examples: awkward to change slide, lost track of time, incomplete URL given

Brad Myers20 Results of Studies l SlideShow Commander fixed to eliminate some of the problems l Added hardware buttons to change slide l Added Timer pane l Test showed that these breakdowns fixed l Future directions: l Control all devices in the room (“univ. remote”) l Integrate task switcher for demos l Audience mode

Brad Myers21 Other Current Work on SlideShow l Better support for on-the-fly presentations l Easier drawing l Saving of scribbles for audience

Brad Myers22 Problems with PowerPoint l Some issues and problems with PowerPoint for our application: l Want to add OLE (ActiveX) objects in slide l Like Word or Excel objects l For dynamic editing at run-time l Zoom property of SlideShowWindow is integer but insufficient resolution--needs to be a float l SlideShowWindow should have properties for actual width and height (issue: black border). l Want to find out where we are in the animation sequence (e.g. an event for next animation, and a query for current-animation) l Want to get a metafile for a slide directly through the OLE interface. Currently, we put it into the clipboard first l Plug-ins don’t get notified when PowerPoint exits.

Brad Myers23 Scribble l Multiple people draw on top of whatever on PC screen, not just PowerPoint l Each user has own cursor and color l Save by PrintScreen l Erase by refresh

Brad Myers24 MultiCursor l For special applications that are aware of multiple inputs l First one: Shared Drawing Tool l “PebblesDraw” l Shared whiteboard application l Single Display Groupware l Multiple people, one display l A number of interesting issues

Brad Myers25 PebblesDraw l Multiple people edit at same time l Independent cursors and selections l Widgets such as button panels used by one person at a time l Undo by user or global undo

Brad Myers26 Chat l Communicate to another PDA user through the PC l PC serves as a conduit l For side notes and messages l For example, in negotiation meetings l Send to all or to a specific person

Brad Myers27 Applications for Individuals l Extra input and output devices have been shown to be useful l But can be expensive and hard to configure l People have PDAs and are attached to PC l For example, cradles for recharging l Customizable, extensible l Extend desktop applications

Brad Myers28 Scrolling with the PDA l For scrolling using the non-dominant hand l Studies showed parallel and efficient uses of both hands together l Generates Windows scrolling events

Brad Myers29 Task and Window Switcher l List on the PDA the tasks on the PC l List each Window for each Task l Click to switch l Unifies the significantly different PC mechanisms

Brad Myers30 Remote Clipboard l Transfer information between PDA and PC l Connects their clipboards together l Invoke file or URL from PDA l Transfer content or reference

Brad Myers31 Web Assistant l Supports “Hub-and-spoke style of browsing l For search results, tables of contents, indices. l Download links from selected part of Web page to PDA l Only reloaded on demand l Pages often have many unuseful links l Click on links to change page on PC l Can save links

Brad Myers32 Shortcutter l User-created panels of controls l Buttons, knobs, sliders, etc. l Create custom interfaces and extensions to PC applications l And then take them with you l Direct manipulation for edit, then set properties

Brad Myers33 Shortcutter Actions l Send any keyboard key, mouse button, scrolling action or string to PC l Open a file or URL l Run an application l Invoke any PC menu or button l Windows message l Switch to a different Shortcutter panel l Control the Mouse

Brad Myers34 More Shortcutter Actions l Macro l Can be multi-application l Application-specific l Same button, different messages l Useful for application sets: browsers, compilers

Brad Myers35 Studies of Two-Handed Use l Summer study of 2 HCI Master’s students l Using PDA in left hand with mouse in right hand l Set up:

Brad Myers36 Study 1 l Speed and error rates with different size buttons Stimulus On PC: 

Brad Myers37 Study 1 Results l Time to tap on button depended on size l Few errors l People often didn’t look at PDA

Brad Myers38 Study 2 l Speed to move hands from keyboard to PDA and mouse l Select cell, then type l Words chosen to alternate starting and ending letters

Brad Myers39 Study 2 Results l Moving to both PDA and mouse only about 15% slower than just moving to the mouse l People are moving both hands at the same time 1H Keyboard->Mouse728 1H Keyboard->PDA744 1H Mouse->Keyboard701 1H PDA->Keyboard639 Keyboard -> Mouse&PDA % Mouse&PDA -> Keyboard % % slower msec

Brad Myers40 Study 3 l Test scrolling with the left hand l Various scrolling mechanisms on PDA l Reproduce study from IBM [Zhai, 97] l Scroll to find link in long Web page and click on it

Brad Myers41 Study 3 Results l Scrolling with buttons on PDA was fastest l PDA scrollers similar to mouse speed l Using 2 hands is effective! Winner!

Brad Myers42 Any PC app. PebblesPC Brad’s PalmPilot Rob’s IBM Workpad Ben’s Windows CE RemoteCmd MultiCursor On the PC PebblesDraw Various PDA apps Slideshow Commander PowerPoint OLE Automation Various dlls General Architecture Windows event stream Direct connection or sockets Serial, IR or sockets

Brad Myers43 Protocols l Can use Pebbles protocols to develop your own application l PalmAmp from IronCreek Software l Ben Bederson’s group at Univ. Maryland l Windows messages or sockets

Brad Myers44 Future: PDAs in a Classroom l Each student has a hand-held l Wireless connection to teacher’s computer l Applications: l Notetaking with reference to teacher’s material l Current slide, web page, any screen l Tests, pop quizzes, surveys, homeworks l Instant feedback and grading l Can be used to influence this lecture l “Concept Tests” l Enforces full class participation

Brad Myers45 Future: Explore Public / Private Information l Private “drill down” to get details of specific items on a public display l Issues: l How specify item? l Reformatting information for small display? l For Command Post and classroom l Privately composing information and then displaying it to all

Brad Myers46 Future: Universal Personal Controller l 2-way communication with smarter devices l Physical devices: copiers, light switch, fridge, furnace, phone, clocks, factory machines, etc. l No display on the device l Devices could up-load specifications of I/O l “Model-based” design of user interface l Customized to user l Based on characteristics of the device l “Universal Plug-and-Play,” Jini, Salutation

Brad Myers47 Possible Future Projects * Most of these useful for classes and business meetings l Taking attendance l Voting (Formal or informal) l Coordinating Schedules and To-Do lists l Automatically distribute correct information l “Structured Idea Generation Process” from Univ. of Arizona l Brainstorming applications

Brad Myers48 Downloads l Most of this software is available for free downloading l First release of RemoteCmd in Feb, 1998 l Release of others in Dec., 1998 l Downloaded over 15,000 times l Possible commercialization of Slide Show, etc.

Brad Myers49 Conclusions l Personal Digital Assistants and similar technologies are becoming ubiquitous. l Connecting techniques will improve. l Important to study how can be used when connected to computers and each other. l The Pebbles research project is off to a good start. l Exciting potential.

Brad Myers50 Funding Supported by grants from: DARPAMicrosoft and equipment grants from: Palm Computing IBM

Using Hand-Held Computers and PCs Together The Pebbles Project Brad A. Myers Human-Computer Interaction Institute School of Computer Science Carnegie Mellon University Brad A. Myers Human-Computer Interaction Institute School of Computer Science Carnegie Mellon University