Jun Rekimoto Sony Computer Science Laboratory Inc Oct.15.1997 Pick-and-Drop: A Direct Manipulation Technique for Multiple Computer Environments Jun Rekimoto Sony Computer Science Laboratory Inc Oct.15.1997

From Single Device UI to multi-Device UI Current PC/GUIs are designed for One-user -- One-computer environment Computer Wall Future Computing Environment must be designed for Multi-user -- Multi-computer environments PDA PDA PDA In old days, a computer is shared by many users. Now, we use a personal computer. So there is a one-to-one mapping between the user and the computer. However, in the near future, we will use a number of computers in a very dynamic way. For example, one may use two or more computers (such as Desk-top PCs and note PCs) simultaneously. Others may use a wall-sized computer as well as a PDA for each participant. So I think that future user interface techniques must be designed for these multi-computer -- multi-user environments. PDA Kiosk PC PC PDA UIST’97

Limitations of Single-Device GUIs One-to-One dogma Mouse and Keyboard are connected to one computer cf. Paper and Pencil No Direct-Manipulation technique that can interrelate two or more computers e.g., Drag-and-drop between two computers However, most of today’s user interface techniques implicitly assume that one computer is used by one user at a same time. For example, mouse and keyboard are connected to the corresponding one computer. It would be quite cumbersome if you have to choose a specific pencil for each piece of paper, but today’s computers have the similar problem. The other problem is the limitation of graphical user interface design. For example, you can easily copy data on a same computer by using cut-and-paste or drag-and-drop, but you have to change the style of interaction to transfer information across the computers. UIST’97

Mouse Jungle This is a photograph of my desk. I am using 3 computers (Unix, Mac, and PC) and share one display by introducing display switcher. However, I regularly misunderstand which mouse is the correct one. UIST’97

Data Copying through Human Body Note: These computers are connected to the Network The next example is a data transfer problem of multi-computer environment. Data copy through human body? What does it mean? I am not talking about BodyNet. Please take a look at this user. What he is doing is, looking at the left display, and read an URL on the display, then type it into the right computer. So, data is transferred through human. But it is of course a miserable solution because [*] these computers are connected to the network. I think that this situation occurs due to the lack of user-interface which is designed for multi-computer environment. UIST’97

How people transfer information between nearby computers? % % % % A survey conducted on 40 engineers in Sony’s Software Labs. I wanted to know the other people are using more clever way to perform information transfer. This is a result of quick survey conducted on 40 software engineers in Sony’s Laboratory. As shown in this graph, more than 60% people are doing data copy by hand. Other methods are also very irregular. Apparently, people have to change the interaction style when dealing with multiple computers. % % UIST’97

Multi-Computer (Device) UIs No cognitive overload as the number of devices increases Virtual / Physical Characteristics Number, size, orientation, and location of devices Effective combination of a variety of devices Boundaries between computers/devices “Seamful but beautifully designed seams” [M.Weiser, Invited Talk at UIST94] An Example: Pick-and-Drop So, When we are designing user interfaces, we have to consider multiple-computer or multiple-device environment. We call this Multi-Device User interface. To deal with multi-device environment, we have to aware of several physical characteristics such as the number, size, orientation, and the location of the devices. We have to carefully aware of several boundaries, or seams between devices. Perhaps our goal is not to varnish the seams of multi-computer environment. Distinction of multiple devices is often very important. Our goal is to design beautiful seams. In this talk, I introduce a new direct manipulation technique called the “Pick-and-Drop” UIST’97

Inter Computer Pick-and-Drop Pick-and-Drop Drag&Drop UIST’97 Pick-and-Drop is a extended concept of drag-and-drop for pen computers. From the view point of multi-device User interface, pen is advantageous because this pen can manipulate both of the computers. So mouse-jungle problem is automatically avoided. With drag-and-drop [*], the user can move data from here to there. Drag-and-drop is quite suitable for mouse, but when applying to pen interface, some users dislike the feeling of scratching the display surface with the pen tip. So I introduced a metaphor of picking up information from the display surface. The user first taps on the display to pick up information, then taps again to drop it. In a same way [*], the user can also transfer information between two computers. Please note that the user can use exactly the same way to transfer information within and across the computer boundaries. UIST’97

Information transition between virtual and physical spaces Pen virtually Carries the data Display Surface Physical Space Pick Drop Computer Space 1 Computer Space 2 It can be said that this technique is adding a physical artifact to virtual computer object. The user feels that he is picking up computer data from here, as if it were a real/physical object, then drop it on another display surface. Just like picking up a French fry from a lunch box, the user can manipulate computer information in a real world. UIST’97

Data Shadow Visual feedback for pen carrying data Display Surface To emphasize that pen virtually holds the data, I introduce a technique called a Data Shadow. When data is picked and the user lifts the pen, an icon with shadow appears on a display. It indicated that pen is carrying the data. When a user releases the pen, this shadow and icon vanishes. UIST’97

Applications of Pick-and-Drop Picking up information from Kiosk (push media) terminal Information exchange between PDAs Since pick-and-drop is an extended concept of drag-and-drop for multi-device environment, there are many applications. You can easily exchange information from your friend, or you can also picking up information from the kiosk terminal (push media terminal) in a public space or office environment. UIST’97

Wall-sized Display + Palmtop Computer Using the combination of multiple-computers can be applied in a different way. This is a whiteboard-sized computer and the user is drawing a picture on it. Putting menu-bar or tool-bar on a wall-sized display introduces several problems. The user might not able to reach the menu-bar at the top of the display. Using multiple-device approach, the user can have a palmtop sized computer, like a palette of oil-painting, to select brush type and color, select a clip-art, or create a text. Like a traditional oil painter, the user can tap a color icon on a palette with the pen to change the pen colors. UIST’97

Video Now, let me show a video UIST’97

System Configuration Network Pen Manager Wall Size Display (or Physical Wall) PDA (Nomadic PCs) Desktop Display Network Pen ID = 1 Internally, Pick-and-Drop is implemented as shown in this picture. The point is that pen does not actually contain the data. Instead, each pen has a unique ID that can be recognized from the computers. When the user picks up a data with the pen, the combination of data name and pen ID is registered to the pen manager on the network. When the user drops the data, actual data transfer occurs through the network. But from the user’s point of view, Pick-and-drop provides an illusion that pen carries the data. Pen ID = 2 ID Data Pen Manager 1 xx.gif UIST’97

Pick-and-Drop between Paper and Computer Object ID Page ID Pick Position Camera Page Code The last example was implemented as shown in this picture. The page ID is recognized by a camera mounted above the desk, and pen position is recognized by the tablet. Printed Objects Tablet UIST’97

Discussions Traditional Data Transfer Pick-and-Drop Symbolic Copy data from C:/ My Document of Machine A to /user/local …. Pick-and-Drop Copy data from here to there Spatial Now, let me discuss the difference between this Pick-and-Drop and traditional data transfer technique such as remote copy command. I think the major difference is in a mental model of the user. With traditional copy technique, the user is more aware of symbolic concept such as a machine’s name or file path. So, it is violating the principle of Graphical User interfaces like “See & Point”. On the other hand, during a Pick-and-Drop operation, the user is more focusing on spatial concept, such as “picking up information from the left computer to the right computer”. UIST’97

Summary Multi-Computer (Multi-Device) UI Pick-and-Drop: as a new paradigm of direct-manipulation techniques Pick-and-Drop: information transfer technique for multi-device environment In conclusion, I proposed a multi-computer UI concept, which will be important for future computing environment. And we also reported an information transfer in a multi-device environment and proposed the Pick-and-Drop as an example of multi-device UI. UIST’97

Future Work Generalized Pick-and-Drop Pick-and-Drop of active objects information exchange between electronic appliance (e.g., VCR), ubiquitous computers, and physical objects (paper, wall, etc.) Pick-and-Drop of active objects e.g., JavaBeans Application to distributed environments Pick-and-drop over video conferencing Currently, Pick-and-Drop mainly deals with information transfer between computers, but we can extend this concept into more general environment. For example, I will be able to pick-up TV program information from Web page with some pointing device, then drop it to the VCT to make reservation. Another direction is to use Pick-and-Drop for object migration. For example, you pick-and-drop a graph-object from my PDA to the whiteboard, then pick-and-drop numerical data on the attached graph to visualize it. The other possible direction is the application to the CSCW domain. For example we will be able to exchange information using Pick-and-Drop, over the shared TV window such as a ClearBoard. UIST’97

Pick-and-Drop over ClearBoard ENVISIONMENT UIST’97

UIST’97

State Transition Diagram PICK DROP DRAG MOVE MOVE DRAG PEN-DOWN PEN-UP DOWN Time Out Time Out MOVE PEN-UP DRAG PEN-UP Time Out UIST’97

Pick-Move, Pick-Copy Move Copy UIST’97

Related Work (1/2) DigitalDesk [Wellner 93] Data transfer from paper to the desk LiveBoard + PARC TAB [Want et.al 95] PARC Tab as a remote pointer for the LiveBoard Graspable UIs [Fitzmaurice et.al 95] Using physical handles (bricks) to manipulate computer objects UIST’97

Related Work (2/2) PDA-ITV [Robertson et.al 96] PDA as a remote controller for the interactive TV PaperLink [Arai et.al 97] Picking up information from paper Ultra Magic Key (UMK) [Usuda et.al 97] TV manipulation using a paper booklet Tangible Bits [Ishii et.al 97] Physical artifact for “Bits” (phicons) UIST’97