1. Jun Rekimoto Sony Computer Science Laboratory Inc Oct.15.1997
Pick-and-Drop: A Direct Manipulation Technique for Multiple Computer Environments
Jun Rekimoto
Sony Computer Science Laboratory Inc
Oct

2. From Single Device UI to multi-Device UI
Current PC/GUIs are designed
for One-user -- One-computer
environment
Computer
Wall
Future Computing Environment
must be designed for
Multi-user -- Multi-computer
environments
PDA
PDA
PDA
In old days, a computer is shared by many users.
Now, we use a personal computer. So there is a one-to-one mapping between the user and the computer.
However, in the near future, we will use a number of computers in a very dynamic way.
For example, one may use two or more computers (such as Desk-top PCs and note PCs) simultaneously. Others may use a wall-sized computer as well as a PDA for each participant.
So I think that future user interface techniques must be designed for these multi-computer -- multi-user environments.
PDA
Kiosk PC PC
PDA
UIST’97

3. Limitations of Single-Device GUIs
One-to-One dogma
Mouse and Keyboard are connected to one computer
cf. Paper and Pencil
No Direct-Manipulation technique that can interrelate two or more computers
e.g., Drag-and-drop between two computers
However, most of today’s user interface techniques implicitly assume that one computer is used by one user at a same time. For example, mouse and keyboard are connected to the corresponding one computer. It would be quite cumbersome if you have to choose a specific pencil for each piece of paper, but today’s computers have the similar problem.
The other problem is the limitation of graphical user interface design. For example, you can easily copy data on a same computer by using cut-and-paste or drag-and-drop, but you have to change the style of interaction to transfer information across the computers.
UIST’97

4. Mouse Jungle
This is a photograph of my desk. I am using 3 computers (Unix, Mac, and PC) and share one display by introducing display switcher. However, I regularly misunderstand which mouse is the correct one.
UIST’97

5. Data Copying through Human Body
Note:
These computers are connected to the Network
The next example is a data transfer problem of multi-computer environment. Data copy through human body? What does it mean? I am not talking about BodyNet. Please take a look at this user. What he is doing is, looking at the left display, and read an URL on the display, then type it into the right computer. So, data is transferred through human. But it is of course a miserable solution because [*] these computers are connected to the network. I think that this situation occurs due to the lack of user-interface which is designed for multi-computer environment.
UIST’97

6. How people transfer information between nearby computers?% % % %
A survey conducted on
40 engineers in
Sony’s Software Labs.
I wanted to know the other people are using more clever way to perform information transfer. This is a result of quick survey conducted on 40 software engineers in Sony’s Laboratory. As shown in this graph, more than 60% people are doing data copy by hand. Other methods are also very irregular. Apparently, people have to change the interaction style when dealing with multiple computers. % %
UIST’97

7. Multi-Computer (Device) UIs
No cognitive overload as the number of devices increases
Virtual / Physical Characteristics
Number, size, orientation, and location of devices
Effective combination of a variety of devices
Boundaries between computers/devices
“Seamful but beautifully designed seams” [M.Weiser, Invited Talk at UIST94]
An Example: Pick-and-Drop
So, When we are designing user interfaces, we have to consider multiple-computer or multiple-device environment. We call this Multi-Device User interface.
To deal with multi-device environment, we have to aware of several physical characteristics such as the number, size, orientation, and the location of the devices.
We have to carefully aware of several boundaries, or seams between devices. Perhaps our goal is not to varnish the seams of multi-computer environment. Distinction of multiple devices is often very important. Our goal is to design beautiful seams.
In this talk, I introduce a new direct manipulation technique called the “Pick-and-Drop”
UIST’97

8. Inter Computer Pick-and-Drop Pick-and-Drop Drag&Drop UIST’97
Pick-and-Drop is a extended concept of drag-and-drop for pen computers.
From the view point of multi-device User interface, pen is advantageous because this pen can manipulate both of the computers. So mouse-jungle problem is automatically avoided.
With drag-and-drop [*], the user can move data from here to there. Drag-and-drop is quite suitable for mouse, but when applying to pen interface, some users dislike the feeling of scratching the display surface with the pen tip.
So I introduced a metaphor of picking up information from the display surface. The user first taps on the display to pick up information, then taps again to drop it.
In a same way [*], the user can also transfer information between two computers. Please note that the user can use exactly the same way to transfer information within and across the computer boundaries.
UIST’97

9. Information transition between virtual and physical spaces
Pen virtually
Carries the data
Display
Surface
Physical Space
Pick
Drop
Computer Space 1
Computer Space 2
It can be said that this technique is adding a physical artifact to virtual computer object. The user feels that he is picking up computer data from here, as if it were a real/physical object, then drop it on another display surface.
Just like picking up a French fry from a lunch box, the user can manipulate computer information in a real world.
UIST’97

10. Data Shadow Visual feedback for pen carrying data Display Surface
To emphasize that pen virtually holds the data, I introduce a technique called a Data Shadow. When data is picked and the user lifts the pen, an icon with shadow appears on a display. It indicated that pen is carrying the data. When a user releases the pen, this shadow and icon vanishes.
UIST’97

11. Applications of Pick-and-Drop
Picking up information from Kiosk (push media) terminal
Information exchange
between PDAs
Since pick-and-drop is an extended concept of drag-and-drop for multi-device environment, there are many applications.
You can easily exchange information from your friend, or you can also picking up information from the kiosk terminal
(push media terminal) in a public space or office environment.
UIST’97

12. Wall-sized Display + Palmtop Computer
Using the combination of multiple-computers can be applied in a different way. This is a whiteboard-sized computer and the user is drawing a picture on it. Putting menu-bar or tool-bar on a wall-sized display introduces several problems. The user might not able to reach the menu-bar at the top of the display. Using multiple-device approach, the user can have a palmtop sized computer, like a palette of oil-painting, to select brush type and color, select a clip-art, or create a text. Like a traditional oil painter, the user can tap a color icon on a palette with the pen to change the pen colors.
UIST’97

13. Video
Now, let me show a video
UIST’97

14. System Configuration Network Pen Manager Wall Size Display
(or Physical Wall)
PDA
(Nomadic PCs)
Desktop
Display
Network
Pen ID = 1
Internally, Pick-and-Drop is implemented as shown in this picture. The point is that pen does not actually contain the data. Instead, each pen has a unique ID that can be recognized from the computers. When the user picks up a data with the pen, the combination of data name and pen ID is registered to the pen manager on the network. When the user drops the data, actual data transfer occurs through the network. But from the user’s point of view, Pick-and-drop provides an illusion that pen carries the data.
Pen ID = 2 ID
Data
Pen Manager 1
xx.gif
UIST’97

15. Pick-and-Drop between Paper and Computer
Object ID
Page ID
Pick Position
Camera
Page Code
The last example was implemented as shown in this picture. The page ID is recognized by a camera mounted above the desk, and pen position is recognized by the tablet.
Printed
Objects
Tablet
UIST’97

16. Discussions Traditional Data Transfer Pick-and-Drop Symbolic
Copy data from C:/ My Document of Machine A to /user/local ….
Pick-and-Drop
Copy data from here to there
Spatial
Now, let me discuss the difference between this Pick-and-Drop and traditional data transfer technique such as remote copy command. I think the major difference is in a mental model of the user. With traditional copy technique, the user is more aware of symbolic concept such as a machine’s name or file path. So, it is violating the principle of Graphical User interfaces like “See & Point”.
On the other hand, during a Pick-and-Drop operation, the user is more focusing on spatial concept, such as “picking up information from the left computer to the right computer”.
UIST’97

17. Summary Multi-Computer (Multi-Device) UI Pick-and-Drop:
as a new paradigm of direct-manipulation techniques
Pick-and-Drop:
information transfer technique for multi-device environment
In conclusion, I proposed a multi-computer UI concept, which will be important for future computing environment.
And we also reported an information transfer in a multi-device environment and proposed the Pick-and-Drop as an example of multi-device UI.
UIST’97

18. Future Work Generalized Pick-and-Drop Pick-and-Drop of active objects
information exchange between electronic appliance (e.g., VCR), ubiquitous computers, and physical objects (paper, wall, etc.)
Pick-and-Drop of active objects
e.g., JavaBeans
Application to distributed environments
Pick-and-drop over video conferencing
Currently, Pick-and-Drop mainly deals with information transfer between computers, but we can extend this concept into more general environment. For example, I will be able to pick-up TV program information from Web page with some pointing device, then drop it to the VCT to make reservation.
Another direction is to use Pick-and-Drop for object migration. For example, you pick-and-drop a graph-object from my PDA to the whiteboard, then pick-and-drop numerical data on the attached graph to visualize it.
The other possible direction is the application to the CSCW domain. For example we will be able to exchange information using Pick-and-Drop, over the shared TV window such as a ClearBoard.
UIST’97

19. Pick-and-Drop over ClearBoard
ENVISIONMENT
UIST’97

20. UIST’97

21. State Transition Diagram
PICK
DROP
DRAG
MOVE
MOVE
DRAG
PEN-DOWN
PEN-UP
DOWN
Time Out
Time Out
MOVE
PEN-UP
DRAG
PEN-UP
Time Out
UIST’97

22. Pick-Move, Pick-Copy
Move
Copy
UIST’97

23. Related Work (1/2) DigitalDesk [Wellner 93]
Data transfer from paper to the desk
LiveBoard + PARC TAB [Want et.al 95]
PARC Tab as a remote pointer for the LiveBoard
Graspable UIs [Fitzmaurice et.al 95]
Using physical handles (bricks) to manipulate computer objects
UIST’97

24. Related Work (2/2) PDA-ITV [Robertson et.al 96]
PDA as a remote controller for the interactive TV
PaperLink [Arai et.al 97]
Picking up information from paper
Ultra Magic Key (UMK) [Usuda et.al 97]
TV manipulation using a paper booklet
Tangible Bits [Ishii et.al 97]
Physical artifact for “Bits” (phicons)
UIST’97