1. ISTD 2003 Interactive Systems Technical Design Lecture #3 Hybrid computing environments

2. ISTD 2003 Hybrid Computing Environments We live in the fusion of real (physical) and virtual (computer) worlds Some systems add virtual properties to physical world Some systems add physical dimension to virtual worlds (e.g. Pick-and-Drop studied here) These two approaches are not contradicting nor mutually exclusive It is important to find the right balance between virtual and physical dimensions

3. ISTD 2003 Case Studies Pick-and-Drop: A Direct Manipulation Technique for Multiple Computer Environments Sony Computer Science Laboratories, Japan Augmented Surfaces: A Spatially Continuous Work Space for Hybrid Computing Environments Sony Computer Science Laboratories, Japan

4. ISTD 2003 Pick-and-Drop: A Direct Manipulation Technique for Multiple Computer Environments Jun Rekimoto, Sony Computer Science Laboratories, Japan Proc. 10th Annual Symposium on User Interface Software and Technology, Banff, Canada

5. ISTD 2003 Pick-and-Drop: A Direct Manipulation Technique for Multiple Computer Environments Jun Rekimoto, Sony Computer Science Laboratories, Japan Proc. UIST’97 Video

6. ISTD 2003 Motivation : Multiple Computer Environments A number of (dedicated) computers within close proximity Desktop PC’s for SW development and office tasks Wall-size computers (displays) for collaborative work Personal PDA’s for mobile use Problems User interface: each computer has dedicated tethered keyboards and pointing devices, ”mouse jungle” Easy natural data transfer between computers

7. ISTD 2003 Survey of Multi-Computer Usage Data: 39 employees at Sony’s SW laboratories Q1. How many computers do you have on your desktop? 3+ 54% 2 39% 1 8% 0 0% Q2. How often do you need to transfer data between computers on the same desktop? Very often 69% Often 25% Sometimes 3% Occasionally 0% Never 3% Q3. (Under situation Q2) How do you transfer data? By hand 63% Shared files 63% FTP 57% e-mail 34% Floppies 20% Other 23%

8. ISTD 2003 Survey of Multi-Computer Usage (cont.) Q4. How often do you need to transfer data from your computer to another’s computer within a short distance? Very often 28% Often 23% Sometimes 36% Occasionally 5% Never 5% Q5. (Under situation Q4) How do you transfer data? By hand 54% Shared files 57% FTP 38% e-mail 73% Floppies 11% Other 19% Lack of easy direct data transfer user interfaces such as copy-and-paste or drag-and-drop between computers

9. ISTD 2003 Tentative Solutions to Problems User interface: each computer has dedicated keyboards and pointing devices, ”mouse jungle”  More sophisticated tetherless input device such as a stylus Can be shared between many pen sensitive displays Data sharing between computers  ”Pick-and-Drop” direct manipulation technique

10. ISTD 2003 Pick-and-Drop Extrapolation of Drag-and-Drop technique Drag-and-Drop vs Pick-and-Drop grabpick up drag hold and move (virtually) drop  No need to drag (slide) the object on the display surface  Virtual hold and move allows extended functionality, e.g. intra-computer operations

11. ISTD 2003 Remote Copy vs Pick-and-Drop Conceptual difference between remote copy and Pick-and-Drop

12. ISTD 2003 State Transition Diagrams Pick-and-Drop can coexist with Drag-and-Drop Time-out between pen-down and pen-up Proximity to screen is detected by combining motion events and a time-out

13. ISTD 2003 System Architecture

14. ISTD 2003 Implementation Details Computers and displays PDA’s: Mitsubishi Amity Desktop screens: Wacom PL300 Wall-sized display: Wacom MeetingStaff + projector Input device: Wacom stylus Can distinguish at most 3 different pens simultaneously Probably not sufficient for practical applications Additional pens via RF tags or IR beacons

15. ISTD 2003 Implementation Details (cont.) Computer network Ethernet for desktops and wall-sized display WLAN for PDA’s Application development in Java Object transfers via Java serialization

16. ISTD 2003 Pen-ID’s Each pen is assigned a unique ID ID is readable when pen is close to display ID represented by modifier buttons in the pens PenManager Binds object ID’s to pen ID’s (~ pen virtually holds the object) Manages object shadows Controls data transfer

17. ISTD 2003 Object Shadows (a)Pen contacts the display (b)Pen is lifted up, but is close to the display -> if pen has data, object shadow appears to indicate this (c) Pen is away from the screen -> no object shadow

18. ISTD 2003 Example Applications Information exchange between PDA’s and kiosks Pick-and-Drop allows easy natural data transfer from one computer (display) to another Transfer of data from a ”push media” information kiosk to PDA Transfer of data between PDA’s

19. ISTD 2003 Example Applications (cont.) Drawing on a wall display with a tablet Simple paint editor on a tablet (color, brush style) No need for a dedicated tool-palette

20. ISTD 2003 Example Applications (cont.) PaperIcons Information exchange between a computer and a book

21. ISTD 2003 Example Applications (cont.) Anonymous Displays Several small tablets are used as ”temporal work buffers” Pick-and-Drop supports intuitive data transfer without bothering with each computer’s symbolic name In comparison to virtual buffers of GUI’s, physical tablets can be arranged freely on physical desktop provide more natural spatial interface allow handling more than two buffers simultaneously are not resricted by the limited and fixed size of main desktop

22. ISTD 2003 Discussion Physical vs Symbolic Functionally, Pick-and-Drop is no more than remote copy However, in terms of UI, Pick-and-Drop is both physical and visible as opposed to symbolic Supports direct manipulation of data instead of needless exchange of symbolic concepts (e.g. drive and file names) Supports collaborative work

23. ISTD 2003 Discussion (cont.) Shared Files vs Pick-and-Drop Shared files force the user to deal with symbolic concepts such as machine and file names Difficult to keep track of multiple computers with shared files The unit of data transfer is not necessarily a file, e.g. text string such as URL Shared files a good solution for geographically separated computers, but not so intuitive between computers within close proximity

24. ISTD 2003 Candidate Improvements Support for a larger number of identifiable pens with RF tags Support for other file transfer protocols Enhanced pen operations such as grouping of objects Integration with a video conferencing system for sharing of data of remote work spaces Wireless mouses instead of a pens Further developed version described in: Jun Rekimoto, ”A Multiple Device Approach for Supporting Whiteboard-based Interaction”, Proc. CHI 98.

25. ISTD 2003 Remarks Pick-and-Drop adds physical dimension to user interface Traditional data transfer methods are too virtual and hard to learn due to their lack of physical aspects Pick-and-Drop does not allow manipulating objects that are out of the user’s physical reach Pick-and-Drop requires a stulys-sensitive surface for operation

26. ISTD 2003 Augmented Surfaces: A Spatially Continuous Work Space for Hybrid Computing Environments Jun Rekimoto, Sony Computer Science Laboratories, Japan Masanori Saitoh, Keio University, Japan Proc. ACM SIGCHI Conference on Human Factors in Computing Systems (CHI 99), Pittsburg, PA

27. ISTD 2003 Augmented Surfaces: A Spatially Continuous Work Space for Hybrid Computing Environments Jun Rekimoto, Sony Computer Science Laboratories, Japan Masanori Saitoh, Keio University, Japan Proc. CHI’99 Video

28. ISTD 2003 Motivation : Hybrid Computing Environments Different types of computers and devices in our everyday working (and living) environments Portable computers Desktops Projectors Digital whiteboards Table and wall displays Challenges Smooth integration of portable/personal and pre- installed/public computers for information exchange Connecting physical and digital (virtual) spaces

29. ISTD 2003 Spatially Continuous Workspace Evolution of spatially continuous workspaces: (a)user performs individual tasks with a portable computer (b)table becomes an extension of the portable computer (c)pre-installed computer displays (table and wall) serve as shared workspaces for collaborative tasks

30. ISTD 2003 Key Features of System Design Environmental computers as extensions of individual computers Supports for links between digital information and physical objects Spatially continuous operations (a)object dragged on computer screen (b)object ”jumps” to table, when cursor reaches edge of the screen (c)object is dragged to the wall (d)object is linked with a physical object

31. ISTD 2003 Prototype Environment: InfoTable & InfoWall

32. ISTD 2003 System Architecture Portable computers and physical objects have visual markers (printed 2D barcode)

33. ISTD 2003 Desksat Problem: how to cover the entire table surface with a single camera and sufficiently high resolution ? Solution : use a combination of two cameras Fixed camera: monitors whole table for changed areas Pan/Tilt camera: scans the table in 36 (6x6) pieces and re-visits changed areas for high resolution images Added/removed object is registered in few seconds

34. ISTD 2003 Visual Marker Recognition Each physical object has a visual marker (2D matrix code) Markers can identify 2 24 different objects Smallest size of a marker is 2cm x 2cm Recognized entities: ID, marker’s position and orientation

35. ISTD 2003 Hyperdragging (a) (b) (c) InfoTable and InfoWall provide spatially continuous workspace (low-resolution peripheral information space) to the laptop (high-resolution focal information space) (a)object is dragged towards the edge of the screen (b)object (and cursor) migrates onto InfoTable (c)object (and cursor) migrates from InfoTable onto InfoWall

36. ISTD 2003 Anchored Cursor Visual feedback: line from laptop to cursor Provided when cursor is manipulated outside the laptop’s screen Allows identifying cursors’ owners Can be used to show the link between information on the table and on the laptop

37. ISTD 2003 Object Aura Represents object’s information field (data space) Visualizes that object has been recognized by the system Digital data can be attached to physical objects by placing them on the object’s aura If object is removed, attached data is saved on network server Attached data is re-displayed, when object is placed on InfoTab´le

38. ISTD 2003 Shared Information Surfaces InfoTable and InfoWall provide shared information space for users InfoTable does not have ”top” or ”bottom” ”Near” sides are determined from visual markers Objects are automatically rotated when brought ”near”

39. ISTD 2003 Object Migration All applications are written in Java Object transfer with Java’s object serilization and RMI (Remote Method Invocation) Supported migratable object classes Text Sound (voice notes) URL’s File short-cuts Image files

40. ISTD 2003 User Experiences and Comments Hyperdragging easily understood and accepted Some users found moving objects across long distances tiresome Allows manipulating objects that are out of the user’s physical reach Mapping scale between pointer movement and the pointing device greatly affects usability InfoTable’s 20 dpi vs laptop’s screen’s 100 dpi Users appreciated attaching objects onto InfoWall while sitting at the table Integration with e.g. printers and scanners?

41. ISTD 2003 More Information Dr. Jun Rekimoto Director Interaction Laboratory Sony Computer Science Laboratories http://www.csl.sony.co.jp/person/rekimoto.html