User Interfaces for Post-Desktop Environments Jan Borchers Stanford Interactivity Lab Advanced Systems Development Lab Talk Series, PARC, 6/3/2002

Jan Borchers, Stanford University: Post-Desktop User Interfaces Outline The Stanford iRoom: A Ubicomp Environment The Stanford iRoom: A Ubicomp Environment User Interface Challenges for Ubicomp environments User Interface Challenges for Ubicomp environments iStuff: Supporting Research in Ubicomp UIs iStuff: Supporting Research in Ubicomp UIs

Jan Borchers, Stanford University: Post-Desktop User Interfaces The Shift to Ubiquitous Computing Desktop-centric Desktop-centric Generic machines Generic machines Client-server Client-server Monolithic apps Monolithic apps One person, one application, one set of hardware One person, one application, one set of hardware Invisible/implicit interaction Dedicated appliances Universal “glue-like” connectivity Infrastructure-centric services Multi-person, multi- application, multi-device integration

Jan Borchers, Stanford University: Post-Desktop User Interfaces What Is an Interactive Room? An augmented dedicated space An augmented dedicated space e.g. conference room e.g. conference room can be used with teleconnected spaces can be used with teleconnected spaces Task-oriented scenarios Task-oriented scenarios e.g. brainstorming meeting, design review e.g. brainstorming meeting, design review Multi-device, multi-user, multi-application Multi-device, multi-user, multi-application Easy to (re)configure Easy to (re)configure Movement of information across multiple devices, and from personal to/from multi-user devices Movement of information across multiple devices, and from personal to/from multi-user devices Integration & augmentation of legacy apps Integration & augmentation of legacy apps

Jan Borchers, Stanford University: Post-Desktop User Interfaces The Stanford iRoom

Jan Borchers, Stanford University: Post-Desktop User Interfaces What’s in the iRoom Today Accessible Infrastructure Accessible Infrastructure 3 4’x5’ Touch Sensitive Smart Boards 3 4’x5’ Touch Sensitive Smart Boards Back Projected 4’x6’ Interactive Mural with 12 tiled projectors (9Mpix) Back Projected 4’x6’ Interactive Mural with 12 tiled projectors (9Mpix) Interactive Table with 3’x4’ Screen and Pen Input Interactive Table with 3’x4’ Screen and Pen Input Control Panel Machine with Wacom LCD Tablet Screen Control Panel Machine with Wacom LCD Tablet Screen PointRight Mouse and Keyboard Redirection System PointRight Mouse and Keyboard Redirection System Laptop video drop switched to all displays Laptop video drop switched to all displays Cluster Control Machine with display to Smart Boards Cluster Control Machine with display to Smart Boards 3+ video cameras Projector Control System Speakers for each Smart Board X-10 Lighting Control iPaq PDA Printer, scanner Anoto pen, Wacom tablets Other custom hardware Behind the Scenes 5 Windows 2000 Boxes 5 Windows NT 4.0 Boxes 2 Windows 98 Boxes 32 processor Linux Cluster 1 Mac OS X Box Lightning 2 Digital Video Switch IEEE Wireless LAN 1000s of feet of cable! + more as time goes on...

Jan Borchers, Stanford University: Post-Desktop User Interfaces iRoom Video

Jan Borchers, Stanford University: Post-Desktop User Interfaces Application Domains 3-D Medical Imaging (Med School/Radiology) Construction project management (CivEng – Center for Integrated Facilities Engineering) Design Brainstorming (Center for Design Resarch, IDEO)

Jan Borchers, Stanford University: Post-Desktop User Interfaces Ubicomp UI Challenges Multiple users, multiple applications, multiple displays Multiple users, multiple applications, multiple displays No simple 2-D "desktop" No simple 2-D "desktop" Window? Focus? Pointing? Selection? Window? Focus? Pointing? Selection? Post-Desktop equivalent of mouse & keyboard? Post-Desktop equivalent of mouse & keyboard?

Jan Borchers, Stanford University: Post-Desktop User Interfaces Some iRoom Technology Solutions Mural cluster: High-resolution interactive wall ("Chromium") Mural cluster: High-resolution interactive wall ("Chromium") Event Heap:"Event queue for the room" Event Heap:"Event queue for the room" MultiBrowse:Sending URLs around the room MultiBrowse:Sending URLs around the room Data Heap: Storing (& converting) information in the room Data Heap: Storing (& converting) information in the room PointRight: One mouse & keyboard for the room PointRight: One mouse & keyboard for the room Room Controller:Web pad to control room lights, projectors,.... Room Controller:Web pad to control room lights, projectors,.... Interface Crafter:Generating interfaces to room services Interface Crafter:Generating interfaces to room services Barehands:Gesture control for Smart Boards Barehands:Gesture control for Smart Boards

Jan Borchers, Stanford University: Post-Desktop User Interfaces iStuff Prototyping desktop GUIs has become easy Prototyping desktop GUIs has become easy VisualBasic, Interface Builder,... VisualBasic, Interface Builder,... Lesson learned: Rapid ubicomp UI prototyping is hard Lesson learned: Rapid ubicomp UI prototyping is hard Example: "Add a feedback light to a tablet-based UI" Example: "Add a feedback light to a tablet-based UI" -> Run wires, solder adapters, write serial drivers,... -> Run wires, solder adapters, write serial drivers,... Idea: A toolkit for building ubicomp UIs? Idea: A toolkit for building ubicomp UIs? Simplify experimentation with ubicomp UI scenarios & designs Simplify experimentation with ubicomp UI scenarios & designs Basic input/output "phidgets" [Greenberg01] Basic input/output "phidgets" [Greenberg01] But: Wireless, design for augmented environment (iRoom) But: Wireless, design for augmented environment (iRoom)

Jan Borchers, Stanford University: Post-Desktop User Interfaces iStuff We've Built Input devices Input devices iButton iButton iSlider iSlider Output devices Output devices iLight iLight iBuzzer iBuzzer iSpeaker iSpeaker Others in the works (sensors, MIDI controllers, microphones, servos,...) Others in the works (sensors, MIDI controllers, microphones, servos,...)

Jan Borchers, Stanford University: Post-Desktop User Interfaces Taking Advantage of Infrastructure Any Event Heap based application can talk to iStuff Any Event Heap based application can talk to iStuff iStuff device = wireless device + PC proxy/daemon iStuff device = wireless device + PC proxy/daemon Lightweight, cheap, low-power,... devices Lightweight, cheap, low-power,... devices Application eheap.jar Event Heap eheap.jar PC daemon iSpeaker Radio Transmit eheap.jar PC proxy iSlider Radio Receiver

Jan Borchers, Stanford University: Post-Desktop User Interfaces Event Heap app to speak text import iwork.eheap2.*; class speaktext { // Connects to event heap in static void main(String []args) // arg[0], and sends AudioEvent static void main(String []args) // arg[0], and sends AudioEvent { // with the text in arg[1]. { // with the text in arg[1]. try{ try{ EventHeap theHeap=new EventHeap(args[0]); // Connect to the Event Heap EventHeap theHeap=new EventHeap(args[0]); // Connect to the Event Heap Event myEvent=new Event("AudioEvent"); // Create an event Event myEvent=new Event("AudioEvent"); // Create an event myEvent.setPostValue("AudioCommand", "Read"); // Set its fields myEvent.setPostValue("AudioCommand", "Read"); // Set its fields myEvent.setPostValue("Text", args[1]); myEvent.setPostValue("Text", args[1]); theHeap.putEvent(myEvent); // Put event into Event Heap theHeap.putEvent(myEvent); // Put event into Event Heap } catch(Exception e) { catch(Exception e) { e.printStackTrace(); e.printStackTrace(); } }} >speaktext iw-room2.stanford.edu "Hello World"

Jan Borchers, Stanford University: Post-Desktop User Interfaces Initial Evaluation Successful sample uses Successful sample uses Start The Room Start The Room iPen iPen iPong iPong Findings and next steps Findings and next steps Allows rapid integration of ubicomp UI components into apps Allows rapid integration of ubicomp UI components into apps Spend time designing the software and interface, not soldering Spend time designing the software and interface, not soldering Advantage of collaboration with Product Design & ME students Advantage of collaboration with Product Design & ME students Abstraction layers: device - normalized - application specific Abstraction layers: device - normalized - application specific Device Classes and IDs, PatchPanel Device Classes and IDs, PatchPanel Applications need to be written differently Applications need to be written differently

Jan Borchers, Stanford University: Post-Desktop User Interfaces Great Unified Input Theory iStuff wireless hardware layer can be RF, X-10, Bluetooth, b, Infrared, Cellular,... iStuff wireless hardware layer can be RF, X-10, Bluetooth, b, Infrared, Cellular,... Offer iStuff as abstraction layer of room devices above this level Offer iStuff as abstraction layer of room devices above this level Adopters can Adopters can Use our hardware and software designs and replicate devices Use our hardware and software designs and replicate devices Use our software and use off-the-shelf components as iStuff Use our software and use off-the-shelf components as iStuff Implement the "plugins" for new hardware technologies Implement the "plugins" for new hardware technologies Make applications listen only to Event Heap Events Make applications listen only to Event Heap Events

Jan Borchers, Stanford University: Post-Desktop User Interfaces Summary The Stanford iRoom The Stanford iRoom Ubicomp requires re-thinking user interface metaphors and architecture Ubicomp requires re-thinking user interface metaphors and architecture iStuff as a tool to facilitate experimentation in ubicomp interfaces iStuff as a tool to facilitate experimentation in ubicomp interfaces Not the answer to "post-desktop mouse & keyboard", but may help finding it... Not the answer to "post-desktop mouse & keyboard", but may help finding it

Bonus Material

Jan Borchers, Stanford University: Post-Desktop User Interfaces PointRight A system for pointer/keyboard redirection among multiple displays and machines (Brad Johanson, Greg Hutchins) A system for pointer/keyboard redirection among multiple displays and machines (Brad Johanson, Greg Hutchins) video

Jan Borchers, Stanford University: Post-Desktop User Interfaces BareHands Gesture-based interaction with a wall- mounted display (Merrie Ringel) Gesture-based interaction with a wall- mounted display (Merrie Ringel)