Mobile Devices for Control Brad A. Myers Brad A. Myers Human Computer Interaction Institute School of Computer Science Carnegie Mellon University

Brad Myers2 Mobile Devices l Mobile Devices for communication l As a “Personal Information Manager” l As a “Personal Digital Assistant” l As a PC replacement l “PocketPC” l For Remote Control of other devices l Remote Control of PCs l Remote Control of appliances l Mobile Devices for communication l As a “Personal Information Manager” l As a “Personal Digital Assistant” l As a PC replacement l “PocketPC” l For Remote Control of other devices l Remote Control of PCs l Remote Control of appliances

Brad Myers3 Problem l Appliances are too complex

Brad Myers4 Problem l Too many remotes

Brad Myers5 Problem l User Interfaces of Mobile Devices themselves are too hard to use

Brad Myers6 Problem l PC’s have only a keyboard and mouse as input devices

Brad Myers7 Approach l Use a personal Mobile Device as an Interface to the PC and to the Appliances

Brad Myers8 History l ParcTab – continuous communication l Early “organizers” – no communication l Newton – extra cost for synchronization l Palm – HotSync once a day l WindowsCE/PocketPC – ActiveSync once a day l ParcTab – continuous communication l Early “organizers” – no communication l Newton – extra cost for synchronization l Palm – HotSync once a day l WindowsCE/PocketPC – ActiveSync once a day

Brad Myers9 Now, Handhelds will be Communicating l (“Wi-Fi”) l BlueTooth l Cell-phone network (G3) l (Infrared) l (Wires or cradle) l (“Wi-Fi”) l BlueTooth l Cell-phone network (G3) l (Infrared) l (Wires or cradle) Toshiba e740 HP iPaq 3870 Microsoft SmartPhone

Brad Myers10 Premises of our Research “With the coming wireless technologies, connecting the PCs and PDAs together will no longer be an occasional event for synchronization. Instead, the devices will frequently be in close, interactive communication.” — Brad Myers, “Using Hand-Held Devices and PCs Together,” Comm. ACM, Vol. 44, No. 11. Nov., pp

Brad Myers11 Pebbles Project l Performed as part of the Pebbles Project l Overall goal: Use of multiple devices at the same time l Palm and desktop PC l Mobile Phone and “Smart Home” l Multiple handhelds in a meeting l Pocket PC and appliances l Multiple users with their devices l Single user with multiple devices l Performed as part of the Pebbles Project l Overall goal: Use of multiple devices at the same time l Palm and desktop PC l Mobile Phone and “Smart Home” l Multiple handhelds in a meeting l Pocket PC and appliances l Multiple users with their devices l Single user with multiple devices

Brad Myers12 Pebbles is: PEBBLESPEBBLES PEBBLESPEBBLES DAs for ntry of oth ytes and ocations from xternalources. DAs for ntry of oth ytes and ocations from xternalources.

Controlling a PC l Remote Commander l Slide Show Commander l “Semantic Snarfing” l Scrolling l Shortcutter l Remote Commander l Slide Show Commander l “Semantic Snarfing” l Scrolling l Shortcutter

Brad Myers14 Important Work by Others l PARCTab [Want 95], [Weiser, CACM’93] l Abowd: eClass using Handhelds [CHI’98] l Greenberg: public and private data [Personal Technologies, 3(1), 1999] l Jun Rekimoto: [UIST’97, CHI’98, CHI’99, …] l Pick and drop; handhelds with big displays l PARCTab [Want 95], [Weiser, CACM’93] l Abowd: eClass using Handhelds [CHI’98] l Greenberg: public and private data [Personal Technologies, 3(1), 1999] l Jun Rekimoto: [UIST’97, CHI’98, CHI’99, …] l Pick and drop; handhelds with big displays

Brad Myers15 Remote Commander l Allow PDAs to control a PC l Can be used with any application l Uses the standard (single) cursor l Don’t have to jump up and grab mouse l All mouse and keyboard functions l Use PDA like touchpad l Graffiti or our own pop-up keyboard l Word prediction and completion l Allow PDAs to control a PC l Can be used with any application l Uses the standard (single) cursor l Don’t have to jump up and grab mouse l All mouse and keyboard functions l Use PDA like touchpad l Graffiti or our own pop-up keyboard l Word prediction and completion

Brad Myers16 For People with Muscular Disabilities l Using handhelds as interface to PCs l People with Muscular Dystrophy have fine-motor control but lose gross motor control l Difficulties with mouse and keyboard, but stylus OK l So use Remote Commander as PC’s keyboard and mouse l Using handhelds as interface to PCs l People with Muscular Dystrophy have fine-motor control but lose gross motor control l Difficulties with mouse and keyboard, but stylus OK l So use Remote Commander as PC’s keyboard and mouse

Brad Myers17 PocketPC version l Get PC’s screen onto PocketPC l Full view, or one-to-one zooming l Scroll with iPaq’s buttons l Control or scribble l Get PC’s screen onto PocketPC l Full view, or one-to-one zooming l Scroll with iPaq’s buttons l Control or scribble

Brad Myers18 Power Point Control l Use PC to give the presentation l Use hand-held to control the PC l Two-way communication l Hand-held shows picture of slide, notes, timer l Use PC to give the presentation l Use hand-held to control the PC l Two-way communication l Hand-held shows picture of slide, notes, timer

Brad Myers19 SlideShow Commander, cont. l See list of slide titles l Scribble l Tap on on-screen buttons and links l Preview other slides l Control demonstrations l See list of slide titles l Scribble l Tap on on-screen buttons and links l Preview other slides l Control demonstrations

Brad Myers20 Control at a Distance l Controlling a PC from across the room l Meetings, etc. l Use a laser pointer? l Studies l Difficult due to: l Jitter of hand (+/- 10 pixels) l No button on the laser pointer l People not accurate at where beam will go on and off l Controlling a PC from across the room l Meetings, etc. l Use a laser pointer? l Studies l Difficult due to: l Jitter of hand (+/- 10 pixels) l No button on the laser pointer l People not accurate at where beam will go on and off

Brad Myers21 Idea:“Semantic Snarfing” l Grab contents to handheld l Picture l “Semantic” because reformat based on the meaning of the contents l Menus l Text l Re-visualization l Grab contents to handheld l Picture l “Semantic” because reformat based on the meaning of the contents l Menus l Text l Re-visualization

Brad Myers22 Command Post of the Future l Large DARPA funded project l Make commanders more effective l Private drill-down of public information l Large DARPA funded project l Make commanders more effective l Private drill-down of public information

Brad Myers23 Classroom Investigations l Investigate improving large lectures with frequent in-class “concept tests” l Provide immediate feedback to instructor l Easy for instructor l Investigate improving large lectures with frequent in-class “concept tests” l Provide immediate feedback to instructor l Easy for instructor

Brad Myers24 Scrolling Desktop Applications l Scrolling using the non-dominant hand l Parallel and efficient use of both hands l Generates Windows scrolling events l Study showed can be faster for some tasks l Scrolling using the non-dominant hand l Parallel and efficient use of both hands l Generates Windows scrolling events l Study showed can be faster for some tasks

Brad Myers25 Shortcutter l User-created panels of controls l Create custom interfaces and extensions to PC applications l And then take them with you l Direct manipulation for edit, then set properties l PocketPC or Palm l User-created panels of controls l Create custom interfaces and extensions to PC applications l And then take them with you l Direct manipulation for edit, then set properties l PocketPC or Palm

Brad Myers26 Shortcutter Widgets l Buttons l Virtual Joy stick l Virtual Knob l Sliders l Mouse pad l Gesture panel l Buttons l Virtual Joy stick l Virtual Knob l Sliders l Mouse pad l Gesture panel

Brad Myers27 Shortcutter Actions l Send any keyboard key, mouse button, scrolling action or string to PC l Open a file or URL l Run an application l Invoke any PC menu or button l Windows message l Recorded l Switch to a different Shortcutter panel l Control the Mouse l Send any keyboard key, mouse button, scrolling action or string to PC l Open a file or URL l Run an application l Invoke any PC menu or button l Windows message l Recorded l Switch to a different Shortcutter panel l Control the Mouse

Brad Myers28 Shortcutter Actions, cont. l Control external devices through PC’s serial port l Directly (e.g., projectors) l X-10 for electrical devices l Macro l Can be multi-application l Application-specific l Same button, different messages l Useful for application sets: browsers, compilers l Control external devices through PC’s serial port l Directly (e.g., projectors) l X-10 for electrical devices l Macro l Can be multi-application l Application-specific l Same button, different messages l Useful for application sets: browsers, compilers

Brad Myers29 More Scenarios of Use l Lean-back mail reading l Controlling media players l … and many others l Lean-back mail reading l Controlling media players l … and many others

Controlling Appliances l PhD research of Jeffrey Nichols

Problem l Too many complex devices, each with its own idiosyncratic interface l Stereo system l Telephones l ATM l Fax machine l Photocopier l Hotel alarm clock l Increasingly computerized l Low usability l Too many complex devices, each with its own idiosyncratic interface l Stereo system l Telephones l ATM l Fax machine l Photocopier l Hotel alarm clock l Increasingly computerized l Low usability

Brad Myers32 Problem

Brad Myers33 Existing “universal” controls l Pre-programmed at the factory with a subset, l Or, Laboriously hand-programmed by the user l Pre-programmed at the factory with a subset, l Or, Laboriously hand-programmed by the user

Brad Myers34 Important Work by Others l Xweb [Olsen Jr. 2000] l INCITS V2 standardization effort: Alternative Interface Access Protocol (AIAP) [Zimmermann, CHI’02] l Stanford iRoom, iCrafter [Ponnekanti, UBICOMP 2001] l Xweb [Olsen Jr. 2000] l INCITS V2 standardization effort: Alternative Interface Access Protocol (AIAP) [Zimmermann, CHI’02] l Stanford iRoom, iCrafter [Ponnekanti, UBICOMP 2001]

Brad Myers35 Our Approach l Handheld is a “Personal Universal Controller” (PUC) l Two-way communication l Appliance describes its functions l Handheld PUC l Automatically creates interface l Controls the appliance l Displays feedback about appliance status l Handheld is a “Personal Universal Controller” (PUC) l Two-way communication l Appliance describes its functions l Handheld PUC l Automatically creates interface l Controls the appliance l Displays feedback about appliance status Specifications Control Feedback

Brad Myers36 Automatic Generation of UIs l Benefits l All interfaces consistent for a user l With conventions of the handheld l Even from multiple manufacturers l Addresses hotel alarm clock problem l Can take into account user preferences l Multiple modalities (GUI + Speech UI) l A Hard Problem l Previous automatic systems have not generated high quality interfaces l Benefits l All interfaces consistent for a user l With conventions of the handheld l Even from multiple manufacturers l Addresses hotel alarm clock problem l Can take into account user preferences l Multiple modalities (GUI + Speech UI) l A Hard Problem l Previous automatic systems have not generated high quality interfaces

Brad Myers37 Research Challenges 1. Automatic Design of Graphical User Interfaces 2. Automatic Design of Speech User Interfaces 3. Connection with real devices l Through various protocols l X-10, AV/C, HAVi, UPnP, etc. l Also, custom hardware and software 1. Automatic Design of Graphical User Interfaces 2. Automatic Design of Speech User Interfaces 3. Connection with real devices l Through various protocols l X-10, AV/C, HAVi, UPnP, etc. l Also, custom hardware and software

Brad Myers38 Hand-Generated Graphical Interfaces l First, Hand-Designed PocketPC interfaces: l AIWA Shelf Stereo (Tape,CD,Tuner) l AT&T Telephone/Answering Machine l Used Embedded Visual Basic l Ensured quality with heuristic analysis and think-aloud studies l Compared with manufacturer’s interfaces l First, Hand-Designed PocketPC interfaces: l AIWA Shelf Stereo (Tape,CD,Tuner) l AT&T Telephone/Answering Machine l Used Embedded Visual Basic l Ensured quality with heuristic analysis and think-aloud studies l Compared with manufacturer’s interfaces

Brad Myers39 Results of Comparison l Using PUC, users took 50% less time & made 50% fewer errors l All differences are significant (p < 0.05) l Using PUC, users took 50% less time & made 50% fewer errors l All differences are significant (p < 0.05)

Brad Myers40 Discussion of Comparison l Our hand designed interfaces succeeded for several reasons l Good organization l Each button has one function l Good labels l Only available functions are active l Others, hidden on tabs or grayed out l Better feedback and error messages l Our hand designed interfaces succeeded for several reasons l Good organization l Each button has one function l Good labels l Only available functions are active l Others, hidden on tabs or grayed out l Better feedback and error messages

Brad Myers41 Current PUC Specification Language l XML l Full documentation for the specification language and protocol: l Contains sample specification for a stereo l XML l Full documentation for the specification language and protocol: l Contains sample specification for a stereo

Brad Myers42 Properties of PUC Language l State variables & commands l Each can have multiple labels l Useful when not enough room l Typed variables l Base types: Boolean, string, enumerated, integers, fixed-point, floating-point, etc. l Optional labels for values l Hierarchical Structure l Groups l State variables & commands l Each can have multiple labels l Useful when not enough room l Typed variables l Base types: Boolean, string, enumerated, integers, fixed-point, floating-point, etc. l Optional labels for values l Hierarchical Structure l Groups

Brad Myers43 Dependency Information l Crucial for high-quality interfaces l Expressed as clauses l Operations: l Equals, Less-Than, Greater-Than l Combined Logically l AND, OR l Used for: l Dynamic graying out l Layout l Widget selection l Crucial for high-quality interfaces l Expressed as clauses l Operations: l Equals, Less-Than, Greater-Than l Combined Logically l AND, OR l Used for: l Dynamic graying out l Layout l Widget selection

Brad Myers44 Specifications l Have working specifications for: l Audiophase stereo l X-10 lights control l Sony CamCorder l Windows Media Player l Audio ReQuest hardware MP3 player l WinAmp Media Player l Elevator l Have working specifications for: l Audiophase stereo l X-10 lights control l Sony CamCorder l Windows Media Player l Audio ReQuest hardware MP3 player l WinAmp Media Player l Elevator

Brad Myers45 Examples of Generated GUIs l Stereo and X-10

Brad Myers46 More Examples l Elevator

Brad Myers47 Generating Speech Interfaces l “Universal Speech Interface” (USI) project l Prof. Roni Rosenfeld of CMU l l Creates grammar, language model and pronunciation dictionary from PUC specification l Pronunciation from labels using phonetic rules l Can provide other pronunciations as labels for fine-tuning l Will use dependency information to help with disambiguation and explanation l Supports queries and spoken feedback l Paraphrases as confirmation l “Universal Speech Interface” (USI) project l Prof. Roni Rosenfeld of CMU l l Creates grammar, language model and pronunciation dictionary from PUC specification l Pronunciation from labels using phonetic rules l Can provide other pronunciations as labels for fine-tuning l Will use dependency information to help with disambiguation and explanation l Supports queries and spoken feedback l Paraphrases as confirmation

Brad Myers48 PUC Architecture

Brad Myers49 Adaptors l “Adaptors” provide the interface to existing (and future) appliances l If do not support specification language directly l Custom hardware l Audiophase Stereo l Custom software l ARQ MP3 player l SIMA MP3 player (future) l X-10 l Light switches, etc. l “Adaptors” provide the interface to existing (and future) appliances l If do not support specification language directly l Custom hardware l Audiophase Stereo l Custom software l ARQ MP3 player l SIMA MP3 player (future) l X-10 l Light switches, etc. l AV/C (standard protocol) l Sony CamCorder l PlayStation, etc. (future) l HAVi (current work) l Mitsubishi HDTV 65” TV l Mitsubishi HDTV VCR l UPnP (future) l ??

Brad Myers50 Adaptor using Custom Hardware for Stereo l Pretends to send IR codes l Reads LED panel signals to decode state l Created by Pittsburgh company: Maya Design l Pretends to send IR codes l Reads LED panel signals to decode state l Created by Pittsburgh company: Maya Design

Brad Myers51 Adaptor for AV/C Hand-written spec of AV/C devices PUC Java Proxy Our Java device code Our C++ code Microsoft DirectShow library AV/C protocol Firewire cable (IEEE 1394) PUC Java Library Automatic GUI generation Digital Video Camera or VCR Multiple device controllers Automatic Speech UI generation

Brad Myers52 Demonstration l X-10 l Camcorder through AV/C protocol for IEEE 1394 (Firewire) l Two way communication: l When state changes from appliance, GUI is updated and can query state with speech l X-10 l Camcorder through AV/C protocol for IEEE 1394 (Firewire) l Two way communication: l When state changes from appliance, GUI is updated and can query state with speech

Brad Myers53 For More Information l Many papers on the Pebbles web site l On PUC and other work l Most programs available for free downloading: l SlideShow Commander available commercially l Many papers on the Pebbles web site l On PUC and other work l Most programs available for free downloading: l SlideShow Commander available commercially

Brad Myers54 Conclusions l Study The Big Picture: l How mobile devices fit into users’ entire information and control space l As more and more electronics are computerized, mobiles devices can help control them l Mobile devices can improve the user interfaces of everything else l Not just be another gadget to be learned l Study The Big Picture: l How mobile devices fit into users’ entire information and control space l As more and more electronics are computerized, mobiles devices can help control them l Mobile devices can improve the user interfaces of everything else l Not just be another gadget to be learned

Brad Myers55 Thanks to Our Sponsors! The Pebbles research project is supported by grants from: DARPA Microsoft NSF Pittsburgh Digital Greenhouse General Motors And equipment grants from: Hewlett Packard Lucent Technologies Mitsubishi Palm Computing Symbol Technologies The Pebbles research project is supported by grants from: DARPA Microsoft NSF Pittsburgh Digital Greenhouse General Motors And equipment grants from: Hewlett Packard Lucent Technologies Mitsubishi Palm Computing Symbol Technologies IBM SMART Technologies VividLogic Synergy Solutions Handango

Mobile Devices for Control Brad A. Myers Brad A. Myers Human Computer Interaction Institute School of Computer Science Carnegie Mellon University