1. Handheld Devices for Control Brad A. Myers bam@cs.cmu.edu http://www.cs.cmu.edu/~pebbles Brad A. Myers bam@cs.cmu.edu http://www.cs.cmu.edu/~pebbles Human Computer Interaction Institute School of Computer Science Carnegie Mellon University

2. Brad MyersCarnegie Mellon2 Handheld Devices l Handheld Devices for: l Communication l As a “Personal Information Manager” l As a “Personal Digital Assistant” l As a PC replacement l “PocketPC” l Handheld Devices for Remote Control of other devices l Remote Control of PCs l Remote Control of appliances l Handheld Devices for: l Communication l As a “Personal Information Manager” l As a “Personal Digital Assistant” l As a PC replacement l “PocketPC” l Handheld Devices for Remote Control of other devices l Remote Control of PCs l Remote Control of appliances

3. Brad MyersCarnegie Mellon3 Problem l Appliances are too complex

4. Brad MyersCarnegie Mellon4 Problem l Too many remotes

5. Brad MyersCarnegie Mellon5 Problem l April 8, 1991

6. Brad MyersCarnegie Mellon6 Approach l Use a personal handheld device as an Interface to the PC and to the Appliances

7. Brad MyersCarnegie Mellon7 History of Communication l ParcTab – continuous communication l Early “organizers” – no communication l Newton – extra cost for synchronization l Palm – HotSync once a day l WindowsCE/PocketPC – ActiveSync once a day l ParcTab – continuous communication l Early “organizers” – no communication l Newton – extra cost for synchronization l Palm – HotSync once a day l WindowsCE/PocketPC – ActiveSync once a day

8. Brad MyersCarnegie Mellon8 Now, Handhelds will be Communicating l 802.11 l BlueTooth l Cell-phone network (G3) l (Infrared) l (Wires or cradle) l 802.11 l BlueTooth l Cell-phone network (G3) l (Infrared) l (Wires or cradle) Toshiba e740 HP iPaq 3870 Microsoft SmartPhone

9. Brad MyersCarnegie Mellon9 Premises of our Research “With the coming wireless technologies, connecting the PCs and PDAs together will no longer be an occasional event for synchronization. Instead, the devices will frequently be in close, interactive communication.” — Brad Myers, “Using Hand-Held Devices and PCs Together,” Comm. ACM, Vol. 44, No. 11. Nov., 2001. pp. 34 - 41.

10. Brad MyersCarnegie Mellon10 Pebbles Project l Performed as part of the Pebbles Project l Overall goal: Use of multiple devices at the same time l Palm and desktop PC l Mobile Phone and “Smart Home” l Multiple handhelds in a meeting l Pocket PC and appliances l Multiple users with their devices l Single user with multiple devices l Performed as part of the Pebbles Project l Overall goal: Use of multiple devices at the same time l Palm and desktop PC l Mobile Phone and “Smart Home” l Multiple handhelds in a meeting l Pocket PC and appliances l Multiple users with their devices l Single user with multiple devices

11. Brad MyersCarnegie Mellon11 Pebbles is: PEBBLESPEBBLES PEBBLESPEBBLES DAs for ntry of oth ytes and ocations from xternalources. DAs for ntry of oth ytes and ocations from xternalources. http://www.pebbles.hcii.edu/

12. Controlling a PC l Remote Commander l Slide Show Commander l “Semantic Snarfing” l Shortcutter l Remote Commander l Slide Show Commander l “Semantic Snarfing” l Shortcutter http://www.pebbles.hcii.edu/

13. Brad MyersCarnegie Mellon13 Remote Commander l Allow PDAs to control a PC l Can be used with any application l Uses the standard (single) cursor l Don’t have to jump up and grab mouse l All mouse and keyboard functions l Use PDA like touchpad l Graffiti or our own pop-up keyboard l Word prediction and completion l Allow PDAs to control a PC l Can be used with any application l Uses the standard (single) cursor l Don’t have to jump up and grab mouse l All mouse and keyboard functions l Use PDA like touchpad l Graffiti or our own pop-up keyboard l Word prediction and completion

14. Brad MyersCarnegie Mellon14 PocketPC version l Get PC’s screen onto PocketPC l Full view, or one-to-one zooming l Scroll with iPaq’s buttons l Control or scribble l Get PC’s screen onto PocketPC l Full view, or one-to-one zooming l Scroll with iPaq’s buttons l Control or scribble

15. Brad MyersCarnegie Mellon15 For People with Muscular Disabilities l Using handhelds as interface to PCs l People with Muscular Dystrophy have fine-motor control but lose gross motor control l Difficulties with mouse and keyboard, but stylus OK l So use Remote Commander as PC’s keyboard and mouse l Using handhelds as interface to PCs l People with Muscular Dystrophy have fine-motor control but lose gross motor control l Difficulties with mouse and keyboard, but stylus OK l So use Remote Commander as PC’s keyboard and mouse

16. Brad MyersCarnegie Mellon16 Alphabet for Accurate Gestures l Use a square hole in a template to increase accuracy l Just as easy to learn as Graffiti™ l 18% more accurate for able novices (p<.02) l Patent Pending l Use a square hole in a template to increase accuracy l Just as easy to learn as Graffiti™ l 18% more accurate for able novices (p<.02) l Patent Pending NEW!

17. Brad MyersCarnegie Mellon17 Shared Clipboard l Transfer information among your personal devices l E.g., between PDA and PC l Also between multiple computers l Connect all clipboards together l Transfer content or reference l Works with all applications l Transfer information among your personal devices l E.g., between PDA and PC l Also between multiple computers l Connect all clipboards together l Transfer content or reference l Works with all applications

18. Brad MyersCarnegie Mellon18 Power Point Control l Use PC to give the presentation l Use hand-held to control the PC l Two-way communication l Hand-held shows picture of slide, notes, timer l Use PC to give the presentation l Use hand-held to control the PC l Two-way communication l Hand-held shows picture of slide, notes, timer

19. Brad MyersCarnegie Mellon19 SlideShow Commander, cont. l See list of slide titles l Scribble l Tap on on-screen buttons and links l Preview other slides l Control demonstrations l See list of slide titles l Scribble l Tap on on-screen buttons and links l Preview other slides l Control demonstrations

20. Brad MyersCarnegie Mellon20 Control at a Distance l Controlling a PC from across the room l Meetings, etc. l Use a laser pointer? l Studies l “Semantic Snarfing” l Grab contents to handheld l Controlling a PC from across the room l Meetings, etc. l Use a laser pointer? l Studies l “Semantic Snarfing” l Grab contents to handheld

21. Brad MyersCarnegie Mellon21 Shortcutter l User-created panels of controls l Create custom interfaces and extensions to PC applications l And then take them with you l Direct manipulation for edit, then set properties l PocketPC or Palm l User-created panels of controls l Create custom interfaces and extensions to PC applications l And then take them with you l Direct manipulation for edit, then set properties l PocketPC or Palm

22. Brad MyersCarnegie Mellon22 Shortcutter Widgets l Buttons l Virtual Joy stick l Virtual Knob l Sliders l Mouse pad l Gesture panel l Buttons l Virtual Joy stick l Virtual Knob l Sliders l Mouse pad l Gesture panel

23. Brad MyersCarnegie Mellon23 Scenarios of Use l Lean-back mail reading l Controlling media players l … and many others l Lean-back mail reading l Controlling media players l … and many others

24. Controlling Appliances l PhD research of Jeffrey Nichols l Assisted by undergrads and MS students l PhD research of Jeffrey Nichols l Assisted by undergrads and MS students http://www.pebbles.hcii.edu/puc/

25. Problem l Too many complex devices, each with its own idiosyncratic interface l Stereo system l Telephones l ATM l Fax machine l Photocopier l Hotel alarm clock l Increasingly computerized l Low usability l Too many complex devices, each with its own idiosyncratic interface l Stereo system l Telephones l ATM l Fax machine l Photocopier l Hotel alarm clock l Increasingly computerized l Low usability

26. Brad MyersCarnegie Mellon26 Existing “universal” controls l Pre-programmed at the factory with a subset, l Or, laboriously hand-programmed by the user l Pre-programmed at the factory with a subset, l Or, laboriously hand-programmed by the user

27. Brad MyersCarnegie Mellon27 Our Approach l Handheld is a “Personal Universal Controller” (PUC) l Two-way communication l Appliance describes its functions l Handheld PUC l Automatically creates interface l Controls the appliance l Displays feedback about appliance status l Handheld is a “Personal Universal Controller” (PUC) l Two-way communication l Appliance describes its functions l Handheld PUC l Automatically creates interface l Controls the appliance l Displays feedback about appliance status Specifications Control Feedback

28. Brad MyersCarnegie Mellon28 Automatic Generation of UIs l Benefits l All interfaces consistent for a user l With conventions of the handheld l Even from multiple manufacturers l Addresses hotel alarm clock problem l Can take into account user preferences l Multiple modalities (GUI + Speech UI) l A Hard Problem l Previous automatic systems have not generated high quality interfaces l Benefits l All interfaces consistent for a user l With conventions of the handheld l Even from multiple manufacturers l Addresses hotel alarm clock problem l Can take into account user preferences l Multiple modalities (GUI + Speech UI) l A Hard Problem l Previous automatic systems have not generated high quality interfaces

29. Brad MyersCarnegie Mellon29 Research Challenges 1. Automatic Design of Graphical User Interfaces 2. Automatic Design of Speech User Interfaces 3. Connection with real devices l Through various protocols l X-10, AV/C, HAVi, UPnP, etc. l Also, custom hardware and software 1. Automatic Design of Graphical User Interfaces 2. Automatic Design of Speech User Interfaces 3. Connection with real devices l Through various protocols l X-10, AV/C, HAVi, UPnP, etc. l Also, custom hardware and software

30. Brad MyersCarnegie Mellon30 Hand-Generated Graphical Interfaces l First, Hand-Designed PocketPC interfaces: l AIWA Shelf Stereo (Tape,CD,Tuner) l AT&T Telephone/Answering Machine l Used Embedded Visual Basic l Ensured quality with heuristic analysis and think-aloud studies l Compared with manufacturer’s interfaces l First, Hand-Designed PocketPC interfaces: l AIWA Shelf Stereo (Tape,CD,Tuner) l AT&T Telephone/Answering Machine l Used Embedded Visual Basic l Ensured quality with heuristic analysis and think-aloud studies l Compared with manufacturer’s interfaces

31. Brad MyersCarnegie Mellon31 Results of Comparison l Using PUC, users took 50% less time & made 50% fewer errors l All differences are significant (p < 0.05) l Using PUC, users took 50% less time & made 50% fewer errors l All differences are significant (p < 0.05)

32. Brad MyersCarnegie Mellon32 Current PUC Specification Language l XML l Full documentation for the specification language and protocol: http://www.pebbles.hcii.cmu.edu/puc/ l Contains sample specification for a stereo l XML l Full documentation for the specification language and protocol: http://www.pebbles.hcii.cmu.edu/puc/ l Contains sample specification for a stereo

33. Brad MyersCarnegie Mellon33 Properties of PUC Language l State variables & commands l Each can have multiple labels l Useful when not enough room l Typed variables l Base types: Boolean, string, enumerated, integers, fixed-point, floating-point, etc. l Optional labels for values l Hierarchical Structure l Groups l State variables & commands l Each can have multiple labels l Useful when not enough room l Typed variables l Base types: Boolean, string, enumerated, integers, fixed-point, floating-point, etc. l Optional labels for values l Hierarchical Structure l Groups

34. Brad MyersCarnegie Mellon34 Dependency Information l Crucial for high-quality interfaces l Expressed as clauses l Operations: l Equals, Less-Than, Greater-Than l Combined Logically l AND, OR l Used for: l Dynamic graying out l Layout l Widget selection l Crucial for high-quality interfaces l Expressed as clauses l Operations: l Equals, Less-Than, Greater-Than l Combined Logically l AND, OR l Used for: l Dynamic graying out l Layout l Widget selection

35. Brad MyersCarnegie Mellon35 Specifications l Have working specifications for: l Audiophase stereo l X-10 lights control l Sony CamCorder l Windows Media Player l Audio ReQuest hardware MP3 player l WinAmp Media Player l Elevator l Parts of GMC Yukon Denali SUV l Etc. l Have working specifications for: l Audiophase stereo l X-10 lights control l Sony CamCorder l Windows Media Player l Audio ReQuest hardware MP3 player l WinAmp Media Player l Elevator l Parts of GMC Yukon Denali SUV l Etc.

36. Brad MyersCarnegie Mellon36 Controller Generators l iPaq PocketPC l SmartPhone l No touchscreen l Desktop (TabletPC) l Speech l iPaq PocketPC l SmartPhone l No touchscreen l Desktop (TabletPC) l Speech

37. Brad MyersCarnegie Mellon37 Examples of Generated GUIs Shelf stereoWindows Media Player

38. Brad MyersCarnegie Mellon38 Generating Speech Interfaces l “Universal Speech Interface” (USI) project l Prof. Roni Rosenfeld of CMU l http://www.cs.cmu.edu/~usi l Creates grammar, language model and pronunciation dictionary from PUC specification l Pronunciation from labels using phonetic rules l Can provide other pronunciations as labels for fine-tuning l Will use dependency information to help with disambiguation and explanation l Supports queries and spoken feedback l Paraphrases as confirmation l “Universal Speech Interface” (USI) project l Prof. Roni Rosenfeld of CMU l http://www.cs.cmu.edu/~usi l Creates grammar, language model and pronunciation dictionary from PUC specification l Pronunciation from labels using phonetic rules l Can provide other pronunciations as labels for fine-tuning l Will use dependency information to help with disambiguation and explanation l Supports queries and spoken feedback l Paraphrases as confirmation

39. Brad MyersCarnegie Mellon39 Adaptors l “Adaptors” provide the interface to existing (and future) appliances l If do not support specification language directly l Custom hardware l Custom software l Lutron l Windows Media Player l X-10 l Light switches, etc. l “Adaptors” provide the interface to existing (and future) appliances l If do not support specification language directly l Custom hardware l Custom software l Lutron l Windows Media Player l X-10 l Light switches, etc. l AV/C (standard protocol) l Sony CamCorder l HAVi l UPnP l Axis Camera

40. Brad MyersCarnegie Mellon40 Demonstration l Sony CamCorder l Windows Media Player l Simulated automobile l Two way communication: l When state changes from application or appliance, GUI is updated l Can query state with speech l Sony CamCorder l Windows Media Player l Simulated automobile l Two way communication: l When state changes from application or appliance, GUI is updated l Can query state with speech

41. Brad MyersCarnegie Mellon41 Future Work on PUC l Controlling User “Experiences” l One command for multiple devices l Play a DVD  sets DVD, cable, TV l Setting alarm clock  sets heat, coffee, car l Consistency for Users l And user preferences l More devices l UPnP l Etc. l Controlling User “Experiences” l One command for multiple devices l Play a DVD  sets DVD, cable, TV l Setting alarm clock  sets heat, coffee, car l Consistency for Users l And user preferences l More devices l UPnP l Etc.

42. Brad MyersCarnegie Mellon42 For More Information l Many papers on the Pebbles web site l On PUC and other work l Most programs available for free downloading: l SlideShow Commander was licensed for commercial sale l Many papers on the Pebbles web site l On PUC and other work l Most programs available for free downloading: l SlideShow Commander was licensed for commercial sale http://www.cs.cmu.edu/~pebbles 

43. Brad MyersCarnegie Mellon43 Conclusions l As more and more electronics are computerized and communicating, mobile devices can help control them l Handheld devices can improve the user interfaces of everything else l Not just be another gadget to be learned l As more and more electronics are computerized and communicating, mobile devices can help control them l Handheld devices can improve the user interfaces of everything else l Not just be another gadget to be learned

44. Brad MyersCarnegie Mellon44 Thanks to Pebbles Students l This research performed by a large number of students since 1997: Htet Htet Aung Rishi Bhatnagar Ben Bostwick Franklin Chen Yu Shan Chuang Karen Cross Carl Evankovich Marc Khadpe Dave Kong Chun-Kwok Lee Joonhwan Lee Jennifer Li Leo Lie Jack Lin Kevin Litwack A. Chris Long Rob Miller Jeff Nichols Choon Hong Peck Mathilde Pignol Suporn Pongnumkul Rajesh Seenichamy Herbert Stiel Jeff Stylos Claire Tokar Marsha Tjandra Adrienne Warmack Jacob O. Wobbrock Jerry Yang Sunny Yang Brian Yeung

45. Brad MyersCarnegie Mellon45 Thanks to Our Sponsors! The Pebbles research is supported by grants from: DARPA Microsoft NSF General Motors NEC Foundation Pittsburgh Digital Greenhouse And equipment grants from: Hewlett Packard Lucent Technologies Lantronix, Inc. Lutron, Inc. Mitsubishi Palm Computing The Pebbles research is supported by grants from: DARPA Microsoft NSF General Motors NEC Foundation Pittsburgh Digital Greenhouse And equipment grants from: Hewlett Packard Lucent Technologies Lantronix, Inc. Lutron, Inc. Mitsubishi Palm Computing Symbol Technologies IBM SMART Technologies VividLogic Synergy Solutions Handango

46. Handheld Devices for Control Brad A. Myers bam@cs.cmu.edu http://www.cs.cmu.edu/~pebbles Brad A. Myers bam@cs.cmu.edu http://www.cs.cmu.edu/~pebbles Human Computer Interaction Institute School of Computer Science Carnegie Mellon University

47. Black, followed by HAVi

48. Pebbles PUC Project use of the Home Audio Visual Interface

49. Brad MyersCarnegie Mellon49 About HAVi l Home Audio Visual Interface l http://www.havi.org l Protocol for controlling appliances over IEEE 1394 (i.LINK® or FireWire®) cables l Mostly used for video equipment l Supported by various Mitsubishi and RCA high-end products l Home Audio Visual Interface l http://www.havi.org l Protocol for controlling appliances over IEEE 1394 (i.LINK® or FireWire®) cables l Mostly used for video equipment l Supported by various Mitsubishi and RCA high-end products

50. Brad MyersCarnegie Mellon50 Our Goals for HAVi l Want to: l Detect HAVi devices on Firewire bus l Automatically create PUC specification from device capabilities l Control all of the appliance’s features using a PC as a server (proxy) l HAVi seems to have appropriate features for these l Want to: l Detect HAVi devices on Firewire bus l Automatically create PUC specification from device capabilities l Control all of the appliance’s features using a PC as a server (proxy) l HAVi seems to have appropriate features for these

51. Brad MyersCarnegie Mellon51 HAVi Experience l MERL donated l A 65-inch high-definition TV (WS-65909) l A high definition digital VCR (HS-HD2000U) l VividLogic FireBus HAVi SDK for Linux, v. 1.0.8 l MERL donated l A 65-inch high-definition TV (WS-65909) l A high definition digital VCR (HS-HD2000U) l VividLogic FireBus HAVi SDK for Linux, v. 1.0.8

52. Brad MyersCarnegie Mellon52 What We’ve Done l Investigated the (several) different ways of controlling HAVi devices l Made simple applications to control HAVi devices l But don’t have a working PUC interface l Have put efforts on hold l Limitations of HAVi l Problems with SDK l Investigated the (several) different ways of controlling HAVi devices l Made simple applications to control HAVi devices l But don’t have a working PUC interface l Have put efforts on hold l Limitations of HAVi l Problems with SDK

53. Brad MyersCarnegie Mellon53 Television l Under the HAVi system, acts as the “master controller” l Refuses to be controlled or to respond to our requests l Even having the TV and PC on same bus causes problems l Under the HAVi system, acts as the “master controller” l Refuses to be controlled or to respond to our requests l Even having the TV and PC on same bus causes problems

54. Brad MyersCarnegie Mellon54 VCR l Several different ways to control VCR l VCR FCM l “Functional Control Module” l Static list of capabilities l Havlet l A Java program uploaded and run on TV (or PC) l DDI l “Data-Driven Interaction” l Panel-based interface for arbitrary control l Several different ways to control VCR l VCR FCM l “Functional Control Module” l Static list of capabilities l Havlet l A Java program uploaded and run on TV (or PC) l DDI l “Data-Driven Interaction” l Panel-based interface for arbitrary control

55. Brad MyersCarnegie Mellon55 VCR FCM l A few more features than AV/C: l Play, Record, Fast Forward, Fast Reverse, Variable Forward, Variable Reverse, Stop, Record-Pause, Skip, Eject Media, Set Recording Mode, ClearRTC l Get State, Get Recording Mode, Get Format, Get Position, Get capability, Get Reject Info l No querying for capabilities l (Apparent) Bugs in SDK on getting status l Have created simple programs to control VCR using the FCM l A few more features than AV/C: l Play, Record, Fast Forward, Fast Reverse, Variable Forward, Variable Reverse, Stop, Record-Pause, Skip, Eject Media, Set Recording Mode, ClearRTC l Get State, Get Recording Mode, Get Format, Get Position, Get capability, Get Reject Info l No querying for capabilities l (Apparent) Bugs in SDK on getting status l Have created simple programs to control VCR using the FCM

56. Brad MyersCarnegie Mellon56 HavLets l Java Applets downloaded to a Java interpreter l No hooks for programmatic control l Very limited functions provided in the VCR’s HavLet l Rewind, Play, Fast Forward, Stop, Pause, Record l Status, Counter, Media Type, Recording Rate of Tape, Rate of Data Source l Java Applets downloaded to a Java interpreter l No hooks for programmatic control l Very limited functions provided in the VCR’s HavLet l Rewind, Play, Fast Forward, Stop, Pause, Record l Status, Counter, Media Type, Recording Rate of Tape, Rate of Data Source

57. Brad MyersCarnegie Mellon57 Data-Driven Interaction (DDI) l We hoped this would provide device querying and control, but… l Only limited features provided in DDI l Same as HavLet interface l Rewind, Play, Fast Forward, Stop, Pause, Record, and some status requests l Provides each UI element as a separate item l E.g., “Time: ” and “54:35:33” may be two separate labels placed next to each other l Need to infer relationships and types by layout l We hoped this would provide device querying and control, but… l Only limited features provided in DDI l Same as HavLet interface l Rewind, Play, Fast Forward, Stop, Pause, Record, and some status requests l Provides each UI element as a separate item l E.g., “Time: ” and “54:35:33” may be two separate labels placed next to each other l Need to infer relationships and types by layout

58. Brad MyersCarnegie Mellon58 Issues with VividLogic’s SDK l Their engineers have been very helpful l SDK incomplete, poorly documented and quite buggy l Only runs with a specific desktop Firewire card, only on Linux (for now) l Slow to come out with new versions l Needed to be “FAV” (Full AV) to use DDI but FAV support not available until Dec’02 l Their engineers have been very helpful l SDK incomplete, poorly documented and quite buggy l Only runs with a specific desktop Firewire card, only on Linux (for now) l Slow to come out with new versions l Needed to be “FAV” (Full AV) to use DDI but FAV support not available until Dec’02

59. Brad MyersCarnegie Mellon59 Demo l Show other devices we have gotten to work l demo demo l Show other devices we have gotten to work l demo demo

60. Black, followed by OLD

61. Brad MyersCarnegie Mellon61 Adaptor for AV/C Hand-written spec of AV/C devices PUC Java Proxy Our Java device code Our C++ code Microsoft DirectShow library AV/C protocol Firewire cable (IEEE 1394) PUC Java Library Automatic GUI generation Digital Video Camera or VCR Multiple device controllers Automatic Speech UI generation

62. Brad MyersCarnegie Mellon62 Adaptor using Custom Hardware for Stereo l Pretends to send IR codes l Reads LED panel signals to decode state l Created by Pittsburgh company: Maya Design l Pretends to send IR codes l Reads LED panel signals to decode state l Created by Pittsburgh company: Maya Design

63. Brad MyersCarnegie Mellon63 Driver Information Console l Simulator for the Driver Information Console of Yukon Denali SUV l All the modes and messages l Full emulation of steering wheel buttons l Simulator for the Driver Information Console of Yukon Denali SUV l All the modes and messages l Full emulation of steering wheel buttons

64. Brad MyersCarnegie Mellon64 Simulator for HVAC