1. Jeffrey Nichols Conference on Human Factors in Computing Systems (CHI) April 8, 2003 Slide #0 Jeffrey Nichols and Brad A. Myers Carnegie Mellon University April 8, 2003 Personal Universal Controllers: Controlling Complex Appliances with GUIs and Speech CHI 2003 Conference on Human Factors in Computing Systems Ft. Lauderdale, Florida

2. Jeffrey Nichols Conference on Human Factors in Computing Systems (CHI) April 8, 2003 Slide #1 The Problem Appliances are too complex

3. Jeffrey Nichols Conference on Human Factors in Computing Systems (CHI) April 8, 2003 Slide #2 The Problem, cont. Each complex appliance has its own idiosyncratic interface! Home and Car Stereos VCRs & Camcorders Car Navigation Systems Answering Machines … Increasingly Computerized Low Usability

4. Jeffrey Nichols Conference on Human Factors in Computing Systems (CHI) April 8, 2003 Slide #3 Our Solution Separate the interface from the appliance! Handheld becomes personal universal controller (PUC) Key Features User interface-independent appliance specification Automatic generation of GUI and speech interfaces Specifications Control Feedback

5. Jeffrey Nichols Conference on Human Factors in Computing Systems (CHI) April 8, 2003 Slide #4 Automatic Generation of UIs Benefits All interfaces consistent for the user With conventions of handheld Other applications and UI guidelines Even from multiple manufacturers Addresses idiosyncracy problem! Multiple modalities (GUI + Speech UI) Can take into account user preferences Will work on special purpose devices (for disabled)

6. Jeffrey Nichols Conference on Human Factors in Computing Systems (CHI) April 8, 2003 Slide #5 Overview Appliances Controller Devices

7. Jeffrey Nichols Conference on Human Factors in Computing Systems (CHI) April 8, 2003 Slide #6 Demonstration #1 Sony Camcorder PocketPC, Smartphone, Speech, Tablet PC

8. Jeffrey Nichols Conference on Human Factors in Computing Systems (CHI) April 8, 2003 Slide #7 - Appliance Adaptors- Comm. Protocol- Interface GeneratorsArchitecture XML-based

9. Jeffrey Nichols Conference on Human Factors in Computing Systems (CHI) April 8, 2003 Slide #8 XML-based Specification Language Describes appliance with these features: Functions of Device State Variables and Commands Labeling Multiple labels are necessary Grouping Hierarchical groups Dependency Information For enabling and structure

10. Jeffrey Nichols Conference on Human Factors in Computing Systems (CHI) April 8, 2003 Slide #9 Demonstration #2 Windows Media Player PocketPC, Smartphone

11. Jeffrey Nichols Conference on Human Factors in Computing Systems (CHI) April 8, 2003 Slide #10 Demonstration #3 GMC Information System / Climate ControlInformation SystemClimate Control PocketPC

12. Jeffrey Nichols Conference on Human Factors in Computing Systems (CHI) April 8, 2003 Slide #11 Important Work By Others INCITS V2 Standardization Effort SIG at CHI 2003 Toward a Unified Universal Remote Console Standard Thursday 9:00-10:30, in Room 122 Xweb (now ICE) [Olsen Jr., UIST 2000] Stanford iRoom, iCrafter [Ponnekanti, Ubicomp 2001] Speakeasy [Newman, UIST 2002]

13. Jeffrey Nichols Conference on Human Factors in Computing Systems (CHI) April 8, 2003 Slide #12 Future Work Improve quality of generated user interfaces More and better rules Specification and generation of more appliances answering machine, navigation system Consistent interface generation New interfaces modeled on past interfaces Generating interfaces for the experience One interface generated for multiple connected appliances General purpose appliance adaptor for HAVi and UPnP protocols

14. Jeffrey Nichols Conference on Human Factors in Computing Systems (CHI) April 8, 2003 Slide #13 Acknowledgements Funding National Science Foundation Microsoft General Motors Pittsburgh Digital Greenhouse Equipment Grants Mitsubishi (MERL) VividLogic Symbol Technologies Hewlett-Packard Lucent PUC Project Members Brad A. Myers Thomas K. Harris Roni Rosenfeld Michael Higgins Joseph Hughes Kevin Litwack Rajesh Seenichamy Mathilde Pignol Stefanie Shriver Jeffrey Stylos Peter Lucas

15. Jeffrey Nichols Conference on Human Factors in Computing Systems (CHI) April 8, 2003 Slide #14 Conference on Human Factors in Computing Systems (CHI) April 8, 2003 Thanks! http://www.cs.cmu.edu/~pebbles/puc/ http://www.cs.cmu.edu/~jeffreyn/

16. Jeffrey Nichols Conference on Human Factors in Computing Systems (CHI) April 8, 2003 Slide #15

17. Jeffrey Nichols Conference on Human Factors in Computing Systems (CHI) April 8, 2003 Slide #16 Rise of Mobile Devices Mobile devices are becoming cheaper and more common

18. Jeffrey Nichols Conference on Human Factors in Computing Systems (CHI) April 8, 2003 Slide #17 Demonstration #4? Two-way X10 Lighting PocketPC, Smartphone

19. Jeffrey Nichols Conference on Human Factors in Computing Systems (CHI) April 8, 2003 Slide #18 X10 Demo #1

20. Jeffrey Nichols Conference on Human Factors in Computing Systems (CHI) April 8, 2003 Slide #19 X10 Demo #2 Back

21. Jeffrey Nichols Conference on Human Factors in Computing Systems (CHI) April 8, 2003 Slide #20 DV Camera #1

22. Jeffrey Nichols Conference on Human Factors in Computing Systems (CHI) April 8, 2003 Slide #21 DV Camera #2 Back

23. Jeffrey Nichols Conference on Human Factors in Computing Systems (CHI) April 8, 2003 Slide #22 Windows Media Player Back

24. Jeffrey Nichols Conference on Human Factors in Computing Systems (CHI) April 8, 2003 Slide #23 GM Driver Information System Back

25. Jeffrey Nichols Conference on Human Factors in Computing Systems (CHI) April 8, 2003 Slide #24 GM Climate Control System Back