1. Automatically Generating High-Quality User Interfaces for Appliances Jeffrey Nichols Doctoral Colloquium Presentation Second International Conference on Pervasive Computing April 18, 2004

2. 1 Problem Appliances are complex and their user interfaces are often hard to use!

3. 2 Thesis Statement A system can automatically generate user interfaces on a wide variety of platforms for remotely controlling appliances where the users performance is better than with the manufacturers interfaces for the appliances.

4. 3 Approach: Use Mobile Devices Common, cheap, and often have ability to communicate 10 million smart phones sold 11.2 million handheld units shipped in 2003 120 million mobile phone subscribers

5. 4 Approach, cont. Specifications Control Feedback Appliances Mobile Devices Use mobile devices to control all appliances in the environment Key Features Two-way communication, Abstract Descriptions, Multiple Platforms, Automatic Interface Generation

6. 5 Automatic Generation of UIs Benefits Multiple modalities (GUI + Speech UI) All interfaces consistent for a user With conventions of the handheld Even from multiple manufacturers Create user interfaces that control multiple connected appliances e.g. a home theater

7. 6 Important Work By Others INCITS/V2 Standardization Effort Alternative Interface Access Protocol (AIAP) [Zimmermann, CHI 2002] Controlling Appliances Xweb [Olsen Jr., UIST 2000] Stanford iRoom, iCrafter [Ponnekanti, Ubicomp 2001] Speakeasy [Newman, UIST 2002] Model-based User Interface Work UIDE [Sukaviriya, CHI 1993], HUMANOID [Szekely, CHI 1993] MASTERMIND [Szekely 1995], TRIDENT [Vanderdonckt 1995] eXtensible Interface Markup Language (XIML) [Puerta, IUI 2002] User Interface Modeling Language (UIML) [Abrams, W3C 1999] None of these systems have been validated to generate high-quality interfaces without designer intervention.

8. 7 Research Approach 1.Hand-design remote control interfaces 2.Determine functional information needed from appliances to design user interfaces 3.Design language for describing appliance functions 4.Build interface generators for multiple platforms 5.Perform user studies to evaluate the interface generators

9. 8 Outline Introduction Personal Universal Controller (PUC) System Handling High-Level Conventions Interface Consistency Generating Interfaces for the Experience Validation Conclusion

10. 9 Architecture PROTOCOL (two-way communication of specification & state) COMMUNICATION (802.11, Bluetooth, Zigbee, etc.) PUC DEVICES (automatic interface generation) PROTOCOL (two-way communication of specification & state) COMMUNICATION (802.11, Bluetooth, Zigbee, etc.) APPLIANCES (Stereo, Alarm Clock, etc.) ADAPTOR (publishes description + appliance state + controls appliance) control device specification & state feedback

11. 10 Architecture PROTOCOL (two-way communication of specification & state) COMMUNICATION (802.11, Bluetooth, Zigbee, etc.) PUC DEVICES (automatic interface generation) PROTOCOL (two-way communication of specification & state) COMMUNICATION (802.11, Bluetooth, Zigbee, etc.) APPLIANCES (Stereo, Alarm Clock, etc.) ADAPTOR (publishes description + appliance state + controls appliance) control device specification & state feedback

12. 11 Architecture PROTOCOL (two-way communication of specification & state) COMMUNICATION (802.11, Bluetooth, Zigbee, etc.) PUC DEVICES (automatic interface generation) PROTOCOL (two-way communication of specification & state) COMMUNICATION (802.11, Bluetooth, Zigbee, etc.) APPLIANCES (Stereo, Alarm Clock, etc.) ADAPTOR (publishes description + appliance state + controls appliance) control device specification & state feedback

13. 12 Architecture PROTOCOL (two-way communication of specification & state) COMMUNICATION (802.11, Bluetooth, Zigbee, etc.) PUC DEVICES (automatic interface generation) PROTOCOL (two-way communication of specification & state) COMMUNICATION (802.11, Bluetooth, Zigbee, etc.) APPLIANCES (Stereo, Alarm Clock, etc.) ADAPTOR (publishes description + appliance state + controls appliance) control device specification & state feedback

14. 13 Architecture PROTOCOL (two-way communication of specification & state) COMMUNICATION (802.11, Bluetooth, Zigbee, etc.) PUC DEVICES (automatic interface generation) PROTOCOL (two-way communication of specification & state) COMMUNICATION (802.11, Bluetooth, Zigbee, etc.) APPLIANCES (Stereo, Alarm Clock, etc.) ADAPTOR (publishes description + appliance state + controls appliance) control device specification & state feedback

15. 14 Language Design Informed by hand-designed interfaces What functional information was needed to create interfaces? Additional Requirements Support complete functionality of appliance No specific layout information Only one way to specify anything Full documentation available at: http://www.cs.cmu.edu/~pebbles/puc/

16. 15 Language Elements Elements State variables & commands Labels Group tree Dependency information Example media player specification Play, stop, pause, next track, previous track Play list

17. 16 Language Elements, cont. State Variables and Commands Represent functions of appliance State variables have types Boolean, Enumeration, Integer, String, etc. Variables sufficient for most functions but not all e.g. seek button on a Radio

18. 17 Language Elements, cont. Label Information One label not suitable everywhere The optimal label length changes with screen size Speech interfaces may benefit from pronunciation and text-to-speech information Label Dictionary A group of semantically similar labels Different lengths Information for different modalities

19. 18 Language Elements, cont. Label Information One label not suitable everywhere The optimal label length changes with screen size Speech interfaces may benefit from pronunciation and text-to-speech information Label Dictionary A group of semantically similar labels Different lengths Information for different modalities

20. 19 Language Elements, cont. Group Tree Specify organization of functions We use n-ary tree with variables or commands at leaves Also used for specifying complex types Lists Unions

21. 20 Language Elements, cont. Group Tree Specify organization of functions We use n-ary tree with variables or commands at leaves Also used for specifying complex types Lists Unions

22. 21 Language Elements, cont. Dependency Information Formulas that specify when a variable or command is active in terms of other state variables Equals, Greater Than, Less Than, Is Defined Linked with logical operators (AND, OR) Allows feedback to user when a function is not available

23. 22 Interface Generators Generators for Two Modalities Graphical Desktop, PocketPC, and Microsoft Smartphone Speech Collaboration with others at Carnegie Mellon Built on top of the PUC framework Implemented using Universal Speech Interface (USI) techniques [Rosenfeld 2001]

24. 23 Controlling Appliances We have built adaptors for many actual appliances Sony Digital Camcorder Windows Media Player Axis UPnP Pan & Tilt Camera Lutron Lighting X10 Lighting Audiophase Shelf Stereo AudioReQuest MP3 player GM Vehicle Information System GM Vehicle Climate Control Written specifications for others Elevator Telephone/Answering Machine GM Navigation System Several Alarm Clocks

25. 24 Outline Introduction Personal Universal Controller (PUC) System Handling High-Level Conventions Interface Consistency Generating Interfaces for the Experience Validation Conclusion

26. 25 High-Level Conventions Problem Human designers rely partly on conventions when making an interface Users expect their appliances to use conventions they know about

27. 26 Smart Templates Need a way to specify high-level information to interface generators Solution Mark groups with tags that identify high- level information media-controls, phone-dialpad, time, date, etc. Restrict the contents of groups so that interface generators can interpret the high-level meaning Standardize the tags and restrictions in advance, so that designers know what interface generators expect

28. 27 Smart Templates, cont. Features Parameterized Specified using primitive elements of specification language Renderable by any interface generator

29. 28 Interfaces with Smart Templates Preliminary Implementation A few templates: image, image-list, media-controls, time-duration

30. 29 Continuing Work Define and implement Smart Templates date, mute, power, time-absolute, volume, etc. Develop more as more appliances are specified Combinations of templates Less implementation cost than a new template e.g. date and time-absolute Use of templates with data already on controller device e.g. calendar and address information Might allow user to enter address from contact list into navigation system

31. 30 Outline Introduction Personal Universal Controller (PUC) System Handling High-Level Conventions Interface Consistency Generating Interfaces for the Experience Validation Conclusion

32. 31 Interface Consistency PUC devices have a unique opportunity to provide consistency for the user Personal device Used for interacting with most appliances Two ways that PUC UIs can be made consistent With other applications on the same device With past interfaces for similar appliances

33. 32 Consistency with Past Interfaces Two sub-problems to address: Similarity Which functions of a new appliance are similar to the functions of interfaces generated in the past? Consistency For similar functions, what rules from previous interfaces can be applied to ensure consistency?

34. 33 Similarity Problem Difficult to conclusively know whether two functions from different appliances are the same Very little semantic information in the specification language Can estimate similarity based on properties of state variables Smart Template Name Group Name Labels Type Improve estimate by looking at relationships between multiple similar variables

35. 34 Similarity & Consistency Problems newprevious newprevious newprevious sparse similarity branch similaritysignificant similarity

36. 35 Outline Introduction Personal Universal Controller (PUC) System Handling High-Level Conventions Interface Consistency Generating Interfaces for the Experience Validation Conclusion

37. 36 The Experience

38. 37 The Experience, cont. (MIT dorm) How can a PUC device provide improved interfaces for connected systems?

39. 38 Improving Interfaces with PUC Generate an interface that aggregates all functions into one set of screens Organized by task instead of appliance Automatically create macros for frequently used functions e.g. Play DVD would: 1. turn on television, DVD player, stereo 2. turn off VCR 3. set the TV and stereo sources to the DVD player 4. instruct the user to insert a DVD (if necessary) 5. play the DVD How can appliance descriptions support these features?

40. 39 Distributed Task Language My plan: Store task information within each appliance description Use an existing task language Combine sub-tasks from each appliance in a connected system to create complete tasks

41. 40 Outline Introduction Personal Universal Controller (PUC) System Handling High-Level Conventions Interface Consistency Generating Interfaces for the Experience Validation Conclusion

42. 41 Two Goals for PUC System Breadth The appliance specification language is capable of describing a wide variety of appliances Quality Interfaces generated for specifications across that range beat the usability of the manufacturers interfaces for the same appliances How do I validate that these goals are met?

43. 42 Evaluation Evaluation Method 1. Develop a list of appliances that are interesting Complexity Unique feature Representative of a class of appliances 2. Specify each appliance 3. Perform a comparative user study on several appliances to test quality

44. 43 Appliance List (so far…comments?) Already Specified & GeneratedPropose to Specify Axis Pan & Tilt CameraGMC Denali Navigation System Audiophase StereoPhone/Answering Machine GMC Denali Driver Information CenterWindows Media Player with playlists GMC Denali Climate Control SystemAutomated Teller Machine (ATM) Sony CamcorderMicrosoft Windows XP Media Center PC Elevator SimulationPhotocopier Lutron Home Lighting SystemTV Tuner Windows Media Player without playlistsPersonal Video Recorder (e.g. TiVO) X10 LightingPowerpoint Alarm Clock Projector Microwave oven

45. 44 Outline Introduction Personal Universal Controller (PUC) System Handling High-Level Conventions Interface Consistency Generating Interfaces for the Experience Validation Conclusion

46. 45 Conclusion Problem Appliances are increasingly complex Appliance user interfaces are often hard to use Solution Move appliance interfaces to a mobile device the user is already carrying Automatically generate interfaces so that: Interfaces are customized for the device and modality that the user prefers Interfaces for similar appliances can be made consistent Interfaces for multiple appliances can be combined into a single interface Validate generated interfaces to prove high quality claim

47. 46 Acknowledgements Thesis Committee Brad A. Myers (chair) Scott Hudson John Zimmerman Dan Olsen Jr. Funding National Science Foundation Microsoft General Motors Intel Pittsburgh Digital Greenhouse Equipment Grants Mitsubishi (MERL) VividLogic Lucent Lutron Lantronix Nokia PUC Project Members Kevin Litwack Thomas K. Harris Michael Higgins Joseph Hughes Roni Rosenfeld Rajesh Seenichamy Pegeen Shen Htet Htet Aung Mathilde Pignol Suporn Pongnumkul Stefanie Shriver Jeffrey Stylos Peter Lucas Collaborators & Friends Naomi Ramos Desney Tan Daniel Avrahami Gaetano Borriello Laura Dabbish Andrew Faulring James Fogarty Krzysztof Gajos Darren Gergle Andy Ko Amy Nichols Mick Nichols Sally Nichols Trevor Pering Fleming Seay Irina Shklovski Roy Want Jake Wobbrock and many others…

48. Thanks for listening! For more information… http://www.cs.cmu.edu/~pebbles/puc/ http://www.cs.cmu.edu/~jeffreyn/

49. Automatically Generating High-Quality User Interfaces for Appliances Jeffrey Nichols Doctoral Colloquium Presentation Second International Conference on Pervasive Computing April 18, 2004

50. 49

51. 50 Problem, cont.

52. 51 Problem, cont. April 29, 1991 CNN – Jan 2004 May 2003

53. 52 Outside the Scope Help systems for generated interfaces Automated trouble-shooting for complex systems Service Discovery Macros and End-User Programming Security Inter-operability with INCITS/V2

54. 53 Outline Introduction Personal Universal Controller (PUC) System Handling High-Level Conventions Interface Consistency Generating Interfaces for the Experience Validation Conclusion

55. 54 Initial Approach What information is needed about the appliance to automatically generate remote control interfaces? Investigate via a design process Create interfaces by hand AIWA Shelf Stereo AT&T Telephone/Answering Machine Improve quality with heuristic analysis and think-aloud studies with several users Compare interfaces with actual appliance interfaces to validate PUC concept Analyze interfaces for functional information

56. 55 Palm Interfaces Initially designed paper-prototype interfaces for Palm telephonestereo

57. 56 Hand-Designed Interfaces Interfaces for Microsofts PocketPC (simulated remote control) telephonestereo Using our interfaces, users were twice as fast and made half as many errors

58. 57 Comparison Study Compared performance of first-time users (not experts) Procedure Each subject worked two sets of tasks both stereo and phone controlled for order and interface Performance Metrics Time to complete all tasks Number of times a user manual was needed Number of missteps ApplianceHand-design Phone Stereo

59. 58 Comparison Study Results Using our interfaces, users were twice as fast and made half as many errors All differences are significant (p < 0.05)

60. 59 Extra Group Tree

61. 60 Graphical Interface Generator Rule-based approach Multiple phases that iteratively transform a specification into a user interface Focuses on panel structure of user interface Small groups of controls have basic layouts Complexity comes from structure of groups Structure can be inferred from dependency info!

62. 61 Generation Process 1.Determine conceptual layout Infer panel structure from dependencies using mutual exclusion property Choose controls (decision tree) Choose row layout (one column, two column, etc.) 2.Allocate space Examine panel contents and choose sizes 3.Instantiate and place controls 4.Fix layout problems

63. 62 Choosing Panel Structure a)b)c) full screen tabbed partial screen

64. 63 Generation Process 1.Determine conceptual layout Infer panel structure from dependencies using mutual exclusion property Choose controls (decision tree) Choose row layout (one column, two column, etc.) 2.Allocate space Examine panel contents and choose sizes 3.Instantiate and place controls 4.Fix layout problems

65. 64 Generation Process 1.Determine conceptual layout Infer panel structure from dependencies using mutual exclusion property Choose controls (decision tree) Choose row layout (one column, two column, etc.) 2.Allocate space Examine panel contents and choose sizes 3.Instantiate and place controls 4.Fix layout problems

66. 65 Generation Process 1.Determine conceptual layout Infer panel structure from dependencies using mutual exclusion property Choose controls (decision tree) Choose row layout (one column, two column, etc.) 2.Allocate space Examine panel contents and choose sizes 3.Instantiate and place controls 4.Fix layout problems

67. 66 Generation Process 1.Determine conceptual layout Infer panel structure from dependencies using mutual exclusion property Choose controls (decision tree) Choose row layout (one column, two column, etc.) 2.Allocate space Examine panel contents and choose sizes 3.Instantiate and place controls 4.Fix layout problems

68. 67 Generation Process 1.Determine conceptual layout Infer panel structure from dependencies using mutual exclusion property Choose controls (decision tree) Choose row layout (one column, two column, etc.) 2.Allocate space Examine panel contents and choose sizes 3.Instantiate and place controls 4.Fix layout problems Without layout fixing rules With layout fixing rules

69. 68 Smart Templates Example

70. 69 Smart Templates, cont. One template can be used across multiple appliances and can be generated on multiple platforms

71. 70 Consistency Problem Apply different consistency techniques depending on similarity Sparse similarity Use similar controls for similar functions e.g. use a scroll bar for new volume, even if this would not be the normal representation (might be a combo box) Branch similarity Use the previous interface structure of branch in the new interface e.g. branch of previous interface was shown in a tabbed panel. In the new interface, separate functions of new interface with tabs and have similar functions in a tabbed panel. Significant similarity Use the group tree from the previously generated interface Interpolate extra functions into the tree

72. 71 The Experience, cont. How can a PUC device provide improved interfaces for these systems? Users often think of multiple connected appliances as one system Home theater CMU lecture room system others…

73. 72 Proposed Schedule March 2004 September 1 2 3 4 5 6 7 8 1Interface consistency 2Smart Templates 3Preliminary user studies with GM vehicle interfaces 4Improve robustness and quality of interface generators on all platforms 5Develop distributed task modeling language and multi-appliance user interface generation 6Implement appliances for final user study 7Conduct final user study and iterate UI generation rules, if necessary 8Write dissertation June DecemberMarch 2005 SeptemberDecember 2005

74. 73 Proposed Contributions An abstract appliance modeling language for describing the functionality of appliances Algorithms for automatically generating high quality interfaces from that language The Smart Templates technique for incorporating high-level conventions Algorithms for ensuring consistency across generated interfaces and…

75. 74 Proposed Contributions, cont. A distributed task-modeling language for describing the sub- tasks specific to an appliance in a multi-appliance system Algorithms for automatically generating a single user interface for multiple appliances using distributed task information Interface generation software on multiple platforms that is shown by user testing to be better than manufacturers interfaces for the same appliances