Automatically Generating High-Quality User Interfaces for Appliances Jeffrey Nichols Doctoral Colloquium Presentation Second International Conference on Pervasive Computing April 18, 2004

1 Problem Appliances are complex and their user interfaces are often hard to use!

2 Thesis Statement A system can automatically generate user interfaces on a wide variety of platforms for remotely controlling appliances where the users performance is better than with the manufacturers interfaces for the appliances.

3 Approach: Use Mobile Devices Common, cheap, and often have ability to communicate 10 million smart phones sold 11.2 million handheld units shipped in million mobile phone subscribers

4 Approach, cont. Specifications Control Feedback Appliances Mobile Devices Use mobile devices to control all appliances in the environment Key Features Two-way communication, Abstract Descriptions, Multiple Platforms, Automatic Interface Generation

5 Automatic Generation of UIs Benefits Multiple modalities (GUI + Speech UI) All interfaces consistent for a user With conventions of the handheld Even from multiple manufacturers Create user interfaces that control multiple connected appliances e.g. a home theater

6 Important Work By Others INCITS/V2 Standardization Effort Alternative Interface Access Protocol (AIAP) [Zimmermann, CHI 2002] Controlling Appliances Xweb [Olsen Jr., UIST 2000] Stanford iRoom, iCrafter [Ponnekanti, Ubicomp 2001] Speakeasy [Newman, UIST 2002] Model-based User Interface Work UIDE [Sukaviriya, CHI 1993], HUMANOID [Szekely, CHI 1993] MASTERMIND [Szekely 1995], TRIDENT [Vanderdonckt 1995] eXtensible Interface Markup Language (XIML) [Puerta, IUI 2002] User Interface Modeling Language (UIML) [Abrams, W3C 1999] None of these systems have been validated to generate high-quality interfaces without designer intervention.

7 Research Approach 1.Hand-design remote control interfaces 2.Determine functional information needed from appliances to design user interfaces 3.Design language for describing appliance functions 4.Build interface generators for multiple platforms 5.Perform user studies to evaluate the interface generators

8 Outline Introduction Personal Universal Controller (PUC) System Handling High-Level Conventions Interface Consistency Generating Interfaces for the Experience Validation Conclusion

9 Architecture PROTOCOL (two-way communication of specification & state) COMMUNICATION (802.11, Bluetooth, Zigbee, etc.) PUC DEVICES (automatic interface generation) PROTOCOL (two-way communication of specification & state) COMMUNICATION (802.11, Bluetooth, Zigbee, etc.) APPLIANCES (Stereo, Alarm Clock, etc.) ADAPTOR (publishes description + appliance state + controls appliance) control device specification & state feedback

10 Architecture PROTOCOL (two-way communication of specification & state) COMMUNICATION (802.11, Bluetooth, Zigbee, etc.) PUC DEVICES (automatic interface generation) PROTOCOL (two-way communication of specification & state) COMMUNICATION (802.11, Bluetooth, Zigbee, etc.) APPLIANCES (Stereo, Alarm Clock, etc.) ADAPTOR (publishes description + appliance state + controls appliance) control device specification & state feedback

11 Architecture PROTOCOL (two-way communication of specification & state) COMMUNICATION (802.11, Bluetooth, Zigbee, etc.) PUC DEVICES (automatic interface generation) PROTOCOL (two-way communication of specification & state) COMMUNICATION (802.11, Bluetooth, Zigbee, etc.) APPLIANCES (Stereo, Alarm Clock, etc.) ADAPTOR (publishes description + appliance state + controls appliance) control device specification & state feedback

12 Architecture PROTOCOL (two-way communication of specification & state) COMMUNICATION (802.11, Bluetooth, Zigbee, etc.) PUC DEVICES (automatic interface generation) PROTOCOL (two-way communication of specification & state) COMMUNICATION (802.11, Bluetooth, Zigbee, etc.) APPLIANCES (Stereo, Alarm Clock, etc.) ADAPTOR (publishes description + appliance state + controls appliance) control device specification & state feedback

13 Architecture PROTOCOL (two-way communication of specification & state) COMMUNICATION (802.11, Bluetooth, Zigbee, etc.) PUC DEVICES (automatic interface generation) PROTOCOL (two-way communication of specification & state) COMMUNICATION (802.11, Bluetooth, Zigbee, etc.) APPLIANCES (Stereo, Alarm Clock, etc.) ADAPTOR (publishes description + appliance state + controls appliance) control device specification & state feedback

14 Language Design Informed by hand-designed interfaces What functional information was needed to create interfaces? Additional Requirements Support complete functionality of appliance No specific layout information Only one way to specify anything Full documentation available at:

15 Language Elements Elements State variables & commands Labels Group tree Dependency information Example media player specification Play, stop, pause, next track, previous track Play list

16 Language Elements, cont. State Variables and Commands Represent functions of appliance State variables have types Boolean, Enumeration, Integer, String, etc. Variables sufficient for most functions but not all e.g. seek button on a Radio

17 Language Elements, cont. Label Information One label not suitable everywhere The optimal label length changes with screen size Speech interfaces may benefit from pronunciation and text-to-speech information Label Dictionary A group of semantically similar labels Different lengths Information for different modalities

18 Language Elements, cont. Label Information One label not suitable everywhere The optimal label length changes with screen size Speech interfaces may benefit from pronunciation and text-to-speech information Label Dictionary A group of semantically similar labels Different lengths Information for different modalities

19 Language Elements, cont. Group Tree Specify organization of functions We use n-ary tree with variables or commands at leaves Also used for specifying complex types Lists Unions

20 Language Elements, cont. Group Tree Specify organization of functions We use n-ary tree with variables or commands at leaves Also used for specifying complex types Lists Unions

21 Language Elements, cont. Dependency Information Formulas that specify when a variable or command is active in terms of other state variables Equals, Greater Than, Less Than, Is Defined Linked with logical operators (AND, OR) Allows feedback to user when a function is not available

22 Interface Generators Generators for Two Modalities Graphical Desktop, PocketPC, and Microsoft Smartphone Speech Collaboration with others at Carnegie Mellon Built on top of the PUC framework Implemented using Universal Speech Interface (USI) techniques [Rosenfeld 2001]

23 Controlling Appliances We have built adaptors for many actual appliances Sony Digital Camcorder Windows Media Player Axis UPnP Pan & Tilt Camera Lutron Lighting X10 Lighting Audiophase Shelf Stereo AudioReQuest MP3 player GM Vehicle Information System GM Vehicle Climate Control Written specifications for others Elevator Telephone/Answering Machine GM Navigation System Several Alarm Clocks

24 Outline Introduction Personal Universal Controller (PUC) System Handling High-Level Conventions Interface Consistency Generating Interfaces for the Experience Validation Conclusion

25 High-Level Conventions Problem Human designers rely partly on conventions when making an interface Users expect their appliances to use conventions they know about

26 Smart Templates Need a way to specify high-level information to interface generators Solution Mark groups with tags that identify high- level information media-controls, phone-dialpad, time, date, etc. Restrict the contents of groups so that interface generators can interpret the high-level meaning Standardize the tags and restrictions in advance, so that designers know what interface generators expect

27 Smart Templates, cont. Features Parameterized Specified using primitive elements of specification language Renderable by any interface generator

28 Interfaces with Smart Templates Preliminary Implementation A few templates: image, image-list, media-controls, time-duration

29 Continuing Work Define and implement Smart Templates date, mute, power, time-absolute, volume, etc. Develop more as more appliances are specified Combinations of templates Less implementation cost than a new template e.g. date and time-absolute Use of templates with data already on controller device e.g. calendar and address information Might allow user to enter address from contact list into navigation system

30 Outline Introduction Personal Universal Controller (PUC) System Handling High-Level Conventions Interface Consistency Generating Interfaces for the Experience Validation Conclusion

31 Interface Consistency PUC devices have a unique opportunity to provide consistency for the user Personal device Used for interacting with most appliances Two ways that PUC UIs can be made consistent With other applications on the same device With past interfaces for similar appliances

32 Consistency with Past Interfaces Two sub-problems to address: Similarity Which functions of a new appliance are similar to the functions of interfaces generated in the past? Consistency For similar functions, what rules from previous interfaces can be applied to ensure consistency?

33 Similarity Problem Difficult to conclusively know whether two functions from different appliances are the same Very little semantic information in the specification language Can estimate similarity based on properties of state variables Smart Template Name Group Name Labels Type Improve estimate by looking at relationships between multiple similar variables

34 Similarity & Consistency Problems newprevious newprevious newprevious sparse similarity branch similaritysignificant similarity

35 Outline Introduction Personal Universal Controller (PUC) System Handling High-Level Conventions Interface Consistency Generating Interfaces for the Experience Validation Conclusion

36 The Experience

37 The Experience, cont. (MIT dorm) How can a PUC device provide improved interfaces for connected systems?

38 Improving Interfaces with PUC Generate an interface that aggregates all functions into one set of screens Organized by task instead of appliance Automatically create macros for frequently used functions e.g. Play DVD would: 1. turn on television, DVD player, stereo 2. turn off VCR 3. set the TV and stereo sources to the DVD player 4. instruct the user to insert a DVD (if necessary) 5. play the DVD How can appliance descriptions support these features?

39 Distributed Task Language My plan: Store task information within each appliance description Use an existing task language Combine sub-tasks from each appliance in a connected system to create complete tasks

40 Outline Introduction Personal Universal Controller (PUC) System Handling High-Level Conventions Interface Consistency Generating Interfaces for the Experience Validation Conclusion

41 Two Goals for PUC System Breadth The appliance specification language is capable of describing a wide variety of appliances Quality Interfaces generated for specifications across that range beat the usability of the manufacturers interfaces for the same appliances How do I validate that these goals are met?

42 Evaluation Evaluation Method 1. Develop a list of appliances that are interesting Complexity Unique feature Representative of a class of appliances 2. Specify each appliance 3. Perform a comparative user study on several appliances to test quality

43 Appliance List (so far…comments?) Already Specified & GeneratedPropose to Specify Axis Pan & Tilt CameraGMC Denali Navigation System Audiophase StereoPhone/Answering Machine GMC Denali Driver Information CenterWindows Media Player with playlists GMC Denali Climate Control SystemAutomated Teller Machine (ATM) Sony CamcorderMicrosoft Windows XP Media Center PC Elevator SimulationPhotocopier Lutron Home Lighting SystemTV Tuner Windows Media Player without playlistsPersonal Video Recorder (e.g. TiVO) X10 LightingPowerpoint Alarm Clock Projector Microwave oven

44 Outline Introduction Personal Universal Controller (PUC) System Handling High-Level Conventions Interface Consistency Generating Interfaces for the Experience Validation Conclusion

45 Conclusion Problem Appliances are increasingly complex Appliance user interfaces are often hard to use Solution Move appliance interfaces to a mobile device the user is already carrying Automatically generate interfaces so that: Interfaces are customized for the device and modality that the user prefers Interfaces for similar appliances can be made consistent Interfaces for multiple appliances can be combined into a single interface Validate generated interfaces to prove high quality claim

46 Acknowledgements Thesis Committee Brad A. Myers (chair) Scott Hudson John Zimmerman Dan Olsen Jr. Funding National Science Foundation Microsoft General Motors Intel Pittsburgh Digital Greenhouse Equipment Grants Mitsubishi (MERL) VividLogic Lucent Lutron Lantronix Nokia PUC Project Members Kevin Litwack Thomas K. Harris Michael Higgins Joseph Hughes Roni Rosenfeld Rajesh Seenichamy Pegeen Shen Htet Htet Aung Mathilde Pignol Suporn Pongnumkul Stefanie Shriver Jeffrey Stylos Peter Lucas Collaborators & Friends Naomi Ramos Desney Tan Daniel Avrahami Gaetano Borriello Laura Dabbish Andrew Faulring James Fogarty Krzysztof Gajos Darren Gergle Andy Ko Amy Nichols Mick Nichols Sally Nichols Trevor Pering Fleming Seay Irina Shklovski Roy Want Jake Wobbrock and many others…

Thanks for listening! For more information…

Automatically Generating High-Quality User Interfaces for Appliances Jeffrey Nichols Doctoral Colloquium Presentation Second International Conference on Pervasive Computing April 18, 2004

49

50 Problem, cont.

51 Problem, cont. April 29, 1991 CNN – Jan 2004 May 2003

52 Outside the Scope Help systems for generated interfaces Automated trouble-shooting for complex systems Service Discovery Macros and End-User Programming Security Inter-operability with INCITS/V2

53 Outline Introduction Personal Universal Controller (PUC) System Handling High-Level Conventions Interface Consistency Generating Interfaces for the Experience Validation Conclusion

54 Initial Approach What information is needed about the appliance to automatically generate remote control interfaces? Investigate via a design process Create interfaces by hand AIWA Shelf Stereo AT&T Telephone/Answering Machine Improve quality with heuristic analysis and think-aloud studies with several users Compare interfaces with actual appliance interfaces to validate PUC concept Analyze interfaces for functional information

55 Palm Interfaces Initially designed paper-prototype interfaces for Palm telephonestereo

56 Hand-Designed Interfaces Interfaces for Microsofts PocketPC (simulated remote control) telephonestereo Using our interfaces, users were twice as fast and made half as many errors

57 Comparison Study Compared performance of first-time users (not experts) Procedure Each subject worked two sets of tasks both stereo and phone controlled for order and interface Performance Metrics Time to complete all tasks Number of times a user manual was needed Number of missteps ApplianceHand-design Phone Stereo

58 Comparison Study Results Using our interfaces, users were twice as fast and made half as many errors All differences are significant (p < 0.05)

59 Extra Group Tree

60 Graphical Interface Generator Rule-based approach Multiple phases that iteratively transform a specification into a user interface Focuses on panel structure of user interface Small groups of controls have basic layouts Complexity comes from structure of groups Structure can be inferred from dependency info!

61 Generation Process 1.Determine conceptual layout Infer panel structure from dependencies using mutual exclusion property Choose controls (decision tree) Choose row layout (one column, two column, etc.) 2.Allocate space Examine panel contents and choose sizes 3.Instantiate and place controls 4.Fix layout problems

62 Choosing Panel Structure a)b)c) full screen tabbed partial screen

63 Generation Process 1.Determine conceptual layout Infer panel structure from dependencies using mutual exclusion property Choose controls (decision tree) Choose row layout (one column, two column, etc.) 2.Allocate space Examine panel contents and choose sizes 3.Instantiate and place controls 4.Fix layout problems

64 Generation Process 1.Determine conceptual layout Infer panel structure from dependencies using mutual exclusion property Choose controls (decision tree) Choose row layout (one column, two column, etc.) 2.Allocate space Examine panel contents and choose sizes 3.Instantiate and place controls 4.Fix layout problems

65 Generation Process 1.Determine conceptual layout Infer panel structure from dependencies using mutual exclusion property Choose controls (decision tree) Choose row layout (one column, two column, etc.) 2.Allocate space Examine panel contents and choose sizes 3.Instantiate and place controls 4.Fix layout problems

66 Generation Process 1.Determine conceptual layout Infer panel structure from dependencies using mutual exclusion property Choose controls (decision tree) Choose row layout (one column, two column, etc.) 2.Allocate space Examine panel contents and choose sizes 3.Instantiate and place controls 4.Fix layout problems

67 Generation Process 1.Determine conceptual layout Infer panel structure from dependencies using mutual exclusion property Choose controls (decision tree) Choose row layout (one column, two column, etc.) 2.Allocate space Examine panel contents and choose sizes 3.Instantiate and place controls 4.Fix layout problems Without layout fixing rules With layout fixing rules

68 Smart Templates Example

69 Smart Templates, cont. One template can be used across multiple appliances and can be generated on multiple platforms

70 Consistency Problem Apply different consistency techniques depending on similarity Sparse similarity Use similar controls for similar functions e.g. use a scroll bar for new volume, even if this would not be the normal representation (might be a combo box) Branch similarity Use the previous interface structure of branch in the new interface e.g. branch of previous interface was shown in a tabbed panel. In the new interface, separate functions of new interface with tabs and have similar functions in a tabbed panel. Significant similarity Use the group tree from the previously generated interface Interpolate extra functions into the tree

71 The Experience, cont. How can a PUC device provide improved interfaces for these systems? Users often think of multiple connected appliances as one system Home theater CMU lecture room system others…

72 Proposed Schedule March 2004 September Interface consistency 2Smart Templates 3Preliminary user studies with GM vehicle interfaces 4Improve robustness and quality of interface generators on all platforms 5Develop distributed task modeling language and multi-appliance user interface generation 6Implement appliances for final user study 7Conduct final user study and iterate UI generation rules, if necessary 8Write dissertation June DecemberMarch 2005 SeptemberDecember 2005

73 Proposed Contributions An abstract appliance modeling language for describing the functionality of appliances Algorithms for automatically generating high quality interfaces from that language The Smart Templates technique for incorporating high-level conventions Algorithms for ensuring consistency across generated interfaces and…

74 Proposed Contributions, cont. A distributed task-modeling language for describing the sub- tasks specific to an appliance in a multi-appliance system Algorithms for automatically generating a single user interface for multiple appliances using distributed task information Interface generation software on multiple platforms that is shown by user testing to be better than manufacturers interfaces for the same appliances