Creating a Lightweight User Interface Description Language An Overview and Analysis of the Personal Universal Controller Project Jeffrey Nichols IBM Research – Almaden ToCHI Track – CHI 2010 April 15, 2010

1 Motivation April 29, 1991 CNN – Jan 2004 Economist – Oct 2004 Nov 27, 2006

2 Personal Universal Controller Personal Mobile Device Abstract Specification Control State Feedback Automatically Generated Interface PUC [Nichols, UIST 02]

3 Automatic Generation of UIs Benefits Generated interfaces can be better than today’s appliance interfaces As shown by user study >

4 Automatic Generation of UIs Benefits Generated interfaces can be better than today’s appliance interfaces As shown by user study All interfaces personally consistent for each user Across functionally similar appliances Original Uniform [Nichols, CHI 06] Consistent

5 Automatic Generation of UIs Benefits Generated interfaces can be better than today’s appliance interfaces As shown by user study All interfaces personally consistent for each user Across functionally similar appliances Aggregate interfaces for systems of multiple appliances Functions from multiple appliances combined into task-based interfaces Aggregate Interface Huddle [Nichols, UIST 06] Flow-Based Interface

6 Today The PUC Specification Language Informative enough to allow generation of high-quality interfaces Easy to learn and use Descriptive enough for a wide- range of complex appliances Supports inclusion of high- level design conventions Formal enough to allow algorithms to reason about appliance behavior Personal Mobile Device Abstract Specification Appliance

7 Design Principles Descriptive enough for any appliance Sufficient detail to generate a high-quality interface No specific layout information Support generation for different devices and modalities Short and concise Only one way to specify anything

8 Design Process Started by exploring appliance user interfaces Created handheld interfaces for two appliances Found these interfaces to be better than manufacturers’ interfaces Analyzed the interfaces for functional information Designed appliance specification language Built interface generators Iterated on the specification design Wrote additional specifications for increasingly complex appliances Modified language as necessary to support additional complexity

9 Specification Language Language Functional information XML-based Elements State variables & commands Group tree Multiple labels per object Dependency information Full documentation available at:

10 Dependency Information Useful for determining structure of user interface Some groups are never available simultaneously Nondependent controls can be split into more accessible side-panels Mutually exclusive groups can be placed on overlapping panels Improves usability and robustness of specification language Reduces emphasis on group tree for structuring interfaces Dependency information is objective

11 Smart Templates Designed Smart Templates technique [Nichols, IUI 04] Allows generated interfaces to include high- level design conventions Flexible parameterization specified using primitive elements of specification language Without Smart Templates With Smart Templates

12 Evaluation Dimensions Completeness What appliances can we write specifications for? What specifications can we generate interfaces for? Usability of Specification Language Usability of Generated Interfaces How good are the generated interfaces?

13 Completeness We have written 33 complete specifications for a wide variety of appliances Home theater appliances InFocus Television, VCRs, DVD players, Sony receiver, Audiophase shelf stereo, Sony camcorder, Windows Media Player Copiers, All-in-one printers Lighting controls (X10, Lutron, UPnP) AT&T telephone/answering machine Alarm clocks Elevator Desktop computer applications PowerPoint, Photo browser, To-do list In-vehicle systems for GMC Yukon Denali Driver Information Console, Climate Controls, Navigation System PUC can generate interfaces for all of these specifications

14 Completeness: Navigation Interface

15 Completeness: Printer Interfaces HP printerCanon printer

16 Usability of Specification Study of Authoring Appliance Specifications Three subjects wrote specifications for a low-end Panasonic VCR All subjects instructed to: Include all functions Be faithful to the existing appliance design Results: Subjects were able to learn the language in a 1.5 hour session About 6 hours was required to create the VCR specification Specifications had some problems, but subjects were not allowed to test their designs Low End: Panasonic VCR What consistency issues arise in our domain?

17 Usability of Generated Interfaces Evaluation of Generated Interfaces Users perform tasks with two all-in-one printer interfaces (HP and Canon) Two comparisons Physical interface to PUC interface Without consistency and with consistency 48 participants (28 male, 20 female) Divided into 6 groups, 8 per group Recruited through CBDR web page Metrics Completion time Failures HP printerCanon printer

18 Usability Results #1 - HP#2 - Canon Built-In Physical AutoGen PUC Consistent PUCUniform #1 - Canon#2 - HP Built-In Physical AutoGen PUC Consistent PUCUniform * = p < 0.05 † = p < 0.1 ** Failures per subject also significantly less for PUC than Physical (Fisher’s Exact Test, p < 0.05) HP: Built-In vs PUC Canon: 2.0 Built-In vs PUC ********* ††

19 Retrospective What Worked: Short and concise principle Use of workarounds for particularly difficult functions (e.g. Smart Templates) What Didn’t Work: “Only one way to specify anything” principle Some features turned out not to be useful Versioning

20 Takeaways Iterative design is important, even for modeling languages Need to choose a work balance between the specification author and interface generator Balance between subjective and objective information High-quality user interfaces can be automatically generated

21 Acknowledgements Thesis Committee Brad A. Myers (chair) Scott Hudson John Zimmerman Dan Olsen Jr. Funding National Science Foundation Microsoft General Motors Intel Pittsburgh Digital Greenhouse Equipment Grants Mitsubishi (MERL) VividLogic Lucent Lutron Lantronix Nokia PUC Project Members Brandon Rothrock Duen Horng Chau Kevin Litwack Thomas K. Harris Michael Higgins Joseph Hughes Roni Rosenfeld Rajesh Seenichamy Pegeen Shen Htet Htet Aung Mathilde Pignol Suporn Pongnumkul Stefanie Shriver Jeffrey Stylos Peter Lucas Collaborators & Friends Naomi Ramos Desney Tan Daniel Avrahami Gaetano Borriello Laura Dabbish Andrew Faulring James Fogarty Krzysztof Gajos Darren Gergle Andy Ko Amy Nichols Mick Nichols Sally Nichols Trevor Pering Fleming Seay Irina Shklovski Rachel Steigerwalt Roy Want Jake Wobbrock and many others…

Thanks for listening! For more information…

23

ToCHI Papers in Process Number of Submissions Months since Submission

Comparison to other Transactions Total Articles Total Downloads Total Cited By Ave. Downloads per Article Ave. Cites per Article ACM Letters on Programming Languages and Systems (LOPLAS) ACM Transactions on Accessible Computing (TACCESS) ACM Transactions on Algorithms (TALG) ACM Transactions on Applied Perception (TAP) ACM Transactions on Architecture and Code Optimization (TACO) ACM Transactions on Asian Language Information Processing (TALIP) ACM Transactions on Autonomous and Adaptive Systems (TAAS) ACM Transactions on Computation Theory (TOCT) ACM Transactions on Computational Logic (TOCL) ACM Transactions on Computer Systems (TOCS) ACM Transactions on Computer-Human Interaction (TOCHI) ACM Transactions on Computing Education (TOCE) ACM Transactions on Database Systems (TODS) ACM Transactions on Design Automation of Electronic Systems (TODAES) ACM Transactions on Embedded Computing Systems (TECS) ACM Transactions on Graphics (TOG) ACM Transactions on Information Systems (TOIS) ACM Transactions on Information and System Security (TISSEC) ACM Transactions on Internet Technology (TOIT) ACM Transactions on Knowledge Discovery from Data (TKDD) ACM Transactions on Mathematical Software (TOMS) ACM Transactions on Modeling and Computer Simulation (TOMACS) ACM Transactions on Multimedia Computing, Communications, and Applications (TOMC ACM Transactions on Programming Languages and Systems (TOPLAS) ACM Transactions on Reconfigurable Technology and Systems (TRETS) ACM Transactions on Sensor Networks (TOSN) ACM Transactions on Software Engineering and Methodology (TOSEM) ACM Transactions on Speech and Language Processing (TSLP) ACM Transactions on Storage (TOS) ACM Transactions on the Web (TWEB) #2 in Ave. Downloads per Article #2 in Ave. Citations per Article