1. Jeffrey Nichols 0 International Workshop on Smart Appliances and Wearable Computing May 19, 2003 Studying The Use of Handhelds To Control Smart Appliances Jeffrey Nichols Carnegie Mellon University May 19, 2003

2. Jeffrey Nichols 1 International Workshop on Smart Appliances and Wearable Computing May 19, 2003 The Problem Appliances are too complex

3. Jeffrey Nichols 2 International Workshop on Smart Appliances and Wearable Computing May 19, 2003 The Problem, cont.  Each complex appliance has its own idiosyncratic interface! Home and Car Stereos Car Navigation Systems Answering Machines …  Increasingly Computerized  Low Usability

4. Jeffrey Nichols 3 International Workshop on Smart Appliances and Wearable Computing May 19, 2003 Our Solution Separate the interface from the appliance! Key Features  User interface-independent appliance specification  Automatic generation of GUI and speech interfaces Specifications Control Feedback

5. Jeffrey Nichols 4 International Workshop on Smart Appliances and Wearable Computing May 19, 2003 Benefits of Our Approach  Handheld has richer interface technology than appliance can afford  Color LCD screen, touch screen, text entry technology  More effort can be put into interface design technology  Appliance manufacturer’s must weigh trade-offs between usability, cost, time-to-market, etc.  Two-way communication channel  Better feedback can be provided to the user regarding the appliance’s state.

6. Jeffrey Nichols 5 International Workshop on Smart Appliances and Wearable Computing May 19, 2003 Automatic Generation of UIs Benefits  All interfaces consistent for the user  With conventions of handheld Other applications and UI guidelines  Even from multiple manufacturers Addresses idiosyncracy problem!  Multiple modalities (GUI + Speech UI)  Can take into account user preferences  Will work on special purpose devices (for disabled)

7. Jeffrey Nichols 6 International Workshop on Smart Appliances and Wearable Computing May 19, 2003 Outline  A First Step  User Studies  Current Work

8. Jeffrey Nichols 7 International Workshop on Smart Appliances and Wearable Computing May 19, 2003 A First Step  Build Reference Interfaces  Remote control interfaces for various appliances that we design manually.  Verify that better interfaces can be created on a handheld  Used for understanding what functional knowledge is necessary to make a good interface.

9. Jeffrey Nichols 8 International Workshop on Smart Appliances and Wearable Computing May 19, 2003 Reference Interfaces  Interfaces were hand-designed for two appliances and two handhelds  Appliances  AIWA CX-NMT70 Shelf Stereo  AT&T 1825 Telephone/Answering Machine  Handhelds  Palm  Microsoft PocketPC

10. Jeffrey Nichols 9 International Workshop on Smart Appliances and Wearable Computing May 19, 2003 Palm Interfaces We initially designed paper-prototype interfaces for Palm telephonestereo

11. Jeffrey Nichols 10 International Workshop on Smart Appliances and Wearable Computing May 19, 2003 PocketPC Interfaces We later implemented interfaces for Microsoft’s PocketPC (simulated remote control). telephonestereo

12. Jeffrey Nichols 11 International Workshop on Smart Appliances and Wearable Computing May 19, 2003 Interface Quality?  We iteratively improved the interfaces using heuristic analysis techniques.  We conducted a think-aloud study with several Carnegie Mellon students to find problems in the interfaces.  Lastly, we conducted a user study that compared our reference interfaces with the manufacturer’s interfaces.

13. Jeffrey Nichols 12 International Workshop on Smart Appliances and Wearable Computing May 19, 2003 Outline  A First Step  User Studies  Current Work

14. Jeffrey Nichols 13 International Workshop on Smart Appliances and Wearable Computing May 19, 2003 User Studies  Two Studies  Study #1: Paper-Prototype Palm vs. Actual Appliance  Study #2: Functional PocketPC vs. Actual Appliance

15. Jeffrey Nichols 14 International Workshop on Smart Appliances and Wearable Computing May 19, 2003 User Studies, cont.  Procedure  We did a between-subjects study.  Each subject worked on two sets of tasks.  In order to minimize subjects, each worked on both the stereo and the phone.  We controlled for order and interface type.

16. Jeffrey Nichols 15 International Workshop on Smart Appliances and Wearable Computing May 19, 2003 Evaluation of Task Performance  Three Metrics:  Time to complete all tasks  Number of times help was requested  How often did the subject need the manual or online help?  Number of missteps  Misstep = the pressing of a button that does not advance the progress on the current task  No missteps were counted after the user requested help.

17. Jeffrey Nichols 16 International Workshop on Smart Appliances and Wearable Computing May 19, 2003 User Study #1: PalmOS  Compared paper prototype interfaces with the interfaces of the actual appliances  Experimenter changed paper as subjects tapped  Control of stereo and phone simulated verbally  When the stereo started playing music, the experimenter said “you now hear music from the stereo”

18. Jeffrey Nichols 17 International Workshop on Smart Appliances and Wearable Computing May 19, 2003 User Study #1, cont.  Participants  13 Carnegie Mellon Graduate Students  Five female, Eight male  Enrolled in School of Computer Science  Volunteers (unpaid)  Seven owned a Palm device  One had no Palm experience  Four owned Aiwa-brand stereo systems

19. Jeffrey Nichols 18 International Workshop on Smart Appliances and Wearable Computing May 19, 2003 User Study #1 Results Users made five times the errors and needed help twice as often with the actual appliances! All results significant (p < 0.001 for all)

20. Jeffrey Nichols 19 International Workshop on Smart Appliances and Wearable Computing May 19, 2003 User Study #2  We implemented the interfaces on a handheld and simulated remote control of an actual appliance.  Remote control applications built in Visual Basic on a PocketPC  Control of stereo and phone simulated in software  Feedback appeared to come from the actual appliance

21. Jeffrey Nichols 20 International Workshop on Smart Appliances and Wearable Computing May 19, 2003 User Study #2, cont.  Participants  Twelve students from Carnegie Mellon  Four female, Eight male  Volunteered in response to a newsgroup advertisement  Paid $7 for their participation (30-45 minutes)  All had limited handheld experience  Half (6) owned Aiwa-brand stereos  Two had AT&T digital answering machines

22. Jeffrey Nichols 21 International Workshop on Smart Appliances and Wearable Computing May 19, 2003 User Study #2 Results All differences are significant (p < 0.05) About ½ the time and ½ the errors!

23. Jeffrey Nichols 22 International Workshop on Smart Appliances and Wearable Computing May 19, 2003 Qualitative Results  Why were the reference interfaces better?  Clear feedback and explanation of the current state was possible.  Elements could be disabled on the screen (graying out)  Functions were separated across multiple screens.

24. Jeffrey Nichols 23 International Workshop on Smart Appliances and Wearable Computing May 19, 2003 Outline  A First Step  User Studies  Current Work

25. Jeffrey Nichols 24 International Workshop on Smart Appliances and Wearable Computing May 19, 2003 Current Work Designed a XML-based Specification Lang  Functions of Device State Variables and Commands  Labeling Multiple labels are necessary  Grouping Hierarchical groups  Dependency Information For enabling and structure

26. Jeffrey Nichols 25 International Workshop on Smart Appliances and Wearable Computing May 19, 2003 Current Work, cont.  Built multiple automatic interface generators  PocketPC  SmartPhone  Tablet PC (desktop)  Speech

27. Jeffrey Nichols 26 International Workshop on Smart Appliances and Wearable Computing May 19, 2003 Control of Real Appliances

28. Jeffrey Nichols 27 International Workshop on Smart Appliances and Wearable Computing May 19, 2003 Acknowledgements Funding  National Science Foundation  Microsoft  General Motors  Pittsburgh Digital Greenhouse Equipment Grants  Mitsubishi (MERL)  VividLogic  Hewlett-Packard PUC Project Members  Brad A. Myers  Thomas K. Harris  Roni Rosenfeld  Michael Higgins  Joseph Hughes  Kevin Litwack  Rajesh Seenichamy  Mathilde Pignol  Stefanie Shriver  Jeffrey Stylos  Peter Lucas

29. Jeffrey Nichols 28 International Workshop on Smart Appliances and Wearable Computing May 19, 2003 Thanks!  For more information see…  http://www.cs.cmu.edu/~jeffreyn/  http://www.cs.cmu.edu/~pebbles/puc/  Or e-mail me at…  jeffreyn@cs.cmu.edu

30. Jeffrey Nichols 29 International Workshop on Smart Appliances and Wearable Computing May 19, 2003

31. Jeffrey Nichols 30 International Workshop on Smart Appliances and Wearable Computing May 19, 2003 Actual Appliance Interfaces  Lots of Problems  Poorly labeled and overloaded buttons  Insufficient feedback  Timer example  Programming the speed-dial  Phone has technical separation between phone and answering machine

32. Jeffrey Nichols 31 International Workshop on Smart Appliances and Wearable Computing May 19, 2003 Qualitative Results  Grouping controls is important  Groups define which elements are placed adjacent to each other and how elements are separated onto pages.  Groupings vary between devices and interface styles.

33. Jeffrey Nichols 32 International Workshop on Smart Appliances and Wearable Computing May 19, 2003 Qualitative Results, cont.  Dual-associated functions are hard to make obvious for users  The record button is associated with both tapes (record onto) and each of the other modes (recorded from).  Some users expected the first mapping to used, whereas the controller used the second mapping.

34. Jeffrey Nichols 33 International Workshop on Smart Appliances and Wearable Computing May 19, 2003 Qualitative Results, cont.  Tree-based structures are not sufficient for fully describing an interface  Some interface concepts, especially dual- associated functions, break the tree because they may interact with the children of several different elements within the tree  The record button breaks the stereo’s tree structure because it is globally accessible but has different local effects.

35. Jeffrey Nichols 34 International Workshop on Smart Appliances and Wearable Computing May 19, 2003 Qualitative Results, cont.  A single function may map to multiple interface widgets (and vice versa)  Example: One state variable could be used to represent all of the playback states of a tape player. The play, stop, fast-forward, and rewind buttons all act on this single variable.

36. Jeffrey Nichols 35 International Workshop on Smart Appliances and Wearable Computing May 19, 2003 Applying These Results  We are actively applying these results to the design of the specification language  A tree-grouping structure is augmented with a dependency graph to help describe dual-mapped functions  Ranking relationships within groups using “priorities”  We will also apply them in the design of the automatic layout engine

37. Jeffrey Nichols 36 International Workshop on Smart Appliances and Wearable Computing May 19, 2003 Future Work  Build the specification language and automatic generation engine

38. Jeffrey Nichols 37 International Workshop on Smart Appliances and Wearable Computing May 19, 2003 A Hard Problem…  Automatically generating a good user interface is hard, but we think we can do it for several reasons:  Remote controls are a special class of user interface that use relatively simple interaction techniques.  Buttons, text fields, and other standard widgets.  Our approach differs from earlier work…

39. Jeffrey Nichols 38 International Workshop on Smart Appliances and Wearable Computing May 19, 2003 The Approach  Study Interfaces  Functional knowledge of the appliance  What must the appliance tell the handheld about itself so that a “good” interface can be constructed.  Design and Layout  How do we turn the knowledge about the appliance into a usable interface?  Design a specification language  Build an automatic interface generator

40. Jeffrey Nichols 39 International Workshop on Smart Appliances and Wearable Computing May 19, 2003 Our Progress…  Study Interfaces  Functional knowledge of the appliance  What must the appliance tell the handheld about itself so that a “good” interface can be constructed.  Design and Layout  How do we turn the knowledge about the appliance into a usable interface?  Design a specification language (in progress)  Build an automatic interface generator

41. Jeffrey Nichols 40 International Workshop on Smart Appliances and Wearable Computing May 19, 2003 Problems with User Study #1  Paper-prototype study introduced a high possibility of experimenter interference.  Solution  Create an environment that completely simulates what one might experience using a personal universal controller  Interfaces running on an actual handheld  Interfaces should appear to control an actual appliance