1. CSCD 487/587 Human Computer Interface Winter 2013 Lecture 19 Evaluation

2. Overview Evaluation is integral to the design process Evaluators collect information about users' or potential users' experiences when interacting with your interface They do this in order to improve its design Evaluation focuses on both usability of system – How easy it is to learn and to use and On users' experience when interacting with system – How satisfying, enjoyable, or motivating

3. Overview Many different evaluation methods Which to use depends on the goals of the evaluation Evaluations can occur in a range of places such as laboratories, people's homes, outdoors, and work settings Evaluations usually involve observing participants and measuring their performance – In usability testing, experiments, or field studies There are other methods, however, that do not involve participants, such as modeling user behavior Tend to be approximations of what users might do when interacting with an interface … not measuring users

4. Why You Need to Evaluate Designers should not presume that everyone is like them or that following set guidelines would guarantee them good usability Evaluation is needed to check that users can use product and like it

5. Why You Need to Evaluate Bruce Tognazzini – Usability Consultant “Iterative design, with its repeating cycle of design and testing, is the only validated methodology in existence that will consistently produce successful results If you don’t have user-testing as a part of your design process you are going to throw buckets of money down the drain”

6. Why You Need to Evaluate Toganazzini’s 5 resons to evaluate 1. Problems are fixed before product is shipped, not after 2. The team can concentrate on real problems, not imaginary ones 3. Engineers code is sharply reduced 4. Time to market is sharply reduced 5. Upon 1 st release, your sales department has a rock solid design it can sell without having to pepper their pitches with how well next release will work

7. Why You Need to Evaluate Evaluation as part of the Design Process design implementationevaluation

8. What to Evaluate It’s a Continuous Process Users and their tasks Observe, measure, and analyze their performance with the system Basically anything for which we need feedback Important for designer to check and make sure that they understand user requirements

9. Where Do We Evaluate Laboratories – Formal experiments On-line – Gather feedback from on-line user statistics Homes – Users can provide feedback in a natural setting

10. When to Evaluate Depends on what we are evaluating Requirements Gather them, evaluate, Did we capture them? Iterate Evaluate again May do this several times Design Evaluate as develop interface and product Known as Formative Evaluation Product Completion Evaluate to assess user satisfaction Known as Sumative Evaluation

11. When to Evaluate New product Use mockups, sketches, and other low fidelity prototyping techniques are used to represent design ideas Upgrading existing products Compare user performance and attitudes and contrast new products with the previous versions Evaluation is a key ingredient for a successful design

12. Evaluation Types 1. Controlled Settings Labs or Living Labs User's activities are controlled Do usability testing and experiments 2. Natural Settings On-line communities and public places No control over user activities Field studies

13. Evaluation Types 3. Settings Not Involving Users Experts and consultants research Predict and model aspects of interfaces You Choose a method based on …. Deciding how much control is needed to find out how an interface is used

14. Controlled Settings Evaluators can control what users do Also reduce outside influences and distractions Usability Testing Purpose - Evaluate User Interfaces Experiments, observations, interviews and questionnaires In a controlled setting Goal Determine whether an interface is usable by intended population

15. Controlled Settings Usability Testing continued Has been done for years as part of Human Computer Interaction Findings of test summarized in a usability specification Changes in design can be specified according to usability spec Example Paper prototype session viewed in class one example

16. Natural Settings Evaluate People in Natural Settings Purpose Identify opportunities for new technology Establish requirements for a new design New context for existing product Field Studies Can be in person or virtual On-line communities Chat rooms Goals Examine social processes, collaborations and cooperation between users

17. Natural Settings Field Studies While techniques, such as focus groups, usability tests, and surveys, can lead to valuable insights, most powerful tool in the toolbox is the 'field study' Field studies get teams immersed in environment of their users and allow them to observe critical details for which there is no other way of discovering Examples: Watch users in shopping malls, watched system administrators, watched paper flow through manufacturing facility What do you think is biggest downside ? Biggest downside to field studies is the cost to the organization, Scheduling the visits, taking team members out of the office for several days, and finishing the analysis can have a huge impact on a project's resources http://www.uie.com/articles/field_studies/ Measuring man or dog?

18. Field Studies The Power of Field Studies Even a short field study, such as two or three half-day visits, can yield tremendous value. From these we can learn: Terminology and processes – What do users do and how do they talk about it? – While users can describe a process or share terms in an interview format, watching them work points out subtleties that they are unaware of

19. Field Studies The Power of Field Studies Even a short field study, such as two or three half- day visits, can yield tremendous value. From these we can learn: Context – What are the external forces that will impact the design? – Do the user's requirements change when they are rushed or up against a deadline? – People have trouble describing the context of their work, however it's easy for outsiders to observe

20. Field Studies The Power of Field Studies Even a short field study, such as two or three half- day visits, can yield tremendous value. From these we can learn: Similarities and differences – Visiting multiple sites can allow a team to collect a rich amount of information about commonalities that appear across environments, along with variations that will impact design decisions – Such as providing switches, options, and optional features

21. Research Not with Users Researcher has to model how interface is likely to be used Predict user behavior Identify usability problems Example Heuristic evaluation of an interface Applies knowledge of user behavior Context system will be used Steps through a scenario and answers set of questions

22. Heuristic Evaluation Originally used in evaluation of screen-based applications Heuristic evaluation is discount usability engineering method for quick, cheap, and easy evaluation of a user interface design Heuristic evaluation is most popular usability inspection method Heuristic evaluation is done as systematic inspection of a user interface design for usability

23. Heuristic Evaluation Jakob Nielsen Most usability engineering methods will contribute substantially to the usability of an interface … …if they are actually used.

24. Heuristic Evaluation Goal of heuristic evaluation is to find usability problems in design so that they can be integrated into iterative design process Heuristic evaluation involves having a small set of evaluators examine interface and judge its compliance with recognized usability principles (the "heuristics") Dates back to the 1990's

25. Heuristic Evaluation How does it work? Evaluators use a checklist of basic usability heuristics Evaluators go through an interface twice 1 st Pass get a feel for the flow and general scope 2 nd Pass refer to checklist of usability heuristics and focus on individual elements The findings of evaluators are combined and assessed http://www.nngroup.com/articles/how-to-conduct-a-heuristic- evaluation/

26. Heuristic Evaluation Usability Heuristics (original, unrevised list) Simple and natural dialogue Speak the users’ language Minimize the users’ memory load Consistency Feedback Clearly marked exits Shortcuts Precise and constructive error messages Prevent errors Help and documentation COMMENTS?

27. Heuristic Evaluation Debriefing session Conducted in brain-storming mode Evaluators rate the severity of all problems identified Use a 0 – 4, absolute scale 0 I don’t agree that this is a prob at all 1 Cosmetic prob only 2 Minor prob – low priority 3 Major prob – high priority 4 Usability catastrophe – imperative to fix

28. Heuristic Evaluation How does H.E. Differ from User Testing? Evaluators have checklists Evaluators are not target users Evaluators decide on their own how they want to proceed Observer can answer evaluators’ questions about the domain or give hints for using the interface Evaluators say what they didn’t like and why; observer doesn’t interpret evaluators’ actions

29. Heuristic Evaluation What are the shortcomings of H.E.? Identifies usability problems without indicating how they are to be fixed “Ideas for appropriate redesigns have to appear magically in the heads of designers on the basis of their sheer creative powers.” Cannot expect it to address all usability issues when evaluators are not domain experts / actual users Have a short video of Heuristic Evaluation … http://www.youtube.com/watch?v=8pDJwjacZnw

30. On-line Analytics Web Analytics Measurement, collection and reporting of user behavior Understand and optimize web usage Tools help analyze log files of interaction data and web traffic http://www.kaushik.net/avinash/qualitative-web- analytics-expert-heuristic-evaluations/

31. Web Analytics What do they tell you? Data such as – Number of visitors to a website, – Where they are from, – Which pages they view, and – Which links they click Measured, collected, analyzed and reported Web analytics are relevant to usability practitioners in that they can provide insight into the large-scale behavior of website users to understand and improve (optimize) the website What wouldn't they tell you? http://www.usabilitybok.org/web-analytics

32. Web Analytics What wouldn't they tell you? Web analytics cannot provide answers – About user motivations or underlying needs and goals – Web analytics may indicate that users are abandoning a checkout process at a particular point, but they cannot be used to explain why this is happening – Usability testing of issues found through web analytics brings deeper understanding needed to fix these usability problems – Additionally, in-person observations of users can lead to insight that informs what metrics are worthwhile to collect, and how to interpret them.

33. Summary User evaluation is a critical part of user centered design Uses many of the techniques we already discussed Plus, can use Heuristics as a quick and cheap way to explore user interface problems Web statistics through website analytics can provide valuable data but is no substitute for more in-depth user testing

34. 34 End Two more Weeks of Classes !!!