1. ©2011 www.id-book.com1 An Evaluation Framework Chapter 13

2. ©2011 www.id-book.com2 Aims of this chapter: o Introduce and explain the DECIDE framework. o Discuss the conceptual, practical, and ethical issues involved in evaluation.

3. ©2011 www.id-book.com3 DECIDE: A Framework to guide Evaluation o Determine the goals. o Explore the questions. o Choose the evaluation methods. o Identify the practical issues. o Decide how to deal with the ethical issues. o Evaluate, analyze, interpret and present the data.

4. ©2011 www.id-book.com4 Determine the goals o What are the high-level goals of the evaluation? o Who wants it and why? o The goals influence the methods used for the study. o Goals vary and could be to:  Identify the best metaphor for the design  Check that user requirements are met  Check for consistency  Investigate how technology affects working practices  Improve the usability of an existing product

5. ©2011 www.id-book.com5 Explore the questions o Questions help to guide the evaluation. o For example: The goal of finding out why some customers prefer to purchase paper airline tickets rather than e-tickets can be broken down into sub-questions:  What are customers’ attitudes to e-tickets?  Are they concerned about security?  Is the interface for obtaining them poor?

6. ©2011 www.id-book.com6 Task What questions might you ask about the design of a cell phone?

7. ©2011 www.id-book.com7 Choose the evaluation approach & methods o The evaluation method influences how data is collected, analyzed and presented. o E.g. field studies typically:  Involve observation and interviews.  Involve users in natural settings.  Do not involve controlled tests.  Produce qualitative data.

8. ©2011 www.id-book.com8 Identify practical issues How to:  Select users  Find evaluators  Select equipment  Schedule & budget constrains

9. ©2011 www.id-book.com9 Decide about ethical issues o Develop an informed consent form Cont..

10. ©2011 www.id-book.com10 o Participants have a right to:  Know the goals of the study.  Know what will happen to the findings.  Privacy of personal information.  Leave when they wish.  Be treated politely. Cont…

11. ©2011 www.id-book.com11 Evaluate, interpret & present data o Methods used influence how data is evaluated, interpreted and presented. o The following need to be considered:  Reliability: can the study be replicated?  Validity: is it measuring what you expected?  Ecological validity: is the environment influencing the findings? i.e. Hawthorne effect  Biases: is the process creating biases?  Scope: can the findings be generalized?

12. ©2011 www.id-book.com12 Key points o Many issues to consider before conducting an evaluation study. o These include: goals of the study; involvment or not of users; the methods to use; practical & ethical issues; how data will be collected, analyzed & presented. o The DECIDE framework provides a useful checklist for planning an evaluation study.

13. ©2011 www.id-book.com13 A project for you Find an evaluation study and use the DECIDE framework to analyze it. Some questions that you should seek to answer are: a)What are the goals and the questions that provide the focus for the evaluation? b)Which evaluation methods are used? c)What data is collected and how it is analyzed? d)What practical and ethical issues have been considered? e)Comment on the reliability, ecological validity, biases, validity and scope of the study.