1. The product of evaluation is knowledge. This could be knowledge about a design, knowledge about the user or knowledge about the task.

2.  Two types of evaluation: 1.Summative Evaluation 2.Formative Evaluation  Is evaluation necessary?  When do we need it?  Quantitative vs. Qualitative Data

3.  To increase the quality of research, we must avoid the following effects during the observational study: 1.Hawthorne effect 2.Observer effect 3.Halo effect

4.  Verbal Protocols  Designing Observations: ◦ Writing a verbal protocol ◦ How to conduct a session ◦ Analyzing a protocol/transcript

5.  We ask users to speak out load and mention why they had done an action.  2 components: ◦ Talk aloud: verbalize silent decision ◦ Think aloud: verbalize whatever thoughts occur during this task.

6.  Select: ◦ Tasks ◦ Users ◦ Environment ◦ One or two significant functional requirements  Observe (at least 3 users)  Give users their set of tasks to complete  Conduct a think-aloud study  Keep protocols (transcripts for each user)  Record users comments, etc. (p. 141)

7.  Description of the environment  List of tasks completed by the user  Users’ background & demographic details  Record and write up users’ comments, body language, facial expressions  The aspects of the interface that these responses relate to should also be detailed.

8.  Choose the tasks  Select users – wrong users lead to misleading information  Explain the purpose to the users  Conduct the evaluation – for example: ◦ What are you looking at now? ◦ What just happened? ◦ What are you going to do next? Why?

9.  The data that you will have at the end of a session is known as the transcript, which details the physical actions and verbal commentary that the user has made.  When analyzing a transcript of an evaluation session, the aim is to categorize the comments according to: ◦ Frequency ◦ Fundamentality

10.  Experiments define a hypothesis  2 stages of experiments: ◦ Implementing the experiment ◦ Analyzing the results  Advantages: systematic with a repeatable approach to testing based on scientific rigor.  Disadvantages: includes a reduced consideration of specific variables, questions which are hard t relate to real-world holistic problems and an artificial setting which may lack real-world validity.  In testing we strive for realism

11.  CW is an approach to formative evaluation without users. ◦ Preparing a CW ◦ Conducting a CW ◦ Experiments in Support of Design ◦ Dependent and Independent Variables ◦ Assigning Subjects ◦ Statistics ◦ Summary of user experimentation

12.  Interviews ◦ Structured interview ◦ Unstructured interview  Questionnaires: ◦ Open Q’s ◦ Closed Q’s:  Simple checklist: Y/N or N/A  Ranked order: (select your preference)  Multi-point rating scale: strongly agree/disagree