The product of evaluation is knowledge. This could be knowledge about a design, knowledge about the user or knowledge about the task.
Two types of evaluation: 1.Summative Evaluation 2.Formative Evaluation Is evaluation necessary? When do we need it? Quantitative vs. Qualitative Data
To increase the quality of research, we must avoid the following effects during the observational study: 1.Hawthorne effect 2.Observer effect 3.Halo effect
Verbal Protocols Designing Observations: ◦ Writing a verbal protocol ◦ How to conduct a session ◦ Analyzing a protocol/transcript
We ask users to speak out load and mention why they had done an action. 2 components: ◦ Talk aloud: verbalize silent decision ◦ Think aloud: verbalize whatever thoughts occur during this task.
Select: ◦ Tasks ◦ Users ◦ Environment ◦ One or two significant functional requirements Observe (at least 3 users) Give users their set of tasks to complete Conduct a think-aloud study Keep protocols (transcripts for each user) Record users comments, etc. (p. 141)
Description of the environment List of tasks completed by the user Users’ background & demographic details Record and write up users’ comments, body language, facial expressions The aspects of the interface that these responses relate to should also be detailed.
Choose the tasks Select users – wrong users lead to misleading information Explain the purpose to the users Conduct the evaluation – for example: ◦ What are you looking at now? ◦ What just happened? ◦ What are you going to do next? Why?
The data that you will have at the end of a session is known as the transcript, which details the physical actions and verbal commentary that the user has made. When analyzing a transcript of an evaluation session, the aim is to categorize the comments according to: ◦ Frequency ◦ Fundamentality
Experiments define a hypothesis 2 stages of experiments: ◦ Implementing the experiment ◦ Analyzing the results Advantages: systematic with a repeatable approach to testing based on scientific rigor. Disadvantages: includes a reduced consideration of specific variables, questions which are hard t relate to real-world holistic problems and an artificial setting which may lack real-world validity. In testing we strive for realism
CW is an approach to formative evaluation without users. ◦ Preparing a CW ◦ Conducting a CW ◦ Experiments in Support of Design ◦ Dependent and Independent Variables ◦ Assigning Subjects ◦ Statistics ◦ Summary of user experimentation
Interviews ◦ Structured interview ◦ Unstructured interview Questionnaires: ◦ Open Q’s ◦ Closed Q’s: Simple checklist: Y/N or N/A Ranked order: (select your preference) Multi-point rating scale: strongly agree/disagree