1 CMT 4240 week 7 Qualitative Evaluation Methods

2 Objectives zAt the end of this lecture you will be able to carry out and analyse the following evaluation methods: yObservation yverbal protocols ycognitive walkthrough

3 Observation zDirect observation zrecorded observation zsoftware logging

4 Direct observation zObservation in the real world and in the lab zresearcher makes notes of user behaviour zquestions: ywhere do you position yourself? yWhat do you watch the screen or the user? ywhich user behaviour is relevant? yHow fast and how much can you write? yDo you have to note conversation literally?

5 Individual Exercise Observation zImagine you are in the science museum zYou evaluate the prototype “secrets behind your screen” zThe design team of the museum has asked you to find out yhow long on average visitors use the exhibit yproblems that visitors experience when using the exhibit

6 Exercise Observation zWhat do you do (step by step)? zWhich problems do you expect? zWhat are you going to take with you? z5 minutes

7 What are you going to take with you? zNotepad and pens zcardboard with clip z(stop-) watch zobservation forms z...

8 Potential Problems zWhere to position yourself? Need to see the user’s actions and his non-verbal behaviour. zUser may feel disturbed, if you watch to closely. zHow to stop the time and to observe potential usability problems at the same time. zHawthorne effect

9 Potential Problems zOnly one pass at the data collection yobserver has to decide in real time what is important and what is not yobserver may miss important data yobserver may misconceive events or misunderstand the user yobserver has to know from the outset what to look out for

10 Potential Problems zYou may get distracted zYour handwriting may be illegible zYou may not remember what you meant with certain words zYou may jot down too few words

11 So what do you do? zPreparation: ymake observation sheets xstart time, end time xuser age (approximately) and gender xactions xproblems

12 What do you do? zPosition sideways between exhibit and user so that you can see both zWrite down everything as explicitly as possible zconversations need to be written down as literal as possible zKeep to the observable FACTS not your interpretation of them

13 Recorded observation zAdvantage: yyou can repeatedly look at the recordings and discover new things every time zDisadvantage: ytechnology is fine, if it works... yYou need electricity, extension leads, several cameras, tapes, etc. yit can be costly

14 Analysing Observation Material zRecordings: make transcripts ztranscripts and notes should be coded zInformal analysis y what are the problems? zTask based analysis yhow does the user tackle a given task? zPerformance based analysis yobtain data on predefined performance measures

15 Verbal protocols zVerbal protocols capture the user’s thoughts and spoken observations zTwo types of verbal protocols y think aloud protocol ypost event protocol zverbal protocols are often used in combination with video recording

16 Think aloud protocol zThe user is being asked to say out loud what he or she think while he or she is carrying out a task

17 Post event protocol zThe user is being asked to think out loud after the task has been completed

18 Exercise Think Aloud Protocol: zGet together in pairs zOne student is the observer and one student is the subject zThe observer makes notes zThe subject solves the following problem while thinking out loud: yhow does the following series continue: x 0, 1/3, 2/4, 3/5, 4/6, ?

19 Solution?

20 Analysis of the protocol zGo through your notes and ask the following questions: zHow many different attempts to solve the problem can you identify? zHow many different problems did your subject encounter? zWhich statement in the protocol supports which problems solving activity and which problem encoutered?

21 Cognitive Walkthrough zEvaluation method based on the idea that… ythe user learns about the interface in an exploratory way, ythe user has goals,  the user is applying simple means-ends reasoning.

22 Cognitive Walkthrough zEvaluation by an expert, who goes through a set task while imitating user performance

23 Preparation: We need four things 1.User description including level of experience with computers and any assumptions made by the designer 2.System description including operations and performance (e.g. paper design) 3.Task description specifying the task that the expert has to carry out from users point of view 4.Action sequence describing the system display and the user actions needed to complete the given task. One system display and one user action together are one step.

24 Process of carrying out a Cognitive Walkthrough zPrototype, user description, system description, task description and action list has to be prepared by the designer/developer zDesigner gives these document to an expert. zThe expert reads the descriptions. zThe expert carries out the task by following the action list. zThe expert asks the following questions with EACH step of the action list:

25 Questions Cognitive Walkthrough 1.Is the next goal clear at this stage? 2.Is the appropriate action obvious? 3.Is it clear that this action leads to the goal? 4.What problems are there in performing the action?

26 Example: the Automatic Nail Painter zThe automatic nail painter is a machine with which one can have one’s nails painted.

27 User Description Automatic Nail Painter zThe users are assumed to have no prior knowledge of how to operate the Automatic Nail Painter. However, they are assumed to have experience in putting nail varnish on nails manually

28 System Description Automatic Nail Painter zThe system allows one finger to be placed into the machine at a time. The user programmes the machine for use by choosing: - the number of fingers to be varnished - the colours of the varnish to be used - the number of coatings to be applied

29 Task Description Automatic Nail Painter zThe user must get all ten fingers varnished with one coat of purple varnish. The system is already switched on when the user begins his/her task. The task is completed, when the tenth fingernail is varnished.

30 Action list Automatic Nail Painter zSystem: System displays ‘How many fingers do you want to varnish?’ User action: user selects number 10 by pressing the + or the - button until the appropriate number is shown System: System responds by displaying the current number of fingers selected to be varnished. User action: User presses ok when the correct number has been displayed

31 Action list Automatic Nail Painter (cont.) zSystem: System displays ‘choose the colour of varnish you wish to apply.’ User action: User must select the colour purple by choosing from the colour menu. This includes scrolling up and down until s/he finds purple. S/he then has to press the OK button. System: Systems displays ‘how many coats do you want to apply?’ User action: User selects the number of coats by pressing the ‘+’ or ‘-’ button until the 1 is shown.

32 Action list Automatic Nail Painter (cont.) zSystem: System displays the number of fingers to be varnished, the colour of the varnish and the number of coats to be applied. System asks for confirmation User action: User presses OK to confirm System: System displays ‘Press start to begin varnishing.’ User action: User presses start button etc.

33 Step 1- Question 1 Automatic Nail Painter zFrom the action list: Step 1 System: System displays ‘How many fingers do you want to varnish? ’ User action: user selects number 10 by pressing the + or the - button until the appropriate number is shown zQuestion 1: Is the goal clear at this stage? Not completely. The user is asked to state the number of fingers to be varnished. He could be thinking that the system deals with one hand at a time.

34 Step 1 - Questions 2, 3, and 4 Automatic Nail Painter zQuestion 2: Is the appropriate action obvious? No. The system dos neither indicate that the user has to use the ‘+’ and ‘-’ buttons nor explain their use. zQuestion 3: Is it clear that this action leads to the goal? Yes If the user presses the ‘+’ and ’-’ button the numbers are clearly displayed. zQuestion 4: zWhat problems are there in performing the action? The action is simple to perform once the user has discovered the use of the buttons.

35 Step 2 – Question 1 Automatic Nailpainter z?z?

36 Summary zDirect and recorded Observation zVerbal Protocol yThink aloud protocol zCognitive Walkthrough yPreparation: System description, user description, task description, Action sequence yProcess: for each step in the action sequence four questions to be answered