1. Evaluation techniques Part 2
Lecture 14
Evaluation techniques Part 2

2. Today’s Topics We will cover today
Evaluation through user participation
Styles of evaluation
Empirical methods: experimental evaluation
Experimental Design
Analysis of data
Experimental studies on groups

3. Today’s Topics Observational Methods Query Techniques
Psychological Methods
Selecting appropriate evaluation technique

4. Evaluating through user Participation
User participation in evaluation tends to occur in the later stages of development when there is at least a working prototype of the system in place.
This may range from a simulation of the system’s interactive capabilities, without its underlying functionality.

5. Styles of evaluation Two distinct styles of evaluation are
those performed under laboratory conditions and
those conducted in the work environment or ‘in the field’

6. Laboratory studies Advantages: Disadvantages: Appropriate
specialist equipment available
uninterrupted environment
Disadvantages:
lack of context
difficult to observe several users cooperating
Appropriate
if system location is dangerous or impractical for constrained single user systems to allow controlled manipulation of use

7. Field Studies Advantages: Disadvantages: Appropriate
natural environment
context retained (though observation may alter it)
longitudinal studies possible
Disadvantages:
distractions
noise
Appropriate
where context is crucial for longitudinal studies

8. Empirical methods: experimental evaluation
One of the most powerful methods of evaluating a design or an aspect of a design is to use a controlled experiment.
This provides empirical evidence to support a particular claim or hypothesis.
It can be used to study a wide range of different issues at different levels of detail.

9. Empirical methods: experimental evaluation
The evaluator chooses a hypothesis to test, which can be determined by measuring some attribute of participant behavior.
A number of experimental conditions are considered which differ only in the values of certain controlled variables.

10. Empirical methods: experimental evaluation
Factors to be considered for experiment design mainly includes,
Participants chosen
Variables tested and manipulated
Hypothesis tested
Experimental Design

11. Participants
In evaluation experiments, participants should be chosen to match the expected user population as closely as possible.
This will involve experimental testing with the actual users but this is not always possible.
If participants are not actual users, they should be chosen to be of a similar age and level of education as the intended user group.

12. Participants
Their experience with computers in general, and with systems related to that being tested, should be similar, as should their experience or knowledge of the task domain.

13. Participants
It is no good testing an interface designed to be used by the general public on a participant set made up of computer science undergraduates: they are simply not representative of the intended user population

14. Participants
Choose enough sample size of population to test all aspects of the system.
Nielsen and Landauer suggest that usability testing with a single participant will find about a third of the usability problems, and that there is little to be gained from testing with more than five.
Always test using large number of participants i.e.; double of five.

15. Variables
Experiments manipulate and measure variables under controlled conditions, in order to test the hypothesis.
There are two main types of variable:
those that are ‘manipulated’ or changed (known as the independent variables) and
those that are measured (the dependent variables).

16. Variables independent variable (IV) dependent variable (DV)
characteristic changed to produce different conditions
e.g. interface style, number of menu items
dependent variable (DV)
characteristics measured in the experiment
e.g. time taken, number of errors.

17. Hypothesis
A hypothesis is a prediction of the outcome of an experiment.
It is framed in terms of the independent and dependent variables, stating that a variation in the independent variable will cause a difference in the dependent variable.
The aim of the experiment is to show that this prediction is correct.

18. Hypothesis
This is done by disproving the null hypothesis, which states that there is no difference in the dependent variable between the levels of the independent variable.
If a result is significant at the given level of certainty, that the differences measured would not have occurred by chance (that is, that the null hypothesis is incorrect).

19. Hypothesis prediction of outcome null hypothesis:
framed in terms of IV and DV
For example “error rate will increase as font size decreases”
null hypothesis:
states no difference between conditions
aim is to disprove this
e.g. null hyp. = “no change with font size”

20. Experimental Design There are two main methods: between-subjects and
within-subjects.

21. Experimental design within groups design between groups design
each subject performs experiment under each condition.
transfer of learning possible
less costly and less likely to suffer from user variation.
between groups design
each subject performs under only one condition
no transfer of learning
more users required
variation can bias results.

22. Between group design
The advantage of a between-subjects design is that any learning effect resulting from the user performing in one condition and then the other is controlled:
each user performs under only one condition.
The disadvantages are that a greater number of participants are required, and that significant variation between the groups can negate any results.

23. Within-group Design Each user performs under each different condition.
This design can suffer from transfer of learning effects, but this can be lessened if the order in which the conditions are tackled is varied between users.

24. Analysis of data Before you start to do any statistics:
look at data
save original data
Choice of statistical technique depends on
type of data
information required
Type of data
discrete - finite number of values
continuous - any value

25. Analysis - types of test
parametric
assume normal distribution
robust
powerful
non-parametric
do not assume normal distribution
less powerful
more reliable
contingency table
classify data by discrete attributes
count number of data items in each group

26. Analysis of data (cont.)
What information is required?
is there a difference?
how big is the difference?
how accurate is the estimate?
Parametric and non-parametric tests mainly address first of these

27. Experimental studies on groups
Experimental studies of groups and of groupware are more difficult than the corresponding single-user experiments already considered.

28. Experimental studies on groups
Problems with:
subject groups
choice of task
data gathering
analysis

29. Experimental studies on groups
To organize, say, 10 experiments of a single-user system requires 10 participants.
For an experiment involving groups of three, we will, of course, need 30 participants for the same number of experiments.

30. Experimental Studies on groups
In addition, experiments in group working are often longer than the single-user equivalents as we must allow time for the group to ‘settle down’ and some rapport to develop.
This all means more disruption for participants and possibly more expense payments.

31. Subject groups
larger number of subjects  more expensive longer time to `settle down’ … even more variation! difficult to timetable so … often only three or four groups

32. The Task Choosing a suitable task is also difficult.
We may want to test a variety of different task types: creative, structured, information passing, and so on.

33. The task must encourage cooperation perhaps involve multiple channels
options:
creative task e.g. ‘write a short report on …’
decision games e.g. desert survival task
control task e.g. ARKola bottling plant

34. Data gathering several video cameras + direct logging of application
problems:
synchronisation
sheer volume!
one solution:
record from each perspective

35. Analysis
In true experimental tradition, we would like to see statistical differences between experimental conditions.
We saw earlier that individual differences made this difficult in single-user experiments. If anything, group variation is more extreme. Given randomly mixed groups, one group will act in a democratic fashion. Thus dominating others.

36. Analysis Vast variation between groups solutions:
within groups experiments
micro-analysis (e.g., gaps in speech)
anecdotal and qualitative analysis
look at interactions between group and media
controlled experiments may `waste' resources!
anecdotal =subjectivee

37. Field studies Experiments dominated by group formation
Field studies more realistic:
distributed cognition  work studied in context
real action is situated action
physical and social environment both crucial
Contrast:
psychology – controlled experiment
sociology and anthropology – open study and rich data

38. Observational Methods
A popular way to gather information about actual use of a system is to observe users interacting with it.
Techniques include
Think Aloud
Cooperative evaluation
Protocol analysis
Automated analysis
Post-task walkthroughs

39. Think Aloud user observed performing task
user asked to describe what he is doing and why, what he thinks is happening etc.
Advantages
simplicity - requires little expertise
can provide useful insight
can show how system is actually use
Disadvantages
subjective
selective
act of describing may alter task performance

40. Cooperative evaluation
variation on think aloud
user collaborates in evaluation
both user and evaluator can ask each other questions throughout
Additional advantages
less constrained and easier to use
user is encouraged to criticize system
clarification possible

41. Protocol analysis paper and pencil – cheap, limited to writing speed
audio – good for think aloud, difficult to match with other protocols
video – accurate and realistic, needs special equipment, obtrusive
computer logging – automatic and unobtrusive, large amounts of data difficult to analyze
user notebooks – coarse and subjective, useful insights, good for longitudinal studies
Mixed use in practice.
audio/video transcription difficult and requires skill.
Some automatic support tools available

42. Automated Analysis – EVA
Workplace project
Post task walkthrough
user reacts on action after the event
used to fill in intention
Advantages
analyst has time to focus on relevant incidents
avoid excessive interruption of task
Disadvantages
lack of freshness
may be post-hoc interpretation of events

43. post-task walkthroughs
transcript played back to participant for comment
immediately  fresh in mind
delayed  evaluator has time to identify questions
useful to identify reasons for actions and alternatives considered
necessary in cases where think aloud is not possible

44. Query Techniques
Relies on asking the user about the interface directly.
Query techniques can be useful in eliciting detail of the user’s view of a system.
Types of Query Techniques are
Interviews
Questionnaires

45. Interviews
analyst questions user on one-to -one basis usually based on prepared questions
informal, subjective and relatively cheap
Advantages
can be varied to suit context
issues can be explored more fully
can elicit user views and identify unanticipated problems
Disadvantages
very subjective
time consuming

46. Questionnaires Set of fixed questions given to users Advantages
quick and reaches large user group
can be analyzed more rigorously
Disadvantages
less flexible
less probing

47. Questionnaires (ctd) Need careful design Styles of question
what information is required?
how are answers to be analyzed?
Styles of question
general
open-ended
scalar
multi-choice
ranked

48. Psychological Methods
One of the problems with most evaluation techniques is that we are reliant on observation and the users telling us what they are doing and how they are feeling.
What if we were able to measure these things directly?

49. Psychological Methods
Interest has grown recently in the use of what is sometimes called objective usability testing, ways of monitoring physiological aspects of computer use.
Potentially this will allow us not only to see more clearly exactly what users do when they interact with computers, but also to measure how they feel.

50. Physiological methods
Areas receiving more attention are,
Eye tracking
Physiological measurement

51. Eye tracking
Head or desk mounted equipment tracks the position of the eye
Eye movement reflects the amount of cognitive processing a display requires
measurements include
fixations: eye maintains stable position. Number and duration indicate level of difficulty with display
saccades: rapid eye movement from one point of interest to another
scan paths: moving straight to a target with a short fixation at the target is optimal

52. Eye Tracking Equipment

53. Calibrating Eye Tracking

54. Eye Tracking Animation

55. Physiological measurements
Emotional response linked to physical changes
These may help determine a user’s reaction to an interface
Measurements include:
heart activity, including blood pressure, volume and pulse.
activity of sweat glands: Galvanic Skin Response (GSR)
electrical activity in muscle: electromyogram (EMG)
electrical activity in brain: electroencephalogram (EEG)
Some difficulty in interpreting these physiological responses - more research needed

56. Selecting an Evaluation Method
We have to decide which method is most appropriate for our needs.
There is no hard and fast rule to select any method.
However, there are a number of factors that should be taken into account when selecting evaluation techniques.

57. Factors distinguishing evaluation techniques
The stage in the cycle at which the evaluation is carried out
The style of evaluation
The level of subjectivity or objectivity of the technique
The type of measures provided
The information provided

58. Factors distinguishing evaluation techniques
The immediacy of the response
The level of interference implied
The resources required.

59. Choosing an Evaluation Method
when in process: design vs. implementation style of evaluation: laboratory vs. field how objective: subjective vs. objective type of measures: qualitative vs. quantitative level of information: high level vs. low level level of interference: obtrusive vs. unobtrusive resources available: time, subjects, equipment, expertise

60. Summary
Evaluation is an integral part of the design process and should take place throughout the design life cycle.
Its aim is to test the functionality and usability of the design and to identify and rectify any problems.
It can also try to determine the user’s attitude and response to the system.

61. Summary
It can take place in a specialist laboratory or in the user’s workplace, and may or may not involve active participation on the part of the user.
Experts can reduce time of evaluation by considering previous experimental results.
The choice of evaluation method is largely dependent on what is required of the evaluation.