1. Human Computer Interaction Lecture 18 Evaluation Through User Participation

2. Experimental Evaluation
- Controlled evaluation of specific aspects of interactive behaviour - Evaluator chooses hypothesis to be tested - A number of experimental conditions are considered which differ only in the value of some controlled variable. - Changes in behavioural measure are attributed to different conditions

3. Experimental factors Participants Variables Hypothesis
Users – representative, sufficient sample
Variables
Things to modify and measure
Hypothesis
What you’d like to show
Experimental design
How you are going to do it

4. Participants
Participants should be chosen to match the expected user population as closely as possible.
If participants are not actual users they should be a similar age, education or profession.
Sample size must be large enough to be considered to represent the actual user population.

5. Variables - independent variable (IV) - dependent variable (DV)
characteristic changed to produce different conditions
e.g. interface style, number of menu items etc.
- dependent variable (DV)
characteristics measured in the experiment
e.g. time taken, number of errors etc.

6. Hypothesis - Prediction of outcome - Null hypothesis:
States that a variation in the independent variables will cause a difference in the dependent variables.
e.g. “error rate will increase as font size decreases”
- Null hypothesis:
- states no difference between conditions
- aim is to disprove this
e.g. null hypothesis. = “no change with font size”

7. Observational Methods
Think Aloud
Cooperative evaluation
Protocol analysis
Post-task walkthroughs

8. Think Aloud - user observed performing task
- user asked to describe what he is doing and why, what he thinks is happening etc.
Advantages
- simplicity - requires little expertise
- can show how system is actually being used
Disadvantages
Subjective

9. Cooperative evaluation
- variation on think aloud
- user collaborates in evaluation
- both user and evaluator can ask each other questions throughout
Additional advantages
- less constrained and easier to use
- user is encouraged to criticize system
- clarification possible

10. Protocol Analysis Methods for recording user actions including:
- paper and pencil – cheap, limited to writing speed
- audio – good for think aloud, difficult to match with other protocols
- video – accurate and realistic, needs special equipment.
- computer logging – automatic and modest, large amounts of data difficult to analyze
- user notebooks – records every few minutes or hourly, good for longitudinal studies
- Mixed use in practice.
audio/video transcription difficult and requires skill.

11. Post-Task walkthroughs
user reacts on action after the event
Advantages
- analyst has time to focus on relevant incidents
avoid excessive interruption of task
Disadvantages
- lack of freshness

12. Interviews Questionnaires
Query Techniques
Interviews
Questionnaires

13. Interviews
analyst questions user on one-to-one basis usually based on prepared questions
Informal and relatively cheap
Advantages
can be varied to suit context
issues can be explored more fully
can extract user views and identify unexpected problems
Disadvantages
time consuming

14. Questionnaires Set of fixed questions given to users Advantages
quick and reaches large user group
Disadvantages
less flexible
less questioning

15. Questionnaires (ctd) Need careful design Styles of question
what information is required?
how are answers to be analyzed?
Styles of question
general
open-ended
multi-choice
ranked

16. Physiological methods
Eye tracking
Physiological measurement

17. Eye Tracking
head or desk mounted equipment tracks the position of the eye
eye movement reflects the amount of cognitive processing a display requires
measurements include
fixations: eye maintains stable position. Number and duration indicate level of difficulty with display
saccades: rapid eye movement from one point of interest to another
scan paths: moving straight to a target with a short fixation at the target is optimal
Still a lot of research is needed in this area

18. Physiological Measurements
emotional response is linked to physical changes
these may help determine a user’s reaction to an interface
Measurements are made by attaching various sensors to users

19. Physiological Measurements
Measurements include:
heart activity, including blood pressure and pulse.
May represent stress or anger
activity of sweat glands: indicated by skin resistance
Represent mental effort needed
electrical activity in muscle: measured by electromyogram (EMG)
Reflect involvement in a task
electrical activity in brain: electroencephalogram (EEG)
Reflect decision making and motivation
Some difficulty in interpreting these physiological responses
Increased pulse rate <- (frustration with the interface OR Stress at being unable to complete the task)
More research needed