1. © De Montfort University, 2001 Questionnaires contain closed questions (attitude scales) and open questions pre- and post questionnaires obtain ratings on an issue before and after an design change can be used to standardise attitude measurement of single subjects following direct observation can be used to survey large user groups

2. © De Montfort University, 2001 Types of rating scales Can you use the following edit commands? yes no don't know duplicate paste A simple checklist

3. © De Montfort University, 2001 Multipoint checklist Rate the usefulness of the duplicate command on the following scale? very of no useful use

4. © De Montfort University, 2001 Likert Scale statement of opinion to which the subject expresses their level of agreement Computers can simplify complex problems very much agree slightly neutral slightly disagree strongly agree agree disagree disagree

5. © De Montfort University, 2001 Caution! The help facility in system A is much better than the help facility in system B very much agree slightly neutral slightly disagree strongly agree agree disagree disagree what does 'strongly disagree' mean?

6. © De Montfort University, 2001 Semantic differential Scale uses a series of bi-polar adjectives and obtains ratings which respect to each Rate the Beauxarts drawing package on the following dimensions extremely quite slightly neutral slightly quite extremely easy difficult clear confusing fun dreary

7. © De Montfort University, 2001 Rank Order Place the following commands in order of usefulness (rank the most useful as 1, the least useful as 4) paste duplicate group clear

8. © De Montfort University, 2001 Do and Don'ts with Questionnaire evaluation do be clear about the information you want to obtain don't risk subjects becoming demotivated don't be lazy do provide specific task reference for questions don’t assume that responses will be positive do pilot the questionnaire first

9. © De Montfort University, 20019 User diaries Used with early releases of complete systems People use system as part of their normal work and keep a log of the tasks they have used the system for and whether or not they were successful in using the system Has the advantage of using real tasks not contrived standard tasks Requires that the system is capable of supporting enough tasks to be useful to the person doing the evaluation Requires input from the evaluator to maintain the person’s motivation to keep using the diary

10. © De Montfort University, 2001 Observation and monitoring usage User trials –direct and indirect observation –verbal protocolls –Collecting user opinions User diaries over period of extended use Surveys Software logging

11. © De Montfort University, 200111 User trials - duration Aimed at observing people who are typically of the intended user group using the interface (often a prototype) People are usually volunteers – this limits the time available for each trial – serious constraint on what can be done How much of your time would you give to help someone test a piece of software? Assume a total trial length of 30 – 45 minutes – this has to include introduction, demonstration, data collection and de-brief If subjects are paid then longer trials are possible

12. © De Montfort University, 2001 User Trials: structured tasks One approach is to give subjects a series of standard tasks to complete using a prototype observe subject completing tasks under standardised conditions data collection aimed at ensuring that qualitative descriptions of problems during task completion are captured Intention is to see whether different people encounter similar problems when using the interface what problems are likely to arise in data recording?

13. © De Montfort University, 2001 Standard tasks in user trials structure tasks into incremental difficulty (easy ones first) have a clear policy on subject becoming stuck and providing help have a reason for including each task (avoid unnecessary duplication) ensure (all) functional areas of interface usage are covered ensure tasks of sufficient complexity are included

14. © De Montfort University, 200114 Example of standard tasks ‘Find the time of the latest train service leaving Leicester that I can take next Tuesday to arrive in Dundee before 8.00 pm’ ‘Find the cost of a return ticket for 2 adults and 2 children for the journey from Leicester to Bristol with no discounts such as saver or supersaver’ ‘Find how many copies of Preece ‘Human-Computer Interaction’ the library currently holds’ Note: each task has a definite end point – the user can provide the answer to the question, which is either correct or incorrect

15. © De Montfort University, 200115 Unstructured user trials Another approach to user trials is to ask the user to browse through information – more appropriate for web-sites or multimedia presentations Browsing behaviour is directed by the user’s interest rather than being asked to retrieve a specific piece of information No guarantee that subject visits all parts of the application or site – how much of the site they visit is often useful information in itself Requires that subject is actually interested in the application or site

16. © De Montfort University, 2001 Indirect observation - video enables post-session debriefing 'talk-through' (post-event protocolls) enables quantitative data to be extracted - e.g. part task timings serves as a diary and visual record of problems usually very time consuming to analyse usability laboratories – facilities to administer standard tasks, record data and analyse these

17. © De Montfort University, 2001 Verbal protocols means of enhancing direct observations user articulates what they are thinking during task completion (think-aloud protocols) but… –doing this can alter normal behaviour –subject likely to stop when undertaking complex cognitive activities –user may rationalise behaviour in post-event protocols get subjects working in pairs - co-discovery can overcome some of these problems.