CITERS2003 Presentation 5 July A Usability Study of a Language Centre Web Site (a presentation of an MSc[ITE] Dissertation) By Andy Morrall

2 Why, in general? Teachers write educational web pages Students use them outside the classroom Teachers don’t always see them being used Hard to see if the pages work properly Students rarely give detailed feedback

3 Why, specifically? I am in charge of an educational website Very little feedback from users Need to write reports about quality control

4 How? Need a tried-and-tested, reliable and valid way to test the site Can’t test if it is good for learning, as there are too many other ways users could be learning English Users opinions of the user-friendliness of the site may be too nice and non-specific So, a need to watch students using the site and analyse the results systematically

5 What is Usability? Rubin (1994, p.22) says that usability testing “employs techniques to collect empirical data while observing representative end users using the product to perform representative tasks”. He describes an approach to usability testing, which “employs an iterative cycle of tests intended to expose usability deficiencies and gradually shape or mould the product in question.”

6 What did the site look like before?

7 Research Questions 1.Do the improvements to the site made during the usability study result in improvements in test participants’ success rate in completing tasks on the CILL site? 2.Do the improvements to the site made during the usability study result in participants being able to carry out these tasks in a shorter time? 3.Do the participants say that they believe that the site is easier to use?

8 Who and what? To learn if the site is usable, I needed to know what it was used for. I asked 5,100 CILL members to take part in the survey by 215 replied filled in the questionnaire at mprovement_questionnaire.htm mprovement_questionnaire.htm mprovement_questionnaire.htm The survey asked users to: –Identify the right site –Say how long they had been using the Internet –When they last used the CILL site –What did they use the site for –If they had any suggestions for improving the site

9 Survey Results 78 users correctly identified the right site Analysis of Responses to Question 7 of the Initial Questionnaire Grammar22Pronunciation12Listening9 Vocabulary9Exercises7Dictionary6 Reference Machine4Job Application Letter 4Reading3 Writing3Oral2Referencing2 Idioms2Newspapers2Academic Writing2 English in the Workplace 2Reports2Interviews2 Finding Materials2Testing Level2CILL Information2 Research1Purdue On-line Writing Lab. 1 CILL teachers ’ 1 Example Assignment1Staff information1Opening Hours1 Games1Resumes/CV1Discussion1 Exit Test1Memos1Speaking1

10 Tasks 1.Find a grammar exercise on the difference between ‘staff’ and ‘staffs’. 2.Find out how to pronounce ‘thorough’. 3.Find a page where you can learn vocabulary about the news. 4.Find a page where you can learn English by listening to pop music. 5.Find a page with some common job interview questions and advice and an exercise about how to answer them. 6.Look up the meaning of the word ‘discharge’. 7.Make a reference (Details provided) 8.Find a page which helps you to write a job application letter. 9.Find a page where you can read about pen-pals you can write to. 10.Find a page that helps you improve your newspaper reading skills.

11 Task Completion Most tasks tested whether test participants could navigate to the right page Only Task 7, on making a reference, tested whether participants could use the contents of a page Timing of participants stopped when the participant –Found the right page –Found a page that they thought was the right page –Gave up without finding the right page –Were obviously lost and in the testers opinion were unlikely to find the right page –Completed Task 7

12 Participants 2 groups: –Re-test group – did the test tasks both before and after the changes –New group – only did the tests after the changes Both groups included: –CILL students and teachers –native speakers of English and non-natives –participants with varying degrees of computer experience and skills

13 How to do a usability test 1.Welcome the participant and check their identity and profile 2.Explain the purpose and duration of the test (1 hour in total, including post-test questionnaire). 3.Elicit any questions. 4.Ask for consent for the participant to take part. 5.Ask the participant to sign the consent form. 6.Participant chooses the browser and version they usually use to access the CILL web site. 7.The participant attempts to carry out the tasks, with the tester watching and taking notes. 8.The participant either completes the final task or runs out of time. 9.Give the participant a short break. 10.Ask the participant to complete a post-test questionnaire. 11.Chat about the test and the site 12.Thank the participant for taking part. 13.Participant departs.

14 Task 1 1.Find a grammar exercise on the difference between ‘staff’ and ‘staffs’. Target page: Easy task first to give users confidence.

15 Task 1: Grammar page before Treatment

16 Task 1: Grammar Page after Treatment

17 Results of Task 1

18 Summary of Improvements Task Number Task SuccessMean Task Duration Improvement or remained at maximum Improvement in percent Improvement or remained the same Improvement in seconds Re-test group New group Re-test group New group Re-test group New group Re-test groupNew group 1. 20% %10% % %10%  40%-5% % %5%  %  -20%0% %10% 4240 Total / Mean Total = 9 Total = 9 Mean = 4% Mean = 5% Total = 10 Total = 9 Mean = 50 Mean = 30

19 Results – Research Question 1 1.Do the improvements to the site made during the usability study result in improvements in test participants’ success rate in completing tasks on the CILL site? Re-test Group:+4% New Group: +5% Re-test Group:+4% New Group: +5%

20 Results – Research Question 2 2. Do the improvements to the site made during the usability study result in participants being able to carry out these tasks in a shorter time? Re-test Group: 50 seconds faster New Group: 30 seconds faster

21 Results – Research Question 3 3. Do the participants say that they believe that the site is easier to use? Re-test group only: Strongly agree: 1 Agree: 1 Neither agree nor disagree: 3

22 Limitations of the Study The Small Number of Tasks in Relation to the Size of the Site (10 tasks, 234 pages) The Small Number of Design Iterations (I did 3, iterative design principles recommend repeated iterations) Debriefing Participants – it’s embarrassing to talk about why they didn’t do or see things - the worse the performance of a participant is, the more valuable their debriefing feedback and further test participation is, but the more difficult they are to get.

23 Recommendations for Further Study Further study could be done into: using more authentic tasks such as a text correction task, with the answers on the site investigating the effects of testers and participants reading the task instructions aloud, because some participants didn’t read all the instructions the effects of participants’ levels of computing skills on their level of task success and task duration the optimum number of tests to run between site design iterations

24 Usability Studies Discount Usability Testing Nielsen, J. (2000). Designing web usability: The practice of usability. Indianapolis: New Riders. Nielsen, J. (1994). Guerilla HCI: Using discount usability engineering to penetrate the intimidation barrier. In Bias, R.G. & Mayhew, D. (Eds.) Cost-justifying usability. Boston: Academic Press. Nielsen, J. (1993). Usability engineering. Boston: AP Professional. Rubin, J. (1994). Handbook of Usability Testing. New York, NY. John Wiley & Sons.