1. Newspaper in Education Web Site (NEWS) Usability Evaluation Conducted by Terry Vaughn School of Information The University of Texas at Austin November 29, 2004

2. NEWS Usability Evaluation What we’ll talk about today Newspaper in Education Program Overview Test Description Participant Background First Impressions Interview Task Performance Problems and Their Sources Final Reactions Conclusions Recommendations Questions from the audience

3. NEWS Usability Evaluation Newspaper in Education Program Overview NIE Program of the Orlando Sentinel (major newspaper) Promotes literacy in the central Florida area Supplies newspapers and teaching materials to area schools NEWS Public can learn more about NIE programs Register for NIE programs Purchase newspapers and other classroom materials http://extra.orlandosentinel.com/nie/

4. NEWS Usability Evaluation Test Description Research Questions Goals Participants Environment and Equipment Test Procedure Data Collection

5. NEWS Usability Evaluation Participant Background Data 4:1 female to male 30 year median age All college educated 10 - 21 years of computer use 8 - 10 years of Web use All used the Web in their work All claimed the Web was important - very important part of their work

6. NEWS Usability Evaluation Participants’ First Impressions Positive “Visually appealing” Appropriate colors and graphic Grouping of left-side navigation Clear audience Negative Headline looks like a run-on sentence Multiple layers of navigation on top may confuse

7. NEWS Usability Evaluation Task Set Tasks 1 - 5 Finding information Example: You are interested in taking a workshop to learn how to incorporate the news into your curriculum. How many participants must you have for a Teacher Workshop? Tasks 6 - 10 Filling out forms Example: You decide to order a newspaper subscription for your classroom starting after the Christmas break (1/4), suspending service over Spring Break (3/14 – 3/18), and ending on the last day of school (5/25). Find the page where you can order a subscription and fill out the form.

8. NEWS Usability Evaluation Task Performance (ranked according to difficulty) Most problematic tasks: Task 6. Find the page where you can order a classroom subscription and fill out the form. Task 2. What is the minimum order of newspapers per classroom? Task 1. In addition to daily newspapers, what else will you receive with your classroom subscription?

9. NEWS Usability Evaluation Most Problematic Tasks Task 6. Find the page where you can order a classroom subscription and fill out the form. Errors: invalid, incorrect, and omitted data Sources: vague labeling and directions; customers had to manually calculate data; unintelligible server error page Quotes: "I don't get this at all.” "I'm supposed to calculate all this myself?” Classroom Subscription Order Form

10. NEWS Usability Evaluation Most Problematic Tasks Task 2. What is the minimum order of newspapers per classroom? Errors: navigation - clicked wrong page Sources: vague labeling; information disorganization - relevant information is located on the FAQ Quotes: “An FAQ is my last resort.” Newspaper in Education FAQ

11. NEWS Usability Evaluation Most Problematic Tasks Task 1. In addition to daily newspapers, what else will you receive with your classroom subscription? Errors: navigation - clicked wrong page; wrong answer Sources: poor information organization Quotes: "I have no idea where to go.” "How do I begin?” Teacher Home Page

12. NEWS Usability Evaluation Errors Ranked According to Criticality Identifying Problems Severity Scale 1 - Minor inconvenience, easy to circumvent 2 3 4 - Major problem prevents user from completing task

13. NEWS Usability Evaluation Identifying Problems Sources of Errors

14. NEWS Usability Evaluation Final Reactions Overall grade: C+ Most appreciated: visual design Least appreciated: information disorganization “Will you return to the site?” Yes - 2 Maybe - 2 No - 1 “Would you recommend the site to a colleague?” Yes - 2 Maybe - 1 No - 2

15. NEWS Usability Evaluation Conclusions Q: Is there enough information for customers to make informed decisions about the purchase of NEWS content and services? A: Yes, but it is difficult to gather because relevant information is not adequately grouped and labeled. Q: Can customers navigate the site and find the information they need? A: Yes, but sometimes customers get disoriented because of the poor labeling and information organization. Q: Can customers find the information in an acceptable time frame? A: Yes, but not without difficulty and/or assistance.

16. NEWS Usability Evaluation Conclusions Q: What problems do customers encounter performing tasks on NEWS? A: Customers mainly encounter problems completing the order forms. Q: Can customers complete the online subscription forms. If so, how many? A: None without assistance. Q: How do users perceive NEWS? A: Due to the flaws in the site and the problems participants encountered, they we’re left questioning the professionalism of the NIE program. Since the forms were so difficult, participants questioned NIE’s commitment to customer service.

17. NEWS Usability Evaluation Recommendations Labeling plain English - no jargon consistent descriptive Information Organization group relevant information according to service offering don’t scatter related information around Hyperlinks change the Contact link to direct to a web page no “click here”

18. NEWS Usability Evaluation Recommendations Form Design improve layout - spacing and grouping clear, descriptive labeling of text boxes don’t make customers do work that a computer could to e.g. calculating data break Classroom Subscription Form into two parts, with an interactive calendar to help customers with start/stop dates Writing style clear and concise employ a copy editor Feedback order form confirmation

19. NEWS Usability Evaluation Questions Thank you!