1. Sandra Martinez, Ph.D. Apply Texas Team

2.  Performed Spring 2010 by Garrett Stettler as his Master’s project for UT School of Information  Garrett observed, interviewed, and videotaped 15 participants  He interviewed 5 high school counselors

3.  Getting a third party to perform a usability study is more effective than using developers because: ◦ They have less personal involvement in the website, which produces less emotional response. ◦ They are not biased by over-familiarity with the website ◦ They have different expectations than developers

4.  They evaluate a product by testing it on actual users.  They discover errors and areas for improvement by observing people using the product.  They measure efficiency, accuracy, recall, and emotional responses of test subjects.

5.  The Apply Texas Committee, in authorizing the study, charged the Apply Texas team with four goals: ◦ Fix simple errors ◦ Fix big, urgent problems ◦ Create list of usability issues for the Apply Texas committee to prioritize ◦ Create list of suggested changes for the Apply Texas committee

6.  8 postsecondary-bound Texas high school students  2 postsecondary-bound high school students from other states  2 current college students (transfer)  1 prospective graduate student  1 “non-traditional” adult returning to school  1 pilot participant

7.  1) Navigation  2) Copy process  3) Help  4) Confusion and frustration  5) Accuracy and completion

8.  Group A (“new” users) 1. Create account, input profile information 2. Start and complete a new application 3. Enter essay information 4. Enter scholarship information 5. Logout  Group B (“returning” users) 1. Retrieve forgotten password 2. View/edit a saved profile 3. View/edit a saved application 4. View/edit saved essays 5. Copy an existing application

9.  Critical concerns  Major concerns  Moderate concerns  Minor concerns

10. Description of critical concerns:  Critical data may be lost, or  The user may not be able to complete the task, or  The user may not want to continue using the application

11. Description of major concerns:  Users can accomplish the task but only with considerable frustration and/or unnecessary steps  Non-critical data may be lost  The user will have great difficulty in circumventing the problem  Users can overcome the problem only after they have been shown how to perform the task

12.  Error handling ◦ Error messages are grouped at the top of the page. ◦ Errors are not highlighted within the page itself ◦ Participants tended to skim over error messages and used a “trial-and-error” approach to correcting the error

14.  Error handling recommendations ◦ Errors should be highlighted at the top of the page and at the location in the page where the error has occurred. ◦ If possible, identify the error as the user is typing

15.  Home / My Account / Logout ◦ Navigating to the main account page was confusing ◦ Attempting to log out could be confusing

16. Current view of top toolbar Currently links to logout page Currently links to application listing page Currently links to help page

17. For new cycle, links to logout page For new cycle, links to application listing page Links to application listing page (no change) For new cycle, name changed to Logoff from Home/Logoff Changes for 2011-2012 cycle

18. Description of moderate concerns:  The user will be able to complete the task in most cases but will undertake some moderate effort in getting around the problem  The user may need to investigate several links or pathways through the system to determine which option will allow them to accomplish the intended task  Users will most likely remember how to perform the task on subsequent encounters with the system

19.  Helper text and other indicators ◦ Every participant failed to notice help text at some point during the test. Helper Text

20. Color of helper text changed from gray to blue

21.  Buttons or important information appear ‘below the fold’  Some participants had difficulty completing a task if the continue button required scrolling

22. Continue button off the screen

23. For new cycle, continue button moved to top of page

24.  Confusing terminology on essay page  Participants were unsure about clicking ‘Save’ if they had not entered an essay.  Some participants were unsure about the difference between save and submit.

25. Use of the term “save” before anything is entered

26. Changed to Enter Topic A

27. Changed to Edit after first save

28.  Difficulty using Perform High School/College Search function ◦ Over one-third of participants clicked the “perform search” button before entering any text ◦ Two participants failed to select the radio button ◦ Several participants did not initially use the search feature

30.  The Apply Texas team recommends reworking this page as follows: 1.Make the “perform high school name search” button more prominent 2.Ask applicants to enter the name of the school in a new window when the button is clicked 3.When the applicant chooses a radio button, automatically close the window and fill in information on the application

31.  Top Menu and Quick Links ◦ Some participants were confused about the Menu bar and the Quick Links menu, and how they differed ◦ Participants generally did not use the navigation bar at the top for moving between Admissions/Scholarships/Essays/Submit

33.  The Apply Texas team recommends reworking the navigation and links: ◦ Make the navigation links as a side navbar ◦ Add quick links to the top menu bar with a “mouseover” dropdown.

34. Example of a page with a side navigation bar

35. Description of minor concerns:  The concern is an irritant for the user  There is a cosmetic problem  There is a typographical error

36.  Confusing/inconsistent terminology ◦ 1) Applicants were unsure what was being asked for “Mini or short” semester

37.  Confusing/inconsistent terminology ◦ 2) Participants were unsure about the difference between ‘community and volunteer service’ and ‘extracurricular activities’ ◦ 3) Under extracurricular activities, participants were unclear what ‘activity level’ meant ◦ 4) One participant was unclear whether ‘household size’ included only children, or all persons in household.

38. For mini/short semester, the following paragraph was added for the new cycle as clarification:

39.  Constraining employment form ◦ Two participants felt the employment form would be difficult to use, especially for applicants who have recurring summer jobs.

40.  Custom Questions ◦ Many participants found the custom question page to be irritating, particularly for institutions with many custom questions. ◦ The page did not cause confusion, just user irritation as it requires scrolling down the page for each question in order to click the ‘save, acknowledge question button.’

42.  Suggestion: Schools with many custom questions may want to consider moving some of their questions to other modules instead of grouping them all on the custom question page.

43.  Yes/No/Reset Answer ◦ Several participants were confused by the Yes/No/Reset Answer response. One suggestion would be to change the button labels from Reset Answer – Not Applicable to No Answer – Not Applicable

44.  Overall, Apply Texas is a fairly usable product, with no critical concerns  Many improvements to reduce frustration could be done fairly easily  Some improvements will require reworking certain modules or processes  And now, let’s hear what the high school counselors had to say --

45.  Suggestions for improving Apply Texas: ◦ Give applicants a warning that if they select to send a paper resume, schools may not process their application until the paper resume is received. ◦ Allow more space in online resumes, or provide an upload option into Apply Texas ◦ Provide another mechanism for retrieval of lost usernames, passwords, and application id numbers. ◦ Provide an online application deadline table equivalent to the one available in the paper copy of the Apply Texas application.

46. Application deadline table

47.  Suggestions for improving the High School Counselor System ◦ Different navigation options ◦ Different documentation on what data is available ◦ The ability to separate four year and two year information

48.  The Apply Texas team would like to thank Garrett Stettler for his time, effort, and excellent results.  We would also like to thank the Apply Texas committee for authorizing this study.  We are confident that implementing his results will make the application easier and less frustrating for our applicants.

49. Questions?