1. Lessons Learned from Summon Usability Testing
Lydia Pyburn
Off-Campus Services Librarian
Mitch Stepanovich
Arts and Humanities Liaison
University of Texas at Arlington
Hello, I’m Lydia Pyburn, the Off-Campus Services Librarian at the University of Texas at Arlington. Thanks for coming to this presentation about the Lessons we learned from Summon Usability study conducted this past spring.

2. Special thanks Xin Wang, University of North Texas
Molly Shade, University of North Texas
Derek Reece, University of Texas at Arlington
Gretchen Trkay, University of Texas at Arlington
Fen Lu, University of Texas at Arlington
Peter Zhang, University of Texas at Arlington

3. Background
In Fall 2013, UTA Libraries launched the discovery tool Summon.
A need to conduct a usability study to determine effectiveness of Summon to gather information to improve the discovery tool.
The study was a collaboration with University of North Texas.
To give you a little background, the library launched Summon as its new discovery tool. There was training for service desk staff and librarians, so they could better assist faculty and staff when it was rolled out in At UTA Libraries, we’re a data-driven library and we decided to conduct a usability study to test the effectiveness of Summon. Dr. Xin Wang of the University of North Texas reached out to us in a partnership to conduct the study as she is focused on the user experience. The study would be coordinated with her student Molly Shade who has previous usability testing experience. She would be working with us to conduct the usability testing, assess the finding and provide a final report.

4. Known item Searches Classic Catalog Summon
One of the reasons leading to study were complaints of known-item searches. Faculty and students would search for an item they knew the library owned in Summon, but they would have difficulty finding. They would have more success using our Classic Catalog. This slide show a comparison searching for one of our faculty. The search was just entering her name and not selecting the field author. In the Classic Catalog, we see four of her items. This doesn’t include the video recording. In Summon, when doing a simple search, only two of her items showed up where she is the author. Summon didn’t include one where she was the author, another where she was the editor, and it didn’t include the video recording.

5. Developing tasks User complaints Helpdesk tickets Chat transcripts
Talk to service desk staff
Curiosity
We knew that we wanted to have a known item search as one of our tasks, but we also wanted to be thorough and look at the data we had to develop other tasks. We looked at user complaints ed to someone directly in the library. Helpdesk tickets regarding problems with Summon were fewer than the user complaints. Se we assumed that staff hadn’t submitted a helpdesk ticket. Then, we looked at chat transcripts to see if users were having problems searching Summon. Next, we talked with our service desk staff. Next, we had a meeting with our service desk staff to ask them any thoughts or problems that they experienced with Summon. Our meeting started off quiet. We didn’t want to lead them in a direction and let them know it was a safe environment to speak freely. Since they weren’t talking I added one of my frustrations with the group and then they opened up. With any complaint or ticket we tried to replicate the search to see the problems they were having as well as what steps they would need to take to get the right results. Lastly we developed tasks out of curiosity. We wanted to know see how students interacted with Summon. If they were aware of certain features of citing the source or how they limited their search.

6. Usability testing tasks
Keywords
Task
Great Gatsby
Locate physical copy of book
Obesity Prevention
Find peer-reviewed journal articles
Microsoft Excel
Find relevant materials in different formats
Martin Luther King, Jr.
Find a newspaper article within a date range
Course Reserves
Identify which editions are reserved for a course
Roommate’s Book
Troubleshoot when a known item doesn’t appear
Of the different tasks we ranked them by their level of importance. We came up with ten tasks total but only used 6 during the testing. The tasks were scenario based. We wanted to include the known item search, so we had them find a certain copy of the Great Gatsby. Next, we had them find peer-reviewed journal articles within Summon. This tasked used different limiters of peer-reviewed. It When developing tasks also consider the time it would take to complete the task. Try some test runs with staff or student workers.

7. Institutional Review Board (IRB)
A board to review human subject research to ensure the rights and welfare of participants. It ensures the research is ethical and protects the privacy of the participants.

8. IRB approval Looked at previous library IRB submissions
Human Subjects Protection training
Everything must be decided on submittal
Overestimate than underestimate
Allow time for submission
Approval in order to publish and present findings,
Determined approval not needed as we were testing the tool and not the students

9. Marketing and recruitment

10. marketing the study
Worked with Marketing and Communications department within the library
Wanted to get faculty and students
Mavwire and Trailblazer
Library digital signage
Library website
Flyers around campus

11. Recruitment Flyer Keep it simple Use official logo
Include time and incentive
Use bitly for the web address

12. Recruitment survey

13. Recruitment survey

14. scheduling
Limits of time, molly’s schedule

15. scheduling testing
Look at school calendar to avoid scheduling during or around midterms or finals
Have a quiet space without distractions
Test the software and hardware
Count on about 20% of who you scheduled to test
Send reminders

16. Scheduling Survey

17. Assessment

18. A combined approach Quantitative Qualitative
Morae-behavior tracking software
Surveys
Think aloud interviewing during tasks
Exit interviews

19. Moving Forward Moved chat box top of page
A/B/N testing of library homepage
Tutorial

20. Questions?