Lessons Learned from Summon Usability Testing Lydia Pyburn Off-Campus Services Librarian Mitch Stepanovich Arts and Humanities Liaison University of Texas at Arlington Hello, I’m Lydia Pyburn, the Off-Campus Services Librarian at the University of Texas at Arlington. Thanks for coming to this presentation about the Lessons we learned from Summon Usability study conducted this past spring.

Special thanks Xin Wang, University of North Texas Molly Shade, University of North Texas Derek Reece, University of Texas at Arlington Gretchen Trkay, University of Texas at Arlington Fen Lu, University of Texas at Arlington Peter Zhang, University of Texas at Arlington

Background In Fall 2013, UTA Libraries launched the discovery tool Summon. A need to conduct a usability study to determine effectiveness of Summon to gather information to improve the discovery tool. The study was a collaboration with University of North Texas. To give you a little background, the library launched Summon as its new discovery tool. There was training for service desk staff and librarians, so they could better assist faculty and staff when it was rolled out in 2013. At UTA Libraries, we’re a data-driven library and we decided to conduct a usability study to test the effectiveness of Summon. Dr. Xin Wang of the University of North Texas reached out to us in a partnership to conduct the study as she is focused on the user experience. The study would be coordinated with her student Molly Shade who has previous usability testing experience. She would be working with us to conduct the usability testing, assess the finding and provide a final report.

Known item Searches Classic Catalog Summon One of the reasons leading to study were complaints of known-item searches. Faculty and students would search for an item they knew the library owned in Summon, but they would have difficulty finding. They would have more success using our Classic Catalog. This slide show a comparison searching for one of our faculty. The search was just entering her name and not selecting the field author. In the Classic Catalog, we see four of her items. This doesn’t include the video recording. In Summon, when doing a simple search, only two of her items showed up where she is the author. Summon didn’t include one where she was the author, another where she was the editor, and it didn’t include the video recording.

Developing tasks User complaints Helpdesk tickets Chat transcripts Talk to service desk staff Curiosity We knew that we wanted to have a known item search as one of our tasks, but we also wanted to be thorough and look at the data we had to develop other tasks. We looked at user complaints emailed to someone directly in the library. Helpdesk tickets regarding problems with Summon were fewer than the user complaints. Se we assumed that staff hadn’t submitted a helpdesk ticket. Then, we looked at chat transcripts to see if users were having problems searching Summon. Next, we talked with our service desk staff. Next, we had a meeting with our service desk staff to ask them any thoughts or problems that they experienced with Summon. Our meeting started off quiet. We didn’t want to lead them in a direction and let them know it was a safe environment to speak freely. Since they weren’t talking I added one of my frustrations with the group and then they opened up. With any complaint or ticket we tried to replicate the search to see the problems they were having as well as what steps they would need to take to get the right results. Lastly we developed tasks out of curiosity. We wanted to know see how students interacted with Summon. If they were aware of certain features of citing the source or how they limited their search.

Usability testing tasks Keywords Task Great Gatsby Locate physical copy of book Obesity Prevention Find peer-reviewed journal articles Microsoft Excel Find relevant materials in different formats Martin Luther King, Jr. Find a newspaper article within a date range Course Reserves Identify which editions are reserved for a course Roommate’s Book Troubleshoot when a known item doesn’t appear Of the different tasks we ranked them by their level of importance. We came up with ten tasks total but only used 6 during the testing. The tasks were scenario based. We wanted to include the known item search, so we had them find a certain copy of the Great Gatsby. Next, we had them find peer-reviewed journal articles within Summon. This tasked used different limiters of peer-reviewed. It When developing tasks also consider the time it would take to complete the task. Try some test runs with staff or student workers.

Institutional Review Board (IRB) A board to review human subject research to ensure the rights and welfare of participants. It ensures the research is ethical and protects the privacy of the participants.

IRB approval Looked at previous library IRB submissions Human Subjects Protection training Everything must be decided on submittal Overestimate than underestimate Allow time for submission Approval in order to publish and present findings, Determined approval not needed as we were testing the tool and not the students

Marketing and recruitment

marketing the study Worked with Marketing and Communications department within the library Wanted to get faculty and students Mavwire and Trailblazer Library digital signage Library website Flyers around campus

Recruitment Flyer Keep it simple Use official logo Include time and incentive Use bitly for the web address

Recruitment survey

Recruitment survey

scheduling Limits of time, molly’s schedule

scheduling testing Look at school calendar to avoid scheduling during or around midterms or finals Have a quiet space without distractions Test the software and hardware Count on about 20% of who you scheduled to test Send reminders

Scheduling Survey

Assessment

A combined approach Quantitative Qualitative Morae-behavior tracking software Surveys Think aloud interviewing during tasks Exit interviews

Moving Forward Moved chat box top of page A/B/N testing of library homepage Tutorial

Questions? llpyburn@uta.edu